Paper: learning gains from frequent computerized exams

In this recently published paper, Jason Morphew and co-authors showed that frequent short PrairieLearn exams in the CBTF (Computer-Based Testing Facility) substantially increased student learning in TAM 251 ("Introductory Solid Mechanics") at the University of Illinois. Most impressively, these learning gains were on an identical pen-and-paper final exam and there were improvements at both high and low grade levels.

How did TAM 251 use the CBTF? In previous semesters the course used a traditional exam schedule with two pen-and-paper midterm exams. Mariana Silva converted it to a frequent exam schedule with seven short CBTF exams and seven optional second-chance retry exams (see Figure 1). The new exams were all implemented in PrairieLearn and allowed students to re-attempt questions for partial credit, with a mastery grading scheme that only awarded points for correct answers.

Figure 1: Traditional versus frequent exam schedule in TAM 251.

How was learning measured? Two semesters were compared, one using the traditional exam schedule and the other using frequent exams. Both semesters finished with an identical pen-and-paper final exam that consisted of 13 multiple-choice questions and three free-response questions in the given–find–solution format. In each semester all students took the exam at the same time in a proctored facility and the exam papers were kept secret and never released to students. The exam was hand-graded using an identical detailed rubric in each semester. The statistical analysis controlled for the ability of the incoming student population using grades in the prerequisite course.

How much did student learning improve? Students using the frequent exam schedule scored 7% higher on average (about 2/3 of a letter grade) on the final exam. This resulted in twice the number of A grades on the final and a drop of 60% in D and F grades (see Figure 2). The improvement was actually higher on the free-response questions (+11.3 percentage points) than the multiple-choice questions (+3.5 points), despite the fact that students had not practiced pen-and-paper free-response questions before in this class. There were over 200 students in each of the traditional- and frequent-exam semesters so all the results are statistically significant. As detailed in the paper, women and minority students had the same increases as the rest of the population.

Figure 2: Histogram of student results on the identical pen-and-paper final exam for the two semesters. Exam letter grades correspond to the following exam score ranges: A (90% or higher), B (80% to 90%), C (70% to 80%), and D/F (below 70%). Error bars show 95% confidence intervals.

What's the theory behind these results? Overall, it seems likely that students both study more and use more effective study strategies when given frequent tests, for at least three reasons. First, having many small exams essentially forces students to use distributed practice rather than cramming. Second, testing is simply more effective at solidifying memory than just studying (the testing effect in psychology). Third, second-chance exams and mastery grading encourage students to learn from their errors on the first-chance exams, rather than simply dismissing the results as a fluke or the result of an unfair test, thereby improving metacognition.