Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms


Large classes, both online and residential, typically demand many graders for evaluating students’ written work. Some classes attempt to use autograding or peer grading, but these both present challenges to assigning grades at for-credit institutions, such as the difficulty of autograding to evaluate free-response answers and the lack of expert oversight in peer grading. In a large, online class at Georgia Tech in Summer 2015, we experimented with a new approach to grading: framing graders as meta-reviewers, charged with evaluating the original work in the context of peer reviews. To evaluate this approach, we conducted a pair of controlled experiments and a handful of qualitative analyses. We found that having access to peer reviews improves the perceived quality of feedback provided by graders without decreasing the graders’ efficiency and with only a small influence on the grades assigned.

Associated People

Associated Projects

Associated Media

Associated News

  • 25 Apr 2016: David, Dana, and Chaohua present at Learning @ Scale 2016. David Joyner, Dana Sheahen, and Chaohua Ou are in Edinburgh, Scotland presenting our five papers. All five presentations are part of the conference's flipped sessions, and the pre-recorded presentations are available here.
  • 15 Apr 2016: Learning @ Scale Flipped Presentations Now Available. For our five Learning @ Scale papers, we are participating in a flipped session. This entails filming the presentation in advance. The presentations are open to the public and now available.
  • 14 Dec 2015: Paper on "Graders as Meta-Reviewers" Accepted to Learning @ Scale 2016. "Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms" has been accepted to Learning @ Scale 2016. The paper is co-authored by David Joyner, Wade Ashby, Liam Irish, Yeeling Lam, Jacob Langston, Isabel Lupiani, Mike Lustig, Paige Pettoruto, Dana Sheahen, Angela Smiley, Amy Bruckman, and Ashok Goel. Congratulations, everyone!