Abstract

In October 2014, one-time MOOC developer Udacity completed its transition from primarily producing massive, open online courses to producing job-focused, project-based microcredentials called “Nanodegree” programs. With this transition came a challenge: whereas MOOCs focus on automated assessment and peer-to-peer grading, project-based microcredentials would only be feasible with expert evaluation. With dreams of enrolling tens of thousands of students at a time, the major obstacle became project evaluation. To address this, Udacity developed a system for hiring external experts as project reviewers. A year later, this system has supported project evaluation on a massive scale: 61,000 projects have been evaluated in 12 months, with 50% evaluated within 2.5 hours (and 88% within 24 hours) of submission. More importantly, students rate the feedback they receive very highly at 4.8/5.0. In this paper, we discuss the structure of the project review system, including the nature of the projects, the structure of the feedback, and the data described above.

Author’s Note: The numbers in this abstract and paper are accurate as of October 2015. As of April 2016, the number of projects evaluated per day has increased to 450, the average turnaround has decreased to 87 minutes, and the average student satisfaction has remained at 4.8/5.0.

Full Paper

The full paper “Expert Evaluation of 300 Projects per Day” can be found here.