As Data Floods In, Massive Open Online Courses Evolve

In 2012, education startups attracted millions of students—and a surge of interest from universities and the media—by offering massive open online courses, or MOOCs. Now some core features of these wildly popular courses are being dissected, enabling the course providers to do some learning of their own. As these companies analyze user data and experiment with different features, they are exploring how to customize students’ learning experiences, and they are amassing a stock of pedagogical tricks to help more students finish their courses.

“The data we are collecting is unprecedented in education,” says Andrew Ng, a cofounder of MOOC provider Coursera and an associate professor at Stanford University. “We see every mouse click and keystroke. We know if a user clicks one answer and then selects another, or fast-forwards through part of a video.”

Ng and other major figures from the MOOC world have long foreseen that MOOCs would provide a wealth of data about how students actually learn. However, until recently these small companies have been too preoccupied with scaling up their infrastructure in order to meet exploding demand (see “The Technology of Massive Open Online Courses”) to investigate this data in depth.

Some recent findings have vindicated aspects of MOOCs’  design. Princeton researchers used data from Coursera to show that the company’s system of peer grading, which calculates grades for coursework based on feedback provided by other students, is effective. Other findings have challenged assumptions about how an online course can successfully cater to hundreds of thousands of students or more.

Since MOOCs first appeared, bite-sized videos have provided the bulk of the teaching, accompanied by online assessments and exercises to help cement the content in students’ minds. However, both Coursera’s data and Udacity’s reveal a large subset of students who prefer to skip videos and fast-forward as much as possible. “We’ve been starting to restructure our courses to have much less video, and to rerecord some videos,” says Sebastian Thrun, a Stanford robotics professor, a vice president at Google, and cofounder and CEO of Udacity. “Our popular courses are really changing a lot based on our data.”

Much of the performance research is motivated by a desire to increase course completion rates, which hover around 10 percent, according to most MOOC providers and figures from academics who have taught using the courses. A Udacity research project recently suggested that technical challenges might be to blame for a significant fraction of dropouts. In the experiment, some Udacity users were invited to text-chat with an “automated” help system that actually used live human operators, and many users mentioned problems related to computer literacy.

“The way that people get stuck is very different to what we expect,” says Thrun. “Some students simply can’t operate a keyboard or a website. It shows that the basic one-size-fits-all MOOC is inadequate to address the retention problem.” Udacity is now working on analysis techniques that could sort students on the basis of their behavior and offer targeted assistance, or adjust courses to better serve them.

Tailoring MOOCs is an idea with merit, says Chris Piech, a Stanford PhD student researching online learning. In a recent study, Piech and two colleagues examined three of Stanford’s computer-science MOOCs and found that dropouts fell into three distinct groups: “auditors,” who had no intention of completing the course but who used it as a resource, like a book; students who participated in the course but gradually fell behind; and those who sporadically “sampled” throughout the course.

Many in the latter two groups would probably complete a course if given the right assistance, says Piech, and data collected in the study suggests that encouraging students to interact with one another via forums or other social features would do it.

Piech anticipates a flood of published and internal research from MOOC providers reporting significant advances in online learning effectiveness. “As the MOOC platforms become more robust and their architecture is worked out, research is going to become more of a priority and more useful,” he says. “They provide the large numbers to answer the hard questions about education.”

Some of the analyses taking place at MOOC companies appear to be answering more-modest questions. “A/B testing,” a methodology common at Internet companies, is being used to try out small design tweaks that might nudge students to do better. A/B testing shows different versions of a service to different segments of a site’s audience to see how they react.

Through A/B testing, says Ng, Coursera recently found that its practice of e-mailing people to remind them of upcoming course deadlines actually made students less likely to continue with the courses. But sending e-mails summarizing students’ recent activity on the site boosted engagement by “several percentage points.” One A/B test by Udacity pitted a colorized version of a lesson against a black-and-white version. “Test results were much better for the black-and-white version,” says Thrun. “That surprised me.”

It’s unclear whether the laundry lists of refinements that result from A/B testing will add up to a grand theory of learning and teaching that challenges tradition. Ng says he doesn’t think a grand theory is needed for MOOCs to succeed. “I read Piaget and Montessori, and they both seem compelling, but educators generally have no way to choose what really works,” he says. “Today, education is an anecdotal science, but I think we can turn education into a data-driven science, where you do what you know works.”