How do you measure success? Many researchers, when evaluating Massive Open Online Courses (MOOCs), tend to measure the success of a course with a single metric, i.e. the completion rate. However, MOOCs generally have a large audience, and given the diverse background of MOOC learners, every learner differs in goals and level of engagement with the course materials. It may not be comprehensive enough to evaluate learner success by simple looking at whether he/she has finished the course; and the overall course completion rate alone may not suffice in evaluating the success of a MOOC.
This is what we learnt from a seminar entitled “Demystifying Learner Success: Before, During, and After a Massive Open Online Course”, delivered by Dr Elle Yuan Wang, a Research Scientist at EdPlus Action Lab, Arizona State University on August 1, 2017. Organized by the Faculty of Education, this seminar offered us a different perspective on how to measure success of a MOOC, an ongoing debate among researchers.
Learner Success in terms of Post-Course Career Development
Dr Wang believes that learner success can take many different forms – traditional assessment scores, or in other forms of post-course development, such as career development. In her 2014 study, she measured the post-course development of a group of MOOC learners two years after the end of the course using two metrics: whether learners (i) joined a relevant professional society; and/or (ii) submitted a paper in a relevant conference. By comparing learners’ post-course career development and in-course performance, she set out to investigate the relation between the two. The ultimate goal is to find out how career advancers differ from other learners in terms of their in-course performance.
The research targeted learners of the first iteration of The Big Data in Education MOOC, a postgraduate-level 8-week course offered on Coursera in 2013. Dr Wang was one of the teaching assistants of the course. (The subsequent iterations of the course have been offered on edX.)
The study revealed that:
- Career advancers who joined a professional society or submitted a paper earned better scores and were more likely to complete the course than non-advancers.
- Career advancers also demonstrated more frequent engagement with course components including course pages, lecture videos, assignment submissions, and discussion forums. For example, the page viewing activities of people who joined a professional society were much higher than non-members.
- However, even though career advancers tended to have more post-reading actions, they were not significantly more likely to post, comment, or vote than their peers.
Significance of the Research
This study enriched our understanding of how MOOCs potentially impact learners’ career development and the possible association between student behaviors and positive developments. All these findings are crucial for educators in developing and improving their MOOCs in the future.
A Special Note of Thanks
Hereby we would like to thank Dr Wang for not only sharing with us her research endeavours and findings, but also inspiring our work in learning analytics in HKU. Our colleague, Dr Leon Lei, completed The Big Data in Education MOOC on Coursera in 2013 and is now applying principles learnt from the course in developing our own MOOCs in HKU. Thank you, Dr Wang, for inspiring us. We look forward to more opportunities to further explore learning analytics and educational data mining with fellow researchers in the future.
*Note: Dr Wang’s research was conducted in collaboration with Ryan Baker, University Pennsylvania, and Luc Paquette, University of Illinois at Urbana-Champaign. For further details, please refer to the original research paper: Wang, Y., Paquette, L., Baker, R. (2014). A longitudinal study on learner career advancement in MOOCs. Journal of Learning Analytics, 1(3), 203-206. [PDF]
