1. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Scoping Review of MOOC
Research & Research Trends
Dr. Mairéad Nic Giolla Mhichíl
National Institute of Digital Learning
Dublin City University, Ireland
@1mngm

2. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Format
• Context
• Overview of Scoping review & Methodology
• Knowledge Representation & Narrative Analysis
– Three Studies
• Insights and Conclusions

3. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Aras Bozkurt, Nilgun Ozdamar Keskin & Inge de Waard (2016) Research Trends in
Massive Open Online Course (MOOC) Theses and Dissertations: Surfing the Tsunami
Wave, Open Praxis, vol. 8 issue 3, July–September 2016, pp. 203–221 (ISSN 2304-
070X)

4. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Adapted from Source: https://www.onlinecoursereport.com/state-of-the-mooc-
2016-a-year-of-massive-landscape-change-for-massive-open-online-courses/

5. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
The MOOC Story Just Got Interesting
9th December 2017

6. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Scoping Review
• Synthesis of Knowledge differs from systematic in
breadth of its focus
• Aims to provide broad review of literature
• Identify gaps and to identify key trends
• Quality assessment
– Peer-reviewed Journals and Conference Proceedings
– Underlying studies not reviewed for quality, quality
assurance applied by journals and institutional quality
control mechanisms in the case of Theses and
Dissertations

7. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Steps conducted as part of Review
• Arksey and O’Malley (2005) Scoping Review
Framework
1. Identify the research questions
2. Find the relevant studies
3. Select the studies that are relevant
4. Chart the data
5. Collate, summarize and report
6. Consult stakeholders i.e. Workshop Participants for
their insights

8. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Objective of Review
• To map MOOC research trends based on a sample
of reviews of the MOOC literature from 2015-:
– Empirical Studies
– Theses and dissertations
– Theories
– Interdisiciplinarity
• Scoping review does not assess the quality of the
underlying studies
• Quality assurance: Peer-reviewed journal and
internal institutional quality procedures

9. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Methodology
• Started with search in IRRODL
– Review of research trends and MOOCs
• Snowballing sampling technique used

10. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
MOOC Research Trends 1 – pre 2015
Author, Year Title & Notes
Liyanagunawardena, T., Adams, A. &
Williams, S. (2013)
A systematic study of the published
literature 2008-2012, IRRODL – Review of
45 articles
Gasevic, D., Kovanovic, V., Joksimovic, S. &
Siemens, G. (2014)
Where is research on massive open online
courses headed? A data analysis of the
MOOC Research Initiative, IRRODL, -
Review of 374 research proposals
submitted to the MRI funded by the Gates
Foundation, administered by Athabasca
University
Ebben, M. & Murphy, J.S. (2014) Unpacking MOOC Scholarly Discourse: A
review of nascent scholarship, Learning
Media and Technology, nine academic
databases interrogated , chronological
framework
Adapted from Source: Bozkurt et al. (2016)

11. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
MOOC Research Trends 2 – pre 2015
Author, Year Title & Notes
Sa’adon, N., Alias, R.A., & Ohshima, N.
(2014)
Nascent Research Trends in MOOCs in
higher educational institutions: a
systematic literature review: Conference
IEEE, 164 Paper data mined , 2008-2014
Kennedy, J. (2014) Characteristics of Massive Open Online
Courses (MOOCs): A research review,
2009-2012, Journal of Interactive Online
Learning – analysis of six peer-reviewed
articles
Adapted from Source: Bozkurt et al. (2016)

12. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 1: Theses (MA) and Dissertations (Doctoral)
• Aras Bozkurt, Nilgun Ozdamar Keskin & Inge de Waard (2016)
Research Trends in Massive Open Online Course (MOOC) Theses
and Dissertations: Surfing the Tsunami Wave, Open Praxis, vol. 8
issue 3, July–September 2016, pp. 203–221 (ISSN 2304-070X)
• Study details
– Sampled published dissertations 2008-2015
– Only those written in English – n=51 (45% MAs, 55% Doctoral)
– Search Criteria: MOOC, Massive Open Online Courses, cMOOC,
xMOOC
– Academic databases – ProQuest, WorldCat, Open Access
Theses and Dissertations, Google Scholar etc.

13. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 1 Findings – Disciplinary Areas
Disciplinary Areas %
Education 51
Computer Science & Engineering 19
Information & Communication Sciences 12
Business & Economics 8
Social Sciences 6
Media Disciplines 4

14. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 1 Findings – Research Methodologies
• Qualitative 49%, Quantitative 21%, Mixed
18%, Review 8 %, Other, 4
Method Frequency
Qual: Case Study 16
Qual: Content Analysis 2
Qual: Best practices 1
Qual: Delphi 1
Quan: Correlational 7
Mixed: Explanatory Sequential Analysis 3
Review: Literature Review 4

15. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study Findings 1 – Theoretical Lenses
• Notably 25 out of the 51 did not employ a
theoretical lens – online learning
communities, social learning, distance
education
Lens Frequency
NA 25
Community of Inquiry 4
Diffusion of Innovations 4
Self-determination theory 2
Social Learning 2
Activity Theory 1
Interaction Equivalency 1

16. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 1: Insights and Conclusions
• Papers directed at xMOOCs in contrast with
previous reviews i.e. cMOOCs
• Assertion that current research directed at
MOOC learners
• More Mixed Methods studies required to
eliminate qual/quan issues
• Meta-analysis/synthesis of studies useful –
more research needed

17. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 2 - Research Methods
• Deng, R. & Benckendorff (2017) A contemporary review of
Research methods adopted to understand students’ and
instructors’ use of massive open online courses (MOOCs),
International Journal of Information and Educational
Technology, 7(8), pp.601-607
Study Details
• Review of Four Academic Databases
• Empirical studies published in English
• Criteria – students/instructors’ experiences, including key
terms i.e. MOOCs
• 53 studies included
• Thematic Analysis

18. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 2: Background
Year of Publication Frequency
2014 14
2015 32
2016 7
Country Frequency
USA 28
UK 6
Canada 5
Australia 3
Israel 3Type Frequency
Journal 40
Thesis/Dissertation 9
Conference 3
Book chapter 1
• Computers and Education
• BJET
• IRRODL
• Distance Education
• Educational Media International

19. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 2: Perspectives & Method
• Perspective of Student Dominant: 48 articles
• Perspective of Instructor: 4 articles
• Perspective of both: 1 article
• Single Method (36)
– Survey
– Interviews
– Logfiles
• Two Methods (15)
• Three+ Methods (2)

20. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 2: Methods
• Survey
– Pre-delivery: course participants’ demographics,
motivation for enrolment & intended study
behaviour, pre-course attitudes
– During-delivery: Patterns of engagement -
behavioural engagement
– Post-delivery: Post-course attitudes, self-reported
engagement, perceived learning outcomes

21. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 2: Methods
• Survey
– Most cross-sectional i.e. data collected at a single
point of time
– Limited linking of pre-entry and exit-surveys when
completed within studies

22. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 2: Methods
• Interviews
– Majority of interviewees – participants
– No study interviewed both instructors and participants
– Number of interviewees range 4-60
– Interviewed by end of MOOC
– Topics Participants: Reflection on experience within
MOOC, motivation, perceived benefits and gains,
engagement, perceptions of everyday learning experiences
– Topics Instructors: Motivation to develop, attitudes to use
of MOOC
– Used to triangulate with survey data: motivation,
engagement, perceptions of learning experiences

23. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 2: Methods
• Log-Files – record of user platform activity
and interaction
• Used as a single method or in combination
• Patterns of social, emotional, and behavioural
engagement
• Social: frequency of engagement and
interaction - measures: initiated forum posts,
discussion participation

24. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 2: Methods
• Emotional: Affective expressions such as likes
etc. in forums
• Behavioural: Clickstream logs, completion of
activities and participation in discussion forum
• Descriptive statistics
• Small number of correlational studies
– Student performance and prior knowledge

25. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 2: Insights and Conclusions
• Ethical Issues not addressed in review literature
• Research Design Simple – Single snap shots, little
methodological innovation
• Longitudinal, comparisons and observations over
time
• Linking of Pre & Test Results - Correlation and/or
Causality
• Holistic Analysis of learning – currently skewed
towards participants

26. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 3: Interdisciplinary MOOC Research
• Veletsianos, G. & Shepherdson, P. (2016) Who Studies
MOOCs? Interdisciplinarity in MOOC Research and its
Changes over Time, IRRODL, 16 (3), pp. 1-17
• Study Details
• Creation of MOOC literature corpus 183 papers: key
words
• Peer-Reviewed Journals, Date-defined, Empirical, In
English
• Databases and search engines: Google scholar, Scopus
etc.
• Corpus statistically compared with two other corpra
(2015, 2014)

27. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 3: Findings
Discipline Frequency
Education 32.8%
Computer Science 23.1%
Engineering 6%
Industry 6.9%
Social Sciences 7.5%
• Study’s corpus less dominated by field of education as opposed
to (2014) corpus
• MOOC research appears to be more interdisiciplinary

28. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 3: Conclusions and Insights
• Corpus included more papers from Computer
Science than previous corpora – quicker
publication routes
• Research design to be expanded to move
beyond – cross-citation analysis to analysis of
integration of knowledge from the various
fields in the design, analysis and reporting

29. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 3: Conclusions and Insights
• MOOC Research Phases
– Phase 1 cMOOCs – Education research dominated
– Phase 2 xMOOCs – Structure reflecting traditional
modes of learning i.e. linear, hierarchical paths?
• Interdisciplinarity at the article level
• Comparison between
– Corpus based on empirical papers with corpus
created on both theoretical and empirical papers

30. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 4: Literature Gaps
• Veletsianos, G., & Shepherdson, P. (2016). A
systematic analysis and synthesis of the
empirical MOOC literature published in 2013–
2015. IRRODL, 17(2), 198-221.
Study Details
• Corpus 183 Papers – Peer-reviewed, in English,
MOOCs, published in time-frame
• Review of databases

31. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 4: Findings

32. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 4: Findings

33. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 5: Findings

34. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 4: Findings Methods
• One method most employed
• Majority using Trace Data,
• Surveys and then Interviews

35. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 4: Findings

36. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 4: Findings
• Student-Related Papers 83.6%
• Learner-behaviours, learner-participation
and interaction
• Learner perceptions, learner experiences
motivation and demographics
• Completion and retention
• Learner sub-populations

37. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Study 4: Insights and Conclusions
• A small number of papers are widely cited
• But most papers are not
• Dominance of positivist approaches and analysis
less interpretative or qualitative approaches i.e.
interviews, focus groups or observations.
• Dearth of instructor-focused research
• Geo-graphical origin concentrated to western
world and China (notably English speaking
regions – English only papers)

38. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Final Thoughts
• Systematic Content Analysis of MOOC
literature needed to investigate and categorise
research trends
• Predominace of research on xMOOCs?
– Related to emergence of trends relating to
monetisation and business models
• Research Design
• English only reviews

39. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Some Studies not Included in This
Review
• Sangrà, A., González-Sanmamed, M., &
Anderson, T. (2015). Meta-analysis of the
research about MOOCs during 2013-2014.
Educación XX1, 1-28.
• Raffaghelli, J., Cucchiara, S., & Persico, D.
(2015), Methodological approaches in MOOC
research: Retracing the myth of Proteus,
British Journal of Educational Technologies,
46(3), 488–509.

40. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Key References 1
• Arksey, H.& O’Malley, L. (2005) Scoping Studies:
Towards a methodological framework.
International Journal of Social Research
Methodology: Theory & Practice 8(1), pp.19-32
• Aras Bozkurt, Nilgun Ozdamar Keskin & Inge de
Waard (2016) Research Trends in Massive Open
Online Course (MOOC) Theses and Dissertations:
Surfing the Tsunami Wave, Open Praxis, vol. 8
issue 3, July–September 2016, pp. 203–221 (ISSN
2304-070X)

41. Training Madrid, 12-16 December 2016
Designing sustainable MOOCs in Europe
Key References 2
• Deng, R. & Benckendorff (2017) A contemporary
review of Research methods adopted to
understand students’ and instructors’ use of
massive open online courses (MOOCs),
International Journal of Information and
Educational Technology, 7(8), pp.601-607
• Veletsianos, G. & Shepherdson, P. (2016) Who
Studies MOOCs? Interdisciplinarity in MOOC
Research and its Changes over Time, IRRODL, 16
(3), pp. 1-17