Dilrukshi Gamage's presentation of her PhD research at the 7th GO-GN Seminar in Delft, April 21-22, 2018.

1. Increasing effectiveness of MOOCs to meet
the 21st century skills
Dilrukshi Gamage
Sri Lanka
dilrukshi.gamage@gmail.com
www.dilrukshigamage.com
VII GO-GN Seminar

2. Increasing effectiveness of MOOCs to meet the 21st century skills

3. Agenda
• 1. Introduction
• 2. Background
• 3. Problem Domain
• 4. Research question and Methods
• 5. Discussion
• 6. Conclusion
creative commons Lincence to add

4. Introduction
• Learning communities are actively engaging E-Learning for more than
10 years
• 2012, MOOCs were introduced and became the trend
• MOOCs found to be following a sound pedagogy
• Some researchers are skeptical

5. J. Young, "Udacity Official Declares MOOCs ‘Dead’ (Though the
Company Still Offers Them)," EdSurge, 12 October 2017.
Content
orient
Instructivist
2012 2018
Passive
Pedagogical
skeptism

6. MOOCs
effectiveness
Quality
21st Century
skills
Creative thinking
Collaboration
Communication
Critical thinking

7. What and why we need?
Creative thinking
Critical thinking
Collaboration
Communication
21st century learning needs
Industries
require wicked
problem solvers
MOOCs are
content focus

8. Introduction
• MOOC effectiveness – Quality
• Online learning increases with introduction of MOOC
• Research findings indicate MOOC is effective –
• Sound pedagogy (Glance, Forsey and Riley,2013)
• Shown higher gains than a regular class (Colvin, Champaign, Zhou, Fredericks
and Pritchard,2014)
• But some researchers are not convinced, state it is not effective

9. Introduction
• Broad view on the proposal –
• What factors affects the effectiveness of MOOCs?
• How can we measure the improvement of effectiveness?
• In what way MOOCs platforms can be improved or what solutions can be
implemented and integrated in order to improve MOOCs platforms using the
dimensions we proposed above in 1, 2?

10. Introduction
• Find solutions incorporating
• Grounded Theory (GT) and Quantitative studies
• Human Computer Interaction (HCI) methodologies and
• Design Based Research (DBR) methodology.
• The above is due to the perception that the problems of MOOCs need
attention with respect to the social behaviors and human interaction
• Broader question of
“How can a large group of humans at any age learn better in online
situations?”

11. Literature review
• eLearning effectiveness – eLearning Quality
• Online learning increases with introduction of MOOC
• We already stated that research findings indicate MOOC is effective –
• Sound pedagogy (Glance, Forsey and Riley,2013)
• Shown higher gains than a regular class (Colvin, Champaign, Zhou, Fredericks
and Pritchard,2014)

12. Literature review
• But some researchers are skeptical –
• No room for creativity (Martin, 2012)
• Follow traditional didactic pedagogy (Daniel,2012)
• Poor quality (Bali, 2013)
J. Young, "Udacity Official Declares MOOCs ‘Dead’ (Though the
Company Still Offers Them)," EdSurge, 12 October 2017.
Content
orient
Instructivist
2012 2018
Passive
Pedagogical
skeptism

13. Literature review
• Number of MOOCs and Courses increases – Quality issue
• Less research conducted to benchmark quality of online learning -
MOOC
• Out of 4745 search hits – only 26 research relevant to Quality
dimensions
• Out of 26 – 2 research work with empirical evidence; 7 proposal
frameworks

14. Literature review
Journal name No of results with search
terms “MOOC quality”
Relevance to
the review
Last access date
Education Specific Journals
British Journal of Education 121 2 19-2-2015
American Journal of Distance Education 8 1 19-2-2015
Journal of Online Learning and Teaching 26 2 19-2-2015
Distance education 14 1 19-2-2015
International Journal of Distance
education
12 0
Publisher Databases
Wiley Online database 89 3 21-2-2015
SpringerLink 209 4 23-2-2015
Elsevier’s ScienceDirect 155 2 23-2-2015
Research Databases
ERIC 156 2 25-2-2015
CiteSeerX 373 2 25-2-2015
Google Scholar 3509 5 25-2-2015
IEEEXplorer 73 2 25-2-2015
Total 4745 26

15. Literature review
• Empirical evidence
• 2 dimensions 6 categories (Yousef,2014)
• used 107 students and 98 professors who take part in
MOOC courses.

16. Literature review
• Proposal Frameworks
• 10 dimensional proposal to evaluate quality in MOOC - Sir John
Daniel and Stamenka Uvaliæ-Trumbiæas “A guide to online
learning” (Butcher and Wilson-Strydom,2014)
• “Quality Matters” program proposes to use its metrics rubric in
assessing MOOCs (Shattuck,2014)
• European Foundation for Quality eLearning(EFQL) (Ehlers,
Ossiannilsson and Creelman,2014)
• 7c’s approach to bring enhancements in leaner experience and
quality assurances (Conole,2013)
• Open Education Initiatives (OEI) (Nagashima,2014)
• Margaryan, Bianco & Littlejhon propose 10 dimensional
instructional design criteria to evaluate MOOCs quality
(Margaryan, Bianco & Littlejhon ,2015)
• framework named mesoMOOC proposed by Schoenak addresses
several challenges that hinder current effective delivery of
MOOCs (Schoenak,2013)

17. Literature review
• Identifying effectiveness factors for MOOC
• Used Grounded theory (GT) process (Glaser,1967)
• Why –
• GT is important for eLearning research since the data produced by GT gives
the best possible insights into the students and their experiences
• Provides flexibility to the researcher as it does not assume hypotheses
• Researchers identified the nature of GT approach to educational research is
not built upon a structured and pre-determined methodology, but rather it
represents a strategy for continually redesigning research in the light of
emergent concepts

18. • Who else used GT in educational research –
• to understand the social context of the UK online community and issues
around the creation and exchange of knowledge within and between online
communities (Cook, John & Smith,2004)
• focus groups to investigate and analyze critical success factors (CSFs) that are
required to deliver eLearning within higher education (HE) courses and
programs (McPherson and Nunes,2008)
• tried to identify the eLearning experience in students using Grounded Theory
(Gliber & Jennifar, 2007)
• a novel analysis using user generated online reviews to find the factors which
make a great MOOC (Adamopoulos,2013)
Literature review

19. Literature review
• Use of Design Based Research Method (DBR)
• DBR is an emerging paradigm
• Is used for the study of learning in context through the systematic
design/ study of instructional strategies/ tools.
• Scholars in DBR collective argue that design-based research can help
create and extend knowledge about developing, enacting, and
sustaining innovative learning environments [DBR Research Collective,
2003].

20. Literature review
• Use of DBR Method
• DBR collective (DBR Research Collective,2003) suggest value of design-based
research should be measured by its ability to improve educational
practice
• four areas where design-based research methods are most promising
(a) exploring possibilities for creating novel learning and teaching environments
(b) developing theories of learning and instruction that are contextually based
(c) advancing and consolidating design knowledge
(d) increasing our capacity for educational innovation.

21. Research Objectives
• Our perceptions
• Improving the effectiveness of Learning experience in MOOCs.
• Existing MOOC model does not support meeting the challenges faced in 21st
century.
• Existing model mostly follow the didactic lecture base learning, leaving less
room for the students to critically think and work collaboratively.
• MOOCs are not effective thus MOOCs are young itself, there are many
opportunities and methods in that can be use to find an effective balance
MOOCs effectiveness
Quality
21st Century skills
Creative thinking
Collaboration
Communication
Critical thinking

22. Research Objectives --> Research Questions
Therefore -
1. Identify what is needed by students to learn best in MOOCs, what
factors does affect the effectiveness?
2. How can we improve the effectiveness of MOOCs using the
discovered factors in 1.
3. Introduce an appropriate pedagogical model/ theoretical framework
to enhance the Learning experience.
4. Design a working porotype model on the enhanced pedagogical
model and empirically test the improvements

23. Research Objectives
• Inline with the Objective 3
• Introduce Network grouping model/ algorithm, where we can group the
students in MOOCs so that they can work effectively than following a typical
MOOC.
• Introduce effective peer reviewing model – We hypnotize an Identified Peer
Reviewing (IPR) model where students can see the peers they grade which is
opposite to the blind peer review.

24. Research Methodology
• Phase 1 - Identifying what factors affect in the effectiveness
• Phase 2 - Developing a pedagogical model to enhance the
effectiveness
• Phase 3 - Modeling a prototype and testing the improvements
• Phase 4 – Implement the prototype and integrate as a learning
intervention

25. Key gap in the pedagogical models
cMOOC follow more connectivist approach.
The lose structure in cMOOCs found to be difficult in following by novice and
typically suited for well self-motivated students.
xMOOCx behavioristic approach
It does not appeared to be providing the skills needed by students who face 21st
century challenges. xMOOCs are more and more lean towards decentralize
system limiting students to student, student to instructor interactions.
Learners are isolated in their learning environment and less motivated to commit
to course activities.

26. Missing key components of MOOCs pedagogy
Brouns,, 2014.
Teacher Learner
Interaction
Building learning
community
Miller, 2015.

27. Key gap in pedagogical model
21st centaury learner need
Collaboration Interaction
Networked
learning

28. key gaps convert to 6 design goals
• 1) Learner Empowerment
• 2) Learner orientation and structured plan
• 3) Collaborating in work with team
• 4) Social Networking
• 5) Peer assistance
• 6) Assessing and Feedback.

29. Outcomes of the research
Designing interventions
Improving MOOCs effectives
Peer
Identity
Incentive Compatible
algorithm
Peer Communication
2017 Learning@Scale
Boston
Evolution of MOOCs in 21st Century
Skills
Quality Feedback
Cluster
Orient
Focus
Network
&
Collabora
te

30. References
[1] M. Bali, "MOOC pedagogy: gleaning good practice from existing MOOCs," Journal of Online Learning and Teaching, vol. 10, no. 1, pp. 44-54, 2014.
[2] J. Young, "Udacity Official Declares MOOCs ‘Dead’ (Though the Company Still Offers Them)," EdSurge, 12 October 2017.
[3] J. Daniel, "Making sense of MOOCs: Musings in a maze of myth, paradox and possibility," Journal of interactive Media in education, vol. 3, 2012.
[4] D. Gamage, I. Perera and S. Fernando, "MOOCs to provide 21st century skills: Learner perspective," in INTED, Valencia, Spain, 2016.
[5] Á. F. Blanco, F. J. García-Peñalvo and M. Sein-Echaluce, "A methodology proposal for developing adaptive cMOOC.," in In Proceedings of the First International Conference on
Technological Ecosystem for Enhancing Multiculturality, 2013.
[6] M. Crosslin and J. Dellinger, "Lessons learned while designing and implementing a multiple pathways xMOOC+ cMOOC," in In Society for Information Technology & Teacher
Education International Conference (pp. 250-255). Association for the Advanceme, Las Vegas, NV, United States, 2015.
[7] D. Gamage, I. Perera and S. Fernando, "Factors leading to an effective MOOC from participiants perspective. In," in IEEE international Conference on Ubi-Media Computing
(UMEDIA), 2015.
[8] R. F. Kizilcec, "Collaborative learning in geographically distributed and in-person groups," AIED 2013 Workshops Proceedings, vol. 67, 2013.
[9] G. Conole, "Designing effective MOOCs," Educational Media International, vol. 54, no. 4, pp. 239-252, 2015.
[10] A. Couros, K. Hildebrandt and G. Veletsianos, "Designing for open and social learning.," in Emergence and innovation in digital learning: Foundations and applications,
Edmonton:Athabasca University Press, 2016, p. 10.
creative commons Lincence to add

31. [11] G. Siemens, "Massive open online courses: Innovation in education."," Open educational resources: Innovation, research and practice, vol. 5, pp. 5-15, 2013.
[12] Editors, "Sociak Networking Wiki," March 2011. [Online].Available: http://socialnetworking.swsi.wikispaces.net/NetworkedLearning. [Accessed October 2017].
[13] G. Sharrock, "Making sense of the MOOCs debate," Journal of Higher Education Policy and Management, vol. 37, no. 5, pp. 597-609, 2005.
[14] K. Handley, A. Sturdy, R. Fincham and T. Clark, "Within and beyond communities of practice: Making sense of learning through participation, identity and practice," Journal of management studies, vol. 43, no. 3, pp. 641-653, 2006.
[15] H. Coates, P. Kelly and R. Naylor, "Leading Online Education for Student Success," International Journal of Chinese Education, vol. 6, no. 1, pp. 105-126, 2017.
[16] D. Gamage, S. Fernando and I. Perera, "Factors leading to an effective MOOC from participiants perspective," in Ubi-Media Computing (UMEDIA) 8th IEEE International Conference, 2015.
[17] M. Bali, "MOOC pedagogy: gleaning good practice from existing MOOCs," Journal of Online Learning and Teaching, vol. 10, no. 1, p. 44, 2014.
[18] P. J. Guo, J. Kim and R. Rubin, "How video production affects student engagement: An empirical study of mooc videos," in first ACM conference on Learning@ scale conference , 2014.
[19] D. Gamage, I. Perera and S. Fernando, "MOOCs to provide 21 Centuary skills:Learners Perspective," in 10th International Technology Education and Development Conference, Valencia,Spain, 2016.
[20] S. Zheng, K. Han, M. B. Rosson and J. M. Carroll, "The role of social media in MOOCs: How to use social media to enhance student retention. In Proceedings of the Third (2016)," in Third ACM Conference on Learning@ Scale, 2016.
[21] A. Anderson, D. Huttenlocher, J. Kleinberg and J. Leskovec, "Engaging with massive online courses," in 23rd international conference on World wide web , 2014.
[22] P. Kim, Massive Open Online Courses: The MOOC Revolution, Routledge , 2014.
[23] R. E. Kraut and A. T. Fiore., "The role of founders in building online groups," in " In Proceedings of the 17th ACM conferenceComputer supported cooperative work & social computing, 2014.
[24] M. Wen, D. Yang and C. P. Rosé, "Virtual teams in massive open online courses," in International Conference on Artificial Intelligence in Education, 2015.
[25] J. W. Strijbos, R. L. Martens, W. M. Jochems and N. J. Broers, "The effect of func-tional roles on perceived group efficiency during computer-supported collaborativelearning: a matter of triangulation," Computers in Human Behavior, vol. 23, no. 1,
353-380, 2007.
[26] Q. Zhang, K. L. Peck, A. Hristova, K. W. Jablokow, V. Hoffman, E. Park and R. Y. Bayeck, "Exploring the communication preferences of MOOC learners and the value of preference-based groups: Is grouping enough?," Educational Technology Research
Development, vol. 64, no. 4, pp. 809-837, 2016.
[27] P. A. G. e. al, "Finding collaborative innovation networks through correlating performance with social network structure," International Journal of Production Research, vol. 46, no. 5, pp. 1357-1371, 2008.
[28] T. Sinha, "Together we stand, Together we fall, Together we win: Dynamic team formation in massive open online courses," in Fifth International Conference on the Applications of Digital Information and Web Technologies (ICADIWT), 2014.
[29] D. Gamage, I. Perera and S. Fernando, "Evaluating effectiveness of MOOCs using empirical tools: learners perspective," in 0th annual International Technology Education and Development Conference, Valencia, Spain, 2016.
[30] Á. Fidalgo-Blanco, M. L. Sein-Echaluce and F. J. García-Peñalvo, "Methodological Approach and Technological Framework to Break the Current Limitations of MOOC Model," Journal of Universal Computer Science, vol. 21, no. 10, pp. 712-734, 2015.
[31] A. Sfard, "On two metaphors for learning and the dangers of choosing just one," Educational Researcher, vol. 27, no. 4, pp. 4-13, 1998.
[32] K. Swan, S. Day, Bogle and V. Prooyen, "AMP: A tool for categorizing the pedagogical approaches of MOOCs.," in (Eds) MOOCs and Open Education Around the World, New York, Routledge, 2015, pp. 105-118.
[33] K. Swan, S. Day and L. Bogle, "Metaphors for Learning and MOOC Pedagogies," in Proceedings of the Third (2016) ACM Conference on Learning@ Scale , 2016.
[34] H. M. Mohamed and M. Hammond, "MOOCs: A Differentiation by Pedagogy, Content and Assessment," The International Journal of Information and Learning Technology , 2017.
[35] S. L. Miller, "Teaching an Online Pedagogy MOOC," Journal of Online Learning & Teaching,, vol. 11, no. 1, 2015.
[36] F. Brouns, J. Mota, L. Morgado, D. Jansen, S. Fano, A. Silva and A. Teixeira, "A networked learning framework for effective MOOC design: the ECO project approach," in 8th EDEN Research Workshop. Challenges for Research into Open & Dista
Learning: Doing Things Better: Doing Better Things , Oxford, UK, 2014.
[37] D. Yang, T. Sinha, D. Adamson and C. P. Rosé, "Turn on, tune in, drop out: Anticipating student dropouts in massive open online courses," in Proceedings of the 2013 NIPS Data-driven education workshop, 2013.
[38] D. Gamage, M. E. Whiting, T. Rajapakshe, H. Thilakarathne, I. Perera and S. Fernando, "Improving Assessment on MOOCs Through Peer Identification and Aligned Incentives," in Proceedings of the Fourth (2017) ACM Conference on Learning@ Sc
Cambridge USA, 2017.
[39] C. Kulkarni, J. Cambre, Y. Kotturi, M. S. Bernstein and S. Klemmer, "Talkabout: Making distance matter with small groups in massive classes," Design Thinking Research, pp. 67-92, 2016.
[40] M. Wen, D. Yang and C. P. Rosé, "Virtual teams in massive open online courses," in International Conference on Artificial Intelligence in Education, 2015.
[41] P. C. Blumenfeld, E. Soloway, R. W. Marx, J. S. Krajcik, M. Guzdial and A. Palincsar, "Motivating project-based learning: Sustaining the doing, supporting the learning," Educational psychologist , vol. 26, no. 3, pp. 369-398, 1991.
[42] D. Johnson and R. T. Johnson, "Making cooperative learning work," Theory into practice, vol. 38, no. 2, pp. 67-73, 1999.
[43] S. L. Miller, "Teaching an Online Pedagogy MOOC," Journal of Online Learning & Teaching, vol. 11, no. 1, 2015.
creative commons Lincence to add