Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research
•Transferir como PPTX, PDF•
1 gostou•697 visualizações
Invited talk given at Declara
![Learning Engineering of MOOClets: Simultaneously
benefiting Professional Learning, Financial Success,
and Cognitive & Learning Sciences Research
Joseph Jay Williams
Lytics Lab, Graduate School of Education
Office of the Vice Provost for Online Learning, Stanford
josephjaywilliams@stanford.edu
www.josephjaywilliams.com
1
[Originally from Trinidad & Tobago – apologies for difficulty in following accent.]](https://image.slidesharecdn.com/learningengineeringofmoocletsforpublic-140604191241-phpapp02/85/Learning-Engineering-of-MOOClets-Simultaneously-benefiting-Professional-Learning-Financial-Success-and-Cognitive-Learning-Sciences-Research-1-320.jpg)
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (19)
Destaque
Semelhante a Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research
Semelhante a Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research (20)
Mais de Joseph Jay Williams
Mais de Joseph Jay Williams (9)
Último
Último (20)
Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research
- 1. Learning Engineering of MOOClets: Simultaneously benefiting Professional Learning, Financial Success, and Cognitive & Learning Sciences Research Joseph Jay Williams Lytics Lab, Graduate School of Education Office of the Vice Provost for Online Learning, Stanford josephjaywilliams@stanford.edu www.josephjaywilliams.com 1 [Originally from Trinidad & Tobago – apologies for difficulty in following accent.]
- 2. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 2
- 3. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 3
- 4. I. Motivation • Beliefs about Intelligence • To what extent do you agree? How smart you are can be significantly changed by your effort and hard work. You have a certain amount of intelligence, and what you do doesn’t really change that much. • Growth Mindset vs. Fixed Mindset • Dweck, 2008; Yeager & Walton, 2012 • Williams, Paunesku, Haley & Sohl-Dickstein, 2013 4
- 5. MOOClet 1: Exercise (Khan Academy) 5 1. Number of Problems Practiced 2. Accuracy [TEXT SHOWN HERE – Randomize]
- 6. Embed motivational messages Practice-as-usual Remember, the more you practice the smarter you become! Growth Mindset Message 6
- 7. Growth Mindset Message helpful? • Growth Mindset Message • "Remember, the more you practice the smarter you become.”, • "Mistakes help you learn. Think hard to learn from them.” • Practice-as-usual Jascha Sohl-Dickstein 7
- 8. Results: More motivated? • Growth Mindset Message > Practice-as- Usual • reduction in dropout rate • increase in Number of problems practiced • increase in Number Correct • Accuracy: Problems correct/Problems attempted • Accuracy improvement 8
- 9. Interpreting results • Obvious? • Skeptical? Anything’s statistically significant with large N… • Second experiment (run simultaneously) 9
- 10. Practice-as-usualGrowth Mindset MessagePositive Message Different motivational messages Some of these problems are hard. Do your best! Remember, the more you practice the smarter you become! 10
- 11. Does any positive message work? • Growth Mindset Message • "Remember, the more you practice the smarter you become.”, • "Mistakes help you learn. Think hard to learn from them.” • Positive Message • "Some of these problems are hard. Just do your best." • "This might be a tough problem, but we know you can do it.” • Practice-as-usual 11
- 12. Effects of Positive Messages? • Positive Messages not better than Practice-as- Usual! • Growth Mindset > Positive Message • How intuitive is this result? • Teachers? • Ongoing: After incorrect answer • www.assistments.org Neil Heffernan • Gates Grant for Modeling Research 12Rapid Experimentation: tiny.cc/mechanicalturk
- 13. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 13
- 15. “MOOClets” or Modular Online Resources • E.g. MOOClet: Math exercise, 5 minute interactive video, 30 minute lesson, Study Strategy coach, Quiz/Assessment • www.josephjaywilliams.com/mooclets • MOOClets Group Tue 12-1 PST (remote) • tiny.cc/joinmooclets mailing list & updates • Williams, J.J. & Williams, B. A. (2013). Randomized Experiments as a Methodological and Conceptual Tool for Improving Design of Online Learning. Data Driven Education, NIPS Workshop. tiny.cc/experimentsonlinelearning 15
- 16. Integrate Diverse Expertise & Innovations 16
- 17. Where is relevant research, best practice, technology, funding, market? tiny.cc/mapofspace, tiny.cc/stanfordmap MOOCs (Coursera, EdX, Udacity) www.KhanAcademy.org www.Assistments.org K-12 Ed-Tech space (Bay Area, Boston...) (www.edsurge.com) (www.edudemic.com ) Higher-Ed Technology/LMS E-Learning & Training for Corporate, State, Federal, Army Psychological Scientists Computer Scientists Education Corporate Training Funding Agencies: IES, Dept. of Education, Dept. of Labor, NIH, NSF Practical Reports: What Works Clearinghouse IES Practice Guides University Teaching & Learning Centers Department of Education reports Think Tanks: SRI International WestEd Pittsburgh Science of Learning Center Learning Sciences Mooc Research Initiative Open Learning Initiative Distance Learning Athabasca, Western Governor’s University Economics of Education Habit & Behavior Change Medical Education Centers/Scholarship of Teaching & Learning Assessment Pearson ETS Open Education Resources Creative Commons Foundations: Gates, Spencer, Hewlett, Kresge Venture Capital Private VCs New Schools Impact Investing Accelerators: Imagine K-12, HCI, Design Thinking Workforce Development 17
- 18. Relevant Resources • www.josephjaywilliams.com/home/useful- software • www.josephjaywilliams.com/education • www.josephjaywilliams.com/online-education- experiments • tiny.cc/mapofspace • Applying Cognitive Science to Online Learning. Paper presented at NIPS Workshop on Data Driven Education (tiny.cc/cognitivescienceinmoocs) • Improving Outcome of Psychosocial Treatments by Enhancing Memory and Learning. Perspectives in Psychological Science. 18
- 19. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 19
- 20. Messages in Coursera? 20 Week 6 Quiz Whether or not you’re correct on the first try, each attempt at a tough problem can make you better at statistics.
- 21. Motivational Introductions 21 Practice-as-usual Have fun in the course! Anne Friedlander Growth MindsetText version of Mindset MessageMindset Message – Another InstructorPositive EncouragementBelonging
- 22. Instructor Guide to foster Growth Mindset • tiny.cc/idstrategymindset 22 Where do you think you will apply this? Do you have any questions or want clarification? Click to see any of the Case Studies below Would you like to?
- 23. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 23
- 24. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 24
- 25. II. Learning through Questions/Assessments • Discovery of patterns (Williams & Lombrozo, 2010, Cognitive Science) • Use of prior knowledge (Williams & Lombrozo, 2013, Cognitive Psychology) • Children’s causal learning (Walker, Williams, Lombrozo, Gopnik, 2012) • Overgeneralize at expense of specific facts (Williams et al, 2013, Journal of Experimental Psychology: General) • Contradictions & belief revision (Williams, et al 2013; 2014; Cog Sci, CHI WiP) • Comparison of examples (Edwards, Williams, Lombrozo, Gentner, 2013) GLORP DRENT Online/Blended Educational Resources
- 26. Adding questions/activities • tiny.cc/augmentedvideo • tiny.cc/augmentedexercise 26
- 27. III. General Learning & Problem-Solving Skills 27
- 28. Learning/Self-Questioning Strategies • Specific Prompt VS. • General Strategy of Self-Questioning? • Teach strategies (McNamara, 2004, Discourse Processes; Palinscar & Brown, 1984, Cognition and instruction; Williams & Lombrozo, 2010, Cognitive Science) • Habit & Behavior Change • Theory & Technology: Embed & Remind • Methodology: “longitudinal”, large N 28e.g. Weinstein & Mayer, 1986 Rothman, Baldwin, & Hertel, 2004
- 29. Embedded “On-Request” Strategy Instruction Clickable link. [Click here to learn about the “What? Why? How?” strategy] 29
- 30. Self-questioning strategy: What? Why? How? 30
- 31. Cues – Reflect & Remind 31
- 32. Preliminary Results 32N = ~ 800 per condition • Prompted Problem-solving better than Practice-as-usual. • Increased Accuracy on intervention and post-intervention problems.
- 34. Learning Assistant – e.g. 1 34
- 35. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 35
- 36. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 36
- 37. IV. Performance Support Tools & Checklists 37
- 39. A Lesson: Concept + Digital Tool to apply Concept in Future 39
- 40. Tool for Difficult Conversations 40
- 41. Topics • I. Motivation • II. Learning through Questions/Assessments • III. General Learning & Problem- Solving Skills • IV. Performance Support Tools & Checklists 41
- 42. Guiding Principles • Multiple Constraints: Practical, Scientific, Financial • Modularity in Product Design: “MOOClets” • Future-thinking but Actionable • Empirical and Integrative Work • Experiment-Focused Instructional Design • Matchmaking Relevant Researchers • Iterative Improvement • Collaborative Development 42
- 43. Acknowledgements • Tania Lombrozo & Tom Griffiths • John Mitchell & Candace Thille • Jascha Sohl-Dickstein • Jace Kohlmeier & Khan Academy • Sam Maldonado • Lytics Lab (lytics.stanford.edu) • VPOL (Vice Provost for Online Learning, online.stanford.edu) 43
Notas do Editor
- Originally from Trinidad. Thank you for having me here, it’s a pleasure because of how this entire movement has impacted me. Two years ago was finishing my dissertation work in cognitive science – psychology experiments on learning, & computational psychology, or using Bayesian statistics & machine learning to model human thinking, but I had been doing work in the education school, and wanted to have a real-world impact, and actually build products. Seemed impossible to do both, but all of a sudden – online education exploded! Tremendously excited, started research in collaboration with Khan Academy, and have been trying to add value by bringing to bear scientific knowledge about learning in a way that’s actually practically useful, and linking researchers to these practical platforms, like writing grants with Khan Academy & Stanford Online. So I will present a subset of that work. But it will be different from a typical research talk, because I took such an amazing group as a chance to share explicitly on the approach I take to developing online educational materials, point to resources people here might also find useful, and emphasize some broad themes in online learning where I think real-world impact and science converge. Interested in discussion & feedback from you on these – you have thought about it a tremendous deal here at Coursera, and much more familiar with your and others’ platforms. Don’t hesitate to make suggestions or state different opinion, I’d like to learn more about your thinking on these issues. Start with practical or “product-first”. Not necessarily a typical focus for researchers, except Candace Thille, Ken Koedinger, ? 50% is primary research, which I’ll present. 50% synthesizing existing work, having broad view of technology, entrepreneurship. I like collecting these list, and think there’s a lot of value to be tapped. Take a business perspective – where is there unrealized value, because of lack of information? So I’m going to discuss ongoing work with Khan Academy & MOOCs and residential education. Atypical Trajectory, which means the talk has many moving pieces: Use digital online educational resources to do use-inspired basic research. Extremely tough to link research and practice, so the last few years I’ve taken positions that allowed me to invest a lot of time into having a broad view of the emerging trends – understanding technology and goals of the MOOC platforms, Khan Academy, hundreds of Startups in Bay Area, as well as e-learning in the corporate training & workforce development world. Also been trying to get a broad view of which research topics and themes are particularly well suited to these contexts, from cognitive psychology to educational topics, social & clinical, health behavior change. Methodologies from HCI, education policy.
- Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
- o.32 0.27
- Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
- Have an unusual background, so I thought it might help to provide some context. Approach: Use digital educational resources to do inspired basic research (e.g. math exercises), understand how cognitive processes like explaining can guide learning & how people’s beliefs influence motivation. (Pasteur’s Quadrant). Principles & Demonstrations for doing this kind of work laid by PSLC. ust to provide some context about Use digital online An interdisciplinary approach to linking research and practice,PhD training is a mixture of Experimental Psychology, Computational Psychology (using methods from machine learning & Bayesian statistics to model cognition),
- Exhausting… but exciting! Criticized for not using literature… but don’t have time to. Researchers haven’t looked at & thought enough about platforms, to synthesize the key ideas. Also rare to have the time to take a broad view.
- Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
- Add textbox below first two questions for people to type into. Add some hyperlink looking things below “Click … case studies” with something like these fields: Encouraging students to see literary skills as malleable through class emails Improving persistence in Database Course programming assignments Enhancing receptiveness to peer feedback in management courses Under would you like to, same kind of thing, but with: Learn more about the research behind this concept Ask a question or share thoughts with the Teach-Net mailing list Request a future reminder for applying this guide
- Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
- Not traditional, but ubiquitous A challenging choice… Take the next theoretical step in this line of research, with a new set of materials. On the other hand, this is venturing into new territory. PSLC & others been successful, they have full control. Also, since there’s little direct past research in many of these contexts, don’t even know yet what form the phenomenon takes. Start with a practical challenge first, seek to improve it by bringing to bear research. Establishing whether there is an effect, sets the stage for using this online context to probe its underlying mechanisms. Jump to the other side, use-inspired basic research.
- Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?
- Collaborative Development Guided by Practical and Research Goals ASSESSMENT/MEASUREMENT/COMPETENCY BASED?