Presentation by Kim Larson and Steven Moore, CMU's Open Learning Initiative, CEBMa's Annual Meeting, Chicago 2018

1. Evidence-Based
Instruction
Kim Larson, Open Learning Initiative
Steven Moore, Eberly Center
1

2. Learning Engineer
“Working in collaboration with members of the faculty
whose interest they can excite, they design and
redesign learning experiences in particular disciplines”
In addition, their role is “to bring the campus into
contact with the lively and significant current activity
in cognitive psychology, and with developments
related to learning machines and computer-aided
instruction”
Herb Simon 1967
2
Bridging learning research and teaching practice for the public good:The learning engineer
Candace Thille, Stanford University

3. Desired Outcome
By the end of this session...
You will have an awareness of a few learning science principles so that you
you can incorporate them into your teaching.
3

4. oli.cmu.edu
Scientifically-based online learning environments based on
the integration of technology and the science of learning
with teaching. OLI is designed to simultaneously improve
learning and facilitate learning research.
What is the Open Learning Initiative?
4

5. oli.cmu.edu

6. oli.cmu.edu
What Are the Affordances of the
Technology?

7. oli.cmu.edu
Data Collection &
Feedback Loops for
Continuous
Improvement
7

8. oli.cmu.edu
OLI STUDY ON ACCELERATING STUDENT LEARNING WITH OLI
STATISTICS
This study, conducted at Carnegie Mellon University, shows that students
using the OLI statistics course at Carnegie Mellon achieved the same or better
learning outcomes as students in the traditional course in half the time.
Lovett, M., Meyer, O., & Thille, C. (2008). The Open Learning Initiative:
Measuring the effectiveness of the OLI statistics course in accelerating
student learning. Journal of Interactive Media in Education.
INDEPENDENT TRIAL OF THE OLI
STATISTICS COURSE
The results of this study are remarkable; they show
comparable learning outcomes for this basic course,
with a promise of cost savings and productivity gains
over time.
Deanna Marcum
Managing Director, Ithaka S+R
Accelerated Student Learning
9

9. oli.cmu.edu
•OLI students completed course in half the time with half the
number of in-person course meetings
•OLI students showed significantly greater learning gains (on
the national standard “CAOS” test for statistics knowledge) and
similar exam scores
•No significant difference between OLI and traditional students
in the amount of time spent studying statistics outside of class
•No significant difference between OLI and traditional students
in follow-up measures given 1+ semesters later
Proven Results
10

10. oli.cmu.edu
Which problem type is most difficult for
beginning algebra students?
Story Problem
As a waiter, Ted gets $6 per hour. One night he
made $66 in tips and earned a total of $81.90. How many
hours did Ted work?
Word Problem
Starting with some number, if I multiply it by 6 and
then add 66, I get 81.90. What number did I start with?
Equation
x * 6 + 66 = 81.90
11

11. oli.cmu.edu
Algebra Student Results:
Story Problems are Easier!
12
Nathan, M.J. & Koedinger, K.R. (2000). Teacher’s and researchers beliefs of early algebra development.
Journal of Mathematics Education Research, 31(2), 168-190

12. oli.cmu.edu
Algebra Student Results:
Story Problems are Easier!
Nathan, M.J. & Koedinger, K.R. (2000). Teacher’s and researchers beliefs of early algebra development.
Journal of Mathematics Education Research, 31(2), 168-190
13

13. oli.cmu.edu
Expert Blindspot
14

14. oli.cmu.edu
Instructor-based, intuitive design lacks
information and is thus flawed
Experts can
describe
only 30% of
what they
know!
What we know
about our
own learning
What we
do not
know we
know
Richard Clark:
http://www.youtube.com/watch?v=SSK63nqEbLQ
15

15. Course Design
16

16. oli.cmu.edu
Course Design Triangle
17

17. oli.cmu.edu
Why Focus on Learning Objectives?
1. They communicate our intentions clearly to students and
to colleagues.
2. They provide a framework for selecting and organizing
course content.
3. They guide in decisions about assessment and
evaluation methods.
4. They provide a framework for selecting appropriate
teaching and learning activities.
5. They give students information for directing their
learning efforts and monitoring their own progress.
Based on A.H. Miller (1987), Course Design for University Lecturers. New York: Nichols Publishing.
Also see, C.I. Davidson & S. A. Ambrose (1994), The New Professor’s Handbook: A Guide to Teaching and Research in Engineering and Sciences. Bolton, MA:
Anker Publishing Company Inc.
18

18. oli.cmu.edu
“. . .
the explicit statement of instructional objectives based on psychological theory
should have beneficial effects both in design of instruction and assessment of
student achievement.
The reason is simple: we can generally do a better job of accomplishing
something and determining how well we have accomplished it when we have a
better understanding of what it is we are trying to accomplish.”
(Greeno, 1976,p. 123)
19
Why Focus on Learning Objectives?

19. oli.cmu.edu 20
What do you want your students to
be able to do by the end of the
course?
Performances &
demonstrations of
knowledge
Performances &
demonstrations of
knowledge

20. oli.cmu.edu
Make Learning something
you can observe
Understand the Federal Reserve system
21
Recognize logical flaws in a written argument
Appreciate the historical context of 1940’s
Apply Newton’s Second Law appropriately

21. oli.cmu.edu
Different types of knowledge
22
Memory/Fluency
Induction/Refinement
Sense making/Understanding
Koedinger, K.R., Corbett, A.T., Perfettic, C. (2012). The Knowledge-Learning-Instruction Framework: Bridging the
Science-Practice Chasm to Enhance Robust Student Learning . Cognitive Science Journal, 36 (2012), 757-798

22. oli.cmu.edu
Different types of assessments for
different types of knowledge
23
Memory/Fluency
Facts- Memorization
Induction/Refinement
Procedural, Rules
Sense making/Understanding
Self reflection
Koedinger, K.R., Corbett, A.T., Perfettic, C. (2012). The Knowledge-Learning-Instruction Framework: Bridging the
Science-Practice Chasm to Enhance Robust Student Learning . Cognitive Science Journal, 36 (2012), 757-798

23. Instructional Content
Worked examples
Step-by-step demonstration of how to perform a task or
solve a problem.
24

24. oli.cmu.edu 25

25. Multimedia Principles
Eight research-based guidelines
26

26. oli.cmu.edu
Multimedia Principles of Learning
● 12 research-based principles handout
● Cognitive load - effort being used in working
memory
● Limited capacity in the audio and visual channels

27. oli.cmu.edu
Multimedia
● Which is better for student learning?
A. Learning from words and pictures
B. Learning from words alone
Example: Description of how lightning works with or without
a graphic
● A. Words & pictures, adding graphics to text improves learning
when graphics are relevant rather than decorative
● Why?
Students can mentally build both a verbal & pictorial model,
then make connections between them

28. oli.cmu.edu
Contiguity
● Which is better for student learning?
A. When corresponding words & pictures are presented
far from each other on the page or screen
B. When corresponding words & pictures are presented
near each other on the page or screen
Example: “Ice crystals” label in text off to the side of the
picture or next to cloud image in the picture
● B. Near
● Why?
Students do not have to use limited mental resources to
visually search the page. They are more likely to hold both
corresponding words & pictures in working memory &
process them at the same time to make connections.

29. oli.cmu.edu
Modality
● Which is better for student learning?
A. Spoken narration & animation
B. On-screen text & animation
Example: Verbal description of lightning process is
presented either in audio or text
● A. Spoken narration & animation
● Why?
Presenting text & animation at the same time can
overload visual working memory & leaves auditory
working memory unused

30. oli.cmu.edu
Image

31. oli.cmu.edu
Coherence
● Which is better for student learning?
A. When extraneous, entertaining material is included
B. When extraneous, entertaining material is excluded
● B. Excluded
● Why?
Extraneous material competes for cognitive resources in
working memory and diverts attention from the important
material

32. oli.cmu.edu
Personalization
● Find the greatest common denominator.
● Close the webpage.
● Next, let’s find the greatest common
denominator.
● Let’s close the webpage and continue.

33. oli.cmu.edu
Segmenting
● Which is better for student learning?
A. Let a lesson “play” like a video so learners have a
continuous picture of the entire procedure
B. Let learners control the sequence by selecting
screens in small parts so they can work at their own rate
● When context is complex, then
B. Break a complex lesson into smaller self-paced parts
● Why?
Segmenting a lesson allows the learner to engage in
essential processing without overloading the cognitive
system

34. oli.cmu.edu
Pretraining
● Confusing image without learning about the parts before
● Trying to present the definitions in tandem can overload
novices

35. Learning Sciences
Additional helpful principles
36

36. oli.cmu.edu
Generation Effect
● apple
● able
● aerosol
● attic
● albert
● b______
● b______
● b______
● b______
● b______

37. oli.cmu.edu
Spacing vs Massed
● Massed practice - no rest between study periods
(cramming)
● Spaced practice - shorter intervals, with rest in
between
● Increased retention and learning gains
● Can lead to overlearning and a lack of retention

38. oli.cmu.edu
Active Learning
● A teaching strategy that requires students to
engage cognitively, above and beyond passive
listening and note taking.
● In STEM classes, there is robust evidence that
active learning:
○ supports increased student performance on
assessments
○ supports decreased failure rates compared to
traditional lecture
● Handout: active learning activities

39. oli.cmu.edu
Games & Simulation
“Games are very motivating and have tremendous
potential in education, but despite a rapid growth in the
research base, there is yet insufficient evidence to draw
definitive conclusions.”
- Hannafin & Vermission, 2008

40. Future Work
Closing notes and useful resources
41

41. oli.cmu.edu
Books
42

42. oli.cmu.edu
Books
43

43. Resources
44
● https://oli.cmu.edu/
● https://www.cmu.edu/teaching/
● https://drive.google.com/drive/folders/1AUB00kvEy2DQVxTJ_4g
JRnp16U6ohSXi

44. oli.cmu.edu
Questions
45

45. oli.cmu.edu
Thank You!
46
kllarson@cmu.edu
stevenmo@andrew.cmu.edu