Deciphering the MOOC ecosystem: what makes successful MOOCs

DECIPHERING THE MASSIVE OPEN ONLINE
COURSES (MOOCS) ECOSYSTEM: WHAT
MAKES SUCCESSFUL MOOCS?
Ido Roll, Will Engle, Simon Bates!
@hummus_monster @infology @simonpbates @CTLT_UBC!
Centre for Teaching, Learning, and Technology!
University of British Columbia
1

@hummus_monster @simonpbates @UBC_CTLT #STLHE2014
Please raise your hand if you have pre-teen
child(ren) at home.!
Please keep it up if you know where the following
sentence is taken from:!
“The snow glows white on the mountain tonight”
2

LEARNING FROM VIDEO !
IS POSSIBLE
3
If someone wants to learn, you cannot stop them.!
@eric_mazur #STLHE2014

WE KNOW A LOT ABOUT
UNIVERSITY TEACHING
What works?!
Small classrooms!
Interactivity!
With material!
With peers!
With instructors!
Feedback!
…
4
MOOCs

THREE KEY QUESTIONS
5
Who are !
our learners?
What deﬁnes !
success?
What course design
supports that?

CONTEXT
Four UBC courses (Spring/Summer 2013)!
Climate Literacy; Useful Genetics; Program Design; Game
Theory!
330,000 learners registered. !
Five Edinburgh courses (Winter/Spring 2013)!
Introduction to Philosophy; Artiﬁcial Intelligence;
Astrobiology; Nutrition; Critical Thinking.!
309,000 learners registered.!
https://www.era.lib.ed.ac.uk/handle/1842/6683!
6

VIDEO PRODUCTION
7
Averagelengthpervideo(min)
0
6
12
18
24
Average # of videos per week
0 4 8 12 16
UBC 1
UBC 2
UBC 3
UBC 4

THREE KEY QUESTIONS
8
Who are
our learners?
What deﬁnes !
success?
What course design
supports that?

WHO ARE OUR LEARNERS?
General demographics consistent across courses
and with Coursera.
9
Education level
%responses
0
25
50
75
100
UBC 1 UBC 2 UBC 3 Coursera
High School Diploma
Professional School
Some College
Bachelor’s Degree
Master’s Degree
Doctorate Degree

WHO ARE OUR LEARNERS?
General demographics consistent across courses
and with Coursera.
10
Ethnicity
%responses
0
25
50
75
100
UBC 1 UBC 2 UBC 3 Coursera
White or Caucasian
South Asian
East Asian
Other Asian
Black or African American
Decline to State

ROLE OF GEOGRAPHY
Does location matter? Are MOOCs “local”?
11
Country of Origin
%responses
0
15
30
45
60
UBC overall Coursera Edinburgh
11
3.6
3.9
United States
India
Canada
United Kingdom
Spain
Brazil
Germany

ROLE OF GEOGRAPHY
Does location matter? Are MOOCs “local”?
12
Country of Origin
%responses
0
15
30
45
60
UBC overall Coursera Edinburgh
43.9
5.2
United States
India
Canada
United Kingdom
Spain
Brazil
Germany

ROLE OF GEOGRAPHY
Break down by course.
weighting by overall
registration, population of
country:!
How many more students
are local, per 10,000
registrants, per capita?!
In Canada, !
~150 additional local
students!
~500 Canadians overall
13
Extralocallearnersper10,000
registrants,1,000,000residents
0
2.5
5
7.5
10
UBC Edinburgh
2.0
1.0
55.7
3.3
5.5
4.8

THREE KEY QUESTIONS
14
Who are !
our learners?
What deﬁnes
success?
What course design
support that?

WHAT DEFINES SUCCESS?
Deﬁnitions:!
Visitors: log in at least once. Our denominator.
15
Activities
Videos
Website
Week
0 1 2 3 4 5 6 7 8

METRICS OF SUCCESS
Deﬁnitions:!
Visitors: log in at least once. Our denominator.!
Tasters: watch a video on Week 2.
16
Activities
Videos
Website
Week
0 1 2 3 4 5 6 7 8

METRICS OF SUCCESS
Deﬁnitions:!
Tasters: watch a video on Week 2.!
Learners: watch videos all the way through.
17
Activities
Videos
Website
Week
0 1 2 3 4 5 6 7 8

METRICS OF SUCCESS
Deﬁnitions:!
Learners: watch videos all the way through. !
Active learners: problem sets, discussion forums.
18
Activities
Videos
Website
Week
0 1 2 3 4 5 6 7 8

METRICS OF SUCCESS
Deﬁnitions:!
Learners: watch videos all the way through. !
Active learners: problem sets, discussion forums.
19
Activities
Videos
Website
Week
0 1 2 3 4 5 6 7 8

METRICS OF SUCCESS
Metrics:!
Attractiveness: convertVisitors to Tasters.!
Retention: convert Tasters to Learners!
Interactivity: convert Learners to Active Learners.
20
Activities
Videos
Website
Week
0 1 2 3 4 5 6 7 8
Visitors
Tasters Learners
Active Learners

METRICS OF SUCCESS
21
0
0.15
0.3
0.45
0.6
Attractiveness Retention Interactivity
48%
54%
35%
50%
31%29%
52%
58%
36% 35%
57%
24%
Activities
Videos
Website
Week
0 1 2 3 4 5 6 7 8
Visitors
Tasters Learners
Active Learners
17k 13k 49k 85k 2k 3k 4k 16k

THREE KEY QUESTIONS
22
Who are !
our learners?
What deﬁnes !
success?
What course design
support that?

WHAT AFFECTS CONVERSION
RATES?
Correlate with:!
Average length per video!
Average length per week
23
Activities
Videos
Website
Week
0 1 2 3 4 5 6 7 8
Visitors
Tasters Learners
Active Learners

ATTRACTIVENESS
Visitors -> Tasters: How many of those who
visited the website watched Week 2 videos?!
By length per week: r = -0.72*, r² = 0.52!
24
%ofVisitorswho
arealsoTasters
0%
20%
40%
60%
Overall legth of videos per week (min)
0 45 90 135 180
UBC
Edinburgh

ATTRACTIVENESS
Visitors -> Tasters: How many of those who
visited the website watched Week 2 videos?!
By length per video: r = -0.33!
25
%ofVisitorswho
arealsoTasters
0%
20%
40%
60%
Average length per video
5 8 11 14 17
UBC
Edinburgh

RETENTION
Tasters -> Learners: How many of those who
watched @ Week 2 also watched @ Week 5?!
By length per week: r = -0.29!
26
%ofTasterswho
arealsoLearners
0%
26.667%
53.333%
80%
Overall legth of videos per week (min)
0 45 90 135 180
UBC
Edinburgh

RETENTION
Tasters -> Learners: How many of those who
watched @ Week 2 also watched @ Week 5?!
By length per video: r = 0.55, r² = 0.30
27
%ofTasterswho
arealsoLearners
0%
20%
40%
60%
80%
Average length per video
5 8 11 14 17
UBC
Edinburgh

INTERACTIVITY
How many of the students who watch videos also
submit problem sets?
28
%ofvideowatchers
whosubmitted
problemsets
20%
30%
40%
50%
60%
70%
Week
0 2 4 6 8
UBC 1
UBC 2
UBC 3
UBC 4

SUMMARY
Local matters!
Courses have unique “signatures”!
Structure, e.g., # and length of videos!
Engagement, e.g., % active learning.!
Overall length of videos is negatively correlated
with retention!
But length of individual videos does not show such
relationship. !
True for both universities
29

DESIGN RECOMMENDATIONS
Long videos may be okay!
Especially later in the course!
Keep overall load reasonable.!
Active learning is not affected by week-by-week
variations!
Two-stage courses: attractiveness and retention.
30

Deciphering the MOOC ecosystem: what makes successful MOOCs

Recommended

Recommended

More Related Content

Similar to Deciphering the MOOC ecosystem: what makes successful MOOCs

Similar to Deciphering the MOOC ecosystem: what makes successful MOOCs (20)

More from Simon Bates

More from Simon Bates (20)

Recently uploaded

Recently uploaded (20)

Deciphering the MOOC ecosystem: what makes successful MOOCs