Presentation from the 2014 New Directions in Online Learning Conference in San Diego. The gist is that faculty can see which students need help early on, intervene with those students, and thereby increase the number of students who successfully complete the course. All data has been anonymized.

1. Getting Students To Do the Work
Using Learner Analytics Data to Inform Instructional Strategies
New Directions in Online Learning
San Diego, 2014
David Lindrum

2. Study Questions vs. Assignments

3. Study Questions vs. Assignments
WEBTEXT

4. Study Questions vs. Assignments
WEBTEXT

5. Study Questions vs. Assignments
WEBTEXT
380 Study Questions

6. Study Questions vs. Assignments
WEBTEXT
• 33 Audio / Video Selections
• 34 Articles and Studies
• 8 Infographics, Websites, etc.

7. Study Questions vs. Assignments
WEBTEXT
Study questions
on each page
Assignments
in each chapter

8. Data Collected From Student Activities
Log-In
- first enrollment
- each log-in time
- most recent date
Active Time
Time spent on
each page typing,
clicking, or scrolling.
Participation
- pages viewed
- questions answered
- number of resets
Scoring
- on multiple choice
- first attempt
- most recent attempt
Higher Order
- assignments
- short answer
- response boards

9. About Our Data
In two studies, assignment scores (non-resettable, application-type
questions) are significant predictors of midterm and final scores. Other
webtext activities are also correlated with course success, including
minutes per question and question completion.
In multiple online courses (over 8,000 students), webtext success
pass closely matches course pass rate.

10. A Tale of Two Courses

11. Percentage
Complete
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Webtext Completion*
PSY Pilot Instructor 1
*N=35 students; excludes 5 students with 0 time.
PSY
Summer 2012
median
✦ 21% webtext
completion
✦ 102 questions
answered
✦ 8.8 hours
active time
✦ 6.2 minutes
per correct
question
median

12. !
Just 20-30%
of students
do assigned
reading.
(Hobson, 2004)

13. 100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Webtext Completion*
*N=46 students; excludes 1 student with 0 time.
PSY
Fall 2012
median
✦ 99% webtext
completion
✦ 475 questions
answered
✦ 16.4 hours
active time
✦ 2.9 minutes
per correct
question
PSY Pilot Instructor 2
median

14. 100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Webtext Completion*
*N=46 students; excludes 1 student with 0 time.
PSY
Fall 2012
median
✦ 99% webtext
completion
✦ 475 questions
answered
✦ 16.4 hours
active time
✦ 2.9 minutes
per correct
question
PSY Pilot Instructor 2
median
30% Course Credit for
Webtext Assignments
and Study Questions

15. Percentage
Complete
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Webtext Completion*
PSY Pilot Instructor 1
*N=35 students; excludes 5 students with 0 time.
PSY
Summer 2012
median
✦ 21% webtext
completion
✦ 102 questions
answered
✦ 8.8 hours
active time
✦ 6.2 minutes
per correct
question
median
0% Course Credit for
Webtext Assignments
and Study Questions

16. What We Learned
Students don’t do optional
Assign credit for formative activities that will prepare students to do well on final.
Credit should generally be between 20-30% of course credit.

17. What We Learned
Students don’t do optional
Assign credit for formative activities that will prepare students to do well on final.
Credit should generally be between 20-30% of course credit.
Students respond to personal outreach (nagging, parenting)
Instructor 2 emailed students at the end of every week
Go get your learning resource. You can’t pass this class without it.
I see that you’re falling behind. If you don’t catch up, you won’t pass this class.

18. Percentage
Complete
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Webtext Completion*
PSY Pilot Instructor 1
What exactly are successful-student behaviors?
*N=35 students; excludes 5 students with 0 time.
PSY
Summer 2012
median
✦ 21% webtext
completion
✦ 102 questions
answered
✦ 8.8 hours
active time
✦ 6.2 minutes
per correct
question
median

19. Successful Student Behaviors

20. 1. Successful students
do optional activities
like study questions.

21. Median Webtext Completion Percentages
Assignments
Study Questions
Polls
Your Turns
By Assignment Scores
Fall 2012
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
A’s B’s C’s D’s F’s
POL
No credit
n=48 at R1, f2f

22. “[L]earning techniques such as taking practice tests and spreading
study sessions out over time — known as distributed practice —
were found to be of high utility because they benefited students of
many different ages and ability levels and enhanced performance in
many different areas.”
Improving Students’ Learning with Effective Learning Techniques
(Dunlosky, et al. 2013)

23. Credit buys time.
!
(and time is fixed.)

24. 2. Successful students
complete more work.

25. Assignments
Study Questions
Average Webtext Completion
By Final Grade POL
0 10 20 30 40 50 60 70 80 90 100
A’s B’s C’s D’s F’s
n=57 at R1, f2f

26. Median Webtext Scores (Lines)
100
90
80
70
60
50
40
30
20
10
0
As of October 28, 2013 PSY
As
Bs
Cs
Fs
Ch 1 Ch 2 Ch 3 Ch 4 Ch 5 Ch 6
Bars represent completion.
!
Lines represent percentage
of points earned per
module.
!
Ds
n=539

27. Median Webtext Scores (Lines) and Completion (Bars)
100
90
80
70
60
50
40
30
20
10
0
As of October 28, 2013 PSY
Ch 1 Ch 2 Ch 3 Ch 4 Ch 5 Ch 6
As
Bs
Cs
Fs
Bars represent completion.
!
Lines represent percentage
of points earned per
module.
!
Ds
n=539

28. 3. Successful students
invest more time.

29. Webtext Hours
Median Active Time in Webtext - Hours
By Webtext Scores
As of October 28, 2013
0 4.5 9 13.5 18
A’s B’s C’s D’s F’s
ORG
n=1,040 at online

30. By Webtext Scores HIS
4
3.2
2.4
1.6
0.8
0
100%
80%
60%
40%
20%
0%
Median Assignment TIme and Scores
As of October 28, 2013
As Bs Cs Ds Fs
Assignment Score Assignment Completion Hours
n=560 at online

31. 500
450
400
350
300
250
200
150
100
50
0
Median Time
Median Correct
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
Active Time in Minutes
Correct Responses
Active Time and Correct Responses
All Students, All Questions (large, 2-year)

32. 4. Successful students
start working before the
first day of class.

33. With Median Hours Per Week PSY
5
3.75
2.5
1.25
0
100%
80%
60%
40%
20%
0%
Median Webtext Scores By Week
As of October 28, 2013
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6
Pre-Term Score Pre-Term Hours Week 1 Score Week 1 Hours Week 2+ Score Week 2+ Hours
n=539 at online

34. Successful Instructor Behaviors

35. Follow up.
(Which is why instructors need actionable analytics.)

39. WHAT IF WE KNEW
STUDENTS WERE
DOING THE WORK?

41. Questions?

42. david.lindrum@soomolearning.com