Talk given at the http://stem2014.ubc.ca/ conference, July 14th Vancouver Canada

1. !
bit.ly/batestalks
Expert-guided crowd-sourced
learning content:
a pilot study
Simon Bates Firas Moosvi
@simonpbates @firasm

2. a"web&based"MCQ"repository"created"by"students"

3. Context
http://goo.gl/MHvDfH

4. Last year….

5. ‘Seeking’ behaviour

6. Challenge: Can we enhance the pre-reading activity to
engage more students more deeply with the material?
• Requirements:
• Outputs: distributed over the whole term
• Load: at least 2h of student time
• Assessment: Submitted online, marked using a
rubric
• TA Load: No increased marking load
!
Pilot study in 3 sections of PHYS101 Jan-Apr 2014

7. PHYS101:
Energy and Waves
http://goo.gl/OgP1Mt

8. PHYS101:
Energy and Waves
http://goo.gl/OgP1Mt

9. PHYS101:
Energy and Waves
http://goo.gl/OgP1Mt

10. PHYS101:
Energy and Waves
http://goo.gl/OgP1Mt

11. PHYS101:
Energy and Waves
http://goo.gl/OgP1Mt

12. Sample Learning Objects
Ted-Ed: A video tutorial
and a series of questions
linked to time-stamps
Screencast: A slideshow
with annotations and
voice narrations
Prezi: A visual slideshow
format that can be used
to break down problems
and many more…

13. Implementation - Logistics
• Cohort split into 4 groups, 2 cycles per semester
• Required to submit two (2) learning objects over the
course of the term, each worth 2.5% of course grade.
• In addition to the two assigned weeks, students can
submit additional LOs
• The two highest scoring learning objects will be used to
determine themark.
• LO topic must be on pre-readings for the next week

14. Implementation - Submission & Survey
• Students submitted attachments or links to LOs online
through the Blackboard LMS
• Upon submission, students complete a short survey:
• How did you come up with your LO topic?
• Time Spent
• How much did you understand the topic before/
after LO creation
• Other feedback

15. physics.lo.open.ubc.ca
Learning Objects are
showcased here:

16. Sample 1 - Were Astronauts really on the moon?

17. Sample 2 - Standing Wave in a bowl

18. Sample 2 - Standing Wave in a bowl

19. Sample 3 - Colour Loss Underwater

20. Sample 4 - “Bathysphere” (a submersible)

21. Results
2. Time spent on Learning Objects
1. Participation Data
3. Benefits for student learning

22. LO submissions were constant through the term
0 100 200 300
LO 1
LO 2
LO 3
LO 4
LO 5
LO 6
LO 7
LO 8
Submitted LOs
Assigned
Optional

23. 0 10 20 30 40 50 60 70 80 90 100
0
100
200
300
Grade (%)
Numberofstudents LO Grade Distribution

24. These students
didn’t even try!
0 10 20 30 40 50 60 70 80 90 100
0
100
200
300
Grade (%)
Numberofstudents LO Grade Distribution

25. Which students aren’t submitting LOs?
Non-participants are from all walks of life!
0 10 20 30 40 50 60 70 80 90 100
0
5
10
15
20
25
Final Exam (%)
Numberofstudents
Final Exam marks of students
that did not participate in LOs

26. 0 100 200 300 400
Less than 0.5h
0.5 to 1 h
1 to 2h
2 to 3h
3 to 4h
4 to 5h
More than 5h
Number of students
Time spent on Learning Objects
Most students tended to work
on each LO for at least 2 hours

27. 0 200 400 600 800
None
Little
Moderate
Good
Excellent
Number of students
0200400600800
Number of students
Student self-reported understanding of
LO content before and after LO submission

28. 0 200 400 600 800
None
Little
Moderate
Good
Excellent
Number of students
0200400600800
Number of students
Students report that they understand
their content MUCH better after creating the LO
Student self-reported understanding of
LO content before and after LO submission
0 200 400 600 800
None
Little
Moderate
Good
Excellent
Number of students

29. • Major Challenge: How to determine whether students that
did an LO on a specific topic are actually primed to
understand content better?
• [Reminder] Requirements of new activity:
✓ Output: distributed over the whole term
✓ Load: at least 2h of student time
✓ Assessment: Submitted online, marked with rubric
★ TA load: No increased marking load on existing TAs
Future Directions

30. ★ TA load: No increased marking load on existing TAs
Option 2: Crowd-source LO assessment by
assigning LOs to be marked by students
Only two choices:
Option 1: Redistribute resources to reallocate
TA hours to marking and curating

31. 0 100 200 300
LO 1
LO 2
LO 3
LO 4
LO 5
LO 6
LO 7
LO 8
Number of students
Assigned
Optional
1
0 100 200 300 400
Less than 0.5h
0.5 to 1 h
1 to 2h
2 to 3h
3 to 4h
4 to 5h
More than 5h
Number of students
2
0 200 400 600 800
None
Little
Moderate
Good
Excellent
Number of students
0200400600800
Number of students
3
Participation Time Spent
Understanding