This document summarizes a study that evaluated the StepUp! learning analytics dashboard. It describes three courses included in the study, an open learning approach that collects learner activity traces, and the StepUp! dashboard. The study used a design-based research methodology to analyze learner issues, identify seven key issues from brainstorming, and determine how the dashboard could address these issues and support self-reflection.

1. ADDRESSING LEARNER
ISSUES WITH STEPUP!: AN
EVALUATION
José Luis Santos, Katrien Verbert,
Sten Govaerts, Erik Duval
http://www.slideshare.net/jlsantoso
1
Wednesday 10 April 13

2. WHAT DO WE HAVE?
• A context
• Courses
• Teaching approach and learning traces
• A learning dashboard
• A research methodology
• A model
• A question
• An evaluation
2
Wednesday 10 April 13

3. 3
Wednesday 10 April 13

4. 4
Wednesday 10 April 13

5. http://lift.do
5
Wednesday 10 April 13

6. 6
Wednesday 10 April 13

7. Govaerts, Sten; Verbert, Katrien, Duval, Erik; Pardo,
Abelardo, The Student Activity Meter for Awareness
and Self-reflection. Proceedings of CHI Conference
on Human Factors in Computing Systems, 2012,
Austin, Texas, ACM.
7
Wednesday 10 April 13

8. THREE COURSES
Course Students Work setup Topic
Thesis 11 Individual Different
Mume 20 Group 2-3 Same
Peno 36 Group 6 Same
8
Wednesday 10 April 13

9. OPEN LEARNING APPROACH
AND LEARNING TRACES
9
Wednesday 10 April 13

10. OPEN LEARNING APPROACH
AND LEARNING TRACES
9
Wednesday 10 April 13

11. OPEN LEARNING APPROACH
AND LEARNING TRACES
9
Wednesday 10 April 13

12. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

13. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

14. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

15. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

16. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

17. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

18. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

19. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

20. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

21. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

22. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

23. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

24. STEPUP!
http://ariadne.cs.kuleuven.be/stepup/
10
Wednesday 10 April 13

25. DESIGN-BASED
RESEARCH
11
Wednesday 10 April 13

26. DESIGN-BASED
RESEARCH
Requirement Design & Deployment
Analysis Evaluation Observation
11
Wednesday 10 April 13

27. DESIGN-BASED
RESEARCH
Surveys
RequirementGoogle Design & based Deployment
on
Analytics
quality and
Analysis
Analysis Evaluation
Tweets, Observation
blogs, time
11
Wednesday 10 April 13

28. 12
Wednesday 10 April 13

29. A MODEL
Verbert, K., Duval, E., Klerkx, J., Govaerts, S., Santos, J.L.: Learning analytics
dashboard applications. American Behavioral Scientist (2013)
http://abs.sagepub.com/content/early/2013/02/27/0002764213479363.abstract
12
Wednesday 10 April 13

30. EARLIER EXPERIMENT
13
Wednesday 10 April 13

31. EARLIER EXPERIMENT
Santos, J.L. , Verbert, K, Duval, E,. Empowering students to reflect on their activity with StepUp!: two case studies with
engineering students, Proceedings of ARETL'12 2nd Workshop on Awareness and Reflection, CEUR WS
13
Wednesday 10 April 13

32. EXPERIMENT
14
Wednesday 10 April 13

33. SET UP THREE
BRAINSTORMING
15
Wednesday 10 April 13

34. Discuss in groups of 3
or 4 their learning issues
15
Wednesday 10 April 13

35. Rate learning issues
15
Wednesday 10 April 13

36. RESULTS
• List of 34 issues
16
Wednesday 10 April 13

37. RESULTS
• List of 7 issues
17
Wednesday 10 April 13

38. 18
Wednesday 10 April 13

39. GOOGLE ANALYTICS
18
Wednesday 10 April 13

40. STEP UP! ENABLES ME TO ANALYZE HOW I SPEND
MY TIME
i3
Step Up! enables me to analyze how others spend their time
Step Up! helps me to identify group members who does not
work well
i1
19
Wednesday 10 April 13

41. I THINK THAT SOME STUDENTS MAY OVER-
REPORT THE TIME THEY WORK ON THIS COURSE
TO GIVE THE IMPRESSION THAT THEY MORE WORK
THAN THEY ACTUALLY DO
20
Wednesday 10 April 13

42. Step Up! helps me to plan my time better
i6
Step Up! helps me to realize when
something goes wrong in the course
i5
21
Wednesday 10 April 13

43. STEP UP! HELPS ME TO BE AWARE
WHICH RESOURCES AND TOOLS I
AND OTHERS USE.
i7
Step Up! helps me to gain some insight on how others
communicate
i2
22
Wednesday 10 April 13

44. I think that I am motivated for this course
StepUp! increases my motivation for the course
i4
23
Wednesday 10 April 13

45. I think that I am motivated for this course
StepUp! increases my motivation for the course
i4
23
Wednesday 10 April 13

46. STEP UP! PROMOTES COMMENTING ON OTHER
BLOGS
Step Up! promotes reading other blogs
StepUp! promotes twitter participation
24
Wednesday 10 April 13

47. WHAT WE HAVE LEARNT
• Methodology
• Different case studies
• individual vs group work
• same vs different work
• New objectives
• efforts vs outcomes
25
Wednesday 10 April 13

48. ONGOING WORK
http://navi-hci.appspot.com/
26
Wednesday 10 April 13

49. ONGOING WORK
27
Wednesday 10 April 13

50. @jlsantoso @svencharleer
Thanks!
Questions?
http://jlsantoso.blogspot.com
http://svencharleer.com
http://www.slideshare.net/jlsantoso
28
Wednesday 10 April 13