This document describes research on developing a Student Activity Meter tool to help students self-reflect and help teachers improve awareness of student activities. The tool tracks student interactions with an online learning system and visualizes patterns of activity. Three evaluations with students and teachers found that the visualizations were generally understandable and useful for the objectives of self-monitoring, awareness, and resource recommendation. Based on feedback, improvements will be made and larger evaluations conducted to further validate the tool's usefulness.

1. VISUALIZING ACTIVITIES FOR
SELF-REFLECTION AND
AWARENESS
Sten Govaerts
Katholieke Universiteit Leuven (Belgium)
http://www.role-project.eu

2. OVERVIEW
• Research outline
• Student Activity Meter
• Objectives
• Design & Implementation
• Three Evaluations
• Conclusion

5. RESEARCH QUESTION?
• canvisualizations assist
teachers to improve
awareness?
• canvisualizations assist
students with self-reflection?

6. PROBLEM

7. PROBLEM

8. PROBLEM

9. PROBLEM

10. OBJECTIVES
• self-monitoring for learners
• awareness for teachers
• time tracking
• learning
resource
recommendation

11. STUDENT ACTIVITY MONITOR

12. STUDENT ACTIVITY MONITOR

13. STUDENT ACTIVITY MONITOR

14. 3 EVALUATIONS
• with CS students
• withCGIAR courses and
teachers
• with
Learning and Knowledge
Analytics course participants

15. WHAT?
• usability
• user satisfaction
• for what can the users use the visualizations? (usefulness)
• does this match the objectives?

16. HOW?
• paper mockups
• implementation
• usability studies
• surveys
• designbased research: give student tool to teachers to
evaluate

17. DATA & DEPLOYMENTS
• Contextualized Attention Metadata
(CAM) data
• scripts
to extract CAM data from
Moodle

18. CS STUDENTS CASE STUDY
• usability and user satisfaction evaluation
• 12 CS students
•2 evaluation sessions:
• task based interview with think aloud
(after 1 week of tracking)
• user satisfaction (SUS & MSDT)
(after 1 month)

19. USABILITY & USER
SATISFACTION
• in general, people understand the visualizations well!
• some small issues were uncovered...
• average SUS score: 73% (stdv: 9,35)

20. USABILITY & USER
SATISFACTION

21. USABILITY & USER
SATISFACTION

22. USABILITY & USER
SATISFACTION

23. USABILITY & USER
SATISFACTION

24. USABILITY & USER
SATISFACTION

25. CGIAR CASE STUDY

26. CGIAR CASE STUDY

27. CGIAR CASE STUDY

28. CGIAR CASE STUDY

29. CGIAR CASE STUDY

30. CGIAR CASE STUDY
wants to more details
search for detect outliers on student
students
good indicator for effort understand the workload
more metrics use for course design optimization
obtain course overview
compare students
increase awareness
want better 1 to 1 progress evolution
comparison tool

31. LAK CASE STUDY
• open course on learning and knowledge analytics
• visual analytics enthousiasts + experts (who can also teach)

32. LAK CASE STUDY
• open course on learning and knowledge analytics
• visual analytics enthousiasts + experts (who can also teach)

33. LAK CASE STUDY
verify the status of the more metrics
classroom activity
chronological course dwell time
self-reflection to measure
progress and increase
motivation find students experiencing
problems and low engagement
more data

34. IDEA’S FROM EXPERTS

35. IDEA’S FROM EXPERTS
2
5
4
3
1

36. FUTURE WORK
• extend SAM with feedback of
experts
• evaluate
the usefulness in a larger
group of students
• offer from Tallinn University
• evaluate
an improved version of
SAM with teachers.

37. THANK YOU!
QUESTIONS?
slides will appear on http://www.slideshare.net/stengovaerts