This document discusses challenges and opportunities in learning analytics. It addresses what should be measured, issues around instrumentation, capturing interventions and social networks. It also discusses challenges around scaling qualitative analysis, temporal processes, longitudinal studies, privacy, ethics and data sharing. Overall, the document advocates that learning analytics should go beyond just measuring outcomes and page access counts, and should seek to understand learning processes, contexts and the effects of students' own decisions.
9. Feedback loops between
students and instructors
are missing!
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational
research, 77(1), 81-112.
17. Population Diversity
100%
90%
80%
70%
ACCT 1 (n = 746)
BIOL 1 (n = 220)
60%
BIOL 2 (n = 657)
50%
COMM 1 (n = 499)
COMP 1 (n = 242)
40%
ECON 1 (n = 661)
30%
GRAP 1 (n = 192)
MARK 1 (n = 723)
20%
MATH 1 (n = 194)
10%
0%
Females
International
students
Other
Living in nonlanguage at
urban
home
Part time
student
Previously
enrolled to a
course
Early access Did not access Late access
18. LMS Functionality Diversity
ACCT 1
Light Box Gallery
Forum
Course
Resource
Turn-it-in
Assignment
Book
Quiz
Feedback
Map
Virtual Classroom
Lesson
Glossary
Chat
X
X
X
X
X
X
X
BIOL 1
X
X
X
BIOL 2
X
X
X
X
X
X
X
X
X
X
COMM 1 COMP 1 ECON 1
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
GRAP 1
X
X
X
MARK 1 MATH 1
X
X
X
X
X
X
X
X
X
X
X
X
X
19. Predictive Power Diversity
100.00%
90.00%
80.00%
70.00%
60.00%
Model 1
50.00%
Moodle
40.00%
Model 1 + Moodle
30.00%
20.00%
10.00%
0.00%
All courses ACCT 1
together
BIOL 1
BIOL 2
COMM 1
COMP 1
Model 1 – demographic and socio-economic variables
* - not statistically significant
ECON 1
* GRAP 1 MARK 1
MATH 1
20. Retention is not
the only challenge
It is important, of course!
But, where is learning?
23. Learning Analytics – What?
Measurement, collection,
analysis, and reporting of data
about learners and their contexts
24. Learning Analytics – Why?
Understanding and optimising
learning and the environments
in which learning occurs
25. Modern Educational Psychology
Human agency is
central to learning
Bandura, A. (1989). Human agency in social cognitive theory. American
psychologist, 44(9), 1175-1184.
28. Why does it matter?!
Challenge
Metacognitive skills
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-Regulated Learning: Beliefs, Techniques, and
Illusions. Annual Review of Psychology, 64, 417-444. doi:10.1146/annurev-psych-113011-143823
29. Why does it matter?!
Challenge
Information seeking skills
Judd, T., & Kennedy, G. (2011). Expediency-based practice? Medical students’ reliance on Google and
Wikipedia for biomedical inquiries. British Journal of Educational Technology, 42 (2), 351-360.
doi:10.1111/j.1467-8535.2009.01019.x
30. Why does it matter?!
Challenge
Sensemaking paradox
Butcher, K. R., & Sumner, R. (2011). Self-Directed Learning and the Sensemaking Paradox. Human–
Computer Interaction, 26(1-2), 123-159. doi:10.1080/07370024.2011.556552
31. Why does it matter?!
Challenge
Asking questions and critical thinking
Graesser, A. C., & Olde, B. (2003). How does one know whether a person understands a device? The
quality of the questions the person asks when the device breaks down. Journal of Educational
Psychology, 95(3), 524–536..
32. Process and context focus
for learning analytics needed
to understand learning
36. Cognitive presence
the extent to which the participants in any
particular configuration of a CoI are able to
construct meaning via sustained communication
Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical Thinking and Computer Conferencing: A Model
and Tool to Assess Cognitive Presence. American Journal of Distance Education ,15(1), 7-23.
37. Effect size of the moderator role on
critical thinking
Cohen’s d = 0.66
38. Effect size of an intervention on
critical thinking in online discussions
d = 0.95 (non-moderators)
and
d = 0.61 (moderators)
39. Cognitive presence
TMA1
TMA2
TMA3
TMA4
Final
Control group
Cognitive Presence in Online
Discussions – Association w/ Grades
Triggering event
Exploration
Integration
Resolution
Other
-.226
-.001
.128
.201
-.028
.005
.141
.060
.027
.078
-.046
.009
.034
-.023
.113
-.050
-.037
.043
-.054
.106
-.010
.048
.113
.074
.154
** p
< 0.01; * p < 0.05
40. Cognitive Presence in Online
Discussions – Association w/ Grades
Intervention
group
Control group
Cognitive presence
** p
TMA1
TMA2
TMA3
TMA4
Final
Triggering event
Exploration
Integration
Resolution
Other
Triggering event
Exploration
-.226
-.001
.128
.201
-.028
.149
.216
.005
.141
.060
.027
.078
-.077
.197
-.046
.009
.034
-.023
.113
-.070
.163
-.050
-.037
.043
-.054
.106
.000
.223
-.010
.048
.113
.074
.154
.016
.243
Integration
.156
.396**
.417**
.338*
.454**
Resolution
Other
-.041
.219
.060
.046
.154
.050
.083
.075
.129
.088
< 0.01; * p < 0.05
41. Integration posts:
effect on final grades
100
80
60
40
20
0
Q1
Q2
Q3
p < .001, Q1 vs. Q2; Q1 vs. Q3, Q1 vs. Q4
Q4
45. Self-reflections in video annotations
1800
1600
1400
1200
Course 1 (non-graded)
1000
Course 2a (graded)
Course 2b (graded)
800
Course 3 (graded)
600
Course 4 (non-graded)
400
200
0
Cognitive processes Perceptual processes
Positive emotions
Negative emotions
Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and
computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24-54.
50. Learning Analytics
What to measure?
We don’t need page access counts only!
Wilson, T.D. (1999). Models in information behaviour research.
Journal of Documentation, 55(3), 249 - 270, doi:10.1108/EUM0000000007145
53. Motivation in
Information Interaction
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced
goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004
54. Motivation in
Information Interaction
Achievement goal
orientation (2x2)
Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced
goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004
55. Technology and
process of self-regulated learning
Siadaty, M. (2013). Semantic Web-Enabled Interventions to Support Workplace Learning,
PhD Thesis, Simon Fraser University, Surrey, BC, Canada.
60. Building data-driven culture in
institutions
Manyika, J., et al., Big Data: The Next Frontier for Innovation, Competition, and Productivity,
2011, McKinsey Global Institute, http://goo.gl/Lue3qs
Students generally have poor self-regulation skills:Weak metacomprehension – assessment of own knowledge – stop learning, when they don’t know enoughConfusion of the rate of learning - stop learning, when they don’t know enoughExternally-generated self-monitoring prompts – AzevedoWeak metacognitive awareness – inefficient study tactics used
Use of unreliable sources Poor querying skills
Students are asked to seek information about domains they do not have sufficient background knowledge They will stop seeking information even if the proper one hasn’t been found
Students are asked to seek information about domains they do not have sufficient background knowledge They will stop seeking information even if the proper one hasn’t been found