Slides accompanying a talk by Ari Bader-Natal (twitter: @aribadernatal) at the 1st International Conference on Learning Analytics and Knowledge.
More information about this work can be found on the Grockit research page: grockit.com/blog/main/research
This PowerPoint helps students to consider the concept of infinity.
Evolving a Learning Analytics Platform
1. Evolving
a Learning Analytics Platform
Ari Bader-Natal Thomas Lotze
ari@grockit.com thomas@grockit.com
@AriBaderNatal
February 28, 2011
2.
3. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
http://grockit.com
4. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
http://grockit.com
5. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
http://grockit.com
6. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
http://grockit.com
7. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
http://grockit.com
8. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
http://grockit.com
9. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
http://grockit.com
10. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
http://grockit.com
11. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
The discovery question
http://grockit.com
12. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
The discovery question
The production questions
http://grockit.com
13. Evolving a Learning Analytics System @AriBaderNatal
Questions to ask when evolving a platform:
The first question
The interesting questions
The infrastructure question
The subset question
The missing data question
The pro-active question
The effort question
The discovery question
The production questions
The audience question
http://grockit.com
14. Evolving a Learning Analytics System @AriBaderNatal
The context questions
What is Grockit?
Where? Why? How?
http://grockit.com
16. Evolving a Learning Analytics System Ari Bader-Natal
E-Learning
Intelligent Tutoring Systems
Computer-Supported
Collaborative Learning
http://grockit.com
18. Evolving a Learning Analytics System Ari Bader-Natal
System control
(adaptive study)
Learner control
(custom-specified study)
System control
(adaptive/IRT assessment)
http://grockit.com
21. Evolving a Learning Analytics System @AriBaderNatal
Social and motivational value of having a cohort of peers:
• opportunities to ask for assistance (just-in-time guidance)
• opportunities for discussion (both on-task and off-task)
• opportunities to earn recognition (for assisting others)
http://grockit.com
22. Evolving a Learning Analytics System @AriBaderNatal
Social and motivational value of having a cohort of peers:
• opportunities to ask for assistance (just-in-time guidance)
• opportunities for discussion (both on-task and off-task)
• opportunities to earn recognition (for assisting others)
http://grockit.com
27. Evolving a Learning Analytics System Ari Bader-Natal
Indicator of helpfulness
Recognition from
Recognition of peer learners
achievements
http://grockit.com
28. Evolving a Learning Analytics System Ari Bader-Natal
The first question
What are people actually doing?
http://grockit.com
29. Evolving a Learning Analytics System Ari Bader-Natal
Just look around in the database.
http://grockit.com
30. Evolving a Learning Analytics System Ari Bader-Natal
Just look around in the database.
•select count(*) from learners;
http://grockit.com
31. Evolving a Learning Analytics System Ari Bader-Natal
Just look around in the database.
•select count(*) from learners;
•select * from discussions;
http://grockit.com
32. Evolving a Learning Analytics System Ari Bader-Natal
Just look around in the database.
•select count(*) from learners;
•select * from discussions;
•select * from responses;
http://grockit.com
33. Evolving a Learning Analytics System Ari Bader-Natal
Just look around in the database.
•select count(*) from learners;
•select * from discussions;
•select * from responses;
•select avg(time) from responses
http://grockit.com
34. Evolving a Learning Analytics System Ari Bader-Natal
The interesting questions
Exploratory Data Analysis...
http://grockit.com
36. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
http://grockit.com
37. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
http://grockit.com
38. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
• Do students who work in groups spend more time on task than those
who choose to work alone?
http://grockit.com
39. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
• Do students who work in groups spend more time on task than those
who choose to work alone?
• Which of the various interventions within Grockit lead to the largest
learning gains?
http://grockit.com
40. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
• Do students who work in groups spend more time on task than those
who choose to work alone?
• Which of the various interventions within Grockit lead to the largest
learning gains?
• Do peer-awarded points motivate more discussion in small group
settings?
http://grockit.com
41. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
• Do students who work in groups spend more time on task than those
who choose to work alone?
• Which of the various interventions within Grockit lead to the largest
learning gains?
• Do peer-awarded points motivate more discussion in small group
settings?
• How does an Item Response Theory model compare to a Knowledge
Tracing model, in terms of predictive accuracy or responses?
http://grockit.com
42. Evolving a Learning Analytics System @AriBaderNatal
What is effective? What is engaging?
• How does frequency and duration of discussions during group study
sessions differ between high school students and post-college
learners?
• Do students who work in groups spend more time on task than those
who choose to work alone?
• Which of the various interventions within Grockit lead to the largest
learning gains?
• Do peer-awarded points motivate more discussion in small group
settings?
• How does an Item Response Theory model compare to a Knowledge
Tracing model, in terms of predictive accuracy or responses?
• Are video explanations more effective than written explanations?
http://grockit.com
44. Evolving a Learning Analytics System @AriBaderNatal
Question:
http://grockit.com
45. Evolving a Learning Analytics System @AriBaderNatal
Question:
Are the questions currently available appropriate for
the current population of learners?
http://grockit.com
46. Evolving a Learning Analytics System @AriBaderNatal
Question:
Are the questions currently available appropriate for
the current population of learners?
• Are there enough hard questions?
http://grockit.com
47. Evolving a Learning Analytics System @AriBaderNatal
Question:
Are the questions currently available appropriate for
the current population of learners?
• Are there enough hard questions?
• Are there enough easy questions?
http://grockit.com
48. Evolving a Learning Analytics System @AriBaderNatal
Question:
Are the questions currently available appropriate for
the current population of learners?
• Are there enough hard questions?
• Are there enough easy questions?
Actionable:
http://grockit.com
49. Evolving a Learning Analytics System @AriBaderNatal
Question:
Are the questions currently available appropriate for
the current population of learners?
• Are there enough hard questions?
• Are there enough easy questions?
Actionable:
If there isn’t a good fit, ask the content authors to start
focusing on writing what is currently most needed.
http://grockit.com
50. Evolving a Learning Analytics System @AriBaderNatal
The infrastructure question
How can we streamline analysis?
http://grockit.com
62. Evolving a Learning Analytics System @AriBaderNatal
The subset question
How can we ask the same question of
different subsets of the available data?
http://grockit.com
63. Evolving a Learning Analytics System @AriBaderNatal
Example views
http://grockit.com
64. Evolving a Learning Analytics System @AriBaderNatal
Multiple views at once.
http://grockit.com
65. Evolving a Learning Analytics System @AriBaderNatal
Multiple views at once.
Subject (e.g. Algebra)
Game User (e.g. Ari)
Question Game Session
Question Session
Response Chat Message
http://grockit.com
66. Evolving a Learning Analytics System @AriBaderNatal
Multiple views at once.
Subject (e.g. Algebra)
Game User (e.g. Ari)
Question Game Session
Question Session
Response Chat Message
http://grockit.com
71. Evolving a Learning Analytics System @AriBaderNatal
The pro-active question
How can we test out new hypotheses?
The effort question
How can we reduce friction in
experimentation and hypothesis testing?
http://grockit.com
72. Evolving a Learning Analytics System Ari Bader-Natal
http://xkcd.com/552/
http://grockit.com
73. Evolving a Learning Analytics System Ari Bader-Natal
R. Kohavi, R. Henne, and D. Sommerfield. Practical guide to controlled
experiments on the web. In Proceedings of the 13th ACM SIGKDD
International Conference on Knowledge Discovery and Data Mining. ACM,
2007.
http://bit.ly/exp-long
http://bit.ly/exp-short
Eric Ries: The one line split-test, or how to A/B test all the time.
http://bit.ly/one-line-split
http://grockit.com
75. Evolving a Learning Analytics System Ari Bader-Natal
InitialQuestions
Should new students be given easier questions when playing alone,
until they answer at least 5 questions correctly?
This test is being assigned and evaluated for new users only.
EasyFirst Normal
Subsequent activity (1657 persons) (1721 persons) (difference)
Participated in further study 73.2% (1213) 67.5% (1161) (2.6% to 8.9%) **
Participated in group study 31.4% (520) 27.0% (464) (1.3% to 7.5%) *
Logged in again later 30.8% (510) 28.4% (489)
Participated in group discussion 22.1% (366) 18.4% (316) (1% to 6.5%) *
Reviewed past questions 13.9% (230) 12.8% (221)
http://grockit.com
76. Evolving a Learning Analytics System @AriBaderNatal
The discovery question
How can we deliver findings effectively?
http://grockit.com
85. Evolving a Learning Analytics System Ari Bader-Natal
Performance (per-skill)
http://grockit.com
86. Evolving a Learning Analytics System @AriBaderNatal
My unresolved question
Is evolving the right approach?
http://grockit.com
87. Evolving a Learning Analytics System @AriBaderNatal
YAGNI: “You Ain’t Gonna Need It”
Ron Jeffries: “Always implement things when you actually need
them, never when you just foresee that you need them.”
http://grockit.com
88. Evolving a Learning Analytics System @AriBaderNatal
In the age of #BigData, how do you choose tools
for analysis before your dataset is undeniably Big?
Start with Big tools or YAGNI?
YAGNI: “You Ain’t Gonna Need It”
Ron Jeffries: “Always implement things when you actually need
them, never when you just foresee that you need them.”
http://grockit.com
89. Evolving a Learning Analytics System @AriBaderNatal
Your unresolved questions
???
http://grockit.com
90. Thanks!
Evolving
a Learning Analytics Platform
Ari Bader-Natal Thomas Lotze
ari@grockit.com thomas@grockit.com
@AriBaderNatal
February 28, 2011