Talk given at the Computers and Learning Research Group (CALRG) annual conference, 12 June 2013, at The Open University, UK. This presentation briefly reviews learning analytics, using some key examples. It then assesses what the OU is doing, and then sets out some ideas for what the OU could do in future to harness the potential of data about our learners to improve their learning.

1. Learning Analytics:
What it is, where we are …
Doug Clow
CALRG Conference, 12 June 2013
This work by Doug Clow is licensed under a Creative Commons Attribution 3.0 Unported License.
… and where we could go.

2. What it is
Where we are
Where we could go

3. Quick Quiz:
Learning Analytics
a) This is the first time I’ve heard of it
b) I know it’s one of the latest buzzwords
c) I’ve heard of some projects
d) I’m doing some projects
e) I have to keep turning down keynote
invitations
on this subject
cc licensed ( BY ) flickr photo by Swaminathan: http://flickr.com/photos/araswami/2168316216/

4. What it is

5. What is learning analytics?
• the measurement, collection, analysis and reporting of
data about learners and their contexts, for purposes of
understanding and optimising learning and the
environments in which it occurs
– First International Conference on Learning Analytics And Knowledge (LAK11), Banff, Alberta,
Feb 27-Mar 1 2011.
cc licensed ( BY ) flickr photo by Cris: http://flickr.com/photos/chrismatos/6917786197/

6. • An abundance of data
• Qualitative and quantitative
• To improve learning

7. • Learning analytics
• Learner analytics
• Academic analytics
• Business Intelligence
• Educational Data Mining
(cc) gareth1953 http://www.flickr.com/photos/gareth1953/5477477947/
• Web analytics
• Social network analysis
• Predictive modeling
• Latent semantic analysis
• … and more!

8. • Predictive modeling, datamining (Blackboard)
• Fit students in to one of three risk groups
=> traffic light
• Trigger for intervention emails
• Consistent grade performance improvement
• Dramatic retention improvements

9. Social Network Analysis
• Social Networks Adapting Pedagogic Practice
• Network visualisations of forum activity data from VLE
• See patterns
• Spot central and
disconnected
• Identify at-risk
• Improve teaching

11. Content/semantic analysis
• Lárusson and White, 2012

12. • Santos, Govaerts,
Verbert and Duval,
2012
Usage tracking

13. Social learning analytics
• Ferguson and Buckingham Shum, 2012

14. Clow, LAK12, 2012

15. (Copyright status hard to verify)

16. Where we are

17. ‘Traditional’ data sources
• IET Student Statistics and Surveys Team
–Student data: demographics, qualification
aim, modules taken, results, etc
–Student feedback data: End of Module
Survey and others (all change)
• Information Office (Planning, Forecasts)
• Marketing
• VOICE
• CIRCE, PLANET
• CAMEL, FRODO, ISDES, PIMS, etc …
cc licensed ( BY ) flickr photo by Walmart: http://flickr.com/photos/walmartcorporate/5391507982/

18. New-ish sources
• Learning delivery reports
• Marketing analytics
• Open media analytics
• Web analytics (DAX) …

19. VLE data

20. VLE data

21. IET Data Wranglers
• Student feedback
• VLE usage
• Learning delivery
reports

23. Where we could go

24. Clow, LAK12, 2012

25. Improving feedback
(cc) Doug Clow http://dougclow.com

26. Student Support Tool
• Team specify the most important activities in a module
–What activities are important, and when
• Tool shows
–Which students have done the important activities
–General usage indicators, with comparison
cc licensed ( BY ) flickr photo by Mike Baird: http://flickr.com/photos/mikebaird/398077070/

27. Course
Signals
for the
OU
cc licensed ( BY SA ) flickr photo by Fin Fahey: http://flickr.com/photos/albedo/97586501/

28. A/B Testing
• Find out what works
–At a smaller, more fine-grained level
–In a matter of weeks, not years
• Settle arguments about how to teach
–With data from actual students!
cc licensed ( BY ) flickr photo by LASZLO ILYES: http://flickr.com/photos/laszlo-photo/4093575863/

29. Bad reasons for not testing Why we should test
We’ve never done it before. Others are doing it. And we have done it,
just small-scale and unsystematically.
We know what’s right already. No. We think we know what’s right.
We might find out we were wrong. Better to find out sooner and improve
faster.
It’ll deskill academics. It’ll help academics do even better.
It’s unethical. It’s unethical not to. Ask doctors.
It’ll cost too much. It’s costing us too much already.
We don’t know how. There is a way to change our capacity to
act: Learning.

30. Good reasons for not testing What we should do about it
Students might not want to be
experimented on.
Explain the benefits. Only test on people
giving informed consent.
It’ll make things more complicated. Design tests carefully to minimise
disruption.
We don’t agree about what’s a
good outcome: retention, pass
rate, progression, attainment,
improvement as a human being, ...
Track multiple outcomes. Work towards
consensus – if we can’t agree what counts
as good we’re in trouble.
You can ‘game’ outcomes – e.g. a
too-easy exam gets ‘good’ results.
Don’t do that. Maintain quality procedures.
What if we start testing and one
version is much, much better?
If the evidence is strong enough, stop the
trial early. (There are sums for this.)
We don’t have the systems, or the
culture, to do systematic testing.
We should build them.

31. Us
Them

35. Learning analytics at the OU:
•We’re doing some interesting things
•We’re going to do more
•We could and should do a lot, lot more
Doug Clow
@dougclow
dougclow.org
doug.clow@open.ac.uk
cc licensed ( BY ) flickr photo by Vince Alongi: http://flickr.com/photos/vincealongi/2537227873/
Systematic innovation & improvement
… through evidence-based practice