Learning analytics and evidence-based teaching and learning
1. Learning analytics and
evidence-based teaching and learning
Doug Clow
Institute of Educational Technology, The Open University, UK
slideshare.net/dougclow
Goldsmiths Learning & Teaching Conference 30 May 2014
2. What do you want to
hear most about?
a) What is learning analytics?
b) Some examples of learning analytics
c) Evidence-based practice
d) What can we do?
e) I’m only here for the next speaker
cc licensed ( BY ) flickr photo by Swaminathan: http://flickr.com/photos/araswami/2168316216/
4. “Big data is like teenage sex: everyone talks
about it, nobody really knows how to do it,
everyone thinks everyone else is doing it, so
everyone claims they are doing it…”
– Dan Ariely, Facebook, 6 Jan 2013
5. “Big data is like teenage sex: everyone talks
about it, nobody really knows how to do it,
everyone thinks everyone else is doing it, so
everyone claims they are doing it…”
– Dan Ariely, Facebook, 6 Jan 2013
… and the world of education seems
obsessed about it, but the little that does go on
is often done badly, and leaves people
disillusioned.
6. “feeding back the
data exhaust”
Big Data in
Education
Photo (CC)-BY Iain Watson http://www.flickr.com/photos/dagoaty/3329699788/
7. It’s not very big
but it may be clever
Photo (CC)-BY Paul and Cathy https://www.flickr.com/photos/becker271/7903353008
8. What is learning analytics?
• the measurement, collection, analysis and reporting of data about
learners and their contexts, for purposes of understanding and
optimising learning and the environments in which it occurs
– First International Conference on Learning Analytics And Knowledge (LAK11), Banff, Alberta, 2011
Photo (CC)-BY Cris: http://flickr.com/photos/chrismatos/6917786197/
9. Photo public domain: http://commons.wikimedia.org/wiki/File:DESYNebelkammer.jpg
Erik Duval
http://erikduval.wordpress.com/2012/01/30/learning-
analytics-and-educational-data-mining/
“collecting traces
that learners leave
behind and using
those traces to
improve learning”
12. • Predictive modeling, datamining (Blackboard)
• Place students in one of three risk groups => traffic light
• Trigger for intervention emails
• Dramatic retention improvements
• Consistent grade performance improvement
13. Social Network Analysis
• Social Networks Adapting Pedagogic Practice
• Network visualisations of forum activity data from VLE
• See patterns
• Spot central and
disconnected
• Identify at-risk
• Improve teaching
18. Evidence-based medicine
is the conscientious, explicit, and judicious use
of current best evidence in making decisions
about the care of individual patients.
–Sackett et al (1996)
Photo (CC)-BY photophilde: https://www.flickr.com/photos/photophilde/8127001284/
19. Evidence-based practice
is the conscientious, explicit, and judicious use
of current best evidence in making decisions
about the care of individual clients.
–Sackett et al (1997)
Photo (CC)-BY photophilde: https://www.flickr.com/photos/photophilde/8127001284/
20. Evidence-based teaching and learning
is the conscientious, explicit, and judicious use
of current best evidence in making decisions
about the care of individual learners.
Photo (CC)-BY photophilde: https://www.flickr.com/photos/photophilde/8127001284/
21. Randomised controlled trial
• Groups vary only in the thing you’re testing
• People assigned to groups at random
• Participants and researchers don’t know which
Photo (CC)-BY Kevin Dooley https://www.flickr.com/photos/pagedooley/6613526021/
22. cc licensed ( BY ) flickr photo by LASZLO ILYES: http://flickr.com/photos/laszlo-photo/4093575863/
Why so few RCTs in HE?
• Ethics
• Impracticality
• Complexity
23. Ethics: HIV/AIDS in 80s/90s
• Poor prognosis for HIV infection
• New, effective treatments
–Untested, unavailable
• New protocols
–Wider access
–Early endpoints to trials
Photo (CC)-BY Swami Stream https://www.flickr.com/photos/araswami/525922259/
24. Ethics: Vioxx
• New drug
• Pain relief & anti-inflammatory
–without stomach damage
• Heart attacks and strokes
• Withdrawn
• Other painkillers now under
suspicion
Photo (CC)-BY xJason.Rogersx https://www.flickr.com/photos/restlessglobetrotter/3058701116/
25. Practicality
• Online learning = data
• A/B testing
Photo (CC)-BY Jonathan Combe https://www.flickr.com/photos/jono566/8489053557/
26. Complexity • Important outcomes long
delayed
• Disagreement about end points
–Medicine: All-cause mortality
–Education: Passes, grades,
employment
Gentian sino-ornata Photo (CC)-BY reurinkjan https://www.flickr.com/photos/reurinkjan/3241158162/
• Richness of humanity
• Assessment Problem
28. Competition:
• Large established companies
• High-tech startups
• Private sector HE providers
Japanese Knotweed Fallopia japonica
Photo (CC)-BY Maja Dumat https://www.flickr.com/photos/blumenbiene/6146039333
30. • See what data you (can) have
about your students
• Move towards evidence-based practice
–Look for evidence before innovation
–Make new evidence
• Find out more about learning analytics
Photo (CC)-BY Wonderlane https://www.flickr.com/photos/wonderlane/3065525293/
31. www.laceproject.eu
• Evidence Hub
• Events: SoLAR Flare, 24 Oct 14
• Publications, briefings, webinars
Learning Analytics Community Exchange (FP7)
32. Thanks to:
People:
• OU Learning Analytics: IET Student Statistics and Survey Team, Gill Kirkup
and the other Data Wranglers, Kevin Mayles, Belinda Tynan, Simon
Buckingham Shum, Rebecca Ferguson, Bart Rientes
• LACE: Rebecca Ferguson, Simon Cross, Michelle Bailey, Rebecca Wilson,
partners.
Funders:
• LACE: EC 619424-FP7-ICT-2013-11
Vote!
I can talk for hours about this. It’s really exciting.
But we have 20 minutes.
I will talk about everything, but what should I spend most time explaining?
Hint: We can talk about what to do in the questions.Or if you lose the vote you can ask a question on the bit you want.
Intro, 3 salient points, conclusion
What is learning analytics, some examples, EBP. Complex.
Then actions.
… Big Data
LA is Big Data in education
And this is a terrible shame because with the right techniques and the right context, where there’s trust and enthusiastic consent, it can be pretty good.
It’s not really Big Data. (It fits on my laptop. Often in Excel.)
For the first time in educational history,
We have more data than theories.
Previously a generous supply of theory and precious little data.
Data mining, academic analytics, learner analytics – focus here is on the learning, not the management and administration of learning
International profile
Photo: Cloud Chamber at the German Electron Synchrotron DESY
Reflective learning cycle, Kolb
Without interventions: still good stuff: computer science, educational research, business intelligence
But only LA if fed back.
Speed and length of cycles: instant feedback as you learn, through to govt policy
This is what teachers – good teachers – have always done.
But more, and more electronic, and (since Piketty has made a knowledge of Marx trendy again):
a big enough quantitative change is a qualitative one.
Things are different.
But how?
Last 25y, transformation – why we’re living even longer.
Some tried and trusted treatments were killing people.
Widened out from just medicine to ‘practice’, but still mostly health (and social care).
Strong parallels.
Our tradition of evidence is different..
idealised
We have the ethics wrong.
Quantitative turn makes more practical – A/B testing
Complexity we have always with us.
Simplifying. Also US healthcare system.
Distinguishing the drugs that actually work (HAART) from comforting and quack treatments
Ethical principle: Get the evidence AND get the benefits.
Practical lesson: engage with RCT process
NOTHING NEW WITHOUT TESTING
Massive trials. Serious testing.
Heart attacks and strokes only came up in extended testing.
Ibuprofen and diclofenac may also have problems
Simplified! More going on here.
UNETHICAL FOR NEW TREATMENT WITHOUT TESTING
Progress towards is also valuable
A/B testing can control for some of it – switching within classes. BUT!
The Assessment Problem:
Not everything that can be measured counts, and not everything that counts can be measured
If we think it’s important but can’t assess it …?
Lenin’s vanguard party. Seize control of the means of production.
Gove is keen on RCTs
Gove is also keen on students not being pushed off cliffs.
You can’t make a reliable indicator by changing the sign of an unreliable one.
Reversed stupidity is not intelligence.
Senior managers care about bums on seats and money. They’re paid to. The good ones – most of them – also care about other things.
You care about teaching and learning. Get involved!
Knowing more about your students is the responsibility of any good teacher.
If Goldsmiths, the people you need to talk to are almost certainly in the room.
Logging facilities, analytics dashboards – may be turned off.