University of Otago, February 2016

1. The University of Sydney Page 1
Bottom-up growth of learning analytics at two
Australian universities:
Empowering staff with actionable intelligence to improve student outcomes
@dannydotliu
danny.liu@mq.edu.au
danny.liu@sydney.edu.au

2. The University of Sydney Page 2
“We found that mature foundations for LA implementations were
identified in institutions that adopted a rapid innovation cycle
whereby small scale projects are initiated and outcomes quickly
assessed within short time frames. The successful projects in this
cycle are then further promoted for scale-ability and mainstream
adoption. In the context of LA, this small-scale seeded approach
appeared more effective in terms of organisational acceptance
and adoption than a whole of institution model attempting to roll
out a single encompassing program.”
Learning Analytics in Australia: Office for Learning & Teaching 2016

3. The University of Sydney Page 3
1. Macquarie: Empowering staff with actionable LMS data
2. Sydney: Learning analytics by stealth
3. Macquarie: Building institutional readiness through openness

4. Empowering staff with actionable LMS data
THE MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP)
February 2016

5. Learning analytics in Moodle
MOTIVATIONS AND BENEFITS
• Large(ish) classes, failure
and retention issues
• Staff familiarity with
Moodle, single point of
access
• Learning experience data
already centralised
• No available learning
analytics tool with
actionable data
Log viewer
Statistics report

6. Enhancing an existing plugin
6
MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP)
• Originally developed by Phillip Dawson, Adam Olley, and Ashley Holman
Moodle
Report
Logins Forums Assessments
Parameters
Traffic
lights
BYO
Moodle Engagement
Analytics Plugin
Indicators
Action

7. Involving staff
7
ACADEMICS AND STUDENT SUPPORT
Staff expectations around an
early alert system
Prototyping and development
User testing
Piloting
Feedback and further development

8. MEAP
8
CAPABILITIES AND APPLICATIONS

9. Stakeholder outcomes
9
PERSONALISED, DATA-DRIVEN INTERVENTIONS
• 3400+ personalised emails sent, average ~46% opened
0
50
100
150
200
250
300
350
01Aug
08Aug
15Aug
22Aug
29Aug
05Sep
12Sep
19Sep
26Sep
03Oct
Numberofstudents
0
20
40
60
80
100
AreyouhavingproblemsaccessingiLearn?
ACCG250-Ifyousnoozeyoulose!
AreyouhavingtroubleloggingintoiLearn?
AreyouhavingtroubleloggingintoiLearninACCG100?
MECH201:EngineeringDynamics-iLearnActivity
ENGG150quizfeedback
ECON110iLearnLogin
AHIS140MythintheAncientWorld
AHIS140MythintheAncientWorld
AreyouhavingtroubleloggingintoECON111
AHIS140Session2Participation
AHIS140MythintheAncientWorld
Quiz1
Quiz1Result-Afin253
Ilearn-Afin253
ACST101-Quiz1
ECON110TutorialSubmissions
ECON111-Tutorialscontributetoyourfinalgrade
ECON111-Tutorialscontributetoyourfinalgrade
MECH202Progressthusfar
MECH203Progressthusfar
ISYS114
ACST101-Lastdaytowithdrawwithoutfailchargeis…
ISYS114Performance
ISYS114weeklytutorialsubmissions
ISYS114AttendanceandPerformance
ISYS114WorkshopAttendanceandPerformance
ISYS114Workshopattendance
Tutorialfeedback
Tutorialparticipation
TutorialparticipationforEUL101SocietiesofEurope
ACST101-Mondayisthelastdaytowithdrawwithout…
ACST101-Mondayisthelastdaytowithdrawwithout…
ACST101-Mondayisthelastdaytowithdrawwithout…
Itsnottoolatetogethelp!
ACST101-Mondayisthelastdaytowithdrawwithout…
ECON111-Pleasecheckyourtutorialrecords
ECON111-pleasecheckyourtutorialrecords
Percentageopened
1 hour 1 day 1 weekEmail opened within:

10. Stakeholder outcomes
10
PERSONALISED, DATA-DRIVEN INTERVENTIONS
• Who: unit
convenors and
student support
staff
• What: census,
updates, reminders
• Why: predominantly
for at-risk students
• How: logins,
assessment
submissions, grades,
attendance
I was surprised someone cared/was actually monitoring, kind
of a weird, I don't know totalitarian/'people are watching you'
feeling? But in this situation I was happy.
He gave me specific advice and encouraged me and it made
me feel much better.
The email basically kicked me into gear and I completed all my
assessments post-email to a high level.
Very useful. I wouldn't have been able to do such a large scale
analysis and identify so many students without MEAP. I
wouldn't have been able to send them such tailored, structured
and consistent messages.

11. Stakeholder outcomes
11
THE DARK SIDE
• Issues around
• Message
composition
• Suggestions
• Student contexts
• Being labelled
I was contacted but in a manner that suggest I should drop out of
the course and not waste the convenor's time. She didn't ask
whether I was experiencing any issues outside of university,
but simply that I should transfer course if I can't handle the
workload.
Being labelled as lazy when you're doing your best and
don't have any other choice is quite sad.
Being aware and then being told of my own inadequacies is
confronting and, yes, does make me feel worse about life in
general. It's something I need to be told and is still that extra bit of
motivation.
The email was worded in a way that it the unit [convenor] was
trying to tell me I was doing horrible and should drop out and
didn't refer any help.

12. Stakeholder outcomes
12
PERSONALISED, DATA-DRIVEN INTERVENTIONS
• Was there an effect?
Online activity Online activityOnline activityOnline activity Online activity
Risk rating Risk rating
Change in risk rating
No email sent
Emailed but not opened
Opened email

13. Stakeholder outcomes
13
PERSONALISED, DATA-DRIVEN INTERVENTIONS
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
Forum Login
Changeinriskrating
*
Missed online quizzes & tutorial submissions
No email sent Emailed but not opened Opened email

14. Back to the users
14
FURTHER ITERATIVE DEVELOPMENT

15. Empowering staff with data
15
MOODLE ENGAGEMENT ANALYTICS PLUGIN (MEAP)
Moodle
Report
Logins Forums Assessments
Parameters
Traffic
lights
Gradebook
Moodle Engagement
Analytics Plugin
Action
Attendance

16. Lessons learned and next steps
16
LOOKING BACK AND LOOKING FORWARD
• Looking back
• Talk to the users
• Find champions
• Staff have varying levels of error tolerance
• LA is a political football
• Looking forward
• Further evaluating impact
• Production and wider trial
• Source code release
• New Moodle LA spec
https://docs.moodle.org/dev/Learning_Analytics_Specification

17. The University of Sydney Page 17
1. Macquarie: Empowering staff with actionable LMS data
2. Sydney: Learning analytics by stealth
3. Macquarie: Building institutional readiness through openness

18. The University of Sydney Page 18
Learning analytics by
stealth
The Student Relationship Engagement System
(SRES)
Standing out from a Crowd
SumAll CC BY-NC-ND 2.0
https://flic.kr/p/kYbv4C

19. The University of Sydney Page 19
The contexts of learning analytics
Common barriers to adoption
– Policy and ethical
challenges
– Culture of resistance to
change
– Vendor solutions
– Data accuracy
– One-size-fits-all
Pressing institutional needs
– ~$7 million/year lost to
attrition
– Larger class sizes
– More disconnected students
– Feedback very generalised

20. The University of Sydney Page 20
The Student Relationship Engagement System
Attendance
Interim
grades
LMS
metrics
Third party
tools
Other data
as needed Student
Relationship
Engagement
System

21. The University of Sydney Page 21
Personalising connections with students
– Empowering staff
– Flexible & intuitive
– Targeted and personalised
– Multi-channel
– Benefits
– Highly customisable
– Efficient – key data in one
place, operating at scale
– Connect staff and all
students (not just at-risk)
Student
Relationship
Engagement
System

22. The University of Sydney Page 22
Stakeholder outcomes
Discontinued
Failed
Passed
F
P
C
D
HD
1st year arts unit ~500 enrolment 1st year science unit ~1000 enrolment
“Many thanks Adam. Yes, things are going much better this semester. I really appreciate
how you keep in contact and keep an eye on us. It's such a big class, I don't know how you
do it.”
“Just to let you know that your emails really helped me survive last semester. I never
realised how big a change it would be from school.”

23. The University of Sydney Page 23
Co-evolution of the SRES
– Organic adoption by academics
– Co-evolution of capabilities
0
5000
10000
15000
20000
25000
0
10
20
30
40
50
60
70
2012 2013 2014 2015
Numberofstudents
Numberofunitsorschools
Number of units Number of schools Number of students
Pilot
EWS
introduced
EWS
integrated
New
analyses
More data
types
Data
import

24. The University of Sydney Page 24
Learning analytics by stealth?
– Co-evolving capabilities and competencies around data-driven
pedagogy and curriculum
Student
Relationship
Engagement
System

25. The University of Sydney Page 25
Learning analytics by stealth?
– Co-evolving capabilities and competencies around data-driven
pedagogy and curriculum
Student
Relationship
Engagement
System

26. The University of Sydney Page 26
The contexts of learning analytics
Common barriers to adoption
– Policy and ethical
challenges
– Culture of resistance to
change
– Vendor solutions
– Data accuracy
– One-size-fits-all
Pressing institutional needs
– ~$7 million/year lost to
attrition
– Larger class sizes
– More disconnected students
– Feedback very generalised

27. The University of Sydney Page 27
Lessons learned and next steps
– Looking back
– It was ugly but it worked
– Ease of use is important – does it save time?
– Attendance was a (surprisingly) popular metric
– Everyone uses a customisable system differently
– Personalisation at scale
– Looking forward
– Facilitate wider roll-out
– Further (re)developments – ML, student view, sub-messages, etc
– Research & evaluation of impact on students and staff

28. The University of Sydney Page 28
1. Macquarie: Empowering staff with actionable LMS data
2. Sydney: Learning analytics by stealth
3. Macquarie: Building institutional readiness through openness

29. Building institutional readiness
through openness
THE MACQUARIE OPEN ANALYTICS TOOLKIT (MOAT)
February 2016

30. Our approach
30
CONNECTING USERS WITH DATA THROUGH ANALYTICS
Macquarie
Open
Analytics
Toolkit
Data
Users
Analytics
LMS
Video
Classrooms
Mobile
Business
systems
External
courses
Understand students
Identify and predict
Code of practice
“LAMP Lighters”

31. Bringing data together
31
NUANCES OF LEARNING DATA
LMS
Video
Classrooms
Mobile
Business
systems
External
courses
Learning
Record
Store
(LRS)
Custom analytics engine

32. Empowering staff with data
32
MACQUARIE OPEN ANALYTICS TOOLKIT (MOAT)

33. Empowering staff with data
33
MACQUARIE OPEN ANALYTICS TOOLKIT (MOAT)

34. Lessons learned and next steps
34
LOOKING BACK AND LOOKING FORWARD
• Looking back
• Multi-disciplinary, multi-level teams
• System architecture choice is important
• Students are very open about data (unless it’s identifiable)
• Staff and students have a (surprisingly) good idea of what they want
• Looking forward
• LRS to production
• Piloting ‘dashboards’* with staff and students
• Working with xAPI community
• Open sourcing analytics tools

35. The University of Sydney Page 35
Lessons learned and
issues raised

36. The University of Sydney Page 36
Lessons learned and issues raised
– Give them what they want vs. build it and they will come
– Customisability is key
– Utility (eventually) trumps aesthetics (to an extent)
– But people still like shiny things
– Data are not enough – connect with pedagogical, pastoral
– Surprisingly little kickback about privacy & ethics
– Tension between research ethics & general ethics
– It’s tricky to measure impact
– Iterate – capabilities, implementation
– Focus on the human

37. The University of Sydney Page 37
Adoption pipeline
Colvin et al. (2015) Student retention and learning analytics: A snapshot of Australian
practices and a framework for advancement. Office of Learning and Teaching, Sydney.
First, implementers require an analytic tool or combination of tools that manage data
inputs and generate outputs in the form of actionable feedback… As these increasingly
meet the real needs of learners and educators the organisational uptake is accelerated.

38. The University of Sydney Page 38
Learning analytics is not an elixir for ineffective teaching,
nor does it reveal an ideal pedagogy; instead, it provides
data-driven tools or suggestions to help instructors make
changes that can be measured in terms of student outcomes.
Pistilli, M. D., Willis III, J. E., & Campbell, J. P. (2014). Analytics Through an Institutional Lens:
Definition, Theory, Design, and Impact. In Learning Analytics (pp. 79-102). Springer New York.

39. The University of Sydney Page 39
1. MEAP Empowering staff with actionable LMS data
Chris Froissard, Deborah Richards, Amara Atif
2. MOAT Building institutional readiness through openness
James Hamilton, Ed Moore, Yvonne Breyer et al.
3. SRES Learning analytics by stealth
Charlotte Taylor, Adam Bridgeman, Kathryn Bartimote-Aufflick et al.
@dannydotliu
danny.liu@mq.edu.au
danny.liu@sydney.edu.au