Analytics4Action Evaluation Framework: a review of evidence-based learning analytics interventions at the Open University UK

1. Keynote address
Analytics4Action Evaluation
Framework: a review of evidence-based
learning analytics interventions at the
Open University UK
20 October 2016
@DrBartRienties
Reader in Learning Analytics
A special thanks to Avinash Boroowa, Aida Azadegan, Shi-Min Chua, Simon Cross, Rebecca Ferguson, Lee Farrington-Flint, Christothea Herodotou, Martin
Hlosta, Wayne Holmes, Garron Hillaire, Simon Knight, Nai Li, Vicky Marsh, Kevin Mayles, Jenna Mittelmeier, Vicky Murphy, Quan Nguygen, Tom Olney, Lynda
Prescott, John Richardson, Jekaterina Rogaten, Matt Schencks, Mike Sharples, Dirk Tempelaar, Lisette Toetenel, Thomas Ullmann, Denise Whitelock, John
Woodthorpe, Zdenek Zdrahal, and others…A special thanks to Prof Belinda Tynan for her continuous support on analytics at the OU UK

2. (Social) Learning Analytics
“LA is the measurement, collection, analysis and reporting of data about learners
and their contexts, for purposes of understanding and optimising learning and the
environments in which it occurs” (LAK 2011)
Social LA “focuses on how learners build knowledge together in their cultural
and social settings” (Ferguson & Buckingham Shum, 2012)

3. Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012). Design and Implementation of a Learning Analytics Toolkit for Teachers. Journal of Educational Technology & Society, 15(3), 58-76.

4. So what does the OU UK do in terms of
interventions on learning analytics?
1) Exemplar of “small
intervention”
2) Large scale adoption
of predictive analytics
to help teachers to
identify “students at
risk”

5. 5

6. Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action Evaluation Framework: a review of evidence-based learning
analytics interventions at the Open University UK. Journal of Interactive Media in Education, 1 (2) 1-13.

7. 7
Analytics4Action framework
Implementation/testing methodologies
• Randomised control trials
• A/B testing
• Quasi-experimental
• Apply to all
Community
of inquiry
framework:
underpinning
typology
Menu of response
actions
Methods of
gathering data
Evaluation Plans
Evidence hub
Key metrics and
drill downs
Deep dive
analysis and
strategic insight
Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action Evaluation Framework: a review of evidence-based learning
analytics interventions at the Open University UK. Journal of Interactive Media in Education, 1 (2) 1-13.

9. 9
Menu of actions
Learning design (before start) In-action interventions (during module)
Cognitive Presence  Redesign learning materials
 Redesign assignments
 Audio feedback on assignments
 Bootcamp before exam
Social Presence  Introduce graded discussion forum activities
 Group-based wiki assignment
 Assign groups based upon learning analytics
metrics
 Emotional questionnaire to gauge students’
emotions
 Introduce buddy system
 Organise additional videoconference sessions
 One-to-one conversations
 Cafe forum contributions
 Support emails when making progress
Teaching Presence  Introduce bi-weekly online videoconference
sessions
 Podcasts of key learning elements in the module
 Screencasts of “how to survive the first two weeks”
 Organise additional videoconference sessions
 Call/text/skype student-at-risk
 Organise catch-up sessions on specific topics that
students struggle with

10. Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action Evaluation Framework: a review of evidence-based learning
analytics interventions at the Open University UK. Journal of Interactive Media in Education, 1 (2) 1-13.

11. Large scale adoption of predictive learning analytics
• 10 modules used predictive learning analytics
• 240 teachers had access to OUA vs. 613
teachers who did not
• 4320 students with OUA, 12713 without
• 70 teachers received a weekly reminder (email)
notifying them that the OUA predictions were
available through OUA dashboard
• 170 received the OUA weekly predictions via
email in excel
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M. (Submitted: 19-10-2016). Using Predictive Learning Analytics to Support Just-in-time
Interventions: The Teachers' Perspectives across a large-scale implementation.

12. So did it make a difference?
• In 7 out of 10 modules
there was no
difference
• In 3 pass rates were
higher

13. So did teachers use PLA?
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M. (Submitted: 19-10-2016). Using Predictive Learning Analytics to Support Just-in-time
Interventions: The Teachers' Perspectives across a large-scale implementation.

14. So what did teachers experience?
Emerging themes Explanation
Actual uses of the OUA dashboard ₋ Features of the OUA dashboard that are used by teachers
₋ Frequency of use
₋ Ease of use
Perceived usefulness of OUA ₋ Teachers' perceptions of the system as being useful or not
₋ In what respect the system support teaching practices
Approaching students at risk ₋ How teachers react to students flagged as being at risk
₋ What intervention strategies they devise to support students
Informing learning design ₋ How OUA could be used to inform the design of a module
Improvements to OUA ₋ Teachers’ suggestions for improving OUA
Intention to use OUA in the future ₋ Teachers’ intentions in terms of using OUA in the future
Table 2: Emerging themes identified through thematic analysis
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M. (Submitted: 19-10-2016). Using Predictive Learning Analytics to Support Just-in-time
Interventions: The Teachers' Perspectives across a large-scale implementation.

15. Conclusions (Part I)
1. Learning analytics can help teachers to
find mismatches in design and
learners’ needs, behaviors, and
expectations
2. Teachers can make a difference when
intervening and LA can help to track
the success of those interventions.

16. Conclusions (Part II)
1. Power of predictive analytics depends on
how teachers (and students) are using data
2. Lack of technology acceptance and
implicit/explicit academic resistance might
limit power of LA
3. Professional development of staff and
strategic support from senior management
needed to make LA a success

17. Keynote address
Analytics4Action Evaluation
Framework: a review of evidence-based
learning analytics interventions at the
Open University UK
20 October 2016
@DrBartRienties
Reader in Learning Analytics
A special thanks to Avinash Boroowa, Aida Azadegan, Shi-Min Chua, Simon Cross, Rebecca Ferguson, Lee Farrington-Flint, Christothea Herodotou, Martin
Hlosta, Wayne Holmes, Garron Hillaire, Simon Knight, Nai Li, Vicky Marsh, Kevin Mayles, Jenna Mittelmeier, Vicky Murphy, Quan Nguygen, Tom Olney, Lynda
Prescott, John Richardson, Jekaterina Rogaten, Matt Schencks, Mike Sharples, Dirk Tempelaar, Lisette Toetenel, Thomas Ullmann, Denise Whitelock, John
Woodthorpe, Zdenek Zdrahal, and others…A special thanks to Prof Belinda Tynan for her continuous support on analytics at the OU UK