Overview of Effective Learning Analytics Using data and analytics to support students at Open University

Jisc Learning Analytics Research
Group
Open University, Milton Keynes,
26th June 2019
Effective Learning Analytics
Using data and analytics to support
students

“The UK experience”: Health and Safety
• No fire drill today ☺
• Toilets
• Jennie Lee Building
• Wifi: Eduroam or free open iCloud

Leading global distance learning, delivering high-quality education to anyone, anywhere, anytime
The Open University
Largest
University
in Europe
No formal
entry
requirements
enter with one
A-level or less
33%
38%
of part-time
undergraduates
taught by OU in UK
173,927 formal
students
55%
of students are
'disadvantaged'
FTSE 100 have
sponsored staff on OU
courses in 2017/8
60%
66%
of new
undergraduates
are 25+ 1,300
Open University students
has a disability (23,630)
1 in 8
Students are
already in work
3 in 4
employers use
OU learning
solutions to
develop
workforce

13,206
learners
6,000
learners
123,000
learners
• A pioneering Moodle-based award winning site providing free access to learning – 1,000+ courses, plus articles and videos.
• The OU publishes curriculum as short courses onto OpenLearn and creates free Badged Open Courses (BOCs) which
reward informal learners with a badge / certificate of achievement.
• Links from BBC broadcasting with themed pages
developed to support the content of many series. In addition,
print resources can be ordered such as posters.
7.8m
new learners
each year
60m
since launch
in 2006
Opening Access: OpenLearn

Innovative and engaging content for
broadcast audiences and rich resources for
students. 2016/17: the OU co-produced 35
series:
• Generated 250 million viewing and
listening events across channels and
platforms.
• Directed 1.2 million viewers to
OpenLearn.
Dedicated channel on YouTube with bite-
sized learning. It is the largest educational
presence on YouTube in EU with:
• Over 1,700 public videos with 50 million
views.
• Over 166,000 subscribers to our channel,
more than any other UK educational
institution
• Reaching over 3 million learners per year.
The OU now engages with learners
on Facebook:
• Viewed 6.2 million times by 2.5
million users.
• We run Facebook live sessions to
engage around topical issues.
• We will be doing more with
Facebook on the DfE Flexible
Learning Fund in 2019.
Reaching out to millions more
250m
views
2016/17
3m
learners
per year
2.5m
in
2017

• 2m transactions a day
• 6m quiz questions answered a year
• 4500 tutors
• Groups of 15-20
• 8-16 hours a week
• Home, work, trains, prisons, submarines
• 130 degrees and other qualifications
• From 450+ modules
Study Experience Overview

Adeniji, B. (2019). A Bibliometric Study on Learning Analytics. Long Island University. Retrieved from https://digitalcommons.liu.edu/post_fultext_dis/16/

Update on LA at the OU
Begona Nunez-Herran and Kevin Mayles (Data and Student Analytics), Rebecca Ward
(Data Strategy and Governance)
-Move towards centralised LA data infrastructure
-Data governance and lessons learned
Prof Bart Rienties & PhD students (Institute of Educational Technology)
-What is the latest “blue sky” learning analytics research from the OU?
Prof John Domingue (Knowledge Media Institute) & Dr Thea Herodotou (IET)
-What have we learned from 5 years of large scale implementation of OU Analyse?
-Where is LA/AI going?

STUDENT SUCCESS ANALYTICS
13
O R G AN I S AT I O N AL C APAB I LT I E S
Productionised
output and MI
Strategic
analysis
Modelling /
AI
Data
collection
Data storage
and access
Technology
architecture
Learning
design and
delivery
Student
lifecycle
managemen
t
Continuous
improvemen
t and
innovation
Creation of
actionable
insight
Availability
of data
Impact the
student
experience
Adapted from Barton and Court (2012) - https://hbr.org/2012/10/making-advanced-analytics-work-for-you

HOW ANALYTICS SUPPORTS STUDENT SUCCESS
STUDENT SUCCESS
ANALYTICS KPIs
STUDENT SUCCESS
PRIORITIES DATA & ANALYTICS WILL SUPPORT, ALIGN TO, AND PROACTIVELY DRIVE THE RECRUITMENT
AND STUDENT SUCCESS STRATEGY THROUGH CONTINUOUS INNOVATION
CONVERSION
STUDY
CONTINUATION
EMPLOYABILITY
STUDY
ENGAGEMENT
WEBSITE APP ALVLECHATBO
T
PHONECOMMUNICATION
CHANNELS
DATA
ANALYTICS
CAPABILITY
CRM & DIGITAL (ANALYTICS) INTEGRATION LAYER
ENTERPRIS
E DATA
HUB
BI
ENABLEMENT
DATA SCIENCE
ENABLEMENT
ANALYTICS
INDUSTRIALISATION
DATA SECURITY, PRIVACY & GOVERNANCE
Build Your Own / Self-serveStrategic Analysis Canned Reporting
BUSINESS INTELLIGENCE
DIGITAL EXPERIENCE OFFLINE CHANNELS
Statutory Reporting
ANALYTICS & DATA SCIENCE
Modelling, AI &
Machine
Learning
Test & LearnKnowledge
Graph
Survey Analytics
Sentiment
Analysis
Trigger Based
Interventions
Digital Journey
Optimisation
Personalised
Prospect Targeting
Personalised
Recommendations
ANALYTICS CAPABILITY
14
SUPPORTED
OPEN ENTRY
IMPROVED
COMMUNICATIONS
FLEXIBLE STUDY
INTENSITY
INCREASED RETURN
RATES
INCREASED YR3
CONTINUATION RATES
INCREASED
SATISFACTION
IMPROVED
PERSONAL
OUTCOMES
INCREASED PASS
RATES
IMPROVED CAREER
OUTCOMES
D ATA & AN ALY T I C S V I S I O N

THE STUDENT SUCCESS ANALYTICS ROADMAP
15
E N T E R P R I S E D A T A H U B P L A T F O R M ( T I T A N ) – D A T A A N A L Y T I C S E N A B L E M E N T
1. BUSINESS INTELLIGENCE 2. STRATEGIC ANALYSIS 3. DATA SCIENCE 4. ANALYTICS
INTEGRATION
5. CONTINUOUS INNOVATION
Build a unified reporting
capability that allows
stakeholders to derive insights
into the key drivers of Student
Success whilst creating more
efficient use of analytics
resource. E.g. what trends are
influencing study intensity, do
we need analysis to see what
this implies for the OU
curriculum and learning
experience?...
Provide in depth, deep-dive
analyses around insights and
hypotheses obtained from the
BI capability to form the basis
of strategic actions. E.g. We
are seeing an increase in
students studying in a full-time
manner, but some are
successful and others less so.
They manner and subjects
studied is clearly a factor in
success outcomes…
Build advanced targeting
capabilities based on the
insights derived from the
analysis deep dives that can be
used in a scalable manner to
support student choices and
experience. E.g. a machine
learning algorithm that
recommends study choices and
patterns based on past student
success trends…
Build the means to turn the
recommendations from data
science techniques into real-life
digital experience and CRM
interactions. E.g. Industrialise
the recommended module
choices through production
level processes and integration
capability into Digital and to
F2F channels…
Continuously enable the latest
data technologies and
techniques to enhance existing
and new means of improving
Student Success. E.g. Build a
Knowledge Graph that enriches
our understanding of data
relationships, improving the
effectiveness of our data
science targeting accuracy…
Deliver a modern data analytics platform that supports all of the analytical capabilities highlighted for the roadmap. It starts by improving the accuracy,
stability and understanding of the OU’s data through effective governance, allows the OU to capture and leverage all types of data assets for use in
data analytics, delivers high performing and flexible BI functionality, offers the latest in open source data science capability, has the ability to integrate
with digital experience and offline CRM channels, and is constantly able to provide the latest in new data analytical technologies…
Create foundational information
and resource efficiency…
Develop in-depth insight to
influence strategic direction…
Create intelligence to support
student learning & decisions…
Deliver intelligence through key
communication channels…
Leverage new analytics trends
to improve student success…
DRIVES RESULTS & SUPPORTS
INNOVATION
DETECTING OPPORTUNITY SIZING OPPORTUNITY MAXIMISING OPPORTUNITY REALISING OPPORTUNITY IMPROVING EFFECTIVENESS

16
PROGRESS AND PLANS
DATA GOVERNANCE IN THE OU
INITIATE DEFINE DEVELOP EMBED
Recognise the need for data
governance and gain agreement
to create a governance
framework for data.
Create policy and the decision-
making framework. Identify,
define and create the core roles
and responsibilities.
Create enterprise-wide artefacts
to support data management.
Embed data governance in
technology and procure
software.
Define and implement data
governance for all core business
processes. Build data quality
dashboards. Transition to BAU.
• Identified and agreed Senior
Information Risk Owner
(SIRO) for the OU.
• Gained agreement to
implement data governance
for the university.
• Aligned to related areas of
work (GDPR…).
• Defined the governance
bodies for data and
information.
• Secured resource to
implement the next phase of
work.
• Created the university’s Data
Governance Policy and
secured approval.
• Formed the Data and
Information Governance
Group and its sub-groups.
• Defined data domains and
agreed domain owners.
• Agreed SMEs to support data
domain owners.
• Trained data domain owners
and local SMEs.
• Initiated next phase.
• Create core artefacts: data
dictionary, MDM approach,
reference data approach,
BIM.
• Procure and implement data
governance software.
• Implement data governance
for single business process,
end-to-end, including:
• Data quality measures
and dashboards
• Local processes
• Workflows within the
software
• Local taxonomy
• Repeat implementation
across core business
processes.
• Secure, train and embed BAU
resources.
• Monitor, review and refine.
• Implement data governance
for non-core/low-priority
business processes.

17
SUPPORTING LEARNING ANALYTICS
Higher
quality
analytics
OU Data
Strategy
Internal
taxonomies
Improved
data
quality
External
alignment

18
SUCCESS FACTORS
• Single most important component
• Enables top-down messaging, support and enforcement
• Local champions help further the cause
Senior management support
• It isn’t all about the technology, but…
• Data governance software makes things easier and technology
projects offer a great opportunity
• If you need a reason to govern data… GDPR
Alignment to technology and
compliance
• Must be wide-reaching and frequent
• Essential to promote the cause – data governance as an
enabler, not a barrier
Communication
• Data governance isn’t expensive but does require some
commitment of resource
• Half-hearted or incomplete attempts more likely to fail
Budget

Update on LA at the OU: slightly more “blue” sky
research
-Rogers Kalissa: Social Learning Analytics to support teaching (University of Oslo)
-Saman Rizvi: Cultural impact of MOOC learning (IET)
-Shi Min Chua: Why does no one reply to my posts (IET/WELS)
-Maina Korir: Ethics and LA (IET)
-Anna Gillespie: Predictive Learning Analytics and role of tutors (EdD)
Not included:
-Quan Nguyen: Impact of learning design on learning analytics
-Garron Hillaire: Role of emotions in learning analytics
-Josmario Albuquerque: Role of social bias in learning analytics
-Pascaline Fresneau: Investigating the implementation of Social Learning Analytics to Increase Online Peer Collaboration
-Jenna Mittelmeier: Using Learning Analytics to implement evidence-based interventions to drive ethnic minority and international
student success
-Simon Knight: Learning analytics for epistemic commitments in collaborative information seeking environment”.

Social Learning Analytics to Support
Teaching and Learning Decisions in
Online Learning Environments
Case study: Bachelors Course
Source of Data: Online Discussion Forums
Analysis Methods: Social Network & Discourse
Analysis
Tools: NodeXL & Coh-Metrix
Rogers Kaliisa, Department of Education, University of Oslo, Norway

FINDINGS
SNA FINDINGS DISCOURSE ANALYSIS FINDINGS
Active Students Less Active Students
SNA
Metrics
S13 S3 S4 S12 S
10
S5 S18 S24 S8 S11
Degree
centrality
5 2 2 2 3 1 1 1 1 1
Betweenn
ess centrality
114 30 30 30 1
5
0.0 0.0 0.0 0.0 0.0
Closeness
centrality
0.01
6
0.01
5
0.01
5
0.01
5
0
.015
0.01
0
0.01
0
0.01
0
0.01
1
0.01
4
Discourse Analysis Results
No of
words
264 212 1006 133 3
73
100 204 114 121 206
Narrativit
y
73 73 47 68 5
3
94 37 64 74 64
Deep
Cohesion
69 48 37 94 9
9
10 55 32 23 62
Referentia
l Cohesion
67 43 62 35 4
6
83 15 31 70 25
Syntax
Simplicity
41 19 62 49 2
5
6 68 17 19 50
Word
Concreteness
4 41 18 5 1
3
13 10 10 17 42
• SLA provides insight and a richer understanding of the students’ cognitive and social learning Processes.
• Students’ network position can determine their discourse features (Kaliisa, Mørch & Kluge, in press)
Fig. 1. Sociogram of week one discussions
Table 1. Week 1 discourse and SNA metrics results

• To understand how learners from various geo-cultural clusters perform in different MOOC
Learning Designs?
• Whether it varies with the discipline?
• Temporal dynamics in learning trajectories?
➢ Progression
➢ Activity Engagement Duration
The Influence of Geo-Cultural Background on
MOOC Learning Trajectories; Mapping
Divergence and Similarities
Saman Rizvi (saman.rizvi@open.ac.uk) https://iet.open.ac.uk/people/saman.rizvi

28/06/2019The Influence of Geo-Cultural Background on MOOC Learning Trajectories
Preliminary Results
Saman Rizvi (saman.rizvi@open.ac.uk) https://iet.open.ac.uk/people/saman.rizvi

5 Types of Comments in FutureLearn
Discussion
Initiating Post
First Reply
First Reply
Further Reply
Initiating Post
Initiator’s Reply
First Reply
Single Post
Shi Min Chua: Discursive practices to achieve dialogic learning and online deliberation in MOOC discussions. https://iet.open.ac.uk/people/shimin.chua

7 Groups of
Social Learners
Loner
26%
Replier
6%
Initiator without
replying
18%
Initiator who respond
6%
Active Social Learner
28%
Active social learners
without turn-taking
15%
Reluctant active
social learners
1%
Language Usage
in Initiating Posts
Shi Min Chua: Discursive practices to achieve dialogic learning and online deliberation in MOOC discussions. https://iet.open.ac.uk/people/shimin.chua

26
STUDY ONE STUDY TWO
2 Privacy
Questionnaires
Vignettes
Follow-up
Interviews
(N=4)
University
QUESTIONS
• Collection of personal data
• Data sharing (identifiable and
anonymised)
• Benefits (w/ & w/o data
sharing)
Lab
Session
Company
QUESTIONS
• Privacy
concepts
LA
Experts (N=12)
Delphi
Study
QUESTIONS
• Privacy
concepts
RQ1: To what extent are students concerned about privacy in the use of
analytics?
RQ2: How do LA experts’ privacy concepts (thinking) compare to
students’ privacy concepts?
Maina Korir: research focuses on ethics and privacy in learning analytics. https://iet.open.ac.uk/people/maina.korir

What factors have
an impact on how
Associate Lecturers
at The Open
University use
Predictive Learning
Analytics?
Anna Gillespie EdD
student year one
Objective: An exploration of the experiences of Open University Associate
Lecturers (ALs) in their use of Predictive Learning Analytics (PLA) to support
distance learning students who are at risk of non-submission of their Tutor
Marked Assignment (TMA)
Sub questions: To consider how individual beliefs, attitudes and intentions,
their knowledge of technology, and the organisational culture/teaching
environment impacts on PLA use and whether is it accepted as an effective
teaching tool. ALs use Early Alert Indicators (EAI) which is the OU specific
PLA dashboard.
Theoretical framework: Using the theoretical models of the Theory of
Planned Behaviour, (Ajzen, 1991) the Decomposed Theory of Planned
Behaviour, (Taylor and Todd, 1995), and the Technology Acceptance Model
(Davis, Bagozzi and Warshaw 1989). I propose that a model based on these
theoretical positions, but with modifications can help us to understand the
role PLA plays in student support and retention and help understand how
ALs use PLA
Anna Gillespie: Predictive learning analytics and the role of teachers. A.Gillespie@open.ac.uk

Initial study
Using a phenomenological qualitative approach, an initial study based on semi structured interviews
of N=5 ALs was conducted. At this stage the findings are limited but have revealed that ALs view the
use of PLA differently according to the number of modules taught, their attitude to technology and
the support offered by their faculty.
Findings indicated that attitudes towards using EAI varied according to each AL’s perception of PLA
and their belief about its value as a tool to assist in supporting students: “It feels a bit like
snooping.... I know the argument is that it is for their own good, but it’s like FB, they are monitoring
us and so are Google…. it feels a bit hollow to me.”
More positively “Using EAI, I could see when he dipped and rose, so I'd email him to tell him what he
needed to do for the TMA and I would see a massive spike in his activity, he didn’t reply to the email
but I knew he was reacting to it”
Further investigation in the main study will look at a larger sample and also use Eye Tracking
technology to give a clear picture of what aspects of the EAI dashboard are most utilised and
hopefully mitigate inconsistencies in self reporting to develop a clear picture of the role of PLA in
supporting ALs delivering tuition at distance.
Anna Gillespie: Predictive learning analytics and the role of teachers. A.Gillespie@open.ac.uk

Prof John Domingue Dr Christothea Herodotou
Director of KMi Senior Lecturer, IET
26 June 2019, JISC LA conference
OU Analyse:
Lessons learnt from predictive analytics so far

OU Analyse History
Pres. Scope Delivery
2014B 2 modules, selected course members Excel spreadsheet, sent
by email, manually
generated
2014J 12 modules, selected course members Automated pred., excel
spreadsheet
2015J Made available to tutors in selected modules Dashboard, within OU
network
2016J ~1000 users (338 accessed) , 785 tutors (305
accessed)
Dashboard on Internet,
no VPN needed, grade
predictions
2017J,
18B
37 modules, 375 users (240 accessed), 323
tutors (204 accessed)
Mostly STEM pilots, user
acceptance
2018J 250 modules in OUA, ~3500 users, including
mostly tutors, module chairs, staff tutors,
cluster managers, SSTs
Dashboard combining
OUA predictions and SIO
Student Probabilities

31
What does it do?
It produces predictions as to
whether students are at risk
of failing their studies.
The model predicts on a
weekly basis whether or not
a given student will submit
their TMA.
It uses a traffic light system
to pinpoint in red students at
risk, in amber those with a
moderate probability of failing
and in green those who are
unlikely to fail.
OU ANALYSE
https://analyse.kmi.open.ac.uk/#dashboard

32
What’s under the bonnet?
Demographics Pre-Course Results VLE DataAssessments Under Testing
- age
- ethnicity
- gender
- highest education
- occupation
- studied credits
- new/continuing
student
- imd band
- region
- country
- the score
- submission
- banking of all the
available
assessments
- activities of students
grouped per week and
activity type (also
known as site type in
SAS)
- also summary activity
counts per week
(these attributes can
grow in dimensionality
as the course progress
and if it has more
activities, a Minimum
Redundancy Maximum
Relevance (mRMR)
algorithm is run to
subset the attributes.)
- linking to the
qualification of a
student
Changes with the module predicted
- number of
previous attempts
- banked
assessments
- number of
previously passed/
failed modules
- number of credits
achieved
Static Data
OU ANALYSE

Prediction scenario – legacy data
Course start
prediction
date
Previous presentation (2016)
Course start
prediction
date
Known past Future to be predicted
Current presentation (2017)
Predictive model
Machine Learning
Prediction
All student data
Student’s data
Wolff, A. et al: Developing predictive models for early detection of at-risk students on distance
learning modules, LAK 2014
Kuzilek et. al: OU Analyse: Analysing At-Risk Students at The Open University. LAK 2015
Martin Hlosta, @mhlosta, martin.hlosta@open.ac.uk
A1 cutoff
A1 cutoff

Under the bonnet - 1
1. Select Data
• Only registered students that haven’t submitted, proper
time-slice of the data
• VLE data grouped by activity type (forum, HTML content,
download resources)
2. Preprocess
• Proper alignment of TMAs (different days in different
presentations)
• Dealing with missing information (previous TMA results)
• Outliers – e.g. extreme number of clicks
• mRMR – Maximum Relevance Minimum Redundancy *
– Most important features that are not correlated
* Peng, H., Long, F., & Ding, C. (2005). Feature selection based on mutual information: criteria of
max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis &
Machine Intelligence, (8), 1226-1238.

3. Train models and vote
• Separate training of 6 Machine Learning
models (various types – various experts)
kNN
Decision Tree
NB
SVM
SUM votes
GBM
S
NS
S/NS
S/NS
S/NS
S/NS
S/NS

3. Scaling up
• Replacing by one ensemble model
• Parallelisation and deployment >= 300
modules (updated within hours)
SUM votes
GBM
S
NS
S/NS

Grades - Beyond submission prediction
• For some modules – S/NS is not a problem
• Predict score & band
• Only students predicted as Submit
Predict
S/NS
SNS
GBR score
0-100

38
2015-2018: A 4-year account of
PLA
Over the last 4 years, OUA reached 23,640 students in 231 undergraduate Open
University courses.

28/06/2019
https://juntosglobal.com/focus-on-customer-journeys-not-touchpoints/
The journey so far

28/06/2019
https://forums.envato.com/t/the-journey-of-becoming-a-musician-share-your-
story/77296
The journey so far
- 2016: planned for
experimental
methodologies
- Tuition policy
changes
- 2016, 2017:
voluntary
participation
- Teachers’ resistance
- Renegotiation of
teachers'
employment
contracts.
- Systematic
engagement with
OUA

42
Systematic engagement of teachers with PLA:
An ongoing challenge
2015/16
2018/19

28/06/2019
What facilitated OUA adoption
over the years…
• Technical developments on OUA
• Access through tutor’s homepage
• Ongoing generation and dissemination of evidence of
impact
• Teachers as champions: training, support, share of
authentic, practice-based
• Educational managers recognising the value and
supporting the initiative
• Development of teaching and intervention strategy and
actively promoted the availability of OUA
• Interdisciplinary team dedicating time on the project

28/06/2019
Herodotou, C., Verdin, B., Boroowa, A., Rienties, B. (2019). Predictive learning analytics ‘at scale’:
Guidelines to successful implementation in Higher Education . Journal of Learning Analytics, 6 (1).

28/06/2019
Improved learning outcomes:
Evidence so far
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M., (In press). A
large-scale implementation of Predictive Learning Analytics in Higher
Education: The teachers' role and perspective. Educational Technology
Research and Development.
Herodotou, C., Hlosta, M., Boroowa, A., Rienties, B., Zdrahal, Z., Mangafa, C. (In
press). Empowering online teachers through predictive learning analytics.
British Journal of Educational Technology.
Herodotou, C., Verdin, B., Boroowa, A., Rienties, B. (2019). Predictive learning
analytics ‘at scale’: Guidelines to successful implementation in Higher
Education . Journal of Learning Analytics, 6 (1).
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., Hlosta, M., Naydenova, G.
(2017). Implementing predictive learning analytics on a large scale: the
teacher's perspective. In: Proceedings of the Seventh International Learning
Analytics & Knowledge Conference, ACM, NY, pp. 267–271.

Study plan
Block 1
Part 1 Part 2
Block 2
Part 1 Part 2 Part 3 Part 1
Block 3
Part 2
Course
Study plan
Block 1
Part 1 Part 2
Block 2
Part 1 Part 2 Part 3 Part 2
Block 3
Part 1
Year
2014
Year
2015
Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7
.
.
.
.
.
.
Course
weeks
At the Open University (OU)
Introduction – OU study plan

Activity of the Top 25%
• describes aggregate information about activities of
students in some cohort (i.e. excellent) in the past
• is calculated for each study material

Feedback from tutors
Dashboard: Tutor View

https://fourweekmba.com/knowledge-graph-linkedin-ai/
https://engineering.linkedin.com/blog/2016/10/building-the-linkedin-knowledge-graph

Thank you
Prof John Domingue Dr Christothea Herodotou
Director of KMi Senior Lecturer, IET
JISC LA event, 26th June

Overview of Effective Learning Analytics Using data and analytics to support students at Open University

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Overview of Effective Learning Analytics Using data and analytics to support students at Open University

Semelhante a Overview of Effective Learning Analytics Using data and analytics to support students at Open University (20)

Mais de Bart Rienties

Mais de Bart Rienties (20)

Último

Último (20)

Overview of Effective Learning Analytics Using data and analytics to support students at Open University