1. This document discusses using learning analytics to gain insights from educational data.
2. Two case studies are described that analyzed institutional data to better understand the impacts of a new virtual learning environment and predictors of student satisfaction in science and engineering courses.
3. Both cases followed a process of appreciating the issue, identifying relevant data sources, summarizing individual data, joining data sources, preparing data for analysis, analyzing and visualizing results, and refining understanding.
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
LRT Talks 2013-03-12 CETIS
1. First steps with learning analytics
CETIS 2013
Birmingham | Mar 2013
Professor Mark Stubbs
Head of Learning & Research Technologies
m.stubbs@mmu.ac.uk http://twitter.com/thestubbs
http://slideshare.net/markstubbs
Tuesday, March 12, 2013 1
2. Analytics to develop understanding
2. Identify
1.Appreciate
relevant data
the issue
sources
Refining
3.Summarise
6.Analyse &
visualise
understanding individual
data sources
of a problem
space
5 Prep the 4 Join on
data for common
analysis identifiers
3. Analytics case study one
Understanding impacts of a new VLE on student success
• PhD Student (Julie Hardman) led evaluation strand for
MMU’s (WebCT Vista) VLE project from 2006-2009
• Stakeholders wanted to understand impacts of new VLE
• Suspicion that some patterns of VLE use were more or
less likely to produce successful outcomes for students
• Lots of small-scale studies, but
lack of institutional-scale research
4. Analytics case study one
Appreciation of 2. Identify Student
1.Appreciate
relevant data
more and less the issue demographics
sources
successful VLE logs
patterns of VLE Exam board
use outcomes
Understanding
3.Summarise
6.Analyse &
visualise
impacts of new individual
data sources
VLE on student Categorise
VLE usage
success & count hits
by category
5 Prep the 4 Join on
data for common
analysis identifiers
5. 3.Summarise
individual
data sources
Analytics case study one
WebCT Vista kept a detailed click-track log (Oracle DB)
• Custom SQL generated 1-row-per-student summaries:
– Number of areas student enrolled on & number active on
– Number of staff actively associated with those areas
– Days from start of term when VLE first & last used
– Percentage of total hits between 9am – 9pm
– Total number of student logons (distinct sessions)
– Total number of document hits by student & by tutors
– Total number of chat hits by student & by tutors
– Total number of assessment hits by student & by tutors
6. Analytics case study one
Appreciation of 2. Identify Student
1.Appreciate
relevant data
more and less the issue demographics
sources
successful VLE logs
patterns of VLE Exam board
use outcomes
Understanding
3.Summarise
6.Analyse &
visualise
impacts of new individual
data sources
VLE on student Categorise
VLE usage
success & count hits
by category
5 Prep the 4 Join on
data for common
analysis identifiers
Join on
student ID
7. 4 Join on
common
identifiers
Analytics case study one
StudentID Demographics a
StudentID Categorised VLE usage b
StudentID Progressed? c
SELECT …
FROM a, b, c
WHERE a.StudentID = b.StudentID
AND b.StudentID = c.StudentID
StudentID Demographics Categorised VLE usage Progressed?
8. Analytics case study one
Appreciation of 2. Identify Student
1.Appreciate
relevant data
more and less the issue demographics
sources
successful VLE logs
patterns of VLE Exam board
use outcomes
Understanding
3.Summarise
6.Analyse &
visualise
impacts of new individual
Interpret data sources
output of VLE on student Categorise
VLE usage
statistical success & count hits
tests
by category
5 Prep the 4 Join on
data for common
Code missing analysis identifiers
Join on
values, collapse student ID
categories…
9. 6.Analyse &
visualise
Analytics case study one
Random forest analysis of factors predicting progression:
• Percentage of student usage between 9am and 9pm
• Number of day since start of term when VLE last used
• Total number of tutors’ document hits
• Total number of students’ document hits
• Total number of tutors’ chat hits
• Total number of student logons (distinct sessions)
• Number of days since start of term when VLE first used
http://onlinelibrary.wiley.com/doi/10.1002/sres.2130/abstract
10. 6.Analyse &
visualise
Analytics case study one
Visualising partial dependence on the top three predictors
9am-9pm usage Last VLE access Tutors’ doc hits
http://onlinelibrary.wiley.com/doi/10.1002/sres.2130/abstract
11. Analytics case study one
Appreciation of 2. Identify Student
1.Appreciate
relevant data
more and less the issue demographics
sources
successful VLE logs
patterns of VLE Exam board
use outcomes
Understanding
3.Summarise
6.Analyse &
visualise
impacts of new individual
Interpret data sources
output of VLE on student Categorise
VLE usage
statistical success & count hits
tests
by category
5 Prep the 4 Join on
data for common
Code missing analysis identifiers
Join on
values, collapse student ID
categories…
12. 1.Appreciate
the issue
Analytics case study one
Enhanced appreciation of relationships between VLE use and
student success
• “At risk of failure” alarm bells
– High usage percentage outside 9am-9pm
– Early finish or late start in year for VLE use
• Categorised VLE use more informative than total hits
– Documents / Content
– Chat / Dialogue
– Assessment
• More questions than answers … we’ll be exploring further
with our new Moodle VLE
13. 1.Appreciate
the issue
Analytics case study two
Understanding predictors of NSS overall satisfaction for
Science and Engineering courses
• Fielding, A.F., P.J.Dunleavy and A.M. Langan (2010)
Effective use of the UK's National Student (Satisfaction)
Survey (NSS) data in science and engineering subjects.
Journal of Further and Higher Education, 33, 347-368.
• Concern that focusing on low mean scores without reference
to subject benchmarks could be ineffective for improving
overall satisfaction
More background…
• http://www.gees.ac.uk/events/2010/feedbknss/documents/MarkLanganInsightsintoNSS_La
ngan_GEES_Nov2010_submitted.ppt
14. Analytics case study two
Appreciation 2. Identify UK NSS
1.Appreciate
relevant data
of how best to the issue dataset filtered
sources
improve by JACS3
satisfaction Science &
Engineering
Understanding
predictors
satisfaction on 3.Summarise
6.Analyse &
individual
Interpret visualise
Science & data sources Question
output of
Engineering responses
statistical
summarised
tests
courses to %satisfied
5 Prep the 4 Join on
data for common
Code missing analysis identifiers
Not required:
values, collapse single dataset
categories…
15. 6.Analyse &
visualise
Analytics case study two
Random forest analysis of Q.s predicting overall satisfaction:
Rank Predicting questionnaire item Inc MSE (%)
1 Q15 - The course is well organised and is running smoothly 119.89
2 Q1 - Staff are good at explaining things 71.45
3 Q4 - The course is intellectually stimulating 66.71
4 Q14 - Any changes in the course or teaching have been communicated effectively 60.79
5 Q10 - I have received sufficient advice and support with my studies 55.34
…
10 Subject 32.35
…
18 Q7 - Feedback on my work has been prompt 10.49
19 Q9 - Feedback on my work has helped me clarify things I did not understand 6.65
20 Q5 - The criteria used in marking have been clear in advance 6.60
21 Q21 - As a result of the course, I feel confident in tackling unfamiliar problems 3.32
22 Q8 - I have received detailed comments on my work 3.04
16. 6.Analyse &
visualise
Analytics case study two
Sector predictions of Q22 from Q1-21 responses for Sci Eng
Under-performing… Actual Predicted Residual SE1 SE2 SE3 Subjects
Over-performing…
17. Analytics case study two
Appreciation 2. Identify UK NSS
1.Appreciate
relevant data
of how best to the issue dataset filtered
sources
improve by JACS3
satisfaction Science &
Engineering
Understanding
predictors
satisfaction on 3.Summarise
6.Analyse &
individual
Interpret visualise
Science & data sources Question
output of
Engineering responses
statistical
summarised
tests
courses to %satisfied
5 Prep the 4 Join on
data for common
Code missing analysis identifiers
Not required:
values, collapse single dataset
categories…
18. 1.Appreciate
the issue
Analytics case study two
Enhanced understanding of the predictors of satisfaction for
Science and Engineering courses
• Statistical demonstration of significance of subject variation
led to a more sophisticated NSS response at MMU
• Course organisation and teaching are significant predictors
• Low feedback scores not a significant predictor
19. Analytics to develop understanding
2. Identify
1.Appreciate
relevant data
the issue
sources
Refining
3.Summarise
6.Analyse &
visualise
understanding individual
data sources
of a problem
space
5 Prep the 4 Join on
data for common
analysis identifiers
20. Next steps with relevant data sources
Exam Board
VLE logs Outcomes
Student & Course
Records
Assignment Student
Submissions Satisfaction
2. Identify 3.Summarise 4 Join on
relevant data individual common
sources data sources identifiers