implementing a university wide learning analytics system. Presentation Overview: - Introduction - Developing the NTU Student Dashboard - Transitioning from pilot phase to whole institution roll-out - Embedding the resource into working practices - Future development

1. The NTU Student Dashboard – implementing a University wide
learning analytics system
ABLE Project: 2015-1-BE-EPPKA3-PI-FORWARD
STELA Project: 562167-EPP-1-2015-1-BE-EPPKA3-PI-FORWARD

2. Presentation Overview
• Introduction
– Description of learning analytics
– Overview of the Dashboard
• Developing the NTU Student Dashboard
– Stakeholder consultation
– Privacy, ethical, and legal consideration
– Data collection challenges and integrating with university systems
• Transitioning from pilot phase to whole institution roll-out
– Scalability and sustainability
• Embedding the resource into working practices
– Collaborating with tutors and study advisors
– Reaching the ‘right’ students
• Future developments
– Where next and why?

3. An introduction to learning analytics
Some of this is very sensitive & confidential

4. "Analytics is a term used in
business and science to refer to
computational support for capturing
digital data to help inform decision-
making … Learning Analytics
appropriates this concept for
education.”
(Buckingham Shum, 2012, p.1)
(Clow, 2012)
Learning Analytics
“Learning analytics uses predictive
models that provide actionable
information. It is a multidisciplinary
approach based on data processing,
technology-learning enhancement,
educational datamining, and
visualization.”
(Scheffel, Drachsler, Stoyanov, &
Specht, 2014).

5. How can institutions use learning analytics?
• HEIs currently possess vast quantities of student data:
– Interaction with facilities/resources
• Attendance / card swipes
• Computer log ins / access to VLE
– Personal, demographic, medical, disciplinary, criminal
– Academic performance
• Data has historically been used for activities such as providing
information during emergency situations, making decisions during
exam boards etc.
• Increasing number of institutions looking to use this data for
learning analytics in order to support students (and/or support staff
in their efforts to support students)
Key idea: interaction with the University systems can be used to
predict the likelihood of student success (and/or spot those
students at risk)

6. Why NTU is interested in learning analytics?
• Reduce barriers to being known by a tutor
• Space for tutors to make notes & plan
• Students as agents
• Can see own grades, and compare self to peers
• Feedback
• Tool for academic use
• Potentially promotes virtuous behaviour
• Alerts tutors if students are high risk of leaving early
• Students can compare their engagement with their peers
Attainment
Progression
Belonging
Strategic
information
• Insights into groups at risk
• Potentially design and delivery of courses

7. Question: How do you use data in your own
practise to support students?

8. Developing the NTU Student Dashboard

9. Stakeholder Consultation
 Pilot year survey & interviews (2013-14)
 Staff briefings & online user survey
(2014-15)
 Workshops (2015-16, 2016-17)
 iPad pilot surveys (2016-17)
 Feedback to service desk (ongoing)
 Governance structure
 Pilot year focus groups (2013-14)
 Transition surveys (2012-13 – onwards)
Feb- Mar each year, approx. 500
respondents
 Questionnaires
 Student research 1 – 1 interviews
 Students involved in ongoing development
Governance group, student placements
Student Consultation Staff Consultation
• Stakeholder consultation is key to creating a fit-for-purpose tool
• Consultation is has been ongoing since the pilot year
• Strategies are needed to maximise stakeholder involvement

10. Dashboard Governance Group
Dashboard Operations Group
Informal student
group
Ethics Group University Systems Group
Academic Standards & Quality
Committee
University Governance &
Management
Academics
Students
Educational developers
Student Support
Student Planning
Information Systems

11. Privacy, Ethics and Legal Considerations
• Complying with the law is vital – but not enough
• Student perceptions about any data kept on them and the data
protection awareness paradox
• Considerations around potential for sophisticated stereotyping
• Considerations around potential to cause students stress
• Staff concerns – hidden agenda/ loss of autonomy
• We have written guidance about how we will use the Dashboard
– (pg 9)
• We have an ethics group and ethics research group
• We need to do more work on student communications
• Open University has good (if lengthy) guidance

12. Data Collection Challenges and Integration
with University Systems
• University systems not built with learning analytics in mind
• University systems are not necessarily used consistently across
whole institutions
• Expertise in a range of areas are required to ensure data quality,
including ensuring that the consequences of other institutional
changes are understood in terms of their impact on the data
• University populations are diverse and dynamic, so data is
constantly changing

13. Question: What concerns do you have about
learning analytics?
 Generally positive about Learning Analytics :)
 Generally negative about Learning Analytics :(
Question: How do you currently feel about
learning analytics?

14. What is the NTU Student Dashboard?

15. Raises
alerts!!
What does the Dashboard do?
NTU
Student
Dashboard
Student biographical
info, e.g. enrolment
status
Evidence of student
engagement
• Door swipes
(where appropriate)
• Library books
• NOW use
• Dropbox
submissions
• Attendance data
• Access to e-
books and
journals through
Shibboleth
authentication
Staff
view
Student
view
Compares student
engagement across
the cohort & gives
rating
Can make
comments in
free text box

16. The NTU Student Dashboard
• The Dashboard monitors students’ engagement with their course
– Door swipes, Library use, log ins to the VLE & submission through NOW Drop
box
• It compares this data to a profile from previous years and assigns
the four ratings:
– High
– Good
– Partial
– Low
– Not fully enrolled (for students who are only eligible to enroll at the start of the
year, or who have withdrawn)
• Students and staff can see exactly the same view
– Staff have a few additional management screens
• Tutors can also make notes in the notes section
• Staff are also sent an alert if there is no engagement for two weeks
during term time

17. The NTU Student Dashboard
• Can be viewed as two products:
Physical
Dashboard
Algorithm
• Staff and students interact with physical dashboard
• Algorithm is the behind the scenes, learning analytics element

18. Data accuracy for algorithm spotting students
at risk
• Two big questions:
1. Can the algorithm correctly identify at risk students?
2. Can it do so on a timescale that allows intervention?

19. Relationship between yearly average engagement &
progression
• Low average (mode) engagement for the year is an indicator of risk

20. Relationship between term one average engagement &
progression
• Low average (mode) engagement for the 1st term is an indicator of risk

21. No engagement alerts
• Any one alert is an indicator of risk
• Students with multiple alerts had lower incidence of progression

22. Tutorial Landing Screen
• Access to tutees in addition to search facilities for other students the
staff member interacts with

23. Links to student’s dashboard Able to sort on headings
Class view
• Designed so staff have easy access to student data.
• Allows staff to quickly identify potentially at risk students

24. Individual student view
Not Fully
Enrolled
Staff and students can
benchmark engagement –
springboard for conversation

25. Notes and referrals
• Notes inputted by
staff only
• Time and date stamp
• Referral to:
 Library Academic
Skills
 Student Support
Services
 Employability
(planned)
Personal tutors can track interactions with students
Personal tutors can make referral whilst in room with student

26. Student profile
• Basic information (ability to
report if this is wrong)
• Engagement summary
• Entry qualification details
• Engagement history (for
previous years)
• Details to help early tutorials

27. • Data drawn
automatically from
the attendance app
• The wheel shows
overall attendance
• The table below
provides more
detailed information
for recent weeks
Attendance
• Springboard for
conversation
(potential context of
School attendance
policy)

28. Assessment & Feedback View
• Only show
assessments and
feedback
submitted
through NOW
(the University’s
VLE)
• Shows
assessments and
feedback for
multiple modules
• Better sight of student performance than only seeing own module results so
tutor can make more informed recommendations

29. Pilots within the Dashboard - Induction Trial
• Designed to give lecturers information for early tutorials and to help students
reflect back later in the year

30. Transitioning from pilot phase to whole institution

31. Developmental Cycle
Sept
2013
Sept
2014
Sept
2015
Pilot Phase Phase One Phase
Two
Phase Three
 Personalisation
 Embedding into University
systems
 4 courses
 40 staff
 500 1st years
Willing participants:
very positive staff
feedback, limited
student awareness
 8 of 9 Schools
 Governance
 Problem solving
 Ethics
Near to whole
University roll out:
increased awareness
 All Schools
 New data
sources
 Assessment
view
Whole
University roll
out: increased
awareness
Feb
2016
Increasing resources (e.g.
how to guides),
communication, guidance
Staff and student consultation ongoing throughout developments
Further details of projects at http://www.ableproject.eu/project-outputs/ and
https://eng.kuleuven.be/english/projects/STELAproject/stela

32. Scalability and Sustainability
• Making a tool that works perfectly for every single student is a
challenge when considering > 25,000 students
• Lots of exceptions to the rule
• As the user-base increases, so do the number of ideas for
development. Where do you stop?

33. Embedding the resource into working practices

34. Getting the word out, the basics…
• Need to communicate existence of resource and how to log in
• Need to communicate why to log in
• Need to communicate what to do once logged in
• Need to keep doing these things
2014/15 – average 11 log-ins per staff member
2016/16 – average 12 log-ins per staff member
2014/15 – average 8 log-ins per student
2016/16 – average 13 log-ins per student
Staff usage Student usage

35. Collaborating with tutors and study advisors
and reaching the ‘right’ students
• There are two change agents in the NTU model
• There needs to be a reason for people to change – communication is
key
Students Staff

36. Students as change agents
• Dashboard as resource to inform students about their engagement
• Dashboard focuses positively on engagement rather than risk
E.g. in FAQ section on Dashboard:
Q: Is there a relationship between engagement and success?
A: Yes. In 2013-14 twice as many final year students with a high
average engagement achieved a 2:1 or 1st compared to those with
low…
• Focus on communication and development of tool in-line with
student views
• https://youtu.be/kDMOsKCXCjc

37. Relationship between number of Dashboard
log-ins and engagement classification

38. Relationship between Dashboard log-ins and
progression

39. Feedback from the Student Transition Survey
• The transition survey was conducted with first year students (Feb/March 2016)
• 91% reported using the Dashboard at least once (90% in 2015)
Have you logged into the NTU student Dashboard?
When using the Dashboard, how often have you explored the following?
Base: 515 (2016), 469 (2015)
4%
5%
14%
15%
13%
24%
5%
6%
26%
24%
35%
36%
12%
13%
33%
34%
34%
36%
79%
77%
28%
28%
19%
5%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Spoke to someone providing specialist help (for
example student support services/ library) as a result
of looking at information on the Dashboard
Spoke to your tutor as a result of looking at
information on the Dashboard
Changed your behaviour to raise or maintain your
engagement score (for example made sure that you
swiped to go into a building)
Compared your engagement score with other students
on your course
Increased the amount of time you spend studying
Checked your own engagement score
Very Often Often Sometimes Never

40. Students as change agents
• Challenges around Success for All
• Highlights importance of staff as change agents

41. Staff as Change Agents
• Personal tutor
– Ongoing support both academically and pastorally
– Induction activity
– Primary contact for alerts
– iPad trial
• Referrals to support services
– Notes and referrals function
• Staff communication
– Briefings and drop-in sessions
– Internal conference/networking events
– Newsletter term 1 – low engagement
– Newsletter term 2 – alerts

42. Newsletter 1 - Relationship between engagement
at key times of year and progression

43. Newsletter 2 - No engagement alerts
• Any one alert is an indicator of risk
• Students with multiple alerts had lower incidence of progression

44. What have we achieved?
• We have a working Dashboard that has been implemented across
the whole University
• Positive feedback from many staff members & students
What would we like to achieve?
• Ideally, we’d like to transform retention, belonging & success
• We’d like to be confident that staff are using the tool consistently
and that all students were taking advantage of the tool (particularly
those who need it most)
• We’d like to use the tool to provide evidence on what works to help
students most at risk

45. Some of our key lessons
• It’s a team game
• Philosophical opposition appears stronger amongst staff, not
students
– Although academics can see the benefits more readily as they appear to have
terrible access to student data
• Whatever you’ve budgeted for communications, double it
• There are two products: the dashboard and the analytics
– The quick wins are as much about the dashboard as they are about the
analytics
• The miraculous quickly becomes the mundane
– Trivial irritants become big problems
• This is disruptive technology
– The dashboard highlights existing problems with systems
– Embedding/ competing/ duplicating

46. Thank you for listening.
Any questions?

47. www.ABLEproject.eu www.stela-project.eu
NTU is involved in two Erasmus+ funded projects to investigate
how learning analytics can help students transition into
University.