Presented at ALT-C 2016
In our increasingly digital world our students leave an ever-growing electronic footprint behind them as they pass through the physical and virtual campus. This data is still a greatly underused asset (Higher Education Commission, 2016) although a number of UK higher education institutions have already implemented descriptive, inferential and/or predictive Learning Analytics (LAs) solutions using a wide variety of approaches, see for example Sclater (2014).
This paper discusses recent research at an English post-92 university aimed at investigating the benefits and challenges of using LAs. Prompted by a perception that some voices had yet to be given a loud enough voice in relation to a systematic use of big data in the higher education sector. It was particularly concerned with gaining a better understanding of the hopes, fears and needs of those on whom it would be most likely to impact.
This presentation will focus on our findings from of a series of focus groups and interviews with students, university governors and academic professional and support staff that took place during the 2015/16 academic year. Questions were framed around understanding views about the purpose of LAs, concerns about the type of data liable to be used, perceptions of how likely being more informed would result in changes in behaviour and outcomes, and finally how should data be presented.
The results indicated that each group had different areas of interest when it came to the type of data of interest. These spanned the range from what may be regarded as strict LAs to Academic Analytics (for an explanation of the differences see Long and Siemens (2014)). A common theme however, was that most felt that being better informed would lead to better decision making. However, having knowledge about one’s own performance, particularly in relation to peers, was unwelcome in some quarters.
When examining student concerns about data, overall there was a low degree of anxiety and a high degree of trust that the institution in general and tutors in particular would behave responsibly. Concerns about legal and ethical problems were most likely to be voiced by academic, professional and support staff. Transparency, and finding creative approaches to promoting it, was identified as vital by most groups.
During this session we will share our findings in more detail and reflect on our understanding of variations in perceptions between and within different stakeholder groups. We will demonstrate and share a checklist of institutional risks and responsibilities that was developed as an aid to identifying, understanding and managing each of these areas.
2. Academic Support, Technology & Innovation
ASTI and Teaching and Learning Support
investigating the potential of using Learning
Analytics as a means of enhancing the student
experience
Grant from the HEA’s Strategic Excellence Initiative
for Vice-Chancellors
Background
Teaching and Learning Support
3. Academic Support, Technology & Innovation
The electronic footprint our students leave behind when they
interact with our digital systems
e.g. digital learning environment, electronic library, ePortfolio
This and other data sources used to track student engagement
and to identify those who may be in danger of failing
Web pages and apps used to present various data visualisations
for personal tutors, students and others
Teaching and Learning Support
Definition of Learning Analytics
4. Academic Support, Technology & Innovation
Teaching and Learning Support
Definition of Learning Analytics
Institution Faculty School Programme Module Individual
Academic
Analytics
Learning
Analytics
Data
Granularity of Data
6. Academic Support, Technology & Innovation
Teaching and Learning Support
Understanding our Stakeholders’ Perspectives
7. Academic Support, Technology & Innovation
Mostly information already collected in a range of ways,
from a range of systems for a range of purposes
Array of challenges
Systematic identification of ‘at risk’ students may place an
unsustainable obligation to act on the University
Challenges:
Ethical
Legal
Data
Technical
Policy
Process
Teaching and Learning Support
Key issues, challenges and concerns
8. Academic Support, Technology & Innovation
Transparency vital to maintain trust
High degree of confidence that staff
would deal with their data in a
‘professional way’
Varied in degree of comfort with easy
access to data about their own
performance - closely monitored
students seemed least worried e.g. health
areas
Most, but not all, keen to compare their
profile with anonymised cohort average or
an ‘ideal’ student, so consider ‘opt out/in’
Could motivate some but discourage others
University’s response to a ‘red flag’ should
not be an automatic process but the start of
a conversation
Teaching and Learning Support
Stakeholder Perspectives:
Students
9. Academic Support, Technology & Innovation
More concerns than students over legal /
ethical data use
Support and training to understand
responsibilities
Need for openness and transparency
Focus should be on benefitting student
not institution
Questioned effect on retention students leave
for many reasons
Policy changes needed, e.g. attendance
monitoring
Much of this data is already collected in
disparate ways
Need to ensure compliance with legislation
for current, retrospective and future use of
data
Teaching and Learning Support
Stakeholder Perspectives:
Academic, Technical & Support Staff
10. Academic Support, Technology & Innovation
Need shared vision of what is meant
by Learning Analytics
Culture of respect for information and
anonymity required
Concerns about the scope and quality
of current data
Analytics data should be triangulated
with other information
Opposing views about students having own and
cohort data
‘why would they want to know?’
‘what are they afraid of?’
Some concerns it could be demotivational, a
distraction or encourage a strategic approach to
study
Could give insight into characteristics of a
successful programme, trajectory of a successful
student, value added over the course of a
programme (Learning Gain)
Teaching and Learning Support
Stakeholder Perspectives:
Senior Leaders
11. Academic Support, Technology & Innovation
Potential to increase retention and enhance performance but would need to
show a return on investment
Varied responses to having personal access
With Analytics, data must become everyone’s responsibility
Academic Analytics could aid institutional decision-making
Enable Plymouth to offer something distinctive to its students
Teaching and Learning Support
Stakeholder Perspectives:
Governors
13. Academic Support, Technology & Innovation
LA owned at a very high level
Plan for success e.g. fewer
withdrawals
Define goal(s) and specify initial
measures
Audit policies to identify
amendments and gaps
Implement single version of
truth for data & policies
Set and resource Institution-
wide standards for responding
Build-in to future procurements
Consider offering ‘opt-out’
Teaching and Learning Support
Recommendations: Policy
14. Academic Support, Technology & Innovation
Consent agreements and
statements in line with planned use
Choice of Learning Analytics
solution
Implement institution-wide
standards for responding
Governance requires a
multidisciplinary team including
students
Digital literacy & training for staff
Be open and transparent,
particularly with students
Be aware there will be false
negatives/ false positives
Staff development to make
responsibilities clear and
support policy changes
Teaching and Learning Support
Recommendations: Process
15. Academic Support, Technology & Innovation
Bring silos together (e.g. data
warehouse)
Single version of truth needed
for chosen data
Specify data currently easily
accessible
Establish ownership,
stewardship and users of data
Agreements with 3rd party
provider to reflect new use
Unique identifier work required
Work out synergies with other
existing projects (e.g. S3, Mobile
With Plymouth app)
Teaching and Learning Support
Recommendations: Technology
16. Academic Support, Technology & Innovation
A response to a ‘red flag’ should not be
an automatic process but the start of a
conversation
Implement single version of truth for
data and policies
Policy changes needed, e.g. attendance
monitoring
Support and training to understand
responsibilities and support policy
changes
Culture of respect for information
and anonymity required
Establish ownership, stewardship and
users of data
Policies and data need to be owned
centrally (i.e. Academic Registry)
With Analytics, data must become
everyone’s responsibility
Teaching and Learning Support
Institutional Checklist
17. Academic Support, Technology & Innovation
Senior Sponsorship is essential
Single Version of the truth for all data and policies
Have an “owner” for data and revised/updated/new policies
Use available solutions (i.e. Jisc toolkit, Mobile With Plymouth, S3)
Use analytics to support personal tutoring and institutional decision making
Learning Analytics is about Culture Change, not technology
Teaching and Learning Support
Making it so