Presentation Slides from ISSOTL 2015.
Bronnimann, J., West, D., Heath, D. & Huijser, H. (2015) Leveraging learning analytics for future pedagogies and scholarship. Paper presented at Leading learning and the scholarship of change: 12th annual ISSOTL conference, Melbourne, Australia.
Fostering Friendships - Enhancing Social Bonds in the Classroom
ISSOTL Presentation
1. Leveraging Learning Analytics for Future Pedagogies
and Scholarship: The Academic Perspective
Jurg Bronnimann, Deborah West, David Heath & Henk Huijser
Acknowledgement of Research Partners:
Professor Carol Miles, Professor Alf Lizzio, Mr Bill Searle, Mr Danny Toohey
Acknowledgement of Funding:
Australian Government: Office for Learning and Teaching
WELCOME
2. Variable Category Absolute
Frequency
Relative
Frequency
Location
(n = 351)
Australia 341 97%
New Zealand 10 3%
Primary Work Role
(n = 353)
Teaching Students 188 53%
Learning Support 47 13%
Management/Administration 37 11%
Other 32 9%
Research 24 7%
Academic Development 18 5%
Student Support 7 2%
LMS at Institution
(n = 353)
Blackboard 203 58%
Moodle 124 35%
Brightspace (D2L) 13 4%
Sakai 2 1%
Other 11 3%
Employment Basis
(n = 351)
Full Time 284 81%
Part Time 44 13%
Casual 18 5%
Other 5 1%
Academic Level
(n = 351)
Lecturer 124 35%
Senior Lecturer 88 25%
Other 59 17%
Associate Professor 30 9%
Associate Lecturer/Tutor 30 9%
Professor 20 6%
Length of employment in
current institution
(n = 324)
Less than 1.5 years 40 12%
1.5 – 5 years 68 21%
5 – 10 years 96 30%
10- 20 years 86 27%
More than 20 years 34 11%
Length of employment in
Higher Education Sector
(n = 345)
Less than 1.5 years 11 3%
1.5 – 5 years 42 12%
5 – 10 years 85 25%
10- 20 years 130 38%
More than 20 years 77 22%
Involvement in teaching
students (n = 353)
Teaches students 276 78%
Does not teach students 77 22%
ACADEMIC
LEVEL
SURVEY
DEMOGRAPHICS
3. WHAT ARE LEARNING ANALYTICS?
‘measurement, collection, analysis and reporting
of data about learners and their contexts, for the
purposes of understanding and optimising
learning and the environments in which it occurs’
4. WHAT ARE LEARNING ANALYTICS?
Is Learning AnalyticsIs not Learning Analytics
5. BIG & SMALL DATA
Ten dimensions where differences between big and small data might be found (Jules Berman, 2013)
Goals – small data often has a specific goal, big data may have goal but it tends to evolve
Location – small data often in one place, big data multiple servers and computers
Structure – small data often highly structured, big data more variety and less structure
Preparation – instead of just end user (small data), the preparer, analyst and user might all be varied and with different disciplines
Longevity – small data often kept for a defined period of time. Big data is expensive and data rolls on
Measurement – small data often uses a single protocol, whereas big data may have multiple protocols as different people in different
places analyse the data. Consistency is more difficult in this instance.
Reproducibility – small data sets can be replicated if something goes wrong, with big data often the best hope is working around an
identified problem area
Stakes – small data when things go wrong the costs are limited it is not an enormous problem. This is more of an issue with big data
where corruptions have vast implications
Introspection – says something about the data itself. Small data sets – often well organised. In big data sets the complexity is more of an
issue. Data may not be locatable or integrated etc.
Analysis – small data often single procedure. With big data however, because things are so enormous and they're spread across lots of
different files and servers, you may have to go through extraction, reviewing, reduction, normalization, transformation, and other
steps and deal with one part of the data at a time to make it more manageable, and then eventually aggregate your results.
6. WHAT ARE LEARNING ANALYTICS?
Is Learning AnalyticsIs not Learning Analytics
7. WHAT ARE LEARNING ANALYTICS?
Is Learning AnalyticsIs not Learning Analytics
8. WHAT ARE LEARNING ANALYTICS?
“learning analytics is a new, expanding field that grows at the
confluence of learning technologies, educational research, and data
science…
“Over time researchers and practitioners with different backgrounds
and methodologies have tried to solve two simple but challenging
questions:
1. How do we measure the important characteristics of the learning
process?
2. How do we use those measurements to improve it?
(Ochoa et al, 2014, p.5)
9. Beer, C., Tickner, R., & Jones, D. (2014). Three paths for learning analytics and beyond: Moving from rhetoric to reality. In B.
Hegarty, J. McDonald & S.-K. Loke (Eds.), Critical Perspectives on Education - From Rhetoric to Reality. Proceedings
ASCILITE 2014 (pp. 242-250). Dunedin, New Zealand. http://ascilite2014.otago.ac.nz/files/fullpapers/185-Beer.pdf
Gaševic, D., Dawson, S., & Siemens, G. (2015). Let's Not Forget: Learning Analytics Are about Learning. TechTrends: Linking
Research and Practice to Improve Learning, 59(1), 64-71.
http://i.unisa.edu.au/Global/LTU/Learning%20analytics/Lets%20not%20forget%20LA%20are%20about%20learning.p
df
Kennedy, G., Corrin, L., Dawson, S., Williams, D., Mulder, R., Khamis, S., & Copeland, S. (2014). Completing the loop: returning
learning analytics to teachers. Paper presented at the Proceedings Rhetoric and Reality: Critical perspectives on
educational technology, 31st ascilite Conference, Dunedin, New Zealand.
http://ascilite.org/conferences/dunedin2014/files/concisepapers/76-Kennedy.pdf
Dawson, S., Rogers, T., Kennedy, G., & Colvin, C. (in press). Student retention and learning analytics: a snapshot of current
Australian practices and a framework for advancement. Strawberry Hills, NSW.
Kitto, K., Cross, S, Waters, Z. & Lupton, M. (2015). Learning Analytics beyond the LMS: the Connected Learning Analytics
Toolkit. Paper presented at the fifth Learning Analytics and Knowledge (LAK’15) conference.
http://eprints.qut.edu.au/81343/1/lak15_submission_116.pdf
West, D., Huijser, H., Heath, D., Lizzio, A., Miles, C., Toohey, D., . . . Bronniman, J. (in press). Learning Analytics: Assisting
Universities with Student Retention. Strawberry Hills, NSW: Office for Learning and Teaching.
www.letstalklearninganalytics.edu.au
MAPPING PROGRESS WITH ANALYTICS
10. “The university has that
information. They do collect it.
It is in the data bank. One of
this biggest problems with data
analytics is that the information
is there but getting the right
algorithms to pull out the
information is very challenging.
INTEGRATING DATA SYSTEMSSYSTEMS INTEGRATION
11. LEARNING ANALYTICS DISCUSSION FREQUENCY
0
30
60
90
120
150
180
210
Teaching Staff Program or
Course Co-
ordinator
Learning support
staff
School or
Faculty
Management
Colleagues in
Communities of
Practice
Central L&T
Group Staff
Student Support
Staff
Students Institutional
management
Daily Weekly Fortnightly Monthly < Monthly Never
From the Academic Level Survey
Notes: n varies between 296 and 319 per group due to missing data
GROUPS
12. LEARNING ANALYTICS ACTIVITY PARTICIPATION
0 30 60 90 120 150
Other
Delivering training on the use of learning analytics
Conducting formal research and/or publishing work on the
topic of learning analytics
Being part of the group that is leading learning analytics at
my institution
Attending conferences/presentations/training specifically to
learn about learning analytics
Advocating for the use of learning analytics to colleagues
(informal or formal)
None of the listed choices*
Using learning analytics to help with analysis and decision
making
Reading about learning analytics for my own professional
development
n
From the Academic Level Survey
Notes:
* denotes mutually exclusive response
n = 346, missing = 7
13. TRAINING ATTENDANCE AND INTERESTS
0
50
100
150
200
Responding to data (n = 217) Interpreting and analysing
data (n = 217)
Accessing data (n = 221) Introduction to learning
analytics concepts and
applications (n = 218)
Overview of institutional plan
for learning analytics use
(n = 215)
Has attended Has not attended but will when available
From the Academic Level Survey
TRAINING TOPICS
14. RATING OF INSTITUTIONAL PROVISIONS
0
40
80
120
160
Ease of learning
analytics data access
Relevance and
comprehensiveness
of data that I can
access
Ease of visualisation
and interpretation of
data
Opportunities to
provide feedback
about learning
analytics
implementation
Professional
development about
learning analytics
Information about
how learning
analytics is being
used
Provision of
information about
how learning
analytics use will
affect me
Poor or Very Poor Fair Good or Very Good
LEARNING ANALYTICS PROVISIONS
Notes:
n varies between 187 and 204 per category due to missing data
Excludes those people who indicated ‘not sure’ (between 85 and 104 per category) to better illustrate trends visually.
n
15. “Tell me what data is available, give me access to it, give me the time to
use it and give me guidance in using it”.
WHAT ACADEMIC STAFF NEED
16. INTEREST IN LEARNING ANALYTICS APPLICATIONS
0
50
100
150
200
250
Identification of at-
risk students with a
view to staff
responding to
address the risk
Teaching staff
evaluating and
improving their own
teaching practice
Students monitoring
their own progress
and identifying
actions they could
take
Development of
broad knowledge
base about how
effective learning
can occur
Program teams
evaluating and
improving their
program curriculum
Informing potential
initiatives to
promote student
retention (e.g.
mentoring)
Informing design
and layout of online
learning sites and
environments
Identification of
student success
with a view to
providing an
affirmation/reward
type of response
Institutional
management
evaluating and
improving teaching
practice across the
institution
Assistance with
decision making
about student
admissions to the
institution
No Interest A little interest A lot of interest
SELECTED LEARNING ANALYTICS APPLICATIONS
Notes:
n varies between 311 and 317 per category due to missing data
Excludes those people who indicated ‘not sure’ (between 12 and 29 per category) to better illustrate trends visually.
17. Being able to better understand who is in their class (demographics, prior academic
history etc.)
Being able to have consolidated information about their individual students at the
touch of a button (e.g. seeing how their students are doing in other units, what their
demographic data is, whether they are using resources etc. all in one place)
INTERESTS OF ACADEMICS
Learning analytics being used by people
centrally to better justify or evidence
directives relating to their teaching (e.g.
when academics are told to respond in 24
hours to students is there evidence for this
being useful?)
Improving BOTH student (e.g. resource
access patterns, socialisation) and teacher
(e.g. teaching style, unit design) behaviour
with respect to learning
18. DEFINITIONS OF SoTL
As a scholarly enterprise, teaching begins with what the teacher knows...
Pedagogical procedures must be carefully planned, continuously examined, and
relate directly to the subject taught.”
(Boyer, 1990)
“systematic reflection on teaching and learning made public.”
(Illinois State University, 1998)
“ongoing learning about teaching and the demonstration of such knowledge.”
(Kreber and Cranton, 2000)
19. SoTL MADE SIMPLE
“taking a systematic, evidence based approach to improve learning and teaching”
“impact and action to improve outcomes for students or student success.”
20. 1. Using a theory, model or framework to ground the initiative and provide
justification of action.
2. Identify an intervention
3. Formulating an investigative question, which is essentially the hypothesis
in research terms.
4. Conducting an investigation
5. Produce a result in the form of a public artefact
6. Inviting peer review/dissemination
SoTL PROCESS (From Trigwell)
21. LEVELS OF INVESTIGATION (From Trigwell)
Level 1 2 3
Purpose of
Investigation
To inform oneself To inform a group within a
shared context
To inform a wider audience
Evidence
gathering methods
and conclusions
will be…
Verified by self Verified by those within the
same context
Verified by those outside that context
Investigation
results in…
Personal knowledge Local knowledge Public knowledge