Ways of seeing learning - 2017v1.0 - NUI Galway University of Limerick postgrad research day

1. • Ways of Seeing Learning
Learning Analytics for Learners
Mary Loftus Michael G Madden
NUI Galway NUI Galway
mary.loftus@nuigalway.ie michael.madden@nuigalway.ie
@marloft
• The authors acknowledge the support of Ireland’s Higher Education Authority through the IT Investment Fund and ComputerDISC in NUI
Galway.

3. And today…
• Data, Artificial Intelligence &
Machine Learning are having a
similar effect across society
• This revolution is not only
showing us to ourselves in new
ways – it is shaping how we live.

4. Overview of this Presentation
• Why Ways of Seeing & What is Learning Analytics
• What kind of Education do we want to have for our children?
• How will AI, Machine Learning & Data Analytics change our world?
• Why try to Measure Learning? What are there downsides?
• Data is political – how do we use it fairly & avoid unintended consequences?
• White-box Algorithms and transparency
• Research Values & Questions
• Bayesian Networks, Open Learner Models & Data Gathering
• Current Research Status & Timeline

5. Learning Analytics – a Definition
• “Learning analytics is the measurement, collection, analysis and
reporting of data about learners and their contexts, for purposes of
understanding and optimising learning and the environments in
which it occurs”
• Call for Papers of the 1st International Conference on Learning Analytics & Knowledge (LAK 2011)
Learning Sciences
Data Mining
Data Visualization
Psychology

6. Central to education’s purpose is “the coming into presence of unique individual beings”
Education “spaces might open up for uniqueness to come into the world”
– Biesta, G. J. J. (2015). Good Education in an Age of Measurement: Ethics, Politics, Democracy. Routledge.

7. Learning Analytics – The Story So Far
• Predicting student outcomes - identifying ‘at-risk students’
• Personalisation of student learning
• Multi-modal analytics – analyses of audio, video, location data
• Discourse and writing analytics
• Measuring ‘student engagement’ & disengagement
• Levels of Learning Analytics:
• Teachers
• Course
• Institution
• National

8. If we can make learning more visible,
can we...
• Reduce the need for formal testing and
examinations?
• Do more problem-based learning & assessment?
• Provide more formative feedback for students?
• Model student’s conceptual understanding?
• Support metacognition?

9. Ethics & Other Tensions in Learning
Analytics
• In the next few slides, I want to examine the values that I am forming
as a researcher that will shape this research:
• Student Vulnerability & Agency
• Measurement as a powerful but double-edged tool
• Recognition of data and algorithms as powerful political tools
• The potential for Unintended Consequences when we build
algorithmic systems in social systems

10. Student Vulnerability, Agency, and
Learning Analytics
• Prinsloo & Slade examine how we:
• decrease student vulnerability,
• increase student agency,
• empower students as participants in learning analytics
• moving students from quantified data objects to qualified and
qualifying selves
• “In light of increasing concerns about surveillance,
higher education institutions (HEIs) cannot afford a
simple paternalistic approach to student data”
• Prinsloo & Slade (2016)

11. Measuring Gets Results – But Care
Needs to be Taken…
• When we measure, we can clearly see
improvement and impact
• However, the Hawthorne effect is a kind of
"tell-me-what-you-measure-I-will-tell-you-
how-people-react-to-it" effect
• Drucker is often quoted as saying: “What gets
measured gets managed” – but Demming
said: “Eliminate management by objective.
Eliminate management by numbers,
numerical goals. Substitute leadership”

12. Data is Political
• We need to take care in our research to ensure fairness and not replicate
societal biases and discrimination
• There is nothing about doing data analysis that is neutral.
What and how data is collected, how the data is cleaned and
stored, what models are constructed, and what questions are
asked—all of this is political.
dana boyd (2017)

13. Unintended Consequences…
• Criminal Justice Systems – discriminating on the basis of race?
• Employment Screening – address, age, gender?
• Advertising – different ads served depending on race? (Sweeney 2013)
• Even if humans are there as a ‘final check’, there is potential for ‘Moral
Crumple Zones (Elish 2016)
Most data analysis makes prejudicial decisions as part of
clustering without having any understanding of the people or
properties that they are using. It’s merely math! But that math—
and the decisions that are determined by it—have serious social
ramifications.
dana boyd (2017)

14. White-Box Algorithms & Transparency
• Students (and citizens) need to be able to ‘see into’ algorithms that impact
their opportunities and quality of life.
The problem with contemporary data analytics is that we’re
often categorizing people without providing human readable
descriptors.
dana boyd (2017)

15. “Action can never manifest through a predictable, deterministic series
of consequences, since the subject, by acting, is placed within a
complicated web of relationships which cannot be predicted before
hand. In the same sense, Action is irreversible.”
Hannah Arendt
“For apart from inquiry, apart from the praxis, individuals cannot be
truly human.
Knowledge emerges only through invention and re-invention, through
the restless, impatient, continuing, hopeful inquiry human beings
pursue in the world, with the world, and with each other.”
Paulo Freire

16. Research Values
• Grounded in the Student perspective
• Students as owners of their learning data
• Links learning analytics to learning design
• Machine Learning with an emphasis on white-box
modelling & visibility as well as prediction
• Data literacy capacity building for students

17. Research Questions
1. Can a Learning Analytics system provide an interface for students to
engage in metacognitive activities around their own learning,
thereby improving individual learning?
2. Can we retool an existing learning analytics system using machine
learning, modelling and classifiers to provide this metacognitive
interface to students?
3. Can such a system help students visualize, track and reflect on their
own learning and development goals and help them to improve
performance?

18. 4. Metacognition
1. Student
Actions
2. Bayesian
Models & Open
Learning Models
3. Student
Reflection

19. Bayesian Networks
Millán et al (2010)

20. Example of Simple Bayesian Network
Madden et al (2008)

21. Open
Learner
Models
Bull et al (2016)

22. Data Gathering
Kitto et al (2016)

23. Current Research Status: Messy!
• Qualitative Research – talking to students
• Student Data being gathered from systems
• Identifying Data Models for this data and trying them for size

24. Research Timeline
• Literature
Review
• Research
Questions
Ethical
Approval
• Data Modelling
• Qualitative
Research
Data
Gathering • Share improved
models with
students
• Assess impact
Write up
2017 2018 2019

25. References
• Biesta, G. J. J. (2015). Good Education in an Age of Measurement: Ethics, Politics, Democracy. Routledge.
• boyd, danah. (2017, April 12). Toward Accountability. Retrieved 18 April 2017, from
https://points.datasociety.net/toward-accountability-6096e38878f0
• Bull, S., Ginon, B., Boscolo, C., & Johnson, M. (2016). Introduction of learning visualisations and metacognitive
support in a persuadable open learner model. In Proceedings of the Sixth International Conference on
Learning Analytics & Knowledge (pp. 30–39). ACM. Retrieved from http://dl.acm.org/citation.cfm?id=2883853
• Elish, M. C. (2016). Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (We Robot 2016)
(SSRN Scholarly Paper No. ID 2757236). Rochester, NY: Social Science Research Network. Retrieved from
https://papers.ssrn.com/abstract=2757236
• Millán, E., Loboda, T., & Pérez-de-la-Cruz, J. L. (2010). Bayesian networks for student model engineering.
Computers & Education, 55(4), 1663–1683. https://doi.org/10.1016/j.compedu.2010.07.010
• Sweeney, L. (2013). Discrimination in Online Ad Delivery. Queue, 11(3), 10:10–10:29.
https://doi.org/10.1145/2460276.2460278
• Madden, Michael G. (NUI, Galway), Lyons, William and Kavanagh, Ita (Limerick Institute of Technology).“A
Data-Driven Exploration of Factors Affecting Student Performance in a Third-Level Institution”, Proceedings of
AICS-2008: 19th Irish Conference on Artificial Intelligence and Cognitive Science, Cork, August 2008.
• Other refs in Abstract

26. Thank you
Mary Loftus
@marloft
mary.loftus@nuigalway.ie