The abundance of data in learning environments poses both a potential and a challenge. Improvements in the student experience need a strong connection between data, learning design and the delivery platform. In this talk we explore some ideas on how to establish this connection with respect to feedback.

1. Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
The role of data in
the provision of feedback at scale
APRU ERT Forum
18 November 2016
FarukAteşflickr.com

2. Abelardo Pardo The role of data in the provision of feedback at scale 2
It’s the learning!
Richard Katz presentation APRU ERT Forum Presentation 16 Nov 2016, Singapore

3. Abelardo Pardo The role of data in the provision of feedback at scale 3

4. Abelardo Pardo The role of data in the provision of feedback at scale 4
TimPierceflickr.com

5. Abelardo Pardo The role of data in the provision of feedback at scale 5
Follow a scholarly approach
JonathanCohenflickr.com

6. Abelardo Pardo The role of data in the provision of feedback at scale 6
YvonneLarssonflickr.com
Pressure
Quality/Cost
Student
Feedback
Data-driven
Support
Examples

7. Abelardo Pardo The role of data in the provision of feedback at scale 7
YvonneLarssonflickr.com
Pressure
Quality/Cost

8. Abelardo Pardo The role of data in the provision of feedback at scale 8
Space&Lightflickr.com
Competition
Funding
Enrolment
Expectations
Digital
Environments

9. Abelardo Pardo The role of data in the provision of feedback at scale 9
TulanePublicRelationsflickr.com
Instructors know how to support students
but not in the current conditions

10. Abelardo Pardo The role of data in the provision of feedback at scale 10
Focus on "good instruction"
DanielKwokflickr.com
Kirschner, P. A., & Neelen, M. (2016). The cold left-overs of inquiry-based learning.
Retrieved from https://3starlearningexperiences.wordpress.com/2016/11/15/the-cold-left-overs-of-inquiry-based-learning/

11. Abelardo Pardo The role of data in the provision of feedback at scale 11
increased engagement = increase in quality of experience
MatthiasRippflickr.com

12. Abelardo Pardo The role of data in the provision of feedback at scale 12
StartupMenaflickr.com
Increase in feedback = increase in engagement

13. Abelardo Pardo The role of data in the provision of feedback at scale 13
YvonneLarssonflickr.com
Pressure
Quality/Cost
Student
Feedback

14. Abelardo Pardo The role of data in the provision of feedback at scale
If you could choose one…
• More than 500 meta-analyses
of student achievements
• 100 factors with potential
influence
• Feedback in top five
• (74 meta-analyses) Most
effective form: video, audio,
computer-assisted
instructional feedback, and/or
related goals
14
Hattie, J. A. (1999). Influences on student learning. Inaugural professorial address, University of Auckland, New Zealand

15. Abelardo Pardo Using technology to provide feedback on learning strategies
The feedback question gets systematically
lower values in student surveys
15
Krause, K.-L., Hartley, R., James, R., & McInnis, C. (2005). The First Year Experience in Australian Universities: Findings from a
decade of National Studies. University of Melbourne: Centre for the Study of Higher Education.
Eleafflickr.com

16. Abelardo Pardo The role of data in the provision of feedback at scale 16
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112. doi:
10.3102/003465430298487
FarukAteşflickr.com
"information provided by an agent (teacher, peer,
book, parent, self, experience) regarding aspects
of one’s performance or understanding”

17. Abelardo Pardo The role of data in the provision of feedback at scale 17
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in
Higher Education: Learning for the Longer Term. London and New York: Routledge.
Perceived as an administrative chore
instead of a pedagogical necessity
MarcinWicharyflickr.com

18. Abelardo Pardo The role of data in the provision of feedback at scale 18
AllenLaiflickr.com
Hounsell, D. (2007). Toward more sustainable feedback to students. In D. Boud & N. Falchikov (Eds.), Rethinking Assessment in
Higher Education: Learning for the Longer Term. London and New York: Routledge.
Downward spiral: Student disenchantment mounts
when feedback is uninformative and still,
less feedback is provided
Solution: provide high-value feedback, rethink the role of
student, enhance relation between guidance and feedback

19. Abelardo Pardo The role of data in the provision of feedback at scale 19
Hattie, J., & Timperley, H. (2007). The Power of Feedback. Review of Educational Research, 77(1), 81-112. doi:
10.3102/003465430298487
Orkomedixflickr.com
Feedback Levels
1. Task Level (understanding,
performance)
2. Process Level (what to do to
understand, perform)
3. Self-regulation level (detecting
and directing effort)
4. Self level (personal evaluation
and affect)

20. Abelardo Pardo The role of data in the provision of feedback at scale 20
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: beliefs, techniques, and illusions. Annu Rev Psychol, 64,
417-444. doi:10.1146/annurev-psych-113011-143823
• May not know how to promote
comprehension, retention, transfer.
• May not assess properly our own
learning
• May be biased when judging our
learning
• May rely too much on social beliefs
• Towards “adaptive learners”
ChristianWeidingerflickr.com

21. Abelardo Pardo The role of data in the provision of feedback at scale 21
JanneMorenflickr.com
Dunlosky, J. (2013). Strengthening the Student Toolbox. Study Strategies to Boost Learning. American Educator, 37(3), 12-21.
Suggest learning strategies to students

22. Abelardo Pardo The role of data in the provision of feedback at scale 22
Personalised suggestions
Woody1778a flickr.com

23. Abelardo Pardo The role of data in the provision of feedback at scale 23
Laverrueflickr.com
Increase in personalisation = increase in knowledge

24. Abelardo Pardo The role of data in the provision of feedback at scale 24
From visualisations to actions
Pardo, A., Mirriahi, N., Martínez-Maldonado, R., Jovanović, J., Dawson, S., & Gasevic, D. (2016). Generating Actionable Predictive
Models of Academic Performance. Paper presented at the International Conference on Learning Analytics and Knowledge,
Edinburgh.
KannanBflickr.com

25. Abelardo Pardo The role of data in the provision of feedback at scale 25
YvonneLarssonflickr.com
Pressure
Quality/Cost
Student
Feedback
Data-driven
Support

26. Abelardo Pardo The role of data in the provision of feedback at scale 26
PNNLPacificNorthwestflickr.com
• Collect data about how students
participate in a learning experience
• Translate observations into actions
• Take into account the instructional
design
• Expertise of the instructor
• Create and deploy personalised
support actions

27. Abelardo Pardo The role of data in the provision of feedback at scale 27
IntanA3flickr.com
1) Go beyond feedback at the task level

28. Abelardo Pardo The role of data in the provision of feedback at scale 28
2) Comprehensive*, multimodal, real-time data collection
bodhithajflickr.com

29. Abelardo Pardo The role of data in the provision of feedback at scale 29
3) Clean, relevant data sources at the instructor’s fingertips
NickReadheadflickr.com

30. Abelardo Pardo The role of data in the provision of feedback at scale 30
SmallRealmflickr.com
4) Capacity to deploy immediate
personalised support actions

31. Abelardo Pardo The role of data in the provision of feedback at scale 31
IANRANSLEYDESIGNPLUSILLUSTRATIONflickr.com
5) Scale to large or highly diverse student cohorts

32. Abelardo Pardo The role of data in the provision of feedback at scale 32
YvonneLarssonflickr.com
Pressure
Quality/Cost
Student
Feedback
Data-driven
Support
Examples

33. Abelardo Pardo The role of data in the provision of feedback at scale 33
Tracking

34. Abelardo Pardo The role of data in the provision of feedback at scale 34
You should take a more
careful look at how symbols
are encoded in the video.
Would you be able to encode/
decode UAL symbols without
looking at the video?
Good initial work. However,
did you understand the trick
to handle encoding with a
variable number of bits?
Would you be able to provide
an example?
Good work. Would you be
able to come up with your
own machine language and
your encoding scheme?
Remember that it has to be
unambiguous.
Thorough work with the task
about machine language
encoding. Give it a quick
review before the midterm.
Q1 Q2 Q3 Q4
Instructor

35. Abelardo Pardo The role of data in the provision of feedback at scale 35
Automatic
Email

36. Abelardo Pardo The role of data in the provision of feedback at scale 36
1) Email remains the primary and preferred channel of communication with the
University
http://libinnovation.blogspot.sg/2016/11/how-do-students-really-prefer-to.html

37. Abelardo Pardo The role of data in the provision of feedback at scale 37
Effect size (Cohen’s d) = 0.49.
Medium positive effect
Effect size (Cohen’s d) = 0.21.
Small positive effect

38. Abelardo Pardo The role of data in the provision of feedback at scale 38
Liu, D. Y.-T., Bartimote-Aufflick, K., Pardo, A., & Bridgeman, A. J. (2016). Data-driven Personalization of Student Learning Support in
Higher Education. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends: A view of the current state of the
art. In preparation: Springer.
Instructors define simple rules to create
personalised emails from multiple data sources
Student Relationship Engagement System
Notsomuchflickr.com

39. Abelardo Pardo The role of data in the provision of feedback at scale 39
ontasklearning.org (in production)
IF [video 3.9.5 not watched] THEN “…."

40. Abelardo Pardo The role of data in the provision of feedback at scale 40
AlanLevineflickr.com
Workshop to
• Expose academics and educational
designers to data sets
• Frame the need for personalised
support
• Explore immediate vs long-term actions
• Jumpstart the conversation
• Identify institutional needs

41. Abelardo Pardo The role of data in the provision of feedback at scale 41
Pardo, A., Jovanović, J., Dawson, S., Gasevic, D., & Mirriahi, N. (2016). Exploring student interactions with preparation activities in a
flipped learning experience. Computers & Education, Manuscript under review.
Identify student interactions in
flipped learning environment
YvonneLarssonflickr.com

42. Abelardo Pardo The role of data in the provision of feedback at scale 42
Jovanović, J., Gasevic, D., Pardo, A., Dawson, S., & Mirriahi, N. (2016). Learning Analytics to Unveil Learning Strategies in a Flipped
Classroom. The Internet and Higher Education, Manuscript under review.
1. identify engagement patterns in active learning
environments
2. Ground for personalised suggestions
3. Increase student awareness about strategies
LucLegayflickr.com

43. Abelardo Pardo The role of data in the provision of feedback at scale 43
EdDunensflickr.com
Focus on improvements on the learning experience
Data provides insight on engagement
Combine data with instructor expertise
Use technology to scale personalisation

44. Abelardo Pardo (@abelardopardo)
Faculty of Engineering and IT
slideshare.net/abelardo_pardo
The role of data in
the provision of feedback at scale
APRU ERT Forum
18 November 2016
FarukAteşflickr.com