Presentation at the LAK Conference 2013 in Leuven, Belgium

1. Live Interest Meter - Learning from Quantified
Feedback in Mass Lectures
INFORMATIK
FZI FORSCHUNGSZENTRUM
V. Rivera-Pelayo, J. Munk, V. Zacharias, and S. Braun
FZI Research Center for Information Technology, Karlsruhe, Germany
LAK Conference 2013 – Leuven, Belgium
10th April 2013

2. Agenda
 Introduction
 Use Case and Motivation
 Live Interest Meter App
 Tests and User Study
 Conclusions
 Ongoing Work
12.04.2013 © FZI Forschungszentrum Informatik 2

3. Use Case and Motivation
Holding a good
presentation is hard!
12.04.2013 © FZI Forschungszentrum Informatik 3

4. Use Case and Motivation
How to get better at it?
• With practice
• Get feedback
Holding a good
• Learn from the feedback!
presentation is hard!
12.04.2013 © FZI Forschungszentrum Informatik 4

5. Use Case and Motivation
 LA for workplace learning
 Capturing of feedback for reflective learning
 Inspired by Quantified Self approaches
 Mass lectures, virtual courses and conferences
12.04.2013 © FZI Forschungszentrum Informatik 5

6. Use Case and Motivation
 LA for workplace learning
 Capturing of feedback for reflective learning
 Inspired by Quantified Self approaches
 Mass lectures, virtual courses and conferences
12.04.2013 © FZI Forschungszentrum Informatik 6

7. Live Interest meter - LIM App
 Gathering, aggregation and visualization of feedback
 Protagonist: the meter
 General
performance
 Speech speed
 Comprehension
12.04.2013 © FZI Forschungszentrum Informatik 7

8. LIM App: Evolution graph
 Available for Android and for Browsers (JavaScript)
 Browser to facilitate the configuration of the presenter
12.04.2013 © FZI Forschungszentrum Informatik 8

9. LIM App: Polls and Questions
 Polling and anonymous rated questions
12.04.2013 © FZI Forschungszentrum Informatik 9

10. LIM App: JavaScript Version
12.04.2013 © FZI Forschungszentrum Informatik 10

11. Experimental tests
 (1) Project meeting
 10 participants used LIM
 Discussion driven
12.04.2013 © FZI Forschungszentrum Informatik 11

12. Experimental tests
 (1) Project meeting
 10 participants used LIM
 Discussion driven
 (2) Lecture at university
Technical results very satisfactory
 15 participants
 Interactive session Acceptance results rather limited
12.04.2013 © FZI Forschungszentrum Informatik 12

13. Experimental tests
 Project meeting
 10 participants used LIM
 Discussion driven
 Lecture at university
Technical results very satisfactory
 15 participants
 Interactive session Acceptance results rather limited
Presenter‘s role well defined Suitable for big audiences
12.04.2013 © FZI Forschungszentrum Informatik 13

14. User Study
 Main goals
 Refine use case
 Guide further development
 In which scenarios and how can the quantification of feedback performed
by the LIM App support reflective learning?
 Which features are more appreciated by users, both presenters and
audience members?
12.04.2013 © FZI Forschungszentrum Informatik 14

15. User Study
 20 qualitative interviews
 Willing to substitute traditional questionnaires for captured feedback
 Anonymity appreciated (for feedback and questions)
  Polls but  Chat
 Reflection needs time – preferred after the session
 Online survey
 1 month (May-June 2012)
 120 participants Presenter
 87 valid responses Audience 44,82%
55,18%
12.04.2013 © FZI Forschungszentrum Informatik 15

16. User Study – Reaction to feedback
What do you consider realistic regarding
how you can react to the feedback (as presenter)?
12.04.2013 © FZI Forschungszentrum Informatik 16

17. User Study – Data collection
64,37 %
Data collected live and continuously is
better and more significant than only periodically collected data.
12.04.2013 © FZI Forschungszentrum Informatik 17

18. User Study – Information needed
Which information
would the presenters
like to know about
and which information
does the audience
want to evaluate?
12.04.2013 © FZI Forschungszentrum Informatik 18

19. User Study – Characteristics of the LIM App
How participants evaluated the LIM App features
12.04.2013 © FZI Forschungszentrum Informatik 19

20. Conclusion
78,16 % found the idea behind 57,47 % would like to test the
the LIM App very positive LIM App
 Readiness of presenters to learn retrospectively
 Motivate students to use the application
 Main concern is distraction
Applying Learning Analytics to informal workplace learning and
data gathered during the learning process
12.04.2013 © FZI Forschungszentrum Informatik 20

21. Towards a second prototype
 Usability improvements
 Support for reflection after the presentation
 From presenter to collaborative approach e.g. topics
 Mock-ups to adapt the results to real features
12.04.2013 © FZI Forschungszentrum Informatik 21

22. Any questions?
THANK YOU!
rivera@fzi.de
@veronicarp
vriverapelayo
12.04.2013 © FZI Forschungszentrum Informatik 22

23. Theoretical Model & Related Work
12.04.2013 © FZI Forschungszentrum Informatik 23

24. Theoretical background
[1] Applying Quantified Self Approaches to Support Reflective Learning. Verónica Rivera-Pelayo, Valentin Zacharias, Lars Müller,
Simone Braun. Learning Analytics and Knowledge 2012 (LAK 2012), Vancouver, Canada
[2] A Framework for Applying Quantified Self Approaches to Support Reflective Learning. Verónica Rivera-Pelayo, Valentin
Zacharias, Lars Müller, Simone Braun. IADIS International Conference on Mobile Learning (Mlearning 2012), Berlin, Germany
12.04.2013 © FZI Forschungszentrum Informatik 24

25. Related Work
 Clickers and feedback tools
 Polling
 Student outcomes and comprehension
 Student attendance and interest on the course
 Reflective Learning in TEL
 Student’s perspective
 Data from LMS, blogs or software used
 LIM App
 Improve lecturer/presenter’s professional performance
 Students give feedback and contribute to it
 Enrich lecture
 Feedback contextualized by polls, questions and chat
 Gathering of data during the lecture
4/12/2013 © FZI Forschungszentrum Informatik 25

26. ARS and Clickers
 [1] M. Akbari, G. Böhm, and U. Schroeder. Enabling communication and feedback in mass
lectures. In ICALT, pages 254{258, 2010.
 [2] M. Bonn, S. Dieter, and H. Schmeck. Kooperationstools der Notebook Universität Karlsruhe
(TH). In Mobiles Lernen und Forschen, pages 63-71. Klaus David, Lutz Wegener (Hrsg.),
November 2003.
 [3] J. E. Caldwell. Clickers in the large classroom: Current research and Best-Practice tips. CBE
Life SciEduc, 6(1):9-20, Mar. 2007.
 [4] D. Duncan and E. Mazur. Clickers in the Classroom: How to Enhance Science Teaching Using
Classroom Response Systems. Pearson Education, 2005.
 [5] J. Hadersberger, A. Pohl, and F. Bry. Discerning actuality in backstage - comprehensible
contextual aging. In EC-TEL, volume 7563 of Lecture Notes in Computer Science, pages 126-
139. Springer, 2012.
 [6] D. Kundisch, P. Herrmann, M. Whittaker, M. Beutner, G. Fels, J. Magenheim, W. Reinhardt, M.
Sievers, and A. Zoyke. Designing a Web-Based Application to Support Peer Instruction for Very
Large Groups. In ICIS '12, Research in Progress, Orlando, USA, December 2012.
 [7] G. Rubner. mbclick - an electronic voting system that returns individual feedback. In WMUTE,
pages 221-222. IEEE, 2012.
 [8] A. Wessels, S. Fries, H. Horz, N. Scheele, and W. Effelsberg. Interactive lectures: Effective
teaching and learning in lectures using wireless networks. Comput. Hum. Behav., 23(5):2524-
2537, Sept. 2007.
4/12/2013 © FZI Forschungszentrum Informatik 26

27. Reflective learning in TEL
 [1] R. Ferguson, S. B. Shum, and R. D. Crick. EnquiryBlogger: using widgets to support
awareness and reflection in a PLE Setting. In ARPLE11, PLE Conference 2011, Southampton,
UK, 11-13 July, 2011.
 [2] S. Govaerts, K. Verbert, E. Duval, and A. Pardo. The student activity meter for awareness and
self-reflection. In CHI '12 Extended Abstracts on Human Factors in Computing Systems, pages
869-884, New York, NY, USA, 2012. ACM.
 [3] J. L. Santos, S. Govaerts, K. Verbert, and E. Duval. Goal-oriented visualizations of activity
tracking: a case study with engineering students. In 2nd International Conference on Learning
Analytics and Knowledge, LAK '12, pages 143-152, New York, NY, USA, 2012. ACM.
4/12/2013 © FZI Forschungszentrum Informatik 27