The ImREAL project (http://www.imreal-project.eu) is researching how to meaningfully augment and extend existing experiential training simulators. The services developed support self-regulated, goal-, and application-oriented learning in adult training. We present results from a study evaluating a medical interview training simulator that has been augmented by an affective metacognitive scaffolding service and by user modelling exploiting social digital traces. Data from 152 medical students participating in this user trial were compared to the results of a prior trial on an earlier technology version. Findings show that students perceived the learning simulator positively and that the enhanced simulator led to increased feelings of success, less frustration, higher technical flow, and more reflection on learning. Interestingly, this cohort of users proved reluctant to provide their open social IDs to enrich their user models.

1. Affective Metacognitive Scaffolding and User Model
Augmentation for Experiental Training Simulators:
A Follow-up Study
Immersive Reflective Experience-based Adaptive Learning
8th European Conference on Technology Enhanced Learning (EC-TEL 2013)
Gudrun Wesiak, Adam Moore, Christina M. Steiner, Claudia
Hauff, Conor Gaffney, Declan Dagger, Dietrich Albert, Fionn Kelly, Gary
Donohoe, Gordon Power, & Owen Conlan

2. Overview
- Intro
- ImREAL
- Challenges and Approach
- ETU Simulator
- Simulator Augmentation
- Empirical study
- Methods and results
- Discussion & Conclusions
EC-TEL 2013 2

3. Intro
- Experiential Training Simulators
- For engaging and motivating learning experiences
- To create situational contexts to practice job-relevant skills
- Adult learning (Andragogy)
- Self-directed, experienced-based, goal-oriented, intrinsically
motivating, relevancy-driven (Knowles, 1984)
- Influenced by learner’s physical, digital and social world
- Should adapt to learner’s previous experiences, knowledge, ….
EC-TEL 2013 3

4. 4
ImREAL: Challenge
Simulated environment
 Augment existing training simulators by...
 Real-world activity modelling
 Enriched user modelling
 Affective-metacognitive scaffolding service
 Develop services for different simulators
Real-world experience
Close the gap
Medical interview
training
Fig. 2. Example learner interaction screen – the simulated situation is in the Dinner episode
where the host has to decide about ordering food for his business guests.
The simulated scenario includes four episodes: Greetings (situations embed arriving
on time, different norms about greetings, first impression, and use of body language),
Dinner (situations embed use of body language and different preferences about food
Business simulation &
Buddy program
Simulated
world
Real world
Forethought
Reflection

5. Experiential Training Simulator:
EmpowerTheUser - ETU
Medical Interview Training:
 Two Psychiatric Interview Scenarios  Mania & Depression
 Basic idea: Watch patient video and select appropriate answer
 2Training modes: Assess mode and practice mode
EC-TEL 2013 5

6. ETU: Assess mode
Performance Prediction and Scores
EC-TEL 2013 6
Interview steps (Process)
Interview skills
ETU scores

7. ETU: Practice mode
EC-TEL 2013 7
Visible Decision scores
Coach: feedback
on decision path

8. Thinking prompts and scaffolding service
ETU: Practice mode
Meta-cognitive scaffolding (MSS)
EC-TEL 2013 8

9. ETU: Practice mode
Meta-cognitive scaffolding (MSS)
EC-TEL 2013 9
Prompts are based on MAI-ROC (Schraw, 194)
Prompts are mapped to different interview situations

10. ETU: Practice mode:
Affective Metacognitive Scaffolding
EC-TEL 2013 10
SAM:
Smiley Affect Measurement technology
• Explicit assessment of current emotional state with SAM
• Selection of displayed prompts influenced by users’ emotional state
• Sentiment analysis of digital traces (U-SEM)
• Mappings of prompts to situations as weighted list

11. Empirical Study: Research Questions
• Simulator Augmentation:
• Is the simulation relevant for users?
• Are the services well integrated?
• Digital Traces:
• Usage, privacy & trust
EC-TEL 2013 12
• AMSS: Are there effect on…
• self-regulated learning?
• users’ learning experience?
• learning performance?
• motivation/affect?

12. Longterm Methodology:
Berthold et al., EC-TEL 2012
Participants
UT1: 143 TCD medical students, 22.7 years, 54.8% male,
97% real world interview experience (15.4% psychiatric)
UT2: 152 TCD medical students, 22.81 years, 50% male,
99% real-world interview experience (25.6% psychiatric)
• 2011
• Pure
Simulator
Baseline
• 2012
• ETU with
MSS
User Trial 1
• 2013
• AMSS, SA
M, U-SEM
User Trial 2
Methods
EC-TEL 2013 13

13. Methods: Procedure and Measures
EC-TEL 2013 14
Cohort survey (N=95)
Training in the simulator
(N=143/152)
Post-sim survey
(N=39/ 40)
Real-world interview
Post-int survey (N=7/19)
• Demographics
• Social Network use & attitudes
• Log-data (Usage, scores)
2 scenarios, 2 modes
• Real-World Relevance
• SRL-strategies
• Motivation & Affect
• Learning Experience
• Learning Performance
• Real-world relevance
• Motivation
• Fb on modes, AMSS, SAM, SN

14. Results: Log Data
Simulator Usage & Training Times
15
Usage (solid lines) and mean training times
(dashed) of assess and practice mode
 UT1: very few users for Mania
 UT2: short times for Depression
Assess vs. Practice:
 Assess: more users and longer times
UT1 vs. UT2:
 Same no. of users, but increased
overall training times (both modes)

15. Results
Which aspects are (not) effected?
No differences in ...
 Integration & perception of thinking prompts
 Self-regulated learning strategies (QSRL)
 Motivation to improve skills, to perform good, to apply what has
been learned (post-sim & post-int)
No difference between UT1 & UT2, but change from Baseline
 Perceived relevance of simulation
 Usability (SUS)
 Overall motivation
 Overall workload (decreased!)
Deacrease from UT1 to UT2
Motivation to learn more about interviewing (post-sim)
EC-TEL 2013 16
Baseline
UT1/UT2
UT1=UT2

16. Results
Relevance for Real-World Interviews
 UT2 students feel better prepared for real-world interviews
 Perception of relevancy & preparedness unchanged after real-world
interview
17

17. Results
Learning Experience
Flow (Flow short scale; Rheinberg et al., 2003)
 increase in overall flow, technical flow & 7 single items
i.e. stronger involvement into the task, greater feeling of smoothness and
fluency, smooth interplay between software components
18
Flow score means on item basis (left fig.) and overall means (and SE) for the
FKS, worry, and tech scales (right fig.)

18. Results
Learning Experience
Workload: Nasa-TLX (Hart & Staveland,1988)
 Lower overall workload in UT2
 UT2 users felt less frustrated & had stronger feelings of success
EC-TEL 2013 19
Mean scores (and SE) from the NASA-TX workload scales for the two user trials.

19. Results
Self-Regulated Learning
Reflection notes and AMSS prompts
• 1092 triggers provided (M=15/student) in UT2, 2001 in UT1
• MAI-prompts: 88.4% of 69 answers positive
• Increase in proportion of reflective notes
EC-TEL 2013 20
Table 1. Rounded percentages of content types for entries from the note-taking tools.
Note: Entries can be coded to more than one type, thus percentages may exceed 100.
Users Text entries Position Technical Notes Reflection
UT1 50 107 17 57 16 66
UT2 35 86 1 13 7 93

20. Results: Log Data
ETU Scores
ETU: predicted and achieved scores in UT1 and UT2
21
 Higher scores in UT2 for
introduction, eliciting and
planning phase
 Mainly underestimation of
scores in UT2

21. Results
Affect
 Affective state reported after simulation: Total affective state scales
(TASS, Nicolescu et al., 2006 )
 Affect during simulation:
 SAM was displayed 352 times (2.23 times / user) – used by 1 person
22
Mean TASS scores on 7 subscales plus overall mean score across all scales.
NO
Yes

22. Results
Social Networks
Usage: 81% use facebook, 20% twitter, < 5% Flickr, MySpace, linkedIn
 Nobody (of 152) provided SN-ID in simulator (13 did in cohort survey; 11 fb)
 For detailed results: see Moore, 2013 (Umap; LBR)
Reasons:
EC-TEL 2013

23. Results
Summary
• Changes from pure (BL) to augmented (UT) simulator
 ↑ Perceived relevance, usability, state motivation
 ↓ Overall workload
• Changes from MSS (UT1) to AMSS (UT2) simulator
 Relevancy: ↑ stronger feeling of preparedness
 SRL: ↑ effort (times), feeling of success, reflective notes;
 Performance: ↑ ETU-Scores
 Learning Experience:
↑ flow
↓ decrease in frustration
 Motivation: ↓ to learn more about interviewing
EC-TEL 2013 24

24. Conclusions & Challenges
• Metacognitive scaffolding supports learners on different levels
• Evaluation of user model augmentation
 Low user acceptance for exploiting their social digital traces
 Need to increase willingness to share SN-IDs
• Study learning activity rather than purely subjective reports
 Analysing learners’ real activities with the simulators
• (Non)-invasiveness vs. (un)-visibility
 SAM, thinking prompts
EC-TEL 2013 25

25. Conclusions & Challenges
• (Non)-invasiveness vs. (un)-visibility
 SAM now with triggers and presented in main window
EC-TEL 2013 26
NO
YES

26. Thanks for listening!
EC-TEL 2013 27