AI and ML methods for
Multimodal Learning Analytics
Kshitij Sharma
Department of Computer Science
Norwegian University of Science and Technology, Trondheim
1

Who am I?
2

Why do I like Multimodal Learning Analytics?
3

Why I like Multimodal Learning Analytics?
4

Context 1: Pacman
5

Context 1: Pacman
• Learning Context → Skill Acquisition
• 19 Participants
• 25-30 Minutes of game play
• Multimodal data → eye-tracking, EEG, EDA, HRV, Facial video, keystrokes
• Outcome → game score
6

Context 1: Pacman
Is multimodal data collection worth
the effort?
7

ML Pipeline
8

Context 1: Pacman
9

Context 2: Self-Assessment
10

Context 2: Self-Assessment
• Learning Context → Self Assessment
• 32 Participants
• 30 Minutes of solving programming problems (adaptive test)
• Multimodal data → eye-tracking, EEG, EDA, HRV, Facial video, keystrokes
• Outcome → test score and effort (guessing/solving)
11

Context 2: Self-Assessment
What about explainability of AI
pipelines with Multimodal data?
12

Context 2: Self-Assessment
13

Context 2: Self-Assessment (Easy result)
NRMSE
test
score
14

Context 2: Self-Assessment (Challenging result)
NRMSE
Effort
15

Context 2: Self-Assessment
Predicting the future!
16

• Attention
• Emotional intensity
• Cognitive load
• Mental workload
• Memory load
• Heart rate
• BVP
• EDA

Context 2: Self-Assessment
Predicting the future!
F-score (effortless/effort) → 0.90
18

Tell about future
19

Context 3: Debugging

Context 3: Debugging
• Learning Context → Programming
• 44 Participants
• 60 Minutes of solving programming problems (adaptive test)
• Multimodal data → eye-tracking, EEG, EDA, HRV, Facial video, keystrokes
• Outcome → debugging performance
21

Context 3: Debugging
Predicting complex performance!
22

ML pipeline
23
Random
forest

ML pipeline (Feature extraction)
• Logs: Reading-writing (R-W) episodes; Use of debugger; Use of variable view
• E4: mean and SD of BVP, TMP, and EDA and the mean of HR
24

Context 3: Debugging
25

Context 4: Game based learning
26

Context 4: Game based learning
• Learning Context → Motion based educational games
• 40 Participants
• 30 Minutes of solving mathematics problems
• Multimodal data → eye-tracking, motion, EDA, HRV, system logs
• Outcome → game performance
27

Context 4: Game based learning
Towards designing AI agent to
support students
28

Context 4: Game based learning

Context 4: Game based learning
• Information processing Index
• Cognitive load
• Mean HR
• Grab-match Differential
Most important features for the agent
30

Context 5: Collaborative Concept Map
• Learning Context → Video based Learning + Synthesis
• 82 Participants
• 20 Minutes of concept map creation
• Multimodal data → eye-tracking, audio, dialogues, system logs
• Outcome → Collaborative Concept map correctness and individual
learning gain
31

Context 6: Collaborative ITS
• Learning Context → Intelligent Tutoring Systems
• 50 Participants
• 45 Minutes of concept map creation
• Multimodal data → eye-tracking, audio, dialogues, system logs
• Outcome → Learning gain
32

LSTM
33
Forget gate Input gate
Cell State Forward
propagation
Output gate

Contexts 5 & 6: Collaborative settings
• Gaze similarity
• Gaze entropy
• Cognitive load
• Joint mental effort
• Audio: autocorrelation
• Audio: energy
• Audio: shape of envelope
• Dialogue codes
• Log events
34

Contexts 5 & 6: Collaborative settings
• Collaborative Concept map
• Best NRMSE: 5.2
• Best combination: audio-gaze
• Collaborative ITS
• Best NRMSE: 5.1
• Best combination: audio-gaze
35

Generalizability across contexts
Deep Features from Facial data
36

Generalizability across contexts
Temporal Features from Physiological data
37

Generalizability across contexts (individual
learning)
Train Using Test Using NRMSE on test dataset
Pacman, Self-Assessment Debugging 9.24 (1.6)
Pacman, Debugging Self-Assessment 8.27 (2.1)
Self-Assessment,
Debugging
Pacman 8.26 (1.9)
Data Used: Facial videos and Wristband data (HRV, EDA)
38

Generalizability across contexts (individual
learning)
Train Using Test Using NRMSE on test dataset
Pacman, Self-Assessment,
Debugging
Motion based
game
10.94 (1.4)
Pacman, Self-Assessment,
Motion based game
Debugging 9.74 (1.1)
Pacman, Debugging, Motion
based game
Self-Assessment 9.27 (0.9)
Self-Assessment, Debugging,
Motion based game
Pacman 10.08 (1.3)
Data Used: Wristband data (HRV, EDA)

Generalizability across contexts (collaborative
learning)
Deep Features from Eye-tracking data
40

Generalizability across contexts (collaborative
learning)
Temporal Features from Eye-tracking measurements
(cognitive load, information processing index, entropy, stability,
anticipation, fixation durations)
41

Generalizability across contexts (collaborative
learning)
Data Used: Eye-tracking
Train Using Test Using NRMSE on test dataset
All Individual Contexts Collaborative
Concept map
19.89 (5.4)
All Individual Contexts Collaborative ITS 21.16 (6.2)
All Individual Contexts +
Collaborative ITS
Collaborative
Concept map
6.7 (1.5)
All Individual Contexts +
Collaborative Concept map
Collaborative ITS 6.5 (1.2)
42

What’s next???
• Online learning → System logs?
• Online learning → System logs and generated multimodal data?
• The variance in these contexts was huge → can we design something
with more similarities within the contexts?
43