1.
Learning Analytics
Ruth Cobos
Biometrics and Behavior
Understanding Technologies for
e-Learning Platforms: Challenges
and Opportunities
Ruth Cobos
ruth.cobos@uam.es
Aythami Morales
aythami.morales@uam.es
1

2.
Outline
➢ Motivation
➢ Biometrics and Behaviour Analysis Technologies
➢ Opportunities in e-learning environments
➢ Multimodal Attention Level Estimation
➢ Case Study:
➢ Context: WebApp MOOC & LA System (edX-LIMS)
➢ Experimental Lab & Data Sets
➢ Initial findings & Research questions
2

3.
Motivation
3

4.
Motivation: E-learning Platforms
Online Learning
• E-learning platforms and virtual education are estimated to grow over a
7% per year between 2018 and 2023, reaching a turnover around 240,000
million dollars.
• E-learning platforms allow to capture information to better understand the
student behavior and create personalized environments.
Market Growth
Emotional State Cognitive Activity
Increased Security
4

5.
Behavioral Understanding
• Action Recognition
• Attention Level Estimation
• Heart Rate Estimation
Biometrics and Behavioral Modelling in e-Learning Platofrms
• Keystroke Dynamics
• Facial Recognition
• Pose Estimation
Biometrics
5

6.
Biometrics and Behavior Analysis
Technologies
6

7.
Biometrics Traits Taxonomy Physiological - Biological Traits
• DNA
• EKG, EEG
• Odor
Behavioral Traits
• Speech - Voice
• Signature
• Handwriting
• Gait
• Keystroke dynamics
• Mouse dynamics
• Web-based biometrics
Physiological - Morphological Traits
• Fingerprints
• Face
• Infrared facial thermography
• Iris
• Ear
• Retinal scan
• Hand & finger geometry
• Blood vessel imaging
• Body profile & body parts
• A. K. Jain, K. Nandakumar and A. Ross, “50 years of biometric research: Accomplishments, challenges, and opportunities,” Pattern Recognition Letters, 79, 80-105, 2016.
7

8.
Biometrics Traits Taxonomy Physiological - Biological Traits
• DNA
• EKG, EEG
• Odor
Behavioral Traits
• Speech - Voice
• Signature
• Handwriting
• Gait
• Keystroke dynamics
• Mouse dynamics
• Web-based biometrics
Physiological - Morphological Traits
• Fingerprints
• Face
• Infrared facial thermography
• Iris
• Ear
• Retinal scan
• Hand & finger geometry
• Blood vessel imaging
• Body profile & body parts
• A. K. Jain, K. Nandakumar and A. Ross, “50 years of biometric research: Accomplishments, challenges, and opportunities,” Pattern Recognition Letters, 79, 80-105, 2016.
8
Something WE ARE
Something WE DO

9.
History of Biometrics
• Evidence suggests fingerprints were used as a personal mark around 500
B.C.
• Early Chinese merchants used fingerprints to settle business
transactions.
• Chinese used fingerprints and footprints to differentiate people.
• Early Egyptian uses:
• Traders were identified by their physical descriptors.
• Differentiate between trusted traders of known reputation and
previous successful transactions, and those new to the market.
Chauvet cave
(France)
9

10.
Biometrics vs. Passwords
It is easy to crack passwords because most of them are weak (related to
personal details, typical words or sequential numbers).
10

11.
Challenges in Biometric Security
Biometrics:
• Cannot be lost or forgotten, but must be enrolled.
But… many challenges:
• Acquisition quality.
• Device interoperability.
• Variability factors.
• Attacks to biometric systems.
• Aging.
• … many more.
• A. K. Jain, K. Nandakumar and A. Ross, “50 years of biometric research: Accomplishments, challenges, and opportunities,” Pattern Recognition Letters, 79, 80-105, 2016. 11
Bad quality samples
Sensor interoperability
Security attacks
Fake samples
Aging

12.
Opportunities in e-learning
environments
12

13.
Understanding and Modeling Student Interaction
J. Hernandez-Ortega, R. Daza, A. Morales, J. Fierrez and J. Ortega-Garcia, “edBB: Biometrics and Behavior for Assessing Remote Education”. Proc.
of AAAI Workshop on Artificial Intelligence for Education (AI4EDU), New York, NY, USA, February 2020.
13

14.
Face Detection and Recognition:
o Continuous monitoring.
o Performances over 99% in controlled
scenarios.
o Privacy ans bias concerns.
Pose Estimation:
o Behavior analysis.
o 3D face modelling from 2D images.
o No extra sensors (only webcam).
Face Biometrics

15.
Keystroke Identification
A. Acien, J.V. Monaco, A. Morales, R. Vera-Rodriguez, J. Fierrez, “TypeNet: Scaling up Keystroke Biometrics,” Proc. of IAPR/IEEE International Joint Conference on
Biometrics (IJCB), Houston, USA, 2020.
A. Morales, A. Acien, J. Fierrez, J.V. Monaco, R. Tolosana, R. Vera-Rodriguez, Javier Ortega-Garcia, “Keystroke Biometrics in Response to Fake News Propagation in a
Global Pandemic,” Proc. of IEEE International Workshop on Secure Digital Identity Management (SDIM), Madrid, Spain, 2020.
×7
State of the art
(before deep learning)
Our current system
• Free text
• 25 keystrokes events
• 168,000 users
• 2M examples
15

16.
Modeling Emotional State through Heart Rate
J. Hernandez-Ortega, J. Fierrez, A. Morales, D. Diaz, "A Comparative Evaluation of Heart Rate Estimation Methods using Face Videos," Proc. of IEEE Intl.
Workshop on Medical Computing (MediComp), Madrid, Spain, 2020.
Altered States can be observed in Heart Rate signals
16

17.
Multimodal Attention Level
Estimation
17

18.
Attention is all you need
• Attention plays a very important role in
students' success in the classroom.
• Attention allows students to “tune out”
unrelated information, background
noise, visual distractions, and even
their own thoughts.
18

19.
• Since the 70s there are studies that connect the eye blink rate
with the cognitive activity like attention:
o Lower eye blink rates can be associated to high attention
o Higher eye blink rates are related to low attention
Attention Level and Blink Rate
Daza, R.; Morales, A.; Fierrez, J.; and Tolosana, R. 2020. mEBAL: A Multimodal Database for Eye Blink Detection and Attention Level Estimation. In Proc. Intl.
Conf. on Multimodal Interaction, 32–36.
However, the potential use of automatic behavior detectors to infer the
attention level of users have not been evaluated in realistic scenarios.
19

20.
https://github.com/BiDAlab/mEBAL
https://github.com/BiDAlab/edBBdb
58 students performing 8 different
tasks categorized in three groups:
• Enrollment Form
• Writing Questions
• Multiple Choice Questions
EEG Band
Mindwave by NeuroSky
Form Logical Problems Crosswords Multiple Choice
15 to 30 minutes
Material: Description of mEBAL Database
20

21.
Attention Levels
PDF
Low and High Attention Level Estimation
Probability Density Function of the attention levels in mEBAL
We argue that it is possible to
predict high/low sustained
levels of attention only with
data from webcam images.
21

22.
Attention Levels
PDF
Low
Attention
High
Attention
Normal
Attention
Low and High Attention Level Estimation
Probability Density Function of the attention levels in mEBAL
22
We argue that it is possible to
predict high/low sustained
levels of attention only with
data from webcam images.

23.
Attention Level Estimation based on Multimodal Behavior Analysis
23
Webcam (input data)
EEG (Groundtruth)
SVM
High/Low
Attention
Eye Blink
Detector
Landmark
Detector
Head Pose
Estimator
Face
Detector
Facial
Expressions
Model
SVM
SVM
SVM
Multimodal Face Analysis
Learning Framework

24.
Challenges
mEBAL presents examples with variations on illuminations, rich in poses, changes
in the distance, etc. The attention level is highly user-dependent.
24
Changes in the Illumination Different Poses and Distances

25.
Unimodal Attention Level Estimation Results
25

26.
Unimodal Attention Level Estimation Results
26
High/Low Blink Rate = Low/High Attention Level
R. Daza, A. Morales, J. Fierrez, R. Tolosana, "mEBAL: A Multimodal Database for Eye Blink Detection and Attention Level Estimation", ACM Intl.
Conf. on Multimodal Interaction (ICMI), Utrecht, The Netherlands, October 2020.

27.
Multimodal Attention Level Estimation Results
27
High attention periods are easier to
recognize than low attention
periods.
Level of attention and behavior
features show a correlation.
The multimodal results outperform
the unimodal

28.
Multimodal Attention Level Estimation Results
28
High attention periods are easier to
recognize than low attention
periods.
Level of attention and behavior
features show a correlation.
The multimodal results outperform
the unimodal performance

29.
Case Study
Learning Analytics + Behavior Analysis
29

30.
2
• Insuficient feedback
• Feeling of insolation
• Lack of interactions with
instructors
MOOC learners
Motivation
https://www.edx.org/school/uamx

31.
WebApp MOOC
“Introduction to Development of Web Applications”
➢ 5-week course, 5 units
➢ Technology to learn: HTML, CSS, Python, JSON, JavaScript and
Ajax
➢ Couse contents in multimedia resources, discussion forums and
course evaluation activities in form of graded assignments
https://www.edx.org/course/introduccion-al-desarrollo-de-aplicaciones-web-2
31

32.
Learning Analytics System: edX-LIMS
System for Learning Intervention and its Monitoring
for edX MOOCs
Cobos, R., Soberón, J. A proposal for Monitoring the Intervention Strategy on the learning of MOOC learners.
Learning Analytics Summer Institute Spain 2020, LASI-Spain 2020. http://ceur-ws.org/Vol-2671/paper07.pdf
edX-LIMS
MOOC
Dashboard Generation
Pascual, I., Cobos, R. A proposal for predicting and intervening on MOOC learners’ performance in real
time. LASI-Spain 2022. http://ceur-ws.org/Vol-3238/paper4.pdf 32

33.
EEG Band
Smart Watch
RGB Camera
NIR Camera
RGB Camera
Screen,
sound,
keyboard,
and mouse
capture
Experimental Lab - edBB Platform
Learner in
the MOOC
Learner in the
Dashboard
RGB Camera
33

34.
EEG Band
• Attention
• Meditation
• Blink
• Brain waves
Smart Watch
• Heart Rate
• Acceleration
• Gyroscope
Sound capture
NIR Camera
RGB Camera
Data Sets - edBB Platform
Learner
Concentration
Learner Behavior
Learning Context
Keyword capture
Mouse capture
• Press/release
• Key Unicode
• Press/release
• Position in screen
• Move
• Drag and drop
• Mouse wheel spin
• Depth
• Infrared
Screen capture
• Color
34

35.
Data Sets – WebApp MOOC & edX-LIMS
edX-LIMS
MOOC
Learner
Performance
Learner
Success Prediction
Learner Intercations
with the course
Learner Intercations
with the LA system
Knowledge extracted
by the LA system
Learner Interest
Learner
Self-Regulation
Learner Problems
Learner Progress
35
Learner Feedback

36.
While student was writing in a
text box where he can give
feedback to instructors, it was
recorded that he had:
• High Attention
• Moderate Heart Rate
Initial Findings
Screen Capture (Video Monitor) Learner Dashboard
The Learner was very
concentrated on the task of giving
the instructors his opinion on the
information provided by the
dashboard
36

37.
About the MOOC:
➢ What parts of the course can be improved?
➢ Can the course videos be improved?
➢ Can we predict which students are at risk of dropping out?
➢ …
About the Learning Analytics System:
➢ Which parts of the Learner Dashboard can be improved?
➢ Are the charts in the Learner Dashboard understandable?
➢ Can the Intervention Strategy be improved?
➢ …
Research questions
37

38.
Learning Analytics
Ruth Cobos
Biometrics and Behavior
Understanding Technologies for
e-Learning Platforms: Challenges
and Opportunities
Ruth Cobos
ruth.cobos@uam.es
Aythami Morales
aythami.morales@uam.es
38