This document provides an overview of multimodal detection of affective states through diverse technologies. It introduces brain-computer interfaces, face-based emotion recognition systems, eye-tracking systems, and physiological sensors as sensing devices that can be used to detect affective states. The document outlines a two-session course that will cover these sensing devices, how to gather and manage the data collected, and approaches to analyze the data to understand users' affective states.
201404 Multimodal Detection of Affective States: A Roadmap Through Diverse Technologies
1. Multimodal Detection of Affective States:
A Roadmap Through Diverse Technologies
Javier Gonzalez-Sanchez, Maria-Elena Chavez-Echeagaray
Robert Atkinson, Winslow Burleson
!
!
School of Computing, Informatics, and Decision Systems Engineering
Arizona State University
Tempe, Arizona, USA
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work
must be honored. For all other uses, contact the owner/author(s). Copyright is held by the author/owner(s).
CHI 2014, April 26–May 1, 2014, Toronto, Ontario, Canada. ACM 14/04
1
2. Advancing Next Generation Learning Environments Lab
&
Motivational Environments Group
!
School of Computing, Informatics, and Decision Systems Engineering
Arizona State University
2
About Us
3. This work was supported by Office of Naval Research under
Grant N00014-10-1-0143
Robert Atkinson
!
Dr. Robert Atkinson is an Associate Professor in the Ira A. Schools of
Engineering and in the Mary Lou Fulton Teacher’s College. His
research explores the intersection of cognitive science, informatics,
instructional design, and educational technology.
His scholarship involves the design of instructional material,
including books, and computer-based learning environments
according to our understanding of human cognitive architecture and
how to leverage its unique constraints and affordances.
His current research focus involves the study of engagement and
flow in games.
3
Principal Investigator
4. This work was supported by Office of Naval Research under
Grant N00014-10-1-0143
Winslow Burleson
!
Dr. Winslow Burleson received his PhD from the MIT Media Lab,
Affective Computing Group.
He joined ASU's School of Computing and Informatics and the Arts,
Media, and Engineering graduate program at ASU in 2006. He has
worked with the Entrepreneurial Management Unit at the Harvard
Business School, Deutsche Telekom Laboratories, SETI Institute,
and IBM's Almaden Research Center where he was awarded ten
patents.
He holds an MSE from Stanford University's Mechanical Engineering
Product Design Program and a BA in Bio-Physics from Rice
University. He has been a co-Principal Investigator on the Hubble
Space Telescope's Investigation of Binary Asteroids and consultant
to UNICEF and the World Scout Bureau.
4
Principal Investigator
5. This work was supported by Office of Naval Research under
Grant N00014-10-1-0143
Javier Gonzalez-Sanchez
!
Javier is a doctoral candidate in Computer Science at Arizona State
University. His research interests are affect-driven adaptation,
software architecture, affective computing, and educational
technology.
He holds an MS in Computer Science from the Center for Research
and Advanced Studies of the National Polytechnic Institute and a BS
in Computer Engineering from the University of Guadalajara.
His experience includes 12+ years as a software engineer and 9+
years teaching undergraduate courses at Tecnologico de Monterrey
and graduate courses at Universidad de Guadalajara.
5
Doctoral Candidate
6. This work was supported by Office of Naval Research under
Grant N00014-10-1-0143
Helen Chavez-Echeagaray
!
Helen is a doctoral candidate in Computer Science at Arizona State
University. Her interests are in the areas of affective computing,
educational technology (including robotics), and learning processes.
The Tecnológico de Monterrey campus Guadalajara conferred upon
her the degree of MS in Computer Science and the degree of BS in
Computer Systems Engineering.
Before starting her PhD program she was a faculty member for 8+
years at Tecnologico de Monterrey. Her experience includes an
administrative position and software development.
6
Doctoral Candidate
10. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Justification
One novel part of the planning and design of the interaction between people
and computers, related to the facet of securing user satisfaction, is the capability
of systems to adapt to their individual users by showing empathy.
!
!
Being empathetic implies that the computer is able to recognize a user’s
affective states and understand the implication of those states.
!
!
Detection of affective states is a step forward to provide machines with
the necessary intelligence to identify and understand human emotions and then
appropriately interact with humans. Therefore, it is necessary to equip computers
with hardware and software to enable them to perceive users’ affective states and
then use this understanding to create more harmonic interactions.
10
11. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Motivation
This course provides a description and demonstration of tools and methodologies
necessary for automatically detecting affective states.
!
Automatic detection of affective states requires the computer
!
(1) to gather information that is complex and diverse;
!
(2) to process and understand information integrating several sources
(senses) that could range from brain-wave signals and biofeedback readings, to
face-based and gesture emotion recognition to posture and pressure sensing; and
!
(3) once the data is integrated, to apply perceiving algorithms software and
data processing tools to understand user’s status.
!
During the course we will review case examples (datasets) obtained from
previous research studies.
11
12. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Objectives
Attendees of this course will
!
Learn about sensing devices used to detect affective states including
brain-computer interfaces, face-based emotion recognition systems, eye-
tracking systems and physiological sensors.
!
Understand the pros and cons of the sensing devices used to detect
affective states.
!
Learn about the data that is gathered from each sensing device and
understand its characteristics. Learn about what it takes to manage (pre-
process and synchronize) affective data.
!
Learn about approaches and algorithms used to analyze affective data and
how it could be used to drive computer functionality or behavior.
12
17. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Concepts
Multimodal Detection
of Affective States
instinctual reaction to stimulation
feelings, emotions
How do you feel?
physiological physical
17
18. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Concepts
18
Until recently much of the affective data gathered by systems has relied heavily on
learner’s self-report of their affective state, observation, or software data
logs [1].
!
But now many systems have started to include data from the physical
manifestations of affective states through the use of sensing devices and the
application of novel machine learning and data mining algorithms to deal with the
vast amounts of data generated by the sensors [2][3].
!
!
!
!
!
!
!
!
!
[1] R.S.J. Baker, M.M.T Rodrigo and U.E. Xolocotzin, “The Dynamics of Affective Transitions in Simulation Problem-solving Environments,” Proc. Affective Computing
and Intelligent Interaction: Second International Conference (ACII ’07), A. Paiva, R. Prada & R. W. Picard (Eds.), Springer Verlag, Vol. Lecture Notes in Computer
Science 4738, pp. 666-677.
!
[2] I. Arroyo, D. G. Cooper, W. Burleson, F. P. Woolf, K. Muldner, and R. Christopherson, “Emotion Sensors Go to School,” Proc. Artificial Intelligence in Education:
Building Learning Systems that Care: from Knowledge Representation to Affective Modelling, (AIED 09), V Dimitrova, R. Mizoguchi, B. du Boulay & A. Grasser (Eds.),
IOS Press, July 2009, vol. Frontiers in Artificial Intelligence and Applications 200, pp. 17-24.
!
[3] R. W. Picard, Affective Computing, MIT Press, 1997.
19. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Concepts
Multimodal Detection
of Affective States
measurement
identify the presence of ...
physiological measures
physical appearance
self-report
19
21. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Sensing Devices
21
Sensing devices obtain data from the user about his physiological responses and
body reactions.
Sensing devices are hardware devices that collect quantitative data as measures
of physiological signals of emotional change. We call the measures provided by
the sensing devices raw data.
Our approach includes the use of brain-computer interfaces, eye-tracking systems,
biofeedback sensors, and face-based emotion recognition systems [4].
The use of several sensing devices either to recognize a broad range of emotions
or to improve the accuracy of recognizing one emotion is referred to as a
multimodal approach.
!
!
!
!
[4] J. Gonzalez-Sanchez, R. M. Christopherson, M. E. Chavez-Echeagaray, D. C. Gibson, R. Atkinson, W. Burleson, “ How to Do Multimodal Detection of Affective
States?,” ICALT, pp.654-655, 2011 IEEE 11th International Conference on Advanced Learning Technologies, 2011
23. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
BCI
Brain-Computer Interfaces (BCI)
!
It is a particular type of a physiological instrument that uses brainwaves as
information sources (electrical activity along the scalp produced by the firing of
neurons within the brain).
!
Emotiv® EPOC headset [5] device will be used to show how to collect and work
with this kind of data.
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
[5] Emotiv - Brain Computer Interface Technology. Retrieved April 26, 2011, from http://www.emotiv.com.
23
24. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
BCI
Wireless Emotiv® EEG Headset.
!
The device reports data with intervals of 125 ms (8 Hz).
!
The raw data output includes 14 values (7 channels on each brain
hemisphere: AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and
AF4) and two values of the acceleration of the head when leaning (GyroX
and GyroY).
!
The Affectiv Suite reports 5 emotions: engagement, boredom,
excitement, frustration, and meditation.
!
And the Expressiv Suite reports facial gestures: blink, wink (left and
right), look (left and right), raise brow, furrow brow, smile, clench, smirk (left
and right), and laugh.
!
!
24
25. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 25
Electrodes are situated and labeled according to the CMS/DRL configuration [6][7]
BCI
[6] Sharbrough F, Chatrian G-E, Lesser RP, Lüders H, Nuwer M, Picton TW. American Electroencephalographic Society Guidelines for Standard Electrode Position
Nomenclature. J. Clin. Neurophysiol 8: 200-2.
!
[7] Electroencephalography. Retrieved November 14th, 2010, from Electric and Magnetic Measurement of the Electric Activity of Neural Tissue: www.bem.fi/book/13/13.htm
26. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Emotiv
Systems
$299
emotions EEG data facial gestures
26
BCI
30. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 30
Field Description Values
Timestamp
It is the timestamp (date and time) of the computer running
the system. It could be used to synchronize the data with
other sensors.
Format "yymmddhhmmssSSS"
(y - year, m - month, d - day, h - hour, m - minutes, s -
seconds, S - milliseconds).
UserID Identifies the user. An integer value.
Wireless Signal Status Shows the strength of the signal. The value is from 0 to 4, 4 being the best.
Blink, Wink Left and Right,
Look Left and Right, Raise
Brow, Furrow, Smile,
Clench, Smirk Left and
Right, Laugh
Part of the expressive suite.
Values between 0 and 1, 1 being the value that
represents the highest power/probability for this
emotion.
Short Term and Long
Term Excitement,
Engagement / Boredom,
Meditation, Frustration
Part of the Affective Suite.
Values between 0 and 1, 1 being the value that
represents the highest power/probability for this
emotion.
AF3, F7, F3, FC5, T7,
P7, O1, O2, P8, T8,
FC6, F4, F8, AF4.
Raw data coming from each of the 14 channels. The name of
these fields were defined according with the CMS/DRL
configuration [XXX][XXXX].
Values of 4000 and higher.
GyroX and GyroY
Information about how the head moves/accelerates according
to X and Y axis accordingly.
BCI
35. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Face-Based Recognition
Face-based emotion recognition systems
!
These systems infer affective states by capturing images of the users’ facial
expressions and head movements.
!
We are going to show the capabilities of face-based emotion recognition systems
using a simple 30 fps USB webcam and software from MIT Media Lab [8].
!
!
!
!
!
!
!
!
!
!
!
!
[8] R. E. Kaliouby and P. Robinson, “Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures,” Proc. Conference on
Computer Vision and Pattern Recognition Workshop (CVPRW ‘04), IEEE Computer Society, June 2004, Volume 10, p. 154.
35
36. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Face-Based Recognition
MindReader API enables the real time analysis, tagging and inference of
cognitive affective mental states from facial video in real-time.
!
This framework combines vision-based processing of the face with predictions
of mental state models to interpret the meaning underlying head and facial
signals over time.
!
It provides results at intervals of approximately 100 ms (10 Hz).
!
!
With this system it is possible to infer emotions such as agreeing,
concentrating, disagreeing, interested, thinking, and unsure.
!
(Ekman and Friesen 1978) – Facial Action Coding System, 46 actions (plus
head movements).
36
37. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Demo
MindReader Software from MIT Media Lab
37
Face-Based Recognition
40. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 40
Field Description Values
Timestamp
It is the timestamp (date and time) of the computer running
the system. It could be used to synchronize the data with
other sensors.
Format "yymmddhhmmssSSS"
(y - year, m - month, d - day, h - hour, m - minutes,
s - seconds, S - milliseconds).
Agreement,
Concentrating,
Disagreement,
Interested, Thinking,
Unsure
This value shows the probability of this emotion being present
on the user at a particular time (frame).
This value is between 0 and 1.
If the value is -1 it means it was not possible to define
an emotion.
This happens when the user's face is out of the
camera focus.
Mind Reader
43. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Eye-tracking systems
!
These are instruments that measure eye position and eye movement in order to
detect zones in which the user has particular interest in a specific time and
moment.
!
Datasets from Tobii®Eye-tracking system [9] data will be shown.
!
!
!
!
!
!
!
!
!
!
!
[9] Tobii Technology - Eye Tracking and Eye Control. Retrieved April 26, 2011, from http://www.tobii.com.
Eye-Tracking System
43
44. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Eye-Tracking System
Tobii® Eye Tracker.
!
The device reports data with intervals of 100 ms (10Hz).
!
The output provides data concerning attention direction (gaze-x, gaze-y),
duration of fixation, and pupil dilation.
44
45. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Demo
Tobii® Eye Tracker
45
Eye-Tracking System
46. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 46
Field Description Values
Timestamp
It is the timestamp (date and time) of the computer running
the system. It could be used to synchronize the data with
other sensors.
Format "yymmddhhmmssSSS"
(y - year, m - month, d - day, h - hour, m - minutes, s -
seconds, S - milliseconds).
GazePoint X
The horizontal screen position for either eye or the average
for both eyes. This value is also used for the fixation definition.
0 is the left edge, the maximum value of the horizontal
screen resolution is the right edge.
GazePoint Y
The vertical screen position for either eye or the average for
both eyes. This value is also used for the fixation definition.
0 is the bottom edge, the maximum value of the
vertical screen resolution is the upper edge.
Pupil Left Pupil size (left eye) in mm. Varies
Validity Left Validity of the gaze data.
0 to 4. 0 if the eye is found and the tracking quality
good. If the eye cannot be found by the eye tracker,
the validity code will be 4.
Pupil Right Pupil size (right eye) in mm. Varies
Validity Right Validity of the gaze data.
0 to 4. 0 if the eye is found and the tracking quality
good. If the eye cannot be found by the eye tracker,
the validity code will be 4.
FixationDuration Fixation duration. The time in milliseconds that a fixation lasts. Varies
Event Events, automatic and logged, will show up under Event. Varies
AOI
Areas Of Interest if fixations on multiple AOIs are to be written
on the same row.
Varies
Eye-Tracking System
50. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Galvanic Skin Conductance
Provide information on the activity of physiological functions of an individual.
!
!
Arousal detection. Measures the electrical conductance of the skin, which
varies with its moisture level that depends on the sweat glands, which are
controlled by the sympathetic and parasympathetic nervous systems. [10]
!
!
Hardware designed by MIT Media Lab.
!
!
It is a Wireless Bluetooth device that reports data in intervals of
approximately 500 ms (2Hz)
!
!
!
!
[10] M. Strauss, C. Reynolds, S. Hughes, K. Park, G. McDarby, and R.W. Picard, “The HandWave Bluetooth Skin Conductance Sensor,” Proc. First International
Conference on Affective Computing and Intelligent Interaction (ACII 05), Springer-Verlang, Oct. 2005, pp. 699-706, doi:10.1007/11573548_90.
50
53. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 53
Field Description Values
Timestamp
It is the timestamp (date and time) of the computer running
the system. It could be used to synchronize the data with
other sensors.
Format "yymmddhhmmssSSS"
(y - year, m - month, d - day, h - hour, m - minutes, s -
seconds, S - milliseconds).
Battery Voltage Level of the battery voltage. 0 - 3 Volts
Conductance Level of arousal. 0 - 3 Volts
Galvanic Skin Conductance
57. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Pressure Sensor
Provide information on the activity of physiological functions of an individual.
!
!
Pressure sensors are able to detect the increasing amount of pressure
(correlated with levels of frustration) that the user puts on a mouse, or
any other controller (such as a game controller) [11].
!
!
Hardware designed by MIT Media Lab.
!
!
It is a serial device that reports data in intervals of approximately 150 ms
(6Hz).
!
!
!
!
[11] Y. Qi, and R. W. Picard, "Context-Sensitive Bayesian Classifiers and Application to Mouse Pressure Pattern Classification," Proc. International Conference on
Pattern Recognition (ICPR 02), Aug. 2002, vol 3, pp. 30448, doi:10.1109/ICPR.2002.1047973.
57
60. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 60
Field Description Values
Timestamp
It is the timestamp (date and time) of the computer running
the system. It could be used to synchronize the data with
other sensors.
Format "yymmddhhmmssSSS"
(y - year, m - month, d - day, h - hour, m - minutes, s -
seconds, S - milliseconds).
Right Rear Sensor positioned in the right rear of the mouse. 0 - 1024. 0 being the highest pressure.
Right Front Sensor positioned in the right front of the mouse. 0 - 1024. 0 being the highest pressure.
Left Rear Sensor positioned in the left rear of the mouse. 0 - 1024. 0 being the highest pressure.
Left Front Sensor positioned in the left front of the mouse. 0 - 1024. 0 being the highest pressure.
Middle Rear Sensor positioned in the middle rear of the mouse. 0 - 1024. 0 being the highest pressure.
Middle Front Sensor positioned in the middle front of the mouse. 0 - 1024. 0 being the highest pressure.
Pressure Sensor
62. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 62
Pressure Sensor
The raw data from the mouse is processed to obtain meaningful
information [12].
The data obtained is a normalized value from the six different sensors
on the mouse.
!
!
!
!
!
!
[12] Cooper, D., Arroyo, I., Woolf, B., Muldner, K., Burleson, W., and Christopherson, R. (2009). Sensors model student self concept in the classroom,
User Modeling, Adaptation, and Personalization, 30--41
66. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Posture detection using a low-cost, low-resolution pressure sensitive seat
cushion and back pad.
!
Developed at ASU based on experience using a more expensive high
resolution unit from the MIT Media Lab [13].
!
!
!
!
!
!
!
!
!
[13] S. Mota, and R. W. Picard, "Automated Posture Analysis for Detecting Learners Interest Level," Proc. Computer Vision and Pattern Recognition Workshop
(CVPRW 03), IEEE Press, June 2003, vol. 5, pp. 49, doi:10.1109/CVPRW.2003.10047.
Posture Sensor
66
69. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 69
Field Description Values
Timestamp
It is the timestamp (date and time) of the computer running
the system. It could be used to synchronize the data with
other sensors.
Format "yymmddhhmmssSSS"
(y - year, m - month, d - day, h - hour, m - minutes, s -
seconds, S - milliseconds).
AccX Value of the X axis of the accelerometer. Varies
AccY Value of the Y axis of the accelerometer. Varies
Right Seat Sensor positioned in the right side of the seat cushion. 0 - 1024. 1024 being the highest pressure.
Middle Seat Sensor positioned in the middle of the seat cushion. 0 - 1024. 1024 being the highest pressure.
Left Seat Sensor positioned in the left side of the seat cushion. 0 - 1024. 1024 being the highest pressure.
Right Back Sensor positioned in the right side of the back pad. 0 - 1024. 1024 being the highest pressure.
Middle Back Sensor positioned in the middle of the back pad. 0 - 1024. 1024 being the highest pressure.
Left Back Sensor positioned in the left side of the back pad. 0 - 1024. 1024 being the highest pressure.
Posture Sensor
71. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 71
Posture Sensor
The raw data from the six chair sensors is processed [13] to obtain net seat
change, net back change, and sit forward values.
!
!
!
!
!
!
!
!
[12] Cooper, D., Arroyo, I., Woolf, B., Muldner, K., Burleson, W., and Christopherson, R. (2009). Sensors model student self concept in the classroom,
73. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 73
Sensing
Device
(rate in Hz)
Legacy Software
Sensing
(Input or Raw Data)
Physiological responses and/or
Emotion reported (output or sensed values)
Emotiv® EEG
headset
(128 Hz)
Emotiv® SDK Brain Waves
EEG activity. Reported in 14 channels [16],: AF3, F7, F3, FC5, T7, P7, O1, O2,
P8, T8, FC6, F4, F8, and AF4.
Face activity. Blink, wink (left and right), look (left and right), raise brow, furrow
brow, smile, clench, smirk (left and right), and laugh.
!
Emotions. Excitement, engagement, boredom, meditation and frustration.
Standard Webcam
(10 Hz)
MIT Media Lab MindReader Facial Expressions Emotion. Agreeing, concentrating, disagreeing, interested, thinking and unsure.
MIT skin conductance sensor
(2 Hz)
USB driver Skin Conductivity Arousal.
MIT pressure sensor
(6 Hz)
USB driver Pressure One pressure value per sensor allocated into the input/control device.
Tobii® Eye tracking
(60 Hz)
Tobii® SDK Eye Tracking Gaze point (x, y).
MIT posture sensor
(6 Hz)
USB driver Pressure
Pressure values in the back and the seat (in the right, middle and left zones) of a
cushion chair.
Summary
75. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Background
75
The datasets shown in the next slides correspond to the data collected in three
studies. The stimuli, protocol, and participants are described briefly in the
following paragraphs.
!
1. Study One: Subjects playing a video game.
!
2. Study Two: Subjects reading documents with and without pictures.
!
3. Study Three: Subjects solving tasks in a Tutoring System.
76. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Study One
76
Stimuli. The high-fidelity graphic, deeply engaging, Guitar Hero® video game.
The game involves holding a guitar interface while listening to music and watching
a video screen.
!
Protocol. The study consists of a one-hour session with (1) 15 minutes to practice
with the objective that the user gets familiar with the game controller and the
environment and (2) 45 minutes when users played four songs of their choice, one
of each level: easy, medium, hard and expert.
!
Participants. The call for participation was an open call among Arizona State
University students. The experiment was run over 21 subjects, 67% of them were
men and 33% women. The age ranged from 18 to 28 years. The study includes
subjects with different (self-reported) experience playing video games, where 15%
never played before, 5% were not skilled in video games, 28% reported being
slightly skilled, 33% somewhat skilled, 14% very skilled, and only 5% reported
themselves as experts.
77. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Study Two
77
Stimuli. On-screen reading material was used for this study. Two different
types of reading material were used, one with either on-task or off-task images,
captions, and drawings; and the second containing only text.
!
Protocol. The study consists of one 60-minute session where the subject is
presented with 10 pages from a popular Educational Psychology textbook and
asked to read for understanding. Each participant was asked to complete a pre
and post-test.
!
Participants. The call for participation also was an open call among Arizona State
University students. The study was run over 28 subjects.
!
78. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Study Three
78
Stimuli. A Tutoring System for system dynamics. The tutor tutors students on
how to model systems behavior using a graphical representation and algebraic
expressions. The model is represented using a directed graph structure that
defines a topology formed by nodes and links. Once there is a complete model,
students can execute and debug it [13].
!
Protocol. Students were asked to perform a set of tasks (about system dynamics
modeling) using the tutoring system. While the student was working on these
tasks, the tutoring system collected emotional state information with the intention
of being able to generate better and more accurate hints and feedback.
!
Participants. Pilot test during Summer 2010 with 2 groups of 30 high-school
students.
!
!
[14] K. VanLehn, W. Burleson, M.E. Chavez-Echeagaray, R. Christopherson, J. Gonzalez-Sanchez, Y. Hidalgo-Pontet, and L. Zhang. “The Affective Meta-Tutoring
Project: How to motivate students to use effective meta-cognitive strategies,” Proceedings of the 19th International Conference on Computers in Education. Chiang
Mai, Thailand: Asia- Pacific Society for Computers in Education. October 2011. In press.
80. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Filtering
80
Filtering the data includes cleaning, synchronizing, and averaging
or passing a threshold or a range.
81. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Filtering and Integration
81
The multimodal emotion recognition considers the existence of different
sources of data that contribute to infer the affective state of an individual.
Each sensing device has its own type of data and sample rate.
The challenge is how to combine the data coming from these diverse
sensing devices that have proven their independent functionality and create an
improved unique output.
82. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Filtering and Integration
Centre
Agent
Agent
Data
Logger
!
Data
Tutoring
Multimodal
82
83. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Framework
83
We developed a framework that follows the organizational strategy of an agent
federation [15].
!
!
The federation assigns one agent to collect raw data from each sensing device.
!
!
That agent implements the perception mechanism for its assigned
sensing device to map raw data into beliefs.
!
!
Each agent is autonomous and encapsulates one sensing device and its
perception mechanisms into independent, individual, and intelligent components.
All the data is timestamped and independently identified by agent.
!
!
!
[15] Gonzalez-Sanchez, J.; Chavez-Echeagaray, M.E.; Atkinson, R.; and Burleson, W. (2011), "ABE: An Agent-Based Software Architecture for a Multimodal Emotion
Recognition Framework," in Proceedings of Working IEEE/IFIP Conference on Software Architecture (June 2011).
84. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Framework
84
When several agents need to interact, it is important to establish an organizational
strategy for them, which defines authority relationships, data flows and
coordination protocols [16] [17].
!
!
These beliefs are reported to a central agent, which integrates them into one
affective state report.
!
!
Third-party systems are able to obtain affective state reports from the centre-agent
using a publish-subscribe style.
!
!
!
!
!
[16] F. Tuijnman, and H. Afsarmanesh, "Distributed objects in a federation of autonomous cooperating agents," Proc. International Conference on
Intelligent and Cooperative Information Systems, May 1993, pp. 256-265, doi:10.1109/ICICIS.1993.291763.
!
[17] M. Wood, and S. DeLoach, “An overview of the multiagent systems engineering methodology,” Agent-Oriented Software Engineering, (AOSE
2000), Springer-Verlag, 2001, pp. 207-221, doi:10.1007/3-540-44564-1_14.
85. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Framework | Architecture
[15] Gonzalez-Sanchez, J.; Chavez-Echeagaray, M.E.; Atkinson, R.; and Burleson, W. (2011), "ABE: An Agent-Based Software Architecture for a Multimodal
Emotion Recognition Framework," in Proceedings of Working IEEE/IFIP Conference on Software Architecture (June 2011).
85
90. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Tool | Eureqa
mathematical relationships in data
90
91. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 91
For reverse engineering searches of the data, the Eureqa tool [19] is
used to discover mathematical expressions of the structural relationships
in the data records.
!
For example, if a record holds information about the physical and emotional
behavior of an individual who was engaged in a single experimental setting,
Eureqa could take all the available sources of data and reveal both how the
measure of engagement is calculated from specific data streams as well as how
other sensors may influence the proposed emotional construct.
!
!
!
!
!
!
!
!
[19] Dubcˇa ́kova ́, R. Eureqa-software review. Genetic programming and evolvable machines. Genet. Program. Evol. Mach. (2010) online first. doi:10.1007/s10710-
010-9124-z .
Tool | Eureqa
95. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray 95
For clustering and classification approaches: Weka [20]
It is a tool that implements a collection of machine learning algorithms for data
mining tasks and is used to explore the data composition and relationships and
derive useful knowledge from data records.
!
!
!
!
!
!
!
[20] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I.H. Witten, “The WEKA Data Mining Software: An Update,” Proc. SIGKDD Explorations, 2009,
Tool | Weka
101. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Visualization
101
BCI and Gaze Points engagement
This figure shows the engagement fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point,
while the level of shading represents the intensity of the emotion.
102. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Visualization
102
BCI and Gaze Points frustration
This figure shows the frustration fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while
the level of shading represents the intensity of the emotion.
103. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Visualization
103
BCI and Gaze Points boredom
This figure shows the boredom fixation points of expert player playing in expert-mode. The size of the circle represents the duration of the fixation in that point, while
the level of shading represents the intensity of the emotion.
104. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Visualization
104
BCI and Gaze Points engagement
This figures shows the engagement gaze points (above a threshold of 0.6) of a user reading material with seductive details (i.e. cartoons).
For this user the text on the bottom part of the first column was engaging.
105. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Visualization
105
BCI and Gaze Points frustration
This figure shows the frustration gaze points (above a threshold of 0.6) of a user reading material with seductive details (i.e. cartoons).
Looking at the cartoon is related with high frustration level.
106. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Visualization
106
BCI and Gaze Points boredom
This figure shows the boredom gaze points (above a threshold of 0.5) of a user while reading material with seductive details (i.e. cartoons).
Notice that the text in the middle part of the second column of that page was boring.
107. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Matching Values
107
Our hypothesis is that, to a great extent, it is possible to infer the values of one source from the other source.
BCI-based values Correlation with face-based model
excitement 0.284
engagement 0.282
meditation 0.188
frustration 0.275
Face-based values Correlation with BCI-based model
agreement 0.76
concentrating 0.765
disagreement 0.794
interested 0.774
thinking 0.78
unsure 0.828
BCI and Face-Based Emotion Recognition
108. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Matching Values
108
…"
[21] L. Breiman. Random forests. Machine Learning, 2001. Volume 45, Number 1. Pages 5–32.
Random Forest
109. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Networks
Structural(Equa+ons,(Adjacency(Matrixes(
and(Networks(Graphs(
Brain(schema+c,(showing(the(channels(
that(contribute(with(engagement(
! !
BCI raw data | Engagement
109
to
110. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Closed-loop
Emotion Adaptation | Games
110
[22] Bernays, R., Mone, J., Yau, P., Murcia, M., Gonzalez-Sanchez, J., Chavez-Echeagaray, M. E., Christopherson, R. M., Atkinson, R., and Yoshihiro, K. 2012. Lost
in the Dark: Emotion Adaption. In Adjunct proceedings of the 25th annual ACM symposium on User interface software and technology, 79–80. New York, NY, USA.
ACM. doi:10.1145/2380296.2380331.
111. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Closed-loop
Emotion Mirroring | Virtual World
111
[23] Gonzalez-Sanchez, J., Chavez-Echeagaray, M. E., Gibson, D., and Atkinson, R. 2013. Multimodal Affect Recognition in Virtual Worlds: Avatars Mirroring User's
Affect. In ACII '13: Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE Computer Society.
112. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Group Discussion
112
devices data inferences
114. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Conclusion
This course aims to provide the attendees with a presentation about tools and
dataset exploration. While the course do not present an exhaustive list of all the
methods available for gathering, processing, analyzing, and interpreting affective
sensor data, the course describe the basis of a multimodal approach that
attendees can use to launch their own research efforts.
!
!
!
!
!
!
!
!
!
!
!
!
!
[22] B. du Boulay, “Towards a Motivationally-Intelligent Pedagogy: How should an intelligent tutor respond to the unmotivated or the demotivated?,” Proc. New
Perspectives on Affect and Learning Technologies, R. A. Calvo & S. D'Mello (Eds.), Springer-Verlag, in press.
114
115. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
References
1. R.S.J. Baker, M.M.T Rodrigo and U.E. Xolocotzin, “The Dynamics of Affective Transitions in Simulation
Problem-solving Environments,” Proc. Affective Computing and Intelligent Interaction: Second
International Conference (ACII ’07), A. Paiva, R. Prada & R. W. Picard (Eds.), Springer-Verlag, Vol.
Lecture Notes in Computer Science 4738, pp. 666-677.
2. I. Arroyo, D. G. Cooper, W. Burleson, F. P. Woolf, K. Muldner, and R. Christopherson, “Emotion Sensors
Go to School,” Proc. Artificial Intelligence in Education: Building Learning Systems that Care: from
Knowledge Representation to Affective Modelling, (AIED 09), V. Dimitrova, R. Mizoguchi, B. du Boulay
& A. Grasser (Eds.), IOS Press, July 2009, vol. Frontiers in Artificial Intelligence and Applications 200,
pp. 17-24.
3. R. W. Picard, Affective Computing, MIT Press, 1997.
4. J. Gonzalez-Sanchez, R. M. Christopherson, M. E. Chavez-Echeagaray, D. C. Gibson, R. Atkinson, W.
Burleson, “ How to Do Multimodal Detection of Affective States?,” icalt, pp.654-655, 2011 IEEE 11th
International Conference on Advanced Learning Technologies, 2011
5. Emotiv - Brain Computer Interface Technology. Retrieved April 26, 2011, from http://www.emotiv.com.
6. F. Sharbrough, G.E. Chatrian, R.P. Lesser, H. Lüders, M. Nuwer, T.W. Picton. American
Electroencephalographic Society Guidelines for Standard Electrode Position Nomenclature. J. Clin.
Neurophysiol 8: 200-2.
7. Electroencephalography. Retrieved November 14th, 2010, from Electric and Magnetic Measurement of
the Electric Activity of Neural Tissue: http://www.bem.fi/book/13/13.htm
115
116. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
References
8. R. E. Kaliouby and P. Robinson, “Real-Time Inference of Complex Mental States from Facial
Expressions and Head Gestures,” Proc. Conference on Computer Vision and Pattern Recognition
Workshop (CVPRW ‘04), IEEE Computer Society, June 2004, Volume 10, p.154.
9. Tobii Technology - Eye Tracking and Eye Control. Retrieved April 26, 2011, from http://
www.tobii.com.
10. M. Strauss, C. Reynolds, S. Hughes, K. Park, G. McDarby, and R.W. Picard, “The HandWave
Bluetooth Skin Conductance Sensor,” Proc. First International Conference on Affective Computing
and Intelligent Interaction (ACII 05), Springer-Verlang, Oct. 2005, pp. 699-706, doi:
10.1007/11573548_90.
11. Y. Qi, and R. W. Picard, "Context-Sensitive Bayesian Classifiers and Application to Mouse Pressure
Pattern Classification," Proc. International Conference on Pattern Recognition (ICPR 02), Aug.
2002, vol 3, pp. 30448, doi:10.1109/ICPR.2002.1047973.
12. D. Cooper, I. Arroyo, B. Woolf, K. Muldner, W. Burleson, and R. Christopherson. (2009). Sensors
model student self concept in the classroom, User Modeling, Adaptation, and Personalization,
30-41.
13. S. Mota, and R. W. Picard, "Automated Posture Analysis for Detecting Learners Interest Level,"
Proc. Computer Vision and Pattern Recognition Workshop (CVPRW 03), IEEE Press, June 2003,
vol. 5, pp. 49, doi:10.1109/CVPRW.2003.10047.
116
117. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
References
14. K. VanLehn, W. Burleson, M.E. Chavez-Echeagaray, R. Christopherson, J. Gonzalez-Sanchez, Y.
Hidalgo-Pontet, and L. Zhang. “The Affective Meta-Tutoring Project: How to motivate students to
use effective meta-cognitive strategies,” Proceedings of the 19th International Conference on
Computers in Education. Chiang Mai, Thailand: Asia- Pacific Society for Computers in Education.
October 2011. In press.
15. J. Gonzalez-Sanchez; M.E. Chavez-Echeagaray; R. Atkinson; and W. Burleson. (2011), "ABE: An
Agent-Based Software Architecture for a Multimodal Emotion Recognition Framework," in
Proceedings of Working IEEE/IFIP Conference on Software Architecture (June 2011).
16. F. Tuijnman, and H. Afsarmanesh, "Distributed objects in a federation of autonomous cooperating
agents," Proc. International Conference on Intelligent and Cooperative Information Systems, May
1993, pp. 256-265, doi:10.1109/ICICIS.1993.291763.
17. M. Wood, and S. DeLoach, “An overview of the multiagent systems engineering methodology,”
Agent-Oriented Software Engineering, (AOSE 2000), Springer-Verlag, 2001, pp. 207-221, doi:
10.1007/3-540-44564-1_14.
18. [18] J. Liu, S. Ji, and J. Ye. SLEP: Sparse Learning with Efficient Projections. Arizona State
University, 2009. http://www.public.asu.edu/~jye02/Software/SLEP.
19. [19] R. Dubcakova. Eureqa-software review. Genetic programming and evolvable machines. Genet.
Program. Evol. Mach. (2010) online first. doi:10.1007/s10710- 010-9124-z .
117
118. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
References
20. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I.H.Witten, “The WEKA Data Mining
Software: An Update,” Proc. SIGKDD Explorations, 2009, Volume 11, Issue 1.
21. L. Breiman. Random forests. Machine Learning, 2001. Volume 45, Number 1. Pages 5–32.
22. R. Bernays, J. Mone, P. Yau, M. Murcia, J. Gonzalez-Sanchez, M.E. Chavez-Echeagaray, R.
Christopherson, R. Atkinson, and K. Yoshihiro. 2012. Lost in the Dark: Emotion Adaption. In Adjunct
proceedings of the 25th annual ACM symposium on User interface software and technology, 79–80.
New York, NY, USA. ACM. doi:10.1145/2380296.2380331.
23. J. Gonzalez-Sanchez, M.E. Chavez-Echeagaray, D. Gibson, and R. Atkinson. 2013. Multimodal Affect
Recognition in Virtual Worlds: Avatars Mirroring User's Affect. In ACII '13: Proceedings of the 2013
Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE Computer
Society. 724-725. doi: 10.1109/ACII.2013.133
24. B. du Boulay, Towards a Motivationally-Intelligent Pedagogy: How should an intelligent tutor respond to
the unmotivated or the demotivated?, Proc. New Perspectives on Affect and Learning Technologies, R.
A. Calvo & S. D'Mello (Eds.), Springer-Verlag.
118
120. Javier Gonzalez-Sanchez | Maria-Elena Chavez-Echeagaray
Acknowledgements
This research was supported by
Office of Naval Research under Grant N00014-10-1-0143
awarded to Dr. Robert Atkinson
120