Emotion recognition from biometrics is relevant to a wide range of application domains, including healthcare and software development. Existing approaches usually adopt multi-electrodes sensors that could be expensive or uncomfortable to be used in real-life situations. We investigate whether we can reliably recognize high vs. low emotional valence and arousal by relying on noninvasive low-cost EEG, EMG, and GSR sensors. We report the results of an empirical study involving 19 subjects in a laboratory setting for emotion elicitation. We achieve state-of-the-art classification performance for both valence and arousal even in a cross-subject classification setting, which eliminates the need for individual training and tuning of classification models. Furthermore, we will discuss our ongoing work on the recognition of affective and cognitive states of software engineer during their daily programming tasks.

1. Emotion Detection Using Noninvasive
Low-cost Sensors
Nicole Novielli
University of Bari, Italy
Collaborative Development Group
@NicoleNovielli nicole.novielli@uniba.it

2. People at Collab
• Faculty
– Filippo Lanubile
– Fabio Calefato
– Nicole Novielli
• Visiting Professors and Researchers
collab.di.uniba.it
• PhD Students
– Giuseppe Iaffaldano
– Daniela Girardi
• Graduate students and
final-year undergraduates

3. Data Science for
(Collaborative) Software
Engineering
Software
Engineering
Social
Computing
Data
Science

4. Research at COLLAB
Software engineering is sociotechnical
H. Erdogmus, N. Medvi“On the social side, it involves a
significant people-and-team component”
dovic and F. Paulisch. 50 Years of Software Engineering. IEEE Software, Sept/Oct
2018
Data Science for
(Collaborative) Software Engineering
Raw data come from various
sources including
sw artifacts, traces of activities
and communication exchanges

5. Research themes
• Role of emotions in the social programmer
ecosystem
• Knowledge sharing in Q&A sites
• Online creative communities
• Sensor-based detection of emotional and cognitive
states
• Information fragmentation and overload for
DevOps
• Personality, trust and project success

6. Sensing Developers’ Emotions
Developer
Emotional self intelligence
Team
Emotional social intelligence
Organization
Emotional organizational intelligence
Scope
EMOTIONAL AWARENESS & MANAGEMENT
Self control
Positive Thinking
Undertake enjoyable actions
Empathy
Dynamics of encouragement
Dynamics of playfulness
Reward and Incentives
Change working conditions
Use family-friendly policies
Improve individual performance Improve group performance Increase job satisfaction
Reduce turnover

7. Role of Emotions in the Social
Programmer Ecosystem
• Research question:
getting emotional while
communicating with developers:
good or bad?
• Method: NLP / ML from
developers’ communication traces
• Contributions:
– Annotated datasets
– SE-specific tools for sentiment analysis
‘What is the best way to kill a critical process’
http://collab.di.uniba.it/emoquest
Sept. 2015 -2018
Emotion Detection with Lo
Noninvasive Biometric Se
Nicole NOVIELLI
COLLAB, Collaborative Development
Dipartimento di Informatica, Università deg
EmoQuest - Investigating the Role of Em
Online Question & Answering Si
Project Website: http://collab.di.uniba.it/

9. The Project Website
• Project Website
http://collab.di.uniba.it/emoquest

10. Building the Community
• SEmotion Workshop series @ICSE, the International
Conference on Software Engineering

11. Sentiment and Emotion in SE

12. Why should we care?
• Proactively providing support to software
developers
• Are they experiencing any difficulties?
– Emotion detection: stuck or ‘in flow’?
• What are they trying to do?
– Automatic identification of task

13. Why should we care?
• Proactively providing support to software
developers
• Are they experiencing any difficulties?
– Emotion detection: stuck or ‘in flow’?
• What are they trying to do?
– Automatic identification of task

14. Emotion Detection Using Noninvasive
Low Cost Sensors
Daniela Girardi, Filippo Lanubile, Nicole Novielli
University of Bari, Italy
ACII 2017, Seventh International Conference
on Affective Computing and Intelligent
Interaction

15. Long-term goals
cognition and mobility, which cannot communicate
their emotions during medical treatments
– Adapting the treatments accordingly
• Sensing emotions of software developers during
their daily programming tasks
– Supporting stuck developers
– Enabling assessment of correct implementation of agile
approaches to software development
– Enhancing effective collaboration

16. Long-term goals
cognition and mobility, which cannot communicate
their emotions during medical treatments
– Adapting the treatments accordingly
• Sensing emotions of software developers during
their daily programming tasks
– Supporting stuck developers
– Enabling assessment of correct implementation of agile
approaches to software development
– Enhancing effective collaborationLong-term goals
• Sensing emotions of patients with impaired
cognition and mobility, which cannot communicate
their emotions during medical treatments
– Adapting the treatments accordingly
• Sensing emotions of software developers during

17. Research Questions
• RQ1: Can we acquire physiological measures from
noninvasive, low cost sensors to accurately predict
emotions?
• RQ2: What is the best combination of physiological
measures for recognizing emotional valence and
arousal?
• RQ3: Can we successfully train a cross-subject
classifiers for valence and arousal, based on
physiological measures?

18. How do we operationalize affective states?
AFFECT

19. Affective States
Duration++
Intensity++
Scherer, 1984. Emotion as a Multicomponent Process: A model and some cross-cultural data. In P. Shaver, ed.,
Review of Personality and Social Psych 5: 37-63.

20. CONTINUOUS VS. DISCRETE
MODELS
37

21. Anger Fear Disgust Surprise Happiness Sadness
Ekman’s Basic Emotions

22. Circumplex Model of Affect
Russel, 1980 - A circumplex model of affect. J.Personality and Social Psychology, 39, 1161-1178.
Activation/Arous
al
Polarity or
Valence
Activation or
Arousal
Polarity or
Valence

23. Continuous vs. Discrete Emotion
Models
Plutchik’s
Wheel

24. Lazarus’ Framework
Based on appraisal theory
Negative Emotions Positive Emotions Mixed
Emotions
anger
fright
anxiety
guilt
shame
sadness
envy
jealousy
disgust
happiness
pride
relief
love
hope
compassion
gratitude

25. OCC Cognitive Model of Emotions
www.id-book.com 42

26. Mapping
Emotions to
the
Circumplex
Model of
Affect
Valence
Arousal

27. Research Questions
• RQ1: Can we acquire physiological measures from
noninvasive, low cost sensors to accurately predict
emotions?
• RQ2: What is the best combination of physiological
measures for recognizing emotional valence and
arousal?
• RQ3: Can we successfully train a cross-subject
classifiers for valence and arousal, based on
physiological measures?

28. INSTRUMENTATION

29. Electroencephalography (EEG)
• Electrical activity of the brain
• Cerebral waves can be categorized based on
frequency
– Delta (<4 Hz): recorded during sleep
– Theta (4-7,5 Hz): decrease of vigilance level
– Alpha (4-12,5 Hz): relax
– Beta (13-30 Hz): mental processes
– Gamma (> 30 Hz): anxiety

30. BrainlinkBrainlink
• Headset for EEG
– Two electrodes in FP1 and FP2 position
according with 10-20 International System
– Reference on earlobe
• Used in combination with
Neuroview software to measures:
– Raw EEG
– Filtered EEG
– Attention and meditation levels

31. Galvanic Skin Respons (GSR)
• Measure of the skin conductance
– i.e. electrical activity due to sweating
• The GSR signal consists of two main components
– Tonic: indicates the basic level (varies from person to
person)
– Phasic:changes according to specific external stimuli such
as sounds, noises, changes in light condition, etc.
• GSR varies with respect to changes in emotional
intensity, especially for emotions with high arousal
– widely employed in emotion recognition

32. Shimmer GSR
• Bracelet for GSR
– two electrodes positioned on the palmar surface of the
first phalanx of two different fingers
• Used in combination with ConsensysPRO software
to measures GSR raw data

33. Hearth- and blood- related
measures
• Metrics related to blood and hearth
activity
– Blood volume, hearth rate, hearth-rate
variability, etc.
• Employed for recognition of emotional
and cognitive states
• Plethysmograph: an optical sensor
usually applied on a finger
– HR usually derived by applying conversion
algorithms to the signal captured
Empatica – www.empatica.com
Bos, EEG-based Emotion Recognition - Human Media Interaction

34. Electromyographic signal (EMG)
• Electrical activity of contracting muscles
– Electricity generated and propagated in tissues, bones and
in the nearby skin area
• Correlated to the amount of muscle contraction and
the number of muscles contracted
– EMG activity is measurable even when no observable
contractions can be seen
• Works for emotions if captured with face electrodes
– very intrusive!

35. Shimmer EMG
• Bracelet for EMG
– two channels, each with a positive and a negative
electrodes positioned parallel to the fibers of the muscle
– Reference positioned in a neutral point in the body, e.g.
on the wrist bone
• Used in combination with
ConsensysPRO software to
measures EMG raw data

36. 19 subjects

37. Emotion elicitation timeline
• 8 videos from DEAP (Database for Emotion Analysis using Physiological signals)
• Each video is associated with different arousal and valence scores
S. Koelstra, C. Muhl, M. Soleymani, J.S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, and I. Patras. “DEAP: A Database for Emotion Analysis Using Physiological Signals”. IEEE Trans. Affect.
Comput. 3, 1, pp. 18-31, 2012.

38. Data Analysis

39. Preprocessing
• Removing noise:
– Loss of contact between the electrodes and skin, eye
blink, or movement artifacts, etc.
• Normalization
– Every person has a unique power spectrum
– Correction of stimulus-unrelated variations by
subtracting the mean value for the signal during baseline
videos associated to neutral emotional states.
• Frequency filters
– Extracting specific waves
Scripts: https://github.com/BioStack/Sensors101/tree/master/Analysis

40. Features
Features
Signal Extracted Features
EEG For the alpha, beta, gamma, delta, theta, attention, and meditation: mean,
minimum, maximum, variance, standard deviation
GSR On the phasic component: mean, minimum, maximum, variance, standard
deviation.
On the corrected phasic component after Wavelet filter: mean of the
derivatives, average of the derivatives of negative values, and the
proportion of negative values.
Considering the peaks: mean, minimum and maximum width, ratio
between number of peaks and minimum width, ratio between sum of
peaks and minimum width
EMG On the integrated signal from both channels: mean, minimum, maximum,
variance, standard deviation.
Scripts: https://github.com/collab-uniba/Biometrics_EmotionDetection

41. Results
• Comparison with the DEAP study
Comparison with the DEAP
study
Signals Study Description
Arousal Valence
Classifier F1 Classifier F1
EEG
Our Study EEG NB/ KNN 0.76 J48 0.90
[14] EEG NB 0.583 NB 0.563
Peripheral
Our Study GSR + EMG KNN 0.79 J48 0.76
[14]
GSR + EMG
+
respiration
+ blood
pressure +
eye blinking
rate
NB 0.553 NB 0.608

42. Improved classification models
• Hyper-parameter tuning
Results
Signals Arousal Valence
Classif Prec Recall F1 Acc. Classif Prec Recall F1 Acc.
Single Sensors
EEG NB/KNN 0.76 0.76 0.76 0.76 J48 0.90 0.90 0.90 0.90
GSR JRIP 0.69 0.68 0.68 0.68 KNN 0.67 0.67 0.67 0.67
EMG KNN 0.80 0.80 0.80 0.80 KNN 0.70 0.70 0.70 0.70
Combined Sensors
EEG+GSR J48 0.98 0.98 0.98 0.98 J48 0.97 0.97 0.97 0.97
GSR+EMG KNN 0.79 0.79 0.79 0.79 J48 0.76 0.76 0.76 0.76
EEG+EMG J48 0.86 0.86 0.86 0.86 J48 0.95 0.95 0.95 0.95
All J48 0.98 0.98 0.98 0.98 NB 0.82 0.82 0.82 0.82
D. Girardi, F. Lanubile, N. Novielli. ‘Classifying Emotional Arousal and Valence Using Noninvasive Low-Cost
Sensors’, under review (IJHCS)

43. Main Findings
• RQ1: Can we acquire physiological measures
from noninvasive, low cost sensors to
accurately predict emotions?

44. Research Questions
• RQ1: Can we acquire physiological measures
from noninvasive, low cost sensors to
accurately predict emotions?
• RQ2: What is the best combination of
physiological measures for recognizing
emotional valence and arousal?
EEG+GSR

45. Research Questions
• RQ1: Can we acquire physiological measures
from noninvasive, low cost sensors to
accurately predict emotions?
• RQ2: What is the best combination of
physiological measures for recognizing
emotional valence and arousal?
• RQ3: Can we successfully train a cross-subject
classifiers for valence and arousal, based on
physiological measures?
EEG+GSR
Research Questions
• RQ1: Can we acquire physiological measures
from noninvasive, low cost sensors to
accurately predict emotions?
• RQ2: What is the best combination of
physiological measures for recognizing
EEG+GSR

46. Identifying Developers’ Emotions Using Biometrics
Davide Fucci
University of Hamburg
Daniela Girardi, Filippo Lanubile, Nicole Novielli, Luigi
Quaranta
University of Bari

47. Emotion detection: stuck or ‘in flow’?
S. Muller, T. Fritz. Stuck and Frustrated or in Flow and Happy: Sensing Developers' Emotions and Progress. ICSE 2015
• Replication of Muller and
Fritz study:
– Which range of emotions
during programming tasks?
– How emotions correlate
with perceived progress?
– To what extent emotions
can be detected using
biometric sensors?

48. Experimental Protocol

49. BrainLink
EEG monitors the electrical
activity of the brain
Instrumentation

50. Self assessment evaluation
1M. M. Bradley and P. J. Lang. 1994. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavioral Therapy and
Experimental Psychiatry, Volume 25, 49–59.

51. Data collection still ongoing
• ~40 participants
– Undergraduate students in Computer Science
– Currently planning a field study with a SME
• Sample by saturation strategy
• Rewarded with 10€ coupon

52. Why should we care?
• Proactively providing support to software
developers
• Are they experiencing any difficulties?
– Emotion detection: stuck or ‘in flow’?
• What are they trying to do?
– Automatic identification of task

53. International Conference on Program Comprehension
• RQ1: Can we classify which task a participant is
undertaking based on signals collected from
lightweight biometric sensors?
– Prose vs. code comprehension
• RQ2: Can we relate expertise to classification
accuracy?

54. Original study
(a) Code Comprehension vs. Prose Review (b) Code Review vs. Prose Review
Fig. 2: Average weight maps for task classifiers. When regions of the brain colored “hot” are active, the decision is pushed
toward Code. The left and right subfigures show a high degree of concordance (r = 0.75, p < .001), quantifying how both
code tasks are distinguished similarly compared to the prose task.
I. Regional Inference
We next sought to determine which regions of the brain
were most involved in discriminating between code and prose.
This involved projecting kernel weights back onto the 3D
brain — for display purposes, we present weight maps that
were averaged across CV folds and participants. It is worth
emphasizing, however, that such multivariate maps do not
lend themselves to simple regional inference: because the
final classification decision depends on information across
all voxels, it is incorrect to assume voxels with high weight
are the “most important.” Nevertheless, we may estimate a
posteriori the total contribution of each anatomical area in the
aforementioned AAL atlas [78]. In this procedure, the absolute
values of all voxel weights within abrain region were summed
and divided by the total number of voxels in the region.
Then, each region’s “contribution strength” was divided by the
sum of strengths for all regions, yielding a proportion that is
directly interpretable as regional importance — a larger value
indicates more total weight represented within a region [66].
These importance maps are also presented as a group average.
(Z = 0.84, p = .400) or Comprehension vs. Prose models
(Z = 0.55, p = .579), suggesting that GPC’s ability to
discriminate between code and prose tasks was not driven by
the number of prose trials completed. This also held when
considering only Prose class accuracy in both Review vs.
Prose models (Z = − 1.87, p = .061) and Comprehension vs.
Prose models (Z = − 1.53, p = .127). A full set of summary
statistics for classifier performance are displayed in Table I.
With regard to overall classifier performance, we employed
nonparametric Wilcoxon signed-rank tests to compare model
BAC against a null median accuracy of 50% (chance for a
binary classifier). For all models, GPC performance washighly
significant. The classifiers accurately discriminated between
Review vs. Prose trials (BAC = 70.83%; Z = 4.00, p <
.001), Comprehension vs. Prose trials (BAC = 79.17%; Z =
4.51, p < .001), and even Review vs. Comprehension trials
(BAC = 61.74%; Z = 3.45, p < .001).
These results suggest that Code Review, Code Compre-
hension, and Prose Review all have largely distinct neural
representations. Inspection of the average weight maps for
Main findings
• The neural representations of programming and natural
languages are distinct
• A classifier can distinguish between these tasks based
solely on brain activity
• Expertise matters: greater skill accompanies a less-
differentiated neural representation.
fMRI

55. Automatic identification of task
Prose vs. code comprehension
Code comprehension:
- 3 tasks per block
- 60 seconds to answer
Prose comprehension:
- 6 tasks per block
- 30 seconds to answer

56. Experimental setting
Tasks
Block
1
Tasks Block
2
Tasks Block
3
6 minutes 10 seconds 6 minutes 10 seconds 6 minutes
fixation fixation
duration: 18’20’’
Each block is composed by 9 tasks showed randomly
30 bachelor students who had passed C
programming courses

57. Signal Features Tot
Brain EEG
• Frequency bin for alpha, beta, gamma, delta and theta waves
• Ratio between frequency bin of each band and one another
• For the attention and meditation measures: min, max, difference
between the mean attention (meditation) during the baseline and
during the task
33
Skin
EDA tonic • mean tonic signal 1
EDA phasic • area under the curve (AUC)
• min, max, mean, sum peaks amplitudes
5
TEMP • min, max, mean temperature
• difference between the min (max, mean) temperature during the
baseline and during the task
6
Heart
BVP
• min, max, mean, sum peak amplitudes
• difference between the mean peak amplitude during baseline and
during the task
5
HR • difference between the mean heart rate during the baseline and
during the task
• difference between the variance heart rate during the baseline
and during the task
2
HRV • standard deviation of beat-to-beat (SDNN) intervals
• root Mean Square of the Successive Differences (RMSSD)
2
Replication package: https://github.com/collab-uniba/Replication_Package_ICPC

58. Training the classifier

59. Results
LORO cross-validation
Hold-out cross-validation

60. Results
f1

61. Main findings
• RQ1: Can we classify which task a
participant is undertaking based on
signals collected from lightweight
biometric sensors?
– Prose vs. code comprehension
Research Questions
• RQ1: Can we acquire physiological measures
from noninvasive, low cost sensors to
accurately predict emotions?
• RQ2: What is the best combination of
physiological measures for recognizing
emotional valence and arousal?
• RQ3: Can we successfully train a cross-subject
classifiers for valence and arousal, based on
physiological measures?
EEG+GSR

62. Main findings
• RQ1: Can we classify which task a
participant is undertaking based on
signals collected from lightweight
biometric sensors?
– Prose vs. code comprehension
• RQ2: Can we relate expertise to
classification accuracy?
– non-significant correlation
– Further replications required with a more
heterogeneous sample
Research Questions
• RQ1: Can we acquire physiological measures
from noninvasive, low cost sensors to
accurately predict emotions?
• RQ2: What is the best combination of
physiological measures for recognizing
emotional valence and arousal?
• RQ3: Can we successfully train a cross-subject
classifiers for valence and arousal, based on
physiological measures?
EEG+GSR
Research Questions
• RQ1: Can we acquire physiological measures
from noninvasive, low cost sensors to
accurately predict emotions?
• RQ2: What is the best combination of
physiological measures for recognizing
emotional valence and arousal?
• RQ3: Can we successfully train a cross-subject
classifiers for valence and arousal, based on
physiological measures?
EEG+GSR

63. Modeling and identifying confusion

64. Modeling and identifying confusion

65. Modeling and identifying confusion

66. SUMMARY

67. Summary

68. Summary

69. Summary

70. Summary