Affective User Modeling

Affective User Modeling
@MEi:CogSci

Univerza v Ljubljani ..: Fakulteta za elektrotehniko:..
[LDOS] ..: Laboratorij za digitalno obdelavo signalov, slik in videa:..

Marko Tkalčič
marko.tkalcic@fe.uni-lj.si
http://ldos.fe.uni-lj.si/markot


Overview

Traditional user modeling
in recommender systems
Need for affective user
modeling!
HOW?

Emotions & detection

The proposed
AUM framework

Example 1
Example 2
Dataset


User modeling
 Prediction of users behavior
 Why?
– Product recommendation


Amazon


Netflix


Recommender systems


Recommender systems

DB

Recommender
System


Recommender systems

Feedback Knowledge

DB

Recommender
System


 In movie recommender systems – Netflix example

XXX [genre = A]
YYY [genre = B]
ZZZ [genre = C]
XYY [genre = B]
XXY [genre = C]

User profile:
A: 0
B: 0
C: 0



XXX [genre = A]
YYY [genre = B]
ZZZ [genre = C] YYY
XYY [genre = B]
XXY [genre = C]

User profile:
A: 0
B: 0
C: 0



XXX [genre = A]
YYY [genre = B]
ZZZ [genre = C] YYY R=5
XYY [genre = B]
XXY [genre = C]

User profile:
A: 0
B: 5
C: 0



YYY [genre = B]
XYY [genre = B]
ZZZ [genre = C]
XXX [genre = A]
XXY [genre = C]

User profile:
A: 0
B: 5
C: 0



YYY [genre = B]
XYY [genre = B]
ZZZ [genre = C] XYY
XXX [genre = A]
XXY [genre = C]

User profile:
A: 0
B: 5
C: 0



YYY [genre = B]
XYY [genre = B]
ZZZ [genre = C] XYY R=3
XXX [genre = A]
XXY [genre = C]

User profile:
A: 0
B: 4
C: 0



YYY [genre = B]
XYY [genre = B]
ZZZ [genre = C]
XXX [genre = A]
XXY [genre = C]

User profile:
A: 0
B: 4
C: 0



YYY [genre = B]
XYY [genre = B]
ZZZ [genre = C] ZZZ
XXX [genre = A]
XXY [genre = C]

User profile:
A: 0
B: 4
C: 0



YYY [genre = B]
XYY [genre = B]
ZZZ [genre = C] ZZZ R=5
XXX [genre = A]
XXY [genre = C]

User profile:
A: 0
B: 4
C: 5


Context-aware user modeling
 Users have different preferences in different contexts

?????

User profile: User profile: User profile:
Context = alone Context = friends Context = children
A: 0 A: 5 A: 1
B: 4 B: 2 B: 5
C: 5 C: 3 C: 1


Context-aware user modeling
 Users have different preferences in different contexts

ZZZ [genre = C]
XXY [genre = C]
YYY [genre = B]
XYY [genre = B]
XXX [genre = A]

Context = alone

User profile: User profile: User profile:
Context = alone Context = friends Context = children
A: 0 A: 5 A: 1
B: 4 B: 2 B: 5
C: 5 C: 3 C: 1


General user modeling framework
 Data-centric = uses data that
– Is available (genres, actors, directors ...)
– Easy to acquire (rating, „liking“ ...)
 But NOT necessarily data that carry information

Controlled variables

USER MODEL
Selected MM items
Huge MM DB 
Prediction accuracy

Uncontrolled variables


General user modeling framework
 Data-centric = uses data that
– Is available (genres, actors, directors ...)
– Easy to acquire (rating, „liking“ ...)
 But NOT necessarily data that carry information

Controlled variables

USER MODEL
Prediction accuracy

?


It is not so simple!
 Bounded rationality theory [Daniel Kahnemann (nobel prize for
economics 2002)]
Decision making = rational + emotional


Need for affective user modeling!
 (Tkalčič et al., 2010)
 Affective + generic variables
>
 Generic) variables

Controlled variables = generic + affective variables

USER MODEL



Overview of emotions
 Emotions are complex human experiences
 Evolutionary based
 Several definitions
 We take with simple models, easy to incorporate in computers:
– Basic emotions
– Dimensional model
– Circumplex model


Basic emotions
 Discrete classes model
 Different sets
 Darwin: Expression of emotions in man and animal
 Ekman definition (6 + neutral):
– Happiness
– Anger
– Fear
– Sadness
– Disgust
– Surprise


Dimensional model
 Three dimensions
– Valence
– Arousal
– Dominance

 Each emotive state is a point in the VAD space


Circumplex model
 Maps basic emotions dimensional model
Arousal
high

joy
anger
surprise

disgust

fear
Valence
neutral
negative positive
sadness

low


How to detect emotions?
 Explicit vs. Implicit
 Explicit
– Questionnaires (SAM)
 Implicit:
– Work done in the affective computing community
– Different modalities (sources):
• Facial actions (video)
• Physiological signals ( GSR, EEG)
• Voice
• Posture
• ...
– ML techniques
• Classification (basic emotions)
• Regression (dimensional model)


Emotion detection from videos of facial expressions
 Problem statement:
– Explicit affective labeling has drawbacks:
• Annoying
• Time consuming
• Potentially inaccurate in real applications
 Proposed solution:
– Implicit affective labeling through emotion detection from facial video
– Aggregation of emotions detected from several users


Experiment
 2 datasets:
– Posed (Kanade Cohn)
– Spontaneous (LDOS-PerAff-1)
 Input: Video streams of facial expressions as responses to visual stimuli
 Output: emotive states as distinct classes

Gabor features kNN

Emotive
state


Results and conclusions
 Posed dataset: accuracy = 92 %
 Spontaneous dataset: accuracy = 62%
 Reasons for bad results:
– Weak learning supervision
– Non optimal video acquisition (face rotation, occlusions, changing lightning ...)
– Non extreme facial expressions


The Affective User Modeling framework
– Research is done in a scattered fashion
– Researchers do not benefit from each other‘s work
 Goal:
– Researchers to identify their position
– To benefit from each other‘s work
– To establish affective user modeling as a (sub)field?


The proposed framework - 1
time

choice

Give Give
recommendations content

Content application

Entry stage Consumption stage Exit stage


time

Entry mood Exit mood

choice

Detect
Give Give
entry
mood

• Context Content application
• Decision making
• Influence
• Diversification
• Decision making profile


time

Entry mood Content-induced affective state

choice

Detect
Give Give
entry Observe user
mood

Content application
• Affective tagging
• Affective user profiles



time

Entry mood Content-induced affective state Exit mood

choice

Detect Detect
Give Give
entry Observe user exit
mood mood

Content application
• Implicit feedback
• Evaluation metrics
(user satisfaction)


time

Entry mood Content-induced affective state Exit mood

choice

Detect Detect
Give Give
entry Observe user exit
mood mood

• Context Content application
• Decision making
• Influence • Affective tagging • Implicit feedback
• Diversification • Affective user profiles • Evaluation metrics
• Decision making profile (user satisfaction)


Profiling in CBR systems

Item Profile (md)
Id 1
Title Girl
Genre Erotic

User Profile (up)
Item Profile (md)
Id 1
Id 2
Action 80
Title Basketball
Erotic 60
Genre Sport
Sport 95
Still life 35
… …

Item Profile (md)
Id 3
Title Kitchen
Genre Still life


Proposed solution
 We propose tu use AFFECTIVE METADATA
 Multimedia content ELICITS (induces) emotions
 Underlying assumption: users differ in their preferences for emotions


Affective modeling
 Emotion description models
– Basic emotions (Ekman: anger, fear, joy, disgust, surprise,sadness)
– Dimensional model (VAD - valence-arousal-dominance)
 We aggregated the emotive responses of many users to a single image:
– First two statistical moments of V, A and D
– Item profile
 The user profile is the result of the training an ML classifier
Arousal
high Valence
mean
<=4.23 >4.23
joy
anger
surprise Class = 0 Valence
mean
disgust <=6.71
>6.71
fear Dominance
Valence Class = 1
neutral mean
negative positive <=5.92 >5.92
sadness
Valence
mean Class = 0
<=5.21 <=5.21

Class = 1 Class = 0

low


Experiment

IAPS Image
Stimuli
generic
metadata

affective
metadata
Consumed Metadata
EMOTION Item (Item Profile)
INDUCTION

Explicit Machine User Profile
Rating Learning

Ground
Predicted
Truth
Ratings
Ratings

Confusion
Matrix


Results
 Pearson chi-square statistical significance test to compare the confusion
matrices
 Scalar measures P, R, F
 Generic+affective metadata > generic metadata
 Avg(v) best feature (71% of users)
 SVM best classifier


Latent factors
Users with personality properties: Users latent factors space
- Extraversion U21
- Agreeableness
- Conscientousness
- Neuroticism
- openness U12 U11

U22

Users – items Matrix
Latent factors
rating matrix factorization

Items latent factors space
I21

Images with affective properties:
- Valence I12 I11
- Arousal
- Dominance

I22


Latent factors - results


A personality-based user similarity measure
 Collaborative filtering recommender (CFR) systems:
– Similar users have similar preferences
– Rating-based similarity measures

Which content should
I watch tonight?


Problem statement
– New user problem: hard to assess user similarities without overlapping ratings 
bad recommendations
 Proposed solution (hypothesis)
– A personality based user similarity measure under cold start conditions


personality
 Personality: accounts for individual differences ( = explains the variance)
– Old greeks: choleric, melancholic, phlegmatic, sanguine
– The five factor model (FFM) – Big5:
• Extraversion
• Agreeableness
• Conscientousness
• Neuroticism
• Openness
 Underlying assumption:
– Users with similar personalities have similar preferences
 Measuring personality:
– the IPIP questionnaire
– For each user u a five tuple b =(b1, b2, b3, b4, b5)


Experiment

 Proposed USM:

 Baseline USM:

Simulate cold-start
stage

Get recommended
Find similar users F measure
items

personality-based
Rating-based USM
USM


Results
 F measures of all users:
– At each cold start stage s we compared both USM with the t-test


The LDOS-PerAff-1 dataset
 Properties of the dataset
– Content items
– End users
– Generic and affective metadata (for content items)
– Personality metadata (for users)
– Video recordings of users during consumption
– Explicit ratings


Data acquisition setup

•Explanations to the user
•Personality assessment with the IPIP questionnaire
•Computer interaction:
•Emotion induction approach
•Images from the IAPS dataset
•Content
•Stimuli
•Explicit Likert ratings
•Matlab GUI
•Webcam recording


Dataset basic statistics
 52 users (avg(age)=18.3 yrs, 37 females)
 IPIP 50 items questionnaire
 70 colour images from the IAPS dataset
 3640 videoclips (320x240 @ 15 fps)


Excerpt


Future work
 Looking for a robust, all-encompassing user model
 Experimental work to prove parts of the model
 Validation in real-world scenarios


Thank you.

Questions?

Affective User Modeling

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (10)

Semelhante a Affective User Modeling

Semelhante a Affective User Modeling (14)

Último

Último (20)

Affective User Modeling