3. 3
Hayley Hung, TUDelft
What is Human Face-to-face interaction for ?
Makes our lives easier:
Relationships
Trust
Co-operation
Persuade/influence others
Information sharing How could technology help
us to understand/interpret/
socially relevant behaviour?
How could this help to
influence/enhance our
experience?
4. 4
Hayley Hung, TUDelft
Research Mission Statement
To develop algorithms that can model and understand non-
verbal human social behaviour in real life situations. And
through this, to understand how to build systems that can
enhance people's quality of life by behaving with socially
aware intelligence.
Develop algorithms that are perceptive to human social
behaviour: Social Signal Processing, Machine Learning
Enhancing people's quality of life: Human Machine Interaction,
Ambient Intelligent Environments, Design, Architecture.
Socially aware intelligence: Social and Behavioural Psychology,
Ethnography.
6. Hayley Hung, TUDelft
What can you say about this picture?
Relaxed postureGestures
Vocal Behaviour
Mutual Gaze
Interpersonal
Distance
7. 7
Hayley Hung, TUDelft
Current Research Frontier
Person
detection
Person
tracking
Gaze
detection
Body pose
estimation
Group
detection
Social and Behavioural
Pscychology, Ethnography
Activity
modelling
Action
recognition
Attraction
Estimation
Rapport
Estimation
Role Recognition
Personality
estimation
Dominance
Estimation
8. 8
Hayley Hung, TUDelft
Current Research Frontier
Relationship intimacy
estimation
Conversation
quality estimation
Person
detection
Person
tracking
Gaze
detection
Body pose
estimation
Group
detection
Conversational
event estimation
Personality
estimation
Relationship
quality estimation
Social and Behavioural
Pscychology, Ethnography
Activity
modelling
Action
recognition
New
Problem
Definitions
9. 9
Hayley Hung, TUDelft
How to model social behaviour
Sensor Data Feature and Cue Extraction
Data Annotation
Social Behaviour
Modelling and
Classification
Model
Performance
Evaluation
10. 10
Hayley Hung, TUDelft
Task 1: Estimating Attraction
Source: http://catinbag.blogspot.nl/2010/07/fatal-attraction.html
Veenstra and Hung, “Do They Like Me? Using Video Cues to Predict Desires
during Speed-dates” in ICCV Workshops 2011
11. 11
Hayley Hung, TUDelft
Speed Dating, Non-verbal cues and Attraction
Can proximity-related video cues be used to automatically
predict attraction in speed-dates?
15. 15
Hayley Hung, TUDelft
Speed Dating Results
Predicting attraction
Variance in position is best
feature predictor for
women (70%).
Variance in position of the
women and synchrony
both perform well (70%)
for men.
Fusion of all synchrony features
Fusion of all movement features
16. 16
Hayley Hung, TUDelft
Speed Date Experiments : Conclusion
The video channel can indeed be a source of valuable
information in speed-dates
Results differ per gender:
Movement synchrony information is more important for
males than females.
For females, information on the movement of their male
counterpart gives good results
17. 17
Hayley Hung, TUDelft
Task 2: Classifiying Social Actions using a
Single Wearable Accelerometer
Hung, Englebienne, Kools, “Estimating Social Actions”, Ubicomp 2013
18. 18
Hayley Hung, TUDelft
Modelling Human Social Behaviour in Dense Crowds
How can we model instantaneous social
behaviour in extremely large crowds?
19. 19
Hayley Hung, TUDelft
Our Goal
To develop methods to automatically measure
socially relevant behaviour and moods in dense crowds
using just a single tri-axial accelerometer
First step: detect socially relevant actions
Speaking; Laughing; Gesturing; Stepping; Drinking
20. 20
Hayley Hung, TUDelft
Our Goal
Use insights from Social
Psychology:
Speakers move more than
listeners (McNeill 2000)
Laughter and joking
correlated with sudden
bursts of motion
(Kendon 1990)
Synchronised motion
during conversation
(Kendon 1990,Chartrand and
Bargh 1999)
21. 21
Hayley Hung, TUDelft
Data: The Scenario
32 volunteers (mostly mutual strangers)
Experiment Stages: Briefing; Meeting and Mingling; Team
formation (groups of 4); Quiz; Award Giving
Prizes for top 3 teams
5mx6m recording area
22. 22
Hayley Hung, TUDelft
Data: The Scenario
32 volunteers (mostly mutual strangers)
Experiment Stages: Briefing; Meeting and Mingling; Team
formation (groups of 4); Quiz; Award Giving
Prizes for top 3 teams
5mx6m recording area
23. 23
Hayley Hung, TUDelft
Data: The Scenario
32 volunteers (mostly mutual strangers)
Experiment Stages: Briefing; Meeting and Mingling; Team
formation (groups of 4); Quiz; Award Giving
Prizes for top 3 teams
5mx6m recording area
Each participant wore a sensing badge
25. 25
Hayley Hung, TUDelft
Social Action Conclusion
It is possible to detect socially relevant behaviour.
Can we detect when people are in the same conversational
group?
Could we even detect
personality traits
quality of people's interaction?
Quality of people's relationships?
...etc
26. 26
Hayley Hung, TUDelft
Looking to the Future...
The way we behave socially can exhibit strong detectable
patterns, which are robust to noise.
How simple can the extracted features be?
How could socially aware systems benefit better design?
28. 28
Hayley Hung, TUDelft
Applications: Urban Planning
Behaviour in Public Spaces
How can we measure statistically generaliseable changes as
a result of interventions?
31. 31
Hayley Hung, TUDelft
Workshop at ACM Multmedia (acmmm13.org/)
Barcelona , October 22
Human Behaviour Understanding for the Interactions in Arts,
Creativity, Entertainment and Edutainment
Albert Ali Salah (Bogazici University, Turkey)
Oya Aran (Idiap Research Institute, Switzerland)
Hatice Gunes (Queen Mary University of London, UK)