This document discusses several studies related to measuring and reconstructing user experiences over time. It first describes research on how users form overall evaluations of interactive products over time. It then examines whether an online survey tool called iScale can help users recollect their experiences with a product by assisting in the reconstruction of memories. Finally, it explores whether mobile sensors and technology can help participants reconstruct their daily experiences and locations through tools like trajectory reminders, photos of emotional expressions, and a footprint tracker using visual, location and context cues.
Measuring users' experience - or, the memory of them?
1. Measuring users’ experiences
or, the memory of them?
Trajectory reminders EmoSnaps Footprint tracker
Evangelos Karapanos
Newcastle, 8 March 2011
2.
3. Nuno
Nunes Vassilis
Kostakos Monchu
Chen
Laura Rodríguez Gonçalo
Gouveia Néstor
Catano
• 20
faculty Pedro
Campos Paulo
Sampaio Eduardo
Fermé
– 14
countries,
8
languages
• Areas: Larry
ConstanHne Jos
van
Leeuwen Barbara
Pizzileo
– 11
CS,
2
physics/electronics,
2
psychology,
2
architecture,
2
design,
2
art,
2
other
Ian
Oakley Luis
Gomes Ron
Salden
Leonel
Nóbrega ValenHna
Nisi Evangelos
Karapanos
David
Aveiro Luis
Gomes Yoram
Chisik
6. My
background
BSc Physics / microelectronics, U Patras, Greece (2004)
Thesis: Model based design and evaluation of walk-up-and-use interfaces (HCI
Group, ECE department)
MSc HCI / UCL Interaction Centre, UK (2005)
Thesis: User acceptance of nomadic user interfaces (Philips Research,
Eindhoven)
PhD HCI / TU Eindhoven, NL (2010)
Title: Quantifying diversity in user experience
7. Some
of
my
current
work
(that
I
will
not
discuss
today)
Socially translucent eco-feedback technologies
How do eco-feedback technologies:
a) raise mutual awareness of family members’ consumption behaviors
b) induce feelings of accountability on individuals regarding their consumption behaviors.
!
Citizen participation on the go
How can we motivate citizen participation through mobile technologies?
•Public transit: The role of psychological empowerment: self-efficacy, sense of community,
and causal importance
•Inclucity: The role of visual and location cues on users’ ability to reconstruct the context,
and form an empathic understanding of the experience of disabled individuals
Location-aware narratives: Does locality matter?
Does the coupling between physical and virtual space result to increased immersion in
the narrative world?
8. Outline
User experience over time
What makes for positive experiences in the long run?
iScale - longitudinal data through memory
Can an online survey tool assist users in recollecting their experiences
with a product?
Technology Assisted Reconstruction
Can mobile sensors assist participants in reconstructing their daily
experiences and whereabouts?
9. Soft Reliability
48% of returned products are not attributed to a
violation of product specifications
12. User
experience
over
Hme
An
exploratory
study
How
do
users
form
overall
evalua/ve
judgments
about
interac/ve
products?
Hassenzahl,
2004
Karapanos, E., Hassenzahl, M., & Martens, J.-B. (2008). User experience over time. CHI ’08 extended abstracts on Human factors in
computing systems (pp. 3561-3566). Florence, Italy: ACM.
13. User
experience
over
Hme
An
exploratory
study
Items
close
together
are
highly
correlated.
Lines
represent
clusters.
Karapanos, E., Hassenzahl, M., & Martens, J.-B. (2008). User experience over time. CHI ’08 extended abstracts on Human factors in
computing systems (pp. 3561-3566). Florence, Italy: ACM.
14. What makes for positive experiences in the long run?
Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2009). User experience over time: an initial framework. In CHI '09:
Proceedings of the 27th international conference on Human factors in computing systems (pp. 729-738). New York, NY, USA: ACM.
15. Now
think
of
the
three
experiences
that
were
for
you
personally
most
sa.sfying
or
unsa.sfying
experiences
of
today.
Please,
use
your
own
feeling
or
a
defini.on
of
what
“sa.sfying”
and
“unsa.sfying
experience”
means.
Take
a
couple
of
minutes
to
be
sure
to
come
up
with
three
most
crucial
experiences;
you
may
also
want
to
write
them
down
for
yourself.
We
want
you
to
be
open
as
to
which
experiences
to
report.
Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2009). User experience over time: an initial framework. In CHI '09:
Proceedings of the 27th international conference on Human factors in computing systems (pp. 729-738). New York, NY, USA: ACM.
16. Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2009). User experience over time: an initial framework. In CHI '09:
Proceedings of the 27th international conference on Human factors in computing systems (pp. 729-738). New York, NY, USA: ACM.
17. Cross-‐sec8onal Repeated
sampling
“Longitudinal”
paradigms
in
HCI
Longitudinal Retrospec8ve
18. Karapanos, E., Martens, J.-B., Hassenzahl, M. (2009) Reconstructing Experiences through Sketching. Arxiv preprint, arXiv:0912.5343.
19. Remembering
is
an
act
of
reconstruc.on
rather
than
reproduc.on
Barlea
(1932)
20. How do we recall experiences?
• Validity? i.e. do memories reflect what we really
experienced?
• Reliability? i.e. in a second trial, will we recall
the same experiences?
Can we assist people in recalling - more
reliably - their experiences with a product?
21. How do we recall emotional
experiences?
Two schools of thought
- The Constructive approach
- The Value-Account approach
22. The Constructive approach
The emotional experience can neither be Design principles
stored nor retrieved. It is reconstructed 1. Feed-forward sketching, as each recalled event
from recalled contextual details will cue more events, eventually resulting to richer
episodic memories from which to infer emotions
Robinson & Clore (2002)
2. Concurrency between sketching and
reporting, as reporting will positively contribute
Type of Knowledge Source of information Type of Self-Report towards the recall of episodic cues
Experiential Online emotion, e.g.
Episodic
Knowledge Experience Sampling
Retrospective, e.g.
Episodic Episodic memory
Day Reconstruction
Situation-specific
Semantic Exit questionnaires
belief
Identity-related
Semantic Exit questionnaires
belief
23. The Value-Account approach
People may recall an overall evaluation of an Design principles
event even when they fail to recall contextual
1. Top-down sketching (i.e., split completed line in
details - “I like it but I don’t know why” parts), as participants are expected to have direct
(Betsch et al., 2001) access to this value-charged information
2. Non-concurrency between sketching and
reporting, as reporting (i.e., thinking about concrete
This type of memory, Value-Account: episodic details) might hinder or bias the recall of
value-charged information
- is more accessible than episodic memory
- can be used to cue the reconstruction
from episodic memory
- is better retained over time
24. Constructive Value-Account Control (no-graphing)
Constructive iScale, but not the Value-Account, performed better than
control condition
•More experience reports
•With more details (references to temporal information, discrete events)
•Higher test-retest consistency of time estimation (i.e., when did
an experience take place)
•Higher test-retest consistency of graphed patterns (over Value-
Account) 24
26. Experience Sampling Method
What variables do ES studies measure?
We distinguish below between self-reported measures of
behavior and experience (see Table 1). This distinction is
sampling method. Two of these provided no justifications
for their choice. Analyzing the remaining 19 papers resulted
to a total of 11 reasons for choosing alternative methods to
relevant as their reconstruction follows a different process ESM (see table 2).
whereas behavioral information may be directly accessible
through episodic memory while experiential information Table 2. Reasons for not selecting the Experience Sampling
Prompts at random, or computationally
has to be further inferred from recalled episodic cues [63]. Method along with frequency of occurence (No of papers).
to self-report on ongoing
estimated times,measures of of ES studies eliciting or both.
Table 1. Number
behavior, experience,
self-reported Reason No
behaviors and experiences. Disrupts the activity 6
Type of measures that studies elicit No
Imposes high burden to participants 3
– Where areSelf-reported measures of behavior
you? 5
Requires high effort from researchers 3
– What are you doing? of experience
Self-reported measures 22
– How far isSelf-reported measures phone?& experience
your mobile of behavior 18
Inappropriate for eliciting rich qualitative data 3
– How do you feel? Misses rare and brief events 3
Behavioral measures related most frequently to the activity
a few relevant venues would still miss a substantial number method while following a user-initiated diary approach.
of studies. We instead used the ACM Guide towas engaged with prior to of the remaining 49 user should be reported
(n=18) that the participant Computing The analysis being The studies is being in control of when, what and 2
Literature querying for the term “experience duration (n=2) (e.g. [35]), the
interrupted (e.g. [30]), its sampling” below. how often to report
No of papers referring to
without constraining to particular venues. This query
participant’s current physical location (n=15) length, sampling frequency, and response rate
returned 284 papers, published in more than fifty venues. Study
(e.g. [17])
Experience Sampling
and the social context (n=10), e.g. the number or nature of of the studies had a duration of several
The majority (80%) Limits sample size 2
60
relationship of people that are in close proximity or
days up to one month with 14 studies (34%) lasting
participate with in a conversation (e.g. [34]). Other seven days (see figure 3). Only two
between four and Depends on participants’ ability to articulate 2
45 measures of behavior related to mode studies had a duration of more than a month.
of transit [25], ongoing experience
participants’ current physical engagement [18] and mode of
15
convrersation (e.g. f2f, fixed/mobile phone etc.) [32].
30 Poses privacy concerns 2
Experiential measures related to:
15
10
• Attitudes towards behaviors or events (n=15) such as Limits number of measured variables 1
0 being interrupted (e.g. [55]), disclosing information to
2001 2003 2005 2007 2009 2011being video recorded (e.g.
relevant others (e.g. [17]), or Technology limitations 1
5
[58]).
Figure 2. 243 papers referring to experience sampling over a
27. Day Reconstruction Method
Kahneman et al. (2004)
Can a retrospective method help participants in
recalling more accurately their experiences?
ry attempt to Type of Knowledge Source of information Type of Self-Report
ntation of the
an unfamiliar Episodic
Experiential
Knowledge
Online emotion, e.g.
Experience Sampling
called stories
tails, altering
in applying Retrospective, e.g.
Episodic Episodic memory
riginal story. Day Reconstruction
h repeated
Situation-specific
Semantic Exit questionnaires
econstruction belief
e support. At
ion between
Identity-related
hile episodic Semantic
belief
Exit questionnaires
om the past,
lar event but Robinson & Clore (2002)
28. Technology
Assisted
ReconstrucHon
Can mobile sensors assist participants in reconstructing their daily experiences and
whereabouts?
Trajectory reminders in location-based preferences
Do trajectory reminders (locations visited before and after) increase the test-retest
reliability of the reconstruction process?
Emosnaps - inferring emotion from self-face pics
Can self-face pictures assist in recalling momentary emotions?
If so, is it through a recognition or a reconstruction process?
Footprint tracker
How do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls
made or received) assist in reconstructing daily behaviors and experiences?
29. Technology
Assisted
ReconstrucHon
Can mobile sensors assist participants in reconstructing their daily experiences and
whereabouts?
Trajectory reminders in location-based preferences
Do trajectory reminders (locations visited before and after) increase the test-retest
reliability of the reconstruction process?
Emosnaps - inferring emotion from self-face pics
Can self-face pictures assist in recalling momentary emotions?
If so, is it through a recognition or a reconstruction process?
Footprint tracker
How do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls
made or received) assist in reconstructing daily behaviors and experiences?
30. Control condition With trajectory reminders
! !
When recalling with trajectory reminders, participants were more consistent over two repeated recalls
separated by 1 week
31. Technology
Assisted
ReconstrucHon
Can mobile sensors assist participants in reconstructing their daily experiences and
whereabouts?
Trajectory reminders in location-based preferences
Do trajectory reminders (locations visited before and after) increase the test-retest
reliability of the reconstruction process?
Emosnaps - inferring emotion from self-face pics
Can self-face pictures assist in recalling momentary emotions?
If so, is it through a recognition or a reconstruction process?
Footprint tracker
How do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls
made or received) assist in reconstructing daily behaviors and experiences?
32. Measuring
emoHons
in
mobile
contexts
Self-Cam ’06 Wearable EMG interface ’10
Teeters, Kaliouby & Picard Gruebler & Suzuki
Can we develop a tool that is truly unobtrusive to daily
life and can be employed in long-term field studies?
Emosnaps - inferring emotion from self-face pics
Can self-face pictures assist in recalling momentary emotions?
If so, is it through a recognition or a reconstruction process?
33. Experience Sampling (Ground truth)
78% of pictures
could be used for
inferring emotions
Time-Day Photo-day Photo-week
control condition Emotion reconstruction Emotion recognition
34. Technology
Assisted
ReconstrucHon
Can mobile sensors assist participants in reconstructing their daily experiences and
whereabouts?
Trajectory reminders in location-based preferences
Do trajectory reminders (locations visited before and after) increase the test-retest
reliability of the reconstruction process?
Emosnaps - inferring emotion from self-face pics
Can self-face pictures assist in recalling momentary emotions?
If so, is it through a recognition or a reconstruction process?
Footprint tracker
How do visual cues (i.e., Sensecam), location cues, and context cues (SMS and calls
made or received) assist in reconstructing daily behaviors and experiences?
35. 1. Sensecam
2. Location logging
3. Context logging
(SMS/calls made or received)
36.
37.
38. Thank you
Evangelos Karapanos
ekarapanos.com
User experience over time
What makes for positive experiences
in the long run?
iScale
Can an online survey tool assist users in
recollecting their experiences with a product?
Tech. Assist. Reconstruction
Can mobile sensors assist participants in
reconstructing their daily experiences and
whereabouts?