1. UX Assessment Techniques
Jen Romano Bergstrom
December 11, 2013
NOVA UX
@forsmarshgroup @romanocog

2. Measuring the UX
“the extent to which a
product can be used by
specified users to
achieve specified goals
with effectiveness,
efficiency, and
satisfaction in a
specified context of
use.” ISO 9241-11
+ emotions
@forsmarshgroup @romanocog
2
• How does it work for the
end user?
• What does the user
expect?
• How does it make the user
feel?

3. Why is it important?
• Put it in the hands of the end user.
• Things may seem straightforward to you but maybe not to
your users.
@forsmarshgroup @romanocog
3

4. Why is it important?
• Put it in the hands of the end user.
• Things may seem straightforward to you but maybe not to
your users.
@forsmarshgroup @romanocog
4

5. Why is it important?
• Put it in the hands of the end user.
• Things may seem straightforward to you but maybe not to
your users.
@forsmarshgroup @romanocog
5

6. Where to test
LABORATORY
•
Controlled environment
•
All participants have the
same experience
•
Record and
communicate from
control room
•
Observers watch from
control room and provide
additional probes (via
moderator) in real time
•
•
Incorporate physiological
measures (e.g., eye
tracking, EDA)
No travel costs
@forsmarshgroup @romanocog
6
REMOTE
•
Participants in their
natural environments
(e.g., home, work)
•
Use video chat
(moderated sessions)
or online programs
(unmoderated)
IN THE FIELD
•
Participants tend to be
more comfortable in
their natural
environments
•
Recruit hard-to-reach
populations (e.g.,
children, doctors)
•
Conduct many sessions
quickly
•
Moderator travels to
various locations
•
Recruit participants in
many locations (e.g.,
states, countries)
•
Bring equipment (e.g.,
eye tracker)
•
Natural observations

7. Where to test
LABORATORY
•
Controlled environment
•
All participants have the
same experience
•
Record and
communicate from
control room
•
Observers watch from
control room and provide
additional probes (via
moderator) in real time
•
•
Incorporate physiological
measures (e.g., eye
tracking, EDA)
No travel costs
@forsmarshgroup @romanocog
7
REMOTE
•
Participants in their
natural environments
(e.g., home, work)
•
Use video chat
(moderated sessions)
or online programs
(unmoderated)
IN THE FIELD
•
Participants tend to be
more comfortable in
their natural
environments
•
Recruit hard-to-reach
populations (e.g.,
children, doctors)
•
Conduct many sessions
quickly
•
Moderator travels to
various locations
•
Recruit participants in
many locations (e.g.,
states, countries)
•
Bring equipment (e.g.,
eye tracker)
•
Natural observations

8. How to test
ONE-ON-ONE SESSIONS
•
•
•
In-depth feedback from
each participant
FOCUS GROUPS
•
No group think
Can allow participants to
take their own route and
explore freely
Participants may be
more comfortable with
others
•
Interview many people
quickly
•
Opinions collide
•
No interference
•
Peer review
•
Remote in participant’s
environment
•
Qualitative
•
Flexible scheduling
•
Qualitative and
Quantitative
@forsmarshgroup @romanocog
8
SURVEYS
•
Representative
•
Large sample sizes
•
Collect a lot of data
quickly
•
No interviewer bias
•
No scheduling sessions
•
Quantitative analysis

9. When to test
@forsmarshgroup @romanocog
9

10. What to measure
EXPLICIT
OBSERVATIONAL
+ Post-task satisfaction
questionnaires
+ In-session difficulty ratings
+ Verbal responses
+ Moderator follow up
+ Real-time +/- dial
+ Ethnography
+ Time to complete task
+ Reaction time
+ Selection/click behavior
+ Ability to complete tasks
+ Accuracy
IMPLICIT
+ Facial expression analysis
+ Eye tracking
+ Electrodermal activity (EDA)
+ Behavioral analysis
+ Linguistic analysis of verbalizations
+ Implicit associations
+ Pupil dilation
@forsmarshgroup @romanocog
10

11. Case 1
• Problems: What do users want? Does the new design work?
• Methods: Focus groups, one-on-one interviews, in-lab
usability testing with eye tracking
@forsmarshgroup @romanocog
1. Participant repeatedly fixated the upper right hand corner. Participant said that he/she was looking for a search tool on the
page. The search tool was in a disappearing banner on the page.
2. Participants had similar fixation counts across bottom links, indicating uncertainty of where to click to get started.
11

12. Case 1
• Problems: What do users want? Does the new design work?
• Methods: Focus groups, one-on-one interviews, in-lab
usability testing with eye tracking
@forsmarshgroup @romanocog
1. Participant repeatedly fixated the upper right hand corner. Participant said that he/she was looking for a search tool on the
page. The search tool was in a disappearing banner on the page.
2. Participants had similar fixation counts across bottom links, indicating uncertainty of where to click to get started.
12

13. Case 2
• Problem: What parts of the form do people actually read?
• Method: In-lab usability testing with eye tracking
@forsmarshgroup @romanocog
13

14. Case 2
Participants did not read the
instructions in their entirety (page 1:
left; page 3: right); rather, they skimmed
and then moved on to the form where
they needed to enter information.
120
Time (seconds)
100
80
60
40
20
Aggregate fixation count heat map across all
participants, Page 1. Participants looked at
‘Purpose of Form’ section the most often.
@forsmarshgroup @romanocog
14
0
Page 1
Page 2
Page 3
Length of time spent on each page of the instructions
before working on form.

15. Case 3
Prototype Grid
@forsmarshgroup @romanocog
15
Old Grid

16. Case 4
Gaze Plot: After getting an error message, the participant
had to search all over the screen to find the missing field.
Smartphone
“How do I advance to the next screen?”
“It seems like it's stuck on the screen.”
@forsmarshgroup @romanocog
16
Tablet

17. Many ways to evaluate the UX
•
•
•
•
Surveys
Focus groups
In-person one-on-one with eye tracking
Analytics
@forsmarshgroup @romanocog
17

18. Obstacles to UX Testing
• There is no time.
– Start early in development process
– One morning a month with 3 users – Krug
– 12 people in 3 days – Anderson Reimer
– 12 people in 2 days – Lebson & Romano Bergstrom
• Can’t find representative users
– Everyone is important
– Travel
– Remote testing
• We don’t have a lab
– Test anywhere
@forsmarshgroup @romanocog
18

19. Thank you!
•
•
•
Twitter: @forsmarshgroup
LinkedIn: http://www.linkedin.com/company/fors-marsh-group
Blog: www.forsmarshgroup.com/index.php/blog
Jennifer Romano Bergstrom
@romanocog
jbergstrom@forsmarshgroup.com
NOVA UX