1.
For a digital product
usability testing
World Usability day colombia 2014
Kyle Soucy
@kylesoucy

2.
Image source: facit digital
What is Usability Testing?

3.
How do you conduct
a usability study?

4.
Lets us begin our
Journey...

5.
Plan Study
Define goals and objectives

6.
What do you want to learn?
How do teachers
shop for books?
Do they use these
slider dots?
Do they notice
these arrows?

7.
WhatYou Can Learn
From A Usability Study

8.
Do users understand how to
complete a desired task?

9.
Does the design prevent users from
completing tasks more effectively?

10.
Do users notice the buttons or links
that you want them to click on?

11.
Are users engaged in the content
and understand it?

12.
How long does it take for a user
to complete a task?

13.
Does the interface cause users undo
frustration or confusion?

14.
WhatYou Can’t Learn
From A Usability Study

15.
Opinions or other subjective data
(We’re not testing hundreds or thousands of people)

16.
The future
(We can only report on what we directly observe)

17.
Design preferences
(You won’t get answers to design questions)

18.
Why can’t I
just use a focus
group?

19.
Qualitative vs.
Quantitative Research
• Qualitative Research
- Data gathered directly by observing the user
- Researcher can ask follow-up questions, probe on behavior
- Analysis of data is not mathematical (Why? How? Compare...)
• Quantitative Research
- Data gathered indirectly through surveys, log files, support
calls, etc.
- Data can be averaged or statistically tested
Source: Christian Rohrer Move, Inc. | Realor.com

20.
Understanding The Domain
And The Product

21.
If you don’t know the product,
you can’t test it!

22.
Plan Study
Define what you are testing

23.
Low vs. High-Fidelity
Prototypes

24.
Plan Study
Task design

25.
1. Write a list of the most important
tasks that users need to accomplish
using the product.
2. Turn each task into a scenario.
a) Provide some context.
b) Supply needed information.
c) Don’t give any clues or hints!
3. Decide on the order of tasks.
a) Make the first task easier.

26.
Bad Task:
Buy a book about genealogy online.

27.
Good Task:
You need to get a book about genealogy that
covers the reliability of different sources.You
want this book in your hands by Thursday.
Find and purchase this book.

28.
What makes a good task:
• You’re asking users to do something
that they would actually do in real life.
• Good scope. Not too broad, not too
specific.
• Has a clear end point.
• Elicits action, not opinion.
Don’t start with
“how would you” or
“where would you”

29.
Good Task:
You need to get a book about genealogy that
covers the reliability of different sources.You
want this book in your hands by Thursday.
Find and purchase this book.Find and purchase this book.
✓ Has a clear end point
✓ Elicits action, not opinion
✓ Good scope

30.
Your Turn...
1. Take out a sheet of paper (or your
computer, phone, or iPad)
2. Write down the most important task
your users perform on your product.
3. Turn the task into a scenario.
a) Provide some context.
b) Supply needed information.
c) Don’t give any clues or hints!

31.
Other things to consider...

32.
Restrictions:
Consider whether or not you want to place
restrictions on how participants try and
accomplish the tasks (e.g.“don’t use search”
or “stay on the website”).

33.
Printing:
If you’re conducting an in-person test, you
may want to print out the scenarios so the
participant can refer back to it.

34.
Time-Fillers:
Create more tasks than you think you have
time to complete just incase someone finishes
early. Every minute with a participant is an
opportunity to learn something.

35.
Visual Aids:
If you’re worried that revealing the name of
something will bias the results, consider
showing the participant a picture instead.

36.
Visual Aid Example:
A friend of yours uses on of these to make baby food,
so you’d like to buy one...

37.
Alternating Tasks/Prototypes:
Depending on what you’re testing, you may
need to alternate tasks. Participants are
learning the interface as they go, so the last
task may always be the easiest to accomplish
unless you alternate them.

38.
Interview-based vs. Predefined Tasks

39.
Interview-based Tasks
1. Interview participant about how they use
the product.
2. Use the answers given in the interview to
create tasks on the fly.
When in doubt just ask,“what was the last thing you did
when you used this product? Can you show me?”

40.
Plan Study
Define participants

41.
Demographics

42.
Cover all user groups
- Prospective as well as
existing users

43.
Be specific and get buy-in
from stakeholders!

44.
Recruiting Options

45.
Do ItYourself

46.
Hire a recruiter
Warning:
It’s never this easy!

47.
Live-intercept Study

48.
How many to test?

49.
Source: Jakob Nielsen

50.
Budget Allocation
What NOT to do...
1 Round of Testing

51.
Budget Allocation
What you SHOULD do...
Round 1 Round 2 Round 3
Design
Iteration
Design
Iteration

52.
What to pay
participants?

53.
Plan Study
define testing method

54.
Vs.
Remote vs. In-person
Testing

55.
Remote Testing
Pros & Cons
Participants stay in their native environment.
Accessible to a larger & more diverse pool of
participants.
Recruiting is easier. Less no shows & easier
last minute replacements.
Opportunity to gather ethnographic data.
Cost & time savings. No travel & lab rental
fees.
Easier for observers to “attend” a session.
Risk of compromising data because the
participant is using a foreign PC is removed.
Pros:
Can’t see the user’s facial expressions.
Can’t see if the user is using peripheral
devices.
Dependent on an Internet connection.
Can only recruit participants with a high-
speed Internet connection.
Cons:

56.
Remote Testing Tools
• GoToMeeting
• WebEx
• Adobe Connect
• Join.me
• Many more...

57.
Remote Testing Tips
• If using a cordless phone, have multiple handsets
charged and ready
• Use a phone recording controller
• Use as few tools as possible
• Back up, back up, back up!
• USB Modem
• Screen sharing backup

58.
• Create a test meeting to make sure the
participant can connect beforehand
• Instruct participants to log on early and use a
landline (noVOIP)
• Turn off entry/exit chimes for conference call
• Allow only one observer to chat to you
• Use a separate chat/IM client

59.
Unmoderated
vs.
Moderated Testing

60.
What is Unmoderated
Remote Testing?

61.
Unmoderated
Testing Tools
(There are a lot!)

62.
We Won’t Talk About...

63.
We Will Talk About...

64.
Has Anyone Here Used
These Tools Before?

65.
WhatYou Can Learn
(Depends on the tool and how you use it!)

66.
Web Analytics
Image source: http://www.hiero.com/web-analytics.html

67.
Task Completion Rates
Image source: Loop11

68.
Number of Clicks/
Clickstream
Image source: Userlytics

69.
Time on Task/Page
Image source: http://blog.usabilla.com/underdogs-beat-expedia-in-usability-test/

70.
Satisfaction Ratings/
Opinion Rankings
Image source: http://wikidoc.org/index.php/Pain
It is simple to use:
Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree

71.
True Intent
Image source: http://ethnio.com/images/live_screener_example_page1.jpg

72.
In-context Feedback
Image source: Loop11
“...I’m only offered a
search by model number.
How ridiculous.”
“...I would like to see the
products availability and delivery/
shipping information listed.”

73.
How Actionable
is the Data?
(Depends on the questions you ask!)

74.
Conducting
Unmoderated Remote
Usability Tests

75.
Process
1. Define the study
2. Recruit participants
3. Launch test and send email invites
4. Analyze results
3. Run a Pilot test!
>
4 /
5 /

76.
Benefits

77.
Test hundreds of people simultaneously while keeping
them in their own natural environment.

78.
Test multiple websites simultaneously.

79.
Reduced costs... depending on the tool you use.

80.
Great way to plant the seed of UCD methodologies
and introduce usability testing into a company.
UX

81.
Fewer logistics to manage and no need to worry
about no-shows and getting last-minute replacements.

82.
Fast Results

83.
Drawbacks

84.
Nothing beats watching participants in real time and
being able to ask probing questions about what they
are doing as it’s happening.

85.
Some participants may only be interested in earning
the honorarium you’ve provided as an incentive.
$$$

86.
You cannot conduct interview-based tasks.

87.
It is simple to use:
Strongly Disagree ---1---2---3---4---5---6---7 Strongly Agree
Image source: http://www.hiero.com/web-analytics.html
What participants report on surveys can be very
different in comparison to what they actually do.

88.
Image source: http://www.hiero.com/web-analytics.html
It’s possible for participants to think they’ve
successfully completed a task when they haven’t.

89.
Image source: http://www.hiero.com/web-analytics.html
Does it matter if it took someone longer to complete
a task?

90.
When to Conduct
Automated Testing?

91.
Not a replacement for
moderated testing or research.
Best used to compliment your
qualitative research.

92.
Unmoderated
Testing Tools
(A closer look...)

93.
Automated Testing Tools
with Recruiting Panels

94.
The danger...
I wonder how a real
customer would actually use
this website?
Panelist

95.
UserTesting.comVideo

96.
BLOCKED
BLOCKED
The danger...

97.
There are no
shortcuts
to conducting
good research.

98.
Automated Testing Tools
with Other Recruiting Options

99.
Loop11

100.
Treejack

101.
Chalkmark

102.
Chalkmark Results

103.
Automated Tools with
Audio andVideo

104.
Handout:
Overview of
Unmoderated
Usability
Testing
Tools
/usableinterface

105.
Plan Study
schedule participants

106.
Example Testing Schedule

107.
Plan Study
invite participants and observers

108.
Key to Success…
Attendance of product
decision makers during testing!

109.
Example Remote Participant Invite

110.
Set Up Study

111.
Usability “Labs”

112.
Image source: facit digital
Source: Kent State University
Vs.

113.
Ideal “Lab-Less” Set Up

114.
Should observers be in the room?

115.
Video Recording Options
• Techsmith Morae - $1,995 USD
• Techsmith Camtasia - $99 USD
• Silverback (Mac only) - v2 Free
• Online Meeting Built-in Recorders

116.
ExampleVideo Consent Form

117.
Example Moderator Guide

118.
Moderate Study

119.
Greet Participant and Give Introduction

120.
Preliminary Interview

121.
Evaluation Instruction

122.
Think Aloud Techniques

123.
Think Aloud Techniques
Concurrent (CTA) Retrospective (RTA)
Vs.

124.
Variation of RTA...
Show the participant their video
after they’re done testing.

125.
Reminder Prompting
• So...?
• So, what are you thinking?
• What are you seeing here?
• What are you looking at?
• Can you tell me what you think is
happening here?
Source: Moderating Usability Testing by Joe Dumas and Beth Loring

126.
Probing Questions
• Is that what you expected?
• What would you do next?
• What did you think about that task?
• Help me to understand more about...
Source: Moderating Usability Testing by Joe Dumas and Beth Loring

127.
Encouraging Statements
• This feedback is really helpful.
• Thank you for thinking out loud.
• That’s good to know.
• You’re doing fine.
Source: Moderating Usability Testing by Joe Dumas and Beth Loring

128.
Remain Neutral
• When in doubt turn the question around:
• “I don’t know what do you think?”
• “What would you do if I wasn’t here?
• Keep acknowledgements limited to:“ah”,
“Mm Hmm”,“Ok”, etc.
Source: Moderating Usability Testing by Joe Dumas and Beth Loring
and Rocket Surgery Made Easy by Steve Krug

129.
Letting Participants Struggle

130.
Note-taking

131.
Analyze & Report
Findings

132.
Formal Usability Testing Report
Findings
categorized by
screens or pages

133.
When, What, Who, Where, !
and Why Statement!
3-4 !
Positive Findings!
3-4 !
Negative Findings!

134.
Major Usability
Problem

135.
Video Clip Example

136.
Mobile
Usability Testing

137.
Image source: http://img.gawkerassets.com/img/181rzskxifcohjpg/original.jpg

138.
“...the mobile web will be
bigger than desktop Internet
use by 2015.”
- Morgan Stanley Study
Source: http://mashable.com/2010/04/13/mobile-web-stats/Source: http://mashable.com/2010/04/13/mobile-web-stats/

139.
Image source: http://3.bp.blogspot.com/-Oh6rZE2fG1k/Uct4T4KhE0I/AAAAAAAAARo/3c_HFDkMwAE/s1600/girls-on-their-phone.jpg
Mobile matters!

140.
Mobile In-person Usability Testing
Image source: http://unmatchedstyle.com/news/usability-testing-on-mobile-devices.php

141.
Image source: http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/

142.
Testing on paper prototypes
Image source: http://www.90percentofeverything.com/2010/05/07/quick-tip-make-your-own-iphone-usability-testing-sled-for-5/

143.
What about remote mobile
usability testing?

144.
Current Navigation Tabbed Navigation Hidden Navigation

145.
What about remote mobile
usability testing?
unmoderated
>

146.
Other Mobile
Usability TestingTools

147.
Image source: http://blog.larrybodine.com/uploads/image/a-99-year-old-lady-using-the-ipad.jpg

148.
Fitting Research
Into An Agile
Process

149.
Waterfall
Analysis Design Coding Testing
Analysis
Design Coding
Testing
Agile
Analysis
Design Coding
Testing
Analysis
Design Coding
Testing
1-4 week sprints

150.
Parallel Sprints
#1

151.
Sprint 1Dev:
UX:
Sprint 2 Sprint 3
Release
Sprint 1 Sprint 2 Sprint 3

152.
Staggered Sprints
#2

153.
Source: http://www.upassoc.org/upa_publications/jus/2007may/agile-ucd.pdf
Tip: Choose a feature to develop first that
has a lengthy dev sprint.

154.
Testing On A
Small Budget

155.
• Conduct your own recruiting
• Don’t offer large honorariums
• Don’t use a lab
• Don’t video record
• Test less people
• Use surrogate users or just conduct
cognitive walkthroughs
• Conduct unmoderated studies
• Don’t write formal reports

156.
Learn more...
• Book: Interviewing Users
by Steve Portigal
• Book: Moderating Usability Tests
by Joe Dumas & Beth Loring
• Book: Paper Prototyping (chapter on task design)
by Caroline Snyder
• Book: Rocket Surgery Made Easy
by Steve Krug
• Book:The Moderator’s Survival Guide
by Donna Tedesco & Fiona Tranquada

157.
• Article: Seven Common Usability Testing Mistakes
by Jared Spool
http://www.uie.com/articles/usability_testing_mistakes/
• Article: Interview-Based Tasks: Learning from Leonardo DiCaprio
by Jared Spool
http://www.uie.com/articles/interview_based_tasks/
• Article: Mobile Testing Toolbox: Part 1&2
by Amber Derosa
http://uxmag.com/articles/mobile-testing-toolbox-part-1
• Article: How To Conduct A Usability Test On A Mobile Device
by Jeff Sauro
http://www.measuringusability.com/blog/mobile-usability-test.php
• Article:Why And How To SegmentYour Customers
by Jeff Sauro
http://www.measuringu.com/blog/segment-customers.php

158.
www.usableinterface.com
@kylesoucy
Thank you.
/usableinterface