Introductory user testing & user research workshop for government staff & civic hackers, delivered at Thoughtworks in NYC on 4/23/14

1. Testing & User Research
Workshop

2. Photo Credit: kaboodle.com

3. “The best tool for resolving disputes within a
design team, for making design decisions based
on data rather than opinion, is sitting next to
someone who is a real person who wants to
accomplish something as they use your design to
do it.”
--Dana Chisnell, August 2, 2010

4. Btw I’m Cyd Harrell
I’m UX Evangelist at Code for America
My favorite tools: empathy and duct tape

5. Today’s Agenda
10:00-10:15: intros
10:15-11:30: core user research technique
11:45-12:30: your research set-up
1:30 - 2:45: recruiting participants
3:00 - 4:00: going beyond usability

6. #1 Find out about the people who will use your thing
#2 Align your core team around what you’re doing to do
#3 Find out what your core team is up against in their work
Research

7. The hardest two things about research:
‣ providing the prompt
‣ collecting the response
OversimplificationAlert

8. The hardest two three things about research:
‣ providing the prompt
‣ collecting the response
‣ getting everyone to buy in
OversimplificationAlert

9. The hardest three four things:
‣ getting the right participants
‣ providing the prompt
‣ collecting the response
‣ getting everyone to buy in
OversimplificationAlert

10. o.k.soafewthingsarehard

11. Let’s jump right in

12. What’s the latest with UX Research?

13. WHAT’S DIFFERENT IN 2014
Source: Pew Internet and American Life Project, January 2014

14. WHAT’S DIFFERENT IN 2014
Source: Knight Foundation Digital Access Update, February 2014

15. TOOLKIT

16. TOOLKIT

17. REMEMBER BE
graceful
When All Your
TECHNOLOGY
BREAKS

18. uximresearch.tumblr.com

19. Now we do a little experiment

20. EXERCISE #1
“Giraffe”
‣ 2 groups per table
‣ Pick one moderator and one participant
‣ 10 minutes of observation

21. MODERATOR Instructions
Your participant has a sheet of paper and that’s it
Goal: find out what kind of physical task instructions work
Hint: ask them to give you a play by play of what they are doing
Try and follow along with your own paper
NO helping, even if you could.

22. It’s A RACE
You only have 10 minutes to do this max.

23. Surprises?

24. Useful phrases 1
the approach
“Hi, would you like to participate in a study
today?”
“I’m working to improve XXXX for citizens and
I’m collecting feedback.”

25. Useful phrases 2
reassurance
“This isn’t a test of you, it’s a test of the system”
“There are no right or wrong answers”
“Positive and negative feedback is equally
valuable”
“Nothing you say is going to hurt my feelings”

26. Useful phrases 3
instructions
“I just need you to be yourself and act as you
naturally would”
“One little thing I’d like you to do differently is
talk aloud as you use this today”

27. Useful phrases 4,5,6,7
keeping the conversation going
“mm-hm” “oh interesting” “sure, I hear you”
“right” “ok” “uh-huh”
(lather, rinse, repeat)

28. Useful phrases 8
probes
“Can you tell me a little more about that?”
“When you did X a minute ago, can you tell me
how you made that choice?”
“Talk me through what happened there.”

29. Useful phrases 9
ratings
“If you had to give this a letter grade, you know
A, B, C, D, F like we used to get in school, what
would you give it?”

30. Useful phrases 10
self-explanatory
“Thank you so much, I really appreciate your
taking the time to talk to me.”

31. DESIGNINGUSER STUDIES

32. FIND THE RIGHT PARTICIPANTS
uximresearch.tumblr.com Photo Credit: Ed Yourdon

33. Photo from blog post by Dustin Larimer http://www.dustinlarimer.com/design-management-summer-camp/
OFFER A TASK

34. SCRIPT FOR CUSTOMIZATION
“Please look for a pair of pink pumps in your size.”
“Please look for a pair of shoes you’d like to buy”

35. MODERATED must-haves
Consent
Tech set-up
The beef
De-installation
Thanks & incentive

36. AUTOMATED must-haves
Expectation setting
Instructions
The beef
Thanks & incentive

37. Photo from blog post by Dustin Larimer http://www.dustinlarimer.com/design-management-summer-camp/
STREAM & RECORD WHAT HAPPENS.

38. BREAK
Photo Credit: Sorin Petcu

39. Testing & User Research
Part 2: your set-up

40. What’s New with ResearchTools?

41. Remote Design Research
Definition:
Research where the participant is not
sharing physical space with the
researcher.
% of qualitative research done remotely*
* Source: My wild guess
In person
Remote (2008)
Remote (2012)

42. Screen
Easier
Sharing

43. Awesome compared to this!

44. Prototype
So Many
Tools
Flinto

45. Experience Sampling
Task-driven
iOS & Android

46. Mobile Client
Simultaneous
Website Testing

47. WHOSE DEVICE?

48. Photo Credit: groundsel, inju
Pro:
‣ better context
‣ more comfortable users
Con:
‣ extra installation time
‣ less control

49. Pro tips for working with users’ own phones:
‣ Have a backup iPhone and backup Android device on hand.
‣ Make a power strip part of your testing station
‣ Have chargers for all the phones you’re expecting
‣ Instruct participants to install necessary apps in advance, but leave time in case
they don’t
‣ Adjust the screen brightness before you start each session

50. Pro tips for offering lab devices:
‣ Have a backup iPhone and backup Android device on hand.
‣ Charge devices between every session.
‣ Remove passwords, lock screens, etc.
‣ Dry-run every aspect of the test, since you have the opportunity.
uximresearch.tumblr.com

51. Physical Setup for MobileTests

52. Photo Credit: Diane Loviglio, Mozilla

53. RealTalk About Holding Phones
Source: Steve Hoober in UX Matters, February 2013

54. TestTargets

55. PROTOTYPES,WIREFRAMES, SKETCHES?

56. HECKYES.

58. Photo from blog post by Dustin Larimer http://www.dustinlarimer.com/design-management-summer-camp/
PAPER PROTOTYPES, POST-IT STYLE

59. Photo from UX Magazine article by Tania Lang
PAPER PROTOTYPES, WITH SCROLLING

60. Lo-fi
clickable
prototypes

61. Photo from UX Magazine article by Tania Lang
Prototyping Apps for Higher Fidelity
Proto.io
Fluid
Flinto
PhoneGap

62. Photo from UX Magazine article by Tania Lang
Cool Hack:
Screenshots in an album or photostream

63. The hardest two things about research:
‣ providing the prompt
‣ collecting the response
InTheLab

65. Photo Credit: Hans Makkee
high resolution
portable
streaming camera

66. Mobile Screen
Recording

67. Source: OpenDeviceLab.com
Device Labs

68. uximresearch.tumblr.com

69. EXERCISE #2
“RollYour Own”
‣ Share out

70. Taking Notes
Second monitor if possible
Excel Hack
(remoteresear.ch/samples)

71. Observer Involvement
IM
Campfire (!)
Integrated Chat
Passing Notes

72. Questions?

73. Let’s do lunch

74. Testing & User Research
Part 3: recruiting participants

76. RECRUITING OPTIONS
‣ Grab someone in the hallway
‣ Grab someone who isn’t involved in your project
‣ IM your outside friends for a quick look
‣ Panels and recruiting agencies
‣ Real citizens from email & consent lists
‣ Physical & web intercepts
realness

77. TWITTER RECRUITING

78. ANATOMY OF A RECRUITING TWEET
@person we’re interviewing [job or characteristic] about [topic] on [dates]
(paid research) -interested? [LINK]
I’m looking for [job or characteristic(s)] for a [research activity] on [dates]
incentive $X [LINK] please RT
Pro tip: edit your profile to mention “UX Researcher” and “Current Study:
Martian Life”

79. BIGGER THAN A TWEET
‣ Why are you doing this work?
‣ What exactly are you asking? (How much time, where, when)
‣ Who do you need? (tasks or demographics)
‣ Who is asking?
‣ Who vouches for you? (official sponsor or trusted leader)
‣ Will there be a reward or incentive?
‣ Can this be confidential or anonymous?

80. Screener constraints
Keep it short
Always ask an open-ended question
Different from surveys
Who gets paid?

81. MOBILE FORMS ARE GETTING WAY BETTER!

82. PANELS

84. “What happens when you let go of a balloon?”
PRO TIP: HOW TO FIND ACTUAL SPEAKERS OF A LANGUAGE:
“ Qu'est-ce qui se passe quand vous lâchez un ballon? ”

85. WHAT PARTICIPANTS WORRY ABOUT

86. Who are you and who sent you?
(can I trust you?)

87. Will participating affect my actual outcome?

88. Is this going to end up on
YouTube?

89. remoteresear.ch/samples
“Online Consent”

90. Paying Incentives
Collect minimal personal info
Deliver quickly
Appropriate for location

91. Paying Incentives
“Free cookies and
lemonade for your
city feedback”

93. EXERCISE
‣ Consider format (poster, landing page, online form)
‣ Cover the questions
draft a recruiting appeal

94. ‣ Why are you doing this work?
‣ What exactly are you asking? (How much time, where, when)
‣ Who do you need? (tasks or demographics)
‣ Who is asking?
‣ Who vouches for you? (official sponsor or trusted leader)
‣ Will there be a reward or incentive?
‣ Can this be confidential or anonymous?

95. share out

96. BREAK
Photo Credit: Mrs Magic

97. User Testing & Research
Part 4: going beyond usability

98. WHAT IF IT’S NOT USABILITY?

99. EXPERIENCE SAMPLING

100. EXPERIENCE SAMPLING
uximresearch.tumblr.com

101. Cyd’sResults

102. Cyd’sResults

103. CONCEPT & NAVIGATION TESTING:
EXERCISE

104. UNMODERATED STUDIES

105. 3 parts
Provide test target
Record user behavior
Capture comments (optional)

106. Prototype
Unmoderated
Testing

108. Mobile Diary Studies
iOS & Android

109. Results

110. Results

111. Results

112. CAPTURE WHAT HAPPENS.

113. Surveys
via SMS

114. About getting buy-in
AFewWords

115. How “Testing” Is Perceived

116. Numbers (& Numbers People)
AreYour Friends
Photo Credit: Search Influence on Flickr

117. Video Is Another Friend
Photo Credit: Search Influence on Flickr

118. Try asking for
One Day
$200
PeopleToWatch
Photo Credit: Search Influence on Flickr

119. “Fix the machine, not the person”
-- Aaron Swartz, 9.25.12
ActualAdvice

120. There’s always a way

121. Questions?

122. cyd@codeforamerica.org
@cydharrell

123. Thanks for coming.
Go run some awesome research.
Photo Credit: Wade Rockett on Flickr