Getting user feedback on your progress is key to making successful interfaces and it doesn’t have to take months. In this session you will learn how setting up regular usability tests can allow you to save time doing the studies and without sacrificing quality. Learn strategies and techniques that can be used for making traditional and remote usability testing methods easier to plan and conduct. We will cover usability testing from planning through analysis, and ways to provide useful and usable recommendations to the team. This session covers the following topics: • Planning tips and tricks • Recruiting methods • Note taking and managing observers • Specific tips for methods (Traditional, Rapid Iterative Testing and Evaluation (RITE) • Specific tips for locations (in-person, on-site, remote) • Analysis and sharing your findings • Making usable recommendations

1. CodeMash 2012
January 12, 2012
EASY & EFFECTIVE USABILITY TESTING
PRESENTED BY CAROL SMITH
@CAROLOGIC

2. "The biggest waste
of all is building something
no one wants"
@ericries #LeanStartupMI via @MelBugai – May 19, 2011

3. FIND OUT WHAT YOUR USERS REALLY NEED
http://www.flickr.com/photos/ableman/534155207/sizes/o/in/photostream/
Page 3
http://www.flickr.com/photos/ableman/

4. AVOID MISTAKES
http://www.flickr.com/photos/dwulff/12256075/sizes/m/in/photostream/
Page 4
http://www.flickr.com/photos/dwulff/

5. MINIMIZE COST & EFFORT
Page 5

6. SAVE TIME & MONEY
Page 6

7. GET HAPPY, SATISFIED USERS
http://www.flickr.com/photos/cogdog/2194678510/sizes/o/in/photostream/
Page 7
http://www.flickr.com/photos/cogdog/

8. Create a great,
usable, accessible,
and relevant experience

9. WHAT IS USABILITY
TESTING?

10. USABILITY TESTING
•Real users doing real tasks
•Not guided
•Using prototypes or live
products
•Observed in the field
Steve Krug - http://www.sensible.com/rsme.html
Page 10

11. L O C AT I O N I S N ’ T I M P O R TA N T
•Anywhere
•Any Stage of dev
•Anytime
•Realistic environment
http://www.flickr.com/photos/nzdave/491411546/sizes/o/in/photostream/
Page 11
http://www.flickr.com/photos/nzdave/

12. PA R T I C I PA N T S
Expert Led Required
• Heuristic reviews • Usability Testing
• Walk-through • RITE Testing
• Accessibility review • Card Sorting
Garrett Goldfield . Fast and Cheap Usability Methods: Using Discount Usability Techniques to Drive
Page 12
Design on Time and Under Budget. http://www.nngroup.com/events/tutorials/guerilla_usability.html

13. WHY DO USABILITY TESTING?
Page 13

14. KISS

15. REGULAR TESTING
(Yes, this is an old idea; a great one!)
Page 15

16. WHY REGULAR?
•Team becomes:
• accustomed to steady stream of qualitative insight
• insight ensures quick decisions
• …line up with business and user goals
Page 16
Adapted from Jeff Gothelf - http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-with-agile/

17. “Teams should stretch
to get work into that day’s
test and use the cadence
to drive productivity.”
Jeff Gothelf - http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-play-nice-with-agile/

18. BRING IT ON!
•Recruitment process
•Small focused tests
•Regular timing or per Sprint
•Same day mid-week (not Monday or Friday)
Page 18

19. W H AT T O T E S T ?
•Select 2 weeks prior to study
•“Focus ruthlessly on a small number of the most important
problems” – Steve Krug
• Work in Progress
• Websites, mobile, products, services
• Prototypes: Paper to hi-fi
• Concepts, rough ideas, brainstorming
• Competing designs (A/B testing)
• Comparative studies across market
• User research
Page 19

20. U S A B I L I T Y T E S T I N G D AY
•Make team aware
•Invite everyone
• Watch remotely
• Recurring meeting invites for stakeholders
Page 20

21. MEASUREMENTS?

22. MEASURE ABILITY TO ACHIEVE GOALS
•Effectiveness
•Efficiency
•Learnability
•Satisfaction
•More…
Page 22

23. MEASUREMENTS
•Success
•Time on task (not generally recommended)
•% of tasks completed/not completed
•Number of steps to accomplish task
•Learning
•Number of errors
•Number of times help consulted
•Satisfaction
Page 23

24. MAKE IT REPEATABLE:
PLAN AHEAD

25. PLANNING IS TIGHT
•UX
• Plan study
• Space and equipment
• Identify work
Page 25

26. PRE-BOOK YOUR ROOMS
•Test & Observation Rooms
•Any location will do:
• Conference rooms
• Offices
• Quiet corner of cafeteria
• Remote
•Time for facilitation, breaks,
post-meeting
• No more than 5 per day
http://creativecommons.org/licenses/by-sa/2.0/
http://www.flickr.com/photos/raphaelquinet/513351385/sizes/l/in/photostream/
Page 26
http://www.flickr.com/photos/raphaelquinet/

27. RECRUITMENT
•Who are your primary users?
• Will they be difficult to recruit?
• How expensive is their time?
Page 27

28. C R E AT E A S C R E E N E R
Guide that helps determine who will participate.
•Get details:
• Education
• Computer/Internet use and expertise
• Knowledge of topic
•Get them talking
• Clear Communicator
• Able to express themselves verbally
•People who pass the screener should closely match your
user group definition
Page 28

29. RULES OF THUMB
Create a Budget
•Recruiting: ~$250 per participant (varies)
•Compensation: Starts at $50 - $100 and up
• Internally - use swag
• “Big fans” – cheaper alternatives ($15 Starbucks card)
Page 29

30. HONORARIUMS
•Make it easy – buy ahead
• Gift cards from retail stores (Target, Starbucks)
•More complex
• Amazon Gift Cards
• Product credits
•Do not recommend Visa gift cards or similar
Page 30

31. HIRE A RECRUITER
•Allows you to focus.
•Good recruiters:
• find right participants
• give regular updates
• take care of directions,
confirmations, incentives, etc.
Page 31

32. DO IT YOURSELF
•Internal resource who REALLY understands who you need
•Create a panel (large recruits, less often)
• Internal customer lists, user groups
• Schedule as needed
• Use cautiously
•Final scheduling ALWAYS by phone
• Ask questions that force them to talk
• Don’t recruit non-talkers
Page 32

33. USE SITE TO RECRUIT
• Pop up on your site:
• http://ethnio.com
Page 33

34. INCLUDE PEOPLE WITH DISABILITIES (PWD)
“We are all only temporarily
able-bodied.
Accessibility is good for us all.”
•Provide the Spirit of the law (Section 508, WCAG 2.0)
Page 34
-@mollydotcom at #stirtrek 2011 via @carologic May 6, 2011

35. N U M B E R O F PA R T I C I PA N T S
Controversy Abounds
•As many as possible (rarely statistically significant)
•Usability Testing Research (in 1990’s)
• 5 from distinct sub-group of the user population will yield
80% of the findings (Nielsen, Virzi, Lewis)
• Assumes expert has reviewed and found obvious issues
•Recommend:
• Early tests with 8 – 12 participants per user group
• Iterative testing (3 per day, iterate, 3 new participants)
Page 35
Barnum, Carol M. (Jan. 2003). What’s in a Number? STC Usability SIG Newsletter, Usability Interface.
http://www.stcsig.org/usability/newsletter/0301-number.html Retrieved: 20080323

36. F O R M A L L A B A P P R O P R I AT E ?
•Your team
• Need them to show up
• Need to be able to schedule as needed
•Recommend laptop/portable lab
• Can be semi-permanent
• Ready when you are (minimal setup)
Page 36

37. C R E AT E R E U S A B L E T E M P L AT E S
•Screener
•Scripts/Guides
• Tasks (same or alternating)
• Pre and Post questions
•Consent Forms
•Data Collection format
Page 37

38. TESTING REMOTELY

39. REMOTE TESTING OPTIONS
Moderated Un-Moderated
• Researcher is present • No Researcher during study
• Same effort as in-person • Minimal effort
• Limited number of participants • High number of participants per
per day (3-5) day
• Immediate feedback • Test data compiled usually
• Get to the “Why?” by asking within 2 days
questions • Reduced or no ability to ask
questions
Page 39

40. R E M O T E - M O D E R AT E D
•Lab setup
• Robust computer
• Screen sharing software
•Participants camera (optional)
•Internet on both ends
• Speedy
• Separate from phone line
Page 40

41. R E M O T E – U N - M O D E R AT E D
•No “Lab” needed – online software
•Participants camera (optional)
•Participant’s internet only
•Recommend to complement Moderated Testing
Page 41

42. BENEFITS OF REMOTE
•No travel
•Easy to set up
•No special room required (advised!)
•Can be conducted from almost anywhere
•See their computer environment
•Drawbacks:
• No F2F
• Missed interaction
• Technology will fail
Page 42

43. S O F T WA R E
Un-moderated Moderated
• Userlytics • GoToMeeting (screen
• Loop 11 sharing)
• UserZoom • Skype
• ChalkMark • Morae
• Silverback
Page 43

44. DURING THE STUDY

45. N O T E TA K I N G
Paper? Computer?
• Printed Script • People are less bothered
• Quiet by typing now
• Analysis may take longer • Can you pay attention?
• Great for remote • Quicker Analysis
Page 45

46. Q U I C K E R – O N B I G PA P E R
Issue P1 P2 P3
Search Used Yes No No
Widget 1 Used N/A Used –
unsure
about
Task 1 Notes 3 – easy 2 – needed 3 – easy
help
Task 2 Notes 2 – needed 2 – easy 2 – needed
help help
Task 3 Notes 2 – needed 3 – easy Ran out of
help time
Task 4 Notes 2 – needed 3 – easy Ran out of
help time
Page 46

47. OBSERVERS
•PO, PM, dev, etc.
• Stakeholders must attend to approve changes
• Training: Set expectations (no interference)
• Take notes
• Make a list of 3 most serious issues
• Help operate software/recording equipment (if in use)
Page 47

48. THEY’RE HERE…
Relax, they are only human
•Get them talking
•Will get frustrated
• Be prepared - how react
• Be supportive
• Guide them back to the task
• Do not lead
•Listen more than you talk
http://www.flickr.com/photos/chuybregts/
Page 48
http://creativecommons.org/licenses/by-nc-sa/2.0/

49. TESTING

50. TRADITIONAL USABILITY TESTING
•Conduct anytime
•One concept tested throughout
•Feedback given when the testing is done
•Qualitative and quantitative study
•Remote and in-person
•“Start earlier than you think makes sense.” – Steve Krug
Page 50

51. STEVE KRUG STYLE
2.5 Weeks to Testing
•January 16 (Monday)
• Start planning what to test and with whom
• Create Screener
• Start Recruit
•January 23
• Determine tasks to test and create guide
• Finish recruiting by Jan. 31
•February 2
• 1st Usability Testing Day
•February 6th
• Begin planning for Feb. 16th UT Day
Page 51

52. STEVE KRUG (CONTINUED)
•3 – 5 users per day
• Recruit loosely and grade
on a curve
•Stakeholders watch tests –
spectator sport
•Debrief at end of study
•Quick analysis
Page 52

53. TWEAK, DON’T REDESIGN
•Small iterative changes
• Make it better now
• Don’t break something else
•Take something away
• Reduce distractions
• Don’t add – question it
Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems.
Page 53 By Steve Krug

54. GUERILLA STUDY
“Hallway Testing”
•Very quick studies
•Very early in the design process (paper or soon after)
•Recruiting is basic
• Not familiar with the project
• Ideally representative of user group
Page 54

55. RITE

56. R A P I D I T E R AT I V E T E S T I N G & E VA L U AT I O N
•RITE is qualitative, exploratory, and conceptual
•Creating conversation around elements
•RITE is not for:
• measuring performance
• validating detailed design decisions
Page 56

57. RITE OVERVIEW
•Most appropriate for:
• early requirements gathering
• design phase of production
•Initial wireframes used to:
• validate information architecture
• gather feedback on proposed functionality
Page 57

58. RITE OVERVIEW
Test Prioritize Update Test
1 High
2 Medium
3 Low
•Series of small usability tests
•Participants attempt tasks on concept
•Qualitative user feedback (actions + comments)
•New version tested with next day’s participants
Page 58

59. R I T E V S . S TA N D A R D U S A B I L I T Y T E S T I N G
Topic RITE Standard Usability Testing
Tested Concept Evolves Mostly static
Suitable stage Must be early Anytime
of project
Dynamics New concepts each day One concept tested
throughout
Feedback Different elements Same elements
throughout
Recommendati Discussed throughout Received in the final
ons report
Data Qualitative Qualitative and
quantitative
Stakeholder Crucial to have daily Preferred
involvement
Page 59

60. LATER THAT SAME DAY

61. D A I LY R E C A P S E S S I O N S
•After the last session
•30 minutes
•Room with a whiteboard or paper
•All decision makers MUST be present
• If not, wait for UX analysis
Page 61

62. GUIDELINES
•Stay on Topic
•Be Constructive
•Don’t get distracted by small problems
•Intense focus on fixing most serious problems first
Page 62

63. GOAL
•Identify top 5 or 10 most serious issues
• Top 3 from each list
• Prioritize from lists
• Make assignments for next piece of work
• Stop
Adapted from: Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing
Page 63
Usability Problems. By Steve Krug

64. DEBRIEF WITH TEAM
•Quick analysis to quick decisions
•Discuss:
• trends seen
• concerns
• recommendations
• prioritize changes for the next round
• list lower priority changes for future iterations
Page 64

65. MAKE USEFUL & USABLE
RECOMMENDATIONS -
QUICKLY

66. A N A LY S I S
Read “between the lines”
•Analysis up to ~1.5 x the facilitation time
•Identify repetition and patterns
•When found, continuation:
• Adds cost
• Delays reporting
• Low probability of many new findings
Page 66

67. T R A N S F O R M D ATA
•Know what you’ve got
• Sort, reorganize, review, repeat
• What refutes your expectations?
• Surprises?
• Outliers?
Page 67

68. DOCUMENT WITH LESS TREES
•Reduce, reuse
•Use email, IM, etc.
•Put it on the wall
• Must be easy to understand
• Quickly absorbable
Page 68

69. E M A I L O R O N E PA G E R
•Your Communication Goal:
• Think about audience
• How will it be used?
•Include:
• Study’s Goal
• Tasks attempted
• Who observed
• What planned to fix and assignment
• Future research/enhancements
Page 69

70. SHARE WHAT YOU LEARN
Page 70

71. MAKE USERS VISIBLE
Information radiators
• Test findings
• Artifacts
• Personas
Page 71

72. T R A N S F O R M D ATA
•Create useful findings and recommendations
• Show screenshots of issues
• Where possible show solutions They liked this!
•Think about audience
• How will it be used?
Make this go
away!
Page 72

73. REVIEW
•Plan out your goals
•Recruit participants
•Remote or in person?
•Look for patterns
•Easy-to-use findings and recommendations
Page 73

74. DO USABILITY TESTING
EARLY & OFTEN

75. RECOMMENDED READINGS
Page 75
7

76. C O N TA C T C A R O L
@carologic
Email: Carol.Smith@perficient.com
slideshare.net/carologic
and
slideshare.net/PerficientInc
speakerrate.com/speakers/15585-caroljsmith
Page 76

77. REFERENCES
•Albert, Bill, Tom Tullis, and Donna Tedesco. Beyond the Usability Lab.
•Gothelf , Jeff. http://blog.usabilla.com/5-effective-ways-for-usability-testing-to-
play-nice-with-agile/
•Henry, S.L. and Martinson, M. Evaluating for Accessibility, Usability Testing in
Diverse Situations. Tutorial, 2003 UPA Conference.
•Krug, Steve. Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding
and Fixing Usability Problems.
•Rubin, Jeffrey and Dana Chisnell. Handbook of Usability Testing: How to Plan,
Design, and Conduct Effective Tests. John Wiley & Sons, Inc.
Page 77

78. T O O L C O N S I D E R AT I O N S
•In-person or remote?
•Lab or on-site?
•Prototype limitations (can it be online?, is it a document or
a clickable site?)
•Number of observers, number of participants?
•Number of facilitators?
•Logging and video editing needs (time on task, highlight
video creation)?
•Surveys before or after?
•Eye tracking?
Page 78

79. U S A B I L I T Y T E S T I N G S O F T WA R E
•Morae
•Ovo
•SilverBack (Mac only)
•UserWorks
•Noldus
•Tobii (Eye-tracker)
•SMI (Eye-tracker)
•SurveyMonkey
Page 79

80. S C R E E N S H A R I N G S O F T WA R E
•GoToMeeting – http://www.gotomeeting.com
•Lotus Sametime Unyte – http://www.unyte.com
•YuuGuu -- http://www.yuuguu.com
•WebEx – http://www.webex.com
•Yugma -- https://www.yugma.com/
•Trouble Shooting: CoPilot - https://www.copilot.com/
Page 80

81. S AT I S FA C T I O N Q U E S T I O N N A I R E S
•Standard Usability Measurement Inventory (SUMI)
• office/desktop software, purchase
• 50 questions
•Website Analysis and Measurement Inventory (WHAMMI)
• Purchase
• 20 questions
•System Usability Scale (SUS)
• Free
• 10 questions
Page 81