Intro talk on lean unmoderated user testing given at General Assembly, Los Angeles in spring 2013. Covers basics, benefits & limitations, when to test, what to test, and a case study.

1. Jessica DuVerneay
Information Architect, The Understanding Group - @undrstndng
LEAN USER
TESTING

2. AGENDA
‣ 6:30 - Agenda & Intro
‣ 6:40 - User Testing Basics
‣ 6:50 - Benefits & Limitations
‣ 7:00 - The Lean Approach
‣ 7:10 - When to Test
‣ 7:20 - What to Test
‣ 7:30 - Case Study
‣ 7:40 - Resources & Next Steps
‣ 7:50 - Q&A
2

3. INTRODUCTION
‣ Clients: Herman Miller, Thompson Reuters /
Techstreet, Macy’s, Metro Health System, University
of Michigan, ASAPS, Cooper Tire
‣ Talks: Taxonomy - WIAD 2013, LAUX Meetup
‣ Workshops: IA/UX Topics including Wireframes,
Performance Continuums, User Testing
‣ Articles: Expert Contributor for UserTesting.com’s
blog, TUG’s site
‣ Global Producer - World Information Architecture
Day (2012)
JESSICA DUVERNEAY - INFORMATION ARCHITECT,
THE UNDERSTANDING GROUP (TUG)
3

4. INTRODUCTION
THE UNDERSTANDING GROUP - @UNDRSTNDNG
4

5. LEAN USER TESTING
USER TESTING BASICS
5

6. USER TESTING BASICS
USER TESTING
‣ Wikipedia says:
“Usability testing is a technique used in user-centered interaction design
to evaluate a product by testing it on users. This can be seen as an
irreplaceable usability practice, since it gives direct input on how real
users use the system.[1] This is in contrast with usability inspection
methods where experts use different methods to evaluate a user interface
without involving users. Usability testing focuses on measuring a human-
made product's capacity to meet its intended purpose. Examples of
products that commonly benefit from usability testing are foods, consumer
products, web sites or web applications, computer interfaces, documents,
and devices. Usability testing measures the usability, or ease of use, of a
specific object or set of objects.”
‣ It is NOT:
A complete set of User Research Activities, Market Research, Statistically
Relevant Research, A/B Testing, Qualitative Formal Research
6
PRO TIP:
Remember, User Testing is
not actually testing the
users - it’s testing the
product from the user’s
actual perspective.

7. USER TESTING BASICS 7
Activities
Deliverables
SME Interviews
Digital Report Card
Experience Map
Score Notes
Owned
Paid
Social
Mobile
Access
Search
B
A
C
B
C
B
ScenarioScenarioScenario
Alignment Session
Heuristic Analysis
Cross Channel Map
Pagename
PHASE 1
PHASE 2
PHASE 3
Channel Channel Channel
Tactics by Channel
Content Audit
Flow Diagrams
User Interviews
Scenario
Collab Design
Competitive
Site Map
You ThemThemFeatures Them
Impressions
1
2.1
2.2
4.1
4.2
4.3
3.1
2 3 4
User Co-Creation
Personas
Wireframes
Persona
Scenarios
2.0 Page Name
Strategic Brief
Content Inventory
Concept Testing
Stakeholder
Needs
Best
Practices
User Needs
b
a
c
d
2.0 Page Name
Functional Spec.
Analytic Report
Workshops
Recommendations
1.0
1.1
1.2
2.0
3.0
User Testing
Roadmap
Analytic Wishlist
1 2 3
Phased Roadmap
2
1
3
4
We wish we knew...

8. USER TESTING BASICS 8
The Elements of User Experience
A basic duality: The Web was originally conceived as a hypertextual information space;
but the development of increasingly sophisticated front- and back-end technologies has
fostered its use as a remote software interface. This dual nature has led to much confusion,
as user experience practitioners have attempted to adapt their terminology to cases beyond
the scope of its original application. The goal of this document is to define some of these
terms within their appropriate contexts, and to clarify the underlying relationships among
these various elements.
Jesse James Garrett
jjg@jjg.net
Visual Design: graphic treatment of interface
elements (the "look" in "look-and-feel")
Information Architecture: structural design
of the information space to facilitate
intuitive access to content
Interaction Design: development of
application flows to facilitate user tasks,
defining how the user interacts with
site functionality
Navigation Design: design of interface
elements to facilitate the user's movement
through the information architecture
Information Design: in the Tuftean sense:
designing the presentation of information
to facilitate understanding
Functional Specifications: "feature set":
detailed descriptions of functionality the site
must include in order to meet user needs
User Needs: externally derived goals
for the site; identified through user research,
ethno/techno/psychographics, etc.
Site Objectives: business, creative, or other
internally derived goals for the site
Content Requirements: definition of
content elements required in the site
in order to meet user needs
Interface Design: as in traditional HCI:
design of interface elements to facilitate
user interaction with functionality
Information Design: in the Tuftean sense:
designing the presentation of information
to facilitate understanding
Web as software interface Web as hypertext system
Visual Design: visual treatment of text,
graphic page elements and navigational
components
Concrete
Abstract
time
Conception
Completion
Functional
Specifications
Content
Requirements
Interaction
Design
Information
Architecture
Visual Design
Information Design
Interface Design Navigation Design
Site Objectives
User Needs
User Needs: externally derived goals
for the site; identified through user research,
ethno/techno/psychographics, etc.
Site Objectives: business, creative, or other
internally derived goals for the site
This picture is incomplete: The model outlined here does not account for secondary considerations (such as those arising during technical or content development)
that may influence decisions during user experience development. Also, this model does not describe a development process, nor does it define roles within a
user experience development team. Rather, it seeks to define the key considerations that go into the development of user experience on the Web today.
task-oriented information-oriented
30 March 2000
© 2000 Jesse James Garrett http://www.jjg.net/ia/

9. USER TESTING BASICS
MODERATED / SYNCHRONOUS TESTING (ONSITE & REMOTE)
9
User
Moderator

10. USER TESTING BASICS
UNMODERATED / ASYNCHRONOUS / AUTOMATED TESTING
10
User
Moderator

11. LEAN USER TESTING
UNMODERATED
BENEFITS &
LIMITATIONS
11

12. UNMODERATED BENEFITS & LIMITATIONS
BENEFITS
‣ Inexpensive, Fast
‣ Can learn high level pain points, random fatal
flaws, percentages of task completion, time on
task, qualitative verbal opinions offered, basic
analytics & demographic info.. more on that
later.
‣ Easy to aggregate concrete numbers if
needed in small or very large test populations
‣ “Quick and dirty”
‣ Can design to validate pretty much anything
‣ Existing tools = Easy to share across
stakeholders
‣ Easy to test a pre-determined flow
‣ Great for iterative design, repeatable
‣ Anonymous comparative analysis & A/B
testing
‣ Easy for one person to write, analyze and
extract findings
‣ Do not need to set up testing lab
‣ No moderator bias / influence
‣ User in natural environment using preferred
technologies & workflows
‣ Asynchronous allows for User Testing to be fit
in with other work responsibilities
‣ Non-local / Global Reach much easier than
moderated testing
12

13. UNMODERATED BENEFITS & LIMITATIONS
LIMITATIONS
‣ Not as empathy driven, less intuitive
‣ Can’t learn deep opinions or user motivations,
non-verbal cues, users values & personality,
organic / natural use case, deep analytics,
detailed statistics, eye mapping
‣ Not able to correct user
‣ Can not restate or correct poorly written test
on the fly
‣ Not able to dig in / probe for subjective issues
or edge cases
‣ Less non-verbal inference (eye tracking, facial
expressions, etc)
‣ Depending on recruits can be biased very
heavily
‣ Easy to misinterpret by external stakeholders
‣ Somewhat complicated to analyze by multiple
team members
‣ Does not build relationship with customer
‣ Does not completely assure confidentiality or
an NDA
13

14. LEAN USER TESTING
THE LEAN APPROACH
14

15. THE LEAN APPROACH
THE ‘MVP’ OF USER TESTING: FIT & FINISH TESTING
If you test nothing else, please test key product
segments & flows for fatal flaws & show
stoppers before you launch.
Common UX & IA Performance Metrics to
consider:
‣ Task Success (Binary Approach for Lean Process)
‣ Time On Task
‣ Errors
‣ Efficiency (Effort / Number of Steps)
‣ Learnability
‣ Perceived Severity of Identified Issues
15
OVERHEARD:
“Friends don’t let friends
launch without user
testing.”
True story.

16. THE LEAN APPROACH
HOW MANY USERS SHOULD WE TEST?
16
Image: Jakob Nielsen Group
Roughly, 5-6 users will
uncover a vast majority of the
issues per scenario, task, or
flow.
The exact number of tests
depends on the number of
scenarios and demographic
segments you need to test.

17. THE LEAN APPROACH 17
WHAT TOOLS SHOULD WE USE?

18. THE LEAN APPROACH 18

19. THE LEAN APPROACH
HOW TO TEST ‘LEAN’: OTHER CONSIDERATIONS
‣ Create a Testing Team (1-4 people)
‣ Create a Testing Plan (Strategy, Budget, Timeline)
‣ Know the Tools Available
‣ Write Effective Tests
‣ Be Cautious About Recruiting & Incentives
‣ Analyze Data Effectively
‣ Present Your Findings (If Needed) Strategically &
Clearly
‣ Iterate, Repeat, and Improve
19
PRO TIP:
A poorly created &
implemented test can be
worse than useless. Take
time to educate yourself &
plan the test.

20. LEAN USER TESTING
WHEN TO TEST
20

21. WHEN TO TEST
TYPICAL PROJECT LIFECYCLE
Discovery: Understanding organizational and business goals for the product or service. This is generally when
the “What & Why” of a product is determined.
User Research: Understanding end user needs and expectations for the same product or service. While the
Discovery Phase established the “What & Why”, this phase is to establish the “Who & Why”
Strategy & Structural Design: Creating a solid strategy & structure for the product or product
improvements. This phase is when the “What, Why & Who” are translated into the strategic and researched “How”.
Launch & Assessment: Building and launching the product, measuring and evaluating success. This is the
continuous improvement phase when efficacy of strategic changes or initial product launches can be measured, and
organizations can begin gathering ideas and creating roadmaps for further improvements or product releases.
21

22. WHEN TO TEST
LAUNCH & ASSESSMENT
‣ Validate visual fit & finish just prior to launch as a lightweight
QA process
‣ Validate improvement over previous product iterations
‣ Augment quantitative A/B testing to investigate user
preference from a qualitative perspective
‣ Identify unnoticed usability issues or awkward product flows
‣ Show instances of a product in multiple environments
(Desktop vs. Mobile)
‣ Provide an area for users to provide opinions, suggestions, and
desired features lists (especially valuable in the startup space)
22

23. WHEN TO TEST
DISCOVERY
‣ Establish successful existing user flows that should be
protected during upcoming versions
‣ Identify show stopping usability issues that may have been
missed in initial product road mapping activities
‣ Benchmark your existing product so that at the end of your
project lifecycle, you have data against which to compare your
improved product.
‣ Provide insight into the competitive landscape
23

24. WHEN TO TEST
USER RESEARCH
‣ Provide a quick and easy way to see a product in front of
multiple user types.
‣ Provide qualitative and quantitative data to flesh out user
research deliverables such as personas (scenarios, tasks,
demographic info, opinions & motivators).
‣ NOTE: Contrary to popular perception, this is actually the
least effective phase to do user testing. User Research, when
done correctly, is far more based on getting to know the user
and their needs, and less on how they would use the existing
product or service.
24

25. WHEN TO TEST
STRATEGY & STRUCTURAL DESIGN
‣ Point out key issues with Strategic or Structural Design
Direction prior to build using prototypes or sandbox
environments
‣ Test and refine complex user flows
‣ Test and refine navigation structures (TreeJack is a great tool
for this)
‣ Identify unanticipated usability issues that may come up after
strategy and structure is approved, during design and dev
phases
25

26. WHEN TO TEST
MAIN POINT
26
No matter where you are in your
product cycle, it is advantageous to
consider lean unmoderated testing.
Unmoderated testing is not just for
usability only - testing can add a
different value and data set at each
stage of the project lifecycle.
PRO TIP:
Test early, test often.

27. LEAN USER TESTING
WHAT TO TEST
27

28. WHAT TO TEST
1. UX / IA / USABILITY ISSUES
What constitutes success for one product may not necessarily
be indicative of optimal state for another product.
Qualitative data is valuable, but do not ignore the insights
extracted from qualitative information (opinions, ratings,
exclamations, and comments) - they can guide some of the
most effective product changes.
28
There are some basic measurable indicators to consider while usability
testing. (Note: The indicators listed below heavily reference Measuring the
User Experience by Tullis and Alberts)
‣ Task Completion: Can a user complete the task? To what percentage can
they move in the correct way in the task completion funnel? What issues
are impeding the completion of the task? Do users want to complete the
task in the first place?
‣ Time On Task: How long does it take the user to complete the task? Does
the time it takes to complete the task allow the user to lose interest or
focus? Does the task take too little time to complete?
‣ Errors: Did users make avoidable errors? Did users make unavoidable
errors? Were they able to recover from the errors? What would make the
errors less likely to occur or easier to recover from?
‣ Efficiency: Was the system efficient in effort and number of steps it took for
users to complete a task? Which steps were superfluous or confusing?
Which steps can be streamlined or eliminated? Which steps need to be
added?
‣ Learnability: For repeat visitors, is the system learnable? Does task
success and perceived efficiency increase? Does time on task and number
of errors decrease? Do certain key actions and flows become tacit?
‣ Perceived Severity of Identified Issues: Are any of the issues noticed by the
testing team not actually registered as issues to the users? Which issues
are most important to achievement of key business flows? Are any of the
issues deal breakers?

29. WHAT TO TEST
2. COMPETITIVE TESTING
A lesser known benefits of lean unmoderated testing
is the ability to show a test subject a competitor’s
product without the risk of moderator bias.
Learning from a competitor – which flows, content,
and design patterns are successful and which
should be avoided – can be particularly useful in the
startup space where lean UX teams may have to
make s product decisions on limited research.
QUESTIONS TO CONSIDER:
‣ Which product do users prefer? Why?
‣ Which product was easier to use? Why?
‣ What are some errors my competitor made that I
can avoid during product development?
‣ What are some successful features that my
competitor has that I might want to consider for
product roadmapping?
‣ How do users of different demographics (Age,
Gender, OS, income, etc) respond to each product?
Is one better for my target market?
29

30. WHAT TO TEST
3. PREFERENCE
Preference testing may provide direct user insights around:
‣ Visual design & Branding
‣ Interaction design
‣ Copy
‣ Navigation approaches
‣ Use of imagery
‣ Page layout and information hierarchy
‣ Any other contentious issues at your organization that are delaying launch.
30
PRO TIP:
Don’t mistake this for A/B
testing. While lean user
preference testing can give
insight, it’s not statistically
significant.

31. WHAT TO TEST
4. VALIDATION OF FIT & FINISH
‣ One to four main user flows
(keep this as simple as
possible)
‣ The main conversion point of
the system
‣ Overall opinions of the visual
design and interaction design
‣ Any crucial UX / IA issues as
previously described
31

32. WHAT TO TEST
5. OTHER USES: COPYWRITING VALIDATION,
LIGHTWEIGHT MULTI PLATFORM QA, PERSONA INFO
‣ Personas: I’ve augmented other user research activities with data I’ve collected
from unmoderated testing to flesh out personas as needed.
‣ QA Testing: I’ve seen unmoderated user testing implemented as a lightweight QA
process – running the same test on a product in multiple desktop and mobile
environments can provide invaluable insights for a small team with limited QA/
UAT capabilities.
‣ Copywriting & Language: Testing copywriting or taxonomies to some extent can
be done in unmoderated testing. Again, the success of these inquiries will depend
largely on the tools used & the manner in which the tests and scenarios are
crafted.
32

33. WHAT TO TEST
MAIN POINT: THE RIGHT TEST AT THE RIGHT TIME
33

34. LEAN USER TESTING
A CASE STUDY
34

35. SUMMARY OF ISSUES KEY QUESTIONS
CASE STUDY
How can we optimize our
eCommerce flow to mitigate
these issues?
What can we learn from Lean
User Testing and how can we
add it into our design cycle?
Well known e-commerce client facing:
• Navigation Issues
• Cart Abandonment
• Small Average Order Size (# of Products per
Conversion)
• Cross Channel Confusion of Digital Properties
(Information Site vs. Store)
• Browse Customers = Lower Conversion On Site
• Brand Perception Suboptimal Online vs. Physical
World Product Interaction
WELL KNOWN FURNITURE COMPANY
35

36. CASE STUDY
SUMMARY: UNMODERATED USABILITY RESEARCH APPROACH
36
PRO TIP:
In hindsight, I should have
tested the proposed
navigation with Treejack
during the Structural Design
Phase.

37. CASE STUDY
DISCOVERY: HIGH LEVEL USABILITY RESEARCH &
COMPETITIVE INSIGHT
‣ Why (Goal) 1: Identify “low hanging fruit” issues and areas for deeper
research
‣ Why (Goal) 2: Competitive Analysis
‣ Who: 18 Users, 3 Tests, 1 focused on competitor
‣ When: 3 weeks quick testing
‣ How: Unmoderated User Testing, UserTesting.com Recruits
‣ Resulted in 20 Quick Fixes and ~5 areas for significant focused future
research, structural design ideas from competitive research
37

38. CASE STUDY
USER RESEARCH: SPECIFIC FEATURE & PAGE
USABILITY RESEARCH
Lightweight Approach:
• 12 users, 2 Tests, Captured through Ethn.io Screener
• Incentive = $75 AMEX Gift Card
• Done in conjunction with Moderated Tests, Existing Personas, and Analytics Work
Multiple Hypotheses Tested Flows & Features:
• Cart Abandonment Motivation
• Wishlist Use
• Listing Page Efficacy
• Feature Value to User
• Check Out Flow
• Also Gathered Qualitative User Opinions
Time = ~1 month
• 1 week to plan
• 1 week to implement
• 1 week to Analyze
38

39. CASE STUDY
RECOMMENDATIONS & OUTCOMES
‣ Multiple UX “Quick Fixes” were launched during the longer project life
cycle
‣ Improved Navigation, Checkout Flow, Product Detail Page, Strategic
Use of Visual Assets
‣ Road map of features & functionality (Faceted Search Results,
Improved Integration of Multiple Sites)
‣ The Preliminary Numbers:
- 16% Increase in order volume
- Organic Traffic increased by 53% (due to improved SEO performance due to navigation changes)
- 84% increase in page views, 28% decline in bounce rate from landing pages
39

40. CASE STUDY
UNMODERATED USABILITY RESEARCH HELPED
‣ Provide Insight into User Behaviors & Expectations
‣ Ensure a Pervasive Customer Focus through Design Cycles
‣ Identify Initial Problem Set
‣ Point out easily implemented immediate “Quick Fixes”
‣ Narrow & Focus Future Research and Development Efforts
‣ Craft a Strategic User Research Phase
‣ Guide Successful Site Changes
‣ Establish IA / UX Road Mapping & Priority for Next Project Iterations
‣ ... All on relatively lean time & money budgets
40

41. LEAN USER TESTING
RESOURCES
& NEXT STEPS
41

42. RESOURCES 42

43. RESOURCES 43
‣ When to Test: Incorporating User
Testing Into Product Design
‣ Lessons From an Expert: What to
Test
‣ How to Test: 9 Steps for Creating
the Perfect User Test
USERTESTING.COM BLOG

44. RESOURCES 44
HTTP://REMOTERESEAR.CH/TOOLS/

45. NEXT STEPS 45
PRO TIP:
5 Free Test Credits for
participants in this class -
use the code
NO CODE AVAILABLE
at usertesting.com
for 5 free tests
TRY IT OUT

46. NEXT STEPS 46
• can do it for you
• can train you to do it yourselves
• can do it with you
SHAMELESS SELF PROMOTION

47. Q&A
LEAN USER TESTING 47
Jessica DuVerneay - Information Architect
The Understanding Group (TUG)
http://www.understandinggroup.com
Twitter: @undrstndng
THANK
YOU!