Given at UXPA-DC's User Focus Conference, Oct. 19, 2012

1. The Use of an Unmoderated,
Online Usability Testing
Service to Test a Website
Brandon Kopp & Bill Mockovak
Research Psychologist
Office of Survey Methods Research
UXPA-DC User Focus Conference
19 October 2012

2. TryMyUI
• Provides panel of participants that are presented with user
defined tasks
• You receive a video screenshot of the participant’s
computer with an audio voiceover of their comments
2

3. Many Alternatives
Userzoom.com
UserTesting.com
UTest.com
OptimalWorkshop.com
OpenHallway.com Loop11.com

4. The BLS.gov Dropdown Usability Test
• During earlier
usability
testing, several
users had
problems with
immediately
activating
dropdown menus
• Users would
accidentally click
menu links and
get lost

5. Prototypes
• Test design alternatives for mega-dropdowns
Type of Menu
Prototype Display Width List of Subject Areas
Control
Partial list of content, with
A Click to show/close Fixed, 3 columns
‘view all’ option
B Hover, no delay Fixed, 1 column Compact
C Hover, with delay Fixed, 3 columns Current content
D Hover, no delay Fixed, 3 columns Current content

6. Prototype A
• Need to click on
tab to open
• Click on X or
Arrow to close
• Abbreviated list
is initially
presented
• User clicks View
All to see entire
list

7. Prototype B
• Dragging mouse
over menu will
open it
• Menu closes
when you move
mouse off it (no
clicking is
necessary)
• Higher-level
categories are
displayed

8. Prototype C
• Current, ‘full’
list of topics
• Menu drops
after delay, with
mouseover
• Moving off
menu makes it
disappear
quickly

9. Prototype D
• Control Condition
• Current menu on
BLS.gov
• Menu drops
immediately (no
delay) with
mouseover
• Moving off menu
makes it disappear
quickly

10. Method
• Define Participant Criteria Prototype
# of
Participants
Gender: Any Age: 18-55 A 11
Country: U.S. Income: Any B 14
Education: Any Employment: Any C 10
Computer Experience: Beginner-Intermediate or Expert D 10
• Participants complete up to 10 tasks while
providing verbal feedback

11. Tasks
1. Find a publication called the Occupational
Outlook Quarterly. This is an online magazine
about jobs and careers.

12. Tasks
1. Occupational Outlook Quarterly
2. Mass Layoffs
3. International Unemployment
4. Green Jobs
5. Survey of Occupational Illness and Injury
6. Strikes and Lockouts
7. Students and Teachers
8. NY State Wages

13. What the Participant Sees
Task
Instructions
Browser /
Webpage View
Recording
Time
14

14. What The Experimenter Sees
Video of Task Index
Participant’s Screen

15. Example Videos
Feedback (A; Task 8)
Feedback (D; Task 6)

16. Task Success and Time To Complete
# of Tasks Successfully Completed Time to Complete Task
8 120
7.4
7 6.6 6.7
Time to Complete Task (in sec.)
96.9
# of Tasks Completed (out of 8)
6.2 100
87.8
6 83.8
77.8
80
5
4 60
3
40
2
20
1
0 0
A B C D A B C D

17. First Click
Subject Databases Economic Elsewhere
Task Home Publications
Areas & Tools Release on page
1 0 3 3 0 32 7
2 3 17 16 7 0 2
3 3 26 9 6 0 1
4 2 33 4 2 0 2
5 1 34 6 1 1 0
6 3 38 3 1 0 0
7 4 25 2 1 4 7
8 1 25 9 1 0 6

18. Method
• Define Participant Criteria Prototype
# of
Participants
Gender: Any Age: 18-55 A 11
Country: U.S. Income: Any B 14
Education: Any Employment: Any C 10
Computer Experience: Beginner-Intermediate or Expert D 10
• Participants complete up to 10 tasks while
providing verbal feedback
• Following testing, participants write responses
to 4 open-ended questions

19. Open-Ended Questions
• How easy were these tasks to complete? Were they very
easy, easy, neither easy nor difficult, difficult, or very difficult? How easy
do you think these tasks would be for an average American citizen using
this website?
80
% of Participants Selecting Rating
70 67 69
61
60
50 47
39
40 33
30 23
20
20 17 17
10 8
0
0
Easy/Very Easy Neither Easy Nor Difficult Difficult/Very Difficult
A B C D

20. Feedback/Recommendations
• Final Recommendation:
Prototype C
• Several participants were
surprised that the Home tab
had a menu
• No items under Workplace
Injuries subcategory on
Subject Areas menu

21. Feedback/Recommendations
• Subject Areas menu extended off the screen
on some participants’ browsers requiring
scrolling

22. Advantages
• Cost ($)
Method Cost
Traditional, In-Lab $40 per participant
TryMyUI.com $35 per participant
UserTesting.com $39 per participant
UTest.com --
OpenHallway.com $49-$199 per month
Loop11.com $350 per project

23. Advantages
• Cost ($)
• Cost (Time)
Task In-Lab Web
20 minutes total; 30 minutes total;
Requesting participants explaining criteria to recruiter specifying test groups and criteria
0 minutes;
Screening participants 10 minutes per participant
done by TryMyUI
0 minutes;
Scheduling participants 15 minutes per participant
study done at participant convenience
60 minutes total;
Preparing for interviews 10 minutes per participant
setting up web survey and tasks
Total (for 45 interviews) 26.6 hours 1.5 hours
3 weeks; 1 – 2 days;
Data collection period based on interviewer schedule Depends on participant criteria

24. Advantages
• Cost ($)
• Cost (Time)
• Short data collection period
• Participants that can be selected based on criteria
• Videos can be shared
• Participants are skilled/trained
– At thinking aloud
– At dealing with problems
– At evaluating websites
• Can get replacements for unusable cases

25. Disadvantages
• No follow-up
• Cannot correct navigation errors
• Limited to 20 minutes*
• Task completion time, success, and other
quantitative methods have to be captured
manually*
• No rating scales*
• Panel participants may have selection biases that
make them different from a ‘typical’ user
* May be different on other testing services

26. Discussion
• What testing service have you used?
• What are the advantages/disadvantages of
that service? Loop11.com
UTest.com
Userzoom.com
UserTesting.com
TryMyUI.com
OpenHallway.com OptimalWorkshop.com

27. Contact Information
Brandon Kopp
Research Psychologist
Office of Survey Methods Research
www.bls.gov/osmr
202-691-7514
kopp.brandon@bls.gov