Slides for my presentation at the IASummit 2012 on crowdsourced remote unmoderated usability testing.

1. Crowdsourced
Remote
Unmoderated
Usability Testing
And how to make it work for you
Inge De Bleecker
@ingedebleecker
IASummit 2012

2. Overview
o What is it
o How to do it
o Get started today (or Monday)

3. What is it
Remote
Unmoderated
Crowdsourced

4. Crowdsourcing
“The act of taking a job traditionally
performed by a designated agent (usually
an employee) and outsourcing it to an
undefined, generally large group of
people in the form of an open call.”
— JEFF HOWE

5. Crowdsourcing
“a model for enabling convenient, on-
demand access to a (shared) pool of
human workers with different skills that
can be rapidly provisioned and released
with minimal management effort”
— PANOS IPEIROTIS

6. Different Flavors including
o Screenshot click test
o Screenshot timed test
o Task-based usability study with
online survey
D
I
Y

7. Good for
o Use of personal devices
o In own environment
o Fast turnaround
o Cheap(er)

8. Not good for
o Can’t observe participants
o No additional questions
o Subject video recording

9. Process
Still the same process
1 Recruit participants Need committed participants!
2 Write task plan Not easy to write a good task plan
3 Run pilot
4 Run test
5 Analyze results
6 Write report

10. Committed participants
Compensa/on
Loyalty
Commitment
Higher
Longer
Higher
comple/on
sessions
quality
rate

11. Tips for writing a task plan
o Participants can’t get blocked
o You only get one shot
o Guide without influencing
o Make all questions required
o Encourage think aloud/write down
o Task and question types depend on
tool

12. Tasks & Questions
o Screenshot click test (@usabilityhub)

13. Tasks & Questions
o Screenshot timed test (@usabilityhub)

14. Tasks & Questions
o Online survey testing (Loop11)

15. Tasks & Questions
o Online survey testing (DIY)

16. Tasks & Questions
o Online survey testing (DIY)

17. Results
Self-reported data
“I find grid views easier to scan because they show
enough detail of several phones on one screen
without forcing me to scroll”
“I would select the "Compare this phone" option
below the phones I wanted. But then I am at a loss
as to what to hit next to actually carry out the
comparison”

18. Getting started
Got money!
o Engage crowdsourcing company
o Use specialized tool
Broke?
o Make your own crowd
o Use generic survey tool (e.g. Surveymonkey,
Google docs)

19. Q&A
Thank you