1. Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014
Lect 13: Theory &
Practice of Usability
Studies
User Experience Design III
John Chin, Ph.D.
jchin@acm.org
973-746-7438
October, 2014
Contact Information
2. Copyright T. Komischke, H. Strub, T. Sachs, 2014
An broad overview of context
- Who, Anyone
- When, Anytime
- Where, Anywhere
- Why, Motivations
- What, Task
The missing link - How
Along with some depth on many topics
2
3. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Doctor Who? Who is John Chin?
Education
Ph.D. in Cognitive Psychology specializing in
Human Factors
Human-Computer Interaction
University of Maryland, College Park (advisor: Kent Norman)
Work Experience
Currently, User Experience Strategist @ Verizon Wireless
User research, Writing UI requirements, Usability testing,
Accessibility, Connected Car, Messaging, 3 patents pending
HiTech – Contractor Homeland Security, TSA
Cognetics – Trizetto Group
Avaya, Alcatel-Lucent, AT&T Bell Labs, Telcordia
Lewis and Clark College
3
4. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Enough about me, what about you?
Who are you?
Education and Work experience
Interests and specialty
Why are you here?
Goals and Objectives
Career development, job?
How can I help?
Needs and Aspirations
Networking
4
5. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Bottom line goals…
1) Be neutral and stick to the facts
2) Advocate for your users, all of them
Recruit representative sample of eventual product/system users
Participants – users testing the design/implementation
Audience – observers (possible development team members)
Report findings and possible solutions to readers or stakeholders
Gain better insights and understanding of the issues
Hope to find possible solutions
Few hard and fast answers: mainly tradeoffs
5
6. Planning a Test
Who: Low vision or Blind user
Where: Standing on the sidewalk at a bus stop
When: During rush hour in the morning
What: Pedestrian getting directions to library
Why: Pick up model created on 3D printer at library
How: Mobile device navigation application
6
7. What are the research questions
Research questions are not tasks
Key questions drive focus of the testing and tasks
Research:
Can people successfully order products?
Will customers be able to pay their credit card bill?
Will radiologists be able to diagnose cases, without errors, efficiently?
Will consumers like this better than their current camcorder?
Tasks:
Typically tasks should be frequent and/or important
How long will it take customers to find the due date for their current
statement?
7
8. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Where is project in the product life
cycle?
Early?
Are there requirements that need to be gathered?
Or are requirements finalized?
What’s the shape of design concepts?
Is this the next version or a new product?
Some will say “too early for usability” since much is undefined
Some will say “Do a (user-centered) focus group instead”
Not yet 100% clear what it does
Bottom line: A lot is vague, testing can have big impact, but make
sure you’re exploring relevant issues
Middle of process?
Requirements finalized
Screens rough, or designed for usability
Lots already determined: Use it. [or not, at your own risk]
Bottom line: Testing can have moderate impact, Fast turnaround
important. Choose issues where you’ll be listened to.
8
9. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Where is the project in the product life
cycle? (continued…)
Late?
UI is likely designed
Possibly even coded
No leeway on requirements (functionality)
Delivery possibly even scheduled
What is the goal?
If the team will update the UI, could have a meaningful impact
Fitting some request “It needs usability”
No-win if in “police” role
Stakeholder may be looking for “lipstick on the pig”, and endorsement that
“Usability says it’s okay”
If it misses the mark, you’re in a no-win situation
Big risk: once it’s delivered, product manager might not want to change UI once
existing customers invested time to learn it.
Bottom line: be diplomatic & realistic. 1) Consider risks to company
if don’t fix enough, 2) what can be fixed later, and 3) process so won’t
repeat this mess next release or product.
9
10. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Where is the project in the product life
cycle?
I prefer early-middle, or early
Beware being brought in late: a no-win game
Especially if project planned without user research.
No resources to create prototype (and no time)
No resources to incorporate results (or time)
No one might care
Even if you have a sponsor, you’re being set up for a battle
Focus on your data. It’s a business decision re what to do.
Be careful – think high level
I’ve been surprised many times – people and organizations that seem to
have mature processes
“organizations” – since companies have silos: one or many organizations
in a company may be awesome. Be careful every time you meet a new
one.
10
11. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
1) The human (in general)
Topics include:
Cognitive processor (and memory)
Perceptual systems
Social interaction (
Emotion
Other topics: Persuasion, attraction, etc.
If you have time to read, get applied books on these topics
For topics you really like, get theoretical books; read research
papers.
If you don’t already have a good book list, I’ll have suggestions next
week
11
12. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
2) Your business:
If a consultancy (or an experienced department)
How does your firm approach problems?
Why do customers choose it?
Know previous reports – especially with your client
Know how to leverage templates: research plan, test documents, reports
Practically: consider ways to promote future work for colleagues – both your team
and other teams [without compromising my integrity]
12
13. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
2) Your business
If you’re an employee:
Get to know others in complementary departments
Market Research (make friends, which also reduces risk of being seen as competition)
Customer Service (Call Center “top 10” lists)
Meet, go to lunch, read reports
VOC (Voice of the Customer) data
Becoming increasingly popular
Whatever you have available.
ForeSee:
Survey at end of visits
13
14. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
2) Your business, if you’re an employee, continued
VOC data, continued
OpinionLab
Input possible on any page
Works best with meaningful URLs
Beware:
Length. Tempting to continue asking questions, but not paying customers
Order matters: What goes early late, which questions go before others
14
15. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
2) Your business, if you’re an employee, continued
Analytics (i.e., big data)
15
Copyright M. Tremaine, T. Sachs, A. Melewski 2012
This is a sample, not an Adobe endorsement
16. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
2) Your business, if you’re an employee, continued
Analytics (i.e., big data)
The ability to track…
Beware: What the data makes easy, versus what is of interest
I have been able to track interactions on a page (sometimes every page), but cannot track a
user through paths
Who are your company’s customers? Who use online systems?
How do people access your stuff?
What are trends?
Social networking analysis: As many as company follows
Reports: JD Power, Forrester, Keynote, etc.
Data collection services, and reports, (usually not cheap)
Technology trends
The continuing rise of mobile, tablets
The impact of OS, browsers, etc.
Zeitgeist: e.g., Responsive Web Design
16
17. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What will you test with? (1 of 3)
Paper prototype?
Cheap, usually easy to create, conceptual
Who will create?
Will it be respected internally?
“Photoshop” [i.e., online] paper prototype more “real” than paper
Paper does communicate “conceptual”, room for change
I usually test paper as concepts that extend prototype(s)
17
18. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What will you test with? (2 of 3)
Visio? (or similar, PowerPoint can be fast)
I’m not a “designer” but I can design fairly well in Visio
If my concepts, I’m in control: can test what I want (or edit myself)
If my concepts, distracting from focusing on testing
Key point: Hyperlinks, so it looks like it’s functioning
More about screens than functionality (won’t function well, i.e., fields
cannot be filled in)
Something that really runs?: Html, iRise, Axure, Dreamweaver,
Ruby, etc.
Functionality is great addition to realism, believability
18
19. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What will you test with? (3 of 3)
Possibly bad addition to cost, time, and pain, since someone has to
build & support the prototype.
Arranging for someone to create the prototype can be hard, takes time.
Don’t assume getting resources will be fast/easy.
For many audiences, many products/systems, this is a necessity
Even more important for mobile UIs – I believe form factor really
matters for mobile
What do you think about where mobile is tested? [lab vs. field]
Trend of hybrid device approach: Big addition to complexity
Prototype web, tablet, phone?
iOS ? (pre-7 & 7?), Android (“flavor”?), Windows mobile, Blackberry?
App versus mobile web?
19
20. Copyright T. Komischke, H. Strub, T. Sachs, 2014
How will you record the test?
Screen critical – see what happened
Audio critical – hear what happened,
High quality audio often undervalued.
Stereo audio recordings can be especially useful, since with headphones
you can pick out voices
Very useful: The participant’s face, to see emotions
Still useful – hands on keyboard, on artifacts, etc.
Mobile technology especially challenging to record
Harder to get good screen image
Want to see what hands do, but hands/fingers cover the screen
Still want to see face of the participant
20
21. Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many participants to run? (1 of 3)
Nielsen’s rules of thumb:
5 or a few more. Go higher if test has truly different groups
Consider more rounds of testing, if prototyping and time are
available
I often have trouble getting prototyping for multiple rounds of
testing)
Bump up numbers a little if only one round of testing
21
22. Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many participants to run? (3 of 3)
Hank’s rules of thumb
Arranging a usability test is hard
Unless really simple, run at least 3 days
60 minute sessions: 15 participants
90 minute sessions: 12 participants
Beware going > 4 days at a crack – exhausting, and too much to analyze
Too much for industry people to pay attention to
If need more people, run another series
Or outsource the usability
22
23. Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many participants to run? (3 of 3)
How many customer “demographic groups” to recruit
Rule of thumb: shoot for at least 3 people per demographic
If differing opinions, still have a tie-breaker
If a no-show, still have 2
If it’s a key customer group (i.e., “prospects”) and site is for them,
increase or subdivide that group
No point in testing 12 groups for study with 15 participants
23
24. Copyright T. Komischke, H. Strub, T. Sachs, 2014
How much time per session?
Guesstimate: How big/complex does the UI feel?
If testing in cafeteria (or some place public) keep sessions really
short
To cover more ground, lots of sessions covering different tasks
In lab, I believe no shorter than an hour
Lots of overhead, effort, to recruit, to prep, to get observers, to analyze
I find it hard to go > 1-1/2 hours
Sessions are tiring, for participant, moderator, observers
Hard for many types of participants to get lots of time during business
hours – might get more strange people
> 90 minutes, schedule in a break (which lowers efficiency)
I’ve seen 2 and 2-1/2 hour sessions (and no break) in last year – and they worked,
but moderator knew domain well
24
25. Copyright T. Komischke, H. Strub, T. Sachs, 2014
When to schedule sessions?
Easiest: during business hours
If live observers, much better to have straight days & business hours
But will you get enough of “right” people?
Who are target participants? What are their days like?
Many jobs: work day
Physicians: One early (before 9) one at lunch, 1-2 starting at 5 pm
Think about yourself. I rarely schedule before 10 in Manhattan since
hard to get there and lots to get lab ready.
If won’t have live observers, can run any time, including a few per
day at odd hours
For some groups, consider weekends
But if you have family or other personal issues (like you have a life), get
more money to pay participants
25
26. Copyright T. Komischke, H. Strub, T. Sachs, 2014
When not to schedule sessions
Consider observers as well as participants
Holiday season is hard
If winter, have backup plan
If big storm might arrive during test, have backup plans
Allow time for analysis, and for results to be incorporated
Often have designers responding before discussion.
26
27. Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many sessions to run per day?
Don’t make testing days too hard on anyone (including the
moderator)
Want to be as alert, as good of a moderator for first and last session
Breaks between every session gives flexibility for buggy prototypes,
late participants, rebooting computer, trying something new, etc
60 minute sessions: up to 5 per day
Breaks of at least 30 minutes per session (even if just to allow for lateness), 60-90
minutes for lunch
90 minute sessions: up to 4 per day
These are more tiring, make sure you have break time
Schedule lightly on first day of testing, to allow updating of everything –
even your discussion guide, and to plan around unexpecteds
When possible, run 1 or 2 participants first day, after lunch
27
28. Copyright T. Komischke, H. Strub, T. Sachs, 2014
How to recruit participants? (1 of 2)
What are you testing? How specialized the community? What’s your
budget? How much time do you have?
Many options, all involve tradeoffs
Professional recruiters: For many types of consumers, and some target
audiences (like MDs)
Most say they’ll recruit anybody for any reason
Realistically, they have specialties
Most have “lists” of known people, who may be semi-professional test takers
(some seem jaded)
Recruiters aren’t cheap
Lists of customers, you contact yourself
If you want to test improvements to Emerson’s HMI, want people who know today’s
Emerson HMI
Beware: lists can take a LONG time to get (6 weeks or more)
Referrals from sales staff, or from technicians who know their “cool” and
“boring” customers
Subscriber lists, Professional Society members (especially valuable if want
special expertise)
28
29. Copyright T. Komischke, H. Strub, T. Sachs, 2014
How to recruit participants? (2 of 2)
Recruiting from your web site
But any issue having “customer information”?
Waiting room at medical clinic
Your card is given out by someone trustworthy (like a doctor)
“Mechanical Turk” at Amazon
Craig’s List, or similar ads
Your friends/acquaintances probably not diverse enough.
Beware effects of participants feeling obligated
Children can be especially challenging
You want parents to be nearby so everyone is trusting, but you have to
keep them occupied
29
30. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Who to recruit
If clear that there’s one demographic (or “persona”), go for it
But consider going for sub-demographics as more participants
I usually shoot for up to 3, maybe 4 max, demographic groups.
Product space may be much broader
Target groups that are truly different, in parameters that matter to your UI
Many factors can be targeted independently, with same participants. E.g.:
Ranges for age, education, income
Separate from experience in what you’re testing
Avoid trap of recruiting only the young/well-educated
They look nice on video, have great things to say, but…
Unless all users will be young and well-educated
I learn a lot from diversity
30
31. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Who to recruit: Common screening
criteria (1 of 2)
No one in your industry, or with a household member in your industry
No one in design or usability
Always ask for a range, within whatever criteria.
If open to unemployed or retired, they’re probably easier to find for
“business hours” studies than people who work, so “up to 2”
Internet skill: Want people who use web themselves, often, familiar
Age: cover diversity of your user community
I’m pushing for older but not above 69
Many older users, but tend to talk more, and therefore get less done (in my
experience)
I’ve had most dishonesty with elders: age (not be too old), internet skills
Consider gender distribution
31
32. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Who to recruit: Common screening
criteria (2 of 2)
Education: I’ve known researchers who screen for minimum of some
college for general consumer products.
Getting some without college is more honest test of usability, for those with
low literacy skills – who buy and use things
I’m often fascinated with responses of those with least education
Ethnic diversity
Income: As low as appropriate. If you want some truly wealthy
people, might need to specially recruit and pay more
But can ask for people whose speech is easy to understand (critical)
Where they live: City vs. suburb; and which city (Philly vs. NY)
32
33. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Special participants: Children, Disabled,
Employees
Children:
Want a witness, at all times
Parent should be nearby
Make it fun
Recruit so you’ll get interested kids
Disabled
Hard enough to get disabled with full cognitive skill
Check literature if recruiting disabled without full cognitive skill
Employees
Is it their choice? Coerced?
Impact of losing time on job
Fear if criticize system, or people, when it’s recorded
33
34. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Possible Forms for participants (1 of 2)
Informed consent form has two goals:
Inform participants of what they’ll be doing, realistically, any risks (even of
embarrassment), and benefits (often, interesting tasks; chance to help
change a product; and get paid)
Consent: So it’s clear what they’re consenting to
Media Release:
That you are recording them
How recordings will be used (for study, nothing else)
Can protect you as well as participants: if colleague loves a clip- wants to
show at a conference (“Sorry!”)
If session is that good, can go back and pay more, or recreate the scene
with actors
34
35. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Possible Forms for participants (2 of 2)
NDA (Non-disclosure Agreement)
If you’re showing proprietary concepts
Explain NDA too – so understand
I’ve heard that a properly written and executed NDA protects even if
people talk – you did your due diligence
Liability release (?)
In case incidentally hurt when coming to your facility
Include conditions that lawyers want
But fight for plain language
Long forms are harder to understand –consider more short forms
than fewer long forms
35
36. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Don’t fight the law (but then I’m married
to a lawyer)
This is probably all new for most corporate council
Draft pretty good sample forms before approaching
Shows you know something
Easier to edit than start from scratch
Be able to explain what’s there, why, who and what your intentions
are
Provide articles on what you’re after, or books
Negotiate, but with the company’s best interests in mind
36
37. Copyright T. Komischke, H. Strub, T. Sachs, 2014
How much to pay participants?
Goldilocks rule of thumb:
Not too much, not to little
If doing internal testing on employees, might be limited by corporate
rules (participants already missing work during work time
For external participants, ask your recruiter for advice
Consider adding in reasonable extra payment for
transportation/parking costs
BTW, if paying a participant more than $500 in a year, check with
company HR about possible need to report “salary” to IRS
Only an issue if have same person return
37
38. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Should I use floaters?
Floaters are backup participants, usually recruited to cover two
sessions
Extremely valuable in case of no-show
Also, for when a participant is totally inappropriate
But, who has time to hang out for 2 sessions? Be careful
I use best recruiter I can, request to go for 100% turnout
I’ve had odd birds as floaters
38
39. Copyright T. Komischke, H. Strub, T. Sachs, 2014
How to record a test?
Morae, by Techsmith: The “Honda Fit” of usability recording:
$1500 for a full license
Records two streams, usually screen and webcam on face, as well as an
audio stream
If built-in mic isn’t awesome, get a good external mic
1copy of Morae includes 1 copy of “Observer”, which lets people in
another room watch live (delayed ~20 seconds)
Very reliable
Not bad to learn, to set up
Can take notes, and make marks, live (but I don’t see that done)
Will make highlights videos
Pretty efficient storage, so many sessions can go on a DVD
39
40. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Recording a test – More on Morae
Morae also good for portable lab – only need 2 laptops
Requirements for computers not clear: doesn’t reflect multi-core
processors
In my experience, Morae has run fine on machines well below the
“minimum recommended” capability
A hack to get 3 images is a hardware video mixer, that will
superimpose for you.
40
41. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Recording a test: Lots of fancier
software, but..
Pricier, need fancier computer and equipment, can be testier
I know many big companies use OvoStudios
My personal experience: very nice & helpful, not as reliable as I’d
like
But I’ve heard of companies that find it trustworthy
Network to find something that works for your needs.
Consider:
HD looks great, awesome detail
Probably not needed for routine recordings of basic web sites
41
42. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Recording a test: Back to mobile
How to get a good screen? Apple to HDMI to DVI cable, or Apple TV
hack if security allows
License supports 3 images – I think all 3 are important
[Note: target area – ask participants to keep the phone THERE
42
43. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Other Technology Issues:
Eye Tracking: compelling to watch, help observers stay awake
Beware of heat maps: If interrupt participants, etc.
Magic is uninterrupted, enough people to get good trends, so software can
analyze what happened
Web site also needs to support eye tracking: overlays, modals mess things
up
Software only knows the URL, not what’s on screen
Emotion tracking: GSR (how emotional) and valence (+/-)
Affectiva is a company to watch
43
44. Copyright T. Komischke, H. Strub, T. Sachs, 2014
The Moderator’s Guide (1 of 4)
Introduction
Why you’re there
That this is usability so
A) Please speak aloud (we’ll practice),
B) Testing the UI, not testing you,
C) Don’t worry about errors since prototype (not real data or system)
Ask questions (though I might not answer)
Can stop at any time, for any reason
Pay at beginning of session: Show trust (people don’t walk away)
Plan the task list
Based on the research questions
Ideally, based on your knowledge:
What are the true top tasks (consider what Gartner, JD Power say)
44
45. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Moderator’s Guide (2 of 4)
Think about time. Consider optional tasks to fill sessions for fast
participants.
Or have an extra task or two, and prioritize list in pilot sessions
Moderator’s guide: By each task I like to have questions to probe,
and suggested things to observe
Participant’s guide just has the tasks – other stuff is deleted
Edit moderator’s guide, either regenerate Participant’s guide, or be careful
on version control.
Questions to ask:
Per task: Make sure to ask the goal of the task
Find a value? Perform a task
Otherwise, think about what you’ll learn
Can ask questions like “How quickly do you expect to do this task? How easy do you expect
this task to be?
After: How easy was it to perform this task? How often do you perform this task?
45
46. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Moderator’s Guide (3 of 4)
Questions to ask: At end of Session
I like to ask:
What did you like best about the prototype system?
What did you like least about the prototype system?
[Sometimes in form of “What were 3 best & 3 worst things?”]
The SUS is trendy [SUS = System Usability Scale]
Original by John Brooke, 1986, DEC
46
48. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Moderator’s Guide (4 of 4)
Thoughts on SUS:
Alternates positive and negative questions, so respondent needs to be
awake
Around since the 80’s, lots of experience in the industry
A few common updates [i.e., “website” instead of “system”]
Like school grading: 70 is okay, 80 good, 90 superb
But maybe UI’s are better these days -> I’ve seen mainly upper 80’s or
better
Net Promoter Score (NPS): How likely are you to recommend this
product/company to a friend?
Trendy, originally published in Harvard B-School Journal
Were participants supposed to learn something?
Though be careful about testing learning in 1-hour long session, unless
that’s what system is for
1 hr not much time to learn, especially when many tasks
48
49. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Cautions on locations (1 of 2)
Dedicated lab:
Will you use it enough? Perhaps try lending it out, but beware…
Sufficient staff support? [reception & waiting area, space for observers who
have meetings, snacks & drinks and meals]
Window: Great, but then keep observer room quiet and dark
Need to keep equipment secure
Conference rooms and a portable Morae lab
Still have to keep laptops secure
Will you always get good rooms for test when you need them?
Others’ office
Conference rooms: Will your Morae computers run on their network? If not,
how will they communicate?
Fairly safe from interruptions, though.
Cubes: Beware interruptions… Phone will ring & cube’nik might need to
pick up; Co-workers don’t recognize you as important
Offices: Probably okay as long as participant respects you
49
50. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Cautions on locations (2 of 2)
Remote Testing
Success depends on installing web-sharing software, but software,
computers and bandwidth are all pretty good
Usually only 5 minutes, sometimes much worse
Better with web cam
Surprisingly to me, rarely interrupted, but phone calls seem to be
respected
3rd party facility (i.e., market research company)
Surprisingly pricey – be prepared for unexpected costs
But they handle the details – REALLY nice
When I use a facility, I have them recruit – so one point of failure
Even professionals sometimes use cafeteria, or Starbucks
Best for quick “first impression” studies, lots of people in little time
Can recruit for more in-depth study (follow-up in lab…)
50
51. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Screen-sharing software, for remote live
observers
Easiest to use the corporate standard
It’s IT-endorsed
No firewall issues
Account likely easy to set up, and/or use
If there’s a problem, you used what company wanted
Otherwise, I believe in WebEx & a telephone bridge
Usually reasonably pain-free to install
Performance (delay, quality, frame-rate) pretty good
Not cheapest, but trustworthy is important to me
51
53. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Be like Johnny Carson
(Used to host The Tonight Show)
Always genial, friendly
Show was about the guest(s), not him
THEIR thoughts & opinions
Johnny didn’t lead
But he got guests to talk
When there was a problem, he took the blame (never the guest’s fault
unless something terrible)
Was incredible at improvising
For more on Johnny, watch for rebroadcast of PBS documentary
53
54. Copyright T. Komischke, H. Strub, T. Sachs, 2014
Taking notes during:
If you follow template strictly and have time, excel template for all
key questions and observation points is neat
Time consuming to set up
Not convenient to use when with participant
Can take notes with computer
Not best for establishing rapport
Take notes at a constant pace (lots of them)
So don’t communicate what you care about
So writing when participant thinks they’ve said something important
Compatriot’s notes (ideally, not in sight of participant) can be great,
but everyone looks for different things
Who will be lead on the report?
54
56. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What to do if: (1 of 2)
Participant is no-show
Ask to arrive 15 minutes early: time for forms, for bathroom, and to call
recruiter before session time
Call recruiter – check on reminders
Backup slots available?
If remote, can’t get to internet
Common to get weird firewall issues
Have a local copy
Prototype fails
Computer fails
Participant arrives quite late
Running really behind. Participant’s a talker, trouble, started late
Participant wants bathroom break, but already behind
56
57. Copyright T. Komischke, H. Strub, T. Sachs, 2014
What to do if: (2 of 2)
Early in session, can see that participant doesn’t fit the study?
It appears that participant lied to recruiter re something key? (After
all, these studies pay well and economy is bad)
You give cash or check to participant at start of session, and he/she
gets up and leaves?
Participant admits to having committed a crime on your video
recording? (“When I do my taxes, I don’t put in all the receipts”)
You have strong evidence that participant is victim of or commits
abuse, or is considering suicide?
57
58. Copyright T. Komischke, H. Strub, T. Sachs, 2014
After a test
Debriefs are great. Consider recording them if you don’t take good
notes.
Write down first impressions of results while they’re fresh in your
mind.
Get invoices in promptly: * participants *, recruiter, and any vendor
you might use again
Return borrowed equipment promptly (before damaged, lost)
Listen to sessions as “background music”. You will hear things
Push self for rapid turn-around.
Consider rehash/highlights meeting immediately afterwards, recorded, to
get impressions of observers
Tight deadline for report doesn’t allow forgetting
But beware getting slammed if rest of your job fell behind due to time away for
testing
58
Alternatives to SUS are SUMI and QUIS. Some versions of SUS make all responses positive to avoid respondent confusion.
There is a formula for converting SUS scores into an equivalent NPS score.