SlideShare uma empresa Scribd logo
1 de 59
Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014
Lect 13: Theory &
Practice of Usability
Studies
User Experience Design III
John Chin, Ph.D.
jchin@acm.org
973-746-7438
October, 2014
Contact Information
Copyright T. Komischke, H. Strub, T. Sachs, 2014
An broad overview of context
- Who, Anyone
- When, Anytime
- Where, Anywhere
- Why, Motivations
- What, Task
The missing link - How
 Along with some depth on many topics
2
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Doctor Who? Who is John Chin?
Education
 Ph.D. in Cognitive Psychology specializing in
 Human Factors
 Human-Computer Interaction
 University of Maryland, College Park (advisor: Kent Norman)
Work Experience
 Currently, User Experience Strategist @ Verizon Wireless
 User research, Writing UI requirements, Usability testing,
 Accessibility, Connected Car, Messaging, 3 patents pending
 HiTech – Contractor Homeland Security, TSA
 Cognetics – Trizetto Group
 Avaya, Alcatel-Lucent, AT&T Bell Labs, Telcordia
 Lewis and Clark College
3
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Enough about me, what about you?
Who are you?
 Education and Work experience
 Interests and specialty
Why are you here?
 Goals and Objectives
 Career development, job?
How can I help?
 Needs and Aspirations
 Networking
4
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Bottom line goals…
 1) Be neutral and stick to the facts
 2) Advocate for your users, all of them
 Recruit representative sample of eventual product/system users
 Participants – users testing the design/implementation
 Audience – observers (possible development team members)
 Report findings and possible solutions to readers or stakeholders
 Gain better insights and understanding of the issues
 Hope to find possible solutions
 Few hard and fast answers: mainly tradeoffs
5
Planning a Test
 Who: Low vision or Blind user
 Where: Standing on the sidewalk at a bus stop
 When: During rush hour in the morning
 What: Pedestrian getting directions to library
 Why: Pick up model created on 3D printer at library
 How: Mobile device navigation application
6
What are the research questions
 Research questions are not tasks
 Key questions drive focus of the testing and tasks
 Research:
 Can people successfully order products?
 Will customers be able to pay their credit card bill?
 Will radiologists be able to diagnose cases, without errors, efficiently?
 Will consumers like this better than their current camcorder?
 Tasks:
 Typically tasks should be frequent and/or important
 How long will it take customers to find the due date for their current
statement?
7
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Where is project in the product life
cycle?
 Early?
 Are there requirements that need to be gathered?
 Or are requirements finalized?
 What’s the shape of design concepts?
 Is this the next version or a new product?
 Some will say “too early for usability” since much is undefined
 Some will say “Do a (user-centered) focus group instead”
 Not yet 100% clear what it does
 Bottom line: A lot is vague, testing can have big impact, but make
sure you’re exploring relevant issues
 Middle of process?
 Requirements finalized
 Screens rough, or designed for usability
 Lots already determined: Use it. [or not, at your own risk]
 Bottom line: Testing can have moderate impact, Fast turnaround
important. Choose issues where you’ll be listened to.
8
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Where is the project in the product life
cycle? (continued…)
 Late?
 UI is likely designed
 Possibly even coded
 No leeway on requirements (functionality)
 Delivery possibly even scheduled
 What is the goal?
 If the team will update the UI, could have a meaningful impact
 Fitting some request “It needs usability”
 No-win if in “police” role
 Stakeholder may be looking for “lipstick on the pig”, and endorsement that
“Usability says it’s okay”
 If it misses the mark, you’re in a no-win situation
 Big risk: once it’s delivered, product manager might not want to change UI once
existing customers invested time to learn it.
 Bottom line: be diplomatic & realistic. 1) Consider risks to company
if don’t fix enough, 2) what can be fixed later, and 3) process so won’t
repeat this mess next release or product.
9
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Where is the project in the product life
cycle?
 I prefer early-middle, or early
 Beware being brought in late: a no-win game
 Especially if project planned without user research.
 No resources to create prototype (and no time)
 No resources to incorporate results (or time)
 No one might care
 Even if you have a sponsor, you’re being set up for a battle
 Focus on your data. It’s a business decision re what to do.
 Be careful – think high level
 I’ve been surprised many times – people and organizations that seem to
have mature processes
 “organizations” – since companies have silos: one or many organizations
in a company may be awesome. Be careful every time you meet a new
one.
10
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 1) The human (in general)
 Topics include:
 Cognitive processor (and memory)
 Perceptual systems
 Social interaction (
 Emotion
 Other topics: Persuasion, attraction, etc.
 If you have time to read, get applied books on these topics
 For topics you really like, get theoretical books; read research
papers.
 If you don’t already have a good book list, I’ll have suggestions next
week
11
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 2) Your business:
 If a consultancy (or an experienced department)
 How does your firm approach problems?
 Why do customers choose it?
 Know previous reports – especially with your client
 Know how to leverage templates: research plan, test documents, reports
 Practically: consider ways to promote future work for colleagues – both your team
and other teams [without compromising my integrity]
12
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 2) Your business
 If you’re an employee:
 Get to know others in complementary departments
 Market Research (make friends, which also reduces risk of being seen as competition)
 Customer Service (Call Center “top 10” lists)
 Meet, go to lunch, read reports
 VOC (Voice of the Customer) data
 Becoming increasingly popular
 Whatever you have available.
 ForeSee:
Survey at end of visits
13
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 2) Your business, if you’re an employee, continued
 VOC data, continued
 OpinionLab
 Input possible on any page
 Works best with meaningful URLs
 Beware:
 Length. Tempting to continue asking questions, but not paying customers
 Order matters: What goes early late, which questions go before others
14
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 2) Your business, if you’re an employee, continued
 Analytics (i.e., big data)
15
Copyright M. Tremaine, T. Sachs, A. Melewski 2012
This is a sample, not an Adobe endorsement
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What do you know?
 2) Your business, if you’re an employee, continued
 Analytics (i.e., big data)
 The ability to track…
 Beware: What the data makes easy, versus what is of interest
 I have been able to track interactions on a page (sometimes every page), but cannot track a
user through paths
 Who are your company’s customers? Who use online systems?
 How do people access your stuff?
 What are trends?
 Social networking analysis: As many as company follows
 Reports: JD Power, Forrester, Keynote, etc.
 Data collection services, and reports, (usually not cheap)
 Technology trends
 The continuing rise of mobile, tablets
 The impact of OS, browsers, etc.
 Zeitgeist: e.g., Responsive Web Design
16
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What will you test with? (1 of 3)
 Paper prototype?
 Cheap, usually easy to create, conceptual
 Who will create?
 Will it be respected internally?
 “Photoshop” [i.e., online] paper prototype more “real” than paper
 Paper does communicate “conceptual”, room for change
 I usually test paper as concepts that extend prototype(s)
17
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What will you test with? (2 of 3)
 Visio? (or similar, PowerPoint can be fast)
 I’m not a “designer” but I can design fairly well in Visio
 If my concepts, I’m in control: can test what I want (or edit myself)
 If my concepts, distracting from focusing on testing
 Key point: Hyperlinks, so it looks like it’s functioning
 More about screens than functionality (won’t function well, i.e., fields
cannot be filled in)
 Something that really runs?: Html, iRise, Axure, Dreamweaver,
Ruby, etc.
 Functionality is great addition to realism, believability
18
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What will you test with? (3 of 3)
 Possibly bad addition to cost, time, and pain, since someone has to
build & support the prototype.
 Arranging for someone to create the prototype can be hard, takes time.
Don’t assume getting resources will be fast/easy.
 For many audiences, many products/systems, this is a necessity
 Even more important for mobile UIs – I believe form factor really
matters for mobile
 What do you think about where mobile is tested? [lab vs. field]
 Trend of hybrid device approach: Big addition to complexity
 Prototype web, tablet, phone?
 iOS ? (pre-7 & 7?), Android (“flavor”?), Windows mobile, Blackberry?
 App versus mobile web?
19
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How will you record the test?
 Screen critical – see what happened
 Audio critical – hear what happened,
 High quality audio often undervalued.
 Stereo audio recordings can be especially useful, since with headphones
you can pick out voices
 Very useful: The participant’s face, to see emotions
 Still useful – hands on keyboard, on artifacts, etc.
 Mobile technology especially challenging to record
 Harder to get good screen image
 Want to see what hands do, but hands/fingers cover the screen
 Still want to see face of the participant
20
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many participants to run? (1 of 3)
 Nielsen’s rules of thumb:
 5 or a few more. Go higher if test has truly different groups
 Consider more rounds of testing, if prototyping and time are
available
 I often have trouble getting prototyping for multiple rounds of
testing)
 Bump up numbers a little if only one round of testing
21
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many participants to run? (3 of 3)
 Hank’s rules of thumb
 Arranging a usability test is hard
 Unless really simple, run at least 3 days
 60 minute sessions: 15 participants
 90 minute sessions: 12 participants
 Beware going > 4 days at a crack – exhausting, and too much to analyze
 Too much for industry people to pay attention to
 If need more people, run another series
 Or outsource the usability
22
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many participants to run? (3 of 3)
 How many customer “demographic groups” to recruit
 Rule of thumb: shoot for at least 3 people per demographic
 If differing opinions, still have a tie-breaker
 If a no-show, still have 2
 If it’s a key customer group (i.e., “prospects”) and site is for them,
increase or subdivide that group
 No point in testing 12 groups for study with 15 participants
23
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How much time per session?
 Guesstimate: How big/complex does the UI feel?
 If testing in cafeteria (or some place public) keep sessions really
short
 To cover more ground, lots of sessions covering different tasks
 In lab, I believe no shorter than an hour
 Lots of overhead, effort, to recruit, to prep, to get observers, to analyze
 I find it hard to go > 1-1/2 hours
 Sessions are tiring, for participant, moderator, observers
 Hard for many types of participants to get lots of time during business
hours – might get more strange people
 > 90 minutes, schedule in a break (which lowers efficiency)
 I’ve seen 2 and 2-1/2 hour sessions (and no break) in last year – and they worked,
but moderator knew domain well
24
Copyright T. Komischke, H. Strub, T. Sachs, 2014
When to schedule sessions?
 Easiest: during business hours
 If live observers, much better to have straight days & business hours
 But will you get enough of “right” people?
 Who are target participants? What are their days like?
 Many jobs: work day
 Physicians: One early (before 9) one at lunch, 1-2 starting at 5 pm
 Think about yourself. I rarely schedule before 10 in Manhattan since
hard to get there and lots to get lab ready.
 If won’t have live observers, can run any time, including a few per
day at odd hours
 For some groups, consider weekends
 But if you have family or other personal issues (like you have a life), get
more money to pay participants
25
Copyright T. Komischke, H. Strub, T. Sachs, 2014
When not to schedule sessions
 Consider observers as well as participants
 Holiday season is hard
 If winter, have backup plan
 If big storm might arrive during test, have backup plans
 Allow time for analysis, and for results to be incorporated
 Often have designers responding before discussion.
26
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How many sessions to run per day?
 Don’t make testing days too hard on anyone (including the
moderator)
 Want to be as alert, as good of a moderator for first and last session
 Breaks between every session gives flexibility for buggy prototypes,
late participants, rebooting computer, trying something new, etc
 60 minute sessions: up to 5 per day
 Breaks of at least 30 minutes per session (even if just to allow for lateness), 60-90
minutes for lunch
 90 minute sessions: up to 4 per day
 These are more tiring, make sure you have break time
 Schedule lightly on first day of testing, to allow updating of everything –
even your discussion guide, and to plan around unexpecteds
 When possible, run 1 or 2 participants first day, after lunch
27
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How to recruit participants? (1 of 2)
 What are you testing? How specialized the community? What’s your
budget? How much time do you have?
 Many options, all involve tradeoffs
 Professional recruiters: For many types of consumers, and some target
audiences (like MDs)
 Most say they’ll recruit anybody for any reason
 Realistically, they have specialties
 Most have “lists” of known people, who may be semi-professional test takers
(some seem jaded)
 Recruiters aren’t cheap
 Lists of customers, you contact yourself
 If you want to test improvements to Emerson’s HMI, want people who know today’s
Emerson HMI
 Beware: lists can take a LONG time to get (6 weeks or more)
 Referrals from sales staff, or from technicians who know their “cool” and
“boring” customers
 Subscriber lists, Professional Society members (especially valuable if want
special expertise)
28
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How to recruit participants? (2 of 2)
 Recruiting from your web site
 But any issue having “customer information”?
 Waiting room at medical clinic
 Your card is given out by someone trustworthy (like a doctor)
 “Mechanical Turk” at Amazon
 Craig’s List, or similar ads
 Your friends/acquaintances probably not diverse enough.
 Beware effects of participants feeling obligated
 Children can be especially challenging
 You want parents to be nearby so everyone is trusting, but you have to
keep them occupied
29
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Who to recruit
 If clear that there’s one demographic (or “persona”), go for it
 But consider going for sub-demographics as more participants
 I usually shoot for up to 3, maybe 4 max, demographic groups.
 Product space may be much broader
 Target groups that are truly different, in parameters that matter to your UI
 Many factors can be targeted independently, with same participants. E.g.:
 Ranges for age, education, income
 Separate from experience in what you’re testing
 Avoid trap of recruiting only the young/well-educated
 They look nice on video, have great things to say, but…
 Unless all users will be young and well-educated
 I learn a lot from diversity
30
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Who to recruit: Common screening
criteria (1 of 2)
 No one in your industry, or with a household member in your industry
 No one in design or usability
 Always ask for a range, within whatever criteria.
 If open to unemployed or retired, they’re probably easier to find for
“business hours” studies than people who work, so “up to 2”
 Internet skill: Want people who use web themselves, often, familiar
 Age: cover diversity of your user community
 I’m pushing for older but not above 69
 Many older users, but tend to talk more, and therefore get less done (in my
experience)
 I’ve had most dishonesty with elders: age (not be too old), internet skills
 Consider gender distribution
31
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Who to recruit: Common screening
criteria (2 of 2)
 Education: I’ve known researchers who screen for minimum of some
college for general consumer products.
 Getting some without college is more honest test of usability, for those with
low literacy skills – who buy and use things
 I’m often fascinated with responses of those with least education
 Ethnic diversity
 Income: As low as appropriate. If you want some truly wealthy
people, might need to specially recruit and pay more
 But can ask for people whose speech is easy to understand (critical)
 Where they live: City vs. suburb; and which city (Philly vs. NY)
32
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Special participants: Children, Disabled,
Employees
 Children:
 Want a witness, at all times
 Parent should be nearby
 Make it fun
 Recruit so you’ll get interested kids
 Disabled
 Hard enough to get disabled with full cognitive skill
 Check literature if recruiting disabled without full cognitive skill
 Employees
 Is it their choice? Coerced?
 Impact of losing time on job
 Fear if criticize system, or people, when it’s recorded
33
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Possible Forms for participants (1 of 2)
 Informed consent form has two goals:
 Inform participants of what they’ll be doing, realistically, any risks (even of
embarrassment), and benefits (often, interesting tasks; chance to help
change a product; and get paid)
 Consent: So it’s clear what they’re consenting to
 Media Release:
 That you are recording them
 How recordings will be used (for study, nothing else)
 Can protect you as well as participants: if colleague loves a clip- wants to
show at a conference (“Sorry!”)
 If session is that good, can go back and pay more, or recreate the scene
with actors
34
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Possible Forms for participants (2 of 2)
 NDA (Non-disclosure Agreement)
 If you’re showing proprietary concepts
 Explain NDA too – so understand
 I’ve heard that a properly written and executed NDA protects even if
people talk – you did your due diligence
 Liability release (?)
 In case incidentally hurt when coming to your facility
 Include conditions that lawyers want
 But fight for plain language
 Long forms are harder to understand –consider more short forms
than fewer long forms
35
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Don’t fight the law (but then I’m married
to a lawyer)
 This is probably all new for most corporate council
 Draft pretty good sample forms before approaching
 Shows you know something
 Easier to edit than start from scratch
 Be able to explain what’s there, why, who and what your intentions
are
 Provide articles on what you’re after, or books
 Negotiate, but with the company’s best interests in mind
36
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How much to pay participants?
 Goldilocks rule of thumb:
 Not too much, not to little
 If doing internal testing on employees, might be limited by corporate
rules (participants already missing work during work time
 For external participants, ask your recruiter for advice
 Consider adding in reasonable extra payment for
transportation/parking costs
 BTW, if paying a participant more than $500 in a year, check with
company HR about possible need to report “salary” to IRS
 Only an issue if have same person return
37
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Should I use floaters?
 Floaters are backup participants, usually recruited to cover two
sessions
 Extremely valuable in case of no-show
 Also, for when a participant is totally inappropriate
 But, who has time to hang out for 2 sessions? Be careful
 I use best recruiter I can, request to go for 100% turnout
 I’ve had odd birds as floaters
38
Copyright T. Komischke, H. Strub, T. Sachs, 2014
How to record a test?
 Morae, by Techsmith: The “Honda Fit” of usability recording:
 $1500 for a full license
 Records two streams, usually screen and webcam on face, as well as an
audio stream
 If built-in mic isn’t awesome, get a good external mic
 1copy of Morae includes 1 copy of “Observer”, which lets people in
another room watch live (delayed ~20 seconds)
 Very reliable
 Not bad to learn, to set up
 Can take notes, and make marks, live (but I don’t see that done)
 Will make highlights videos
 Pretty efficient storage, so many sessions can go on a DVD
39
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Recording a test – More on Morae
 Morae also good for portable lab – only need 2 laptops
 Requirements for computers not clear: doesn’t reflect multi-core
processors
 In my experience, Morae has run fine on machines well below the
“minimum recommended” capability
 A hack to get 3 images is a hardware video mixer, that will
superimpose for you.
40
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Recording a test: Lots of fancier
software, but..
 Pricier, need fancier computer and equipment, can be testier
 I know many big companies use OvoStudios
 My personal experience: very nice & helpful, not as reliable as I’d
like
 But I’ve heard of companies that find it trustworthy
 Network to find something that works for your needs.
 Consider:
 HD looks great, awesome detail
 Probably not needed for routine recordings of basic web sites
41
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Recording a test: Back to mobile
 How to get a good screen? Apple to HDMI to DVI cable, or Apple TV
hack if security allows
 License supports 3 images – I think all 3 are important
 [Note: target area – ask participants to keep the phone THERE
42
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Other Technology Issues:
 Eye Tracking: compelling to watch, help observers stay awake
 Beware of heat maps: If interrupt participants, etc.
 Magic is uninterrupted, enough people to get good trends, so software can
analyze what happened
 Web site also needs to support eye tracking: overlays, modals mess things
up
 Software only knows the URL, not what’s on screen
 Emotion tracking: GSR (how emotional) and valence (+/-)
 Affectiva is a company to watch
43
Copyright T. Komischke, H. Strub, T. Sachs, 2014
The Moderator’s Guide (1 of 4)
 Introduction
 Why you’re there
 That this is usability so
 A) Please speak aloud (we’ll practice),
 B) Testing the UI, not testing you,
 C) Don’t worry about errors since prototype (not real data or system)
 Ask questions (though I might not answer)
 Can stop at any time, for any reason
 Pay at beginning of session: Show trust (people don’t walk away)
 Plan the task list
 Based on the research questions
 Ideally, based on your knowledge:
 What are the true top tasks (consider what Gartner, JD Power say)
44
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Moderator’s Guide (2 of 4)
 Think about time. Consider optional tasks to fill sessions for fast
participants.
 Or have an extra task or two, and prioritize list in pilot sessions
 Moderator’s guide: By each task I like to have questions to probe,
and suggested things to observe
 Participant’s guide just has the tasks – other stuff is deleted
 Edit moderator’s guide, either regenerate Participant’s guide, or be careful
on version control.
 Questions to ask:
 Per task: Make sure to ask the goal of the task
 Find a value? Perform a task
 Otherwise, think about what you’ll learn
 Can ask questions like “How quickly do you expect to do this task? How easy do you expect
this task to be?
 After: How easy was it to perform this task? How often do you perform this task?
45
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Moderator’s Guide (3 of 4)
 Questions to ask: At end of Session
 I like to ask:
 What did you like best about the prototype system?
 What did you like least about the prototype system?
 [Sometimes in form of “What were 3 best & 3 worst things?”]
 The SUS is trendy [SUS = System Usability Scale]
 Original by John Brooke, 1986, DEC
46
Copyright T. Komischke, H. Strub, T. Sachs, 2014
47
The original SUS
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Moderator’s Guide (4 of 4)
 Thoughts on SUS:
 Alternates positive and negative questions, so respondent needs to be
awake
 Around since the 80’s, lots of experience in the industry
 A few common updates [i.e., “website” instead of “system”]
 Like school grading: 70 is okay, 80 good, 90 superb
 But maybe UI’s are better these days -> I’ve seen mainly upper 80’s or
better
 Net Promoter Score (NPS): How likely are you to recommend this
product/company to a friend?
 Trendy, originally published in Harvard B-School Journal
 Were participants supposed to learn something?
 Though be careful about testing learning in 1-hour long session, unless
that’s what system is for
 1 hr not much time to learn, especially when many tasks
48
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Cautions on locations (1 of 2)
 Dedicated lab:
 Will you use it enough? Perhaps try lending it out, but beware…
 Sufficient staff support? [reception & waiting area, space for observers who
have meetings, snacks & drinks and meals]
 Window: Great, but then keep observer room quiet and dark
 Need to keep equipment secure
 Conference rooms and a portable Morae lab
 Still have to keep laptops secure
 Will you always get good rooms for test when you need them?
 Others’ office
 Conference rooms: Will your Morae computers run on their network? If not,
how will they communicate?
 Fairly safe from interruptions, though.
 Cubes: Beware interruptions… Phone will ring & cube’nik might need to
pick up; Co-workers don’t recognize you as important
 Offices: Probably okay as long as participant respects you
49
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Cautions on locations (2 of 2)
 Remote Testing
 Success depends on installing web-sharing software, but software,
computers and bandwidth are all pretty good
 Usually only 5 minutes, sometimes much worse
 Better with web cam
 Surprisingly to me, rarely interrupted, but phone calls seem to be
respected
 3rd party facility (i.e., market research company)
 Surprisingly pricey – be prepared for unexpected costs
 But they handle the details – REALLY nice
 When I use a facility, I have them recruit – so one point of failure
 Even professionals sometimes use cafeteria, or Starbucks
 Best for quick “first impression” studies, lots of people in little time
 Can recruit for more in-depth study (follow-up in lab…)
50
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Screen-sharing software, for remote live
observers
 Easiest to use the corporate standard
 It’s IT-endorsed
 No firewall issues
 Account likely easy to set up, and/or use
 If there’s a problem, you used what company wanted
 Otherwise, I believe in WebEx & a telephone bridge
 Usually reasonably pain-free to install
 Performance (delay, quality, frame-rate) pretty good
 Not cheapest, but trustworthy is important to me
51
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Running a Test52
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Be like Johnny Carson
(Used to host The Tonight Show)
 Always genial, friendly
 Show was about the guest(s), not him
 THEIR thoughts & opinions
 Johnny didn’t lead
 But he got guests to talk
 When there was a problem, he took the blame (never the guest’s fault
unless something terrible)
 Was incredible at improvising
 For more on Johnny, watch for rebroadcast of PBS documentary
53
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Taking notes during:
 If you follow template strictly and have time, excel template for all
key questions and observation points is neat
 Time consuming to set up
 Not convenient to use when with participant
 Can take notes with computer
 Not best for establishing rapport
 Take notes at a constant pace (lots of them)
 So don’t communicate what you care about
 So writing when participant thinks they’ve said something important
 Compatriot’s notes (ideally, not in sight of participant) can be great,
but everyone looks for different things
 Who will be lead on the report?
54
Copyright T. Komischke, H. Strub, T. Sachs, 2014
Things to watch during session55
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What to do if: (1 of 2)
 Participant is no-show
 Ask to arrive 15 minutes early: time for forms, for bathroom, and to call
recruiter before session time
 Call recruiter – check on reminders
 Backup slots available?
 If remote, can’t get to internet
 Common to get weird firewall issues
 Have a local copy
 Prototype fails
 Computer fails
 Participant arrives quite late
 Running really behind. Participant’s a talker, trouble, started late
 Participant wants bathroom break, but already behind
56
Copyright T. Komischke, H. Strub, T. Sachs, 2014
What to do if: (2 of 2)
 Early in session, can see that participant doesn’t fit the study?
 It appears that participant lied to recruiter re something key? (After
all, these studies pay well and economy is bad)
 You give cash or check to participant at start of session, and he/she
gets up and leaves?
 Participant admits to having committed a crime on your video
recording? (“When I do my taxes, I don’t put in all the receipts”)
 You have strong evidence that participant is victim of or commits
abuse, or is considering suicide?
57
Copyright T. Komischke, H. Strub, T. Sachs, 2014
After a test
 Debriefs are great. Consider recording them if you don’t take good
notes.
 Write down first impressions of results while they’re fresh in your
mind.
 Get invoices in promptly: * participants *, recruiter, and any vendor
you might use again
 Return borrowed equipment promptly (before damaged, lost)
 Listen to sessions as “background music”. You will hear things
 Push self for rapid turn-around.
 Consider rehash/highlights meeting immediately afterwards, recorded, to
get impressions of observers
 Tight deadline for report doesn’t allow forgetting
 But beware getting slammed if rest of your job fell behind due to time away for
testing
58
Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014
Questions?

Mais conteúdo relacionado

Mais procurados

Seeing the Forest for the Trees: Visual Problem Solving as a Design and Usabi...
Seeing the Forest for the Trees: Visual Problem Solving as a Design and Usabi...Seeing the Forest for the Trees: Visual Problem Solving as a Design and Usabi...
Seeing the Forest for the Trees: Visual Problem Solving as a Design and Usabi...Christina York
 
Watson DevCon 2016 - Engage and Be Engaging: Building Compassionate and Perso...
Watson DevCon 2016 - Engage and Be Engaging: Building Compassionate and Perso...Watson DevCon 2016 - Engage and Be Engaging: Building Compassionate and Perso...
Watson DevCon 2016 - Engage and Be Engaging: Building Compassionate and Perso...IBM Watson
 
Choosing the Right Research Methods for Your Project (webinar)
Choosing the Right Research Methods for Your Project (webinar)Choosing the Right Research Methods for Your Project (webinar)
Choosing the Right Research Methods for Your Project (webinar)Susan Mercer
 
Faster Usability Testing in an Agile World presented at Agile2011
Faster Usability Testing in an Agile World presented at Agile2011Faster Usability Testing in an Agile World presented at Agile2011
Faster Usability Testing in an Agile World presented at Agile2011Carol Smith
 
What your employees need to learn to work with data in the 21 st century
What your employees need to learn to work with data in the 21 st century What your employees need to learn to work with data in the 21 st century
What your employees need to learn to work with data in the 21 st century Human Capital Media
 
Watson DevCon 2016 - From Jeopardy! to the Future
Watson DevCon 2016 - From Jeopardy! to the FutureWatson DevCon 2016 - From Jeopardy! to the Future
Watson DevCon 2016 - From Jeopardy! to the FutureIBM Watson
 
Brightfind world usability day 2016 full deck final
Brightfind world usability day 2016   full deck finalBrightfind world usability day 2016   full deck final
Brightfind world usability day 2016 full deck finalBrightfind
 
HR Technology In the Era of Drones, Robots, and Infinite Data
HR Technology In the Era of Drones, Robots, and Infinite DataHR Technology In the Era of Drones, Robots, and Infinite Data
HR Technology In the Era of Drones, Robots, and Infinite DataeCornell
 
BAQMaR - Conference Evening
BAQMaR - Conference EveningBAQMaR - Conference Evening
BAQMaR - Conference EveningBAQMaR
 
SIMS Quantitative Course Lecture 1
SIMS Quantitative Course Lecture 1SIMS Quantitative Course Lecture 1
SIMS Quantitative Course Lecture 1Rashmi Sinha
 
Test your bot_x_conf
Test your bot_x_confTest your bot_x_conf
Test your bot_x_confShama Ugale
 
Watson - a supercomputer
Watson - a supercomputerWatson - a supercomputer
Watson - a supercomputerHarshil Darji
 
Bits of Evidence
Bits of EvidenceBits of Evidence
Bits of EvidenceGreg Wilson
 
10 tips for a better UX survey
10 tips for a better UX survey10 tips for a better UX survey
10 tips for a better UX surveyCaroline Jarrett
 
Communicating data: Reporting user research
Communicating data: Reporting user researchCommunicating data: Reporting user research
Communicating data: Reporting user researchPuja Parakh
 
Beyond usability testing: Getting a holistic view of the user experience
Beyond usability testing: Getting a holistic view of the user experienceBeyond usability testing: Getting a holistic view of the user experience
Beyond usability testing: Getting a holistic view of the user experienceAmanda Nance
 
Choosing Technical Interview Questions (2006)
Choosing Technical Interview Questions (2006)Choosing Technical Interview Questions (2006)
Choosing Technical Interview Questions (2006)Adam Barr
 
The State of Creative Technology - 2019
The State of Creative Technology - 2019The State of Creative Technology - 2019
The State of Creative Technology - 2019TBD Labs
 
Bundledarrows160 bit.ly/teamcaptainsguild
Bundledarrows160 bit.ly/teamcaptainsguildBundledarrows160 bit.ly/teamcaptainsguild
Bundledarrows160 bit.ly/teamcaptainsguildshadowboxingtv
 

Mais procurados (20)

Seeing the Forest for the Trees: Visual Problem Solving as a Design and Usabi...
Seeing the Forest for the Trees: Visual Problem Solving as a Design and Usabi...Seeing the Forest for the Trees: Visual Problem Solving as a Design and Usabi...
Seeing the Forest for the Trees: Visual Problem Solving as a Design and Usabi...
 
Watson DevCon 2016 - Engage and Be Engaging: Building Compassionate and Perso...
Watson DevCon 2016 - Engage and Be Engaging: Building Compassionate and Perso...Watson DevCon 2016 - Engage and Be Engaging: Building Compassionate and Perso...
Watson DevCon 2016 - Engage and Be Engaging: Building Compassionate and Perso...
 
Choosing the Right Research Methods for Your Project (webinar)
Choosing the Right Research Methods for Your Project (webinar)Choosing the Right Research Methods for Your Project (webinar)
Choosing the Right Research Methods for Your Project (webinar)
 
Faster Usability Testing in an Agile World presented at Agile2011
Faster Usability Testing in an Agile World presented at Agile2011Faster Usability Testing in an Agile World presented at Agile2011
Faster Usability Testing in an Agile World presented at Agile2011
 
What your employees need to learn to work with data in the 21 st century
What your employees need to learn to work with data in the 21 st century What your employees need to learn to work with data in the 21 st century
What your employees need to learn to work with data in the 21 st century
 
Watson DevCon 2016 - From Jeopardy! to the Future
Watson DevCon 2016 - From Jeopardy! to the FutureWatson DevCon 2016 - From Jeopardy! to the Future
Watson DevCon 2016 - From Jeopardy! to the Future
 
Brightfind world usability day 2016 full deck final
Brightfind world usability day 2016   full deck finalBrightfind world usability day 2016   full deck final
Brightfind world usability day 2016 full deck final
 
HR Technology In the Era of Drones, Robots, and Infinite Data
HR Technology In the Era of Drones, Robots, and Infinite DataHR Technology In the Era of Drones, Robots, and Infinite Data
HR Technology In the Era of Drones, Robots, and Infinite Data
 
BAQMaR - Conference Evening
BAQMaR - Conference EveningBAQMaR - Conference Evening
BAQMaR - Conference Evening
 
SIMS Quantitative Course Lecture 1
SIMS Quantitative Course Lecture 1SIMS Quantitative Course Lecture 1
SIMS Quantitative Course Lecture 1
 
Test your bot_x_conf
Test your bot_x_confTest your bot_x_conf
Test your bot_x_conf
 
Watson - a supercomputer
Watson - a supercomputerWatson - a supercomputer
Watson - a supercomputer
 
Bits of Evidence
Bits of EvidenceBits of Evidence
Bits of Evidence
 
10 tips for a better UX survey
10 tips for a better UX survey10 tips for a better UX survey
10 tips for a better UX survey
 
Communicating data: Reporting user research
Communicating data: Reporting user researchCommunicating data: Reporting user research
Communicating data: Reporting user research
 
Http1
Http1Http1
Http1
 
Beyond usability testing: Getting a holistic view of the user experience
Beyond usability testing: Getting a holistic view of the user experienceBeyond usability testing: Getting a holistic view of the user experience
Beyond usability testing: Getting a holistic view of the user experience
 
Choosing Technical Interview Questions (2006)
Choosing Technical Interview Questions (2006)Choosing Technical Interview Questions (2006)
Choosing Technical Interview Questions (2006)
 
The State of Creative Technology - 2019
The State of Creative Technology - 2019The State of Creative Technology - 2019
The State of Creative Technology - 2019
 
Bundledarrows160 bit.ly/teamcaptainsguild
Bundledarrows160 bit.ly/teamcaptainsguildBundledarrows160 bit.ly/teamcaptainsguild
Bundledarrows160 bit.ly/teamcaptainsguild
 

Semelhante a Lect2-3 Theory-Practice_UsabiilityStudies-Oct2014(1)

Enterprise User Experience in Higher Education
Enterprise User Experience in Higher EducationEnterprise User Experience in Higher Education
Enterprise User Experience in Higher EducationTarik (Rick) Dzekman
 
Requirements Engineering for the Humanities
Requirements Engineering for the HumanitiesRequirements Engineering for the Humanities
Requirements Engineering for the HumanitiesShawn Day
 
UX Research Methods: Behind the Scenes At Process Street
UX Research Methods: Behind the Scenes At Process StreetUX Research Methods: Behind the Scenes At Process Street
UX Research Methods: Behind the Scenes At Process StreetQuekelsBaro
 
Interactive XAI for ODSC East 2023
Interactive XAI for ODSC East 2023Interactive XAI for ODSC East 2023
Interactive XAI for ODSC East 2023Meg Kurdziolek
 
Symplicit Ark Persona Presentation V2.1
Symplicit   Ark Persona Presentation   V2.1Symplicit   Ark Persona Presentation   V2.1
Symplicit Ark Persona Presentation V2.1jodie moule
 
Focus on Experiences, not Products
Focus on Experiences, not ProductsFocus on Experiences, not Products
Focus on Experiences, not ProductsJoel Eden, PhD
 
Best Practice Information Architecture
Best Practice Information ArchitectureBest Practice Information Architecture
Best Practice Information ArchitecturePatrick Kennedy
 
Snowforce 2017 Keynote - Peter Coffee
Snowforce 2017 Keynote - Peter CoffeeSnowforce 2017 Keynote - Peter Coffee
Snowforce 2017 Keynote - Peter CoffeePeter Coffee
 
PROMISE 2011: Seven Habits of High Impactful Empirical Software Engineers (La...
PROMISE 2011: Seven Habits of High Impactful Empirical Software Engineers (La...PROMISE 2011: Seven Habits of High Impactful Empirical Software Engineers (La...
PROMISE 2011: Seven Habits of High Impactful Empirical Software Engineers (La...CS, NcState
 
"Open" includes users - Leverage their input
"Open" includes users - Leverage their input"Open" includes users - Leverage their input
"Open" includes users - Leverage their inputRandy Earl
 
Optimize Customer Experiences with Design Thinking
Optimize Customer Experiences with Design ThinkingOptimize Customer Experiences with Design Thinking
Optimize Customer Experiences with Design ThinkingJared Hill
 
Optimize Customer Experiences with Design Thinking
Optimize Customer Experiences with Design ThinkingOptimize Customer Experiences with Design Thinking
Optimize Customer Experiences with Design ThinkingLima Consulting Group
 
Web 2.0 in Plain English
Web 2.0 in Plain EnglishWeb 2.0 in Plain English
Web 2.0 in Plain Englishtroyangrignon
 
When Too Many is Just Enough: Citizen Engagement and Federal Government Websites
When Too Many is Just Enough: Citizen Engagement and Federal Government WebsitesWhen Too Many is Just Enough: Citizen Engagement and Federal Government Websites
When Too Many is Just Enough: Citizen Engagement and Federal Government WebsitesJeffrey Ryan Pass
 
Sweeny group think-ias2015
Sweeny group think-ias2015Sweeny group think-ias2015
Sweeny group think-ias2015Marianne Sweeny
 

Semelhante a Lect2-3 Theory-Practice_UsabiilityStudies-Oct2014(1) (20)

Enterprise User Experience in Higher Education
Enterprise User Experience in Higher EducationEnterprise User Experience in Higher Education
Enterprise User Experience in Higher Education
 
Requirements Engineering for the Humanities
Requirements Engineering for the HumanitiesRequirements Engineering for the Humanities
Requirements Engineering for the Humanities
 
UX Research Methods: Behind the Scenes At Process Street
UX Research Methods: Behind the Scenes At Process StreetUX Research Methods: Behind the Scenes At Process Street
UX Research Methods: Behind the Scenes At Process Street
 
Interactive XAI for ODSC East 2023
Interactive XAI for ODSC East 2023Interactive XAI for ODSC East 2023
Interactive XAI for ODSC East 2023
 
Hybrid Publishing Design Methods For Technical Books
Hybrid Publishing Design Methods For Technical BooksHybrid Publishing Design Methods For Technical Books
Hybrid Publishing Design Methods For Technical Books
 
Symplicit Ark Persona Presentation V2.1
Symplicit   Ark Persona Presentation   V2.1Symplicit   Ark Persona Presentation   V2.1
Symplicit Ark Persona Presentation V2.1
 
Focus on Experiences, not Products
Focus on Experiences, not ProductsFocus on Experiences, not Products
Focus on Experiences, not Products
 
Best Practice Information Architecture
Best Practice Information ArchitectureBest Practice Information Architecture
Best Practice Information Architecture
 
Snowforce 2017 Keynote - Peter Coffee
Snowforce 2017 Keynote - Peter CoffeeSnowforce 2017 Keynote - Peter Coffee
Snowforce 2017 Keynote - Peter Coffee
 
UCIDesign.ppt
UCIDesign.pptUCIDesign.ppt
UCIDesign.ppt
 
PROMISE 2011: Seven Habits of High Impactful Empirical Software Engineers (La...
PROMISE 2011: Seven Habits of High Impactful Empirical Software Engineers (La...PROMISE 2011: Seven Habits of High Impactful Empirical Software Engineers (La...
PROMISE 2011: Seven Habits of High Impactful Empirical Software Engineers (La...
 
User centric design (ucd)
User centric design (ucd)User centric design (ucd)
User centric design (ucd)
 
Hci Overview
Hci OverviewHci Overview
Hci Overview
 
"Open" includes users - Leverage their input
"Open" includes users - Leverage their input"Open" includes users - Leverage their input
"Open" includes users - Leverage their input
 
Optimize Customer Experiences with Design Thinking
Optimize Customer Experiences with Design ThinkingOptimize Customer Experiences with Design Thinking
Optimize Customer Experiences with Design Thinking
 
Optimize Customer Experiences with Design Thinking
Optimize Customer Experiences with Design ThinkingOptimize Customer Experiences with Design Thinking
Optimize Customer Experiences with Design Thinking
 
Web 2.0 in Plain English
Web 2.0 in Plain EnglishWeb 2.0 in Plain English
Web 2.0 in Plain English
 
When Too Many is Just Enough: Citizen Engagement and Federal Government Websites
When Too Many is Just Enough: Citizen Engagement and Federal Government WebsitesWhen Too Many is Just Enough: Citizen Engagement and Federal Government Websites
When Too Many is Just Enough: Citizen Engagement and Federal Government Websites
 
6p model of research
6p model of research6p model of research
6p model of research
 
Sweeny group think-ias2015
Sweeny group think-ias2015Sweeny group think-ias2015
Sweeny group think-ias2015
 

Lect2-3 Theory-Practice_UsabiilityStudies-Oct2014(1)

  • 1. Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014 Lect 13: Theory & Practice of Usability Studies User Experience Design III John Chin, Ph.D. jchin@acm.org 973-746-7438 October, 2014 Contact Information
  • 2. Copyright T. Komischke, H. Strub, T. Sachs, 2014 An broad overview of context - Who, Anyone - When, Anytime - Where, Anywhere - Why, Motivations - What, Task The missing link - How  Along with some depth on many topics 2
  • 3. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Doctor Who? Who is John Chin? Education  Ph.D. in Cognitive Psychology specializing in  Human Factors  Human-Computer Interaction  University of Maryland, College Park (advisor: Kent Norman) Work Experience  Currently, User Experience Strategist @ Verizon Wireless  User research, Writing UI requirements, Usability testing,  Accessibility, Connected Car, Messaging, 3 patents pending  HiTech – Contractor Homeland Security, TSA  Cognetics – Trizetto Group  Avaya, Alcatel-Lucent, AT&T Bell Labs, Telcordia  Lewis and Clark College 3
  • 4. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Enough about me, what about you? Who are you?  Education and Work experience  Interests and specialty Why are you here?  Goals and Objectives  Career development, job? How can I help?  Needs and Aspirations  Networking 4
  • 5. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Bottom line goals…  1) Be neutral and stick to the facts  2) Advocate for your users, all of them  Recruit representative sample of eventual product/system users  Participants – users testing the design/implementation  Audience – observers (possible development team members)  Report findings and possible solutions to readers or stakeholders  Gain better insights and understanding of the issues  Hope to find possible solutions  Few hard and fast answers: mainly tradeoffs 5
  • 6. Planning a Test  Who: Low vision or Blind user  Where: Standing on the sidewalk at a bus stop  When: During rush hour in the morning  What: Pedestrian getting directions to library  Why: Pick up model created on 3D printer at library  How: Mobile device navigation application 6
  • 7. What are the research questions  Research questions are not tasks  Key questions drive focus of the testing and tasks  Research:  Can people successfully order products?  Will customers be able to pay their credit card bill?  Will radiologists be able to diagnose cases, without errors, efficiently?  Will consumers like this better than their current camcorder?  Tasks:  Typically tasks should be frequent and/or important  How long will it take customers to find the due date for their current statement? 7
  • 8. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Where is project in the product life cycle?  Early?  Are there requirements that need to be gathered?  Or are requirements finalized?  What’s the shape of design concepts?  Is this the next version or a new product?  Some will say “too early for usability” since much is undefined  Some will say “Do a (user-centered) focus group instead”  Not yet 100% clear what it does  Bottom line: A lot is vague, testing can have big impact, but make sure you’re exploring relevant issues  Middle of process?  Requirements finalized  Screens rough, or designed for usability  Lots already determined: Use it. [or not, at your own risk]  Bottom line: Testing can have moderate impact, Fast turnaround important. Choose issues where you’ll be listened to. 8
  • 9. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Where is the project in the product life cycle? (continued…)  Late?  UI is likely designed  Possibly even coded  No leeway on requirements (functionality)  Delivery possibly even scheduled  What is the goal?  If the team will update the UI, could have a meaningful impact  Fitting some request “It needs usability”  No-win if in “police” role  Stakeholder may be looking for “lipstick on the pig”, and endorsement that “Usability says it’s okay”  If it misses the mark, you’re in a no-win situation  Big risk: once it’s delivered, product manager might not want to change UI once existing customers invested time to learn it.  Bottom line: be diplomatic & realistic. 1) Consider risks to company if don’t fix enough, 2) what can be fixed later, and 3) process so won’t repeat this mess next release or product. 9
  • 10. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Where is the project in the product life cycle?  I prefer early-middle, or early  Beware being brought in late: a no-win game  Especially if project planned without user research.  No resources to create prototype (and no time)  No resources to incorporate results (or time)  No one might care  Even if you have a sponsor, you’re being set up for a battle  Focus on your data. It’s a business decision re what to do.  Be careful – think high level  I’ve been surprised many times – people and organizations that seem to have mature processes  “organizations” – since companies have silos: one or many organizations in a company may be awesome. Be careful every time you meet a new one. 10
  • 11. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  1) The human (in general)  Topics include:  Cognitive processor (and memory)  Perceptual systems  Social interaction (  Emotion  Other topics: Persuasion, attraction, etc.  If you have time to read, get applied books on these topics  For topics you really like, get theoretical books; read research papers.  If you don’t already have a good book list, I’ll have suggestions next week 11
  • 12. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  2) Your business:  If a consultancy (or an experienced department)  How does your firm approach problems?  Why do customers choose it?  Know previous reports – especially with your client  Know how to leverage templates: research plan, test documents, reports  Practically: consider ways to promote future work for colleagues – both your team and other teams [without compromising my integrity] 12
  • 13. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  2) Your business  If you’re an employee:  Get to know others in complementary departments  Market Research (make friends, which also reduces risk of being seen as competition)  Customer Service (Call Center “top 10” lists)  Meet, go to lunch, read reports  VOC (Voice of the Customer) data  Becoming increasingly popular  Whatever you have available.  ForeSee: Survey at end of visits 13
  • 14. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  2) Your business, if you’re an employee, continued  VOC data, continued  OpinionLab  Input possible on any page  Works best with meaningful URLs  Beware:  Length. Tempting to continue asking questions, but not paying customers  Order matters: What goes early late, which questions go before others 14
  • 15. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  2) Your business, if you’re an employee, continued  Analytics (i.e., big data) 15 Copyright M. Tremaine, T. Sachs, A. Melewski 2012 This is a sample, not an Adobe endorsement
  • 16. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What do you know?  2) Your business, if you’re an employee, continued  Analytics (i.e., big data)  The ability to track…  Beware: What the data makes easy, versus what is of interest  I have been able to track interactions on a page (sometimes every page), but cannot track a user through paths  Who are your company’s customers? Who use online systems?  How do people access your stuff?  What are trends?  Social networking analysis: As many as company follows  Reports: JD Power, Forrester, Keynote, etc.  Data collection services, and reports, (usually not cheap)  Technology trends  The continuing rise of mobile, tablets  The impact of OS, browsers, etc.  Zeitgeist: e.g., Responsive Web Design 16
  • 17. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What will you test with? (1 of 3)  Paper prototype?  Cheap, usually easy to create, conceptual  Who will create?  Will it be respected internally?  “Photoshop” [i.e., online] paper prototype more “real” than paper  Paper does communicate “conceptual”, room for change  I usually test paper as concepts that extend prototype(s) 17
  • 18. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What will you test with? (2 of 3)  Visio? (or similar, PowerPoint can be fast)  I’m not a “designer” but I can design fairly well in Visio  If my concepts, I’m in control: can test what I want (or edit myself)  If my concepts, distracting from focusing on testing  Key point: Hyperlinks, so it looks like it’s functioning  More about screens than functionality (won’t function well, i.e., fields cannot be filled in)  Something that really runs?: Html, iRise, Axure, Dreamweaver, Ruby, etc.  Functionality is great addition to realism, believability 18
  • 19. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What will you test with? (3 of 3)  Possibly bad addition to cost, time, and pain, since someone has to build & support the prototype.  Arranging for someone to create the prototype can be hard, takes time. Don’t assume getting resources will be fast/easy.  For many audiences, many products/systems, this is a necessity  Even more important for mobile UIs – I believe form factor really matters for mobile  What do you think about where mobile is tested? [lab vs. field]  Trend of hybrid device approach: Big addition to complexity  Prototype web, tablet, phone?  iOS ? (pre-7 & 7?), Android (“flavor”?), Windows mobile, Blackberry?  App versus mobile web? 19
  • 20. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How will you record the test?  Screen critical – see what happened  Audio critical – hear what happened,  High quality audio often undervalued.  Stereo audio recordings can be especially useful, since with headphones you can pick out voices  Very useful: The participant’s face, to see emotions  Still useful – hands on keyboard, on artifacts, etc.  Mobile technology especially challenging to record  Harder to get good screen image  Want to see what hands do, but hands/fingers cover the screen  Still want to see face of the participant 20
  • 21. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How many participants to run? (1 of 3)  Nielsen’s rules of thumb:  5 or a few more. Go higher if test has truly different groups  Consider more rounds of testing, if prototyping and time are available  I often have trouble getting prototyping for multiple rounds of testing)  Bump up numbers a little if only one round of testing 21
  • 22. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How many participants to run? (3 of 3)  Hank’s rules of thumb  Arranging a usability test is hard  Unless really simple, run at least 3 days  60 minute sessions: 15 participants  90 minute sessions: 12 participants  Beware going > 4 days at a crack – exhausting, and too much to analyze  Too much for industry people to pay attention to  If need more people, run another series  Or outsource the usability 22
  • 23. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How many participants to run? (3 of 3)  How many customer “demographic groups” to recruit  Rule of thumb: shoot for at least 3 people per demographic  If differing opinions, still have a tie-breaker  If a no-show, still have 2  If it’s a key customer group (i.e., “prospects”) and site is for them, increase or subdivide that group  No point in testing 12 groups for study with 15 participants 23
  • 24. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How much time per session?  Guesstimate: How big/complex does the UI feel?  If testing in cafeteria (or some place public) keep sessions really short  To cover more ground, lots of sessions covering different tasks  In lab, I believe no shorter than an hour  Lots of overhead, effort, to recruit, to prep, to get observers, to analyze  I find it hard to go > 1-1/2 hours  Sessions are tiring, for participant, moderator, observers  Hard for many types of participants to get lots of time during business hours – might get more strange people  > 90 minutes, schedule in a break (which lowers efficiency)  I’ve seen 2 and 2-1/2 hour sessions (and no break) in last year – and they worked, but moderator knew domain well 24
  • 25. Copyright T. Komischke, H. Strub, T. Sachs, 2014 When to schedule sessions?  Easiest: during business hours  If live observers, much better to have straight days & business hours  But will you get enough of “right” people?  Who are target participants? What are their days like?  Many jobs: work day  Physicians: One early (before 9) one at lunch, 1-2 starting at 5 pm  Think about yourself. I rarely schedule before 10 in Manhattan since hard to get there and lots to get lab ready.  If won’t have live observers, can run any time, including a few per day at odd hours  For some groups, consider weekends  But if you have family or other personal issues (like you have a life), get more money to pay participants 25
  • 26. Copyright T. Komischke, H. Strub, T. Sachs, 2014 When not to schedule sessions  Consider observers as well as participants  Holiday season is hard  If winter, have backup plan  If big storm might arrive during test, have backup plans  Allow time for analysis, and for results to be incorporated  Often have designers responding before discussion. 26
  • 27. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How many sessions to run per day?  Don’t make testing days too hard on anyone (including the moderator)  Want to be as alert, as good of a moderator for first and last session  Breaks between every session gives flexibility for buggy prototypes, late participants, rebooting computer, trying something new, etc  60 minute sessions: up to 5 per day  Breaks of at least 30 minutes per session (even if just to allow for lateness), 60-90 minutes for lunch  90 minute sessions: up to 4 per day  These are more tiring, make sure you have break time  Schedule lightly on first day of testing, to allow updating of everything – even your discussion guide, and to plan around unexpecteds  When possible, run 1 or 2 participants first day, after lunch 27
  • 28. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How to recruit participants? (1 of 2)  What are you testing? How specialized the community? What’s your budget? How much time do you have?  Many options, all involve tradeoffs  Professional recruiters: For many types of consumers, and some target audiences (like MDs)  Most say they’ll recruit anybody for any reason  Realistically, they have specialties  Most have “lists” of known people, who may be semi-professional test takers (some seem jaded)  Recruiters aren’t cheap  Lists of customers, you contact yourself  If you want to test improvements to Emerson’s HMI, want people who know today’s Emerson HMI  Beware: lists can take a LONG time to get (6 weeks or more)  Referrals from sales staff, or from technicians who know their “cool” and “boring” customers  Subscriber lists, Professional Society members (especially valuable if want special expertise) 28
  • 29. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How to recruit participants? (2 of 2)  Recruiting from your web site  But any issue having “customer information”?  Waiting room at medical clinic  Your card is given out by someone trustworthy (like a doctor)  “Mechanical Turk” at Amazon  Craig’s List, or similar ads  Your friends/acquaintances probably not diverse enough.  Beware effects of participants feeling obligated  Children can be especially challenging  You want parents to be nearby so everyone is trusting, but you have to keep them occupied 29
  • 30. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Who to recruit  If clear that there’s one demographic (or “persona”), go for it  But consider going for sub-demographics as more participants  I usually shoot for up to 3, maybe 4 max, demographic groups.  Product space may be much broader  Target groups that are truly different, in parameters that matter to your UI  Many factors can be targeted independently, with same participants. E.g.:  Ranges for age, education, income  Separate from experience in what you’re testing  Avoid trap of recruiting only the young/well-educated  They look nice on video, have great things to say, but…  Unless all users will be young and well-educated  I learn a lot from diversity 30
  • 31. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Who to recruit: Common screening criteria (1 of 2)  No one in your industry, or with a household member in your industry  No one in design or usability  Always ask for a range, within whatever criteria.  If open to unemployed or retired, they’re probably easier to find for “business hours” studies than people who work, so “up to 2”  Internet skill: Want people who use web themselves, often, familiar  Age: cover diversity of your user community  I’m pushing for older but not above 69  Many older users, but tend to talk more, and therefore get less done (in my experience)  I’ve had most dishonesty with elders: age (not be too old), internet skills  Consider gender distribution 31
  • 32. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Who to recruit: Common screening criteria (2 of 2)  Education: I’ve known researchers who screen for minimum of some college for general consumer products.  Getting some without college is more honest test of usability, for those with low literacy skills – who buy and use things  I’m often fascinated with responses of those with least education  Ethnic diversity  Income: As low as appropriate. If you want some truly wealthy people, might need to specially recruit and pay more  But can ask for people whose speech is easy to understand (critical)  Where they live: City vs. suburb; and which city (Philly vs. NY) 32
  • 33. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Special participants: Children, Disabled, Employees  Children:  Want a witness, at all times  Parent should be nearby  Make it fun  Recruit so you’ll get interested kids  Disabled  Hard enough to get disabled with full cognitive skill  Check literature if recruiting disabled without full cognitive skill  Employees  Is it their choice? Coerced?  Impact of losing time on job  Fear if criticize system, or people, when it’s recorded 33
  • 34. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Possible Forms for participants (1 of 2)  Informed consent form has two goals:  Inform participants of what they’ll be doing, realistically, any risks (even of embarrassment), and benefits (often, interesting tasks; chance to help change a product; and get paid)  Consent: So it’s clear what they’re consenting to  Media Release:  That you are recording them  How recordings will be used (for study, nothing else)  Can protect you as well as participants: if colleague loves a clip- wants to show at a conference (“Sorry!”)  If session is that good, can go back and pay more, or recreate the scene with actors 34
  • 35. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Possible Forms for participants (2 of 2)  NDA (Non-disclosure Agreement)  If you’re showing proprietary concepts  Explain NDA too – so understand  I’ve heard that a properly written and executed NDA protects even if people talk – you did your due diligence  Liability release (?)  In case incidentally hurt when coming to your facility  Include conditions that lawyers want  But fight for plain language  Long forms are harder to understand –consider more short forms than fewer long forms 35
  • 36. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Don’t fight the law (but then I’m married to a lawyer)  This is probably all new for most corporate council  Draft pretty good sample forms before approaching  Shows you know something  Easier to edit than start from scratch  Be able to explain what’s there, why, who and what your intentions are  Provide articles on what you’re after, or books  Negotiate, but with the company’s best interests in mind 36
  • 37. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How much to pay participants?  Goldilocks rule of thumb:  Not too much, not to little  If doing internal testing on employees, might be limited by corporate rules (participants already missing work during work time  For external participants, ask your recruiter for advice  Consider adding in reasonable extra payment for transportation/parking costs  BTW, if paying a participant more than $500 in a year, check with company HR about possible need to report “salary” to IRS  Only an issue if have same person return 37
  • 38. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Should I use floaters?  Floaters are backup participants, usually recruited to cover two sessions  Extremely valuable in case of no-show  Also, for when a participant is totally inappropriate  But, who has time to hang out for 2 sessions? Be careful  I use best recruiter I can, request to go for 100% turnout  I’ve had odd birds as floaters 38
  • 39. Copyright T. Komischke, H. Strub, T. Sachs, 2014 How to record a test?  Morae, by Techsmith: The “Honda Fit” of usability recording:  $1500 for a full license  Records two streams, usually screen and webcam on face, as well as an audio stream  If built-in mic isn’t awesome, get a good external mic  1copy of Morae includes 1 copy of “Observer”, which lets people in another room watch live (delayed ~20 seconds)  Very reliable  Not bad to learn, to set up  Can take notes, and make marks, live (but I don’t see that done)  Will make highlights videos  Pretty efficient storage, so many sessions can go on a DVD 39
  • 40. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Recording a test – More on Morae  Morae also good for portable lab – only need 2 laptops  Requirements for computers not clear: doesn’t reflect multi-core processors  In my experience, Morae has run fine on machines well below the “minimum recommended” capability  A hack to get 3 images is a hardware video mixer, that will superimpose for you. 40
  • 41. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Recording a test: Lots of fancier software, but..  Pricier, need fancier computer and equipment, can be testier  I know many big companies use OvoStudios  My personal experience: very nice & helpful, not as reliable as I’d like  But I’ve heard of companies that find it trustworthy  Network to find something that works for your needs.  Consider:  HD looks great, awesome detail  Probably not needed for routine recordings of basic web sites 41
  • 42. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Recording a test: Back to mobile  How to get a good screen? Apple to HDMI to DVI cable, or Apple TV hack if security allows  License supports 3 images – I think all 3 are important  [Note: target area – ask participants to keep the phone THERE 42
  • 43. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Other Technology Issues:  Eye Tracking: compelling to watch, help observers stay awake  Beware of heat maps: If interrupt participants, etc.  Magic is uninterrupted, enough people to get good trends, so software can analyze what happened  Web site also needs to support eye tracking: overlays, modals mess things up  Software only knows the URL, not what’s on screen  Emotion tracking: GSR (how emotional) and valence (+/-)  Affectiva is a company to watch 43
  • 44. Copyright T. Komischke, H. Strub, T. Sachs, 2014 The Moderator’s Guide (1 of 4)  Introduction  Why you’re there  That this is usability so  A) Please speak aloud (we’ll practice),  B) Testing the UI, not testing you,  C) Don’t worry about errors since prototype (not real data or system)  Ask questions (though I might not answer)  Can stop at any time, for any reason  Pay at beginning of session: Show trust (people don’t walk away)  Plan the task list  Based on the research questions  Ideally, based on your knowledge:  What are the true top tasks (consider what Gartner, JD Power say) 44
  • 45. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Moderator’s Guide (2 of 4)  Think about time. Consider optional tasks to fill sessions for fast participants.  Or have an extra task or two, and prioritize list in pilot sessions  Moderator’s guide: By each task I like to have questions to probe, and suggested things to observe  Participant’s guide just has the tasks – other stuff is deleted  Edit moderator’s guide, either regenerate Participant’s guide, or be careful on version control.  Questions to ask:  Per task: Make sure to ask the goal of the task  Find a value? Perform a task  Otherwise, think about what you’ll learn  Can ask questions like “How quickly do you expect to do this task? How easy do you expect this task to be?  After: How easy was it to perform this task? How often do you perform this task? 45
  • 46. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Moderator’s Guide (3 of 4)  Questions to ask: At end of Session  I like to ask:  What did you like best about the prototype system?  What did you like least about the prototype system?  [Sometimes in form of “What were 3 best & 3 worst things?”]  The SUS is trendy [SUS = System Usability Scale]  Original by John Brooke, 1986, DEC 46
  • 47. Copyright T. Komischke, H. Strub, T. Sachs, 2014 47 The original SUS
  • 48. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Moderator’s Guide (4 of 4)  Thoughts on SUS:  Alternates positive and negative questions, so respondent needs to be awake  Around since the 80’s, lots of experience in the industry  A few common updates [i.e., “website” instead of “system”]  Like school grading: 70 is okay, 80 good, 90 superb  But maybe UI’s are better these days -> I’ve seen mainly upper 80’s or better  Net Promoter Score (NPS): How likely are you to recommend this product/company to a friend?  Trendy, originally published in Harvard B-School Journal  Were participants supposed to learn something?  Though be careful about testing learning in 1-hour long session, unless that’s what system is for  1 hr not much time to learn, especially when many tasks 48
  • 49. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Cautions on locations (1 of 2)  Dedicated lab:  Will you use it enough? Perhaps try lending it out, but beware…  Sufficient staff support? [reception & waiting area, space for observers who have meetings, snacks & drinks and meals]  Window: Great, but then keep observer room quiet and dark  Need to keep equipment secure  Conference rooms and a portable Morae lab  Still have to keep laptops secure  Will you always get good rooms for test when you need them?  Others’ office  Conference rooms: Will your Morae computers run on their network? If not, how will they communicate?  Fairly safe from interruptions, though.  Cubes: Beware interruptions… Phone will ring & cube’nik might need to pick up; Co-workers don’t recognize you as important  Offices: Probably okay as long as participant respects you 49
  • 50. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Cautions on locations (2 of 2)  Remote Testing  Success depends on installing web-sharing software, but software, computers and bandwidth are all pretty good  Usually only 5 minutes, sometimes much worse  Better with web cam  Surprisingly to me, rarely interrupted, but phone calls seem to be respected  3rd party facility (i.e., market research company)  Surprisingly pricey – be prepared for unexpected costs  But they handle the details – REALLY nice  When I use a facility, I have them recruit – so one point of failure  Even professionals sometimes use cafeteria, or Starbucks  Best for quick “first impression” studies, lots of people in little time  Can recruit for more in-depth study (follow-up in lab…) 50
  • 51. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Screen-sharing software, for remote live observers  Easiest to use the corporate standard  It’s IT-endorsed  No firewall issues  Account likely easy to set up, and/or use  If there’s a problem, you used what company wanted  Otherwise, I believe in WebEx & a telephone bridge  Usually reasonably pain-free to install  Performance (delay, quality, frame-rate) pretty good  Not cheapest, but trustworthy is important to me 51
  • 52. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Running a Test52
  • 53. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Be like Johnny Carson (Used to host The Tonight Show)  Always genial, friendly  Show was about the guest(s), not him  THEIR thoughts & opinions  Johnny didn’t lead  But he got guests to talk  When there was a problem, he took the blame (never the guest’s fault unless something terrible)  Was incredible at improvising  For more on Johnny, watch for rebroadcast of PBS documentary 53
  • 54. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Taking notes during:  If you follow template strictly and have time, excel template for all key questions and observation points is neat  Time consuming to set up  Not convenient to use when with participant  Can take notes with computer  Not best for establishing rapport  Take notes at a constant pace (lots of them)  So don’t communicate what you care about  So writing when participant thinks they’ve said something important  Compatriot’s notes (ideally, not in sight of participant) can be great, but everyone looks for different things  Who will be lead on the report? 54
  • 55. Copyright T. Komischke, H. Strub, T. Sachs, 2014 Things to watch during session55
  • 56. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What to do if: (1 of 2)  Participant is no-show  Ask to arrive 15 minutes early: time for forms, for bathroom, and to call recruiter before session time  Call recruiter – check on reminders  Backup slots available?  If remote, can’t get to internet  Common to get weird firewall issues  Have a local copy  Prototype fails  Computer fails  Participant arrives quite late  Running really behind. Participant’s a talker, trouble, started late  Participant wants bathroom break, but already behind 56
  • 57. Copyright T. Komischke, H. Strub, T. Sachs, 2014 What to do if: (2 of 2)  Early in session, can see that participant doesn’t fit the study?  It appears that participant lied to recruiter re something key? (After all, these studies pay well and economy is bad)  You give cash or check to participant at start of session, and he/she gets up and leaves?  Participant admits to having committed a crime on your video recording? (“When I do my taxes, I don’t put in all the receipts”)  You have strong evidence that participant is victim of or commits abuse, or is considering suicide? 57
  • 58. Copyright T. Komischke, H. Strub, T. Sachs, 2014 After a test  Debriefs are great. Consider recording them if you don’t take good notes.  Write down first impressions of results while they’re fresh in your mind.  Get invoices in promptly: * participants *, recruiter, and any vendor you might use again  Return borrowed equipment promptly (before damaged, lost)  Listen to sessions as “background music”. You will hear things  Push self for rapid turn-around.  Consider rehash/highlights meeting immediately afterwards, recorded, to get impressions of observers  Tight deadline for report doesn’t allow forgetting  But beware getting slammed if rest of your job fell behind due to time away for testing 58
  • 59. Copyright T. Komischke, H. Strub, T. Sachs, J. Chin 2014 Questions?

Notas do Editor

  1. Who, Anyone - When, Anytime - Where, Anywhere - Why, Motivations - What, Task
  2. Alternatives to SUS are SUMI and QUIS. Some versions of SUS make all responses positive to avoid respondent confusion. There is a formula for converting SUS scores into an equivalent NPS score.