It’s very easy for User Experience researchers to get stuck in the rut of using your favorite research methods for gathering information and getting user feedback. But, are you really gathering the best information that you can? Or are there other methods that are better suited for your project’s specific needs?
Or, if you’re just starting out – how do you know whether you should conduct interviews, run a survey or a card sort, or something different all together?
Don’t stress – in this webinar, we’ll cover the most popular user research methods and discuss their strengths and weaknesses. Each method shines in different circumstances, and we’ll highlight the factors that will make each successful. We will also present a structured approach to helping you choose the best method or methods for a particular situation.
10 Best WordPress Plugins to make the website effective in 2024
Choosing the Right Research Methods for Your Project (webinar)
1. Choosing The Right Research
Methods For Your Project
Susan Mercer – Senior Experience Researcher
smercer@madpow.com
@susanamercer
October 29, 2013
2. Hello, I’m Susan Mercer
BA and MSc in Geophysics
19 years in software and web UI and UX design
Developer
Designer
Web Producer
Product Manager
Researcher
MS Human Factors, Bentley University
Twitter: @susanamercer
2
3. Introduction
“It is tempting, if the only tool you have is a hammer, to
treat everything as if it were a nail.”
– Abraham Harold Maslow
Agenda:
Market Research & User Research
Study Goals
User Research Methods
Choosing the Right Methods
3
6. Defining Goals
“The most serious mistakes are not being made as a
result of wrong answers. The truly dangerous thing is
asking the wrong questions.”
– Peter Drucker
6
7. Defining Goals
“The most serious mistakes are not being made as a
result of wrong answers. The truly dangerous thing is
asking the wrong questions.”
– Peter Drucker
The first key step of a good study is to determine your goals
Good Study Goals:
Provide focus to the study
Become more specific as the project progresses
Focus on user behavior, not on interface specifics
Help drive actionable results to impact business or design decisions
7
8. Defining Goals
Consider:
Where are you in the project?
Identifying problems with current behaviors to identify opportunities?
Evaluating the appetite for a product feature?
Gathering feedback on an early design?
Testing an existing product?
What are you trying to learn?
Is there an existing artifact to test?
How much time do you have?
How much of a budget do you have?
8
12. User Interviews
What is a User Interview:
A focused one-on-one conversation
In-person or via telephone
Familiar format for both participant and interviewer
12
13. User Interviews
How to ask good interview questions:
Open-Ended Questions
Who, What, When, Where, Why, How
Start with broad questions, then use prompts to fill in gaps
Let the participant guide the discussion at first to what they think is important
Example:
Tell me about sharing the printer and copier with others.
What works well?
What doesn’t work?
Do you need an ID, card, or code to use the shared printer/copier?
Does your printer keep track of how many copies/dollars you have left on your “budget”?
Do you know how much it costs per page to print?
Is your company actively trying to manage printing costs? If so, how?
13
14. User Interviews
Pros
Cons
Moderator can ask follow on questions
to focus on topics of interest
Qualitative data takes longer to analyze
than quantitative
Gather rich, detailed data
Questions may not completely consistent
from participant to participant
Good for exploratory, formative
research
Need to manage talkative and quiet
participants
Small sample size yields effective data
Not as efficient when there are many
different demographics with truly
different needs
Can be done remotely via telephone
Relatively inexpensive to conduct
14
15. User Interviews
Additional Resources:
Understanding Your Users: A practical guide to user requirements. Catherine Courage & Kathy
Baxter, Morgan Kaufmann, 2005. Chapter 7
Interviewing Users: How to uncover compelling insights. Steve Portigal, Rosenfeld Media, 2013.
Mental Models: Aligning Design Strategy with Human Behavior. Indi Young, Rosenfeld Media, 2008.
Chapter 7
15
17. Ethnography
What is ethnography?
Observing users in their natural environment while they perform the tasks of
interest
Big “E” Ethnography
“A research strategy that allows researchers to explore and examine the cultures and
societies that are a fundamental part of the human experience” (Murchison)
Generally involves immersion of the researcher within the culture to be studied
Longitudinal and can take many years
Little “e” ethnography
UX Researchers perform little “e” ethnography
Typically involves a period of observation followed by an interview
Focuses on how users perform the tasks of interest and their overall environment
17
18. Ethnography
Good to examine:
What information users look for and where they find it
Who and what they interact with to get the task done
The order in which things happen, and whether that is important
Distractions, barriers, and interruptions that the participants encounter
18
19. Ethnography
Pros
Cons
See interactions with other people and
other products/systems
Logistics of setting up
observations/interviews in someone’s
home/office
Users will interact with their own items,
including cheat sheets or other “coping
mechanisms”
May be difficult to gain access to
individuals or certain environments, or for
extended periods of time
Good to understand how things are
really done vs. how they are supposed
to be done
Observing/interviewing many people
requires more time
Can yield insightful “aha” moments and
innovative ideas
On-site recording logistics can be
challenging
Observe natural interruptions and other
events which participants forget to
report in interviews
19
20. Ethnography
Additional Resources:
Ethnography in UX, Nathanael Boehm, uxmatters.com,
http://www.uxmatters.com/mt/archives/2010/06/ethnography-in-ux.php
Practical Ethnography: A guide to doing ethnography in the private sector, Sam
Ladner, Expected soon. http://www.practicalethnography.com/
Ethnography Essentials: Designing, conducting, and presenting your research,
Julian M. Murchison. Josey-Bass, 2010.
20
22. Collaging
What is Collaging:
A creative activity to help uncover thoughts and attitudes.
Participants are given photographs, large paper and art supplies and asked to
create a collage about a central theme. They then explain their collage.
Can be done one-on-one or as a focus group activity
Can make some participants conscience of their lack of creative skills
Their storytelling about the college is the key
Followed by a short interview
22
23. Collaging
How to do it:
1.
Prepare materials
2.
Define focus
3.
Create the collage
4.
Have them tell their story
What does
“Saving Energy”
mean to you?
23
24. Collaging
Pros
Cons
Gather rich, detailed data
Qualitative data takes longer to analyze
Small sample size
Questions are not consistent from
participant to participant
Good for exploratory research – when
you don’t know the questions to ask
Some participants are not as open to
creative exercises as others
Good for uncovering values,
motivations, attitudes that are not topof-mind – participants may not even be
consciously aware of them
Preparation can be time-consuming
Good seeding activity for focus groups
Can help some participants verbalize
thoughts
24
25. Collaging
Additional Resources:
Collaging: Getting answers to the questions you don’t know to ask. Kyle Soucy,
Smashing Magazine, February 6, 2012.
http://uxdesign.smashingmagazine.com/2012/02/06/collaging-getting-answersquestions-you-dont-know-ask/
McKay, D., Cunningham, S. J., Thomson, K. Exploring the user experience through
collage. CHINZ ’06 Proceedings of the 7th ACM SIGCHI New Zealand’s chapter’s
international conference on Computer-human interaction: design centered HCI,
p. 109-115.
25
27. Diary Studies
What is a Diary Study:
A semi-longitudinal study used gather user behavior over time
Users are asked to keep a diary for a certain topic
When they interact with a device/website/etc.
Health, food, study habits, banking habits, etc.
When to use a diary study:
When a single research session will not truly capture users’ interactions
When the users’ environment plays a role in how they use the interface
If there is plenty of time in the project timeline
27
28. Diary Studies
Pros
Cons
Captures users’ data in their natural
setting
Reliant on participants keeping up
with their diaries
Users will be interacting with their
own items
Participants self-select content
Can capture data over time
Management of the data collection
mechanism
Must have time in the project timeline
28
29. Diary Studies
Additional Resources:
Observing the User Experience: A Practitioner’s Guide to User Research, Elizabeth Goodman,
Mike Kuniavsky, & Andrea Moed, Morgan Kaufman, 2012.
Diary Studies in HCI & Psychology, Demetrios Karis, UPA Boston 2011 Conference.
http://www.slideshare.net/UPABoston/diary-studies-in-hci-psychology
29
34. Desirability Testing
What is Desirability Testing?
A set of tools for exploring participants’ emotional reactions to different visual
designs.
Use this method to:
Select a visual design which most closely aligns with desired traits
Evaluate a product’s visual design vs. competition to uncover differences in
brand perceptions or user experience
Evaluate a single design for emotional response
34
35. Desirability Testing
Open-Ended Questions:
Ask for initial reactions
Show a design to a participant and ask for their immediate reactions
Let them tell you whatever they want to tell you
Sentence Completions:
Start 5-10 sentences and ask them to finish them based on their reactions to the
design
Product Reaction Words:
Show them ~ 60 words – both positive and negative connotations
Ask them to select a small number which best represent the design
35
36. Sentence Completion
Have participants complete sentences
Gather focused feedback without being leading
Examples:
This page makes me feel…
Reading information on this page is…
If I used this website all day, I would…
I want to use this website because…
I do not want to use this website because…
36
37. Product Reaction Words
Accessible
Dated
Hard to Use
Simplistic
Appealing
Dull
Helpful
Sophisticated
Approachable
Easy to Use
Impersonal
Sterile
Boring
Effective
Innovative
Stimulating
Busy
Efficient
Inspiring
Straight Forward
Clean
Energetic
Intimidating
Stressful
Clear
Engaging
Intuitive
Time-consuming
Comfortable
Enthusiastic
Inviting
Time-saving
Compelling
Exciting
Motivating
Too Technical
Complex
Familiar
Organized
Trustworthy
Confident
Fast
Overwhelming
Understandable
Confusing
Flexible
Patronizing
Usable
Convenient
Fresh
Predictable
Useful
Creative
Friendly
Professional
Valuable
Cutting Edge
Frustrating
Reliable
37
39. Product Reaction Words
Existing Design
Customers
Customers & Non-Customers
Non-Customers
Reliable
Accessible
Appealing
Confusing
Comfortable
Organized
Professional
Busy
Confident
Straight Forward
Approachable
Complex
Effective
Understandable
Efficient
Dull
Friendly
Easy to Use
Flexible
Hard to Use
Inviting
Clear
Helpful
Impersonal
Time-saving
Convenient
Overwhelming
Trustworthy
Clean
Boring
Useful
Familiar
Frustrating
Simplistic
Intimidating
Valuable
Stressful
Usable
Time-consuming
39
40. Product Reaction Words
Number of positive vs. negative words
Number of Words
70
60
50
40
Positive
30
Negative
20
10
0
Existing
Design A
Design B
Design C
40
41. Product Reaction Words
Words that the client desires the look and feel to portray
Existing
Design A
Design B
Design C
Appealing
Appealing
Appealing
Appealing
Clean
Clean
Clean
Clean
Efficient
Efficient
Efficient
Efficient
Engaging
Engaging
Engaging
Intuitive
Professional
Professional
Useful
Total
Professional
Useful
Useful
9
20
Useful
16
9
41
42. Desirability Testing
Pros
Cons
Gather qualitative feedback on
emotional response to designs
Choice of adjectives could introduce
subtle bias
Gather focused feedback in a neutral,
open-ended fashion
Generally, relatively small sample size
Product reaction words provides some
level of quantitative measurement, but
should be used carefully
Qualitative feedback can take longer to
analyze
42
43. Desirability Testing
Additional Resources:
Benedeck, J. & Miner, T. (2002) Measuring Desirability: New methods for evaluating
desirability in a usability lab setting. Proceedings of Usability Professionals Association,
2003, 8-12.
http://www.pagepipe.com/pdf/microsoft-desirability.pdf
Rapid Desirability Testing: A Case Study, Michael Hawley, uxmatters.com,
http://www.uxmatters.com/mt/archives/2010/02/rapid-desirability-testing-a-casestudy.php
Users play cards We keep score Magic Results, presentation by Carol Barnum and Laura
Palmer
http://www.slideshare.net/cbarnum/barnum-palmer-stc-2011-presentation
Kuhala, S. & Nurkka, P. (2012) Sentence Completion for Evaluating Symbolic Meaning.
International Journal of Design, 6(3), 1525.http://www.ijdesign.org/ojs/index.php/IJDesign/article/view/1166/523
43
45. Usefulness Testing
What is Usefulness Testing?
Usability testing identifies issues that prevent users from performing tasks
Usefulness testing evaluates the match between user needs and system features
1. Investigate how users do tasks today
2. Have them use the system or prototype
3. Ask questions about usefulness
45
46. Usefulness Testing
1. Background Questions:
Investigate how users do things today – what works well and what doesn’t
Sample Questions:
Tell me about your process for ordering equipment and supplies from XYZ Co.
What do you typically need to order?
How do you determine whom to contact?
How often do you order supplies?
What happened last time you felt your resolve to eat well sagged?
What sources of support did you rely on?
How did they help you?
How did you get “back on track”?
46
47. Usefulness Testing
2. Show them the product / prototype
Watch for:
Comments about usefulness
“Wow. This would really make it easier to reorder the same things.”
“I don’t like this. It would take me too long to do this part.”
Neutral reactions
“Eh. This is OK.”
Probe: “Might this help you get your work done? Or not?”
Be wary of “This isn’t how we do it today” comments
Business processes may need to change in the future, so try to find out why something
would not work.
47
48. Usefulness Testing
3. Follow up Questions:
After they have seen the system/prototype, how do they think it might work
for them?
Sample Questions:
What are your overall impressions of the website?
Based on what you saw today, how helpful would the system be to you? Why?
What could we do to make this more helpful to you?
48
51. How to Choose the Right Method(s)
1.
Make sure your research goals are clear
Know what you want to learn
Know what you will do with the information you gather
Know what decisions the business needs to make
2.
Know your constraints
Timeline?
Resources?
Budget?
Access to Users?
3.
Create a Methods Chart
The answers will become clear
51
52. Methods Chart
Method
Method 1
Pros
• List the advantages for
this method for this
specific project
Cons
• List the disadvantages
for this method for this
specific project
Candidate
Yes / No –
Would this method be a
good candidate for this
specific project
Method 2
Method 3
Etc…
52
53. Methods Chart Example
Method
Pros
Cons
Candidate
Interviews
• Good for capturing
motivations for behavior
change
• Can get good qualitative
details on why users
prefer each concept or
not
• Good to explore issues to
fine tune survey
questions
• Can be done via phone
(get broad geographic
sample)
• Can be done quickly
• Small numbers – client
wants large numbers
Y – would be good as a
qualitative method to pair
with a larger quantitative
method
Focus Groups
• Good for qualitative
information gathering
• Could generate some
interesting conversations
about behavior change
motivations
• Wouldn’t get as much
detail as interviews
• Concern about group
think when evaluating
concepts
• Concern about not
sharing details of
personal goals in front of
others
N – interviews would be
better for qualitative
Survey
• Good for large numbers
• Easily replicated across
different
countries/languages
• Can be done online for
broad geographic
distribution
• Can be done quickly
• Unclear what exact
questions to ask
• Doesn’t provide detailed
insights into qualitative
topics
Y – good paired with
qualitative method
53
54. Want to learn more?
Attend our Workshop
Choosing The Right Methods For Your Research Project
Susan Mercer & Dan Berlin
Full Day Workshop, IA Summit – San Diego, CA
March 26, 2014
http://www.iasummit.org
Contact Us
Susan Mercer
Senior Experience Researcher
smercer@madpow.com
@susanamercer
Dan Berlin
Experience Researcher Director
dberlin@madpow.com
@banderlin
54
Usability testing & Interviews are very popular, but are not always the best method.Goal: Show you some lesser-known methods and when they are appropriate.
People unfamiliar with the field sometimes ask what is the difference between Marketing Research and User Research?Sometimes the difference is a bit fuzzy, particularly because we sometimes use similar methodsThey like on a continuumMarketing Research primarily focuses on whether people will BUY a productUser Research focuses on how people USE a product
Before we can talk about methods, you first have to be clear about what you want to learnFocus on answering questions that will help your business make decisions
Before we can talk about methods, you first have to be clear about what you want to learnFocus on answering questions that will help your business make decisions
Generative = Background = Before you have an artifact to testEvaluative = Get feedback on some stage of prototype or productRecognize overlap with Marketing Research – Interviews, Surveys, Focus Groups
Not good when:Need consistency in questionsNeed quantitative, statistically significant dataHave a lot of disparate user groups (required sample size increases)
ethnography:
Good for:Identifying key pain points to address with new product or new feature setDetermining how a new product might fit into a complex ecosystem (e.g. operating room)Understanding how people do things today – and what product needs to compete with (e.g. iPad vs. hand-writing notes)
Other creative activities work too: Lego Serious Play
Key part is hearing their storyThat is where the symbolic meaning comes outThat is where you get them to articulate their thoughts that they just surfaced with the creative activityWhat is most important to them?
Generative = Background = Before you have an artifact to testEvaluative = Get feedback on some stage of prototype or productRecognize overlap with Marketing Research – Interviews, Surveys, Focus Groups
We think Usability Testing first – but, that is just one piece of the puzzleUsefulness & Desirability Testing use similar methods, but complement by focusing questions differently