Defining User Research Methodologies: A Pragmatic Approach

DEFINING USER RESEARCH METHODOLOGIES
A PRAGMATIC APPROACH
MINNEWEBCON 2012

Monday, April 30, 2012

DEFINING USER RESEARCH METHODOLOGIES
A PRAGMATIC APPROACH
MINNEWEBCON 2012

#mwcresearch


ZACK
SENIOR USER EXPERIENCE DESIGNER
USER RESEARCH PRACTICE LEAD

@zacknaylor


DAVE
USER EXPERIENCE DESIGNER

PHD CANDIDATE
Tech Comm & User Experience

@Dave_L_Jones


WHAT ABOUT YOU?


WHAT WE’LL COVER


WHAT WE’LL COVER
Common user research methods


WHAT WE’LL COVER
Common user research methods

Identifying the appropriate use and combination of methods


WHAT WE’LL COVER

Identifying the appropriate use and combination of methods


WHAT WE’LL COVER

Goals in research planning


WHAT WE’LL COVER

Research data analysis


USER RESEARCH
IS NOT

MARKETING RESEARCH


USER RESEARCH
IS NOT

MARKETING RESEARCH

...but there is some overlap


USER RESEARCH
We do this
IS NOT

MARKETING RESEARCH



USER RESEARCH
We do this
IS NOT
We DON’T do this

MARKETING RESEARCH



USER RESEARCH MARKETING RESEARCH



Preferences
Opinions
Likes
Desires



Behavior Preferences
Needs Opinions
Goals Likes
Tasks Desires
Mental & Physical Context


RESEARCH SPECTRUM

Indi Young - Mental Models: Rosenfeld Media http://www.flickr.com/photos/rosenfeldmedia/2159500714/in/set-72157603511616271/

RESEARCH SPECTRUM
overlap

Indi Young - Mental Models: Rosenfeld Media http://www.flickr.com/photos/rosenfeldmedia/2159500714/in/set-72157603511616271/

WHY DO USER RESEARCH?


Remember when we said that
design solves problems?


Remember when we said that
design solves problems?

Remember when we said that a UX
process starts with defining the problem?



Remember when we said that a UX
process starts with defining the problem?


The systematic investigation into and study of
materials and sources in order to establish
facts and reach new conclusions.



Research tells us what the problem is.




Research tells us why it’s a problem.




Research tells us why it’s a problem.

Research shows us how to fix it.


BENEFITS OF USER RESEARCH


Throughly defines the problem.



Informs design decisions.




Provides direction & priority.


Throughly defines the problem. SHOWS US THE “WHAT”





Informs design decisions. TEACHES US THE “WHY”




Informs design decisions. TEACHES US THE “WHY”

Provides direction & priority. GUIDES US TO THE “HOW”


EXAMPLE PROJECT NEEDS
Stated: What Research Tells Us:

“WHAT”

“WHY”

“HOW”



“WHAT” “We need a website redesign”

“WHY”

“HOW”


The website doesn’t meet
users’ expectations

“WHY”

“HOW”



“The current site isn’t
“WHY” working for us”

“HOW”



“The current site isn’t There are usability
“WHY” working for us” flaws in the design

“HOW”




“We should update
“HOW” the look & feel”




“We should update We need to design clearer
“HOW” the look & feel” call to action buttons.





Vague & Undefined





Vague & Undefined
Clear & Actionable

BEWARE OF THE DATA TYPE

QUALITATIVE vs. QUANTITATIVE



QUALITATIVE X
vs. QUANTITATIVE



QUALITATIVE X
vs. QUANTITATIVE

Just remember



QUALITATIVE X
vs. QUANTITATIVE

Just remember

We need both!



QUANTITATIVE
Objective
X
vs. QUALITATIVE
Subjective
“Measurable” “Non-measurable”
Numerical Data NOT Numerical
Statistics Concepts



QUANTITATIVE QUALITATIVE


QUANTITATIVE

QUALITATIVE


Page Views
Bounce Rate
Time On Site
Yes/No
True/False


Page Views Expectations
Bounce Rate Reactions
Time On Site Confusion
Yes/No Comprehension
True/False Behavior


QUALITATIVE

QUANTITATIVE



QUANTITATIVE informs QUALITATIVE




“WHAT”




“WHAT” “WHY”




“WHAT” informs “WHY”




i.e. Sign-up page bounce rate it high.




i.e. Sign-up page bounce rate it high. i.e. Call to action text is confusing.


WHEN SHOULD YOU DO
USER RESEARCH?

Timeframe Beginning of the project During Design Production Project Completion

- re-enforce direction
- inform direction & scope - gauge progress/success
Benefit - validate design decisions
- throughly defines the problem - guides product direction
- acquire design feedback at
- provides insights for next steps - discover areas for improvement
significantly lower cost

- Contextual Inquiry - Usability Testing - Usability Testing
Example - Field Study/Ethnography - Card Sorting - Site Search Analytics
Method/Use - Stakeholder Interviews - Personas - Web Analytics
- Surveys - Mental Models - A/B Testing/Multivariate Testing


BONUS ROUND


BONUS ROUND
ONGOING RESEARCH


BONUS ROUND
ONGOING RESEARCH
How:


BONUS ROUND
ONGOING RESEARCH
How: - Create a solid and sustainable research plan for a continuous
feedback loop from your customers/users


BONUS ROUND
ONGOING RESEARCH

Why:


BONUS ROUND
ONGOING RESEARCH

Why: - Your audience changes and evolves over time


BONUS ROUND
ONGOING RESEARCH

- Your product/service/website will attract new audience segments


BONUS ROUND
ONGOING RESEARCH

- Your product/service/website will attract new audience segments
-Other products/services/websites introduce new expectations for
interacting with your information


COMMON RESEARCH SKILLS
INTERVIEWING

OBSERVATION

LISTENING

ANALYSIS (SENSE-MAKING)

T L
IA RESEARCH SKILLS
COMMON
S E N
E S
INTERVIEWING

OBSERVATION

LISTENING

ANALYSIS (SENSE-MAKING)

NERDERY RESEARCH METHODS
Usability Testing Field Study/Ethnography

Surveys Site Search Analytics
Card Sorting
Web Analytics
Stakeholder Interviews
A/B Testing : Multivariate Testing
User Interviews
Mental Models
Contextual Inquiry

Personas


“WHAT”

Card Sorting
Web Analytics
User Interviews
Mental Models
Contextual Inquiry

Personas


“WHY”

Card Sorting
Web Analytics
User Interviews
Mental Models
Contextual Inquiry

Personas


“HOW”

Card Sorting
Web Analytics
User Interviews
Mental Models
Contextual Inquiry

Personas


STAKEHOLDER INTERVIEWS


WHAT ARE
STAKEHOLDER INTERVIEWS?
Definition:


WHAT ARE
STAKEHOLDER INTERVIEWS?
Definition: one on one conversations with client champions of a project aimed
at gaining understanding of three overarching themes of
information:

1. Project Context

2. Target Audience

3. Project Success


THE “HOW-TO” OF
STAKEHOLDER INTERVIEWS
Identify the stakeholders on the project, schedule ONE HOUR with each stakeholder
1. Recruiting separately.

2. Research Plan Establish a clear focus for what you expect to find & what you hope to learn.

3. Conduct Interviews Watch users completing relevant goals & tasks in their own context.

4. Analyze Review what you learned. Did it match your hypothesis? What patterns emerged?

5. Insights Report Create appropriate documentation to communicate what you found to the team.


WEB ANALYTICS


WHAT ARE
WEB ANALYTICS?
Definition:


WHAT ARE
WEB ANALYTICS?
Definition: Web analytics is the measurement, collection, analysis and
reporting of internet data for purposes of understanding and
optimizing web usage. (Wikipedia)


THE “HOW-TO” OF
WEB ANALYTICS
1. Identify data needs What information and data are useful for defining the problem?

2. Gather data Collect that data and information within a determined time period for analysis.

Segment where relevant and cross reference the raw data to find patterns and/or
3. Analyze correlations.

4. Insights Report Review what you learned. Did it match your hypothesis? What patterns emerged?

Target unknowns and craft a qualitative research plan to fill known gaps in
5. Pinpoint knowledge gap(s) understanding. Outline how you plan to uncover the “why” of the problem.


CONTEXTUAL INQUIRY


WHAT IS CONTEXTUAL INQUIRY?

Fancy Definition:



Fancy Definition: a field data-gathering technique that studies a few carefully
selected individuals in depth to arrive at a fuller understanding of
the work practice across all customers.



- Hugh Beyer and Karen Holtzblatt - Contextual Design: Defining Customer-Centered Systems




Non-Nerd Version:




Non-Nerd Version: go watch people work in their own context.




Non-Nerd Version: go watch people work in their own context.
- Us


THE “HOW-TO” OF
CONTEXTUAL INQUIRY
1. Recruiting Contact & schedule the people you will observe. THIS IS THE MOST IMPORTANT STEP.

2. Research Plan Establish a clear focus for what you expect to find & what you hope to learn.

3. Observe Watch users completing relevant goals & tasks in their own context.

4. Analyze Review what you learned. Did it match your hypothesis? What patterns emerged?

5. Report Create appropriate documentation to communicate what you found to the team.


WHAT’S THE BENEFIT OF
CONTEXTUAL INQUIRY?
Are there current frustrations or problems with the existing design? What about with their
Issues physical environment or other systems and processes? Can the new design support those?

What are the high level priorities of the people using the current design?
Goals What are they trying to accomplish?

Tasks What are the steps people are taking to accomplish those goals?

Environment What is their physical location like? How does it impact the design or how they use it?

What other hardware or software are they using to do their work? Can (or should) they be
Applications integrated? Can the new design eliminate the need of these factors?


CONTEXTUAL INQUIRY?
Work-Arounds Are people creating ways to work around a poor design now?

Triggers What causes someone to begin down a path of completing a goal?

Are there several ways in which people are accomplishing the same goal or task? Should the
Variation(s) design support one? Both?

Partners Who does the person work with to accomplish a goal or task?

“Crutches” Do people have “cheat-sheets” or other materials to help them accomplish goals and tasks?


RECOMMENDATIONS FOR
CONTEXTUAL INQUIRY
Recruiting Be sure to observe an appropriate, representative sample of your target audience.
Aim for 3-5 participants separately.

Timeline Allow 1-2 weeks for recruiting effort. (Varies depending on the project & participant availability)
Allow 1-2 weeks for conducting the research. (Assuming 5 participants)
Allow AT LEAST 1 week for analysis
Allow 1 week to create a report.

Approximately 4-8 weeks total


CARD SORTING


WHAT IS CARD SORTING?

Definition:


WHAT IS CARD SORTING?

Definition: method of gathering data to inform the information architecture,
navigation, taxonomy and labeling of a design


THE “HOW-TO” OF
CARD SORTING
Determine who you will conduct card sorts with, write a screener to ensure you recruit your
1. Recruiting target audience from the responses & schedule the participants.

Where will the sorts take place? (remote or in person? with how many?)
2. Research Plan As before, Establish a clear focus for what you expect to find & what you hope to learn.
Choose your data, expected analysis method and sort method (open/closed sort).
2.1 Logistics Single or group sorts?

3. Conduct Sorts Conduct the card sort(s) with the target audience and selected data/content.

4. Analyze Collect your findings and perform exploratory analysis or statistical analysis (or both).

5. Report Create appropriate documentation that conveys the findings from the research.


CARD SORTING?
Navigation
Hierarchy

Grouping

Labeling

Categorization


CARD SORTING?

Informed Information Architecture


USABILITY TESTING


WHAT IS USABILITY TESTING?

Definition:


WHAT IS USABILITY TESTING?

Definition: a form of gathering feedback from actual users of a design by
having them attempt to complete intended goals and tasks with
said design.


THE “HOW-TO” OF
USABILITY TESTING
Determine who you will conduct usability testing with, write a screener to ensure you recruit
1. Recruiting your target audience from the responses & schedule the participants. Again, a CRITICAL step.

Where will the tests take place? What will the research cover?
2. Research Plan As before, Establish a clear focus for what you expect to find & what you hope to learn.

3. Conduct Tests Conduct the test (ideally with an experience moderator) & observe, while taking notes.

4. Analyze Discuss test results with any participating team members & review recordings if available.

5. Report Create appropriate documentation that conveys the findings from the research.


USABILITY TESTING?
Expectations Is the solution designed meeting expectations of those intended to use it?

Task Completion Can the users complete the available tasks of the design?

Level of difficulty If so, how difficult was it for them to complete the task(s)? Why?

Path Taken What were the steps involved in completing a given task?

Impression Did the user(s) understand the overall message and intent that the design meant to convey?


VARIATIONS OF
USABILITY TESTING
In-Person, Moderated Sessions are conducted at a physical location with a live, in-person moderator
leading the session(s) with participants.

Research is conducted via “live recruiting” from an existing website.
Remote, Moderated Participants are immediately connected with a moderator from a remote location.

Sessions are conducted using an online service that allows users to
Remote, Un-moderated participate at their convenience, without a moderator.

Usability testing done with minimal recruiting effort and logistical
Guerilla planning. Common locations are coffee shops, bars, offices, etc.


VARIATIONS OF
USABILITY TESTING
Good





VARIATIONS OF
USABILITY TESTING
Better





VARIATIONS OF
USABILITY TESTING
Best





RECOMMENDATIONS FOR
USABILITY TESTING
Do your own recruiting if possible. If this is not possible, work closely with a recruiting agency
Recruiting to ensure the participants being recruited match your target audience.
(especially for the particular research you’re doing)

Aim for 5-10 participants. (dependent on the study)

Timeline Allow 1-2 weeks for recruiting effort. (Varies depending on the recruiting method)
Allow 1 week for conducting the tests. (Assuming 5 participants)
Allow 3 days to 1 week for analysis.
Allow 3 days to 1 week to create a report.

Approximately 3-6 weeks total


WHERE TO BEGIN


STEP 1:
UNDERSTANDING BUSINESS GOALS


STEP 1:
UNDERSTANDING BUSINESS GOALS

What are the stated problems?

What impact (and implications will our design have on those goals?

How do we arrive at the core problem or root cause?


STEP 2:
DEFINING YOUR INFORMATION NEEDS


STEP 2:
DEFINING YOUR INFORMATION NEEDS
What info is necessary for you to meet those business goals?

What information do you need to successfully design?

What information do you have available?

Where are you knowledge “blind spots”?


STEP 3:
CRAFT YOUR APPROACH


STEP 3:
CRAFT YOUR APPROACH
Which methods are applicable to reach your information needs?

How will you gather information about the target audience?

How do you plan to use that information?

What are the impacts of that data in design?


ACTIVITY:
USER RESEARCH PLAN


ACTIVITY:
USER RESEARCH PLAN

30-40 minutes


ACTIVITY:
USER RESEARCH PLAN

30-40 minutes

GO!


PUTTING IT ALL TOGETHER


FICTIONAL PROJECT:
WEB APP REDESIGN


FICTIONAL PROJECT:
WEB APP REDESIGN

“We need to redesign our web application
to increase customer engagement”


FICTIONAL PROJECT:
WEB APP REDESIGN

“We need to redesign our web application
to increase customer engagement”

GO!


FICTIONAL PROJECT:
WEB APP REDESIGN
Let’s get started:


FICTIONAL PROJECT:
WEB APP REDESIGN

THE “WHAT”


FICTIONAL PROJECT:
WEB APP REDESIGN

THE “WHAT” Targeted Surveys


FICTIONAL PROJECT:
WEB APP REDESIGN

THE “WHAT” Targeted Surveys Web Analytics


FICTIONAL PROJECT:
WEB APP REDESIGN

THE “WHY”


FICTIONAL PROJECT:
WEB APP REDESIGN

THE “WHY” User Interviews


FICTIONAL PROJECT:
WEB APP REDESIGN

THE “WHY” User Interviews Contextual Inquiry


FICTIONAL PROJECT:
WEB APP REDESIGN

THE “HOW”


FICTIONAL PROJECT:
WEB APP REDESIGN

THE “HOW” Usability Testing


FICTIONAL PROJECT: WEB APP REDESIGN
Project Overview:

“WHAT” “WHY” “HOW”

Prototype
[design & development}

iteration(s)

Targeted Web User Contextual Usability
Surveys Analytics Interviews Inquiry Testing

Time


DATA ANALYSIS


STEP 1:
REVISIT YOUR RESEARCH GOALS


STEP 1:
REVISIT YOUR RESEARCH GOALS

Did you collect the information necessary to design?

Were the goals of your research met?

Have you gained an understanding of the problem space?


STEP 2:
BUILD A MODEL - DATA INTO KNOWLEDGE


STEP 2:
BUILD A MODEL - DATA INTO KNOWLEDGE
Craft a visual representation of all the raw data you collected.

Pull out patterns, problems and context.

Prioritize the issues and patterns.

Brainstorm an approach to design from your new found knowledge.


ACTIVITY:
DATA ANALYSIS


ACTIVITY:
DATA ANALYSIS

30-40 minutes


ACTIVITY:
DATA ANALYSIS

30-40 minutes

GO!


QUESTIONS?
Fire away...

or

Keep the party goin’...

@zacknaylor

@Dave_L_Jones


THANKS


Defining User Research Methodologies: A Pragmatic Approach

Recomendados

Recomendados

Mais conteúdo relacionado

Semelhante a Defining User Research Methodologies: A Pragmatic Approach

Semelhante a Defining User Research Methodologies: A Pragmatic Approach (10)

Mais de Zack Naylor

Mais de Zack Naylor (7)

Último

Último (20)

Defining User Research Methodologies: A Pragmatic Approach