Advanced usability testing - moderating

Advanced Usability Testing
Telle Zeiler & Rebecca Destello
Moderating

Telle Zeiler
Zeiler & Destello | Advanced Usability Testing - Moderating
● Program Management (1999 - 2011)
○ Microsoft vendor
○ Seattle Times
○ Nordstrom.com
● M.S. in HCDE, concentration in
user-centered design (2011)
● UX Researcher, support lecturer, UX
mentor (2011 - present)

Rebecca Destello
Front-end Web Developer
(1998 - 2011)
Bachelor of Arts
@ UW Art
(2008)
Researcher / Mentor / Teacher
(2011 - now)
Master of Science
@ UW HCDE
(2011)
The Jakob Nielsen
Experience

What we’ll cover today
Moderating a research study
● What works
● What to avoid
● How to work with the participant during the study
● Taking notes
● Unpacking data

Q. What makes a good
moderator?

Q. What makes a great
moderator?

What makes a great moderator?
● Uses good timing for prompting
● Knowing when to go deeper
● Knowing how to pull participant back from tangents
● Coming into study with the biz. goals in mind and
being able to explore when relevant/have time
● Getting data from difficult participants
● “Knows when to hold ‘em, knows when to fold ‘em”

On Listening
The first rule of interviewing is that if the respondent has something to say, the respondent must be able to
say it. If you find yourself talking over the respondent, interruption or holding the floor while the respondent
tries to interrupt, something is going wrong in the interview. You might want to withdraw some of your
attention from the respondent for a moment or two to figure out why you are competing for the floor. But
whether you figure it out or not, you ought to stop doing it.
It is easy to intrude in an interview. You can interrupt the respondent. You can finish the respondent’s
sentences. You can offer your associations to what the respondent is saying. You can suggest explanations
for observations about which the respondent is perplexed; for example, if the respondent shrugs and says,
“ I don’t know why he said that,” you could propose, “well maybe he was trying to defend himself.” You can
insist on completing your questions even if the respondent has already started to answer. You can hop
from issue to issue following your own train of thought rather than the respondent’s. With any and all of
these, don’t do it.
- Weiss, Learning from Strangers: Interviewing (p78)
All this to say…
Don’t interrupt the participant!

The role of the
Moderator

Moderating
The most important thing to remember about
moderating is this:
• Being in a study is stressful for a participant
• The participant’s comfort level and state of mind
are paramount
• When people are stressed, their behavior and
memory are impaired

Why is moderating challenging?
• Goal is unbiased feedback, but your presence causes bias
• Participants need to be comfortable, but the study situation is
often unsettling
• You need to know what the participant is thinking, but you can’t
read their minds
• It’s not like a normal conversation, you don’t
• Answer questions (usually)
• Interrupt
• Take an equal role in conversation

Golden rule: The newer
you are to moderating, the
less you should talk.
Guidelines for Moderating
• Keep yourself “small” in relationship to the
participant
• Be as neutral as possible
• Be professional
• Err on the side of taking care of the participant

How to keep them talking:
Prompting

Prompting guidelines
When prompting, you want to get the user
to elaborate on something that is unclear without
● Introducing bias
● Influencing the user’s behavior
● Taking over as the expert or primary speaker
● Unnerving the user through implication that she did
something wrong
Q. What are some ways
a moderator could
introduce bias?

Prompting techniques
Provide Acknowledgment Tokens at natural intervals
• “Mm hmm”, or “Uh huh”
Focus on questions
• What did you expect to happen?
Repeat their word or phrase back to them as a question
• They say: That was confusing
• You say: “confusing…?”
Make the alternative explicit
• Was this task easy or difficult?

Handling questions
Participants sometimes ask questions - try to turn their question
into a question.
How would you respond to the following:
● Is that right? Did I do it right?
● Am I supposed to keep going? I’ve been looking for a long time.
● What should I do now?
● Should I click here?
● What is this?

When and how to intervene
In general, it’s preferable to let participants decide
how much time to work on a task.
Some exceptions
● If they are getting very frustrated
● If you feel that you’ve got data on the task at
hand and you want to keep going and get to
other tasks

Prompting: word of warning
Remember, people cannot reliably reflect on their own
cognition. Questions like...
• “What were you thinking when…?”
• “Why do you think you…?”
...often result in bad data. Participants will try to come
up with a believable rationale while being unaware of
the true cause for their actions.

Reading non-verbal communication

Nonverbal communication
The process of communication through sending and receiving wordless
(mostly visual) cues between people.
• Posture. General body structure held in a certain way.
• Bearing. Orientation of the body.
• Body Movement. Movement over time.
• Stance. The way a person stands.
• Gestures. Expressions using arms, hands and legs.
• Facial Expressions. Expressions using face.
• Eyes. Eye movements.
• Touch. Touch and personal space.
• Voice. Tone and pitch of voice as well as speed of speech.

Let’s critique

Moderator:
● What prompting techniques does she use?
● How does she help the participant feel comfortable?
● How does she make sure she is getting good, unbiased data from the
participant?
Participant:
● Was anything she said surprising or unexpected?
● What did she say or do that required the moderator to bring her back
to the task at hand?
Usability session role play critique

Let’s debrief

Collecting data

Data collection
• Know what you are looking for
• Define success criteria
• Determine how structured to be up front

Data collection
During the usability study, you’ll need to take good,
complete, unbiased notes.
A few tips:
• The more you capture during the study, the less time
you have to spend reviewing tapes
• Get organized - use codes and shortcuts
• If you’re doing data collection - do a quick “clean up”
after each session

Data collection
Do capture
● Observations - what you see the participant do
● Quotes - what you hear the participant say
● Questions - actions or statements that were unclear or that you didn’t
capture completely
Don’t capture
● Inferences - why you think they are doing what they are doing.
● Broad generalizations that do not contain description (e.g. “She had a
lot of trouble finding the link”)

Data collection tricks
Use codes to note issues that you’re tracking.
Example:
● F = task failure
● E = error message
● H = asks for help
● P = moderator prompted
● U = gave up

Use checkboxes in your notetaking script to check off
the things you’re looking for.
Example:

Data analysis
Convert data you collect into an easy-to-analyze format
Example:

Use sticky notes or Trello.com to collect observations,
quotes, and questions.
Example:

Putting it all together
Collecting data during a study
• Setup the script to make it easier to capture data
• Use codes to capture data during the study
• Convert data into an easy to analyze format

Unpacking data: exercise

In groups of 2
• Participant A, Participant B
• Practice run through your scenario and note
potential issues along the way
• Run scenario on your partner – take notes!
• Switch
• We’ll report out as a class
Let’s try it out!

• Task Success: The ability of participants to successfully
complete the task without assistance or indicating they
would abandon the task.
• Error on Task: Identify the number of participants who
performed an action that created unexpected results.
• Confusion on Task: Identify the number of participants
who either verbalized confusion, or who re-entered
information to get proper results
Things to look for

Scenario: You and your significant other are spending a long
weekend in San Francisco March 27th – 30th and want to book a city
tour one of the days you are there.
Task: Using your smartphone book a city tour from Travelocity.com.
When you get to the checkout page you have finished the task.
At the end ask: On a scale of 1-5 with 1 being very difficult and 5
being very easy, how easy or difficult was it to complete that task?
Participant A

Scenario: You and your significant other are spending a long
weekend in San Francisco and you want to rent a compact car to get
around the city. You fly into SFO March 23rd at 10am and leave from
the same airport on March 31st at 5pm.
Task: Use your smartphone to reserve a car on Travelocity.com.
When you get to the checkout page you have finished the task.
At the end ask: On a scale of 1-5 with 1 being very difficult and 5
being very easy, how easy or difficult was it to complete that task?
Participant B

Let’s unpack the data together
• Task Success
• Error on Task
• Confusion on Task
• Quotes
• Post task ratings
Analysis!

Some words of advice

● Be user focused - as a researcher, your users are your
stakeholders - learn their needs
● Be flexible - you win some, you lose some
● Be fast - find ways to speed up your study and report out
● Be mindful - choose your words wisely
● Be nosey - get involved in product discussions and proactively
look for ways to inform the design process
● Be a partner - you are the design team’s partner
Work well with others

● Know your stakeholder’s expectations and particular needs for
consuming your research findings
● Combine and hack research methods to fit your needs
● Use a matrix figure out the # of participants you need
● Use a spreadsheet to analyze findings and look for patterns
● Find easy ways to code during your study
● Build buy-in from the start
More: Assessing interview techniques, Chauncey Wilson: http://bit.ly/2qkKZxI
Research tips

Questions?
TelleZeiler@gmail.com
RebeccaDestello@gmail.com

Advanced usability testing - moderating

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Advanced usability testing - moderating

Similar to Advanced usability testing - moderating (20)

More from Rebecca Destello

More from Rebecca Destello (6)

Recently uploaded

Recently uploaded (20)

Advanced usability testing - moderating