1.
Advanced Usability Testing
Telle Zeiler & Rebecca Destello
Moderating

2.
Telle Zeiler
Zeiler & Destello | Advanced Usability Testing - Moderating
● Program Management (1999 - 2011)
○ Microsoft vendor
○ Seattle Times
○ Nordstrom.com
● M.S. in HCDE, concentration in
user-centered design (2011)
● UX Researcher, support lecturer, UX
mentor (2011 - present)

3.
Rebecca Destello
Zeiler & Destello | Advanced Usability Testing - Moderating
Front-end Web Developer
(1998 - 2011)
Bachelor of Arts
@ UW Art
(2008)
Researcher / Mentor / Teacher
(2011 - now)
Master of Science
@ UW HCDE
(2011)
The Jakob Nielsen
Experience

4.
What we’ll cover today
Moderating a research study
● What works
● What to avoid
● How to work with the participant during the study
● Taking notes
● Unpacking data
Zeiler & Destello | Advanced Usability Testing - Moderating

5.
Zeiler & Destello | Advanced Usability Testing - Moderating
Q. What makes a good
moderator?

6.
Zeiler & Destello | Advanced Usability Testing - Moderating
Q. What makes a great
moderator?

7.
What makes a great moderator?
● Uses good timing for prompting
● Knowing when to go deeper
● Knowing how to pull participant back from tangents
● Coming into study with the biz. goals in mind and
being able to explore when relevant/have time
● Getting data from difficult participants
● “Knows when to hold ‘em, knows when to fold ‘em”
Zeiler & Destello | Advanced Usability Testing - Moderating

8.
On Listening
The first rule of interviewing is that if the respondent has something to say, the respondent must be able to
say it. If you find yourself talking over the respondent, interruption or holding the floor while the respondent
tries to interrupt, something is going wrong in the interview. You might want to withdraw some of your
attention from the respondent for a moment or two to figure out why you are competing for the floor. But
whether you figure it out or not, you ought to stop doing it.
It is easy to intrude in an interview. You can interrupt the respondent. You can finish the respondent’s
sentences. You can offer your associations to what the respondent is saying. You can suggest explanations
for observations about which the respondent is perplexed; for example, if the respondent shrugs and says,
“ I don’t know why he said that,” you could propose, “well maybe he was trying to defend himself.” You can
insist on completing your questions even if the respondent has already started to answer. You can hop
from issue to issue following your own train of thought rather than the respondent’s. With any and all of
these, don’t do it.
- Weiss, Learning from Strangers: Interviewing (p78)
All this to say…
Don’t interrupt the participant!
Zeiler & Destello | Advanced Usability Testing - Moderating

9.
The role of the
Moderator
Zeiler & Destello | Advanced Usability Testing - Moderating

10.
Moderating
The most important thing to remember about
moderating is this:
• Being in a study is stressful for a participant
• The participant’s comfort level and state of mind
are paramount
• When people are stressed, their behavior and
memory are impaired
Zeiler & Destello | Advanced Usability Testing - Moderating

11.
Why is moderating challenging?
• Goal is unbiased feedback, but your presence causes bias
• Participants need to be comfortable, but the study situation is
often unsettling
• You need to know what the participant is thinking, but you can’t
read their minds
• It’s not like a normal conversation, you don’t
• Answer questions (usually)
• Interrupt
• Take an equal role in conversation
Zeiler & Destello | Advanced Usability Testing - Moderating

12.
Golden rule: The newer
you are to moderating, the
less you should talk.
Guidelines for Moderating
• Keep yourself “small” in relationship to the
participant
• Be as neutral as possible
• Be professional
• Err on the side of taking care of the participant
Zeiler & Destello | Advanced Usability Testing - Moderating

13.
How to keep them talking:
Prompting
Zeiler & Destello | Advanced Usability Testing - Moderating

14.
Prompting guidelines
When prompting, you want to get the user
to elaborate on something that is unclear without
● Introducing bias
● Influencing the user’s behavior
● Taking over as the expert or primary speaker
● Unnerving the user through implication that she did
something wrong
Q. What are some ways
a moderator could
introduce bias?
Zeiler & Destello | Advanced Usability Testing - Moderating

15.
Prompting techniques
Provide Acknowledgment Tokens at natural intervals
• “Mm hmm”, or “Uh huh”
Focus on questions
• What did you expect to happen?
Repeat their word or phrase back to them as a question
• They say: That was confusing
• You say: “confusing…?”
Make the alternative explicit
• Was this task easy or difficult?
Zeiler & Destello | Advanced Usability Testing - Moderating

16.
Handling questions
Participants sometimes ask questions - try to turn their question
into a question.
How would you respond to the following:
● Is that right? Did I do it right?
● Am I supposed to keep going? I’ve been looking for a long time.
● What should I do now?
● Should I click here?
● What is this?
Zeiler & Destello | Advanced Usability Testing - Moderating

17.
When and how to intervene
In general, it’s preferable to let participants decide
how much time to work on a task.
Some exceptions
● If they are getting very frustrated
● If you feel that you’ve got data on the task at
hand and you want to keep going and get to
other tasks
Zeiler & Destello | Advanced Usability Testing - Moderating

18.
Prompting: word of warning
Remember, people cannot reliably reflect on their own
cognition. Questions like...
• “What were you thinking when…?”
• “Why do you think you…?”
...often result in bad data. Participants will try to come
up with a believable rationale while being unaware of
the true cause for their actions.
Zeiler & Destello | Advanced Usability Testing - Moderating

19.
Reading non-verbal communication
Zeiler & Destello | Advanced Usability Testing - Moderating

20.
Nonverbal communication
The process of communication through sending and receiving wordless
(mostly visual) cues between people.
• Posture. General body structure held in a certain way.
• Bearing. Orientation of the body.
• Body Movement. Movement over time.
• Stance. The way a person stands.
• Gestures. Expressions using arms, hands and legs.
• Facial Expressions. Expressions using face.
• Eyes. Eye movements.
• Touch. Touch and personal space.
• Voice. Tone and pitch of voice as well as speed of speech.
Zeiler & Destello | Advanced Usability Testing - Moderating

21.
Let’s critique
Zeiler & Destello | Advanced Usability Testing - Moderating

22.
Zeiler & Destello | Advanced Usability Testing - Moderating
Moderator:
● What prompting techniques does she use?
● How does she help the participant feel comfortable?
● How does she make sure she is getting good, unbiased data from the
participant?
Participant:
● Was anything she said surprising or unexpected?
● What did she say or do that required the moderator to bring her back
to the task at hand?
Usability session role play critique

23.
Let’s debrief
Zeiler & Destello | Advanced Usability Testing - Moderating

24.
Collecting data
Zeiler & Destello | Advanced Usability Testing - Moderating

25.
Data collection
• Know what you are looking for
• Define success criteria
• Determine how structured to be up front
Zeiler & Destello | Advanced Usability Testing - Moderating

26.
Data collection
During the usability study, you’ll need to take good,
complete, unbiased notes.
A few tips:
• The more you capture during the study, the less time
you have to spend reviewing tapes
• Get organized - use codes and shortcuts
• If you’re doing data collection - do a quick “clean up”
after each session
Zeiler & Destello | Advanced Usability Testing - Moderating

27.
Data collection
Do capture
● Observations - what you see the participant do
● Quotes - what you hear the participant say
● Questions - actions or statements that were unclear or that you didn’t
capture completely
Don’t capture
● Inferences - why you think they are doing what they are doing.
● Broad generalizations that do not contain description (e.g. “She had a
lot of trouble finding the link”)
Zeiler & Destello | Advanced Usability Testing - Moderating

28.
Data collection tricks
Use codes to note issues that you’re tracking.
Example:
● F = task failure
● E = error message
● H = asks for help
● P = moderator prompted
● U = gave up
Zeiler & Destello | Advanced Usability Testing - Moderating

29.
Data collection tricks
Use checkboxes in your notetaking script to check off
the things you’re looking for.
Example:
Zeiler & Destello | Advanced Usability Testing - Moderating

30.
Data analysis
Convert data you collect into an easy-to-analyze format
Example:
Zeiler & Destello | Advanced Usability Testing - Moderating

31.
Data collection tricks
Use sticky notes or Trello.com to collect observations,
quotes, and questions.
Example:
Zeiler & Destello | Advanced Usability Testing - Moderating

32.
Putting it all together
Collecting data during a study
• Setup the script to make it easier to capture data
• Use codes to capture data during the study
• Convert data into an easy to analyze format
Zeiler & Destello | Advanced Usability Testing - Moderating

33.
Unpacking data: exercise
Zeiler & Destello | Advanced Usability Testing - Moderating

34.
In groups of 2
• Participant A, Participant B
• Practice run through your scenario and note
potential issues along the way
• Run scenario on your partner – take notes!
• Switch
• We’ll report out as a class
Zeiler & Destello | Advanced Usability Testing - Moderating
Let’s try it out!

35.
• Task Success: The ability of participants to successfully
complete the task without assistance or indicating they
would abandon the task.
• Error on Task: Identify the number of participants who
performed an action that created unexpected results.
• Confusion on Task: Identify the number of participants
who either verbalized confusion, or who re-entered
information to get proper results
Zeiler & Destello | Advanced Usability Testing - Moderating
Things to look for

36.
Scenario: You and your significant other are spending a long
weekend in San Francisco March 27th – 30th and want to book a city
tour one of the days you are there.
Task: Using your smartphone book a city tour from Travelocity.com.
When you get to the checkout page you have finished the task.
At the end ask: On a scale of 1-5 with 1 being very difficult and 5
being very easy, how easy or difficult was it to complete that task?
Zeiler & Destello | Advanced Usability Testing - Moderating
Participant A

37.
Scenario: You and your significant other are spending a long
weekend in San Francisco and you want to rent a compact car to get
around the city. You fly into SFO March 23rd at 10am and leave from
the same airport on March 31st at 5pm.
Task: Use your smartphone to reserve a car on Travelocity.com.
When you get to the checkout page you have finished the task.
At the end ask: On a scale of 1-5 with 1 being very difficult and 5
being very easy, how easy or difficult was it to complete that task?
Zeiler & Destello | Advanced Usability Testing - Moderating
Participant B

38.
Let’s unpack the data together
• Task Success
• Error on Task
• Confusion on Task
• Quotes
• Post task ratings
Zeiler & Destello | Advanced Usability Testing - Moderating
Analysis!

39.
Some words of advice
Zeiler & Destello | Advanced Usability Testing - Moderating

40.
● Be user focused - as a researcher, your users are your
stakeholders - learn their needs
● Be flexible - you win some, you lose some
● Be fast - find ways to speed up your study and report out
● Be mindful - choose your words wisely
● Be nosey - get involved in product discussions and proactively
look for ways to inform the design process
● Be a partner - you are the design team’s partner
Work well with others
Zeiler & Destello | Advanced Usability Testing - Moderating

41.
● Know your stakeholder’s expectations and particular needs for
consuming your research findings
● Combine and hack research methods to fit your needs
● Use a matrix figure out the # of participants you need
● Use a spreadsheet to analyze findings and look for patterns
● Find easy ways to code during your study
● Build buy-in from the start
More: Assessing interview techniques, Chauncey Wilson: http://bit.ly/2qkKZxI
Research tips
Zeiler & Destello | Advanced Usability Testing - Moderating

42.
Questions?
TelleZeiler@gmail.com
RebeccaDestello@gmail.com
Zeiler & Destello | Advanced Usability Testing - Moderating