The Note-Taker’s Perspective During Usability Testing: Recognizing What’s Important, What’s Not

www.userworks.com
(301) 431-0500
Kristen Davis & Dick Horst
UXPA International Conference 2013
July 10, 2013
The Note-Taker's PerspectiveThe Note-Taker's Perspective
During Usability Testing:During Usability Testing:
Recognizing What's Important, What'sRecognizing What's Important, What's
NotNot

The Note-Taker's Perspective During Usability Testing
Kristen Davis and Dick Horst - UserWorks, Inc.
OverviewOverview
Presentation Objectives:
To collectively explore ways to become a more efficient and effective
note-taker by:
• Developing the skills and planning necessary
• Recognizing what’s important to note and what’s not
• Understanding the pros and cons of various note-taking styles
• Determining what metrics to define and utilize in note-taking
• Creating a list of note-taking best practices and tips
2

Why is Effective Note-TakingWhy is Effective Note-Taking
Important?Important?
 To improve the data analysis process by:
• Capturing the appropriate data the first time; less need for reviewing
recordings
• Making analyses more efficient by categorizing observations on the fly
• Increasing accuracy and completeness of notes (so you don’t miss
anything important)
• Identifying trends across participants
3

Typical ContextTypical Context
 One-on-one, moderated sessions (with the participant in-person or
remote)
 Moderator interacting with the participant, note-taker in the
background taking notes
 Participant attempting typical task scenarios at the direction of
the moderator
 Participant thinking aloud
Many of the same principles and best practices would also apply to
taking notes in focus groups, user interviews, or ethnographic
settings, but not all
4

ExerciseExercise
 Think about and/or record your observations as you watch this
video clip
6

DiscussionDiscussion
 What did you note?
 Did you capture what you thought were the key findings?
 Did you miss anything?
7

Note-Taking MisconceptionsNote-Taking Misconceptions
 No preparation needed
 Take the notes, then figure out how to analyze and summarize the
data
 Anyone can do it
 Attention to detail, fast typing are the key skills
 A one-size-fits-all approach will work for all practitioners and for all
products under evaluation
8

The ContextThe Context
 Product: NIHSeniorHealth.gov website
 Participant: Seniors who look for health information online
 Task: What types of food or drink are related to balance problems?
 Study goals:
• What navigation path do participants use when looking for specific
pieces of information?
• Do participants notice and use the pagination?
• Does the visual treatment used in the left hand navigation clearly
indicate location within a specific health topic?
 Task goals:
 Are participants able to locate information on the 2nd
page?
 Are the left hand navigation labels clear?
10

ExerciseExercise
video clip
11

 Did you capture the key findings?
 Did you note any of these:
• Study goals:
 What navigation path do participants use when looking for specific
pieces of information?
 Do participants notice and use the pagination?
 Does the visual design treatment used in the left hand navigation
clearly indicate location within the topic?
• Task goals:
 Are participants able to locate information on the 2nd
page?
 Are the left hand navigation labels clear?
12

Lessons Learned: Understand theLessons Learned: Understand the
GoalsGoals
 Important to be familiar with the product and understand the
overall goals of the study and the specific goals of the tasks
 Goals should be set prior to data collection in order to:
• Record appropriate data
• Target the type of data that will be useful when summarizing and
reporting the findings
13

Different Note-Taking ApproachesDifferent Note-Taking Approaches
 Would you try to note:
• Everything the participant says?
• Everything the participant does?
 Pages visited
 Links visited
 Scrolling, hovering
 Participant’s demeanor
 Non-verbal signs
 Or would you be more selective?
• Task scoring (success vs failure)
• Errors, wrong paths
• Pre-identified issues of interest
• Occasional verbatims
14

Different Note-Taking StylesDifferent Note-Taking Styles
 Moderator also takes notes
 The court stenographer, verbatim
 High level interpretation only, not behavior or comment
 Coding behaviors in addition to free form notes
 Working from a checklist
 Multiple note-takers looking for different things
 Multiple note-takers working redundantly
 Taking notes from playback of a video/audio recording of the
session
 Using a tool like Morae, Ovo, Silverback, free form typing into a
word processing or spreadsheet program, or paper-based
(checklist)
15

Practice ExercisePractice Exercise
 Product: HealthIT.gov website
 Participant: Private physician
 Task: Medical professionals and hospitals are encouraged to
participate in the Government EHR reimbursement programs, early,
to receive the maximum payment. Find out when the Government will
start or has started these reimbursement payments.
 Study goals:
• Are the labels used for navigation clear?
• Does the information architecture make sense?
• What suggestions do participants have to improve the content?
 Task goal:
• What navigation path do participants use when looking for specific pieces of
information?
17

ExerciseExercise
video clip
18

19

Lessons Learned: RecordLessons Learned: Record Only KeyOnly Key
PointsPoints
 The method of note-taking impacts the type of data collected
during the evaluation
 Depending on the evaluation objective, one style or a combination
of styles may be more appropriate
20

Challenges for the Note-taker (andChallenges for the Note-taker (and
Moderator)Moderator)
 Participant says one thing and does another
 Participant thinks they have been successful when they really
haven’t
 Participant doesn’t want to look bad, so bluffs
 Participant initially struggles, then succeeds, and speaks highly of
the product
 Participant blames self and rates product highly despite
disastrous task performance
 Participant is overly chatty and goes off on tangents
21

More Challenges for the Note-TakerMore Challenges for the Note-Taker
 Being selective in the information you note, without introducing
your biases (seeing what you want/expect to see)
 Noting participant behavior versus comments
 Whether to try to capture participant’s clickpath, menu choices,
data entry
 Whether to capture timing
 Whether to use a shorthand “code” for observations
 Whether to capture participant’s non-verbal behavior
 Whether to note good clips for a highlights video
22

Practice ExercisePractice Exercise
 Product: Blood glucose meter
 Participant: Person with diabetes who uses a meter
 Task: Change default settings in the meter
 Rating scale for scoring task completion:
• Completed with ease: Participant easily completed the task (Score: 3)
• Completed with minor difficulty: Participant somewhat struggled to
complete the task (e.g., attempted more than two paths) (Score: 2)
• Completed with major difficulty: Participant significantly struggled
to complete the task (e.g., attempted numerous paths and/or used
customer service) (Score: 1)
• Failed to complete: Participant was unable to complete the task
(Score: 0)
24

ExerciseExercise
video clip
25

 What score did you give the participant?
• Completed with ease: Participant easily completed the task (Score: 3)
• Completed with minor difficulty: Participant somewhat struggled to
complete the task (e.g., attempted more than two paths) (Score: 2)
• Completed with major difficulty: Participant significantly struggled to
complete the task (e.g., attempted numerous paths and/or used customer
service) (Score: 1)
• Failed to complete: Participant was unable to complete the task (Score: 0)
26

Lessons Learned:Lessons Learned: Define MetricsDefine Metrics
 Using a task completion rating scale allows:
• Participant performance to be categorized
• Note-taker to record the scores in a systematic and consistent way
 Carefully defining the performance categories before the fact
helps the note-taker
• Know what to look for
• Efficiently categorize on the fly
 Some additional metrics to consider (facilitated by commercial
data logging software):
• Clickpaths
• Time on task
• Shorthand codes for behaviors or incidents
 Example: Number of times a participant clicked the pagination
27

"Take-Aways" for Effective Note-"Take-Aways" for Effective Note-
TakingTaking
 Create a mindset that values the role of the note-taker
 Choose someone with appropriate education and experience to
appreciate the design issues of interest
 Include the note-taker in test planning discussions and debriefs
 Establish project-specific note-taking procedures and coding
schemes; anticipate analysis needs when planning note-taking
 Be sure the note-taker is thoroughly familiar with the product
interface, task scenarios, and project goals
29

And a Few Tips for the EffectiveAnd a Few Tips for the Effective
Note-TakerNote-Taker
 Focus on what the participant does, not just what they say
 Strive to interpret what you are seeing on the fly and take notes
about that, not just “raw” observations
 Review your notes and fill in any gaps as soon as possible after
the session while observations are fresh in your mind
 Consult with the team and client mid-course to be sure your notes
reflect project goals and team intent
 Check available storage space on recording devices before each
session
 Spot check recordings throughout the study to ensure that
everything is working as intended
30

Questions/Comments?Questions/Comments?
Kristen Davis
UserWorks, Inc.
User Experience Specialist/
Note-Taker Extraordinaire
kdavis@userworks.com
31
Dick Horst
UserWorks, Inc.
President/
Principal User Experience
Specialist
dhorst@userworks.com
www.userworks.com
(301) 431-0500

The Note-Taker’s Perspective During Usability Testing: Recognizing What’s Important, What’s Not

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a The Note-Taker’s Perspective During Usability Testing: Recognizing What’s Important, What’s Not

Semelhante a The Note-Taker’s Perspective During Usability Testing: Recognizing What’s Important, What’s Not (20)

Mais de UXPA International

Mais de UXPA International (20)

Último

Último (20)

The Note-Taker’s Perspective During Usability Testing: Recognizing What’s Important, What’s Not

Notas do Editor