Exploring Usability Testing

TI
Half-day Tutorials
5/6/2014 8:30:00 AM
Exploring Usability Testing
Presented by:
Rob Sabourin
AmiBug.com
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Rob Sabourin
AmiBug.com
Rob Sabourin, P. Eng., has more than thirty years of management experience leading teams of software
development professionals. A well-respected member of the software engineering community, Rob has managed,
trained, mentored, and coached hundreds of top professionals in the field. He frequently speaks at conferences and
writes on software engineering, SQA, testing, management, and internationalization. Rob wrote I am a Bug!, the
popular software testing children's book; works as an adjunct professor of software engineering at McGill University;
and serves as the principle consultant (and president/janitor) of AmiBug.Com, Inc. Contact Rob at Contact Rob
at rsabourin@amibug.com.

© 2013 Robert Sabourin 1
© 2013 Robert Sabourin EUT v1.1 1
Robert Sabourin
President
AmiBug.Com, Inc.
Montreal, Canada
rsabourin@amibug.com
WELCOME

• Robert Sabourin ,
Software Evangelist
• President
• AmiBug.Com Inc.
• Montreal, Quebec,
Canada
• rsabourin@amibug.com
• www.amibugshare.com
Exploring Usability
Testing
• Pain points?
– What hurts?
– How Much?

SOME PHILOSOPHY
Conforming to
requirements
Suiting purpose
Delivering value
to stakeholders

Edsger W. Dijkstra
• “Program testing can be used to show the
presence of bugs, but never to show their
absence”
NON-FUNCTIONAL
TESTING

Non-Functional Testing
Quality Factors
Attributes
Characteristics
Reliability
Other “-ilities”
Challenges
Goals
• Elicit
• Articulate
• Quantify
Subjective
• Needs or wants?
• Relative or absolute?
• What is good enough?
Tests
• Tricky to orchestrate
• Difficult to interpret results
• Challenging to baseline or regress
10© 2013 Robert Sabourin EUT v1.1

Non-Functional Agile
Challenges
Time
• Analysis
• Model
• Prepare
• Run
• Interpret
Change
• Baselines
• Goals
Software
• Continuous integration
• Incomplete product builds
ISO 9126 Quality
Factors
•Attributes of the effort
needed for use, and on the
individual assessment of
such use, by a set of users
•Learnability
•Understandibility
•Operability
Usability

Quality Factors
Product Backlog
Constraints
Tom Gilb, Competitive Engineering, suggests defining
Scale: "What is measured"
Meter: "How to measure (method)"
Target: "Level we're aiming for. Success"
Constraint: "Level we're seeking to avoid.
Failure"
Benchmark: "Where we are today"

NET PROMOTER SCORE

Net Promoter Score
Net Promoter Score

Net Promoter Score
•11 values
•0 not at all likely
•10 extremely likely
Response Scale
Net Promoter Score
• respondents giving a 9 or 10 score
Promoters
• respondents giving a 7 or 8 score
Passives
• respondents giving a 0 to 6 score
Detractors

Net Promoter Score
•The difference between the percentage
of Promoters and Detractors
•Is not expressed as a percentage
•Is an absolute number lying between
-100 and +100
•uTest suggests “any positive number is
good” (ref: TK Maxx Mobile Usability
Testing Results Report)
NPS
Net Promoter Score

SYSTEM USABILITY SCALE

System Usability Scale
• Measures usability in terms of
effectiveness, efficiency and satisfaction
• SUS can be read as a percentage
• SUS can be used as a benchmarking
mechanism
• SUS can be used to compare the current
system to competitors’ systems
• SUS can be used to compare the current
system to future versions
SUS - System Usability Scale

1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought the system was easy to use
4. I think that I would need the support of a technical
person to be able to use this system
5. I found the various functions in this system were well
integrated
6. I thought there was too much inconsistency in this
system
7. I would imagine that most people would learn to use
this system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get
going with this system
SUS Questions
SUS Responses

• For odd items: subtract one from the user
response.
• For even-numbered items: subtract the
user responses from 5
• This scales all values from 0 to 4 (with
four being the most positive response).
• Add up the converted responses for each
user and multiply that total by 2.5. This
converts the range of possible values from
0 to 100 instead of from 0 to 40.
Scoring SUS
Scoring SUS

EXPLORATORY TESTING
Exploratory Testing
© 2013 Robert Sabourin EUT v1.1
Learning
DesignExecution
32

Charter Statement
• Statement of mission
• Ties to purpose
• Focuses work
• Confirms understanding
• Delineates scope
• Analogy to test story
Charter Statement
• Short, terse
• To the point
• Inclusions
• Exclusions
• Limits

eBay Usability Charters

Pairing Testers With …
Non Functional Testing
•Usability Gurus
•SMEs
•Human Factors
Experts
Heuristics

Heuristics
Guide
Exploration
Rules of
thumb
Fallible but
useful
WHITEBOARDING

Prototype Styles
• Paper and pencil
• Comic books style story boards
• Wire frame models
• Mock ups
• Prototypes
• Site maps
• User flow
• Wire flow
• Story boards
• Wire frame
• Page description diagram
• Functional specification
Whiteboarding

Whiteboarding
What is Whiteboarding?
Collaboration
Coordination
Conceptualization
Tony Buzan
Mind Maps

Whiteboarding
Whiteboarding
UserMindMap

Buying a Book
Usage Scenarios
Whiteboarding
StoryBoard

Wrap-O-Matic
Usage Scenarios
Scenario Based Testing

Buying a Book
Usage Scenarios
© 2013 Robert Sabourin
EUT v1.1
Slide 54

© 2013 Robert Sabourin
EUT v1.1
Slide 55

Social Networking
Usage Scenarios

TOFT Testing
• Task Oriented
Functional Testing
– Can the user
accomplish useful
tasks correctly?
© 2013 Robert Sabourin EUT v1.1 Slide 67
USABILITY HEURISTICS

Jakob Nielsen
• Usability heuristics
• Rules of thumb
• General principles
of user interface
design
• www.nngroup.com
Usability Heuristics
• Visibility of system status
– Always keep users informed
– Provide appropriate feedback
– Respond in a reasonable timeframe

• Match between system and the real
world
– Speak the users' language
– Use familiar concepts
– Follow a natural order
• User control and freedom
– Provide means to exit from unwanted states
– Provide undo and redo capabilities

• Consistency and standards
– Same words mean same thing in different
contexts
– Follow environment conventions
• Error prevention
– Prevents problems from occurring
– Eliminate error-prone conditions
– Have a confirmation option

• Recognition rather than recall
– Making objects visible
– Make instructions available
• Flexibility and efficiency of use
– Accelerators for the expert user
– Make common options adaptable

• Aesthetic and minimalist design
– Dialogues should contain
relevant information
• Help users recognize, diagnose, and
recover from errors
– Use plain language for error messages
– Constructively suggest solutions

• Help and documentation
– Easy to search
– Focused on the user's task
– Offer concrete steps to be carried out
– Short and to the point
USABILITY SEVERITY
LEVELS

Usability Severity
ISO 9421

Usability ISO 9421
•the extent to which the intended goals of use are achieved
Effectiveness
•the resources that have been expended to achieve the intended goals
Efficiency
•the extent to which the user finds the use of the product acceptable
Satisfaction
•can user be harmed
Safety
Usability ISO 9421
•the extent to which the system is complete in design and meets its desired
goals
Completeness
•Is the capability of a software product to enable the user to learn how to
use it
Learn-ability
•the extent to which the product fits into normal course of daily life
Convenience
•Confusing or vague usability. Misleading information that might also result
in providing no usability at all
Ambiguity

USABILITY TEST
FRAMEWORK
Usability Test Framework
Basic Usability Test Steps
• Select representative users
• Complete defined set of tasks
• Facilitator observes user behaviour
• Interview user after task is completed
• User is asked to think aloud (recorded)
• Review findings to identify concerns
• Generate actionable recommendations

User groups
•Select 5 or 6 users for each group
•All should attempt the same tasks
•Complex applications may need many groups
•Groups can be based
•Domain (subject matter) expertise
•Experience level (task or professional)
•Solution familiarity
Where to perform usability tests?
• Control lab
• Conference room with recording
equipment
• Work area with
• Local
• Distributed with local observer
• Distributed without local observer

Prepare Usability Scenario
•Identify project usability test objectives
•Select relevant user tasks
•Keep user instructions short and to the point
•Do not over complicate instructions
•Use the language of the user not the
language of the system
•Make task parameters specific
•Use a logical flow
Running a test
•Let participants make mistakes
•Take note of questions
•Take note of decisions points
•Take note of alternative
selections

Think Aloud
Approach
• the user
verbalizes actions
and decisions
• user verbalizes
questions
concerns or
ambiguities
Retrospective
Think Aloud
• records and eye
tracking software
record event
• review with user
and ask them to
explain their
actions
Reporting Findings
• List of usability concerns
• Summary of user questions
• Description of users behaviour
• Tasks of concern
• Too long
• Awkward
• Unsuccessful
• Time to complete
• Feedback
• Expected
• Unexpected

Measures and metrics
•Effectiveness
•Efficiency
•Satisfaction
•Error frequency
•Memorability
Measures and metrics
• Successful task completion
• Error rates
• Time on task
• Subjective comments
• Likes
• Dislikes
• Recommendations

Facilitating a test
• Make participant comfortable with environment and
technology
• Give participants a change to ask questions
• Explain process
• Ask participants about the amount of experience they
have with the technology
• Ask participants about the amount of experience they
have with the domain
• Review task description with participant
• Explain facilitators involvement
FORMS AND WORKFLOWS

Form Design Principles
• Minimize pain
• Illuminate path
• Consider context
• Ensure consistent communication
Luke Wroblewski
Web Form Design
Eye Tracking

Form Usability
• Number and location of errors
• Severity of errors
• Completion rates
• Time to complete forms
• Satisfaction scores
• Subjective comments
Awkward Form Path

Clear Form Path
Form Usability
• Eye tracking
– What people looked at
– Number of eye fixations
– Length of eye fixations

Form Usability Checklist
Any unnecessary questions
Succinct questions
Sincere voice
Natural language
Logical groups of information
Structure form as a conversation
Ask optional questions separately
Form Usability Checklist
Name the form
Name sections of the form
Clear scan line
Spacing between sections
Minimal distractions
Start page guidance for long forms
Indicate progress

Progress Indication
Input Validation
Suggest valid answers
Illustrate valid format
Validate syntax after user enters data
Indicate limits or ranges of inputs
Provide smart defaults
Hide unneeded controls

Error Messages
Provide clear error messages
Indicate if error blocks completion
Guide users to resolve error
Provide visual emphasis of error
Use red text and icons for errors
Indicate successful completion
DO IT YOURSELF
USABILITY TESTING

Do it yourself
Do it yourself
A morning a month
Get insights from users
Watch real users

Do it yourself
Recruit diversely
Find important problems
Improve design
Do it yourself
Three testers
Test on site
Observe shared screens

Do it yourself
Observers take notes
User questions recorded
Action decided at lunch
SOME MOBILE USABILITY
CONCERNS

Mobile Usability
•Small Screens
•Limited bandwidth
•Fingers
•No Flash Like GUI
Difficulties
Mobile Usability
•Bad first impression
•Operator errors
•Negative feedback / reviews
•Competitor fills gap
Risks

Mobile Usability
• Important stuff on top
• Easy to find stuff
• Single column layout
• Minimize navigation
• Light data
Guidelines
Mobile Usability
• Finger taps
• Avoid fancy web gui technologies
• Minimize forms
• Minimize feature sets
• Separate mobile web resources
• Fast
Guidelines

Mobile Usability
•In the lab
•Beta
•Crowd source
Mobile Usability Test
Mobile Usability
• Vary and control
• Technologically
• OS
• Browser
• Device
• Carrier
• Bandwidth
Mobile Usability Test

Mobile Usability
• Geographically
• Continent
• Country
• City
• Language
• Locale
Vary and control
Mobile Usability
• Demographically
• Age
• Gender
• Education
• Employment
• Industry
• Computer savvy
• Domain expertise
Vary and control

LEAN UX
Lean UX

Lean UX
• Cross functional teams
• Small teams
• Dedicated teams
• Collocated teams
• Progress equals Outcomes
• Problem focused teams
Lean Principles
Lean UX
• Remove waste
• Small batch size
• Continuous discovery
• GOOB Get out of the building
• Shared understanding
• No Rock Stars Gurus or Ninjas
Lean Principles

Lean UX
• Externalize Your Work
• Making over Analysis
• Learning over Growth
• Permission to Fail
• Getting out of the Deliverables
Business
Lean Principles
Lean UX
Assumptions
Hypotheses
Outcomes
Personas
Features

Lean UX
•Business
•User
Assumptions
Lean UX
We believe that
creating this
experience for these
personas will achieve
this outcome. We will
know this to be true
when we see this
feedback or measure.
Hypotheses

Lean UX
Outcomes
More business
More referrals
More fun
Lean UX
Personas
• Sketch and Name
• Demographic
• Pain points & needs
• Potential solutions

Lean UX
•Capabilities
•Tactics
•Products
Features are
Lean UX
•Business
•Customer
•User
Features serve

Lean UX
Features drive
customer
behaviour
Lean UX
Process

Thank You
• Questions?

Exploring Usability Testing

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (13)

Similar to Exploring Usability Testing

Similar to Exploring Usability Testing (20)

More from TechWell

More from TechWell (20)

Recently uploaded

Recently uploaded (20)

Exploring Usability Testing