2. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Goal of Usability Assessments
• Can users complete tasks?
• Appropriateness of mental models
• Comparative efficiency
• Subjective satisfaction
• How do we assess?
3. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
No bright lines
Design
Functional
systemPaper
Prototype
Release
Usability Inspections
Usability Studies
Empirical User Studies, Case Studies,
Longitudinal Studies, Acceptance
Tests
Low
cost,
low
validity Higher
cost,
validity
4. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Usability Inspections
• “clean-room” static examination of usability
• Methodically scrutinize interfaces in search of potential
problems
• Pros:
– Inexpensive – no users, relatively easy
– Identify major issues at a relatively early stage
• Cons:
– May miss problems: generally find < 50%
– All results are hypothetical – don’t know which problems might
really lead to errors
5. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Common Questions across inspection
types
●
Who inspects?
●
Background
●
How many?
●
Which tasks?
●
How to interpret findings?
●
Problems are not “real” problems experience by
users
●
How important are they?
6. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Broad classes of inspections
Heuristic Evaluations: How well does an interface conform
to guidelines for interface design?
Walkthrough: Analytic examination of interface and
interaction requirements, usually informed by some model
of the user
Many variants...
7. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Who inspects?
Heuristic inspection
Usability experts
Domain experts
Combination? (Double experts)
Users should participate as users
when possible
3-5 experts? (Nielsen)
Or more...
Work alone, or in teams..
Walkthroughs
May require more cognitive
background
Domain expert feedback
helpful
Conducted by a team?
8. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Tasks
Heuristic inspections
Set tasks
Open-ended exploration
Walkthroughs
Generally, specific tasks
9. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Nielsen's Heuristics
Nielsen, 1994 - http://www.nngroup.com/articles/ten-usability-heuristics/
• Visibility of system status
• Match between system and real world
• User control and freedom
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of use
• Aesthetic and minimalist design
• Help users recognize, diagnose, and recover from errors
• Help and documentation
10. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Results
Nielsen and Molich 1990 , Nielsen 1993
• Few false positives
• Find < 50% of errors
• “In general, we would expect aggregates of five
evaluators to find about two thirds of the usability
problems which is really quite good for an informal and
inexpensive technique like heuristic evaluation.”
12. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Results
Nielsen and Molich 1990
Source of the “you only need 5 evaluators” rule…
Hotly contested…
13. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Which Evaluators?
Nielsen, 1992
• “Double Experts” - Domain and Usability expertise – considered
best
• Not always available – use some of each, or teams
14. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Heuristic Evaluation Procedure
Evaluators work alone
(except for when they work in teams)
Optional observer can help explain confusing issues and to
record issues.
Go through interface several times
List specific tasks
Note discrepancies between interface and heuristic
Note any concerns - even those that don’t exactly fit heuristics
Individual evaluators meet to aggregate results
agree on interpretation and prioritization
15. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Nielsen's Heuristics
Nielsen, 1994 - http://www.nngroup.com/articles/ten-usability-heuristics/
• Visibility of system status
•“The system should always keep users informed about what is going on,
through appropriate feedback within reasonable time”
• Match between system and real world
“The system should speak the users’ language, with words, phrases, and
concepts familiar to the user, rather than system-oriented terms. Follow real-
world conventions, making information appear in a natural and logical order.”
• User control and freedom
“Users often choose system functions by mistake and will need a clearly marked
"emergency exit" to leave the unwanted state without having to go through an
extended dialogue. Support undo and redo.”
16. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Nielsen's Heuristics
Nielsen, 1994 - http://www.nngroup.com/articles/ten-usability-heuristics/
• Consistency and standards
“Users should not have to wonder whether different words, situations,
or actions mean the same thing. Follow platform conventions.”
• Error prevention
“Even better than good error messages is a careful design which prevents a
problem from occurring in the first place. Either eliminate error-prone
conditions or check for them and present users with a confirmation option
before they commit to the action.”
• Recognition rather than recall
“Minimize the user's memory load by making objects, actions, and options
visible. The user should not have to remember information from one part of
the dialogue to another. Instructions for use of the system should be visible or
easily retrievable whenever appropriate.”
17. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Nielsen's Heuristics
Nielsen, 1994 - http://www.nngroup.com/articles/ten-usability-heuristics/
• Flexibility and efficiency of use
“Accelerators -- unseen by the novice user -- may often speed up the
interaction for the expert user such that the system can cater to both
inexperienced and experienced users. Allow users to tailor frequent
actions.”
• Aesthetic and minimalist design
• “Dialogues should not contain information which is irrelevant or
rarely needed. Every extra unit of information in a dialogue competes
with the relevant units of information and diminishes their relative
visibility.”
18. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Nielsen's Heuristics
Nielsen, 1994 - http://www.nngroup.com/articles/ten-usability-heuristics/
• Help users recognize, diagnose, and recover from errors
“Error messages should be expressed in plain language (no codes), precisely
indicate the problem, and constructively suggest a solution.”
• Help and documentation
“Even though it is better if the system can be used without documentation, it
may be necessary to provide help and documentation. Any such information
should be easy to search, focused on the user's task, list concrete steps to be
carried out, and not be too large.”
19. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
How to Interpret?
Use severity judgments to prioritize fixes
Frequency of problem
Impact of problem
Persistence – will users be repeatedly bothered?
Multiple independent raters increase reliability
Bigger questions – does this design work at all?
As with usability studies, try to generalize
Don't solve lots of small problems if the design is inherently problematic
20. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Severity Ratings
0. No problem
1. Cosmetic
2. Minor
3. Major
4. Catastrophe
Measure both impact and frequency?
21. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Reporting heuristic violationshttp://ocw.mit.edu/courses/electrical-engineering-and-computer-science/
6-831-user-interface-design-and-implementation-spring-2011/lecture-notes/MIT6_831S11_lec23.pdf
• Goal - communicate problems
• Identify successes
• Be constructive
• Be specific
22. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Heuristic Evaluation reportshttp://ocw.mit.edu/courses/electrical-engineering-and-computer-science/
6-831-user-interface-design-and-implementation-spring-2011/lecture-notes/MIT6_831S11_lec23.pdf
• Problem
• Heuristic
• Description
• Severity
• Recommendations
• Screenshot
24. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Example: NIH Undiagnosed
Disease Program UDPICs LIMS tool
Heuristic Visibility of system status; Consistency and standards; Recognition rather than
recall
Description The “gear” icon is often used to indicate a menu of systems settings and
preference. Although this is not a standard, it is a widely-used convention.
UDPICS uses this gear icon to access a menu that contains a variety of options
- some of which are settings and others providing additional system functionality
Figure 2). This may confuse users who select the menu looking only for settings
selections, or for those who look elsewhere for options found in this menu. The
use of the gear item also raises concerns for administrative users, who might be
confused by the gear icon on the one hand and the “settings and preferences”
menu on the left-hand navigation bar.
Severity Minor
Suggested
Revisions
One possible solution would be to remove all non-settings items from this menu,
adding additional choices to the otherwise underutilized menu bar. Even if the
left-hand navigation bar is available only to administrators, placing all settings
menu in a single place and adjusting the available options based on user
privileges might be easier for users.
25. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Making use of heuristic evaluation
results?
●
Have member of development team present during
inspection?
Can be done with usability studies and other inspections as well
They should observe, not analyze
●
Debriefing session to explore fixes
Consider redesigns where appropriate.
26. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Other Inspection Methods
http://www.useit.com/papers/heuristic/inspection_summary.html
Heuristic estimation – compare interfaces
Feature Inspection – look for long sequences
of operations, unnatural sequences,
those requiring extensive skill...
Consistency inspection – compare designs for consistency
Standards Inspection
27. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Which Heuristics?
• Multiple versions of Nielsen’s list
• Other versions
– Special cases?
• More heuristics – more focus
– ..but less study
• Too many heuristics – cumbersome, complicated
28. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Web Usability Heuristics
http://www.andybudd.com/archives/2007/01/
heuristics_for_modern_web_application_development/index.htm
• Design for user expectations
• Clarity
• Minimize Unnecessary Complexity and Cognitive Load
• Efficiency and Task Completion
• Provide Users with Context
• Consistency and Standards
• Prevent Errors
• Help users, notice, understand, and recover from errors
• Promote a pleasurable and positive web experience
29. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Usability Evaluation of an Electronic Medication
Administration Record (eMAR) Application
Guo, et al. 2011
• eMAR tasks
– Access to drug information, policies, and procedures related to
meds
– Context-specific patient clinical results
– Facilitate communication between nurses and pharmacists
– Verification of the five “rights” (patient, drug, dose, route, and
time)
• Heuristic Evaluation for usability
• Using Zhang’s heuristics and severity ratings
30. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Heuristic evaluation of eMAR
Guo, et al. 2011
• Tasks
– Login
– Order and modify medication
– Verify medication orders
– Access drug references
– Administer medication
– Edit eMAR
– Generate reports and review
• Procedure
– Train evaluators
– Conduct evaluation
– Calculate agreement
– Build consensus
– Review with staff – conduct tasks
31. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Heuristic evaluation of eMAR
Guo, et al. 2011
• Observations
– classified by heuristic,
– grouped into usability problems
– Organized by task
34. Harry Hochheiser, harryh@pitt.edu Baobab Health, February 2015
Agreement?
• What if multiple reviewers disagree?
• Can calculate inter-rater reliability
• Cohen’s kappa
• Raw agreement
• Goal here is to find potential problems -
• being thorough is more important than agreeing