Week 4 lecture, CN5111 Usability Engineering (2014-15)

1. CN5111 – Week 4: Usability
evaluation methods (part 2)
and performance metrics
Dr. Andres Baravalle

2. Lecture content
• Usability testing (review & scenarios)
• Usability inspection
• Usability inquiry
• Performance metrics
2

3. Usability testing (review and
scenarios)
3

4. Usability testing
• When: common for comparison of products or
prototypes
• Tasks & questions focus on how well users
perform tasks with the product
– Focus is on time to complete task & number & type of
errors
• Data collected by video & interaction logging
• Experiments are central in usability testing
– Usability inquiry tends to use questionnaires &
interviews
4

5. Testing conditions
• Usability lab or other controlled space
• Emphasis on:
– Selecting representative users
– Developing representative tasks
• Small sample (5-10 users) typically selected
• Tasks usually last no longer than 30 minutes
• The test conditions should be the same for every
participant
5

6. Some type of data
 Time to complete a task
 Time to complete a task after a specified time
away from the product
 Number and type of errors per task
 Number of errors per unit of time
 Number of navigations to online help or manuals
 Number of users making a particular error
 Number of users completing task successfully
6

7. How many participants is
enough for user testing?
• The number is a practical issue
• Depends on:
– Schedule for testing
– Availability of participants
– Cost of running tests
• Typically 5-10 participants
– Some experts argue that testing should
continue with additional users until no new
insights are gained
7

8. Examples
• The next slides describe 2 experiments:
the one behind the book Prioritizing Web
Usability and a fictional one on
OpenSMSDroid
• Both use Thinking Aloud and video/screen
recording for data collection
8

9. Prioritizing Web Usability
• Prioritizing Web Usability (Nielsen and Loranger, 2006)
used the Thinking Aloud method to collect insight on
user behaviour:
– 69 users, all with at least one year experience in using the
web
– Broad range of job backgrounds and web experience – but no
one working in IT or marketing
– 25 web sites tested with specific tasks
– Windows desktops with 1024x768 resolution running Internet
Explorer
– Recordings of monitor and upper body for each session
– Broadband speed between 1 and 3 Mbps
9

10. Prioritizing Web Usability (2)
• The tasks that the users were asked to perform
included:
– Go to ups.com and find how much does it cost to
send a postcard to China
– You want to visit the Getty Museum this weekend. Go
to getty.edu and find opening times/prices
– Go to nestle.com and find a snack to eat during
workouts
– Go to bankone.com and find best savings account if
you have a $1,000 balance
10

11. Prioritizing Web Usability (3)
• The result of the research is presented as
a book:
– Organising the finding in categories (including
searching, navigation, typography and
writing style)
– Using plenty of examples and screenshots to
demonstrate the usability issues that were
identified
11

12. Prioritizing Web Usability:
findings
• People succeed 66% of the time when
working on “single site” activities and 60%
of the time when having to browse through
the internet for information
12

13. Prioritizing Web Usability:
findings (2)
• Experienced users spend about 25
seconds in a homepage and 45 in an
interior page (35 and 60 for
inexperienced users)
• Only 23% of users scroll on their first
visit of a homepage
– The number decreases after the first visit
– The average scroll for first visit is 0.8 of a
screen
13

14. Prioritizing Web Usability:
findings (3)
• 88% of users go to search engines to find
information
• Font face and size: different font faces for
print and screen
– Different font size depending on target
audience
• More in the book…
14

15. OpenSmsDroid evaluation
• You have been tasked to evaluate the usability
for a new (fictional) Android application to write
short text messages, OpenSMSDroid
• You have decided to set up an experiment
– The next experiment is (loosely) adapted from
“Experimental Evaluation of Techniques for Usability
Testing of Mobile Systems in a Laboratory Setting”
(Beck, Christiansen, Kjeldskov, Kolbe and Stage,
2003)
15

16. OpenSmsDroid evaluation
• Your test users will be perform a set of
tasks in specific configurations using the
thinking aloud method for data collection
– A constraint of 5 minutes has been set for
each of the tasks
– The usability researcher will record the
session and take notes
16

17. OpenSmsDroid evaluation:
testing configurations
• Configurations for the test (tentative list):
– Sitting on a chair at a table
– Walking on a treadmill at constant speed
– Walking on a treadmill at varying speed
– Walking on an 8-shaped course that is changing as
obstructions are being moved, within 2 meters of a
person that walks at constant speed
– Walking on an 8-shaped course that is changing as
obstructions are being moved, within 2 meters of a
person that walks at varying speed
– Walking in Westfield Stratford at 16:00 on Saturday
17

18. OpenSmsDroid evaluation:
testing configurations (2)
• For practical reasons and after reviewing the
literature, these settings have been selected for
this evaluation:
– Sitting on a chair at a table
– Walking on a treadmill at constant speed
– Walking in Westfield Stratford at 16:00 on
Saturday
18

19. OpenSmsDroid evaluation:
tasks
• Writing a new SMS containing the phrase “The quick
brown fox jumps over the lazy dog” repeated 2 times to
an existing contact (without using predictive text
features)
• Writing a new SMS containing the phrase “The quick
brown fox jumps over the lazy dog” repeated 2 times to
an existing contact (using predictive text features)
• Taking a picture and sending it to an existing contact
• Taking a short 1 minute video and sending it to an
existing contact
19

20. OpenSmsDroid evaluation:
tasks (2)
• In each test, you can collect:
– Quantitative data: time needed to perform the
task, and if the task has been completed
– Qualitative data: asking the user to think
aloud while interacting with the device and
recording the interaction
20

21. OpenSmsDroid evaluation: data
analysis
• The evaluation will analyse the data
collected and report on any findings,
informing on any difference in
performance and suggesting possible
changes to the interface
– An experiment can also generate further
hypothesis which will be used in further
experiments
21

22. OpenSmsDroid experiment:
what's missing?
• Something to compare to!
– Otherwise you cannot know if the interface is
better or not
22

23. Usability inspections
23

24. Usability inspection methods
• Heuristic evaluation and walkthroughs
are the most common usability inspection
methods
– We'll also see several other methods
24

25. Usability inspections and
heuristics (2)
• Usability inspection methods are based on
having evaluators inspecting an user
interface
• Usability inspection methods aim to
examine usability-related aspects of an
user interface, even if the interface has not
been yet developed
– Can be used to perform usability evaluations
in the initial stages of the development
25

26. Heuristic evaluations
• Heuristic evaluation is a method that
requires usability specialists to judge
whether each element of an user interface
follows established usability principles and
guidelines
– E.g. Jakob Nielsen’s heuristics
• Heuristics are being developed for mobile
devices, wearables, virtual worlds, etc.
26

27. 27
Nielsen’s heuristics: discount
evaluations
• An heuristic evaluation is referred to as
discount evaluation when 5 evaluators
are used
– Empirical evidence suggests that on
average 5 evaluators identify 75-80% of
usability problems on generalist web
sites

28. 28
Heuristic evaluations: stages
• Briefing session to tell experts what to do.
• Evaluation period of 1-2 hours in which:
– Each expert works separately
– Take one pass to get a feel for the product
– Take a second pass to focus on specific features
• Debriefing session in which experts work
together to prioritize and categorise the
problems

29. Relevant standards and
guidelines
• Relevant standards and guidelines
include:
– W3C HTML and CSS standards
– W3C WAI guidelines
– Mobile Web Best Practices guidelines
– ISO 9271
29

30. Heuristic evaluations:
advantages and problems
• Few ethical & practical issues to consider
because users not involved
– Can be difficult & expensive to find experts
– Experts should have knowledge of application domain
& of the evaluation method used
• Critical points:
– Important problems may get missed
– Focus can be lost on trivial problems
– Experts have biases
30

31. Cognitive walkthroughs
• Focus on ease of learning
• Designer presents an aspect of the design
& usage scenarios
• Expert is told the assumptions about user
population, context of use, task details.
• One or more experts walk through the
design prototype with the scenario.
31

32. Cognitive walkthroughs (2)
• Experts are guided by 3 questions:
– Will the correct action be sufficiently evident to
the user?
– Will the user notice that the correct action is
available?
– Will the user associate and interpret the
response from the action correctly?
• As the experts work through the scenario
they note problems.
32

33. Pluralistic walkthrough
• Variation on the cognitive walkthrough
theme.
– Performed by a team
• The panel of experts begins by working
separately
• Then there is managed discussion that
leads to agreed decisions.
• The approach lends itself well to
participatory design
33

34. Feature inspection
• Feature inspection is a technique that focuses
on the features of a product or of a web site
– A group of inspectors that are given some use cases
and are asked to analyse each feature of the web site
for what regards availability, understandability, and
other aspects of usability
– This technique is better in the middle stages of
development, when features are known but the
artefact cannot be evaluated with methods as lab
experiments.
34

35. Standards inspection
• Standards inspection is a technique used to
ensure the compliance of a web site against
some standard
• A usability professional with extensive
knowledge of the relevant standards inspects a
web site for compliance
• Different standard inspections can be run on the
same artefact
– Nielsen’s heuristics include standards inspection
35

36. Usability inquiry
36

37. Truckers and mobile devices
[...] a major mobile device company was trying to understand why there were
so many data entry errors on a mobile device for long-haul truck drivers. Many
people in the company tended to blame the truckers, whom they assumed were
un educated.
None of them had ever actually met a trucker, but they figured it couldn’t be too
hard to type in a word or two. One winter, a senior user interface (UI) designer
decided to see for himself.
The designer spent a week at a truck stop watching truck drivers use the
device and talking to them about it. He quickly discovered that the truckers
could spell perfectly. Instead, the problem was the device. The truckers tended
to be big men, with big fingers. To make matters worse, they often wore bulky
gloves in the winter.
The device had tiny buttons, making typing with big fingers in warm gloves
frustrating.
(Observing the User's Experience; Goodman, Kuniavsky and Moed 2012)
37

38. Truckers and mobile devices (2)
• The team realized it had been basing
important design decisions on faulty
assumptions
• The team redesigned the UI so that it
required less typing and increased the size
of buttons.
– The error rates dropped dramatically
38

39. Truckers and mobile devices (2)
• That's an example of how usability enquiry
methods work
39

40. Usability inquiry
• Usability inquiry methods focus (at
different degrees) on analysing an artefact
either from “the native point of view" or
looking for “the native point of view"
– Used to obtain information about users' likes,
dislikes, needs, and understanding of the
system
40

41. Usability inquiry (2)
• They may use one or more of these
tecniques:
– Talking to users
– Observing users using a system in a real
working situation
– Letting the users answer questions (verbally
or in written form)
41

42. Data collection & analysis
• Data collection:
– Observation & interviews (e.g. contextual inquiry)
– Notes, pictures, recordings, diaries
– Video
– Logging
• Analysis
– Categorizing the findings
– Using existing categories can be provided by pre-
existing research
42

43. Usability inquiry methods
• The next slides will cover some popular
usability inquiry methods:
– Diary
– Contextual inquiry
– Interviews and focus groups
– Surveys
43

44. Diary method
• The diary method requires users to keep a
diary of their interactions
• Diaries can be free form or structured
– The diary method is best used when the
researcher does not have the time, the
resources or the possibility to use user
monitoring methods or when the level of detail
provided by user monitoring methods is not
needed
44

45. Contextual inquiry
• Contextual inquiry is a structured field
interviewing method which typically evaluates:
– User opinions
– User experience
– Motivation
– Context
• It is a study based on dialogue and interaction between
interviewee and user, and it is one of the best methods
to use when researchers need to understand the users'
work context.
45

46. Interviews and focus groups
• Interviews and focus groups are research
methods based on interaction between
researchers and users
– The researcher facilitates the discussion
about the issues rose by the questions
– In focus groups (multiple users present), the
interaction among the users may raise
additional issues, or identify common
problems that many persons experience
46

47. Surveys
• Surveys are a quantitative research
method, where a set list of questions are
asked and the users' responses recorded
– When the questions are administered by a
researcher, the survey is called a structured
interview
– When the questions are administered by the
respondent, the survey is referred to as a
questionnaire
47

48. And now…
• You have had an overview of a wide
selection of usability evaluation methods
– And you are ready to use them in your
assignment
48

49. Measuring the User Experience
• The next slides are based on the core text
book for this module, "Measuring the User
Experience"
49

50. Performance metrics
50

51. Types of performance metrics
• Task success
• Level of success
• Errors
• Efficiency
• Learnability
51

52. Task success
• Binary success
• Level of success
52

53. Task success
• It measures how effectively users are able
to complete a given set of tasks
• To measure task success, each task that
users are asked to perform must have a
clear end state or goal
53

54. Task success (2)
• You can:
– Ask users to articulate the answer verbally
– Ask users to provide an answer in a
structured way (e.g. using an online tool or
paper)
– Use proxy measures (e.g. when the correct
solution is not easily identifiable as it may
depend from individual circumstances)
54

55. Task failure
• There are many different ways in which a
participant might fail
– Giving up: participants indicate that they would not
continue with the task if they were doing this on their
own
– Moderator “calls” it because the participant is not
making any progress
– Too long: the participant completed the task but not
within a predefined time
– Wrong: participants thought that they completed the
task successfully, but they actually did not
55

56. Binary success
• Binary success is the simplest and most
common way of measuring task success
– Each time users perform a task, they should
be given a “success” or “failure” score (0 or 1)
– Users either completed a task successfully or
they didn’t
– That's the case, for example. for e-commerce
– Sometimes, you will should measure
perceived success rather than factual
success (why?)
56

57. Binary success: analysing and
presenting data
• The most common way to analyse and
present binary success rates is by task.
• This involves simply presenting the
percentage of participants who completed
each task successfully
57

58. Binary success: analysing and
presenting data by task
58

59. Binary success: presenting data
by user
• Frequency of use (infrequent users versus
frequent users)
• Previous experience using the product
• Domain expertise
• Age group
59

60. Levels of success
• Identifying levels of success is useful
when there are reasonable shades of grey
associated with task success.
60

61. Levels of success: complete,
partial and failure
• Complete success
– With assistance
– Without assistance
• Partial success
– With assistance
– Without assistance
• Failure
– User thought it was complete, but it wasn’t
– User gave up
61

62. Levels of success: analysing
and presenting data
62

63. Time on task
• In most situations, the faster a user can
complete a task, the better the experience
– Time on task is particularly important for
products where tasks are performed
repeatedly by the user.
63

64. Time on task: analysing and
presenting data
• The most common way is to look at the
average amount of time spent on any
particular task
– You should always report a confidence
interval to show the variability in the time data
64

65. Time on task: range and
threshold
• A variation is to create ranges and report
the frequency of users who fall into each
interval
• Another useful way to analyse task time
data is by using a threshold.
– In many situations, the only thing that matters
is whether users can complete certain tasks
within an acceptable amount of time.
65

66. Errors
• Errors reflect the mistakes made during a
task. Errors can be useful in pointing out
particularly confusing or misleading parts
of an interface.
66

67. Errors (2)
• Examples of errors include:
– Entering incorrect data into a form field
– Making the wrong choice in a menu or drop-
down list
– Taking an incorrect sequence of actions
– Failing to take a key action
67

68. Errors: analysing and
presenting data
• The most common approach is to look at
average error rates per task or per
participant
68

69. Efficiency
• Efficiency can be assessed by examining
the amount of effort a user expends to
complete a task, such as the number of
clicks in a website or the number of button
presses on a mobile phone.
69

70. Efficiency (2)
• Efficiency typically measures the number
of actions or steps that users took in
performing each task.
– Identify the action(s) to be measured: for
websites, it's typically mouse clicks or page
views
– Define the start and end of an action
– Count the action
70

71. Learnability
• Learnability is a way to measure how
performance improves or fails to improve
over time.
71

72. Learnability (2)
• Learnability is normally measured using
performance metrics: time on task, errors,
number of steps, or task success per
minute.
72

73. Learnability: analysing and
presenting data
• The most common way to analyse and
present learnability data is by examining a
specific performance metric (such as time
on task, number of steps, or number of
errors) by trial for each task or aggregated
• across all tasks.
73

74. Learnability: analysing and
presenting data (2)
74

75. References
• Beck, E., Christiansen, M., Kjeldskov, J.,
Kolbe, N. and Stage, J. (2003).
‘Experimental Evaluation of Techniques for
Usability Testing of Mobile Systems in a
Laboratory Setting’, OzCHI 2003.
• Nielsen, J. and Loranger, H. (2006).
Prioritizing Web Usability.
75