The document discusses using quantitative usability testing data to understand why users behave in certain ways when interacting with websites and digital products. It emphasizes that quantitative research is more powerful when combined with qualitative insights into why users encounter issues. The document presents several hypotheses that can be derived from design assumptions, other data sources, stakeholder views, and usability evaluation methods to help explain quantitative findings. It also discusses using remote unmoderated usability testing to test hypotheses about user behavior by observing their interactions with tasks and noting where problems occur.

1. Discovering
WHY
from numbers
1234567890123456789012345678901234567890123456789012345678901234567890
©2013 Webnographer Limited
1

2. Why Numbers?
Really, the question is: Why not?
©2013 Webnographer Limited
2

3. 20th Anniversary
of Nielson’s Law.
20 years ago it
was expensive.
There was a
diminishing
return.
N (1-(1- .31 ) n )
©2013 Webnographer Limited
3

4. Discoverability Gap based on the Theory
Likelihood of discovering an issue
100%
Lab Testing with 10 users
80%
Remote Testing with 80 users
60%
40%
N (1-(1- .31 ) n )
20%
0%
0%
5%
10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
Visibility of the issue
The above Lab testing numbers mean that 10 users will find 85% of the issues with a visibility of 17%. The these numbers use
the formula from Nielsen, Jakob, and Landauer, Thomas K.: "A mathematical model of the finding of usability problems,“.
©2013 Webnographer Limited
4

5. 56%
18%
30%
Issues that can be reliably found in the Lab
(5 respondents)
27%
25%
5 respondents find 80% of issues with 30% visibility
BUT that is only 18% of ALL issues
21%
Issue Distribution
Issue distribution is the % of issues that fall into each category of issue visibility
Discoverability Gap based on data
20%
18% of interactions with issues identified
27% of interactions have no issue
= 45.5% coverage
15%
14%
56% of interactions that have issues get missed.
10%
8%
5%
6%
5%
2%
3%
3%
2%
1%
1%
2%
1%
0%
1%
0%
1%
1%
1%
1%
0%
0%
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
©2013 Webnographer Limited
Likelihood of an issue occurring
5

6. 45%
28%
30%
Issues that can be reliably found in the Lab
(10 respondents)
27%
25%
10 respondents find 85% of issues with 17% visibility
BUT that is only 28% of ALL issues
21%
Issue Distribution
Issue distribution is the % of issues that fall into each category of issue visibility
Discoverability Gap based on data
20%
28% of interactions with issues identified
27% of interactions have no issue
= 55% coverage
15%
14%
45% of interactions that have issues get missed.
10%
8%
5%
6%
5%
2%
3%
3%
2%
1%
1%
2%
1%
0%
1%
0%
1%
1%
1%
1%
0%
0%
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100%
©2013 Webnographer Limited
Likelihood of an issue occurring
6

7. Benefits: Prioritise findings
Margin of error
45%
Because Remote Un-moderated
40%
Testing evaluates designs with
40%
large numbers of users, actionable
35%
metrics have a small margin of
30%
error. This allows the prioritisation
25%
of insights and high level
decision making.
20%
The findings from the lab have a
15%
large margin of error because they
9%
10%
are evaluated with only a few
5%
users. This makes it hard to
Number of participants
0%
0
50
Lab Testing
©2013 Webnographer Limited
100
150
200
250
Remote Un-moderated Testing
300
350
prioritise the findings.
The chart uses Sauro’s 20/20 rule. The rule states
that Testing with 20 users will produce a margin of
error of +/-20%. Also see
http://www.measuringusability.com/test-margin.php
for an explanation of the 20/20 rule.
7

8. Why, why?
Why care about WHY in
quantitative research?
©2013 Webnographer Limited

9. Quant research is powerless without why
• If you know 50% of customers can’t find what they
need. What can you change?
• If you cannot explain a finding, it is powerless.
©2013 Webnographer Limited

10. The 2 Cultures
Science vs. humanities
One without the other is dangerous
C P Snow
©2013 Webnographer Limited
10

11. How to get WHY
from numbers?
©2013 Webnographer Limited
11

12. WISDOM
KNOWLEDGE
HOW
INFORMATION
WHAT
DATA
©2013 Webnographer Limited
WHY
THAT
12

13. Theory first
Socrates
Karl Popper
The Socratic method influenced scientific
method where you start with a hypothesis.
A hypothesis is proven in the negative.
©2013 Webnographer Limited
13

14. Finding out
WHY
customers
want your
product
©2013 Webnographer Limited
14

15. Nobel Prize
Winners of the Nobel prize in science were able to explain
phenomenons in the world through theory first
©2013 Webnographer Limited

16. 4 types of hypothesis for UX research
Design
as hypothesis
Hypothesis for
why you are RIGHT
©2013 Webnographer Limited
Other data
as hypothesis
Stakeholder
Views
as hypothesis
Review
methods
as hypothesis
Hypothesis for
why you are WRONG
16

17. Design as hypothesis
Optimum Path:
• Is the shortest possible path to the target page.
• Is the hypothesis of the designer of how customers behave.
1
Click on
Broadband &
Internet in Help
and Support
©2013 Webnographer Limited
2
Reached the
Help hub page
3
Click on My
connection is
slow
4
Reach the “Slow
connection” page
17

18. Design as hypothesis
Lostness
There may be other alternative routes to the target
page which are longer.
x
x
However, the interface should help people find the
shortest path to the information. Customers do not
want the scenic route. They want to get stuff done.
x
1
2
Click on
Broadband &
Internet in Help
and Support
3
Reached the
Help hub page
x
x
©2013 Webnographer Limited
4
Reach the “Slow
connection” page
Click on My
connection is
slow
x
x
x
Longer routes can be
an indication for the
interface facilitating
error recovery.
18

19. Other Data as Hypothesis
Survey data
©2013 Webnographer Limited
Lab test findings
Web analytics data
19

20. Stakeholder Hypothesis
CONFUSION
©2013 Webnographer Limited
UNDERSTANDING
20

21. Review Methods as Hypothesis
Heuristic Evaluation, Cognitive Walkthrough,
or GOMS for each step of the design hypothesis
©2013 Webnographer Limited
21

22. Create a UX Hypothesis
The hypothesis is the theory
WHY something happens.
•
Write down a hypothesis for
how customers behave
•
What evidence do you need
to support your hypothesis?
•
What evidence do you need
to disprove your hypothesis?
©2013 Webnographer Limited
22

23. Test your hypothesis through un-moderated remote usability testing
1
2
To evaluate the research questions,
participants are asked to complete a
Webtask on the website.
The Webtask presents participants with
a task to perform using the website.
While they navigate through the site
their interactions are recorded with our
remote usability testing tool.
3
4
Remote usability testing records all
interactions with the website, such as
clicks, hovering, keystrokes, back press,
page scrolling, time in each form item,
errors per item, time on page, pages
visited, task completion rate, time on
task.
Participant’s interaction with the website
allows us to collect comprehensive
information regarding website usability,
and customer behaviours.
Webtasks provide thus valuable insights
on where and why customers are
experiencing problems, and casts light
on ways to improve website usability.
©2013 Webnographer Limited
23

24. ©2013 Webnographer Limited
24

25. Remote Usability Testing Can
When using the right method
©2013 Webnographer Limited
25

26. Thank you.
sabrina@webnographer.com
@sabrinamach
james@webnographer.com
@jamespage
©2013 Webnographer Limited
26