How do you know if the user experience your team is slaving over is succeeding or failing? How can your team use data to make better decisions? Quantitative measures can help answer these questions and can complement more traditional qualitative research methods like usability testing.

1. Measures of Success: How to Quantitatively
Measure Your User Experience
Richard Dalton, @mauvyrusset
http://www.flickr.com/photos/torontorob/4044565681/
1

2. 2

3. #1 Evaluate your experience against
something you care about – is it
meeting it’s objectives?
@mauvyrusset #WVpdx
3

4. 4

5. #2 The objectives of your experience
will likely differ from those of the
person sitting next to you
@mauvyrusset #WVpdx
5

6. Users have goals
Goals
… and the business has goals
Are realized through
Users do things to try to meet their goals
Tasks + Emotions
… and the business wants users to do things so it can meet its goals
Are enabled & encouraged by
Capabilities help users to do the tasks they want to do
Capabilities
… and encourage users to do the tasks the business wants them to do
Are created & changed by
Projects create new capabilities
Projects
Projects change existing capabilities
6

7. #3 Measure how well tasks are
satisfied by capabilities, not
projects – otherwise you have
no baseline
@mauvyrusset #WVpdx
7

8. 8

9. User-driven tasks
Find an investment company Capabilities Business-driven tasks
Make good investment decisions
Web Follow Vanguard’s investing principles
Monitor my investments
Phone Learn why Vanguard is great
Act on my investments
Paper
Bring assets to Vanguard
Stay current on news and commentary E-mail
Use Vanguard's products & services
Deal with taxes Mobile devices
Help other people be successful investors Radio/TV Self-provision on the web
Help my heirs be successful at Vanguard Spread the word about Vanguard
Trust Vanguard
90 tasks grouped 635 capabilities and 45 tasks grouped
into 8 categories counting … into 7 categories
9

10. http://filmfanatic.org/reviews/wp-content/uploads/2008/01/anfscd-parrot.png
10

11. User-driven tasks
Research an item Capabilities Business-driven tasks
Get information on the item
Find out how much the item costs Close the sale
Item profile
Compare the item to others like it (web) Buy the item
See if other people like the item
Save the item to look at later Cross sell
Print details about the item See related items
See related items
Trust us
Ready to buy Believe that the site is safe and secure
Buy the item
Spread the word
Find out shipping costs and times
Tell other people about the item
Find out how to pay
Tell others
Tell a friend about the item
11

12. Buy the item
1
Buy the item Emotional considerations
Shoppers are commonly fearful or unsure that they have chosen the
2 Get information on the item
best product for their needs and want to “comparison shop” the price
3 Find out how much the item costs and/or features of several products.
4 See if other people like the item
5 Find out shipping costs and times High-level design & content approach
Show recently viewed items and provide access to a “compare to
6 Compare the item to others like it similar items” tool.
7 Believe that the site is safe and secure
8 Find out how to pay
Success criteria & measures – “what is good?”
9 Save the item to look at later
The ratio of users looking at recently viewed items via the “compare
See related items to similar items” tool vs. pogo-sticking back to the gallery page should
10 be 20:1 or better.
See related items
11 Print details about the item
Tell a friend about the item
12
Tell other people about the item
12

13. Measures
13

14. Measures vs. Success Criteria
Measure Success Criteria
http://3.bp.blogspot.com/_iUXX3rF1Axo/TC0fGhmlV0I/AAAAAAAAAMw/atKJuTja0AM/s1600/butterfly+growth+chart2.jpg
http://www.clarklings.com/uploaded_images/IMG_1100-732092.JPG 14

15. S3.01 Tell a friend about the item S2.03 Find out how to pay S1.01 Get information on the item
S2.02 Find out S1.02 Find out S1.03 Compare the
shipping costs how much the item to others like it
and times item costs
S1.04 See if
other people
like the item
S2.01 Buy the item
P1.01 Buy the item
P4.02 Tell other P3.01 Believe P2.01 See related items
people about that the site is
the item safe and secure
Drivers Outcome
Ishikawa (fishbone) diagram
http://en.wikipedia.org/wiki/Ishikawa_diagram
15

16. #4 Measuring outcomes can tell you
if a capability is failing. Measuring
drivers can tell you why
@mauvyrusset #WVpdx
16

17. What’s being evaluated? Measure
A. General effectiveness in satisfying the task 1. Completion or conversion rates
“I’ve done it” or “Now I understand” 2. Did future user behavior change as a
result of this interaction?
B. Findability of a known item Speed to find the first task
“I know where I’m going, don’t get in my way”
C. Were the user expectations met? Link bounce rate
“Oh, that wasn’t what I wanted, let me go back …”
D. Client satisfaction 1. “Was this useful/helpful” surveys
“Huh, that information was useless” 2. “Loss of sale” surveys
E. Was enough information provided? Usage of a second source of information
“Don’t leave me wanting more info”
F. Are users travelling the paths we expected them to? Relative usage of one path vs. another to
“I can’t find the link, I’ll go back to the homepage first” identical destination
17

18. Success Criteria
Enduring Temporary
Use for on-going monitoring Use for point-in-time improvement
Measure a single solution Compare two or more solutions
against a predefined criteria against one another using A/B testing
Criteria set by past user Point-in-time “winners”
behavior or future expectations. identified by test results.
Example Example
Task: Get information on the item Task: Print details about the item
Loss of sale surveys should show that A/B test two versions of the
problems with item information printer friendly entry point (link vs.
account for less than 5% of lost sales button)
18

19. #5 Ask; how would the user
behave if we nailed the design?
How would they behave if we
screwed it up?
@mauvyrusset #WVpdx
19

20. Cultural Challenges
20

21. Sometimes people aren’t ready to listen
21

22. #6 Be open about the uses &
limitations of data & involve
people early – it helps gain buy-in
@mauvyrusset #WVpdx
22

23. Beware good intentions
23

24. #7 Avoid misleading measures -
temptation to use the data is
too strong. Ask “what if the
result is X?”
@mauvyrusset #WVpdx
24

25. The only thing we have to fear is fear itself
Ostridge
http://fc05.deviantart.net/fs36/f/2008/285/4/f/You_make_kitty_scared_by_GreenLabRat.jpg
25

26. #8 Be unbiased. Don’t be afraid to
measure things that might
contradict your own opinion
@mauvyrusset #WVpdx
26

27. I’m sorry Dave. I’m afraid I can’t do that
http://img395.imageshack.us/img395/3899/hal90001600en6.jpg
27

28. #9 Don’t lose your perspective
about how data fits into your
decision-making process
@mauvyrusset #WVpdx
28

29. But we have over 635 capabilities!
http://www.flickr.com/photos/tammra/279392432/
29

30. #10 Start small. Pick a capability,
identify objectives, define
measures and watch what
happens
@mauvyrusset #WVpdx
30

31. Thank you
Richard Dalton
richard@mauvyrusset.com
mauvyrusset.com
@mauvyrusset
31