Mais conteúdo relacionado Semelhante a Is Design Metrically Opposed? - Jared Spool (ProductCamp Boston) (20) Mais de ProductCamp Boston (20) Is Design Metrically Opposed? - Jared Spool (ProductCamp Boston)1. @jmspool
Jared M. Spool
Is Design
Metrically Opposed?
1Are we
measuring the
right thing?
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 1 • [R3]
2. Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 2 • [R3]
3. @ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
33 Email Addresses
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
77 Email Addresses
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
33 Email Addresses 77 Email Addresses
More email addresses are better
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
All email addresses are equal
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 3 • [R3]
4. @ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
@ @ @ @ @ @ @ @ @ @ @
More email addresses are better
All email addresses are equal
Inferences
Observations
Inferences
Observations
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 4 • [R3]
6. Design DecisionsInferencesObservations
Why we think
it happened?
What did we
see?
How will we
improve the
design?
Log file filled with “ ” (blanks)
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 6 • [R3]
7. Enter button focus was broken
Didn’t enter any search text
Was looking for advanced search
Log file is broken
Design DecisionsInferencesObservations
Broken focus
Log file
contained
blanks
Fix Focus
No search text
Advanced
search
Log file bug
Add label to
search box
Build advanced
search feature
Fix bug
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 7 • [R3]
8. The best designers
never stop at the
first inference.
Log file
contained
blanks
Design DecisionsInferencesObservations
Broken focus Fix Focus
No search text
Advanced
search
Log file bug
Add label to
search box
Build advanced
search feature
Fix bug
Users didn’t
know to
type query
into box
Log file
contained
blanks
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 8 • [R3]
10. SnapChat AirBNB Uber Facebook
MarketValuation
InstancesofletterEonhomepage
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 10 • [R3]
11. Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 11 • [R3]
12. Counting the letter E is a stupid metric.
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 12 • [R3]
13. Measure:
Metric:
Analytic:
Something we can count.
A measure we track.
A measure software tracks.
Time on page
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 13 • [R3]
14. Time on page
Bounce rate
% Exit
Pageviews
Sessions w/search
Time after search
% Search Refinements
Bounce rate
What should we do differently?
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 14 • [R3]
15. “
R.H. Coase,
British economist
If you torture data
long enough, it will
confess to anything
you’d like.”
What Google Analytics
can’t tell you:
‣What content was useful?
‣What people found confusing?
‣Who is your site’s biggest spender?
‣What do big spenders do that others don’t?
‣What should you do to improve your content?
‣Why did someone click?
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 15 • [R3]
16. What Google Analytics
can’t tell you:
Why?
Conversion Rate
Conversion Rate
# of people who purchase
# of people who visited
=
1.0 %
10,000
1,000,000
=
Conversion Rate
# of people who purchase
# of people who visited
=
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 16 • [R3]
17. Conversion Rate
1.0 %
10,000
1,000,000
=
2.0 %
20,000
1,000,000
=
2.0 %
10,000
500,000
=
10,000
Conversion Rate
1,000,000
=
1,000,000
=
500,000
=
10,000
=
x $100
$1,000,000
20,000 x $100
$2,000,000
=
=
10,000 x $100
$1,000,000
1.0 %
2.0 %
2.0 %
10,00010,000
20,00020,000
10,00010,000
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 17 • [R3]
18. Conversion Rate
20,000
1,000,000
=
10,000
500,000
=
$2,000,000=
= $1,000,000
20,000
10,000
2.0 %
2.0 %
What should we design for?
$2,000,000
$1,000,000
2.0 %
2.0 %
Conversion Rate? Money?
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 18 • [R3]
19. Conversion Rate
Conversion Rate
# of people who purchase
# of people who visited
=
Visit Visit Visit Purchase
Conversion Rate = 25 % ?
Conversion Rate = 100 % ?
What conversion rate can’t
tell you:
Why?
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 19 • [R3]
21. Achieving satisfaction
is too low a bar to set
for our designs.
We can do better.
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 21 • [R3]
23. Pro tip:
Nothing says “we
don’t care” like a
page of 10-point
satisfaction scales.
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 23 • [R3]
24. 10-point scales make
noise feel like
science.
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 24 • [R3]
25. What satisfaction surveys
and Net Promoter Score
can’t tell you:
Why?
We need metrics that
help us improve our
users’ experience.
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 25 • [R3]
27. ☹
Thingsthatusersfindfrustrating
Content that is confusing
☹ Incomplete information
☹ Can’t remember password
☹ Hidden features
☹ Hidden navigation
☹ Confusing navigation
☹ Error messages
Thingsthatusersfindfrustrating
Phone numbers can’t have dashes or spaces.
☹ Error messages
The credit card security code is required (again).
Username and password do not match.
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 27 • [R3]
28. Case study:
Improve checkout process for a
major e-commerce site.Shippinginformation
Billinginformation
Paymentinformation
Confirmpurchase
Done!
100%
50%
Pageviews
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 28 • [R3]
31. Qualitative usability research
$300,000,000
Quantitative custom metrics
+
Mostimportantcustommetric:
Unrealized shopping cart value from password issues.
Qualitative findings
must drive our
quantitative research
agenda.
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 31 • [R3]
32. The team’s initial
inferences
‣The checkout steps were where we’d find the
biggest improvement.
‣It was “normal” to lose customers before checkout.
‣Cart review went straight to checkout without
additional steps.
‣All of the screens were instrumented.
Shippinginformation
Billinginformation
Paymentinformation
Confirmpurchase
50%
Pageviews
Shopforproduct
ReviewShoppingCart
Logintoaccount
Requestpasswordreset
Clickonlinkinemailtoreset
Putinnewpassword
100%
Done!
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 32 • [R3]
34. “Analytics are controlled by a
different group.”
Nolongeracceptable:
Owned by the UX Team
Owned by the Analytics Team
Under new management:
the UX Team
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 34 • [R3]
35. DJ Patil, US Chief Data Scientist
“Analytics are controlled by a
different group.”
Data science is now an essential
skill for every UX team.
Nolongeracceptable:
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 35 • [R3]
36. Continually question what the
metric is trying to tell you.
Nolongeracceptable:
“I don’t understand what the
metrics mean.”
Keep the numbers simple, so you
can focus on the behaviors.
Nolongeracceptable:
“I’m not good with numbers.”
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 36 • [R3]
37. Applying targeted
metrics to qualitative
research: a powerful
addition to the UX
designer’s toolkit.
4Who collects
the metrics?
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 37 • [R3]
38. Design:
The Rendering
of Intent
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 38 • [R3]
39. Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 39 • [R3]
41. “
Robert Fabricant
The medium of
design is behavior”
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 41 • [R3]
42. Design must drive
metric collection, not
the other way around.
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 42 • [R3]
44. Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 44 • [R3]
45. Gallup CE11
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 45 • [R3]
47. Loyalty
Confidence
Integrity
Pride
Passion
Gallup CE11
I am always proud to be
a customer of this
product’s company.
This product’s company
always treat me with
respect.
Loyalty
Confidence
Integrity
Pride
Passion
Gallup CE11
This product’s company
is perfect for people like
me.
I can’t imagine a world
without this product’s
company.
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 47 • [R3]
48. Loyalty
Confidence
Integrity
Pride
Passion
Gallup CE11
I Agree
I’m Not Sure
I Don’t Agree
+1
0
-1
Result scale: +11 to -11
6.2
5.5
4.5
4.3 3.0
1.4
1.4
-1.0
0.5
1.1
Measuring Engagement
While Buying Electronics
Amazon Best Buy Dell HP Walmart
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 48 • [R3]
49. Metrics must drive us
to eliminate frustration
and deliver delight.
Delight and engagement
go hand in hand.
Gallup CE11
http://bit.ly/gallup-ce11
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 49 • [R3]
50. Is Design
Metrically Opposed?
‣We need metrics that help us improve the experience.
‣Avoid jumping from observations to inference too soon.
‣ Make sure you’re testing alternative inferences.
‣Customize metrics to match experience objectives.
‣Data science is now an essential UX skill.
@jmspool
Go ahead! Follow me on the Twitters.
jspool@uie.com
Find me at:
Copyright © 2014 User Interface Engineering
Don’t forget to connect to me on the LinkedIn.
uie.com
Is Design Metrically Opposed? User Interface Engineering - www.uie.com
© Copyright 2015, User Interface Engineering. All Rights Reserved. Page 50 • [R3]