1. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Measure what Matters
Crafting UX Success Metrics
Wednesday, May 23, 2019 | Kate Rutter | @katerutter
2. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Say “hi” and
introduce yourselves.
(These are the folks who will
make your work awesome.)
3 min
Let’s get to know each other.
Hello!
3. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
tweet-friendly
@katerutter
4. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
5. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
How do you
measure UX
now?
6. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
I am not an expert on your business.
You are.
Today’s goal is to help you explore
ways to measure how well your product
serves real humans...
so you can use metrics to design
& improve the products you build.
What to expect...
7. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
It’s NOT about
Right vs Wrong.
It’s about
Weak ideas made Strong.
What to expect...
8. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
You may not like the pace/process/
pressure/perspective, etc. etc.
That’s okay.
All I ask is that you
keep an open mind and try it.
What to expect...
9. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Let’s get warmed up
Pick an idea to use
for the day
Task-a-
Doodle
write it on a
stickynote
A product you work on
An idea for a new product
or
Product • Feature • Interaction • Customer problem
1 min
10. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
a mobile app for
delegating tasks
Working parents with
with kids
need to know when
something’s done
Too much to do, not enough time
need to share
tasks on the go
Task-a-doodle 5 3 2017Who are the people?
(the users)
What problems do
they have?
What is a proposed
solution?
Product Snapshot
BE SPECIFIC
2 min
11. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Our work today…
•Choose your focus
•Identify your customer
•Envision a Key Use
•Establish a Key Metric
•Capture the data
•Track over time
•Tell the story
meaningful… metrics…
12. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
None of these numbers help us know
our product is working for our users.
“Over 10,000 downloads!”
“Avg time on site is 22 minutes!”
“+450 new sign-ins this week!”
Nope.
13. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Surveys
Usability
Testing
Interviews
Observation
Field
research
Metrics &
Analytics
Quantitative
tells you what is
happening
Qualitative
tells you why
things happen
Generative
helps you come up
with new ideas
Evaluative
helps you assess or
validate something
Big data?
best source
of possible
metrics
NUMBERS STORIESUser
Research
15. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
DAU Daily Active Users
MAU Monthly Active Users
CPA Cost per Acquisition
ARPDAU Average Revenue per Daily Active User
Retention How long they stay & how
frequently they interact
Churn How many users you lose vs. keep
LTV LifeTime Value
NPS Net Promoter Score Sketch via Build Better Products by Laura Klein, Rosenfeld Media
Common metrics
16. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Product
Traffic
Retention
17. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Product
Traffic
Retention
use
19. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
“ ~ Joshua Porter
Your metrics will be as
unique as your business.
20. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Our work today…
•Choose your focus
•Identify your customer
•Envision a Key Use
•Establish a Key Metric
•Capture the data
•Track over time
•Tell the story
meaningful… metrics…
21. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Features
Users
Uses
Product
Needs
Company
Purpose
sketches,
prototypes,
pixels
UI
via @luxrco
UX Stack
22. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Who is our user?
What do they think/
feel/say?
What do they need?
Ask:
23. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Features
Users
Uses
Product
Needs
Company
Purpose
sketches,
prototypes,
pixels
UI
via @luxrco
UX Stack
24. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Meet Mary, a working professional with two kids.
Mary needs to share the load of household tasks
with her family, and get help completing tasks.
using the task scheduler, task sharer and done
notifications.
in a way that’s friendly, clear, visual, and fast.
With Task-a-doodle, Mary can create tasks, share
them with her spouse and kids, and know when
something gets done
Task-a-doodle
25. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
5 min
Who is this person?
(name & picture)
What is their story?
What are their
needs & goals?
Customer Portrait
BE SPECIFIC
26. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
What can our
customer do with our
product that they
can’t do without it?
Ask:
27. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Features
Users
Uses
Product
Needs
Company
Purpose
sketches,
prototypes,
pixels
UI
via @luxrco
UX Stack
28. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
using the task scheduler, task sharer and done
notifications.
in a way that’s friendly, clear, visual, and fast.
Meet Mary, a working professional with two kids.
Mary needs to share the load of household tasks
with her family, and get help completing tasks.
With Task-a-doodle, Mary can create tasks, share
them with her spouse and kids, and know when
something gets done
Task-a-doodle
29. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
An example from Foodspotting
30. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
An example from Foodspotting
31. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
rapid viz
32. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
smile frown
up brows
neutral
down
brows
no brows
add a mouth...
add eyebrows...
sketch via @luxrco; also see @austinkleon
Expressions Matrix
33. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
You can draw a person in 4 simple steps....
Star People
34. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Speed-sketch
5 star people!
1 min
35. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
http://bit.ly/sketchingpeople
Video examples here:
36. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Overhead gallery!
37. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Key Uses 6-up
OK to be obvious
5 3 2017
•1 idea per panel
•6 different ideas
(not a sequence)
•Sketch AND write
What can someone
DO with your
product?
- Solves the problem
- Addresses a need
10 min
Task-a-doodle, Mary
38. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Dot-vote top pick
.5 min
Pick one to focus on: use a colored dot
39. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Key Use
• This is a key use
for your product
• Make the sketch
clean & clear
1 min
Task-a-doodle,
Mary
5 3 2017
Share tasks
with others.
RE-Draw your key use
OK to be obvious
40. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
What you have now...
What people?
What problems?
What product?
Product
Snapshot
Who is your
customer? User?
Customer
Portrait
What can a
user do with
our product?
Key
Use
Gives
meaning to
our metrics}
41. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Break!
30 min
42. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Our work today…
•Choose your focus
•Identify your customer
•Envision a Key Use
•Establish a Key Metric
•Capture the data
•Track over time
•Tell the story
meaningful… metrics…
43. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
44. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
45. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
…measures the
usage of your
product by a
person.
a good metric…
Clear &
specific Comparative
Normalized
Actionable
Changes
your behavior
46. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
• Know the desired user behaviors.
• Know the specific interactions
that support the action
or behavior.
• Know what numbers track
these interactions.
Clear & specific
share a task
*
a good metric…
47. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Normalized
(rate or ratio)
• Percentages are
inherently comparative
• They balance out big
changes in total
quantities.
782 total users
420 share a task
1,275 share a task
6,835 total users
18.6%
53.7%
a good metric…
48. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Comparable
(time-stamped)
• Captured at regular
intervals
• Directly comparable
over a span of time What?!?!
6
a good metric…
49. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Actionable, not vanity
• Total number of users
• Downloads
• Page views
• Likes
• Funding press releases
• Articles in media
• etc…
a good metric…
50. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
“ ~ Alistair Croll
with Ben Yoskovitz
Don’t just ask questions. Know how
the answers to the questions will
change your behavior.
51. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
vanity
total number
of registered
users
unhelpful
sign-ups
% of users
who share a
task 3+
times a day,
per week
awesome
better
% of users
who sign in 3+
times a day,
per week
good
% of new
users per
week
an awesome metric…
52. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
So…
How do we get
our awesome
metric?
53. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Brainstorm 3
ideas for metrics
DONE!
What could you count to make sure
the Key Use is happening?
54. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Pick one to focus on
.05 min
55. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
3. Is there an object
basis?
• “per user” [ ___/user]
• “per user per week” (fancy!)
• Examples: per user, per session,
or by interaction or item
(messages per file, emails
per thread)
* not required, but super-helpful
• “per week” [ ___/wk ]
• “per month” [ ___/mo]
2. Is there a time basis?
• “number of...” [ # ]
• “average number of...” [avg ]
• “percent of...” [ % ]
1. Does the metric
begin with a number?
Make it awesome
% of users
who share a
task 3+
times a day,
per week
awesome
better
% of users
who sign in
3+ times a
day,
per week
good
% of new
users per
week
5 min
56. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Share your work
Review the snapshot
& use.
Peers: make
recommendations
Improve
as a pair
Key Metric 2 min
each
57. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Put it on your dashboard
.5 min
58. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Pick another one
.05 min
59. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
3. Is there an object
basis?
• “per user” [ ___/user]
• “per user per week” (fancy!)
• Examples: per user, per session,
or by interaction or item
(messages per file, emails
per thread)
* not required, but super-helpful
• “per week” [ ___/wk ]
• “per month” [ ___/mo]
2. Is there a time basis?
• “number of...” [ # ]
• “average number of...” [avg ]
• “percent of...” [ % ]
1. Does the metric
begin with a number?
Make it awesome
% of users
who share a
task 3+
times a day,
per week
awesome
better
% of users
who sign in
3+ times a
day,
per week
good
% of new
users per
week
3 min
60. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Key Metric 2 min
each
Share your work
Review the snapshot
& use.
Peers: make
recommendations
Improve
as a pair
61. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
.5 min
Put it on your dashboard
62. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Homework
Pick the third one
63. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
.5 min
Pick ONE to work on next
64. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Questions &
Discussion
65. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
So…
now
what?
66. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Three ways to gather data:
• Non-technical
• Technical “by hand”
• Instrumented with software
The process of setting up the technology to capture
the numbers that matter to you. This often means
using an analytics package or writing custom code.
Instrumentation
67. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
http://www.businessofapps.com/big-list-app-analytics-tools/
https://blog.bufferapp.com/social-media-analytics-tools
Instrumentation
68. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
UI Sketch
69. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
1.Sketch the UI -
simple is good!
2.Annotate interactions
to capture
3.Make notes on how
to calculate.
Where in the UI
can you capture
data?
UI Sketch
Homework
70. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
How do we
track progress
over time?
Ask:
71. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
•Set a goal
•Set a time span
•Think about how to
collect the data
How will you track
progress?
Metrics Plan
72. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
•Set a goal
•Set a time span
•Think about how to
collect the data
How will you track
progress?
Metrics Plan
Homework
73. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Track over time
74. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Track over time
75. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
What you have now...
measures
meaningful
metrics}Metric
Tracker
How will we
track progress?
Metrics UI
Sketch
Where is it in
the UI?
Key Metric
Plan
How will we get
the data?
Metrics
Dashboard
What metric
matters?
76. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Tell the story
77. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
4 Metrics
Dashboard
Put in this order:
1 Product
Snapshot
2 Customer
Portrait
3 Key
Use
5 Metrics UI
Sketch
78. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Geek out & learn more
79. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Questions &
Discussion
80. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Kate Rutter
@katerutter
http://intelleto.com
Thanks!
Swearing, drinking & UX…give it a listen!
http://bit.ly/whatiswrongwithux
81. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) 2019
Handouts &
Templates
82. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
Handouts
Crafting UX Success Metrics
Measure what Matters
83. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
Features
Users
Uses
Product
Needs/Goals
Company
Purpose
sketches,
prototypes,
pixels
UI
UX Stack
* Via Luxr, Janice Fraser
84. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
Product Snapshot
people
problem solution
have
addresses
use
Product name date
/ /
85. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
Their story...
Meet
Customer Portrait date
/ /
Needs & goals...
86. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
With _________________ , _________________ can...
Key Uses 6-up sketches
date
/ /
87. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
With
can...
,
Key Use date
/ /
88. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
Feature Set date
/ /
To meet these needs, we have these features:
feature
Notes
feature
Notes
feature
Notes
89. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
a great metric...
...makes you look at all the other metrics
and say “none of those other numbers
matter if we don’t get this right first.”
a good metric...
...measures the usage of the product by
a person. The usage should be specific to
features that deliver value to the user.
Making metrics actionable
vanity good betterunhelpful
total number
of registered
users
% of users
who share a
task 3+
times a day,
per week
sign-ups
awesome
% of users
who sign in 3+
times a day,
per week
% of new
users per
week
Does the metric begin with
a number?
• “percent of...” [ % ]
• “average number of...” [avg ]
• “number of...” [ # ]
Is there a time basis?
Is it comparative?
• “per week” [ ___/wk ]
• “per month” [ ___/mo]
Is there an object basis?
* not required, but super-helpful
• “per user” [ ___/user]
• “per user per week” (fancy!)
1
Metrics Check
2 3
90. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
metric
by
Goal
/ /
Notes
Metrics Dashboard date
/ /
To assess progress, we will measure these metrics:
metric
by
Goal
/ /
Notes
metric
by
Goal
/ /
Notes
91. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
Metrics UI Sketch
date
/ /
Here’s what we’ll collect and measure.
92. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
Metrics UI Sketch
date
/ /
Here’s what we’ll collect and measure.
93. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
by
Goal
/ /
Key Metric plan date
/ /
To assess progress, we
will measure this metric:
Measured
Baseline
Daily Weekly Monthly
Here’s how we’ll collect the data:
metric
94. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
Metric Tracker
start date
/ /
by
Goal
/ /
Acquisition
Monthly / weekly
/ /
/ /
/ /
/ /
metric
to
end date
/ /
as of
Baseline
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
/ /
Activation
Retention
Referral
Revenue
95. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
Metric Tracker
96. UX Lisbon 2019 | Measure What Matters | @katerutter | May 23, 2019 | (c) Kate Rutter
What behavior are you trying to introduce / increase / decrease / change?
Analytics Strategy date
/ /
What is the financial impact of the behavior?
Who owns the financials for this behavior? Who else is involved?