The document discusses the problems that can arise from focusing on a single metric, referred to as "the one metric that matters". It notes that companies like Etsy and Yahoo have run into issues by focusing only on gross merchandise sales or revenue per user, respectively. The document recommends using a framework with three types of metrics: overall evaluation criteria to measure long-term business value, feature metrics to understand specific product changes, and guardrail metrics to avoid degradation. It provides examples of metrics that could be used for companies like Uber/Lyft and Twitter.
1. The lure of “the one metric that matters”
Adil Aijaz (@adilaijaz)
Co-founder & CEO
Split Software
2. Lure of One Metric That Matters
Software Engineer Sr Staff Software
Engineer
Principal Member
Technical Staff
Co-founder & CEO
3. Lure of One Metric That Matters
Split is a product decisions platform
that lets engineering & product teams
rapidly - and safely - deliver valuable software
4. Lure of One Metric That Matters
We power your product decisions
5. Lure of One Metric That Matters
… by providing a unified platform for
feature flags and experimentation
6. Lure of One Metric That Matters
Output
factory
Outcome
factory
We change the way engineering teams operate
#allaboutmeasurement
7. Lure of One Metric That Matters
Measurement is all about metrics
8. Lure of One Metric That Matters
It’s a topic that eludes many
9. Lure of One Metric That Matters
Common problems with metrics
10. Lure of One Metric That Matters
#1 Opinion driven cultures - zero metrics
11. Lure of One Metric That Matters
#2 The lure of one metric that matters
12. Lure of One Metric That Matters
#3 The tyranny of metrics - fixation with
metrics
13. Lure of One Metric That Matters
The lure of one metric that matters
14. How Etsy’s New CEO Threw
Out All of the Company’s
Key Metrics, Except One
“I observed that GMS [gross merchandise sales]
is the one metric that matters," says Silverman.
Before the company was measuring 15 or so
metrics, which justified a lot of projects that
may or may not have been worthwhile. In fact,
he asked his new team to "stack rank" the 800
product initiatives in Etsy's pipeline, all with an
eye to what could goose GMS quickly.”
Fortune.com March 5th, 2018
Lure of One Metric That Matters
Sometimes, it is explicit
15. Yahoo! Mail’s display ad
problem due to one metric
that matters:
revenue / user
Lure of One Metric That Matters
Sometimes, you just
know it
27. Lure of One Metric That Matters
● Overall Evaluation
Criteria
● Feature Metrics
● Guardrail Metrics
Three types of
metrics*
*based on work done by Microsoft Analysis & Experimentation team
29. The (OEC) is the metric your product
development is designed to improve.
What is an OEC?
Lure of One Metric That Matters
measure of long-term
business value or
satisfaction for user,
rather than granular
feature specific metric.
30. ● Sensitivity - The metric should change due
to small changes in user satisfaction or
business value so that the impact of
product development is clear within a
reasonable time period.
● Directionality - If the business value
increases, the metric should move
consistently in one direction. If it
decreases, the metric should move in the
opposite direction.
● Understandability - OEC ties product
development to business value, so it
should be easily understood by business
executives.
Lure of One Metric That Matters
Properties of a
Good OEC
31. Lure of One Metric That Matters
What’s a good OEC for Uber / Lyft?
32. Example for riders - Uber / Lyft
Lure of One Metric That Matters
Revenue / User Rides / User
Distance / User Sessions / User
33. Example for riders - Uber / Lyft
Lure of One Metric That Matters
Revenue / User Rides / User
Distance / User Sessions / User
34. Lure of One Metric That Matters
What’s a good OEC for Twitter?
35. Example for consumers - Twitter
Lure of One Metric That Matters
Tweets / User Time Spent / User
Sessions / User Interactions /
User
36. Example for consumers - Twitter
Lure of One Metric That Matters
Tweets / User Time Spent / User
Sessions / User Interactions /
User
37. ● Click through rate on a button is not a
good OEC.
● One way to recognize whether you have a
good OEC is to intentionally experiment
with a bad feature you know your users
would not like.
● For advanced companies, OEC can be the
output of a model that takes multiple
input signals.
● It’s best to normalize the OEC “/customer”.
Lure of One Metric That Matters
Some rules of
thumb
39. What is a feature
metric?
Lure of One Metric That Matters
Like the OEC, but of
localized interest to a
specific team.
Often explains why
the OEC moved a
certain way
40. ● Click through rate on a button
can be a good feature metric.
● Should be directional and
sensitive
● Understandability is not critical.
● Tie back to business value is
not critical
Lure of One Metric That Matters
Some rules of
thumb
42. What is a
guardrail metric?
Lure of One Metric That Matters
Metric that should not
degrade in the pursuit
of OEC
43. Examples
Lure of One Metric That Matters
● Page load time
● Ride cancellations / user
● Revenue / user
● Content quality / content
● Unsubscribe rates
44. ● One team’s OEC can be another
team’s guardrail
● Engineering system performance
is always a good guardrail
● Should be directional and
sensitive
● Understandability is not critical.
● Tie back to business value is not
critical
Lure of One Metric That Matters
Some rules of
thumb
46. Lure of One Metric That Matters
A framework gives a vocabulary to your team
to talk about metrics
47. Lure of One Metric That Matters
HEART Metrics*
*Based on work done by Google’s UX team
48. ● Metrics that measure user attitude
● Often subjective
● Usually measured via surveys
● Examples: NPS score, ease of use,
satisfaction scores.
Lure of One Metric That Matters
Happiness
49. ● Metrics that measure user involvement
● Frequency, intensity, or depth of
interaction over some period of time
● Usually measured by Product Analytics
tools
● Examples: daily active users, sessions /
user, number of shares / user / week
Lure of One Metric That Matters
Engagement
50. ● Metrics that measure how new users are
adopting a feature or product
● Often tied to the OEC
● Examples: # of rides taken by a Lyft rider
in their first month.
Lure of One Metric That Matters
Adoption
51. ● Metrics that measure whether users who
adopted a feature in one time period,
continue using it in subsequent
time period
● Often tied to the OEC
● Examples: # of rides taken by a Lyft rider
in their second, third, and fourth month.
Lure of One Metric That Matters
Retention
52. ● Metrics that measure user efficiency,
effectiveness, and error rate in
completing a critical task
● Often tied to the OEC
● Examples: wait time / ride measures
efficiency, , # of rides cancelled by a rider
measures effectiveness, & app crashes
measure error rate for Lyft
Lure of One Metric That Matters
Task Success