Valerio Zanini is a product innovation and Agile trainer. The document discusses various metrics for measuring product success, including customer metrics to measure outcomes, business metrics to measure results, and telemetry metrics to ensure technical performance. It emphasizes the importance of understanding customer needs through discovery and providing value throughout the development process, rather than just focusing on output and speed to launch. Key failures discussed include shortcutting discovery of customer needs and using Agile practices like Scrum only in development without taking an Agile approach across the full product life cycle.
19. @vzanini 19
“We just finished our product,
we need help with marketing”
Failure #1:
Shortcutting Discovery
20. @vzanini
The team was focused on execution
20
DISCOVER DESIGN DEVELOP DEPLOY DELIVER
Source: book “Deliver great products that customers love”, V. Zanini 2018
?
21. @vzanini
Understanding your customers helps you understand how to
deliver value
21
DISCOVER DESIGN DEVELOP DEPLOY DELIVER
Source: book “Deliver great products that customers love”, V. Zanini 2018
22. @vzanini 22
“Of course we are Agile, we
use Scrum in development”
Failure #2: Big Waterfall
using Scrum
23. @vzanini
Agile is used in Development but everything else is set upfront
23
DISCOVER DESIGN
6+ months to define all
the requirements
24. @vzanini
Agile is used in Development but everything else is set upfront
DISCOVER DESIGN
DEVELOP
SCRUM
1 year to build it
using “Agile”
24
25. @vzanini
Agile is used in Development but everything else is set upfront
25
DISCOVER DESIGN
DEVELOP
DEPLOY DELIVER
SCRUM
?
27. @vzanini
The 5 Dimensions of great products
27
DISCOVER DESIGN DEVELOP DEPLOY DELIVER
SolutionsProblems, needs,
opportunities
Products Market launch Customer
adoption and
feedback
Measure value to customers all the way
28. @vzanini
Work across the 5 Dimensions in rapid iterations and deliver
value along the way
28
MVP
V.2.0
V.2.1
29. @vzanini 29
In your project:
Which of the 5
Dimensions are you
least prepared for?
Which needs most
attention?
35. @vzanini
Three primary responsibilities of a Product Manager
Understand
the
customer
Drive product
development
Measure
the
outcomes
This is also well
represented by the
Learn, Build,
Measure cycle
36. @vzanini
In reality, it often feels
this way
Understand
the
customer
Drive product
development
Measure
the
outcomes
Too often we focus on
just getting the
product out the door,
instead of making sure
we build the right
product
37. @vzanini
It gets even worse
Understand
the
customer
Drive product
development
Measure
the
outcomes We do
“Agile”
38. @vzanini 38
When we build products, we try to go from
idea to launch in the shortest time possible
Image: Created by Alekksall - Freepik.com
Image: Created by Cornecoba - Freepik.com
From Idea To Launch
Agile DevOps
Build, Measure, Learn
MVP
Prototypes
Design Thinking
41. @vzanini
Customer focus
41
Can you articulate?
Who the customer is
What they need
Why they buy your product
What value they get from it
Too often teams shortcut the
Discovery dimension jumping
straight to the Solution
42. @vzanini 42
The goal should be to go from Idea to Customer
Image: Created by Alekksall - Freepik.com
Image: Created by Vectorpocket - Freepik.com
From Idea To Customer
43. @vzanini 43
DEPLOYMENT DELIVERY
Focus is on getting
product out
(outputs)
Focus is on customers,
needs, CX, value
(outcomes)
DELIVERY
GAP
Deployment is not the real goal Delivery is the goal
44. @vzanini
Employ a healthy balance of activities to build the right product
Understand
the
customer
Drive product
development
Measure
the
outcomes
Build
Measure
Learn
48. @vzanini 48
Is it meaningful?
You can influence it
You can measure it
Value
metrics
If you improve it, your
outcomes improve
You can do something
about it
Otherwise, how do you
know?
Select a metric that:
49. @vzanini
Measure value delivered
49
Measuring delivery on time, on budget, scope, quality, etc.: all OK
But, do these matter if people don’t want your product?
Focus should be on value delivered.
Establish
baseline
metric
Establish
target /
desired
outcome
Build an MVP
or MVF
Measure
progress
towards target
/ desired
outcome
Pivot
Success
Failure
Innovation accounting, adapted from “The Lean Startup”, Eric Ries
50. @vzanini 50
Backlog of requirements
Measure outputs
Feedback is confirmatory
What’s in your backlog?
51. @vzanini 51
Backlog of experimentsBacklog of requirements
Measure outputs
Feedback is confirmatory
Measure outcomes
Feedback is a learning
opportunity
What’s in your backlog?
52. @vzanini 52
Most people use analytics the
way a drunk uses a lamppost, for
support rather than illumination
“
”
– David Ogilvy
57. @vzanini
STORY
57
We focused on registered users and bounce rate,
instead of metrics that drove real value
Online e-commerce company
58. @vzanini
The HEART framework from Google
58
H E A R T
Happiness Engagement Adoption Retention Task success
Level of user
involvement, typically
measured via
behavioral proxies
such as frequency,
intensity, or depth of
interaction over some
time period. Examples
might include the
number of visits per
user per week or the
number of photos
uploaded per user
per day.
New users of a
product or feature.
For example: the
number of accounts
created in the last
seven days or the
percentage of Gmail
users who use labels.
The rate at which
existing users are
returning. For
example: how many
of the active users
from a given time
period are still present
in some later time
period? You may be
more interested in
failure to retain,
commonly known as
“churn.”
This includes
traditional behavioral
metrics of user
experience, such as
efficiency (e.g. time
to complete a task),
effectiveness (e.g.
percent of tasks
completed), and
error rate. This
category is most
applicable to areas
of your product that
are very task-focused,
such as search or an
upload flow.
Measures of user
attitudes, often
collected via survey.
For example:
satisfaction,
perceived ease of
use, and net-
promoter score.
60. @vzanini
System and security logs
Real-time telemetry
Test and re-test
Flow break-points
Team delivery charts
Is this thing working?
Problems can arise at anytime, whether as
united consequences of your actions or
due to externalities
Telemetry
metrics
61. @vzanini
Telemetry helps you monitor the health of your app
61
Errors in production
HTTP 5xx
Performance
Events
Broken links
Server failures
Dedicated tools and system logs
Image courtesy of Splunk
E.g. Capital
One Mobile
Deposit
62. @vzanini
Telemetry helps you identify your flow break points
62
Customer journey maps help you
understand the journey users take
with your product/service and
identify breaking points
Telemetry provides real-use
information of where users stumble
63. @vzanini 63
Burndown charts can offer useful
insights into team impediments and
actions product owners can take
Look for unusual paths and changes in
scope. Overcommitment and
changing priorities are usual culprits.
64. @vzanini
What will you measure tomorrow?
Are we
delivering
business
results?
Are we
delivering
outcomes?
Is this thing
working?
Business
metrics
Customer
metrics
Telemetry
metrics
