Lean Analytics is more than just understanding what to track and when. Lean Analytics (and data in general) is about communication within an organization. This is a 1-day workshop I conducted at CrunchConf 2016 in Budapest with a group of data analysts and data scientists to help them understand their role, through the use of analytics, within a larger organization.
6. Agenda
09:00-11:00 - Tools & frameworks for “talking business”
• Lean Startup (it’s not just for startups!)
• Business Mapping (Lean Canvas)
• Product Management & Data (roadmaps & feature prioritization)
11:00-11:30 - Coffee break
11:30-12:30 - Lean Analytics: The basics
• Making data simpler for everyone else
• What makes a good metric
• Types of metrics
12:30-13:30 - Lunch
13:30-15:00 - Lean Analytics II
• Lean Analytics framework
• One Metric that Matters
• Lean Analytics Cycle
15:00-15:30 - Coffee break
15:30-17:30 - Interactive session & Q&A
13. ENTERPRISES ARE GOING LEAN…
“By 2021, more than 50% of established corporations will
be leveraging lean startup techniques.” (Gartner)
14. Core Adjacent Transformative
Do the same thing
better.
Nearby product, market,
or method.
Start something
entirely new.
Regional
optimizations.
Innovation, go-to-market
strategies.
Reinvent the business
model.
• Get there faster
• Smaller batches
• Solution, then testing
• Increased accountability
• Customer development
• Test similar cases
• Parallel deployment
• Analytics & cycle time
• Fail fast
• Skunkworks/R&D
• Focus on the search
• Ignore the current model &
margins
Many models for enterprise innovation
15. Core Adjacent Transformative
Know the problem
(customers tell you it)
Know the solution
(customers/regulations/
norms dictate it.)
Know the problem (market
analysis)
Don’t know the solution (non-
obvious innovation confers
competitive advantage.)
Don’t know the problem (just
an emerging need/change)
Don’t know the solution.
Waterfall:
Execution
matters
Agile/scrum:
Iteration
matters
Lean Startup:
Discovery
matters
Another way to look at it
17. Improvement Adjacency Remodelling
Do the same,
only better.
Explore what’s
nearby quickly
Try out new
business models
Lean approaches apply, but the metrics vary widely.
Sustain /
core
Innovate /
adjacent
Disrupt /
transformative
20. Don’t sell what you can make. Make what you can sell.
Kevin Costner is a lousy entrepreneur.
21. The core of Lean
is iteration.
TALKING BUSINESS: LEAN STARTUP
22. TALKING BUSINESS: LEAN STARTUP
Key principles of Lean Startup
• “Get out of the building”
• Customer discovery is key
• Identify and solve the riskiest thing first
• Reduce waste
• Most important: learning
23. TALKING BUSINESS: LEAN STARTUP
“THERE ARE NO
ANSWERS INSIDE
THE BUILDING.”
- Steve Blank, author of Four Steps to the Epiphany
24. TALKING BUSINESS: LEAN STARTUP
Build Measure Learn
• What do I build?
• How do I know if it’s the
right thing?
• How do I start?
25. TALKING BUSINESS: LEAN STARTUP
START WITH A PROBLEM:
What problem are you trying
to solve?
27. LEAN STARTUP IS ABOUT
FINDING THE TRUTH,
AND DOING SO AS
QUICKLY AS POSSIBLE.
28. SO HOW DO WE FIGURE
OUT WHAT PROBLEMS
TO SOLVE?
29. TALKING BUSINESS: LEAN STARTUP
http://talkingtohumans.com https://leanstack.com/running-lean-book/ http://keytakeaways.io/books/four-steps-epiphany/
A few good references on customer development, problem/
solution interviews and more
30. TALKING BUSINESS: ASSUMPTIONS
Understand your business assumptions
• Make a list of your riskiest
assumptions first
• Assumptions have to be
testable
• “We believe that…”
Do you know the definition of
“Assume”?
31. BUSINESS ASSUMPTIONS Day 2 - Lean Startup
ZxLERATOR | NYC | SUMMER 2016 31
Understand your business assumptions
• My target customer will be? (Tip: how would you describe your primary
target customer)
• The problem my customer wants to solve is? (Tip: what does your
customer struggle with or what need do they want to fulfill?)
• My customer’s need can be solved with? (Tip: give a very concise
description / elevator pitch of your product)
• Why can’t my customer solve this today? (Tip: what are the obstacles
that have prevented my customer from solving this already?)
• The measurable outcome my customer wants to achieve is? (Tip:
what measurable change in your customer’s life makes them love your
product?)
• My primary customer acquisition tactic will be? (Tip: what is your
current guest for the top 1 or 2 marketing channels?)
TALKING BUSINESS: ASSUMPTIONS
32. BUSINESS ASSUMPTIONS Day 2 - Lean Startup
ZxLERATOR | NYC | SUMMER 2016 32
Understand your business assumptions
• My earliest adopter will be? (Tip: remember that you can’t get to the
mainstream customer without getting early adopters first)
• I will make money (revenue) by? (Tip: don’t list all the ideas for making
money, but pick your primary one)
• My primary competition will be? (Tip: think about both direct and
indirect competition?)
• I will beat my competitors primarily because of? (Tip: what truly
differentiates you from the competition?)
• My biggest risk to financial viability is? (Tip: what could prevent you
from getting to breakeven? Is there something baked into your revenue
or cost model that you can de-risk?)
• My biggest technical or engineering risk is? (Tip: is there a major
technical challenge that might hinder building your product?)
TALKING BUSINESS: ASSUMPTIONS
35. TALKING BUSINESS: MAPPING YOUR BUSINESS
• It’s difficult to create value for a business without understanding how
the business functions
• There are several systems for designing/mapping a business
• The goal is to understand all the actors involved, their motivations,
interests and biases
• Common questions you’ll want to answer: Where does the money
come from? Who is the customer? Who are the partners?
The importance of mapping your business
37. The leader in predictive analytics for people. Clearfit
helps thousands of companies build better teams. As
featured in:
CASE STUDY
10x
revenue increase
off of 3x in sales
volume
“People don’t do subscriptions for haircuts, hamburgers or
hiring. You have to understand your customer, who they are,
how and why they buy, and how they value your product or
service.” - Ben Baldwin
39. TALKING BUSINESS: MAPPING YOUR BUSINESS
http://leanstack.com/lean-canvas/
• A 1-page “Business Plan”
• Helps identify key aspects of your business and biggest risks
• Should be doable in 20-minutes
• A visualization of the business assumptions list from Talking
to Humans
• Ideally you update it over time at major milestones / key
discoveries
Introducing Lean Canvas
40. TALKING BUSINESS: MAPPING YOUR BUSINESS
Lean Canvas
Day 3 - Business Models &
Value Propositions
DESIGNING A BUSINESS MODEL
1
2
3
4
5
67
8
9
41. TALKING BUSINESS: MAPPING YOUR BUSINESS
http://leanstack.com/lean-canvas/
Lean Canvas Best Practices
1
2
3
Try and test one thing at a time
Have a hypothesis around how to test each section
and decide on an MVP to experiment with
Update the Lean Canvas whenever you’ve learned
something significant
42. TALKING BUSINESS: MAPPING YOUR BUSINESS
1. Testing the Problem
• Ongoing customer discovery; problem interviews -- largely
collecting qualitative feedback
• Remember: No one pays for their 5th problem to be solved
• For existing alternatives define one clear, direct competitor.
• Consider other ways users/customers can address their problems
• What products or services exist as alternatives to what you’re offering?
• Be careful about problems that are too high level (“universal
truths”)
43. TALKING BUSINESS: MAPPING YOUR BUSINESS
2. Identifying Customer Segments
• Define 3-4 specific user personas for your early adopter groups
• Who is your intended audience?
• What type of person do you anticipate benefiting most from your
product?
• Don’t forget there may be a difference between the users of your product
and the buyer
• For early adopters: what makes them different?
• Understand your best users: http://www.instigatorblog.com/your-best-users/
2014/06/20/
44. TALKING BUSINESS: MAPPING YOUR BUSINESS
3. Testing the Unique Value Proposition
• What do you do, why are you different, and why are you worth
investing in?
• What’s the high level concept?
• Conduct solution interviews (this is where you’ll start to test your
UVP)
• Create landing pages with different messaging
45. TALKING BUSINESS: MAPPING YOUR BUSINESS
More on Value Propositions
Steve Blank’s XYZ
Template: “We help X do Y doing Z”.
Sample: We help non-technical
marketers discover return on investment
in social media by turning engagement
metrics into revenue metrics
Dave McClure’s Elevator Ride
Template:
• Short, simple, memorable: what, how, why.
• 3 keywords or phrases
• KISS (no expert jargon)
Sample: Mint.com is the free, easy way to
manage your money online.
http://torgronsund.com/2011/11/29/7-proven-templates-for-creating-value-
propositions-that-work/
46. TALKING BUSINESS: MAPPING YOUR BUSINESS
4. Testing Solutions
• Conduct solution interviews — get qualitative feedback from
users/customers
• Measure interest in solutions (before building anything)
• Start mapping out your MVPs
• Map key features to key problems
47. TALKING BUSINESS: MAPPING YOUR BUSINESS
5. Testing Channels
• How will users/customers come in contact with your brand?
Where will they first learn about your business?
• Think about the various touch points before people buy, during
purchase, and after
• Don’t worry too much about cost (initially), focus more on
channel success in terms of conversion and quality of users/
customers
48. TALKING BUSINESS: MAPPING YOUR BUSINESS
6. Testing Revenue
• How will you make money? Who will pay?
• Early on, revenue is a proxy of value creation, but now you have
to test the economic engine of the business more seriously
• Remember: users of your product may not be the customers, or
you may have multiple customers (e.g. top-down enterprise
software)
• Metrics of importance: ARPU, LTV, CAC, MRR, sales, etc.
49. TALKING BUSINESS: MAPPING YOUR BUSINESS
6. Testing Pricing
http://download.red-gate.com/ebooks/DJRTD_eBook.pdf
“Prices are a shortcut to our most sensitive emotional
responses.” – Tom Whitwell
https://medium.com/dreamit-perspectives/founders-guide-to-product-pricing-
b187093e2483#.vdpwcl752
! Don’t overcomplicate too early
! Sell based on aspirational value (not negative value)
! Don’t ask customers what they’d pay, they don’t know
! Test it (over and over again)
50. TALKING BUSINESS: MAPPING YOUR BUSINESS
7. Measuring Costs
• What are the costs?
• Where do they come from?
• How are they impacted by growth/scale?
• For Web-related businesses, costs are usually low and not
worth focusing on initially; but for other types of businesses
(manufacturing, hardware) costs are much higher. For Web-
related businesses, costs are mostly in acquisition (where LTV >
CAC becomes important)
52. TALKING BUSINESS: MAPPING YOUR BUSINESS
9. Defining an Unfair Advantage
• What makes you stand out from competitors?
• What do you know or have access to that no one else does?
• This is very hard to come up with; there are very few legitimate
unfair advantages
• Insider information
• Single-minded obsession with the “One Thing”
• Personal authority
• Existing customers / distribution
http://blog.asmartbear.com/unfair-advantages.html
53. ZxLERATOR | NYC | SUMMER 2016 53
Day 3 - Business Models &
Value Propositions
LEAN CANVAS
Example Lean Canvas: Freckle
http://blog.asmartbear.com/unfair-advantages.html
http://www.slideshare.net/de-pe/lean-canvas-process-and-examples
54. TALKING BUSINESS: MAPPING YOUR BUSINESS
1
2
3
Fill out your Lean Canvas in 20 minutes
Think through the business assumption
questions from Talking to Humans
We’ll share and discuss after
Time to give it a try!
56. Product management is
the “glue” between
everyone and everything
TALKING BUSINESS: PRODUCT MANAGEMENT
57. PRODUCT MANAGEMENT IS THE
PROCESS BY WHICH WE TURN
VISION INTO REALITY
PRODUCT MANAGEMENT =
STRATEGY - PRIORITIZATION EXECUTION
58. TALKING BUSINESS: PRODUCT MANAGEMENT
The role of product managers
• Product managers are the “glue” between all stakeholders
(internal & external)
• Work towards achieving the company’s vision through strategy
& execution
• Empower others to get things done
• Prioritize feature development based on all the inputs
• Ensure deliverability as expected (on time)
• Measure results (success or failure)
59. TALKING BUSINESS: PRODUCT MANAGEMENT
“The job of a product manager is
to: Help your team (and company)
ship the right product to your
users.”
- Josh Elman
https://medium.com/@joshelman/a-product-managers-job-63c09a43d0ec#.5mhlul6l1
60.
61.
62. DATA IS A KEY INPUT AND
FILTER IN PRODUCT
MANAGEMENT
65. @byosko
Company vision
Internal and
external inputs
Perpetual problem / solution validation
Project Scope Creation
Sprints
Quarterly roadmap Quarterly roadmap Quarterly roadmapQuarterly roadmapBacklog
How it all comes together
TALKING BUSINESS: PRODUCT MANAGEMENT
66. @byosko
Company vision
Internal and
external inputs
Perpetual problem / solution validation
Project Scope Creation
Sprints
Quarterly roadmap Quarterly roadmap Quarterly roadmapQuarterly roadmapBacklog
How it all comes together
TALKING BUSINESS: PRODUCT MANAGEMENT
DATA HAS AN IMPACT ON EVERY STEP IN THE
PRODUCT DEVELOPMENT PROCESS
67. TALKING BUSINESS: PRODUCT MANAGEMENT
Product roadmaps (problem discovery)
Product roadmaps need to answer
three critical questions:
1. Where are we going? (Vision)
2. Why are we going there? (Business
Goals / Value Creation)
3. And how are we going to do it?
(Resources & Planning)
68. TALKING BUSINESS: PRODUCT MANAGEMENT
Best practices for great product roadmaps
DON’T
FORGET
TO KISS!
• Focus on goals/outcomes not features
• Categorize and organize by themes
• Make sure problems are clearly defined and
understood
• Timelines are loose at best (this isn’t the core focus)
• Use as a conversation starter, not as something writ
in stone
69. TALKING BUSINESS: PRODUCT MANAGEMENT
Identifying the core problems / goals (objectives)
• Collect and analyze feedback / inputs
• Identify the core problems (goals) for the business
(problem discovery)
• Identify the One Metric That Matters for each
problem/goal and draw a line in the sand
• Hypothesize potential solutions
• Be outcome driven
70. Collecting input
Product & Design
(defining goals /
objectives)
User & customer
feedback
Sales
Marketing
Customer Support
Etc.
! In-person interviews
! Surveys
! Customer support
inquiries
! Real-time online
Supported
by data
Your gut
Company vision
Your own ideas
71. TALKING BUSINESS: PRODUCT MANAGEMENT
Collecting input
• Do you have a customer panel? (e.g. private FB
group, mailing list, etc.)
• Can you schedule regular in-person sessions?
• Be wary of salespeople suggesting, “If we only built
feature X, I could close this deal.”
• Attend customer and sales meetings; spend time on
the help desk.
• Work with the product team to segment your user/
customer base.
72. TALKING BUSINESS: PRODUCT MANAGEMENT
Intercom’s product roadmap inputs
“Building a great product
is an art as much as a
science. It requires
making hard decisions
and trade-offs, in
circumstances ranging
from being overwhelmed
with data to having no
data.”
https://blog.intercom.io/where-do-product-roadmaps-come-from/
73. Roman Pichler’s GO Product Roadmap
! Blends high-level roadmap and
release planning
! Name can function as a theme
! Goal is a high-level purpose
aligned with the stage you’re at
(e.g. Stickiness, Virality, etc. or
use AARRR)
! Roadmap leads to epics,
stories, specs & MVPs
http://www.romanpichler.com/blog/working-go-
product-roadmap/
The release date or
timeframe
The reason for
creating the release
The metrics to
determine if the goal
has been met
The high-level
features necessary to
meet the goal
The name of the new
release
Date or timeframe Date or timeframe Date or timeframe Date or timeframe
Name / version Name / version Name / version Name / version
FeaturesFeaturesFeaturesFeatures
Goal Goal Goal Goal
MetricsMetricsMetricsMetrics
74. TALKING BUSINESS: PRODUCT MANAGEMENT
Feature prioritization
1. Start with the key goals / outcomes & metrics
2. Rank order the solutions (features) based on value creation
a. Identify a “line in the sand” for each solution (target)
3. Rank order the solutions based on effort
Ultimately, what will create
the MOST value for the LEAST effort?
75. Idea / Solution Score Target Engagement
Value
Effort
Send more push notifications at regular
intervals regarding key actions we want users
to take inside the app
9 +10% MAU 3 1
Improve first user experience by providing
users with a key action they need to take right
away (no “white screen of death”)
5 +15% MAU 5 3
Add more robust reporting features for date
range searches, saved reports
1.5 +5% MAU 2 4
Leaderboard for users hitting sales targets 1.5 +3% MAU 1 2
Goal: Increase Engagement
(target: +15% MAU)
Try using a “dot voting”
system per idea/solution
Only a rough estimate is
needed here; goal is to
have comparative values
@byosko@byosko
Use a simple weighted
formula to score/rank items
(e.g. Engagement Value *
3 / Effort)
TALKING BUSINESS: PRODUCT MANAGEMENT
Sample ranking system
76. TALKING BUSINESS: PRODUCT MANAGEMENT https://blog.intercom.io/rice-simple-prioritization-for-product-managers/
! Reach: How many people will be impacted by the new
feature over a given period of time?
! Impact: How much will this new feature impact an
individual user/customer? 3 for “massive impact”, 2 for
“high”, 1 for “medium”, 0.5 for “low” and finally 0.25 for
“minimal”.
! Confidence: How confident are we in our estimates?
100% is “high confidence”, 80% is “medium”, 50% is
“low”.
! Effort: How much time will it take to deliver the new
feature? Measure in “person months”.
Reach x Impact x Confidence
Effort
RICE
Score
Intercom’s RICE model for feature prioritization
77. TALKING BUSINESS: PRODUCT MANAGEMENT
Data’s role in product management
• As an input for discovering interesting trends that are
worth pursuing in product development
• As a filter for validating the relative importance of
features
• As a check throughout the entire product
development process from establishing a product
roadmap, to feature prioritization, to scope creation
and sprint management
• This is all about effective communication
84. LEAN ANALYTICS: THE BASICS
1
2
3
Think about a project or product you’re involved
with today
Can you write down the key metrics that
matter?
We’ll compare after going through the next
section
Quick exercise
85. LEAN ANALYTICS: THE BASICS
Analytics is the measurement of
movement towards business goals.
86. LEAN ANALYTICS: THE BASICS
A GOOD METRIC IS:
Understandable
If you’re busy explaining
the data, you won’t be
busy acting on it.
Comparative
Active Users vs.
Active Users/month
A ratio or rate
% Monthly Active
Users
Behavior changing
You know how you’ll
change your business
based on what the
metric tells you.
87. LEAN ANALYTICS: THE BASICS Day 3 - Lean Analytics
ZxLERATOR | NYC | SUMMER 2016 87
Herbert Simon
If a metric won’t
change how you
behave, it’s a…
BAD
METRIC
88. Metrics help you know yourself
LEAN ANALYTICS: THE BASICS Day 3 - Lean Analytics
Acquisition
Hybrid
70%
of retailers
You are just
like
Customers that buy
>1x in 90d
Once
2-2.5
per year
Your customers will
buy from you
Then you are in
this mode
1-15%
15-30%
Low acquisition cost,
high checkout
Focus on
20%
of retailers
Increasing return
rates, market share
Loyalty>30%
Loyalty, selection,
inventory size
>2.5
per year
10%
of retailers
(Thanks to Kevin Hillstrom for this.)
90. LEAN ANALYTICS: THE BASICS
VANITY vs. ACTIONABLE
Vanity Actionable
Makes you feel good but
doesn’t change how you’ll act.
Helps you pick a direction and
change your behavior.
“Up and to the right” These are good.
91. LEAN ANALYTICS: THE BASICS
QUALITATIVE vs. QUANTITATIVE
Qualitative Quantitative
Unstructured, anecdotal,
revealing, hard to aggregate.
Numbers and stats; hard
facts, but less insights.
Warm and fuzzy. Cold and hard.
93. LEAN ANALYTICS: THE BASICS
Do Airbnb hosts get more
business if their property
is professionally
photographed?
94. LEAN ANALYTICS: THE BASICS
Case study: Does professional photography
make a difference?
Gut instinct (hypothesis)
Professional photography helps Airbnb’s business
Built a Concierge MVP
Sent 20 photographers out into the field
Measured results
Compared photographed listings to control group
Made a decision
Launched photography as a new feature to all hosts
96. LEAN ANALYTICS: THE BASICS
EXPLORATORY vs. REPORTING
Exploratory Reporting
Speculative. Tries to find
unexpected insights. Source
of unfair advantage.
Predictable. Keeps you
abreast of normal, day-to-day
operations. Can be managed
by exception.
Cool. Necessary.
97. LEAN ANALYTICS: THE BASICS
Case study: Finding insights in the data
• Started as Circle of Friends
• Leveraged Facebook early
• Grew to 10M users fast
ENGAGEMENT SUCKED!
98. LEAN ANALYTICS: THE BASICS
Case study: Moms are crazy (but in a good
way!)
ENGAGEMENT SOLVED!
• Messages to one another were ~50% longer
• 115% more likely to attach a picture to a post
• 110% more likely to engage in a threaded conversation
• Invited friends were 50% more likely to become engaged users
• 60% more likely to accept invitations to the app
99. LEAN ANALYTICS: THE BASICS
LAGGING vs. LEADING
Lagging Leading
Historical metric that shows
you how you’re doing: reports
the news.
Number today that shows a
metric tomorrow: makes the
news.
Start here. Try and get here.
100. LEAN ANALYTICS: THE BASICS
Examples of leading metrics
A Facebook user reaching 7 friends within 10 days of signing up.
(Chamath Palihapitiya)
A Dropbox user who puts at least 1 file in 1 folder on 1 device. (ChenLi
Wang)
A Twitter user who follows a certain number of people, and a certain
percentage of those people follow the user back. (Josh Elman)
A LinkedIn user getting to X connections in Y days. (Elliot Schmukler)
101. LEAN ANALYTICS: THE BASICS
Case study: Buffer’s leading metrics
revealed
Buffer discovered 3 leading metrics for long-term engagement:
1 People who install the Chrome extension
2 People who connect more than 1 social account
3 People who share 15 pieces of content in 7 days
102. Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Correlation vs. causation
103. LEAN ANALYTICS: THE BASICS
CORRELATED vs. CAUSAL
Correlated Causal
Two variables that are related
(but may be dependent on
something else.)
An independent variable that
directly impacts a dependent
one.
Ice cream and
drowning.
Ice cream and summertime.
Drowning and summertime.
105. Case study: Putting basic data to use
Ricky (product manager) had some ideas for improving the “Proposal Send Screen” (based on
qualitative feedback & gut), but before prioritizing this work, he digs into the data.
• 50% of people send proposals through Proposify
(50% don’t) (quantitative) — Is this good or bad?
Ricky isn’t sure. So he’s going to need to look at
additional data (exploratory):
• Churn
• Size of customer
• Proposal won rate
• Any correlations here?
• Can also do some direct customer developer to
learn more (qualitative)
• Might lead to additional, meaningful product dev
(actionable)
106. LEAN ANALYTICS: THE BASICS
1
2
Look at the metrics you wrote down before,
how many of them stand up?
How can you make your metrics simpler for
people to understand?
Quick exercise
109. LEAN ANALYTICS: THE FRAMEWORK
Lean Analytics Framework
What business
are you in?
What stage are
you at?
• E-Commerce
• SaaS (freemium)
• Mobile app (gaming)
• Two-sided marketplace
• Media
• User-generated content
• Empathy
• Stickiness
• Virality
• Revenue
• Scale
113. LEAN ANALYTICS: THE FRAMEWORK
Eric Ries’s Three Engines of Growth
Stickiness Virality Price
Approach
Math that
matters
Keep people
coming back.
Get customers
faster than you
lose them.
Make people invite
friends.
How many they
tell, how fast they
tell them.
Spend money to
get customers.
Customers are
worth more than
they cost.
114. LEAN ANALYTICS: THE FRAMEWORK
Lean Analytics Stages
Empathy
Stickiness
Virality
Revenue
Scale
Stage
You’ve found a real, poorly-met need that a reachable market
faces.
You’ve figured out how to solve the problem in a way that users
will adopt, keep using, and pay for.
Your users and features fuel growth organically and artificially.
You’ve found a sustainable, scalable business with the right
margins in a healthy ecosystem.
Gate
115. LEAN ANALYTICS: THE FRAMEWORK
Lean Analytics Stages
Empathy
Stickiness
Virality
Revenue
Scale
Stage
You’ve found a real, poorly-met need that a reachable market
faces.
You’ve figured out how to solve the problem in a way that users
will adopt, keep using, and pay for.
Your users and features fuel growth organically and artificially.
You’ve found a sustainable, scalable business with the right
margins in a healthy ecosystem.
Gate
Most projects/products/
startups fail here
116. LEAN ANALYTICS: THE FRAMEWORK
Case Study: Jumping the gun on product
development
• Stage: Empathy/Stickiness
• Model: E-Commerce
• Originally tied to Instagram with
an “Insta-Order” feature
117. LEAN ANALYTICS: THE FRAMEWORK
Case Study: Optimize for 1st purchase or
repeat orders?
LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics
With Insta-Order feature Without Insta-Order
• 2x transactions
• Lower bounce rate
• Sign-in goals increased
118. ZxLERATOR | NYC | SUMMER 2016 118
“THERE ARE NO SHORTCUTS TO
ANY PLACE WORTH GOING.”
- Beverly Sills
119. LEAN ANALYTICS: THE FRAMEWORK
Case Study: Localmind hacks Twitter
(use a proxy to test your hypotheses)
BIGGEST RISK:
Would people be willing to answer
questions about a place in real-
time?
120. LEAN ANALYTICS: THE FRAMEWORK
Case Study: Localmind hacks Twitter
(use a proxy to test your hypotheses)
Herbert Simon Tested results
• The response rate to tweeted questions was very high
• Good enough proxy to de-risk the solution and convince the team to continue
Ran an experiment on Twitter
• Tracked geolocated tweets in Times Square
• Sent @ messages to people who had just tweeted, asking questions about the area
Would people answer questions?
• Before writing a line of code, they wanted to answer this question
• This was their biggest risk; if questions went unanswered, the experience would suck
126. LEAN ANALYTICS: THE FRAMEWORK
Case Study: WineExpress A/B tests what
really matters
LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics
Think of this over time…A B
127. Case Study: WineExpress A/B tests
what really matters
B
• 41% increase in revenue per
customer! (People bought a lot
more product.)
• Conversion also went up, but
was secondary in importance
128. LEAN ANALYTICS: THE FRAMEWORK
All business models have issues
E-Commerce
SaaS
Mobile Apps
2-Sided Marketplace
Media
UCG
CAC vs. LTV — margins are usually small. A $10M e-commerce business is
often considered quite small.
Freemium requires tens of millions of free users. They can be expensive to
support. Will enough convert?
The average # of apps downloaded by North Americans per month is now 0.
Monetizing is incredibly hard. Popularity is fleeting.
Chicken & egg problem. Supply and demand. How do you build up enough of
both?
Real monetization requires hundreds of millions of engaged visitors. People’s
attention is hard to capture and keep.
Content creation. Will it be good enough? Will enough people do it? Why?
129. YOU KNOW WHAT BUSINESS YOU’RE
IN AND WHAT STAGE YOU’RE AT.
NOW WHAT?
130. LEAN ANALYTICS: THE FRAMEWORK
The One Metric That Matters
The business you’re in
E-Commerce SaaS Mobile 2-Sided Marketplace UCG Media
Thestageyou’reat
Empathy
Stickiness
Virality
Revenue
Scale
ONE METRIC THAT MATTERS
131. LEAN ANALYTICS: THE FRAMEWORK
The One Metric That Matters
• The metric should indicate that your user experienced the
core value of the product
• It should reflect user’s engagement and activity level
• It should be the “one thing” that indicates the business is
heading in the right direction
• The metric ideally should be easy to understand &
communicate across teams
• The OMTM will change over time as the business evolves
132. LEAN ANALYTICS: THE FRAMEWORK
Case study: Moz cuts down on metrics to
track
SaaS-based SEO toolkit in the Scale stage.
Focused on net adds.
Net adds up:
Was a marketing campaign successful?
Were customer complaints lowered?
Was a product upgrade valuable?
Net adds flat:
Can we acquire more valuable customers?
What product features can increase engagement?
Can we improve customer support?
Net adds down:
Are the new customers not the right segment?
Did a marketing campaign fail?
Did a product upgrade fail somewhere?
Is customer support falling apart?
133. Examples of the One Metric That Matters
LEAN ANALYTICS: THE FRAMEWORK
# of transactions (for
merchants)
# of nights booked sales
total time reading
https://medium.com/data-lab/mediums-metric-that-matters-total-time-reading-86c4970837d5#.tidx5bunj
http://quibb.com/links/metrics-to-inform-your-model-lessons-from-square-stripe-and-quora
http://500.co/aircall-growth-uber/
monthly active users monthly recurring
revenue (MRR)
136. LEAN ANALYTICS: THE FRAMEWORK
The Layer Cake of Metrics
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Project
OMTM
Department OMTM Department OMTM Department OMTM
OMTM: Business Help Indicator
138. LEAN ANALYTICS: THE FRAMEWORK
Some interesting benchmarks
Growth
5% / week (revenue or
active users)
Time on site
17 minutes
Free to paid
2% of free users
Mobile file size
< 50MB
Engaged visitors
30% monthly users
10% daily users
Paid load time
< 5 seconds
Churn
2% / month
CLV:CAC
3:1
142. Draw a new line
ZxLERATOR | NYC | SUMMER 2016 142
Lean Analytics Cycle
LEAN ANALYTICS: THE FRAMEWORK Day 4 - Lean Analytics
Pivot or
give up
Try again
Success!
Did we move the
needle?
Measure the
results
Make changes in
production
Design a test
Hypothesis
With data:
find a
commonality
Without data:
make a good
guess
Find a potential
improvement
Draw a linePick a OMTM
145. LEAN ANALYTICS: THE BASICS
Map your business model
1
2
Map your business as a systems diagram
Think about all the touch points of a user /
customer and how they interact with your
business (idea, product, people, etc.)
3
What are the riskiest areas / biggest areas of
uncertainty?
4
Where are you going to focus? Can you identify
the OMTM at each stage of your business?
148. THE END!
1
Data people need to be able to
“talk business”
So where are we at now?
• Understand the types of innovation and how a
company is approaching new product development
& business models
• Lean Startup applies beyond startups, but it’s not
easy
• Understand how the sausage is made (product
management)
• Product isn’t built in a vacuum; there’s a rigorous,
experiment-oriented process that helps
• Get involved in product roadmaps & feature
prioritization
• Data is both an input and a filter into the entire
process of building a business
149. THE END!
1
Data people need to be able to
“talk business”
So where are we at now?
• Understand the types of innovation and how a
company is approaching new product
development & business models
• Lean Startup applies beyond startups, but it’s not
easy
• Understand how the sausage is made (product
management)
• Product isn’t built in a vacuum; there’s a rigorous,
experiment-oriented process that helps
• Get involved in product roadmaps & feature
prioritization
• Data is both an input and a filter into the entire
process of building a business
2
Everyone else needs to be able to
“speak data”
• Data is the common language that everyone
needs to be reasonably fluent in
• We need to simplify the data (not what we track)
for others
• Data needs to be tied to real/core business goals
• The One Metric That Matters is a good construct
for forcing simplification and focus
• It’s important to understand the business you’re in
and the stage you’re at in order to identify the right
metrics to focus on
150. Once, a leader convinced others
in the absence of data.