Startup metrics toronto March 19

Lean Analytics
Use data to build a
better business faster.

www.leananalyticsbook.com
@byosko | @acroll
@leananalytics
Sunday, 24 March, 13

Some housekeeping.


Kevin Costner is a lousy entrepreneur.

Don’t sell what you can make.
Make what you can sell.


Analytics is the measurement
of movement towards your
business goals.

http://www.ﬂickr.com/photos/itsgreg/446061432/


Small business example:
Solare watches the
numbers

• Stage: Revenue
• Model: Retailer
• Solare is an Italian ﬁne-dining restaurant under new management. The new team
is trying to identify the key metrics and leading indicators


Solare watches the numbers

• A line in the sand: Gross Revenue to Labor Cost
• Under 30% is good
• Below 24% is great
• Lower than 20% and you may be under-stafﬁng, leading to dissatisﬁed
customers
• A leading indicator: Total covers is 5x reservations at 5PM
• If you have 50 reservations at 5, you’ll have 250 covers that night.
• This ratio varies by restaurant.


In a startup, the purpose of
analytics is to iterate to a
product/market ﬁt before
the money runs out.


Most startups don’t know what they’ll be
when they grow up.

Freshbooks Mitel
was invoicing Wikipedia was a
Paypal
for a web was to be lawnmower
ﬁrst built for company
design ﬁrm written by
Palmpilots experts only

Flickr
Hotmail Twitter Autodesk
was going to
was a was a made desktop
be an MMO
database podcasting automation
company company


Five things you need to know

•What makes a good metric

•Understanding cohorts and segments

•The business model ﬂipbook

•The Lean Analytics framework

•Picking One Metric That Matters


Clear, comparable ratios
Tied to your business model
A good metric is Actionable, not vain
Correlated or Causal
Leading or Lagging


Comparable ratios: think about a car

• Clear: You know 60MPH is twice as fast as 30MPH
• In a country, speed limits and mileage are well
understood
• Kilometers are conveniently decimal; miles map to
hours
• Rates: Miles travelled is good; miles per hour is better;
accelerating or decelerating changes your gas pedal
• Business model: You can measure “MPH divided by
speeding tickets” as a metric of “driving fast without
losing my license”

Vanity Actionable
Picks a
direction.

Makes you feel
good, but doesn’t
change how you’ll
act.
http://www.ﬂickr.com/photos/lostseouls/807253220/ http://www.ﬂickr.com/photos/aussiegall/6382775153/

A metric from the early, foolish days of the Web.
Hits
Count people instead.
Marginally better than hits. Unless you’re displaying
Page views
ad inventory, count people.
Is this one person visiting a hundred times, or are a
Visits
hundred people visiting once? Fail.
This tells you nothing about what they did, why they
Unique visitors
stuck around, or if they left.
Followers/ Count actions instead. Find out how many followers
friends/likes will do your bidding.
Time on site, or Poor version of engagement. Lots of time spent on
pages/visit support pages is actually a bad sign.
How many recipients will act on what’s in them?
Emails collected

Number of Outside app stores, downloads alone don’t lead to
downloads lifetime value. Measure activations/active accounts.

http://www.ﬂickr.com/photos/circasassy/7858155676/

If it won’t change
how you behave,
it’s a

bad
metric.


2-sided market model:
AirBnB and photography

• Stage: Revenue
• Model: 2-sided marketplace
• Rental-by-owner marketplace that allows property owners to list and market
their houses. Offers a variety of related services as well.


AirBnB tests a hypothesis

• The hypothesis: “Hosts with professional photography will get more business.
And hosts will sign up for professional photography as a service.”

• Built a concierge MVP

• Found that professionally photographed listings got 2-3x more bookings than the
market average.

• In mid-to-late 2011, AirBnB had 20 photographers in the ﬁeld taking pictures for
hosts.


NIGHTS BOOKED

10 million

8 million

6 million

20 photographers
4 million

2 million

2008 2009 2010 2011 2012

Friday, November 9, 12

A few words on causality.

http://www.ﬂickr.com/photos/roryﬁnneren/65729247

50

37.5

25

12.5

0
1 2 3 4 5 6 7 8 9 10
Seat rentals
http://www.rvca.com/anp/wp-content/plugins/wp-o-matic/cache/57226_07+proof+1a+hb+beach+day.jpg

http://www.imdb.com/media/rm3768753408/tt0073195

http://www.ﬂickr.com/photos/kapungo/2287237966

10000

1000

100

10

1
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec

Ice cream consumption Drownings

http://www.ﬂickr.com/photos/25159787@N07/3766111564

http://www.ﬂickr.com/photos/wheressteve/3284532080

http://www.ﬂickr.com/photos/wtlphotos/1086968783

10000

1000

100

10

1
Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Ice cream consumption Drownings Temperature


http://www.ﬂickr.com/photos/stuttermonkey/57096884

http://www.ﬂickr.com/photos/germanuncut77/3785152581

http://www.ﬂickr.com/photos/fasteddie42/2421039207

Correlated Causal
Two variables that An independent
change in similar factor that directly
ways , perhaps impacts a
because they’re dependent one.
linked to something
else.
Summer
al

Ca
us

us
Ca

Correlated al Drowning
Ice cream
consumption

http://www.ﬂickr.com/photos/bootbearwdc/1243690099/

Causality is a superpower, because it lets you
change the future.

Correlation lets you Causality lets you
predict the future change the future

“I will have 420 “If I can make more
engaged users and ﬁrst-time visitors stay
75 paying customers on for 17 minutes I
next month.” will increase sales in
90 days.”

Optimize the
Find correlation Test causality
causal factor

Leading Lagging
Number today that Historical metric that
shows metric shows how you’re
tomorrow—makes doing—reports the
the news. news.


What mode of e-commerce are you?

How many of
your customers Then you are in Your customers You are just
Focus on
buy a second this mode will buy from you like
time in 90 days?

Low CAC,
1-15% Acquisition Once 70% high
of retailers checkout

15-30% Hybrid 2-2.5 20% Increasing
per year of retailers returns

Loyalty,
>30% Loyalty >2.5 10% inventory
per year of retailers expansion
(Thanks to Kevin Hilstrom for this.)

• A Facebook user reaching 7 friends within 10 days of
signing up (Chamath Palihapitiya)
• If someone comes back to Zynga a day after signing up
for a game, they’ll probably become an engaged, paying
user (Nabeel Hyatt)
• A Dropbox user who puts at least one ﬁle in one folder
on one device (ChenLi Wang)
• Twitter user following a certain number of people, and a
certain percentage of those people following the user
back (Josh Elman)
• A LinkedIn user getting to X connections in Y days (Elliot
Schmukler)

(These are also great segments to analyze.)
(from the 2012 Growth Hacking conference)

So how do you
test things?
Segmentation.
http://www.ﬂickr.com/photos/zlakfoto/5294803278/

Segments, cohorts, A/B, and multivariates

Cohort:
Comparison of
similar groups
along a timeline.

Segment: A/B test: ☀ Multivariate
Cross-sectional ☀ Changing one analysis
comparison of all thing (i.e. color)
☁ Changing several
people divided by and measuring ☀ things at once to
some attribute
☁ the result (i.e. see which correlates
☁
(age, gender, etc.) revenue.) with a result.


Why use cohorts? Here’s an example.

Is this
January February March April May
company
growing or
Rev/customer $5.00 $4.50 $4.33 $4.25 $4.50
stagnating?

Cohort 1 2 3 4 5

January $5 $3 $2 $1 $0.5

How about February $6 $4 $2 $1
now?
March $7 $6 $5

April $8 $7

May $9


Why use cohorts? Here’s an example.

Cohort 1 2 3 4 5

January $5 $3 $2 $1 $0.5

Look at the February $6 $4 $2 $1
same data
in cohorts March $7 $6 $5

April $8 $7

May $9

Averages $7 $5 $3 $1 $0.5


What business are you in?


Business model ﬂipbook
Revenue model: How you take money from someone
Product type: What you give them in return Together, these
Delivery model: How you get it to them make up a
Acquisition channel: How they learn about you business model
Selling tactic: How you convince them to buy


Paid advertising Banner on Informationweek.com
Search Engine Mgmt. High pagerank for ELC in kid’s toys
Acquisition Social media outreach Active on Twitter i.e. Kissmetrics

channel
How the visitor, Inviting team member to Asana
Inherent virality
customer, or user finds
out about the startup. Artificial virality Rewarding Dropbox user for others’ signups
Affiliate marketing Sharing a % of sales with a referring blogger
Public relations Speaker submission to SXSW
App/ecosystem mkt. Placement in the Android market

Simple purchase Buying a PC on Dell.com
What the startup does Discounts & incentives Black Friday discount, loss leader, free ship
Selling
tactic

to convince the visitor Free trial Time-limited trial such as fitbit Premium
or user to become a Freemium Free tier, relying on upgrades, like Evernote
paying customer.
Pay for privacy Free account content is public, like Slideshare
Free-to-play Monetize in-app purchases, like Airmech
One-time transaction Single purchase from Fab
How the startup Recurring subscription Monthly charge from Freshbooks
Revenue
model

extracts money from its Consumption charges Compute cycles from Rackspace
visitors, users, or
Advertising clicks PPC revenue on CNET.com
customers.
Re-sale of user data Twitter’s firehose license
Donation Wikipedia’s annual campaign

Software Oracle’s accounting suite
What the startup does Platform Amazon’s EC2 cloud
in return. May be a
Product

Merchandising Thinkgeek’s retail store
type

product or service; may
be hardware or User-generated content Facebook’s status update
software; may be a Marketplace AirBnB’s list of house rentals
mixture. Media/content CNN’s news page
Service A hairstylist
Delivery

Hosted service Salesforce.com’s CRM
model

How the product gets Digital delivery Valve purchase of desktop game
to the customer. Physical delivery Knife shipped from Sur La Table

Business Flipbook
Dropbox example
aspect page(s)

Acquisition Inherent virality. Sharing ﬁles with others.
channel Artiﬁcial virality. Free storage when others sign up.

Selling Limited-capacity accounts are free;
Freemium.
tactic subscribe when you need more.

Revenue Recurring $99/year, monthly fees, enterprise
model subscription. tiers.

Product Storage-as-a-service with APIs,
Platform.
type collaboration, synchronization tools.

Delivery Hosted service. Cloud storage, web interface.
model Digital delivery. Desktop client software.


The Lean Analytics Framework.


Eric Ries’
Three engines

Stickiness Virality Price

Approach Keep people Make people Spend revenue
coming back. invite friends. getting customers.

Math that Get customers How many they Customers are
matters faster than you tell, how fast worth more than
lose them. they tell them. they cost to get.


The ﬁve Stages of Lean Analytics

The business you’re in

E- 2-sided Mobile User-gen
SaaS Media
commerce market app content
Empathy
The stage you’re at

One Metric
Stickiness

Virality

Revenue That Matters.
Scale

Example: a restaurant

• Empathy: Before opening, the owner ﬁrst learns about the diners in its area,
their desires, what foods aren’t available, and trends in eating.
• Stickiness: Then he develops a menu and tests it out with consumers, making
frequent adjustments until tables are full and patrons return regularly. He’s giving
things away, testing things, asking diners what they think. Costs are high
because of variance and uncertain inventory.
• Virality: He starts loyalty programs to bring frequent diners back, or to
encourage people to share with their friends. He engages on Yelp and
Foursquare.
• Revenue: With virality kicked off, he works on margins—fewer free meals,
tighter controls on costs, more standardization.
• Scale: Finally, knowing he can run a proﬁtable business, he pours some of the
revenues into marketing and promotion. He reaches out to food reviewers, travel
magazines, and radio stations. He launches a second restaurant, or a franchise
based on the initial one.


Example: a software company

• Empathy: The founder finds an unmet need, often because she has a background in a
particular industry or has worked with existing solutions that are being disrupted.
• Stickiness: She meets with an initial group of prospects, and signs contracts that look more
like consulting agreements, which she uses to build an initial product. She’s careful not to
commit to exclusivity, and tries to steer customers towards standardized solutions, charging
heavily for custom features. She supports the customers directly from the engineering team
until the product is stable and usable.
• Virality: Product in hand, she asks for references from satisfied customers, and uses them as
testimonials. She starts direct sales, and grows the customer base. She launches a user group,
and starts to automate support. She releases an API, encouraging third-party development and
scaling potential market size without direct development.
• Revenue: She focuses on growing the pipeline, sales margins, and revenues while controlling
costs. Tasks are automated, outsourced, or offshored. Feature enhancements are scored
based on anticipated payoff and development cost. Recurring license and support revenue
becomes an increasingly large component of overall revenues.
• Scale: She signs deals with large distributors, and works with global consulting firms to have
them deploy and integrate her tool. She attends trade shows to collect leads, carefully
measuring cost of acquisition against close rate and lead value.


Empathy stage:
Localmind hacks Twitter

• Stage: Empathy
• Model: UGC/mobile
• Real-time question and answer platform tied to locations.
• Needed to ﬁnd out if a core behavior—answering questions about a place—
happened enough to make the business real


Localmind hacks Twitter

• Before writing a line of code, Localmind was concerned that people would never
answer questions.
• This was their biggest risk: if questions went unanswered users would have a
terrible experience and stop using Localmind.
• Ran an experiment on Twitter
• Tracked geolocated tweets in Times Square
• Sent @ messages to people who had just tweeted, asking questions about
the area: how busy is it; is the subway running on time; is something open;
etc.
• The response rate to their tweeted questions was very high.
• Good enough proxy to de-risk the solution, and convince the team and
investors that it was worth building Localmind.


Stickiness stage:
WP Engine discovers the
2% cancellation rate

• Stage: Stickiness
• Model: SaaS
• Wordpress hosting company founded in July 2010, it raised $1.2M in November
2011


WP-Engine discovers the 2%
cancellation rate

• All companies have cancellations, but founder Jason Cohen was alarmed that he
was losing a quarter of customers every year.

• Jason called customers himself. “Not everyone wanted to speak with me, but
enough people were willing to talk, even after they had left, that I learned a lot
about why they were leaving.”

• Asked around. Turns out 2% is best case for most hosting companies.

• Without this, the company would have been getting diminishing returns over-
optimizing churn; instead, they could focus on maximizing revenues or lowering
acquisition costs.


Virality stage:
qidiq streamlines invites

• Stage: Virality
• Model: SaaS
• Tool to poll small groups, built in the Year One Labs accelerator


Initial design Redesigned workflow
Survey owner adds recipient to group Survey owner adds recipient to group

70-90% RESPONSE RATE
Survey owner asks question Survey owner asks question

Recipient gets invite Recipient reads survey question
10-25% RESPONSE RATE

Recipient installs mobile app Recipient responds to question

Recipient sees survey results
Recipient creates account, profile

Recipient can edit profile, view past (Later, if needed…)
questions, etc.
Recipient visits website
Recipient reads survey question
Recipient has no password!
Recipient responds to question
Recipient does password recovery
Recipient sees survey results
One-time link sent to email

Recipient creates password

Recipient can edit profile, view past
questions, etc.


Revenue stage:
Backupify’s customer
lifecycle

• Stage: Scale
• Model: SaaS
• Leading backup provider for cloud based data.
• The company was founded in 2008 by Robert May and Vik Chadha
• Has gone on to raise $19.5M in several rounds of ﬁnancing.


Shifting to Customer Acquisition
Payback as a key metric

• Initially focused on site visitors

• Then focused on trials

• Then switched to signups

• Today, MRR

• In early 2010, CAC was $243 and ARPU was only $39

• Pivoted to target business users

• CLV-to-CAC today is 5-6x

• Now they track Customer Acquisition Payback

• Target is less than 12 months


What’s your OMTM?
E- 2-sided Mobile User-gen
SaaS Media
commerce market app content

Empathy Interviews; qualitative results; quantitative scoring; surveys

Loyalty, Inventory, Engagement, Downloads, Content, Traffic, visits,
Stickiness conversion listings churn churn, virality spam returns

CAC, shares, Inherent WoM, app Invites, Content
Virality reactivation
SEM, sharing
virality, CAC ratings, CAC sharing virality, SEM

(Money from transactions) (Money from active users) (Money from ad clicks)

Transaction, Transactions, Upselling, CLV, Ads, CPE, affiliate
Revenue CLV commission CAC, CLV ARPDAU donations %, eyeballs

Affiliates, Other API, magic Spinoffs, Analytics, Syndication,
Scale white-label verticals #, mktplace publishers user data licenses


B2B and intrapreneurs


The B2B stereotype

• Domain expert knows
industry and the problem
domain. Has a Rolodex;
proxy for customers.

http://www.techdigest.tv/2007/02/im_a_pc_im_a_ma.html
• Disruption expert knows
tech that will produce a
change Sees beyond the
current model.

Domain Disruption
expert expert Operations

Three typical approaches

Create a popular consumer Dropbox
Enterprise pivot product then pivot to tackle the
enterprise

Take an existing consumer or Yammer,
Copy and rebuild open source idea and make it MapR
enterprise-ready

Convince the enterprise to Taleo,
Disrupt a problem discard the old way because of Google
overwhelming advantages. Apps


Lean Analytics lifecycle
for an enterprise-focused startup

Stage Do this Fear this
Consulting to test ideas and Lock-in, IP
Empathy bootstrap the business control, overﬁtting

Standardization and integration; Ability to
Stickiness shift from custom to generic integrate; support

Word of mouth, references, case Bad vibes;
Virality studies exclusivity

Growing direct sales, professional Pipeline, revenue
Revenue services, support recognition, comp

Channels, analysts, ecosystems, Crossing the
Scale APIs, vertically targeted products chasm; Gorillas


The Zero Overhead principle

A central theme to this new wave of
innovation is the application of core product
tenets from the consumer space to the
enterprise.
In particular, a universal lesson that I keep
sharing with all entrepreneurs building for the
enterprise is the Zero Overhead Principle: no
feature may add training costs to the
user. DJ Patil


Intrapreneur


Skunk Works for intrapreneurs

• The Lockheed Martin Skunk Works


Span of control and the railroads

• Daniel C. McCallum


The BCG matrix

• How businesses think
about products or Question marks! increase 
Pivot to  
Stars!
companies (low market share, market  (high growth rate,
share 
high growth rate)
through  high market share)
May be the next big thing. virality,  What everyone wants. As
• Lean is about moving Consumes investment, but attention
market invariably stops
will require money to growing, should become
up and to the right Growth rate
increase market share.
cash cows.

Milk with 
Pivot to  Pivot to 
revenue 
redeﬁne problem/  increase growth 
optimization as 
solution through  rate through 
growth slows
empathy
disruption

Dogs! Cash cows!
(low market share, (high market share,
low growth rate)
low growth rate)
Barely breaks even, may Boring sources of cash, to
be a distraction from better be milked but not worth
opportunities. Sell off or additional investment.
shut down.

Market share

Intrapreneur example:
P&G changes the mop
instead of the soap

• Stage: Empathy
• Model: Retail/consumer packaged goods
• P&G is constantly looking for better soaps. But innovation was slowing.
Frustrated, they hired a design team to help them.


P&G changes the mop
instead of the soap

• Heavy internal investment in R&D, but limited results
• Brought in an outside agency (Continuum) to help
• The team watched people as they mopped, recording and iterating their
research approach

• Watched someone pick up spilled coffee. Rather than mopping, the person
swept up with a broom, then wiped with a cloth
• Realized the mop, not the liquid, mattered
• Studied the makeup of ﬂoor dirt; realized much of it is dust

• Swiffer is a $500M innovation in a stalled industry


The Lean Analytics lifecycle
for an Intrapreneur
Stage Do this Fear this
Get buy-in Political fallout
Beforehand
Find problems; don’t test demand. Entitled, aggrieved
Empathy Skip the business case, do analytics customers

Know your real minimum based on Hidden “must haves”,
Stickiness expectations, regulations feature creep

Build inherent virality in from the Luddites who don’t
Virality start; attention is the new currency understand sharing

Consider the ecosystem, channels, Channel conﬂict,
Revenue and established agreements resistance, contracts

Hand the baton to others gracefully Hating what happens
Scale to your baby

“The most important ﬁgures that
one needs for management are
unknown or unknowable, but
successful management must
nevertheless take account of
them.”

Lloyd S. Nelson

Choose only one metric.


Yes, one metric.


It will soon change.


In a startup,
focus is hard to achieve.


Having only one metric
addresses this problem.


Metrics are like
squeeze toys.

http://www.ﬂickr.com/photos/connortarter/4791605202/

ARCHIMEDES
HAD TAKEN
BATHS BEFORE.


Once, a leader convinced others in
the absence of data.


Now, a leader knows what
questions to ask.


Ben Yoskovitz
byosko@gmail.com
@byosko

Alistair Croll
acroll@gmail.com
@acroll


Startup metrics toronto March 19

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (8)

Semelhante a Startup metrics toronto March 19

Semelhante a Startup metrics toronto March 19 (20)

Último

Último (20)

Startup metrics toronto March 19