1. (Lean) Startup Metrics & Analytics
Nicola Junior Vitto - @njvitto @blomming
September 26, 2012 - Better Software - Firenze, Italy

2. Entrepreneurship
...Entrepreneurs recognize they are going to have
to follow their own internal compass and embrace
the uncertainty as part of the journey.
- Steve Blank -
more on: http://steveblank.com/2009/06/29/agile-opportunism-entrepreneurial-dna/

3. Get out of the building (by Steve Blank)
Your business assumptions can be wrong
Go out and speak with your customers

4. Vanity Metrics

5. Vanity Metrics - Timelines
There will always be something up and to the right

6. Vanity Metrics - Google Analytics
Forget it!

7. Vanity Metrics - Google Analytics
...and this could become like a drug...killing you!

8. Vanity Metrics - TechCrunch & Co
“mDialog raises 5 Million
from Blackberry partners”
“GroupMe is now sending
one Million texts every day”
“Google Plus hits 10 Million
users in 2 weeks”

9. Lean Metrics

10. What’s a startup?
A startup is first of all an experiment, it is a human
institution designed to deliver a new product or
service under conditions of extreme uncertainty
At its heart, a startup is a catalyst that transforms
ideas into product.
(From Lessons Learned blog By Eric Ries)

11. What’s a Lean Startup?
Lean startup is a rigorous process
for iterating from Plan A
to a plan that works.
(by Ash Maurya)

12. What Metrics Matter?
Example: Facebook on early days
150k users, little revenue
Many superior competitors
What would indicate their future success?

13. What Metrics Matter?
Example: Facebook on early days
75% of users visit one or more times per day
Within one month of launching on a new
campus, can acquire 90% of the students

14. What Metrics Matter?
Good Metrics (Three A’s)
Are Actionable
Can be Audited
Are Accessible

15. Actionable Metric
An actionable metric is one that ties
specific and repeatable actions
to observed results
(by Ash Maurya)

16. A/B Testing

17. A/B Testing - New Blomming home
VS
Define your Goals

18. A/B Testing - Define your goals
A/B Testing Goals
One single important Goal:
% Users that sign-up for the service (as a seller)
Secondary Goal:
% Buyers that buy something
after landing on the homepage

19. Multivariate Testing

20. Multivariate Testing at Google
“Yes, it’s true that a team at Google couldn’t
decide between two blues, so they’re testing 41
shades between each blue to see whic one
performs better. I has a recent debate over wheter
a border should be 3, 4 or 5 pixels wide, and was
asked to prove my case.
I can’t operate in an environment like that.”
(by Doug Bowman on Design at Google)

21. Multivariate Testing at Google

22. Multivariate Testing at Google

23. Multivariate Testing at Google
Google Bing

24. Multivariate Testing at Google
“That blue was worth at least $80M”
(by Paul Ray - Product Manager at Bing)

25. Funnels

26. David Cancel’s Funnel
The Problem:
98% of people who visit your site,
leave never to return

27. David Cancel’s Funnel
2%
“The average conversion rate in the United States of
America wheter you’re selling elephants or iPods”
(by Avinash Kaushik - Analytics Evangelist, Google)

28. David Cancel’s Funnel

29. David Cancel’s Funnel
The Solution:
To find, engage and convert more
of your website visitors into customers

30. Online Marketing
is shifting

31. Online Marketing Paradigm Shift
by David Cancel

32. Online Marketing Paradigm Shift
by David Cancel

33. Online Marketing Paradigm Shift
by David Cancel

34. Online Marketing Paradigm Shift
Internet Marketing 3.0 (different goal)
by David Cancel

35. Lifecycle Marketing
by David Cancel

36. Optimize

37. Optimize for learning
Optimize for learning
by creating a data-driven marketing framework

38. Optimize for learning
Analytics & A/B Testing are also
not exactly the answer

39. Optimize for learning
Use quantitative & qualitative data
to test *your* assumptions

40. Optimize for learning
The point of testing is to prove or disprove
assumptions quickly

41. Optimize for learning
Pay attention to “overtesting” using Multivariate testing
because it generally takes too long

42. The Local Maxima Problem

43. Frameworks

44. AARRR

45. AARRR
Dave McClure’s Startup Metrics for Pirates is a good
framework for thinking about macro metrics

46. AARRR - Model

47. AARRR - Conversions Example

48. AARRR - Different orders (example)
1. Activation, Retention (happiness)
2. Revenue (sustainability)
3. Acquisition, Referral (virality)
(this is the Ash Maurya or 37signals order:
“Charge from day one”)

49. Acquisition

50. Tracking Marketing Channels
Google Adwords
Banners
Social Media
Partnerships
PR
Blogging
...anything that links to your site!

51. Tracking Marketing Channels
Use unique urls (tracking parameters)
on every url you create/give-out/pay for
Use bit.ly
Use HTTP referrers to group PR

52. Measuring Cost of Visits
Cost per click (site visits)
Cost of campaign / Traffic generated
(Social media) Cost of effort / Traffic generated

53. Measuring Cost of Acquisition
Cost of one visit / conversion rate =
Cost of an Acquisition
Acquisition could be paid for acquisition
of free sign-up
...unfortunately is not so simple

54. Marketing Channels
Per channel track:
Scale
Cost
Conversion
Optimise and scale channels
with best Cost of Acquisition (CoA)
and potential to Scale

55. Acquisition
Cost of Acquisition < Customer Lifetime Profit
Then scale acquisition at a similar or lower rate
A/B test acquisition channels to optimise

56. Acquisition including Referral
If it costs €3 to acquire a customer through advertising
But a customer brought
in 2 other customers through referral
The COA of that channel is €1

57. Activation

58. Activation
Sure, they signed-up!!!

59. Activation
Sure, they signed-up!!!
BUT did the like it???

60. Activation at Facebook
(i) which data points predict whether a user will stay?
(ii) if they stay, which data points predict how active they’ll
be after three months?

61. Activation at Facebook
(i) which data points predict whether a user will stay?
having more than one session as a new user
and entering basic profile information.
(ii) if they stay, which data points predict how active they’ll
be after three months?
how often a user was reached out by others, frequency of
third party application use, and how available a user was
on the site.

62. Activation at Facebook
“At the time, I heard many people criticize
Facebook’s early investors, claiming that Facebook had
‘no business model’ and only modest revenues
relative to the valuation offered by its investors...”
“...Facebook was different, because it employed a different
Engine of Growth. It paid nothing for customer
acquisition, and its high engagement meant that
it was accumulating massive amount of customer
attention every day.”
(Eric Ries - The Lean Startup)

63. Activation Metrics
Activation metrics often drive other metrics
in Referral, Retention, Revenue

64. Retention

65. Returning Users
Tracking unregistered users
that return often (useful for content sites with advertising)
Tracking registered users
that return and take some sort of action.

66. Cohort Analysis

67. Twitter Cohorts (example)
by Fred Wilson’s blog
(more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)

68. Twitter Cohorts (example)
by Fred Wilson’s blog
(more on: http://www.avc.com/a_vc/2009/10/the-cohort-analysis.html)

69. Referral

70. Referral Mechanism
Social Media Sharing
Invite a friend type mechanism
App Reviews
Word of Mouth

71. Viral Coefficient
The average number of customers
that each customer refers

72. Viral Coefficient - Example
Each company customer
invites 5 other customers via email
20% click on that email link
50% of those users convert into customers

73. Viral Coefficient - Example
Viral Coefficient = Referrals x click through x conversion
5 x 0.2 x 0.5 = 0.5

74. Viral Coefficient
A viral co-efficient greater than 1
means that every customer
gets more than one other customer on average

75. Viral Coefficient
A viral co-efficient greater than 1
means that every customer
gets more than one other customer on average
Therefore your product will grow virally

76. Viral Coefficient
Measure automatically for all
your built in referral mechanism
Give customers unique invite URLs
and track conversions

77. Viral Cycle Time
Viral Cycle Time = the average time taken for
a referral to turn into a customer
With the viral coefficient
and viral cycle time
you can determine your viral growth

78. Net Promoter Score
Net Promoter Score: Number of people who would
recommend your product to a friend
A single question survey

79. Net Promoter Score
“How likely is it you would recommend my product to a
friend or colleague?”
0 1 2 3 4 5 6 7 8 9 10
NPS = % Promoters - % Detractors

80. Net Promoter Score
Survey a small percentage
of your customers by email weekly
Record who you sent to so
you don’t repeat the same people
every week

81. Net Promoter Score
NPS is not a direct measurement
that translates to key metrics
Can act as a leading indicator of other metrics

82. Revenue

83. Making Money
Key for any business is to find out how much profit
they make for every customer
and scaling the number of customers

84. Lifetime Customer Value
How much money you make
for every customer you acquire
For some this a single transaction
For subscription businesses it depends on retention

85. Cancellation Rate
Cancellation rate = is the percentage number of
customers who cancel in any given month compared to
total (paying) customers
[Cancellation rate] =
[product utility] + [service quality] + [acceptable price]

86. Lifetime Customer Value (LTV)
LTV = monthly revenue x n° months in lifetime
N° months = 1 / cancellation rate
LTV = monthly revenue / cancellation rate

87. Lifetime Customer Value (LTV)
For example:
If €20 a month and cancellation rate is 10% a month
LTV = €200

88. Lifetime Customer Value (LTV)
Often not that simple.
Short term cancellation rates are
much higher than long term
Use Cohort Analysis to determine

89. Lifetime Customer Value (LTV)
r = short-term cancellation rate (e.g. 0,15)
p = long-term cancellation rate (e.g. 0.03)
s = number of months in the “short-term” age group
(e.g. 3)
(1-r)^s x (s + 1/p) = expected months

90. Some Tools

91. Event based tracking

92. Event based tracking

93. Event based tracking

94. Customer lifecycle
management

95. Customer lifecycle management

96. Customer lifecycle management

97. Tips

98. OODA Loop

99. Innovation Accounting
“To improve entrepreneurial outcomes, and to
hold entrepreneurs accountable, we need to focus
on the boring stuff: how to measure progress,
how to setup milestones, how to prioritize
work. This requires a new kind of accounting,
specific to startups”
(Eric Ries - The Lean Startup)

100. Innovation Accounting
Concentrate on macro metrics
Everything you do should attempt to change a metric
Try to use as few tools as possible

101. One Page Dashboard

102. Customer Development: pivoting

103. (Lean) Product Development
Problem:
unknown
Solution:
unknown

104. Scalable (lean) startup

105. SCRUM
Do you know SCRUM?

106. Kanban Board
I know: this isn’t an example of an exciting and well done Sprint :)

107. (Lean) Product Management
Well, trying to connect the dots:
what we learnt so far is how the Product Owner
can define and prioritize stories at every Sprint
in a Startup that wants to be called, you know, Lean! :)

108. Learning Boards - Lean Stack
We will talk about the Lean Stack of Ash Maurya
or other possible “lean mapping” frameworks
in a future presentation

109. Focus focus focus

110. Credits
Stuart Eccles - Lean Startup Metrics
Joshua Porter - Metrics-Driven Design
David Cancel - Data-Driven Marketing
Lean Startup Machine -
How to measure the metrics that determine real progress
(and the others specified directly in the slides when picking
some parts of books or blogs)

111. The Lean Startup

112. Running Lean

113. Business Models

114. An alternative?
Lifestyle business

115. Thanks!
Blomming.com
Company.Blomming.com
Magazine.Blomming.com
Help.Blomming.com
Twitter.com/Blomming
Facebook.com/Blomming
nicola@blomming.com
On FB & Twitter as: njvitto