1.
1
Josh Aberant
CMO
t.co/Josh
@jaberant
Being Data Driven

2.
@jaberant 2
Bio - Josh Aberant
SparkPost CMO & Growth
Twitter Postmaster
RestEngine CoFounder
Marketo Director of Privacy

3.
@jaberant 3
Postmaster Role @Twitter
• Growth team
Early member of formalized Growth team built around acquihires
• Mixer Labs, Summify, Fluther, RestEngine & more
• Build & optimize growth surfaces
• Work on Growth surfaces
• NUX, RUX, messaging, logged out, distribution/viral
• Provide growth services across enterprise
Data as a service
Experiments as a service
• Lots of teams needed/consumed growth services
Led to a lot of security work
Security discussion is for another day. We are hear to talk #Growth

4.
4
#GrowthHacking
Data in Action
Being Data Driven

5.
@jaberant 5

6.
@jaberant 6
#GrowthHacking - Data in Action
Growth
User State
Models
Data Experiments

7.
7

8.
@jaberant 8
#GrowthHacking for CX
#GrowthHacking is a capable CX framework (Customer Experience)
• CX is marketing teams embracing both data & empathy for the user
• The whole breadth of user interactions
• Acquisition, Engagement, Loyalty & Advocacy
• Listening to and co-designing with the users
• #GrowthHacking is methodology for scaling empathy
• #GrowthHacking measurable & repeatable way to get at the
essence of what the user wants, enjoys & needs

9.
9
You are all #GrowthHackers
here, even if you don’t
know it yet

10.
@jaberant 10
The Growth Role
Blend data & deep understanding of user
• Needs, habits, perception, tastes, preferences
Answer questions on user experience. e.g.:
• Why are users dropping out of the sign up
experience?
• Why don’t users come back to the application after
the initial download?
• Why aren’t users responding to special offers?
Experiment to collect real meaningful feedback
• Innovate through experimentation

11.
@jaberant 11
The Growth Role
Data & Experiments Infrastructure Services
• Analytics & experiments
Roll your own (work with the Data scientist & devs) or
Commercially available solutions - Google Analytics to
Optimizely
Dashboards
• State Models
Growth Plan
• Define the growth objectives
• Decide where in the tornado (funnel) to focus
e.g. User acquisition or fighting churn

12.
@jaberant 12
Funnels Have Become Tornados
• Funnels have been blown to bits
• User’s don’t take linear paths to and
through apps or products
• They get to your app or product how they
want to, when they want to and do
whatever they want to do
• Users own the relationship

13.
@jaberant 13
Managing Tornadoes
• Breakdown all user mechanics
• Measure & instrument
• Experiment to improve
The classic funnel stages still matter - users just don’t go
through them linearly
• Acquisition, activation, engagement, retention,
resurrection

14.
@jaberant 14
Experimental Results = User Insights
Through experiments we use real world data that to
generate user insights that are fed into the enterprise
• Empathy
• Thinking like the users think
• Understanding essence of users environment, tasks &
goals
• Living in the world of users & enhancing their experience
This isn’t just polling & surveys - experiments enable to see
what users truly do when changes are made.

15.
15
There’s no magic code
“There is no code. Ultimately growth is a
function of the right mix of a product that
meet important user needs and tenacious
creative data driven execution that
leverage principles of growth and often
exploits psychological triggers.”
-Sean Ellis

16.
#GrowthHacking is
getting users hooked on
your apps & products

17.
More
scientifically...

18.
18
#GrowthHacking = Science
Growth Hacking is a framework to study how users
engage so as to give them a more meaningful app or
product that they'll want to use & share.
If you want to build apps or products that people are
hooked to then make growth hacking core to your
development.

19.
@jaberant 19
Systematic & Repeatable
There’s really 3 main things #GrowthHacking does
• See how users activate & engage
• Get them to activate & engage more
• Get them to share

20.
@jaberant 20
To Grow
Instrument & measure
• Observe & understand
Innovate through experimentation
• Try things
• Optimize things
• Throw out what doesn’t work, do what does
Be a champion of the user

21.
21
The1%experiments

22.
@jaberant 22
#BeingDataDriven
Life inside a data driven organization
• Anyone can do a 1% experiment
• Indeed, don’t show up to present to execs without
experimental data
• Innovate through experimentation!

23.
23
Experiments

24.
24
Experiments are what
make #GrowthHacking
scientific

25.
@jaberant 25
Experiments Made Easy

26.
@jaberant 26
Experiments Made Easy
Test Group

27.
@jaberant 27
Experiments Made Easy
Test Group
Run the change on the test group
Leave the rest alone (they are the control)
Measure the change in the metrics you care about

28.
@jaberant 28
Working With Experiments
#BeingDataDriven means a regular cadence of test &
reviews
• Daily standups, weekly team review, set schedule of
reviews around initiatives, backlog mgmt
Learn to have instincts point to experimental success
• Often an experiment need more time to show results
• Or it needs to be expanded beyond 1%
Make decisions on how to proceed
• Expand, reformulate, place on hold, kill

29.
@jaberant 29
Focus on what works
Double down on what already works
• With limited resources you don’t have time to work on
everything
• It’s easier to build on a strength than to improve a
weakness
• It’s often easier to improve a conversion rate that’s
already doing well
• Easier to increase engagement of active users compared
to people who are inactive

30.
Instrumentation &
User State Models

31.
31
Instrument the metrics
that tell you if your users
are healthy

32.
“Based on experience of which companies stuck with us and
which didn't, we decided that any team that has exchanged
2,000 messages in its history has tried Slack — really tried
it. For a team around 50 people that means about 10 hours’
worth of messages. For a typical team of 10 people, that’s
maybe a week’s worth of messages. But it hit us that,
regardless of any other factor, after 2,000 messages, 93% of
those customers are still using Slack today.”
- Stewart Butterfield, Slack Founder
32
Healthy User Example

33.
@jaberant 33
Example Healthy User Metrics
Facebook
• Getting a user to 7 friends in 10 days
Dropbox
• Getting a new user to link a computer and add a file

34.
@jaberant 34
Some Common Engagement Metrics
• Logins past 30 days
• Days since last login
• Session length
• Session interval frequency (how often user launches app)
• App screens per session
• Interactions
• Post
• Shares
• Views

35.
@jaberant 35
Metrics That Matter
Ask
• Do metrics indicate that a user is healthy?
• Which phase of the tornado (funnel)?
Activation, activation, engagement, vitality,
resurrection
Build
• User state models
Specific to your user paths (tornado)

36.
@jaberant 36
User State Models
User state models simplify working with a lot of metrics by
aggregating metrics them into meaningful models that can
be used to quickly make decisions on the health of an
experimental result
• Examples
Healthy vs Unhealthy
Casual or Core

37.
37
User State Models
Growth is all about the quality
and quantity of the user base.
User state models provide a
way of measuring the quality.

38.
@jaberant 38
New
Casual Core
At
Risk
User State Models Examples

39.
@jaberant 39
Measuring User State Transitions Example
Example from Pinterest: Measuring flows through the Tornado
• New signup: User joins
• New -> Dormant: New users doesn’t engage for 28 days
• MAU -> Dormant: User was an MAU, but doesn’t engage for 28
days
• Dormant -> MAU: Users engages after not using app for 28 days

40.
@jaberant 40
Days since last login
Logins
in past
30 days
Healthy Users
At risk:
focus
engagement
experiments
here
Transitioning out:
focus Resurrected User
Experience (RUX)
experiments here
User State Models Examples

41.
@jaberant 41
RUX while you can
• User transitioning out are about to be churned users. Apply
resurrection messaging & experiments to them before they
leave while it is still much easier to reach them.
• RUX is generally a subset - the best parts of - the NUX
(New User Experience)

42.
@jaberant 42
User State Models Example
Pyramid example (for Twitter):
• Heavy tweeter
• Heavy non-tweeter
• Medium tweeters
• Medium non-tweeters
• Light users
• Very light
• Nearly light (new users)

43.
@jaberant 43
User State Models Example
Xd28s example (from Pinterest):
# of users Pinterest in past 28 days
• 4d28s+ = # of users that used Pinterest 4 or more days in
last 28
Major categories
• Core = 14d28s+
• Casual = 4d28s+
• Marginal = <4d28
Example provided by Pinterest Eng Blog

44.
@jaberant 44
User State Models Consideration
There’s a lot of debate about whether “middle” user states
are meaningful
• Medium, casual
Or wether is best to always think of these states as
transitioning out
• “I’m about out of here” group

45.
@jaberant 45
User State Models Consideration
There’s no one model that meets all experimental cases
• All models have limits
• Different models work better in different cases
Always consider whether a given model is right for the
experiments

46.
@jaberant 46
Counter Metrics
• Don’t lose sight of the forest in the trees. There’s a risk
your user state model doesn’t apply well to the
experiment
• Best practice: Pair every metric with appropriate counter
metric
Signups with activations
Activations with churn
New paid customers with total revenue
This encourages a holistic approach to growth & will help
keep you on the right track on when your user state models
or serving you well or when you should be using different
user state models.

47.
@jaberant 47
More Meaningful Cohorts w/ User States
Often cohorts are looked at by individual metrics
• In any given experiment the metrics might move different
metrics in different directions
• Difficult to evaluate an experiment’s results
Use user state models in your cohort analysis
• Scale decision making
• Incorporate multiple metrics
• Tells you what to do next
If the experiment makes more users healthy its a good
experiment

48.
@jaberant 48
Standard Cohort Example
A cohort is a group of people that share something in common
Default cohort analysis often means clustering users by when they
started & seeing how behavior changes over time
• E.g. 1 week old cohorts in Feb vs March when a big app change was
made at beginning of March
Lots of great commercial solutions
• Google Analytics, Kissmetrics, Keen.io, Segment, Mixpanel

49.
@jaberant 49
Cohorts with Users States
If the experiment makes more users healthy it’s a good
experiment
Cohort
Health
Session
length
Session
interval
App
Screes
per
session
Interactions Views Post
+ + + + + - +
- - - + + + +
- + - - - - -
+ + + - - - -

50.
@jaberant 50
Interactive Dashboard Best Practices
Use dashboards!
• Relevant metrics
• Visually engaging
You’ve to to get everyone on the team looking at & using
these dashboards
• Intuitive default selections
• Super easy basics analytical tasks
Filtering view, adjusting parameters, drill to examine
underlying data
Allow team members to understand data and drill
down in 1-2 clicks

51.
@jaberant 51
Interactive Dashboard Best Practices
Keep it current!! - up-to-date data
• Min, hour, week, month, quarter - whatever the right
timeline is for best managing this growth function
Label metrics meaningfully
• Meaningful to others
• Make sure metrics support the objective & the audience)
Stay away from chart junk
• Low value graphics or unintelligible widgets

52.
@jaberant 52
Interactive Dashboard Best Practices
Alerts
• Altering on growth systems
• E.g. email volume & push volume (on per type basis.)
• In addition to core product alerting
• Need good dashboards & alerting so when alerts go off
you can figure out what went wrong

53.
@jaberant 53
Keep Dashboards Relevant
Key metrics can shift as growth happens and challenges
change
• Maintain your dashboards
• Are they meeting the current needs of the team?

54.
@jaberant 54
Growth Surfaces
NUX
RUX
• The best parts of NUX
Messaging
• Email, Push, app updates
Logged out/ unauthed user state
• SEO
• Signups
Deeplinking (distribution)
• App indexing
Viral mechanism

55.
@jaberant 55
New User Experience (NUX)
Often the Growth team owns the NUX and it is a focus of
intense experimentation
• Activate Users
Quickly orient & teach users to use your app our
service
• Move users to a healthy state
E.g. Follow people, add friends, link a file

56.
@jaberant 56
Activation in the NUX
What is the “aha” moment or “must have” experience?
• How do you get users to this point as fast as possible (i.e.
in seconds)?
• What actions should users take to put them in a healthy
state?

57.
57
Growth comes from
delivering the core product
value to users as early and
often as possible.

58.
@jaberant 58
Resurrected User Experience (RUX)
Often the Growth team own the RUX and it is a focus of
another focus intense experimentation
• How do you bring users back and reactivate them?
• What’s changed since the users was last there that they’ll
want to stick around longer?
• Do you have a way of reaching churned users?
Email
Notifications
App store updates
Custom channels

59.
59
Resurrection is hard
It’s very hard to resurrect people without being
pushy. But if done right emails, notifications and app
store updates can do much for bringing users back
#KeepItClassy

60.
@jaberant 60
Messaging
The Growth team often provides messaging capabilities
across the enterprise
• Infrastructure, Services & best practices
• Email, push, sms, in-app, app store updates, custom
channels
• Policy & controls (based on fatigue models) to prevent
different teams from over-mailing or over-pushing
Prevent user fatigue

61.
61
App Indexing
via
Deeplinking

62.
@jaberant 62
SEO is back
With App Indexing Google will deeplink from search results
into content in your app.
• 30+ billion deeplinks & counting
• Powerful distribution
• Think about the landing experience
• The years sent on developing best practices for landing
pages on the web...
App Indexing + Adwords = Search download ads

63.
@jaberant 63
• Why would users want to share your app or product’s
atomic unit?
• How can users share the atomic unit?
• What are the mechanics of sharing?
• How do you reduce sharing friction?
• How do you get users to recommend your app?
Virality

64.
@jaberant 64
Vitality - Your app’s atomic unit
What is the fundamental object that people are consuming
and/or creating?
Some examples:
• Twitter – tweet
• Soundcloud - audio track
• Instagram – photo
• Tumblr – post
• Dropbox – file
• Codecademy – lesson

65.
65

66.
@jaberant 66
Social Proof - Virality Best Practice
Example from Nuzzel App - nuzzel.com

67.
@jaberant 67
Social Proof Best Practices
Pictures
People
• Connections
• Experts
• Celebrities
• Similar people
Don’t include if poor quality

68.
68
“We bet heavily on Twitter. Even if
someone is incredibly enthusiastic about a
product, literal word of mouth will only
get to a handful of people — but if
someone tweets about us, it can be seen
by hundreds, even thousands.”
- Stewart Butterfield, Slack Founder
Share via Twitter

69.
69
Listen

70.
70

71.
@jaberant 71
Listen To Your Users
App & product users will communicate
• They want the stuff they are using to get better
Give them ways to talk back to you
• Button to talk to you, in-app prompt, surveys
Find guidance on what they want
• Then experiment to confirm
Tag, ticket & study messages from users
• Be fastidious

72.
72
“If you put that all together, we probably
get 8,000 Zendesk help tickets and 10,000
tweets per month, and we respond to all
of them.”
” - Stewart Butterfield, Slack Founder
Listen with Twitter

73.
73
Finally a word of warning

74.
74
Don’t Spam

75.
@jaberant 75
Don’t Spam
Spam ruins the internet
Users don’t like it
Can easily waste so much good work
Virality doesn’t matter until you can deliver a compelling
benefit to users
• Nothing worth sharing means just that
Avoid gimmicks that drive sign-ups in the short-term but
alienate users in the long-run

76.
76
Now go forth & #GrowthHack

77.
77
Josh Aberant
CMO
t.co/Josh
@jaberant
Being Data Driven