Discusses techniques I've used to do Lean Startup style pivots in an agile team and web application. Presented at LAST Conference 2015. We use impact mapping, geckoboard, datahero to show how having a hypothsis backed by data has helped us make decisions to improve bidcontender.com.
2. CONFIDENTIAL | 2
Featured Project:
The Wroclaw West Bypass, Poland
Trusted by the world’s largest projects
Pivots for real: how we've
hacked up some data to
know we've hit the mark
Mark Smith @markjdsmith
23. CONFIDENTIAL | 23
Discover the drivers for your goals (Impact Mapping)
The Goal
Who needs to
change
Ways we can
drive change
Down here –
you’re just
guessing
(hypothesis!)
http://www.impactmapping.org/
24. CONFIDENTIAL | 24
The ‘guesses’ down the tree can be stated as
hypothesis
.
We believe that if we
implement semi
automated invites we
will see the number of
invited friends increase
x% within 1 month
25. CONFIDENTIAL | 25
Goals are the outcome at the end of a funnel
Step 1
Step 2
Step
3
% that get to
this step is the
goal
100 users
50 users
10 users
(10%)
27. CONFIDENTIAL | 27
We took a flow from our app and created
hypothesis for change
STEPS
Registered
Profile Updated
Subscribed
Subbie connected
to builder
Labelled by
builder
Invited to quote
quoted
28. CONFIDENTIAL | 28
Hypothesis for changes to a step
STEPS
Registered
Profile Updated
Subscribed
Subbie connected
to builder
Labelled by
builder
Invited to quote
quoted
Adding a list of ‘builders in
their state’ will increase
connections to builders for
subbies by 50% within 1
month of introduction
Hypothesis
29. CONFIDENTIAL | 29
Hypothesis for changes to a step
STEPS
Registered
Profile Updated
Subscribed
Subbie connected
to builder
Labelled by
builder
Invited to quote
quoted
Adding a list of ‘builders in
their state’ will increase
connections to builders for
subbies by 50% within 1
month of introduction
Hypothesis
Search filter for builders in
Network directory
Send URL to builders list in
email
Static list (A-Z) of builders
Options to solve
32. CONFIDENTIAL | 32
Increasing Activation to 90% will
increase retention by 20% for pro
subscribers within a quarter
Ensuring Subbies are being added to
builders networks will increase
invitations to quote by 30% for subbies
within 1 month of introduction
Adding a list of builders in their
state will increase connections to
builders for subbies by 50% within
1 month of introduction
If this holds
Then this
may hold
And this will
hold
(invitations create activation)
33. CONFIDENTIAL | 33
And then build it…
Adding a list of builders in their
state will increase connections to
builders for subbies by 50% within
1 month of introduction
As a subbie I want to
see a list of
contractors in my
state so that I can
connect to them
As a builder I want an
indication that subbies are
trying to connect with me
so that I will go and
complete the connection
As a product manager I
want to know how many
subbies use the ‘find
contractor feature’ so I can
measure the outcome
Etc…
35. CONFIDENTIAL | 35
Indira Gandhi International Airport, Terminal 3, India | US $2B Value
Data informs hypotheses.
Passing Hypotheses deliver
your goals.
Failing Hypotheses don’t.
Find out which is which!
36. CONFIDENTIAL | 36
Further Reading
http://www.businessmodelgeneration.com/book
http://practicetrumpstheory.com/http://www.amazon.com/Value-Proposition-Design-Customers-Strategyzer/dp/1118968050
http://shop.oreilly.com/product/0636920033851.do
37. CONFIDENTIAL | 37
The Center to Grand Forks (CGF) Project, U.S.A. | US $350M Value
We’re Hiring!
http://Aconex.com/careers
39. CONFIDENTIAL | 39
Preparation (for 2nd Session – if you’re coming along!)
• Bring your laptop (or use one of the computers in the lab)
• Ensure you know your Google Analytics credentials.
• Have a google apps account so you can use google sheets
• Bring a list of 5 epics from your current backlog
41. CONFIDENTIAL | 41
Dubai International Airport, U.A.E | US $4.5B Value
Session 2: Using
Geckoboard and
DataHero to measure
your hypothesis
42. CONFIDENTIAL | 42
Yas Island, U.A.E. | US $40B Value
How do I turn my backlog
of stories into
hypotheses?
43. CONFIDENTIAL | 43
Some techniques
• Extract the assumptions/intention from the story
• As a <who>
• I want to <the change>
• So that <the measure>
As a player
I want to have
semi automated
invites sent
So that my list of
friends increases
44. CONFIDENTIAL | 44
But Why?
• The 5 why’s exercise on a stories ass:
– Q: Why do I want to get semi automated invites?
– A: so I increase the list of my friends
– Q: why do you want to increase the list of my friends?
– A: so that we get more players
– Q: Why do I want more players?
– A: so that we get to 1M players
– Q: why 1M players
– A: Cos then I get a Ferrari….
As a player
I want to have
semi automated
invites sent
So that my list of
friends increases
45. CONFIDENTIAL | 45
Check the (HIPPO’s) bullshit
Bullshit! Will semi
automated REALLY
drive more friends?
• The 5 why’s exercise on a stories ass:
– Q: Why do I want to get semi automated invites?
– A: so I increase the list of my friends
– Q: why do you want to increase the list of my friends?
– A: so that we get more players
– Q: Why do I want more players?
– A: so that we get to 1M players
– Q: why 1M players
– A: Cos then I get a Ferrari….
Bullshit! Will more
friends in the list really
get more players?
The implied hierarchy of the 5
Why’s and the bullshit detector
drives a natural inference tree
46. CONFIDENTIAL | 46
Let’s try it…. Volunteer?
• Let’s take someone’s backlog and have a go at converting it to an inference
tree:
– Volunteer – bring out your top 5 cards
– Let’s do it together.
47. CONFIDENTIAL | 47
Measures are easy in geckoboard
• Pick any piece of data you need for your inference tree
• Sign in to Geckoboard.
• Create a dashboard
• Put the data there.
51. CONFIDENTIAL | 51
Get going!
Your turn – in groups of 4:
– Take your backlog, and find the real goal
– Find the assumptions
– Write down the hypothesis as a tree of inference
– Work out the data you need
– Connect it to your geckoboard
52. CONFIDENTIAL | 52
Objections to Dashboards
• Data Objections:
– Don’t have access. (you don’t have permissions to that data)
– Can’t get that yet (integration problem)
– It’s in an email.
• Answer:
– Google Sheets.
• IT objection
– We’re not paying till you do a business case.
– OFFS – buy it on your credit card and then show the improvement, then get the
corporate card on it…..
• Priority Objection
– We don’t have time to build measures, we need to build feature X
• (if nobody uses it, if it doesn’t convert, what’s the point!)
• A good product manager knows to kill features as well as make them.
59. CONFIDENTIAL | 59
Seattle Tunnel State Route 99, U.S.A | US $1.4B Value
Never forget you are
telling a story
60. CONFIDENTIAL | 60
Pro Activation Funnel has improved from our work….
• We’ve moved from 25% (28-May-2015) to 41% (3-Jun-2015) to 47% (17-Jun-
2015)
• We did these changes:
– Introduced auto-labelling
– Pending network connections show as icon on network directory
28-may-2015 3-Jun-2015 17-Jun-2015
61. CONFIDENTIAL | 61
And look at starter!
• This has been sitting
on 7% for some
weeks (haven’t
captured the
progression).
• Look at the numbers
on it now!
62. CONFIDENTIAL | 62
Still a problem, but not enough to finish?
• Drop off on connection
requests
– so we’re doing
reminder email tonight.)
• Drop off on invite to
quoted
– Will cover this as part
of builder onboarding
(they want to get
coverage).