4. Outcomes
• Theory
• What are analytics & experimentation?
Saturday, 16 February 13
5. Outcomes
• Theory
• What are analytics & experimentation?
• Why are they important
Saturday, 16 February 13
6. Outcomes
• Theory
• What are analytics & experimentation?
• Why are they important
• How do you do them?
Saturday, 16 February 13
7. Outcomes
• Theory
• What are analytics & experimentation?
• Why are they important
• How do you do them?
• Practical
Saturday, 16 February 13
8. Outcomes
• Theory
• What are analytics & experimentation?
• Why are they important
• How do you do them?
• Practical
• Analytics plan
Saturday, 16 February 13
9. Outcomes
• Theory
• What are analytics & experimentation?
• Why are they important
• How do you do them?
• Practical
• Analytics plan
• Experiment plan
Saturday, 16 February 13
33. Segmentation
• Grouping users together
based on some characteristic
of the users
Saturday, 16 February 13
34. Segmentation
• Grouping users together
based on some characteristic
of the users
• Shows patterns otherwise
hidden by noise averages
Saturday, 16 February 13
35. Segmentation
• Grouping users together
based on some characteristic
of the users
• Shows patterns otherwise
hidden by noise averages
• Focus on those who are
most important to you
Saturday, 16 February 13
43. Cohorts
• Grouping by point in time
Saturday, 16 February 13
44. Cohorts
• Grouping by point in time
• How does behaviour change over
time?
Saturday, 16 February 13
45. Cohorts
• Grouping by point in time
• How does behaviour change over
time?
• Relative or absolute
Saturday, 16 February 13
46. Cohorts
• Grouping by point in time
• How does behaviour change over
time?
• Relative or absolute
• Comparison between cohorts
Saturday, 16 February 13
63. KPIs
• The KPIs (key performance indicators) are
derived from the product vision
Saturday, 16 February 13
64. KPIs
• The KPIs (key performance indicators) are
derived from the product vision
• Need to take into account at what stage
your product is
Saturday, 16 February 13
76. Data Points
• What makes up metrics
• What makes up funnels
• Achievable
• Segments
Saturday, 16 February 13
77. Data Points: Example
• Number of new visitors
• Number who register
• Number who reach each step of the
• Value of each checkout chart
• Number of checkouts
• Number of documents uploaded
Saturday, 16 February 13
88. Why What?
• Apply scientific method to get a better
product
Saturday, 16 February 13
89. Why What?
• Apply scientific method to get a better
product
• Test different assumptions made about the
product
Saturday, 16 February 13
90. Why What?
• Apply scientific method to get a better
product
• Test different assumptions made about the
product
• Test different variations to see what effects
the KPIs
Saturday, 16 February 13
103. Hypothesis
• Proposed reason or explanation for a
phenomena
Saturday, 16 February 13
104. Hypothesis
• Proposed reason or explanation for a
phenomena
• An answer to the question or explanation
of the question
Saturday, 16 February 13
105. Hypothesis
• Proposed reason or explanation for a
phenomena
• An answer to the question or explanation
of the question
• Testable with independent variable that can
be controlled and a dependent variable that
can be measured
Saturday, 16 February 13
106. Hypothesis: Example
• The call to action button should be red
• The message is not clear about the value of
registering
• There are too many different call-to-actions
on the page
Saturday, 16 February 13
108. Hypothesis: Structure
• IF I water the plants THEN the plants will
grow
Saturday, 16 February 13
109. Hypothesis: Structure
• IF I water the plants THEN the plants will
grow
• IF I don’t water the plants THEN the plants
will not grow
Saturday, 16 February 13
111. Hypothesis: Example
• IF the call to action button is red THEN
the number of people registering will
increase
Saturday, 16 February 13
112. Hypothesis: Example
• IF the call to action button is red THEN
the number of people registering will
increase
• IF we change the copy explaining the value
of registering THEN the number of
people registering will go up
Saturday, 16 February 13
113. Hypothesis: Example
• IF the call to action button is red THEN
the number of people registering will
increase
• IF we change the copy explaining the value
of registering THEN the number of
people registering will go up
• IF we remove all but one call-to-action on
the page THEN the number of people
registering will increase
Saturday, 16 February 13
132. Running Experiment
• Record the experiments results
Saturday, 16 February 13
133. Running Experiment
• Record the experiments results
• Segment the traffic
Saturday, 16 February 13
134. Running Experiment
• Record the experiments results
• Segment the traffic
• Traffic (random split to remove bias)
Saturday, 16 February 13
135. Running Experiment
• Record the experiments results
• Segment the traffic
• Traffic (random split to remove bias)
• How long to run?
Saturday, 16 February 13
136. Running Experiment
• Record the experiments results
• Segment the traffic
• Traffic (random split to remove bias)
• How long to run?
• Who to test?
Saturday, 16 February 13
137. Running Experiment
• Record the experiments results
• Segment the traffic
• Traffic (random split to remove bias)
• How long to run?
• Who to test?
• Test well - don’t take short cuts
Saturday, 16 February 13
138. Running Experiment
• Record the experiments results
• Segment the traffic
• Traffic (random split to remove bias)
• How long to run?
• Who to test?
• Test well - don’t take short cuts
• Negative result
Saturday, 16 February 13
140. Closing the Loop
• What do these results mean for the
product development prioritisation?
Saturday, 16 February 13
141. Closing the Loop
• What do these results mean for the
product development prioritisation?
• Why did I get these results? or,
Saturday, 16 February 13
142. Closing the Loop
• What do these results mean for the
product development prioritisation?
• Why did I get these results? or,
• Why didn’t I get the results expected?
Saturday, 16 February 13
143. Closing the Loop
• What do these results mean for the
product development prioritisation?
• Why did I get these results? or,
• Why didn’t I get the results expected?
• Revamp priorisation produce the next
set of experiments
Saturday, 16 February 13