Conversion Conference Berlin
•
1 gostou•1,199 visualizações
A walkthrough of the most common statistical pitfalls that marketers encounter in CRO testing.
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Conversion Conference Berlin
Semelhante a Conversion Conference Berlin (20)
Mais de Tom Capper
Mais de Tom Capper (13)
Último
Último (20)
Conversion Conference Berlin
- 2. A Good A/B Test Result: “10% Uplift, With 95% Significance”
- 3. ● What does this mean? ● Is this correct? “10% Uplift, With 95% Significance”
- 7. Risk
- 8. Marketer: “Roll it out!” Statistician (me): *sobs*
- 11. “Advanced”
- 12. You will learn today: ● The most common serious errors in A/B testing ● How to avoid them ● How to interpret your result ● Whether to roll it out
- 13. How to Run an A/B Test
- 14. 1. Test design 2. Results interpretation 3. Decision
- 16. Jargon: Null Hypothesis ● The hypothesis that your variant and original are functionally equivalent
- 17. e.g. an A/A Test vs. A A
- 18. Jargon: P-Value
- 19. Jargon: P-Value ● The chance of a result this extreme if the null hypothesis is true ● E.g. 0.05 for 95% significance
- 21. Jargon: Critical Value ● What you compare your p-value with when deciding whether to reject the null hypothesis
- 22. 1. Test Design
- 23. How Many Tests?
- 24. A B C D E F Multivariate Testing Landing Page: Product Pages:
- 25. Multivariate Testing A B C D Landing Page: Product Page:
- 27. Multivariate Testing A C D BA C D B
- 28. Multivariate Testing A C D BA C D BA C D B
- 29. Multivariate Testing A C D B A C D BA C D BA C D B
- 30. Multivariate Testing A BLanding Page: A: 5% B: 7.5%
- 31. Multivariate Testing C D C: 5% D: 7.5% Product Page:
- 32. Multivariate Testing A C D B A C D BA C D BA C D B
- 33. A C Multivariate Testing D B A C D BA C D BA C D B AC: 0% BD: 5% BC: 10% AD: 10%
- 34. A B
- 37. Multiple Testing A B C D E F
- 39. False Positives Test: Healthy Test: Ill False Negatives
- 40. False Positives Test: Healthy Test: Ill False Positives False Negatives
- 41. Multiple Testing 1 A/A test: 5% chance of achieving 95% significance.
- 42. Multiple Testing 1 A/A Test: 5% chance
- 43. Multiple Testing 1 A/A Test: 2 A/A Tests: 5% chance 9.75% chance
- 44. Multiple Testing 1 A/A Test: 2 A/A Tests: 3 A/A Tests: 5% chance 9.75% chance 14.26% chance
- 45. Multiple Testing 1 A/A Test: 2 A/A Tests: 3 A/A Tests: 4 A/A Tests: 5% chance 9.75% chance 14.26% chance 18.55% chance
- 46. Multiple Testing 1 A/A Test: 2 A/A Tests: 3 A/A Tests: 4 A/A Tests: n A/A Tests: 5% chance 9.75% chance 14.26% chance 18.55% chance 1-0.95^n
- 47. Multiple Testing Solutions: 1. Accept risk of false positives
- 48. Multiple Testing Solutions: 1. Accept risk of false positives 2. Bonferroni correction
- 49. Bonferroni Approximation Standard: P-value vs………..…. 0.05
- 50. Bonferroni Approximation Standard: P-value vs………..…. Approximation: P-value vs…... 0.05 0.05/N
- 51. Bonferroni Correction Standard: P-value vs………..…. Bonferroni: P-value vs………. 0.05 1-(1-0.05)^(1/N)
- 52. Multiple Testing Solutions: 1. Accept risk of false positives 2. Bonferroni correction 3. Holm-Bonferroni correction
- 53. Choosing the Right Metric
- 54. Choosing the Right Metric Conversion Rate vs. Average Session Value
- 55. Choosing the Right Metric Conversion Rate vs. Average Session Value Profit?
- 56. Stopping Rules
- 57. Stopping Rules Common: When my test reaches significance.
- 58. “Significance so far” varies over time.
- 59. Stopping Rules Y Y Y Y Y N N N N N
- 60. Stopping Rules Y Y Y Y Y Y YN N N
- 62. 20000
- 64. Stopping Rules Solutions: 1. Sequential testing - e.g. Optimizely 2. Bayesian testing - e.g. VWO 3. Predetermined sample size
- 66. Sample Size for Average Session Value Testing
- 72. 1. Test design 2. Results interpretation 3. Decision
- 75. Interpreting the P-value 1 test reaches 95% significance: 5% chance of data this extreme if variants functionally equivalent.
- 76. 0
- 77. Analogy
- 78. Analogy Question: How likely is it that my analytics or site are broken?
- 79. Analogy Question: How likely is it that my analytics or site are broken? Non-Answer: We only go a whole day with no conversions once every 2 months.
- 80. Analytics is broken with probability 1 or 0.
- 81. Interpreting the P-value Question: How likely is it that this variation actually does nothing? Non-Answer: We’d only see a difference this big 5% of the time.
- 82. Meanwhile in Industry Tools: ● “Chance to beat baseline” ● “We are 95% certain that the changes in test “B” will improve your conversion rate”
- 84. Unanswered Questions Question: How likely is it that the increase will be less than predicted?
- 85. Unanswered Questions Question: How likely is it that the increase will be negative?
- 86. One Mistake Probability of Outcome given Data vs. Probability of Data given Null
- 87. Unanswered Questions Question: How likely is it that these results are a fluke?
- 89. Confidence Interval of Conversion Rate
- 91. Everything Else Still Applies
- 92. Choosing the Right Metric
- 95. 1. Test design 2. Results interpretation 3. Decision
- 96. 3. Decision
- 97. A Good A/B Test Result: “10% Uplift, With 95% Significance”
- 98. But what about this? “10% Uplift, With 60% Significance”
- 99. Jargon: P-Value ● The chance of a result this extreme if the null hypothesis is true ● E.g. 0.05 for 95% significance
- 100. “10% Uplift, With 60% Significance” ● 40% chance of data at least this extreme if variation functionally identical
- 101. “10% Uplift, With 60% Significance” ● 40% chance of data at least this extreme if variation functionally identical ● The variation is probably better than the baseline
- 103. Are You OK With False Positives?
- 106. Data is Expensive
- 107. Data is Expensive: ● Opportunity Cost ● Exploration vs. Exploitation
- 109. Hang on… Why Should I Care About Significance?
- 110. 1. Ignoring Significance Doesn’t Allow You to Ignore Statistics
- 111. 2. Risk Aversion
- 112. Risk Factors: ● Agility ● Business attitudes ● What’s the worst that could happen?
- 113. Decision Recap Significant vs. Winning Risk Exploration vs. Exploitation
- 115. 1. Think about significance and risk during test design
- 116. 2. Remember your real KPI: Profit
- 117. 3. You’re not testing medicines
- 118. @THCapper Takeaways: 1. Think about significance and risk during test design 2. Remember your real KPI: Profit 3. You’re not testing medicines