O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

A/B Puzzle

187 visualizações

Publicada em

This presentation is from Performance Marketing Summit 2017 (March 14, 2017 in Austin, TX). Session description: How A/B tests lie to us, and how to drive genuine improvement.

Publicada em: Marketing
  • Entre para ver os comentários

  • Seja a primeira pessoa a gostar disto

A/B Puzzle

  1. 1. A/B PUZZLE How A/B tests lie to us, and how to drive genuine improvement Jason Cohen, founder & CTO @asmartbear http://wpengine.com
  2. 2. The puzzle
  3. 3. 758 out of 24,483 visits 85% confidence
  4. 4. What do stats really tell us?
  5. 5. 758 out of 24,483 visits 85% confidence
  6. 6. Finding the Cheaters 105 athletes. 100 are clean. 5 are juicing. Test is 95% accurate.
  7. 7. Finding the Cheaters IN REALITY Clean (100) Juicing (5) TEST RESULT (95% accuracy) Clean Juicing 5 095 5
  8. 8. Insight: Rare events are indistinguishable from random noise
  9. 9. 758 out of 24,483 visits 85% confidence
  10. 10. Easy significance testing with hamsters Trust in the Hamster. bit.ly/abhamster
  11. 11. Easy significance testing N = A+B = total # of conversions.
 D = (A–B)/2 = half the difference.
 Significant if D2 > N. bit.ly/abhamster N = 758 + 685 = 1443
 D = ( 758 — 685 ) / 2 = 37
 1369 > 1443?
 No! Not significant.
  12. 12. Final Result: As predicted Wasted another month
  13. 13. 30 40 50 60 70 12/2/09 12/16/09 1/5/10 1/13/10 2/2/10 4/9/10 4/27/10 9/24/10 9/14/11 11/8/11 OtherInbox Splash Page
  14. 14. Trust the Hamster
  15. 15. Beware the Rare
  16. 16. Defeat rare by
 seeking only huge results
  17. 17. Can you A/B test your way to huge results?
  18. 18. 1,116 250 4,100 1,151 Individuals: 67% off Group: 3% off
  19. 19. Wisdom of the Crowd “I’m as clueless as you, but in the equal and opposite direction.”
  20. 20. What’s the funniest joke?
  21. 21. And the winner is…
  22. 22. Jokes, Pizza, Design, Features, Voice, Values, Culture
  23. 23. Crowds find objective truth but kill creativity
  24. 24. Beware the rare. Seek huge results. Don’t let data kill creativity. Jason Cohen, founder & CTO @asmartbear http://wpengine.com

×