Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Próximos SlideShares
Carregando em…5
×

A/B Puzzle

187 visualizações

This presentation is from Performance Marketing Summit 2017 (March 14, 2017 in Austin, TX). Session description: How A/B tests lie to us, and how to drive genuine improvement.

• Full Name
Comment goes here.

Are you sure you want to Yes No
• Entre para ver os comentários

• Seja a primeira pessoa a gostar disto

A/B Puzzle

1. 1. A/B PUZZLE How A/B tests lie to us, and how to drive genuine improvement Jason Cohen, founder & CTO @asmartbear http://wpengine.com
2. 2. The puzzle
3. 3. 758 out of 24,483 visits 85% confidence
4. 4. What do stats really tell us?
5. 5. 758 out of 24,483 visits 85% confidence
6. 6. Finding the Cheaters 105 athletes. 100 are clean. 5 are juicing. Test is 95% accurate.
7. 7. Finding the Cheaters IN REALITY Clean (100) Juicing (5) TEST RESULT (95% accuracy) Clean Juicing 5 095 5
8. 8. Insight: Rare events are indistinguishable from random noise
9. 9. 758 out of 24,483 visits 85% confidence
10. 10. Easy significance testing with hamsters Trust in the Hamster. bit.ly/abhamster
11. 11. Easy significance testing N = A+B = total # of conversions.  D = (A–B)/2 = half the difference.  Significant if D2 > N. bit.ly/abhamster N = 758 + 685 = 1443  D = ( 758 — 685 ) / 2 = 37  1369 > 1443?  No! Not significant.
12. 12. Final Result: As predicted Wasted another month
13. 13. 30 40 50 60 70 12/2/09 12/16/09 1/5/10 1/13/10 2/2/10 4/9/10 4/27/10 9/24/10 9/14/11 11/8/11 OtherInbox Splash Page
14. 14. Trust the Hamster
15. 15. Beware the Rare
16. 16. Defeat rare by  seeking only huge results
17. 17. Can you A/B test your way to huge results?
18. 18. 1,116 250 4,100 1,151 Individuals: 67% off Group: 3% off
19. 19. Wisdom of the Crowd “I’m as clueless as you, but in the equal and opposite direction.”
20. 20. What’s the funniest joke?
21. 21. And the winner is…
22. 22. Jokes, Pizza, Design, Features, Voice, Values, Culture
23. 23. Crowds find objective truth but kill creativity
24. 24. Beware the rare. Seek huge results. Don’t let data kill creativity. Jason Cohen, founder & CTO @asmartbear http://wpengine.com