[DSC Adria 23]Rok Piltaver Lessons Learned From 10 Years of AB Testing.pptx
•Transferir como PPTX, PDF•
0 gostou•28 visualizações
A/B testing helps organisations identify and evaluate potential product improvements using data and statistics, which eliminates the guesswork and helps them understand what makes users tick. In this talk, I’ll explain what A/B testing is and give examples of how it can be used. Then I’ll share lessons learned from 10 years of AB tests on millions of users and some of the world's most popular mobile games: what are prerequisites for a successful A/B test, A/B test process that ensure actionable results, how A/B test infrastructure speeds up implementation and analysis, and how to avoid the common pitfalls.
Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a [DSC Adria 23]Rok Piltaver Lessons Learned From 10 Years of AB Testing.pptx
Semelhante a [DSC Adria 23]Rok Piltaver Lessons Learned From 10 Years of AB Testing.pptx (20)
Mais de DataScienceConferenc1
Mais de DataScienceConferenc1 (20)
Último
Último (20)
[DSC Adria 23]Rok Piltaver Lessons Learned From 10 Years of AB Testing.pptx
- 1. Lessons Learned From 10 Years of A/B Testing Rok Piltaver, Senior Data Scientist
- 3. What is A/B testing
- 4. What is A/B Testing Variant A Variant B
- 5. What is A/B Testing Variant A Variant B
- 6. What is A/B Testing Variant A Variant B data
- 7. What is A/B Testing Variant A Variant B data Users Randomly assigned into 2 groups Each group uses one variant Observe, measure, and record data Use statistics to compare variants
- 8. Examples: a bad example configcat.com A B
- 9. Examples: call to action configcat.com S A V E L E O A B
- 10. Examples: steps to action … heartlandanimalshelter.org A B
- 12. Examples: unlock with ad My Talking Angela 2 A B
- 13. Why do we do A/B tests
- 14. Data driven decisions based on statistical methodology to: ● Improve product ● Improve monetisation ● Learn Why? revenue game difficulty or rewards generosity
- 15. Learn: value of new content Talking Tom Hero Dash
- 16. Learn: value of new content Talking Tom Hero Dash A B C D E F G H
- 17. Learn: value of small app size My Talking Hank
- 18. A: 106 MB B: 600 MB Learn: value of small app size My Talking Hank
- 19. How to A/B test
- 24. Process ` problem, metric A/B test ideas implement collect data analyse
- 25. Process ` problem, metric A/B test ideas implement collect data analyse decide
- 26. Process ` problem, metric A/B test ideas implement collect data analyse decide profit
- 27. Start with users’ or business problem, not just a test idea! Who: ● Controlling ● Product ● Users ● Analytics Define metric(s) to be improved 1 .Identify problems
- 28. Select ideas by estimating: ● impact ● test power ● cost ● needed time 2. A/B test ideas
- 29. 2. Select A/B test: estimate impact $ $/user # users % viewers $/ad ads/user banner $/user installs/day retention by DSI users by DSI interstitial $/user video $/user IAP $/user
- 30. 2. Select A/B test: estimate impact $ $/user # users % viewers ads/user $/ad +5% banner $/user installs/day retention by DSI users by DSI interstitial $/user video $/user IAP $/user
- 31. 2. Select A/B test: estimate impact $ $/user # users $/ad +5% banner $/user installs/day retention by DSI users by DSI interstitial $/user video $/user IAP $/user ads/user -1% % viewers -1%
- 32. 2. Select A/B test: estimate impact $ # users $/ad +5% installs/day retention by DSI users by DSI interstitial $/user video $/user IAP $/user ads/user -1% % viewers -1% +2.6% banner $/user +0.8% $/user
- 33. 2. Select A/B test: estimate impact $/ad +5% installs/day users by DSI interstitial $/user video $/user IAP $/user ads/user -1% % viewers -1% +2.6% banner $/user +0.8% $/user retention by DSI +2% +1% # users $ +1.8%
- 34. 2. Select A/B test: is it measurable Small difference Big difference Big deviation Small deviation
- 35. ● Implementation ● Coordination ● User acquisition ● Opportunity cost ● Negative impact on users 2. Select A/B test: estimate cost
- 36. 2. Select A/B test: estimate time define implement get users collect data analyse decide time
- 37. 3. Implement A/B test ● Clear goal and spec ● QA ● Handle UX after the test ● BE configurable tests
- 38. 3. BE configurable A/B test ID Price default 90
- 39. 3. BE configurable A/B test ID Price Test ID black list Test ID white list default 90 prices_B prices_C cheap_prices 50 prices_B expensive_prices 200 prices_C
- 40. 4. Collecting users: infrastructure `
- 41. 4. Collect data ● How many users ● Time to collect users ● Time to collect data ● Which data to track ● Which users `
- 42. ● Monetisation data ● Activity data ● Test group id ● User id ● Product version ● Market segment 4. Collect data: which data `
- 43. 4. Collect data: which users ` never in test A1 B1
- 44. 4. Collect data: which users ` never in test A1 B1 A2 B2
- 45. 4. Collect data: which users ` A2 B2 never in test A1 B1
- 46. 4. Collect data: which users ` A2 B2 A2 B2 never in test A1 B1
- 47. 5. Analyse data: automation
- 48. 5. Analyse data: automated plots Daily automated plots about all important metrics for each test
- 49. 6. Decide ● Write executive report ● Export important plots ● Analysis code ● Peer review results x2 ● Present to decision makers ● Decide ● Switch to new version ● Clean-up test
- 50. ` A1 B1 A2 B2 A2 B2 A B