SlideShare uma empresa Scribd logo
1 de 34
Baixar para ler offline
1
@PaulCutsinger
Russ Carroll
A|B Testing
2
3
4
Surface the Value
Image from Diner Dash by PlayFirst
Source: Amazon Appstore, January 2013
+75%
ARPPU
5
Show Users How to “Consume”
Image from Cut the Rope by ZeptoLab
Source: Amazon Appstore, January 2013
+65%
repeat orders
6
Experiment and react quickly
7 day retention
Time
Active
Players
7
A|B Testing
8
This is Air Patriots
Case Study: Store Optimization
10
• Goal – Easier to access store content
• Platform – Android & iOS
• Test – Make it possible to unlock content on Main Menu
STORE OPTIMIZATION
11
LOCKED ON MAIN
MENU
UNLOCKABLE >
<
12
STORE OPTIMIZATION
Test Case
Conversion to
Customer
7-Day Retention
Locked on Main Menu
(Control)
– –
Unlockable on Main
Menu
20.93% 4.17%
13
• Goal – Drive players to our other games
• Platform – Android
• Test – Which image is clicked on more
CROSS PROMOTION
CROSS PROMOTION
15
CROSS PROMOTION
16
Air Patriots: Cross Promotion
B
A 1.77%
0.93%
ClickThrough
19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. Ads Clicked +1.89% +3.07%
43,120
The Producer’s Perspective
Ads on menu or in game
19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In GameOn MenuNo Ads
The Producer’s Perspective
19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In GameOn MenuNo AdsGoals
The Producer’s Perspective
19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In GameOn MenuNo AdsGoals
The Producer’s Perspective
19.94%1. 3 day Retention -1.07% +0.83%
8.06%2. 7 day Retention +0.74% -10.11%
2.05%3. Units Purchased +0.34% -12.07%
0%4. Ads Clicked +1.09% +4.07%
43,120
Ads on menu or in game
In GameOn MenuNo AdsGoals
The Producer’s Perspective
22
Retrieve a variation
Objective C
23
Record a view and conversion
Objective C
bit.ly/ABTestXPlat
bit.ly/ABTestiOS
bit.ly/ABTestJava
More details at
Case Study: Junk Race by GameNug
Junk Race Overview
26
Junk Race Overview
27
Tips:
Producers in control: “With two-way services you can request a value from the
server that you control. Two-way services let you make changes to your live game
instantly.
Free: Amazon is the only free two-way service that we have found.”
Multi-variable: “We don't test variables, we test user experiences.”
Watch sample size: “If a couple users can change your conclusion, you need a
larger sample”
Deploy Instantly Cross Platform: “A/B testing on iOS has the advantage of
skipping the week long review period ”
-John Goforth
28
Case Study: Difficulty Level
29Amazon Services Confidential
GAME DIFFICULTY TEST
7 day retention
Time
Active
Players
30Amazon Services Confidential
GAME DIFFICULTY TEST
• Goal – A Dramatic Increase!
• Platform – Android & iOS
• Test – A/B/n Test – 5 Difficulty levels!
OR?
31Amazon Services Confidential
GAME DIFFICULTY TEST
32Amazon Services Confidential
Test 1:
A
Control
B
2% easier
C
4% easier
D
6% easier
E
8% easier
7-Day Retention – tbd tbd tbd tbd
Conversion to customer – -1.71% 16.11% 19.23% 4.84%
GAME DIFFICULTY TEST
33Amazon Services Confidential
GAME DIFFICULTY TEST
• How much would you pay to increase retention by 20%?
• How much time would you invest?
Our Take-Aways!
• Never launch a game in future w/o AB difficulty test!
• Low-hanging fruit!
• Opportunities abound!
34Amazon Services Confidential
A|B Testing
• Monetization – minimize luck and maximize
skill
• Delight your customers
• test customer experiences
• AB testing tools are easy to use
• Cross Platform
• Deploy changes instantly
• Free
• Low-hanging fruit is waiting to be harvested
Paul Cutsinger
Russell Carroll
Find me at:
• @PaulCutsinger
• @AmazonAppDev
• developer.amazon.com/blog
• bit.ly/ABTestXPlat

Mais conteúdo relacionado

Semelhante a A/B Testing with Air Patriots - GDC 2014

Quiz app android ppt
Quiz app android pptQuiz app android ppt
Quiz app android ppt
Aditya Nag
 
Browserscope oscon 2011
Browserscope oscon 2011Browserscope oscon 2011
Browserscope oscon 2011
lsimon
 
From Point & Click to Pinch & Zoom by Dan Rabinovitz
From Point & Click to Pinch & Zoom by Dan RabinovitzFrom Point & Click to Pinch & Zoom by Dan Rabinovitz
From Point & Click to Pinch & Zoom by Dan Rabinovitz
Sauce Labs
 

Semelhante a A/B Testing with Air Patriots - GDC 2014 (20)

Draftkings: Launching w/ Confidence at Scale, FutureStack17 NYC
Draftkings: Launching w/ Confidence at Scale, FutureStack17 NYCDraftkings: Launching w/ Confidence at Scale, FutureStack17 NYC
Draftkings: Launching w/ Confidence at Scale, FutureStack17 NYC
 
Gaming Testing
Gaming TestingGaming Testing
Gaming Testing
 
Measure camp tools of the cro rabble
Measure camp   tools of the cro rabbleMeasure camp   tools of the cro rabble
Measure camp tools of the cro rabble
 
Front-End Test Fest Keynote: The State of the Union for Front End Testing.pdf
Front-End Test Fest Keynote: The State of the Union for Front End Testing.pdfFront-End Test Fest Keynote: The State of the Union for Front End Testing.pdf
Front-End Test Fest Keynote: The State of the Union for Front End Testing.pdf
 
Product Madness - A/B Testing
Product Madness - A/B TestingProduct Madness - A/B Testing
Product Madness - A/B Testing
 
Mobile User Experience: Auto Drive through Performance Metrics
Mobile User Experience:Auto Drive through Performance MetricsMobile User Experience:Auto Drive through Performance Metrics
Mobile User Experience: Auto Drive through Performance Metrics
 
Quiz app android ppt
Quiz app android pptQuiz app android ppt
Quiz app android ppt
 
Real User Monitoring: Getting Real Data from Real Users in the Real World - S...
Real User Monitoring: Getting Real Data from Real Users in the Real World - S...Real User Monitoring: Getting Real Data from Real Users in the Real World - S...
Real User Monitoring: Getting Real Data from Real Users in the Real World - S...
 
Failure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature DeliveryFailure is an Option: Scaling Resilient Feature Delivery
Failure is an Option: Scaling Resilient Feature Delivery
 
Amazon Services for Data Sync, Player Engagement & A/B Testing (MBL309) | AWS...
Amazon Services for Data Sync, Player Engagement & A/B Testing (MBL309) | AWS...Amazon Services for Data Sync, Player Engagement & A/B Testing (MBL309) | AWS...
Amazon Services for Data Sync, Player Engagement & A/B Testing (MBL309) | AWS...
 
Ship quality mobile apps with speed [Webinar]
Ship quality mobile apps with speed [Webinar]Ship quality mobile apps with speed [Webinar]
Ship quality mobile apps with speed [Webinar]
 
Browserscope oscon 2011
Browserscope oscon 2011Browserscope oscon 2011
Browserscope oscon 2011
 
How we improved i os delivery speed from qa side by Kateryna Sprynsian
How we improved i os delivery speed from qa side by Kateryna SprynsianHow we improved i os delivery speed from qa side by Kateryna Sprynsian
How we improved i os delivery speed from qa side by Kateryna Sprynsian
 
Sydney Continuous Delivery Meetup May 2014
Sydney Continuous Delivery Meetup May 2014Sydney Continuous Delivery Meetup May 2014
Sydney Continuous Delivery Meetup May 2014
 
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
How The Zebra Utilized Feature Experiments To Increase Carrier Card Engagemen...
 
Data Science Popup Austin: Predicting Customer Behavior & Enhancing Customer ...
Data Science Popup Austin: Predicting Customer Behavior & Enhancing Customer ...Data Science Popup Austin: Predicting Customer Behavior & Enhancing Customer ...
Data Science Popup Austin: Predicting Customer Behavior & Enhancing Customer ...
 
Uncovering breaking changes behind UI on mobile applications
Uncovering breaking changes behind UI on mobile applicationsUncovering breaking changes behind UI on mobile applications
Uncovering breaking changes behind UI on mobile applications
 
From Point & Click to Pinch & Zoom by Dan Rabinovitz
From Point & Click to Pinch & Zoom by Dan RabinovitzFrom Point & Click to Pinch & Zoom by Dan Rabinovitz
From Point & Click to Pinch & Zoom by Dan Rabinovitz
 
Google Glass Meetup 3/3 - 8 Janvier 2014
Google Glass Meetup 3/3 - 8 Janvier 2014Google Glass Meetup 3/3 - 8 Janvier 2014
Google Glass Meetup 3/3 - 8 Janvier 2014
 
testCloud & Crittercism: How to Continuously Ensure Mobile App Quality
testCloud & Crittercism: How to Continuously Ensure Mobile App QualitytestCloud & Crittercism: How to Continuously Ensure Mobile App Quality
testCloud & Crittercism: How to Continuously Ensure Mobile App Quality
 

Mais de Amazon Web Services

Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
Amazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
Amazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
Amazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
Amazon Web Services
 

Mais de Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Último

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Último (20)

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKSpring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 

A/B Testing with Air Patriots - GDC 2014

  • 2. 2
  • 3. 3
  • 4. 4 Surface the Value Image from Diner Dash by PlayFirst Source: Amazon Appstore, January 2013 +75% ARPPU
  • 5. 5 Show Users How to “Consume” Image from Cut the Rope by ZeptoLab Source: Amazon Appstore, January 2013 +65% repeat orders
  • 6. 6 Experiment and react quickly 7 day retention Time Active Players
  • 8. 8 This is Air Patriots
  • 9. Case Study: Store Optimization
  • 10. 10 • Goal – Easier to access store content • Platform – Android & iOS • Test – Make it possible to unlock content on Main Menu STORE OPTIMIZATION
  • 12. 12 STORE OPTIMIZATION Test Case Conversion to Customer 7-Day Retention Locked on Main Menu (Control) – – Unlockable on Main Menu 20.93% 4.17%
  • 13. 13 • Goal – Drive players to our other games • Platform – Android • Test – Which image is clicked on more CROSS PROMOTION
  • 16. 16 Air Patriots: Cross Promotion B A 1.77% 0.93% ClickThrough
  • 17. 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.89% +3.07% 43,120 The Producer’s Perspective Ads on menu or in game
  • 18. 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo Ads The Producer’s Perspective
  • 19. 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo AdsGoals The Producer’s Perspective
  • 20. 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo AdsGoals The Producer’s Perspective
  • 21. 19.94%1. 3 day Retention -1.07% +0.83% 8.06%2. 7 day Retention +0.74% -10.11% 2.05%3. Units Purchased +0.34% -12.07% 0%4. Ads Clicked +1.09% +4.07% 43,120 Ads on menu or in game In GameOn MenuNo AdsGoals The Producer’s Perspective
  • 23. 23 Record a view and conversion Objective C bit.ly/ABTestXPlat bit.ly/ABTestiOS bit.ly/ABTestJava More details at
  • 24. Case Study: Junk Race by GameNug
  • 27. 27 Tips: Producers in control: “With two-way services you can request a value from the server that you control. Two-way services let you make changes to your live game instantly. Free: Amazon is the only free two-way service that we have found.” Multi-variable: “We don't test variables, we test user experiences.” Watch sample size: “If a couple users can change your conclusion, you need a larger sample” Deploy Instantly Cross Platform: “A/B testing on iOS has the advantage of skipping the week long review period ” -John Goforth
  • 29. 29Amazon Services Confidential GAME DIFFICULTY TEST 7 day retention Time Active Players
  • 30. 30Amazon Services Confidential GAME DIFFICULTY TEST • Goal – A Dramatic Increase! • Platform – Android & iOS • Test – A/B/n Test – 5 Difficulty levels! OR?
  • 32. 32Amazon Services Confidential Test 1: A Control B 2% easier C 4% easier D 6% easier E 8% easier 7-Day Retention – tbd tbd tbd tbd Conversion to customer – -1.71% 16.11% 19.23% 4.84% GAME DIFFICULTY TEST
  • 33. 33Amazon Services Confidential GAME DIFFICULTY TEST • How much would you pay to increase retention by 20%? • How much time would you invest? Our Take-Aways! • Never launch a game in future w/o AB difficulty test! • Low-hanging fruit! • Opportunities abound!
  • 34. 34Amazon Services Confidential A|B Testing • Monetization – minimize luck and maximize skill • Delight your customers • test customer experiences • AB testing tools are easy to use • Cross Platform • Deploy changes instantly • Free • Low-hanging fruit is waiting to be harvested Paul Cutsinger Russell Carroll Find me at: • @PaulCutsinger • @AmazonAppDev • developer.amazon.com/blog • bit.ly/ABTestXPlat