1.
Creating an Effective A/B
Testing Strategy for App Store
Live Webinar + Q&A
Natalie Ostapovich
Customer Support & Account Manager
@ SplitMetrics
Elisa Mourao
Marketing Manager
@ App Radar

2.
30 minutes A/B Testing
Natalie Ostapovich Elisa Mourao
Live Q&A
splitmetrics.com appradar.com

3.
Agenda
Pre-Launch A/B Test
A/B Testing in App Stores
01
02
splitmetrics.com appradar.com
Timeline
Hypothesis
What to test
How to run A/B tests
How to Analyze Results

4.
A/B Testing Timeline
Research Brainstorming Creating Variations Running Experiments Analysis
splitmetrics.com appradar.com

5.
Pre-Launch A/B Test
App Pre-Launch Marketing Plan

6.
Why and How to A/B test in Pre-Launch Phase?
Validate your app idea;
Find positioning;
Find target audience;
Prioritize app features;
Polish store listing.
splitmetrics.com appradar.com

7.
How to interpret results and move on?
Conversion Rate;
Performance in different
audiences;
Bounce Rate;
Explored sections from
the store listing.
splitmetrics.com appradar.com

8.
A/B Testing
in App Stores

9.
How to come up with a strong hypothesis?
Research Phase
App Product Page vs. Category and Search Page
What type of experiment should be aimed for clear results ?
How many hypothesis should be tested at the same time?
splitmetrics.com appradar.com

10.
What elements to test?
Icon, screenshots, video
How about the other elements in the store listing?
splitmetrics.com appradar.com

11.
Testing the App Icon
splitmetrics.com appradar.com
22%
Uplift
Color palette

12.
Testing the App Screenshots
splitmetrics.com appradar.com
24%
Uplift
Portrait vs. Landscape
Colors
Order
Features vs. Gameplay

13.
Testing the App Video
splitmetrics.com appradar.com
17%
Uplift
Does an app video increase
conversion rates?
Which videos show better
performance?
How many videos should I
have in my store listing?

14.
How to Run A/B Tests
splitmetrics.com appradar.com
For how long should I run my test?
How does targeting work with A/B testing?
What is enough traffic for an experiment?

15.
splitmetrics.com appradar.com
“It normally takes at least 50 ‘Clicks to Install’ per
variation to get statistically significant results. On
average, you need 400-500 visitors per variation
for a test on your app’s product page and
800-1000 users per variation if you run an
experiment on the search or category page.
SplitMetrics
”

16.
Analyzing Results
What is and how to interpret the Confidence Level?
How to choose a winner?
Main KPIs
01
02
splitmetrics.com appradar.com
Visitors per variation
Clicks to install per variation
Time to click to install
Engagement rate in the store listing
Improvement Metric
Retention rate
03

17.
Analyzing Results – Best Practices
What if my new variation loses?
What if I don't have an answer after the test?
Should we use the same winning creatives to the App Store
and Google Play or should we a/b test per store?
splitmetrics.com appradar.com

18.
Thank You!
Questions?
Natalie Ostapovich
Customer Support & Account
Manager @ SplitMetrics
Elisa Mourao
Marketing Manager
@ App Radar
Sign up: appradar.com
Follow us on social media:
appradar
appradar
appradar_com
App_rad ar
App Radar
splitmetrics.com
searchadshq.com
splitmetrics.com/agency/