AppPromotionSummit London 2016 #APSLondon Appstore A/B test ASO Conversion Play Store Experiments iOS Search Ads A/A Test

1. 1
1
A/B testing your store listing
@Thomasbcn #APSLondon

2. 2
2
@Thomasbcn #APSLondon

3. 3
3
ASO has changed…
ASO 2.0 / AMO / AVO
• App Store search Optimization (ASO 1.0)
• Conversion Rate Optimization (CRO)
• Search Ads (SEM)
• Featuring
• App Indexing (SEO)
@Thomasbcn #APSLondon

4. 4
4
https://twitter.com/Thomasbcn/status/690109233510105089
Why?
70% to 90% of App Stores “views”
don’t generate an install
Testing the listing
> Raise install conversion
> More installs / less UA costs
Sources: Branch / Salesforce / Splitmetrics / SensorTower @Thomasbcn #APSLondon

5. 5
5
https://twitter.com/Thomasbcn/status/690109233510105089"We have increased install conversions
by more than 25% testing content for our app stores”
Maria Waters (Kabam)
“If we can improve the app store conversion,
we actually reduce the load on our marketing team
and significantly reduce the budget that we spend on user acquisition”
Rajeev Girdhar (Rovio)
“I changed my icon to an icon with a unicorn.
My daily downloads as well as daily revenue doubled!”
Per Haglund (Super Puzzle)
Source: Splitmetrics, VB, StoreMaven
Why?
@Thomasbcn #APSLondon

6. 6
6
+44% conversion*
(but…)
@Thomasbcn #APSLondon

7. 7
7
Gut choices vs testing
“Design by committee produces disastrous results”
Michael D'Ulisse (TapTapTap)
“A/B testing revealed that the images the team preferred reduced the
conversion rate of installs by 15%”
Fat Fish Games (VB)
“I’ve seen pageview to download conversion rates
rise double digit percentages through icon optimization”
Wilhelm That (Rovio)
@Thomasbcn #APSLondon

8. 8
8
https://twitter.com/Thomasbcn/status/690109233510105089
What to test
• Icon
• First screenshot (+video?)
• Title
• (Short) description
• Feature graphics
@Thomasbcn #APSLondon

9. 9
9
A/B testing iOS Search Ads
“The top app achieved approximately the same amount of clicks
on the ‘Get’ button regardless of whether it was a ‘search ad’
or a regular search result”
“Users don’t see the difference between
sponsored and organic listings”
@Thomasbcn #APSLondon

10. 10
10
How to? Apple AppStore
• No official tool
• Send paid & owned traffic
to an AppStore replica (web)
• Providers:
Build your own? (Soundcloud)
youtube.com/watch?v=v8MXAzYPKFY
@Thomasbcn #APSLondon

11. 11
11
@Thomasbcn #APSLondon

12. 12
12
Apple AppStore A/B testing caveats…
• Misleading users
• Organic traffic excluded
• Added costs: tool + dropoff
“You’ll be losing some users who’ve decided to download your app”
Nick Kurat (TestNest)
“There was a big drop-off which made the cost to acquire each
sample user much higher than expected:
greater than $10 on Google Play.”
Ilya Shereshevsky (DeNA)
Source: TestNest, AppAnnie @Thomasbcn #APSLondon

13. 13
13
How to? Google Play Store
Play Store Experiments
• Built-in (Dev. Console)
• Includes organic traffic
• Test visuals & texts
• Up to 5 localized experiments at a time
• FREE
@Thomasbcn #APSLondon

14. 14
14
Source: RedKaraoke @Thomasbcn #APSLondon

15. 15
15
Source: 8fit @Thomasbcn #APSLondon

16. 16
16
Sources: Iteratingfun, TC, TNW
+30%
@Thomasbcn #APSLondon

17. 17
17
Sources: Peter Fodor, FatFish, @Thomasbcn #APSLondon

18. 18
18
Google Play Store A/B testing limits…
• Android & iOS are different
• No testing on title
• Traffic split uncontrolled & undisclosed
• No user behaviour details
in app / on bottom line
on page
• Unreliable statistical relevance…
Read more: GP Experiments limits acc. to
@Thomasbcn #APSLondon

19. 19
19
Measuring user behaviour on your listing
@Thomasbcn #APSLondon

20. 20
20
Testing the tool itself : the A/A test
@Thomasbcn #APSLondon

21. 21
21
Counter testing:
@Thomasbcn #APSLondon

22. 22
22
The issue with GP Experiment:
Undisclosed statistical significance
@Thomasbcn #APSLondon

23. 23
23
A/B testing listing caveats…
• Most A/B tests fail!
• Closed variants bring little learnings
• Volume required for significance can be high
• Improvements curve
• Bottom line impact?
• Bias/externalities
“Only 1 out of 8 A/B tests have driven significant change”
@noahkagan
“We waste our time testing variants
without any real meaningful differences”
@justinrmegahan
Read more: ConversionXL, Formisimo, Groove @Thomasbcn #APSLondon

24. 24
24
@Thomasbcn #APSLondon
… Believe your guts
… Design by committee
… Start with close variants
… Assume iOS & Android behave the same
… Mimick your competition
… Conclude on early data
… Believe minor uplifts
… Believe GP Experiments stats relevance
… Test & iterate
… Make bold assumptions
… Focus on visuals
… Localize your tests
… Counter-test results
… Double-check with tracked sources
… Challenge the tools
… Challenge yourself

25. 25
25
A/B testing your store listing
@Thomasbcn #APSLondon