Talk at DevTribe gathering 3. 10 minutes about how feature toggling enable A/B testing.

1. A/B testing(on multiple platforms with recurring and paying users)
at Spotify
DevTribe Gathering 3
2016-12-06. Oscar Carlsson, lad@spotify.com
A/B testing at Spotify(on multiple platforms with recurring and paying users)

2. Oscar Carlsson
Data Scientist

3. Feature toggling
+
A/B testing

4. picture from https://www.elegantthemes.com/blog/tips-tricks/easy-ab-testing-with-divi-leads
The classic example of
A/B testing
Winner if CTR/Conversion is up.

5. Oftenly no requirement to
understanding the
why

11. New users
Existing
users
Multiple platforms
Premium
& Free

12. New users
Existing users
Platforms

13. New users
Newly registered users with no habits
or routines in the app.
They have never seen the UI and any
impression or interaction is guaranteed
to be the first.
Relatively low traffic.
Activation:
How many registered users use the
app on first day?
Engagement:
How much are they using the product?
Retention:
How many registered users use the
app second week/month?
Conversion:
Do they turn premium?

14. New users
Newly registered users with no habits
or routines in the app.
They have never seen the UI and any
impression or interaction is guaranteed
to be the first.
Relatively low traffic.
Activation:
How many registered users use the
app on first day?
Retention:
How many registered users use the
app second week/month?
Engagement:
How much are they using the product?
Conversion:
Do they turn premium?

15. Existing users
Users who are using your product
already. They probably have habits and
routines and they might have seen the
surface before.
Relatively high traffic.
Engagement:
How much are they using the product?
Conversion:
Do they turn premium?

16. Existing users
Users who are using your product
already. They probably have habits and
routines and they might have seen the
surface before.
Relatively high traffic.
Engagement:
How much are they using the product?
Conversion:
Do they turn premium?

17. Platforms
(Often) test separately
Not fully aligned - We’re testing in different phase on all platforms
Users can use multiple platforms and usage can shift
Android
iPhone
Desktop
Web player
and more.

18. Every team should
be testing!
.. could there be challenges with that?

19. A/B
testing
A/B
testing
A/B
testing
A/B
testing

20. I want long term effect!
I want your change and
I want long term effect!
I also want those changes and
I want long term effect!
Me too and
I want long term effect!

21. I want long term effect!
I want your change and
I want long term effect!
I also want those changes and
I want long term effect!
Me too and
I want long term effect!
Dependencies!
&
Complicated
code?

22. New bold font
Center header
Remove card background
Center follower count

23. Fully scientific? Quick product development?

24. Fully scientific? Quick product development?

25. Fully scientific? Quick product development?
A mix of both?

26. How many tests can a user live
through?

27. How do you track, and
do you track what
happens to the users
when the test is over?

28. picture from http://codevoyagers.com/2016/03/16/design-like-youre-right-test-like-youre-wrong/

29. Thanks for listening
Oscar Carlsson
lad@spotify.com