The presentation for students about a/b testing process, its advantages, best practices and common (and not that common) pitfalls.

1. A/B Testing
Ivan Mylyanyk
@ Booking.com

2. Prologue

3. Mylyanyk Ivan
Lviv Physics and Mathematics lyceum
Ivan Franko National University of L’viv
Freelance
Outsource
Startup
Booking.com

4. Booking.com
planet’s #1 accommodation website
HQ @ Amsterdam
10K+ employees
1M+ open properties
1M+ room nights / day

5. What is A/B Testing?

7. What you can test?
● color for your button
● title for your announcement
● new data science model
● algorithms
● fonts
● new functionality
● different photo resolutions
● email campaigns
● many more...

8. What can be your metric?
● #pageviews
● #signups
● profit
● something very specific:
○ number of uploaded profile pics

9. Do I need A/B Testing?
● customers “decide” what stays
● opinions and assumptions aren’t involved*
● easy to measure the business impact
● you have a shelter when something is failing*
● great way to learn your customers
○ not all experiments are charged for succeed
● general product deployment is easy easier
*almost

10. Trade offs
● the “space” is limited
○ test one idea at time
■ not really
● the codebase can really get overcomplicated
● things change over time
● ethical part: experiments on people?!

11. Experiment workflow

12. Most important steps of the experiment
1. Problem
2. Hypothesis
3. Prediction
4. Experiment
5. Analysis
6. Decision

13. Problem

14. Hypothesis

15. Prediction

16. Experiment

17. Analysis

18. Decision

19. Why experiments fail?

20. Technical issues
● bugs
● increase in the page load time
● check different browsers
● mobile?!

21. Keep context in mind
● something became less notable
● where did you get space for your feature?
● [design] does it fit the overall experience?

22. Audience
Do we show changes to the right audience?
… but really, do we show it to the right audience?

23. Low traffic
Did you collect enough of data?
https://www.qualtrics.com/blog/determining-sample-size/

26. Exercise time!

27. New block on the
page

28. What can go wrong?

30. Dart Vader
paradox (c)

31. Experiments with
translations

32. Summary

33. Should I use A/B testing?
opinions and assumptions aren’t involved*
easy to measure the business impact
great way to learn your customers and their
preferences
not all experiments are charged for succeed
general product deployment is easy easier
takes some time
things change over time
ethical questions
a lot of pitfalls
complicated code on scale

34. Why is nobody clicking that button?
it’s the wrong color
it’s not big enough
it says “buy now” not “add to cart”
it should be round
it should be flat
Your product is ...less than perfect
Nobody know what your product does
Your product is overpriced
Your website doesn’t look professional
Your website is slow
You charge too much
Your product is out of season
Source: http://www.slideshare.net/hayluke/ab-testing-uxcamp

35. Fancy stuff

36. Experiment game: http://lukasvermeer.github.io/confidence/
Booking.com AI game: https://workingatbooking.com/blog/hackman/
Careers: https://workingatbooking.com/

37. Thank you for
your attention!
Q&A time!
Ivan Mylyanyk
Blog @ Medium
Social networks