1. SUBTITLE BELOW
Optimize for €€€
HOW WE OPTIMIZE AT THE BIGGEST HOTEL CHAIN IN THE NETHERLANDS

2. @AM_Klaassen / @tomvdberg1
How we got here…

3. @AM_Klaassen / @tomvdberg1
The quest for female speakers…

4. @AM_Klaassen / @tomvdberg1
Female speakers…
?

5. @AM_Klaassen / @tomvdberg1
Our lovely colleagues

6. What we do…

7. @AM_Klaassen / @tomvdberg1
Conversion rate optimization
Analytics Psychology

8. Lots of A/B-tests

9. @AM_Klaassen / @tomvdberg1
Our clients

10. @AM_Klaassen / @tomvdberg1
Adding
direct value
Learning
user behavior

11. @AM_Klaassen / @tomvdberg1

12. @AM_Klaassen / @tomvdberg1

13. @AM_Klaassen / @tomvdberg1
Van der Valk
 Unique website for each hotel
 1 centralized team
 Paid by % of their turnover

14. @AM_Klaassen / @tomvdberg1
Optimization of Valk
 Increase revenue through CRO
 Reduce costs per experiment

15. “We’re not in the business of
science, we’re in the business
of making money”

16. 3 Tactics

17. 1.
Free test software

18. @AM_Klaassen / @tomvdberg1
Jorrin Quest
GTM testing
http://gtmtesting.com

19. @AM_Klaassen / @tomvdberg1
GTM testing
 Easy setup
 Free
 Preview mode

20. 2.
Change of statistics

21. @AM_Klaassen / @tomvdberg1
Focus on finding proof

22. @AM_Klaassen / @tomvdberg1
Example test: Order flow

23. @AM_Klaassen / @tomvdberg1
Example test: Set completion
A B

24. @AM_Klaassen / @tomvdberg1
Test result

25. @AM_Klaassen / @tomvdberg1
• No effect on conversion
 Don’t implement
 Re-test with higher volumes
Conclusion

26. @AM_Klaassen / @tomvdberg1

27. @AM_Klaassen / @tomvdberg1
What’s the alternative?
Frequentist
statistics
Bayesian
statistics

28. @AM_Klaassen / @tomvdberg1
Example test: Set completion
A B

29. @AM_Klaassen / @tomvdberg1
Bayesian Test evaluation

30. @AM_Klaassen / @tomvdberg1
Focus on risk assessment
88,4%
A test result is the probability that B outperforms A:
ranging from 0% - 100%

31. @AM_Klaassen / @tomvdberg1
IMPLEMENT B PROBABILITY * EFFECT ON REVENU
Expected risk 11,6% - € 69.791
Expected uplift 88,4% € 213.530
Contribution € 180.733
* Based on 6 months and an average order value of € 175
Make a risk assessment

32. @AM_Klaassen / @tomvdberg1
Implement B
B

33. @AM_Klaassen / @tomvdberg1
abtestguide.com/bayesian/
Roy Schieving
Annemarie Klaassen

34. @AM_Klaassen / @tomvdberg1
Conclusion Bayesian
 Easier to understand and
communicate
 Better suits the business
 Don’t throw away good ideas
(indicatively significant)
 Higher implementation rate and
revenue
€ 0
€ 1.000.000
€ 2.000.000
€ 3.000.000
€ 4.000.000
€ 5.000.000
€ 6.000.000
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39
Expected impact on revenue in 1 year
€ Frequentist € Bayesian

35. 3.
Change of test method

36. @AM_Klaassen / @tomvdberg1
50%
50%
A/B-test

37. @AM_Klaassen / @tomvdberg1
Traffic allocation over time
A
B
Traffic
Time
Profit or loss?

38. @AM_Klaassen / @tomvdberg1
What’s the alternative?
A/B-test Bandits

39. If you have multiple options:
which bandit
will pay-out the
most?

40. @AM_Klaassen / @tomvdberg1
Explore Exploit
?
Explore / Exploit dilemma

41. @AM_Klaassen / @tomvdberg1
Bandit
100%

42. @AM_Klaassen / @tomvdberg1
Bandit
80%
20%
10%
90%

43. @AM_Klaassen / @tomvdberg1
Traffic allocation over time
A
B
Traffic
Time
More Profit!

44. @AM_Klaassen / @tomvdberg1
Limited time frame Automation
When use Bandits?
Earn over learn

45. How?

46. @AM_Klaassen / @tomvdberg1
Example 1

47. @AM_Klaassen / @tomvdberg1
Example 1
 Older target group
 Comparing price on
other sites
 Behavior of others

48. @AM_Klaassen / @tomvdberg1
Example 1
Control No message
Variation B Copy default
Variaton C
Variation D

49. 11,1%
10,6%
10,4%
10,5%

50. @AM_Klaassen / @tomvdberg1
Example 1 – Effect on revenue
A
B
Traffic
Time
+ € 6.637

51. @AM_Klaassen / @tomvdberg1
Example 2
Control New control
Variation B Copy control
Variation C
Variation D

52. 9,2%
8,9%
7,8%
8,4%

53. @AM_Klaassen / @tomvdberg1
Example 2 – Effect on revenue
A
B
Traffic
Time
+ € 19.862

54. @AM_Klaassen / @tomvdberg1
Conclusion Bandits
 Less regret during testing
 No implementation costs
 Optimizing per user segment

55. 1. Free test software
2. Bayesian statistics
3. Bandit testing

56. revenue
Maximize

57. @AM_Klaassen / @tomvdberg1

58. “We’re not in the business of
science, we’re in the business
of making money”

59. TÄNAN VÄGA!
Bayesian calculator: abtestguide.com/bayesian/
Slide deck: ondi.me/elitecamp2016
@AM_Klaassen
annemarie@onlinedialogue.com
nl.linkedin.com/in/amklaassen
@tomvdberg1
tom@onlinedialogue.com
nl.linkedin.com/in/vandenbergtom