April 14th - 09:45 - 12 hours earlier Ton Wesseling was asked to replace Craig Sullivan as morning keynote of Emerce Conversion 2015. Based on the ROAR model he created for CROday 2015 and the analytics tips on A/B-testing slidedeck he created voor DDTT on A/B-testing, he presented these slides for a 350 people audience in Amsterdam.
Any questions?
Email: ton@onlinedialogue.com
Twitter: @tonw
1. ONLINE DIALOGU E
G u i d i n g y o u t o d i g i t a l m a t u r i t y
D e m y t h e n , d e l e u g e n s e n d e i l l u s i e s
# C o n v e r s i o n 2 0 1 5 – A m s t e r d a m – A p r i l 1 4 t h 2 0 1 5
To n W e s s e l i n g
A /B-testen
13. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
More experiments
“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon – 2004!
14. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Like Amazon, Like Zalando
GermanEcommercemarket,thankyouAndréMorys
40. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ROAR - Optimization
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
41. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Warning: low significance levels
ü Big chance the winner is not a real winner
42. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Warning: low power levels
ü Big chance a real winner is not recognized
43. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
ü 20 out of 100 ideas is a winner…
ü …with an average uplift of 10%
45. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized(40% power)
ü 8 real winners & 10 false positives
ü NOT 18 improvements of 10%+
46. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
You need big numbers per variation
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
48. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Tests with impact, on average
1 out of 3
Simply truth: you’re just not always able to create a winner with enough impact!
51. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
What to test? Business cases!
Delivery –test with
in stock. 4 weeks
or 2 days?!
52. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
What to test? Business cases!
Service charge, do
it or not? Include
in price or not?!
54. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Warning: choose what to analyze up front
ü You just go and dig unitil you find something
(you will always find something that seems true BUT is not!)
58. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
You just let it run until it’s significant
Your test has a fixed lenght!
It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!
60. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ROAR – Optimization – moving up
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
61. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Finally you can test details…
Is Multi Variatie Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!
66. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Data & Psychology
Digital Data
Persuasion Psychology
Analyze
&
Experiment
70. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
FA C T & A C T – 7 s t e p s t o l e a r n
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
79. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
FA C T & A C T – 7 s t e p s t o l e a r n
&
Tell
Conclude
Analyze
Test
Create
Analyze
Find
80. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Now you’re testing
On your way to
4 tests a week
Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst!
81. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Your winners are still spare?
Less than 25%?
Please higher your significance level to 95% or more if it drops below this 25 number!!
82. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Tip: Your test takes too long
ü Cookie deletion
(you will loose 10% in two weeks)
They will re-enter the the and pollute your samples!
83. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Tip: Don’t confuse cross device users
ü Only test 1 device at a time!
85. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Tip: You forgot to leave out some visitors
ü Only test on those who can be changed
(if you test a sales promotion – leave out your current clients who just logon)
86. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Tip: You’re not slowing the control down
ü Add the same code to the control
87. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Tip: You’re not testing on fresh people
ü Start with control for a full purchase cycle
(and then start sending traffic to your variations – setting the control to 0%)
88. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Tip: You forgot to give them time to buy
ü Stop getting fresh visitors after your test time
(send new traffic to the control, but give you tested people time to finish converting)
89. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ROAR – Automation
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
90. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
You’re already thinking of more teams
And still have test capacity
*test capacity – the number of tests you can start per week based on your audience & conversions!
94. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Now you’ve moved to
Algorithm testing
Which internal and external influencers do have an impact on which segment of users?!
95. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ROAR – Automation
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
99. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Customer knowledge center
Making money from conversion improvements
Creating more and more customer insights
Which can be monetized
on even a larger scale
101. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
So you keep on groing
GermanEcommercemarket,thankyouAndréMorys
102. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
Done testing? No!
TESTING NEVER STOPS
Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!
103. onlinedialogue.com
#Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015
ROAR – when to start with what
Time span!
Conversionspermonth!
Risk! + Optimization! + Automation! Re-think!
1.000 conversions!
per month!
10.000 conversions
per month!
104. ONLINE DIALOGU E
G u i d i n g y o u t o d i g i t a l m a t u r i t y
D e m y t h e n , d e l e u g e n s e n d e i l l u s i e s
# C o n v e r s i o n 2 0 1 5 – A m s t e r d a m – A p r i l 1 4 t h 2 0 1 5
To n W e s s e l i n g
A /B-testen