Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

< H a v e m o r e f u n A / B - t e s t i n g >
A/B-t e s t i n g Errors
To n We s s e l i n g – C EO Te s t i n g . A g e n c y
#DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4

Why me?
ü 10 years+ A/B-test experience
ü I always loved the numbers side
ü I’ve made all these mistakes
Email: ton@testing.agency Twitter: @TonW #DDTT

We optimize

Nature

A better example

Groundhog Day – The A/B-test movie

Trying to get the girl

Constant testing environment
A / B Testing

Better Dialogue Optimization
Default
Variation
B
Variation
Winner!"
A

This is why you’re A/B-testing
Small steps to learn how
to make the next big step

Testing is not free
Use it to LEARN

50% of the A/B-tests I get to see is
wrong

Look at your A/B-testing results!
+24%
+52%
+126%
+83%
+12%
+16%

Did you push the winners live to your
site?
You must be rich now!

Or is this reality?

Or even worse:
A/B-test winners are letting you down!

Are these real winners?
+24% +52%
+83% +12% +126%
+16%

20 reasons why
Your A/B-test
analyzing
sucks

Get your statistics right!

1: low significance levels
ü Big chance the winner is not a real winner

2: low power levels
ü Big chance a real winner is not recognized

Example test set-up
ü 90% significance level
ü 100 conversions per variation
ü Average conversion rate: 2%
ü 20 out of 100 ideas is a winner
ü With an average uplift of 10%

Average: 75% significance & 40% power

So:
ü 100 tests will give 10 false positives (90% significance)
ü 40% of the 20 winners are recognized (40% power)
ü 8 real winners & 10 false positives
ü NOT 18 improvements of 15%+

3: Not calculating traffic up front
ü You thought the page had thousands of
visitors

4: Not calculating what you can improve
ü You thought the page had lots and lots
conversions

ABTestGuide.com/calc

5: Not choosing your segments up front
ü You just go and dig unitil you find something
(you will always find something that seems true BUT is not!)

6: You stop the test once it is significant

7: You just let it run until it’s significant

7: You just let it run until it’s significant
Your test should have a fixed lenght!

8: Your test takes too long
ü Cookie deletion
(you will loose 10% in two weeks)
They will re-enter the the and pollute your samples!

9: You just choose a test length
ü At least 1 full purchase cycle!

10: You don’t test full weeks
ü At least 1 full week
(if your not testing for time of day / week effects)

11: You think they always use one decive
ü Only test 1 device at a time!

12: You’re testing broken stuff
ü Always check browser compatibility!
ü Dont’ break dynamic stuff!

13: You have multiple tests that overlap
ü Your variation did not won
(it was a combination of variations)

14: You thought this page matters
ü But it did not
(so you decide to use some other metric to declare the winner)

15: You forgot to leave out some visitors
ü Only test on those who can be changed
(if you test a sales promotion – leave out your current clients who just logon)

16: You’re not slowing the control down
ü Add the same code to the control

17: You’re not testing on fresh people
ü Start with control for a full purchase cycle
(and then start sending traffic to your variations – setting the control to 0%)

18: You forgot to give them time to buy
ü Stop getting fresh visitors after your test time
(send new traffic to the control, but give you tested people time to finish converting)

19: You’re not diving into your analytics
ü Always measure with analytics software
(You will find conversions that are not supposed to be there!)

20: You’re not measuring the right stuff
ü Use unique visitors
ü Against lifetime value predictors!

If you keep on making mistakes:
Stuff like this happens

Bonus: You did not re-test your winners
ü Always re-test to be sure it was not a glitch

So when you implement winners
This will happen!

< H a v e m o r e f u n A / B - t e s t i n g >
A/B-t e s t i n g Errors
To n We s s e l i n g – Te s t Wi n n e r
#DDTT A m s t e r d a m 2 0 1 4 / 0 6 / 2 4

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (18)

Similar to Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors

Similar to Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors (20)

More from Online Dialogue

More from Online Dialogue (20)

Recently uploaded

Recently uploaded (20)

Keynote Digital Data Tips Tuesday - Amsterdam - June 24th 2014 - A/B-testing analytics errors