1.
Sample Size
The indispensable A/B test calculation
that you’re not making.

2.
As Marketers,
many of us run A/B Tests

3.
We test copy

4.
We test design

5.
We test subject lines

6.
We choose winners

7.
Version A is converting better than
Version B and statistical significance
has breached 95%.
So, Version A won.

8.
Version A is converting better than
Version B and statistical significance
has breached 95%.
So, Version A won.
OR DID IT?

9.
That math is half-baked

10.
Suppose you check an A/B Test twice:
Once after 200 impressions and then after 500.
Then you end the test.

11.
Now, instead, suppose you stop the test once
you reach significance:

12.
Now, suppose you stop the experiment as soon
as there is a significant result:
FALSE POSITIVE!

13.
How often will you get
a false positive?

14.
Assuming you check results after every impression and
stop once you reach significance….
26.1%
So you just went from 95% confidence to 74%
This is a worst-case scenario. BUT, some test platforms do this automatically!

15.
OK…well, then when should I stop an
A/B test?

16.
SAMPLE SIZE
Dictates how long to run a test

17.
SAMPLE SIZE
• Used religiously in the pharmaceutical Industry,
economic studies, etc…

18.
https://www.optimizely.com/resources/sample-size-calculator

19.
Agenda
1. How we put this into practice on a website test
2. How we applied these learnings to email testing:
• Open rates
• Click to Open Rates
• Conversion Rates

20.
A/B Testing on your website
Here’s your new test process:
1. Determine your baseline conversion rate (or click rate, or
download rate, etc..)
2. Decide how long you are willing to wait for a result. Convert your
unique traffic metric to a sample size.
3. Adjust MDE (Minimum Detectable Effect) until your Sample Size is
just under the target you determined in #2 above.
4. Re-adjust MDE until you are content.
5. Start the test, and don’t stop until you hit the sample size.

21.
Case Study: Item Urgency

22.
Case Study: Item Urgency
TEST (VERSION A):
INVENTORY NOTIFICATION
CONTROL (VERSION B):
NO INVENTORY NOTIFICATION

23.
STEP 1 – We determined our baseline conversion rate

24.
STEP 2 – Calculate Target Sample Size
We initially decided we wanted a result in 2 weeks.
So we took the last 2 weeks of unique product page views:

25.
STEP 2 – Calculate Target Sample Size
We then divided that number by two (since we’ll have two test
segments)
Divided by two again to account for desktop traffic only
Then multiplied by 5% (since the message only displays on 5% of
product pages)
Sample Size -> 12,351

26.
This gave us 30% MDE (Conversion Lift). This is unrealistic

27.
How about 10% ?

28.
107,105 unique visits ~ 17 weeks

29.
Wow, that’s a long time…

30.
Yep.

31.
You’re probably not
running your tests long
enough

32.
WAIT A MINUTE.
MY A/B TEST PLATFORM SAYS NOTHING
ABOUT SAMPLE SIZE…

33.
EVERYONE WANTS INSTANT GRATIFICATION

34.
YOUR A/B TEST PLATFORM IS HAPPY TO SELL IT

35.
Quietly assuming you have calculated
sample size on your own

36.
Item Urgency - Test Results
We are over 4 weeks in….
*Conv. rate is higher than expected because test platform runs on 7 day conversion window.

37.
Item Urgency - Test Results
Lift is over 10%
Note the spike in the beginning and the increased stabilization with time

38.
Test Results
The effect is slowly approaching the MDE

39.
Test Results
Significance is now over 95%, but it’s been up and down.
Many marketers would stop the test on 9/5 and declare a 57% Lift.

40.
Email Testing

41.
After learning about Sample Size,
we reconsidered our email testing strategy
• Open Rate (Subject line testing)
• Click-to-Open (CTO) Rate
• Conversion Rate

42.
OPEN RATE
We used sample size to gut check the
size of our subject line test segments

43.
OPEN RATE
Remember, for the sample size calculator,
you need the baseline conversion rate and
then the sample size, and that will give you
MDE.

44.
OPEN RATE
First, we needed the baseline conversion open rate

45.
OPEN RATE
Our open rates typically end up ~ 17% , but when
we make the call on our winning subject line, open
rates are usually around 7%.

46.
OPEN RATE
Next we need the sample size

47.
OPEN RATE
We always test 4 different subject lines.
We had been sending each subject line to 10,000
customers.
So, sample size ~ 10,000

48.
OPEN RATE
Plugging these numbers in, this would only detect 13% open rate lift or higher

49.
OPEN RATE
13% lift on 17% open rate is 19.2%.
We rarely see subject lines this high
We needed a lower MDE to make sure we could detect
more winners…

50.
OPEN RATE
We ended up doubling our subject line segment to
80,000, giving us an MDE ~ 9.2%

51.
CTO
First we needed the baseline

52.
CTO
We averaged the last 10 weeks -> 11% CTO

53.
CTO
Sample size = ½ of the avg opens count

54.
CTO
We averaged the last 10 weeks -> Avg
opens = 107,000 / 2 = 53,500

55.
CTO

56.
CTO
4.4% CTO lift is a very reasonable goal for a test.
This showed us that we could trust most of the
results of our past CTO tests.

57.
GRID vs. FREE FORM
15.7% CTO Lift

58.
PRODUCT NAMES vs. NO PRODUCT NAMES
22.6% CTO Lift

59.
Conversion Rate
We had been making many email decisions
after reaching significance
on a conversion rate lift

60.
Conversion Rate
Time for a reality check.

61.
Conversion Rate
Baseline Conversion Rate ~ 1.5%

62.
Conversion Rate
Sample Size = ½ Average # Clicks -> 6,000

63.
Conversion Rate

64.
Conversion Rate
38% is ASTRONOMICAL

65.
Conversion Rate
To get meaningful results for conversion rate,
consider running an email test many times, so that
you can eventually reach the necessary sample
size.

66.
Takeaways
This is the MDE curve again. Remember what this looks like.
The longer you run a test, the lower MDE will be.
The more traffic volume you have, the faster MDE will drop

67.
Takeaways
For Web Testing
• If you stop your A/B tests once you reach statistical significance, you are increasing your chances of finding
false positives
• Calculating sample size will give you a clear stop date and an MDE
• MDE and sample size are inversely related – The lower the MDE, the larger the sample size
• Most likely, your A/B tests need to run much longer than you realize
For Email Testing
• Use sample size to determine the size of your subject line test segments
• Your CTO tests are probably reaching the necessary sample size
• Your Conversion tests are probably not hitting sample size

68.
Sources
Kyle Rush – Mozcon 2014 Presentation
https://seomoz.box.com/shared/static/2fw6yevkkmmdum
z431j4.pdf
Evan Miller – How not to run an AB test
http://www.evanmiller.org/how-not-to-run-an-ab-test.
html

69.
Zack Notes
Digital Marketing
Manager
zack@uncommongoods.com
@zacknotes
slideshare.net/zacknotes1/presentations

70.
Appendix

71.
GRID vs. FREE FORM

72.
PRODUCT NAMES vs. NO PRODUCT NAMES

73.
What do you do if a test reaches sample size
and your lift < MDE?

74.
You can either extend the test and accept a
lower MDE or Move On.