This document discusses best practices for running A/B tests, including collecting enough data over multiple weeks and business cycles, avoiding relying solely on significance levels to determine a winner, and integrating test data with web analytics. It also emphasizes formulating a clear hypothesis by outlining why a change is needed and how its impact will be measured. The presenter advocates getting familiar with key statistical concepts and learning statistics.

2. Your%Presenter
Michael%Aagaard
4 Senior%Conversion%Op:mizer%at%Unbounce%
4 Full4:me%CRO%for%the%lat%seven%years%
4 Interna:onal%speaker%
4 Founder%of%ContentVerve.com

3. Join%the%conversa:on
4%Include%the%hashtag%#thinkppc)in%your%TwiJer%tweets.%
4%Or%use%the%webinar%ques:on%box%to%send%us%ques:ons.

4. “When can I stop my A/B Test?”

5. “Why run an A/B Test?”

6. To get a representative picture of how a
variant would perform if implemented
permanently “in the wild”.

7. To get a representative picture of how a
variant would perform if implemented
permanently “in the wild”.

8. Do Not Use 95% Significance
Level as a Stopping Rule
1

9. After 1 day
+151.4% / significance 97%
655 visitors / 18 conversions

10. After 2 days
+98.4% / significance 95%
988 visitors / 23 conversions

11. After 1 week
+0.2% / significance 50%
3031 visitors / 58 conversions

12. After 2 weeks
+18.1% / significance 77%
5277 visitors / 78 conversions

13. After 4 weeks
+36.2% / significance 96%
Total sample: 9396 visitors / 122 conversions

14. What we have now:
9,396
What we need:
34,304 visitors in total
What we’re missing:
24,908 visitors
73% of the required
sample size.
http://www.evanmiller.org/ab-testing/sample-size.html

15. http://abtestguide.com/calc/

16. “Formulas don’t know whether they are being used properly,
and they don’t warn you when your results are incorrect.”
- Deborah J. Rumsey, PhD

17. “Real accuracy depends on the quality of data
as well as on the sample size.”
- Deborah J. Rumsey, PhD

18. Worth Getting Familiar With:
- Power Level
- P-Value
- Margin of Error
- Confidence Interval
- Null hypothesis / alternative hypothesis

19. Make Sure You Collect Enough
Data to Yield Meaningful Insight
2

20. Full weeks
Full business cycles

21. Weekly Business Cycle

22. Full weeks
Full business cycles
Week One Week Two Week Three Week Four

23. Test duration: 2-4 weeks (2 or more business cycles)
Sample size: Pre-calculated sample size (users)
400+ conversions (goals)
Significance: 95% or higher
Power: 80% P-Value: 0%
What I look for:

24. Be Careful With Averages -
They Hide the Truth
3

25. Devices
Browsers
Channels
Country
Days of
the week
Hours of
the day
User
types
Motivation
Campaigns
Age
Gender
Gender
BrowsersDevices
Browsers
Channels
Countries
Days of
the week
Hours of
the day
User
types
Motivation
Campaigns
Age
Days of
the month
Gender
Devices
Experiment

28. Integrate Test Data with
Your Web Analytics Setup
And Get the Full Picture

29. Device?
Channel?
Campaign?
User Type?

30. Formulate a Clear Test
Hypothesis
4

31. Why do we think we need to make a change?
What is it that we want to change?
What impact do we expect to see?
How will we measure this impact?

32. 1. Because we saw [data/feedback]
2. We expect that [change] will cause [impact]
3. We’ll measure this using [data metric]

33. Because [exit surveys indicated that saving money is
most important to our users.]
We expect that [tweaking our value prop to reflect this]
will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our
key metric.]

34. Because [exit surveys indicated that saving money is
most important to our users.]
We expect that [tweaking our value prop to reflect this]
will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our
key metric.]

35. Because [exit surveys indicated that saving money is
most important to our users.]
We expect that [tweaking our value prop to reflect this]
will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our
key metric.]

36. Because [exit surveys indicated that saving money is
most important to our users.]
We expect that [tweaking our value prop to reflect this]
will cause [more people to fill out our lead form.]
We’ll measure this using [form submission rate as our
key metric.]
We expect to see reliable results in [four weeks.]

37. Learn Statistics!!!
5

38. http://amzn.to/1RFNBwA

39. Live%Q&A%Time

40. Have%more%questions?
Contact%Us%Directly
• Webinar%Feedback:%marketing@hanapinmarketing.com
____________________________________________
Thank'you'for'attending'our'webinar!'#thinkppc