This document discusses A/B testing of emails and landing pages. It explains that A/B testing involves showing different versions of a site to visitors to see which performs better according to goals like purchases or leads. Effective A/B tests identify goals, choose a page or email to test, select an element to vary like headlines or buttons, state a hypothesis, and track results to determine a winner. Examples of testing ideas included formatting, subject lines, "from" lines, button placement, headers and banners. The document emphasizes using data from A/B tests to improve performance rather than personal preferences.

1. Test for Success
A Guide to A/B Testing on Emails & Landing Pages

2. #abtesting
Our presenters
Jessica Langensand
Marketing Programs Manager
Marketo
Allison Sparrow
Marketing Automation Manager
Optimizely

3. A few things first...
● Slides and recording emailed to you immediately
#abtesting
after the webinar.
● Ask questions in the chat box & on Twitter!
○ Use #abtesting.

4. #abtesting
What you’ll learn
● Why should you A/B test?
● Setting up your first (effective) A/B test
● Testing Ideas and Success Stories

5. #abtesting
Why Should You Care About
A/B tests?

6. #abtesting

7. #abtesting

8. #abtesting
There are 4.1 Billion email
accounts in existence.
Source: The Radicati Group

9. #abtesting
In 2013, SMBs voted that email
marketing was #2 most
important marketing channel.
#1, having a website.
Source: eMarketer

10. #abtesting
2.3% {

11. A/B Testing is the process of
showing different versions of a site
to visitors, in order to see which
one better achieves your goals.
#abtesting

12. #abtesting
An A/B test is...

13. #abtesting

14. #abtesting
One Page/Email

15. #abtesting
One Component
One Page/Email

16. #abtesting
One Component
One Page/Email
One Split Audience
(50/50)

17. #abtesting
A B

18. #abtesting
A B
43.4% more pre-orders

19. Why should you care about A/B tests?
● Let the data do the work
● Build internal alignment
● Your website and email channels are invaluable
#abtesting
(make the most of it)
● It’s easy, and cheap

20. #abtesting
What is an (effective) A/B test?

21. #abtesting
What is an (effective) A/B test?
● Identify your company’s goals
● Choose where to test
● Choose what to test
● State your hypothesis
● Track and report results

22. #abtesting
What are my company’s goals?

23. #abtesting
What are my company’s goals?
● Purchases/conversions
● Revenue per visitor
● Lead generation
● Pageviews

24. #abtesting
Where to test?
Welcome Page Product Page

25. #abtesting
Where to test?
● Welcome email
● Home page
● Product page
● Contact Form page
● 30 day expiration email
● Download confirmation
email

26. #abtesting
What to test?

27. #abtesting
What to test?
● Headline
● Subject Line
● Value Proposition
● CTA

28. #abtesting
Create a Hypothesis
“Visitors are getting distracted and feeling
overwhelmed with too many competing elements
on the page. Removing elements will reduce the
clutter, focus user attention on article content and
result in an increase in article pageviews.”

29. #abtesting
State a Hypothesis
● Identify a theory about the specific user
experience
● Define the metrics for success
● Define the quantifiable change in the key metric

30. #abtesting
Track and Report Your Results

31. #abtesting
How to Determine a Winner

32. #abtesting
Email

33. #abtesting
Statistical Significance
Source:
ZettaSphere.com

34. #abtesting
Statistical Significance
Source:
ZettaSphere.com

35. #abtesting
Landing Pages

36. #abtesting
Significant?
Source: SplitTestCalculator.com

37. #abtesting
Testing Ideas

38. #abtesting
Format
• Font
• Text Size
• Color
• Branding
• Images :)

39. #abtesting
Subject Lines
A
B

40. #abtesting
A
Subject Lines
B + 33% click through rate

41. Subject Lines
● Vary a Replicable Element
[New Report] 2014 Marketing Automation Vendor Comparison
vs.
[Download] 2014 Marketing Automation Vendor Comparison
#abtesting
● Complete Subject Line Test
○ Test on a small group, wait 24 hours and send winner remainder of your list

42. #abtesting
“From” Line
“Scott Magri”
+ 4% opens +3% click-through
vs.
“Marketo Premium Content”

43. #abtesting
Button Placement

44. #abtesting
Button Placement
+6% CTR

45. #abtesting
Headers

46. #abtesting
Banners

47. #abtesting
Banners
10% CTR 5% CTR

48. #abtesting
Button Color

49. #abtesting
Button Color
+21% CTR

50. A/B testing isn’t about what you
like. It’s about what your
audience responds to.
#abtesting

51. #abtesting

52. #abtesting
Questions?
Use Chat

53. #abtesting
Thank you!
Recording is on the way!

54. #abtesting
Appendix - Resources for A/B tests
- statistical significance
- sample size calculator
- a/b test archive template
- variation calculator
- toolkit

55. #abtesting
No one has time to test everything...
Use Data to Determine what to test.