1. Testing for Success
How to Effectively Utilize A/B Split Testing

2. Fishbowl Best Practices Webinar
Hosted by:
Drew Hebble
Senior Marketing Specialist

3. AGENDA
• The Basics
• Getting Started
• Improving Open Rates
• Improving ClickThrough Rates
• After the A/B Split Test
• Q&A

4. THE BASICS:
WHAT IS AN A/B SPLIT TEST & HOW DOES IT WORK

5. What is an A/B Split Test?
A simple method for comparing the performance of
two different versions of an e-mail campaign

6. How Does an A/B Test Split Work?
• A small subset of the total member list is selected as a test group and divided
into two equal parts.
• Each half of the test group receives a different version of the same message
and a winner is identified
The higher-performing message is blasted to the remainder of the list.

7. How Will A/B Split Testing Help my eClub?
Understanding the behavior of your e-club members will help you
construct a messaging strategy that better engages them
Engaged members drive:
– Higher open rates
– Higher click-through rates
– Lower unsubscribe rates
THE BETTER YOUR PROGRAM PERFORMS AGAINST THESE
KEY METRICS, THE GREATER THE SALES IMPACT!

8. GETTING STARTED:
CHOOSING THE APPROPRIATE A/B SPLIT TEST

9. Getting Started: What Should You Test?
Each A/B split test should isolate a specific message component
that has the potential to provide actionable results
Common test variables include:
 Subject lines
 Creative design
 Call to action language
 Call to action placement
 Call to action style
Actions based on
 Send times (time of day or week) findings
 From lines

10. Getting Started: What Should I Measure?
Open Rate
• Key Test Variables
– Subject line Open CTR
– Time of day & day of week
Click-through Rate (CTR)
• Key Test Variables
– Call to action
What’s Most
• Language, Common
• Style
• Placement

11. Getting Started: Guidelines for a Clean & Effective Test
1. Restrict each test to 1 variable
A comparison of two subject lines sent at different
times of day makes it difficult to directly correlate
results with to one of the variables
2. Make sure that your results can be measured
Comparing two non-clickable calls to action would not
provide an action to measure
3. Construct a fair comparison
“Free appetizer today!” vs. “Your October Restaurant
Newsletter” are too different to provide a fair comparison.

12. IMPROVING OPEN RATES:
A/B SPLIT TESTING OF SUBJECT LINES

13. Subject Line Testing: Good Examples
• Offer “teaser” vs. Specific offer
Enjoy this Special Birthday Gift!
Enjoy your Free Entrée Birthday Gift!
• Personalization
Drew, enjoy a free dessert on us!
Enjoy a free dessert on us!
• Subject line length
We’ve got it all!
We’ve got it all! Good food, music, giveaways and more!

14. Subject Line Testing: Good Examples
• Symbols
Rainy day special!
Rainy day special!
• Placement of offer copy
Free Appetizer for you and a Friend!
You and a Friend can enjoy a Free Appetizer!
Adjust testing types, based on your branding, assumptions
and historical data

15. Subject Line Testing: Is It a “TRUE” Test?
Have a Free Appetizer on us!
vs.
Drew, we’d like to offer you a Free Appetizer – today only!
You’re going to love this great deal!
vs. CAUTION…
UNFAIR
A Free Entrée for you and a friend! COMPARISON
AHEAD

16. Subject Line Testing: “I’ve sent my test, now what?”
1. Review key performance metrics for each version of the message
(opens, click-throughs, and unsubscribes)
2. Send the top performing subject line to remaining members
3. Monitor winning subject line performance?
4. Analyze the results and use them to improve on next campaign?
The results of each test should
add one more piece to the puzzle

17. What Does Winning Performance Look Like…..
25%
20%
IMPORTANT! 15%
Consider ALL messaging
metrics, NOT just open rate 10%
5%
0%
Open Rate ClickThrough Rate Unsubscribe Rate
Subject Line A: Enjoy this Special
19% 1.20% 0.12%
Birthday Gift!
Subject Line B: Enjoy your Free Entrée
23% 1.40% 0.13%
Birthday Gift!

18. IMPROVING CLICKTHROUGH RATES:
A/B TESTING OF CALL TO ACTION

19. Call to Action Testing: Language
• Question vs. statement
• Click Here vs.  (an implied clickthrough icon)
Want a Free Click Here for your
Appetizer? Free Appetizer

20. Call to Action Testing: Placement
• Left vs. right
• Top vs. bottom
• Inside an image vs. stand alone
Happy Birthday Happy
on Us! Birthday
on Us!
Click Here for a free
brownie during the Click Here for a
week of free brownie
your birthday. during the week of
your birthday.

21. Call to Action Testing: Visual Style
• Clickable body copy vs. button
• Animation vs. static image
• Different color copy/button
To help you through To help you through
this rainy this rainy weather
weather, Click here
to enjoy a free Click Here for your
appetizer with any free appetizer with
entrée purchase. any entrée purchase.

22. Call to Action Testing: Is It A “TRUE” Test
• Different images on each version
• One has a clickable call to action, the other has no measurable variable
AVOID UNFAIR
COMPARISONS
Click Here for your
free appetizer with
any entrée purchase.

23. Improving CTR: “I’ve sent my test, now what?”
1. Review click-through performance metrics for each version of the message
2. Send top performing call to action to remaining members
3. Monitor winning call to action performance across the remaining audience?
4. Analyze the results and use them to improve on next campaign?
Want a Free Click Here for your
Free Appetizer
Appetizer?
Version A (2.8% CTR) Version B (1.7% CTR)

24. AFTER THE A/B TEST:
ANALYZING RESULTS & APPLYING LEARNING’S

25. There Was No Clear Winner…What Should I Do?
• Both versions performed below your average?
– Negative results can be as useful as positive ones. Learn from what your guests
don’t respond to, and try a different strategy on your next campaign.
• Click-through results don’t lean one way or the other?
– Review your testing method to ensure that there the difference between the two
message versions is significant.
A “Click here to RSVP” vs. “Click
below to RSVP” call to action may
not yield a different result

26. There Was No Clear Winner…What Should I Do?
One version had a much higher open rate, while the other version had a
far superior clickthrough rate?
It’s important to consider the amount
of variation between metrics.
Depending on the focus of your
message, it’s sometimes better to Subject line test results
sacrifice one slightly lower metric for
another that is much greater. A) 23% open rate & 1.1% CTR
B) 21% open rate & 3.3% CTR

27. Acting on Your Results
• Shorter subject line outperformed a longer subject line?
• Red call to action yielded a higher click-through than a blue one?
Make Assumptions based on learning and test on future campaigns
Are you able to replicate the results
If YES – You’re on the right track to better understanding your audience.
If NO –
Was there a difference on this test that may have swayed results?
Refine your methodology and continue to test until results are consistent.

28. Three Key A/B Testing Takeaways
1. ALWAYS test variable at a time
2. ALWAYS review the performance of your entire mailing, NOT
just the A/B test
3. ALWAYS apply your learning's to future campaigns
4. NEVER stop testing and refining

29. Questions?
Thanks for your time!

30. Upcoming Fishbowl Webinars
October 18th Optimizing your website to attract new e-club members
November 15th Messaging Best Practices

31. Resources
For more information on Fishbowl’s Creative and strategic
Professional Services offerings:
Enterprise Clients: Please contact your Account Director
Local Clients: Please contact your Sales Representative
Technical questions on how to do A/B subject line tests?
Call our Support line 800.883.1984 or
email clientsupport@fishbowl.com
Sign up for a webinar:
http://www.fishbowl.com/feature/webinar_enterprise_series