What is A/B testing? A SCIENTIFIC METHOD OF PROOF WHAT IS A/B TESTING FOR MARKETING • A scientific method adapted for marketing • Compare two or more versions of an asset to each other • Use data and statistics to determine which version is better • Implement winner and increase conversions HOW DOES A/B TESTING WORK? SO WHY DO WE A/B TEST? • Lowers risk • Helps us realize the actual impact of proposed changes • Increase learnings • Takes the guess work out of optimization How do you start testing? THE DON’TS WHERE DO YOU START TESTING? DON’T JUST GUESS SUB OPTIMIZE • Button colors • Small copy changes • Small layout changes • Fonts, font size, etc. * at least not until you have found 
 a great variation and have lots of traffic WATCH OUT FOR BUZZ • Don’t just test what the gurus say • What works for others may not work for you PREMATURE TESTING • Testing without an audience/traffic • Before you know what and why you are testing Where do you start testing? THE DOS DO YOUR HOMEWORK You have to put in the work to get great results • Learn as much as you can about A/B testing • Study the great people out there • What’s their process and insights? A framework for success The basis of a successful testing program Research Know what to test and see the greatest improvements DIVE INTO YOUR DATA FIND YOUR BIGGEST OPPORTUNITIES INVESTIGATE THE OPPORTUNITY TO UNDERSTAND IT FORM A HYPOTHESIS By doing A, B will happen, Because of C By focusing on the benefits of Kissmetrics instead its features (A), We will get more prospects to sign up for a trial (B), Because people understand the value of the product (C). BUILD THE TEST VS. Control Variant MEASURE THE TEST VS. Control Variant LAUNCH THE TEST CALL THE TEST VS. Control Variant +17.74% increase in registrations!! 98% Significance DOCUMENT AND SHARE YOUR FINDINGS Documenting all your tests and results • Increases company transparency • Helps others know where tests are going on • Makes it easier to call tests • Makes learnings available for everyone in company REPEAT THE PROCESS AS OFTEN AS POSSIBLE A STORY OF OUR FAILURE VS -13.68% increase in registrations!! 80% Significance NOT ALL TESTS ARE WINNERS ““Only 1 out of 8 A/B tests have driven significant change.” Noah Kagan AppSumo* ““The goal of a test is to get a learning, not a lift. With enough learnings, you can get the real lift,” Dr. Flint McGlaughlin MECLABS RULES OF THUMB FOR TESTING Tooltips AND BURGERS How I like my testing burger Tooltips for your testing framework Tool Company Why do I like it? Not everyone likes their burger the same way How do you take yours? Trust the process Share your leanings Stack your results A/B TESTING FOR SUCCESS REPEAT THE PROCESS AS OFTEN AS POSSIBLE FEEL LIKE A WINNER AND TAKE A VACATION Get Our Free PDF Guide to A/B Testing

1. `A/B Testing 101:
Off To A Great Start
THUE MADSEN

2. Thue is the Kissmetrics Webinar Wizard and Marketing Ops Manager. Before
joining forces with Kissmetrics, he was a Lyft driver in SF, which is also how
he ended up as a KISSmetrics marketer. Whenever Thue is not trying to
automate everything around him, you can find him hiking in the Sierras.
THUE MADSEN
Marketing Operations Manager, Kissmetrics
@ThueLMadsen

3. @thuelmadsen
#Kisswebinar
@Kissmetrics

4. The don’ts…
The dos…
1 What is A/B testing?
Do your homework
Repetition
Succeed
2 Where do you start testing?
TABLE OF CONTENTS
3 A framework for success
Tooltips and Burgers4

5. What is A/B testing?
A SCIENTIFIC METHOD OF PROOF

6. WHAT IS A/B TESTING FOR MARKETING
• A scientific method adapted for marketing
• Compare two or more versions of an asset to each other
• Use data and statistics to determine which version is better
• Implement winner and increase conversions

7. *Image from Optimizely
HOW DOES A/B TESTING WORK?

8. *Image from Kikolani
HOW DOES A/B TESTING WORK?

9. *Image from Dynamicyield
HOW DOES A/B TESTING WORK?

10. SO WHY DO WE A/B TEST?
• Lowers risk
• Helps us realize the actual impact of proposed changes
• Increase learnings
• Takes the guess work out of optimization

11. How do you start testing?
THE DON’TS

12. *Image from Optimizely
WHERE DO YOU START TESTING?

13. DON’T JUST GUESS

14. SUB OPTIMIZE
• Button colors
• Small copy changes
• Small layout changes
• Fonts, font size, etc.
* at least not until you have found
a great variation and have lots of traffic

15. WATCH OUT FOR BUZZ
• Don’t just test what the gurus say
• What works for others may not work for you

16. PREMATURE TESTING
• Testing without an audience/traffic
• Before you know what and why you are testing

17. Where do you start testing?
THE DOS

18. DO YOUR HOMEWORK
You have to put in the work to get great results
• Learn as much as you can about A/B testing
• Study the great people out there
• What’s their process and insights?

19. A framework for success
The basis of a successful testing program

20. Research
Know what to test and see the greatest improvements

21. DIVE INTO YOUR DATA
PaidEntered CCAdded to CartViewed Product

22. FIND YOUR BIGGEST OPPORTUNITIES
PaidViewed CheckoutAdded to CartViewed Product

23. INVESTIGATE THE OPPORTUNITY TO UNDERSTAND IT
And/Or

24. INVESTIGATE THE OPPORTUNITY TO UNDERSTAND IT
* peek.usertesting.com/

25. INVESTIGATE THE OPPORTUNITY TO UNDERSTAND IT

26. FORM A HYPOTHESIS
By doing A,
B will happen,
Because of C

27. FORM A HYPOTHESIS
By focusing on the benefits of Kissmetrics instead its features (A),
We will get more prospects to sign up for a trial (B),
Because people understand the value of the product (C).

28. BUILD THE TEST

29. BUILD THE TEST
VS.
Control Variant

30. MEASURE THE TEST
VS.
Control Variant

31. LAUNCH THE TEST

32. CALL THE TEST
VS.
Control Variant
+17.74% increase
in registrations!!
98% Significance

33. DOCUMENT AND SHARE YOUR FINDINGS

34. DOCUMENT AND SHARE YOUR FINDINGS
Documenting all your tests and results
• Increases company transparency
• Helps others know where tests are going on
• Makes it easier to call tests
• Makes learnings available for everyone in company

35. *Image from Optimizely
REPEAT THE PROCESS AS OFTEN AS POSSIBLE

36. A STORY OF OUR FAILURE
VS
Control Variant

37. A STORY OF OUR FAILURE
VS

38. A STORY OF OUR FAILURE
VS
-13.68% increase
in registrations!!
80% Significance

39. *Quote found on VWO
NOT ALL TESTS ARE WINNERS
““Only 1 out of 8 A/B tests have driven
significant change.”
Noah Kagan
AppSumo*
““The goal of a test is to get a learning,
not a lift. With enough learnings, you
can get the real lift,”
Dr. Flint McGlaughlin
MECLABS**
**Quote found on MarketingSherpa

40. RULES OF THUMB FOR TESTING
Things to remember when we are testing
• Min. 2000 visitors
• Min. a few hundred conversions
• At least one full cycle > 2 week
• >99% statistical significance
• Keep a backlog of test hypothesis
• Build next tests as the current one is running
*More info at http://kiss.ly/1vF4wUc and http://kiss.ly/1Q4vTm0

41. Tooltips
AND BURGERS

42. How I like my testing burger
Tooltips for your testing framework

43. Tool Company Why do I like it?
Analytics Kissmetrics
Shows me my actual
customers and their
behavior
Qualitative Research
Qualaroo,
SurveyMonkey,
Usertesting
Easy to use/Deep
qualitative insights
Project Management Trello
Manage my time +
communicate with
collaborators

44. Tool Company Why do I like it?
Company Wiki Confluence
Document initial
thoughts for everyone
Testing platform*
Optimizely, Pardot,
Adwords
Easily build tests and
get data
Analytics Kissmetrics
Let’s me confirm and
verify down my funnel
Company Wiki Confluence
Documentation and
learnings for
everyone
*Includes actual testing platform and tools that have testing capabilities

45. Not everyone likes their
burger the same way
How do you take yours?

46. Trust the process
Share your leanings
Stack your results
A/B TESTING FOR SUCCESS

47. *Image from Optimizely
REPEAT THE PROCESS AS OFTEN AS POSSIBLE

48. FEEL LIKE A WINNER AND TAKE A VACATION

49. Get Our Free PDF Guide to A/B Testing

50. Appsumo reveals its A/B testing secret: only 1 out of 8 tests produce results
MORE RESOURCES…
A Beginner’s Guide To A/B Testing: An Introduction
How to Run AB Tests that Get REAL Results
Statistical Significance Does Not Equal Validity (or Why You Get Imaginary Lifts)
How Big is the Difference Between Being 90% and 99% Certain?
My 7 Rules for A/B Testing That Triple Conversion Rates
Optimizely: What is A/B Testing
How To Determine The Statistical Significance Of An A/B Test – Basic Introduction
How to Determine Statistical Significance When A/B Testing With Divi Leads

51. THUE MADSEN
Marketing Operations Manager, Kissmetrics
@ThueLMadsen
tmadsen@kissmetrics.com
Questions?