3. We’ll cover...1 State of A/B Testing2 Q&A3 Demo 4. Let’s start with the data! 5. WHO AREWe? 6. Who are we? 176 in total 60 45 30 15 0 Ecommerce SaaS Agency/Cons Publishing Mobile Other 7. What’s the size of our teams? 90 67.5 45 22.5 0 1-5 6-10 11-25 26-60 51-200 200+ 8. What testing tools does everyone use? 90 67.5 45 22.5 0 Visual Website GA Content Optimizely None Other Optimizer Experiments 9. How many A/B tests did we run last month? 150112.5 75 37.5 0 0 1-4 5-9 10-19 20+ 10. It’s easy to get ahead of our competitors 11. How do we generate ideas for testing? 60 45 30 15 0 Test Experience Best Practices Research Needs Randomly 12. Step 1: Get Qualitative 13. Step 2: Make a prediction 14. Step 3: Confirm with an A/B test 15. How KISSmetrics generates testing ideas 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test 16. OURResults 17. Do we regularly increase conversions? 80 60 40 20 0 Always Sometimes Rarely Never 18. The revenue we gain from A/B testing 70 52.5 35 17.5 0 $1001- $10,001- Over None $1-1,000 No Idea 10,000 100,000 $100,000 19. 2 ways to make money with A/B testing 20. Option 1: Understand your customers first 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test 21. You’ll find the BIG wins much faster 22. Option 2: Get relentless 23. How pros like Facebook and Twitter do it 1 Build an entire growth team 2 Back-to-back testing 3 Look for 100s of small improvements 24. How do we A/B test with small amounts of data? 25. Do tests with customer interviews 1 Talk to people, give different options 2 Look for trends 3 Build and see if actions match 26. Do we track results over the long term? 130 97.5 65 32.5 0 Yes No 27. The pitfalls of not tracking long term results 28. Variation B performs the best in this funnel 29. Variation A gives the most revenue 30. LET’S TRACK THE RETURN ON A/B Testing 31. First, make sure you can track lifetime value You’ll need customer analytics that can track multiple purchases from the same person. 32. Where do we find customer analytics? 33. Second, connect A/B tools to customer analytics OR 34. What about Google Analytics? Connects tests to goals, not ecommerce transactions.

1. SUMMER 2012 STATE OF A/B TESTING
Lars Lofgren
Marketing Analyst - August 2012 info@kissmetrics.com - Confidential - Do not distribute

2. @larslofgren #KISSwebinar

3. We’ll cover...
1 State of A/B Testing
2 Q&A
3 Demo

4. Let’s start with the data!

5. WHO ARE
We?

6. Who are we? 176 in total
60
45
30
15
0
Ecommerce SaaS Agency/Cons Publishing Mobile Other

7. What’s the size of our teams?
90
67.5
45
22.5
0
1-5 6-10 11-25 26-60 51-200 200+

8. What testing tools does everyone use?
90
67.5
45
22.5
0
Visual Website GA Content
Optimizely None Other
Optimizer Experiments

9. How many A/B tests did we run last month?
150
112.5
75
37.5
0
0 1-4 5-9 10-19 20+

10. It’s easy to get ahead of our competitors

11. How do we generate ideas for testing?
60
45
30
15
0
Test
Experience Best Practices Research Needs
Randomly

12. Step 1: Get Qualitative

13. Step 2: Make a prediction

14. Step 3: Confirm with an A/B test

15. How KISSmetrics generates testing ideas
1 Get qualitative insights
2 Predict how to improve
3 Confirm prediction with an A/B test

16. OUR
Results

17. Do we regularly increase conversions?
80
60
40
20
0
Always Sometimes Rarely Never

18. The revenue we gain from A/B testing
70
52.5
35
17.5
0
$1001- $10,001- Over
None $1-1,000 No Idea
10,000 100,000 $100,000

19. 2 ways to make money with A/B testing

20. Option 1: Understand your customers first
1 Get qualitative insights
2 Predict how to improve
3 Confirm prediction with an A/B test

21. You’ll find the BIG wins much faster

22. Option 2: Get relentless

23. How pros like Facebook and Twitter do it
1 Build an entire growth team
2 Back-to-back testing
3 Look for 100s of small improvements

24. How do we A/B test with small amounts of data?

25. Do tests with customer interviews
1 Talk to people, give different options
2 Look for trends
3 Build and see if actions match

26. Do we track results over the long term?
130
97.5
65
32.5
0
Yes No

27. The pitfalls of not tracking long term results

28. Variation B performs the best in this funnel

29. Variation A gives the most revenue

30. LET’S TRACK THE RETURN ON
A/B Testing

31. First, make sure you can track lifetime value
You’ll need customer analytics that
can track multiple purchases from the
same person.

32. Where do we find customer analytics?

33. Second, connect A/B tools to customer analytics
OR

34. What about Google Analytics?
Connects tests to goals, not
ecommerce transactions.

35. Q&A Time!
Lars Lofgren
llofgren@kissmetrics.com
@larslofgren