Most businesses know they should be A/B testing their website but few actually do it on a regular basis. One of their biggest challenges is knowing what to test. In this talk Dan will share the best practices & lessons learned from working with over 10,000 users who have created more than 30,000 experiments using a dead-simple A/B testing tool called Optimizely. Dan will share key insights from these experiments and leave you inspired to start testing today.

1. Best Practices & Lessons
Learned from 30,000 A/B
and Multivariate Tests
Dan Siroker
Co-Founder & CEO

2. Lesson #1:
Nothing is sacred. Question assumptions.

3. Media
Button

4. Splash page experiment
Variations:
Button: Media:
1. Sign Up 1. Get Involved Image
2. Learn More 2. Family Image
3. Join Us Now 3. Change Image
4. Sign Up Now 4. Barack’s Video
5. Springfield Video
6. Sam’s Video

5. Button: “Sign Up”

6. Button: “Learn More”

7. Button: “Join Us Now”

8. Button: “Sign Up Now”

9. Splash page experiment
Variations:
Button: Media:
1. Sign Up 1. Get Involved Image
2. Learn More 2. Family Image
3. Join Us Now 3. Change Image
4. Sign Up Now 4. Barack’s Video
5. Springfield Video
6. Sam’s Video

10. Media: “Get Involved”

11. Media: “Family”

12. Media: “Change”

13. Media: “Barack’s Video”

14. Media: “Springfield Video”

15. Media: “Sam’s Video”

16. Splash page experiment
Variations:
Button: Media:
1. Sign Up 1. Get Involved Image
2. Learn More 2. Family Image
3. Join Us Now 3. Change Image
4. Sign Up Now 4. Barack’s Video
5. Springfield Video
6. Sam’s Video

17. Splash page experiment
Variations:
Button: Media:
1. Sign Up 1. Get Involved Image
2. Learn More 2. Family Image
3. Join Us Now 3. Change Image
4. Sign Up Now 4. Barack’s Video
5. Springfield Video
6. Sam’s Video

18. Splash page experiment
Variations:
Button: Media:
1. Sign Up 1. Get Involved Image
2. Learn More 2. Family Image
3. Join Us Now 3. Change Image
4. Sign Up Now 4. Barack’s Video
5. Springfield Video
6. Sam’s Video

19. Splash page experiment results

20. Splash page experiment results

21. Splash page experiment results

22. Splash page experiment results
Email
Volunteers Amount Raised
Subscriptions
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000

23. Splash page experiment results
Email
Volunteers Amount Raised
Subscriptions
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000

24. Splash page experiment results
Email
Volunteers Amount Raised
Subscriptions
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000

25. vs.

26. vs.
+600%
engagement

27. Lesson #1:
Nothing is sacred. Question assumptions.

28. Lesson #2:
Less is more. Reduce choices.

29. vs.

30. vs.
+8.4%
signups

31. vs.

32. vs.
+10.97%
$ per visitor

33. vs.

34. vs.
+8%
engagement

35. Lesson #2:
Less is more. Reduce choices.

36. Lesson #3:
Words matter. Focus on your call to action.

37. Donation button experiment
Variations

38. Donation button experiment results
Never Signed Up, Previously
Variations Signed Up Never Donated Donated
0.0% 0.0% 0.0%
+2.3% +27.8% +16.3%
-27.8% N/A N/A
+15.2% -24.6% +11.9%
+8.5% +2.9% +18.4%

39. vs.

40. vs.
+10%
clicks

41. vs.

42. vs.
+14.6%
clicks

43. vs.

44. vs.
+15.75%
$ per visitor

45. Lesson #3:
Words matter. Focus on your call to action.

46. Lesson #4:
Fail fast.

47. vs.

48. vs.
-92.3%
video
clicks

49. Lesson #4:
Fail fast.

50. Lesson #5:
Start today.

52. Happy Customers
Featured In

53. Lesson #5:
Start today.

54. Lessons Learned
1. Nothing is sacred. Question assumptions.
2. Less is more. Reduce choices.
3. Words matter. Focus on your call to action.
4. Fail fast.
5. Start today.

55. Email me: Follow me:
dan@optimizely.com @dsiroker