1. The Art of A/B Testing
@mike_greenfield
numeratechoir.com
500 Startups, 2013-05-08
2. What’s an A/B Test?
Definition: An A/B Test is a means by which
a product’s users are randomly given
one of two or more experiences.
Usage: Companies use A/B Tests to
discover which experience is most
effective.
3. A/B Testing Overview
Why You Should A/B Test
What You Should A/B Test
When You Should A/B Test
How You Should A/B Test
What I’ve Learned from A/B Testing
5. Humans’ quantitative intuition is poor
Beliefs:
1) “I have great product intuition”
2) “This is business, not science”
3) “It leads to a local maximum”
Reality:
At Internet scale, testing and measuring
properly has a huge return on investment.
6. “I have great product intuition”
• Improved signup flow: often a disaster
• Sharing on Facebook: can be good or
bad
• Home page changes: usually a mixed
bag
…but product changes != progress
7. “This is business, not science”
• Requires oversight and sleuthing
• Only works for big improvements
• Requires other factors to stay the same
…but change-then-measure is flawed
8. Test Test Type Goal
Change Button
Color
Optimization Increased clickthrough
Old Site Design
vs. New Site
Design
Holistic (aka don’t
shoot yourself in
the foot)
Make an informed
decision on old vs. new
“It leads to a local maximum”
…so test holistically
16. Validate your new design*
* if you have scale
Scale What to Do Reasoning
Early stage Just change it. Nothing to lose, no data to
test.
Something to lose A/B test it. The existing product is
probably more effective
than you think.
Understand the
consequences of an
“upgrade.”
18. Test only if you can get to
statistical significance
(Google: “split test calculator”)
Big: 200,000
emails with a 3%
CTR option and a
4% CTR option
-------------------------
6000 vs. 8000
Small: 200 emails
with a 3% CTR
option and a 4%
CTR option
--------------------------
6 vs. 8
19. Test only if you can get to business
significance
• Only things that can cumulatively
have a meaningful impact on your
business
• For emails, a small list means small
improvements: don’t test
• For virality, small changes matter if and
only if you’re close to K=1
20. Rule of thumb: test every user-
facing change that will be seen
by 10,000-100,000 people
22. Okay Choice: Use Commercial
Tools
• MixPanel
• Unbounce
• Optimizely
• Google Analytics
23. Best Choice: Build Your Own
Framework
• Yep, it’s work with no immediate
payoff
• Your mom won’t care
• Your users won’t care
But…
• There are simple ways to get started
• It gives you tons of flexibility
24. Why Build Your Own
• Incorporate tests in many places
(page ordering, new designs, email
content, email strategy, mobile)
• Look at results holistically
• Go back and see how any test
influences anything, not just the stats
you’re tracking
25. Code
It needs to be super simple to create a test
{ab_test_if test=“signup_reason”
option=“awesome” user=$viewer}
because it’s awesome!
{/ab_test_if}
{ab_test_if test=“signup_reason” option=“free”
user=$viewer}
because it’s free!
{/ab_test_if}
26. Data Structure
AB_TESTS (id, name, time_created)
AB_TEST_OPTIONS
(id, ab_test_id, weight, name)
USER_AB_TEST_OPTIONS
(id, user_id/visitor_id, ab_test_option_id, t
ime_created)
28. Scaling
• An A/B testing system can yield a lot of
DB writes
• Reporting means many long-running
SQL queries
• Need to batch several aspects
29. Run Some A/A Tests
• Test two versions of the same thing
• If results are wildly off, something’s
wrong with the testing system
• Deciding too early is a major issue: it’s
usually best to be conservative before
choosing a winner
32. Focus on 1 item in emails
• Clear subject focused on that item (Why
the Giants will win the World Series)
• Body of text focused on that item
(peripheral content is okay on the
periphery)
• Clear, big clickthrough action in the
email body (See why the Giants will win)
• Require clickthrough to get the full story
33. In signup, minimize distraction
• Provide context/messaging of what
the product is, but don’t make it
clickable
• Clear “next” or “continue” steps to
guide user through the process
• Remove unnecessary navigation
34. Highlight friends, not your product
• Most effective: your friends are doing
something; you should join them
• Unless you’re Apple, no one cares
about your new feature or new design
• People probably don’t care about
your fancy new social network
• “Join my circle because I trust you”
beats “check out this great product”
37. A/B testing = good culture
• Data trump opinions
• Iterate quickly but intelligently
• Everyone gets better at predicting
product success
38. Test changes if they’re likely to
have both statistical and business
significance
39. Validate the Big Stuff
A/B Test Holistically; testing
isn’t a substitute for product
vision.
Optimize the Small Stuff
The details matter more than you
think.