A/B Testing Beyond Headlines and Button Colors -- ideas for tests (particularly for B2B SaaS), common pitfalls in organizations, and how to overcome them.
7. What Did We Learn From That Test?
• ~60% increase in sales going from 15 -> 8 cards
• Synergistic effects can occasionally actually happen.
• You don’t know what you don’t know until you test
for it.
8. Key Points With Testing Quotas
• Grandfather in existing users.
• Set quota levels intelligently.
• Be ready to tolerate small uptick in support load.
• Your product is still too cheap.
12. What We Learned From This Test
• A single A/B test can help your bottom line more
than a product launch.
• Dealing with synergistic effects in testing can
occasionally be maddening.
• The data is never wrong, but sometimes the data
isn’t telling you what you think it is.
13. Key Points With Testing Checkout Flow
• Make absolutely sure you don’t break this!
• Have UX-focused team (product / marketing / etc)
go over every step of checkout funnel with zero
tolerance for bad UX.
• Checkout flows go “stale” at many successful
companies. Schedule a deep dive on them twice a
year, at least.
• Upsells and cross selling are classics for a reason!
17. What We Learned From This Test
• Important to communicate with sales/CS.
• Lead to resource-intensive but hugely successful
lead-scoring efforts.
• Justified very expensive interventions like e.g.
onboarding help for everybody.
18. Key Points With Testing Offline Processes
• Your software/workflow has to close the loop back
to the testing software. This can be tricky.
• Virtually no CRM or A/B testing software accurately
models enterprise decisionmaking process. You
might be forced to optimize for proxy conversions.
• Volumes required for statistical significance are
really punishing, so sometimes we “proxy and
pray.”
21. Key Goals of Onboarding Process
• Get user to a concrete success as soon as possible.
• For more complex sales, lay groundwork for future
success by e.g. inviting in team or getting
permission to follow-up.
• For B2C, especially in social, you live and die on
virality during the onboarding experience.
24. What We Learned From This Test
• Users who start onboarding tour 25% to 100%+
more likely to purchase software
• Users sometimes have visceral reaction to elements
of the onboarding experience.
• Engineers prefer in-app onboarding but it is
expensive as heck to build and does not always
justify itself versus e.g. a Youtube video.
25. Key Points With Testing Offline Processes
• Your software/workflow has to close the loop back
to the testing software. This can be tricky.
• Virtually no CRM or A/B testing software accurately
models enterprise decisionmaking process. You
might be forced to optimize for proxy conversions.
• Volumes required for statistical significance are
really punishing, so sometimes we “proxy and
pray.”
31. What We Learned From This Test
• Just hiding the advanced features increased task
success by 10%, which directly improved sales
• Further jump of 15% when we did the redesign
– Absolutely no increase in sales…
– … but we saw in the funnel that this was solely due to
less people hitting the trial limits …
– … which we were able to profitably exploit with the
quota limit test described earlier
32. Key Points With Testing Core Interactions
• Test the entire funnel. Sometimes task success
doesn’t (unfortunately) help bottom line metrics.
• A good understanding of where people spend their
time/energy is vital – test the 90% interaction, not
the 2%.
37. Gets A Wee Bit Trickier With 2 Sites
Our Marketing Site
A/B
The Web Application
38. Total Freaking Madness
Our Marketing Site
A/B
The Web Application
iPhone app
Offline processes
Android app
Partner site
Content minisite
Offsite checkout
Japanese subsidiary
40. Lessons Learned With This
• Think about strategy for transitioning people between
systems – easier to build into design earlier than later
• Cookies with unique identities are a stopgap, not a
solution. You need to understand the sales process.
• This is a good opportunity to standardize on an A/B
testing tool and workflow… or you’ll hate your life
• This is a “high class problem” which can be deferred
until $50M+ in revenue, but when you solve it, it is
going to cost serious time/money.
42. Single Most Important Lesson:
One Line of Code
@user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
43. Other Lessons Learned
• Organization should learn rather than guess better.
– Best investment ever: organization-wide wiki.
– Next best: weekly “What we tried this week” email.
• Engineers love A/B testing when you describe it like
a puzzle or a video game, with bragging rights.
• Marketers love A/B testing when it’s positioned as a
way to increase their status in the organization
rather than a replacement for their talents.
44. Things To Never Be Without
• CS should be able to force-override a user’s
participation in an A/B test (and, relatedly, “see
what users is seeing” is great if you can get it).
• A/B tests deeper than surface level need unit/etc
tests. Make sure your framework supports it.
• Test switcher UI in development/staging is a huge
win.
• URL query parameter to flip tests a huge win.
45. Thanks for Listening!
• I love talking about this and you cannot possibly
waste my time: @patio11 or
patrick@kalzumeus.com
• Free email mini-course on CRO in software
companies: http://bit.ly/patrick-optimizely
• Liked this presentation? I deliver something like
this twice a month, free, to my email list.
http://training.kalzumeus.com to sign up