The best A/B test idea
you haven’t thought of PHIL SHARP

@NEILPATEL
@Kissmetrics
#KissWebinar
@ThueLMadsen

Thue is the Kissmetrics Webinar
Wizard and Marketing Ops Manager. Before joining forces with Kissmetrics, he was a Lyft driver in SF, which is also how he ended up as a KISSmetrics marketer. Whenever Thue is not trying to automate everything around him, you can find him hiking in the Sierras. THUE MADSEN Marketing Operations Manager, Kissmetrics @ThueLMadsen Phil leads the UserTesting marketing team in helping companies discover the importance of user experience and usability testing. PHIL SHARP VP of Marketing, UserTesting @iamphilsharp

@NEILPATEL
@Kissmetrics
#KissWebinar
@IAmPhilSharp

WATCH WEBINAR RECORDING NOW

The problem

Not so much Winner! Not
so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much

The real problem

One month Winner! Two months
Three months Four months Five months Six months Seven months Eight months Nine months

1 winner per year

QUOTE FROM ELON MUSK BIOGRAPHY
“ He would place this urgency that he expected the revenue in ten years to be ten million dollars a day and that every day we were slower to achieve our goals was a day of missing out on that money.

$10,000,000 days

We’re missing out on this

But…why?

We’re smart

And hard working

And nice :)

And know our products

Why?

TOO MANY IDEAS!

170 ideas in 3 minutes

Try new CTA Try new
CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price

Try new CTA Try new
CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price

Try new CTA Try new
CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price

Er.

We need a process

Process = Sleepy time

Process = More revenue

Process = More
experiments

The core of our process…

Find out what’s stopping folks
from converting…

…fix it.

Thanks

I’ll be here all night

Three Steps

Step 1: Take these ideas…

Try new CTA Try new
CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price

…and throw them away.

Step 2: Listen to your
customers and prospects

Find out what’s stopping folks
from converting…

…fix it.

Step 2: Listen to your
customers and prospects

Try new CTA Try new
CTA location Test multiple CTAs Test different button colors Add an “About” section Try a different headline Add an image to the homepage Shorten my form Auto play the video Try modals for forms Test a free trial Try annual billing Try a high anchor price These ideas are good. They just might not be for you.

Gut Your gut is good.
It just knows too much.

Curse of knowledge
HTTPS://HBR.ORG/2006/12/THE-CURSE-OF-KNOWLEDGE

How to listen

QUALAROO

QUALAROO - USERTESTING EXAMPLE

QUALAROO Is anything preventing you
from signing up?

QUALAROO “Trying to work out
a password that meets your criteria — it’s taking me a while.”

QUALAROO “A 10 character minimum
for a password! Seriously? That’s insane.”

QUALAROO “10 characters for the
password? That’s way too many. This isn’t a bank site or something…”

CHAT TRANSCRIPTS

SALES PEOPLE AND SUPPORT

USER TESTING

Send 3 users through your
funnel

USER TESTING

USER TESTING

Send 3 users to your
competition

Send 3 users to a
HUGE company in your space

PEEK

Step 3: Use a framework
to make decisions

Test summary Test objective Hypothesis
Opportunity size Time to test Likely scenarios Next steps What worked? What didn’t work? HTTP://BIT.LY/ABTEST-FRAMEWORK

Test Objective In 1-2 sentences,
why are we running this test?

Hypothesis If X, then Y
because Z

Opportunity Size If this test
works, what’s the total impact we could expect to see over a year?

Time to Test How long
will it take to run this test?

Likely Scenarios Write out your
assumptions and 3 of the most likely scenarios.

Next Steps Now that you
know the results, what will you do? If the campaign worked, how can you expand it? What can you test that is similar?

What worked? What didn’t? What
did you learn?

The problem

Not so much Winner! Not
so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much

The problem opportunity

Not so much Winner! Not
so much Not so much Not so much Not so much Not so much Not so much Not so much Not so much

Not so much Winner! Not
so much Not so much Not so much Winner! Not so much Not so much Not so much Not so much

Not so much Not so
much Not so much Winner! Not so much Not so much Winner! Not so much Not so much Winner!

LOG IN WITH GOOGLE Start
Your Free KISSmetrics Trial

PHIL SHARP VP of Marketing,
UserTesting @iamphilsharp phil@usertesting.com THUE MADSEN Marketing Operations Manager, Kissmetrics @ThueLMadsen tmadsen@kissmetrics.com Questions?