Hiten Shah, President and Co-Founder, KISSmetrics and Crazy Egg The majority of A/B tests that you run end up failing. Wouldn't it be great if you could increase your chance of success? In this session Hiten Shah, President and Co-Founder of KISSmetrics and Crazy Egg provides a framework and examples of how to increase your success rate by using both qualitative and quantitative tactics. Learn how to design great experiments by understanding more about your visitors, users and customers.

1. How
to
Increase
Your
Tes0ng
Success
by
Combining
Qualita0ve
and
Qualita0ve
Data
Hiten
Shah,
KISSmetrics

2. Hiten
Shah
Co-­‐Founder
KISSmetrics
@hnhshah

3. Hiten
Shah
Co-­‐Founder
KISSmetrics
@hnhshah

4. There’s
a
big
problem
with
A/B
tes5ng!

5. You’ve
tested
the
color
of
your
bu?ons

6. You’ve
watched
your
compe5tors

7. You’ve
tried
tes5ng
everyone’s
opinion

8. You’ve
hit
the
limits
of
your
current
design

9. You’ve
run
experiments
inspired
by
blog
posts

10. But…

11. Only
1
out
of
5
tests
win.
4
out
of
5
tests
cost
you
money.
$

12. There
are
no
magic
bullets
or
pixie
dust

13. You
need
to
double
your
win
rate.

14. Tools

15. Usability
Tes5ng
hEp://usertes0ng.com

16. Mouse
Tracking
hEp://crazyegg.com

17. On-­‐Page
Surveying
hEp://qualaroo.com

18. Live
Chat
hEp://olark.com

19. Long
Form
Surveying
hEp://surveymonkey.com

20. 5
tac5cal
5ps
to
double
your
win
rate

21. 1.
Find
out
why
people
don’t
convert
2.
Use
the
exact
words
customers
do
3.
Start
with
a
hypothesis
then
priori5ze
4.
Create
experiments
that
are
small
5.
Con5nuously
experiment

22. Discover
what
informa5on
is
missing

23. Ask
people
when
they
try
to
leave

24. Email
a
survey

25. Ask
people
why
they
cancelled

26. Get
feedback
right
when
people
cancel

27. 1.
Find
out
why
people
don’t
convert
2.
Use
the
exact
words
customers
do
3.
Start
with
a
hypothesis
then
priori5ze
4.
Create
experiments
that
are
small
5.
Con5nuously
experiment

28. Learn
what
frustrates
them

29. Find
out
how
they
describe
your
product

30. Test
very
specific
call
to
ac5ons
(CTAs)

31. 1.
Find
out
why
people
don’t
convert
2.
Use
the
exact
words
customers
do
3.
Start
with
a
hypothesis
then
priori5ze
4.
Create
experiments
that
are
small
5.
Con5nuously
experiment

32. “A solid test hypothesis is an informed solution to
a real problem – not an arbitrary guess. The more
research and data you have to base your
hypothesis on, the be er it will be.”
!
— Michael Aagaard, h p://kiss.ly/abthypo

33. Answer
key
ques5ons
for
each
experiment
1.
Which
top
metric
are
we
trying
to
improve?
2.
What
can
you
anchor
your
metric
against?
3.
What
is
your
hypothesis
about
why
this
metric
is
at
x%?
4.
What
do
we
expect
from
this
experiment?
5.
What
was
the
result
and
what
did
we
learn
from
it?
hEp://kiss.ly/6qsab

34. Priori5ze
your
experiments
hEp://kiss.ly/abpie

35. 1.
Find
out
why
people
don’t
convert
2.
Use
the
exact
words
customers
do
3.
Start
with
a
hypothesis
then
priori5ze
4.
Create
experiments
that
are
small
5.
Con5nuously
experiment

36. Waste
less
code
!
Small
experiments
help
you
focus
on
learning
instead
of
wri0ng
code.

37. Know
what
works
!
Small
experiments
help
you
isolate
and
understand
what
impacts
conversions.

38. Validate
your
ideas
!
Small
experiments
help
you
validate
assump0ons
about
your
product.

39. 1.
Find
out
why
people
don’t
convert
2.
Use
the
exact
words
customers
do
3.
Start
with
a
hypothesis
then
priori5ze
4.
Create
experiments
that
are
small
5.
Con5nuously
experiment

40. Con5nuous
Experimenta5on
Small,
measurable
changes.
Keeps
us
honest.
Prevents
us
from
breaking
things.
hEp://kiss.ly/etsycont
Dan
McKinley
@mcfunley
@etsy

41. One
more
thing...
Learn
more…
hEp://grow.kissmetrics.com/op0con
NEW!
KISSmetrics
Customizable
A/B
Tes5ng
Report

42. THANK
YOU
Hiten
Shah
@hnshah

43. Hiten
Shah
Co-­‐Founder
KISSmetrics
@hnhshah