What, why and how to A/B test with AI

Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
Patrik Matell
CONVERSION SPECIALIST
CONVERSIONISTA
What, Why and How
to Test with AI
WEBINAR
Jonathan Epstein
CHIEF MARKETING OFFICER
SENTIENT TECHNOLOGIES

•  What is artificial intelligence (AI)? Can it make better decisions than me?
•  Do we need another type of testing? Why?
•  How can AI be used for A/B and multivariate testing? Can I trust it?
•  When should I use AI Testing or not? What does it require?
•  Is it possible to get faster results when “AI testing”?
What kind of results?
Agenda

Copyright © 2016 Sentient Technologies LLC. All rights reserved.
The CRO Wheel
- How it’s done today
The reality AI will start at

Experimentation is
the phase where
the optimization
wheel hits the
reality.
“Where the rubber hits the road”

Two things will happen - both important
Movement forward
Gravel spray

The A/B tests two results
Movement forward
Gravel spray
The A/B test will show a positive result, and when you
permanently implement the new version you get instant effect on
your business.
Data from the A/B test give you new, and often unexpected, user
behaviour insights.
You can use those insights to create new experiments, and let the
wheel roll forward.

The Optimization
Wheel
Data Hypothesis
Experiments
Plan &
Prioritize
Research &
Analysis
Behaviour Analysis

What is Artificial Intelligence (AI)?
Can it Really Make Better
Decisions than Me?

•  A field of computer science focused on building “intelligent” machines
•  An ideal “intelligent” machine
•  Perceives its environment
•  Takes actions that maximize its success against some goal
•  Learns from the results of prior actions and adapts
•  AI is all around you already!
•  Siri, Waze, Airline pricing
•  Is it smarter than me?
•  Depends on what you mean by smart. But it is a lot faster.
What is Artificial Intelligence?

AI comes in many shapes and forms…
…each of which excels at certain problems.
DEEP LEARNING EVOLUTIONARY
ALGORITHMS
NEURO-
EVOLUTION
BAYESIAN

Do we need another
type of testing?
Why?

Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.
“
“
Agility and experimentation make up the
fundamental oxygen for corporate
success going forward.
If you’re not trying 100 ideas or even
1,000 ideas, you’ll get stuck.
Peter Diamandis
FOUNDER OF THE X PRIZE

But…
Only 1 in 7 A/B
tests is a success.
– Visual Website Optimizer

And…
Traffic required to get from 5.0% to 5.25% using A/B tests:
136,000 per sample
- Optimizely calculator
Copyright © 2016 Sentient Technologies LLC. All rights reserved.Copyright © 2016 Sentient Technologies LLC. All rights reserved.

How can AI be used for A/B
and multivariate testing?

Evolution: A Proven Efficient Design System

FITNESS
What makes one candidate
(website, trading strategy,
animal) more viable than
another?
COMBINATION
By combining successful
candidates, you can achieve
greater fitness, and also
prune out bad attributes.
MUTATION
An element of randomness
helps discover new
successful combinations and
ensures you find the best
solution—not just a good
one.
Evolutionary Algorithms:
AI Based on Natural Selection

After the operator inputs the various test hypotheses, Ascend tests a
first generation – each individual change is measured against the
goal…
Each box represents a design
Each letter represents a hypothesis
Sentient Ascend Applies AI to Testing

How Sentient Ascend Works
+ 0 - + - + 0 + +
…and determines which ones have good “genetic material.”

The system then combines the successful candidates
into a new generation to test…

…and again selects the winners in that population.

The breeding continues, with mutations to ensure we
explore all corners.

Within 4 generations you see massive impact; by 8 to 10
generations Ascend converges on the winning design

What is Massively Multivariate Testing?
•  Test dozens of site
improvements at once
•  Determine the best
combination of those
changes to achieve your
growth goals
•  Evolutionary algorithms
can determine the best
of 1 million designs while
only testing 500-600 of
them

Easy Multipage Funnel Optimization
•  Optimize your website funnels—whether they are a single page, or
multiple pages
•  Test global site elements, too
Home PDP Cart

Always-On Continuous Optimization
•  Enables true always-on
site optimization
•  Build on top of your
existing tests and add new
ideas into the mix
•  Whether you add new
ideas or not, Ascend
always tests in the
background and adapts
your site presentation to
provide the best results if
your audience changes for
any reason

When should I use AI Testing
or not? What does it require?

1.  Do I have enough traffic?
2.  Can I get the ROI I need?
•  Base-level Ascend subscription =
USD 3,000 per month
•  0 to 300,000 unique users/month to
your properties; no limit on testing
•  Consider cost/design tested, cost/
hypothesis tested, or cost/increase
in KPIs
Is my site large enough to use this type of testing?

•  How long is a test cycle?
•  What sort of feedback does the product provide?
•  What sorts of goals can I optimize for?
•  Will I understand what hypotheses worked, as well as
what combinations worked?
•  How would AI testing change my workflow?
How long are AI tests?
How are results shown?

•  Does Ascend’s AI testing replace
A/B testing or would I use both?
•  Will this replace my job or make it
better? Should I be concerned?
What does this mean to me & my A/B tests?

•  What sort of information/analytics
should I be gathering to set up my
hypotheses for AI testing?
•  What sort of ideas can I test?
•  What sort of ideas should I test?
•  Can I come up with enough ideas to
use AI testing?
Should I think about
hypotheses differently?

The CRO Wheel
- How is it affected?
AI optimization’s effect on your
workflow

The
Prioritization
Loop
Feasibility
Analysis
P.I.E
Prioritization
I.A.R Formula
Experiment
Roadmap

Design
Develop
Deploy &
Monitor
Follow-up
Experiment
Execution

Thank you!
LEARN MORE
conversionista.se
PATRIK@CONVERSIONISTA.SE
sentient.ai/ascend
JONATHAN.EPSTEIN@SENTIENT.AI

What, why and how to A/B test with AI

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (15)

Semelhante a What, why and how to A/B test with AI

Semelhante a What, why and how to A/B test with AI (20)

Mais de Conversionista

Mais de Conversionista (20)

Último

Último (20)

What, why and how to A/B test with AI