How to Balance
Innovation and Optimization
in your CRO program
+

1. All attendees will be placed on mute during
the webinar
1. In case of any queries/technical issues,
please use the chat window, and we will be
happy to assist.
1. If you have any questions, send them via
chat, and we will take them up after the
webinar.
1. In case your question isn't covered, we will
reach out to you via email with the answer.
Housekeeping reminders

Over the last 10 years, I've helped companies like
Abercrombie & Fitch, Under Armour, Intercontinental
Hotel Group, and Fidelity make incremental changes
to their ecommerce businesses to increase sales.
As the founder of Sam Baker Consulting, my team and
I use data-driven, digital marketing decisions to put
together effective, practical revenue-driving
strategies. We guide globally-known companies as
they increase their website revenue by building
and growing their testing & optimization programs.
Hi, I’m Sam!

1. How to strike a balance between innovation
and optimization
1. Why having a consistent testing roadmap is
the key to optimization
1. How upfront user research can support the
innovation process
1. Questions to ask when testing new
innovative ideas
At the end of this webinar you’ll know…

Innovation is among the top 5 priorities for 96% of
companies.

However.. 57% of companies say that it’s a struggle to
foster an internal culture of experimentation and
innovation.

ORIGINAL NEW

The CRO Team

1. Every experiment must only test a
single variable
1. Must have enough site traffic to see
significant changes within a reasonable
amount of time
1. Setting up the test must be technically
possible.
Three things you learn early
as a CRO

ORIGINAL NEW

Innovative ideas tend to be:
● Multi-variable
● Difficult to A/B Test
● Often lack a true “control”
experience
The problem with “test
everything”

82% of organizations run innovation in the same way
they would go about achieving any incremental
performance gain.

How do we make sure this guy’s innovative idea has a
sound analytics strategy to support it?

Commit to a
testing roadmap

Testing allows us to get to know our customers on a deeper level
through data.
A consistent testing roadmap allows us to consistently gain
insights that will help us and our business stakeholders create
new innovations based on data.
Innovation should start with insights from the CRO
program.

1. Test, test, test!
2. Build a results repository that can be
referenced when new innovative
ideas are needed.
3. Commit to finding at least one
customer insight from each test.
How to use CRO to inspire innovation.

54% of innovating organizations have trouble bridging
the gap between innovation strategy and the larger
business strategy.

● Conversion Drop lead to a panic
● We need to be innovative, NOW!
C-Suite push for Innovation

1. Executive pushes for something
“innovative”
2. Other projects go on hold
3. Competitive research
4. Analysis is rushed
5. Launch with something that may or
may not work.
Historic approach to innovation

A new process to limit the panic
Always
have an
innovation
ready
01
02
03
04

➔ Results repository filled with
meaningful, actionable insights
➔ A roadmap that already pushes
innovative ideas forward
➔ A deep organizational
understanding of the core customer
What this looks like now…

Conduct up-front
user research

Pre-test data can be in the form of:
● Surveys
● Focus Groups
● User Testing
● Observation
Require pre-test data for innovative ideas.

User research is used to:
● Try out ideas before committing time and
money to a big project
● Ask questions that will uncover insights
into ways to make the new product better
● Help your optimization analyst know what
to look for once an innovation is live
Don’t skip this step because it’s not “statistically valid”
data.

● The majority of revenue came from a 1-
month window at the end of the year
● Difficult to test due to low traffic and
low conversion rate throughout the
first 3 quarters of the year.
● The team wanted to be innovative, but
felt bottlenecked by short timeframe
and inability to pivot quickly
Working toward a yearly deadline

● The team knew it was even more
important to get things right.
● Used the first half of the year to
analyze and research.
Working toward a yearly deadline

Working toward a yearly deadline
JAN FEB MAR APR MAY JUN
Review
results
from
previous
period
User research
Concepts
developed
User
testing
Dev work
begins

● This process allowed the team to prove
consistent revenue increases after
every open enrollment period.
Consistent CVR increases YOY

Break new innovations
into several tests

A few questions to ask:
● Are there any pieces of this innovation that
don’t need to be tested?
● Are there any functional changes that can be
tested on their own?
● Are there any pieces of the innovation that are
particularly risky? Can those pieces be tested
on their own?
● Can a multivariate test be used to lower risk?
Sometimes it’s possible to test into an innovation.

● The product team responsible for the
product detail page wanted to
completely overhaul the PDP
● The CRO team saw this as a high-risk
change that would be difficult to test
Redesigning an outdated product page

Redesigning an outdated product page

Redesigning an outdated product page
● New image functionality
● Afterpay
● Icons in the buy grid
● Sticky Add to Cart Button

How to test efficiently
● Broke new features into individual
tests
● Lower confidence threshold to 80%
● Ran 2-week directional tests for each
new element
Most importantly… we tested everything
together once we were confident that
each element would work on its own.

We’ve covered 3 strategies that will help you balance
innovation and optimization:
1. Commit to a testing roadmap
2. Conduct up-front user research
3. Break new innovations into several tests

Following these three strategies will allow the CRO
and UX teams:
➔ Develop innovative strategies based on customer
data and insights
➔ Launch innovative ideas that are user tested so
that the team will have confidence in the idea
before it goes live
➔ Build a test roadmap that allows the team to
break the innovation down into several small
tests

1. How to strike a balance between innovation
and optimization
1. Why having a consistent testing roadmap is
the key to optimization
1. How upfront user research can support the
innovation process
1. Questions to ask when testing new
innovative ideas
Now you know…

Questions?