Still A/B testing button colors?
BIGGER
Think

3 Questions

Who works at a
company/agency
that is A/B testing?
Q1

Who is involved in
the A/B testing
process themselves?
Q2

Who runs 10+ A/B
tests a month?
Q3

To give you some context…
Guido X Jansen
Global Business & Technology Evangelist @ Spryker
● Applied Cognitive Psychologist
● Working in E-com & Conversion Optimization (CRO) since 2008
● Ran 1000+ experiments
● Build several international CRO teams at Ecom merchants
● Won the individual Experimentation Culture Award in 2020
● Host of the CRO.CAFE Podcast
gui.do/linkedin
@guido

I’m fascinated by the brain
and human behaviour

Why think BIGGER?

I made a lot of mistakes…

I liked the details a bit too much…

I was a control freak

I got too scientific about it

I forgot about the stakeholders

And I feel I’m not the only one…

So let’s talk about Small vs Big…

Sidenote: Framing your Experiments

Everyone is already experimenting (unwittingly)

CRO is about validation and raising the quality

Part 1: Small versus Big

I might get in trouble for this…

Small vs Big: Validation types
Classic A/B test
Small = single change
Classic Split-URL test
Big = Multiple changes all at once
Classic Multivariate testing
(MVT)

Small vs Big: Impact
source:
.org
+4.3%
Median Impact from
376 “Small” experiments
+15%
Median Impact from
43 “Big” experiments
Database of 419 Experiments

Small vs Big: Required Sample Size
4.3
162,289
15
13,533
Small test
Big test
source:
EvanMiller.org

Small vs Big: Test Duration
Duration of 3 small A/B tests (4.3% compounded to 13.5%)
Duration of 1 big A/B test (15%)
973.734 sample size
27.066 sample size

Running BIG experiments…
Decrease
Time
Increase
Impact

Perfect for smaller companies
Lower
Traffic
Higher
Upside

TIME IS THE MOST
VALUABLE
RESOURCE
CLEAR FOCUS IS
MORE NEEDED
THAN EVER

Part 2: Big

Big Example: Amazon product page redesign
Full case: https://goodui.org/leaks/amazons-beautifully-designed-and-failed-three-vs-two-column-layout-experiment/
Version B
Rejected

Big Example: Netflix homepage redesign
Full case: https://goodui.org/leaks/netflix-rejected-all-these-tested-homepage-variations-perhaps-this-one-will-help/
Version B
Rejected

Mixing the odds isn’t great…
Mixed
Probabilities
✘
+
+
+ -
-
?
?
Positive
Probabilities
✔
+
+
+ +

Increase your probability for success
● Combine small and big tests
○ Run small experiments for Cause-Effect
○ Run big experiments for big impact
● Apply patterns that worked before
○ GoodUI.org Leaks & Patterns
○ Extrapolate your own successful patterns
○ Properly document your own experiments

Example: Thomasnet.com Signup Page UI Changes
Full case: https://goodui.org/blog/6-signup-form-ui-changes-with-17-more-signups-for-thomasnet/
Test result details
+17%
Signups
source:
.org

Example: KensingtonTours.com 12 landingpage changes
Full case: https://goodui.org/blog/12-landing-page-patterns-for-42-more-leads-on-kensingtontours.com/
Test result details
+42%
Leads
source:
.org

Increase your probability for success
For Cause-Effect
Run Small Experiments
For Biggest Impact
Run Big Experiments

Build pattern prioritization feedback loops
Prioritize based on and adjust weights based on your own test outcomes.
Device
Page type
Metric
Tag
…
Template available > DM me!

Part 3:

Don’t stop at the frontend!

Business Capabilities & Features to A/B test
Cross-sell &
Upsell
Catalog
Management
CMS
Internationalization
Discounts &
Promotions
Cart
Gift Cards
Checkout CRM
Company
Account
Back Office
Forms
Order Management
Search & Filter
CMS
SEO
Payment
Personalization
Mailing &
Communication
Media
Management
Navigation
Multi-
Channel
Technology Partner Integrations Wish List
Workflow & Process
Management
Product
Management
Payments
Tax
Shipment Ratings & Reviews
Shopping List
Frontend
Frontend
Price

How enables you to do this
Spryker Cloud Commerce OS is a B2B,
B2C and marketplace solution renowned
for its ease of use, flexibility, and speed.
Fully modular system
● 120+ Features
● 30+ Business Capabilities
Spryker.com

Spryker solution partners that can help you run BIGGER experiments
40+ Solution
Partners @
Spryker.com

14K+ Subscribers, 250+ Episodes on tap
English | Spanish | Dutch | German

So can we NEVER test button colors…?

Thanks for listening!
Guido X Jansen
Global Business & Technology Evangelist @ Spryker
gui.do/linkedin
Find me and my team at the booth!

End

Templates

Intro
"Stop running minor frontend experiments: experiment with ALL your business capabilities!"
or "Still A/B testing your buttons? You need to think (much) bigger..."
● CRO/CXO, Experiments and A/B testing have gained a lot of traction in the last years, hopefully also within your company;
● When optimizing, a lot of commerce teams seem stuck experimenting with minor frontend changes like tekst, images or layout.
● While still valuable to do this, you could benefit much more from the experimentation mindset when applying this to not just your
frontend, but to across all aspects of your business
● Guido shows you how using Packaged Business Capabilities can give your commerce team the freedom to do just that.

Title
● …

Real-life A/B test example
source:
.org