Mais conteúdo relacionado

Apresentações para você(20)

Similar a Still A/B testing your buttons? You need to think (much) bigger...(20)


Still A/B testing your buttons? You need to think (much) bigger...

  1. Still A/B testing button colors? BIGGER Think
  2. 3 Questions
  3. Who works at a company/agency that is A/B testing? Q1
  4. Who is involved in the A/B testing process themselves? Q2
  5. Who runs 10+ A/B tests a month? Q3
  6. To give you some context… Guido X Jansen Global Business & Technology Evangelist @ Spryker ● Applied Cognitive Psychologist ● Working in E-com & Conversion Optimization (CRO) since 2008 ● Ran 1000+ experiments ● Build several international CRO teams at Ecom merchants ● Won the individual Experimentation Culture Award in 2020 ● Host of the CRO.CAFE Podcast @guido
  7. I’m fascinated by the brain and human behaviour
  8. Why think BIGGER?
  9. I made a lot of mistakes…
  10. I liked the details a bit too much…
  11. I was a control freak
  12. I got too scientific about it
  13. I forgot about the stakeholders
  14. And I feel I’m not the only one…
  15. So let’s talk about Small vs Big…
  16. Sidenote: Framing your Experiments
  17. Everyone is already experimenting (unwittingly)
  18. CRO is about validation and raising the quality
  19. Part 1: Small versus Big
  20. I might get in trouble for this…
  21. Small vs Big: Validation types Classic A/B test Small = single change Classic Split-URL test Big = Multiple changes all at once Classic Multivariate testing (MVT)
  22. Small vs Big: Impact source: .org +4.3% Median Impact from 376 “Small” experiments +15% Median Impact from 43 “Big” experiments Database of 419 Experiments
  23. Small vs Big: Required Sample Size 4.3 162,289 15 13,533 Small test Big test source:
  24. Small vs Big: Test Duration Duration of 3 small A/B tests (4.3% compounded to 13.5%) Duration of 1 big A/B test (15%) 973.734 sample size 27.066 sample size
  25. Running BIG experiments… Decrease Time Increase Impact
  26. Perfect for smaller companies Lower Traffic Higher Upside
  28. Part 2: Big
  29. Big Example: Amazon product page redesign Full case: Version B Rejected
  30. Big Example: Netflix homepage redesign Full case: Version B Rejected
  31. Mixing the odds isn’t great… Mixed Probabilities ✘ + + + - - ? ? Positive Probabilities ✔ + + + +
  32. Increase your probability for success ● Combine small and big tests ○ Run small experiments for Cause-Effect ○ Run big experiments for big impact ● Apply patterns that worked before ○ Leaks & Patterns ○ Extrapolate your own successful patterns ○ Properly document your own experiments
  33. Example: Signup Page UI Changes Full case: Test result details +17% Signups source: .org
  34. Example: 12 landingpage changes Full case: Test result details +42% Leads source: .org
  35. Increase your probability for success For Cause-Effect Run Small Experiments For Biggest Impact Run Big Experiments
  36. Build pattern prioritization feedback loops Prioritize based on and adjust weights based on your own test outcomes. Device Page type Metric Tag … Template available > DM me!
  37. Part 3:
  38. Don’t stop at the frontend!
  39. Business Capabilities & Features to A/B test Cross-sell & Upsell Catalog Management CMS Internationalization Discounts & Promotions Cart Gift Cards Checkout CRM Company Account Back Office Forms Order Management Search & Filter CMS SEO Payment Personalization Mailing & Communication Media Management Navigation Multi- Channel Technology Partner Integrations Wish List Workflow & Process Management Product Management Payments Tax Shipment Ratings & Reviews Shopping List Frontend Frontend Price
  40. How enables you to do this Spryker Cloud Commerce OS is a B2B, B2C and marketplace solution renowned for its ease of use, flexibility, and speed. Fully modular system ● 120+ Features ● 30+ Business Capabilities
  41. Spryker solution partners that can help you run BIGGER experiments 40+ Solution Partners @
  42. 14K+ Subscribers, 250+ Episodes on tap English | Spanish | Dutch | German
  43. So can we NEVER test button colors…?
  44. Thanks for listening! Guido X Jansen Global Business & Technology Evangelist @ Spryker Find me and my team at the booth!
  45. End
  46. Templates
  47. Intro "Stop running minor frontend experiments: experiment with ALL your business capabilities!" or "Still A/B testing your buttons? You need to think (much) bigger..." ● CRO/CXO, Experiments and A/B testing have gained a lot of traction in the last years, hopefully also within your company; ● When optimizing, a lot of commerce teams seem stuck experimenting with minor frontend changes like tekst, images or layout. ● While still valuable to do this, you could benefit much more from the experimentation mindset when applying this to not just your frontend, but to across all aspects of your business ● Guido shows you how using Packaged Business Capabilities can give your commerce team the freedom to do just that.
  48. Title ● …
  49. Real-life A/B test example source: .org