For more ecommerce webinars visit us at http://www.elasticpath.com/webinars/archive

1. Taking Your Site Performance to the
Next Level with Optimization Testing
Presented by Linda Bustos,
Director of Ecommerce Research, Elastic Path Software
June 25, 2010
Listen to audio using your computer’s microphone & speakers (VoIP) or dial in:
United States: + 1 323 417 4600
United Kingdom: + 44 (0) 121 368 0265
Access Code: 646-797-773
Elastic Path™

2. Introducing
Elastic Path Software
• Enterprise ecommerce solutions for software, media,
telco, online services, and digital reseller markets
• Trusted ecommerce experts
• #1 ecommerce blog www.getelastic.com
• For more on-demand ecommerce resources:
www.elasticpath.com/resources
Elastic Path™

3. Agenda
• What is optimization testing?
– A/B split testing vs. MVT (multivariate testing)
• What to optimize
• Designing your test
• Testing “gotchas”
• Understanding tools (free vs. paid)
Elastic Path™

4. What is optimization testing?
Elastic Path™

5. What is optimization testing?
• Not user testing
– Qualitative data
– Goal is to identify usability issues
Elastic Path™

6. What is optimization testing?
• A/B split and multivariate testing (MVT)
– Quantitative data
– Goal is to improve the performance of a KPI (key
performance indicator, metric of success)
– Hypothesis testing (user testing may help form
your hypothesis)
Elastic Path™

7. What is optimization testing?
• Why test?
– The only way to truly know what works and what
doesn’t for your industry, customers and products
• No “gut feel” or opinions
• No “best practices”
• No “blind redesign”
• No missed opportunities
Elastic Path™

8. A/B
A B
Elastic Path™

9. A/B
A B C D
Elastic Path™

10. A/B
A B
Elastic Path™

11. MVT
A B C D
V1a V1b V2a V2b
Elastic Path™

12. A/B or MVT?
• A/B • MVT
– Broad decisions – Tweaking a “winner”
– Just getting feet wet – Large traffic/conversions
– Limited – Need to test >10 recipes
traffic/conversions at one time
– Tool restrictions • More accurate
• Faster
Elastic Path™

13. What to optimize
Elastic Path™

14. What to optimize
• Processes/workflows
Elastic Path™

15. What to optimize
• Processes/workflows
• “Landing pages”
– Any individual page,
even if it’s not a landing
page
Elastic Path™

16. What to optimize
• Processes/workflows
• “Landing pages”
– Any individual page,
even if it’s not a landing
page
• Campaigns
– Email
– Affiliate
– Pay Per Click
Elastic Path™

17. What to optimize
• Processes/workflows
– Checkout/sales funnels
– Account registration
– Product finder
– Product configurator
Elastic Path™

18. Control checkout
Elastic Path™

19. Treatment checkout
+ 257.26% Checkout
Completion
+ 8.54% Average
Order Value
+ 0.54% Overall Site
Conversion
Elastic Path™

20. What to optimize
• Landing pages
– Top content
– Top landing pages (may not be home page!)
– Highest bounce rates
• Segment!
– Top exit pages
– Flagship products
– Underperforming products/categories
Elastic Path™

21. Design
• Colors
• Navigation
• Tabbed information
• Search
• Placement (what’s above fold, etc)
• Imagery
• Banners
• Content spaces
• Calls to action
Elastic Path™

22. Content
• Images (zoom, 360 rotate,
color swatch, multiple views)
• Headlines
• Descriptions / Copy
• Call to action text
(button labels, links)
• Review feature
• Video
• Interactive tools
• Interstitials
Elastic Path™

23. Merchandising /Promotions
• Banners
• Cross-sell, up-sell
• Bundles
• Pricing
• Offers (% off, $ off, free gift, free shipping etc)
• Products on promotion
Elastic Path™

24. Designing your test
Elastic Path™

25. Start with web analytics
• Look for areas to improve
• Funnel abandonment
• Top landing pages, bounce rate
• Top content, call to action
• Identify your conversion goal/KPIs
• Make sure you’ve segmented your data
Elastic Path™

26. Elastic Path™

27. Elastic Path™

28. Analyze your [page]
Elastic Path™

29. Elastic Path™

30. Select variables that influence each other
Elastic Path™

31. When to stop
http://tinyurl.com/gwo-calculator
Elastic Path™

32. Statistical significance
Elastic Path™

33. Testing “gotchas”
Elastic Path™

34. Testing “gotchas”
• Can’t measure the “why,” only the “what”
– (Supplement with user testing)
• “Half-life”
– Results may move back to baseline over time (traffic
mix changes, economy (offers), etc.)
• Can’t tell you if you should build a feature
– User testing
• Your application
• A competitor
• A site with a similar feature
• Problem might be product related
– Inventory, assortment
Elastic Path™

35. Comparing the tools
Elastic Path™

36. Which MVT?
http://www.whichmvt.com/
Elastic Path™

37. Major players
• Google
– Free
– 95% CI only
– “Full factorial” only
– Can’t handle segmentation or personalization
– Limits on variables/branching factors, recipes
– Can’t use on email campaigns
– Can’t use offline data
– Can’t self-host (but less reliance on IT)
– No customer support (unless you have Google rep)
– Requires a web developer to help set up
Elastic Path™

38. Major players
• Omniture Test&Target
– Plays nicely with other products in Omniture suite
– “Predictive learning” accepts offline data (e.g.
product holdings, credit score, time on file)
– Design tests right on the site
– Usually a four-figure monthly fee
Elastic Path™

39. Should you buy a tool?
• If you bought your tool, don’t change
– Omniture Test&Target plays nicely with other products
in Omniture suite
• If you haven’t bought, use Google (“Get good at
free”)
• Who owns testing? Is IT-free important?
• Do you need to test email? (most email platforms
have at least A/B testing)
• Who owns the data? Could budget constraints
force you to cancel the tool?
• Some tools have API to port into other tools
(analytics, spreadsheets)
Elastic Path™

40. Thank you!
http://www.getelastic.com/
For more on-demand ecommerce resources:
www.elasticpath.com/resources
Questions?
linda.bustos@elasticpath.com
Elastic Path™