Technical overview about how optimizely works.

1. A/B Testing
what’s that and how it works
Konrad Synoradzki Allegro Group 2013

2. What is A/B testing?
A/B testing is comparing two versions of a web
page to see which one performs better.
Konrad Synoradzki Allegro Group 2013

3. web-page visitors
Konrad Synoradzki Allegro Group 2013

4. slice visitors
Konrad Synoradzki Allegro Group 2013

5. Then compare two web pages by showing the two
variants (let's call them A and B) to similar visitors at
the same time.
Konrad Synoradzki Allegro Group 2013

6. example
Konrad Synoradzki Allegro Group 2013

7. example
Konrad Synoradzki Allegro Group 2013

8. example
Konrad Synoradzki Allegro Group 2013

9. What can you test?
Headlines
Sub headlines
Paragraph Text
Testimonials
Call to Action text
Call to Action Button
Links
Images
Content above or below
the fold
• Social proof
•
•
•
•
•
•
•
•
•
Konrad Synoradzki Allegro Group 2013
•
•
•
•
•
•
Media mentions
Awards and badges
Banners efficiency
Rotation speed
New version of page
Removing ads

10. Types of tests
Konrad Synoradzki Allegro Group 2013

11. A/B testing
Best used to measure the
impact of a two to four
variables on interactions with
the page.
Examples:
• new layout
• Labels on button
Konrad Synoradzki Allegro Group 2013

12. Multivariate Testing
• compares a higher
number of variables,
• reveals more information
about how these
variables interact with
one another.
• traffic to a page is split
between different
versions of the design.
Examples
• Test headline, button
color, button text and
image on landing page
Konrad Synoradzki Allegro Group 2013

13. Multi-Page Testing
• A/B test on multi-page
Example:
• Colors on primary
action in process
• Badges on promo items
Konrad Synoradzki Allegro Group 2013

14. Getting Started
What is and how to use optimizely?
Konrad Synoradzki Allegro Group 2013

15. Getting Started
Step 0: Implement the snippet
Konrad Synoradzki Allegro Group 2013

16. Getting Started
Step 1: Enter your URL on Optimizely.com
Konrad Synoradzki Allegro Group 2013

17. Getting Started
Step 2: Add page variations
Konrad Synoradzki Allegro Group 2013

18. Getting Started
Step 3: Set up goals for tracking
Konrad Synoradzki Allegro Group 2013

19. Getting Started
Step 4: Target and allocate
Konrad Synoradzki Allegro Group 2013

20. Report page
Konrad Synoradzki Allegro Group 2013

21. Short movie
https://fast.wistia.net/embed/iframe/gxttknv8aa?auto
Play=true&controlsVisibleOnLoad=true&version=v1&vo
lumeControl=true
Konrad Synoradzki Allegro Group 2013

22. Technical details
but how??
Konrad Synoradzki Allegro Group 2013

23. Organizing Experiments
Konrad Synoradzki Allegro Group 2013

24. User Roles
Konrad Synoradzki Allegro Group 2013

25. Each Project Has Its Own
Project Code (Snippet)
or asynchronous version
Konrad Synoradzki Allegro Group 2013

26. Optimizely Snippet Contents
•
•
•
•
•
Active Experiments
Draft Experiments
Project Goals
Optimizely execution and tracking logic
Optimizely jQuery Settings (optional – version 1.6.4 included by
default)
Konrad Synoradzki Allegro Group 2013

27. Two modes
Konrad Synoradzki Allegro Group 2013

28. Optimizely translates every change made in the visual editor
into a line of jQuery code, which we call Variation Code:
Konrad Synoradzki Allegro Group 2013

29. Optimizely Testing Process
Konrad Synoradzki Allegro Group 2013

30. Order of Execution
Konrad Synoradzki Allegro Group 2013

31. Snippet Timing Logic
Konrad Synoradzki Allegro Group 2013

32. Optimizely Architecture Overview
Konrad Synoradzki Allegro Group 2013

33. Optimizely Architecture Overview
• Google App Engine is used to store and manage your account and
experiment configuration settings, including your account login
and password information. Passwords are hashed and stored
securely using 160 bit cryptographic hashes.
• Amazon Elastic Compute Cloud and Simple Storage Service is
used to store your conversion/traffic logging data, which
Optimizely uses to compute experiment results.
• Akamai is used to store and serve your Optimizely snippet
JavaScript file(s). This file is publicly accessible by necessity.
Konrad Synoradzki Allegro Group 2013

34. Collecting data
The visitor’s browser makes an asynchronous request to
https://log.optimizely.com, which logs the following information:
•
•
•
•
•
•
•
The Optimizely­­assigned unique user ID
IDs of the experiments and variations this visitor has seen
Your Optimizely account number
Visitor’s browser/OS version
Visitor’s IP address, with last octet removed
A string identifying this conversion goal, such as the page URL
Date and time
Konrad Synoradzki Allegro Group 2013

35. that’s all
Konrad Synoradzki Allegro Group 2013

36. question?
Konrad Synoradzki Allegro Group 2013

37. Konrad Synoradzki Allegro Group 2013