Start moving your experimentation program to the next level
Many organizations are already testing and experimenting on a small scale. So now the big question is: what’s stopping you from moving to the next level?
In this webinar, we’ll guide you through the framework that helps you to assess where you are on your journey – plus the four essential building blocks for success.
Building your team and making the time. Defining your strategy. Integrating the technology that unlocks benefits way beyond Google Analytics. And developing the culture of experimentation that drives innovation - and powers growth.
Discover real-world, actionable insights that allow you to successfully grow your program, deliver more value to your business and maximise ROI.
Lisa Rohlfs will be your expert guide:
- Discover a framework to evaluate your journey so far.
- Gain actionable insight to move your program to the next level.
- Create the culture of experimentation that drives innovation – and powers growth.
6. 1. Discover a framework to evaluate your
journey so far.
2. Gain actionable insight to move your
program to the next level.
3. Create the culture of experimentation that
drives innovation – and powers growth.
10. Level 2
● You can report on business metrics
● Your Analytics are integrated with
your testing platformTechnology
● Standardized process across a
team
● Roadmap document that includes
structured hypothesis & prioritization
criteria
● Experiments are more often
targeted to specific
personas/segments
● Using indirect and direct data for
ideation
● Non-standardized process
● Roadmap is a simple list of
experiment ideas
● Experiments are targeted
infrequently to specific audiences
● Sometimes leverages web analytics
or past experimentation results for
ideation
Strategy
● Management actively drives
experimentation and leverages
results for decision making
● Teams experiment cross-
functionally
● Management is aware but not
actively driving experimentation
● More than one team in one part of
the business is experimenting
Culture
● More than 1 full-time resource on
experimentation, access to
developers
● Variety of roles involved
● Less than 1 full-time resource on
experimentation
● Foundational roles in place
Team
Level 3
11. Level 2
● You can report on business metrics
● Your Analytics are integrated with
your testing platform
Technolog
y
● Standardized process across a
team
● Roadmap document that includes
structured hypothesis & prioritization
criteria
● Experiments are more often
targeted to specific
personas/segments
● Using indirect and direct data for
ideation
● Non-standardized process
● Roadmap is a simple list of
experiment ideas
● Experiments are targeted
infrequently to specific audiences
● Sometimes leverages web analytics
or past experimentation results for
ideation
Strategy
● Management actively drives
experimentation and leverages
results for decision making
● Teams experiment cross-
functionally
● Management is aware but not
actively driving experimentation
● More than one team in one part of
the business is experimenting
Culture
● More than 1 full-time resource on
experimentation, access to
developers
● Variety of roles involved
● Less than 1 full-time resource on
experimentation
● Foundational roles in place
Team
Level 3
13. 13
Increasing Time Spent on experimentation
Team Culture Strategy Technology
Link experimentation to existing strategic
initiatives
Existing reach
14. 14
Increasing Time Spent on experimentation
Team Culture Strategy Technology
Link experimentation to existing strategic
initiatives
Showcase experiment ROI
Existing reach
16. 16
Getting more teams to experiment
Analyze required Skill Sets
Create internal Onboarding Guide
Share Optimization Successes
Team Culture Strategy Technology
Expand reach
17. 17
Example testing charter/Onboarding Guide
Team Culture Strategy Technology
CompanyName Testing Methodology
Introduction & Program Goals
[Add an introduction paragraph]
•● Problems: The Optimizely testing platform was purchased to help us [describe the rationale for purchase here]
•● Outcomes & Success: Our testing program will be successful if we are able to [describe what success looks like
here]
•● This Document: This document is the source of truth for our organization when it comes to all things related to
our testing program. If you have additional suggestions for it please don’t hesitate to edit this document or let
[Testing Program Manager’s Name] know so it can be added to this page.
•Team
•● Executive Sponsor - [name and email]
•● Program Manager/Power User - [name and email]
•● Primary Developer - [name and email]
•● Primary Designer - [name and email]
•● Testing QA Lead - [name and email]
•● [add other team members as needed]
•Process
•[Add an introduction paragraph describing the testing process generally.]
•[Example image]
•[Add your steps from your process below]
•● Brainstorm: [who, where, why, what, when, how]
•● Prioritization: [who, where, why, what, when, how]
•● Documentation & Design: [who, where, why, what, when, how]
•● Setup: [who, where, why, what, when, how]
•● QA & Approval: [who, where, why, what, when, how]
•● Run: [who, where, why, what, when, how]
•● Analysis & Review: [who, where, why, what, when, how]
•● Results Sharing: [who, where, why, what, when, how]
in the testing team and their respective answers]
•Meetings & Meeting Cadence
•[You may include this as part of your process described above or you may note it here separately.]
•Test Velocity
•The aim is to run at least [##] tests per [timeframe]. However testing quality is important as well and we should aim
to learn something from every test. We’ll start with this initial goal but testing velocity may [increase/decrease]
depending on our team’s agility and it’s goals for any given quarter.
•Test Plan & Results Templates
•Each test plan should include the following elements and be based off [link to a downloadable template].
Consistently documenting the tests we run so we can have quarterly summary meetings and effectively learn from
our tests. We aim to have a strong testing results archive.
•Links Key
•● Roadmap
•● Templates
• ○ Test Plan
• ○ Test Results
•● Results Archive
•● Goals Definitions
•● Targeting Definitions
•● Audience Definitions
•● Optimizely Knowledge Base
•● Optimizely Community
•● Optimizely Learning Academy
•● [add other items as needed]
•
•Other Guidelines and FAQs
•● [list general guidelines for the testing program]
•● [list common questions for people
23. 23
Integrating Business Metrics
Revenue:
$2.0m
Revenue Per
Visitor:
$1.00
Average
Order Value:
$50.00
Average
quantity:
2.00
Average per
unit price:
$25.00
Conversion
Rate:
2.00%
Add to Cart
Cart
Checkout
Rate
Visitors:
2.0m
User
acquisition:
1.5m
User
retention
33%
Team Culture Strategy Technology
24. 24
Integrating Business Metrics
Revenue:
$2.0m
Revenue Per
Visitor:
$1.00
Average
Order Value:
$50.00
Average
quantity:
2.00
Average per
unit price:
$25.00
Conversion
Rate:
2.00%
Add to Cart
Cart
Checkout
Rate
Visitors:
2.0m
User
acquisition:
1.5m
User
retention
33%
Define strategic program KPIs
(company/team)
Ways to leverage a goal tree
Compare QoQ/YoY snapshots
Identify metric ‘trade-offs’
Team Culture Strategy Technology
26. 26
Process Best Practices
Direct data
● Web analytics
● Experiment results
● Voice of the customer
● Heatmaps
● User personas
● Usability tests
● Customer decision model
● Metrics
Indirect data
● Competitor sites
● Best-in-class experiences
● Blogs and webinars
● User groups
● Academic literature
Leverage Data
01
Team Culture Strategy Technology
30. 30
Ideation Workshop
Team Culture Strategy Technology
Post-Workshop
1. Compile ideas for a
prioritized
backlog/roadmap
2. Get attendees involved
in execution of
experiments
3. Keep everyone
informed on progress &
successes!
Preparation
1. Invite different areas of
the business
2. Set up
whiteboards/virtual
collaboration
3. Define area of website
to ideate on (page-
specific)
Execution
1. Introduce hypothesis
concept (*see slide 25)
2. Assign area of the
website to the team
3. Identify all ‘user
problems’
4. Brainstorm possible
solutions
5. Define metrics of
success
31. 31
Ideation Workshop
Team Culture Strategy Technology
Post-Workshop
1. Compile ideas for a
prioritized
backlog/roadmap
2. Get attendees involved
in execution of
experiments
3. Keep everyone
informed on progress &
successes!
Preparation
1. Invite different areas of
the business
2. Set up
whiteboards/virtual
collaboration
3. Define area of website
to ideate on (page-
specific)
Execution
1. Introduce hypothesis
concept (*see slide 25)
2. Share area of focus on
your website
3. Identify all ‘user
problems’
4. Brainstorm possible
solutions
5. Define experiment
metrics
32. 32
Ideation Workshop
Team Culture Strategy Technology
Post-Workshop
1. Compile ideas for a
prioritized
backlog/roadmap
2. Get attendees involved
in execution of
experiments
3. Keep everyone
informed on progress &
successes!
Preparation
1. Invite different areas of
the business
2. Set up
whiteboards/virtual
collaboration
3. Define area of website
to ideate on (page-
specific)
Execution
1. Introduce hypothesis
concept (*see slide 25)
2. Share area of focus on
your website
3. Identify all ‘user
problems’
4. Brainstorm possible
solutions
5. Define experiment
metrics
37. 37
When
● Timeline
● Highlight interferences
What
○ Prioritized experiments
○ Balanced testing schedule
Roadmap Best Practices:
What to include
Team Culture Strategy Technology
38. 38
Who
○ Availability of resources
○ Dependencies
When
● Timeline
● Highlight interferences
What
● Prioritized experiments
● Balanced testing schedule
Roadmap Best Practices:
What to include
Team Culture Strategy Technology
39. 39
ObjectivesWho
● Availability of resources
● Dependencies
When
● Timeline
● Highlight interferences
What
● Prioritized experiments
● Balanced testing schedule
Roadmap Best Practices:
What to include
● Funnel coverage
● Strategic theme
Team Culture Strategy Technology
44. 44
Process Best Practices
Analysis
01
Be strict on statistical significance
Understand why certain changes lead to your desired outcomes (wins)
Team Culture Strategy Technology
45. 45
Process Best Practices
Analysis
01
Be strict on statistical significance
Understand why certain changes lead to your desired outcomes (wins)
Analyse Inconclusive & Losing Tests
Team Culture Strategy Technology
46. 46
Process Best Practices
Analysis
01
Be strict on statistical significance
Understand why certain changes lead to your desired outcomes (wins)
Analyse Inconclusive & Losing Tests
Document and share takeaways
Team Culture Strategy Technology
48. 48
Targeting
1st & 3rd
Party Data
Location
Based
Targeting
Demographic
Targeting
DCP Data
Real Time
Information
Behavioral
Targeting
Team Culture Strategy Technology
50. 50
Integrating Business Metrics
Clarify the metrics you need to track
Evaluate if and where you are able to track them
Team Culture Strategy Technology
Revenue:
$2.0m
Revenue Per
Visitor:
$1.00
Average
Order Value:
$50.00
Average
quantity:
2.00
Average per
unit price:
$25.00
Conversion
Rate:
2.00%
Add to Cart
Cart
Checkout
Rate
Visitors:
2.0m
User
acquisition:
1.5m
User
retention
33%
52. 52
Watch your program mature!
Leverage the recording of this
session & our academy for
guidance on activities
Identify areas of opportunity
Take the Optimizely Maturity
Assessment (it’s free!)
53. 53
Webinar 3: Been there, done that.
Learn how leading brands have taken their digital
to new heights.
Friday 25th January 2019
12:00 GMT / 13:00 CET
54. Thank you for listening.
• Lisa Rohlfs
Manager, EMEA Customer Success