As digital marketers and experience leaders, we have increasingly reduced our customers to merely data points, line graphs and bar charts. The problem is that we are losing the necessary insight and experimentation to understand human behaviour across digital channels. To uncover our customers’ true intent, and ultimately understand the behavioural impact on the bottom line, we need to start asking WHY. Hear from experts from Clicktale and Optimizely as they share experiences from working with brands like Samsung, Missguided and RBS to uncover: - What data and insights can uncover about customers’ digital behaviour - How to align metrics that look at measuring the experience, not just conversion - How brands are scaling an approach to data, insights and analytics across their organisations - Best practices in ideation, A/B testing and experimentation

1. 1
What Your Customers
Really Do Online:
5 Ways to Remove
the Guesswork

2. 2
Kenneth Kutyn
Optimizely Solutions
Engineer
Welcome
Jonathan Bhonker
Clicktale Product
Manager

3. Digital Experience Optimization:
Digital Products, commerce & campaigns
Up to 5X Increase in Yield:
Revenue, share of wallet, funnel conversion,
risk mitigation, ops efficiency
Partner of Choice:
Work with leading global enterprises & “digital
disruptors” including 26 of F100
OUR COMPANY
Digital Experimentation Platform:
Next gen “Test and Learn” system
Replaces Digital Guesswork:
with evidence-based optimization
Speeds Innovation & Optimization:
Single platform for marketing & product teams
Best-in-class stats & machine learning
Consumer-grade usability
Enterprise program management & prof. services
OUR SOLUTION

4. 4
Companies don’t lose in the digital age
because of technology
They lose because they’re not customer
centric and they don’t focus on
experience
The world today…

5. 5
80% of everything you
put online doesn’t work
in the way that you think
it would do.
“
”
The rule of digitization:
Giles Richardson
Vice President - Digital Journeys at T-Mobile

6. 6
1.
Using Data to
kick off ideation

7. 7
Do Experiment Wins
Depend on Analytics?
No Analytics Heatmaps Only
Analytics Only Analytics & Heatmaps
33% 37%
25% 31%

8. 8
Looking from different perspective
• Solutions that offer session replay show real
visitors on your website.
• Watching the different ways in which people
interact with your webpages or app lets you
identify areas for improvement.
• One Clicktale customer has digital teams view
session replays on their pages once
a week to get an outside perspective on how
well their experiences are built.

9. 9
Regular post match review

10. 10
Example: not all filters are created equal

11. 11
Thinking Bigger
Start with the low hanging fruit, but don’t get stuck
on a single page or flow
Every interaction is an opportunity to optimize,
surprise and delight your customers
Test your business model, omni channel journeys,
tech stack, pricing and packaging, and product
launches

12. 12
Problem Solution Result Framework
The <user persona>..
• Cannot find..
• Does not know…
• Does not see…
• Wants to…
• Is distracted by…
• Is forced to…
• Needs...
Problem Solution Result
• Clearly define the problem
• Validate it with qualitative or
quantitative data
• Describe the proposed solution(s)
• Provide rationale (why will it work?)
• Choose metrics to measure results
• Set criteria for success and failure
By <doing solution> the user will... If <solution> solves <problem>, then
<metric> will increase/decrease

13. 13
2.
Experiment
Prioritization

14. 14
Where can your efforts have the biggest impact?
Identify Opportunities
for Improvement
Simple formulas to help you start prioritizing where to focus:
WHERE?
• Homepage: 30,000 Visitors x 50% Exit Rate = 15,000 Visitors (opportunity)
• Product Detail Pages: 15,000 Visitors x 25% Exit Rate = 3,750 Visitors (opportunity)

15. 15
Monetizing behavior - eCommerce checkout

16. 16
Revenue Tree
Benefits
Comprehensive:
• Visually analyzes the entire site’s
opportunity, and help avoid
opportunity cost
Alignment:
• Allows teams to explicitly align
testing goals to business goals;
keeps ideas tightly focused
Productive Brainstorming:
• Strategic foundation for creative
brainstorming, helping to garner
a volume of ideas for
prioritization
*Orange = Areas for digital
optimization

17. 17
Learning from the Best
Use weighted scoring systems to rank
experiment ideas.
Get input from all CRO stakeholders
Open up to business stakeholders
Scoring should reflect what you learn from
analytics, session recordings, & heatmaps.

18. 18
Prioritization should be repeatable
and scalable
What factors are used to prioritize experiments. How do more advanced programs do it?
• Trello
• Excel
• Jira
• Optimizely Program Management
• Build your own

19. 19
Hotwire Case Study
A good prioritization framework “removes the emotion from A/B testing.”
- Lead Product Manager Pauline Marol
Rule 1 Point 0 Points
Main Metric Supports the company’s main metric – new bookings Supports a secondary metric
Location Tests a change to the results or billing pages Tests a change located on any other pages
Fold Makes a change above the fold Makes a change below the fold
Targeting Targets 100% of customers Targets a subset of customers
New Information Adds new information or a new element Makes a change to the existing elements (copy, color, UI, etc.)
Benchmarking Borrows from a success on family sites No benchmarking best practice
Conversion Veins Applies to two or more conversion vein themes Applies to one or fewer conversion vein themes
Strategic Topic Supports a strategic company goal Doesn’t map to a company-level goal
Mobile Would change an element of the mobile web No mobile component
Opaque Potential to increase of opaque share for a line of business
(travel industry-specific value metric)
No influence on opaque share of business

20. 20
Hotwire Case Study
1. Correlation vs. causation - how good is the evidence for this change affecting the root cause?
2. Measure experience - define and use quantifiable experience KPIs in addition to raw conversion to
prioritize
3. Monetization (quantification) - putting a monetary value on an issue can help get buy-in from
stakeholders to push for a change
4. Easy change - small changes are easier to promote than big changes. Score each experiment for
ease of change (to site/app) vs. expected impact

21. 21
3.
Measure the Full
Experience

22. 22
Conversion does not tell the whole story

23. 23
Hello!Bank case study
Question: Why are 95% dropping at 1st stage of enrollment form?
Answer: Social connect buttons were causing 90% of attempts to
register to end in failure
Solution: Enhanced enrollment page to allay visitor concerns

24. 24
Where does Analysis Happen?
PM BA DS

25. 25
Measure the Experience
Analysis is more than just winners and losers:
- Non-conclusive tests still teach us something
- Segmentation can reveal hidden winners
Dave Powell
Ecommerce
Conversion Manager
“People are starting to come to me and say ‘we want to test
this’ and opinions are starting to fly around. It’s beginning to
take hold that the data will tell the truth,”

26. 26
4.
Democratize
Experimentation

27. 27
Get multiple perspectives

28. 28
Case study: Large NA bank
The content team ran a test to serve its images from a new
platform. Using Clicktale, the were able to identify that 8% of
visitors were receiving blank space instead of the images. This
reduced the number of logins from the homepage.
A second team focusing on improving account opens reviewed
the test results and measured different KPIs and experience
metrics.
They discovered that prospects receiving the blank boxes
showed improved engagement with the homepage and were
more likely to open an account. This was due to removal of
distracting content from an already cluttered page.
Rather than simply changing content-serving platforms, as the
test results suggested, the company redesigned the homepage
for simplicity.

29. 29
Experimentation is a Team Sport
Learn from the best:
- Program-level reporting
- Regular updates sent to all stakeholders
- Experiment idea forms open to all
- Accessible platform

30. 30
5.
Connect the
voice of customer

31. 31
Give your customers a seat at the table

32. 32
Case Study : Apparel retailer
A major US apparel retailer received dozens of surveys during Black through a third
party platform. Visitors reported technical problems they ran into or usability issues
encountered. Using an integration with Clicktale, the retailer was able to see playback of
visitors who experienced these issues.
Technical issues and errors were fixed at high priority.
Usability issues were set aside in a backlog. Following Black Friday, the VoC team
shared these findings with the separate Product teams.
Product teams examined visitor playback to understand the causes behind the
comments, generating ideas for A/B tests. These causes were quantified using Clicktale
to prioritize the tests.

33. 33
Case study: Samsung
“Decision making on how to improve customer
experience is always data-driven now at Samsung,
from a very small in-page change, all the way through
to the whole digital customer journey and we try and
use those learnings from previous launches of products
and push that across our future planning, to what our
customer best responds to. We feed that into the
next launch to ensure that the customer gets the best
experience and, of course, that we optimize our own
sales.”
Richard Wilson,
Director of Digital Analytics,
Optimization and Personalization
Samsung

34. 34
Wrap up

35. 35
Summary
Move beyond the guesswork:
1. Examine all customer touchpoints in detail
2. Base your decisions on data
3. Leverage tools to understand intent
4. Measure beyond basic KPIs
5. Connect your programs

36. 36
Q&A

37. 37
Further reading / content
Resources from Clicktale / Optimizely
Building your own prioritization framework:
https://help.optimizely.com/Ideate_and_Hypothesize/Build_a_basic_pri
oritization_framework?mt-learningpath=optimizationmethodology
Testing Toolkit:
https://www.optimizely.com/resources/testing-toolkit/