1.
Laura Howard, Dennis Meisner
Making your hypothesis work harder
to inform future product strategy

2.
Dennis Meisner
Senior Product Analyst
Laura Howard
Lead Product Manager

3.
Treatwell is Europe’s
leading marketplace for
booking hair & beauty.
>30,000
Salons and Spas
11
Countries across Europe
>250,000
Users per day
A booking every
1.6 seconds

4.
What you’ll learn
Beyond the business metric: Using experimentation to better understand your
customers and inform your future product strategy.
How Treatwell began to learn about the ideal booking flow through:
User centric experimentation
Well structured hypotheses
In depth results analysis

5.
Why do we experiment?
Experimentation is not only about finding a solution for a business problem,
it’s about informing a future product strategy - one built on user insights.
Classic approach:
Choosing experiments to move a business goal
Single test series
“I want to move conversion”
“I want to prove this idea is right”
Our approach:
Explorative experimentation to inform future decisions
Longer running test series
“I want to give users the ideal browse
experience”
“What is the future of the checkout?”

6.
Case study: The basket page experiment
How can we improve funnel progression versus what’s the future of the
appointment selection?

7.
Experiments at Treatwell go through a three step cycle.
Generate
testable
hypothesis
Run disciplined
experiments
Learn
meaningful
insights
The Experimentation Wheel

8.
In the definition phase we define what we want to learn and what we want
to accomplish.
Generate
testable
hypothesis
Run disciplined
experiments
Learn
meaningful
insights
The Experimentation Wheel

9.
Building a hypothesis backlog
A clear definition of the goal and the problem areas are imperative to create a rich
hypothesis backlog.
Learn about user needs between
treatment selection and checkout.
Problem/
Opportunity Area 1
Basket Page
Problem/
Opportunity Area 3
Hypothesis 1
Removing the basket
page
Hypothesis 3
Remove only
Remove +
Add continue
button
Test Cell C Test Cell D
Goal
Problem/
Opportunity Areas
Hypothesis
Variants

10.
What makes a good hypothesis?
A good hypothesis outlines the change, the target group, the projected outcome
and the reason why we think this will work.
We predict that [removing the basket page] for [mobile web users]
will [bring more users to the checkout page] because
[users are fatigued of unnecessary steps in the funnel].
We will know this is true when we see an
[increase in salon page-to-checkout conversion].
outcome
action target group
reasoning
measurable outcome

11.
How do we choose the hypothesis to test?
Usually we can only test a subset of the hypothesis we came up with and
therefore have to make a choice.
Possible criteria for selecting the hypothesis to test:
Goal
Problem/
Opportunity Area 1
Problem/
Opportunity Area 2
Problem/
Opportunity Area 3
Hypothesis 1 Hypothesis 2 Hypothesis 3
Test Cell A Test Cell B Test Cell C Test Cell D
Testable: Do we have the means to test this?
ROI: Where could our efforts produce the
biggest business impact? Is this learning
actually worth the effort?
Duration: Focus on Fast Feedback to keep
momentum and achieve relevant results.

12.
In the execution phase, the hypotheses are crafted into test cells and launched
to our users.
Generate
testable
hypothesis
Run disciplined
experiments
Learn
meaningful
insights
The Experimentation Wheel

13.
What do we need to validate a hypothesis?
Things to keep in mind before starting implementation of an experiment:
Results oriented decisions: If the proposed initiative is a done deal, why go through
all the hard work to conduct an experiment?
Consider required resources: Can we increase our resources, even a little, to ensure
the optimum outcome?
Start with low fidelity: Try to break down the required change into pieces and start
testing low fidelity, low effort versions of those pieces.
Run concurrent experiments: Sequential experiments with the same goal do not
operate in the same environment.

14.
Setting up the experiment
Our insights and knowledge about the basket page led to two concurrent variants.
Goal:
Learn about user needs between treatment
selection and checkout.
Problem/
Opportunity Area: Basket Page
Hypothesis:
Removal of Basket Page
Data & Insights:
15% of users bounce
on the basket page.
Users don’t like
being directly
forwarded on the
availability page.
Users don’t engage
with the
components on the
basket page.
Cell A:
Removal of Basket Page
Cell B:
Removal of Basket Page
+ Continue on Availability
Page

15.
Variant 1 Variant 2
Primary Metric: Salon Page-to-Checkout Conversion Rate
Secondary Metric: Overall Conversion Rate
Control

16.
The Experimentation Wheel
In the analysis phase, we take a close look at the test results and decide on
the next steps.
Generate
testable
hypothesis
Run disciplined
experiments
Learn
meaningful
insights

17.
First Glance Test Results
Removing the basket page does increase the number of users proceeding to
checkout.
Variant 1 Variant 2
● +6.3% Users seeing Checkout Page
● No significant impact on CVR
● No significant impact on Salon-Checkout CVR
● No significant impact on CVR
But… is this everything we can learn from this test?

18.
Deep Dive Results
Having a closer look revealed more surprising & impactful insights!
Better Experience
Users in Variant 2 were less
likely to jump back and forth
between checkout and
availability page
Psychological reassurance lead
to more confident progression
More Net Bookings
Users in Variant 2 were less likely to
cancel and more likely to show up at
their salon and get the treatment.
Extra consideration removed
impulsive bookers

19.
Learnings & Next Steps
The learnings from this test give ample inspiration for new experiments and the
evolution of the website.
Experiment
New Testable
Ideas
Shorter flow is possible but not without
risks: We can safely remove the basket page
from the product, simplifying the UX and
code
Positive friction can ensure quality bookings:
Some user friendly positive friction will
ensure bookings are intentional and
considered, leading to less progression but
better quality
New lever for post booking success: A new,
unexpected lever for reduced cancellations
and less no shows
“What do our users really need after selecting
their treatment on the Salon Page? Do we have
a bigger opportunity than we thought?”

20.
The learnings of this test become a driver of our future product strategy.
Generate
testable
hypothesis
Run disciplined
experiments
Learn
meaningful
insights
The Experimentation Wheel

21.
Thanks for listening!