1.
www.productschool.com
How to Effectively Experiment in PM
by LendingTree Sr PM

2.
Join 35,000+Product
Managers on
Free Resources
Discover great job
opportunities
Job Portal
prdct.school/PSJobPortalprdct.school/events-slack

3.
C O U R S E S
Product
Management
Learn the skills you need to land
a Product Manager job

4.
C O U R S E S
Coding
for Managers
Build a website and gain the
technical knowledge to lead
software engineers

5.
C O U R S E S
Data Analytics
for Managers
Learn the skills to understand web
analytics, SQL and machine learning
concepts

6.
C O U R S E S
Learn how to acquire more users
and convert them into clients
Digital Marketing
for Managers

7.
C O U R S E S
UX Design
for Managers
Gain a deeper understanding of your
users and deliver an exceptional end-to-
end experience

8.
C O U R S E S
For experienced Product Managers
looking to gain strategic skills needed
for top leadership roles
Product
Leadership

9.
C O U R S E S
Corporate
Training
Level up your team’s Product
Management skills

10.
Lokesh Manohar
T O N I G H T ’ S S P E A K E R

11.
What is Experimentation?
Experiment
“an experiment is a procedure carried out to
support, refute or validate a hypothesis” – Wiki
Experimentation
Experiment
• The process of improving an outcome
• Experiment is the initial building block of
experimentation
Experimentation
“Scientific approach to understanding the needs
of customers and building products that they will
find valuable”
Experimentation in Product Management

12.
Basic Building Block of Experimentation - A/B Testing
• Compare performance of test samples against control samples
• Samples are identical except for one aspect whose effect is being tested
Population
Random Sampling
Control Test
A B
Performance
Metric
Change Statistically
Significant?
Control x
Test y y ÷ x (%) yes / no

13.
A/B Testing - Examples
vs.
vs.

14.
A/B Testing - Examples
vs.

15.
A/B Testing - Examples
vs.

16.
A/B Testing - Examples
vs.

17.
A/B Testing - Examples
vs.

18.
Why Experiment?
Pick the Winner!
vs.
2 steps & no auto-complete Single step with auto-
complete

19.
Why Experiment?
Pick the Winner!

20.
Why Experiment?
vs.
Pick the Winner!

21.
Why Experiment?
Problem
A
Solution
Product Process without
Experimentation
Optimal
Solution
X
Reaching the Optimal Solution is difficult -
• Predictions are highly sensitive to assumptions
and assumptions are never truly robust
• Actual customer behavior can deviate widely
from stated opinion
• “not everyone is Steve Jobs”
MM of $$

22.
Why Experiment?
Problem
Solution
v1
Product Process with Experimentation
Optimal
Solution
Solution
v2
• Deeper customer empathy leading to
understanding of the Why and not just the
What
• Data driven (scientific approach) hence more
robust
• Breaks HiPPO bias
• Ideas multiply and helps avoiding local
maxima in product development

23.
Why Experiment?
Number of experiments
• LendingTree – 800 in
2018
• Intuit: 1,300 (in 2015)
• P&G: 7,000 – 10,000
• Google : 7,000
• Amazon – 1,976
• Netflix – 1,000
“We design, analyze, and execute experiments
with rigor so that we have confidence that the
changes we’re making are the right ones for our
members and our business.”
-
Netflix
“Even well-researched products can suffer due
to the gap between what customers think they
want and what their behaviors reveal they
actually want.”
-
Optimizely
“Our success at Amazon is a function of how
many experiments we do per year, per month,
per week, per day.”
-
Jeff Bezos
“[Facebook] conducts tens of thousands of
experiments at any given time.”
-
Mark Zuckerberg
“[we continually] optimize our conversion
funnel all the way through loan funding, which
is something that we believe will substantially
move our company forward”
- Doug Lebda, CEO of
LendingTree
“it’s critical to adopt an “experiment with
everything” approach”
-
Ron Kohavi, Microsoft

24.
Experimentation Program Maturity
• Startups
• Few thousand
customers per year
• Early stages of instituting
an experimentation
program
• High growth companies
• Few million customers per
year
• Experimentation program
established but lacking
complete integration and
automation
• High and medium growth
companies
• Several million customers
per year
• Experimentation culture
adopted company wide

25.
Experimentation Process
Define
Objective
Experimentation
Objectives should align with product goals
Objectives should be measurable through core
performance metrics
Set high and broad goals (don’t be conservative)
How to establish objective(s)?
1. Define Objective(s)
Increase conversion rate by 10%
Improve NPS by 5 points
Reduce customer churn by 25%
Examples

26.
Experimentation Process
Define
Objective
Ideation &
Prioritization
Idea sources –
• Quantitative research (funnel analysis,
segmentation)
• Qualitative research (user interviews)
• Product roadmap
• Competitive analysis
Experimentation
2a. Ideation

27.
Experimentation Process
2b. Prioritization
Idea Description Idea Source Product Goal Performance Metric Expected Impact Priority Score Status
Add progress bar Link to quantitative analysis
Increase conversion
rate
Conversion Rate 5% 2 Backlog
Native input options for date of
birth
Link to competiive analysis
Increase conversion
rate
Conversion Rate 2% 6 In Dev
Auto sign-in for returning users
Link to user interview
videos
Increase customer LTV
Revenue per Visit (returning
users)
10% 10
Ready to
launch
Priority Score = F (impact, effort, bandwidth)

28.
Experimentation Process
ABT - Always Be Testing
X ✔

29.
Experimentation Process
Define
Objective
Ideation &
Prioritization
Form
Hypothesis
If <this is done>,
Then the <metric> will increase/decrease,
Because <expected customer behavior>
Experimentation
How to form a hypothesis?
3. Form Hypothesis
Examples
If slider is provided as input option for loan
amount step
Then the conversion rate would increase
Because of decreased abandons in that step
Why
?
✔ Helps determine what should be measured
✔ Helps avoid HARKing

30.
Experimentation Process
Define
Objective
Ideation &
Prioritization
Form
Hypothesis
Experiment
✔ Communication
✔ Platform (build vs. buy)
✔ Client side vs. Server side
✔ Target segment
✔ Statistical power
• Size of the effect vs. sample size
✔ Other live experiments
✔ Reporting
✔ Monitor (but don’t analyze, don’t p-hack)
Experimentation
Key considerations
4. Run Experiment

31.
Experimentation Process
Define
Objective
Ideation &
Prioritization
Form
Hypothesis
Experiment
Analyze
Experimentation
5. Analyze
Performance
Metric
Change Statistically
Significant?
Control x
Test y y ÷ x (%) yes / no
✔ Analyze segments and funnels
✔ Check other KPIs
✔ Beware of HARKing
✔ No win ≠No value
✔ Generate ideas for future experiments

32.
Experimentation Process
Define
Objective
Ideation &
Prioritization
Form
Hypothesis
Experiment
Analyze
Publish
Experimentation
6. Publish

33.
Experimentation Infrastructure
✔ Experimentation Platform (server, sampling,
targeting)
✔ Analytics & Reporting
✔ Program Management Tools (ticketing,
documentation, communication) Netflix Experimentation Platform

34.
Resources
1. https://medium.com/search?q=experimentation
2. https://theblog.adobe.com/experience-cloud/target/
3. https://www.widerfunnel.com/blog/
4. A/B Testing: The Most Powerful Way to Turn Clicks Into Customers - https://www.amazon.com/Testing-Most-
Powerful-Clicks-Customers/dp/1118792416
5. You Should Test That: https://www.amazon.com/You-Should-Test-That-Optimization/dp/1118301307/

35.
Thank you
www.lendingtree.com
https://www.linkedin.com/in/hsekol/
@lokeshhhsekol

36.
www.productschool.com
Part-time Product Management, Coding, Data Analytics, Digital
Marketing, UX Design, Product Leadership courses and
Corporate Training