Main takeaways:
- How to build Product Roadmap together with Data Science
- How to Prioritize Machine Learning features
- Measuring success on Machine Learning models
3. C O U R S E S
Product
Management
Learn the skills you need to land
a Product Manager job
4. C O U R S E S
Coding
for Managers
Build a website and gain the
technical knowledge to lead
software engineers
5. C O U R S E S
Data Analytics
for Managers
Learn the skills to understand web
analytics, SQL and machine learning
concepts
6. C O U R S E S
Learn how to acquire more users
and convert them into clients
Digital Marketing
for Managers
7. C O U R S E S
UX Design
for Managers
Gain a deeper understanding of your
users and deliver an exceptional end-to-
end experience
8. C O U R S E S
For experienced Product Managers
looking to gain strategic skills needed
for top leadership roles
Product
Leadership
9. C O U R S E S
Corporate
Training
Level up your team’s Product
Management skills
10. T O N I G H T ’ S S P E A K E R S
Mahesh Chayel
&
Tim Conklin
11. Building scalable Machine
Learning products
@Product School, Boston
Mahesh Chayel
https://www.linkedin.com/in/maheshchayel/
Tim Conklin
https://www.linkedin.com/in/tconklin91/
12. “
Three takeaways today:
- How to build product roadmap together with
Data Science
- How to prioritize Machine Learning features
- Measuring success on Machine Learning
models
12
16. When do Product and DS
work together?
Enhanced experience
and functionality
Finding relevant
product or estimates of
different values e.g.
Zillow estimate
Internal functions and
business logic
Improving the way
internal complex
process work e.g.
Reserve price
calculation at Ebay
Expansion to new
verticals and products
Building insights from
data to serve new
segments or product
verticals e.g. Zillow
recommends real estate
developers on what
features buyers care
about
16
17. Start from the basics
Define vision of
your product
Identify
user pain
points
(JTBD)
Explore
Solutions
17
Define problem
to be solved
Understand
user needs
Investigate
current
alternatives
18. Understand key concepts
18
Types of Learning
1. Supervised Learning
a. Regression
b. Classification
2. Unsupervised Learning
a. Clustering
b. Anomaly detection
3. Semi-supervised Learning
4. Reinforced Learning
What matters?
1. Objective function selection
2. Algorithm selection: depending
on problem to solve
3. Feature selection: depending on
availability and usefulness of
data
4. Explainability and
interpretability
20. Considerations for both
PM and DS
Real time
requirements
Data and model
dependencies
Data collection
frequency
20
Data collection
methods
21. Prioritization depends on
your test/ launch plan
Prepare for a
soft launch
Plan for
experiment
s
Iterate after
launch
21
22. Example: Reserve Price
22
Progression stage Features required
Optimal reserve price theory Business rules
First basic data science model
Most impactful features + simple
model
Iterate on model A/B testing framework
More complex model versions As needed
25. Parameters to check for ML
model success
1. The same business
metrics
a. Acquisition
b. Activation
c. Retention
d. Revenue
e. Referral
2. ML specific metrics
Measuring success of ML
models
Frameworks for running ML
tests
1. Traffic slice based
testing
2. Product specific testing
3. Market specific testing
25
26. Example: Reserve Price
26
Testing Framework When to Use
Traffic Split Testing User behavior, no feedback loop
Market Testing Anticipated feedback loop
Product Testing Focus/small group testing
27. The Obvious reading
list
- “Data Science
for Business:
What You Need
to Know about
Data Mining
and Data-
Analytic
Thinking” by
Foster Provost
- “Storytelling
with Data: A
Data
Visualization
Guide for
Business
Professionals”
by Nussbaumer
Knaflic, Cole
- “Data Smart:
Using Data
Science to
Transform
Information into
Insight” by John
W. Foreman
27
"As you checked in we sent you an email to join our online communities, events, and to apply for product management jobs. As members of the Product School community we'd like to provide you with these resources at your disposal."
Mahesh/ Tim
3 minutes
Questions for audience
Mahesh
2 min
3 key pieces of Product Manager workstream
Tim
1 min
Tim
3 min
Slight background into what auctions are and how they’re used at TA and other companies
What reserve prices are in an auction and why they’re needed
Some explanation in how this may vary based on a number of predictors
Mahesh
Mahesh
2 min
3 usual scenarios
Mahesh
3 min
Cover each topic and how DS and PM work together
Tim
3 min
Some background on what questions are the right ones to ask and the wrong ones and how to define the focus of the problem you’re trying to solve
How to choose the proper algorithm
Some TripAdvisor specifics on algorithms and choices that people might need to know about
User Likelihood to Book
What attributes does this restaurant have (i.e. is it a romantic restaurant, casual, etc)
Where users stay in NY
Do predictors for model variables change significantly WoW or DoD
Ad selection
Mahesh
Mahesh
3 min
Real time: can results be computed in advance or do they need to be computed real-time. Architecture changes a lot
Data and model dependencies: Do models run into each other? Do we need to wait for some data? Is there an SLA to be met?
Data collection frequency: How much data to store? How much recent data do you need? What is the frequency that the data must be available to you?
Data collection methods: Batch or stream? Push or pull? API, file upload, model input mechanisms?
Mahesh
2 min
Soft launch: To understand the impact and catch blind spots. B2B vs B2C approach
Plan for experiments: Have a plan B. Have monitoring and alerting in place. Logging
Iterate after launch: Have a framework for iterating. Plan on adding new kind of models and/or signals.
Tim
3 min
Tim
Mahesh
1 min
Mahesh
2 min
Tim
3 min
Considerations for the testing
How big is the expected effect size?
Can long term effects be ignored?
What are the anticipated feedback loops? Will users that have seen the treatment behave differently in the future? Will partner behavior be different in the future?
What are the differences between markets? Can the learnings in market A be applied to market B?
Some TripAdvisor/reserve price applications
TST: Update reserve prices on some fraction of traffic. Long term effects can be ignored and assumed that savings/losses will be invested accordingly
MT: Partners expected to bid differently once Return on Ad Spend Changes (reserve price example)
Seeing the flaws/hangups when users are trying to perform tasks on our site to guide future testing
Tim + Mahesh
3 min
Lots of pop culture type books like Nate Silver and Tufte. Short 30 second blurb on what I `like` about each book.