Choose the Right Problems to Solve with ML
by Spotify PM
www.productschool.com

CERTIFICATES
Your Product Management Certificate Path
Product Leadership
Certificate™
Full Stack Product
Management Certificate™
Product Management
Certificate™

Corporate
Training
Level up your team’s Product
Management skills

Free Product Management Resources
BOOKS
EVENTS
JOB PORTAL
COMMUNITIES
bit.ly/product_resources
COURSES

Choosing the Right Problems to
Solve with Machine Learning (ML)
Angela Hu Spotify Product Manager

Angela Hu
S P E A K E R
Product Manager

Why does finding the right problems matter?
● Building ML to solve a business problem, not just to build
it
● Ensures we have stakeholder buy-in and interest
● Ensure we’re investing time and resources in the right
area
● Avoid cost of error from choosing problems that should
not be handled by ML
● Ensure we have the right resources (skills, data,
technology) to invest in this problem space
● Prevent production & scaling blocks by checking that
system maturity levels can handle ML products

Evaluating the landscape
Business Impact. Technical Feasibility.

What criteria should I consider?
Business Impact. Technical Feasibility.
Data Availability
Do we have the right training data
(eg. labelled data for supervised
problems?).
ML Solvability
Can this problem be solved with ML
and what exists for ML solutions
today?
System maturity
Can we put the ML into production?
Can we scale? Can we get feedback
for this ML problem?
Business Impact
What business problem does this
solve? How will this help us reach our
KPIs?
Application vs. Insights
Are we developing an ML application
or insights? Who do we need to get
on board?
Risk
Do we have the right data to avoid
bias? How high is the cost of error?

ML Scorecard for Multiple Options
What: ML scorecard or weighted decision matrix
When: Use when prioritizing between multiple
problem spaces or to get alignment on problem
selection with multiple groups
How: Add in criteria for both categories, business
impact and technical feasibility. Decide on
weighting. Evaluate what you can with a score from
1-5 and compare final scores of all options
Note:
This isn’t met to be an exact science. Think of it as
a relative comparison to prioritize options

Examples of good vs. bad
Example 1
A team wants to automate
classification of customer
support tweets as needing
advisor support or not.
Landscape
● Automation can save
substantial cost
● Labelled training data is
available
● Heuristics system in place
is complex to manage
Example 2
A team wants to use ML to
provide recommendations on
an infrequently visited part of
site
Landscape
● No way to capture user
signals on recos
● Existing website cannot
handle dynamic inputs
● Low business impact
✅ Great Fit 🚩Bad Fit 🟡 Uncertain
Example 3
A team wants to use ML to
provide recommendations on a
frequently visited part of site
Landscape
● Feedback mechanism to
capture user signals
● Existing website CAN
handle dynamic inputs
● HIGH business impact
● Uncertain stakeholders

You’ve found a great problem! Now what?

Designing the UX for your ML product
Use cases
What are edge use cases do we
need to consider? What needs to
be rules-based vs. ML driven?
UI Design
What do we need to change in our
UI? How do we set the right
expectations for our end users on ML
capability?
Cost of Error
What happens to your end user if
something goes wrong? If the cost
is large (eg. blocking user account),
change precision vs. recall balance
Metrics (w/ DS or MLE)
What metrics should we use and
what do we prioritize (eg. precision
vs. recall)? How much error are
stakeholders okay with?

Frequent Challenges
Longer development time without knowing outcome
● You’re teaching a product to learn the system
● May impact stakeholder buy-in
● Derisk by:
○ Performing offline analysis
○ Pilot or A/B testing for initial online analysis
Data availability
● Setting up predictive ML problems requires us to have labelled data to train
the model
○ Often requires manual tagging or hiring of data curators
Feedback Loop
● There isn’t always a clear signal from end users on whether or not prediction
is correct OR system does not capture feedback

Summary
● What problems are best solved with ML and what problems are
NOT
■ High business impact and technical feasibility
■ Use a scorecard when prioritizing multiple initiatives
● What you need to understand and how technical you need to get as
a PM of an ML product
■ Consider and plan for edge cases
■ Work with DS / MLE to define metrics and risk tolerance
■ Derisk in the product life cycle

Thank You!
WWW.PRODUCTSCHOOL.COM
Find me on Linkedin at:
https://www.linkedin.com/in/ahu29

www.productschool.com
Part-time Product Management Training Courses
and
Corporate Training