Mais conteúdo relacionado

Apresentações para você(20)

Similar a Choose the Right Problems to Solve with ML by Spotify PM(20)


Mais de Product School(20)


Choose the Right Problems to Solve with ML by Spotify PM

  1. Choose the Right Problems to Solve with ML by Spotify PM
  2. CERTIFICATES Your Product Management Certificate Path Product Leadership Certificate™ Full Stack Product Management Certificate™ Product Management Certificate™
  3. Corporate Training Level up your team’s Product Management skills
  5. Choosing the Right Problems to Solve with Machine Learning (ML) Angela Hu Spotify Product Manager
  6. Angela Hu S P E A K E R Product Manager
  7. Why does finding the right problems matter? ● Building ML to solve a business problem, not just to build it ● Ensures we have stakeholder buy-in and interest ● Ensure we’re investing time and resources in the right area ● Avoid cost of error from choosing problems that should not be handled by ML ● Ensure we have the right resources (skills, data, technology) to invest in this problem space ● Prevent production & scaling blocks by checking that system maturity levels can handle ML products
  8. Evaluating the landscape Business Impact. Technical Feasibility.
  9. What criteria should I consider? Business Impact. Technical Feasibility. Data Availability Do we have the right training data (eg. labelled data for supervised problems?). ML Solvability Can this problem be solved with ML and what exists for ML solutions today? System maturity Can we put the ML into production? Can we scale? Can we get feedback for this ML problem? Business Impact What business problem does this solve? How will this help us reach our KPIs? Application vs. Insights Are we developing an ML application or insights? Who do we need to get on board? Risk Do we have the right data to avoid bias? How high is the cost of error?
  10. ML Scorecard for Multiple Options What: ML scorecard or weighted decision matrix When: Use when prioritizing between multiple problem spaces or to get alignment on problem selection with multiple groups How: Add in criteria for both categories, business impact and technical feasibility. Decide on weighting. Evaluate what you can with a score from 1-5 and compare final scores of all options Note: This isn’t met to be an exact science. Think of it as a relative comparison to prioritize options
  11. Examples of good vs. bad Example 1 A team wants to automate classification of customer support tweets as needing advisor support or not. Landscape ● Automation can save substantial cost ● Labelled training data is available ● Heuristics system in place is complex to manage Example 2 A team wants to use ML to provide recommendations on an infrequently visited part of site Landscape ● No way to capture user signals on recos ● Existing website cannot handle dynamic inputs ● Low business impact ✅ Great Fit 🚩Bad Fit 🟡 Uncertain Example 3 A team wants to use ML to provide recommendations on a frequently visited part of site Landscape ● Feedback mechanism to capture user signals ● Existing website CAN handle dynamic inputs ● HIGH business impact ● Uncertain stakeholders
  12. You’ve found a great problem! Now what?
  13. Designing the UX for your ML product Use cases What are edge use cases do we need to consider? What needs to be rules-based vs. ML driven? UI Design What do we need to change in our UI? How do we set the right expectations for our end users on ML capability? Cost of Error What happens to your end user if something goes wrong? If the cost is large (eg. blocking user account), change precision vs. recall balance Metrics (w/ DS or MLE) What metrics should we use and what do we prioritize (eg. precision vs. recall)? How much error are stakeholders okay with?
  14. Frequent Challenges Longer development time without knowing outcome ● You’re teaching a product to learn the system ● May impact stakeholder buy-in ● Derisk by: ○ Performing offline analysis ○ Pilot or A/B testing for initial online analysis Data availability ● Setting up predictive ML problems requires us to have labelled data to train the model ○ Often requires manual tagging or hiring of data curators Feedback Loop ● There isn’t always a clear signal from end users on whether or not prediction is correct OR system does not capture feedback
  15. Summary ● What problems are best solved with ML and what problems are NOT ■ High business impact and technical feasibility ■ Use a scorecard when prioritizing multiple initiatives ● What you need to understand and how technical you need to get as a PM of an ML product ■ Consider and plan for edge cases ■ Work with DS / MLE to define metrics and risk tolerance ■ Derisk in the product life cycle
  16. Thank You! WWW.PRODUCTSCHOOL.COM Find me on Linkedin at:
  17. Part-time Product Management Training Courses and Corporate Training