This talk covers the PM framework needed to lead AI incubations. Product school webinar video at https://www.linkedin.com/video/live/urn:li:ugcPost:6690684172895322113/

1. LEADING AI (ML)
INCUBATIONS
DEBAPRIYA BASU
PRINCIPAL PRODUCT MANAGER @ ZILLOW

2. HELLO!
I am Debapriya Basu
I am here to share my learnings and experiences
on how to drive AI incubations.
You can find me at
https://www.linkedin.com/in/debapriyabasu/
Current: PM@Zillow Group
Former: PM@Microsoft
2

3. WHAT I WILL
COVER
Journey of an AI(ML) PM, using real life examples to
illustrate
◎ When to use AI(ML) to solve customer problems
◎ What is a generic PM framework for an AI (ML)
incubation
◎ What skills help PMs to succeed in AI (ML)
incubations
3

4. WHEN TO USE
AI(ML) 4

5. IDENTIFY CUSTOMER
PROBLEM
1

6. IDENTIFY
CUSTOMER
PROBLEM
◎ What is the problem
◎ How can the problem be solved
without AI(ML)
◎ How can AI(ML) add value
◎ Identifies complex patterns to predict
outcomes
◎ Adapts outcome to inputs in real time
◎ Scales on vast datasets fast
◎ Enables personalization
◎ Adapts to post launch improvements on
model through feedback flywheel generated
by user’s interaction with UX + AI
6
ZESTIMATE

7. AI(ML) IN ACTION
2 PREDICTION
RECOMMENDATION
CLASSIFICATION
ANOMALY DETECTION
NATURAL LANGUAGE PROCESSING
CONVERSATIONAL AI ETC

8. USE AI (ML)
TECH FOR
◎ Predictions
8ZILLOW

9. USE AI (ML)
TECH FOR
◎ Predictions (ZESTIMATE)
◎ Recommendations
9
NORDSTROM

10. USE AI (ML)
TECH FOR
◎ Predictions
◎ Recommendations
◎ Classification
10
MICROSOFT

11. USE AI (ML)
TECH FOR
◎ Predictions
◎ Recommendations
◎ Classification
◎ Natural Language Processing
11
MICROSOFT

12. ◎ Predictions
◎ Recommendations
◎ Classification
◎ Natural Language Processing
◎ Anomaly Detection
USE AI (ML)
TECH FOR
12
GOOGLE

13. ◎ Predictions
◎ Recommendations
◎ Classification
◎ Natural Language Processing
◎ Anomaly Detection
◎ Conversational AI
◎ etc
USE AI (ML)
TECH FOR
13
GOOGLE

14. USE AI(ML) STRATEGY
TO DRIVE BUSINESS
OUTCOMES3

15. BUSINESS
OUTCOMES
◎ AI(ML) solutions provide such
enhanced productivity for
end-users that it ends up changing
their Human Computer Interaction
and becomes table stakes for
businesses
Search
15

16. BUSINESS
OUTCOMES
◎ Customized User Experiences leads to
engaged and retained users
◎ Recommendations
◎ Custom Personalization
◎ Conversational AI
Netflix artwork is personalized to
Individuals
16NETFLIX

17. BUSINESS
OUTCOMES
◎ Identification of insights using
AI(ML) help optimize for business
goals (like saving costs or deciding
more effectively where to invest)
17
GOOGLE
Google used DeepMind’s ML to reduce
energy to cool its data centers by 40%

18. BUSINESS
OUTCOMES
◎ Strategy helps expand to new
surfaces and customers
quicker
Google cloud search is Google search for
enterprise content and is available in Gmail,
Drive, Docs, Sheets, Slides, Calendar here
18
GOOGLE

19. PM FRAMEWORK
FOR AI(ML)
INCUBATIONS
19

20. STAGES
20
Identify Data
and AI
Technology
Ideate PrototypeDefine
Vision
Build TeamGet
Sponsors
hip
ArchitectureUX Experiment
and
Productionize
we
Develop
Model

21. IDEATE ALIGN
Align stakeholders to a
common definition of
the problem
OBJECTIVE FUNCTION
Is it Prediction, Classification
or something else
MODEL QUALITY
METRICS
Classification Metrics like
Precision, Recall etc
Rank Aware Metrics like
NDCG, MRR etc
Trade-offs
21
EXPERIENCES
Identify hero
experience with
implicit and explicit
user feedback
mechanism to ideate
SUCCESS
DEFINITION
Define success in terms
of online metrics

22. IDEATE
22
GOAL:
How popular will a listing be?
OBJECTIVE FUNCTION:
Predict the likelihood of a listing to drive a
contact request tomorrow, given
● the listing content features (price,
size, location etc) and
● historical engagement (the number
of views, saves and contacts a listing
has received on the previous days)
OFFLINE METRIC:
AUC and NDCG
ONLINE METRIC:
#contacts, #views
Ref

23. IDENTIFY
DATA
AND AI
TECHNOLOGY
DATA
◎ 1st Party vs 3rd Party
◎ Privacy and Security
◎ Fresh
◎ Unbiased
◎ Representative
◎ Relevant
◎ Missing data
◎ Noisy data
◎ Inconsistent
23
AI TECH
◎ Machine learned
◎ Supervised
◎ Unsupervised
◎ Reinforced
◎ Modeling Techniques
◎ Deep learning
◎ NLP
◎ Ensemble Learning
◎ Pre-trained Models

24. DEFINE
VISION
24
SurfacesValue
Proposition
Prototype
Crawl,
Walk,
Run
Metric and
Tradeoff for
Success
Long-term
Success
Examples:
Summarization to help user triage work content effectively
User UnderstandingAI to understand and predict customer behaviour
for deeper personalization.

25. PROTOTYPE
25
Validate
● If supervised
learning, validate
offline metrics
against ground
truth
● If clustering,
determine how well
situated a point is in
a cluster or how
distant clusters are
Build
● Simplify objective
function
● Select most relevant
features
● Test multiple algos if
needed
Iterate till
acceptance
criteria reached
● Define behavior in
edge cases
● Define fallback if
model does not
work as needed
Not often time-bound
and maynot yield best
results in get go
Don’t forget to eyeball
Are you improving
performance on existing
non-AI orAI solution
Analyze
● Analyze every
important attribute
of dataset
● Identify
relationships
between attributes
in the dataset
● Analyze in context of
time, user etc
● Visualize

26. GET
SPONSORSHIP
26
ROI
In terms of monetary gain,
users acquired, retained, cost
saving, gain or save on
company OKRs
Resourcing
How many
What type
Value for the business
Align with overall strategy
Competing / Differentiator
Value for the customer
Productivity, Engagement etc
SELL
PRODUCT/TECH
VISION

27. BUILD TEAM Immediate
Skills
◎ Applied Science
◎ Data Science
◎ Data Pipeline Engineering
◎ E2E and online Product Engineering
◎ Machine Learning Engineering
27
V-Team
◎ Annotators
◎ Experience Teams
◎ Architecture Teams
◎ Data Engineering Teams
◎ Experimentation Platform
Teams
◎ Legal, Privacy, Security
Attitude
● Comfort with ambiguity
● Willingness and ability to shape the problem
● Ability to look at the problem from different
angles and dimensions
● Ability to think laterally
● Success comes in iterations

28. 28
UX FOR AI -> TRANSPARENT + EXPLAINABLE AI
MODEL CONFIDENCE
EXPLICIT FEEDBACK
LIKE/DISLIKE
DATA SOURCES
MODEL
FEATURES
Ref
IMPLICIT FEEDBACK IS CRITICAL

29. AI
ARCHITECTURE
29
Ref Offline Modeling pipeline
E2E Architecture
3 levels
◎ Offline model training pipeline
◎ Online serving infrastructure
◎ E2E architecture
◎ UX
◎ Online serving infra
◎ Offline Batch processing
◎ Other dependent systems
PM considerations
- Mechanism of Data collection
(batched/streamed)
- Frequency of Data Collection
- Dependency systems
- How are predictions processed
(real-time/offline)

30. SHIP A MODEL Refine prototype or build new model
◎ DataAnalysis
◎ Algorithm selection
◎ Objective function selection
◎ Training and test set selection
◎ Feature selection
◎ Expand data coverage
◎ Check for outliers
◎ Retrain, retest your model
◎ Scale to new markets and languages
30
Operationalize
◎ Validate via online Experiments
◎ Ship in production
◎ Learn from causal uplift analysis
◎ Scale data collection
◎ Scale models to new markets and
languages
◎ Incorporate feedback either in
real time or offline
◎ Reduce friction to deploy models
◎ Version control models
Success => Online metric improvements,
Offline metric improvement

31. SUSTAINED
SUCCESS
CRITERIA
AI Models powering
Individual Product or Feature
Established Relation between
online metric and offline metric
31
AI Models for understanding
a class of entities
Adoption
User UnderstandingAI is
a class of models that
helps understand
customer’s state, segment
and journey

32. ETHICAL AI Privacy
AI models should be built within
guardrails to ensure users cannot be
identified or their details inferred
from model output
Researchers were able to identify Netflix
users by correlating anonymized test data
provided in the Netflix Prize competition with
publicly available IMDB movie review
database.
32
Fairness
AI/ML models used for making decisions or
predictions should not be biased with respect
to protected attributes (latent bias) such as
race, gender and sexuality. It should be aware
and counter impacts of interaction and
selection bias also.
A MIT study Project Gender Shades uncovered
the bias that facial analysis technologies have
a heavy bias towards white males.
JoyAdowaa Buolamwini foundedAlgorithmic
Justice League, an organisation that looks to
challenge bias in decision making software

33. SKILLS THAT
HELP AI PMs
SUCCEED
33

34. SKILLS
34
Clarify
Clarify goals, assumptions
Understand what the data says
Simplify problem, process,
solution
Be resilient
Learn from failures, course
correct, reshape problem,
remodel, reengineer
Success is a process, not an
end goal
Focus
Keep an eye on customer problem at
all times
Refine optimization function and
online metric until crisp, offline
metrics until relation between online
and offline metric is identified
Zoom in-out
(+) Guide product direction and
discussion as needed
(-)Align product to broader
business strategy

35. WHAT’S NEXT
AFTER A
SUCCESSFUL
INCUBATION
35
Life post incubation (launch)
◎ Analyze user feedback data, both implicit and explicit
◎ Understand user pain-points with the product and work to mitigate it
As a PM, this is the moment
where you decide to take the
shipped technology to next
phase or move to shape a new
idea to solve a customer
problem
◎ Continue to iterate model with fresh data and feedback
◎ Build a strategy for improved model quality, by training on sophisticated
frameworks, more data etc
◎ Understand and analyze industry and expert sentiments on the product
◎ Build a strategy of the shipped product/technology ROI
◎ Should it be expanded to new surfaces, new markets
◎ Does the engagement and feedback funnel provide enough data to
retrain models
◎ Should UX be updated to ensure better quality feedback
USER
IMPACT
MODEL
QUALITY
PRODUCT-
TECH
STRATEGY

36. HOW IS
LEADING
AN AI (ML)
INCUBATION
DIFFERENT
FROM LEADING
OTHER
PROJECTS
AS A PM
36
What’s same, what’s different
- Understand customer problem,
- Define strategy
- Define vision and success criteria of product
- Drive alignment and collaboration
- Execute
- UnderstandAI technology, process and needs of building models
- Define optimization goal of the model, success KPIs
- Understand Datasets, learn relationship between online and offline metric
- Design UX that works for the non-deterministic aspect ofAI
- Understand Privacy and Legal impact onAI models and how they affect customers
Core PM
skills
AI PM
skills
- Shape a problem from ambiguity
- Find, stitch data, identify insights from data
- Envision and convince others of the vision
- Collaborate, collaborate, collaborate
- Think laterally and think iteratively, reshape, re-pivot as needed
- Be resilient (failures form the way to ultimate success)
AI incubation
PM skills

37. SPECIAL
THANKS
◎ Amit Mondal, Google
◎ Kieran Mcdonald, Microsoft
◎ Ondrej Linda, Zillow
◎ Sangdi Lin, Zillow
37
Slide template

38. WHAT I
COVERED
Takeaways and food for thought
◎ When do you use AI (ML) to solve customer
problems
◎ What are the stages you as a PM go through in an
AI (ML) incubation
◎ What skills and mindset will help you to succeed
in AI (ML) incubations
Please reach out with your comments and thoughts at
tutulpriya@gmail.com
38