Mais conteúdo relacionado Semelhante a Machine Learning for Product Managers (20) Mais de Thoughtworks (20) Machine Learning for Product Managers3. ©ThoughtWorks 2020 Commercial in Confidence 3
What is
Machine
Learning?
Technical
Feasibility
Customer
Value
Commercial
Viability
What we’ll cover today
A look at ML through six product management lenses
©ThoughtWorks 2020 Commercial in Confidence
Usability Strategy Delivery
4. ©ThoughtWorks 2020 Commercial in Confidence 4
What is
Machine
Learning?
Technical
Feasibility
Customer
Value
Commercial
Viability
What we’ll cover today
A look at ML through six product management lenses
©ThoughtWorks 2020 Commercial in Confidence
Usability Strategy Delivery
6. ©ThoughtWorks 2020 Commercial in Confidence 6
Breaking down the jargon
A house price example
distance material age
31 BRICK 12
2 WOOD 7
15 BRICK 13
sale price
$250,000
$670,000
????????
.
7. ©ThoughtWorks 2020 Commercial in Confidence 7
Technical
Feasibility
Customer
Value
Commercial
Viability
What we’ll cover today
A look at ML through six product management lenses
©ThoughtWorks 2020 Commercial in Confidence
Usability Strategy Delivery
What is
Machine
Learning?
.
8. ©ThoughtWorks 2020 Commercial in Confidence
Start small and add more data &
model complexity in increments.
8
How models &
data interact
Getting the balance right
MLMODELCAPACITY BESPOKE DATA ESTATE
Investment
.
9. ©ThoughtWorks 2020 Commercial in Confidence 9
Two dimensions of data
Volume vs complexity.
distance material age
31 BRICK 12
2 WOOD 7
15 BRICK 13
sale price
$250,000
$670,000
$300,000
.
10. ©ThoughtWorks 2020 Commercial in Confidence 10
Two dimensions of data
Volume vs complexity.
distance material age
31 BRICK 12
2 WOOD 7
15 BRICK 13
17 WOOD 12
sale price
$250,000
$670,000
$300,000
$250,000
.
11. ©ThoughtWorks 2020 Commercial in Confidence 11
Two dimensions of data
Volume vs complexity.
distance material age yard
31 BRICK 12 SMALL
2 WOOD 7 LARGE
15 BRICK 13 MEDIUM
17 WOOD 12 SMALL
sale price
$250,000
$670,000
$300,000
$250,000
.
12. ©ThoughtWorks 2020 Commercial in Confidence 12
Two dimensions of data
Volume vs complexity.
distance material age yard
31 BRICK 12 SMALL
2 WOOD 7 LARGE
15 BRICK 13 MEDIUM
17 WOOD 12 SMALL
days to sell
7
14
6
5
.
14. ©ThoughtWorks 2020 Commercial in Confidence
Model options
Commodity vs bespoke models.
Cloud services and existing models
provide a commodity set of capability,
but don't differentiate.
Bespoke models give the most
flexibility, but come at non-trivial
engineering cost.
.
14
15. ©ThoughtWorks 2020 Commercial in Confidence 15
Customer
Value
Commercial
Viability
What we’ll cover today
A look at ML through six product management lenses
©ThoughtWorks 2020 Commercial in Confidence
Usability Strategy Delivery
What is
Machine
Learning?
Technical
Feasibility
16. ©ThoughtWorks 2020 Commercial in Confidence 16
Mapping value
to investment
Getting the balance right
Customer value should increase to
justify the investment in both data and
models.
(Just like all products)
A clear measure of customer value
and a non-ML baseline is essential.
CUSTOMER VALUE 16MLMODELCAPACITY BESPOKE DATA ESTATE
Investment
17. ©ThoughtWorks 2020 Commercial in Confidence 17
Mapping value
to investment
Getting the balance right
If additional investment leads to no
growth in user value, look for
opportunities on the other dimension.
MLMODELCAPACITY BESPOKE DATA ESTATECUSTOMER VALUE
Balanced
Zone
Over-engineered
model
Redundant
data
18. ©ThoughtWorks 2020 Commercial in Confidence
The stalled value
trap
Enough already.
Customer value plateaus as better
models and more data provide no
perceivable improvement.
18CUSTOMER VALUE 18MLMODELCAPACITY BESPOKE DATA ESTATE
No growth
in value
19. ©ThoughtWorks 2020 Commercial in Confidence
The hype trap
We just need some ML!
The opportunity for ML is
overestimated. Simple rules-based
systems already deliver high value and
a major ML investment is required to
match the customer value.
19CUSTOMER VALUE 19MLMODELCAPACITY BESPOKE DATA ESTATE
Marginal value
growth after
major
investment
20. ©ThoughtWorks 2020 Commercial in Confidence
The moonshot
True faith.
Value relies on complex models and
high volume and complexity of data.
High risk, with no feedback on
progressing customer value.
20CUSTOMER VALUE 20MLMODELCAPACITY BESPOKE DATA ESTATE
21. ©ThoughtWorks 2020 Commercial in Confidence
Complementary
Approaches
The most reliable path.
Most successful ML products combine
multiple models and datasets, often at
different stages of development,
which maximise customer value over
the life of the product.
21CUSTOMER VALUE 21MLMODELCAPACITY BESPOKE DATA ESTATE
22. ©ThoughtWorks 2020 Commercial in Confidence 22
Commercial
Viability
What we’ll cover today
A look at ML through six product management lenses
©ThoughtWorks 2020 Commercial in Confidence
Usability Strategy Delivery
What is
Machine
Learning?
Technical
Feasibility
Customer
Value
23. ©ThoughtWorks 2020 Commercial in Confidence
Platforms and
tooling
How to take advantage of the
burgeoning ecosystem.
● cloud APIs
● host your own model
● download and tune model
● train existing from own data
● novel model with own data
.
23
24. ©ThoughtWorks 2020 Commercial in Confidence
Data Cost
Structure
How to manage costs
Expect high people costs in data
engineering when first accessing data.
Storage is cheap but grows as
complexity and volume increases
2424
BESPOKE DATA ESTATE
DATA ENGINEERS
DATA STORAGE
Money Icon by Matlo from the Noun Project
25. ©ThoughtWorks 2020 Commercial in Confidence
ML Model cost
Structure
People and technology.
Compute costs can be significant but
typically have gradual growth.
Data Scientists are expensive and can
cause explosive increases during the
transition from generic models to
custom models.
2525MLMODELCAPACITY
DATA
SCIENTISTS
COMPUTE
COSTS
MLaaS
Generic
Custom
Money Icon by Matlo from the Noun Project
Free!
26. ©ThoughtWorks 2020 Commercial in Confidence 26
Privacy & Ethics
Land Mine Icon by Ethan Clark from the Noun Project
Function
ML unlocks new opportunities for
powerful new products that are ethically
debatable. Self driving vehicle safety,
facial recognition in crime prevention.
Explainability
Many ML models lack explainability;
compounding biases and frustrating
those seeking redress. Fraud
prevention
Data Security
All ML requires lots of data; holding this
data is a privacy risk. Re-identification is a
risk when PI data points can be
reproduced from a model.
Bias
Models reflect their training data. Biased
datasets produce biased models. Facial
analysis model have perpetuated racism
and sexism in hiring and criminal justice.
.
27. ©ThoughtWorks 2020 Commercial in Confidence 27
What is
Machine
Learning?
Technical
Feasibility
Customer
Value
Commercial
Viability
What we’ll cover today
A look at ML through six product management lenses
©ThoughtWorks 2020 Commercial in Confidence
Usability Strategy Delivery
28. ©ThoughtWorks 2020 Commercial in Confidence 28
Robust to failure
Combine algorithms for flexibility and nuance.
➜ Zidane
➜ Humour
➜ Sport
➜ Football
➜ Football
➜ Zidane
29. ©ThoughtWorks 2020 Commercial in Confidence 29
Augmenting vs
Automating
Start with a humble UI
29MLMODELCAPACITY BESPOKE DATA ESTATE
Confident
UI
Humble UI
AUGMENTING
AUTOMATING
30. ©ThoughtWorks 2020 Commercial in Confidence 30
What is
Machine
Learning?
Technical
Feasibility
Customer
Value
Commercial
Viability
What we’ll cover today
A look at ML through six product management lenses
©ThoughtWorks 2020 Commercial in Confidence
Usability Strategy Delivery
31. ©ThoughtWorks 2020 Commercial in Confidence
Custom combinations of
known
models
Novel model IP
Competitive
Advantage
Where are the moats?
Best opportunities for competitive
advantage are in:
● Data, rather than models
● Combinations
● Usage feedback loops creating a
flywheel effect.
31HIGHEST COMPETITIVE. AD. 31MLMODELCAPACITY BESPOKE DATA ESTATE
Generic
models
as a
service
Spreadsheet models
Existing
models on
exclusive
data
Usage
Data
feed-
back
loops
Existing
models on
replicable
data
32. ©ThoughtWorks 2020 Commercial in Confidence 32
ML’s double flywheel
How usage can create unassailable competitive advantage.
EXPERIENCE
QUALITY
USAGE
$ UNIQUE DATA
(Volume & Complexity)
MODEL
INNOVATION
33. ©ThoughtWorks 2020 Commercial in Confidence 33
Value creation vs efficiency
Where to look for opportunities.
Value Creation Efficiency
Definition Create customer new customer
value which can be monetised.
Reduce costs or mitigate risks
through efficiency
Examples Voice services, self driving
vehicles
Process prioritisation, fraud
detection.
Competitive
Advantage
Potential for flywheel. Incremental and replicable.
Timing Explore urgently Build gradually.
Likelihood of success Risky, moonshot More proven.
34. ©ThoughtWorks 2020 Commercial in Confidence 34
Tech-led or customer-led?
What if your CEO just wants some AI.
● Generally we advise customer-led to ensure we are solving real problems
● But ML is a big enough step from known tech that there is the risk of Product
teams not knowing what they don’t know.
● ML bootcamps, which are tech-led, for all teams to understand the potential of the
tech and re-assess existing Opportunities with new ML-based solutions.
35. ©ThoughtWorks 2020 Commercial in Confidence 35
What is
Machine
Learning?
Technical
Feasibility
Customer
Value
Commercial
Viability
What we’ll cover today
A look at ML through six product management lenses
©ThoughtWorks 2020 Commercial in Confidence
Usability Strategy Delivery
.
36. ©ThoughtWorks 2020 Commercial in Confidence
ML Maturity
Where are the skills?
There's usually a mix of skills and
impacts within an organisation.
Cross team
Company
wide
Single team
Descriptive Predictive Prescriptive
Many teams
doing simple
things
Deep
expertise
in few
teams
Capture
expertise
in tooling
.
36
37. ©ThoughtWorks 2020 Commercial in Confidence
Low risk with low impact?
High risk with high impact?
A mix?
Moonshots or
incremental?
Research portfolio
Portfolio Evolution
.
37
38. ©ThoughtWorks 2020 Commercial in Confidence
feedback?
https://l.ead.me/bbRXBr
Thank you!
Matt Travers
matt.travers@thoughtworks.com
Mat Kelcey
mkelcey@thoughtworks.com