Excitement about AI and Machine Learning is tremendous. But how many mainstream products with AI functions do you know? This talk discusses the importance to have a technical strategy for AI projects to avoid pitfalls.
How to Troubleshoot Apps for the Modern Connected Worker
How to approach Machine Learning for innovation projects? (by Jochem Grietens)
1. 3.1 How to approach Machine Learning for innovation projects? 1
CONFIDENTIAL Template Innovation Day 2018CONFIDENTIAL
MACHINE LEARNING:
PROJECT APPROACH
Jochem Grietens
System Engineer – A.I. passionate
Jochem.Grietens@verhaert.com
TRACK 3 - ROBOTICS FOR A BETTER WORLD
2. 3.1 How to approach Machine Learning for innovation projects? 2
CONFIDENTIAL
Introduction
3. 3.1 How to approach Machine Learning for innovation projects? 3
CONFIDENTIAL
Goal
4. 3.1 How to approach Machine Learning for innovation projects? 4
CONFIDENTIAL
AI and ML ?
5. 3.1 How to approach Machine Learning for innovation projects? 5
CONFIDENTIAL
AI and ML ?
6. 3.1 How to approach Machine Learning for innovation projects? 6
CONFIDENTIAL
AI and ML ?
ML
• Unstructured data structured data
• Learn from data
• Pattern recognition/detection
• …
AI
• Reasoning systems
• Planning, searching, …
• Expert systems
• Understanding ?
• …
7. 3.1 How to approach Machine Learning for innovation projects? 7
CONFIDENTIAL
Biggest drivers
1. Data availability
2. Computational performance.
Why now
8. 3.1 How to approach Machine Learning for innovation projects? 8
CONFIDENTIAL
Framing the A.I. roadmap
9. 3.1 How to approach Machine Learning for innovation projects? 9
CONFIDENTIAL
“Let’s collect as much data as possible and apply A.I. later”
“We have a bunch of data laying around … “
“A.I. as a solution to everything…”
Framing the A.I. roadmap
10. 3.1 How to approach Machine Learning for innovation projects? 10
CONFIDENTIAL
“Goal oriented”
Vs
“Explorative”
Framing the A.I. roadmap
dzdz
11. 3.1 How to approach Machine Learning for innovation projects? 11
CONFIDENTIAL
ML, where do we begin ?
12. 3.1 How to approach Machine Learning for innovation projects? 12
CONFIDENTIAL
Machine learning process
TRAINING THE MODEL
USING THE MODEL
13. 3.1 How to approach Machine Learning for innovation projects? 13
CONFIDENTIAL
Unfamiliar technology
Risk project failure
Having the right team with the right skills
Careful execution
Machine learning – The new kid on the block
14. 3.1 How to approach Machine Learning for innovation projects? 14
CONFIDENTIAL
Development cycle
15. 3.1 How to approach Machine Learning for innovation projects? 15
CONFIDENTIAL
Machine learning project steps
Verhaert project flow:
based on CRISP-DM
16. 3.1 How to approach Machine Learning for innovation projects? 16
CONFIDENTIAL
1. Business understanding
17. 3.1 How to approach Machine Learning for innovation projects? 17
CONFIDENTIAL
• “Tech driven” vs “business driven”
• Analyse the opportunity first before investing resources
and infrastructure.
Business understanding
18. 3.1 How to approach Machine Learning for innovation projects? 18
CONFIDENTIAL
Rules of ML
19. 3.1 How to approach Machine Learning for innovation projects? 19
CONFIDENTIAL
Concept on product level & Model level
Algorithms don’t live in a box
20. 3.1 How to approach Machine Learning for innovation projects? 20
CONFIDENTIAL
“Sometimes A.I. isn’t up for the job”
• Problem too complex
• Never seen something like this before
Human in the loop (HITL)
HITL + Active (incremental) Learning
A.I. failure and human interaction
21. 3.1 How to approach Machine Learning for innovation projects? 21
CONFIDENTIAL
Human interaction
22. 3.1 How to approach Machine Learning for innovation projects? 22
CONFIDENTIAL
Explainable AI (XAI )
• Decision trees
• Bayesian reasoning systems
• Expert systems
Reverse engineering
• Internal state reflection
• Parameter variation
Divide and conquer
Lack of transparency
23. 3.1 How to approach Machine Learning for innovation projects? 23
CONFIDENTIAL
2. Concept generation
24. 3.1 How to approach Machine Learning for innovation projects? 24
CONFIDENTIAL
Before understanding explore the problem
Automated car:
Traffic sign detection, Free Road detection, Traffic lights, object recognition,
Traffic rules, Distance, ….
Input data:
Visual, Lidar, HD maps, traffic rules, gps, …
Output data: steering, breaking and driving, …
Understanding of the problem
Explore the problem
25. 3.1 How to approach Machine Learning for innovation projects? 25
CONFIDENTIAL
Concept generation
26. 3.1 How to approach Machine Learning for innovation projects? 26
CONFIDENTIAL
3. Data understanding
27. 3.1 How to approach Machine Learning for innovation projects? 27
CONFIDENTIAL
Before understanding explore the problem
Automated car:
Traffic sign detection, Free Road detection, Traffic lights, object recognition,
Traffic rules, Distance, ….
Traffic sign detection:
Lighting, surroundings, color, angle, image sharpness , ….
Understanding of the problem AND data
Explore the problem
28. 3.1 How to approach Machine Learning for innovation projects? 28
CONFIDENTIAL
Creating data = expensive
Gathering low cost data & learn
• Create toy datasets
• Use existing (online) databases
• Augment limited databases
Learn as much as you can this way.
• What's missing in the database ?
• Which are the hard cases ?
…
Data Gathering – First Runs
29. 3.1 How to approach Machine Learning for innovation projects? 29
CONFIDENTIAL
So you need more data…
Questions:
• Is the needed information present in the data provided ?
• What data input could provide this information
Recruit expert knowledge.
This requires engineering/system/physics insight.
Data gathering – AI engineering
30. 3.1 How to approach Machine Learning for innovation projects? 30
CONFIDENTIAL
“Learning from a dataset that is skewed will predict skewed”
Fixes:
• Over sampling
• Under sampling
• Create new data
• Weighting techniques
• …
Training data balance
0
1
2
3
4
5
Traffic agent identification
31. 3.1 How to approach Machine Learning for innovation projects? 31
CONFIDENTIAL
Data Veracity = Accuracy
Data fidelity
• Quality in context
• Unintended Bias
Data understanding
Quality in the context it’s being used Unintended Bias
32. 3.1 How to approach Machine Learning for innovation projects? 32
CONFIDENTIAL
Learning by heart vs. learning general features
Algorithm should have learned general features :
• Recognizing 4 wheels vs 2 wheels
• Shape of person
• size vs environment
Instead it learned:
• Red/blue = car
• Tree in the background
Explaining overtraining
33. 3.1 How to approach Machine Learning for innovation projects? 33
CONFIDENTIAL
4. Data Preparation
34. 3.1 How to approach Machine Learning for innovation projects? 34
CONFIDENTIAL
Feature extraction vs raw data usage
35. 3.1 How to approach Machine Learning for innovation projects? 35
CONFIDENTIAL
Labeling is expensive!
Crowd sourcing: HUMAN INTELLIGENCE
Data labeling
36. 3.1 How to approach Machine Learning for innovation projects? 36
CONFIDENTIAL
4 C = Clean, Correct, Consistent, Complete
1. Data formatting
2. Data cleaning (usually 90% of the job),
3. Data anonymization (i.e. when working with healthcare and banking data)
4. Data augmentation
5. Feature engineering
6. …
Data preparation
37. 3.1 How to approach Machine Learning for innovation projects? 37
CONFIDENTIAL
Data augmentation
38. 3.1 How to approach Machine Learning for innovation projects? 38
CONFIDENTIAL
Artificial data synthesis
“Pick up box A ”
Factory noise
39. 3.1 How to approach Machine Learning for innovation projects? 39
CONFIDENTIAL
5. Modeling
40. 3.1 How to approach Machine Learning for innovation projects? 40
CONFIDENTIAL
1. Determine ML algorithm performance metrics
2. Create a Data pipeline
3. Build a simple model first
4. Compare models
Build a simple model
41. 3.1 How to approach Machine Learning for innovation projects? 41
CONFIDENTIAL
Off the shelf A.I. software
• From open source to pay-per-use
• Vendor support
• Clean and supply data
• Capabilities match
• Beware: Ideas with a website
AI PLATFORMS
• IBM, Amazon, Google
• off the shelf AI building tools for researchers.
Custom build
• Only when needed
• Great flexibility
BUILD vs. BUY vs. PLATFORM
42. 3.1 How to approach Machine Learning for innovation projects? 42
CONFIDENTIAL
End-to-end vs Ensembling
43. 3.1 How to approach Machine Learning for innovation projects? 43
CONFIDENTIAL
End-To-End vs Ensemble learning
Acc.
Break
Steering
44. 3.1 How to approach Machine Learning for innovation projects? 44
CONFIDENTIAL
Autonomous vehicles
Subdivide the problem in to many subcomponents
End-To-End vs Ensemble learning
Acc
Break
Steering
45. 3.1 How to approach Machine Learning for innovation projects? 45
CONFIDENTIAL
Perception
Acc
Break
Steering
46. 3.1 How to approach Machine Learning for innovation projects? 46
CONFIDENTIAL
Localization
Acc.
Break
Steering
47. 3.1 How to approach Machine Learning for innovation projects? 47
CONFIDENTIAL
Planning
Acc.
Break
Steering
48. 3.1 How to approach Machine Learning for innovation projects? 48
CONFIDENTIAL
End-to-end
- Let the machine figure the
problem out…
- More data needed
- Can improve performance
End-to-End vs Ensemble methods
Person
Ensemble methods
- Systems can be trained separately
- Less data needed
- Algorithm insight
- Levels of abstraction allow freedom:
49. 3.1 How to approach Machine Learning for innovation projects? 49
CONFIDENTIAL
5. Evaluation
50. 3.1 How to approach Machine Learning for innovation projects? 50
CONFIDENTIAL
Defining evaluation metrics
51. 3.1 How to approach Machine Learning for innovation projects? 51
CONFIDENTIAL
Define success metrics @ 2 levels
Business Level
i.e. Decrease # interventions of driver in assisted driving.
Model (ML) Level
Comparing models
i.e. Accuracy object detection, Accuracy lane detection
Guard relation between the two !
Success metrics
52. 3.1 How to approach Machine Learning for innovation projects? 52
CONFIDENTIAL
Find a metric that represents your goal & easy to compare
Multiple metrics
Single metric: Accuracy = 50 %
Single metric combo
Model level success metric
53. 3.1 How to approach Machine Learning for innovation projects? 53
CONFIDENTIAL
When ML automates things humans do well.
1. Humans are good labelers
2. Error analysis can draw on human intuition
3. Use human-level performance is often a good first goal.
Automating things humans don’t do well don’t have these advantages
to draw from – Creativity needed !
Comparing to human level performance
“This recipe calls for a pear of apples”
mistaking “pair” for “pear.”
54. 3.1 How to approach Machine Learning for innovation projects? 54
CONFIDENTIAL
Acceptance level of a solution ?
• Rational threshold ?
• Consumer acceptance threshold ?
• Legal context ?
Guard relation between the two !
Business Level
Model
success
Business
success
55. 3.1 How to approach Machine Learning for innovation projects? 55
CONFIDENTIAL
Recommendations
56. 3.1 How to approach Machine Learning for innovation projects? 56
CONFIDENTIAL
One group, five brands
Our services are marketed through 5 brands each
addressing specific missions in product development.
INTEGRATED PRODUCT DEVELOPMENT
ON-SITE
PRODUCT
DEVELOPMENT
DIGITAL
PRODUCTS
DEVELOPMENT
OPTICAL
PRODUCTS
DEVELOPMENT
VENTURING