Presentation held during Belgrad AI's opening event in March 2019. Terminology and workflows for a regular ML or data science project, top points from selecting an AI project and a little framework to help you get everything on the same page for the top levels details of your ML project.
4. Unless you have a huge dataset (‘Big Data’), it’s
not worth attempting ML or data science projects
on your problem.
TRUE
FALSE
5. Unless you have a huge dataset (‘Big Data’), it’s
not worth attempting ML or data science projects
on your problem.
TRUE
FALSE
6. “The future is independent of the past given the
present.”
𝑃(𝑋𝑡|𝑋𝑡−1,…,𝑋0)=𝑃(𝑋𝑡|𝑋𝑡−1)P(Xt|Xt−1,…,X0)=P(Xt|Xt−1).
7.
8. WHAT’S AN AI-FIRST COMPANY?
Unified data warehouse2
Strategic data acquisition 1
New roles (MLE) & division
of labor
4
Pervasive automation 3
9. WHAT’S AN AI-FIRST COMPANY?
Any company + #<deep learning> AI-first company
10. Workflow of a ML project
Collect data
Train model
*iterate many times until it’s good
enough
Deploy model
*get back + Maintain/update model
11. Workflow of a ML project
Collect data
Train model
*iterate many times until it’s good
enough
Deploy model
*get back + Maintain/update model
Workflow of a data science project
Collect data
Analyze data
*iterate many times until good
insights
Suggest hypotheses/actions
*deploy changes + re-analyze data
periodically
12. SELECTING AN AI PROJECT
Valuable
for your
business
What AI can
do
13. SELECTING AN AI PROJECT
Valuable
for your
business
What AI can
do
AI Experts Domain Experts
14. SELECTING AN AI PROJECT -
due diligence
TECHNICAL
DILIGENCE
BUSINESS
DILIGENCE
Can AI system meet
desired
performance?
Lower costs
How much data is
needed?
Increase revenue
Engineering timeline Launch a new
product or business
15. SELECTING AN AI PROJECT -
due diligence
TECHNICAL
DILIGENCE
BUSINESS
DILIGENCE
Can AI system meet
desired
performance?
Lower costs
How much data is
needed?
Increase revenue
Engineering timeline Launch a new
product or business
BONUS: Brainstorming framework
1. Think automating tasks rather
than automating jobs.
1. What are the main drivers of
business value?
1. What are the main pain points
in your business?
*Having more data almost never
hurts, but you can make progress
w/o big data. Small datasets can be
enough to get started on some
projects.
16. Working with an AI Team
● Specify your acceptance criteria (eg: detect defects with
95% accuracy)
● Write good specifications by providing your AI team a
dataset on which to measure performance (eg. test set
~1000 pics)
● Involve the AI Team with the IT Team in calibrating your
data collection
17. AI Canvas was built by: Ajay Agrawal, Joshua Gans and Avi Goldfarb, Rotman School of Management, University of Toronto.
ML CANVAS
Prediction Judgement Action Outcome
what do you need
to know to make
the decision?
Ie. 95% accuracy
how do you value
different outcomes
and errors?
Ie: retrain, keep it
actual.
what are you trying to do?
Ie. Yes/No or more nuanced
what are your metrics for task success?
Ie. action always leads to outcome
Input Training Feedback
What data do you need to run the
predictive algorithm?
Ie. what’s going on at the time a decision
needs to be made
What do you need to train the predictive
algorithm?
Ie. the richer and varied the dataset, the
better.
Have you used the outcomes to improve the algorithm?
Ie. data in real situations, form a richer set of
environments that training data.
18. “If you train a learning algorithm and every single picture you
show it is a picture of a cat, then it thinks every single thing in
the world is a cat, because it’s never seen anything that’s not a
cat before. And I think in a similar way, if you show Executives
a sequence of success stories but no failure stories, then it
creates an impression that AI can do anything, and that’s just
not true.”
Andrew Ng - Google Brain CoFounder, ex-Baidu Chief AI
Scientist, Coursera, Deeplearning.ai
Editor's Notes
Markov Chains help build the concept of decision making in a very simplistic way. It’s also one of the most important deterministic processes used in AI.
How could we bring Markov Chains’ simplicity to business? Connecting the dots between data, AI, and value creation are essential for working on the right probleM.
Pic 3 main points: 1. 1950 alan turing propose the test for machine intelligence ; 2. 2011 Siri - intelligent virtual assistant making AI for everyone; 3. 2016 - Microsoft’s Tay chatbot builds its own identity on social media and goes rogue.
Internet-first companies had 3 main things in common: A/B testing; short iteration time and decision making being pushed down to engineers and PMs or other roles.
Replace DL with any AI tech
Ethical diligence: will it make society/humanity better?
Prediction - a machine can potentially tell you this
Judgement - no prediction is 100% accurate so in order to determine the value of investing in better prediction you need to know the cost of different outcomes. It depends on the situation and requires human judgement. The value of human judgement can change the nature of the prediction machine. To give you a feel, Prediction within 5% accuracy - as soon as you put it in the wild, you have to retrain it, keep it actual, stick on the right path. These have to be considered
Action - what is the action dependant on the predictions generated? It may be a simple yes/no or more nuanced.
Outcome - an action always leads to an outcome. If you do not know what outcome you want, improvement is difficult, if not impossible
Last 3 relate to data: Input - To generate a useful prediction, you need to know what is going on at the time a decision needs to be made.
Training - to develop the prediction machine in the first place, you need to train a machine learning model. The richer and more varied that training data, the better your predictions will be out of the gate. If that data is not available, then you might have to deploy a mediocre prediction machine and wait for it to improve over time.
Feedback - This is data that you collect when the prediction machine is operating in real situations. Feedback data is often generated from a richer set of environments than training data.