SlideShare a Scribd company logo
1 of 31
Explainable Machine Learning with Shapley Values
Scott Lundberg
Senior Researcher
Microsoft
Explainable AI in practice
Model
development
model 22%
chance John will have
repayment problems
John, a bank customer
No loan
Why was I denied?
What is our
financial risk?
How do I
debug?
Accuracy = $
Interpretable Accurate
Complex model ✘ ✔
Simple model ✔ ✘
Interpretable or accurate: choose one.
😀 ⚖️ 💰
3
?
?
Complex models are
inherently complex!
But a single prediction involves only a
small piece of that complexity.
Inputvalue
Outputvalue
5
6
How did we get here?
Base rate Prediction for John
22%
𝑓 𝑥0
16%
𝐸[𝑓 𝑋 ]
7
16% 22%
𝐸[𝑓 𝑋 ] 𝑓 𝑥0
Base rate
𝜙0
Income not verified
18.2%
𝐸 𝑓 𝑋 𝑑𝑜(𝑋1 = 𝑥1)]
𝜙1
No recent account openings
21%
𝐸 𝑓 𝑋 𝑑𝑜(𝑋1,2 = 𝑥1,2)]
22.5%
𝐸 𝑓 𝑋 𝑑𝑜(𝑋1,2,3 = 𝑥1,2,3)]
18.5%
𝐸 𝑓 𝑋 𝑑𝑜(𝑋1,2,3,4 = 𝑥1,2,3,4)]
𝜙2
𝜙3𝜙4
𝜙5
DTI = 30
Delinquent 10 months ago
46 years of credit history
The order matters!
*Janzing et al. 2019
*
8
𝐸[𝑓 𝑋 ] 𝑓 𝑥0
𝜙0
𝜙1
No recent account openings
𝜙2
𝜙3
𝜙4
𝜙5
46 years of credit history
The order matters!
Nobel Prize in 2012
Lloyd Shapley
9
𝐸[𝑓 𝑋 ] 𝑓 𝑥0
𝜙0
𝜙1
𝜙2
𝜙3
𝜙4
𝜙5
Shapley properties
Additivity (local accuracy) – The sum of the local
feature attributions equals the difference between
rate and the model output.
1
10
𝐸[𝑓 𝑋 ] 𝑓 𝑥0
𝜙0
𝜙1
𝜙2
𝜙3
𝜙4
𝜙5
Shapley properties
Monotonicity (consistency) – If you change the
original model such that a feature has a larger
possible ordering, then that input’s attribution
decrease.
2
Violating consistency means you can’t trust feature orderings
based on your attributions.
…even within the same model!
11
𝐸[𝑓 𝑋 ] 𝑓 𝑥0
𝜙0
𝜙1
𝜙2
𝜙3
𝜙4
𝜙5
Shapley values result from averaging over all N! possible orderings.
(NP-hard)
ex = shap.TreeExplainer(model, …)
shap_values = ex.shap_values(X)
shap.force_plot(ex.expected_value, shap_values[john_ind,:], X.iloc[john_ind,:])
Why does 46 years of credit history increase
the risk of payment problems?
shap.dependence_plot(“Months of credit history”, shap_values, X)
The model is identifying
retirement-age individuals based
on their long credit histories!
Explain and
debug your models!
Explainable AI in practice
Model
development
Debugging/exploration
Monitoring
Model monitoring
Time
Training performance Test performance
Can you find where we introduced the bug?
16
Model monitoring
Now can you find where we introduced the bug?
17
False True
Model monitoring
Time
Transient electronic medical record
Time
18
False True
Model monitoring
Time
Gradual change in atrial fibrillation
ablation procedure durations
Time
19
False True
Explainable AI in practice
Model
development
Human/AI
collaboration
Regulatory
compliance
Debugging/exploration Customer retention Consumer explanations
Encoding prior beliefs
Monitoring Decision support Anti-discrimination
Risk managementHuman risk oversight
Scientific
discovery
Pattern discovery
Population subtyping
Signal recovery
Thank You
github.com/slundberg/shap
Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Mortality risk model
Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Mortality risk model
Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Mortality risk model
Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Mortality risk model
Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Mortality risk model
Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Mortality risk model
Reveal rare high-magnitude mortality effects
Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Conflates the
prevalence of an effect
with the
magnitude of an effect
Mortality risk model
Reveal rare high-magnitude mortality effects
Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Reveal rare high-magnitude mortality effects
Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Rare high magnitude effects
Reveal rare high-magnitude mortality effects
Global feature importance Local explanation summary(A)
(log relative risk of mortality)
Mortality model
(F/M)
Lots of ways to die young…Not many ways to live longer…

More Related Content

What's hot

Explainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretableExplainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretableAditya Bhattacharya
 
Large Language Models Bootcamp
Large Language Models BootcampLarge Language Models Bootcamp
Large Language Models BootcampData Science Dojo
 
An Introduction to XAI! Towards Trusting Your ML Models!
An Introduction to XAI! Towards Trusting Your ML Models!An Introduction to XAI! Towards Trusting Your ML Models!
An Introduction to XAI! Towards Trusting Your ML Models!Mansour Saffar
 
Explainability and bias in AI
Explainability and bias in AIExplainability and bias in AI
Explainability and bias in AIBill Liu
 
Explainable AI in Industry (KDD 2019 Tutorial)
Explainable AI in Industry (KDD 2019 Tutorial)Explainable AI in Industry (KDD 2019 Tutorial)
Explainable AI in Industry (KDD 2019 Tutorial)Krishnaram Kenthapadi
 
Explainable AI
Explainable AIExplainable AI
Explainable AIDinesh V
 
Explainable Machine Learning (Explainable ML)
Explainable Machine Learning (Explainable ML)Explainable Machine Learning (Explainable ML)
Explainable Machine Learning (Explainable ML)Hayim Makabee
 
Explainable AI in Industry (AAAI 2020 Tutorial)
Explainable AI in Industry (AAAI 2020 Tutorial)Explainable AI in Industry (AAAI 2020 Tutorial)
Explainable AI in Industry (AAAI 2020 Tutorial)Krishnaram Kenthapadi
 
Explainable AI in Industry (FAT* 2020 Tutorial)
Explainable AI in Industry (FAT* 2020 Tutorial)Explainable AI in Industry (FAT* 2020 Tutorial)
Explainable AI in Industry (FAT* 2020 Tutorial)Krishnaram Kenthapadi
 
Machine Learning Explanations: LIME framework
Machine Learning Explanations: LIME framework Machine Learning Explanations: LIME framework
Machine Learning Explanations: LIME framework Deep Learning Italia
 
Explainable AI in Healthcare
Explainable AI in HealthcareExplainable AI in Healthcare
Explainable AI in Healthcarevonaurum
 
Towards Human-Centered Machine Learning
Towards Human-Centered Machine LearningTowards Human-Centered Machine Learning
Towards Human-Centered Machine LearningSri Ambati
 
Responsible AI in Industry: Practical Challenges and Lessons Learned
Responsible AI in Industry: Practical Challenges and Lessons LearnedResponsible AI in Industry: Practical Challenges and Lessons Learned
Responsible AI in Industry: Practical Challenges and Lessons LearnedKrishnaram Kenthapadi
 
Responsible AI in Industry (ICML 2021 Tutorial)
Responsible AI in Industry (ICML 2021 Tutorial)Responsible AI in Industry (ICML 2021 Tutorial)
Responsible AI in Industry (ICML 2021 Tutorial)Krishnaram Kenthapadi
 
Explainable AI (XAI) - A Perspective
Explainable AI (XAI) - A Perspective Explainable AI (XAI) - A Perspective
Explainable AI (XAI) - A Perspective Saurabh Kaushik
 
Interpretability of machine learning
Interpretability of machine learningInterpretability of machine learning
Interpretability of machine learningDaiki Tanaka
 
Interpretable Machine Learning
Interpretable Machine LearningInterpretable Machine Learning
Interpretable Machine Learninginovex GmbH
 

What's hot (20)

Explainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretableExplainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretable
 
Large Language Models Bootcamp
Large Language Models BootcampLarge Language Models Bootcamp
Large Language Models Bootcamp
 
An Introduction to XAI! Towards Trusting Your ML Models!
An Introduction to XAI! Towards Trusting Your ML Models!An Introduction to XAI! Towards Trusting Your ML Models!
An Introduction to XAI! Towards Trusting Your ML Models!
 
Explainability and bias in AI
Explainability and bias in AIExplainability and bias in AI
Explainability and bias in AI
 
Explainable AI in Industry (KDD 2019 Tutorial)
Explainable AI in Industry (KDD 2019 Tutorial)Explainable AI in Industry (KDD 2019 Tutorial)
Explainable AI in Industry (KDD 2019 Tutorial)
 
Explainable AI
Explainable AIExplainable AI
Explainable AI
 
Explainable Machine Learning (Explainable ML)
Explainable Machine Learning (Explainable ML)Explainable Machine Learning (Explainable ML)
Explainable Machine Learning (Explainable ML)
 
Explainable AI in Industry (AAAI 2020 Tutorial)
Explainable AI in Industry (AAAI 2020 Tutorial)Explainable AI in Industry (AAAI 2020 Tutorial)
Explainable AI in Industry (AAAI 2020 Tutorial)
 
Explainable AI
Explainable AIExplainable AI
Explainable AI
 
Explainable AI
Explainable AIExplainable AI
Explainable AI
 
Explainable AI in Industry (FAT* 2020 Tutorial)
Explainable AI in Industry (FAT* 2020 Tutorial)Explainable AI in Industry (FAT* 2020 Tutorial)
Explainable AI in Industry (FAT* 2020 Tutorial)
 
Machine Learning Explanations: LIME framework
Machine Learning Explanations: LIME framework Machine Learning Explanations: LIME framework
Machine Learning Explanations: LIME framework
 
Explainable AI in Healthcare
Explainable AI in HealthcareExplainable AI in Healthcare
Explainable AI in Healthcare
 
Towards Human-Centered Machine Learning
Towards Human-Centered Machine LearningTowards Human-Centered Machine Learning
Towards Human-Centered Machine Learning
 
Responsible AI in Industry: Practical Challenges and Lessons Learned
Responsible AI in Industry: Practical Challenges and Lessons LearnedResponsible AI in Industry: Practical Challenges and Lessons Learned
Responsible AI in Industry: Practical Challenges and Lessons Learned
 
Responsible AI in Industry (ICML 2021 Tutorial)
Responsible AI in Industry (ICML 2021 Tutorial)Responsible AI in Industry (ICML 2021 Tutorial)
Responsible AI in Industry (ICML 2021 Tutorial)
 
Explainable AI (XAI) - A Perspective
Explainable AI (XAI) - A Perspective Explainable AI (XAI) - A Perspective
Explainable AI (XAI) - A Perspective
 
Interpretability of machine learning
Interpretability of machine learningInterpretability of machine learning
Interpretability of machine learning
 
Explainable AI (XAI)
Explainable AI (XAI)Explainable AI (XAI)
Explainable AI (XAI)
 
Interpretable Machine Learning
Interpretable Machine LearningInterpretable Machine Learning
Interpretable Machine Learning
 

Similar to Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shapley Values - H2O World NYC 2019

Practical Aspects of Stochastic Modeling.pptx
Practical Aspects of Stochastic Modeling.pptxPractical Aspects of Stochastic Modeling.pptx
Practical Aspects of Stochastic Modeling.pptxRon Harasym
 
Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)OPEN KNOWLEDGE GmbH
 
Using Machine Learning on AWS for Continuous Sentiment Analysis from Labeling...
Using Machine Learning on AWS for Continuous Sentiment Analysis from Labeling...Using Machine Learning on AWS for Continuous Sentiment Analysis from Labeling...
Using Machine Learning on AWS for Continuous Sentiment Analysis from Labeling...Amazon Web Services
 
Revolutionizing your Business with AI (AUC VLabs).pdf
Revolutionizing your Business with AI (AUC VLabs).pdfRevolutionizing your Business with AI (AUC VLabs).pdf
Revolutionizing your Business with AI (AUC VLabs).pdfOmar Maher
 
JDO 2019: Data Science for Developers - Matthew Renze
JDO 2019: Data Science for Developers -  Matthew RenzeJDO 2019: Data Science for Developers -  Matthew Renze
JDO 2019: Data Science for Developers - Matthew RenzePROIDEA
 
Rsqrd AI: Exploring Machine Learning Model Predictions
Rsqrd AI: Exploring Machine Learning Model PredictionsRsqrd AI: Exploring Machine Learning Model Predictions
Rsqrd AI: Exploring Machine Learning Model PredictionsSanjana Chowdhury
 
Machine learning in credit risk modeling : a James white paper
Machine learning in credit risk modeling : a James white paperMachine learning in credit risk modeling : a James white paper
Machine learning in credit risk modeling : a James white paperJames by CrowdProcess
 
Regoli fairness deep_learningitalia_20220127
Regoli fairness deep_learningitalia_20220127Regoli fairness deep_learningitalia_20220127
Regoli fairness deep_learningitalia_20220127Matteo Testi
 
Human-Machine Collaboration in Organizations: Impact of Algorithm Bias on De...
 Human-Machine Collaboration in Organizations: Impact of Algorithm Bias on De... Human-Machine Collaboration in Organizations: Impact of Algorithm Bias on De...
Human-Machine Collaboration in Organizations: Impact of Algorithm Bias on De...Anh Luong
 
Marketing Analytics with R Lifting Campaign Success Rates
Marketing Analytics with R Lifting Campaign Success RatesMarketing Analytics with R Lifting Campaign Success Rates
Marketing Analytics with R Lifting Campaign Success RatesRevolution Analytics
 
205250 crystall ball
205250 crystall ball205250 crystall ball
205250 crystall ballp6academy
 
Alexandr Honchar. Financial ML != ML and Finance
Alexandr Honchar. Financial ML != ML and FinanceAlexandr Honchar. Financial ML != ML and Finance
Alexandr Honchar. Financial ML != ML and FinanceLviv Startup Club
 
Patrick Hall, H2O.ai - The Case for Model Debugging - H2O World 2019 NYC
Patrick Hall, H2O.ai - The Case for Model Debugging - H2O World 2019 NYCPatrick Hall, H2O.ai - The Case for Model Debugging - H2O World 2019 NYC
Patrick Hall, H2O.ai - The Case for Model Debugging - H2O World 2019 NYCSri Ambati
 
When the AIs failures send us back to our own societal biases
When the AIs failures send us back to our own societal biasesWhen the AIs failures send us back to our own societal biases
When the AIs failures send us back to our own societal biasesClément DUFFAU
 
Project presentation by Debendra Adhikari
Project presentation by Debendra AdhikariProject presentation by Debendra Adhikari
Project presentation by Debendra AdhikariDEBENDRA ADHIKARI
 
Scale your Testing and Quality with Automation Engineering and ML - Carlos Ki...
Scale your Testing and Quality with Automation Engineering and ML - Carlos Ki...Scale your Testing and Quality with Automation Engineering and ML - Carlos Ki...
Scale your Testing and Quality with Automation Engineering and ML - Carlos Ki...QA or the Highway
 
How Four Statistical Rules Forecast Who Wins a Competitive Bid
How Four Statistical Rules Forecast Who Wins a Competitive BidHow Four Statistical Rules Forecast Who Wins a Competitive Bid
How Four Statistical Rules Forecast Who Wins a Competitive BidIntelCollab.com
 
Stochastic Modeling - Financial Reporting
Stochastic Modeling - Financial ReportingStochastic Modeling - Financial Reporting
Stochastic Modeling - Financial ReportingRon Harasym
 
Module 2: Machine Learning Deep Dive
Module 2:  Machine Learning Deep DiveModule 2:  Machine Learning Deep Dive
Module 2: Machine Learning Deep DiveSara Hooker
 

Similar to Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shapley Values - H2O World NYC 2019 (20)

Practical Aspects of Stochastic Modeling.pptx
Practical Aspects of Stochastic Modeling.pptxPractical Aspects of Stochastic Modeling.pptx
Practical Aspects of Stochastic Modeling.pptx
 
Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)Der Spagat zwischen BIAS und FAIRNESS (2024)
Der Spagat zwischen BIAS und FAIRNESS (2024)
 
Using Machine Learning on AWS for Continuous Sentiment Analysis from Labeling...
Using Machine Learning on AWS for Continuous Sentiment Analysis from Labeling...Using Machine Learning on AWS for Continuous Sentiment Analysis from Labeling...
Using Machine Learning on AWS for Continuous Sentiment Analysis from Labeling...
 
Revolutionizing your Business with AI (AUC VLabs).pdf
Revolutionizing your Business with AI (AUC VLabs).pdfRevolutionizing your Business with AI (AUC VLabs).pdf
Revolutionizing your Business with AI (AUC VLabs).pdf
 
JDO 2019: Data Science for Developers - Matthew Renze
JDO 2019: Data Science for Developers -  Matthew RenzeJDO 2019: Data Science for Developers -  Matthew Renze
JDO 2019: Data Science for Developers - Matthew Renze
 
Rsqrd AI: Exploring Machine Learning Model Predictions
Rsqrd AI: Exploring Machine Learning Model PredictionsRsqrd AI: Exploring Machine Learning Model Predictions
Rsqrd AI: Exploring Machine Learning Model Predictions
 
Machine learning in credit risk modeling : a James white paper
Machine learning in credit risk modeling : a James white paperMachine learning in credit risk modeling : a James white paper
Machine learning in credit risk modeling : a James white paper
 
Regoli fairness deep_learningitalia_20220127
Regoli fairness deep_learningitalia_20220127Regoli fairness deep_learningitalia_20220127
Regoli fairness deep_learningitalia_20220127
 
Human-Machine Collaboration in Organizations: Impact of Algorithm Bias on De...
 Human-Machine Collaboration in Organizations: Impact of Algorithm Bias on De... Human-Machine Collaboration in Organizations: Impact of Algorithm Bias on De...
Human-Machine Collaboration in Organizations: Impact of Algorithm Bias on De...
 
Marketing Analytics with R Lifting Campaign Success Rates
Marketing Analytics with R Lifting Campaign Success RatesMarketing Analytics with R Lifting Campaign Success Rates
Marketing Analytics with R Lifting Campaign Success Rates
 
205250 crystall ball
205250 crystall ball205250 crystall ball
205250 crystall ball
 
Alexandr Honchar. Financial ML != ML and Finance
Alexandr Honchar. Financial ML != ML and FinanceAlexandr Honchar. Financial ML != ML and Finance
Alexandr Honchar. Financial ML != ML and Finance
 
Patrick Hall, H2O.ai - The Case for Model Debugging - H2O World 2019 NYC
Patrick Hall, H2O.ai - The Case for Model Debugging - H2O World 2019 NYCPatrick Hall, H2O.ai - The Case for Model Debugging - H2O World 2019 NYC
Patrick Hall, H2O.ai - The Case for Model Debugging - H2O World 2019 NYC
 
Introduction To Six Sigma
Introduction To  Six  SigmaIntroduction To  Six  Sigma
Introduction To Six Sigma
 
When the AIs failures send us back to our own societal biases
When the AIs failures send us back to our own societal biasesWhen the AIs failures send us back to our own societal biases
When the AIs failures send us back to our own societal biases
 
Project presentation by Debendra Adhikari
Project presentation by Debendra AdhikariProject presentation by Debendra Adhikari
Project presentation by Debendra Adhikari
 
Scale your Testing and Quality with Automation Engineering and ML - Carlos Ki...
Scale your Testing and Quality with Automation Engineering and ML - Carlos Ki...Scale your Testing and Quality with Automation Engineering and ML - Carlos Ki...
Scale your Testing and Quality with Automation Engineering and ML - Carlos Ki...
 
How Four Statistical Rules Forecast Who Wins a Competitive Bid
How Four Statistical Rules Forecast Who Wins a Competitive BidHow Four Statistical Rules Forecast Who Wins a Competitive Bid
How Four Statistical Rules Forecast Who Wins a Competitive Bid
 
Stochastic Modeling - Financial Reporting
Stochastic Modeling - Financial ReportingStochastic Modeling - Financial Reporting
Stochastic Modeling - Financial Reporting
 
Module 2: Machine Learning Deep Dive
Module 2:  Machine Learning Deep DiveModule 2:  Machine Learning Deep Dive
Module 2: Machine Learning Deep Dive
 

More from Sri Ambati

H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo DayH2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo DaySri Ambati
 
Generative AI Masterclass - Model Risk Management.pptx
Generative AI Masterclass - Model Risk Management.pptxGenerative AI Masterclass - Model Risk Management.pptx
Generative AI Masterclass - Model Risk Management.pptxSri Ambati
 
AI and the Future of Software Development: A Sneak Peek
AI and the Future of Software Development: A Sneak Peek AI and the Future of Software Development: A Sneak Peek
AI and the Future of Software Development: A Sneak Peek Sri Ambati
 
LLMOps: Match report from the top of the 5th
LLMOps: Match report from the top of the 5thLLMOps: Match report from the top of the 5th
LLMOps: Match report from the top of the 5thSri Ambati
 
Building, Evaluating, and Optimizing your RAG App for Production
Building, Evaluating, and Optimizing your RAG App for ProductionBuilding, Evaluating, and Optimizing your RAG App for Production
Building, Evaluating, and Optimizing your RAG App for ProductionSri Ambati
 
Building LLM Solutions using Open Source and Closed Source Solutions in Coher...
Building LLM Solutions using Open Source and Closed Source Solutions in Coher...Building LLM Solutions using Open Source and Closed Source Solutions in Coher...
Building LLM Solutions using Open Source and Closed Source Solutions in Coher...Sri Ambati
 
Risk Management for LLMs
Risk Management for LLMsRisk Management for LLMs
Risk Management for LLMsSri Ambati
 
Open-Source AI: Community is the Way
Open-Source AI: Community is the WayOpen-Source AI: Community is the Way
Open-Source AI: Community is the WaySri Ambati
 
Building Custom GenAI Apps at H2O
Building Custom GenAI Apps at H2OBuilding Custom GenAI Apps at H2O
Building Custom GenAI Apps at H2OSri Ambati
 
Applied Gen AI for the Finance Vertical
Applied Gen AI for the Finance Vertical Applied Gen AI for the Finance Vertical
Applied Gen AI for the Finance Vertical Sri Ambati
 
Cutting Edge Tricks from LLM Papers
Cutting Edge Tricks from LLM PapersCutting Edge Tricks from LLM Papers
Cutting Edge Tricks from LLM PapersSri Ambati
 
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...Sri Ambati
 
Open Source h2oGPT with Retrieval Augmented Generation (RAG), Web Search, and...
Open Source h2oGPT with Retrieval Augmented Generation (RAG), Web Search, and...Open Source h2oGPT with Retrieval Augmented Generation (RAG), Web Search, and...
Open Source h2oGPT with Retrieval Augmented Generation (RAG), Web Search, and...Sri Ambati
 
KGM Mastering Classification and Regression with LLMs: Insights from Kaggle C...
KGM Mastering Classification and Regression with LLMs: Insights from Kaggle C...KGM Mastering Classification and Regression with LLMs: Insights from Kaggle C...
KGM Mastering Classification and Regression with LLMs: Insights from Kaggle C...Sri Ambati
 
LLM Interpretability
LLM Interpretability LLM Interpretability
LLM Interpretability Sri Ambati
 
Never Reply to an Email Again
Never Reply to an Email AgainNever Reply to an Email Again
Never Reply to an Email AgainSri Ambati
 
Introducción al Aprendizaje Automatico con H2O-3 (1)
Introducción al Aprendizaje Automatico con H2O-3 (1)Introducción al Aprendizaje Automatico con H2O-3 (1)
Introducción al Aprendizaje Automatico con H2O-3 (1)Sri Ambati
 
From Rapid Prototypes to an end-to-end Model Deployment: an AI Hedge Fund Use...
From Rapid Prototypes to an end-to-end Model Deployment: an AI Hedge Fund Use...From Rapid Prototypes to an end-to-end Model Deployment: an AI Hedge Fund Use...
From Rapid Prototypes to an end-to-end Model Deployment: an AI Hedge Fund Use...Sri Ambati
 
AI Foundations Course Module 1 - Shifting to the Next Step in Your AI Transfo...
AI Foundations Course Module 1 - Shifting to the Next Step in Your AI Transfo...AI Foundations Course Module 1 - Shifting to the Next Step in Your AI Transfo...
AI Foundations Course Module 1 - Shifting to the Next Step in Your AI Transfo...Sri Ambati
 
AI Foundations Course Module 1 - An AI Transformation Journey
AI Foundations Course Module 1 - An AI Transformation JourneyAI Foundations Course Module 1 - An AI Transformation Journey
AI Foundations Course Module 1 - An AI Transformation JourneySri Ambati
 

More from Sri Ambati (20)

H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo DayH2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
H2O.ai CEO/Founder: Sri Ambati Keynote at Wells Fargo Day
 
Generative AI Masterclass - Model Risk Management.pptx
Generative AI Masterclass - Model Risk Management.pptxGenerative AI Masterclass - Model Risk Management.pptx
Generative AI Masterclass - Model Risk Management.pptx
 
AI and the Future of Software Development: A Sneak Peek
AI and the Future of Software Development: A Sneak Peek AI and the Future of Software Development: A Sneak Peek
AI and the Future of Software Development: A Sneak Peek
 
LLMOps: Match report from the top of the 5th
LLMOps: Match report from the top of the 5thLLMOps: Match report from the top of the 5th
LLMOps: Match report from the top of the 5th
 
Building, Evaluating, and Optimizing your RAG App for Production
Building, Evaluating, and Optimizing your RAG App for ProductionBuilding, Evaluating, and Optimizing your RAG App for Production
Building, Evaluating, and Optimizing your RAG App for Production
 
Building LLM Solutions using Open Source and Closed Source Solutions in Coher...
Building LLM Solutions using Open Source and Closed Source Solutions in Coher...Building LLM Solutions using Open Source and Closed Source Solutions in Coher...
Building LLM Solutions using Open Source and Closed Source Solutions in Coher...
 
Risk Management for LLMs
Risk Management for LLMsRisk Management for LLMs
Risk Management for LLMs
 
Open-Source AI: Community is the Way
Open-Source AI: Community is the WayOpen-Source AI: Community is the Way
Open-Source AI: Community is the Way
 
Building Custom GenAI Apps at H2O
Building Custom GenAI Apps at H2OBuilding Custom GenAI Apps at H2O
Building Custom GenAI Apps at H2O
 
Applied Gen AI for the Finance Vertical
Applied Gen AI for the Finance Vertical Applied Gen AI for the Finance Vertical
Applied Gen AI for the Finance Vertical
 
Cutting Edge Tricks from LLM Papers
Cutting Edge Tricks from LLM PapersCutting Edge Tricks from LLM Papers
Cutting Edge Tricks from LLM Papers
 
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
Practitioner's Guide to LLMs: Exploring Use Cases and a Glimpse Beyond Curren...
 
Open Source h2oGPT with Retrieval Augmented Generation (RAG), Web Search, and...
Open Source h2oGPT with Retrieval Augmented Generation (RAG), Web Search, and...Open Source h2oGPT with Retrieval Augmented Generation (RAG), Web Search, and...
Open Source h2oGPT with Retrieval Augmented Generation (RAG), Web Search, and...
 
KGM Mastering Classification and Regression with LLMs: Insights from Kaggle C...
KGM Mastering Classification and Regression with LLMs: Insights from Kaggle C...KGM Mastering Classification and Regression with LLMs: Insights from Kaggle C...
KGM Mastering Classification and Regression with LLMs: Insights from Kaggle C...
 
LLM Interpretability
LLM Interpretability LLM Interpretability
LLM Interpretability
 
Never Reply to an Email Again
Never Reply to an Email AgainNever Reply to an Email Again
Never Reply to an Email Again
 
Introducción al Aprendizaje Automatico con H2O-3 (1)
Introducción al Aprendizaje Automatico con H2O-3 (1)Introducción al Aprendizaje Automatico con H2O-3 (1)
Introducción al Aprendizaje Automatico con H2O-3 (1)
 
From Rapid Prototypes to an end-to-end Model Deployment: an AI Hedge Fund Use...
From Rapid Prototypes to an end-to-end Model Deployment: an AI Hedge Fund Use...From Rapid Prototypes to an end-to-end Model Deployment: an AI Hedge Fund Use...
From Rapid Prototypes to an end-to-end Model Deployment: an AI Hedge Fund Use...
 
AI Foundations Course Module 1 - Shifting to the Next Step in Your AI Transfo...
AI Foundations Course Module 1 - Shifting to the Next Step in Your AI Transfo...AI Foundations Course Module 1 - Shifting to the Next Step in Your AI Transfo...
AI Foundations Course Module 1 - Shifting to the Next Step in Your AI Transfo...
 
AI Foundations Course Module 1 - An AI Transformation Journey
AI Foundations Course Module 1 - An AI Transformation JourneyAI Foundations Course Module 1 - An AI Transformation Journey
AI Foundations Course Module 1 - An AI Transformation Journey
 

Recently uploaded

Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsNanddeep Nachan
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century educationjfdjdjcjdnsjd
 
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Bhuvaneswari Subramani
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityWSO2
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MIND CTI
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodJuan lago vázquez
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamUiPathCommunity
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxRustici Software
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusZilliz
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra
 

Recently uploaded (20)

Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin WoodPolkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
 
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptxCorporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with Milvus
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 

Scott Lundberg, Microsoft Research - Explainable Machine Learning with Shapley Values - H2O World NYC 2019

  • 1. Explainable Machine Learning with Shapley Values Scott Lundberg Senior Researcher Microsoft
  • 2. Explainable AI in practice Model development
  • 3. model 22% chance John will have repayment problems John, a bank customer No loan Why was I denied? What is our financial risk? How do I debug? Accuracy = $
  • 4. Interpretable Accurate Complex model ✘ ✔ Simple model ✔ ✘ Interpretable or accurate: choose one. 😀 ⚖️ 💰 3 ? ?
  • 5. Complex models are inherently complex! But a single prediction involves only a small piece of that complexity. Inputvalue Outputvalue 5
  • 6. 6 How did we get here? Base rate Prediction for John 22% 𝑓 𝑥0 16% 𝐸[𝑓 𝑋 ]
  • 7. 7 16% 22% 𝐸[𝑓 𝑋 ] 𝑓 𝑥0 Base rate 𝜙0 Income not verified 18.2% 𝐸 𝑓 𝑋 𝑑𝑜(𝑋1 = 𝑥1)] 𝜙1 No recent account openings 21% 𝐸 𝑓 𝑋 𝑑𝑜(𝑋1,2 = 𝑥1,2)] 22.5% 𝐸 𝑓 𝑋 𝑑𝑜(𝑋1,2,3 = 𝑥1,2,3)] 18.5% 𝐸 𝑓 𝑋 𝑑𝑜(𝑋1,2,3,4 = 𝑥1,2,3,4)] 𝜙2 𝜙3𝜙4 𝜙5 DTI = 30 Delinquent 10 months ago 46 years of credit history The order matters! *Janzing et al. 2019 *
  • 8. 8 𝐸[𝑓 𝑋 ] 𝑓 𝑥0 𝜙0 𝜙1 No recent account openings 𝜙2 𝜙3 𝜙4 𝜙5 46 years of credit history The order matters! Nobel Prize in 2012 Lloyd Shapley
  • 9. 9 𝐸[𝑓 𝑋 ] 𝑓 𝑥0 𝜙0 𝜙1 𝜙2 𝜙3 𝜙4 𝜙5 Shapley properties Additivity (local accuracy) – The sum of the local feature attributions equals the difference between rate and the model output. 1
  • 10. 10 𝐸[𝑓 𝑋 ] 𝑓 𝑥0 𝜙0 𝜙1 𝜙2 𝜙3 𝜙4 𝜙5 Shapley properties Monotonicity (consistency) – If you change the original model such that a feature has a larger possible ordering, then that input’s attribution decrease. 2 Violating consistency means you can’t trust feature orderings based on your attributions. …even within the same model!
  • 11. 11 𝐸[𝑓 𝑋 ] 𝑓 𝑥0 𝜙0 𝜙1 𝜙2 𝜙3 𝜙4 𝜙5 Shapley values result from averaging over all N! possible orderings. (NP-hard)
  • 12.
  • 13. ex = shap.TreeExplainer(model, …) shap_values = ex.shap_values(X) shap.force_plot(ex.expected_value, shap_values[john_ind,:], X.iloc[john_ind,:]) Why does 46 years of credit history increase the risk of payment problems?
  • 14. shap.dependence_plot(“Months of credit history”, shap_values, X) The model is identifying retirement-age individuals based on their long credit histories! Explain and debug your models!
  • 15. Explainable AI in practice Model development Debugging/exploration Monitoring
  • 16. Model monitoring Time Training performance Test performance Can you find where we introduced the bug? 16
  • 17. Model monitoring Now can you find where we introduced the bug? 17 False True
  • 18. Model monitoring Time Transient electronic medical record Time 18 False True
  • 19. Model monitoring Time Gradual change in atrial fibrillation ablation procedure durations Time 19 False True
  • 20. Explainable AI in practice Model development Human/AI collaboration Regulatory compliance Debugging/exploration Customer retention Consumer explanations Encoding prior beliefs Monitoring Decision support Anti-discrimination Risk managementHuman risk oversight Scientific discovery Pattern discovery Population subtyping Signal recovery
  • 22. Global feature importance Local explanation summary(A) (log relative risk of mortality) Mortality model (F/M) Mortality risk model
  • 23. Global feature importance Local explanation summary(A) (log relative risk of mortality) Mortality model (F/M) Mortality risk model
  • 24. Global feature importance Local explanation summary(A) (log relative risk of mortality) Mortality model (F/M) Mortality risk model
  • 25. Global feature importance Local explanation summary(A) (log relative risk of mortality) Mortality model (F/M) Mortality risk model
  • 26. Global feature importance Local explanation summary(A) (log relative risk of mortality) Mortality model (F/M) Mortality risk model
  • 27. Global feature importance Local explanation summary(A) (log relative risk of mortality) Mortality model (F/M) Mortality risk model
  • 28. Reveal rare high-magnitude mortality effects Global feature importance Local explanation summary(A) (log relative risk of mortality) Mortality model (F/M) Conflates the prevalence of an effect with the magnitude of an effect Mortality risk model
  • 29. Reveal rare high-magnitude mortality effects Global feature importance Local explanation summary(A) (log relative risk of mortality) Mortality model (F/M)
  • 30. Reveal rare high-magnitude mortality effects Global feature importance Local explanation summary(A) (log relative risk of mortality) Mortality model (F/M) Rare high magnitude effects
  • 31. Reveal rare high-magnitude mortality effects Global feature importance Local explanation summary(A) (log relative risk of mortality) Mortality model (F/M) Lots of ways to die young…Not many ways to live longer…

Editor's Notes

  1. We'll start with a simple motivational example. This is John, a typical bank customer. Like many consumers today, when he applies for a loan, information about him is sent through a predictive model XX. This model is designed to calculate the risk that John will have repayment problems, XX which, unfortunately for John, is 55%. And because his risk is high, the bank declines his loan application XX. A natural first question John has XX is “Why?” The bank XX also want to know why, because these are key business decisions. But unfortunately for the data scientist that built this model XX, she used a complex model and so these questions are hard to answer XX …. XX
  2. The reason the data scientist used a complex model is because they are often very accurate on large data sets XX, but that same complexity also makes them hard to interpret XX. In contrast, simple models are easy to interpret XX, but are often less accurate XX. This leads to a trade-off XX between interpretability and accuracy. This trade-off is particularly painful for the bank since accuracy directly corresponds to profitability XX, while interpretability has important implications for customer satisfaction XX and even legality. This tradeoff effects a wide range of applications, and so many recent methods have been developed to address it. XX
  3. These methods do not try and make an entire complex model interpretable, because there is inherently too much to succinctly explain. Instead they focus on explaining a single prediction XX, because mapping a single input to an output involves only a small part of the complexity of the overall model. XX
  4. To see, let’s return to John and answer why his loan application was denied. To explain his denial it is important to start with the base rate of loan repayment problems XX, denoted here by the expected value of the model’s output. To explain John's risk XX, we need to explain how we got from the base rate XX, to his risk of 55%. XX
  5. To see, let’s return to John and answer why his loan application was denied. To explain his denial it is important to start with the base rate of loan repayment problems XX, denoted here by the expected value of the model’s output. To explain John's risk XX, we need to explain how we got from the base rate XX, to his risk of 55%. XX
  6. To see, let’s return to John and answer why his loan application was denied. To explain his denial it is important to start with the base rate of loan repayment problems XX, denoted here by the expected value of the model’s output. To explain John's risk XX, we need to explain how we got from the base rate XX, to his risk of 55%. XX
  7. To see, let’s return to John and answer why his loan application was denied. To explain his denial it is important to start with the base rate of loan repayment problems XX, denoted here by the expected value of the model’s output. To explain John's risk XX, we need to explain how we got from the base rate XX, to his risk of 55%. XX
  8. To see, let’s return to John and answer why his loan application was denied. To explain his denial it is important to start with the base rate of loan repayment problems XX, denoted here by the expected value of the model’s output. To explain John's risk XX, we need to explain how we got from the base rate XX, to his risk of 55%. XX
  9. To see, let’s return to John and answer why his loan application was denied. To explain his denial it is important to start with the base rate of loan repayment problems XX, denoted here by the expected value of the model’s output. To explain John's risk XX, we need to explain how we got from the base rate XX, to his risk of 55%. XX
  10. Make title shorter
  11. Focus more on for each
  12. What is an EMR?
  13. Make title shorter
  14. Make title shorter
  15. Make title shorter
  16. Make title shorter
  17. Make title shorter
  18. Make title shorter
  19. Make title shorter
  20. Make title shorter
  21. Make title shorter
  22. Make title shorter
  23. Make title shorter