Responsible AI in Industry: Practical Challenges and Lessons Learned

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Krishnaram Kenthapadi
Principal Scientist, Amazon AWS AI
Responsible AI in Industry:
Practical Challenges and Lessons Learned
Invited talks @
• Responsible Use of Data series, The Academic Fringe Festival, Delft
University of Technology, Jul’2021
• Flipkart Data Science Conference, Jun’2021
• Syracuse University, May’2021
• The University of Waterloo ML+Logic Colloquium, Apr’2021
• The University of British Columbia, Vancouver, Mar’2021

Massachusetts Group
Insurance Commission
(1997): Anonymized
medical history of state
employees
William Weld vs
Latanya Sweeney
Latanya Sweeney (MIT
grad student): $20 –
Cambridge voter roll
born July 31, 1945
resident of 02138

64%
Uniquely identifiable with ZIP
+ birth date + gender (in the
US population)
Golle, “Revisiting the Uniqueness of Simple Demographics in the US Population”, WPES 2006

A History of Privacy Failures …
Credit: Kobbi Nissim,Or Sheffet

• Ethical challenges
posed by AI systems
• Inherent biases present
in society
• Reflected in training
data
• AI/ML models prone to
amplifying such biases
Algorithmic Bias

Laws against Discrimination
Immigration Reform and Control Act
Citizenship
Rehabilitation Act of 1973;
Americans with Disabilities Act
of 1990
Disability status
Civil Rights Act of 1964
Race
Age Discrimination in Employment Act of
1967
Age
Equal Pay Act of 1963;
Civil Rights Act of 1964
Sex
And more...

Fairness Privacy
Transparency Explainability

Motivation & Business Opportunities
Regulatory. We need to understand why the ML model made a given prediction
and also whether the prediction it made was free from bias, both in training and at
inference.
Business. Providing explanations to internal teams (loan officers, customer service
rep, compliance teams) and end users/customers
Data Science. Improving models through better feature engineering and training
data generation, understanding failure modes of the model, debugging model
predictions, etc.

© 2020, Amazon Web Services, Inc. or its Affiliates.
Amazon
SageMaker
VISION SPEECH TEXT SEARCH CHATBOTS PERSONALIZATION FORECASTING FRAUD CONTACT CENTERS
Deep
Learning
AMIs &
Containers
GPUs &
CPUs
Elastic
Inference
Trainium Inferentia FPGA
AI SERVICES
ML SERVICES
FRAMEWORKS & INFRASTRUCTURE
DeepGraphLibrary
Amazon
Rekognition
Amazon
Polly
Amazon
Transcribe
+Medical
Amazon
Lex
Amazon
Personalize
Amazon
Forecast
Amazon
Comprehend
+Medical
Amazon
Textract
Amazon
Kendra
Amazon
CodeGuru
Amazon
Fraud Detector
Amazon
Translate
INDUSTRIAL AI CODE AND DEVOPS
NEW
Amazon
DevOps Guru
Voice ID
For AmazonConnect
Contact Lens
NEW
Amazon
Monitron
NEW
AWS Panorama
+ Appliance
NEW
Amazon Lookout
for Vision
NEW
Amazon Lookout
for Equipment
Scaling Fairness, Explainability & Privacy across the AWS ML Stack
NEW
Amazon
HealthLake
HEALTHCARE AI
NEW
Amazon Lookout
for Metrics
ANOMALY DETECTION
Amazon
Transcribe
for Medical
Amazon
Comprehend
for Medical
Label
data
NEW
Aggregate &
prepare data
NEW
Store & share
features
Auto ML Spark/R
NEW
Detect
bias
Visualize in
notebooks
Pick
algorithm
Train
models
Tune
parameters
NEW
Debug &
profile
Deploy in
production
Manage
& monitor
NEW
CI/CD
Human review
review
NEW: Model management for edge devices
NEW: SageMaker JumpStart
SAGEMAKER STUDIO IDE

LinkedIn operates the largest professional
network on the Internet
Tell your
story
740M members
55M+
companies are
represented
on LinkedIn
90K
schools listed
(high school &
college)
36K
skills listed
14M+
open jobs
on
LinkedIn
Jobs
280B
Feed updates

Talk Outline*
• Fairness-aware ML: An overview
• Explainable AI: An overview
• Privacy-preserving ML: An overview
• Responsible AI tools
• Case studies
• Key takeaways * For a longer version, please see our tutorial titled,
“Responsible AI in Industry”:
https://sites.google.com/view/ResponsibleAITutorial

What is Explainable AI?
Data
Opaque
AI
AI
product
Confusion with Today’s Opaque AI
● Why did you do that?
● Why did you not do that?
● When do you succeed or fail?
● How do I correct an error?
Opaque AI
Prediction,
Recommendation
Clear & Transparent Predictions
● I understand why
● I understand why not
● I know why you succeed or fail
● I understand, so I trust you
Explainable AI
Data
Explainable
AI
Explainable
AI Product
Prediction
Explanation
Feedback

- Humans may have follow-up questions
- Explanations cannot answer all users’ concerns
Weld, D., and Gagan Bansal. "The challenge of crafting
intelligible intelligence." Communications of the ACM (2018).
Example of an End-to-End XAI System

Neural Net
CNN
GAN
RNN
Ensemble
Method
Random
Forest
XGB
Statistical
Model
AOG
SVM
Graphical Model
Bayesian
Belief Net
SLR
CRF HBN
MLN
Markov
Model
Decision
Tree
Linear
Model
Non-Linear
functions
Polynomial
functions
Quasi-Linear
functions
Accuracy
Explainability
Interpretability
Learning
• Challenges:
• Supervised
• Unsupervised learning
• Approach:
• Representation
Learning
• Stochastic selection
• Output:
• Correlation
• No causation
How to Explain? Accuracy vs. Explainability

Text
Tabular
Images
On the Role of Data in XAI

Explainable AI Methods
Approach 1: Post-hoc explain a given AI model
● Individual prediction explanations in terms of input features, influential examples,
concepts, local decision rules
● Global prediction explanations in terms of entire model in terms of partial
dependence plots, global feature importance, global decision rules
Approach 2: Build an interpretable model
● Logistic regression, Decision trees, Decision lists and sets, Generalized Additive
Models (GAMs)
19

Credit: Slide based on https://twitter.com/chandan_singh96/status/1138811752769101825
Integrated
Gradients
Opaque,
complex
model

Explaining Model Predictions: Credit Lending
21
Fair lending laws [ECOA, FCRA] require credit decisions to be explainable

Feature Attribution Problem
Attribute a model’s prediction on an input to features of the input
Examples
● Attribute a lending model’s prediction to its features
● Attribute an object recognition model’s prediction to its pixels
● Attribute a text sentiment model’s prediction to individual words
Several attribution methods:
● Ablations
● Gradient based methods (specific to differentiable models)
● Score back-propagation based methods (specific to neural networks)
● Game theory based methods (Shapley values)
22

Application of Attributions
● Debugging model predictions
E.g., Attribution an image misclassification to the pixels responsible for it
● Generating an explanation for the end-user
E.g., Expose attributions for a lending prediction to the end-user
● Analyzing model robustness
E.g., Craft adversarial examples using weaknesses surfaced by attributions
● Extract rules from the model
E.g., Combine attribution to craft rules (pharmacophores) capturing prediction
logic of a drug screening network
23

Next few slides
We will cover the following attribution methods**
● Ablations
● Gradient based methods (specific to differentiable models)
● Score Backpropagation based methods (specific to NNs)
We will also discuss game theory (Shapley value) in attributions
**Not a complete list!
See Ancona et al. [ICML 2019], Guidotti et al. [arxiv 2018] for a comprehensive survey 24

Ablations
Drop each feature and attribute the change in prediction to that feature
Pros:
● Simple and intuitive to interpret
Cons:
● Unrealistic inputs
● Improper accounting of interactive features
● Can be computationally expensive
25

Feature*Gradient
Attribution to a feature is feature value times gradient, i.e., xi* 𝜕y/𝜕xi
● Gradient captures sensitivity of output w.r.t. feature
● Equivalent to Feature*Coefficient for linear models
○ First-order Taylor approximation of non-linear models
● Popularized by SaliencyMaps [NeurIPS 2013], Baehrens et al. [JMLR 2010]
26
Gradients in the
vicinity of the input
seem like noise?

Local linear approximations can be too local
27
score
“fireboat-ness” of image
Interesting gradients
uninteresting gradients
(saturation)
1.0
0.0

Score Back-Propagation based Methods
Re-distribute the prediction score through the neurons in the network
● LRP [JMLR 2017], DeepLift [ICML 2017], Guided BackProp [ICLR 2014]
Easy case: Output of a neuron is a linear function
of previous neurons (i.e., ni = ⅀ wij * nj)
e.g., the logit neuron
● Re-distribute the contribution in proportion to
the coefficients wij
28
Image credit heatmapping.org

Tricky case: Output of a neuron is a non-linear
function, e.g., ReLU, Sigmoid, etc.
● Guided BackProp: Only consider ReLUs
that are on (linear regime), and which
contribute positively
● LRP: Use first-order Taylor decomposition to
linearize activation function
● DeepLift: Distribute activation difference
relative a reference point in proportion to
edge weights 29

Pros:
● Conceptually simple
● Methods have been empirically validated to
yield sensible result
Cons:
● Hard to implement, requires instrumenting
the model
● Often breaks implementation invariance
Think: F(x, y, z) = x * y *z and
G(x, y, z) = x * (y * z)

Baselines and additivity
● When we decompose the score via backpropagation, we imply a normative
alternative called a baseline
○ “Why Pr(fireboat) = 0.91 [instead of 0.00]”
● Common choice is an informationless input for the model
○ E.g., All black or all white image for image models
○ E.g., Empty text or zero embedding vector for text models
● Additive attributions explain F(input) - F(baseline) in terms of input features

score
intensity
Interesting gradients
uninteresting gradients
(saturation)
1.0
0.0
Baseline … scaled inputs ...
… gradients of scaled inputs
….
Input
Another approach: gradients at many points

IG(input, base) ::= (input - base) * ∫0 -1▽F(𝛂*input + (1-𝛂)*base) d𝛂
Original image Integrated Gradients
Integrated Gradients [ICML 2017]
Integrate the gradients along a straight-line path from baseline to input

34
© 2020 Amazon Web Services, Inc. or its affiliates. All rights reserved |
Feature Attributions using Shapley Values
Coalition Game
● Players collaborating to generate some payoff
● Shapley value: Fair way to quantify each player’s contribution to the game’s
payoff
34

35
Shapley Value: Intuition
35

36
Shapley Value [Annals of Mathematical studies,1953]
=
Average over
all player
orderings &
subsets
SC M / {i}
Marginal contribution
Coalition contribution with and without this player
f – payoff of the game
M – all players
S – subset of players
i – specific player
Fair way to quantify each player’s
contribution to the game’s payoff

37
Shapley Value Justification
Shapley values are unique under four simple axioms
● Dummy: If a player never contributes to the game then it must receive zero
attribution
● Efficiency: Attributions must add to the total payoff
● Symmetry: Symmetric players must receive equal attribution
● Linearity: Attribution for the (weighted) sum of two games must be the same
as the (weighted) sum of the attributions for each of the games
37

38
Explaining Model Predictions using Shapley Values
f
Game payoff
Game theory
All players
Subset of players
Specific player
Instance being explained
Model prediction
All features
Subset of features
Specific feature
Instance being explained
M
S
i
x
Machine learning
https://github.com/slundberg/shap
Lundberg et al: NeurIPS (2017)

39
Explaining Model Predictions: SHAP Algorithm
How it works
=
Average over
all feature
orderings &
subsets
SC M / {i}
f – model prediction
M – all features
S – subset of features
i – specific feature
x – instance being explained

40
Different solutions for runtime and algorithmic challenges
=
Average over
all feature
orderings &
subsets
SC M / {i}
M – all features
Exponential time How to run a model
with missing features?

41
Different solutions for runtime and algorithmic challenges
=
Average over
all feature
orderings &
subsets
SC M / {i}
M – all features
Exponential time
How to run a model
with missing features?
Shap.KernelExplainer
• All models
• Explanations
Sample subsets,
defaults to
2|M| + 2048
Fill missing
values from a
background dataset
• Training set
• Median of the dataset
• K-means representatives
• All-black image
Kernel
Open-source SHAP implementation

Baselines (or Norms) are essential to explanations [Kahneman-Miller 86]
● E.g., A man suffers from indigestion. Doctor blames it to a stomach ulcer. Wife
blames it on eating turnips. Both are correct relative to their baselines.
● The baseline may also be an important analysis knob.
Attributions are contrastive, whether we think about it or not.
Lesson learned: Baselines are important

Some limitations and caveats for attributions

Some things that are missing:
● Feature interactions (ignored or averaged out)
● What training examples influenced the prediction (training agnostic)
● Global properties of the model (prediction-specific)
An instance where attributions are useless:
● A model that predicts TRUE when there are even number of black pixels and
FALSE otherwise
Attributions don’t explain everything

Attributions are for human consumption
Naive scaling of attributions
from 0 to 255
Attributions have a large
range and long tail
across pixels
After clipping attributions
at 99% to reduce range
● Humans interpret attributions and generate insights
○ Doctor maps attributions for x-rays to pathologies
● Visualization matters as much as the attribution technique

Privacy-preserving ML: An Overview

What is Privacy?
• Right of/to privacy
• “Right to be let alone” [L. Brandeis & S. Warren, 1890]
• “No one shall be subjected to arbitrary interference with [their] privacy,
family, home or correspondence, nor to attacks upon [their] honor and
reputation.” [The United Nations Universal Declaration of Human Rights]
• “The right of a person to be free from intrusion into or publicity concerning
matters of a personal nature” [Merriam-Webster]
• “The right not to have one's personal matters disclosed or publicized; the
right to be left alone” [Nolo’s Plain-English Law Dictionary]

Data Privacy (or Information Privacy)
• “The right to have some control over how your personal information is
collected and used” [IAPP]
• “Privacy has fast-emerged as perhaps the most significant consumer
protection issue—if not citizen protection issue—in the global
information economy” [IAPP]

Data Privacy vs. Security
• Data privacy: use & governance of personal data
• Data security: protecting data from malicious attacks & the exploitation
of stolen data for profit
• Security is necessary, but not sufficient for addressing privacy.

Data Privacy:Technical Problem
Given a dataset with sensitive personal information, how can we compute
and release functions of the dataset while protecting individual privacy?
Credit: Kobbi Nissim

Lessons Learned …
• Attacker’s advantage: Auxiliary information; high dimensionality;
enough to succeed on a small fraction of inputs; active; observant …
• Unanticipated privacy failures from new attack methods
• Need for rigorous privacy notions & techniques

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Threat Models

Threat Models
User Access Only
• Users store their
data
• Noisy data or
analytics transmitted
Trusted Curator
• Stored by organization
• Managed only by a
trusted curator/admin
• Access only to noisy
analytics or synthetic
data
External Threat
• Stored by organization
• Organization has
access
• Only privacy enabled
models deployed

Defining Privacy
59
Curator
Curator
+ your data
- your data

Differential Privacy
60
Databases D and D′ are neighbors if they differ in one person’s data.
Differential Privacy: The distribution of the curator’s output M(D) on database
D is (nearly) the same as M(D′).
Curator
+ your data
- your data
Dwork, McSherry, Nissim, Smith [TCC 2006]
Curator

(ε, 𝛿)-Differential Privacy: The distribution of the curator’s output M(D) on
database D is (nearly) the same as M(D′).
Differential Privacy
61
Curator
Parameter ε quantifies
information leakage
∀S: Pr[M(D)∊S] ≤ exp(ε) ∙ Pr[M(D′)∊S]＋𝛿.
Curator
Parameter 𝛿 gives
some slack
Dwork, Kenthapadi, McSherry, Mironov, Naor [EUROCRYPT 2006]
+ your data
- your data

Differential Privacy: Random Noise Addition
If ℓ1-sensitivity of f : D → ℝn:
maxD,D′ ||f(D) − f(D′)||1 = s,
then adding Laplacian noise to true output
f(D) + Laplacen(s/ε)
offers (ε,0)-differential privacy.

Examples of Attacks
• Membership inference attacks
• Reconstruction attacks
• Model inversion attacks
• Different attacks correspond to different threat models

Privacy-preserving MLTechniques /Tools
• Differentially private model training
• Differentially private stochastic gradient descent
• PATE
• Differentially private synthetic data generation
• Federated learning
• Trusted execution environments (secure enclaves)
• Secure multiparty computation
• Homomorphic encryption
• Different techniques applicable under different threat models
• May need a hybrid approach

Fairness in
AI @
LinkedIn
Fairness-aware
Talent Search
Ranking*
*Work done while at LinkedIn

Guiding Principle:
“Diversity by Design”

Insights to
Identify Diverse
Talent Pools
Representative
Talent Search
Results
Diversity
Learning
Curriculum
“Diversity by Design” in LinkedIn’s Talent
Solutions

Representative Ranking for Talent Search
S. C. Geyik, S. Ambler,
K. Kenthapadi, Fairness-
Aware Ranking in Search &
Recommendation Systems with
Application to LinkedIn Talent
Search, KDD’19.
[Microsoft’s AI/ML
conference
(MLADS’18). Distinguished
Contribution Award]
Building Representative
Talent Search at LinkedIn
(LinkedIn engineering blog)

Intuition for Measuring and Achieving Representativeness
Ideal: Top ranked results should follow a desired distribution on
gender/age/…
E.g., same distribution as the underlying talent pool
Inspired by “Equal Opportunity” definition [Hardt et al, NIPS’16]
Defined measures (skew, divergence) based on this intuition

Measuring (Lack of) Representativeness
Skew@k
(Logarithmic) ratio of the proportion of candidates having a given attribute
value among the top k ranked results to the corresponding desired proportion
Variants:
MinSkew: Minimum over all attribute values
MaxSkew: Maximum over all attribute values
Normalized Discounted Cumulative Skew
Normalized Discounted Cumulative KL-divergence

Fairness-aware Reranking Algorithm (Simplified)
Partition the set of potential candidates into different buckets for
each attribute value
Rank the candidates in each bucket according to the scores assigned
by the machine-learned model
Merge the ranked lists, balancing the representation requirements
and the selection of highest scored candidates
Representation requirement: Desired distribution on gender/age/…
Algorithmic variants based on how we choose the next attribute

Validating Our Approach
Gender Representativeness
Over 95% of all searches are representative compared to the qualified
population of the search
Business Metrics
A/B test over LinkedIn Recruiter users for two weeks
No significant change in business metrics (e.g., # InMails sent or accepted)
Ramped to 100% of LinkedIn Recruiter users worldwide

Lessons
learned
• Post-processing approach desirable
• Model agnostic
• Scalable across different model
choices for our application
• Acts as a “fail-safe”
• Robust to application-specific business
logic
• Easier to incorporate as part of existing
systems
• Build a stand-alone service or
component for post-processing
• No significant modifications to the
existing components
• Complementary to efforts to reduce bias
from training data & during model
training
• Collaboration/consensus across key

Evaluating Fairness Using Permutations Tests
[DiCiccio, Vasudevan, Basu, Kenthapadi, Agarwal, KDD’20]
• Is the measured discrepancy across different groups statistically
significant?
• Use statistical hypothesis tests!
• Can we perform hypothesis tests in a metric-agnostic manner?
• Non-parametric tests can help!
• Permutation testing framework

Brief Review of Permutation Tests
Observe data from two populations:
and
Are the populations the same?
A reasonable test statistic might be

Brief Review of Permutation Tests (Continued)
A p-value is the chance of observing a test statistic at least as
“extreme” as the value we actually observed
Permutation test approach:
●Randomly shuffle the population designations of the
observations
●Recompute the test statistic T
●Repeat many times
Permutation p-value: the proportion of permuted datasets resulting in
a larger test statistic than the original value
This test is exact!

A Fairness Example
Consider testing whether the true positive rate of a classifier is equal
between two groups
Test Statistic: difference in proportion of negative labeled
observations that are classified as positive between the two groups
Permutation test: Randomly reshuffle group labels, recompute test
statistic

Permutations Tests for Evaluating Fairness in ML Models
• Issues with classical permutation test
• Want to check: just equality of the fairness metric (e.g., false positive rate)
across groups, and not if the two groups have identical distribution
• Exact for the strong null hypothesis …
• … but may not be valid (even asymptotically) for the weak null hypothesis
• Our paper: A fix for this issue
• Choose a pivotal statistic (asymptotically distribution-free; does not
depend on the observed data’s distribution)
• E.g., Studentize the test statistic

Engineering for Fairness in AI Lifecycle
S.Vasudevan, K. Kenthapadi, LiFT: A Scalable Framework for Measuring Fairness in ML Applications, CIKM’20
https://github.com/linkedin/LiFT

LiFT System Architecture [Vasudevan & Kenthapadi, CIKM’20]
•Flexibility of Use
(Platform agnostic)
•Ad-hoc exploratory
analyses
•Deployment in offline
workflows
•Integration with ML
Frameworks
•Scalability
•Diverse fairness
metrics
•Conventional fairness
metrics
•Benefit metrics
•Statistical tests

Acknowledgements
LinkedIn Talent Solutions Diversity team, Hire & Careers AI team, Anti-abuse AI team, Data Science
Applied Research team
Special thanks to Deepak Agarwal, Parvez Ahammad, Stuart Ambler, Kinjal Basu, Jenelle Bray, Erik
Buchanan, Bee-Chung Chen, Fei Chen, Patrick Cheung, Gil Cottle, Cyrus DiCiccio, Patrick Driscoll,
Carlos Faham, Nadia Fawaz, Priyanka Gariba, Meg Garlinghouse, Sahin Cem Geyik, Gurwinder
Gulati, Rob Hallman, Sara Harrington, Joshua Hartman, Daniel Hewlett, Nicolas Kim, Rachel Kumar,
Monica Lewis, Nicole Li, Heloise Logan, Stephen Lynch, Divyakumar Menghani, Varun Mithal,
Arashpreet Singh Mor, Tanvi Motwani, Preetam Nandy, Lei Ni, Nitin Panjwani, Igor Perisic, Hema
Raghavan, Romer Rosales, Guillaume Saint-Jacques, Badrul Sarwar, Amir Sepehri, Arun Swami, Ram
Swaminathan, Grace Tang, Ketan Thakkar, Sriram Vasudevan, Janardhanan Vembunarayanan, James
Verbus, Xin Wang, Hinkmond Wong, Ya Xu, Lin Yang, Yang Yang, Chenhui Zhai, Liang Zhang, Yani
Zhang

88
88
Detect bias in ML models and understand model predictions
Amazon SageMaker Clarify

89
Predictive
Maintenance
Manufacturing,
Automotive, IoT
Demand
Forecasting
Retail, Consumer
Goods, Manufacturing
Fraud
Detection
Financial Services,
Online Retail
Credit Risk
Prediction
Financial Services,
Retail
Extract and
Analyze Data
from Documents
Healthcare, Legal,
Media/Ent, Education
Computer
Vision
Healthcare, Pharma,
Manufacturing
Autonomous
Driving
Automotive,
Transportation
Personalized
Recommendations
Media & Entertainment,
Retail, Education
Churn
Prediction
Retail, Education,
Software & Internet
https://aws.amazon.c
om/sagemaker/gettin
g-started
Amazon SageMaker Customer ML Use cases

90
Bias and Explainability: Challenges
1 Without detection, it is hard to know if bias has entered an ML model:
• Imbalances may be present in the initial dataset
• Bias may develop during training
• Bias may develop over time after model deployment
2 Machine learning models are often complex & opaque, making explainability critical:
• Regulations may require companies to be able to explain model predictions
• Internal stakeholders and customers may need explanations for model behavior
• Data science teams can improve models if they understand model behavior

91
Amazon
SageMaker
Clarify
Detect bias in ML models
and understand model
predictions
Detect bias during data preparation
Identify imbalances in data
Evaluate the degree to which various types of bias are present in your model
Check your trained model for bias
Understand the relative importance of each feature to your model’s behavior
Explain overall model behavior
Understand the relative importance of each feature for individual inferences
Explain individual predictions
Provide alerts and detect drift over time due to changing real-world conditions
Detect drift in bias and model behavior over time
Generate automated reports
Produce reports on bias and explanations to support internal presentations

92
SageMaker Clarify works across the ML lifecycle
Collect and prepare
training data
Train and tune
model
Evaluate and qualify
model
Deploy model
in production
Monitor model
in production
Measure Bias
Metrics
Measure and Tune
Bias Metrics
Measure
Explainability
Metrics
Catalog Model
Metrics
Measure Bias
Metrics
Measure
Explainability
Metrics
Monitor Bias Metric
Drift
Monitor
Explainability Drift
SageMaker
Data Wrangler
SageMaker
Training
Autopilot
Hyperparameter Tuning
SageMaker
Processing
SageMaker
Hosting
SageMaker
Model Monitor

93
How SageMaker Clarify works
Amazon SageMaker
Clarify

94
SageMaker Clarify – Detect Bias During Data Preparation
Bias report in
SageMaker Data Wrangler

95
SageMaker Clarify – Check Your Trained Model for Bias
Bias report in
SageMaker Experiments

96
SageMaker Clarify – Monitor Your Model for Bias Drift
Bias Drift in
SageMaker Model Monitor

97
SageMaker Clarify – Understand Your Model
Model Explanation in
SageMaker Experiments

98
SageMaker Clarify – Monitor Your Model for Drift in Behavior
Explainability Drift in
SageMaker Model Monitor

99
Case Study & Demo: https://youtu.be/cQo2ew0DQw0

100
SageMaker Clarify Use Cases
Regulatory
Compliance
Internal Reporting Operational
Excellence
Customer
Service

Lessons
learned
• Fairness as a Process
• Notions of bias & fairness are highly application
dependent
• Choice of the attribute(s) for which bias is to be
measured & the choice of the bias metrics to be
guided by social, legal, and other non-technical
considerations
• Collaboration/consensus across key
stakeholders
• Wide spectrum of customers with different
levels of technical background
• Managed service vs. open source packages
• Monitoring of the deployed model
• Fairness & explainability considerations
across the ML lifecycle

102
Best Practices
• Fairness as a Process:
• The notions of bias and fairness are highly application dependent and the
choice of the attribute(s) for which bias is to be measured, as well as the choice
of the bias metrics, may need to be guided by social, legal, and other non-
technical considerations.
• Building consensus and achieving collaboration across key stakeholders (such
as product, policy, legal, engineering, and AI/ML teams, as well as end users
and communities) is a prerequisite for the successful adoption of fairness-aware
ML approaches in practice.
• Fairness and explainability considerations may be applicable during each
stage of the ML lifecycle.

103
Fairness and Explainability by Design in the ML Lifecycle

104
Additional Pointers
For more information on Amazon SageMaker Clarify, please refer:
• https://aws.amazon.com/sagemaker/clarify
• Amazon Science / AWS Articles
• https://aws.amazon.com/blogs/aws/new-amazon-sagemaker-clarify-detects-
bias-and-increases-the-transparency-of-machine-learning-models
• https://www.amazon.science/latest-news/how-clarify-helps-machine-learning-
developers-detect-unintended-bias
• Technical papers: (1) Amazon SageMaker Clarify [KDD’21] (2) Fairness
Measures for Machine Learning in Finance
• https://github.com/aws/amazon-sagemaker-clarify
Acknowledgments: Amazon SageMaker Clarify core team, Amazon AWS AI
team, and partners across Amazon

105
Additional Pointers
For more information on Amazon SageMaker Clarify, please refer:
• https://aws.amazon.com/sagemaker/clarify
• Amazon Science / AWS Articles
• https://aws.amazon.com/blogs/aws/new-amazon-sagemaker-clarify-detects-
bias-and-increases-the-transparency-of-machine-learning-models
• https://www.amazon.science/latest-news/how-clarify-helps-machine-learning-
developers-detect-unintended-bias
• Technical papers: (1) Amazon SageMaker Clarify [KDD’21] (2) Fairness
Measures for Machine Learning in Finance
• https://github.com/aws/amazon-sagemaker-clarify
Acknowledgments: Amazon SageMaker Clarify core team, Amazon AWS AI
team, and partners across Amazon

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark
Amazon SageMaker Debugger
Debug and profile ML model training and get real-time insights

Why debugging and profiling
Training bugs Large compute instances Long training times
Training ML models is difficult and compute intensive

SageMaker Debugger
Real-time
monitoring
Relevant
data capture
Automatic error
detection
SageMaker Studio
integration
Debug data while training
is ongoing
Zero code change
Persistent in your
S3 bucket
Built-in and custom rules
Early termination
Alerts about rule status
Save time
and cost
Find issues early
Accelerate prototyping
Detect performance bottlenecks
View suggestions on
resolving bottlenecks,
Interactive visualizations
Monitor utilization Profile
by step or
time duration
Right size instance
Improve utilization
Reduce cost
System resource usage
Time spent by
training operations

https://www.amazon.science/publications/amazon-sagemaker-debugger-a-system-for-real-time-insights-into-machine-learning-model-
training

110
Additional Pointers
For more information on Amazon SageMaker Debugger, please refer:
• https://aws.amazon.com/sagemaker/debugger
• AWS Articles
• https://aws.amazon.com/blogs/aws/amazon-sagemaker-debugger-debug-your-
machine-learning-models
• https://aws.amazon.com/blogs/machine-learning/detecting-hidden-but-non-trivial-
problems-in-transfer-learning-models-using-amazon-sagemaker-debugger
• Technical paper: Amazon SageMaker Debugger: A System for Real-Time Insights
into Machine Learning Model Training (MLSys 2021)
• https://pypi.org/project/smdebug
Acknowledgments: Amazon SageMaker Debugger core team, Amazon AWS AI team,
and partners across Amazon

Fairness for Opaque Models via
Model Tuning (Hyperparameter
Optimization)
• Can we tune the hyperparameters of a model to achieve both
accuracy and fairness?
• Can we support both opaque models and opaque fairness
constraints?
• Use Bayesian optimization for HPO with fairness constraints!
• Explore hyperparameter configurations where fairness constraints are
satisfied
V. Perrone, M. Donini, M. B. Zafar, R. Schmucker, K. Kenthapadi, C. Archambeau,
Fair Bayesian Optimization, AIES 2021
(Best paper award @ ICML 2020 AutoML workshop)

Fairness Definitions - Examples

Global Optimization for Opaque Functions

Results: Model-specific vs. Model-agnostic Methods

Human-in-the-loop frameworks
Desirable to augment ML model predictions with expert inputs
Popular examples – Healthcare models
Content moderation tasks
Useful for improving accuracy, incorporating additional information, and auditing models.
Child maltreatment hotline screening

Errors and biases in human-in-the-loop frameworks
ML tasks often suffer from group-specific bias, induced due to misrepresentative data or models.
Human-in-the-loop frameworks can reflect biases or inaccuracies of the human experts.
Concerns include:
• Racial bias in human-in-the-loop framework for recidivism risk assessment (Green, Chen – FAccT 2019)
• Ethical concerns regarding audits of commercial facial processing technologies (Raji et al. – AIES 2020)
• Automation bias in time critical decision support systems (Cummings – ISTC 2004)
Can we design human-in-the-loop frameworks that take into account the expertise and biases of
the human experts?

Model
𝑋 − non-protected attributes; 𝑌 − class label; 𝑍 − protected attribute/group membership
Number of experts available = 𝑚 − 1
Input −𝑋
Classifier
𝐹: 𝑋 → 𝑌
Deferrer
𝐷: 𝑋 → 0,1 𝑚
.
.
If 𝐷1 = 1
If 𝐷2 = 1
If 𝐷3 = 1
If 𝐷4 = 1
If 𝐷𝑚−1 = 1
If 𝐷𝑚 = 1
Deferrer 𝐷 choose a committee of
experts. The majority decision of
committee is the final prediction
Final output
of selected
committee
Experts might have access to additional information, including group membership 𝑍.
There might be a cost/penalty associated with each expert review.

Fairness in human-in-the-loop settings
• Joint learning framework to learn a classifier and a deferral system
for multiple experts simultaneously
• Synthetic and real-world experiments on the efficacy of our
method
V. Keswani, M. Lease, K. Kenthapadi, Towards Unbiased and
Accurate Deferral to Multiple Experts, AIES 2021.

Minimax Group Fairness: Algorithms and
Experiments
• Equality of error rates: an intuitive & well-studied group fairness
notion
• May require artificially inflating error on easier-to-predict groups 
• Undesirable when most/all of the targeted population is disadvantaged
• Goal: minimize maximum group error [Martinez et al, 2020]
• “Ensure that the worst-off group is as well-off as possible”
• Our work: algorithms based on a zero-sum game between a Learner
and a Regulator
• Theoretical results and experimental evaluation
E. Diana, W. Gill, M. Kearns, K. Kenthapadi, A. Roth, Minimax Group Fairness:
Algorithms and Experiments, AIES 2021.

Robust Interpretability of Neural Text Classifiers
• Feature attribution methods used for understanding model
predictions
• How robust are feature attributions for neural text classifiers?
• Are they identical under different random initializations of the same model?
• Do they differ between a model with trained parameters and a model with
random parameters?
• Common feature attribution methods fail both tests!
M. B. Zafar, M. Donini, D. Slack, C. Archambeau, S. Das, K. Kenthapadi, On the
Lack of Robust Interpretability of Neural Text Classifiers, Findings in ACL 2021.

Privacy in
AI @
LinkedIn
PriPeARL: Framework to
compute robust,
privacy-preserving
analytics

Analytics & Reporting Products at LinkedIn
Profile View
Analytics
126
Content
Analytics
Ad Campaign
Analytics
All showing
demographics of
members engaging with
the product

Admit only a small # of predetermined query types
Querying for the number of member actions, for a specified time period,
together with the top demographic breakdowns

Admit only a small # of predetermined query types
Querying for the number of member actions, for a specified time period,
together with the top demographic breakdowns
E.g., Title = “Senior
Director”
E.g., Clicks on a
given ad

Privacy Requirements
Attacker cannot infer whether a member performed an action
E.g., click on an article or an ad
Attacker may use auxiliary knowledge
E.g., knowledge of attributes associated with the target member (say,
obtained from this member’s LinkedIn profile)
E.g., knowledge of all other members that performed similar action (say, by
creating fake accounts)

Possible Privacy Attacks
130
Targeting:
Senior directors in US, who studied at Cornell
Matches ~16k LinkedIn members
→ over minimum targeting threshold
Demographic breakdown:
Company = X
May match exactly one person
→ can determine whether the person
clicks on the ad or not
Require minimum reporting threshold
Attacker could create fake profiles!
E.g. if threshold is 10, create 9 fake profiles
that all click.
Rounding mechanism
E.g., report incremental of 10
Still amenable to attacks
E.g. using incremental counts over time to
infer individuals’ actions
Need rigorous techniques to preserve member privacy
(not reveal exact aggregate counts)

Problem Statement
Compute robust, reliable analytics in a privacy-
preserving manner, while addressing the product
needs.

PriPeARL: A Framework for Privacy-Preserving Analytics
K. Kenthapadi, T. T. L. Tran, ACM CIKM 2018
132
Pseudo-random noise generation, inspired by differential privacy
● Entity id (e.g., ad
creative/campaign/account)
● Demographic dimension
● Stat type (impressions, clicks)
● Time range
● Fixed secret seed
Uniformly Random
Fraction
● Cryptographic
hash
● Normalize to
(0,1)
Random
Noise
Laplace
Noise
● Fixed ε
True
Count
Noisy
Count
To satisfy consistency
requirements
● Pseudo-random noise → same query has same result over time, avoid
averaging attack.
● For non-canonical queries (e.g., time ranges, aggregate multiple entities)
○ Use the hierarchy and partition into canonical queries
○ Compute noise for each canonical queries and sum up the noisy
counts

Lessons Learned from Deployment (> 1
year)
Semantic consistency vs. unbiased, unrounded noise
Suppression of small counts
Online computation and performance requirements
Scaling across analytics applications
Tools for ease of adoption (code/API library, hands-on how-to tutorial) help!
Having a few entry points (all analytics apps built over Pinot)  wider adoption

Summary
Framework to compute robust, privacy-preserving analytics
Addressing challenges such as preserving member privacy, product
coverage, utility, and data consistency
Future
Utility maximization problem given constraints on the ‘privacy loss budget’
per user
E.g., noise with larger variance to impressions but less noise to clicks (or conversions)
E.g., more noise to broader time range sub-queries and less noise to granular time
range sub-queries
Reference: K. Kenthapadi, T. Tran, PriPeARL: A Framework for Privacy-
Preserving Analytics and Reporting at LinkedIn, ACM CIKM 2018.

Acknowledgements
Team:
AI/ML: Krishnaram Kenthapadi, Thanh T. L. Tran
Ad Analytics Product & Engineering: Mark Dietz, Taylor Greason, Ian
Koeppe
Legal / Security: Sara Harrington, Sharon Lee, Rohit Pitke
Acknowledgements
Deepak Agarwal, Igor Perisic, Arun Swami

LinkedIn Salary (launched in Nov, 2016)

Data Privacy Challenges
Minimize the risk of inferring any one
individual’s compensation data
Protection against data breach
No single point of failure

Problem Statement
How do we design LinkedIn Salary system taking into
account the unique privacy and security challenges,
while addressing the product requirements?
K. Kenthapadi, A. Chudhary, and
S. Ambler, LinkedIn Salary: A
System for Secure Collection and
Presentation of Structured
Compensation Insights to Job
Seekers, IEEE PAC 2017
(arxiv.org/abs/1705.06976)

Title Region
$$
User Exp
Designer
SF Bay
Area
100K
User Exp
Designer
SF Bay
Area
115K
... ...
...
Title Region
$$
User Exp
Designer
SF Bay
Area
100K
De-identification Example
Title Region Company Industry Years of
exp
Degree FoS Skills
$$
User Exp
Designer
SF Bay
Area
Google Internet 12 BS Interactive
Media
UX,
Graphics,
...
100K
Title Region Industry
$$
User Exp
Designer
SF Bay
Area
Internet
100K
Title Region Years of
exp $$
User Exp
Designer
SF Bay
Area
10+
100K
Title Region Company Years of
exp $$
User Exp
Designer
SF Bay
Area
Google 10+
100K
#data
points >
threshold?
Yes ⇒ Copy to
Hadoop (HDFS)
Note: Original submission stored as encrypted objects.

Acknowledgements
Team:
AI/ML: Krishnaram Kenthapadi, Stuart Ambler, Xi Chen, Yiqun Liu, Parul
Jain, Liang Zhang, Ganesh Venkataraman, Tim Converse, Deepak Agarwal
Application Engineering: Ahsan Chudhary, Alan Yang, Alex Navasardyan,
Brandyn Bennett, Hrishikesh S, Jim Tao, Juan Pablo Lomeli Diaz, Patrick
Schutz, Ricky Yan, Lu Zheng, Stephanie Chou, Joseph Florencio, Santosh
Kumar Kancha, Anthony Duerr
Product: Ryan Sandler, Keren Baruch
Other teams (UED, Marketing, BizOps, Analytics, Testing, Voice of
Members, Security, …): Julie Kuang, Phil Bunge, Prateek Janardhan, Fiona
Li, Bharath Shetty, Sunil Mahadeshwar, Cory Scott, Tushar Dalvi, and team
Acknowledgements
David Freeman, Ashish Gupta, David Hardtke, Rong Rong, Ram

Privacy Research @ Amazon -
Sampler

Differentially Private Query Release
• Problem: answering marginal queries privately with high accuracy
• Marginal queries are a special class of linear queries that count slices of a
dataset.
• “How many authors have visited Charlotte, graduated in the last two year
and work in the Bay Area?” – A 3-way marginal query on the below dataset.
• Privacy: Marginals computed against our dataset should protect
against inferences on an individual’s membership (using Differential
Privacy)

Differentially Private Query Release
• Projection Mechanism: Evaluate noisy answers to all queries in a
query class Q and find the synthetic dataset (D′) in the space of
feasible datasets that minimizes error with respect to some norm. Q
is a class of queries.
Nikolov, Talwar, Zhang, The geometry of differential privacy: the sparse and approximate cases, STOC 2013

Differentially Private Query Release: Key Ideas
1. Relax the data domain: one-hot encode the non-continuous data
and expand the domain to real numbers. Extend the differentiable
queries to the new domain.
2. Adaptively select queries: repeatedly choose the k worst performing
queries privately and optimize D′ to answer those well.
S. Aydore, W. Brown, M. Kearns, K. Kenthapadi, L. Melis, A. Roth, A. Siva,
Differentially Private Query Release Through Adaptive Projection, ICML 2021.

Simple but effective, privacy-preserving mechanism
Task: subsample from dataset using additional information in privacy-
preserving way.
Building on existing exponential analysis of k-anonymity, amplified by
sampling…
Mechanism M is (β, ε, δ)-differentially private
Model uncertainty via Bayesian NN
”Privacy-preserving Active Learning on Sensitive Data for User Intent
Classification” [Feyisetan, Balle, Diethe, Drake; PAL 2019]

Analysis of DP sampling
Avg impact privacy budget
budget applied to downstream
classification task.
Relative trade-off on accuracy
for privacy gains (from β & k)
in downstream classification
task

Differentially-private text redaction
Task: automatically redact sensitive text for privatizing various ML models.
 Perturb sentences but maintain meaning
e.g. “goalie wore a hockey helmet”  “keeper wear the nhl hat”
Apply metric DP and analysis of word embeddings to scramble sentences
Mechanism M is d χ – differentially private
Establish plausible deniability statistics:
Nw := Pr[M(w ) = w ]
Sw := Expected number of words output by M(w)
“Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate
Perturbations” [Feyisetan, Drake, Diethe, Balle; WSDM 2020]

Analysis of DP redaction
Show plausible deniability via dist. of Nw & Sw for ε:
ε  0 : Nw decreases, Sw increases
ε  inf : Nw increases, Sw decreases.
Impact of accuracy given ε (epsilon) on
multi-class classification and question
answering tasks, respectively:

Improving data utility of DP text redaction
Task: redact text, but use additional structured information to
better preserve utility.
Can we improve redaction for models that fail for extraneous words?
~Recall-sensitive
Extend d χ privacy to hyperbolic embeddings [Tifrea 2018] via
Hyperbolic: utilize high-dimensional geometry to infuse embeddings
with graph structure
E.g. uni- or bi-directional syllogisms from WebIsADb
New privacy analysis of Poincaré model and sampling procedure
Mechanism takes advantage of density in data to apply
perturbations more precisely.
“Leveraging Hierarchical Representations for Preserving Privacy
and Utility in Text” Feyisetan, Drake, Diethe; ICDM 2019
Tiling in Poincaré disk
Hyperbolic Glove emb.
projected into B2 Poincaré disk

Analysis of Hyperbolic redaction
New method improves over
privacy and utility because
of ability to encode
meaningful structure in
embeddings.
Accuracy scores on classification tasks. * indicates results better than 1 baseline, ** better than 2
baselines
Plausible deniability stat Nw (Pr[M(w ) = w) improved.

Beyond
Accuracy
Performance and Cost
Fairness and Bias
Transparency and Explainability
Privacy
Security
Safety
Robustness

Fairness, Explainability &
Privacy: Opportunities

Fairness in ML
Application specific challenges
Conversational AI systems: Unique bias/fairness/ethics considerations
E.g., Hate speech, Complex failure modes
Beyond protected categories, e.g., accent, dialect
Entire ecosystem (e.g., including apps such as Alexa skills)
Two-sided markets: e.g., fairness to buyers and to sellers, or to content
consumers and producers
Fairness in advertising (externalities)
Tools for ensuring fairness (measuring & mitigating bias) in AI lifecycle
Pre-processing (representative datasets; modifying features/labels)
ML model training with fairness constraints
Post-processing
Experimentation & Post-deployment

Explainability in ML
Actionable explanations
Balance between explanations & model secrecy
Robustness of explanations to failure modes (Interaction between ML
components)
Application-specific challenges
Conversational AI systems: contextual explanations
Gradation of explanations
Tools for explanations across AI lifecycle
Pre & post-deployment for ML models
Model developer vs. End user focused

Privacy in ML
Privacy for highly sensitive data: model training & analytics using
secure enclaves, homomorphic encryption, federated learning / on-
device learning, or a hybrid
Privacy-preserving model training, robust against adversarial
membership inference attacks (Dynamic settings + Complex data /
model pipelines)
Privacy-preserving mechanisms for data marketplaces

Reflections
“Fairness, Explainability, and Privacy by
Design” when building AI products
Collaboration/consensus across key
stakeholders
NYT / WSJ / ProPublica test :)

Related Tutorials / Resources
• ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT)
• AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES)
• Sara Hajian, Francesco Bonchi, and Carlos Castillo, Algorithmic bias: From
discrimination discovery to fairness-aware data mining, KDD Tutorial, 2016.
• Solon Barocas and Moritz Hardt, Fairness in machine learning, NeurIPS Tutorial, 2017.
• Kate Crawford, The Trouble with Bias, NeurIPS Keynote, 2017.
• Arvind Narayanan, 21 fairness definitions and their politics, FAccT Tutorial, 2018.
• Sam Corbett-Davies and Sharad Goel, Defining and Designing Fair Algorithms, Tutorials
at EC 2018 and ICML 2018.
• Ben Hutchinson and Margaret Mitchell, Translation Tutorial: A History of Quantitative
Fairness in Testing, FAccT Tutorial, 2019.
• Henriette Cramer, Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III,
Miroslav Dudík, Hanna Wallach, Sravana Reddy, and Jean Garcia-Gathright, Translation
Tutorial: Challenges of incorporating algorithmic fairness into industry practice, FAccT
Tutorial, 2019.

Related Tutorials / Resources
• Sarah Bird, Ben Hutchinson, Krishnaram Kenthapadi, Emre Kiciman, Margaret Mitchell,
Fairness-Aware Machine Learning: Practical Challenges and Lessons Learned, Tutorials
at WSDM 2019, WWW 2019, KDD 2019.
• Krishna Gade, Sahin Cem Geyik, Krishnaram Kenthapadi, Varun Mithal, Ankur Taly,
Explainable AI in Industry, Tutorials at KDD 2019, FAccT 2020, WWW 2020.
• Himabindu Lakkaraju, Julius Adebayo, Sameer Singh, Explaining Machine Learning
Predictions: State-of-the-art, Challenges, and Opportunities, NeurIPS 2020 Tutorial.
• Kamalika Chaudhuri, Anand D. Sarwate, Differentially Private Machine Learning:
Theory, Algorithms, and Applications, NeurIPS 2017 Tutorial.
• Krishnaram Kenthapadi, Ilya Mironov, Abhradeep Guha Thakurta, Privacy-preserving
Data Mining in Industry, Tutorials at KDD 2018, WSDM 2019, WWW 2019.
• Krishnaram Kenthapadi, Ben Packer, Mehrnoosh Sameki, Nashlie Sephus, Responsible
AI in Industry, Tutorials at AAAI 2021, FAccT 2021, WWW 2021, ICML 2021.

Responsible AI in Industry: Practical Challenges and Lessons Learned

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Responsible AI in Industry: Practical Challenges and Lessons Learned

Semelhante a Responsible AI in Industry: Practical Challenges and Lessons Learned (20)

Mais de Krishnaram Kenthapadi

Mais de Krishnaram Kenthapadi (11)

Último

Último (20)

Responsible AI in Industry: Practical Challenges and Lessons Learned