MIS QUARTERLY Fernandez, Carlos & Provost, Foster & Han, Xintian Presenter ：CHEN,YOU-SHENG (Shane) 2022/12/15

1. /37
0
MIS QUARTERLY
Fernandez, Carlos & Provost, Foster &
Han, Xintian
Presenter ：CHEN,YOU-SHENG
(Shane) 2022/12/15

2. /37
1
JCR For MIS QUARTERLY

3. /37
2
Introduction
Limitations of
importance
weights
AI Systems and
Explanations,
Counterfactual
explanations
Case Studies
C O N T E N T S
Discussion Conclusion

4. /37
3
We examine counterfactual
explanations, which are becoming an
increasingly accepted alternative for
explaining AI decisions
Purpose
To point fundamental reasons why
importance-weight explanations may
not be well-suited to explain data-
driven decisions made by AI systems
Findings
Abstract

5. /37
4
• Authors explain system decisions rather
than model predictions
• Present 3 detailed studies using real-world
data to compare the counterfactual
approach with SHAP
Methodology
Resulting in a framework
(a) is model-agnostic
(b) can address features with arbitrary data types
(c) may explain decisions made by complex AI
systems that incorporate multiple models
(d) is scalable to very large numbers of features
Originality
Abstract

6. /37
5
Situation
• I have burned my tongue
• A person (P) loan application
was rejected
Counterfactual Explanation(CE)
“If I hadn't taken a sip of this hot coffee, I wouldn't
have burned my tongue”
“If P had a higher salary and less outstanding loans,
his loan application would have been approved”
Counterfactual Explanation

7. /37
6
Introduction
Data and predictive models are used by artificial
intelligence (AI) systems to make decisions across many
applications and industries
In fact, as predictive models become more complex and
difficult to understand
The stakeholders often become more skeptical and
reluctant to adopt or use them, even if the models have
been shown to improve decision-making performance
(Arnold et al., 2006; Kayande et al., 2009)

8. /37
7
The importance-weight explanations may not be well-suited to
explain data-driven decisions made by AI systems
Features have large weight but different decisions result in features may
not playing out thus identifying important features is not sufficient to
explain system decisions (may have lots of CE)
Introduction

9. /37
8
AI Systems and Explanations
Explaining predictive models
• Rule-based explanations have been a
popular approach to explain black-box
models (Jacobsson ,2005; Martens et al.,2007) but
the methods are not tailored to explain
individual decisions
Explaining model predictions (Fig.1)
• Framing the explanations in terms of
feature importance by associating a
weight to each feature in the model

10. /37
9
AI Systems and Explanations
( SHAP
The SHAP value quantifies the contribution of each feature to the
prediction made by the model
* contribution margin
(Age、Gender、Job) → 2^3 is equal to 8 possibilities
* SHAP value
CSDN- 机器学习模型的解释-SHAP
https://blog.csdn.net/weixin_41851055/article/details/106146098

11. /37
10
AI Systems and Explanations
SHAP (SHapley Additive exPlanations)
The SHAP value quantifies the contribution of each feature to the
prediction made by the model
(Age、Gender、Job) → 2^3 is equal to 8 possibilities
= 50k + (-11.33k - 2.33k + 46.66k) = 83k
* SHAP value (apply weight)
* Result
-15
-9
-10
-12
w ₁= w ₂+ w ₃= w ₄
w ₂= w ₃
CSDN- 机器学习模型的解释-SHAP
https://blog.csdn.net/weixin_41851055/article/details/106146098

12. /37
11
AI Systems and Explanations
Find a simple and understandable model of an individual in a local
area to answer the question "Why does the model classify an
individual into a particular category?"
Sherry Su- Local Interpretable Model-agnostic Explanations (LIME)
https://medium.com/sherry-ai/xai-透過-lime-解釋複雜難懂的模型-23898753bea5
The yellow area (stars) is a tree frog, and the green area (triangles) is a Mike Wazowski.
LIME (Local Interpretable Model-agnostic Explanations)

13. /37
12
AI Systems and Explanations
LIME (Local Interpretable Model-agnostic Explanations)
Find a simple and understandable model of an individual in a local
area to answer the question "Why does the model classify an
individual into a particular category?"
Sherry Su- Local Interpretable Model-agnostic Explanations (LIME)
https://medium.com/sherry-ai/xai-透過-lime-解釋複雜難懂的模型-23898753bea5

14. /37
13
Counterfactual explanations
• Causal means that removing the set of features from the instance causes the system decision to change
• Irreducible means that removing any proper subset of the explanation would not change the system decision
consider an instance I consisting of a set of m features, I = {1, 2, ..., m},
for which the decision-making system C : I → {1, 2, ..., k} gives decision c.
A feature i is an attribute taking on a particular value
I = Instance ; E = feature ; C = decision-making system (classifier)
E’ is a counterfactual explanations of "C"
: To make the

15. /37
14
Counterfactual explanations
The algorithm proposed by Martens and Provost (2014) finds counterfactual explanations by using a heuristic
search that requires the decision to be based on a scoring function, such as a probability estimate from a
predictive model

16. /37
15
Counterfactual explanations
DEMO : Tutorial_BehavioralData_SEDC
https://github.com/yramon/edc/blob/master/tutorials/Tutorial_BehavioralDataMovielens_MLP_SEDC.ipynb

17. /37
16
Counterfactual explanations
DEMO : Tutorial_BehavioralData_SEDC
https://github.com/yramon/edc/blob/master/tutorials/Tutorial_BehavioralDataMovielens_MLP_SEDC.ipynb
Explain why the user with index = 17 is
predicted as a 'FEMALE' user by the
model.

18. /37
17
Counterfactual explanations
Algorithm : Heuristic best-first search algorithm for finding Evidence Counterfactuals (SEDC)
data - run each [combo_set]
If the classification does not change *pass
If the classification changes R will *add
new features (important)
< max_features

19. /37
18
Limitations of importance weights
To point out that SHAP has several advantages for
explaining data-driven model predictions
1) it produces numeric “importance weights” for
each feature at an instance-level
2) it is model-agnostic
3) its importance weights tie instance-level
explanations to cooperative game theory,
providing a solid theoretical foundation
4) SHAP unites several feature importance
weighting methods (Ribeiro, Singh and Guestrin, 2016)

20. /37
19
Limitations of importance weights
Decision procedure Ci as defined (4)(5)
This example illustrates this by
defining Yˆ 1 as follows (6)
Example 1: Distinguishing between predictions and decisions
Orig = 0 Res = 22
*the large “importance” of a feature for a model prediction may
not imply an impact on a decision made with that prediction

21. /37
20
Limitations of importance weights
Decision procedure Ci as defined (4)(5)
This example illustrates this by
defining Yˆ 1 as follows (6)
Example 1: Distinguishing between predictions and decisions
Orig = 0 Res = 1
*do not capture well how features affect decisions

22. /37
21
Limitations of importance weights
Decision procedure Ci as defined (4)(5)
This example illustrates this by
defining Yˆ 2 as follows (7)
Example 2: Multiple interpretations for the same weights
Orig = 0 Res = 1
*do not communicate how removing (or changing)
the features may change the decision

23. /37
22
Limitations of importance weights
Decision procedure Ci as defined (4)(5)
This example illustrates this by
defining Yˆ 3 as follows (8)
Example 3: Positive impact of non-positive weights
Orig = 0 Res = 1
* a feature that we might mistakenly deem as
irrelevant due to its non-positive weight

24. /37
23
Importance Weights vs
Counterfactual
Explanations
- Lending Club
Accept or deny credit
Predict Facebook post
click like who >age 50
Predict the amount
that a potential
target will donate
High-dimensional and
Context-specific
Explanations
- myPersonality
System Decisions with
Multiple Models
- KDD Cup 1998
Case Studies
1 2 3

25. /37
24
1 - Accept or deny credit
Case Studies
• Data is publicly available and
contains comprehensive information
on all loans issued starting in 2007
• Focus on loans with a 13% annual
interest rate and a duration of three
years (the most common loans)
• Resulting in 71,938 loans
• 70% of this data set to train , 30% for
test
• Denies credit to loan applicants with
a probability of default above 20%

26. /37
25
1 - Accept or deny credit
Case Studies
• SHAP may be adjusted
further to compute
weights only for a
subset of features
• This would make
sense in our context if
customers can only
ask for less money or
show additional
sources of income to
get their credit
approved

27. /37
26
2 - Predict Facebook post
click like who >age 50
Case Studies
• Use a sample that contains
information on 587,745 individuals
from the United States
• Including their Facebook Likes and a
subset of their Facebook profiles
• Leaving us with 10,822 binary
features
• 70% of this data set to train , 30% for
test

28. /37
27
2 - Predict Facebook post
click like who >age 50
Case Studies
• Using the heuristic search
procedure proposed by Martens
and Provost (2014), which does
not consider the relevance of the
various possible explanations and
was designed to find the smallest
explanations first
• To adjust the heuristic search so
that it penalizes less-popular
pages (those with fewer total Likes)
by assigning them a higher cost

29. /37
28
3 - Predict the amount
that a potential target will
donate
Case Studies
• Data set was originally provided
by a national veterans
organization
• 70% of this data set to train ,
30% for test
• Target the 5% of households
with the largest (estimated)
expected donations
• Computed SHAP values for its
predicted probability of donating
(classification model) and its
predicted donation amount
(regression model)

30. /37
29
3 - Predict the amount
that a potential target will
donate
Case Studies
• Counterfactual explanations
can transparently be applied
to system decisions that
involve more than one model
• In fact, AVGGIFT(Average
dollar amount of gifts to date)
had a negative SHAP value in
the regression model, but it
appears in all explanations

31. /37
30
• Case 1 : The importance weight of features is not enough to
determine how the features affect system decisions
• Case 2 : Sampling-based approximations of importance weights
get worse as the number of features increases
-small subsets of features are usually enough to explain decisions
• Case 3 : Weights may be misleading when decisions are made
using multiple models (negative SHAP value)
Disadvantage of counterfactual explanations
 People may prefer simple explanations over the complexity of the
real world
 The number of counterfactual explanations may grow
exponentially
Discussion

32. /37
31
• If features are correlated, mean imputation and
retraining the model without the removed feature
may produce different results
-future research should assess the advantages of each
approach in different settings
• A counterfactual explanation could be defined as a set
of “minimal” feature adjustments that changes the
decision
-future research is to study how users actually perceive
these different sorts of explanations in practice
Discussion

33. /37
32
 This paper shows that explaining model predictions is
not the same as explaining system decisions
 Increasingly popular approach of explaining model
predictions using importance weights has significant
drawbacks when repurposed to explain system
decisions
 Use counterfactual explanations
Conclusion
1. Explain system decisions rather than model predictions
2. Do not enforce any specific method to remove features
3. Our explanations can deal with feature sets with
arbitrary dimensionality and data types.

34. /37
33
RESOURCES
• Fernandez, Carlos & Provost, Foster & Han, Xintian. (2022). Explaining Data-Driven
Decisions made by AI Systems: The Counterfactual Approach. MIS Q. 46, 3
(September 2022), 1635-1660. https://doi.org/10.25300/MISQ/2022/16749
• David Martens and Foster Provost. 2014. Explaining data-driven document
classifications. MIS Q. 38, 1 (March 2014), 73–100.
https://doi.org/10.25300/MISQ/2014/38.1.04
• PPT template- Application Analysis Presentation Template
https://googleslides.org/application-analysis-presentation-template/2011
• Microsoft Stock images (royalty-free images)
• Bing CC images

35. /37
34
Extended learning
• 公平公正的AI，可能嗎？
https://highscope.ch.ntu.edu.tw/wordpress/?p=80057
• 反事实解释（Counterfactual Explanation, CE）
https://zhuanlan.zhihu.com/p/524030270?utm_id=0
• 因果推断可解释性 之 反事实解释综述（一）
https://zhuanlan.zhihu.com/p/441307638?utm_id=0
• 机器学习模型的解释-SHAP
http://t.csdn.cn/W8gi5
• XAI| 透過 LIME 解釋複雜難懂的模型
https://medium.com/sherry-ai/xai-透過-lime-解釋複雜難懂的模型-23898753bea5
• 黑盒模型事後歸因解析（二）： LIME方法
https://ppfocus.com/hk/0/di2ea81b4.html

36. /37
35
Extended learning
• 你真的熟悉所使用的工具嗎？
https://highscope.ch.ntu.edu.tw/wordpress/?p=83733
• 反事實思維（Counterfactual Thinking）
https://wiki.mbalib.com/zh-tw/%E5%8F%8D%E4%BA%8B%E5%AE%9E%E6%80%9D%E7%BB%
• 机器学习中的特征空间
https://blog.csdn.net/google19890102/article/details/49359161
• 機器學習常勝軍 - XGBoost
https://ithelp.ithome.com.tw/articles/10273094?sc=hot
• Heuristic best-first algorithm for computing Evidence Counterfactuals (SEDC)
https://github.com/yramon/edc
DEMO
https://github.com/yramon/edc/blob/master/tutorials/Tutorial_BehavioralData_SEDC.ipynb