We propose an abstract framework for XAI based on MAS encompassing the main definitions and results from the literature, focussing on the key notions of interpretation and explanation.

1. An Abstract Framework for
Agent-Based Explanations in AI
Giovanni Ciatto∗ Davide Calvaresi†
Michael I. Schumacher† Andrea Omicini∗
∗Dipartimento di Informatica – Scienza e Ingegneria (DISI)
Alma Mater Studiorum – Universit`a di Bologna
{giovanni.ciatto , andrea.omicini}@unibo.it
†University of Applied Sciences and Arts Western Switzerland
{davide.calvaresi, michael.schumacher}@hevs.ch
International Conference on
Autonomous Agents and Multi-Agent Systems (AAMAS)
May 9 – 13, 2020, Auckland, New Zeland
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 1 / 17

2. Motivation & Context
Next in Line. . .
1 Motivation & Context
2 Fundamentals
3 Understandability in data-driven intelligent systems
4 Conclusions
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 1 / 17

3. Motivation & Context
Context
Some well known facts:
Increasing adoption of autonomous intelligent systems
for automation, monitoring, and decision support
⇒ Increasing amounts of activities delegated to autonomous agents
! even critical ones, e.g., finance, healthcare, etc
Increasing exploitation of ML to let agents learn tasks from data
alternative to manual programming
Involving black-box predictors which are inherently opaque
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 2 / 17

4. Motivation & Context
Motivation
Opaqueness of ML-based predictors brings several drawbacks [5, 7]:
difficulty in understanding what agents learn from data
e.g. “snowy background” problem [9]
difficulty in spotting “bugs” w.r.t. expected behaviour
because such knowledge is not explicitly represented
several failures of ML-based systems reported so far [2, 9, 11]
lawmakers recognised citizens’ right to meaningful explanations [10]
about the logic behind automated decision making
e.g. in General Data Protection Regulation (GDPR) [4]
=⇒ need to make AI more understandable [6]
understandable → control / robustness → trust
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 3 / 17

5. Motivation & Context
The eXplanable AI (XAI) approach [6]
The XAI community is nowadays facing such understandability issues
Focus on techniques easing the interpretation of numeric predictors
a.k.a. “opening the black box”, or look into it [5]
From [7]
In particular, most efforts are devoted to:
specific sorts of tasks, e.g. classification and regression
specific sorts of data, e.g. images, text, or tables
specific sorts of predictors, e.g. neural networks, SVM
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 4 / 17

6. Motivation & Context
Contribution of the paper
Discussions in the field of XAI are often ambigous
Due to the strong reliance on informal notions
such as explanation, interpretation, or understandability
Which are often used interchangeably
In this work
We provide a clear unambiguous definition of two fundamental notions:
explanation
interpretation
proposing an abstract framework leveraging on the MAS background
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 5 / 17

7. Fundamentals
Next in Line. . .
1 Motivation & Context
2 Fundamentals
3 Understandability in data-driven intelligent systems
4 Conclusions
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 5 / 17

8. Fundamentals
About interpretation
Definition: interpretability
A fuzzy and subjective property any object X may satisfy into some agent
A’s perspective
interpretability of some object X is not an absolute property
it only makes sense in presence of an observer A
we model interpretation as an observer-specific function
IA(X) → [0, 1]
the particular value of IA for some X is not that relevant
as long as comparisons are possible: IA(X) > IA(X ) ≥ IA(X ) . . .
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 6 / 17

9. Fundamentals
About explanation
Definition: explanation
An objective activity any agent may perform to make an object X more
interpretable
explaining an object X = searching for another object X s.t.
X is more interpretable than X, and
X has an high fidelity w.r.t. X
we model explanation as a function
E(X) → X
whereas difference in fidelity is measured through a function
∆f (X, X ) → [0, ∞)
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 7 / 17

10. Fundamentals
Understandability in a nutshell
Definition: understandability
The soft goal pursued by an agent A willing to make some object X
interpretable to some observer B, by looking for the right explanation
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 8 / 17

11. Understandability in data-driven intelligent systems
Next in Line. . .
1 Motivation & Context
2 Fundamentals
3 Understandability in data-driven intelligent systems
4 Conclusions
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 8 / 17

12. Understandability in data-driven intelligent systems
ML-based intelligent systems
A common situation for intelligent agents is to leverage on ML to
learn tasks from data
This implies a predictor M to be trained on some dataset (X, Y )
For any given task, many families of predictors may be suitable
eg neural networks, SVM, decision trees, linear models, etc.
In particular, training aims at selecting the best predictor M w.r.t.
some predictive performance measure of choice
the data at hand
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 9 / 17

13. Understandability in data-driven intelligent systems
The role of representations in ML
However, interpretability of predictors is an important feature as well
Predictors, as abstract objects, are not directly interpretable
Definition: Predictor representations
A predictor M may have one or more representation R = r(X, M),
describing its behaviour for some input data X
eg heatmaps, feature importance vectors, decision boundary plots, etc
! representations are actually interpretable by observers
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 10 / 17

14. Understandability in data-driven intelligent systems
Global vs. local representation
Local representations
Describe a predictor behaviour
w.r.t. some portion of the input
space (e.g. 1 instance)
Global representations
Describe a predictor behaviour
w.r.t. the whole input space
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 11 / 17

15. Understandability in data-driven intelligent systems
Representations interpretability
Not all representations are equally interpretable
Nor can a representation fit all possible cases
eg heatmaps are better suited for image classifiers
Predictor families come with some natural representation
some are considered more interpretable than others
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 12 / 17

16. Understandability in data-driven intelligent systems
Representations vs Explanation
To make some predictor M more interpretable for an agent A, one
may either:
change representation, or
search for a better explanation M = E(M)
The latter case make sense if
M has an high fidelity to M for some input data X
r(X, M ) is more interpretable than any other r(X, M)
Takeaway
Explaining a black-box predictor is about searching approximate models
amenable of more interpretable representations
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 13 / 17

17. Understandability in data-driven intelligent systems
Example: Symbolic Knowledge Extraction (SKE)
Neural network
explanation
−−−−−−→
Decision tree/rules
eg symbolic knowledge extraction out of neural networks [1, 5]
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 14 / 17

18. Conclusions
Next in Line. . .
1 Motivation & Context
2 Fundamentals
3 Understandability in data-driven intelligent systems
4 Conclusions
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 14 / 17

19. Conclusions
Summing up
ML-powered AI is everywhere but it not the silver-bullet
Increasing demand of understandability for ML-based systems
XAI mostly focus on building more interpretable representation
a.k.a. opening the black-boxes [5]
Most discussions are imprecise as they leverage on ambiguous notions
and terms
→ Abstract framework deeply rooted in the MAS, to properly define
interpretation and explanation
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 15 / 17

20. Conclusions
Future Works
Extension of the conceptual framework towards the multi-agent case
user-2-agents and agent-2-agents cases
Design, development, and validation of protocols for
cooperative/competitive best explanation search
Comparison, assessment, and generalisation of SKE algorithms
development of software libraries for SKE
e.g. extending Sci-Kit Learn [8]
Technological integration of SKE with symbolic frameworks
e.g. the tuProlog engine [3]
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 16 / 17

21. An Abstract Framework for
Agent-Based Explanations in AI
Giovanni Ciatto∗ Davide Calvaresi†
Michael I. Schumacher† Andrea Omicini∗
∗Dipartimento di Informatica – Scienza e Ingegneria (DISI)
Alma Mater Studiorum – Universit`a di Bologna
{giovanni.ciatto , andrea.omicini}@unibo.it
†University of Applied Sciences and Arts Western Switzerland
{davide.calvaresi, michael.schumacher}@hevs.ch
International Conference on
Autonomous Agents and Multi-Agent Systems (AAMAS)
May 9 – 13, 2020, Auckland, New Zeland
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020 17 / 17

22. Bibliography
References I
[1] Robert Andrews, Joachim Diederich, and Alan B. Tickle.
Survey and critique of techniques for extracting rules from trained artificial neural
networks.
Knowledge-Based Systems, 8(6):373–389, December 1995.
[2] Kate Crawford.
Artificial intelligence’s white guy problem.
The New York Times, 25, 2016.
[3] Enrico Denti, Andrea Omicini, and Roberta Calegari.
tuProlog: Making Prolog ubiquitous.
ALP Newsletter, October 2013.
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020

23. Bibliography
References II
[4] General Data Protection Regulation (GDPR).
Regulation (eu) 2016/679 of the european parliament and of the council of 27 april
2016 on the protection of natural persons with regard to the processing of personal
data and on the free movement of such data, and repealing directive 95/46/ec.
https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32016R0679.
Online; accessed on October 11, 2019.
[5] Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and Fosca
Giannotti.
A survey of methods for explaining black box models.
CoRR, abs/1802.01933, 2018.
[6] David Gunning.
Explainable artificial intelligence (XAI).
Funding Program DARPA-BAA-16-53, DARPA, 2016.
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020

24. Bibliography
References III
[7] Zachary Chase Lipton.
The mythos of model interpretability.
CoRR, abs/1606.03490, 2016.
[8] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel,
M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos,
D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.
Scikit-learn: Machine learning in Python.
Journal of Machine Learning Research, 12:2825–2830, 2011.
[9] Marco T´ulio Ribeiro, Sameer Singh, and Carlos Guestrin.
Why should I trust you? Explaining the predictions of any classifier.
CoRR, abs/1602.04938, 2016.
[10] Andrew D Selbst and Julia Powles.
Meaningful information and the right to explanation.
International Data Privacy Law, 7(4):233–242, 12 2017.
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020

25. Bibliography
References IV
[11] Rebecca Wexler.
When a computer program keeps you in jail: How computers are harming criminal
justice.
New York Times, 2017.
Ciatto et al. (UNIBO, HES-SO) Abstract Framework for XAI AAMAS, 2020