Slides of the talk at the 14th annual International Conference of Education, Research and Innovation, ICERI 2021, November 8th-9th 2021 (online).

1. The Intelligence Corpus,
an annotated corpus of
definitions of intelligence
Annotation, guidelines, and student research projects
@dmonett
Dagmar Monett1, Luisa Hoge2, Laura Haase3, Lena Schwarz4, Marc Normann5, Linus Scheibe6
1Berlin School of Economics and Law
2Robert Koch Institute
3Technical University of Applied Sciences Wildau
4Hochschule Stralsund
5NORDAKADEMIE Graduate School
6DB Systel GmbH
14th annual International Conference of Education, Research and Innovation
8th - 9th of November, 2021

2. How to cite this work
Monett, D., Hoge, L., Haase, L., Schwarz, L., Normann, M., & Scheibe, L. (2021).
The Intelligence Corpus, an Annotated Corpus of Definitions of Intelligence:
Annotation, Guidelines, and Student Research Projects.
In Proceedings of the 14th annual International Conference of Education, Research
and Innovation, ICERI 2021, November 8th-9th 2021 [online].
Available at: https://www.slideshare.net/dmonett/monett-etal-2021-iceri
(Accessed: access date).
2

3. Annotating definitions
of intelligence
- Fact: Lack of consensus on defining
intelligence
- Needed: To provide better insights into
definitions and how to define them
- Why: It is central to the basics of AI
literacy
- How: In this work, by evaluating the
properties of available definitions
3

4. Annotation and Annotators
Automatic data annotation
still biased, error prone, and
far from being entirely
satisfactory.
Domain knowledge
The annotation might require
special insights into the
problem domain.
Software solutions
Software solutions are available for
supporting annotators in their work
(Neves and Ševa, 2021), but not for
all kinds of data and domains.
Annotation
The annotation of data either
its nature can be a very
challenging and time
consuming process.
Annotators
Undergraduate 3rd year
Computer Science students
Annotators
Parallel course on AI and
student research projects
M. Neves and J. Ševa, “An extensive review of tools for manual annotation of documents,” Briefings in Bioinformatics, vol. 22, no. 1, pp. 146–163, 2021.
4

5. The Annotation Data
A: 213 new, suggested definitions of machine or
artificial intelligence by participants to the survey
on defining intelligence (Monett & Lewis, 2018)
B: 125 new, suggested definitions of human
intelligence by participants to the survey on
defining intelligence (Monett & Lewis, 2018)
C: 34 definitions of intelligence from the literature to
agree upon in the initial edition of the survey on
defining intelligence (Monett & Lewis, 2018)
D: 71 definitions of intelligence from the collection
presented in (Legg & Hutter, 2007)
D. Monett and C.W.P. Lewis, “Getting clarity by defining Artificial Intelligence—A Survey,” in Philosophy and Theory of Artificial Intelligence (V.C. Müller, ed.),
SAPERE vol. 44, pp. 212–214. Springer, Berlin, 2018.
S. Legg and M. Hutter, “A Collection of Definitions of Intelligence,” in Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms (B. Goertzel
and P. Wang, eds.), vol. 157, pp. 17–24. IOS Press, UK, 2007.
443 definitions of (machine) intelligence
5

6. Examples of Definitions
From collection C
“Intelligence measures an agent’s ability to
achieve goals in a wide range of
environments.”
From collection D
“[Intelligence is] the capacity to learn,
reason, and understand.”
From collection B
“[Human intelligence is] the
ability to use information to
accomplish goals.”
From collection A
“Machine Intelligence is concerned
with building systems that can adapt
and learn in unstructured noisy
domains.”
6

7. Example:
„A good
definition of
intelligence
is
affirmative.“
for definitions as suggested in
(Monett & Lewis, 2020)
Quality criteria
D. Monett and C.W.P. Lewis, “Definitional Foundations for Intelligent Systems,
Part I: Quality Criteria for Definitions of Intelligence,” in Proceedings of The 10th
Anniversary Conference of the Academic Conference Association (J. Vopava, V.
Douda, R. Kratochvil, and M. Konecki, eds.), pp. 73–80, Prague, Czech
Republic. MAC Prague Consulting Ltd., 2020.
7

8. Definitions and Annotators
Distribution of definitions per groups of annotators
8

9. Annotation Guidelines
… on a cell if the
corresponding
definition fulfils the
quality criterion
Write a 1
E.g. by column (i.e.
one quality criterion
at a time); by row (i.e.
definition by definition)
How to proceed
… you spend
annotating
whenever possible
Record the time
… when no idea
about how to
evaluate a given
quality criterion
Set background color
… grammatical
errors you might
find in the
definitions.
Do not fix
Do not discuss with
other; this could
introduce some bias
Annotate alone
… among others from the literature!
9

10. Examples of Annotations
Definition It is affirmative It is short
It includes
cognitive
abilities or
functions
It defines the
“what”
It does not
distinguish
between
human and
machine
intelligence
“An intelligent machine must first
of all be a machine with interests,
otherwise there are no interests
to be served by its intelligence.
Human interests can be served
by machine ‘competence,’ not by
machine intelligence.”
1
“[Human intelligence is the]
ability to achieve objectives in a
variety of environments.”
1 1 1
“Intelligence is the computational
part of the ability to achieve
goals in the world.”
1 1 1 1
“[Intelligence is] the capacity to
acquire and apply knowledge.”
1 1 1 1 1
10

11. Inter-Annotator Agreement (IAA)
Avg. Cohen’s κ = 0.4
(per group per
collection)
J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, pp. 37–46, 1960.
J.R. Landis and G.G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, pp. 159–174, 1977.
 IAA between fair and moderate for all collections (i.e.
Cohen’s κ ranging from 0.344 to 0.465, interpretation
according to (Landis and Koch, 1977)).
 The number of agreements among annotators was higher for
the collection containing definitions of human intelligence
(collection B).
 The quality criteria with the highest IAA values were those
simpler, more intuitive, and easier to understand.
 The annotators were more agreeable when evaluating the
fulfilment of the quality criteria for the following definition:
“Intelligence is a very general mental capability that, among
other things, involves the ability to reason, plan, solve
problems, think abstractly, comprehend complex ideas, learn
quickly and learn from experience.”
11

12. The “best” definition: Gottfredson’s
L.S. Gottfredson, “Mainstream science on intelligence: An editorial with 52 signatories, history, and bibliography,”
Intelligence, vol. 24, pp. 13–23, 1997.
R.J. Haier, “The Neuroscience of Intelligence,” Cambridge University Press, New York, NY, 2017.
12

13. Usage: Examples
Part of the
Intelligence Corpus
All definitions from
(Monett and Lewis, 18)
 71 definitions of machine or
artificial intelligence (from a
total of 213) from collection A.
 42 definitions of human
intelligence (from a total of
125) from collection B.
 12 definitions of intelligence
(from a total of 34) from
collection C.
 23 definitions of intelligence
(from a total of 71) from
collection D.
https://bit.ly/AnnotatedDefsIntelligence
https://goo.gl/KDPtKT
The collection of definitions of
intelligence used in the research
survey “Defining (machine)
Intelligence”
D. Monett and C.W.P. Lewis, “Getting clarity by defining Artificial Intelligence—A Survey,” in Philosophy and Theory of Artificial Intelligence (V.C. Müller, ed.),
SAPERE vol. 44, pp. 212–214. Springer, Berlin, 2018.
13

14. Conclusions
Prior experience available in mentoring and
supervising student research projects.
Parallel course on AI delivered by the same
instructor; ad hoc discussions in class.
A peculiar annotation case study that evaluates
whether definitions of human and machine
intelligence satisfy desirable properties or quality
criteria of good definitions.
Information, materials, and tools carefully
prepared and discussed in advance, also
throughout the project’s execution.
AI course
AI
Experience
Intelligence Corpus Project management
Training on the process of defining a good
definition of any concept, which could be of
interest to regulators or lawyers, for instance.
E.g. detailed, manually-conducted quality control
of all available annotations.
Further work
Other uses
14

15. Contact
/monettdiaz
@dmonett
Prof. Dr.
Dagmar Monett
15
http://monettdiaz.com
dagmar.monett-diaz@hwr-berlin.de
Prof. Dr. Computer Science (Artificial Intelligence, Software
Engineering)
Computer Science Dept.
Co-Director M.Sc. Digital Transformation
Berlin School of Economics and Law (HWR Berlin)
Faculty of Cooperative Studies
Template: www.allppt.com