Formal concept analysis (FCA) derives a hierarchy of concepts in a formal context that relates objects with attributes. This approach is very well aligned with the traditions of Frege, Saussure and Peirce, which relate a signifier (e.g. a word/an attribute) to a mental concept evoked by this word and meant to refer to a specific object in the real world. However, in the practice of natural languages as well as artificial languages (e.g. programming languages), the application context often constitutes a latent variable that influences the interpretation of a signifier. We present some of our current work that analyzes the usage of words in natural language in varying application contexts as well as the usage of variables in programming languages in varying application contexts in order to provide conceptual constraints on these signifiers.

1. Steffen Staab 1Institute for Web Science and Technologies · University of Koblenz-Landau, Germany
Web and Internet Science Group · ECS · University of Southampton, UK &
Concepts in Application Context
( How we may think conceptually )
Steffen Staab
Including joint work with Lukas Schmelzeisen, Martin Leinberger,
Ralf Lämmel, Claudia Schon, Philipp Seifer & further team
Several slides adapted from Lukas & Martin

2. Steffen Staab 2
Pre-existing hypotheses
• Formal concept analysis is a useful tool to
explore natural and artificial languages
• Formal concept analysis is a useful tool to
explore our thinking using language
Recent paradigm changes (2018 / 2019)
– Understanding natural language text
– Understanding software code
Open question to you:
• How does / how can FCA help?
Main thrust of this talk

3. Steffen Staab 3
From one forefather of FCA
Charles Saunders Peirce
Four incapacities:
3. No power of thinking without signs. A cognition must be interpreted in a subsequent
cognition in order to be a cognition at all.

4. Steffen Staab 4
Publicdomain,
https://commons.wikimedia.org/w/index.php?curid=2567
0405
Concepts in natural languages

5. Steffen Staab 5
Example: WordNet
5
event
miracle
act,
human action,
human activity
happening,
occurrence,
occurent,
natural event
change,
alteration,
modification
miracle
transition
damage,
harm,
impairment
increase
leap,
hump,
saltation
jump,
leap
forfeit,
forfeiture,
sacrifice
action group action
resistance,
opposition
change transgression
motion,
movement,
move
demotion variation
locomotion,
travel
descent
run,
running
jump,
parachuting
dash,
sprint
Example from the WordNet Assignment in
the Princeton Algorithm Course COS 226.
hypernym–
hyponym
relation synset =
sets of
synonyms

6. Steffen Staab 6
Morning star, most commonly used as a name for the
planet Venus when it appears in the east before
sunrise
The Egyptians knew the morning star as Tioumoutiri
and the evening star as Ouaiti.
Example text
https://en.wikipedia.org/wiki/Morning_Star
https://en.wikipedia.org/wiki/Venus

7. Steffen Staab 7
Subtasks:
 Term Extraction:
Input: text corpus and domain
Output: terms that should become part of taxonomy
Taxonomy learning from text
7
Morning star, most commonly used as a name for the
planet Venus when it appears in the east before
sunrise.

8. Steffen Staab 8
Subtasks:
 Term Extraction:
Input: text corpus and domain
Output: terms that should become part of taxonomy
 Hypernym Detection:
Input: pair of terms
Output: relationship (hypernym / hyponym / other)
Taxonomy learning from text
8
Morning star, most commonly used as a name for the
planet Venus when it appears in the east before
sunrise.

9. Steffen Staab 9
Subtasks:
 Term Extraction:
Input: text corpus and domain
Output: terms that should become part of taxonomy
 Hypernym Detection:
Input: pair of terms
Output: relationship (hypernym / hyponym / other)
 Taxonomy Construction:
Input: set of hypernym-hyponym-pairs
Output: high quality taxonomy (cycle-free, etc.)
Taxonomy learning from text
9
Morning star, most commonly used as a name for the
planet Venus when it appears in the east before
sunrise.

10. Steffen Staab 10
„Venus“
The second
planet in our
star system
The signifier
The concept
The signified
„morning
star“
The signifier
Peirce: Sign, interpretant, object
Frege: Zeichen (sign),
Sinn (sense), Bedeutung (meaning)
Semiotic triangle

11. Steffen Staab 11
• Concepts are related
in a heterarchy / Hasse diagram
• Concept comprises objects (the signified)
• Concept comprises attributes (the signifiers)
Possible correspondence
between FCA and the Semiotic Triangle

12. Steffen Staab 12
An early approach (Cimiano et al 2005)
„The museum houses an impressive
collection of medieval and modern art.
The building combines geometric abstraction
with classical references that allude to the
Roman influence on the region.“

13. Steffen Staab 13
An early approach (Cimiano et al 2005)

14. Steffen Staab 14
An early approach (Cimiano et al 2005)

15. Steffen Staab 15
 Word Embeddings
 Word2vec, GloVe, fastText, …
 Dominating paradigm until mid 2018
 Map a term 𝑡 ∈ 𝑉 onto a vector in concept space ℝ 𝑛
:
𝑓: 𝑉 → ℝ 𝑛
 Just one sense per term
Context-free Word Representations
15
„morning
star“
𝐶 ∈ ℝ 𝑛
The signifier
The concept
The signified

16. Steffen Staab 16
Explorative analysis

17. Steffen Staab 17
Morning star, most commonly used as a name for the
planet Venus when it appears in the east before sunrise
The Egyptians knew the morning star as Tioumoutiri and
the evening star as Ouaiti.
A morning star is any of several medieval club-like
weapons consisting of a shaft with an attached ball
adorned with one or more spikes
The Morning star is normally considered to be a one
handed weapon but it was also a polearm weapon
Example text continued
https://en.wikipedia.org/wiki/Morning_star_(weapon)
http://medieval.stormthecastle.com/armorypages/polearms/morning-star.htm

18. Steffen Staab 18
Issues
 Lack of accounting for ambiguous terms
 Lack of quality of taxonomy (e.g. non-transitivity)
What about ambiguity?
18
„Venus“
The second
planet in our
star system
The signifier
The concept
The signified
A mace, a
weapon
with spiked
ball
The concept
„morning
star“
The signifier
The signified

19. Steffen Staab 19
Planet
2nd in solar
system
Can be
used to kill
Used by
knights Has spikes
Venus x x
Evening star x x
Morning star x x x x x
Mace x x x
Weapon x x
weaponVenus, Evening star
Morning star mace
⊤
⊥
A too shallow analysis
cannot reveal the wrong
place of „morning star“

20. Steffen Staab 20
„Steffen went to Universität Koblenz-Landau.“
Interpretation 1: Steffen studied at Universität Koblenz-Landau.
Interpretation 2: Steffen went to the location
where the premises of Universität Koblenz-Landau are.
Interpretation 2a: Steffen went to Koblenz.
Interpretation 2b: Steffen went to Landau.

21. Steffen Staab 21
From one forefather of FCA
Charles Saunders Peirce
Four incapacities:
3. No power of thinking without signs. A cognition must be interpreted in a subsequent
cognition in order to be a cognition at all.
However: our thinking allows for ambiguity.
Just as our language does.

22. Steffen Staab 22
 Paradigm-changing results:
 2018: ULMFiT, ELMo, OpenAI GPT, BERT
 2019: OpenAI GPT-2
 Map a sequence of terms from vocabulary 𝑉
onto a sequence of vectors in concept space ℝ 𝑛:
𝑓: 𝑉 𝑙
→ ℝ𝑙+𝑛
 Representation of a term in concept space is relative to all
other representations of terms in concept space
Contextualized Word Representations
22
In recent evaluations of natural language systems
(QA, SWAG, Entity recog.,...),
proper usage of BERT led to best overall systems
Ongoing
competition!!

23. Steffen Staab 23
 Map a sequence of terms from vocabulary 𝑉
onto a sequence of vectors in concept space ℝ 𝑛:
𝑓: 𝑉 𝑙
→ ℝ𝑙+𝑛
 Is this useful to deal with ambiguity of attributes?
Contextualized Word Representations vs.
Concepts in Application Context
23
Our (ongoing) objectives
 Derive hierarchically ordered synonym sets
 Allow inference for similarity and inclusion
 Create taxonomy

24. Steffen Staab 24
 Input
 term sequence
 Method
 Compute contextualized representations of terms in
concept space
 Cluster representations in concept space
 Output: Representations for concept clusters
 E.g. multi-variate mixtures of Gauß distributions
Concept Representations
24

25. Steffen Staab 25
Explorative Analysis
25

26. Steffen Staab 26
Preliminary observations
26
 Averages of concept vectors are good representations
 Concept vectors of hyponyms show less variance than of hypernyms
 Ambiguous words
 Individual concept representations are close to concept representations
of similar terms

27. Steffen Staab 27
Intermediate summary
• Recent deep
learning/numerical methods
– hugely successful for
addressing a range of
natural language processing
tasks
– Because they are good at
representing
similarity/analogy
– Because they deal with
ambiguity
• But: it is hard to understand
what they do
• FCA working with
application context might be
useful
– Explicit
– Explaining
Let‘s use FCA for analysing vocabulary

28. Steffen Staab 28
Concepts in software languages
Public Domain,
https://commons.wikimedia.org/w/index.php?curid=358991

29. Steffen Staab 29
Data Base 1 Data Base 2 Data Base 3
App A App B App B
Scenario

30. Steffen Staab 30
Example: How old are these students?
Query for all students, access age
Query fails during evaluation
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.age)
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person

31. Steffen Staab 31
Example: How old are these students?
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.age)
Should we use this relation on this signifier?
Depends on application contexts:
1. Conceptualization of data source
2. Query of data source
3. Software code
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person

32. Steffen Staab 32
• alice and bob are Persons
– Implies having a name
• bob is a Student
– Implies being a Person and having a place to studyAt
• No restrictions with respect to age or matrNr
1 Conceptualization of data source
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person

33. Steffen Staab 33
1 Conceptualization of data source
SHACL: Shapes constraint language
SHACL shapes are integrity constraints
(Namespaces omitted for brevity)
:StudentShape a :NodeShape;
:targetClass :Student;
:class :Person;
:property [
:path :studiesAt;
:minCount 1;
:class :University;
].
:PersonShape a :NodeShape;
:targetClass :Person;
:property [
:path :name;
:minCount 1;
:datatype xsd:string;
].

34. Steffen Staab 34
1 Conceptualization of data source
Type checking discovers (potential) run-time errors
• Types interpreted as sets of values
• Based on tests for subsets
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.age)
Set of all students (StudentShape)
One value of
StudentShape
set
Not allowed since
StudentShape ⊈ ≥ 𝟏age.⊤
when considering
all conceptually possible RDF graphs

35. Steffen Staab 35
1 Conceptualization of data source
• Access: matrNr
• No error during evaluation
• Unsafe: Rejected by type checking,
conceptualization not guaranteed
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.matrNr)
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person

36. Steffen Staab 36
2 Query context
• Query for: matrNr
• Type safe access:
matrNr inferred to be given for all values of student
let students = query { SELECT ?x WHERE {?x matrNr ?y. } }
for student in students do
printfn „%A“ (student.matrNr)
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person

37. Steffen Staab 37
3 Type safety in one code context
• Accessing studiesAt relation
• Accepted as type safe
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.studiesAt)
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person

38. Steffen Staab 38
3 Lacking type safety in another code context
• Accessing studiesAt relation
• Not type safe for type Person
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ ( (Person)student . studiesAt)
bobalice 𝑏1
Student
UniversitysubClass
type
studiesAt
type
211... "Bob"
matrNr name
25 "Alice"
age name
Person

39. Steffen Staab 39
1. Use available SHACL constraints
2. Infer additional SHACL constraints from queries
3. Type check using inference
Determine type safety in context
let students = query { SELECT ?x WHERE {?x a Student. } }
for student in students do
printfn „%A“ (student.name)
Query shape(2) including StudentShape (1)
One value of
StudentShape
set
StudentShape ⊆ PersonShape and
PersonShape ⊆ ≥1name. ⊤ in all possible graphs
Inference (3)

40. Steffen Staab 40
Inference for type checking
Abstraction: If 𝜆 𝑥: 𝑇 . 𝑡 is a function
• 𝑥 is a variable of type 𝑇, 𝑡 is the body
• Type 𝑇 constitutes a set of values
(domain of the function)
Application: 𝜆 𝑥: 𝑇1 . 𝑡1 𝑡2
• Then question is: What type 𝑇2 is 𝑡2
(to what values can 𝑡2 evaluate to)?
• Can the function be applied to all possible evaluation results
(is 𝑇2 ⊆ 𝑇1 true)?

41. Steffen Staab 41
Intermediate summary
Application contexts
1. Conceptualization
2. Queries
3. Code
Open Questions
• Inference
– Sound
– Complete
– Efficient
• More polymorphism
Data integration leads to ambiguity

42. Steffen Staab 42
Conclusion

43. Steffen Staab 43
Pre-existing hypotheses
• Formal concept analysis is a useful tool to
explore natural and artificial languages
• Formal concept analysis is a useful tool to
explore our thinking using language
Recent paradigm changes (2018 / 2019)
– Understanding natural language text
– Understanding software code
Open question to you:
• How does / how can FCA help?
Main thrust of this talk

44. Steffen Staab 44
Open questions
• Representation of
ambiguities
– Representation of
underspecification
(as in Computational
Linguistics)
– Representation of
alternatives
• Sound and complete
inference for subsumption
between SHACL shapes
• Ambiguities →
Polymorphisms and more
type inference
Claim
Talking and thinking with ambiguities is efficient and effective!
How to support it?

45. Steffen Staab 45
Thank you very much!

46. Steffen Staab 46
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova: BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding. NAACL-HLT (1) 2019: 4171-4186
P. Cimiano, A. Hotho, S. Staab. Learning Concept Hierarchies from Text Corpora using Formal Concept
Analysis. JAIR - Journal of AI Research. 24: 305-339, 2005.
M. Leinberger, P. Seifer, C. Schon, R. Lämmel, S. Staab. Type Checking Program Code using SHACL.
In: Proc. of ISWC-2019. New Zealand, 2019.
Seifer, P., Leinberger, M., Lämmel, R., Staab, S.: Semantic query integration with reason. The Art, Science, and
Engineering of Programming 3(3) (2019). https://doi.org/10.22152/programming-journal.org/2019/3/13
References