Presented at COLING'18 (21-08-2018)

1. Systematic Study of
Long Tail Phenomena in
Entity Linking
Filip Ilievski, Piek Vossen, Stefan Schlobach

2. Entity Linking (EL)
“Washington announces Alex Smith trade
It seems like months ago that the Chiefs traded Alex Smith to Washington...
Smith, 33, originally entered ...”
(https://profootballtalk.nbcsports.com/2018/03/14/washington-announces-alex-smith-trade/)
surface form
instance
interpretation

3. State-of-the-art Entity Linking
SotA: High F1-scores by probabilistic optimization
F1-score
=> system skills ??
=> errors ??
~ data properties ??
“Washington announces Alex Smith trade
It seems like months ago that the Chiefs traded Alex Smith to Washington...
Smith, 33, originally entered ...”
(https://profootballtalk.nbcsports.com/2018/03/14/washington-announces-alex-smith-trade/)

4. Head and tail of Entity Linking
Claim: performance (head) >> performance (tail)
(Ilievski et al., 2016; van Erp et al., 2016; Esquivel et al., 2017)
head =? ∧ tail=?
=> performance (head) >> performance (tail) ??
=> how to improve performance (tail) ??

5. Contributions of this work
1. Description and hypotheses on the long tail properties of EL
2. Analysis of EL datasets WRT the long tail properties
3. Analysis of system performance WRT the long tail properties
4. Recommended actions

6. Ambiguity of forms
(number of different instances that a form refers to)
“Washington “
Variance of instances
(number of distinct forms that refer to an instance)
“... U.S. federal government” “Washington” “... government of U.S.
...”
Frequency of forms/instances
(number of occurrences in a corpus)
“Washington announces Alex Smith trade
It seems like months ago that the Chiefs traded Alex Smith to Washington.
Smith, 33, originally entered ...”
Popularity of instances
(PageRank in a knowledge graph)
Definition of long tail properties

7. Hypotheses and setup
16 hypotheses
2 data collections (CoNLL-AIDA and N3), 5 corpora in total
3 SotA systems: AGDISTIS MAG, DBpedia Spotlight, and WAT
Precision, recall and F1-score

8. Hypotheses on the data properties
Positive correlation between ambiguity and frequency of forms amb(f) ~ freq(f)
Positive correlation between variance, frequency, and popularity of instances var(i) ~ freq(i)
var(i) ~ pop(i)
freq(i) ~ pop(i)
Zipfian frequency distribution within all forms that refer to an instance freq(f|I) ~ zipfian
Zipfian frequency distribution within all instances that refer to a form freq(i|F) ~ zipfian

9. amb(f) ~ freq(f) var(i) ~ freq(i)
freq(i) ~ pop(i)var(i) ~ pop(i)

10. freq(f) ~ zipfian freq(i) ~ zipfian

11. Hypotheses on system performance
Systems perform worse on forms that are ambiguous than overall. f1(AMF) << f1(ALL)
Best performance on frequent, non-ambiguous forms;
worst performance on infrequent, highly ambiguous forms.
f1(freq, ⅂amb) = MAX(f1)
f1(⅂freq, amb) = MIN(f1)
Performance is inversely proportional with entropy. f1(AMF) ~ ⅂entropy(AMF)
Systems perform better on frequent/popular instances of ambiguous forms,
compared to their infrequent/unpopular instances.
f1(i|F) ~ freq(i|F)
f1(i|F) ~ pop(i|F)

12. f1(AMF) << f1(ALL)

13. f1(freq, ⅂amb) = MAX(f1)
f1(⅂freq, amb) = MIN(f1)

14. S4: Systems perform better on ambiguous forms with imbalanced,
compared to balanced, instance distribution
f1(AMF) ~ ⅂entropy(AMF)

15. f1(amb)<< f1(all)
f1(i|F) ~ freq(i|F)
f1(i|F) ~ pop(i|F)

16. Recommendations
[Dataset creation]
● statistics on the head and the tail
● most-frequent-value baseline
[Evaluation]
● evaluate on the head and the tail
● use macro F1-score
[System development]
● which heuristics target which cases
● which resources optimize for the head/tail

17. Conclusions
First work that systematically describes the relation of surface forms in EL corpora and their
instances in DBpedia, through long tail properties.
We measured expected inter-correlations between long tail phenomena in EL datasets.
System performance correlates positively with frequency and popularity of instances, and
negatively with ambiguity of forms.
We listed recommended actions to influence future designs of systems and datasets in EL.

18. Thanks for your attention!
Questions?
Github: cltl/EL-long-tail-phenomena
Twitter: @earthling91