Broad-coverage knowledge bases (KBs) such as Wikipedia, Freebase, Microsoft's Satori and Google's Knowledge Graph contain structured data describing real-world entities. These data sources have become increasingly important for a wide range of intelligent systems: from information retrieval and question answering, to Facebook's Graph Search, IBM's Watson, and more. Previous work on learning to populate knowledge bases from text has, for the most part, made the simplifying assumption that facts remain constant over time. But this is inaccurate - we live in a rapidly changing world. Knowledge should not be viewed as a static snapshot, but instead a rapidly evolving set of facts that must change as the world changes. In this paper we demonstrate the feasibility of accurately identifying entity-transition-events, from real-time news and social media text streams, that drive changes to a knowledge base. We use Wikipedia's edit history as distant supervision to learn event extractors, and evaluate the extractors based on their ability to predict online updates. Our weakly supervised event extractors are able to predict 10 KB revisions per month at 0.8 precision. By lowering our confidence threshold, we can suggest 34.3 correct edits per month at 0.4 precision. 64% of predicted edits were detected before they were added to Wikipedia. The average lead time of our forecasted knowledge revisions over Wikipedia's editors is 40 days, demonstrating the utility of our method for suggesting edits that can be quickly verified and added to the knowledge graph.

1. Learning to Extract Events from
Knowledge Base Revisions
Alexander Konovalov
Ohio State University
konovalov.2@osu.edu
Benjamin Strauss
Ohio State University
strauss.105@osu.edu
Alan Ritter
Ohio State University
ritter.1492@osu.edu
Brendan O'Connor
University of Amherst
brenocon@cs.umass.edu

2. Knowledge Bases
Some notable examples:
● Google Knowledge Graph
● Microsoft Satori KB
● Wolfram Alpha
● Wikidata
● DBPedia
● Wikipedia Infoboxes
● Freebase

3. Knowledge Bases
Some notable examples:
● Google Knowledge Graph
● Microsoft Satori KB
● Wolfram Alpha
● Wikidata
● DBPedia
● Wikipedia Infoboxes
● Freebase

4. Extracting KBs from
Text
Prior work assumes static text
corpora and knowledge bases:
● NELL [AAAI 2015]
● Mintz et al. [ACL 2009]
● DeepDive [VLDS 2012]
● Knowledge Vault [KDD 2014]

5. Real-time Knowledge Bases
Knowledge Bases are not static snapshots!
Events are constantly changing the world.

6. Real-time Knowledge Bases
Knowledge Bases are not static snapshots!
Events are constantly changing the world.

7. Wikipedia: a dynamically evolving knowledge base

8. Wikipedia: a dynamically evolving knowledge base

9. Wikipedia: a dynamically evolving knowledge base

10. Wikipedia: a dynamically evolving knowledge base

11. 50% of deaths updated within
a couple of days
Manual Editing…

12. 50% of deaths updated within
a couple of days
... but for scientists take 31 days to
reach 50% coverage!*
Manual Editing…
can not Scale Up!
* https://research.googleblog.com/2013/05/distributing-edit-history-of-wikipedia.html
Goal: extract KB revisions from text as
soon as public knowledge is available

13. Infobox
Revisions
News
Event
extraction
pipeline
Training
Social Media
In a nutshell

14. Prediction
Infobox
Revisions
News
Event
extraction
pipeline
Social Media
In a nutshell

15. In a nutshell

16. X welcomes Y as our new mayor
X sworn in as Y mayor
In a nutshell

17. X welcomes Y as our new mayor
X sworn in as Y mayor
In a nutshell

18. In a nutshell
X welcomes Y as our new mayor
X sworn in as Y mayor{{infobox Settlement
| name=Pittsburgh
| Leader=Bill Peduto

19. In a nutshell: formally.

20. In a nutshell: formally.
● An event is a quadruple of a
predicate, two entities, and a
time.

21. In a nutshell: formally.
● Features are collected from tweets written
before time t.
Prediction time
Feature window

22. In a nutshell: formally.
● A bag of Mintz et al. like features.
State arg1 wins in arg2
cnn projects arg1 wins IN arg2
0.333
0.115
CurrentTeam arg1 to arg2
arg1 joins arg2
arg1 signs...with arg2
1.291
0.513
0.419
DeathPlace arg1 found dead in arg2
arg1 found JJ IN arg2
arg1 killed in arg2
arg1 memorial...in arg2
0.269
0.260
0.195
0.125

23. In a nutshell: formally.
● L₂ regularized log-linear model.
● Optimize conditional likelihood with
respect to θ.

24. Challenge: Non-Semantic Edits
| spouse = Soumaya Domit (1967–1999; her
death); 6 children
| spouse = Soumaya Domit
<small>(1967-1999; her death)</small>

25. Challenge: Vandalism
|leader_name = Chuck Norris|leader_name = Dave Paulekas

26. Revision as of 17:27,
7 April 2015
➕ | death_date = {{death date and age
|2015|02|03|1978|05|29}}}
➕ | death_place = [[Valhalla, New
York|Valhalla, NY]], U.S.
Challenge: Edits After Event

27. Experiments: datasets.
● Twitter (Social Media)
○ Training: Sep. 2008 - Jun. 2011
○ Test: Jun. 2011 - Jan. 2012
○ 10% firehose.
● Annotated Gigaword (News)
○ Training: Jun. 2004 - Jun. 2009
○ Test: Jun 2009 - Jan. 2011
○ [Napoles et al. 2012]

28. Experiments: datasets.
● Twitter:
○ langid.py [IJCNLP 2011]
○ Domain-specific Twitter NLP
[Ritter et al. 2011]
○ Down-sampling
Twitter
Firehose
langid.py
Tokens,
NE, POS
Down
sampling
Annotated
Samples

29. Experiments: datasets.
● Annotated Gigaword
○ POS, NER, Syntactic trees, etc...
○ Each sentence is a single data-point.
Annotated
Gigaword
Split into sentences
Annotated
Samples

30. Attributes
Six Wikipedia
infobox attributes:
● CurrentTeam
● LeaderName
● StateRepresentative
● Spouse
● Predecessor
● DeathPlace

31. Attributes
Six Wikipedia
infobox attributes:
● CurrentTeam
● LeaderName
● StateRepresentative
● Spouse
● Predecessor
● DeathPlace

32. Attributes
Six Wikipedia
infobox attributes:
● CurrentTeam
● LeaderName
● StateRepresentative
● Spouse
● Predecessor
● DeathPlace

33. Experiments: datasets.
● Intuition: Samples near the time of a
revision are likely to mention an event
that causes the change.
Annotated
Samples
Matched
Samples
Match NE
Aligned Samples
Use
intuition!
Unaligned Samples

34. Experiments: datasets.
● Intuition: Samples near the time…
Use temporal alignment (-10...+3 days).
Annotated
Samples
Matched
Samples
Match NE
Aligned Samples
Unaligned Samples
temporal
alignment

35. Experiments: baseline.
● Baseline: a traditional relation
extraction system that uses
no temporal information.
Annotated
Samples
Matched
Samples
Match NE
Aligned Samples
Unaligned Samples
temporal
alignment

36. Experiments: automatic evaluation.
● Evaluate assuming that
events = edits (positive).
● Mix 90% randomly sampled as negative
samples.

37. Experiments: automatic evaluation.
Twitter Gigaword
Attribute Our System Baseline Our System Baseline
Current Team 0.53 0.28 0.03 0.03
Leader Name 0.50 0.12 0.27 0.05
Spouse 0.27 0.22 0.08 0.05
State
Representative
0.01 0.002 0.43 0.29
Predecessor 0.02 0.09 0.24 0.17
Death Place 0.49 0.63 0.41 0.35

38. Experiments: human evaluation.
Automatic evaluation:
● Noise - underestimates precision & recall.
● Missing edits - overestimates recall.
● Unrealistic ratio of signal / noise.

39. Experiments: human evaluation.
● Split the predictions into month-long bins.
● Annotate the top predictions from both systems.

40. Experiments: evaluation.
● Human judgement via Amazon Mechanical Turk,
7 workers.

41. Experiments: evaluation.
● Inter-Annotator Agreement (Fleiss-Kappa):
○ 0.64 on Twitter
○ 0.30 on Gigaword
Context matters:
Much of Wednesday was about pomp
and circumstance for Rand and Ron
Paul. They were sworn in, [...] But the
newly minted senator from Kentucky
also tended to some business [...]

42. Experiments: All attributes.
Predictions per month
Precision
Twitter Gigaword

43. Experiments: All attributes.
Predictions per month
Precision
Twitter Gigaword
Our system
Baseline

44. Experiments: All attributes.
Predictions per month
Precision
Twitter Gigaword

45. Conclusions
● Revisions - a source for distant
supervision for event extraction.

46. Conclusions
● Revisions - a source for distant
supervision for event extraction.
● Use temporal and surface matching to
identify reliable infobox edits
corresponding to real world events.

47. Conclusions
● Revisions - a source for distant
supervision for event extraction.
● Use temporal and surface matching to
identify reliable infobox edits
corresponding to real world events.
● Automate KB updates by learning
predictors from past KB updates.

48. Conclusions
● We generate on average 34.3 edits per
month with high precision for 6
attributes.
● Often beat human KB contributors in
terms of recall (64%⇑ before) and lead
time (40 days).

49. Future Work
● Database revisions are not always
well-aligned:

50. Future Work
● Align edits to the actual events using
latent variable alignment model.
With an
offset

51. Q & A
Learning to Extract Events from Knowledge Base Revisions
You can reach me via konovalov.2@osu.edu.
Code and data will be available soon at github.com/alexknvl/dsup.