This paper describes VIStology's HADRian system for semantically integrating disparate information sources into a common operational picture (COP) for humanitarian assistance/disaster relief (HADR) operations. Here the system is applied to the task of determining where unexploded or additional bombs were being reported via Twitter in the hours immediately after the Boston Marathon bombing in April, 2013. We provide an evaluation of the results and discuss future directions.

1. Situational Awareness from
Social Media
V I S T O L O G Y, I N C
BRIAN ULICNY
JAKUB MOSKAL
M I E C Z Y S L AW M . ( M I T C H ) K O K A R ( N O R T H E A S T E R N )
SEMANTIC TECHNOLOGIES FOR INTELLIGENCE DEFENSE
AND SECURITY (STIDS 2013)
GEORGE MASON UNIVERSITY
NOVEMBER 13, 2013

2. About VIStology, Inc.
 Past and present R&D contracts
 ONR, Army RDECOM , AFOSR, AFRL, DARPA, MDA
 Professional Collaborations
 Northeastern University, W3C, Lockheed
Martin, OMG, Referentia
Systems, BBN, Raytheon, Vulcan, Inc.
 Products & Services
 BaseVISor: highly efficient inference engine
 ConsVISor: consistency checker of ontologies
 PolVIsor: policy-compliant information exchange
 HADRian: next-generation smart COP for HA/DR ops
 Ontology Engineering and Systems Design
 Areas of Expertise
 Level 2+ Information Fusion, Situation Awareness,
Formal Reasoning Systems, Artificial Intelligence,
Ontology Engineering, Object-Oriented Design
2

3. VIStology HADRian
 HADRian is a next-generation COP for HA/DR ops
 HADRian enables an Incident Commander to
 Find
 Filter
 Geocode
and
 Display
the information that he or she needs to make the best
decisions about the situation on the basis of semantic
annotation of information repositories.
 Repositories may contain text
reports, photos, videos, tracks (KML), chemical
plumes, 3D building models (SketchUp), etc.
4

4. HADrian: Concept of Operations:
Find, Filter, Map and Display Info Re:Disasters
 COP Operator annotates repositories, using ontology
 Info from new repositories can be potentially integrated by annotating
metadata
 COP operator formulates High Level Query to describes info
needs for current operation
 System infers repositories that may contain relevant info by
reasoning over metadata that the repository has been
annotated with.


Information remains in place until need (not ETL);
Users upload data wherever they want; ingested as needed
 System issues appropriate low level query to repositories

Repositories contain disparate data in disparate formats
 System filters out irrelevant data
 System aggregates and displays data in Google Earth COP
 Users interact with data in COP
 COP Operator can send information from COP to responder phone
5

5. Dataflow for HADRian Situation Awareness
from Social Media
Twitter
Status Updates
Civilians/Media/
Others
Ontologybased
annotation
of metadata
Identifies relevant
repositories based on
metadata and high
level queries.
Here, only one
repository is relevant
Processes content of relevant
tweets
HADRian
Formulates high
level queries
based on
Commander
needs:
Here, “Where are
unexploded or
additional bombs
reported after
Boston
Marathon?”
Uses COP to
make decisions.
COP Operator
Identifies information needs.
Conveys to COP Operator
Field Operators
1. Identifies tweets that report
an additional or unexploded
bomb.
2. Identifies where the bomb is
reported
3. Geolocates the reported
location
4. Anonymizes the reporter‟s
identity
5. Presents the information in
HADRian COP
Commander
6

6. Current State of the Art: Fixed Feeds, On/Off
VIStology, Inc

7. High Level Query
“Show me Locations mentioned in Tweets
about Boston Marathon and Unexploded
Bombs”

8. Repository Annotation

9. Filter Processing Chain
The bulk of 0.5 million tweets were collected on April 15, 2013 in 3 hours after
the Boston Marathon bombing and stored in a giant CSV file. After this file is
determined to be relevant:
1) Using metadata description of schema and rules, the 0.5m tweets were filtered to only those
that contained mentions of unexploded or undetonated bombs, and converted into an OWL
representation. This file is identified as being relevant to the HLQ “find locations mentioned
in tweets about Boston Marathon and unexploded bombs at Boston Marathon on April
15, 2013”
2) The OWL-formatted tweets were loaded into BaseVISor along with rules. These rules
extract information about locations mentioned in the tweets - we call them "location
phrases", for instance: "jfk", "jfk library", "kennedy library", "#boston", etc.

10. Location Phrases
Locations referenced in tweet; NOT location of user, although that can help disambiguate
INFO Initializing BaseVISor..
INFO There are 26679 asserted facts in the knowledge
base.
INFO Initialization complete. Running inference...
INFO There is a total of 44414 facts in the knowledgebase
after running the inference.
INFO Done.
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
INFO
'mandarin hotel' -> 3
'bostonmarathon' -> 798
'bpd commissioner ed davis' -> 1
'back bay' -> 1 …
'jfk library' -> 311
'jfk' -> 73
'jfklibrary' -> 2
'jkf library' -> 1 …
'copley place' -> 8 …
„st. ignatius catholic church‟ -> 47

11. Mapping Location Phrases to Placemarks
For each extracted location phrase a rule with a special procedural
attachment is fired. This attachment, takes the location phrase and
tries to locate the place on the map using the following algorithm:
 1. (Places API, Exact) Lookup Google Maps Places API
(https://developers.google.com/places) and see if it returns exact
match for the phrase. If it does, use its Lat/Lon to assign the tweet
to that location, otherwise, move to next step.
 2. (Geocoding API) Lookup Google Geocoding API
(https://developers.google.com/maps/documentation/geocoding)
and see if it returns exact match for the phrase, if it does, use its
Lat/Lon to assign the tweet to that location, otherwise move to the
next step.
 3. (Places API, First on the list) Pick the first result in the Google
Maps Places API result (from 3a), if there were no results, ignore
the location.

12. 3 tweets mention Mandarin Hotel, 2 Copley
Place, 1 Back Bay Station,…

13. Information in Placemark
 We indicate the source of





the location inside the
placemark. (Here, Places
API first result)
Corresponding location
phrases
Photo
Number of tweets (1158)
Place type
(here, “library, museum,
establishment”)
Sample Tweets (not
shown)

14. Placemark Size and Color
The number next to the
placemark's name indicates
the number of tweets that
used on of the location
phrases mapping to this
location. The higher the
number, the more tweets
were talking about the same
spot.
We emphasize this fact by
rendering polygons
underneath the placemarks the higher and darker the
color, the more frequently
mentioned was the location.

15. 47 Tweets mention St Ignatius Church,
at Boston College

16. Relevant Tweet Retrieval (Finding) Evaluation
 Corpus: ~500K tweets
 Identified 7,748 tweets that were about additional or
unexploded bombs with a precision of 94.5%, based on
a random sample of 200 tweets identified as such.

That is, only 1.5% of the original corpus was identified as referring to
additional bombs, using our pattern matching.
 Based on a random sample of 236 tweets from the
original corpus, our recall (identification of tweets that
discussed additional bombs) was determined to be 50%.



That is, there were many more ways to refer to additional bombs
than our rules considered.
Thus, our F1 measure for accurately identifying tweets about
additional bombs was 65%.
Nevertheless, because of the volume of tweets, this did not affect the
results appreciably.

17. Location Phrase (Filtering) Evaluation
 Location phrases were identified purely by means of generic pattern






matching.
We did not use any list of known places. Nor did we include any
scenario-specific patterns.
The precision with which we identified location phrases was 95%.
That is, in 95% of the cases, when we identified a phrase as a
location phrase, it actually did refer to a location in that context.
Mistakes included temporal references and references to online
sites.
Our recall was only 51.3% if we counted uses of #BostonMarathon
that were locative. (We mishandled hashtags with camel case.)
If we ignore this hashtag, then our recall was 79.2%.

That is, of all the locations mentioned in tweets about additional bombs at the
Boston Marathon, we identified 79.2% percent of the locations that were
mentioned.
 Using the more lenient standard, our F1 measure for identifying
location phrases in the text was 86.3%.
18

18. GeoCoding Evaluation
 Our precision in associating tweets with known places via the
Google APIs was 97.2%.
 Our precision in assigning unique location phrases to
known places via Google APIs was 50%.
 That is, there were many location phrases that were repeated
several times that we assigned correctly to a known place, but
half of the unique phrase names that we extracted were not
assigned correctly.
 Ten location phrases that were extracted corresponded to no
known locations identified via the Google APIs.

These included location phrases such as “#jfklibrary” and “BPD
Commissioner Ed Davis”. The former is a phrase we would like to
geolocate, but lowercase hashtags which concatenate several words
are challenging. The latter is the sort of phrase that we expect would
be rejected as non-geolocatable.
19

19. Top 20 Locations by Frequency
Known Place
JFK Library
1158
Boston
629
Boston Marathon
325
St Ignatius Catholic Church
47
PD
29
Boylston
8
CNN
5
Copley Sq
4
Huntington Ave
4
Iraq
3
Mandarin Hotel
3
Dorchester
3
Marathon
3
US Intelligence
3
Copley Place
2
Boston PD
2
BBC
2
Cambridge
2
John
2
St James Street #Boston
20
#Tweets
2

20. Comparing Locations Mentioned in Media Blogs
 For three of these sites – Mass.
Location [Source]: (# of Tweets Identified with That Location
Boylston Street [Globe, CNN]: 8
Commonwealth Ave near Centre Street, Newton [Globe]: 0
Commonwealth Ave (Boston) [Globe]: 0
Copley Square [NYT]: 4
Harvard MBTA station [Globe]: 0
JFK Library [CNN, Globe, NYT]: 1158
Mass. General Hospital [Globe, NYT]: 0
(glass footbridge over) Huntington Ave near Copley place
[Globe]: 4
Tufts New England Medical Center [NYT]: 0
Washington Square, Brookline [NYT]: 0
21
General Hospital, Tufts Medical
Center and Washington Square,
Brookline -- reports of
unexploded bombs or suspicious
packages occurred after the end
of the tweet collection period, at
7:06 PM.
 Otherwise, the recall of our
system was good, missing only
the report of unexploded bombs
at the Harvard MBTA station.
 Media failed to report other
locations prominent to us (e.g. St
Ignatius Catholic Church)
 In our corpus, but missed, due to
capitalization.

21. Qualitative Evaluation
 On average, tweets reflecting same locations as
media blogs were produced 11 minutes prior to their
being reported on the sites mentioned.
 Thus, the tweet processing was more timely and
more comprehensive (included more locations) than
simply relying on a handful of news sites alone for
situational awareness
22

22. Conclusion
 We described a system for integrating disparate information sources into a





COP for Humanitarian Assistance/Disaster Relief operations by means of
semantic annotations and queries, using a common ontology.
We applied our technology to a repository of tweets collected in the
immediate aftermath of the Boston Marathon bombings in April, 2013, and
demonstrated that a ranked set of places could be incorporated into the
COP, showing the prominence of each site by tweet volume that was reported as
being the site of an additional unexploded bomb or bombs.
We evaluated the results formally and compared the results with the
situational awareness that could be gleaned only from mainstream media blogs
being updated at the same time.
On average, the automatic processing would have had access to locations from
tweets eleven minutes before these sites were mentioned on the mainstream
media blogs.
Additionally, sites that were prominent on Twitter (e.g. St Ignatius Church at
Boston College or the Mandarin Oriental Hotel in Boston) were not mentioned
on the news blog sites at all.
We believe that these results show that this approach is a promising one for
deriving situational awareness from social media going forward.
23