2. Narrative Map Augmentation with Automated Landmark Extraction and Path Inference 51
shopping in supermarkets and showed that independent blind travelers can navigate
modern supermarkets given adequate route descriptions.
Indoor and outdoor localization technologies can be augmented to better utilize
the traveler’s cognition. One approach to maximizing the traveler’s cognitive and
physical skills is narrative maps (www.clickandgomaps.com), i.e., verbal, egocentric
or allocentric, descriptions of specific environments. Narrative maps are written by
O&M professionals to take advantage of perceptual abilities of blind travelers, i.e.,
transitions from carpet to tile, obstacle detection, localization, shorelining, contextual
cues for orientation and re-orientation, etc. The production of narrative maps requires
the expertise of O&M professionals who must travel to designated environments and
describe large numbers of routes. Complete route coverage is rarely feasible due to
the sheer complexity of many environments. However, existing narrative maps can be
augmented by automated landmark extraction and path inference. In this paper, we
propose an algorithm that uses scalable natural language processing (NLP) to extract
landmarks and their connectivity from verbal route descriptions. Extracted landmarks
can be subsequently annotated with sensor readings (e.g., Wi-Fi clusters or digital
compass readings), used to find new routes, or track the traveler’s progress en route.
The paper is organized as follows. In Section 3, we outline the algorithm. In Sec-
tion 4, we present the experiments with the algorithm and discuss the results.
2 Landmark Extraction and Path Inference Algorithm
The conceptual basis of our algorithm is Kuipers’ Spatial Semantic Hierarchy (SSH)
[6], a hybrid knowledge representation framework for spatial cognition. In our pre-
vious study [2], the SSH was shown to be appropriate for the communication of ver-
bal routes to blind supermarket shoppers. The SSH represents environments in terms
of four levels: sensory, causal, topological, and metric. Of specific relevance to this
paper is the topological level of the SSH that describes the environment as maps of
places, paths, regions, and their connectivity and containment.
Fig. 1. Partial route description from Caribbean Ballroom 6 to Caribbean foyer at Caribe
Royale Convention Center Orlando from www.clickandgomaps.com
The input of our algorithms is verbal route descriptions, one of which is shown in
Fig. 1. The descriptions are split into sentences and the sentences are tokenized. The
tokenized sentences are tagged with parts of speech (POS) and parsed to identify noun
phrases (NPs) and verb phrases (VPs). We have used the Stanford Parser
(nlp.stanford.edu) for both POS tagging and parsing. Fig. 2 shows a parse tree with
POS tags for the sentence “Grand Sierra Ballroom foyer begins 75 feet ahead as the
3. 52 V. Kulyukin and T. Reddy
carpet changes to tile.” Landmarks are extracted by finding NP nodes from parse trees
and applying regular expressions to the corresponding text segments. Each landmark
receives a unique ID and is saved in an SQL database. VP nodes from parse trees and
regular expressions are used to extract actions and their parameters as well as land-
mark connectivity information. For example, from the sub-tree (VP (VB walk) (NP
(CD 3) (NNS steps))), the algorithm extracts the action WALK that can be paramete-
rized by the unit STEP quantified by numeral 3. If at least one action is detected be-
tween two landmarks, the landmarks are considered connected in a directed graph that
represents the connectivity of the environment. If it cannot be determined which
landmarks are connected by an extracted action, two virtual landmarks are generated
and stored in the database. New paths are inferred from landmark nodes and action
edges by finding landmarks common to a pair of routes, as shown in Fig. 3. The read-
er may consult [5] for more details.
Fig. 2. Results of POS Tagging and Parsing
Fig. 3. Path Inference
4. Narrative Map Augmentation with Automated Landmark Extraction and Path Inference 53
3 Experiments
The algorithm is implemented in Java and tested on 272 verbal route directions for
Caribe Royale Convention Center Orlando from www.clickandgomaps.com. The
algorithm extracted 421 landmarks and 884 action edges. Of 421 landmarks, 361
(86%) were true positives and 60 (14%) were false positives. Of 884 actions, 873
(98%) were true positives and 11 (2%) false positives. The algorithm also inferred
2,210 new paths.
References
1. Goldsmith, A.: Wireless Communications. Cambridge Press (2005)
2. Nicholson, J., Kulyukin, V., Coster, D.: ShopTalk: Independent Blind Shopping Through
Verbal Route Directions and Barcode Scans. The Open Rehabilitation Journal 2, 11–23
(2009)
3. Golledge, R.G., Klatzky, R.L., Loomis, J.M.: Cognitive Mapping and Wayfinding by
Adults without Vision. In: Portugali, J. (ed.) The Construction of Cognitive Maps. Kluwer
Academic Publishers, Dordrecht (1996)
4. Gaunet, F.: Verbal Guidance Rules for a Localized Wayfinding and Intended for Blind Pe-
destrians in Urban Areas. Universal Access in the Information Society 4(4), 338–353
(2006)
5. Kulyukin, V., Nicholson, J.: Toward Blind Travel Support through Verbal Route Direc-
tions: A Path Inference Algorithm for Inferring New Route Descriptions from Existing
Route Directions. The Open Rehabilitation Journal 5, 22–40 (2012)
6. Kuipers, B.: The Spatial Semantic Hierarchy. Artificial Intelligence 119, 191–233 (2000)