O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.1 © 2016 AGERO, INC. PROPRIETARY AND CONFID...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.2
• Agero, Inc. is a leading provider of ve...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.3
Who We Are
44 Years of
Experience
Financi...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.4
Creating Strong and Lasting Connections
B...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.5
Agero Connects Service Providers with Dri...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.6
Community Driven (Crowdsourced)
• Mappers...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.7
‘way’‘node’ ‘relation’
Hierarchy of Struc...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.8
<node id="261728686" lat="54.0906309" lon...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.9
Node and Ways are Relationships
1 Cabot R...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.10
OSM Complex and Dynamic Information
• Ro...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.11
Know about way segments
• Surface / spee...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.12
Maps are graphs
• Travel from start (nod...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.13
Bolt interface
• Native drivers for pyth...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.14
• Rectangular ‘bounding boxes’
• Points ...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.15
Map 2D to 1D space
• Create a bit string...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.16
Map 2D to 1D space
• Create a bit string...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.17
Encoding is Simple and Fast
Lat/Lon bits...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.18
Reverse the Process to Decode
• Base32 B...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.19
Quick to Determine Closeness
CITY GEOHAS...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.20
Trie data structure using GeoHash
• Walk...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.21
drt3j2yc
drt3j2y8
drt3j2ww
drt3j2wx
drt3...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.22
Supports and Models our Business Domain
...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.23
© 2016 AGERO, INC. PROPRIETARY AND CONFI...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.24
© 2016 AGERO, INC. PROPRIETARY AND CONFI...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.25
Select Node Component
Terminal nodes of ...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.26
Node is Displayed – Part of 2 Ways
2 (co...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.27
Connected Way
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.28
Property Change for Way – One is a Bridge
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.29
Graph Data Model Superior for Relationsh...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.30
The division of storage
facilitates grap...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.31
Neo4j Physical Storage
Properties
Node
L...
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.32
Open Street Map in Neo4j
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.33
Neo4j OSM Spatial Plugin
© 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.34
Neo4j Spatial Stored Procedures
Próximos SlideShares
Carregando em…5
×

Driving Predictive Roadway Analytics with the Power of Neo4j

624 visualizações

Publicada em

This talk will present how a small start-up (Waveonics) with no prior graphical database experience employed the simplicity, versatility and flexibility of Neo4j to handle the complex and dynamic roadway conditions found in crowdsourced street map data. We map the route taken to transit from a relational to graphical data model in support of the next generation roadside service applications for an established leader in the roadside service industry (Agero). We will show how Waveonics and Agero used Neo4j to detect and understand changing roadway conditions in order to identify emerging trends and thereby improve driver safety and the driving experience.

Recent advances in mobile device sensors, GPS location services, communication technology, crowdsourced data acquisition, affordable cloud storage, machine learning and data analytics, has opened the door to an exciting world of new services. Capitalizing on these opportunities requires providers continually organize, store, manage, access and update terabytes of information efficiently. They need databases that are powerful enough to model the complexities of the business environment and easy to modify as the environment evolves - without sacrificing the performance demanded by data analytics. Waveonics is working with Agero Mobile to explore and validate the use of graphical databases (Neo4j) to drive Agero’s mission of delivering a “holistic approach [which] helps anticipate and meet customers’ needs in a way that forms lasting connections with them.” Agero is building on Neo4j to further its leadership position in the automotive, insurance, finance and roadside service industries.

Waveonics will outline best practices used to convert 175 GB of XML relational US street map data obtained from the Open Street Map (OSM) project into a Neo4j graph database in order to successfully enable predictive roadway analytics for Agero. Attendees will learn how Neo4j’s graph data model and Cypher query language effortlessly supported an elegant representation of street map data, continually updated from customer mobile sensors, to reflect evolving road conditions.

Publicada em: Software
  • Seja o primeiro a comentar

Driving Predictive Roadway Analytics with the Power of Neo4j

  1. 1. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.1 © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY. Blake Nelson | Principal of Waveonics Deve Palakkattukudy|Principal Software Engineer, Mobile Engineering Agero October 13, 2016 Driving Predictive Roadway Analytics with the Power of Neo4j 1
  2. 2. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.2 • Agero, Inc. is a leading provider of vehicle and driver safety, security and information services. • Waveonics, LLC is a software development and consulting firm • Together, they are leveraging Neo4j 3.x, open source Spatial plug-in and crowdsourced Open Street Map data to: - Detect changing roadway and driving conditions - Analyze dynamic conditions for developing trends - Predict potential consequences of developing trends - Improve driver safety and the driving experience Enhance Driving Safety and Experience: Detect, Analyze, Predict, Improve
  3. 3. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.3 Who We Are 44 Years of Experience Financial Institutions Dealers & Repair Shops Auto Manufacturers Insurance Companies Road & Tow Companies
  4. 4. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.4 Creating Strong and Lasting Connections Between Our Clients and Their Drivers 9.5MILLION EVENTS SERVICES IN 75%VEHICLES 20CONNECTED VEHICLE YEARS 40YEARS INDUSTRY LEADER 350APIs
  5. 5. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.5 Agero Connects Service Providers with Drivers • Protecting over 85 million drivers • Over 9.5 million service dispatches each year What Are We Dealing With? (Business) Multiple Platforms • Mobile • Web & Cloud • Telephone Multiple Needs • Roadside Service • Data Analytics
  6. 6. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.6 Community Driven (Crowdsourced) • Mappers, GIS professionals, engineers, humanitarians providing accurate and up to date global map data • Databases, Local aerial imagery, GPS devices, low-tech field maps • Free to Use Credit to OSM and contributors http://www.openstreetmap.org Open Street Map (OSM)
  7. 7. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.7 ‘way’‘node’ ‘relation’ Hierarchy of Structural Information Open Street Map Data Files ‘way’ (route) are ordered sequence of ‘nodes’ • ‘way’ ID and sequence of ‘node’ IDs • Maybe ‘tags’ which are key/value properties ‘node’ (points) are the lowest level • Latitude, Longitude and ‘node’ ID • Maybe ’tags’ which are key/value properties ‘relation’ is highest (semantic) level • ‘relationship’ ID and… 1. Sequence of other relations 2. Sequence of ways 3. Sequence of nodes • Maybe tags which are key/value properties 1 2 3
  8. 8. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.8 <node id="261728686" lat="54.0906309" lon="12.2441924" /> <node id="1831881213" lat="54.0900666" lon="12.2539381"> <tag k="name" v="Neu Broderstorf"/> <tag k="traffic_sign" v="city_limit"/> </node> <way id="26659127" > <nd ref="292403538"/> <nd ref="298884289"/> ... <nd ref="261728686"/> <tag k="highway" v="unclassified"/> <tag k="name" v="Pastower Straße"/> </way> What does OSM data look like? Note: a number of attributes removed for brevity Timestamps, version, user, changeset, etc. Lat/Lon Points Road Segment
  9. 9. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.9 Node and Ways are Relationships 1 Cabot Rd #4 OSM way nodes in way Tags (properties) Revere Beach Parkway
  10. 10. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.10 OSM Complex and Dynamic Information • Roads, trails, waterway, regions, points of interest • 1.2M adds, 302K mods, 120K dels daily Scale and Complexity OSM is a Large Rich Data Set. For North America: • 862.4M ‘nodes’ (lat/lon points) • 60.4M ‘ways’ (sequence of related nodes) • 332M ‘tags’ (properties of nodes & ways) • 972K ‘relations’ (between ways and nodes)
  11. 11. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.11 Know about way segments • Surface / speeds / widths • Lanes / bridges / tunnels • Intersections / access • Hazards / obstructions What Do We Want To Do? EVERY WAY IS UNIQUE
  12. 12. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.12 Maps are graphs • Travel from start (node) to end (node) along roadways (edges) • Turns only possible at intersections (nodes) • Road segments have properties (speed, surface, lanes, etc.) Why Graph Database https://www.researchgate.net/figure/221252013_fig1_Fig-1-a-Example-of-road-map- extracted-from-a-city-street-map-b-Zones-shown-in https://neo4j.com/blog/neo4j-3-0-massive-scale-developer-productivity/
  13. 13. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.13 Bolt interface • Native drivers for python, java, javascript, .Net • Our Data Scientists work in Python Stored Procedures • Callable from Cypher through Bolt • Develop on client Migrate to server Why Neo4j 3? Applications Neo4j Execution EngineJava Stored Procedure
  14. 14. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.14 • Rectangular ‘bounding boxes’ • Points close to each other often in same bounding box • Performance problems at scale - Bboxes can overlap - Close points in can be in different bboxes - Poor worst-case performance • Splitting, rebalancing, etc. Neo4j Spatial Uses RTree Indexing https://en.wikipedia.org/wiki/R-tree
  15. 15. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.15 Map 2D to 1D space • Create a bit string by slicing the world by longitude & latitude • Convert the bit string into characters • Similar strings usually close Use GeoHash 1D Index https://mapzen.com/blog/geohashes-and-you/
  16. 16. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.16 Map 2D to 1D space • Create a bit string by slicing the world by longitude & latitude • Convert the bit string into characters • Similar strings usually close Use GeoHash 1D Index
  17. 17. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.17 Encoding is Simple and Fast Lat/Lon bits base32 Division and bit shift • (37.77564, -122.41365) • “0100110110010001111011110” • “9q8yy” Binary 01001 10110 01000 11110 11110 Decimal 9 22 8 30 30 Base 32 9 q 8 y y
  18. 18. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.18 Reverse the Process to Decode • Base32 Bits Latitude/Longitude Decoding is Simple and Fast Base 32 9 q 8 y y Decimal 9 22 8 30 30 Binary 01001 10110 01000 11110 11110 Longitude 0-0-1 -0-1- 0-0-0 -1-1- 1-1-0 Latitude -1-0- 1-1-0 -1-0- 0-1-0 -1-1-
  19. 19. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.19 Quick to Determine Closeness CITY GEOHASH LATITUDE LONGITUDE San Francisco 9q8yym901hw 37.77926 -122.41923 Oakland 9q9p1d5zfks 37.80531 -122.27258 Berkeley 9q9p3tvj8uf 37.86947 -122.27093 Los Angeles 9q5ctr60zyr 34.05366 -118.24276 New York City dr5regw2z6y 40.71273 -74.00599 London gcpvn0ntjut 51.50479 -0.07871 Greenwich u10hb5403uy 51.47651 0.00283 Things close are in same GeoHash region or a neighboring GeoHash region
  20. 20. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.20 Trie data structure using GeoHash • Walk tree using GeoHash string • Leaf node identifies Bbox Close things are identified by: • Same leaf node • Leaf node of Neighboring GeoHash ‘Fits’ with Neo4j relationships Fast Indexing with GeoHash Trie San Francisco 9q8yy 9 q 8 y y GeoHash Root
  21. 21. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.21 drt3j2yc drt3j2y8 drt3j2ww drt3j2wx drt3j2wm Walk Relationships Using GeoHash String One OSM Way Intersects Several GeoHash Regions
  22. 22. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.22 Supports and Models our Business Domain • Directed Graph Models Roadways with Properties • Spatial Plugin Supports OSM Data We Depend Upon Bolt Supports our Data Scientists • Python for Machine Learning and Predictive Analytics • Java for our Developers • Access to Stored Procedures when Needed Open Source Code / Open Source Community • Ability to Customize for Data Model and Performance Needs (e.g. Indexing) • Add Features with Plugin Technology Why Graph Database? Why Neo4j? https://github.com/codeforamerica/DemoDexter
  23. 23. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.23 © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY. Questions 23
  24. 24. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.24 © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY. Backup & Errata 24
  25. 25. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.25 Select Node Component Terminal nodes of this way are shared with connecting ways Different ways when Different property (properties)
  26. 26. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.26 Node is Displayed – Part of 2 Ways 2 (connecting) Ways for Terminal Node
  27. 27. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.27 Connected Way
  28. 28. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.28 Property Change for Way – One is a Bridge
  29. 29. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.29 Graph Data Model Superior for Relationships
  30. 30. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.30 The division of storage facilitates graph traversals • Nodes and Relationships Graph DB Model Optimized for Traversals Graph is node (vertex) and edges (relations) Storage structured on nodes connected by relations Locality of reference optimizes graph traversal
  31. 31. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.31 Neo4j Physical Storage Properties Node Label Relation
  32. 32. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.32 Open Street Map in Neo4j
  33. 33. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.33 Neo4j OSM Spatial Plugin
  34. 34. © 2016 AGERO, INC. PROPRIETARY AND CONFIDENTIAL. A CROSS COUNTRY GROUP COMPANY.34 Neo4j Spatial Stored Procedures

×