Graph Search and Discovery for your Dark Data
To achieve new insights within your company you don't necessarily have to accumulate and crunch a lot of new tracking and tracing data - much key information already exists in disconnected data silos, stored in different databases and systems. By bringing the relevant pieces together in a single place and connecting the pieces of the puzzle at the right points you can gain new insights into your existing data and business. Graph databases offer the unique opportunity to easily cross-reference and connect disparate, variably structured datasets from many sources in a single place. It is then available for ad-hoc querying and exploration as well as strategic decision making. As a graph is a flexible and malleable data structure, it can evolve as you add new datasets or draw new connections from the insights or assumptions gained. This source of decision making will not be your old-school MDM data cemetery, but a flexible instrument that you can use, reuse, discard and recreate for different goals and purposes. Getting the data into the graph is only one interesting side of the coin, the other similarly challenging one is how to make the freshly spun web of knowledge available for different types of users. Technical users have it easy - they can yield Cypher, a powerful graph query language for arbitrary graph pattern matching, filtering, projection and aggregation of relevant data. For non-technical users a variety of tools and toolkits are available that use the inherent structures and meta information of the graph (labels, types, relationships and properties) to provide visual or natural language interaction to drill down, look at interesting facets, connect and correlate interesting information. Under the hood those tools use the same graph query language to do their job. This talk will cover both aspects of this exciting opportunity. We'll look at how to aggregate, import and connect information from disparate data-sources into a dynamic, flexible graph model and the means, tools and techniques for graph search and discovery on top of that fabric. The tools we use are the graph database Neo4j, its query language Cypher which also provides comprehensive import and data transformation abilities, as well as several tools and libraries from the Neo4j ecosystem to provide visual and textual graph search and exploration. Storyline Draft: http://gist.asciidoctor.org/?dropbox-14493611%2Ftalk_javazone.txt
Recomendados
Recomendados
Mais conteúdo relacionado
Destaque
Destaque (20)
Semelhante a Graph Search and Discovery for your Dark Data
Semelhante a Graph Search and Discovery for your Dark Data (15)
Mais de Neo4j
Mais de Neo4j (20)
Último
Último (20)
Graph Search and Discovery for your Dark Data
- 11. The+world+is+a+graph+–+everything+is+connected+ • people,&places,&events& • companies,&markets& • countries,&history,&poli4cs& • sciences,&art,&teaching& • technology,&networks,&machines,&& applica4ons,&users& • so7ware,&code,&dependencies,&& architecture,&deployments&
- 15. Meet+Jan+
- 16. Kontext+is+King+
- 24. NODE& key:&“value”& proper4es& Property+Graph+Model+ Nodes& • The&en44es&in&the&graph& • Can&have&name?value&proper%es' • Can&be&labeled& RelaEonships& • Relate&nodes&by&type&and&direc4on& • Can&have&name?value&proper%es' RELATIONSHIP& NODE& NODE& key:&“value”& proper4es& key:&“value”& proper4es& key:&“value”& proper4es&
- 25. CAR& name:&“Jan”& last:&Hansson& born:&1978& name:&“Anne”& last:&Olsen& since:&& Jan&10,&2011& brand:&“Volvo”& model:&“V70”& Property+Graph+Model+&+Example+ Nodes& • The&en44es&in&the&graph& • Can&have&name?value&proper%es' • Can&be&labeled& RelaEonships& • Relate&nodes&by&type&and&direc4on& • Can&have&name?value&proper%es' LOVES& LOVES& LIVES&WITH& PERSON& PERSON&
- 28. Your+friend+Neo4j+ An&openLsource$graph$database& • Manager+and+store&your&connected+ data+as&a&graph+ • Query+relaEonships&& easily&and&quickly& • Evolve+model+and+applicaEons++ to&support&new&requirements&and& insights& • Built&to&solve&relaEonal+pains++ &
- 29. Your+friend+Neo4j+ An&openLsource$graph$database$ & • Built+for+RelaEonships+ • Open+Source+ • Java+&+Scala+ • High+Performance+ • ACIDSDatabase+ &
- 32. Graph+Query+Language:+Cypher+ &&&&&&&&&&&&&&(jan:Person&{&name:“Jan”}&)&?[:LOVES]?>&(anne:Person&{&name:“Anne”}&)&& LOVES& Jan+ Anne+ NODE& NODE& LABEL& PROPERTY&LABEL& PROPERTY& MATCH+
- 35. Cypher:+Clauses+ • WITH&expr&AS&alias,&...& • UNWIND&coll&AS&item& • LOAD&CSV&FROM&„url“&AS&row&
- 37. GeYng+Data+into+Neo4j+ Load+JSON+with+Cypher+ • Send&JSON&as¶meter& • Deconstruct&the&document& • Into&a&non?duplicated&graph&model& UNWIND%{json}.items%as%item% MERGE%(:Person%{name:item.name,%% %%%%%%%%%%%%%%%%%age:toInt(item.age)});%
- 38. GeYng+Data+into+Neo4j+ CSV+Bulk+Loader++++neo4j/import' • For&ini4al&database&popula4on& • For&loads&with&10B+&records& • Up&to&1M&records&per&second& bin/neo4jOimport%–Ointo%people.db%% %OOnodes:Person%people.csv% %OOrelationship:FRIEND_OF%friendship.csv%
- 40. Let‘s+LOAD+
- 43. Twi]er+ • Run&a&twiber&search,&exclude&Neo4j&sources& • „neo4j&OR&#neo4j&OR&@neo4j&& &?from:@neo4j&?from:@neotechnology“& • Pass&resul4ng&JSON&directly&to&Cypher& (:Person)O[:TWEETED]O>(:Tweet:Content)O[:TAGGED]O>(:Tag)% (:Tweet)O[:MENTIONS]O>(:Person)% (:Tweet)O[:RETWEET]O>(:Tweet)% (:Tweet)O[:REPLY]O>(:Tweet)% (:Tweet)O[:CONTAINS]O>(:Link)%
- 45. Data+imported&
- 46. Connect!+& • twiber&handle& • email& • website& • name& • tags& • url&
- 53. So_ware+AnalyEcs+ So7ware&is&connected&informa4on& • Source&?>&AST& • Inheritance,&Composi4on,&Delega4on& • Call&Trees& • Run4me&Memory& • Dependencies& • Modules,&Libraries& • Tests& • ...& & hbps://jqassistant.org&
- 54. jQAssistant+ • GeekCruise:&My&first&Neo4j&project& • So_ware+deteriorates+ • Develop&rules&and&enforce&them& • Commercial&Tools&too&inflexible& • Open&Source&So7ware&...&& • Scanner&?>&Enhancer&?>&Analyzer& • Enrichment,&Concepts&and&Rulez&in&Cypher& • Scanner&Plugins& • Integrate&in&Build&Process& • Fail,&Generate&Reports,&...& hbp://jqassistant.org&
- 58. Background& Business&problem& Solu4on&&&Benefits& ©&All&Rights&Reserved&2014&|&Neo&Technology,&Inc.& Industry:'''Communica%ons' Use'case:''Network'Management' Paris,&France& • Large&French&communica4ons&company& • Tons&of&various&networking&infrasctructure& • Has&to&shut&down&equipment&for&maintenance& • Who+is+impacted?+How+to+compensate?+ • Data&lives&in&30+&systems& • Took&a&week&to&print,&reseach,&inform&...& • Un4l&...& 5 8
- 60. Background& Business&problem& Solu4on&&&Benefits& ©&All&Rights&Reserved&2014&|&Neo&Technology,&Inc.& Industry:'''Communica%ons' Use'case:''Network'Management' Paris,&France& • Second&largest&communica4ons&company& in&France& • Part&of&Vivendi&Group,&& partnering&with&Vodafone& 6 0