Recomendados
Recomendados
Mais conteúdo relacionado
Semelhante a Introduction to Graph Databases @ SAI
Semelhante a Introduction to Graph Databases @ SAI (20)
Mais de datablend
Último
Último (20)
Introduction to Graph Databases @ SAI
- 2. aboutme whoami... DavySuvee @DSUVEE ➡ bigdataarchitect@datablend-continuum • providebigdataandnosqlconsultancy • sharepracticalknowledgeandbigdatausecasesviablog
- 6. Whatisbigdata... BigData ...largeandcomplexdatasetsthataredifficultto processwithtraditionaldatabasemanagementtools... ➡ store (nosql) ➡ enrich (datamining,ml,nlp,...) ➡ visualize (d3,gephi,mapbox,tableau,...) ➡ process/analyze (map/reduce,cep,storm,...)
- 8. Tacklingthevolumeproblem... ➡ Throwingourdataaway :-( Whatwearecurrentlydoing... ➡ Storingpreprocesseddata :-/ ➡ Trytostoreitanyway ;-( Butwhy?
- 13. Tacklingthevolumeproblem... VerticalScaling € 4 HorizontalScaling € x #nodes Your database NoSQL
- 16. Tacklingthevelocityproblem... ➡ Collect Wewantto ... ➡ Process ➡ Query in Real-Time MASSIVEamountsof Unstructured data ➡ Analyze
- 17. Tacklingthevelocityproblem... Slowandoutdatedinformation Fastandrealtime Your stack NoSQL & Big Data BI ETL APP SYNC SYNC APP Map-Reduce BI (+ANALYTICS)
- 19. alittlebitofgraphtheory... Davy age = 33 Datablend btw = 123... node/vertex Janssen sector = pharma Kim age = 26 gender = F edge founded in: 2011 worked_for from: 2008 to: 2013 knowssince: 2013
- 20. Advantages...? ➡ whiteboardfriendly ➡ schema-less ➡ index-freeadjacency(nojoins!) Graph Database ➡ queriesastraversals ➡ queriesaspatternmatching
- 21. Advantages...?
- 22. Products/projects...? ➡ databases:neo4j,orientdb,allegrograph,dex,... ➡ processing:pregel,giraph,hama,goldenorb,... ➡ APIs:blueprints Graph Database ➡ querylanguages:gremlin,cypher,sparql
- 23. Graphdatabase101 (neo4j) GraphDatabaseService graph = ... Node davy = graph.createNode(); davy.setProperty(“name”,”Davy”); Davy KimNode kim = graph.createNode(); kim.setProperty(“name”,”Kim”);
- 24. Graphdatabase101 (neo4j) enum RelTypes implements RelationshipType { KNOWS, WORKED_FOR, FOUNDED } Davy Kim knows Relationship davy_kim = davy.createRelationshipTo(kim, RelTypes.KNOWS) davy_kim.setProperty(“since”, 2013);
- 25. Graphdatabase101 (neo4j) Relationship davy_datablend = davy.createRelationshipTo( datablend, RelTypes.FOUNDED) davy_datablend.setProperty(“in”, 2011); Davy Datablend founded ➡ howtoaccessthedatablend node?
- 26. Graphdatabase101 (neo4j) Index<Node> nodeIndex = graph.index().forNodes(“nodes”); Node datablend = graph.createNode(); datablend.setProperty(“name”,”Datablend”); nodeIndex.add(datablend, “name”, “Datablend”); Node found = nodeIndex.get(“name”,”Datablend”).getSingle();
- 27. Graphdatabase101 (neo4j) ➡ findfriendsofmyfriends... TraversalDescription td = Traversal.description() .breadthFirst() .relationships(RelTypes.KNOWS, Direction.OUTGOING) .evaluator(Evaluators.toDepth(2)); Traverser traverser = td.traverse(davy); for (Path path : traverser) { ... }
- 28. Graphdatabase101 (neo4j) ➡ findfriendsofmyfriends... START davy=node:node_auto_index(name = “Davy”) MATCH davy-[:KNOWS]->()-[:KNOWS]->fof RETURN davy, fof ExecutionEngine engine = new ExecutionEngine(graph); ExecutionResults result = engine.execute(query); for(Map<String,Object> row : result) { ... }
- 29. Usecases...? ➡ recommendations ➡ accesscontrol ➡ routing Graph Database ➡ socialcomputing/networks ➡ genealogy
- 30. insightsinbigdata ➡ typicalapproachthroughwarehousing ★ starschemawithfacttablesanddimensiontables
- 31. insightsinbigdata ➡ typicalapproachthroughwarehousing ★ starschemawithfacttablesanddimensiontables
- 32. insightsinbigdata ➡ typicalapproachthroughwarehousing ★ starschemawithfacttablesanddimensiontables
- 33. insightsinbigdata ★ real-timevisualization ★ filtering ★ metrics ★ layouting ★ modular1,2 1.http://gephi.org/plugins/neo4j-graph-database-support/ 2.http://github.com/datablend/gephi-blueprints-plugin
- 34. geneexpressionclustering ★ 4.800samples ★ 27.000genes ➡ oncologydataset: ➡ Question: ★ for a particular subset of samples, whichgenesareco-expressed?
- 35. mongodbforstoringgeneexpressions { "_id" : { "$oid" : "4f1fb64a1695629dd9d916e3"} , "sample_name" : "122551hp133a21.cel" , "genomics_id" : 122551 , "sample_id" : 343981 , "donor_id" : 143981 , "sample_type" : "Tissue" , "sample_site" : "Ascending colon" , "pathology_category" : "MALIGNANT" , "pathology_morphology" : "Adenocarcinoma" , "pathology_type" : "Primary malignant neoplasm of colon" , "primary_site" : "Colon" , "expressions" : [ { "gene" : "X1_at" , "expression" : 5.54217719084415} , { "gene" : "X10_at" , "expression" : 3.92335121981739} , { "gene" : "X100_at" , "expression" : 7.81638155662255} , { "gene" : "X1000_at" , "expression" : 5.44318512260619} , … ] }
- 36. pearsoncorrelationthroughmap-reduce pearson correlation x y 43 99 21 65 25 79 42 75 57 87 59 81 0,52
- 44. analyzingrunningdata <trkpt lon="4.723870977759361" lat="51.075748661533"> <ele>29.799999237060547</ele> <time>2011-11-08T19:18:39.000Z</time> </trkpt> <trkpt lon="4.724105251953006" lat="51.075623352080584"> <ele>29.799999237060547</ele> <time>2011-11-08T19:18:45.000Z</time> </trkpt> <trkpt lon="4.724143054336309" lat="51.07560558244586"> <ele>29.799999237060547</ele> <time>2011-11-08T19:18:46.000Z</time> </trkpt>
- 45. analyzingrunningdatathroughneo4j ➡ usingneo4jspatialextension ➡ createanodeforeachtrackedpoint List<GeoPipeFlow> closests = GeoPipeline.startNearestNeighborLatLonSearch( runningLayer, to, 0.02). sort("OrthodromicDistance"). getMin("OrthodromicDistance").toList(); ➡connectsucceedingtrackingnodesinagraph
- 48. graphsandtime... ➡ fluxgraph:ablueprints-compatiblegraphontopofDatomic ➡ makeFluxGraphfullytime-aware ★ travelyourgraphthroughtime ★ time-scopediterationofverticesandedges ★ temporalgraphcomparison ➡ towardsatime-awaregraph... ➡ reproduciblegraphstate
- 49. travelthroughtime FluxGraph fg = new FluxGraph();
- 50. travelthroughtime FluxGraph fg = new FluxGraph(); Vertex davy = fg.addVertex(); davy.setProperty(“name”,”Davy”); Davy
- 51. travelthroughtime FluxGraph fg = new FluxGraph(); Vertex davy = fg.addVertex(); davy.setProperty(“name”,”Davy”); Davy Kim Vertex kim = ...
- 52. travelthroughtime FluxGraph fg = new FluxGraph(); Vertex davy = fg.addVertex(); davy.setProperty(“name”,”Davy”); Peter Davy Kim Vertex kim = ... Vertex peter = ...
- 53. travelthroughtime FluxGraph fg = new FluxGraph(); Vertex davy = fg.addVertex(); davy.setProperty(“name”,”Davy”); Peter Davy Kim Vertex kim = ... Vertex peter = ... Edge e1 = fg.addEdge(davy, kim, “knows”); knows
- 55. travelthroughtime Date checkpoint = new Date(); Peter Davy Kim knows
- 56. travelthroughtime Date checkpoint = new Date(); davy.setProperty(“name”,”David”); Peter Davy Kim knows
- 57. travelthroughtime Date checkpoint = new Date(); davy.setProperty(“name”,”David”); Peter Kim knows David
- 58. travelthroughtime Date checkpoint = new Date(); davy.setProperty(“name”,”David”); Peter Kim Edge e2 = fg.addEdge(davy, peter, “knows”); knows David knows
- 62. tcurrrentt3t2 time-scopediteration change change change Davy’’’Davy’ Davy’’ t1 Davy ➡howtofindtheversionofthevertexyouareinterestedin?
- 64. next next next previouspreviousprevious tcurrrentt3t2 time-scopediteration Davy’’’Davy’ Davy’’ t1 Davy
- 65. next next next previouspreviousprevious tcurrrentt3t2 time-scopediteration Davy’’’Davy’ Davy’’ t1 Davy Vertex previousDavy = davy.getPreviousVersion();
- 66. next next next previouspreviousprevious tcurrrentt3t2 time-scopediteration Davy’’’Davy’ Davy’’ t1 Davy Vertex previousDavy = davy.getPreviousVersion(); Iterable<Vertex> allDavy = davy.getNextVersions();
- 67. next next next previouspreviousprevious tcurrrentt3t2 time-scopediteration Davy’’’Davy’ Davy’’ t1 Davy Vertex previousDavy = davy.getPreviousVersion(); Iterable<Vertex> allDavy = davy.getNextVersions(); Iterable<Vertex> selDavy = davy.getPreviousVersions(filter);
- 68. next next next previouspreviousprevious tcurrrentt3t2 time-scopediteration Davy’’’Davy’ Davy’’ t1 Davy Vertex previousDavy = davy.getPreviousVersion(); Iterable<Vertex> allDavy = davy.getNextVersions(); Iterable<Vertex> selDavy = davy.getPreviousVersions(filter); Interval valid = davy.getTimerInterval();
- 70. temporalgraphcomparison ➡difference(A,B) = union(A,B)-B ➡...asa(immutable)graph! difference ( , ) = David knows
- 72. usecase:longitudinalpatientdata ➡ historicaldatafor15.000patientsoveraperiodof10years(2001-2010) ➡ exampleanalysis: ★ ifamalepatientisnolongersmokingin2005 ★ whatarethechancesofgettinglungcancerin2010,comparing patientsthatsmokedbefore2005 patientsthatneversmoked
- 74. OpenInnovationNetworkingTool ➡ Manydifferentprojects,manydifferentpartners,manydifferentdomains... ★ howdowekeeptrack? ★ howcanwelearnfromthedata? ➡ Storethedateinit’smostnaturalform,agraph ➡usegraphalgorithmstoidentifytheimportanceofeachnodeandtheirrelatedones
- 77. Moregraphs... ➡ pharma ➡ geospatial ➡ dependencyanalysis ➡ ontology ➡ ...
- 78. Questions?