1. FluxGraph: A time-machine for your graphs
Davy Suvee
Michel Van Speybroeck
Janssen Pharmaceutica

2. about me
who am i ...
➡ working as an it lead / software architect @ janssen pharmaceutica
• dealing with big scientific data sets
• hands-on expertise in big data and NoSQL technologies
➡ founder of datablend
• provide big data and NoSQL consultancy
Davy Suvee • share practical knowledge and big data use cases via blog
@DSUVEE

4. graphs and time ...
➡ graphs are continuously changing ...

5. graphs and time ...
➡ graphs are continuously changing ...
➡ graphs and time ...
★ neo-versioning by david montag 1
2
★ representing time dependent graphs in neo4j by the isi foundation
★ modeling a multilevel index in neo4j by peter neubauer 3
1. http://github.com/dmontag/neo4j-versioning 2. http://github.com/ccattuto/neo4j-dynagraph/wiki 3. http://blog.neo4j.org/2012/02/modeling-multilevel-index-in-neoj4.html

6. graphs and time ...
➡ graphs are continuously changing ...
➡ graphs and time ...
★ neo-versioning by david montag 1
2
★ representing time dependent graphs in neo4j by the isi foundation
★ modeling a multilevel index in neo4j by peter neubauer 3
copy and relink semantics
๏ graph size
๏ object identity
๏ mixing data-model and time-model
1. http://github.com/dmontag/neo4j-versioning 2. http://github.com/ccattuto/neo4j-dynagraph/wiki 3. http://blog.neo4j.org/2012/02/modeling-multilevel-index-in-neoj4.html

7. FluxGraph ...
➡ towards a time-aware graph ...

8. FluxGraph ...
➡ towards a time-aware graph ...
➡ implement a blueprints-compatible graph on top of Datomic

9. FluxGraph ...
➡ towards a time-aware graph ...
➡ implement a blueprints-compatible graph on top of Datomic
➡ make FluxGraph fully time-aware
★ travel your graph through time
★ time-scoped iteration of vertices and edges
★ temporal graph comparison

10. travel through time
FluxGraph fg = new FluxGraph();

11. travel through time
FluxGraph fg = new FluxGraph();
Davy
Vertex davy = fg.addVertex();
davy.setProperty(“name”,”Davy”);

12. travel through time
FluxGraph fg = new FluxGraph();
Davy
Vertex davy = fg.addVertex();
davy.setProperty(“name”,”Davy”);
Peter
Vertex peter = ...

13. travel through time
FluxGraph fg = new FluxGraph();
Davy
Vertex davy = fg.addVertex();
davy.setProperty(“name”,”Davy”);
Peter
Vertex peter = ...
Vertex michael = ...
Michael

14. travel through time
FluxGraph fg = new FluxGraph();
Davy
kn
ow
Vertex davy = fg.addVertex();
s
davy.setProperty(“name”,”Davy”);
Peter
Vertex peter = ...
Vertex michael = ...
Edge e1 = Michael
fg.addEdge(davy, peter,“knows”);

15. travel through time
Davy
Date checkpoint = new Date();
kn
ow
s
Peter
Michael

16. travel through time
David
Date checkpoint = new Date();
kn
ow
s
davy.setProperty(“name”,”David”); Peter
Michael

17. travel through time
David
Date checkpoint = new Date();
kn
ow
s
davy.setProperty(“name”,”David”); Peter
kn
Edge e2 =
ow
fg.addEdge(davy, michael,“knows”);
s
Michael

18. travel through time
time
kn
Davy ow
s
Peter
Michael

19. travel through time
time
kn
Davy ow
s
checkpoint
Peter
Michael

20. travel through time
time
kn
Davy ow David
Davy
s
kn
ow
checkpoint
s
Peter Peter
kn
ow
s
Michael Michael

21. travel through time
time
kn
Davy ow David
Davy
s
kn
ow
checkpoint
s
current
Peter Peter
kn
ow
s
Michael Michael

22. travel through time by default
time
kn
Davy ow David
Davy
s
kn
ow
checkpoint
s
current
Peter Peter
kn
ow
s
Michael Michael

23. travel through time
time
kn
Davy ow David
Davy
s
kn
ow
checkpoint
s
current
Peter Peter
kn
ow
s
Michael Michael
fg.setCheckpointTime(checkpoint);

24. time-scoped iteration
t1
Davy

25. time-scoped iteration
t1 t2
change
Davy Davy’

26. time-scoped iteration
t1 t2 t3
change change
Davy Davy’ Davy’’

27. time-scoped iteration
t1 t2 t3 tcurrrent
change change change
Davy Davy’ Davy’’ Davy’’’

28. time-scoped iteration
t1 t2 t3 tcurrrent
change change change
Davy Davy’ Davy’’ Davy’’’
➡ how to find the version of the vertex you are interested in?

29. time-scoped iteration
t1 t2 t3 tcurrrent
Davy Davy’ Davy’’ Davy’’’

30. time-scoped iteration
t1 t2 t3 tcurrrent
next next next
Davy Davy’ Davy’’ Davy’’’
previous previous previous

31. time-scoped iteration
t1 t2 t3 tcurrrent
next next next
Davy Davy’ Davy’’ Davy’’’
previous previous previous
Vertex previousDavy = davy.getPreviousVersion();

32. time-scoped iteration
t1 t2 t3 tcurrrent
next next next
Davy Davy’ Davy’’ Davy’’’
previous previous previous
Vertex previousDavy = davy.getPreviousVersion();
Iterable<Vertex> allDavy = davy.getNextVersions();

33. time-scoped iteration
t1 t2 t3 tcurrrent
next next next
Davy Davy’ Davy’’ Davy’’’
previous previous previous
Vertex previousDavy = davy.getPreviousVersion();
Iterable<Vertex> allDavy = davy.getNextVersions();
Iterable<Vertex> selDavy = davy.getPreviousVersions(filter);

34. time-scoped iteration
t1 t2 t3 tcurrrent
next next next
Davy Davy’ Davy’’ Davy’’’
previous previous previous
Vertex previousDavy = davy.getPreviousVersion();
Iterable<Vertex> allDavy = davy.getNextVersions();
Iterable<Vertex> selDavy = davy.getPreviousVersions(filter);
Interval valid = davy.getTimerInterval();

35. time-scoped iteration
➡ When does an element change?

36. time-scoped iteration
➡ When does an element change?
➡ vertex:
★ setting or removing a property
★ add or remove it from an edge
★ being removed

37. time-scoped iteration
➡ When does an element change?
➡ vertex: ➡ edge:
★ setting or removing a property ★ setting or removing a property
★ add or remove it from an edge ★ being removed
★ being removed

38. time-scoped iteration
➡ When does an element change?
➡ vertex: ➡ edge:
★ setting or removing a property ★ setting or removing a property
★ add or remove it from an edge ★ being removed
★ being removed
➡ ... and each element is time-scoped!

39. temporal graph comparison
David
Davy Davy
kn
kn
ow
ow
s
s
Peter what changed? Peter
kn
ow
s
Michael Michael
current checkpoint

40. temporal graph comparison
➡ difference (A , B) = union (A , B) - B
➡ ... as a (immutable) graph!

41. temporal graph comparison
➡ difference (A , B) = union (A , B) - B
➡ ... as a (immutable) graph!

42. temporal graph comparison
➡ difference (A , B) = union (A , B) - B
➡ ... as a (immutable) graph! David
difference ( , )=
kn
ow
s

43. FluxGraph ...
➡ available on github
http://github.com/datablend/fluxgraph

44. use case: longitudinal patient data
t1 t2 t3 t4 t5
smoking smoking death
patient patient patient patient patient
cancer cancer

45. use case: longitudinal patient data
➡ historical data for 15.000 patients over a period of 10 years (2001- 2010)

46. use case: longitudinal patient data
➡ historical data for 15.000 patients over a period of 10 years (2001- 2010)
➡ example analysis:
★ if a male patient is no longer smoking in 2005
★ what are the chances of getting lung cancer in 2010, comparing
patients that smoked before 2005
patients that never smoked

47. use case: longitudinal patient data
➡ get all male non-smokers in 2005
fg.setCheckpointTime(new DateTime(2005,12,31).toDate());

48. use case: longitudinal patient data
➡ get all male non-smokers in 2005
fg.setCheckpointTime(new DateTime(2005,12,31).toDate());
Iterator<Vertex> males =
fg.getVertices("gender", "male").iterator()

49. use case: longitudinal patient data
➡ get all male non-smokers in 2005
fg.setCheckpointTime(new DateTime(2005,12,31).toDate());
Iterator<Vertex> males =
fg.getVertices("gender", "male").iterator()
while (males.hasNext()) {
Vertex p2005 = males.next();
boolean smoking2005 =
p2005.getEdges(OUT,"smokingStatus").iterator().hasNext();
}

50. use case: longitudinal patient data
➡ which patients were smoking before 2005?
boolean smokingBefore2005 =
((FluxVertex)p2005).getPreviousVersions(new TimeAwareFilter() {
public TimeAwareElement filter(TimeAwareVertex element) {
return element.getEdges(OUT, "smokingStatus").iterator().hasNext()
? element : null;
}
}).iterator().hasNext();

51. use case: longitudinal patient data
➡ which patients have cancer in 2010
working set of smokers
Graph g =
fg.difference(smokerws,
time2010.toDate(),
time2005.toDate());

52. use case: longitudinal patient data
➡ which patients have cancer in 2010
working set of smokers
Graph g =
fg.difference(smokerws,
time2010.toDate(),
time2005.toDate());
➡ extract the patients that have an edge to the cancer node

54. gephi plugin for fluxgraph 2010

55. gephi plugin for fluxgraph 2001

56. gephi plugin for blueprints!
1
➡ available on github
http://github.com/datablend/gephi-blueprints-plugin
➡ Support for neo4j, orientdb, dex, rexter, ...
1. Kudos to Timmy Storms (@timmystorms)

57. Questions?