Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2nSKaat. Jim Webber explores the new Causal clustering architecture for Neo4j, showing that, despite the mixture of consensus protocols and asynchronous replication, Neo4j allows users to read their own writes straightforwardly, and discusses why this is such a difficult achievement in distributed systems. Filmed at qconlondon.com. Jim Webber is Chief Scientist with Neo Technology, the company behind the popular open source graph database Neo4j. He works on R&D for highly scalable graph databases and writes open source software. He has written two books on integration and distributed systems: “Developing Enterprise Web Services” and “REST in Practice”. His latest book is “Graph Databases” which focuses on the Neo4j database.

1. Causal Consistency For Large Neo4j Clusters
Jim Webber
Chief Scientist, Neo4j
QCon London

2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations
/causal-clustering-neo4j
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

8. Leads to a social graph

10. Motivation
Why do we need clusters of Neo4j?

11. Massive Throughput

12. Data Redundancy

13. Data Redundancy

14. Data Redundancy

15. Data Redundancy

16. High Availability

17. High Availability

18. High Availability
Error!
503: Service Unavailable

19. High Availability
Error!
503: Service Unavailable

20. High Availability
Error!
503: Service Unavailable

21. High Availability
Error!
503: Service Unavailable

22. High Availability
✓
Error!
503: Service Unavailable

23. Data RedundancyMassive Throughput High Availability

24. Data RedundancyMassive Throughput High Availability
3.0

25. Data RedundancyMassive Throughput High Availability
3.0
Bigger Clusters Consensus Commit Built-in load balancing
3.1
Causal
Clustering

26. Roles for safety and scale
Divide and conquer complexity

27. Read Replicas
Core

28. • Small group of Neo4j databases
• Fault-tolerant Consensus Commit
• Responsible for data safety
Core

29. Writing to the Core Cluster
Neo4j Driver Neo4j Cluster

30. Writing to the Core Cluster
Neo4j Driver
CREATE (:User {...})
✓
Neo4j Cluster

31. Writing to the Core Cluster
Neo4j Driver
CREATE (:User {...})
✓
Neo4j Cluster

32. Writing to the Core Cluster
Neo4j Driver
CREATE (:User {...})
✓
✓
✓
Neo4j Cluster

33. Writing to the Core Cluster
Neo4j Driver
CREATE (:User {...})
✓
✓
✓
Neo4j Cluster

34. Writing to the Core Cluster
Neo4j Driver
CREATE (:User {...})
✓
✓
✓
Neo4j Cluster

35. Writing to the Core Cluster
Neo4j Driver
✓
✓
✓
Success
Neo4j Cluster

36. Writing to the Core Cluster
Neo4j Driver
✓
✓
✓
Success
Neo4j Cluster
✓
✓

37. Raft Protocol
Non-Blocking Consensus for Humans

39. Raft Protocol
https://github.com/ongardie/raftscope

40. Raft in a Nutshell
• Raft keeps logs tied together (geddit?)
• Logs contain entries for both the database and the cluster membership
• Entries are appended and subsequently committed if a simple majority agree
• Implication: majority agree with the log as proposed
• Anyone can call an election: highest term (logical clock) wins, followed by highest committed,
followed by highest appended.
• Appended, but not committed, entries can be truncated, but this is safe (translates as transaction
aborted)
https://github.com/ongardie/raftscope

41. Consensus Log → Committed Transactions → Updated Graph
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11 12 13
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
Transaction log: the same transactions appear
in the same order on all members
Consensus log: stores both committed and
uncommitted transactions
Uncommitted
entries may differ
between
members
Transactionsareonlyappendedtothetransaction
logwhencommittedaccordingtoRaft
Transactionsareapplied,updatingthegraph
Neo4j Raft implementation

42. • Small group of Neo4j databases
• Fault-tolerant Consensus Commit
• Responsible for data safety
Core

43. • For massive query throughput
• Read-only replicas
• Not involved in Consensus Commit
• Disposable, suitable for auto-scaling
Read Replicas

44. Propagating updates to the Read Replicas
Neo4j Driver Neo4j Cluster

45. Propagating updates to the Read Replicas
Neo4j Driver Neo4j Cluster
Write

46. Propagating updates to the Read Replicas
Neo4j Driver Neo4j Cluster
Write

47. Reading from the Read Replicas
Neo4j Driver Neo4j Cluster
Read

48. Updating the graph Querying the graph

49. Read
Replicas
Core Updating the graph
Queries, analysis, reporting

50. ESTATE=$(neo-workbench estate add database -p Local -b core-block -s 3)
neo-workbench estate add database -p Local -b edge-block -s 10 $ESTATE
neo-workbench database install -m Core
--package-uri file:///Users/jim/Downloads/neo4j-enterprise-3.1.1-unix.tar.gz
-b core-block $ESTATE
neo-workbench database install -m Read_Replica
--package-uri file:///Users/jim/Downloads/neo4j-enterprise-3.1.1-unix.tar.gz
-b edge-block $ESTATE
neo-workbench database start $ESTATE

51. :sysinfo

52. Building an App
Computer science meets technology

53. App
Server
Neo4j
Driver
Bolt protocol

54. Java
<dependency>
<groupId>org.neo4j.driver</groupId>
<artifactId>neo4j-java-driver</artifactId>
</dependency>
Python
pip install neo4j-driver
.NET
PM> Install-Package Neo4j.Driver
JavaScript
npm install neo4j-driver

55. https://neo4j.com/developer/language-guides

56. bolt://
GraphDatabase.driver( "bolt://aServer" )

57. bolt+routing://
GraphDatabase.driver( "bolt+routing://aCoreServer" )

58. GraphDatabase.driver( "bolt+routing://aCoreServer" )
Bootstrap: specify any
core server to route load
across the whole cluster
bolt+routing://

59. Application
Server
Neo4j
Driver
Max
Jim
Jane
Mark

60. Routed write statements
driver = GraphDatabase.driver( "bolt+routing://aCoreServer" );
try ( Session session = driver.session( AccessMode.WRITE ) )
{
try ( Transaction tx = session.beginTransaction() )
{
tx.run( "MERGE (user:User {userId: {userId}})",
parameters( "userId", userId ) );
tx.success();
}
}

61. Routed read queries
driver = GraphDatabase.driver( "bolt+routing://aCoreServer" );
try ( Session session = driver.session( AccessMode.READ ) )
{
try ( Transaction tx = session.beginTransaction() )
{
tx.run( "MATCH (user:User {userId: {userId}})-[*]-(:Product) RETURN *",
parameters( "userId", userId ) );
tx.success();
}
}

62. Consistency models
Can you read what you write?

63. Cluster members slightly “ahead” or “behind” of each other
0 1 2 3 4 5 6 7 8 9 10
11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
If I query this server I
won’t see the updates
from transaction .
If I query this server,
I’ll see all updates
from all committed
transactions
11
11

64. Updating the graph Querying the graph

65. Register
Login
You need
to login in
to continue
your
purchase!

66. Register
Login
You need
to login in
to continue
your
purchase!
Username:
Password:
Create Account

67. Register
Login
You need
to login in
to continue
your
purchase!
Username:
jim_w
Password:
********
Create Account

68. Register
Login
You need
to login in
to continue
your
purchase!
Username:
Password:
Login

69. Username:
jim_w
Password:
********
Login

70. Purchase
Login
Successful
Try again
No account
found!Username:
jim_w
Password:
********
Login
𝙓

71. Username:
jim_w
Password:
********
A few moments later...
✓
Login

72. Purchase
Login
SuccessfulUsername:
jim_w
Password:
********
Login
A few moments later...
✓

73. Q Why didn’t this work?
A Eventual Consistency

74. 0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
Create
Account
App
Server A Driver

75. 0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver

76. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver

77. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver
11

78. 0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User)
Create
Account
App
Server A Driver
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
11

79. 0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User)
Create
Account
App
Server A Driver
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
11

80. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver
11

81. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver
MATCH (:User)
Login
App
Server B Driver
11

82. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver
MATCH (:User)
Login
App
Server B Driver
11

83. Bookmark
Session token
String (for portability)
Opaque to application
Represents ultimate user’s most recent view
of the graph
More capabilities to come

84. Let’s try again, with Causal Consistency

85. 0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
Create
Account
App
Server A Driver

86. 0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver

87. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver

88. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver
11

89. 0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User)
Create
Account
App
Server A Driver
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
11

90. 0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User)
Create
Account
App
Server A Driver
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9
11

91. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver
11

92. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9
CREATE (:User)
Create
Account
App
Server A Driver
MATCH (:User)
Login
App
Server B Driver
11

93. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
CREATE (:User)
Create
Account
MATCH (:User)
Login
App
Server A
App
Server B
Driver
Driver

94. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
CREATE (:User)
Create
Account
MATCH (:User)
Login
App
Server A
App
Server B
Driver
Driver
11

95. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
CREATE (:User)
Create
Account
MATCH (:User)
Login
App
Server A
App
Server B
Driver
Driver
11

96. Obtain bookmark
try ( Session session = driver.session( AccessMode.WRITE ) )
{
try ( Transaction tx = session.beginTransaction() )
{
tx.run( "CREATE (user:User {userId: {userId}, passwordHash:
{passwordHash})",
parameters( "userId", userId, "passwordHash", passwordHash ) );
tx.success();
}
String bookmark = session.lastBookmark();
}

97. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
CREATE (:User)
Create
Account
MATCH (:User)
Login
App
Server A
App
Server B
Driver
Driver
11
Obtain bookmark

98. Use a bookmark
try ( Session session = driver.session( AccessMode.READ ) )
{
try ( Transaction tx = session.beginTransaction( bookmark ) )
{
tx.run( "MATCH (user:User {userId: {userId}}) RETURN *",
parameters( "userId", userId ) );
tx.success();
}
}

99. 0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10 11
0 1 2 3 4 5 6 7 8 9 10
0 1 2 3 4 5 6 7 8 9 10
CREATE (:User)
Create
Account
MATCH (:User)
Login
App
Server A
App
Server B
Driver
Driver
11
Use bookmark

100. Read
Replicas
Core Updating the graph
Queries, analysis, reporting

101. Thank you for listening
@jimwebber

102. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/causal-
clustering-neo4j