O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

Causal Consistency for Large Neo4j Clusters

549 visualizações

Publicada em

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2nSKaat.

Jim Webber explores the new Causal clustering architecture for Neo4j, showing that, despite the mixture of consensus protocols and asynchronous replication, Neo4j allows users to read their own writes straightforwardly, and discusses why this is such a difficult achievement in distributed systems. Filmed at qconlondon.com.

Jim Webber is Chief Scientist with Neo Technology, the company behind the popular open source graph database Neo4j. He works on R&D for highly scalable graph databases and writes open source software. He has written two books on integration and distributed systems: “Developing Enterprise Web Services” and “REST in Practice”. His latest book is “Graph Databases” which focuses on the Neo4j database.

Publicada em: Tecnologia
  • Seja o primeiro a comentar

Causal Consistency for Large Neo4j Clusters

  1. 1. Causal Consistency For Large Neo4j Clusters Jim Webber Chief Scientist, Neo4j QCon London
  2. 2. InfoQ.com: News & Community Site Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations /causal-clustering-neo4j • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week
  3. 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
  4. 4. Leads to a social graph
  5. 5. Motivation Why do we need clusters of Neo4j?
  6. 6. Massive Throughput
  7. 7. Data Redundancy
  8. 8. Data Redundancy
  9. 9. Data Redundancy
  10. 10. Data Redundancy
  11. 11. High Availability
  12. 12. High Availability
  13. 13. High Availability Error! 503: Service Unavailable
  14. 14. High Availability Error! 503: Service Unavailable
  15. 15. High Availability Error! 503: Service Unavailable
  16. 16. High Availability Error! 503: Service Unavailable
  17. 17. High Availability ✓ Error! 503: Service Unavailable
  18. 18. Data RedundancyMassive Throughput High Availability
  19. 19. Data RedundancyMassive Throughput High Availability 3.0
  20. 20. Data RedundancyMassive Throughput High Availability 3.0 Bigger Clusters Consensus Commit Built-in load balancing 3.1 Causal Clustering
  21. 21. Roles for safety and scale Divide and conquer complexity
  22. 22. Read Replicas Core
  23. 23. • Small group of Neo4j databases • Fault-tolerant Consensus Commit • Responsible for data safety Core
  24. 24. Writing to the Core Cluster Neo4j Driver Neo4j Cluster
  25. 25. Writing to the Core Cluster Neo4j Driver CREATE (:User {...}) ✓ Neo4j Cluster
  26. 26. Writing to the Core Cluster Neo4j Driver CREATE (:User {...}) ✓ Neo4j Cluster
  27. 27. Writing to the Core Cluster Neo4j Driver CREATE (:User {...}) ✓ ✓ ✓ Neo4j Cluster
  28. 28. Writing to the Core Cluster Neo4j Driver CREATE (:User {...}) ✓ ✓ ✓ Neo4j Cluster
  29. 29. Writing to the Core Cluster Neo4j Driver CREATE (:User {...}) ✓ ✓ ✓ Neo4j Cluster
  30. 30. Writing to the Core Cluster Neo4j Driver ✓ ✓ ✓ Success Neo4j Cluster
  31. 31. Writing to the Core Cluster Neo4j Driver ✓ ✓ ✓ Success Neo4j Cluster ✓ ✓
  32. 32. Raft Protocol Non-Blocking Consensus for Humans
  33. 33. Raft Protocol https://github.com/ongardie/raftscope
  34. 34. Raft in a Nutshell • Raft keeps logs tied together (geddit?) • Logs contain entries for both the database and the cluster membership • Entries are appended and subsequently committed if a simple majority agree • Implication: majority agree with the log as proposed • Anyone can call an election: highest term (logical clock) wins, followed by highest committed, followed by highest appended. • Appended, but not committed, entries can be truncated, but this is safe (translates as transaction aborted) https://github.com/ongardie/raftscope
  35. 35. Consensus Log → Committed Transactions → Updated Graph 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 11 12 13 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 Transaction log: the same transactions appear in the same order on all members Consensus log: stores both committed and uncommitted transactions Uncommitted entries may differ between members Transactionsareonlyappendedtothetransaction logwhencommittedaccordingtoRaft Transactionsareapplied,updatingthegraph Neo4j Raft implementation
  36. 36. • Small group of Neo4j databases • Fault-tolerant Consensus Commit • Responsible for data safety Core
  37. 37. • For massive query throughput • Read-only replicas • Not involved in Consensus Commit • Disposable, suitable for auto-scaling Read Replicas
  38. 38. Propagating updates to the Read Replicas Neo4j Driver Neo4j Cluster
  39. 39. Propagating updates to the Read Replicas Neo4j Driver Neo4j Cluster Write
  40. 40. Propagating updates to the Read Replicas Neo4j Driver Neo4j Cluster Write
  41. 41. Reading from the Read Replicas Neo4j Driver Neo4j Cluster Read
  42. 42. Updating the graph Querying the graph
  43. 43. Read Replicas Core Updating the graph Queries, analysis, reporting
  44. 44. ESTATE=$(neo-workbench estate add database -p Local -b core-block -s 3) neo-workbench estate add database -p Local -b edge-block -s 10 $ESTATE neo-workbench database install -m Core --package-uri file:///Users/jim/Downloads/neo4j-enterprise-3.1.1-unix.tar.gz -b core-block $ESTATE neo-workbench database install -m Read_Replica --package-uri file:///Users/jim/Downloads/neo4j-enterprise-3.1.1-unix.tar.gz -b edge-block $ESTATE neo-workbench database start $ESTATE
  45. 45. :sysinfo
  46. 46. Building an App Computer science meets technology
  47. 47. App Server Neo4j Driver Bolt protocol
  48. 48. Java <dependency> <groupId>org.neo4j.driver</groupId> <artifactId>neo4j-java-driver</artifactId> </dependency> Python pip install neo4j-driver .NET PM> Install-Package Neo4j.Driver JavaScript npm install neo4j-driver
  49. 49. https://neo4j.com/developer/language-guides
  50. 50. bolt:// GraphDatabase.driver( "bolt://aServer" )
  51. 51. bolt+routing:// GraphDatabase.driver( "bolt+routing://aCoreServer" )
  52. 52. GraphDatabase.driver( "bolt+routing://aCoreServer" ) Bootstrap: specify any core server to route load across the whole cluster bolt+routing://
  53. 53. Application Server Neo4j Driver Max Jim Jane Mark
  54. 54. Routed write statements driver = GraphDatabase.driver( "bolt+routing://aCoreServer" ); try ( Session session = driver.session( AccessMode.WRITE ) ) { try ( Transaction tx = session.beginTransaction() ) { tx.run( "MERGE (user:User {userId: {userId}})", parameters( "userId", userId ) ); tx.success(); } }
  55. 55. Routed read queries driver = GraphDatabase.driver( "bolt+routing://aCoreServer" ); try ( Session session = driver.session( AccessMode.READ ) ) { try ( Transaction tx = session.beginTransaction() ) { tx.run( "MATCH (user:User {userId: {userId}})-[*]-(:Product) RETURN *", parameters( "userId", userId ) ); tx.success(); } }
  56. 56. Consistency models Can you read what you write?
  57. 57. Cluster members slightly “ahead” or “behind” of each other 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 If I query this server I won’t see the updates from transaction . If I query this server, I’ll see all updates from all committed transactions 11 11
  58. 58. Updating the graph Querying the graph
  59. 59. Register Login You need to login in to continue your purchase!
  60. 60. Register Login You need to login in to continue your purchase! Username: Password: Create Account
  61. 61. Register Login You need to login in to continue your purchase! Username: jim_w Password: ******** Create Account
  62. 62. Register Login You need to login in to continue your purchase! Username: Password: Login
  63. 63. Username: jim_w Password: ******** Login
  64. 64. Purchase Login Successful Try again No account found!Username: jim_w Password: ******** Login 𝙓
  65. 65. Username: jim_w Password: ******** A few moments later... ✓ Login
  66. 66. Purchase Login SuccessfulUsername: jim_w Password: ******** Login A few moments later... ✓
  67. 67. Q Why didn’t this work? A Eventual Consistency
  68. 68. 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 Create Account App Server A Driver
  69. 69. 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver
  70. 70. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver
  71. 71. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver 11
  72. 72. 0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User) Create Account App Server A Driver 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 11
  73. 73. 0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User) Create Account App Server A Driver 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 11
  74. 74. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver 11
  75. 75. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver MATCH (:User) Login App Server B Driver 11
  76. 76. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver MATCH (:User) Login App Server B Driver 11
  77. 77. Bookmark Session token String (for portability) Opaque to application Represents ultimate user’s most recent view of the graph More capabilities to come
  78. 78. Let’s try again, with Causal Consistency
  79. 79. 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 Create Account App Server A Driver
  80. 80. 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver
  81. 81. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver
  82. 82. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver 11
  83. 83. 0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User) Create Account App Server A Driver 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 11
  84. 84. 0 1 2 3 4 5 6 7 8 9 10 11CREATE (:User) Create Account App Server A Driver 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 11
  85. 85. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver 11
  86. 86. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 CREATE (:User) Create Account App Server A Driver MATCH (:User) Login App Server B Driver 11
  87. 87. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 CREATE (:User) Create Account MATCH (:User) Login App Server A App Server B Driver Driver
  88. 88. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 CREATE (:User) Create Account MATCH (:User) Login App Server A App Server B Driver Driver 11
  89. 89. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 CREATE (:User) Create Account MATCH (:User) Login App Server A App Server B Driver Driver 11
  90. 90. Obtain bookmark try ( Session session = driver.session( AccessMode.WRITE ) ) { try ( Transaction tx = session.beginTransaction() ) { tx.run( "CREATE (user:User {userId: {userId}, passwordHash: {passwordHash})", parameters( "userId", userId, "passwordHash", passwordHash ) ); tx.success(); } String bookmark = session.lastBookmark(); }
  91. 91. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 CREATE (:User) Create Account MATCH (:User) Login App Server A App Server B Driver Driver 11 Obtain bookmark
  92. 92. Use a bookmark try ( Session session = driver.session( AccessMode.READ ) ) { try ( Transaction tx = session.beginTransaction( bookmark ) ) { tx.run( "MATCH (user:User {userId: {userId}}) RETURN *", parameters( "userId", userId ) ); tx.success(); } }
  93. 93. 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 11 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 CREATE (:User) Create Account MATCH (:User) Login App Server A App Server B Driver Driver 11 Use bookmark
  94. 94. Read Replicas Core Updating the graph Queries, analysis, reporting
  95. 95. Thank you for listening @jimwebber
  96. 96. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/causal- clustering-neo4j

×