O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

The Future of Distributed Databases Is Relational

45 visualizações

Publicada em

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2KljrPR.

Sumedh Pathak talks about his team’s journey to create a more modern relational database. He talks about the distributed systems problems they had to solve in order to scale out the Postgres open source database, in order to achieve parallelism and a concomitant increase in performance. He describes the architecture of the distributed query planner, the distributed deadlock detection, and more. Filmed at qconlondon.com.

Sumedh Pathak is a co-founder and the VP of Engineering at Citus Data, where he leads the effort to make sure the Citus scale-out database is useful for application developers, so they can focus on their application and not their infrastructure. Before Citus, he worked as a software engineer at Amazon on the shopping cart app.

Publicada em: Tecnologia
  • Seja o primeiro a comentar

  • Seja a primeira pessoa a gostar disto

The Future of Distributed Databases Is Relational

  1. 1. Sumedh Pathak, Co-Founder & VP Engineering, Citus Data QCon London 2018 The Future of Distributed Databases is Relational @moss_toss | @citusdata
  2. 2. InfoQ.com: News & Community Site Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ future-distributed-database-relational • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week
  3. 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon London www.qconlondon.com
  4. 4. About Me • Co-Founder & VP Engineering at Citus Data • Amazon Shopping Cart (former) • Amazon Supply Chain & Order Fulfillment (former) • Stanford Computer Science Citus Data co-founders, left-to-right Ozgun Erdogan, Sumedh Pathak, & Umur Cubukcu Photo cred: Willy Johnson, Monterey CA, Sep 2017 Sumedh Pathak | Citus Data | QCon London 2018
  5. 5. Why RDBMS? Sumedh Pathak | Citus Data | QCon London 2018
  6. 6. Streaming Storage Map Reduce NoSQL SQL Database SELECT ... Application Application Sumedh Pathak | Citus Data | QCon London 2018 Because your architecture could be simpler
  7. 7. An RDBMS is a general-purpose data platform Sumedh Pathak | Citus Data | QCon London 2018 Fast writes Real-time & bulk High throughput High concurrency Data consistency Query optimizers
  8. 8. PostgreSQL MySQL MongoDB SQL Server + Oracle Source: Hacker News, https://news.ycombinator.com Startups Are Choosing Postgres% database job posts mentioning each database, across 20K+ job posts Sumedh Pathak | Citus Data | QCon London 2018
  9. 9. but RDBMS’s don’t scale, right? Sumedh Pathak | Citus Data | QCon London 2018
  10. 10. but RDBMS’s don’t scale RDBMS’s are hard to scale Sumedh Pathak | Citus Data | QCon London 2018
  11. 11. What exactly needs to Scale? - Tables (Data) - Partitioning, Co-location, Reference Tables - SQL (Reads) - How do we express and optimize distributed SQL - Transactions (Writes) - Cross Shard updates/deletes, Global Atomic Transactions Sumedh Pathak | Citus Data | QCon London 2018 1 2 3
  12. 12. Scaling Tables Sumedh Pathak | Citus Data | QCon London 2018
  13. 13. How to partition the data? - Pick a column - Date - Id (customer_id, cart_id) - Pick a method - Hash - Range
  14. 14. Partition data across nodes R1 R2 R3 R4 R5 R6 R7 Coordinator Node Worker Nodes Shards Sumedh Pathak | Citus Data | QCon London 2018
  15. 15. Worker → RDBMS, Shard → Table
  16. 16. Reference Tables N1 N1 N1 N1 Copies of same table Sumedh Pathak | Citus Data | QCon London 2018 Coordinator Node Worker Nodes
  17. 17. Co-Location R1 R2 R3 R4 S1 S2 S3 S4 Explicit Co-Location API. E.g. Partition by Tenant Sumedh Pathak | Citus Data | QCon London 2018 Coordinator Node Worker Nodes
  18. 18. What about Foreign Keys?
  19. 19. The key to scaling tables... - Use relational databases as a building block - Understand semantics of application—to be smart about partitioning - Multi-tenant applications
  20. 20. Scaling SQL
  21. 21. FROM table R SELECT x Projectx (R) → R’ WHERE f(x) Filterf(x) (R) → R’ … JOIN … R × S → R’ SQL ↔ Relational Algebra Sumedh Pathak | Citus Data | QCon London 2018
  22. 22. FROM sharded_table Collect(R1 ,R2 ,...) → R Distributed Relational Algebra R1 R2 C Sumedh Pathak | Citus Data | QCon London 2018
  23. 23. Projectx (Collect(R1 ,R2 ,...)) = Collect(Projectx (R1 ), Projectx (R2 )...) Commutative property R1 R2 C R1 R2 C Px Px Px Sumedh Pathak | Citus Data | QCon London 2018
  24. 24. Collect(R1 ,R2 ,...) x Collect(S1 ,S2 ,...) = Collect(R1 × S1 ,R2 × S2, ...) Distributive property R1 R2 C × C S1 S2 R1 R2 C × S1 S2 × X = Join Operator
  25. 25. SUM(x)(Collect(R1 ,R2 ,...)) = SUM(Collect(SUM(R1 ), SUM(R2 )...)) Associative property R1 R2 C R1 R2 C Sumx Sumx Sumx Sumedh Pathak | Citus Data | QCon London 2018 Sumx
  26. 26. SELECT sum(price) FROM orders, nation WHERE orders.nation = nation.name AND orders.date >= '2012-01-01' AND nation.region = 'Asia'; Sumedh Pathak | Citus Data | QCon London 2018
  27. 27. Sumedh Pathak | Citus Data | QCon London 2018
  28. 28. Sumedh Pathak | Citus Data | QCon London 2018 Volcano style processing Data flows from bottom to top
  29. 29. Sumedh Pathak | Citus Data | QCon London 2018
  30. 30. Sumedh Pathak | Citus Data | QCon London 2018
  31. 31. Sumedh Pathak | Citus Data | QCon London 2018 Parallelize Aggregate Push Joins & Filters below collect. Run in parallel across all nodes Filters & Projections done before Join
  32. 32. SELECT sum(intermediate_col) FROM <concatenated results>; SELECT sum(price) FROM orders_2 JOIN nation_2 ON (orders_2.name = nation_2.name) WHERE orders_2.date >= '2017-01-01' AND nation_2.region = 'Asia'; SELECT sum(price) FROM orders_2 JOIN nation_1 ON (orders_2.name = nation_1.name) WHERE orders_2.date >= '2017-01-01' AND nation_2.region = 'Asia'; Sumedh Pathak | Citus Data | QCon London 2018
  33. 33. Executing Distributed SQL SQL database orders_1 nation_1 orders nation SELECT sum(price) FROM orders_2 o JOIN nation_1 ON (o.name = n.name) WHERE o.date >= '2017-01-01' AND n.region = 'Asia'; SELECT sum(price) FROM orders_1 o JOIN nation_1 ON (o.name = n.name) WHERE o.date >= '2017-01-01' AND n.region = 'Asia'; SELECT sum(price) FROM <results>; SQL database orders_2 nation_1
  34. 34. The key to scaling SQL... - New relational algebra operators for distributed processing - Relational Algebra Properties to optimize tree: Commutativity, Associativity, & Distributivity - Map / Reduce operators
  35. 35. Scaling Transactions
  36. 36. Money Transfer, as an example BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; Sumedh Pathak | Citus Data | QCon London 2018
  37. 37. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 Coordinator Sumedh Pathak | Citus Data | QCon London 2018
  38. 38. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; Sumedh Pathak | Citus Data | QCon London 2018 Coordinator
  39. 39. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; COMMIT; Sumedh Pathak | Citus Data | QCon London 2018 Coordinator
  40. 40. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 Coordinator BEGIN; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; Sumedh Pathak | Citus Data | QCon London 2018 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’;
  41. 41. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 BEGIN; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT Sumedh Pathak | Citus Data | QCon London 2018 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; Coordinator
  42. 42. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 PREPARE TRANSACTION ‘citus_...98’; PREPARE TRANSACTION ‘citus_...98’; Sumedh Pathak | Citus Data | QCon London 2018 Coordinator
  43. 43. What happens during PREPARE? State of transaction stored on a durable store Locks are maintained &
  44. 44. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 Coordinator PREPARE TRANSACTION ‘citus_...98’; ROLLBACK TRANSACTION ‘citus_...98’; Sumedh Pathak | Citus Data | QCon London 2018
  45. 45. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 PREPARE TRANSACTION ‘citus_...98’; PREPARE TRANSACTION ‘citus_...98’; Sumedh Pathak | Citus Data | QCon London 2018 Coordinator
  46. 46. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 COMMIT PREPARED ‘citus_...98’; COMMIT PREPARED ‘citus_...98’; Sumedh Pathak | Citus Data | QCon London 2018 Coordinator
  47. 47. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 COMMIT PREPARED ‘citus_...98’; COMMIT PREPARED ‘citus_...98’; Sumedh Pathak | Citus Data | QCon London 2018 Coordinator
  48. 48. A1 A2 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; COMMIT; A3 A4 COMMIT PREPARED ‘citus_...98’; COMMIT PREPARED ‘citus_...98’; Sumedh Pathak | Citus Data | QCon London 2018 Coordinator
  49. 49. But.... Deadlocks!
  50. 50. Example // SESSION 1 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; (LOCK on ROW with id ‘ALICE’) // SESSION 2 Sumedh Pathak | Citus Data | QCon London 2018
  51. 51. Example // SESSION 1 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; (LOCK on ROW with id ‘ALICE’) // SESSION 2 BEGIN; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; (LOCK on ROW with id ‘BOB’) Sumedh Pathak | Citus Data | QCon London 2018
  52. 52. Example // SESSION 1 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; (LOCK on ROW with id ‘ALICE’) UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; // SESSION 2 BEGIN; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; (LOCK on ROW with id ‘BOB’) Sumedh Pathak | Citus Data | QCon London 2018
  53. 53. Example // SESSION 1 BEGIN; UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’; (LOCK on ROW with id ‘ALICE’) UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; // SESSION 2 BEGIN; UPDATE accounts SET balance = balance + 100 WHERE id = ‘BOB’; (LOCK on ROW with id ‘BOB’) UPDATE accounts SET balance = balance - 100 WHERE id = ‘ALICE’ Sumedh Pathak | Citus Data | QCon London 2018
  54. 54. How do Relational DB’s solve this? S1 S2 - Construct a Directed Graph - Each node is a session/transaction - Edge represents a wait on a lock Waiting on ‘Bob’ Waiting on ‘Alice’
  55. 55. S1 S2 S1 S2 S2 S1 S2 Waits on S1 S1 Waits on S2 Sumedh Pathak | Citus Data | QCon London 2018
  56. 56. S1 S2 S1 S2 S2 S1 S2 Waits on S1 S1 Waits on S2 Distributed Deadlock Detector S1 S2 S2 S1 Sumedh Pathak | Citus Data | QCon London 2018
  57. 57. keys to scaling transactions - 2PC to ensure atomic transactions across nodes - Deadlock Detection—to scale complex & concurrent transaction workloads - MVCC - Failure Handling 4
  58. 58. It’s 2018. Distributed can be Relational - Scale tables—via sharding - Scale SQL—via distributed relational algebra - Scale transactions—via 2PC & Deadlock Detection Sumedh Pathak | Citus Data | QCon London 2018 Now, how do we implement all of this?
  59. 59. Postgres Sumedh Pathak | Citus Data | QCon London 2018
  60. 60. PostgreSQL Planner Executor Custom scan Commit / abort Extension (.so) Access methods Foreign tables Functions ... ... ... ... ... ... ... CREATE EXTENSION ...
  61. 61. PostgreSQL PostgreSQL PostgreSQL shardsshardsshard shardsshardsshard SELECT … FROM distributed_table … SELECT … FROM shard… SELECT … FROM shard… Citus
  62. 62. PostgreSQL Citus PostGIS PL/Python JSONB 2PC Replication... Sequences Indexes Full-Text SearchTransactions … dblink Sumedh Pathak | Citus Data | QCon London 2018 Rich ecosystem of tooling, data types, & extensions in PostgreSQL
  63. 63. PostgreSQL can be extended into an all-purpose distributed RDBMS
  64. 64. I believe the future of distributed databases is relational.
  65. 65. Thank you! Sumedh Pathak sumedh@citusdata.com @moss_toss | @citusdata | citusdata.com QCon London 2018 | The Future of Distributed Databases is Relational
  66. 66. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/future- distributed-database-relational

×