Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2KljrPR. Sumedh Pathak talks about his team’s journey to create a more modern relational database. He talks about the distributed systems problems they had to solve in order to scale out the Postgres open source database, in order to achieve parallelism and a concomitant increase in performance. He describes the architecture of the distributed query planner, the distributed deadlock detection, and more. Filmed at qconlondon.com. Sumedh Pathak is a co-founder and the VP of Engineering at Citus Data, where he leads the effort to make sure the Citus scale-out database is useful for application developers, so they can focus on their application and not their infrastructure. Before Citus, he worked as a software engineer at Amazon on the shopping cart app.

1. Sumedh Pathak, Co-Founder & VP Engineering, Citus Data
QCon London 2018
The Future of Distributed
Databases is Relational
@moss_toss | @citusdata

2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
future-distributed-database-relational
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. About Me
• Co-Founder & VP
Engineering at Citus Data
• Amazon Shopping Cart
(former)
• Amazon Supply Chain &
Order Fulfillment (former)
• Stanford Computer Science
Citus Data co-founders, left-to-right
Ozgun Erdogan, Sumedh Pathak, & Umur Cubukcu
Photo cred: Willy Johnson, Monterey CA, Sep 2017
Sumedh Pathak | Citus Data | QCon London 2018

6. Why
RDBMS?
Sumedh Pathak | Citus Data | QCon London 2018

7. Streaming Storage
Map
Reduce
NoSQL
SQL
Database
SELECT ...
Application
Application
Sumedh Pathak | Citus Data | QCon London 2018
Because your architecture could be simpler

8. An RDBMS is a
general-purpose
data platform
Sumedh Pathak | Citus Data | QCon London 2018
Fast writes
Real-time & bulk
High throughput
High concurrency
Data consistency
Query optimizers

9. PostgreSQL
MySQL
MongoDB
SQL Server +
Oracle
Source: Hacker News, https://news.ycombinator.com
Startups Are Choosing Postgres% database job posts mentioning each database, across 20K+ job posts
Sumedh Pathak | Citus Data | QCon London 2018

10. but RDBMS’s
don’t scale, right?
Sumedh Pathak | Citus Data | QCon London 2018

11. but RDBMS’s don’t scale
RDBMS’s are hard to scale
Sumedh Pathak | Citus Data | QCon London 2018

12. What exactly needs to Scale?
- Tables (Data)
- Partitioning, Co-location, Reference Tables
- SQL (Reads)
- How do we express and optimize distributed SQL
- Transactions (Writes)
- Cross Shard updates/deletes, Global Atomic Transactions
Sumedh Pathak | Citus Data | QCon London 2018
1
2
3

13. Scaling
Tables
Sumedh Pathak | Citus Data | QCon London 2018

14. How to partition the data?
- Pick a column
- Date
- Id (customer_id, cart_id)
- Pick a method
- Hash
- Range

15. Partition data across nodes
R1
R2
R3
R4
R5
R6
R7
Coordinator
Node
Worker
Nodes
Shards
Sumedh Pathak | Citus Data | QCon London 2018

16. Worker → RDBMS, Shard → Table

17. Reference Tables
N1
N1
N1
N1
Copies of same table
Sumedh Pathak | Citus Data | QCon London 2018
Coordinator
Node
Worker
Nodes

18. Co-Location
R1
R2
R3
R4
S1
S2
S3
S4
Explicit Co-Location API.
E.g. Partition by Tenant
Sumedh Pathak | Citus Data | QCon London 2018
Coordinator
Node
Worker
Nodes

19. What about Foreign Keys?

20. The key to scaling tables...
- Use relational databases as a building block
- Understand semantics of application—to be smart about
partitioning
- Multi-tenant applications

21. Scaling
SQL

22. FROM table R
SELECT x Projectx
(R) → R’
WHERE f(x) Filterf(x)
(R) → R’
… JOIN … R × S → R’
SQL ↔ Relational Algebra
Sumedh Pathak | Citus Data | QCon London 2018

23. FROM sharded_table Collect(R1
,R2
,...) → R
Distributed Relational Algebra
R1
R2
C
Sumedh Pathak | Citus Data | QCon London 2018

24. Projectx
(Collect(R1
,R2
,...)) = Collect(Projectx
(R1
), Projectx
(R2
)...)
Commutative property
R1
R2
C
R1
R2
C
Px
Px
Px
Sumedh Pathak | Citus Data | QCon London 2018

25. Collect(R1
,R2
,...) x Collect(S1
,S2
,...) = Collect(R1
× S1
,R2
× S2,
...)
Distributive property
R1
R2
C
×
C
S1
S2
R1
R2
C
×
S1
S2
×
X = Join Operator

26. SUM(x)(Collect(R1
,R2
,...)) = SUM(Collect(SUM(R1
), SUM(R2
)...))
Associative property
R1
R2
C
R1
R2
C
Sumx
Sumx
Sumx
Sumedh Pathak | Citus Data | QCon London 2018
Sumx

27. SELECT sum(price) FROM orders, nation
WHERE orders.nation = nation.name AND
orders.date >= '2012-01-01' AND
nation.region = 'Asia';
Sumedh Pathak | Citus Data | QCon London 2018

28. Sumedh Pathak | Citus Data | QCon London 2018

29. Sumedh Pathak | Citus Data | QCon London 2018
Volcano style processing
Data flows
from
bottom
to top

30. Sumedh Pathak | Citus Data | QCon London 2018

31. Sumedh Pathak | Citus Data | QCon London 2018

32. Sumedh Pathak | Citus Data | QCon London 2018
Parallelize
Aggregate
Push Joins & Filters
below collect. Run in
parallel across all
nodes
Filters & Projections
done before Join

33. SELECT sum(intermediate_col)
FROM <concatenated results>;
SELECT sum(price)
FROM orders_2 JOIN nation_2
ON (orders_2.name = nation_2.name)
WHERE
orders_2.date >= '2017-01-01'
AND
nation_2.region = 'Asia';
SELECT sum(price)
FROM orders_2 JOIN nation_1
ON (orders_2.name = nation_1.name)
WHERE
orders_2.date >= '2017-01-01'
AND
nation_2.region = 'Asia';
Sumedh Pathak | Citus Data | QCon London 2018

34. Executing Distributed SQL
SQL database
orders_1
nation_1
orders
nation
SELECT sum(price)
FROM orders_2 o JOIN nation_1
ON (o.name = n.name)
WHERE o.date >= '2017-01-01'
AND n.region = 'Asia';
SELECT sum(price)
FROM orders_1 o JOIN nation_1
ON (o.name = n.name)
WHERE o.date >= '2017-01-01'
AND n.region = 'Asia';
SELECT sum(price) FROM <results>;
SQL database
orders_2
nation_1

35. The key to scaling SQL...
- New relational algebra operators for
distributed processing
- Relational Algebra Properties to optimize
tree: Commutativity, Associativity, &
Distributivity
- Map / Reduce operators

36. Scaling
Transactions

37. Money Transfer, as an example
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
Sumedh Pathak | Citus Data | QCon London 2018

38. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
Coordinator
Sumedh Pathak | Citus Data | QCon London 2018

39. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
Sumedh Pathak | Citus Data | QCon London 2018
Coordinator

40. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
COMMIT;
Sumedh Pathak | Citus Data | QCon London 2018
Coordinator

41. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
Coordinator
BEGIN;
UPDATE accounts SET balance =
balance + 100 WHERE id = ‘BOB’;
Sumedh Pathak | Citus Data | QCon London 2018
BEGIN;
UPDATE accounts SET balance = balance
- 100 WHERE id = ‘ALICE’;

42. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
BEGIN;
UPDATE accounts SET balance =
balance + 100 WHERE id = ‘BOB’;
COMMIT
Sumedh Pathak | Citus Data | QCon London 2018
BEGIN;
UPDATE accounts SET balance = balance
- 100 WHERE id = ‘ALICE’;
Coordinator

43. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
PREPARE TRANSACTION ‘citus_...98’; PREPARE TRANSACTION ‘citus_...98’;
Sumedh Pathak | Citus Data | QCon London 2018
Coordinator

44. What happens during PREPARE?
State of transaction stored
on a durable store
Locks are
maintained
&

45. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
Coordinator
PREPARE TRANSACTION ‘citus_...98’; ROLLBACK TRANSACTION ‘citus_...98’;
Sumedh Pathak | Citus Data | QCon London 2018

46. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
PREPARE TRANSACTION ‘citus_...98’; PREPARE TRANSACTION ‘citus_...98’;
Sumedh Pathak | Citus Data | QCon London 2018
Coordinator

47. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
COMMIT PREPARED ‘citus_...98’; COMMIT PREPARED ‘citus_...98’;
Sumedh Pathak | Citus Data | QCon London 2018
Coordinator

48. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
COMMIT PREPARED ‘citus_...98’; COMMIT PREPARED ‘citus_...98’;
Sumedh Pathak | Citus Data | QCon London 2018
Coordinator

49. A1
A2
BEGIN;
UPDATE accounts SET balance = balance -
100 WHERE id = ‘ALICE’;
UPDATE accounts SET balance = balance +
100 WHERE id = ‘BOB’;
COMMIT;
A3
A4
COMMIT PREPARED ‘citus_...98’; COMMIT PREPARED ‘citus_...98’;
Sumedh Pathak | Citus Data | QCon London 2018
Coordinator

50. But....
Deadlocks!

51. Example
// SESSION 1
BEGIN;
UPDATE accounts SET balance
= balance - 100 WHERE id =
‘ALICE’;
(LOCK on ROW with id
‘ALICE’)
// SESSION 2
Sumedh Pathak | Citus Data | QCon London 2018

52. Example
// SESSION 1
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE
id = ‘ALICE’;
(LOCK on ROW with id ‘ALICE’)
// SESSION 2
BEGIN;
UPDATE accounts SET balance
= balance + 100 WHERE id =
‘BOB’;
(LOCK on ROW with id ‘BOB’)
Sumedh Pathak | Citus Data | QCon London 2018

53. Example
// SESSION 1
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE
id = ‘ALICE’;
(LOCK on ROW with id ‘ALICE’)
UPDATE accounts SET balance
= balance + 100 WHERE id =
‘BOB’;
// SESSION 2
BEGIN;
UPDATE accounts SET balance = balance + 100 WHERE
id = ‘BOB’;
(LOCK on ROW with id ‘BOB’)
Sumedh Pathak | Citus Data | QCon London 2018

54. Example
// SESSION 1
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE
id = ‘ALICE’;
(LOCK on ROW with id ‘ALICE’)
UPDATE accounts SET balance = balance + 100
WHERE id = ‘BOB’;
// SESSION 2
BEGIN;
UPDATE accounts SET balance = balance + 100 WHERE
id = ‘BOB’;
(LOCK on ROW with id ‘BOB’)
UPDATE accounts SET balance
= balance - 100 WHERE id =
‘ALICE’
Sumedh Pathak | Citus Data | QCon London 2018

55. How do Relational DB’s solve this?
S1
S2
- Construct a
Directed Graph
- Each node is a
session/transaction
- Edge represents a
wait on a lock
Waiting
on ‘Bob’
Waiting on
‘Alice’

56. S1
S2
S1 S2
S2 S1
S2 Waits on S1 S1 Waits on S2
Sumedh Pathak | Citus Data | QCon London 2018

57. S1
S2
S1 S2
S2 S1
S2 Waits on S1 S1 Waits on S2
Distributed
Deadlock
Detector
S1
S2 S2
S1
Sumedh Pathak | Citus Data | QCon London 2018

58. keys to scaling transactions
- 2PC to ensure atomic transactions across nodes
- Deadlock Detection—to scale complex & concurrent
transaction workloads
- MVCC
- Failure Handling
4

59. It’s 2018. Distributed can be Relational
- Scale tables—via sharding
- Scale SQL—via distributed relational algebra
- Scale transactions—via 2PC & Deadlock Detection
Sumedh Pathak | Citus Data | QCon London 2018
Now, how do we implement all of this?

60. Postgres
Sumedh Pathak | Citus Data | QCon London 2018

61. PostgreSQL
Planner
Executor
Custom scan
Commit / abort
Extension (.so)
Access methods
Foreign tables
Functions
...
...
...
...
...
...
...
CREATE EXTENSION ...

62. PostgreSQL PostgreSQL
PostgreSQL
shardsshardsshard
shardsshardsshard
SELECT … FROM distributed_table …
SELECT … FROM shard… SELECT … FROM shard…
Citus

63. PostgreSQL
Citus
PostGIS PL/Python
JSONB
2PC
Replication...
Sequences
Indexes
Full-Text SearchTransactions
…
dblink
Sumedh Pathak | Citus Data | QCon London 2018
Rich
ecosystem of
tooling, data
types,
& extensions
in
PostgreSQL

64. PostgreSQL can be
extended into an all-purpose
distributed RDBMS

65. I believe the future of distributed
databases is relational.

66. Thank you!
Sumedh Pathak
sumedh@citusdata.com
@moss_toss | @citusdata | citusdata.com
QCon London 2018 | The Future of Distributed Databases is Relational

67. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/future-
distributed-database-relational