Presentation at PGConf India 2017.

1. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 1
Partition and Conquer large data in
PostgreSQL 10
• Ashutosh Bapat | 2017.02.03 @ PGConf India

2. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 2
• Manageability
– Smaller data is easier to manage
– Partition-wise utility commands
– Easy bulk load and deletes
• Query/DML Performance
– Partition elimination (pruning)
– Partition-wise joins, aggregate
– Distribute DMLs across partitions
• Data storages based on partition properties
– “hot” and “cold” partitions
– Foreign partitions
Why to partition tables?

3. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 3
• Maintenance
– No triggers, rules or constraints to maintain
– Adding/deleting partition does not require more than a
single command
• Faster queries and DMLs
– No trigger execution overhead
●
DMLs are 10-20 times faster
– Simpler partition bounds representation
– Faster partition pruning
– Partition-wise join, aggregation
Why declarative partitioning?
(instead of inheritance based partitioning)

4. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 4
Declarative partitioning in
PostgreSQL 10

5. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 5
●
Partitioning methods: List, range, (WIP. hash)
●
Partition key: Single or multiple columns, expressions
●
Sub-partitioning
– Partitioned partitions
– Mixed sub-partitioning
●
Development efforts
– Several previous attempts by many people
– Amit Langote proposed first patch in August 2015
– Got committed in Dec 2016, bug fixes, doc changes continue ...
PostgreSQL Declarative partitioning

6. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 6
CREATE TABLE part_tab (c1 int, c2 int, …) PARTITION BY RANGE (c1);
Creating partitioned table
part_tab (c1 int, c2 int, ...)

7. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 7
CREATE TABLE part_tab (c1 int, c2 int, …) PARTITION BY RANGE (c1);
CREATE TABLE part1 PARTITION OF part_tab FOR VALUES FROM (0) TO (100);
CREATE TABLE part2 PARTITION OF part_tab FOR VALUES FROM (100) TO (200);
Creating partitioned table
part_tab (c1 int, c2 int, ...)
Part1
0 to 100
Part2
100 to 200
Tablespaces

8. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 8
ALTER TABLE part_tab ATTACH PARTITION ext_part FOR VALUES FROM (400) to (500);
ATTACH partition
part_tab (c1 int, c2 int, ...)
Part1
0 to 100
Part2
100 to 200
ext_part
400 to 500
ext_part (c1 int, c2 int, ...)
CHECK c1 >= 400 AND c1 < 500
Bulk load
data.csv

9. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 9
ALTER TABLE part_tab DETACH PARTITION ext_part;
DETACH partition
part_tab (c1 int, c2 int, ...)
Part1
0 to 100
Part2
100 to 200
ext_part (c1 int, c2 int, ...)
CHECK c1 >= 400 AND c1 < 500

10. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 10
●
Same columns as parent table
●
Partition specific constraints, defaults, indexes,
storage parameters
●
Tablespace separate from that of parent
– “hot” and “cold” partitions
●
VACUUM, ANALYZE, CLUSTER can be run
separately
– Utilities don't block the whole table
– Work where it's required
Partitions are tables

11. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 11
CREATE TABLE part_tab (c1 int, c2 int, …) PARTITION BY RANGE (c1);
...
CREATE FOREIGN TABLE part3 PARTITION OF part_tab FOR VALUES FROM (300) TO
(400) SERVER xyz_server;
CREATE FOREIGN TABLE part4 PARTITION OF part_tab FOR VALUES FROM (400) TO
(500) SERVER abc_server;
Foreign partitions
part_tab (c1 int, c2 int, ...)
Part1
0 to 100
Part2
100 to 200
Part3
300 to 400
fpart3 (c1 int, c2 int, ...)
c1 >= 300 AND c1 < 400
xyz_server
xyz_fdw
Part4
400 to 500
fpart3 (c1 int, c2 int, ...)
c1 >= 400 AND c1 < 500
abc_server
abc_fdw

12. © Copyright EnterpriseDB Corporation, 2015. All Rights Reserved. 12
Query optimizations
partition pruning
partition-wise join
partition-wise aggregation

13. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 13
Query optimization: partition pruning
SELECT * FROM t1 WHERE c1 = 350;
Partition 1
FOR VALUES
FROM (0) TO (100)
Partitioned table
t1 (c1 int, c2 int, …)
Partition 4
FOR VALUES
FROM (300) TO (400)
Partition 3
FOR VALUES
FROM (200) TO (300)
Partition 2
FOR VALUES
FROM (100) TO (200)
SELECT * FROM t1
WHERE c1 BETWEEN 150 AND 250;
Constraintexclusion

14. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 14
Query optimization: partition-wise join
Partition 1
FOR VALUES
(0) TO (100)
Partitioned table
t1 (c1 int, ...)
Partition 3
FOR VALUES
(200) TO (300)
Partition 2
FOR VALUES
(100) TO (200)
Partition 1
FOR VALUES
(0) TO (100)
Partition 3
FOR VALUES
(200) TO (300)
Partition 2
FOR VALUES
(100) TO (200)
Partition 1
FOR VALUES
(0) TO (100)
Partitioned table
t2 (c1 int, ...)
Partition 3
FOR VALUES
(200) TO (300)
Partition 2
FOR VALUES
(100) TO (200)
t1 JOIN t2
ON t1.c1 = t2.c1
Partitioned join
Partition 3
FOR VALUES
(200) TO (300)
Partition 1
FOR VALUES
(0) TO (100)
Partition 3
FOR VALUES
(200) TO (300)
Partition 2
FOR VALUES
(100) TO (200)
Partition 3
FOR VALUES
(200) TO (300)
Partition 1
FOR VALUES
(0) TO (100)
Partition 3
FOR VALUES
(200) TO (300)
Partition 2
FOR VALUES
(100) TO (200)

15. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 15
Query optimization: partition-wise
aggregation
Partitioned table
t1 (c1 int, ...)
Partition 1
Partition 3
Partition 2
Partition 1
Partitioned table
t2 (c1 int, ...)
Partition 3
Partition 2
t1 JOIN t2
ON t1.c1 = t2.c1
Partition 1
Partition 3
Partition 2
Partition 1
Partition 3
Partition 2
Agg/Group
Agg/Group
Agg/Group
Agg/Group

16. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 16
• Bad partitioning is worse than no partitioning
• Query optimization
– Partition key is the key to success
– Columns in conditions
– include it in the query
• Storage management
– Columns that decide the storage
– Usually timestamp of the row
– Easy to add and drop partitions
• Keep an eye on current limitations
– Expected to reduce with next few releases
Choosing partitioning scheme

17. © Copyright EnterpriseDB Corporation, 2017. All Rights Reserved. 17
• We have just begun …
• No triggers on partitioned table
– Need to create those for each of the partitions
• No SPLIT, MERGE, EXCHANGE partition
• No global indexes
– No primary key, unique constraints
• Foreign partitions
– INSERTs via parent table not supported
– No ACID support for transactions involving multiple foreign servers
Limitations