Dat305 Deep Dive on Amazon Aurora PostgreSQL

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AmazonRDS is. . . Cloud native engine Open source engines Commercial engines Amazon RDS platform • Automatic failover • Backup & recovery • X-region replication • Isolation & security • Industry compliance • Automated patching • Advanced monitoring • Routine maintenance • Push-button scaling Image credit: By Mackphillips - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=55946550

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AmazonRDS PostgreSQLUniverse CLIENTS RDS PostgreSQL Aurora PostgreSQL EBS Aurora Storage Postgres 9.6/10 — same extensions Backup/Recovery - PITR High Availability & Durability Secure – IAM Auth Read Replicas Cross Region Snapshots Scale Compute – Online Scale Storage Cross Region Replication Outbound Logical Replication Preview 11

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Concurrency—Remove log buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage A Queued Work Storage B C D E 4 3 4 2 4 A B C D E Durability Tracking

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Concurrency—Remove log buffer Queued Work Log Buffer PostgreSQL Aurora PostgreSQL Storage A Queued Work Storage B C D E 6 5 6 3 5 A B C D E Durability Tracking

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL—Writing less Aurora Block in Memory t-v1 Aurora Storage Block in Memory PostgreSQL t-v1 t-v2 t-v1 t-v2 Full Block WAL update t set y = 6;

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL—Writing less Aurora Block in Memory t-v1 Aurora Storage Block in Memory PostgreSQL t-v1 t-v2 t-v3 t-v1 t-v2 Full Block t-v3 WAL update t set y = 6;

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL—Writing less Aurora Block in Memory t-v1 Aurora Storage Block in Memory PostgreSQL t-v1 t-v2 t-v3 checkpoint datafile t-v1 t-v2 Full Block t-v3 WAL archive update t set y = 6;

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL—Writing less Aurora Block in Memory t-v1 Aurora Storage Block in Memory PostgreSQL t-v1 t-v2 t-v3 checkpoint datafile t-v1 t-v2 Full Block t-v3 WAL archive 4K 4K 8K update t set y = 6;

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL—Writing less Aurora Block in Memory t-v1 Aurora Storage Block in Memory PostgreSQL t-v1 t-v2 t-v3 checkpoint datafile t-v1 t-v2 Full Block t-v3 WAL archive 4K 4K 8K update t set y = 6;

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora PostgreSQL—Writing less Aurora update t set y = 6; Block in Memory t-v1 t-v2 t-v3 Aurora Storage t-v1 t-v2 t-v3 no checkpoint = no FPW Block in Memory PostgreSQL t-v1 t-v2 t-v3 checkpoint datafile t-v1 t-v2 Full Block t-v3 WAL archive 4K 4K 8K update t set y = 6;

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. InsertTest Test Table • UUID PK—Random • ID int—Right Lean Sequence • VARCHAR(100)—Random • VARCHAR(50)—Small Set of Words • INT—Random • INT—Random (smaller set) • BOOLEAN—Random (50/50) • BOOLEAN—Somewhat Random (75/25) • Timestamp—Right Lean

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 0 5,000 10,000 15,000 20,000 25,000 30,000 1 31 61 91 121 151 181 211 241 271 InsertsPerSecond Minutes Insert Workload—PostgreSQL 9.6 BASE 16GB Max WAL Aurora PostgreSQL

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 3,729 4,871 17,158 - 5,000 10,000 15,000 20,000 TPS(2UpdatesperTransaction) Update Workload—PostgreSQL 9.6 BASE 16GB Max WAL Aurora PostgreSQL

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AmazonAuroraRecoversUpto97%Faster 3 GiB Redo Recovered in 19 seconds 10 GiB Redo Recovered in 50 seconds 30 GiB Redo Recovered in 123 seconds 0 20 40 60 80 100 120 140 160 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 RecoveryTimeinSeconds(lessisbetter) Writes / Second (more is better) RECOVERY TIME FROM CRASH UNDER LOAD Bubble size represents redo log, which must be recovered As PostgreSQL throughput goes up, so does log size and crash recovery time Amazon Aurora has no redo. Recovered in 3 seconds while maintaining significantly greater throughput.

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurorastorageandreplicas RW Application Application Write log records Automatic scalable to 64TB Availability zone 1 Availability zone 3Availability zone 2 Aurora storage

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurorastorageandreplicas RW Application Application Write log records Read blocks Automatic scalable to 64TB Availability zone 1 Availability zone 3Availability zone 2 Aurora storage

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Aurorastorageandreplicas RW Application RO Application Async Invalidation & Update Async Invalidation & Update Write log records Read blocks Automatic scalable to 64TB Availability zone 1 Availability zone 3Availability zone 2 Aurora storage RORORORO

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Aurorastorageandreplicas RW Application RO Application Async Invalidation & Update Async Invalidation & Update Write log records Read blocks Automatic scalable to 64TB Availability zone 1 Availability zone 3Availability zone 2 Aurora storage RORORORO

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Aurorastorageandreplicas RW Application RO Application Async Invalidation & Update Async Invalidation & Update Write log records Read blocks RW Automatic scalable to 64TB Availability zone 1 Availability zone 3Availability zone 2 Aurora storage RORORORO

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. EBS EBS Availability Zone 1 Availability Zone 2 Availability Zone 3 EBS Typicalsynchronousreplication–3 locations COMMI T Region

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 6 10 21 31 7 12 28 123 0 20 40 60 80 100 120 140 50 90 99.9 99.99 Latency(ms) Percentile High Concurrency Sync Write Test 2 Node (4 copy) 3 Node (6 Copy) Cost of additional synchronous replicas

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora Storage Availability Zone 1 Availability Zone 2 Availability Zone 3 AmazonAurora– 3AZ’s–6 copies COMMI T Region Aurora Storage Aurora Storage Aurora Storage Aurora Storage Aurora Storage

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AmazonAuroragives>2xlowerresponsetimes 0.00 100.00 200.00 300.00 400.00 500.00 600.00 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 responsetime,ms minute sysbench response time (p95), 30 GiB, 1024 clients PostgreSQL (Single AZ, No Backup) Amazon Aurora (Three AZs, Continuous Backup)

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. pgbenchbenchmark Async replication and apply PostgreSQL Aurora PostgreSQL RW RO accounts tellers branches history accounts RW RO accounts tellers branches history

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. pgbenchbenchmark Async replication and apply PostgreSQL Aurora PostgreSQL RW RO accounts tellers branches history accounts tellers branches history RW RO accounts tellers branches history

60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. pgbenchbenchmark Async replication and apply PostgreSQL Aurora PostgreSQL RW RO accounts tellers branches history accounts tellers branches history Async replication and memory update RW RO accounts tellers branches history accounts

63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application FastClones RW Application RW Reporting Application Read blocks Availability Zone 1 Availability Zone 3Availability Zone 2 Aurora storage Primary storage Clone storage Clone

69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application FastClones RW Application RW Reporting Application Write log records Read blocks Availability Zone 1 Availability Zone 3Availability Zone 2 Aurora storage Primary storage Clone storage Clone

72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fast Clone example 0 5000 10000 15000 20000 25000 0 3 6 9 12 15 18 21 24 27 30 33 36 39 42 45 48 51 54 57 60 63 66 69 72 75 78 TransactionsPerSecond(TPS) Minutes PGBench RW Scale 10K - Target Rate 20K TPS Main Database Clone Database

79. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Crossregionreplication Aurora Storage RO Application RW Application RO Application Availability zone 1 Availability zone 3Availability zone 2 Region A Region B Aurora Storage RO Application Availability zone 1 Availability zone 3Availability zone 2

84. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Crossregionreplication Aurora Storage RO Application RW Application RO Application Availability zone 1 Availability zone 3Availability zone 2 Region A Region B Aurora Storage RO Application Application RO Application Availability zone 1 Availability zone 3Availability zone 2 RO

88. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Crossregionreplication Aurora Storage RO Application RW Application RO Application Availability zone 1 Availability zone 3Availability zone 2 Region A Region B Aurora Storage RO Application Application RO Application Availability zone 1 Availability zone 3Availability zone 2 RORW

91. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Caching changes—Nodoublebuffering 488 GB RAMPG + OS processes shared_buffers 25% Linux Pagecache 50+% select of data – check for block in shared_buffers if not in shared_buffers load from pagecache/disk EBS PostgreSQL Aurora PostgreSQL

92. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Caching changes—Nodoublebuffering 488 GB RAMPG + OS processes shared_buffers 25% Linux Pagecache 50+% select of data – check for block in shared_buffers if not in shared_buffers load from pagecache/disk EBS duplicate buffers PostgreSQL Aurora PostgreSQL

93. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Caching changes—Nodoublebuffering 488 GB RAMPG + OS processes shared_buffers 25% Linux Pagecache 50+% select of data – check for block in shared_buffers if not in shared_buffers load from pagecache/disk EBS duplicate buffers PG + OS processes shared_buffers 75% PostgreSQL Aurora PostgreSQL Aurora Storage select of data – check for block in shared_buffers or load from Aurora storage

94. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Caching changes—Nodoublebuffering 488 GB RAMPG + OS processes shared_buffers 25% Linux Pagecache 50+% select of data – check for block in shared_buffers if not in shared_buffers load from pagecache/disk EBS PG + OS processes shared_buffers 75% PostgreSQL Aurora PostgreSQL Aurora Storage select of data – check for block in shared_buffers or load from Aurora storage

95. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Caching changes—Nodoublebuffering 488 GB RAMPG + OS processes shared_buffers 25% Linux Pagecache 50+% select of data – check for block in shared_buffers if not in shared_buffers load from pagecache/disk EBS PG + OS processes shared_buffers 75% PostgreSQL Aurora PostgreSQL Aurora Storage select of data – check for block in shared_buffers or load from Aurora storage Survivable cache

96. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Caching changes—Nodoublebuffering 689,068 417,496 334,691 682,931 - 100,000 200,000 300,000 400,000 500,000 600,000 700,000 800,000 TransactionsPerSecond(tps) pgbench read only - scale 22,000 - r4.16xlarge Aurora 75% Cache PostgreSQL 25% Cache PostgreSQL 10% Cache PostgreSQL 75% Cache 1.6x 2.0x 18K read iops no reads heavy double buffering no double buffering no survivable cache Approx 350GB working set

97. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. ClusterCacheManagement-Failover 0 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 0 60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960 1020 1080 1140 1200 TransactionsperSecond(TPS) Seconds PGBench 20X RO / 1X RW 160GB Cached - Failover at 600 Seconds Baseline 32 seconds

98. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. ClusterCacheManagement-Failover 0 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 0 60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960 1020 1080 1140 1200 TransactionsperSecond(TPS) Seconds PGBench 20X RO / 1X RW 160GB Cached - Failover at 600 Seconds Baseline 340 seconds

99. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application ClusterCache Management(CCM)Feature RW Application RO Application Async Invalidation & Update Availability Zone 1 Availability Zone 3Availability Zone 2 Aurora storage RORORORO

100. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application ClusterCache Management(CCM)Feature RW Application RO Application Async Invalidation & Update Availability Zone 1 Availability Zone 3Availability Zone 2 Aurora storage RORORORO apg_ccm_enabled=on

103. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. ClusterCache Management 0 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 0 60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960 1020 1080 1140 1200 TransactionsperSecond(TPS) Seconds PGBench 20X RO / 1X RW 160GB Cached - Failover at 600 Seconds Baseline CCM Enabled 32 seconds 340 seconds

108. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Plan change Before Aggregate (cost=3804.15..3804.16 rows=1 width=16) -> Nested Loop (cost=12.67..3802.61 rows=307 width=8) -> Index Scan using pgbench_branches_pkey on pgbench_branches b (cost=0.29..16.60 rows=2 width=8) Index Cond: (bid = ANY ('{1,4}'::integer[])) -> Bitmap Heap Scan on pgbench_history h (cost=12.39..1891.47 rows=154 width=8) Recheck Cond: (bid = b.bid) Filter: ((mtime >= (now() - '01:00:00'::interval)) AND (mtime <= (now() - '00:30:00'::interval))) -> Bitmap Index Scan on i_p_bid (cost=0.00..12.35 rows=522 width=0) Index Cond: (bid = b.bid) After Aggregate (cost=171092.96..171092.97 rows=1 width=16) -> Hash Join (cost=329.02..171091.42 rows=307 width=8) Hash Cond: (h.bid = b.bid) -> Seq Scan on pgbench_history h (cost=0.00..166712.20 rows=1542280 width=8) Filter: ((mtime >= (now() - '01:00:00'::interval)) AND (mtime <= (now() - '00:30:00'::interval))) -> Hash (cost=329.00..329.00 rows=2 width=8) -> Seq Scan on pgbench_branches b (cost=0.00..329.00 rows=2 width=8) Filter: (bid = ANY ('{1,4}'::integer[])) • enable_bitmapscan=off • enable_indexscan=off

118. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4,000 5,000 6,000 7,000 8,000 9,000 10,000 1 61 121 181 241 301 361 421 481 541 601 661 721 781 841 901 961 1021 1081 1141 1201 1261 TPS Minutes Updates—No Vacuum running transaction id wrap around

126. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora Serverless  Starts up on demand, shuts down when not in use  Scales up/down automatically  No application impact when scaling  Pay per second, 1 minute minimum WARM POOL OF INSTANCES AURORA STORAGE AURORA REQUEST ROUTER DATABASE END-POINT

127. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora Serverless  Starts up on demand, shuts down when not in use  Scales up/down automatically  No application impact when scaling  Pay per second, 1 minute minimum WARM POOL OF INSTANCES APPLICATION AURORA STORAGE AURORA REQUEST ROUTER DATABASE END-POINT

132. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora Serverless  Starts up on demand, shuts down when not in use  Scales up/down automatically  No application impact when scaling  Pay per second, 1 minute minimum WARM POOL OF INSTANCES APPLICATION AURORA STORAGE AURORA REQUEST ROUTER DATABASE END-POINT AURORA STORAGE

133. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aurora Serverless  Starts up on demand, shuts down when not in use  Scales up/down automatically  No application impact when scaling  Pay per second, 1 minute minimum WARM POOL OF INSTANCES APPLICATION AURORA STORAGE AURORA REQUEST ROUTER DATABASE END-POINT AURORA STORAGE

136. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. MigrationtoAuroraPostgreSQL Methods • PostgreSQL - pg_dump / pg_restore • AWS Data Migration Service (DMS) • RDS PostgreSQL - Snapshot import • RDS PostgreSQL – Read replica

137. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customer Premises, EC2, RDS Application Users Aurora PostgreSQLVPN/Network AWSDMS—Logical replication Start a replication instance

138. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customer Premises, EC2, RDS Application Users Aurora PostgreSQLVPN/Network AWSDMS—Logical replication Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data

139. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customer Premises, EC2, RDS Application Users Aurora PostgreSQLVPN/Network AWSDMS—Logical replication Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in sync

140. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Customer Premises, EC2, RDS Application Users Aurora PostgreSQLVPN/Network AWSDMS—Logical replication Start a replication instance Connect to source and target databases Select tables, schemas, or databases Let the AWS Database Migration Service create tables and load data Uses change data capture to keep them in sync Switch applications over to the target at your convenience

Dat305 Deep Dive on Amazon Aurora PostgreSQL

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Dat305 Deep Dive on Amazon Aurora PostgreSQL

Similar to Dat305 Deep Dive on Amazon Aurora PostgreSQL (20)

Recently uploaded

Recently uploaded (20)

Dat305 Deep Dive on Amazon Aurora PostgreSQL

Editor's Notes