AWS Community Day 2018 re:Invent 특집 김영헌 - AWSKRUG 구로디지털모임

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Aurora Serverless - 서버리스 RDBMS의 서막
AWS re:Invent 2017
김 영 헌
me@younghun.kim

2. Topics
● AWS re:Invent 2017
● Aurora Parallel Query Processing
● Aurora Multi-Master
● Aurora Serverless

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

6. New Service ! New Sessions !!

7. https://youtu.be/rPmKo2g9znA

8. ● Aurora & MySQL
● Aurora Parallel Processing
● Aurora Multi-Master
● Aurora Serverless

9. Aurora & MySQL

10. Aurora & MySQL
BINLOG DATA DOUBLE-WRITELOG FRM FILES
TYPE OF WRITE
MYSQL WITH REPLICA
EBS MIRROREBS MIRROR
AZ 1 AZ 2
AMAZON S3
EBS
AMAZON
ELASTIC BLOCK
STORE (EBS)
PRIMARY
INSTANCE
REPLICA
INSTANCE
AZ 1 AZ 3
PRIMARY
INSTANCE
Amazon S3
AZ 2
REPLICA
INSTANCE
AMAZON AURORA
ASYNC
4/6 QUORUM
DISTRIBUTED
WRITES
REPLICA
INSTANCE

11. Aurora & MySQL
AZ 1 AZ 3
PRIMARY
INSTANCE
Amazon S3
AZ 2
REPLICA
INSTANCE
AMAZON AURORA
ASYNC
4/6 QUORUM
DISTRIBUTED WRITES
REPLICA
INSTANCE
Head Node
Database Node
Compute Node
Storage Node

12. https://aws.amazon.com/ko/about-aws/whats-new/2017/11/sign-up-for-the-preview-of-amazon-aurora-serverless/
https://aws.amazon.com/ko/about-aws/whats-new/2017/11/sign-up-for-the-preview-of-amazon-aurora-multi-master/

13. Quorum based voting
AZ 1 AZ 2 AZ 3
Quorum
break on
AZ failure
2/3 read
2/3 write
AZ 1 AZ 2 AZ 3
Quorum
survives
AZ failure
3/6 read
4/6 write
AZ 1 AZ 2 AZ 3
Rules
● Vr + Vw > V
● Vw > V/2
AZ 1 AZ 2 AZ 3
● V = 3, Vr = 2, Vw = 2
● V = 6, Vr = 3, Vw = 4

14. AURORA PARALLEL QUERY

15. Parallel query processing
Aurora storage has thousands of CPUs
▪ Presents opportunity to push down and parallelize
query processing using the storage fleet
▪ Moving processing close to data reduces network
traffic and latency
However, there are significant challenges
▪ Data stored in storage node is not range partitioned –
require full scans
▪ Data may be in-flight
▪ Read views may not allow viewing most recent data
▪ Not all functions can be pushed down to storage nodes
DATABASE NODE
STORAGE NODES
PUSH DOWN
PREDICATES
AGGREGATE
RESULTS

16. Processing at head node
STORAGE
NODES
OPTIMIZER
EXECUTOR
INNODB
NETWORK STORAGE DRIVER
AGGREGATOR
APPLICATION
PARTIAL
RESULTS
STREAM
RESULTS
IN-FLIGHT
DATA
PQ CONTEXT
PQ PLAN
Query Optimizer produces PQ Plan and
creates PQ context based on leaf page
discovery
PQ request is sent to storage node along
with PQ context
Storage node produces
▪ Partial results streams with processed
stable rows
▪ Raw stream of unprocessed rows with
pending undos
Head node aggregates these data
streams to produce final results

17. Processing at storage node
Each storage node runs up to 16 PQ
processes, each associated with a
parallel query
PQ process receives PQ context
▪ List of pages to scan
▪ Read view and projections
▪ Expression evaluation byte code
PQ process makes two passes through
the page list
▪ Pass 1: Filter evaluation on
InnoDB formatted raw data
▪ Pass 2: Expression evaluation
on MySQL formatted data
PQ PROCESS
PQ PROCESS
Up to 16
TO/FROM HEAD NODE
STORAGE
NODE
PROCESS
PAGE LISTS

18. Parallel query performance
Latency (seconds)
Decision support benchmark, R3.8xlarge, cold buffer cache
Improvement factor
with Parallel Query
24.6x
18.3x
5.0x

19. AURORA MULTI-MASTER

21. Shared Disk vs Shared Nothing
SQL
TRANSACTIONS
CACHING
LOGGING
SQL
TRANSACTIONS
CACHING
LOGGING
SHARED DISK CLUSTER
STORAGE
APPLICATION
SHARED NOTHING
SQL
TRANSACTIONS
CACHING
LOGGING
SQL
TRANSACTIONS
CACHING
LOGGING
APPLICATION
STORAGE STORAGE

22. Today Aurora
Shared Distributed Storage Volume
APPLICATION
Read/Write
Master 1
Read
Master 2
Read
Master 3
AZ 1 AZ 2 AZ 3

23. Aurora Multi-Master
Shared Distributed Storage Volume
APPLICATION
Read/Write
Master 2
Read/Write
Master 1
Read/Write
Master 3
AZ 1 AZ 2 AZ 3

24. Conflict resolution using distributed ledgers
There are many “oases” of
consistency in Aurora
The database nodes know
transaction orders from that node
The storage nodes know transaction
orders applied at that node
Only have conflicts when data
changed at both multiple database
nodes AND multiple storage nodes
Much less coordination required
2 3 4 5 61
BT1 [P1]
BT2 [P1]
BT3 [P1]
BT4 [P1]
BT1
BT2
BT3
BT4
Page1
Quorum
OT1
OT2
OT3
OT4
Page 1 Page 2
2 3 4 5 61
OT1[P2]
OT2[P2]
OT3[P2]
OT4[P2]
PAGE1 PAGE
2
MASTE
R
MASTE
R
Page 2

25. Hierarchical conflict resolution
Both masters are writing to two
pages P1 and P2
BLUE master wins the quorum at
page P1; ORANGE master wins
quorum at P2
Both masters recognize the conflict
and have two choices: (1) roll back
the transactions or (2) escalate to
regional resolver
Regional arbitrator decides who
wins the tie breaker
2 3 4 5 61
BT1 [P1]
OT1 [P1]
2 3 4 5 61
OT1[P2]
BT1[P2]
PAGE1 PAGE2
MASTER
MASTER
BT1 OT1
Regional
resolver
Page 1 Page 2 Page 1 Page 2
Quorum
X X

26. AURORA SERVERLESS

28. Aurora Serverless
▪ Starts up on demand, shuts down
when not in use
▪ Scales up/down automatically
▪ No application impact when scaling
▪ Pay per second, 1 minute minimum
WARM POOL
OF INSTANCES
APPLICATION
DATABASE STORAGE
SCALABLE DB CAPACITY
REQUEST ROUTER
DATABASE END-
POINT

29. Database end-point provisioning
When you provision a database, Aurora
Serverless:
▪ Provisions VPC end-points for the
application connectors
▪ Initializes request routers to accept
connections
▪ Creates an Aurora storage volume
A database instance is only provisioned
when the first request arrives
APPLICATION
CUSTOMER VPC
VPC
END-POINTS
VPC
END-POINTS
NETWORK LOAD BALANCER
STORAGE
VOLUME
REQUEST
ROUTERS

30. Instance provisioning and scaling
▪ First request triggers instance
provisioning. Usually 3-5 seconds
▪ Instance auto-scales up and down as
workload changes. Usually 1-3 seconds
▪ Instances hibernate after user-defined
period of inactivity
▪ Scaling operations are transparent to the
application – user sessions are not
terminated
▪ Database storage is persisted until
explicitly deleted by user
DATABASE STORAGE
WARM POOL
APPLICATION
REQUEST
ROUTER
CURRENT
INSTANCE
NEW
INSTANCE

31. Use Cases
Infrequently Used/
New Applications
Batch / Analytics
Variable/
Unpredictable
Workloads
Dev/Test

32. ● Pay per second, 1 minute minimum
● ACU = Aurora Capacity Unit
● 1 ACU
○ approximately 2 GB of memory
○ corresponding CPU and networking
● Price
○ $ 0.06 / 1 hour, N. Virginia
● Scaling
○ 1 ~ 256 ACU
Pricing

33. Pricing
Type vCPU Memory (GiB) Price ($/hour)
r4.large 2 15.25 0.258
r4.xlarge 4 30.5 0.57
● Aurora, (N. Virginia)
ACU vCPU Memory (GiB) Price ($/hour)
5 ACU ?? ≈ 10 0.3
10 ACU ?? ≈ 20 0.6
● Aurora Serverless, (N. Virginia) *
≈ x5
≈ x10
* Official specification of ACUs is not unveiled yet. All the information of Aurora Serverless is based on my estimation.

34. Aurora Parallel Processing
Aurora Multi-Master
Aurora Serverless

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

36. Web | Preview Request - Amazon Aurora Serverless
Web | Amazon Aurora Serverless
Web | In The Works - Amazon Aurora Serverless
Youtube | AWS re:Invent 2017 Keynote - Andy Jassy
Youtube | AWS re:Invent 2017: Deep Dive on the Amazon Aurora MySQL-
compatible Edition (DAT301) - Anurag Gupta
Slide | AWS re:Invent 2017: Deep Dive on the Amazon Aurora MySQL-compatible
Edition (DAT301) - Anurag Gupta
References