Databases-on-AWS-Purpose-built-databases,-the-right-tool-for-the-right-job

Jayson,
Solutions Architect, AWS
Databases on AWS
Purpose-built databases,
the right tool for the right job

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2
User Interface
Business Layer
Data Interface
Relational DB
單體式架構微服務架構
Microservices UI
Microservices
Microservices
Microservices Microservices
Relational DB Key-Value DB In-Memory DB
隨著時代演進，系統架構也不斷優化

2007年，GILT的單體架構

2016年，GILT在AWS上的微服務架構
参考：
https://www.slideshare.net/AmazonWebServices/aws-reinvent-2016-from-monolithic-to-microservices-evolving-architecture-patterns-in-the-cloud-arc305

“Because we heard X is the best new thing.”
“Because we have a site license for X.”
“Because X is what we know how to use.”

“Because this database is purpose built to support
what my application is designed to do.”

“Lift and shift” existing
apps to the cloud
原有應用程式搬遷
Quickly build new
apps in the cloud
建置新應用程式或
重構既有應用程式

apps to the cloud
原有應用程式搬遷
Quickly build new
apps in the cloud

Traditional Application Requirements
Users: 10–100k
Data volume: GB–TB
Locality: HQ
Performance: Seconds
Request Rate: Tens of thousands
Access: Internal servers, PCs
Scale: Up
Economics: Pay up front
Developer Access: Days/weeks/months
HR Payroll …
CRM ERP

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Aurora
MySQL and PostgreSQL-compatible relational database built for the cloud
Performance and availability of commercial-grade databases at 1/10th the cost
Performance
and scalability
Availability
and durability
Highly secure Fully managed
5x throughput of standard
MySQL and 3x of standard
PostgreSQL; scale-out up to
15 read replicas
Fault-tolerant, self-healing
storage; six copies of data
across three Availability Zones;
continuous backup to Amazon
S3
Network isolation,
encryption at
rest/transit
Managed by RDS:
No hardware provisioning,
software patching, setup,
configuration, or backups

Amazon Relational Database Service (RDS)
Managed relational database service with a choice of six popular database engines
Easy to administer Available and durable Highly scalable Fast and secure
No need for infrastructure
provisioning, installing, and
maintaining DB software
Automatic Multi-AZ data
replication; automated backup,
snapshots, failover
Scale database
compute and storage
with a few clicks with
no app downtime
SSD storage and
guaranteed provisioned
I/O; data encryption at rest
and in transit

AWS Database Migration Service (AWS DMS)
M I G R A T I N G
D A T A B A S E S
T O A W S
Migrate between on-premises and AWS
Migrate between databases
Automated schema conversion
Data replication for
zero-downtime migration
100,000+
databases migrated

apps to the cloud
Quickly build new
apps in the cloud
建置新應用程式或
重構既有應用程式

Modern apps create new requirements
Users: 1 million+
Data volume: TB–PB–EB
Locality: Global
Performance: Milliseconds–microseconds
Request rate: Millions
Access: Web, mobile, IoT, devices
Scale: Up-down, Out-in
Economics: Pay for what you use
Developer access: No assembly requiredSocial mediaRide hailing Media streaming Dating

Common data categories and use cases
Relational
Referential
integrity, ACID
transactions,
schema-
on-write
Lift and shift, ERP,
CRM, finance
Key-value
High
throughput, low-
latency reads
and writes,
endless scale
Real-time bidding,
shopping cart,
social, product
catalog, customer
preferences
Document
Store
documents
and quickly
access
querying on
any attribute
Content
management,
personalization,
mobile
In-memory
Query by key
with
microsecond
latency
Leaderboards,
real-time analytics,
caching
Graph
Quickly and
easily create
and navigate
relationships
between
data
Fraud detection,
social networking,
recommendation
engine
Time-series
Collect, store,
and process
data
sequenced by
time
IoT applications,
event tracking
Ledger
Complete,
immutable, and
verifiable history
of all changes to
application data
Systems
of record, supply
chain, health care,
registrations,
financial
Aurora, RDS DynamoDB DocumentDB ElastiCache Neptune Timestream QLDB
AWS
Service(s)
Common
Use Cases

1. Relational

Relational
Referential
integrity, ACID
transactions,
schema-on-
write
CRM, finance
time Joe Larry
1 $100 $1000
2 +$1000 -$1000
3 $1100 $0

Relational
Customer
name
notify_phone
type
PK
Shipments
date_shipped
state
PK
Customer Addresses
address_id
customer_id
label
street
City
Postal_code
PK
Line_item
order_id
shipment_id
cost
PK
Order
customer_id
time_ordered
complete
ship_date
order_total
tax
shipping
PK
Products
product_id
sku
inventory
name
description
PK
FK
FK
FK
FK
FK
FK
customer_id
order_id
customer_id
line_item_id
product_id
order_id
Referential
integrity, ACID
transactions,
schema-on-
write
CRM, finance

2. Key-value

Key-value
Table
1
Table 2 Table N
…
…
…
… … …
Partitions Partitions Partitions
… …
…
Highly partitionable data
Low-latency,
key look-ups
with high
throughput and
fast ingestion
of data
Real-time bidding,
shopping cart, IoT
device tracking

Key-value
Low-latency,
key look-ups
with high
throughput and
fast ingestion
of data
Real-time bidding,
shopping cart, IoT
device tracking
key
value
Table
“People”

Amazon DynamoDB
Fast and flexible NoSQL database service for any scale
Handles millions of requests per
second
Delivers microsecond latency
Automated global replication
ACID transactions
Encryption at rest
On-demand backup and restore
Maintenance free
Auto scaling
On-demand capacity mode

Netflix uses DynamoDB as a massive-
scale metadata store for A/B testing
Netflix selected DynamoDB for
operational resiliency:
• Operate their DynamoDB metadata store
across multiple AWS Regions
Selected DynamoDB to handle Netflix
scale:
• Netflix runs hundreds of A/B tests at any
given time and DynamoDB can handle that
scaling up and down
• Across millions of user accounts
Selected DynamoDB for performance:
• Netflix requires low latency across millions
of JSON documents

Profile, match, and swipe data
25 TB data, 20 billion matches,
190 countries
Migrated from MongoDB to
DynamoDB
• 60 percent cost savings
“DynamoDB helps us achieve greater
developer efficiency…at a lower cost.”
—Jun-young Kwak
Tinder

3. Document

Document
(client) (app) (database)
JSON !JSON
!=
Indexing and
storing
documents with
support
for query on
any attribute
Content management,
personalization,
mobile

Document
{id: 34,
name: larry,
url: ‘www.aws.amazon.com”,
attributes:
[{
project: alpha,
location: kumo floor 5,
team: [{id: 3}, {id: 1}]
}]
}
Indexing and
storing
documents with
support
for query on
any attribute
Content management,
personalization,
mobile

4. In-memory

Microseconds
latency,
key-based
queries, and
specialized
data structures
Leaderboards, real-
time analytics, caching
(Leaderboard)
Microseconds
are now the new
milliseconds
In-memory

Database
Memory
(Buffer pool)
Disk
Query processor
Get/Put APIs
Memory
Milliseconds to microseconds (10x faster)
Storage Engine
Microseconds
latency,
key-based
queries, and
specialized
data structures
Leaderboards, real-
time analytics, caching
In-memory

Introducing Amazon ElastiCache
Fully-managed, Redis or Memcached compatible, low-latency, in-memory data store
Fully
Managed
Extreme
Performance
Easily
Scalable
AWS manages all
hardware and software
setup, configuration,
monitoring
In-memory data store and
cache for sub-millisecond
response times
Read scaling with
replicas. Write and memory
scaling with sharding.
Non disruptive scaling

RDS and ElastiCache are Behind Grab’s
Taxi-Booking App
Grab is a popular taxi hailing app in
southeast Asia.
Average response time of the API layer is
<40ms, mandating an in-memory layer to
achieve such performance.
A small devops team that tried running
Redis on EC2 before, but that was too much
work. Using both RDS and ElastiCache in
Multi-AZ allowed them to outsource all the
management to AWS.
The latency of a cab call must be low, and remain low
even in times of peak traffic of hundreds of thousands
of cab requests per minute. We use ElastiCache for
Redis in front of RDS MySQL to keep our systems’
real time performance at any scale.”
–Ryan Ooi
Sr. Devops Engineer, Grab
“

5. Graph

Graph
Creating and
navigating
relations
between data
easily
and quickly
Fraud detection, social
networking,
recommendation
engine

Graph use case
// Product recommendation to a user
gremlin> V().has(‘name’,’sara’).as(‘customer’).out(‘follows’).in(‘follows’).out(‘purchased’)
( (‘customer’)).dedup() (‘name’) ('name')
PURCHASED PURCHASED
FOLLOWS
PURCHASED
KNOWS
PRODUCT
SPORT
FOLLOWS
FOLLOWS
// Identify a friend in common and
make a recommendation
gremlin> g.V().has('name','mary').as(‘start’).
both('knows').both('knows’).
where(neq(‘start’)).
dedup().by('name').properties('name')

FAST RELIABLEOPEN
Query billions of
relationships with
millisecond latency
6 replicas of your data
across 3 AZs with full
backup and restore
Build powerful
queries easily with
Gremlin and SPARQL
Supports Apache
TinkerPop & W3C
RDF graph models
EASY

6. Time-Series

Application
events
IoT Sensor
Readings
DevOps data
Humidity
% WATER VAPOR
91.094.086.093.0

Existing time-series databasesRelational databases
Difficult to
maintain high
availability
Difficult to
scale
Limited data
lifecycle
management
Inefficient
time-series data
processing
Unnatural for
time-series
data
Rigid schema
inflexible for fast
moving time-
series data

1,000x faster and 1/10th
the cost of relational
databases
Collect data at the rate of
millions of inserts per
second (10M/second)
Trillions of
daily events
Adaptive query processing
engine maintains steady,
predictable performance
Time-series
analytics
Built-in functions for
interpolation, smoothing,
and approximation
Serverless
Automated setup,
configuration, server
provisioning, software
patching

7. Ledger

Challenges with building ledgers
Adds unnecessary
complexity
BlockchainRDBMS - audit tables
Difficult to
maintain
Hard to use
and slow
Hard to build
Custom audit functionality using
triggers or stored procedures
Impossible to verify
No way to verify changes
made to data by sys admins

ID Manufacture
r
Model Year VIN Owner
ID Versio
n
Start End Manufactur
er
How it works
ID Manufacture
r
1 Tesla Model S 201
2
12345678
9
Traci Russell
INSERT INTO cars <<
{ 'Manufacturer': 'Tesla',
'Model': 'Model S',
'Year': '2012',
'VIN': '123456789',
'Owner': 'Traci Russel' }
>>
FROM cars WHERE VIN = '123456789' UPDATE owner = 'Ronnie Nash'FROM cars WHERE VIN = '123456789' UPDATE owner = 'Elmer Hubbard'
J
ID Versio
n
er
1 1 07/16/201
2
NULL Tesla Model S 201
2
12345678
9
Traci Russell
current.cars
C
history.cars
H ID Versio
n
er
1 1 07/16/201
2
08/03/201
3
Tesla Model S 201
2
12345678
9
Traci Russell
1 2 08/03/201
3
2
12345678
9
Ronnie Nash
ID Versio
n
er
1 1 07/16/201
2
08/03/201
3
Tesla Model S 201
2
12345678
9
Traci Russell
1 2 08/03/201
3
09/02/201
6
Tesla Model S 201
2
12345678
9
Ronnie Nash
1 3 09/02/201
6
2
12345678
9
Elmer Hubbard
ID Manufacture
r
1 Tesla Model S 201
2
12345678
9
Ronnie Nash
ID Manufacture
r
1 Tesla Model S 201
2
12345678
9
Elmer Hubbard
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Owner: Traci Russell
Metadata: {
Date:07/16/2012
}
H (x) UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H (x) UPDATE cars
ID:1
Owner: Elmer Hubbard
Metadata: {
Date: 09/02/2016
}
H (x)

Amazon Quantum Ledger Database (QLDB) (Preview)
Fully managed ledger database
Track and verify history of all changes made to your application’s data
Immutable
Maintains a sequenced record
of all changes to your data,
which cannot be deleted or
modified; you have the ability
to query and analyze the full
history
Cryptographically
verifiable
Uses cryptography to
generate a secure
output file of your
data’s history
Easy to use
Easy to use, letting you
use familiar database
capabilities like SQL APIs
for querying the data
Highly scalable
Executes 2–3X as many
transactions than ledgers
in common blockchain
frameworks

???
???
這麼多資料庫類型到底要怎麼選？！

The Iron Triangle of Data - All About CAP
C
A
PConsistency:
all clients always
have the same view
of data
Partition tolerance:
the system works well despite
physical network partitions
Availability:
all clients can always
read and write
CA
MSSQL
Oracle
DB2
MySQL
Aster Data
Greenplum
Postgres
CP
Big Table
Hypertable
HBase
MongoDB
Terastore
Couchbase
Scalaris
DynamoDB
BerkeleyDB
Memcached
Redis
Pick Two
AP
Voldemort
Tokyo Cabinet
KAI
DynamoDB
Cassandra
SimpleDB
CouchDB
Riak
Data Models:
Relational
Wide Column
Document
Key/Value

Infinite Scale:
The database can gracefully increase
size and throughput without practical
limits
The Iron Triangle of Purpose (The PIE Theorem)
I
P
E Efficiency:
The database will deliver required
query latency for the workload at all
times
Pattern Flexibility:
The database supports random access
patterns and ad hoc queries
PI
Amazon RDS
Elasticsearch
Aurora Serverless
Neptune
IE
Pick Two
PE
Data Models:
Relational
Wide Column
Document
Graph
Columnar
Unstructured
Amazon DynamoDB
Document DB (Mongo)
Amazon Redshift
Athena

Airbnb uses different databases based
on the purpose
User search history: Amazon DynamoDB
• Massive data volume
• Need quick lookups for personalized search
Session state: Amazon ElastiCache
• In-memory store for submillisecond site rendering
Relational data: Amazon RDS
• Referential integrity
• Primary transactional database

反向思考 - 你要解決的問題是什麼？
Choose the right tool for each job

THANK YOU!

Databases-on-AWS-Purpose-built-databases,-the-right-tool-for-the-right-job

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Databases-on-AWS-Purpose-built-databases,-the-right-tool-for-the-right-job

Semelhante a Databases-on-AWS-Purpose-built-databases,-the-right-tool-for-the-right-job (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Databases-on-AWS-Purpose-built-databases,-the-right-tool-for-the-right-job