Databases on AWS - The right tool for the right job - ADB203 - Santa Clara AWS Summit.pdf

© 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Databases on AWS: The right tool for
the right job
Joseph Idziorek
Principal Product Manager
Amazon Web Services
A D B 2 0 3

Purpose built
The right tool for
the right job
https://www.allthingsdistributed.com/2018/06/purpose-built-databases-in-aws.html

Agenda
1. Common data categories and how we got here
2. Discuss the purpose and use cases of each data model
3. Demo
4. Where to learn more

Data categories and common use cases
Relational Key value Document In-memory Graph
Referential
integrity, ACID
transactions,
schema-
on-write
Low-latency,
key lookups
with high
throughput and
fast ingestion
of data
Indexing and
storing
documents
with support
for query on
any attribute
Microseconds
latency, key-
based queries,
and specialized
data structures
Creating and
navigating
data relations
easily and quickly
Lift and shift,
EMR, CRM,
finance
Real-time bidding,
shopping cart,
social
Content
management,
personalization,
mobile
Leaderboards,
real-time
analytics, caching
Fraud detection,
social networking,
recommendation
engine
Search
Indexing and
searching
semistructured
logs and data
Product catalog,
help, and FAQs,
full text
Time series Ledger
Collect, store,
and process data
sequenced by
time
IoT applications,
event tracking
Complete,
immutable,and
verifiablehistory of
all changes to
application data
Systems
of record,
supply chain,
healthcare,
registrations,
financial

AWS: Purpose-built databases
Relational Key value Document In-memory Graph Search
Amazon
DynamoDB
Amazon
Neptune
AmazonRDS
Aurora CommercialCommunity
Amazon
ElastiCache
Amazon
Elasticsearch
Service
Amazon
DocumentDB
Time series Ledger
Amazon
Timestream
Amazon
Quantum
Ledger
DatabaseMemcachedRedis

1970 1980 1990 2000
Oracle DB2
SQL Server
MySQL
PostgreSQL
DynamoDB
Redis
MongoDB
Elasticsearch
Neptune
CassandraAccess
Aurora
2010
Timestream
QLDB
Amazon
DocumentDB

400,000+ customers using AWS databases
and analytics services

Two fundamental areas of focus
“Lift and shift” existing
apps to the cloud
Quickly build new
apps in the cloud

AWS Database Migration Service (AWS DMS)
M i g r a t i n g
d a t a b a s e s
t o A W S
Migrate between on-premises and AWS
Migrate between databases
Automatedschema conversion
Datareplication for
migration with zero downtime
100,000+
databases migrated

Modern apps create new requirements
Users: 1 million+
Data volume: TB–PB–EB
Locality: Global
Performance: Milliseconds–microseconds
Request rate: Millions
Access: Web, mobile, IoT, devices
Scale: Up-down, Out-in
Economics: Pay for what you use
Developer access: No assembly requiredSocial mediaRide hailing Media streaming Dating

Airbnb uses different databases based
on the purpose
User search history: Amazon DynamoDB
• Massive data volume
• Need quick lookups for personalized search
Session state: Amazon ElastiCache
• In-memory store for submillisecond site rendering
Relational data: Amazon RDS
• Referential integrity
• Primary transactional database

Challenge
Wanted to enable anyoneto learn a
language for free.
Solution
Purpose-built databases from AWS:
• DynamoDB: 31B items tracking which
language exercises completed
• Aurora:Primary transactionaldatabase
for user data
• ElastiCache:Instant access to common
words and phrases
Result
More people learning a languageon
Duolingothan entire US school system
300M total users
7B exercises per month

Relational model
Data model
• Data is stored in rows and tables
• Data is normalized
• Strict schema
• Relationships established via
keys enforced by the system
• Data accuracy and consistency
• Complex queries
Patient
* Patient ID
FirstName
LastName
Gender
DOB
* Doctor ID
Visit
* Visit ID
* Patient ID
* Hospital ID
Date
* Treatment ID
Medical Treatment
* Treatment ID
Procedure
How Performed
AdverseOutcome
Contraindication
Doctor
* Doctor ID
FirstName
LastName
Medical Specialty
* Hospital Affiliation
Hospital
* Hospital ID
Name
Address
Rating

Relational model Patient
* Patient ID
FirstName
LastName
Gender
DOB
* Doctor ID
Visit
* Visit ID
* Patient ID
* Hospital ID
Date
* Treatment ID
Medical Treatment
* Treatment ID
Procedure
How Performed
AdverseOutcome
Contraindication
Doctor
* Doctor ID
FirstName
LastName
Medical Specialty
* Hospital Affiliation
Hospital
* Hospital ID
Name
Address
Rating
Query model: SQL
SELECT
d.first_name, d.last_name, count(*)
FROM
visit as v,
hospital as h,
doctor as d
WHERE
v.hospital_id = h.hospital_id
AND h.hospital_id = d.hospital
AND v.t_date > date_trunc('week’,
CURRENT_TIMESTAMP - interval '1 week')
GROUP BY
d.first_name, d.last_name;

Use cases
Amazon.com is the world’s leading
online retailer. The Amazon
Transaction Risk Management Services
(TRMS) team migrated more than 100
on-premises Oracle databases to
Amazon Aurora.
Enterprise applications Software as a service (SaaS)
applications
Web and mobile gaming
“We are excited to announce the
availability of Remedy ITSM on
AWS Cloud. Our customers can
now benefit from best-in-class
cloud service, installation time
that’s three times faster, and lower
cost of ownership by supporting
migration to Aurora PostgreSQL."
“At the UN, we operate multiple
websites with global reach that
require mission-critical reliability
and consistent performance. We
were able to achieve superb
performance, even with Amazon
Aurora’s smallest database engine.”

Amazon Aurora
MySQL and PostgreSQL-compatible relational database built for the cloud
Performance and availability of commercial-grade databases at 1/10 the cost
Performance
and scalability
Availability
and durability
Highly secure Fully managed
5x throughput of standard MySQL
and 3x of standardPostgreSQL;
scale-out up to
15 read replicas
Fault-tolerant,self-healing storage;
six copies of data
across three Availability Zones;
continuousbackup to Amazon S3
Network isolation,
encryption at rest/transit
Managedby Amazon RDS:
No hardwareprovisioning,software
patching, setup, configuration,or
backups

Amazon Aurora ascendant: How we designed a cloud-
native relational database
Blog: https://www.allthingsdistributed.com/

Amazon Relational Database Service (Amazon RDS)
Managed relational database service with a choice of six popular database engines
Easy to administer Available and durable Highly scalable Fast and secure
No need for infrastructure
provisioning,installing,and
maintainingDB software
Automatic Multi-AZ datareplication;
automatedbackup, snapshots,
failover
Scale database compute
and storage with a few
clicks with no app
downtime
SSD storage and guaranteed
provisionedI/O; data
encryption at rest and in transit

Key-value data
• Simple key-value pairs
• Partitioned by keys
• Resilient to failure
• High throughput, low-
latency reads and writes
• Consistent performance at
scale
Table 1
…
…
Partitions
…
Highly partitionable data

Gamers
PrimaryKey
Attributes
Gamer Tag Type
Hammer57
Rank
Level Points Tier
87 4050 Elite
Status
Health Progress
90 30
Weapon
Class Damage Range
Taser 87% 50
FluffyDuffy
Rank
Level Points Tier
5 1072 Trainee
Status
Health Progress
37 8
// Status of Hammer57
GET {
TableName:"Gamers",
Key: {
"GamerTag":"Hammer57",
"Type":"Status” } }
// Return all Hammer57
Gamers
GamerTag = :a
:a Hammer57
Key-value data

Use cases for key-value data
Social media
Capital One uses DynamoDB to reduce
latency for its mobile applications by
moving its mainframe transactions to a
serverless architecture for unbound scale
Lyft leverages the scalability of DynamoDB
for multiple data stores, including a ride-
tracking system that stores GPS
coordinates for all rides
Snap migrated its largest storage
workload, Snapchat Stories, to
DynamoDB and improved
performance while reducing costs
Mobile IoT

Amazon DynamoDB
Fast and flexible key value database service for any scale
Performance at scale Serverless Comprehensive
security
Global database for
global users and apps
Consistent, single-digit millisecond
response times at any scale; built
applicationswith virtuallyunlimited
throughput
No server provisioning,software
patching, or upgrades; scales up
or down automatically;
continuouslybacks up your data
Encrypts all data by defaultand
fully integrateswith AWS identify
and Access Managementfor
robust security
Build global applicationswith
fast access to localdata by
easily replicatingtables
across multiple AWS regions

Document databases
• Data is stored in JSON-like
documents
• Documents map naturally to
how humans model data
• Flexible schema and indexing
• Expressive query language built
for documents (ad hoc queries
and aggregations)
JSON documents are
first-class objects
of the database
{
id: 1,
name: "sue",
age: 26,
email: "sue@example.com",
promotions: ["new user", "5%", "dog lover"],
memberDate: 2018-2-22,
shoppingCart: [
{product:"abc", quantity:2, cost:19.99},
{product:"edf", quantity:3, cost: 2.99}
]
}

Evolution of document databases
JSON became the de
facto data
interchange format
Friction when
converting JSON to
the relational
model
Object-relational
mappings (ORMs) were
created to help with
this friction
Document
databases solved
the problem
(Client)
(App) (Database)
JSON
!=
RelationalJSON

Use cases for document data
User profiles
{
id: 181276,
username: "sue1942",
name: {first: "Susan",
last: "Benoit"}
}
{
id: 181276,
last: "Benoit"}
}
{
id: 181276,
last: "Benoit"},
ExploidingSnails: {
hi_score: 3185400,
global_rank: 5139,
bonus_levels: true
},
promotions: ["new user","5%","snail lover"]
}
{
id: 181276,
last: "Benoit"},
ExploidingSnails: {
hi_score: 3185400,
global_rank: 5139,
bonus_levels: true
}
}

Use cases for document data
Mobile
Retail and
marketing
User profilesCatalog
Content
management
Personalization

Amazon DocumentDB
Fast,scalable, and fullymanaged MongoDB-compatible databaseservice
Fast Scalable Fully managed MongoDB
compatible
Millionsof requests per
second with millisecondlatency;
twice the
throughput of MongoDB
Separation of compute and
storage enablesboth layersto
scale independently; scale out
to 15 read
replicasin minutes
Managed by AWS:
no hardware provisioning; auto
patching,quick setup,secure,
and automatic backups
Compatible with MongoDB3.6;
use the same SDKs, tools,and
applications with Amazon
DocumentDB

In-memory
• No persistence, in-memory
• Microsecond performance
• Simple commands for
manipulating in memory data
structures
• Strings, hashes, lists, sets,
and sorted sets
Database
Memory
(buffer pool)
Disk
Query processor Get/Put APIs
Memory
Milliseconds to microseconds (10x faster)
Storage engine

In-memory
set a "hello" // Set key "a" with a string value and no expiration OK
get a // Get value for key "a"
"hello"
get b // Get value for key "b" results in miss
(nil)
set b "Good-bye" EX 5 // Set key "b" with a string value and a 5 second expiration
"Good-bye"
get b // Get value for key "b" "Good-bye"
// wait >= 5 seconds
get b (nil) // key has expired, nothing returned

Use cases for in-memory data
Caching
Reducing our database queries up to
95% with simple caching
McDonald’s uses a number of AWS
services, including Amazon EC2, Elastic
Load Balancing, Amazon EBS, and Amazon
ElastiCache, to support its global POS
system, including 200,000 registers and
300,000 POS devices
Airbnb uses ElastiCache for site-
wide caching
Real-time bidding Real time

Amazon ElastiCache
Redis and Memcached compatible, in-memory datastoreand cache
Redis and Memcached
compatible
Extreme
performance
Secure and
reliable
Easily scalable
Fully compatible with open source
Redisand Memcached
In-memory datastore and
cache for microsecond
response times
Network isolation,encryption
at-rest/in-transit,HIPAA, PIC,
FedRAMP,Multi-AZ, and
automatic failover
Scales writesand reads with
shardingand replicas

Search: Full text search

Search
The bright
blue
butterfly
hangs on the
breeze
It’s best to
forget the
great sky and
to retire from
every wind.
Under blue sky,
in bright
sunlight, one
need search
around.
Document 1
Document 2
Document 3
a
and
around
every
for
from
in
is
it
not
on
one
the
to
under
ID Term Document
1 best 2
2 blue 1, 3
3 bright 1,3
4 breeze 1
5 butterfly 1
6 forget 2
7 great 2
8 hangs 1
9 need 3
10 retire 2
11 search 3
12 sky 2, 3
13 wind 2
Inverted indexStopword list

Search
_search?q=house
"hits": {
"total": 85,
"max_score": 6.6137657,
"hits": [{
"_index": "movies",
"_type": "movie",
"_id": "tt0077975",
"_score": 6.6137657,
"_source": { "directors": [ "John Landis" ],
"release_date": "1978-07-27T00:00:00Z",
"rating": 7.5,
"genres": [ "Comedy", "Romance" ],
"image_url": "http://ia.jpg
"plot": "At a 1962 College, Dean Vernon Wormer…",
"title": "Animal House",
"rank": 527,
"running_time_secs": 6540,
"actors": [ "John Belushi","Karen Allen","Tom Hulce" ],
"year": 1978, "id": "tt0077975" }
},

Search use cases
Log analytics
Adobe uses Amazon Elasticsearch
Service to cost-effectively analyze
and visualize large amount of log
data for its developer platform,
which at peak receives over 200K
API calls per second.
Full text search Clickstream analytics
Hearst Corporation built a
clickstream analytics platform using
Amazon Elasticsearch Service,
Amazon Kinesis Data Streams, and
Amazon Kinesis Data Firehose to
transmit and process 30 terabytes
of data a day from 300+ Hearst
websites worldwide.
MirrorWeb uses Amazon Elasticsearch
Service to make the UK Government
and UK Parliament’s web archives
searchable. With Amazon
Elasticsearch Service, MirrorWeb
indexed 1.4B documents for just $337
and indexed 146M docs per hour—14x
faster than the previously used
technology.

Amazon Elasticsearch Service
Fully managed, reliable, and scalableElasticsearch service
Easy to use Scalable Highly available Secure
Deploy a production-ready
Elasticsearch
cluster in minutes
Resize your cluster
with a few clicks
or a single API call
Replicate across
Availability Zones,with
monitoringand
automated self-healing
Deploy into your VPC
and restrict access
using security groups
and IAM policies

Graph data
• Relationships are first-class objects
• Data is modeled and queries as a
graph
• Vertices connected by Edges
• Creating and navigating relations
between data easily
and quickly
Purchased Purchased
Follows
Purchased
Knows
Product
Sport
Follows

Graph

Graph
Social networking
Life sciences Network & IT operationsFraud detection
Recommendations Knowledge graphs

Graph
Fraud detection,
social networking,
recommendation engine

Graph use case
// Product recommendationto a user PURCHASED PURCHASED
PURCHASED
PURCHASED
KNOWS
BOOK#1
BOOK#2
PURCHASED
BOOK#3

Amazon Neptune
Fully managed graph database
Fast Reliable Easy Open
Query billionsof relationships
with millisecond latency
Six replicasof your data across
three AZs with fully backup and
restore
Build powerful querieseasily
with Gremlin and SPARQL
SupportsApache TinkerPop & W3C
RDF graph models

Retail demo application
Demo application:
1. Available today
2. On GitHub:
/aws-samples/aws-
bookstore-demo-app
3. One-click
AWS CloudFormation
deployment
Search
Indexing and
searching
semistructured logs
and data
Product
search
Amazon
Neptune
AmazonElasticsearch
Service
Key-value
High
throughput, Low-
latency
reads
and writes, endless
scale
Shopping cart, user
profile
Graph
Quickly and
easily create and
navigate
relationships
between
data
Product
recommendation
In-memory
Query by key
with
microsecond
latency
Product leaderboard
DynamoDB ElastiCache

Time series data
What is time series data?
What is special about a time
series database? A sequence of data points
recorded over a time interval
Time is the
single primary axis
of the data model
t

Time series use case
Application events
IoT sensor readings
DevOps data
Humidity
% Water vapor
91.094.086.093.0

Existing time-series databasesRelational databases
Difficult to
maintainhigh
availability
Difficult to scale Limited data
lifecycle
management
Inefficient
time series data
processing
Unnaturalfor
time series data
Rigid schema
inflexiblefor
fast moving
time series data
Building with time series data is challenging

Amazon Timestream (sign up for the preview)
Fast, scalable, fully managed time-series database
1,000xfaster and 1/10 the cost
of relational databases
Collect data at the rate of
millionsof inserts per second
(10M/second)
Trillions of
daily events
Adaptivequery processing
engine maintainssteady,
predictableperformance
Time-seriesanalytics
Built-in functionsfor
interpolation,smoothing, and
approximation
Serverless
Automatedsetup, configuration,
server provisioning,software
patching

Common customer use cases
Ledgers with centralized control
Healthcare
Verify and track hospital
equipment inventory
Manufacturers
Track distribution of a
recalled product
HR & payroll
Track changes to an
individual’s profile
Government
Track vehicle title history

Challenges with building ledgers
Adds unnecessary
complexity
BlockchainRDBMS – audit tables
Difficult to
maintain
Hard to use and
slow
Hard to build
Custom audit functionality using
triggers or stored procedures
Impossible to verify
No way to verify changes made to
data by sys admins

ID Manufacturer Model Year VIN Owner
ID Version Start End Manufacturer Model Year VIN Owner
How it works
1 Tesla Model S 2012 123456789 Traci Russell
INSERTINTO cars <<
{ 'Manufacturer': 'Tesla',
'Model': 'Model S',
'Year': '2012',
'VIN': '123456789',
'Owner': 'Traci Russel'}
>>
FROMcars WHEREVIN= '123456789' UPDATEowner ='Ronnie Nash'FROMcars WHEREVIN= '123456789' UPDATEowner ='Elmer Hubbard'
J
1 1 07/16/2012 NULL Tesla Model S 2012 123456789 Traci Russell
current.cars
C
history.cars
H ID Version Start End Manufacturer Model Year VIN Owner
1 1 07/16/2012 08/03/2013 Tesla Model S 2012 123456789 Traci Russell
1 2 08/03/2013 NULL Tesla Model S 2012 123456789 Ronnie Nash
1 1 07/16/2012 08/03/2013 Tesla Model S 2012 123456789 Traci Russell
1 2 08/03/2013 09/02/2016 Tesla Model S 2012 123456789 Ronnie Nash
1 3 09/02/2016 NULL Tesla Model S 2012 123456789 Elmer Hubbard
1 Tesla Model S 2012 123456789 Ronnie Nash
1 Tesla Model S 2012 123456789 Elmer Hubbard
INSERT cars
ID:1
Manufacturer: Tesla
Model: Model S
Year: 2012
VIN: 123456789
Owner: Traci Russell
Metadata: {
Date:07/16/2012
}
H (x) UPDATE cars
ID:1
Owner: Ronnie Nash
Metadata: {
Date:08/03/2013
}
H (x) UPDATE cars
ID:1
Owner: Elmer Hubbard
Metadata: {
Date: 09/02/2016
}
H (x)

Amazon Quantum Ledger Database (Amazon QLDB)
(preview)
Fully managed ledger database
Track and verify history of all changes made to your application’sdata
Immutable
Maintainsa sequenced record of all
changes to your data,which cannot
be deleted or modified;you can to
query and analyze the full history
Cryptographically
verifiable
Uses cryptography to
generate a secure output
file of your data’s history
Easy to use
Easy to use, letting you
use familiar database
capabilitieslike SQL APIs for
querying the data
Highly scalable
Executes 2–3x as many
transactions than ledgers
in common blockchain
frameworks

Benefits of purpose-built databases
Better
performance Better scale
More
functionality
Easier to
debug
Independence
between teams

Team Internet
AWS re:Invent 2017: Running Lean Architectures:
How to Optimize for Cost Efficiency (ARC303)
“For every single purpose within our application, we
our now using different databases”
“…now we can pick the right tool for every job we
have”

https://aws.amazon.com/this-is-my-architecture/

AWS: Purpose-built databases
Relational Key-value Document In-memory Graph Search
Amazon
DynamoDB
Amazon
Neptune
AmazonRDS
Aurora CommercialCommunity
Amazon
ElastiCache
Amazon
Elasticsearch
Service
Amazon
DocumentDB
Time series Ledger
Amazon
Timestream
Amazon
QLDB
MemcachedRedis

Additional resources
Andy Jassy’s re:Invent 2017/2018 keynote:
Databases
2017: https://www.youtube.com/watch?v=1IxDLeFQKPk&feature=youtu.be&t=37m47s
2018: https://youtu.be/ZOIkOnW640A?t=3238
Werner Vogel’s blog: A one size fits all database
doesn't fit anyone
https://www.allthingsdistributed.com/2018/06/purpose-built-databases-in-aws.html
https://aws.amazon.com/products/databases/
AWS Databases

Additional resources
https://www.youtube.com/watch?v=hwnNbLXN4vA
https://www.youtube.com/watch?v=-pb-DkD6cWg
AWS re:Invent 2018: Databases on AWS: The Right
Tool for the Right Job (DAT205-R1)
AWS re:Invent 2018: Building with AWS Databases: Match
Your Workload to the Right Database (DAT301)

Databases on AWS - The right tool for the right job - ADB203 - Santa Clara AWS Summit.pdf

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Databases on AWS - The right tool for the right job - ADB203 - Santa Clara AWS Summit.pdf

Semelhante a Databases on AWS - The right tool for the right job - ADB203 - Santa Clara AWS Summit.pdf (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Databases on AWS - The right tool for the right job - ADB203 - Santa Clara AWS Summit.pdf