Database Week at the San Francisco Loft: Non-Relational Revolution A decade ago, relational databases were used for nearly every use case. Today, new technologies are enabling a revolution in databases, creating new options for document, key: value, in-memory, search, and graph capabilities that do not use relational tables. We’ll discuss this revolution in database options and who is using them. Speaker: Joseph Idziorek - Sr. Product Manager, Amazon Chimera

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Joseph Idziorek
Non-relational Revolution

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1.Purpose-built/work backwards
2.Common data models/use cases
3.Demo
4.Customer examples
5.Additional resources
Agenda

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A common question that I get is why do we offer so many database products? The answer for me is
simple: Developers want their applications to be well architected and scale effectively. To do this, they
need to be able to use multiple databases and data models within the same application.
https://www.allthingsdistributed.com/2018/06/purpose-built-databases-in-aws.html

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Oracle DB2
SQL Server
MySQL
PostgreSQL
DynamoDB
Redis
Neptune
Cassandra
Aurora
1970 1980 1990 2000 2010
MongoDB
Elasticsearch
Amazon Redshift
Evolution of databases

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Modern application requirements
Users: 1M+
Data volume: TB–PB–EB
Locality: Global
Performance: Milliseconds–microseconds
Request rate: Millions
Access: Mobile, IoT, devices
Scale: Up-out-in
Economics: Pay as you go
Developer access: Instant API access
The picture can't be
displayed.
Relational Key-value Document
In-memory GraphSearch

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
purpose
noun | pur·pose | ˈpər-pəs
The reason for which something is done or created
or for which something exists

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Things with
purpose

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Things with
Purpose

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Work backward from
the problem you are
trying to solve
Choose the right tool for each job

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Purpose-built databases example: Airbnb
Airbnb uses different databases based
on the purpose
User search history: DynamoDB
• Massive data volume
• Need quick lookups for personalized search
Session state: ElastiCache
• In-memory store for sub-millisecond site rendering
Transactional data: RDS
• Referential integrity
• Primary transactional database

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Purpose-built databases example: Expedia
The picture can't be displayed.
Expedia uses Aurora, ElastiCache,
and Redshift for inventory and pricing analytics
Real-time stream processing: ElastiCache
• Multi-stream union and self-join
• 24-hour look-back window
Transactional data: Aurora
• Operational queries
Data warehousing: Redshift
• Analytical queries

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data models and common use cases
Relational Key-value Document In-memory Graph Search
Referential
integrity, ACID
transactions,
schema-on-
write
Low-latency,
key look-ups
with high
throughput and
fast ingestion
of data
Indexing and
storing
documents with
support
for query on
any attribute
Microseconds
latency,
key-based
queries, and
specialized
data structures
Creating and
navigating
relations
between data
easily
and quickly
Indexing and
searching
semistructured
logs and data
Lift and shift, EMR,
CRM, finance
Real-time bidding,
shopping cart, IoT
device tracking
Content management,
personalization, mobile
Leaderboards, real-
time analytics, caching
Fraud detection, social
networking,
recommendation
engine
Product catalog,
help/FAQs, full-text
Amazon Aurora
Amazon RDS
Amazon Redshift
Amazon
DynamoDB
Amazon
DynamoDB
Amazon
ElastiCache for
Redis &
Memcached
Amazon Neptune Amazon
Elasticsearch
Service
The picture can't be displayed.

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Relational
The picture can't be displayed.
Referential
integrity, ACID
transactions,
schema-on-
write
Lift and shift, EMR,
CRM, finance
time Joe Shawn
1 $100 $1000
2 +$1000 -$1000
3 $1100 $0

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Relational
Customer
name
notify_phone
type
PK
Shipments
date_shipped
state
PK
Customer Addresses
address_id
customer_id
label
street
City
Postal_code
PK
Line_item
order_id
shipment_id
cost
PK
Order
customer_id
time_ordered
complete
ship_date
order_total
tax
shipping
PK
Products
product_id
sku
inventory
name
description
PK
FK
FK
FK
FK
FK
FK
customer_id
order_id
customer_id
line_item_id
product_id
order_id
The picture can't be displayed.
Referential
integrity, ACID
transactions,
schema-on-
write
Lift and shift, EMR,
CRM, finance

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Columnar
ID Product Cost Quantity Total_cost
1 Pencil $2.50 4 $10.00
2 Eraser $1.75 1 $1.75
3 Paper $3.00 3 $9.00
4 Pen $1.00 5 $5.00
5 Stapler $2.50 3 $7.50
6 Marker $6.00 2 $12.00
7 Glue $2.00 5 $10.00
Product 1:Pencil, 2:Eraser, 3:Paper, 4:Pen, 5:Stapler,
6:Marker, 7:Glue
Cost 1:$2.50, 2:$1.75, 3:$3.00, 4:$1.00, 5:$2.50,
6:$6.00, 7:$2.00
Quantity 1:4, 2:1, 3:3, 4:5, 5:3, 6:2, 7:5
Total_cost 1:$10.00, 2:$1.75, 3:$9.00, 4:$5.00, 5:$7.50,
6:$12.00, 7:$10.00
SELECT
sum(Quantity)
FROM
product_table
SELECT
sum(Quantity)
FROM
product_table

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key-value

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key-value
Table 1 Table 2 Table N
…
…
…
… … …
Partitions Partitions Partitions
… …
…
Highly partitionable data
Low-latency,
key look-ups
with high
throughput and
fast ingestion
of data
Real-time bidding,
shopping cart, IoT
device tracking

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Document
The picture can't be
displayed.
(client) (app) (database)
JSON !JSON
The picture can't be
displayed.
!=
Indexing and
storing
documents with
support
for query on
any attribute
Content management,
personalization, mobile

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Document
Indexing and
storing
documents with
support
for query on
any attribute
Content management,
personalization, mobile
{id: 34,
name: joe,
url: ‘www.aws.amazon.com”,
attributes:
[{
project: alpha,
location: floor 3,
team: [{id: 3}, {id: 1}]
}]
}

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
In-memory
Microseconds
latency,
key-based
queries, and
specialized
data structures
Leaderboards, real-
time analytics, caching
(Leaderboard)
Microseconds
are now the new
milliseconds

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
In-memory
Database
Memory
(Buffer pool)
Disk
Query processor
Get/Put APIs
Memory
Milliseconds to microseconds (10x faster)
Microseconds
latency,
key-based
queries, and
specialized
data structures
Leaderboards, real-
time analytics, caching
Storage Engine

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Graph
Creating and
navigating
relations
between data
easily
and quickly
Fraud detection, social
networking,
recommendation
engine

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Graph
Creating and
navigating
relations
between data
easily
and quickly
Fraud detection, social
networking,
recommendation
engine

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Search
Indexing and
searching
semistructured
logs and data
Product catalog,
help/FAQs, full-text

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Search
The bright
blue
butterfly
hangs on
the breeze
It’s best to
forget the
great sky
and to retire
from every
wind.
Under blue
sky, in bright
sunlight, one
need search
around.
Document 1
Document 2
Document 3
a
and
around
every
for
from
in
is
it
not
on
one
the
to
under
ID Term Document
1 best 2
2 blue 1, 3
3 bright 1,3
4 breeze 1
5 butterfly 1
6 forget 2
7 great 2
8 hangs 1
9 need 3
10 retire 2
11 search 3
12 sky 2, 3
13 wind 2
Inverted indexStopword list
Indexing and
searching
semistructured
logs and data
Product catalog,
help/FAQs, full-text

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data models and common use cases
Relational Key-value Document In-memory Graph Search
Referential
integrity, ACID
transactions,
schema-on-
write
Low-latency,
key look-ups
with high
throughput and
fast ingestion
of data
Indexing and
storing
documents with
support
for query on
any attribute
Microseconds
latency,
key-based
queries, and
specialized
data structures
Creating and
navigating
relations
between data
easily
and quickly
Indexing and
searching
semistructured
logs and data
Lift and shift, EMR,
CRM
Real-time bidding,
shopping cart, IoT
device tracking
Content management,
personalization, mobile
Leaderboards, real-
time analytics, caching
Fraud detection, social
networking,
recommendation
engine
Product catalog,
help/FAQs, full-text
Amazon Aurora
Amazon RDS
Amazon Redshift
Amazon
DynamoDB
Amazon
DynamoDB
Amazon
ElastiCache for
Redis &
Memcached
Amazon Neptune Amazon
Elasticsearch
Service
The picture can't be displayed.

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo Architecture
Amazon
DynamoDB
Amazon ES
AWS
Lambda
Amazon
Neptune
streams

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of purpose-built databases
Better
performance
Better scale
More
functionality
Easier to debug
Independence
between teams

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Team Internet
AWS re:Invent 2017:
Running Lean
Architectures: How to
Optimize for Cost Efficiency
(ARC303)
“For every single purpose within our application,
we our now using different databases”
“…now we can pick the right tool for every
job we have”

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
https://aws.amazon.com/this-is-my-architecture/

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
RDS
Amazon
ElastiCache
Amazon
DynamoDB
Amazon
RDS
10 Steps to Using
Amazon ECS, the Sixth
will Blow Your Mind!
Building Cloud
Microservices with
an on-premises
system of record

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon
DynamoDB
Amazon
ElastiCache
Amazon
RDS
Amazon
ElastiCache
Amazon
Redshift
From Monolith, to Less
Than 100 ms with
Amazon ECS and
Kinesis
London Stock Exchange
Group: Building a Data
Warehouse Using AWS
Amazon ES

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Additional resources
Andy Jassy’s re:Invent 2017 keynote: Databases
https://www.youtube.com/watch?v=1IxDLeFQKPk&feature=youtu.be&t=37m47s
Werner Vogel’s blog: A one size fits all database
doesn't fit anyone
https://www.allthingsdistributed.com/2018/06/purpose-built-databases-in-aws.html
https://aws.amazon.com/products/databases/
AWS Databases

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Additional resources
https://www.youtube.com/watch?v=Z3VmcWlIp84
https://www.youtube.com/watch?v=KWOSGVtHWqA
AWS re:Invent 2017: Which Database to
Use When? (DAT310)
AWS Summit SF 2018: Applying AWS Purpose-Built
Database Strategy (SRV307)

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
re:cap
1.Purpose-built/work backwards
2.Common data models/use cases
3.Demo
4.Customer examples
5.Additional resources

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Appendix

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
400,000+ customers using AWS databases
and analytics services