In this presentation you'll learn about the decisions that went into designing and building Amazon DynamoDB, and how it allows you to stay focused on your application while enjoying single digit latencies at any scale. We'll dive deep on how to model data, maintain maximum throughput, and drive analytics against your data, while profiling real world use cases, tips and tricks from customers running on Amazon DynamoDB today. Phil Fitzsimons, Solution Architect, AWS Rob Greig, CTO, Royal Opera House

1. Under the Covers of DynamoDB
Philip Fitzsimons
Manager, Solution Architecture

2. Overview
1. Getting started
2. Customer story: Royal Opera House
3. Data modeling
4. Partitioning
5. Reporting & Analytics

3. Getting started

4. DynamoDB is a managed
NoSQL database service.
Store and retrieve any amount of data.
Serve any level of request traffic.

5. Without the operational burden.

6. Consistent, predictable performance.
Single digit millisecond latency.
Backed on solid-state drives.

7. Flexible data model.
Key/attribute pairs. No schema required.
Easy to create. Easy to adjust.

8. Seamless scalability.
No table size limits. Unlimited storage.
No downtime.

9. Durable.
Consistent, disk only writes.
Replication across data centers and availability zones.

10. Without the operational burden.

11. Focus on your app.

12. Two decisions + three clicks
= ready for use

13. Level of throughput
Primary keys
Two decisions + three clicks
= ready for use

14. Level of throughput
Primary keys
Two decisions + three clicks
= ready for use

15. Provisioned throughput.
Reserve IOPS for reads and writes.
Scale up for down at any time.

16. Pay per capacity unit.
Priced per hour of provisioned throughput.

17. Write throughput.
Size of item x writes per second
$0.0065 for 10 write units

18. Consistent writes.
Atomic increment and decrement.
Optimistic concurrency control: conditional writes.

19. Transactions.
Item level transactions only.
Puts, updates and deletes are ACID.

20. Strong or eventual consistency
Read throughput.

21. Strong or eventual consistency
Read throughput.
Provisioned units = size of item x reads per second
$0.0065 per hour for 50 units

22. Strong or eventual consistency
Read throughput.
Provisioned units = size of item x reads per second
2
$0.0065 per hour for 100 units

23. Strong or eventual consistency
Read throughput.
Same latency expectations.
Mix and match at ‘read time’.

24. Customer story
Royal Opera House

25. Royal Opera House
Rob Greig
CTO
@rob_greig

26. Strategy
Innovation using
open data and
collaborative working
Web 3.0?

27. >600 pages
Inconsistent structure
…and this doesn’t include microsites

28. Manual interlinking
Poor discoverability

29. Coherent and consistent linking
throughout the complete
production lifecycle

30. iPad

31. Open Data

34. Customer
Service

36. DynamoDB

41. Provisioned throughput is
managed by DynamoDB.

42. Data is partitioned and
managed by DynamoDB.

43. Reserved capacity.
Up to 53% for 1 year reservation.
Up to 76% for 3 year reservation.

44. Authentication.
Session based to minimize latency.
Uses the Amazon Security Token Service.
Handled by AWS SDKs.
Integrates with IAM.

45. Monitoring.
CloudWatch metrics:
latency, consumed read and write throughput,
errors and throttling.

46. Libraries, mappers and mocks.
ColdFusion, Django, Erlang, Java, .Net,
Node.js, Perl, PHP, Python, Ruby
http://j.mp/dynamodb-libs

47. Data Modeling

48. date = 2012-05-16-
id = 100 09-00-10 total = 25.00
date = 2012-05-15-
id = 101 15-00-11 total = 35.00
date = 2012-05-16-
id = 101 12-00-10 total = 100.00

49. Table
date = 2012-05-16-
id = 100 09-00-10 total = 25.00
date = 2012-05-15-
id = 101 15-00-11 total = 35.00
date = 2012-05-16-
id = 101 12-00-10 total = 100.00

50. date = 2012-05-16-
id = 100 09-00-10 total = 25.00
Item
date = 2012-05-15-
id = 101 15-00-11 total = 35.00
date = 2012-05-16-
id = 101 12-00-10 total = 100.00

51. date = 2012-05-16-
id = 100 09-00-10 total = 25.00
Attribute
date = 2012-05-15-
id = 101 15-00-11 total = 35.00
date = 2012-05-16-
id = 101 12-00-10 total = 100.00

52. Where is the schema?
Tables do not require a formal schema.
Items are an arbitrarily sized hash.

53. Indexing.
Items are indexed by primary and secondary keys.
Primary keys can be composite.
Secondary keys index on other attributes.

54. ID Date Total
id = 100 date = 2012-05-16-09-00-10 total = 25.00
id = 101 date = 2012-05-15-15-00-11 total = 35.00
id = 101 date = 2012-05-16-12-00-10 total = 100.00
id = 102 date = 2012-03-20-18-23-10 total = 20.00
id = 102 date = 2012-03-20-18-23-10 total = 120.00

55. Hash key
ID Date Total
id = 100 date = 2012-05-16-09-00-10 total = 25.00
id = 101 date = 2012-05-15-15-00-11 total = 35.00
id = 101 date = 2012-05-16-12-00-10 total = 100.00
id = 102 date = 2012-03-20-18-23-10 total = 20.00
id = 102 date = 2012-03-20-18-23-10 total = 120.00

56. Hash key Range key
ID Date Total
Composite primary key
id = 100 date = 2012-05-16-09-00-10 total = 25.00
id = 101 date = 2012-05-15-15-00-11 total = 35.00
id = 101 date = 2012-05-16-12-00-10 total = 100.00
id = 102 date = 2012-03-20-18-23-10 total = 20.00
id = 102 date = 2012-03-20-18-23-10 total = 120.00

57. Hash key Range key Secondary range key
ID Date Total
id = 100 date = 2012-05-16-09-00-10 total = 25.00
id = 101 date = 2012-05-15-15-00-11 total = 35.00
id = 101 date = 2012-05-16-12-00-10 total = 100.00
id = 102 date = 2012-03-20-18-23-10 total = 20.00
id = 102 date = 2012-03-20-18-23-10 total = 120.00

58. Programming DynamoDB.
Small but perfectly formed API.

59. CreateTable PutItem
UpdateTable GetItem
DeleteTable UpdateItem
DescribeTable DeleteItem
ListTables BatchGetItem
Query BatchWriteItem
Scan

60. CreateTable PutItem
UpdateTable GetItem
DeleteTable UpdateItem
DescribeTable DeleteItem
ListTables BatchGetItem
Query BatchWriteItem
Scan

61. CreateTable PutItem
UpdateTable GetItem
DeleteTable UpdateItem
DescribeTable DeleteItem
ListTables BatchGetItem
Query BatchWriteItem
Scan

62. Conditional updates.
PutItem, UpdateItem, DeleteItem can take
optional conditions for operation.
UpdateItem performs atomic increments.

63. One API call, multiple items
BatchGet returns multiple items by key.
BatchWrite performs up to 25 put or delete operations.
Throughput is measured by IO, not API calls.

64. CreateTable PutItem
UpdateTable GetItem
DeleteTable UpdateItem
DescribeTable DeleteItem
ListTables BatchGetItem
Query BatchWriteItem
Scan

65. Query vs Scan
Query for Composite Key queries.
Scan for full table scans, exports.
Both support pages and limits.
Maximum response is 1Mb in size.

66. Query patterns
Retrieve all items by hash key.
Range key conditions:
==, <, >, >=, <=, begins with, between.
Counts. Top and bottom n values.
Paged responses.

67. EXAMPLE 1:
Mapping relationships.

68. Players
user_id = location = joined =
mza Cambridge 2011-07-04
user_id = location = joined =
jeffbarr Seattle 2012-01-20
user_id = location = joined =
werner Worldwide 2011-05-15

69. Players
user_id = location = joined =
mza Cambridge 2011-07-04
user_id = location = joined =
jeffbarr Seattle 2012-01-20
user_id = location = joined =
werner Worldwide 2011-05-15
Scores
user_id = game = score =
mza angry-birds 11,000
user_id = game = score =
mza tetris 1,223,000
user_id = location = score =
werner bejewelled 55,000

70. Players
user_id = location = joined =
mza Cambridge 2011-07-04
user_id = location = joined =
jeffbarr Seattle 2012-01-20
user_id = location = joined =
werner Worldwide 2011-05-15
Scores Leader boards
user_id = game = score = game = score = user_id =
mza angry-birds 11,000 angry-birds 11,000 mza
user_id = game = score = game = score = user_id =
mza tetris 1,223,000 tetris 1,223,000 mza
user_id = location = score = game = score = user_id =
werner bejewelled 55,000 tetris 9,000,000 jeffbarr

71. Players
user_id = location = joined =
mza Cambridge 2011-07-04
user_id = location = joined =
Query for scores
jeffbarr Seattle 2012-01-20 by user
user_id = location = joined =
werner Worldwide 2011-05-15
Scores Leader boards
user_id = game = score = game = score = user_id =
mza angry-birds 11,000 angry-birds 11,000 mza
user_id = game = score = game = score = user_id =
mza tetris 1,223,000 tetris 1,223,000 mza
user_id = location = score = game = score = user_id =
werner bejewelled 55,000 tetris 9,000,000 jeffbarr

72. Players
user_id = location = joined =
mza Cambridge 2011-07-04
user_id = location = joined = High scores by game
jeffbarr Seattle 2012-01-20
user_id = location = joined =
werner Worldwide 2011-05-15
Scores Leader boards
user_id = game = score = game = score = user_id =
mza angry-birds 11,000 angry-birds 11,000 mza
user_id = game = score = game = score = user_id =
mza tetris 1,223,000 tetris 1,223,000 mza
user_id = location = score = game = score = user_id =
werner bejewelled 55,000 tetris 9,000,000 jeffbarr

73. EXAMPLE 2:
Storing large items.

74. Unlimited storage.
Unlimited attributes per item.
Unlimited items per table.
Maximum of 64k per item.

75. Split across items.
message =
message_id = 1 part = 1
<first 64k>
message =
message_id = 1 part = 2
<second 64k>
joined =
message_id = 1 part = 3
<third 64k>

76. Store a pointer to S3.
message =
message_id = 1
http://s3.amazonaws.com...
message =
message_id = 2
http://s3.amazonaws.com...
message =
message_id = 3
http://s3.amazonaws.com...

77. EXAMPLE 3:
Time series data

78. April Hot and cold tables.
event_id = timestamp = key =
1000 2013-04-16-09-59-01 value
event_id = timestamp = key =
1001 2013-04-16-09-59-02 value
event_id = timestamp = key =
1002 2013-04-16-09-59-02 value
March
event_id = timestamp = key =
1000 2013-03-01-09-59-01 value
event_id = timestamp = key =
1001 2013-03-01-09-59-02 value
event_id = timestamp = key =
1002 2013-03-01-09-59-02 value

79. December January February March April

80. Archive data.
Move old data to S3: lower cost.
Still available for analytics.
Run queries across hot and cold data
with Elastic MapReduce.

81. Partitioning

82. Uniform workload.
Data stored across multiple partitions.
Data is primarily distributed by primary key.
Provisioned throughput is divided evenly across partitions.

83. To achieve and maintain full
provisioned throughput, spread
workload evenly across hash keys.

84. Non-Uniform workload.
Might be throttled, even at high levels of throughput.

85. BEST PRACTICE 1:
Distinct values for hash keys.
Hash key elements should have a
high number of distinct values.

86. Lots of users with unique user_id.
Workload well distributed across hash key.
user_id = first_name = last_name =
mza Matt Wood
user_id = first_name = last_name =
jeffbarr Jeff Barr
user_id = first_name = last_name =
werner Werner Vogels
user_id = first_name = last_name =
simone Simone Brunozzi
... ... ...

87. BEST PRACTICE 2:
Avoid limited hash key values.
Hash key elements should have a
high number of distinct values.

88. Small number of status codes.
Unevenly, non-uniform workload.
status = date =
200 2012-04-01-00-00-01
status = date =
404 2012-04-01-00-00-01
status date =
404 2012-04-01-00-00-01
status = date =
404 2012-04-01-00-00-01

89. BEST PRACTICE 3:
Model for even distribution.
Access by hash key value should be evenly
distributed across the dataset.

90. Large number of devices.
Small number which are much more popular than others.
Workload unevenly distributed.
mobile_id = access_date =
100 2012-04-01-00-00-01
mobile_id = access_date =
100 2012-04-01-00-00-02
mobile_id = access_date =
100 2012-04-01-00-00-03
mobile_id = access_date =
100 2012-04-01-00-00-04
... ...

91. Sample access pattern.
Workload randomized by hash key.
mobile_id = access_date =
100.1 2012-04-01-00-00-01
mobile_id = access_date =
100.2 2012-04-01-00-00-02
mobile_id = access_date =
100.3 2012-04-01-00-00-03
mobile_id = access_date =
100.4 2012-04-01-00-00-04
... ...

92. Reporting & Analytics

93. Seamless scale.
Scalable methods for data processing.
Scalable methods for backup/restore.

94. Amazon Elastic MapReduce.
Managed Hadoop service for
data-intensive workflows.
aws.amazon.com/emr

95. create external table items_db
(id string, votes bigint, views bigint) stored by
'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler'
tblproperties
("dynamodb.table.name" = "items",
"dynamodb.column.mapping" =
"id:id,votes:votes,views:views");

96. select id, likes, views
from items_db
order by views desc;

97. Summary
1. Getting started
2. Customer story: Royal Opera House
3. Data modeling
4. Partitioning
5. Reporting & Analytics

98. Free tier.

99. aws.amazon.com/dynamodb

100. Thank you!
philipfs@amazon.co.uk