Elastic storage and compute services provide a firm foundation on which to build systems to drive value from data. This presentation discuss how to run analytics pipelines on the AWS Cloud, from data storage with S3 and DynamoDB, to high scale computation with Elastic MapReduce and Cluster Compute instances on EC2.

1. Storage
Tools &
Compute
Support
Databases

2. Compute
Storage
Tools &
Support
Databases

3. Analytics

4. Let’s talk about data

5. Data is valuable

6. Data is plentiful

7. Data is complex

8. Data is in flux

9. Data is fast moving

10. Capturing and managing
data is challenging

11. Lots of data

12. Lots of data,
lots of uses

13. Lots of data,
lots of uses,
lots of users

14. Lots of data,
lots of uses,
lots of users,
lots of locations

15. Cost

16. Force multiplier

17. Additional value

18. Click through
Social graph Log files
Additional value
Audit trails Customer usage
Transcoding

19. Remove constraints

20. Analytics
Data intensive, Tightly
scale out coupled

21. Analytics
Data intensive, Tightly
scale out coupled

22. Hadoop

23. Elastic MapReduce
Managed Hadoop

24. Undifferentiated
heavy lifting

25. S3
Input data

26. S3
Input data
Code Elastic
MapReduce

27. S3
Input data
Code Elastic Name
MapReduce node

28. S3
Input data
Code Elastic Name
MapReduce node
Elastic
cluster

29. S3
Input data
Code Elastic Name
MapReduce node
HDFS
Elastic
cluster

30. S3
Input data
Code Elastic Name
MapReduce node
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

31. S3
Input data
Code Elastic Name Output
MapReduce node S3 + SimpleDB
Queries
HDFS
+ BI
Via JDBC, Pig, Hive
Elastic
cluster

32. S3
Input data
Output
S3 + SimpleDB

47. It’s all just
Hadoop

48. Hive, Pig,
Cascading,
Streaming

49. API driven

50. Data movement

51. Import/Export

52. Large object
support

53. Multipart upload

54. Scale control

55. Resize running
job flows

56. 14 hours
Time remaining: 14 hours

57. 14 hours
Time remaining: 7 hours

58. Time remaining: 3 hours

59. Balance cost and
performance

60. Resize based on
usage patterns

61. Steady state Steady state
Batch processing

62. Spot

63. Integrated with
DynamoDB

64. Integrate

65. Backup and
restore

66. HiveQL

67. Live data in DynamoDB
CREATE EXTERNAL TABLE orders_ddb_2012_01 ( order_id
string, customer_id string, order_date bigint, total
double )
STORED BY
'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler
' TBLPROPERTIES (
"dynamodb.table.name" = "Orders-2012-01",
"dynamodb.column.mapping" = "order_id:Order
ID,customer_id:Customer ID,order_date:Order
Date,total:Total"
);

68. Query DynamoDB
SELECT customer_id, sum(total) spend, count(*)
order_count
FROM orders_ddb_2012_01
WHERE order_date >= unix_timestamp('2012-01-01', 'yyyy-
MM-dd')
AND order_date < unix_timestamp('2012-01-08', 'yyyy-MM-
dd')
GROUP BY customer_id
ORDER BY spend desc
LIMIT 5 ;

69. Archived data in S3
CREATE EXTERNAL TABLE orders_s3_export ( order_id
string, customer_id string, order_date int, total
double )
PARTITIONED BY (year string, month string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY 't'
LOCATION 's3://elastic-mapreduce/samples/ddb-orders' ;

70. Query S3
SELECT year, month, customer_id, sum(total) spend,
count(*) order_count
FROM orders_s3_export
WHERE customer_id = 'c-2cC5fF1bB'
AND month >= 6
AND year = 2011
GROUP BY customer_id, year, month
ORDER by month desc;

71. Export to S3
CREATE EXTERNAL TABLE orders_s3_new_export ( order_id
string, customer_id string, order_date int, total
double )
PARTITIONED BY (year string, month string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION 's3://';
INSERT OVERWRITE TABLE
orders_s3_new_export
PARTITION (year='2012', month='01')
SELECT * from orders_ddb_2012_01;

72. Perfect match

73. Analytics
Data intensive, Tightly
scale out coupled

74. Parallel computation

75. Drug discovery
Financial risk Social media &
analysis gaming
Parallel computation
Manufacturing Transcoding &
& design rendering
Genomics

76. CC1 + GPU
Cluster compute instances

77. CC2

78. 16 Intel Xeon cores Placement groups
CC2
Non-blocking, fully bisectional 10 gig E network

79. 240 TFLOPS
42nd faster supercomputer

80. StarCluster
web.mit.edu/star/cluster

81. CloudFormation
aws.amazon.com/hpc

82. Q&A
matthew@amazon.com