Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select & Amazon Glacier Select - STG313 - re:Invent 2017

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Big Data Breakthroughs
Process and Query Data In Place with
Amazon S3 & Glacier
R a h u l B h a r t i a R a s h i m G u p t a
P r i n c i p a l P r o d u c t M a n a g e r P r i n c i p a l P r o d u c t M a n a g e r
A m a z o n S 3 A m a z o n G l a c i e r
S T G 3 1 3
N o v e m b e r 2 9 , 2 0 1 7

What to expect from the session
1. Introduction to Amazon S3 Select and Amazon Glacier Select
2. Dive into Use-cases
3. Key features and demos

Unmatched durability,
availability, and scalability
Best security, compliance, and audit
capability
Object-level control
at any scale
Business insight into
your data
Twice as many partner
integrations
Most ways to bring
data in
Building a data lake with Amazon S3

Amazon Athena Amazon Redshift
Spectrum
Amazon EMR AWS Glue
With a wide variety of in-place tools…
Amazon S3

Today: Compute scales based…
on object size instead of the amount of data you want to process
DATA COMPUTE

Today: All of these tools…
retrieve a lot of data they don’t need and
do the heavy lifting

Today: You need to restore…
entire object from Amazon Glacier to Amazon S3
and then use it.
Amazon
S3
Amazon
Glacier

Introducing…
Amazon S3 Select and Amazon Glacier Select
Select subset of data from an object based on a SQL expression

Available as an API—no
infrastructure or administration
Faster performance as compared
to doing it yourself
Pay as you go. The less you
retrieve the more you save.
Simple, faster, and cheaper!

Amazon S3 Select

Simple to use
Standard SQL expression
Familiar
Work and scales like GET requests
Integrated
AWS SDK and Presto
(others coming soon)
Amazon S3 Select
Select contents from object instead of retrieving the object

Amazon S3 Select
Output
Format: delimited text (CSV,
TSV), JSON …
Clauses Data types Operators Functions
Select String Conditional String
From Integer, Float, Decimal Math Cast
Where Timestamp Logical Math
Boolean String (Like, ||) Aggregate
Input
TSV), JSON …
Compression: GZIP …

Amazon S3 Select: Simple pattern matches
…get-object …object… | awk -F ’{ if($4=="x") print $1}’
...select-object …object… ‘SELECT o._1 WHERE o._4 == “x”…’

Amazon S3 Select: Serverless applications
Amazon
S3
AWS
Lambda
Amazon
SNS
S3
Select

200 seconds and 11.2 cents
# Download and process all keys
for key in src_keys:
response = s3_client.get_object(Bucket=src_bucket,
Key=key)
contents = response['Body'].read()
for line in contents.split('n')[:-1]:
line_count +=1
try:
data = line.split(',')
srcIp = data[0][:8]
….
95 seconds and costs 2.8 cents
# Select IP Address and Keys
for key in src_keys:
response = s3_client.select_object_content
(Bucket=src_bucket, Key=key, expression =
SELECT SUBSTR(obj._1, 1, 8), obj._2 FROM
s3object as obj)
contents = response['Body'].read()
for line in contents:
line_count +=1
try:
….
A f t e rB e f o r e
Amazon S3 Select: Serverless MapReduce
2X Faster at 1/5 of the cost

Up to 400% Faster
Up to 80% Cheaper
Amazon S3 Select: Accelerating Big Data
Amazon S3
Before:
Amazon S3
S3 Select
After:

DEMO: Amazon S3 Select with Presto
Works with your existing Hive Metastore
Automatically converts predicates into S3 Select requests
Amazon S3
S3 Select

A f t e rB e f o r e
Amazon S3 Select: Accelerating big data
5X Faster with 1/40 of the CPU

Amazon Athena Amazon EMR Amazon Redshift
Spectrum
Amazon S3 Select: Will be supported by…

Amazon S3 Select: Preview starts today
• Formats: CSV, JSON
• Compression: GZIP
• Encryption: None
• Encoding: UTF-8
• Integration: AWS SDK for Java and Python and Presto Connector
• Availability: Northern Virginia, Ohio, Oregon, Dublin, and Singapore
Apply at: https://pages.awscloud.com/amazon-s3-select-preview.html

Amazon Glacier Select

Amazon Glacier
• Extremely low-cost archive storage service, starting at $0.004 GB/mo
• Allows you to retrieve data from minutes to hours
• 99.999999999% of durability (five to six orders of magnitude higher
than two copies of tape)
• All data is encrypted at rest
• Features: compliance, data management, cost management, audit
logging

Choice of storage classes
Active data Archive dataInfrequently accessed data
Milliseconds Minutes to hoursMilliseconds
From 2.1¢-GB/mo. 0.4¢-GB/mo.1.25¢-GB/mo.
Amazon S3 Standard
Amazon S3 Standard–
Infrequent Access
Amazon Glacier

Retrievals using Amazon Glacier
Expedited Standard Bulk
Data Access Time 1 - 5 minutes 3 - 5 hours 5 - 12 hours
Data Retrievals $0.03 per GB $0.01 per GB $0.0025 per GB
Retrieval Requests $0.01 per request $0.05 per 1,000 requests $0.025 per 1,000 requests
• Three flexible and powerful retrieval options to access any of your Amazon
Glacier data

Simple SQL Expression
SELECT and WHERE
Familiar semantics
Work and scales like RESTORE requests
Integrated
AWS SDK and CLI
Select relevant contents from object instead of restoring entire object

Using Glacier API
Data directly uploaded to Amazon
Glacier
Using S3 API
For data that is lifecycled to Amazon Glacier
from S3
How to use Amazon Glacier Select?
Two ways to use Amazon Glacier Select

Object id
How to use Amazon Glacier Select?
Current restore-object API arguments
Tier
SQL query
New (optional) restore-object API arguments to use Glacier Select
Output S3 location SNS topic

Using Glacier Select
…restore-object …object… | …get-object … object …. | awk -F ’{ if($4==“id") print $1}’
...restore-object …object… ‘SELECT o._1 WHERE o._4 == “id”…’ | …get-object … object ….

How Glacier Select works
App Amazon Glacier Amazon S3Glacier-Select (ArchiveId, SQL, Tier,
S3 bucket to write output)
200 OK
Read data and
Perform filtering
Write output to S3
Notify app using SNS, that output ready

Billing Inquiries Auditing/Compliance Building Training Models
Amazon Glacier Select use cases

Input
TSV, PSV, etc.)
Encryption: SSE-KMS, SSE-S3
Output
TSV, PSV, etc.)
Clauses Data types Operators Functions
Select String Conditional String
From Integer, Float, Decimal Math Cast
Where Timestamp Logical
Boolean String (Like, ||)

Demo: Glacier Select
App
Glacier Select
Amazon Glacier

Amazon Glacier Select pricing
• Data scanned: per GB bill for size of data read
• Data returned: per GB charge based on amount of output data returned to S3
• Request cost: charge for each Glacier Select request

Amazon Athena
Coming in 2018: Amazon Athena with
Amazon Glacier
• Query data in Amazon Glacier directly from Amazon Athena
• Enables customers to use Amazon Glacier as a data source for their SQL queries in Amazon
Athena

Amazon Glacier Select: GA starts today
• Formats: CSV, Any delimiter separated file
• Encryption: SSE- KMS, SSE-S3
• Encoding: UTF-8
• Integration: AWS SDK, CLI, Athena integration (expected 2018)
• Availability: All commercial regions where Amazon Glacier is launched

Summary

Simple Faster Cheaper
Now: Your data lake on AWS
Amazon
Glacier
Amazon
S3
Amazon Redshift
Spectrum
Amazon Athena Amazon EMR
AWS
Lambda
ISVs and Custom
Applications
SELECT

STG303 – Deep Dive on Amazon Glacier – Thurs, 1:45 PM
STG312 – Best Practices for Building a Data Lake in Amazon S3 & Amazon
Glacier – Thurs, 3:15 PM
Learn more…

Q&A
Amazon S3 Amazon Glacier

Thank you!

Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select & Amazon Glacier Select - STG313 - re:Invent 2017

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select & Amazon Glacier Select - STG313 - re:Invent 2017

Semelhante a Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select & Amazon Glacier Select - STG313 - re:Invent 2017 (20)

Mais de Amazon Web Services

Mais de Amazon Web Services (20)

Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select & Amazon Glacier Select - STG313 - re:Invent 2017