SlideShare uma empresa Scribd logo
1 de 44
Baixar para ler offline
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Susan Chan
Rahul Bhartia
December 13, 2017
New Features in Amazon S3
and Amazon Glacier
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What’s New on Amazon S3 and Amazon Glacier?
• Amazon S3 Select and Amazon Glacier Select
• S3 Inventory Enhancements
• S3 Default Encryption
• S3 Bucket Permissions Check
• S3 Cross-Region Replication Enhancements
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon S3 Select and
Amazon Glacier Select
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon S3 Select and Amazon Glacier Select
Simple to use
Standard SQL expression
Familiar
Work and scales like GET requests or
RESTORE requests
Integrated
AWS SDK
Select contents from object instead of retrieving or restoring the
object
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon S3 Select
Output
Format: delimited text (CSV,
TSV), JSON …
Clauses Data types Operators Functions
Select String Conditional String
From Integer, Float, Decimal Math Cast
Where Timestamp Logical Math
Boolean String (Like, ||) Aggregate
Input
Format: delimited text (CSV,
TSV), JSON …
Compression: GZIP …
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Glacier Select
Input
Format: delimited text (CSV,
TSV, PSV, etc.)
Encryption: SSE-KMS, SSE-S3
Output
Format: delimited text (CSV,
TSV, PSV, etc.)
Clauses Data types Operators Functions
Select String Conditional String
From Integer, Float, Decimal Math Cast
Where Timestamp Logical
Boolean String (Like, ||)
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Simple pattern matches
…get-object …object… | awk -F ’{ if($4=="x") print $1}’
...select-object …object… ‘SELECT o._1 WHERE o._4 == “x”…’
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Using Glacier API
Data directly uploaded to Amazon Glacier
Using S3 Restore API
For data that is lifecycle
to Amazon Glacier from Amazon S3
How to use Glacier Select?
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
With Amazon Glacier Select
App Amazon Glacier Amazon S3Glacier-Select (ArchiveId, SQL, Tier,
S3 bucket to write output)
200 OK
Read data and
Perform filtering
Write output to S3
Notify app using SNS, that output ready
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless applications
Amazon
S3
AWS
Lambda
Amazon
SNS
S3
Select
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
200 seconds and 11.2 cents
# Download and process all keys
for key in src_keys:
response = s3_client.get_object(Bucket=src_bucket,
Key=key)
contents = response['Body'].read()
for line in contents.split('n')[:-1]:
line_count +=1
try:
data = line.split(',')
srcIp = data[0][:8]
….
95 seconds and costs 2.8 cents
# Select IP Address and Keys
for key in src_keys:
response = s3_client.select_object_content
(Bucket=src_bucket, Key=key, expression =
SELECT SUBSTR(obj._1, 1, 8), obj._2 FROM s3object
as obj)
contents = response['Body'].read()
for line in contents:
line_count +=1
try:
….
AfterBefore
Amazon S3 Select: Serverless MapReduce
2X Faster at 1/5 of the cost
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Up to 400% Faster
Up to 80% Cheaper
Accelerating Big Data
Amazon S3
Before:
Amazon S3
S3 Select
After:
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon S3 Select with Presto
Works with your existing Hive Metastore
Automatically converts predicates into S3 Select requests
Amazon S3
S3 Select
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AfterBefore
Amazon S3 Select: Accelerating big data
5X Faster with 1/40 of the CPU
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Athena Amazon EMR Amazon Redshift
Spectrum
Amazon S3 Select: Will be supported by…
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon S3 Select available in Preview
• Formats: CSV, JSON
• Compression: GZIP
• Encryption: None
• Encoding: UTF-8
• Integration: AWS SDK for Java and Python and Presto Connector
• Availability: Northern Virginia, Ohio, Oregon, Dublin, and Singapore
Apply at: https://pages.awscloud.com/amazon-s3-select-preview.html
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Glacier Select: Generally available
• Formats: CSV, Any delimiter separated file
• Encryption: SSE- KMS, SSE-S3
• Encoding: UTF-8
• Integration: AWS SDK, CLI, Athena integration (expected 2018)
• Availability: All commercial regions where Amazon Glacier is launched
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S3 Inventory
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S3 Inventory
Save time Daily or Weekly delivery Delivery to S3 bucket
• Same set of metadata as the LIST API
• Can add size, last modified date, storage class, etag, or replication status
Trigger business workflows and applications such as secondary index,
garbage collection, data auditing, and offline analytics
Delivery notification
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S3 Inventory
Object level
Encryption Status
CSV or ORC
output format
Query with Athena,
Redshift Spectrum or
any Hive tools
Encrypt inventory
with SSE-S3 or
SSE-KMS
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
S3 Inventory
 Bucket name
 Key name
 Version ID
 IsLatest
 Size
 Last modified date
 ETag
 Storage class
 Multipart upload flag
 Delete marker
 Replication status
 Encryption Status
destination-bucket
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
"The rich information generated by AWS
through the new object encryption status in S3
Inventory has been instrumental in helping us to
automate and streamline daily reporting on
compliance controls."
– John Andrukonis
Chief Architect, Capital One.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Query S3 Inventory with Amazon Athena
CREATE EXTERNAL TABLE my_inventory_table(
`bucket` string,
key string,
version_id string,
is_latest boolean,
is_delete_marker boolean,
size bigint,
last_modified_date timestamp,
e_tag string,
storage_class string,
is_multipart_uploaded boolean,
replication_status string,
encryption_status string)
PARTITIONED BY (dt string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION 's3://bucketname/inventory/output_destination/hive';
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Query S3 Inventory with Amazon Athena
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Visualize in Amazon QuickSight
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bucket Permissions Check
Default Encryption
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS Trusted
Advisor
Bucket Permissions Check
S3 Console
Object Encryption status
S3 Inventory
Security Inspection
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bucket Permissions Check
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Default Encryption
Automatically encrypts all objects written to your
Amazon S3 bucket
• Choose SSE-S3 or SSE-KMS
• Makes it easy to satisfy
compliance needs
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cross-Region Replication
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cross-Region Replication (CRR)
Use cases:
What is CRR?
Automated, fast, and reliable asynchronous
replication of data across AWS regions
Compliance Lower latency Security
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How does CRR work?
Bucket B
Region A Region B
• All uploads into source
bucket are replicated
• Entire bucket or prefix
• Choose any AWS region as
your target region
• Secure transfer via SSL
• Exact replicas including
object ACL and tags
Bucket A
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CRR across account
Region A
Region B
Why?
• Additional protection on your
back up to prevent malicious
delete
Ownership overwrite
• Replica are owned by
destination bucket
• Maintain 2 distinct and
independent stacks of
ownership
Primary
Account
Primary
Account
Secondary
Account
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Choose any AWS
region as target
Lifecycle policy
Support SSE-KMS
Encrypted objects
Ownership overwrite
for cross-account CRR
Choose any S3 Storage
Class as target
More with Cross-Region Replication
Bi-directional replication
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Getting Started - CRR
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
my_source_bucket
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Destination
KMS master key
Ownership overwrite
My_destination_bucket
arn:aws:kms:us-east-2:123456789:/abc12345t234-1234-5678-a12b-a12b34cd5678
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cross Account CRR-Destination Set Up
Destination bucket
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cross Account CRR –Destination Set Up
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Recap
• S3 Select and Glacier Select
• S3 Inventory Enhancements
• S3 Default Encryption
• S3 Bucket Permissions Check
• S3 Cross-Region Replication Enhancements
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Learn more
• S3 Select and Glacier Select https://youtu.be/p-JkncBZcc4
• S3 Select preview https://pages.awscloud.com/amazon-s3-select-
preview.html
• Amazon Macie: AWS Security Service Powered by Machine Learning
https://youtu.be/LCjX2rsQ2wA
• Deep Dive on Storage Management https://youtu.be/SUWqDOnXeDw
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!

Mais conteúdo relacionado

Mais procurados

Armazenamento em Amazon Web Service para Setor Publico
 Armazenamento em Amazon Web Service para Setor Publico  Armazenamento em Amazon Web Service para Setor Publico
Armazenamento em Amazon Web Service para Setor Publico Amazon Web Services LATAM
 
AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, an...
AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, an...AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, an...
AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, an...Amazon Web Services
 
SRV334-Making Things Right with AWS Config Rules and AWS Lambda
SRV334-Making Things Right with AWS Config Rules and AWS LambdaSRV334-Making Things Right with AWS Config Rules and AWS Lambda
SRV334-Making Things Right with AWS Config Rules and AWS LambdaAmazon Web Services
 
Deep Dive: Building Hybrid Cloud Storage Architectures with AWS Storage Gatew...
Deep Dive: Building Hybrid Cloud Storage Architectures with AWS Storage Gatew...Deep Dive: Building Hybrid Cloud Storage Architectures with AWS Storage Gatew...
Deep Dive: Building Hybrid Cloud Storage Architectures with AWS Storage Gatew...Amazon Web Services
 
Set it and Forget it: Auto Scaling Target Tracking Policies - AWS Online Tech...
Set it and Forget it: Auto Scaling Target Tracking Policies - AWS Online Tech...Set it and Forget it: Auto Scaling Target Tracking Policies - AWS Online Tech...
Set it and Forget it: Auto Scaling Target Tracking Policies - AWS Online Tech...Amazon Web Services
 
Building High Availability Apps on Lightsail: Load Balancing and Block Storag...
Building High Availability Apps on Lightsail: Load Balancing and Block Storag...Building High Availability Apps on Lightsail: Load Balancing and Block Storag...
Building High Availability Apps on Lightsail: Load Balancing and Block Storag...Amazon Web Services
 
STG311_Deep Dive on Amazon S3 & Amazon Glacier Storage Management
STG311_Deep Dive on Amazon S3 & Amazon Glacier Storage ManagementSTG311_Deep Dive on Amazon S3 & Amazon Glacier Storage Management
STG311_Deep Dive on Amazon S3 & Amazon Glacier Storage ManagementAmazon Web Services
 
STG206_Big Data Data Lakes and Data Oceans
STG206_Big Data Data Lakes and Data OceansSTG206_Big Data Data Lakes and Data Oceans
STG206_Big Data Data Lakes and Data OceansAmazon Web Services
 
Optimising Cost and Efficiency on AWS
Optimising Cost and Efficiency on AWSOptimising Cost and Efficiency on AWS
Optimising Cost and Efficiency on AWSAmazon Web Services
 
Deep Dive: Hybrid Cloud Storage with AWS Storage Gateway - AWS Online Tech Talks
Deep Dive: Hybrid Cloud Storage with AWS Storage Gateway - AWS Online Tech TalksDeep Dive: Hybrid Cloud Storage with AWS Storage Gateway - AWS Online Tech Talks
Deep Dive: Hybrid Cloud Storage with AWS Storage Gateway - AWS Online Tech TalksAmazon Web Services
 
Adding Search to Relational Databases
Adding Search to Relational DatabasesAdding Search to Relational Databases
Adding Search to Relational DatabasesAmazon Web Services
 
Hybrid Cloud Storage for Recovery & Migration with AWS Storage Gateway (STG30...
Hybrid Cloud Storage for Recovery & Migration with AWS Storage Gateway (STG30...Hybrid Cloud Storage for Recovery & Migration with AWS Storage Gateway (STG30...
Hybrid Cloud Storage for Recovery & Migration with AWS Storage Gateway (STG30...Amazon Web Services
 
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPTHow TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPTAmazon Web Services
 
Data Storage for the Long Haul: Compliance and Archive
Data Storage for the Long Haul: Compliance and ArchiveData Storage for the Long Haul: Compliance and Archive
Data Storage for the Long Haul: Compliance and ArchiveAmazon Web Services
 
Adding Search to Amazon DynamoDB
Adding Search to Amazon DynamoDBAdding Search to Amazon DynamoDB
Adding Search to Amazon DynamoDBAmazon Web Services
 
Building a Strong Foundation with AWS Storage Services
Building a Strong Foundation with AWS Storage ServicesBuilding a Strong Foundation with AWS Storage Services
Building a Strong Foundation with AWS Storage ServicesAmazon Web Services
 
DEV337_Deploy a Data Lake with AWS CloudFormation
DEV337_Deploy a Data Lake with AWS CloudFormationDEV337_Deploy a Data Lake with AWS CloudFormation
DEV337_Deploy a Data Lake with AWS CloudFormationAmazon Web Services
 
AWS Data Lifecycle and Storage Management Demo
AWS Data Lifecycle and Storage Management DemoAWS Data Lifecycle and Storage Management Demo
AWS Data Lifecycle and Storage Management DemoAmazon Web Services
 

Mais procurados (20)

Armazenamento em Amazon Web Service para Setor Publico
 Armazenamento em Amazon Web Service para Setor Publico  Armazenamento em Amazon Web Service para Setor Publico
Armazenamento em Amazon Web Service para Setor Publico
 
AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, an...
AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, an...AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, an...
AWS Data Transfer Services - AWS Gateway, AWS Snowball, AWS Snowball Edge, an...
 
AWS for Backup and Recovery
AWS for Backup and RecoveryAWS for Backup and Recovery
AWS for Backup and Recovery
 
SRV334-Making Things Right with AWS Config Rules and AWS Lambda
SRV334-Making Things Right with AWS Config Rules and AWS LambdaSRV334-Making Things Right with AWS Config Rules and AWS Lambda
SRV334-Making Things Right with AWS Config Rules and AWS Lambda
 
Deep Dive: Building Hybrid Cloud Storage Architectures with AWS Storage Gatew...
Deep Dive: Building Hybrid Cloud Storage Architectures with AWS Storage Gatew...Deep Dive: Building Hybrid Cloud Storage Architectures with AWS Storage Gatew...
Deep Dive: Building Hybrid Cloud Storage Architectures with AWS Storage Gatew...
 
Set it and Forget it: Auto Scaling Target Tracking Policies - AWS Online Tech...
Set it and Forget it: Auto Scaling Target Tracking Policies - AWS Online Tech...Set it and Forget it: Auto Scaling Target Tracking Policies - AWS Online Tech...
Set it and Forget it: Auto Scaling Target Tracking Policies - AWS Online Tech...
 
Building High Availability Apps on Lightsail: Load Balancing and Block Storag...
Building High Availability Apps on Lightsail: Load Balancing and Block Storag...Building High Availability Apps on Lightsail: Load Balancing and Block Storag...
Building High Availability Apps on Lightsail: Load Balancing and Block Storag...
 
STG311_Deep Dive on Amazon S3 & Amazon Glacier Storage Management
STG311_Deep Dive on Amazon S3 & Amazon Glacier Storage ManagementSTG311_Deep Dive on Amazon S3 & Amazon Glacier Storage Management
STG311_Deep Dive on Amazon S3 & Amazon Glacier Storage Management
 
STG206_Big Data Data Lakes and Data Oceans
STG206_Big Data Data Lakes and Data OceansSTG206_Big Data Data Lakes and Data Oceans
STG206_Big Data Data Lakes and Data Oceans
 
Optimising Cost and Efficiency on AWS
Optimising Cost and Efficiency on AWSOptimising Cost and Efficiency on AWS
Optimising Cost and Efficiency on AWS
 
Deep Dive: Hybrid Cloud Storage with AWS Storage Gateway - AWS Online Tech Talks
Deep Dive: Hybrid Cloud Storage with AWS Storage Gateway - AWS Online Tech TalksDeep Dive: Hybrid Cloud Storage with AWS Storage Gateway - AWS Online Tech Talks
Deep Dive: Hybrid Cloud Storage with AWS Storage Gateway - AWS Online Tech Talks
 
Adding Search to Relational Databases
Adding Search to Relational DatabasesAdding Search to Relational Databases
Adding Search to Relational Databases
 
Hybrid Cloud Storage for Recovery & Migration with AWS Storage Gateway (STG30...
Hybrid Cloud Storage for Recovery & Migration with AWS Storage Gateway (STG30...Hybrid Cloud Storage for Recovery & Migration with AWS Storage Gateway (STG30...
Hybrid Cloud Storage for Recovery & Migration with AWS Storage Gateway (STG30...
 
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPTHow TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
 
Data Storage for the Long Haul: Compliance and Archive
Data Storage for the Long Haul: Compliance and ArchiveData Storage for the Long Haul: Compliance and Archive
Data Storage for the Long Haul: Compliance and Archive
 
Adding Search to Amazon DynamoDB
Adding Search to Amazon DynamoDBAdding Search to Amazon DynamoDB
Adding Search to Amazon DynamoDB
 
Building a Strong Foundation with AWS Storage Services
Building a Strong Foundation with AWS Storage ServicesBuilding a Strong Foundation with AWS Storage Services
Building a Strong Foundation with AWS Storage Services
 
DEV337_Deploy a Data Lake with AWS CloudFormation
DEV337_Deploy a Data Lake with AWS CloudFormationDEV337_Deploy a Data Lake with AWS CloudFormation
DEV337_Deploy a Data Lake with AWS CloudFormation
 
Amazon RDS_Deep Dive - SRV310
Amazon RDS_Deep Dive - SRV310 Amazon RDS_Deep Dive - SRV310
Amazon RDS_Deep Dive - SRV310
 
AWS Data Lifecycle and Storage Management Demo
AWS Data Lifecycle and Storage Management DemoAWS Data Lifecycle and Storage Management Demo
AWS Data Lifecycle and Storage Management Demo
 

Semelhante a Deep Dive on New Features in Amazon S3 & Glacier - AWS Online Tech Talks

Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select...
Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select...Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select...
Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select...Amazon Web Services
 
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...Amazon Web Services
 
Serverless Architectural Patterns
Serverless Architectural PatternsServerless Architectural Patterns
Serverless Architectural PatternsAmazon Web Services
 
Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdf
Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdfBuilding+your+Data+Project+on+AWS+-+Luke+Anderson.pdf
Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdfsaidbilgen
 
Building Data Lakes That Cost Less and Deliver Results Faster - AWS Online Te...
Building Data Lakes That Cost Less and Deliver Results Faster - AWS Online Te...Building Data Lakes That Cost Less and Deliver Results Faster - AWS Online Te...
Building Data Lakes That Cost Less and Deliver Results Faster - AWS Online Te...Amazon Web Services
 
STG302_Best Practices for Amazon S3
STG302_Best Practices for Amazon S3STG302_Best Practices for Amazon S3
STG302_Best Practices for Amazon S3Amazon Web Services
 
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...Amazon Web Services
 
Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS O...
Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS O...Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS O...
Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS O...Amazon Web Services
 
Storage Data Management: Tools and Templates to Seamlessly Automate and Optim...
Storage Data Management: Tools and Templates to Seamlessly Automate and Optim...Storage Data Management: Tools and Templates to Seamlessly Automate and Optim...
Storage Data Management: Tools and Templates to Seamlessly Automate and Optim...Amazon Web Services
 
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017Amazon Web Services
 
China Gaming Industry Experience and Architecture Sharing
China Gaming Industry Experience and Architecture SharingChina Gaming Industry Experience and Architecture Sharing
China Gaming Industry Experience and Architecture SharingAmazon Web Services
 
中國AWS遊戲業經驗和架構分享
中國AWS遊戲業經驗和架構分享中國AWS遊戲業經驗和架構分享
中國AWS遊戲業經驗和架構分享Amazon Web Services
 
AWSomeday Brussels Technical Track
AWSomeday Brussels Technical TrackAWSomeday Brussels Technical Track
AWSomeday Brussels Technical TrackAmazon Web Services
 
Deep Dive on Amazon S3 & Amazon Glacier Storage Management - STG311 - re:Inve...
Deep Dive on Amazon S3 & Amazon Glacier Storage Management - STG311 - re:Inve...Deep Dive on Amazon S3 & Amazon Glacier Storage Management - STG311 - re:Inve...
Deep Dive on Amazon S3 & Amazon Glacier Storage Management - STG311 - re:Inve...Amazon Web Services
 
Serverless Architectural Patterns 
and Best Practices - Madhu Shekar - AWS
Serverless Architectural Patterns 
and Best Practices - Madhu Shekar - AWSServerless Architectural Patterns 
and Best Practices - Madhu Shekar - AWS
Serverless Architectural Patterns 
and Best Practices - Madhu Shekar - AWSCodeOps Technologies LLP
 
SRV208 S3 One Zone-IA and S3 Select GA
SRV208 S3 One Zone-IA and S3 Select GASRV208 S3 One Zone-IA and S3 Select GA
SRV208 S3 One Zone-IA and S3 Select GAAmazon Web Services
 
Data Lake Implementation: Processing and Querying Data in Place (STG204-R1) -...
Data Lake Implementation: Processing and Querying Data in Place (STG204-R1) -...Data Lake Implementation: Processing and Querying Data in Place (STG204-R1) -...
Data Lake Implementation: Processing and Querying Data in Place (STG204-R1) -...Amazon Web Services
 
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...Amazon Web Services
 

Semelhante a Deep Dive on New Features in Amazon S3 & Glacier - AWS Online Tech Talks (20)

Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select...
Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select...Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select...
Big Data Breakthroughs: Process and Query Data In Place with Amazon S3 Select...
 
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
Best Practices for Building a Data Lake in Amazon S3 and Amazon Glacier, with...
 
Serverless Architectural Patterns
Serverless Architectural PatternsServerless Architectural Patterns
Serverless Architectural Patterns
 
Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdf
Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdfBuilding+your+Data+Project+on+AWS+-+Luke+Anderson.pdf
Building+your+Data+Project+on+AWS+-+Luke+Anderson.pdf
 
Building Data Lakes That Cost Less and Deliver Results Faster - AWS Online Te...
Building Data Lakes That Cost Less and Deliver Results Faster - AWS Online Te...Building Data Lakes That Cost Less and Deliver Results Faster - AWS Online Te...
Building Data Lakes That Cost Less and Deliver Results Faster - AWS Online Te...
 
STG302_Best Practices for Amazon S3
STG302_Best Practices for Amazon S3STG302_Best Practices for Amazon S3
STG302_Best Practices for Amazon S3
 
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
 
Storage Data Management
Storage Data ManagementStorage Data Management
Storage Data Management
 
Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS O...
Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS O...Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS O...
Transforming Data Lakes with Amazon S3 Select & Amazon Glacier Select - AWS O...
 
Storage Data Management: Tools and Templates to Seamlessly Automate and Optim...
Storage Data Management: Tools and Templates to Seamlessly Automate and Optim...Storage Data Management: Tools and Templates to Seamlessly Automate and Optim...
Storage Data Management: Tools and Templates to Seamlessly Automate and Optim...
 
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017
Migrating Large Scale Data Sets to the Cloud - STG204 - re:Invent 2017
 
China Gaming Industry Experience and Architecture Sharing
China Gaming Industry Experience and Architecture SharingChina Gaming Industry Experience and Architecture Sharing
China Gaming Industry Experience and Architecture Sharing
 
中國AWS遊戲業經驗和架構分享
中國AWS遊戲業經驗和架構分享中國AWS遊戲業經驗和架構分享
中國AWS遊戲業經驗和架構分享
 
AWSomeday Brussels Technical Track
AWSomeday Brussels Technical TrackAWSomeday Brussels Technical Track
AWSomeday Brussels Technical Track
 
Deep Dive on Amazon S3 & Amazon Glacier Storage Management - STG311 - re:Inve...
Deep Dive on Amazon S3 & Amazon Glacier Storage Management - STG311 - re:Inve...Deep Dive on Amazon S3 & Amazon Glacier Storage Management - STG311 - re:Inve...
Deep Dive on Amazon S3 & Amazon Glacier Storage Management - STG311 - re:Inve...
 
Serverless Architectural Patterns 
and Best Practices - Madhu Shekar - AWS
Serverless Architectural Patterns 
and Best Practices - Madhu Shekar - AWSServerless Architectural Patterns 
and Best Practices - Madhu Shekar - AWS
Serverless Architectural Patterns 
and Best Practices - Madhu Shekar - AWS
 
SRV208 S3 One Zone-IA and S3 Select GA
SRV208 S3 One Zone-IA and S3 Select GASRV208 S3 One Zone-IA and S3 Select GA
SRV208 S3 One Zone-IA and S3 Select GA
 
Data Lake Implementation: Processing and Querying Data in Place (STG204-R1) -...
Data Lake Implementation: Processing and Querying Data in Place (STG204-R1) -...Data Lake Implementation: Processing and Querying Data in Place (STG204-R1) -...
Data Lake Implementation: Processing and Querying Data in Place (STG204-R1) -...
 
Hybrid and Edge Architectures
Hybrid and Edge ArchitecturesHybrid and Edge Architectures
Hybrid and Edge Architectures
 
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...
 

Mais de Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateAmazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSAmazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 

Mais de Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Deep Dive on New Features in Amazon S3 & Glacier - AWS Online Tech Talks

  • 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Susan Chan Rahul Bhartia December 13, 2017 New Features in Amazon S3 and Amazon Glacier
  • 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What’s New on Amazon S3 and Amazon Glacier? • Amazon S3 Select and Amazon Glacier Select • S3 Inventory Enhancements • S3 Default Encryption • S3 Bucket Permissions Check • S3 Cross-Region Replication Enhancements
  • 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon S3 Select and Amazon Glacier Select
  • 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon S3 Select and Amazon Glacier Select Simple to use Standard SQL expression Familiar Work and scales like GET requests or RESTORE requests Integrated AWS SDK Select contents from object instead of retrieving or restoring the object
  • 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon S3 Select Output Format: delimited text (CSV, TSV), JSON … Clauses Data types Operators Functions Select String Conditional String From Integer, Float, Decimal Math Cast Where Timestamp Logical Math Boolean String (Like, ||) Aggregate Input Format: delimited text (CSV, TSV), JSON … Compression: GZIP …
  • 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Glacier Select Input Format: delimited text (CSV, TSV, PSV, etc.) Encryption: SSE-KMS, SSE-S3 Output Format: delimited text (CSV, TSV, PSV, etc.) Clauses Data types Operators Functions Select String Conditional String From Integer, Float, Decimal Math Cast Where Timestamp Logical Boolean String (Like, ||)
  • 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Simple pattern matches …get-object …object… | awk -F ’{ if($4=="x") print $1}’ ...select-object …object… ‘SELECT o._1 WHERE o._4 == “x”…’
  • 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Using Glacier API Data directly uploaded to Amazon Glacier Using S3 Restore API For data that is lifecycle to Amazon Glacier from Amazon S3 How to use Glacier Select?
  • 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. With Amazon Glacier Select App Amazon Glacier Amazon S3Glacier-Select (ArchiveId, SQL, Tier, S3 bucket to write output) 200 OK Read data and Perform filtering Write output to S3 Notify app using SNS, that output ready
  • 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless applications Amazon S3 AWS Lambda Amazon SNS S3 Select
  • 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 200 seconds and 11.2 cents # Download and process all keys for key in src_keys: response = s3_client.get_object(Bucket=src_bucket, Key=key) contents = response['Body'].read() for line in contents.split('n')[:-1]: line_count +=1 try: data = line.split(',') srcIp = data[0][:8] …. 95 seconds and costs 2.8 cents # Select IP Address and Keys for key in src_keys: response = s3_client.select_object_content (Bucket=src_bucket, Key=key, expression = SELECT SUBSTR(obj._1, 1, 8), obj._2 FROM s3object as obj) contents = response['Body'].read() for line in contents: line_count +=1 try: …. AfterBefore Amazon S3 Select: Serverless MapReduce 2X Faster at 1/5 of the cost
  • 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Up to 400% Faster Up to 80% Cheaper Accelerating Big Data Amazon S3 Before: Amazon S3 S3 Select After:
  • 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon S3 Select with Presto Works with your existing Hive Metastore Automatically converts predicates into S3 Select requests Amazon S3 S3 Select
  • 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AfterBefore Amazon S3 Select: Accelerating big data 5X Faster with 1/40 of the CPU
  • 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Athena Amazon EMR Amazon Redshift Spectrum Amazon S3 Select: Will be supported by…
  • 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon S3 Select available in Preview • Formats: CSV, JSON • Compression: GZIP • Encryption: None • Encoding: UTF-8 • Integration: AWS SDK for Java and Python and Presto Connector • Availability: Northern Virginia, Ohio, Oregon, Dublin, and Singapore Apply at: https://pages.awscloud.com/amazon-s3-select-preview.html
  • 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Glacier Select: Generally available • Formats: CSV, Any delimiter separated file • Encryption: SSE- KMS, SSE-S3 • Encoding: UTF-8 • Integration: AWS SDK, CLI, Athena integration (expected 2018) • Availability: All commercial regions where Amazon Glacier is launched
  • 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. S3 Inventory
  • 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. S3 Inventory Save time Daily or Weekly delivery Delivery to S3 bucket • Same set of metadata as the LIST API • Can add size, last modified date, storage class, etag, or replication status Trigger business workflows and applications such as secondary index, garbage collection, data auditing, and offline analytics Delivery notification
  • 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. S3 Inventory Object level Encryption Status CSV or ORC output format Query with Athena, Redshift Spectrum or any Hive tools Encrypt inventory with SSE-S3 or SSE-KMS
  • 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. S3 Inventory  Bucket name  Key name  Version ID  IsLatest  Size  Last modified date  ETag  Storage class  Multipart upload flag  Delete marker  Replication status  Encryption Status destination-bucket
  • 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. "The rich information generated by AWS through the new object encryption status in S3 Inventory has been instrumental in helping us to automate and streamline daily reporting on compliance controls." – John Andrukonis Chief Architect, Capital One.
  • 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Query S3 Inventory with Amazon Athena CREATE EXTERNAL TABLE my_inventory_table( `bucket` string, key string, version_id string, is_latest boolean, is_delete_marker boolean, size bigint, last_modified_date timestamp, e_tag string, storage_class string, is_multipart_uploaded boolean, replication_status string, encryption_status string) PARTITIONED BY (dt string) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.SymlinkTextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat' LOCATION 's3://bucketname/inventory/output_destination/hive';
  • 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Query S3 Inventory with Amazon Athena
  • 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Visualize in Amazon QuickSight
  • 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Bucket Permissions Check Default Encryption
  • 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Trusted Advisor Bucket Permissions Check S3 Console Object Encryption status S3 Inventory Security Inspection
  • 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Bucket Permissions Check
  • 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Default Encryption Automatically encrypts all objects written to your Amazon S3 bucket • Choose SSE-S3 or SSE-KMS • Makes it easy to satisfy compliance needs
  • 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cross-Region Replication
  • 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cross-Region Replication (CRR) Use cases: What is CRR? Automated, fast, and reliable asynchronous replication of data across AWS regions Compliance Lower latency Security
  • 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How does CRR work? Bucket B Region A Region B • All uploads into source bucket are replicated • Entire bucket or prefix • Choose any AWS region as your target region • Secure transfer via SSL • Exact replicas including object ACL and tags Bucket A
  • 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CRR across account Region A Region B Why? • Additional protection on your back up to prevent malicious delete Ownership overwrite • Replica are owned by destination bucket • Maintain 2 distinct and independent stacks of ownership Primary Account Primary Account Secondary Account
  • 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Choose any AWS region as target Lifecycle policy Support SSE-KMS Encrypted objects Ownership overwrite for cross-account CRR Choose any S3 Storage Class as target More with Cross-Region Replication Bi-directional replication
  • 36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Getting Started - CRR
  • 37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. my_source_bucket
  • 38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  • 39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Destination KMS master key Ownership overwrite My_destination_bucket arn:aws:kms:us-east-2:123456789:/abc12345t234-1234-5678-a12b-a12b34cd5678
  • 40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cross Account CRR-Destination Set Up Destination bucket
  • 41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cross Account CRR –Destination Set Up
  • 42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Recap • S3 Select and Glacier Select • S3 Inventory Enhancements • S3 Default Encryption • S3 Bucket Permissions Check • S3 Cross-Region Replication Enhancements
  • 43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Learn more • S3 Select and Glacier Select https://youtu.be/p-JkncBZcc4 • S3 Select preview https://pages.awscloud.com/amazon-s3-select- preview.html • Amazon Macie: AWS Security Service Powered by Machine Learning https://youtu.be/LCjX2rsQ2wA • Deep Dive on Storage Management https://youtu.be/SUWqDOnXeDw
  • 44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!