Optimize data lakes with Amazon S3 - STG302 - Santa Clara AWS Summit

© 2019, Amazon Web Services, Inc. orits affiliates. All rights reserved.S UM M I T
Optimizing data lakes with Amazon S3
John Mallory
Storage Business Development Manager
AWS
S T G 3 0 2

125+ million players
Data provides a constant feedback loop
for game designers
Up-to-the-minute analysis of gamer
satisfaction to drive gamer engagement
Resulting in the most popular
game played in the world
Fortnite

Data is a strategic asset for every organization
The world’s most valuable
resource is no longer oil, but
data.*
*Copyright: The Economist, 2017,DavidParkins
“
”

Finding value in data is a journey
Business monitoring
Business insights
New business opportunity
Business optimization
Business transformation
Evolving tools and methods
AI/MLSQL query

Why use AWS for big data & analytics?
Agility Scalability
Get to insights faster
Broadest and deepest
capabilities
Low cost
Data migrations made easy

More data lakes and analytics than anywhere else
More than 10,000data lakesonAWS

Defining the AWS data lake
Data lakes provide:
Relational and nonrelational data
Scale out to Amazon EBS
Diverse set of analytics and machine learning tools
Work on data without any data movement
Designed for low-cost storage and analytics
OLTP ERP CRM LoB
Data warehouse
Business
intelligence
Data lake
100110000100101011100101010
111001010100001011111011010
0011110010110010110
0100011000010
Devices Web Sensors Social
Catalog
Machine
learning
DW queries Big data
processing
Interactive Real time

Data lake on AWS
Catalog & search Access & user interfaces
Data ingestion
Analytics & serving
S3
Amazon
DynamoDB
Amazon Elasticsearch
Service
AWS
AppSync
Amazon
API Gateway
Amazon
Cognito
AWS
KMS
AWS
CloudTrail
Manage & secure
AWS
IAM
Amazon
CloudWatch
AWS
Snowball
AWS Storage
Gateway
Amazon
KinesisData
Firehose
AWS Direct
Connect
AWS DatabaseMigration
Service
Amazon
Athena
Amazon
EMR
AWS
Glue
Amazon
Redshift
Amazon
DynamoDB
Amazon
QuickSight
Amazon
Kinesis
Amazon
Elasticsearch
Service
Amazon
Neptune
Amazon
RDS
Central storage
Scalable, secure, cost-
effective
AWS
Glue

User-definedfunctions
• Bring your own functions & code
• Execute without provisioning servers
Processing and querying in place
Fully managed process & query
• Catalog, transform & query data in Amazon S3
• No physical instances to manage
Lambda function

Amazon S3 is the best place for data lakes
Most ways to
bring data in
Best security,
compliance,
and audit
capabilities
Object-level
controls
Unmatched
durability,
availability,
and scalability
Business
insights
into your data

Optimize costs with data tiering
Hot
Cold
Amazon S3
Standard
Amazon S3
Standard-
Infrequent Access
Amazon S3
Glacier
HDFS ✓ Use EMR/Hadoop with local HDFS for
hottest data sets
✓ Store cooler data in Amazon S3 and
cold in Amazon S3 Glacier to reduce
costs
✓ Use Amazon S3 analytics to optimize
tiering strategy
Amazon S3 Analytics

Your choice of Amazon S3 storage classes
Access frequencyFrequent Infrequent
• Active, frequently
accessed data
• Milliseconds access
• >3 AZ
• From: $0.0210/GB
• Data with changing access
pattern
• >3 AZ
• From: $0.0210 to
$0.0125/GB
• Monitoring fee per obj.
• Min. storage duration
• Infrequently accessed
data
• >3 AZ
• From: $0.0125/GB
• Retrieval fee per GB
• Min. object size
S3 Standard S3 Standard-IA S3 One Zone-IA S3 Glacier
• Recreatable less accessed
data
• 1 AZ
• From: $0.0100/GB
• Archive data
• Minutes to hours
access
• >3 AZ
• From: $0.0040/GB
• Retrieval fee
per GB
S3 Intelligent-
Tiering
S3 Glacier Deep
Archive
• Archive data
• Hours access
• >3 AZ
• From: $0.00099/GB
N E W ! N E W !

Amazon S3 Intelligent-Tiering
automates cost savings
Automatically optimizes storage costs for
data with changing access patterns
Moves objects between two storage tiers:
• Frequent access tier
• Infrequent access tier
Monitors access patterns and auto-tiers on
granular object level
Milliseconds access, >3 AZ, monitoring fee
per object, minimum storage duration
NEW!

Amazon S3 Glacier Deep Archive NEW!
No tape to
manage
$0.00099/GB/month
Less than 1/4 the cost of
Amazon S3 Glacier
Designed for
119s durability
Recover data in
hours
Lowest cost storage available in the cloud
C o m i n g s o o n

A data lake needs to
accommodate a wide
variety of concurrent data
sources
Rapidly ingest all data sources
IoT, sensor data, clickstreamdata,
social media feeds, streaming logs
Oracle, MySQL, MongoDB, DB2,
SQL Server, Amazon RDS
On-premises ERP, mainframes,
lab equipment, NAS storage
Offline sensor data, NAS,
On-premises Hadoop
On-premises data lakes, EDW,
large-scale data collection
Ingest
methods

AWS Transfer for SFTP NEW!
Fully managed service enabling transfer
of data over SFTP while stored in Amazon S3
Seamless migration
of existing workflows
Native integration
with AWS services
Simple
to use
Cost
effective
Secure and compliantFully managed
in AWS

AWS
integrated
AWS
Transfer service that simplifies, automates, and accelerates data movement
Transfers up
to 10 Gbps
per agent
Pay as
you go
Secure and
reliable
transfers
Replicate data to AWS for
business continuity
Transfer data for timely
in-cloud analysis
Migrate active application
data to AWS
Combines the speed and reliability of network acceleration software
with the cost-effectiveness of open source tools
Simple data
movement to
Amazon S3 or
Amazon EFS
AWS DataSync NEW!

Process data in place
Amazon S3
Amazon Athena Amazon Redshift
Spectrum
Amazon SageMaker AWS Glue

Amazon S3 Select
Select a subset of your object’s data using a SQL expression

Improved performance for data lakes
As customers store larger and larger datasets in Amazon S3,
Amazon S3 Select offers up to a 400% performance improvement

Amazon S3 Select enhancements NEW!
Now supports:
CSV, JSON, JSON arrays, and Parquet formats
GZIP, BZIP2, and Snappy compression
Integrated with Spark, Hive, and Presto on Amazon EMR

Amazon FSx for Lustre
For compute-intensive data processing
use cases like HPC or machine learning
Raw data stored in Amazon S3 is loaded to
FSx for Lustre for processing
Output of processing returned to
Amazon S3 for retention

Amazon FSx for Lustre performance
Massively scalable performance
100+ GB/s throughput | Millions of IOPS |
Consistent submillisecond latencies
Parallel file system Supports hundreds of
thousands of cores
SSD-based

Choosing the right data formats
There is no such thing as the “best” data format
• All involve tradeoffs, depending on workload & tools
• CSV, TSV, JSON are easy but not efficient
• Compress & store or archive as raw input
• Columnar compressed are generally preferred
• Parquet or ORC
• Smaller storage footprint = lower cost
• More efficient scan & query
• Row-oriented (AVRO) good for full data scans
• Organize into partitions
• Coalescing to larger partitions over time
Key considerations are cost, performance, & support

Data prep is ~80% of data lake work
Building training sets
Cleaning and organizing data
Collecting datasets
Mining data for patterns
Refining algorithms
Other

Set up a catalog, ETL, and data prep
with AWS Glue
Serverless provisioning, configuration, and
scaling to run your ETL jobs on Apache Spark
Pay only for the resources used for jobs
Crawl your data sources, identify data
formats, and suggest schemas and
transformations
Automates the effort in building, maintaining
and running ETL jobs

Security challenges with data lakes
Data challenges
• Controlling access to data
• Data masking, row / column / cell level
encryption, key management
• Data loss / exfiltration
• Loss of data integrity
• Data provenance
• Compliance requirements (GDPR
and others)
Management challenges
• Central administration
• Federated authentication,
typically with Active Directory
• Role-based access control (RBAC)
• Centralized audit
• End-to-end data protection (at-
rest and in-transit)

AWS helps you secure
Compliance
AWS Artifact
Amazon Inspector
AWS CloudHSM
Amazon Cognito
AWS CloudTrail
Security
Amazon GuardDuty
AWS Shield
AWS WAF
Amazon Macie
Amazon VPC
Encryption
AWS Certification Manager
AWS Key Management Service
Encryption at rest
Encryption in transit
Bring your own keys, HSM
support
Identity
AWS IAM
AWS SSO
Amazon Cloud Directory
AWS Directory Service
AWS Organizations
Customers need multiple levels of security, identity and access management, encryption, and
compliance to secure their data lake

Data lake security
• Data storage
• Metadata

Control access to data
Configure Amazon S3 permissions
• Implement your access control matrix using IAM
policies
• Use S3 bucket policies for easy cross-account data
sharing
• Limit role-based access from an Amazon EMR
cluster’s Amazon Elastic Compute Cloud (Amazon
EC2) instance profile
• Authorize access from other tools such as Amazon
Redshift using IAM roles
IAM principals Amazon EMR Amazon Redshift

Block public access to Amazon S3
Amazon S3 provides four settings
• BlockPublicAcls – Rejects new public object or bucket ACLs
• IgnorePublicAcls – Ignores existing public object or bucket ACLs
• BlockPublicPolicy – Rejects new public bucket access policy
• RestrictPublicBuckets – Restricts access to only AWS services and authorized users
within the bucket owner's account
But, what is “public”?
• Public object (or bucket) ACL → Grants permissions to members of the
predefined AllUsers or AuthenticatedUsers groups (grantees)
• Public bucket policy → Doesn’t grant permissions to only fixed values in Principal and
Condition elements

Amazon S3 Object Lock NEW!
Immutable Amazon S3 objects
• Write once read many (WORM) protections for Amazon S3 objects
• Object or bucket control of WORM & retention attributes
Retention managementcontrols
• Define retention periods in your app or with bucket-level defaults
• objects locked for the duration of the retention period
• Support for legal hold scenarios
Data protection and compliance
• Assessedfor use in SEC 17a-4, CFTC, and FINRA environments
• Extra protection against accidental or malicious delete

Metadata security
AWS Glue Data Catalog
• Apache Hive metastorecompatible
• Track data evolution using schema versioning
• Integrates with Hive, Spark, Presto, Amazon
Athena and Amazon Redshift Spectrum
• Use crawlers classify your data in one central list
that is searchable

Metadata security
Key learnings
• Create and maintain centralized data catalog
• Enable cross account access
• Use IAM policies to control catalog access—similar to
Amazon S3 bucket policies
• Encrypt metadata in AWS Glue Data Catalog

AWS Glue Data Catalog resource policies
• Fine-grained access control to Data Catalog using IAM policies
• Restrict what they can view and query

Typical steps of building a data lake
Setup storage1
Move data2
Cleanse, prep, and
catalog data
3
Configureand enforce
security and compliance
policies
4
Make data available
for analytics
5

Building data lakes can still take months

Enforce security policies across
multiple services
Gain and manage new insights
Identify, ingest, clean, and
transform data
Build a secure data lake in days
AWS Lake Formation

How it works

Three simple steps to an AWS data lake
Remove data silos
Aggregate data
Better agility
More data = > Insights
Know what you have
Better datamanagement
Quicker time to results
Higher quality data
Extract value from data
Analyze & report on data
Apply machine learning
Visualize & consume results
Amazon ingest & storage
Amazon S3, Amazon S3 Glacier,
AWS Sync,
AWS Storage Gateway,
AWS Snow Family,
Amazon Kinesis
AWS Glue
Crawl, discover & catalogdata
ETL data
Amazon analytics & ML
Amazon Athena, EMR,
Amazon Redshift,Amazon
SageMaker, Amazon Rekognition,
Amazon EC2 + FSx for Lustre
Collect & centralize Catalog & transform Analytics & insights

Optimize data lakes with Amazon S3 - STG302 - Santa Clara AWS Summit

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Optimize data lakes with Amazon S3 - STG302 - Santa Clara AWS Summit

Similar to Optimize data lakes with Amazon S3 - STG302 - Santa Clara AWS Summit (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Optimize data lakes with Amazon S3 - STG302 - Santa Clara AWS Summit