Mais conteúdo relacionado Semelhante a What's New with Amazon S3, Amazon EFS, and Other AWS Storage Services - STG201 - Anaheim AWS Summit (20) Mais de Amazon Web Services (20) What's New with Amazon S3, Amazon EFS, and Other AWS Storage Services - STG201 - Anaheim AWS Summit1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What’s new with AWS storage services
Robbie Wright
AWS Storage
S T G 2 0 1
2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda
Key use cases for AWS storage services
What’s new in Amazon Simple Storage Service (Amazon S3)
What’s new in Amazon Elastic Block Store (Amazon EBS)
What’s new in Amazon Amazon Elastic File System (Amazon EFS)
Amazon FSx
AWS Backup
AWS Hybrid-Edge and Data Transfer
3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon FSx for
Lustre
More choice for more applications
Object storage
S3 Standard
S3 Glacier Deep Archive
S3 Glacier
S3 Intelligent-Tiering
S3 One Zone-IA
S3 Standard-IA
Block storage
Provisioned-IOPS SSD
Cold HDD
Throughput-optimized HDD
File storage
EFS Standard
EFS Standard-Infrequent Access
Elastic
Amazon EFS AWS Storage
Gateway Family
Amazon S3
Amazon FSx for
Windows File
Server
Amazon EBS
Amazon EC2
Backup
AWS Backup
NEW!
NEW!
NEW!
NEW!
NEW!
4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
New AWS storage features in the last year
Block storage
Data transfer
Object storage
S3 Block Public Access
S3 Select GA (+Parquet, bzip2, JSON Arrays)
S3 performance increase to 5,500 reads/ 3,500
writes/sec
CRR based on object tags
S3 Glacier in S. America (Sao Paulo) Region
S3 Glacier CloudWatch metrics
S3 Batch Operations
S3 Glacier Restore Notifications
S3 Glacier Restore Speed Upgrade
S3 Object Lock
Direct Glacier access through S3 API
File storage
EFS in US West (N. California),
Asia Pacific (Seoul), Asia Pacific
(Tokyo), and Asia Pacific (Singapore)
Regions
EFS encryption in transit
EFS HIPAA eligibility
EFS provisioned throughput
EFS PCI-DSS compliance
EFS access over AWS VPN
EFS locks per file increase
EFS multi-VPC access
EBS Elastic Volumes for Standard Magnetic
EBS optimized performance Increase for c5/m5 to
14 gbps
EBS resource-level permissions for snapshots
Incremental encrypted snapshot copy with
custom CMKs
EBS snapshots tag on create
EBS data lifecycle manager
2X EBS provisioned IOPs performance
improvement
60% EBS gp2 SSD volumes performance
improvement
Storage Gateway hardware appliance
File Gateway SMB support
NEW!
NEW!
NEW!
NEW!
PREVIEW!
NEW!
NEW!
NEW!
Tape Gateway in two new regions
Snowball Edge Compute Optimized
5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS has been helping enterprises for 13 years
Increase agility Accelerate
innovation
Strengthen
security
Reduce
cost
6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What are customers building?
Backup &
restore
Non-disruptive
Easy place to start
Integrated with all
major vendors
Archive &
compliance
Media workflows
Tape replacement
Public Sector,
FinServ,
Healthcare/Life
Sciences
Home
directories
Simple to move
Not sensitive to
latency
Significant cost
savings
Data lakes
Variety of analytics
tools
Built for
streaming data
Data visualization
Business-
critical
applications
Integrated with
major vendors
Fully managed
infrastructure
Lift-and-shift
migrations
7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Backup, archive, and home directories
Backup & archive data
Home directories
Amazon S3
Amazon EFS
Amazon FSx for
Windows File Server
8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Business-critical applications
Amazon
EFS
Amazon FSx
for Windows
File Server
Amazon
EBS
Block storage
Windows
file storage
Linux/NFS file
storage
9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Data lakes
10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
On-premises
data center
Archival
Processing
Amazon S3AWS Storage Gateway
or
AWS DataSync
Hybrid cloud storage
Analytics
11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
More options for data transfer
AWS
Direct Connect
Amazon Kinesis
Data Firehose
AWS
Snowball
AWS Snowmobile
AWS Storage
Gateway
Amazon S3
Transfer
Acceleration
AWS
DataSync
AWS Transfer
for SFTP
AWS
Snowball Edge
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
12. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon S3
13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon S3
Media master files
DNA sequences
Mobile sync
and storage
Machine learning
training dataMedia assets
IoT sensor data
Data
lakes
Durable backups
Digital record
preservation
Seismic and
reservoir
simulation data
Home
recording video
Meteorological and
environmental research
Origin storage
for CDN
Financial
transaction
records Website
hosting
Surveillance
video/closed
circuit television
Medical imagery
and records
Autonomous
vehicle data
Oil and gas topography
Customer call
center records
Mapping data
Geospatial or lunar imagery
Compliance
records
Pharmaceutical
study data
User-generated
contentLog files
Analytics
14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S3 is the best place for data lakes
Most ways to
bring data in
Best security,
compliance,
and audit
capabilities
Object-level
controls
Unmatched
durability,
availability,
and scalability
Business
insights
into your data
15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S3 Batch Operations
Amazon S3: Service benefits
FPO
The most features to cost-effectively store,
manage, audit, secure, and query data
– at virtually any scale.
S3 Standard
S3 Standard-IA
S3 Intelligent-Tiering
S3 One Zone-IA
S3 Glacier
S3 Glacier
Deep Archive
Use S3 Storage Class Analysis to
learn access patterns and S3
Lifecycle policies to move objects
between classes
S3 Storage Classes
Configure access to S3 resources and
define user access. Block all public access
requests with S3 Block Public Access.
Access management
Replicate objects to a region of your choice
to reduce latency and for compliance.
Cross-region replication
Append up to 10 metadata tags to an object.
Use tags, buckets, and prefixes to organize
data. Audit and report on access requests and
activities.
Data management tools
Execute tasks and invoke AWS Lambda across billions of
objects—with a single API call or a few clicks in the
console.
S3-integrated analytics applications
AWS Lake Formation to stand up a data lake in days
S3 Select to query data in place
FSx for Lustre for HPC, ML, and media data processing
Analytics & file systems integrations
Supported by the most secure, durable, and performant storage infrastructure
Security by design Compliance programs 99.999999999% durability Multi-AZ resiliency Limitless scalability
S3 Standard
S3 Standard-IA
S3 Intelligent-Tiering
S3 One Zone-IA
S3 Glacier
S3 Glacier
Deep Archive
16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Security is job #1
Free checks with
AWS Trusted
Advisor
S3 default
encryption
S3 bucket permissions checkEncryption status
in S3 inventory
17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon S3 Block Public AccessS3bucket
AWS account
S3 Block Public
Access
18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
designed for
99.99%
durability
Two copies on one site
designed for
99.999%
durability
One copy on two sites
designed for
99.999999999%
durability
GlacierStandard IA
AWS Region
Understanding durability
19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How do AWS object storage classes differ in design?
Availability Zone
Availability Zone
Availability Zone
S3 Standard
S3 Intelligent-Tiering
S3 Standard-IA
Availability Zone
S3 One Zone-IA
AWS Region AWS Region
S3 Glacier
S3 Glacier Deep Archive
20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Your choice of Amazon S3 storage classes
Access frequencyFrequent Infrequent
• Active, frequently
accessed data
• Milliseconds access
• ≥3 AZ
• $0.0210/GB
• Data with changing access
patterns
• Milliseconds access
• ≥3 AZ
• $0.0210 to $0.0125/GB
• Monitoring fee per Obj.
• Min storage duration
• Infrequently accessed
data
• Milliseconds access
• ≥3 AZ
• $0.0125/GB
• Retrieval fee per GB
• Min storage duration
• Min object size
S3 Standard S3 Standard-IA S3 One Zone-IA S3 Glacier
• Re-creatable, less
accessed data
• Milliseconds access
• 1 AZ
• $0.0100/GB
• Retrieval fee per GB
• Min storage duration
• Min object size
• Archive data
• Select minutes or hours
• ≥3 AZ
• $0.0040/GB
• Retrieval fee per GB
• Min storage duration
• Min object size
S3 Intelligent-Tiering
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S3 Glacier
Deep Archive
• Archive data
• Select 12 or 48 hours
• ≥3 AZ
• $0.00099/GB
• Retrieval fee per GB
• Min storage duration
• Min object size
21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon S3 One Zone-IA
An S3 storage class built for easily re-creatable data
Designed on a single Availability Zone
Still 99.9999999% durable but less resilient—for 20% less cost
Use it for:
Mobile or enterprise backup data
Off-site compliance data
Disaster recovery data
Derived analysis data
22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S3 Intelligent-Tiering
Automated storage tiering for data
with changing access patterns
23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S3 Glacier Deep Archive
Lowest cost storage class for long-term
archiving and digital asset preservation
Fully managed
without tape
burden
$0.00099 per GB-
month
Designed for
99.999999999%
durability
Recover
data in 12
hours
24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S3 performance increase
S3 for data analytics
Before
Compute
Wr i t e t i m e Re a d t i m e
5 TB of 2-MB objects
S3
1with prefix
25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S3 performance increase
S3 request performance increase
NOW
Compute
W R I T E T I M E R E A D T I M E
S3
41m 40s 13m 52s
5TB of 2MB objects
1with prefix
26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S3 performance increase
S3 request performance increase
Parallel processing
Compute
Wr i t e t i m e Re a d t i m e
S3
12m 00s 7m 00s
41m 40s 13m 52s
5 TB of 2-MB objects
1with 0 prefix
27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S3 Glacier Restore Speed
Upgrade
S3 Object Lock
S3 Batch Operations 12
Manageability
PREVIEW!
Cross-Region Replication to
S3 Glacier
Direct PUT to S3 GlacierS3 Glacier Restore
Notifications
28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S3 Batch Operations
Takelarge-scaleactions onS3 objects
Now in Preview!
29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
S3 Batch Operations
Takelarge-scaleactions onS3 objects
Now in Preview!
30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Archival storage through S3 API
Writeto S3 Glacier directly via the S3 API—no Lifecyclepolicies necessary
Amazon GlacierS3 Storage Class
Move via Lifecycle PolicyPUT via S3 API
31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Archival Storage Through S3 API
New features
Direct PUT to Glacier
S3 Object Lock (WORM Storage)
S3 Glacier Restore Speed Upgrade
S3 Glacier Restore Notifications
CRR Direct to S3 Glacier
Write to S3 Glacier directly via the S3 API—no Lifecycle policies necessary
S3 Glacier
Direct PUT via S3 API
32. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon EBS: Built for dynamic workloads
2x performance improvement
for PIOPS SSD volumes
60% improvement
in gp2 SSD volumes
performance
High performance Reliable
Back up data on EBS volumes
using point-in-time snapshot
capability
Data Lifecycle Manager
Simple
34. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon EFS
Automatically
grows and shrinks
Lower TCO than
DIY or on-premises
Fully managed
Highly reliable
regional design
Secure
No re-architecting
required
Consistent IOPS
Consistent throughput
Flexible client
connectivity
Elastic ScalableSimple
36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EFS Infrequent Access
Lower-cost storage for EFS file data
as it becomes less frequently accessed
EFS Lifecycle Policy
85% savings for files accessed <1 time per
month
EFS Standard EFS IA
Frequently
accessed
file system
Infrequently
accessed
file system
37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EFS Bursting Throughput
Data ingest
Compute
D u ra t i o n
Av e ra g e
t h ro u g h p u t
1 TB of 1-MB files
from 1 c5.2xlarge instance
EFS
38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
EFS Provisioned Throughput
Data ingest
Compute
D u ra t i o n
Av e ra g e
T h ro u g h p u t
1 TB of 1-MB files
from 1 c5.2xlarge instance
2h 53m 105 MB/s
EFS
39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Compute
Compute
Compute
EFS Provisioned Throughput
Data ingest
Compute
D u ra t i o n
Av e ra g e
T h ro u g h p u t
1 TB of 1-MB files
from 4 c5.2xlarge instances
1h 12m 250 MB/s
EFS
2h 53m 105 MB/s
40. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon FSx for Windows File Server
Lift and shift your Windows file storage with
fully managed Windows file servers
Handles
patching and
other
maintenance
Fast and flexible
performance
Native
Windows
compatibility
back to
Windows 7
Connect to
Amazon EC2,
WorkSpaces,
Appstream 2.0
and VMware
Cloud on AWS
Ready for
enterprise apps
like ERP and
CRM
42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon FSx for Lustre
For compute-intensive data processing
use cases like HPC or machine learning
Raw data stored in S3 is loaded to
FSx for Lustre for processing
Output of processing returned to
S3 for retention
43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon FSx for Lustre Performance
Massively scalable performance
100+ GB/s throughput | Millions of IOPS |
Consistent sub-millisecond latencies
Parallel file system Supports hundreds of
thousands of cores
SSD-based
44. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Backup
Centrally manage backups acrossAWSservices
Automated backup
scheduling
Centralized
backup
management
Policy and tag
based backup
solution
Automated retention
and lifecycle
management
46. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
More options for data transfer
AWS
Direct Connect
Amazon Kinesis
Data Firehose
AWS
Snowball
AWS Snowmobile
AWS Storage
Gateway
Amazon S3
Transfer
Acceleration
AWS
DataSync
AWS Transfer
for SFTP
AWS
Snowball Edge
Amazon Kinesis
Data Streams
Amazon Kinesis
Video Streams
48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Storage Gateway
Accelerates
data center
shutdowns
Eliminates new
NAS/SAN
purchases
Incorporates
cloud scale and
durability with
on-premises
operations
Seamless, non-
disruptive
integration
Suitable for a vast
range of use cases
Simple Economical Hybrid
49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS DataSync
Simplifies and acceleratesonline data transfer—
up to 10X faster than open-sourcetools
AWS Region
Network attached
storage
NFS
DataSync agent AWS
DataSync
S3 bucket
EFS file system
Corporate data center
AWS Direct Connect or WAN
Encryption with TLS
Migrate active application
data to AWS
Transfer data for timely
in-cloud analysis
Replicate data to AWS for
business continuity
50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Transfer for SFTP
Fully managed SFTPservicefor S3
Your SFTP
end users
Archival
Analytics
Processing
S3AWS Transfer
for SFTP
Integrates with
existing Active
Directory or LDAP
Data available immediately
for archiving and
processing in S3
Highly available, and
elastically scalable
51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Snow family
• 80 TB capacity/10 G network
• Data encryption, end to end
• Rugged 8.5 G impact case
• Rain and dust resistant
• 100 TB capacity/10 Gb network
• Data encryption, end to end
• Rugged 8.5 G impact case
• Rain and dust resistant
• Compute and storage for
hybrid/edge workloads
• Rack-mountable, clusterable
• Exabyte-scale 45 ft container
• Data encryption, end to end
• Dedicated security personnel
• GPS tracking, alarm monitoring,
24/7 surveillance, and optional
additional security
AWS Snowball
AWS Snowball Edge –
Storage Optimized
AWS Snowball Edge –
Compute Optimized
AWS Snowmobile
• 52 vCPUs or NVIDIA G100 GPU
• 42 TB capacity/100 Gb/s networking
• Data encryption, end to end
• Rugged 8.5 G impact case
• Rain and dust resistant
• Compute and storage for
hybrid/edge workloads
• Rack-mountable, clusterable
52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon S3 Transfer Acceleration
Rio De Janeiro Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los Angeles Seattle Tokyo Singapore
Time[hrs]
500 GB upload from clients in these locations to a bucket in Singapore
Public internetAccelerated Transfer
Up to 300% faster
171% on average
53. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Robbie Wright