This document provides an overview and summary of recent innovations to Amazon S3. It discusses new storage classes like Standard - Infrequent Access storage, data ingestion options like S3 Transfer Acceleration and Amazon Kinesis Firehose, enhanced visibility and control of data, and expanded integration with services like AWS CloudTrail and CloudWatch. It also provides examples of how to use lifecycle policies to transition objects between storage classes and automatically delete incomplete multipart uploads or expired object delete markers.
2. Recent innovations on S3
Visibility & control
of your data
New storage
offering
More data
ingestion options
• Standard -
Infrequent Access
• Amazon CloudWatch
integration
• AWS CloudTrail integration
• New lifecycle policies
• Event notifications
• Bucket limit increases
• Read-after-write consistency
• IPv6 support
• AWS Snowball (80 TB)
• S3 Transfer Acceleration
• Amazon Kinesis Firehose
• Partner integration
3. Choice of storage classes on S3
Standard
Active data Archive dataInfrequently accessed data
Standard - Infrequent Access Amazon Glacier
4. File sync and share
+
consumer file
storage
Backup and archive +
disaster recovery
Long-retained
data
Use cases for Standard-Infrequent Access
5. Designed for 11 9s of
durability
Standard - Infrequent Access storage
Designed for
99.9% availability
Durable Available
Same as Standard storage
High performance
• Bucket policies
• AWS Identity and Access
Management (IAM) policies
• Many encryption options
Secure
• Lifecycle management
• Versioning
• Event notifications
• Metrics
Integrated
• No impact on user
experience
• Simple REST API
Easy to use
6. - Directly PUT to Standard - IA
- Transition Standard to Standard - IA
- Transition Standard - IA to Amazon Glacier
storage
- Expiration lifecycle policy
- Versioning support
Standard - Infrequent Access storage
Integrated: Lifecycle management
Standard - Infrequent Access
8. Lifecycle policy
Standard Storage -> Standard - IA
<LifecycleConfiguration>
<Rule>
<ID>sample-rule</ID>
<Prefix>documents/</Prefix>
<Status>Enabled</Status>
<Transition>
<Days>30</Days>
<StorageClass>STANDARD-IA</StorageClass>
</Transition>
<Transition>
<Days>365</Days>
<StorageClass>GLACIER</StorageClass>
</Transition>
</Rule>
</LifecycleConfiguration>
Standard - Infrequent Access storage
9. Standard Storage -> Standard - IA
<LifecycleConfiguration>
<Rule>
<ID>sample-rule</ID>
<Prefix>documents/</Prefix>
<Status>Enabled</Status>
<Transition>
<Days>30</Days>
<StorageClass>STANDARD-IA</StorageClass>
</Transition>
<Transition>
<Days>365</Days>
<StorageClass>GLACIER</StorageClass>
</Transition>
</Rule>
</LifecycleConfiguration>
Standard - IA Storage -> Amazon Glacier
Standard - Infrequent Access storage
Lifecycle policy
10. S3 support for IPv6
Dual-stack endpoints support both IPv4 and IPv6
Same high performance
Integrated with most S3 features
Manage access with IPv6 addresses
Easy to adopt, just change your endpoint.
No additional charges
11. IPv6 - Getting started
Update your endpoint to
• virtual hosted style address
http://bucketname.s3.dualstack.aws-region.amazonaws.com
Or
• path style address
http://s3.dualstack.aws-region.amazonaws.com/bucketname
15. 15 – COMCAST
IPV6 @ COMCAST
"Route 6 runs uncertainly from nowhere to nowhere,
scarcely to be followed from one end to the other,
except by some devoted eccentric”
George R. Stewart
AWS NYC 2016
16. 16 – COMCAST
BACKGROUND
• The IPv6 program at Comcast began in 2005
• Seamlessness is a cornerstone of our program
• Motivation
• IPv4 is not adequate, could not support near or long term
growth requirements
• IPv6 is inevitable
• Scope
• Everything, over time!
17. 17 – COMCAST
THE FIRST IPV6 ONLY SERVICE…
• 98+% of devices are
managed using IPv6
only
• Management use of
IPv6 (only) is one of the
largest deployments of
IPv6 worldwide
• Trending towards 100%
of all new and existing
devices managed
using IPv6 only, no IPv4
GROWTH
20. 20 – COMCAST
NEXT…
• Minimizing and reducing IPv4 dependencies
• IPv6 is used to manage the majority (and growing)
of our business needs today
• IPv6 utilization continues to grow
• Currently ~30% of our Internet facing
communications is over IPv6
• Leverage IPv6 as a platform for innovation
23. S3 Transfer Acceleration
S3 Bucket
AWS Edge
Location
Uploader
Optimized
Throughput!
Typically 50%-400% faster
Change your endpoint, not your code
No firewall exceptions
No client software required
59 global edge locations
24. Rio De
Janeiro
Warsaw New York Atlanta Madrid Virginia Melbourne Paris Los
Angeles
Seattle Tokyo Singapore
Time[hrs.]
500 GB upload from these edge locations to a bucket in Singapore
Public Internet
How fast is S3 Transfer Acceleration?
S3 Transfer Acceleration
25. Getting started
1. Enable S3 Transfer Acceleration on
your S3 bucket.
2. Update your endpoint to
<bucket-name>.s3-accelerate.amazonaws.com.
3. Done!
27. Tip: Parallelizing PUTs with multipart uploads
• Increase aggregate throughput by
parallelizing PUTs on high-bandwidth
networks
• Move the bottleneck to the network,
where it belongs
• Increase resiliency to network errors;
fewer large restarts on error-prone
networks
Best Practice
28. Incomplete multipart upload expiration policy
• Partial upload does incur storage charges
• Set a lifecycle policy to automatically make
incomplete multipart uploads expire after a
predefined number of days
Incomplete multipart
upload expiration
Best Practice
31. Tip #1: Use versioning
• Protects from accidental overwrites and
deletes
• New version with every upload
• Easy retrieval of deleted objects and roll
back to previous versions
Best Practice
Versioning
32. Tip #2: Use lifecycle policies
• Automatic tiering and cost controls
• Includes two possible actions:
• Transition: archives to Standard - IA or Amazon
Glacier based on object age you specified
• Expiration: deletes objects after specified time
• Actions can be combined
• Set policies at the bucket or prefix level
• Set policies for current version or non-
current versions
Lifecycle policies
34. Expired object delete marker policy
• Deleting a versioned object makes a
delete marker the current version of the
object
• Removing expired object delete marker
can improve list performance
• Lifecycle policy automatically removes
the current version delete marker when
previous versions of the object no
longer exist
Expired object delete
marker
36. Tip #3: Restrict deletes
• Bucket policies can restrict deletes
• For additional security, enable MFA (multi-factor
authentication) delete, which requires additional
authentication to:
• Change the versioning state of your bucket
• Permanently delete an object version
• MFA delete requires both your security credentials and a
code from an approved authentication device
Best Practice