2. AWS Storage Options
Object Storage Block Storage
Connect AWS Storage with On-Premises Environments
For NoSQL and Relational Database storage, catch the dedicated sessions
+ +
3. We are sincerely eager to hear
your feedback on this
presentation and on re:Invent.
Please fill out an evaluation form
when you have a chance.We are constantly producing more data
4. We are sincerely eager to hear
your feedback on this
presentation and on re:Invent.
Please fill out an evaluation form
when you have a chance.
From all types of industries
10. 1. High Durability
2. Unlimited Capacity
3. Low latency
4. High Volume Static Websites
5. Security & Compliance
6. Delivery via CDN
S3 gives you So you can build
Mobile Apps
Music Apps
Video networks
Video Apps
Backup Apps
Photo Apps
Static Websites
Ad networks
Anything else you’d like!
29. nearmap Aerial Imagery
nearmap routinely captures hi-res aerial images
covering 75% of Australia’s population
nearmap provides browser-based access to up-to-
date and historical aerial imagery to thousands of
companies, large and small
29
31. Current
Images are delivered within
7-14 days, rather than 3-6
months after flying
6mo
Traditional
nearmap
Processing Time
3mo 9mo 1yr
Brisbane floods – nearmap was the only company to capture the
flooded areas and publish within days
31
44. nearmap Tools
The photomap is
correctly scaled for
accurate
measurements
Objects are shown in
their correct real-
world position
44
45. Why Use nearmap
• Less time driving
• Less time on site
• Reduce fleet costs and environmental impact
• Eliminates existing spend on imagery
• Integrates with enterprise systems
45
46. nearmap and AWS
Q. What does nearmap use AWS for?
A. Everything
• Production
• Development and QA
• Corporate systems
• Image archival
46
47. Live on AWS October-2012
Transition went very smoothly
Local AWS team provided fantastic support
AWS systems continue to work flawlessly
47
48. AWS Features Used
EC2 - both Reserved and Spot Instances
S3
Glacier
RDS
Route53
CloudFormation
CloudWatch
IAM
SES
Direct Connect
7 May, 2013
49. Image Data Storage
6 Terabytes of new image data per month
We have 4+ years of existing image data
S3: 150 Terabytes and counting
Glacier: 300 Terabytes and counting
7 May, 2013
59. Workloads on EBS Provisioned IOPS
P-IOPS is designed to run transactional applications that require high and
consistent IO such as:
• Relational databases
• NoSQL databases (e.g. MongoDB)
• High performance file systems
• Productivity applications (e.g. Microsoft Exchange)
• Enterprise applications (e.g. CRM, ERP)
62. Use Amazon EBS When You Need
• Filesystem for an instance NTFS, ExtFS, RAID, LVM
• Long-term persistent storage
• Ability to change data frequently
• Access to raw, unformatted block-level storage
64. Access both via SDKs and via third-party software
S3 Glacier
Internet Web
Services API
HTTP(S)
Via SDK Via 3rd parties
65. Common Data Storage Challenges and the AWS Opportunity
Internet Web
Services API
HTTP(S)
AWS Direct
Connect
Block File
Customer
Data
Center
Storage
Use Cases
Archive Backup
Disaster
Recovery
NextGenerationEnterpriseStorage
S3 Glacier
66. Common Data Storage Challenges and the AWS Opportunity
Primary
Block
Storage
Primary
File
Storage
Archival
Storage
Disk-Based
Backup
Storage
Tape
Infrastructure
&
Management
Replicated
Storage for
Disaster
Recovery
Offsite
Locations
Geo-
Resilience
Next Generation: Gateway & Cloud Storage
Reduce
SAN
Footprint
Reduce
NAS
Footprint
Eliminate
Architectur
e
Hardware
& Software
Eliminate
Backup
Hardware
& Software
Eliminate
Tape
Recover in
Cloud or
any Data
Center
Eliminate
Secondary
Data
Centers
Reduce Complexity. Reduce Cost.
NextGenerationEnterpriseStorage-Benefits
67. Common Data Storage Challenges and the AWS Opportunity
S3 & Glacier - Designed to work with other AWS Services
68. Storage Gateway – Connect On-Prem with the AWS Cloud
1. Local, low-latency access to the
most frequently used files while
storing all data in Amazon S3
(Cached-Volumes)
Or
2. Scheduled off-site backups to
Amazon S3 for on-premises data
(Stored-Volumes)
69. Use AWS Storage Gateway When You Need To
• Backup your data
• Synchronize data
• Export data for migration
• Set up a departmental fileshare
• Conduct simple disaster recovery
Why are we going through the different storage solutions? Data comes in many shapes and sizes – this gives you different scale issues when you are looking at volumes of dataData also comes in a different rates – this is you a scale problem of how you ingest the data.It is important to understand what technology will meet your data needs, giving the right performance at the right price
One of the most common workloads in the mobile sector is multi-media.With the proliferation of smartphones, the amount of data that can be sent in simultaneously from a globally distributed network of phones can be huge. Both in total volume and ingest
It is not just the consumer segment that is producing masses of data. Oil and gas companies, such as our customer Shell for example, have embedded sensors all the way down the drill heads – this is producing huge amounts of data that needs to be ingested and stored
We are durable: One of its key value propositions is durability, 11 9sYour data is stored across three separate facilities giving you geo-redundancy and we can sustain data loss in two facilities simultaneously and your data is still safe.
We are BIG. We manage this level of durability across over 1 trilion unique customer objects. This does not include all the objects that AWS stores like snapshots.
We heard the feedback around S3, that it was a great product, but when you looked at infrequently accessed data that was to be archived for a long time, you wanted a cheaper solution
Glacier is 1c per GB per month…
That equates to $120 per TB/Year.
Whilst keeping the 11 x 9s of durability so your data is safe.Important to remember that Glacier is designed for the long term – if you store data on tape for 50 years, you will have to refresh the tapes every 5-10 years as otherwise the hardware will end of life.With Glacier we manage that for you.If you store data today, and retrieve it in 50 years, the chances are every part of hardware except for the concrete floor will have been swapped out
In contrast to S3, the data in Glacier needs 3-5 hoursSend in a request to retrieve data,We then notify you when it is there.Then you can download
Important to manage the retrieval costs. If your data is too hot, it will be more economic to store in S3You have 5% of your data per month (prorated to the day) available to retrieve.
Fractal pic under creative commons: http://www.flickr.com/photos/jayakody2000lk/7216399294/
TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
Different pricing around the globe.Also note RRS.This gives lower durability, but at a lower cost
Can also serve HTML and CSSS3 can be your web server for static content
If you have common content that is delivered to a high number of peopleCloudfront is our CDN. This can cache content locally, and the when you ask for the content you will be directed to the nearest POP
https://na3.salesforce.com/a0750000008gz0tAAA
One of our customers is Spotify, an on demand music service who adds over 20,000 music tracks a day and delivers that music to its customersCall out RRS in description
File systems have been the bedrock of computing for decades.
For more persistent data you need EBS
Each server comes with Ephemeral storage,However this cannot be transferred between instances, and is lost when the instance is disposed of.Great for temp data etc
You can therefore attach the drive to the instance
However with EBS, you can then reattach that EBS volume to a new instance. Your data is now persistent
You can also add multiple drives. Maybe you need it to segment your data – different DBs on different drives.Or you can get the OS to create a RAID drive and therefore see it as a single volume, which gives a larger max volume size, and can increase performance.
Allows you to create a volume of up to 1TB, that you can either use as a block device, or you can format for a file system
EBS is great for spiky loads, it will allow you to flex up. However for more consistent loads – ie databases you should look at PIOPs that allow you to configure drives of up to 2000IOPS
What is really important is to get point in time snapshots – this is your backup
You can take snapshots of your EBS drives to allow you to roll back to an older version if needed -all kept in S3
00:47:00
Important requirement is for people to be able to extend to the cloud and sync data between the twoTalk about Storage Gateway and RMAN
TRANSITION:Previous examples showed 3 ways to use S3…To store objects in S3, you can create buckets.These are containers for your objects, you can have up to 100 buckets – but unlimited files within those bucketsThese objects, with the right permissions, are addressable over the internet.This means S3 can be used for everything from storing images, to application data, to backups
Storage GatewayDownload a VM image that runs on your environmentGive it some local storage to use as a cache and upload buffer.I can now sync the data into the cloud as EBS snapshots.Which means that you can either download the snapshot and recover form that, or attach it to an EC2 instanceAlso have a cached volume version to enable thin provisioning or departmental / remote office file-sharingRemove some workloads from expensive file systems to simple/scalable gateway
00:47:00
Why are we going through the different storage solutions? Data comes in many shapes and sizes – this gives you different scale issues when you are looking at volumes of dataData also comes in a different rates – this is you a scale problem of how you ingest the data.It is important to understand what technology will meet your data needs, giving the right performance at the right price
It is all about choice.Pick the technology that delivers the right performance at the right price.
https://na3.salesforce.com/a0750000006xf87AAA
https://na3.salesforce.com/a0750000006xf9iAAA
Shazam at superbowl during the adverts managed to hit 500,000 write a second – maintaining that sub 10ms write time. They actually called us 3 days prior to the SuperBowl with the goal to make the impossible, possible and did so with great success