This document discusses various media and online advertising use cases on AWS. It begins with an overview of how media companies can build applications for music streaming, video streaming, digital publishing, and media sharing on AWS. It then discusses WeTransfer's use of AWS services like S3, EC2, RDS, and DynamoDB to support their file transfer needs at scale. The document also covers how online advertising platforms can use AWS services to address challenges around device and media fragmentation, scalability, and availability. It provides examples of how services like S3, Elastic Transcoder, CloudFront, EC2, Route 53, DynamoDB, and EMR can be used at various stages of an advertising request lifecycle for storage
12. WeTransfer
• Send big files anywhere
• Up to 2GB per transfer
• Files are stored for 7 days
• Beautiful backgrounds
(50/50 split between ads & art)
2
13. WeTransfer
• WeTransfer Plus: € 120 per year
• Up to 5GB per transfer
• Store up to 50GB permanently
• Your own backgrounds (no ads)
14. WeTransfer
• 1.8 Million transfers per day
• Downloads: 25 Gigabit per second
• 1.5 Million requests per hour (site + API)
• Over a Petabyte of storage used on S3
(Peak measurements - July 2013)
15. Company
• 15 wonderful, dedicated people
• Founded & based in Amsterdam
• Originated from Oy Communications
16. Origins
• Purely a functional tool
• Design company needs to send big files to clients
• FTP & friends “technical”, “confusing”
18. Growth
• Double the transfers every
3 months
• Previous hoster:
• could not match growth
tempo adequately
• hardware-based platform;
adds maintenance & development costs
20. Growth
• AWS: on-demand, available
right away
• Initial migration:
• development time: 1 month
• using S3 through EC2 instances
21. The new WeTransfer
• Built for & with AWS: uses RDS, EC2, S3,
CloudWatch, DynamoDB, Route53, ElastiCache
• Ruby + HTML5 + JavaScript (frontend)
• Backend tailored around S3
• Launched January of this year
22. WeTransfer and S3
• Virtually unlimited storage capacity
• Redundancy: always available
• Fast, and cheap compared to similar offers
• Dramatically less costs
23. WeTransfer and S3
• Uses the multipart upload mechanism (where possible)
• Resumable uploads
• Uploads go directly to S3
thanks to CORS support
• Worker instances to process
uploaded content
24. WeTransfer and S3
• Secure upload / download, and encryption
• Regionalized: storage facilities all over the world to
ensure proper speeds to end users
• No maintenance
25. So why S3?
• Fast & flexible
• Almost no time spent on maintenance
• Virtually limitless capacity at the tips of your fingers
31. 503
Service Temporarily Unavailable
The server is temporarily unable
to service your request due to
maintenance downtime or capacity
problems. Please try again later.
32. 503
Service Temporarily Unavailable
The server is temporarily unable
to service your request due to
maintenance downtime or capacity
problems. Please try again later.
46. Dallas(2)
St.Louis
Miami
JacksonvilleLos Angeles (2)
Palo Alto
Seattle
Ashburn(2)
Newark
New York (2)
Dublin
London(2)
Amsterdam
Stockholm
Frankfurt(2)
Paris(2)
Singapore(2)
Hong Kong
Tokyo
Sao Paulo
South Bend
San Jose
Osaka
Milan
Sydney
Reach a global audienceReach a global audience
48. Simple HLS video streaming architecture
In-house content
publication server
Source Video
Assets in S3
S3
49. Simple HLS video streaming architecture
In-house content
publication server
Source Video
Assets in S3
Video
transcoded into
HLS
S3Elastic Transcoder
50. Simple HLS video streaming architecture
In-house content
publication server
Source Video
Assets in S3
Video
transcoded into
HLS
Edge Delivery
using CloudFront
Stockholm
NY
CloudFront S3Elastic Transcoder
56. Amazon Route 53
Highly available and scalable Domain Name System
Extremely reliable and cost effective
Feature Details
Global Supported from AWS global edge locations for fast and reliable
domain name resolution
Scalable Automatically scales based upon query volumes
Latency
based
routing
Supports resolution of endpoints based upon latency, enabling
multi-region application delivery
Integrated Integrates with other AWS services allowing Route 53 to front load
balancers, S3 and EC2
Reach a global audience
90. • Cloud-based Real Time Advertising
Technology
• Focus on the premium publisher / media
owner
• Integrations with thousands of Demand
Partners
• Decision driven by Real Time Data
• Offices in UK, NL, DE & ES
• +100 Premium Publishers
IMPROVE
DIGITAL
ABOUT US
91. Use AWS in conjunction with dedicated physical
infrastructure
2 sides to the story
• Front end: serving of ads to end-users
• Back-end: Data processing and dev/test
Use of AWS
92. • Fleet of ad servers running mostly on EC2
• Ad serving process is computationally expensive and has strict time constraints
• Need ability to spin up additional instances based on demand: horizontally
scalable system
• Place ad servers in different regions to reduce serving latency; big benefit of
EC2 over physical kit
• Grow fleets in different regions separately
Serving ads
93. • S3/Glacier used for policy-driven data retention
• S3 is the starting point for AWS and on-premises data processing jobs
• S3 used as a shared storage space between distributed components
• VPC used to integrate AWS and on-premises flows and systems
• Automated deployment of dev/test software into VPC EC2 has been
great
Backend systems
94. As a startup it was almost a no-brainer
• Didn't want/need overhead of own physical infrastructure
• Pricing model with hugely reduced (or zero) up-front cost is an easy sell
• Coordinating the ability to quickly grow the ad server fleet is *hard* with a physical data centre
• As a more mature company the above still apply
• In addition our needs have also matured from the lower level "give us servers and storage"
Why do we use AWS?
95. Lessons learned:
It works!
Service integration is often ridiculously easy: pull S3 data into EMR, set
up auto-scaling etc
Geographic data locality -- helps with compliance
Automatic cost reductions does wonders for corporate acceptance
Continuous evolution of the services means that they suddenly can be
a great fit
96. Lessons learned:
It works in its own way
Need to understand exactly what each service offers
Need to design for fault tolerance; instances can fail at high scale
Had to work hard to get our network to integrate with VPC
Can't save you from yourself; poor design is poor design
Would still love to see another region in EU
97. • Growth means more of all the above
• Want to re-evaluate services that weren't a great fit for us in the past
(RDS, DynamoDB)
• Believe we can use data processing services (Elastic MapReduce in
particular) alongside on-premises systems
• Looking to Cloud Formation/ Elastic Beanstalk and Opsworks to
extend automation much further
The future