Michelle Munson, CEO & Co-Founder of Aspera, is joined by Jay Migliaccio, Director or Cloud Technologies at Aspera, and Stephane Houet, Product Manager at EVS Broadcast Equipment, for the following session: MED305- Achieving Consistently High Throughput for Very Large Data Transfers with Amazon S3; Media Production & Distribution Track, on Wednesday, Nov 12, 3:30 PM - 4:15 PM – Level 4 - Delfino 4102
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
AWS re:Invent - Med305 Achieving consistently high throughput for very large data transfers with amazon s3 (aspera)
1. Michelle Munson - Co-Founder & CEO, Aspera
Jay Migliaccio - Director of Cloud Technologies, Aspera
Stéphane Houet – Product Manager, EVS Broadcast Equipment
2. PRESENTERS
Michelle Munson
Co-founder and CEO Aspera
michelle@asperasoft.com
Jay Migliaccio
Director Cloud Services, Aspera
jay@asperasoft.com
Stephane Houet
Product Manager, EVS
s.houet@evs.com
AGENDA
• Quick Intro to Aspera
• Technology Challenges
• Aspera Direct to Cloud Solution
• Demos
• FIFA Live Streaming Use Case
• Q & A
3.
4.
5.
6.
7.
8. HTTP
Master Database Mapping Object ID to Data Nodes
File Data Replicas and Storing Metadata
Key Value
H(URL) R1, R2, R3
15. TRANSFER DATA TO CLOUD OVER WAN EFFECTIVE THROUGHPUT
Multi-part HTTP
Typical internet conditions
• 50–250ms latency & 0.1–3% packet loss
15 parallel http streams
<10 to 100 Mbps
depending on distance
Aspera FASP Aspera FASP transfer over WAN to Cloud Up to 1Gbps *
10 TB transferred per 24 hours
* Per EC2 Extra Large Instance -independent of distance
16. LOCATION AND AVAILABLE
BANDWIDTH AWS ENHANCED UPLOADER ASPERA FASP
Montreal to AWS East
• 100 Mbps Shared Internet
Connection
30 minutes (7-10 Mbps)
3.7 minutes (80 Mbps)
9X Speed Up
Rackspace in Dallas to AWS
East
• 600 Mbps Shared Internet
Connection
7.5 minutes (38 Mbps)
0.5 minutes (600 Mbps)
15X Speed Up
Other pains … “Enhanced Bucket Uploader” requires java applet, very large
transfers time out, no good resume for interrupted transfers, no downloads
17. EFFECTIVE THROUGHPUT & TRANSFER TIME FOR 4.4 GB/15691 FILES (AVERAGE SIZE 300KB)
LOCATION AND AVAILABLE
BANDWIDTH AWS HTTP MULTIPART ASPERA ASCP
New York to AWS East Coast
• 1 Gbps Shared Connection
334 seconds (113 Mbps)
107 seconds (353 Mbps)
3.3X Speed Up
New York to AWS West Coast
• 1 Gbps Shared Connection
8.7 GB in 1032 seconds (36 Mbps)
8.7 GB in 110 seconds (353 Mbps)
9.4 X Speed Up
EFFECTIVE THROUGHPUT & TRANSFER TIME FOR 8.7 GB/18,995 FILES (AVERAGE SIZE 9.6MB)
LOCATION AND AVAILABLE
BANDWIDTH AWS HTTP MULTIPART ASPERA ASCP
New York to AWS East Coast
• 1 Gbps Shared Connection
477 seconds (156 Mbps)
178 seconds (420 Mbps)
2.7 X Speed Up
New York to AWS West Coast
• 1 Gbps Shared Connection
967 seconds (77 Mbps)
177 seconds (420 Mbps)
5.4 X Speed Up
18.
19.
20.
21. – Maximum speed single stream transfer
– Support for large files and directory sizes in a single transfer
– Network and disk congestion control provides automatic
adaptation of transmission speed to avoid congestion and overdrive
– Automatic retry and checkpoint resume of any transfer from point of
interruption
– Built in over-the-wire encryption and encryption-at-rest (AES 128)
– Support for authenticated Aspera docroots using
private cloud credentials and platform-specific role based access control including Amazon IAMS.
– Seamless fallback to HTTP(s) in restricted network
environments
– Concurrent transfer support scaling up to ~50 concurrent transfers
per VM instance
22. New Clients connect
to “available” pool
Existing client
transfers
Utilization > high w/m Available Pool
Console
• Collect / aggregate
transfer data
• Transfer activity /
reporting (UI, API)
Shares
• User management
• Storage access
control
KEY COMPONENTS
• Cluster Manager for Auto-scale and
Scaled DB
• Console Management UI + Reporting API
• Enhanced Client for Shares
Authorizations
• Unified Access to Files/Directories
(Browser, GUI, Commend Line, SDK)
Scaling Parameters
• Min/max number of t/s
• Utilization low/high
watermark
• Min number of t/s in
“available” pool
• Min number of idle t/s in
”available” pool
Management and Reporting
Cluster Manager
• Monitor cluster nodes
• Determine eligibility for
transfer scale up / down
• Create / remove db with
replicas
• Add / remove node
Scale DB Persistence Layer
26. • Near Live experiences have highly bursty processing and
distribution requirements
• Transcoding alone is expected to generate 100s of varieties of bitrates
and formats for a multitude of target devices
• Audiences peak to millions of concurrent streams and die off shortly
post event
• Near “Zero Delay” in the video experience is expected
• “Second screen” depends on near instant access / instant replay, which
requires reducing
• Linear transcoding approaches simply can not meet demand (and
are too expensive for short term use!)
• Parallel, “cloud” architectures are essential
• Investing in on premise bandwidth for distribution is also
impractical
• Millions of streams equals terabits per second
27. FASP
Scale Out
High-Speed Transfer by
Aspera
Scale Out
Transcoding by
On Demand Elemental
Multi-screen
capture and
distribution
by EVS
28. Belgian
company
+90%
market share of sports
OB trucks
21
offices
+500
employees
(+50% in R&D)
32. Live Streaming :
REAL TIME CONSTRAINT!
6 feeds @ 10 Mbps = 60 Mbps
X 2 games at the same time
X 2 for safety
WE NEED A SOLUTION !
240 Mbps
VOD Multicam Near-live replays :
Up to 24 clips @ 10 Mbps = 240 Mbps
X 2 games at the same time
480 Mbps
Maximum Throughput (bps) =
TCP-Window-Size (b) / Latency (s)
(65535 * 8) / 0.2 s =
2621400 bps = 2.62 Mbps
33.
34. 6 Live Streams
HLS streaming of 6 HD streams to
tablets & mobiles per match
+20 Replay cameras
On-demand replays of selected events
from up to 20+ cameras on the field
+4000 VoD elements
Exclusive on-demand multimedia
exclusive edits
35. FASP
Scale Out
High-Speed Transfer by
Aspera
Scale Out
Transcoding by
On Demand Elemental
Multi-screen
capture and
distribution
by EVS
36. + 27 TB of video data Key Metrics Total over
62 games
Average
per Game
Transfer Time (in hours) 13,857 216
Number of GB
Transferred 27,237 426
Number of Transfers 14,073 220
Number of Files
Transferred 2,706,922 42,296
< 14,000 hrs video transferred
200 ms of latency over WAN
10% packet loss over WAN
37. Live Streams
660,000 Minutes
Transcoded
Output
x 4.3 =
2.8 Million Minutes
Delivered
Streams
x 321 =
15 Million Hours
35 Million
Unique
Viewers