SlideShare uma empresa Scribd logo
1 de 38
Baixar para ler offline
Michelle Munson - Co-Founder & CEO, Aspera 
Jay Migliaccio - Director of Cloud Technologies, Aspera 
Stéphane Houet – Product Manager, EVS Broadcast Equipment
PRESENTERS 
Michelle Munson 
Co-founder and CEO Aspera 
michelle@asperasoft.com 
Jay Migliaccio 
Director Cloud Services, Aspera 
jay@asperasoft.com 
Stephane Houet 
Product Manager, EVS 
s.houet@evs.com 
AGENDA 
• Quick Intro to Aspera 
• Technology Challenges 
• Aspera Direct to Cloud Solution 
• Demos 
• FIFA Live Streaming Use Case 
• Q & A
HTTP 
Master Database Mapping Object ID to Data Nodes 
File Data Replicas and Storing Metadata 
Key Value 
H(URL) R1, R2, R3
With Direct-to-CLOUD
TRANSFER DATA TO CLOUD OVER WAN EFFECTIVE THROUGHPUT 
Multi-part HTTP 
Typical internet conditions 
• 50–250ms latency & 0.1–3% packet loss 
15 parallel http streams 
<10 to 100 Mbps 
depending on distance 
Aspera FASP Aspera FASP transfer over WAN to Cloud Up to 1Gbps * 
10 TB transferred per 24 hours 
* Per EC2 Extra Large Instance -independent of distance
LOCATION AND AVAILABLE 
BANDWIDTH AWS ENHANCED UPLOADER ASPERA FASP 
Montreal to AWS East 
• 100 Mbps Shared Internet 
Connection 
30 minutes (7-10 Mbps) 
3.7 minutes (80 Mbps) 
9X Speed Up 
Rackspace in Dallas to AWS 
East 
• 600 Mbps Shared Internet 
Connection 
7.5 minutes (38 Mbps) 
0.5 minutes (600 Mbps) 
15X Speed Up 
Other pains … “Enhanced Bucket Uploader” requires java applet, very large 
transfers time out, no good resume for interrupted transfers, no downloads
EFFECTIVE THROUGHPUT & TRANSFER TIME FOR 4.4 GB/15691 FILES (AVERAGE SIZE 300KB) 
LOCATION AND AVAILABLE 
BANDWIDTH AWS HTTP MULTIPART ASPERA ASCP 
New York to AWS East Coast 
• 1 Gbps Shared Connection 
334 seconds (113 Mbps) 
107 seconds (353 Mbps) 
3.3X Speed Up 
New York to AWS West Coast 
• 1 Gbps Shared Connection 
8.7 GB in 1032 seconds (36 Mbps) 
8.7 GB in 110 seconds (353 Mbps) 
9.4 X Speed Up 
EFFECTIVE THROUGHPUT & TRANSFER TIME FOR 8.7 GB/18,995 FILES (AVERAGE SIZE 9.6MB) 
LOCATION AND AVAILABLE 
BANDWIDTH AWS HTTP MULTIPART ASPERA ASCP 
New York to AWS East Coast 
• 1 Gbps Shared Connection 
477 seconds (156 Mbps) 
178 seconds (420 Mbps) 
2.7 X Speed Up 
New York to AWS West Coast 
• 1 Gbps Shared Connection 
967 seconds (77 Mbps) 
177 seconds (420 Mbps) 
5.4 X Speed Up
– Maximum speed single stream transfer 
– Support for large files and directory sizes in a single transfer 
– Network and disk congestion control provides automatic 
adaptation of transmission speed to avoid congestion and overdrive 
– Automatic retry and checkpoint resume of any transfer from point of 
interruption 
– Built in over-the-wire encryption and encryption-at-rest (AES 128) 
– Support for authenticated Aspera docroots using 
private cloud credentials and platform-specific role based access control including Amazon IAMS. 
– Seamless fallback to HTTP(s) in restricted network 
environments 
– Concurrent transfer support scaling up to ~50 concurrent transfers 
per VM instance
New Clients connect 
to “available” pool 
Existing client 
transfers 
Utilization > high w/m Available Pool 
Console 
• Collect / aggregate 
transfer data 
• Transfer activity / 
reporting (UI, API) 
Shares 
• User management 
• Storage access 
control 
KEY COMPONENTS 
• Cluster Manager for Auto-scale and 
Scaled DB 
• Console Management UI + Reporting API 
• Enhanced Client for Shares 
Authorizations 
• Unified Access to Files/Directories 
(Browser, GUI, Commend Line, SDK) 
Scaling Parameters 
• Min/max number of t/s 
• Utilization low/high 
watermark 
• Min number of t/s in 
“available” pool 
• Min number of idle t/s in 
”available” pool 
Management and Reporting 
Cluster Manager 
• Monitor cluster nodes 
• Determine eligibility for 
transfer scale up / down 
• Create / remove db with 
replicas 
• Add / remove node 
Scale DB Persistence Layer
.mp2ts 
HLS 
adaptive 
bitrate 
FASPStream
• Near Live experiences have highly bursty processing and 
distribution requirements 
• Transcoding alone is expected to generate 100s of varieties of bitrates 
and formats for a multitude of target devices 
• Audiences peak to millions of concurrent streams and die off shortly 
post event 
• Near “Zero Delay” in the video experience is expected 
• “Second screen” depends on near instant access / instant replay, which 
requires reducing 
• Linear transcoding approaches simply can not meet demand (and 
are too expensive for short term use!) 
• Parallel, “cloud” architectures are essential 
• Investing in on premise bandwidth for distribution is also 
impractical 
• Millions of streams equals terabits per second
FASP 
Scale Out 
High-Speed Transfer by 
Aspera 
Scale Out 
Transcoding by 
On Demand Elemental 
Multi-screen 
capture and 
distribution 
by EVS
Belgian 
company 
+90% 
market share of sports 
OB trucks 
21 
offices 
+500 
employees 
(+50% in R&D)
With the kind permission of HBS
Live Streaming : 
REAL TIME CONSTRAINT! 
6 feeds @ 10 Mbps = 60 Mbps 
X 2 games at the same time 
X 2 for safety 
WE NEED A SOLUTION ! 
240 Mbps 
VOD Multicam Near-live replays : 
Up to 24 clips @ 10 Mbps = 240 Mbps 
X 2 games at the same time 
480 Mbps 
Maximum Throughput (bps) = 
TCP-Window-Size (b) / Latency (s) 
(65535 * 8) / 0.2 s = 
2621400 bps = 2.62 Mbps
6 Live Streams 
HLS streaming of 6 HD streams to 
tablets & mobiles per match 
+20 Replay cameras 
On-demand replays of selected events 
from up to 20+ cameras on the field 
+4000 VoD elements 
Exclusive on-demand multimedia 
exclusive edits
FASP 
Scale Out 
High-Speed Transfer by 
Aspera 
Scale Out 
Transcoding by 
On Demand Elemental 
Multi-screen 
capture and 
distribution 
by EVS
+ 27 TB of video data Key Metrics Total over 
62 games 
Average 
per Game 
Transfer Time (in hours) 13,857 216 
Number of GB 
Transferred 27,237 426 
Number of Transfers 14,073 220 
Number of Files 
Transferred 2,706,922 42,296 
< 14,000 hrs video transferred 
200 ms of latency over WAN 
10% packet loss over WAN
Live Streams 
660,000 Minutes 
Transcoded 
Output 
x 4.3 = 
2.8 Million Minutes 
Delivered 
Streams 
x 321 = 
15 Million Hours 
35 Million 
Unique 
Viewers
AWS re:Invent - Med305 Achieving consistently high throughput for very large data transfers with amazon s3 (aspera)

Mais conteúdo relacionado

Mais procurados

L’odyssée d’une requête HTTP chez Scaleway
L’odyssée d’une requête HTTP chez ScalewayL’odyssée d’une requête HTTP chez Scaleway
L’odyssée d’une requête HTTP chez Scaleway
Scaleway
 
RedisConf18 - Serving Automated Home Valuation with Redis & Kafka
RedisConf18 - Serving Automated Home Valuation with Redis & KafkaRedisConf18 - Serving Automated Home Valuation with Redis & Kafka
RedisConf18 - Serving Automated Home Valuation with Redis & Kafka
Redis Labs
 
Building a CICD Pipeline for Containers - DevDay Austin 2017
Building a CICD Pipeline for Containers - DevDay Austin 2017Building a CICD Pipeline for Containers - DevDay Austin 2017
Building a CICD Pipeline for Containers - DevDay Austin 2017
Amazon Web Services
 

Mais procurados (20)

Walmart & IBM Revisit the Linear Road Benchmark- Roger Rea, IBM
Walmart & IBM Revisit the Linear Road Benchmark- Roger Rea, IBMWalmart & IBM Revisit the Linear Road Benchmark- Roger Rea, IBM
Walmart & IBM Revisit the Linear Road Benchmark- Roger Rea, IBM
 
Netflix viewing data architecture evolution - EBJUG Nov 2014
Netflix viewing data architecture evolution - EBJUG Nov 2014Netflix viewing data architecture evolution - EBJUG Nov 2014
Netflix viewing data architecture evolution - EBJUG Nov 2014
 
Aws multi-region High Availability
Aws multi-region High Availability Aws multi-region High Availability
Aws multi-region High Availability
 
AWS reInvent 2016 recap Taiwan
AWS reInvent 2016 recap TaiwanAWS reInvent 2016 recap Taiwan
AWS reInvent 2016 recap Taiwan
 
Network Latency
Network LatencyNetwork Latency
Network Latency
 
Engineering Leader opportunity @ Netflix - Playback Data Systems
Engineering Leader opportunity @ Netflix - Playback Data SystemsEngineering Leader opportunity @ Netflix - Playback Data Systems
Engineering Leader opportunity @ Netflix - Playback Data Systems
 
L’odyssée d’une requête HTTP chez Scaleway
L’odyssée d’une requête HTTP chez ScalewayL’odyssée d’une requête HTTP chez Scaleway
L’odyssée d’une requête HTTP chez Scaleway
 
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
 
Network latency - measurement and improvement
Network latency - measurement and improvementNetwork latency - measurement and improvement
Network latency - measurement and improvement
 
Keystone - ApacheCon 2016
Keystone - ApacheCon 2016Keystone - ApacheCon 2016
Keystone - ApacheCon 2016
 
RedisConf18 - Serving Automated Home Valuation with Redis & Kafka
RedisConf18 - Serving Automated Home Valuation with Redis & KafkaRedisConf18 - Serving Automated Home Valuation with Redis & Kafka
RedisConf18 - Serving Automated Home Valuation with Redis & Kafka
 
Cloudsolutionday 2016: DevOps workflow with Docker on AWS
Cloudsolutionday 2016: DevOps workflow with Docker on AWSCloudsolutionday 2016: DevOps workflow with Docker on AWS
Cloudsolutionday 2016: DevOps workflow with Docker on AWS
 
AWS re:Invent 2016 Fast Forward
AWS re:Invent 2016 Fast ForwardAWS re:Invent 2016 Fast Forward
AWS re:Invent 2016 Fast Forward
 
Multi-master, multi-region MySQL deployment in Amazon AWS
Multi-master, multi-region MySQL deployment in Amazon AWSMulti-master, multi-region MySQL deployment in Amazon AWS
Multi-master, multi-region MySQL deployment in Amazon AWS
 
Monitoring microservices platform
Monitoring microservices platformMonitoring microservices platform
Monitoring microservices platform
 
AWS guerrilla orchestration
AWS guerrilla orchestrationAWS guerrilla orchestration
AWS guerrilla orchestration
 
Running Containerised Applications at Scale on AWS
Running Containerised Applications at Scale on AWSRunning Containerised Applications at Scale on AWS
Running Containerised Applications at Scale on AWS
 
Navigate Data Service using AWS
Navigate Data Service using AWSNavigate Data Service using AWS
Navigate Data Service using AWS
 
Building a CICD Pipeline for Containers - DevDay Austin 2017
Building a CICD Pipeline for Containers - DevDay Austin 2017Building a CICD Pipeline for Containers - DevDay Austin 2017
Building a CICD Pipeline for Containers - DevDay Austin 2017
 
Keystone - Leverage Big Data 2016
Keystone - Leverage Big Data 2016Keystone - Leverage Big Data 2016
Keystone - Leverage Big Data 2016
 

Destaque

OneLeap Solutions (1)
OneLeap Solutions (1)OneLeap Solutions (1)
OneLeap Solutions (1)
Amol Shenvi
 
AWS for Media: Content in the Cloud, Miles Ward (Amazon Web Services) and Bha...
AWS for Media: Content in the Cloud, Miles Ward (Amazon Web Services) and Bha...AWS for Media: Content in the Cloud, Miles Ward (Amazon Web Services) and Bha...
AWS for Media: Content in the Cloud, Miles Ward (Amazon Web Services) and Bha...
Amazon Web Services
 

Destaque (17)

AWS 2013 LA Media Event: Scalable Media Processing
AWS 2013 LA Media Event: Scalable Media ProcessingAWS 2013 LA Media Event: Scalable Media Processing
AWS 2013 LA Media Event: Scalable Media Processing
 
Aspera - Bridging On Premise and Cloud Deployments for Broadcast IT
Aspera - Bridging On Premise and Cloud Deployments for Broadcast ITAspera - Bridging On Premise and Cloud Deployments for Broadcast IT
Aspera - Bridging On Premise and Cloud Deployments for Broadcast IT
 
Aspera Solution Overview - IBM Software
Aspera Solution Overview - IBM SoftwareAspera Solution Overview - IBM Software
Aspera Solution Overview - IBM Software
 
The Pandora Security Model - Alessandro Confetti
The Pandora Security Model -  Alessandro ConfettiThe Pandora Security Model -  Alessandro Confetti
The Pandora Security Model - Alessandro Confetti
 
Diventare famosi con lo stack ELK - Alfonso Iannotta
Diventare famosi con lo stack ELK - Alfonso IannottaDiventare famosi con lo stack ELK - Alfonso Iannotta
Diventare famosi con lo stack ELK - Alfonso Iannotta
 
Telestream Vidchecker
Telestream VidcheckerTelestream Vidchecker
Telestream Vidchecker
 
OneLeap Solutions (1)
OneLeap Solutions (1)OneLeap Solutions (1)
OneLeap Solutions (1)
 
TELESTREAM Vantage - VIDELIO Cap'Ciné
TELESTREAM Vantage - VIDELIO Cap'CinéTELESTREAM Vantage - VIDELIO Cap'Ciné
TELESTREAM Vantage - VIDELIO Cap'Ciné
 
AWS Partner Presentation - Aspera
AWS Partner Presentation - Aspera AWS Partner Presentation - Aspera
AWS Partner Presentation - Aspera
 
L'architettura di classe enterprise di nuova generazione - Massimo Brignoli
L'architettura di classe enterprise di nuova generazione - Massimo BrignoliL'architettura di classe enterprise di nuova generazione - Massimo Brignoli
L'architettura di classe enterprise di nuova generazione - Massimo Brignoli
 
Automation in Post-Production — Boris Polyak for NATEXPO 2016
Automation in Post-Production — Boris Polyak for NATEXPO 2016Automation in Post-Production — Boris Polyak for NATEXPO 2016
Automation in Post-Production — Boris Polyak for NATEXPO 2016
 
Machine Learning Real Life Applications By Examples - Mario Cartia
Machine Learning Real Life Applications By Examples - Mario CartiaMachine Learning Real Life Applications By Examples - Mario Cartia
Machine Learning Real Life Applications By Examples - Mario Cartia
 
SoftLayer Storage Services Overview
SoftLayer Storage Services OverviewSoftLayer Storage Services Overview
SoftLayer Storage Services Overview
 
AWS for Media: Content in the Cloud, Miles Ward (Amazon Web Services) and Bha...
AWS for Media: Content in the Cloud, Miles Ward (Amazon Web Services) and Bha...AWS for Media: Content in the Cloud, Miles Ward (Amazon Web Services) and Bha...
AWS for Media: Content in the Cloud, Miles Ward (Amazon Web Services) and Bha...
 
Aspera on demand for AWS (S3 inc) overview
Aspera on demand for AWS (S3 inc) overviewAspera on demand for AWS (S3 inc) overview
Aspera on demand for AWS (S3 inc) overview
 
Polyglot Persistence e Big Data: tra innovazione e difficoltà su casi reali -...
Polyglot Persistence e Big Data: tra innovazione e difficoltà su casi reali -...Polyglot Persistence e Big Data: tra innovazione e difficoltà su casi reali -...
Polyglot Persistence e Big Data: tra innovazione e difficoltà su casi reali -...
 
The Top Skills That Can Get You Hired in 2017
The Top Skills That Can Get You Hired in 2017The Top Skills That Can Get You Hired in 2017
The Top Skills That Can Get You Hired in 2017
 

Semelhante a AWS re:Invent - Med305 Achieving consistently high throughput for very large data transfers with amazon s3 (aspera)

Delivering on the promise of the cloud for digital media, aspera on demand
Delivering on the promise of the cloud for digital media, aspera on demandDelivering on the promise of the cloud for digital media, aspera on demand
Delivering on the promise of the cloud for digital media, aspera on demand
Amazon Web Services
 
Hp aspera-big data cloud-v2
Hp aspera-big data cloud-v2Hp aspera-big data cloud-v2
Hp aspera-big data cloud-v2
dkumiaspera
 
Aspera bt-big-data-cloud
Aspera bt-big-data-cloudAspera bt-big-data-cloud
Aspera bt-big-data-cloud
dkumiaspera
 

Semelhante a AWS re:Invent - Med305 Achieving consistently high throughput for very large data transfers with amazon s3 (aspera) (20)

Delivering on the promise of the cloud for digital media, aspera on demand
Delivering on the promise of the cloud for digital media, aspera on demandDelivering on the promise of the cloud for digital media, aspera on demand
Delivering on the promise of the cloud for digital media, aspera on demand
 
Hp aspera-big data cloud-v2
Hp aspera-big data cloud-v2Hp aspera-big data cloud-v2
Hp aspera-big data cloud-v2
 
Aspera bt-big-data-cloud
Aspera bt-big-data-cloudAspera bt-big-data-cloud
Aspera bt-big-data-cloud
 
Intel aspera-medical-v1
Intel aspera-medical-v1Intel aspera-medical-v1
Intel aspera-medical-v1
 
IBM Aspera overview
IBM Aspera overview IBM Aspera overview
IBM Aspera overview
 
Shoot the Bird: Linear Broadcast Distribution on AWS by Usman Shakeel of Amaz...
Shoot the Bird: Linear Broadcast Distribution on AWS by Usman Shakeel of Amaz...Shoot the Bird: Linear Broadcast Distribution on AWS by Usman Shakeel of Amaz...
Shoot the Bird: Linear Broadcast Distribution on AWS by Usman Shakeel of Amaz...
 
Wowza Ultra-Low Latency Streaming
Wowza Ultra-Low Latency StreamingWowza Ultra-Low Latency Streaming
Wowza Ultra-Low Latency Streaming
 
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022
 
Nokta techpresentation
Nokta techpresentationNokta techpresentation
Nokta techpresentation
 
In search of the perfect IoT Stack - Scalable IoT Architectures with MQTT
In search of the perfect IoT Stack - Scalable IoT Architectures with MQTTIn search of the perfect IoT Stack - Scalable IoT Architectures with MQTT
In search of the perfect IoT Stack - Scalable IoT Architectures with MQTT
 
AWS re:Invent 2016: Accelerating the Transition to Broadcast and OTT Infrastr...
AWS re:Invent 2016: Accelerating the Transition to Broadcast and OTT Infrastr...AWS re:Invent 2016: Accelerating the Transition to Broadcast and OTT Infrastr...
AWS re:Invent 2016: Accelerating the Transition to Broadcast and OTT Infrastr...
 
Lightweight and scalable IoT Architectures with MQTT
Lightweight and scalable IoT Architectures with MQTTLightweight and scalable IoT Architectures with MQTT
Lightweight and scalable IoT Architectures with MQTT
 
Не так страшен терабит / Вячеслав Ольховченков (Integros)
Не так страшен терабит / Вячеслав Ольховченков (Integros)Не так страшен терабит / Вячеслав Ольховченков (Integros)
Не так страшен терабит / Вячеслав Ольховченков (Integros)
 
Maximize Application Performance and Bandwidth Efficiency with WAN Optimization
Maximize Application Performance and Bandwidth Efficiency with WAN OptimizationMaximize Application Performance and Bandwidth Efficiency with WAN Optimization
Maximize Application Performance and Bandwidth Efficiency with WAN Optimization
 
Qnap iei partners_day_2016 1108
Qnap iei partners_day_2016 1108Qnap iei partners_day_2016 1108
Qnap iei partners_day_2016 1108
 
Kafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid CloudKafka for Real-Time Replication between Edge and Hybrid Cloud
Kafka for Real-Time Replication between Edge and Hybrid Cloud
 
The Microsoft Cloud Partner
The Microsoft Cloud PartnerThe Microsoft Cloud Partner
The Microsoft Cloud Partner
 
From Device to Data Center to Insights
From Device to Data Center to InsightsFrom Device to Data Center to Insights
From Device to Data Center to Insights
 
Scale Your Load Balancer from 0 to 1 million TPS on Azure
Scale Your Load Balancer from 0 to 1 million TPS on AzureScale Your Load Balancer from 0 to 1 million TPS on Azure
Scale Your Load Balancer from 0 to 1 million TPS on Azure
 
[AWS LA Media & Entertainment Event 2015]: Shoot the Bird: Linear Broadcast o...
[AWS LA Media & Entertainment Event 2015]: Shoot the Bird: Linear Broadcast o...[AWS LA Media & Entertainment Event 2015]: Shoot the Bird: Linear Broadcast o...
[AWS LA Media & Entertainment Event 2015]: Shoot the Bird: Linear Broadcast o...
 

Último

Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
 

Último (20)

[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​Elevate Developer Efficiency & build GenAI Application with Amazon Q​
Elevate Developer Efficiency & build GenAI Application with Amazon Q​
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with Milvus
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 

AWS re:Invent - Med305 Achieving consistently high throughput for very large data transfers with amazon s3 (aspera)

  • 1. Michelle Munson - Co-Founder & CEO, Aspera Jay Migliaccio - Director of Cloud Technologies, Aspera Stéphane Houet – Product Manager, EVS Broadcast Equipment
  • 2. PRESENTERS Michelle Munson Co-founder and CEO Aspera michelle@asperasoft.com Jay Migliaccio Director Cloud Services, Aspera jay@asperasoft.com Stephane Houet Product Manager, EVS s.houet@evs.com AGENDA • Quick Intro to Aspera • Technology Challenges • Aspera Direct to Cloud Solution • Demos • FIFA Live Streaming Use Case • Q & A
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8. HTTP Master Database Mapping Object ID to Data Nodes File Data Replicas and Storing Metadata Key Value H(URL) R1, R2, R3
  • 9.
  • 10.
  • 12.
  • 13.
  • 14.
  • 15. TRANSFER DATA TO CLOUD OVER WAN EFFECTIVE THROUGHPUT Multi-part HTTP Typical internet conditions • 50–250ms latency & 0.1–3% packet loss 15 parallel http streams <10 to 100 Mbps depending on distance Aspera FASP Aspera FASP transfer over WAN to Cloud Up to 1Gbps * 10 TB transferred per 24 hours * Per EC2 Extra Large Instance -independent of distance
  • 16. LOCATION AND AVAILABLE BANDWIDTH AWS ENHANCED UPLOADER ASPERA FASP Montreal to AWS East • 100 Mbps Shared Internet Connection 30 minutes (7-10 Mbps) 3.7 minutes (80 Mbps) 9X Speed Up Rackspace in Dallas to AWS East • 600 Mbps Shared Internet Connection 7.5 minutes (38 Mbps) 0.5 minutes (600 Mbps) 15X Speed Up Other pains … “Enhanced Bucket Uploader” requires java applet, very large transfers time out, no good resume for interrupted transfers, no downloads
  • 17. EFFECTIVE THROUGHPUT & TRANSFER TIME FOR 4.4 GB/15691 FILES (AVERAGE SIZE 300KB) LOCATION AND AVAILABLE BANDWIDTH AWS HTTP MULTIPART ASPERA ASCP New York to AWS East Coast • 1 Gbps Shared Connection 334 seconds (113 Mbps) 107 seconds (353 Mbps) 3.3X Speed Up New York to AWS West Coast • 1 Gbps Shared Connection 8.7 GB in 1032 seconds (36 Mbps) 8.7 GB in 110 seconds (353 Mbps) 9.4 X Speed Up EFFECTIVE THROUGHPUT & TRANSFER TIME FOR 8.7 GB/18,995 FILES (AVERAGE SIZE 9.6MB) LOCATION AND AVAILABLE BANDWIDTH AWS HTTP MULTIPART ASPERA ASCP New York to AWS East Coast • 1 Gbps Shared Connection 477 seconds (156 Mbps) 178 seconds (420 Mbps) 2.7 X Speed Up New York to AWS West Coast • 1 Gbps Shared Connection 967 seconds (77 Mbps) 177 seconds (420 Mbps) 5.4 X Speed Up
  • 18.
  • 19.
  • 20.
  • 21. – Maximum speed single stream transfer – Support for large files and directory sizes in a single transfer – Network and disk congestion control provides automatic adaptation of transmission speed to avoid congestion and overdrive – Automatic retry and checkpoint resume of any transfer from point of interruption – Built in over-the-wire encryption and encryption-at-rest (AES 128) – Support for authenticated Aspera docroots using private cloud credentials and platform-specific role based access control including Amazon IAMS. – Seamless fallback to HTTP(s) in restricted network environments – Concurrent transfer support scaling up to ~50 concurrent transfers per VM instance
  • 22. New Clients connect to “available” pool Existing client transfers Utilization > high w/m Available Pool Console • Collect / aggregate transfer data • Transfer activity / reporting (UI, API) Shares • User management • Storage access control KEY COMPONENTS • Cluster Manager for Auto-scale and Scaled DB • Console Management UI + Reporting API • Enhanced Client for Shares Authorizations • Unified Access to Files/Directories (Browser, GUI, Commend Line, SDK) Scaling Parameters • Min/max number of t/s • Utilization low/high watermark • Min number of t/s in “available” pool • Min number of idle t/s in ”available” pool Management and Reporting Cluster Manager • Monitor cluster nodes • Determine eligibility for transfer scale up / down • Create / remove db with replicas • Add / remove node Scale DB Persistence Layer
  • 23.
  • 24.
  • 25. .mp2ts HLS adaptive bitrate FASPStream
  • 26. • Near Live experiences have highly bursty processing and distribution requirements • Transcoding alone is expected to generate 100s of varieties of bitrates and formats for a multitude of target devices • Audiences peak to millions of concurrent streams and die off shortly post event • Near “Zero Delay” in the video experience is expected • “Second screen” depends on near instant access / instant replay, which requires reducing • Linear transcoding approaches simply can not meet demand (and are too expensive for short term use!) • Parallel, “cloud” architectures are essential • Investing in on premise bandwidth for distribution is also impractical • Millions of streams equals terabits per second
  • 27. FASP Scale Out High-Speed Transfer by Aspera Scale Out Transcoding by On Demand Elemental Multi-screen capture and distribution by EVS
  • 28. Belgian company +90% market share of sports OB trucks 21 offices +500 employees (+50% in R&D)
  • 29.
  • 30. With the kind permission of HBS
  • 31.
  • 32. Live Streaming : REAL TIME CONSTRAINT! 6 feeds @ 10 Mbps = 60 Mbps X 2 games at the same time X 2 for safety WE NEED A SOLUTION ! 240 Mbps VOD Multicam Near-live replays : Up to 24 clips @ 10 Mbps = 240 Mbps X 2 games at the same time 480 Mbps Maximum Throughput (bps) = TCP-Window-Size (b) / Latency (s) (65535 * 8) / 0.2 s = 2621400 bps = 2.62 Mbps
  • 33.
  • 34. 6 Live Streams HLS streaming of 6 HD streams to tablets & mobiles per match +20 Replay cameras On-demand replays of selected events from up to 20+ cameras on the field +4000 VoD elements Exclusive on-demand multimedia exclusive edits
  • 35. FASP Scale Out High-Speed Transfer by Aspera Scale Out Transcoding by On Demand Elemental Multi-screen capture and distribution by EVS
  • 36. + 27 TB of video data Key Metrics Total over 62 games Average per Game Transfer Time (in hours) 13,857 216 Number of GB Transferred 27,237 426 Number of Transfers 14,073 220 Number of Files Transferred 2,706,922 42,296 < 14,000 hrs video transferred 200 ms of latency over WAN 10% packet loss over WAN
  • 37. Live Streams 660,000 Minutes Transcoded Output x 4.3 = 2.8 Million Minutes Delivered Streams x 321 = 15 Million Hours 35 Million Unique Viewers