Moving Your Media Supply Chain to the AWS Cloud
•
4 gostaram•5,443 visualizações
Usman Shakeel, AWS Tech Lead's presentation to the Media Supply Chain track at the Media & Entertainment Cloud Symposium on November 4, 2016
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (6)
Semelhante a Moving Your Media Supply Chain to the AWS Cloud
Semelhante a Moving Your Media Supply Chain to the AWS Cloud (20)
Mais de Amazon Web Services
Mais de Amazon Web Services (20)
Último
Último (20)
Moving Your Media Supply Chain to the AWS Cloud
- 4. Media Supply Chains Playout & Distribution Production/Post Production B2B/Cinematic Delivery DAM & Archive Digital Supply Chain Publishing OTT Content Acquisition Ingestion Manual QC Auto-QC Metadata Extraction Transcoding Delivery Packaging Clips/Edits Complex workflow/State management Bursty sub-workflow tasks (transcoding/auto-QC, Packaging) Elastic Licensing Content movement between workflow tasks (a common repository) Different delivery formats for each affiliate and downstream Manual tasks – (content movement or interactivity) CMS/DAM Normalization
- 6. Ingest Store Edit/Manual Tasks Process PUSH OR PULL CONTENT & ARTIFACTS ACCESS SOURCE FILES LOCATED IN S3 CREATE, EDIT, QC & COLLABORATE IN THE CLOUD SCALE OUT ON ELASTIC CAPACITY FOR PROCESSING (Transcoding, QC, Packaging and distribution) Media production and delivery companies are leveraging AWS to accelerate their media supply chains, ingestion, storage, editing/manual QC of content and processing with highly scalable cloud computing and storage. Using Cloud for Media Supply Chains
- 7. Ingest PUSH OR PULL CONTENT & ARTIFACTS AWS Services • Amazon S3 Transfer Acceleration: Fast data transfers • AWS Direct Connect: Dedicated network connectivity • AWS Snowball: Petabyte scale data transfer • Amazon Cloudfront: Global content delivery network • Amazon Storage Gateway: On-prem to cloud storage connectivity • Amazon Marketplace: One-click partner software deployments • Elemental: Content contribution from on-premises encoders Partners Media Supply Chains
- 8. AWS Services • Amazon S3/SIA: Highly scalable, durable object storage • AWS Storage Gateway: On-prem to cloud storage connectivity • Amazon Glacier: Secure, durable, petabyte-scale data archival • Amazon EFS: Simple, elastic, scalable file system • Amazon EBS: Persistent block storage Partners Store ACCESS SOURCE FILES LOCATED IN S3 Media Supply Chains
- 9. Edit/Manual Tasks CREATE, EDIT, Clips, Manual QC & COLLABORATE IN THE CLOUD AWS Services • Amazon EC2: Scalable CPU/GPU computing capacity • Amazon Workspaces: Virtual desktops in the cloud • Amazon Appstream: Remote Application Streaming • Amazon EFS: Simple, elastic, scalable file system • Amazon EBS: Persistent block storage • Amazon S3: Highly scalable, durable object storage Partners Media Supply Chains
- 10. Process SCALE OUT ON ELASTIC CAPACITY FOR PROCESSING AWS Services • Amazon EC2: Scalable CPU/GPU/Spot computing capacity • Amazon EFS: Simple, elastic, scalable file system • Amazon EBS: Persistent block storage • Amazon S3: Highly scalable, durable object storage • Elemental Cloud: On-demand, scalable video processing Partners Media Supply Chains
- 11. AWS Services • Amazon S3/SIA: Highly scalable, durable object storage • AWS Storage Gateway: On-prem to cloud storage connectivity • Amazon Glacier: Secure, durable, petabyte-scale data archival • Amazon EFS: Simple, elastic, scalable file system • Amazon EBS: Persistent block storage Partners Asset Management Metadata for Assets and supply chain workflow Media Supply Chains
- 12. INFRASTRUCTURE
- 14. Reserved Make a low, one-time payment and receive a significant discount on the hourly charge For committed utilization On-Demand Pay for compute capacity by the hour with no long-term commitments For spiky workloads, or to define needs Spot Bid for unused capacity at a Spot Price - fluctuates based on supply & demand For time-insensitive or transient workloads Consumption Models
- 15. AWS Storage Services Maturity Amazon EFS File Amazon EBS Amazon EC2 Instance Store Block Amazon S3 Amazon Glacier Object Data Transfer (Ingest/Egress) AWS Direct Connect AWS Snowball ISV Connectors Amazon Kinesis Firehose S3 VPC EndPoint AWS Storage Gateway S3 – Infrequent Access Events S3 Event Notifications S3 Transfer Acceleration
- 17. Availability Zone A Private subnet Private subnet AWS region Virtual Private Gateway Content Value-add Service Content Value-add Service Availability Zone B Amazon S3 VPC VPN connection VPC Endpoints No IGW No NAT No public IPs Free Robust access control Customer network Access Control with S3 VPC Endpoint
- 18. corporate data center AWS cloud users Content Servers disk tape storage Processing Layer Amazon S3 Amazon EBS Amazon Glacier KMS/ HSMClient side encryption role IAM Encrypted Content role Direct Connect S3 VPC Endpoint Hybrid Media Network
- 20. EBS Volumes SSD Based HDD Based gp2 General Purpose SSD io1 Provisioned IOPS SSD st1 Throughput Optimized HDD sc1 Cold HDD Baseline: 3IOPS per GB up to 10,000 Burst: 3000 IOPS (for volumes up to 1TB) Throughput: 160 MB/s Latency: Single digit ms Capacity: 1GB to 16TB Usage Boot Volumes Bursty workloads (Rendering) Databases (Transactional) Low latency operations (Streaming) Baseline: 100IOPS up to 20,000 Throughput: 320 MB/s Latency: Single digit ms Capacity: 4GB to 16TB Usage Sustained IOPS Critical Applications with IOPS Guarantee (no burst) Baseline: 40MB/s per TB up to 500MB/s Burst: 250MB/s per TB up to 500MB.s Capacity: 500GB to 16TB Usage Large Block, high throughput sequential Transcoding, QC, Packaging Rendering Baseline: 12MB/s per TB up to 192MB/s Burst: 80MB/s per TB up to 250MB.s Capacity: 500GB to 16TB Usage Sequential Logging, Active Archive/Backup
- 21. Object store as Processing tier Partition 1 Partition 2 Partition n An S3 Backend Storage Auto-scaled/HA Front-end TCP Distributed Key-map Optimizations Replication across multiple Availability Zones No FS – everything GET/PUT/DELETE • Write once – Read many • Globally unique namespaces (object names unique within a bucket) Requests Per Second (>300) • Partitioning • Random Prefix Puts vs. Gets • CDN – GET heavy workloads Network Throughput • TCP Window Scaling • TCP Selective Acknowledgements • S3 Session Usage • Multi-part Uploads • Buffered Writes • Range GETs Consistency/Commit Latency • Read after Write • Objects replicated across multiple AZs • Not suitable today for workloads with <MS latency/IO requirement Client(s) Client-side bandwidth
- 22. Elastic File System (EFS) • Designed to support Peta-Byte scale file systems • Throughput scales linearly to storage – Burst: 100MB/sec per TB of storage for 12 hours each day – Baseline: 50MB/sec per TB of storage continuously A 10 TB filesystem can burst up to 1GB/s for 12 hours each day and drive500MB/sec aggregate read / write throughput continuously. • Same latency spec across each AZ • Thousands of concurrent NFS connections • Works great for Large I/O sizes • Pay for only what you use not what you provision • Managed with multi-copy durability Render Farms Big data analytics Transcode Farms Home directories Content management Web serving Metadata- intensive jobs High throughput and parallel I/O Low latency and serial I/O
- 23. Workload Specific Storage Object Store (Amazon S3) Block Store (Amazon EBS) Shared FS (Amazon EFS) Performance Per Operation Latency Variable Lowest, Consistent Low Consistent Throughput Scale Multiple GBs per sec Single GBs per sec Multiple GBs per sec Data Availability/Durability Across Multi-AZ Redundant within an AZ Across Multi-AZ Characteristics Access 1-1000’s of Clients A single EC2 instance 1-1000’s of Clients Use Cases Active Archive and lifecycle to Cold Archive Media Processing (Transcoding, rendering, QC, Farms) depending on Application conformity Databases Boot Volumes Media Processing (Transcoding, rendering, QC, Farms) Singleton Streaming Data Analytics Media Processing (Transcoding, rendering, QC, Farms) Distributed Content Management Web Serving/Publishing
- 24. SOLUTIONS
- 26. AWS Import/ Export S3 multi-part API S3 as backend storage for Content Files accessible to other processing tasks Amazon Elastic Transcoder S3 Notification Trigger a Lambda Function to Start a transcoding job Ingest S3 Notification Lambda function to generate a signed URL to share the file Update CMS or Metadata Serverless Media Supply Chain (concept)
- 27. § Expand compute capacity of Manager node(s) § 100ms billing & short run-time § Ideal for chunk storage, pre- & post- processing § Don’t forget binary encapsulation! https://github.com/binoculars/aws-lambda-ffmpeg Lambda Pre and Post Processing
- 28. A high-level Media Supply Chain in the cloud (concept) EC2, ETS can request the data-key on behalf of customer Provide CPK for S3 encryption at rest AWS Import/ Export S3 multi-part API Amazon S3 For Source, Renditions, Metadata, Sidecar Files Ingest AWS Elastic Beanstalk CloudFront Distribution DynamoDB CloudSearch S3 Notification Content Processing Amazon SWF, SQS, SNS and AWS Lambda can be used to orchestrate the processing workflow tasks CloudFront Distribution User Logs Amazon DynamoDB Amazon Redshift Users Connected Devices Analytics and Recommendations Engine Amazon EMR Amazon DataPipeline Elastic Transcoder EC2 ASG Rendition Workers EC2 ASG Metadata Workers Metadata Proc queueRenditions Queue Amazon Elastic Transcoder Rights Management EC2 ASGELB DRM Wrap Encrypt Package Watermark QC EC2 ASG Amazon SNS Topic DAM Web Service Personalization Recommendations EC2 ASG RDS Front End Web Application DAM Catalog Content Archive Amazon Glacier Key Management Service Content owner provides the master key BI & Reporting Users Web Browser Content Owner
- 30. Cloud Based Editing • NVIDIA GPU based EC2 instances • Teradici PCoIP • Amazon AppStream/Nice DCV • Amazon Workspaces Artist applications Artist workstation (Dumb Client) Remote Application running on a G2 instance G2
- 31. Thank You!