Scalable Media Workflows in the Cloud
•Transferir como PPTX, PDF•
19 gostaram•39,068 visualizações
This session will cover the approaches for a cloud-based workflow: media ingest, storage, processing and delivery scenarios on the AWS cloud. We will cover solutions for high speed file transfer, cloud-based transcoding, tiered storage, content processing, application deployment and global low-latency delivery, as well as the orchestration and management of the entire media workflow.
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (19)
Semelhante a Scalable Media Workflows in the Cloud
Semelhante a Scalable Media Workflows in the Cloud (20)
Mais de Amazon Web Services
Mais de Amazon Web Services (20)
Último
Último (20)
Scalable Media Workflows in the Cloud
- 1. Scalable Media Workflows in the Cloud Bhavik Vyas - AWS Stephane Houet - EVS Media &Entertainment Product Manager Partner Ecosystem Manager C-Cast © 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
- 2. Media Workflows are Complex Featurettes Interviews 2D Movie 3D Movie Archive Materials Stills Archive Networks Theatrical DVD/BD Online MSOs Mobile Apps Process Store Deliver
- 3. Tools Need to Scale with Media …
- 4. Managed IT Services Secure High Availability Global Scale
- 5. A Typical Media Workflow Ingest Index Process QC Package Protect Auth. Track Deliver Orchestrate Store
- 6. Where AWS Fits Analytics and Monetization Amazon Web Services Ingest Index Process Package Protect QC Auth. Track Playback Media Asset Management
- 7. Mapping the Workflow to AWS Services Ingest Store Process/ Manage Deliver AWS Direct Connect Elastic Load Balancing AWS Import/ Export Amazon S3 AWS Storage Gateway Amazon EBS Amazon Glacier CloudFront Amazon Amazon Elastic Transcoder CloudSearch Amazon SQS Amazon SNS Amazon SWF Amazon EC2 Amazon EMR DynamoDB Amazon VPC Amazon RDS Elasti Cache Amazon Redshift AMI Route 53 Amazon EC2 Amazon EC2 Amazon Workspaces Amazon Kinesis Elastic Load Balancing Elastic Beanstalk CloudWatch CloudFormation
- 8. AWS SDKs Java Python (boto) PHP .NET Ruby Node.js iOS Android AWS Toolkit for Visual Studio AWS Toolkit for Eclipse AWS Tools for Windows PowerShell AWS CLI JavaScript
- 9. AWS Media Partners: Powering the Workflow INGEST STORE PROCESS MANAGE SECURE CREATE DELIVER INTEGRATE MONETIZE
- 10. Media S/W in the AWS Marketplace AWS Marketplace: Online Software Store • Find, research, try & buy software • 1-Click launch – application running minutes • S/W Billing fully integrated into AWS account • 1700 products across 25 categories Developer Tool Categories Include • Media Servers • Transcoding • Media Publishing • Digital Asset Management • Image Resizing
- 12. Media Ingest for AWS AWS Import/ Export Amazon S3 (multi-part Upload) AWS Storage Gateway Service AWS Direct Connect
- 13. Media Storage on AWS On-premises Data Center AWS Cloud Amazon Glacier Gateway Appliance/ AWS Storage Gateway Amazon S3 Block File Archive Backup Disaster Recovery Amazon EBS
- 14. Choosing the Right Storage Block (CIFS/NFS) Object Storage - http(s) Object Storage - http(s) Amazon S3 Amazon Glacier >Durability >>Scale >>>Cost EBS Elastic Block Storage
- 15. EBS Options: • Three types: – General Purpose (SSD) volumes • Ideal for a broad range of use cases such as boot volumes, small and medium size databases, and development and test – Provisioned IOPS (SSD) volumes • You specify the I/O performance, up to 4000 IOPS per volume – Magnetic volumes • Ideal for workloads where data is accessed infrequently, and applications where the lowest storage cost is important. • Encrypted Amazon EBS volumes to meet a wide range of data-at-rest encryption requirements Block (CIFS/NFS) EBS Elastic Block Storage
- 16. Reducing Object Storage Costs <$0.03/GB Amazon S3 <$0.01/GB Amazon Glacier Single API access (regardless of the storage class) > Durability >> Scale <$0.024/GB >>> Cost Amazon S3 (RRS)
- 17. S3 & Working with Large Files S3 Bucket S3 SDK AWS S3 S3 multi-part Upload Files corporate data center S3 Object <= 5TB
- 18. S3 Encryption Options 1. Sever-Side Encryption (SSE) – Provides an integrated solution where Amazon handles key management 2. SSE-Customer (SSE-C) – Perform the encryption and decryption of your objects while retaining control of the keys used to encrypt objects. 3. Client library (e.g. Amazon S3 Encryption Client) – Objects encrypted before they are sent to Amazon S3 for storage.
- 19. Media Processing Amazon S3 Instance Store EC2 Instance AMI EBS
- 20. Ensure Flexibility: EBS Backed Instances AMI Instance Store EBS EC2 Instance SIZE INSTANCE BY APPLICATION NEED
- 21. Size AWS Instance Type to Application I/O EC2 Instance Type
- 22. Scale Capacity Up & Down as Needed Traditional IT capacity Elastic CLOUD capacity Your needs Time Capacity
- 23. Use What You Need, When You Need To Capacity Time 1000’s of transcoders Launch of new device
- 24. Amazon EC2 Instance Types m1.small new existing Entry into GPU m1.xlarge m1.large m1.small space m2.2xlarge m2.4xlarge c1.medium c1.xlarge m1.xlarge m1.large m1.small G2 Instances cc2.8xlarge cc1.4xlarge cg1.4xlarge t1.micro m2.xlarge m2.2xlarge m2.4xlarge c1.medium c1.xlarge m1.xlarge m1.large m1.small hs1.8xlarge m3.xlarge m3.2xlarge hi1.4xlarge m1.medium cc2.8xlarge cc1.4xlarge cg1.4xlarge t1.micro m2.xlarge m2.2xlarge m2.4xlarge c1.medium c1.xlarge m1.xlarge m1.large m1.small cc1.4xlarge cg1.4xlarge t1.micro m2.xlarge m2.2xlarge m2.4xlarge c1.medium c1.xlarge m1.xlarge m1.large m1.small c3.large c3.xlarge c3.2xlarge c3.4xlarge c3.8xlarge i2.large i2.xlarge i2.2xlarge i2.4xlarge i2.8xlarge g2.2xlarge cr1.8xlarge hs1.8xlarge m3.xlarge m3.2xlarge hi1.4xlarge m1.medium cc2.8xlarge cc1.4xlarge cg1.4xlarge t1.micro m2.xlarge m2.2xlarge m2.4xlarge c1.medium c1.xlarge m1.xlarge m1.large m1.small c1.medium c1.xlarge m1.xlarge m1.large m1.small 2006 2007 2008 2009 2010 2011 2012 2013
- 25. 2014 EC2 Instance Highlights • HI1 – 120,000 random read IOPS on SSDs • I2 – Next generation I/O optimized (350,000+ random read IOPS) • M3 – Next generation general purpose • HS1 – 48 TB of storage per instance • CR1 – 244 GiB of RAM &Intel Xeon E5 2670 (Sandy Bridge) CPUs • C3 – 2.8 GHz Intel Xeon E5-2680v2 (Ivy Bridge) CPUs (SSD) • R3 – Next generation memory optimized
- 26. EC2 GPU Instances (G2) g2.2xlarge 1 NVIDIA GK104 GPU (Kepler / GRID) 2.6 GHz Sandy Bridge CPU w/ Turbo enabled 8 vCPUs, 15 GiB of RAM 60GB SSD storage Frame Capture and Encoding APIs
- 27. On-demand instances Unix/Linux instances start at $0.02/hour Pay as you go for compute power Low cost and flexibility Pay only for what you use, no up-front commitments or long-term contracts Use Cases: Applications with short term, spiky, or unpredictable workloads; Application development or testing Reserved instances 1- or 3-year terms Pay low up-front fee, receive significant hourly discount Low Cost / Predictability Helps ensure compute capacity is available when needed Use Cases: Applications with steady state or predictable usage Applications that require reserved capacity, including disaster recovery Spot instances Bid on unused EC2 capacity Spot Price based on supply/demand, determined automatically Cost / Large Scale, dynamic workload handling Use Cases: Applications with flexible start and end times Applications only feasible at very low compute prices EC2 Instance Options
- 28. Managed Database Services Amazon RDS Amazon DynamoDB Managed relational database service Managed NoSQL database service Amazon ElastiCache In-Memory Caching Service
- 29. Workflow Building Blocks Amazon Simple Queue Service (SQS) Amazon SWF Amazon Simple Email Service (SES) Amazon Simple Notification Service (SNS) Amazon CloudSearch CloudWatch AWS CloudTrail Amazon ElastiCache Amazon Elastic MapReduce Elastic Load Balancing Amazon Relational Database Store Deliver Service (RDS) Process
- 30. AWS Media Delivery Processed Media Files HTTP/HTTPS Amazon S3 Streaming or Distribution Amazon CloudFront
- 31. ABR VOD & Live Streaming
- 32. Content: Consumption & Reporting Amazon CloudFront Access Logs Amazon S3 EMR Amazon Redshift Business Smarts
- 33. Mapping the Workflow to AWS Services Ingest Store Process/ Manage Deliver AWS Direct Connect Elastic Load Balancing AWS Import/ Export Amazon S3 AWS Storage Gateway Amazon EBS Amazon Glacier CloudFront Amazon Amazon Elastic Transcoder CloudSearch Amazon SQS Amazon SNS Amazon SWF Amazon EC2 Amazon EMR DynamoDB Amazon VPC Amazon RDS Elasti Cache Amazon Redshift AMI Route 53 Amazon EC2 Amazon EC2 Amazon Workspaces Amazon Kinesis Elastic Load Balancing Elastic Beanstalk CloudWatch AWS CloudFormation
- 34. AWS Media Partners: Powering the Workflow INGEST STORE PROCESS MANAGE SECURE CREATE DELIVER INTEGRATE MONETIZE
- 35. Media Workflows in the Cloud: Customer Examples
- 36. in the Cloud AKA “Video Factory”
- 37. Video Factory – Workflow SDI Broadcast Video Feed x 24 Broadcast Encoder Live Ingest Logic Playout Data Feed Amazon Elastic Transcoder Elemental Cloud DRM QC Editorial Clipping MAM Amazon S3 Mezzanine Time Addressable Media Store Amazon S3 Distribution Renditions RTP Chunker Transcode Abstraction Layer SMPTE Timecode Mezzanine Video Capture Mezzanine Playout Video Transcoded Video Metadata
- 38. • The UK’s biggest video & audio on-demand service – And it’s free! • Over 7 million requests every day – ~2% of overall consumption of BBC output • Over 500 unique hours of content every week – Available immediately after broadcast, for at least 7 days • Available on over 1000 devices including – PC, iOS, Android, Windows Phone, Smart TVs, Cable Boxes… • Both streaming and download (iOS, Android, PC) • > 20 million app downloads Sources: BBC iPlayer Performance Pack August 2013 http://www.bbc.co.uk/blogs/internet/posts/Video-Factory
- 39. POWERING MULTIMEDIA CONTENT DELIVERY AT 2014 FIFA WORLD CUP BRAZIL™ www.evs.com
- 40. FIFA WORLD CUP 2014 June 12 - July 13 2014 - 12 Stadiums - 64 Games IBC1 in Rio de Janeiro EVS appointed for multilateral production - On site2 live production - IBC file-based video management - Multimedia production and distribution EVS SPORTS Notes: 1. The International Broadcast Centre (IBC) is a temporary hub for broadcasters during major sport events. 2. Remote broadcast television studio where the event is produced and then beamed to broadcasters via satellite.
- 41. EVS ENABLED A NEVER-SEEN-BEFORE MULTIMEDIA EXPERIENCE 6 Live Streams HLS streaming of 6 HD streams to tablets & mobiles per match 24 Replay cameras On demand replay of selected events from any of the 20+ cameras on the field. +4000 VoD elements On demand multimedia exclusive edits.
- 42. EVS SPOR
- 44. Max of 12 streams at 10Mbps each. Max of 36 streams at 3,5-10Mbps 9 HLS encoding, from 0.4 to 3.5Mbps + 2 thumbs 1 RTMP EVS C-CAST LIVE STREAMING WORKFLOW
- 45. EVS C-CAST VOD WORKFLOW 9 HLS encoding, from 0.4 to 3.5Mbps + 2 thumbs 1 RTMP
- 46. EVS C-CAST MANAGES ALL FROM PRODUCTION TO VIEWERS Cloud Platform Linking mobile trucks, studios and viewers Content selection and approval process Metadata editing station Automatic translations in 20+ languages Automatic HD backup on top Workflow engine, including - VoD file transfers and transcoding management - Live streaming activation and transcoding management Output: API mode or white label embedded EVS SPORTS
- 47. MULTIMEDIA PRODUCTION HBS OVERVIEW MAP
- 48. MULTIMEDIA PRODUCTION IN THE AWS CLOUD Aspera for transfers into S3 Store and deliver data using S3 Create image thumbnails (EC2, SQS & S3) Transcode VoD files using Zencoder Transcode Live using Elemental Cloud - Up to 128 concurrent live streams encoded Operator GUI for Rio and international sites - S3, Cloudfront, EC2 and RDS Manage all workflows at IBC and Venue - EC2, SQS & RDS 1Gbps Direct Connect from Rio to Dublin
- 49. EVS SPORTS 6 MULTILATERAL STREAMS 17 INDIVIDUAL MRL STREAM 4 INDIVIDUAL COMMENTARY 243 DIFFERENT LIVE STREAMS /GAME 2,799,360 MINUTES OF ENCODED STREAMS LIVE STREAMING
- 50. MULTI-ANGLE CLIPS On average 60 CLIPS RECORDED PER MATCH, overall UP TO 20,000 MULTILATERAL CLIPS throughout the tournament Roughly a third of clips from multicam with 14 DIFFERENT ANGLES on average
- 51. COMBINED DATA FOR FIFA WORLD CUP WEB PLAYER AND APP AFTER 64 MATCHES FOR ALL MARKETS TOTAL HOURS WATCHED: >15 MILLION UNIQUE USERS: >35 MILLION Available in more than 10 TERRITORIES Peak ACTIVE CONCURRENT USERS 500,000
- 52. FIN & THANK YOU! Hall #3.B16 Hall #7.G30 Hall #4.B75 Hall #8.B90 Hall #5.B20
Notas do Editor
- Start by talking about media workflows. Main point is there are many workflows. Use media workflows to go from what’s on the the left to what’s on the right. Steps themselves are generally pretty straightforward. Industry trends that are making workflows more complex: More content: at the pro end look at all the content on the left. On the consumer end, everyone is carrying around a 1080p camcorder. And the more content there is, the greater the opportunity to monetize it. Bigger content: the industry is moving to some combination of more pixels, faster pixels and better pixels. More pixels: 4K and beyond (4x pixels compared to 1080p). Faster pixels: higher frame rates. 48fps is 2x current cinema frame rate. Better pixels: higher dynamic range and brighter pixels, increase bit-depth. More processing: the amount of processing going up not down. At the high end, whether it is a commercial, a TV show or a movie, most shows contain visual effects. Even in corporate video, color correction is becoming a standard part of the workflow. And at the consumer level, all those Instagram like filters require processing. More output formats: not just renditions based on devices but also versions. ? One senior industry figure recently told me that a piece of finished content will have been converted 1000 times! So all of these trends have an impact on workflows especially when you factor in constrained budgets and timeframes.
- To give you context for what follows in Phil’s session, I thought I’d cover where AWS fits and then some approaches we’ve seen for doing media processing at scale in the cloud. As you know, AWS provides infrastructure services: compute, networking, database, storage and delivery and so on. We also provide application services and deployment and management services. Using these services, as your “software defined datacenter”, you can build media processing workflows. Typical operations in a media workflow would run on top of the AWS services. These operations could be provided by software that you’ve developed or they might be from another vendor like Aspera for ingest or Tektronix for video QC. On top of all that you’d have media applications – perhaps an Online Video Platform, a production management application, a digital dailies system or visual effects. So that’s where AWS fits. Now let’s look at some approaches for doing media processing on AWS.
- To give you context for what follows in Phil’s session, I thought I’d cover where AWS fits and then some approaches we’ve seen for doing media processing at scale in the cloud. As you know, AWS provides infrastructure services: compute, networking, database, storage and delivery and so on. We also provide application services and deployment and management services. Using these services, as your “software defined datacenter”, you can build media processing workflows. Typical operations in a media workflow would run on top of the AWS services. These operations could be provided by software that you’ve developed or they might be from another vendor like Aspera for ingest or Tektronix for video QC. On top of all that you’d have media applications – perhaps an Online Video Platform, a production management application, a digital dailies system or visual effects. So that’s where AWS fits. Now let’s look at some approaches for doing media processing on AWS.
- [Mention that Sony MCS went through the full MPAA audit]
- Amazon S3 Reduced Redundancy 99.99% durability vs. 99.999999999% Up to 20% savings Great for everything that is easy to reproduce Amazon Glacier Same durability as S3 3 to 5 hours restore time Up to 89% savings Great for archiving, long-term backups and old data
- Amazon EC2 offers a broad selection of virtual server configurations, or instance types, to satisfy the resource needs of a diversity of workloads. And we have been iterating quickly on behalf of our customers. These can be purchased on-demand, meaning you only pay for the hours of time that you use them. This has the huge advantage of turning capital expenditures into variable costs, one reason why AWS has been so popular amongst small businesses and enterprises. And when you look as our instance types, each has a different focus – memory, cpu, storage, IOPS, and GPUs each of which align with different use cases. We got into the GPU family 3 years ago to provide customers on-demand access to the massive parallel processing capability of GPUs. For workloads that fit into the GPU co-processor programming model, the performance boost can be huge.
- After launching our first GPU instance – which supported GPU compute programming models only - we started getting customer feedback about the Cluster GPU instances. For some, they were too expensive. For others, they were overkill. As programming two GPUs was more cumbersome at the time. And the last and most prominent piece of feedback was the desire for graphics API. As an interesting side, some customers found ways to run low versions of OpenGL, providing us demonstrated demand for the graphics APIs. We knew we had to evolve the instance family. Roughly four months ago we launched the G2 instances type, which we believe accommodates the customer feedback we’ve received. It has newer GPU hardware from NVIDIA, a single GPU per instance, supports graphics APIs in addition to GPU compute programming models, and it comes at a significantly reduced price of $0.65 per hour, Linux, on-demand in our Northern Virginia and Oregon datacenters.
- Cost is the conversation starter when it comes to cloud. There are many pieces to cost conversation when it comes to AWS and your own infrastructure. The first advantage you get in the cloud is that you don’t have to lay out capital expense for hardware and infrastructure before you know the demand. In essence you convert your capital expense into variable expense. And then that variable expense on AWS is lower than what most companies can do on their own because AWS runs at a massive scale and we pass that scale to our customers in the form of lower pricing. There are multiple pricing models in AWS, so you can optimize your spend depending on what your workloads requirements are. And the more you use AWS, the less your costs are. We have tiered pricing and for customers doing large data center migrations, we have negotiated custom pricing to make their transitions cost-effective.
- NOTES: This architecture shows how to deliver content in popular formats including: Apple’s HTTP Live Streaming (HLS) Adobe’s HTTP Dynamic Streaming (HDS) Microsoft’s Smooth Streaming This diagram doesn’t explicitly call out the upcoming MPEG-DASH format, but it would be very similar since it is based on the 3 formats shown Transcoding is outside the scope of this diagram. There are many transcoding options available For live streaming, the transcoder will push the content to the segmenters via different protocols depending on the type of ABR streaming: For HLS, MPEG2 Transport Streams are used to deliver to the Apple Stream Segmenter tool For HDS, RTMP is used to deliver to the Adobe Live Packager tool For Smooth Streaming, the Smooth Streaming Live Protocol is used After encoding, but prior to packaging, you may also choose to encrypt the content using one of many popular DRM technologies, which are also specific to the various ABR technologies: HDS uses Adobe Access or Protected HDS HLS uses HLS encryption Smooth Streaming uses PlayReady It should also be noted that both Adobe and Microsoft added the ability to deliver HLS formats to Adobe Media Server and IIS Media Services, respectively
- [Mention that Sony MCS went through the full MPAA audit]
- Introduction: Phil Cluff Principal Software Engineer & Team Lead @ BBC Media Services Been with BBC for 3 ½ years, focused in Transcode architectures, Message Orientated Middleware & Reliable, Distributed systems in the cloud! I’m going to talk to you about BBC iPlayer and our journey into the Cloud.
- Be really clear on Mezzanine definition since next 4 slides depend on it. Mention Mez video capture is classic broadcast technologies. Make note of the “Time addressable media store”
- Hopefully you’ve all heard of the BBC, but you may not have all heard of iPlayer. So What is BBC iPlayer? UK online population is about 40m which is the size of the state of California.
- Warning: confidential content