Netflix and Containers: Not A Stranger Thing
Denunciar
Compartilhar
•5 gostaram•2,310 visualizações
Netflix and Containers: Not A Stranger Thing
•5 gostaram•2,310 visualizações
Denunciar
Compartilhar
Baixar para ler offline
Customers from over all over the world streamed Forty Two Billion hours of Netflix content last year. The Netflix streaming service had been powered by the Amazon cloud with virtual machines for over five years, blazing a trail for similar architectures. In the last year, it invested in containers for batch-style jobs and service-style applications. Andrew Spyker will explain the potential containers have to help Netflix create a more productive development experience while simultaneously deepening its control over resource management. Join Andrew to see why Netflix is moving forward with containers, how it can leverage its existing operational machinery, and how it’s running containers with a similar guarantee of high availability as current Netflix infrastructure provides.
Mais conteúdo relacionado
Mais procurados
Mais procurados(20)
Similar a Netflix and Containers: Not A Stranger Thing
Similar a Netflix and Containers: Not A Stranger Thing(20)
Mais de aspyker
Mais de aspyker(9)
Último
Último(20)
Netflix and Containers: Not A Stranger Thing
- 1. and containers Andrew Spyker (@aspyker) - Engineering Manager Not
- 2. About Netflix ● 86.7M members ● A few thousand employees ● 190+ countries ● > ⅓ NA internet download traffic ● 500+ Microservices ● Many 10’s of thousands VM’s ● 3 regions across the world
- 3. Netflix has a elastic, cloud native, immutable microservice architecture using full devops built on VM’s! 3 Why are we messing around with containers?
- 4. Technical motivating factors for containers ● Simpler management of compute resources ● Simpler deployment packaging artifacts for compute jobs ● Need for a consistent local developer environment
- 5. Sampling of realized container benefits Media Encoding - encoding research development time ● VM’s platform to container platform - 1 month vs. 1 week Continuous Integration Testing ● Build all Netflix codebases in hours ● Saves development 100’s of hours of debugging Edge Re-architecture using NodeJS ● Focus returns to app development ● Simplifies, speeds test and deployment 5
- 7. Multi-tenant (cgroups/Mesos) historically used for batch Linux cgroups
- 8. What do batch users want? ● Simple shared resources, run till done, job files ● NOT ○ EC2 Instance sizes, autoscaling, AMI OS’s ● WHY ○ Offloads resource management ops, Simpler
- 9. Introducing Titus Batch Job Management Resource Management & Optimization Container Execution Integration Workflow, Data Analysis, Adhoc Upstream Systems
- 10. Netflix Batch Job Examples ● Algorithm Model Training (with GPU’s)
- 11. Netflix Batch Job Examples ● Media Encoding ● Digital Watermarking 1 1
- 12. Netflix Batch Job Examples Open Connect CDN Reporting Adhoc Reporting
- 13. ● Docker helped generalize use cases ● Scheduling required (GPU, elastic) ● Initially ignored failures (with retries) ● Time sensitive batch came later Lessons Learned from Batch
- 14. Current Container Usage - Batch ● 1000’s of container hosts (g2, m4, r3 instances) ● 1000’s containers / hour average ● Large spikes of CI testing and Digital Watermarking From day of 10/26 ● 47K containers ● Bursts of 1000 containers in a minute
- 16. Why Services in containers? Theory Reality Developer
- 17. Opportunities to evolve our baking ● Java focused supported AMI, baking works well! ● However, wanted to allow ○ other stacks to evolve independent of OS updates ○ simplified builds (vs. Java and OS based tooling) ○ reliable smaller instances for dynamic languages ○ ability to develop locally with same image
- 18. Services are just long running batch? Services Job Management Resource Management & Optimization Container Execution Integration Service Apps Batch
- 19. 19 Nope, not that easy - Titus Details 19 Titus UITitus UI Docker Registry Docker Registry Rhea container container container docker Titus Agent metrics agent Titus executor logging agent zfs Mesos agent docker RheaTitus API Cassandra Titus Master Job Management & Scheduler S3 Zookeeper Docker Registry EC2 Autocaling API Mesos Master Titus UI Fenzo container Pod & VPC network drivers containercontainer AWS metadata proxy Integration AWS VM’sCI/CD
- 20. Services more complex ● Services resize constantly and run forever ○ Autoscaling ○ Hard to upgrade underlying hosts ● Require IPC integration ○ Routable IPs, service discovery ○ Ready for traffic vs. just started/stopped ● Existing well defined dev, deploy, runtime & ops tools
- 21. Real networking is hard
- 22. Multi-tenant Need IP per container - in VPC Using security groups Using IAM roles Considering network resource isolation
- 23. Enabling VPC Networking No IP, SecGrp A Task 0 SecGrp Y,Z Task 1 Task 2 Task 3 Titus EC2 Host VMeth1 ENI1 SecGrp=A eth2 ENI2 SecGrp=X eth3 ENI3 SecGrp=Y,Z IP 1 IP 2 IP 3 pod root veth<id> app SecGrp X pod root veth<id> app SecGrp X pod root veth<id> appapp veth<id> Linux Policy Based Routing + Traffic Control Titus EC2 Metadata Proxy 169.254.169.254 IPTables NAT (*) * ** 169.254.169.254 Non-routable IP *
- 24. Solutions ● VPC Networking driver ○ Supports ENI’s - full IP functionality ○ Scheduled security groups ○ Support traffic control (resource isolation) ● EC2 Metadata proxy ○ Adds container “node” identity ○ Delivers IAM roles
- 25. Reuse existing infrastructure services VMVM EC2 AWSAutoScaler VMs App Cloud Platform (metrics, IPC, health) VPC Netflix Cloud Infrastructure (VM’s + Containers) Atlas Eureka Edda
- 26. Enable them for containers VMVM EC2 AWSAutoScaler VMs App Cloud Platform (metrics, IPC, health) VPC Netflix Cloud Infrastructure (VM’s + Containers) VMVM Atlas TitusJobControl Containers App Cloud Platform (metrics, IPC, health) Eureka Edda VMVM Batch Containers
- 27. Spinnaker
- 28. Deploy based on new images tags
- 29. Basic resource requirements IAM Roles & Sec Groups per container Deploy Strategies Same as VM’s
- 33. Secure Multi-tenancy Common to VM’s and tiered security needed ● Protect the reduced host IAM role, Allow containers to have specific IAM roles ● Needed to support same security groups in container networking as VM’s User namespacing ● Docker 1.10 - Introduced User Namespaces ● Didn’t work /w shared networking NS ● Docker 1.11 - Fixed shared networking NS’s ● But, namespacing is per daemon, Not per container, as hoped ● Waiting on Linux ● Considering mass chmod / ZFS clones
- 34. Titus Advanced Scheduling ● Support for AZ balancing ● Multiple instance types selected based on workload ● Elastic underlying common resource pool ○ Bin packing managed transparently across all apps ● Hard and soft constraints ● Resource affinity and task affinity ● Capacity guarantees (critical tier) 34
- 35. Fenzo - Keep resource scheduling extensible Fenzo - Extensible Scheduling Library Features: ● Heterogeneous resources & tasks ● Autoscaling of mesos cluster ○ Multiple instance types ● Plugins based scheduling objectives ○ Bin packing, etc. ● Plugins based constraints evaluator ○ Resource affinity, task locality, etc. ● Scheduling actions visibility
- 36. Current Container Usage - Service ● Still small ~ 2000 long running containers ● NodeJS Device UI Scripts Apps ● Stream Processing Jobs - Flink ● Various Internal Dashboards
- 37. Questions? Andrew Spyker (@aspyker) - Engineering Manager