Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2iUKlD8. Sharma Podila reviews the state of containers usage in Netflix. With references to projects Titus and Mantis, he highlights the integrations with AWS to run Apache Mesos clusters and how they adapted resource scheduling via NetflixOSS Fenzo to run an elastic infrastructure for a varied mix of workloads.Topics include capacity guarantees, network isolation, modeling multiple EC2 instance types etc. Filmed at qconsf.com. Sharma Podila is the author of NetflixOSS Fenzo scheduling library for Apache Mesos frameworks. He is a Software Engineer in the Edge Engineering team at Netflix. His current work includes distributed resource scheduling, reactive stream processing, and services optimization.

1. Elastic Efficient Execution of
Varied Containers
Sharma Podila
Nov 7th 2016, QCon San Francisco

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-containers-2016

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. How do we efficiently run
heterogeneous workloads
on an elastic pool of
heterogeneous resources,
with capacity guarantees?
In other words...

5. Topics
● Containers, Mesos, Fenzo - where are we today?
● Modeling an elastic Mesos cluster
● Capacity guarantees for varied applications
● Network resource and security groups
● Ongoing and future work

6. About Me
● Software engineer
○ Resource scheduling, stream processing,
distributed systems
○ Netflix Edge Engineering
○ Sun Microsystems + Oracle Corp.
● Author of Fenzo scheduling library
https://github.com/Netflix/Fenzo

7. Source: https://www.sandvine.com/news/global_broadband_trends.asp
81 Million subscribers worldwide and growing!

8. Microservices architecture on AWS EC2

9. Containers, Apache Mesos, Fenzo -
where are we today?

10. Reactive stream processing: Mantis
Zuul
Cluster
API
Cluster
Mantis
Stream processing
Cloud native service
● Configurable message delivery guarantees
● Heterogeneous workloads
○ Real-time dashboarding, alerting
○ Anomaly detection, metric generation
○ Interactive exploration of streaming data
Anomaly
Detection

11. Current Mantis usage
● Peak of 1,800 EC2 instances
○ M3.2xlarge instances
● Peak of 3,700 concurrent containers
○ Trough of 2,700 containers
● Mix of perpetual and interactive exploratory jobs
● Peak of 11 Million events / sec

12. EC2
VPC
VMVM
TitusJob
Control
Containers
App
Cloud Platform
(metrics, IPC, health)
VMVM
Batch
Containers
Eureka Edda
Container deployment: Titus
Atlas &
Insight

13. Current Titus usage
#Containers (tasks) for the week of 10/24 in one of the regions
● Peak of ~1,800 instances
○ Mix of m4.4xl, r3.8xl, g2.8xl
○ ~800 instances at trough
● Mix of batch, stream
processing, and some
microservices

14. Core architectural components
AWS EC2
Apache Mesos
Titus/Mantis Framework
Fenzo
Fenzo at
https://github.com/Netflix/Fenzo
Apache Mesos at
http://mesos.apache.org/

15. Jobs, tasks, instances, containers
Jobs can be one of batch, service, or stream
processing type of jobs
A jobs has one or more tasks to run
An instance is equivalent to a task
A task runs one container

16. A few common themes
Heterogeneous mix of jobs and resources
Resource Task request Agent sizes
CPU 1 - 32 CPUs 8 - 32 CPUs
Memory 2 - 200+ GB 32 - 244 GB
Network
bandwidth
10 - 2000 Mbps 1024 - 10240
Resource affinity based on task type
Task locality

17. A few common themes
Large variation in peak to trough resource
requirements
Mantis
events/sec
11M
2M
Titus
concurrent
containers
1000s
10s

18. Can we resize agent cluster based
on demand?
Modeling an elastic Mesos cluster

19. Task assignments in a cluster
Consider a cluster with 4-slot hosts

20. “Random” assignments in a cluster
An EC2
instance
with 4 slots
Used slot
Idle slot
Cluster starts random assignments of
resources to tasks

21. “Random” assignments in a cluster
Cluster starts to fill up...

22. “Random” assignments in a cluster
Cluster somewhat full.
But, only 1 agent can be terminated for scale
down without losing jobs
About 50%
utilized

23. “Random” assignments in a cluster
Cluster is now full
100%
utilized

24. “Random” assignments in a cluster
Cluster partially used as jobs finish...
About 65%
utilized

25. “Random” assignments in a cluster
Cluster partially used, but, can’t terminate any
instance without losing jobs
About 25%
utilized

26. Ideal assignments in a cluster
Cluster utilized to the same level as previous,
but, can now terminate 9 of the 12 instances!
Similarly,
25% utilized

27. Ideal assignments in a cluster
Cluster scaled down easily due to “bin packing”

28. EC2 ASG attributes for setting
number of servers in cluster
EC2 AutoScalingGroups have three attributes to set
● Min - minimum number of instances to have
● Max - maximum number of instances
● Desired - current number of instances to have
Fenzo sets the “Desired” count based on demand

29. EC2 AutoScalingGroup for Mesos agents
Min
Desired
Max

30. Min
Desired
Max
EC2 AutoScalingGroup for Mesos agents

31. Min
Desired
Max
EC2 AutoScalingGroup for Mesos agents

32. Using multiple instance types

33. Amazon EC2 provides a variety of servers
a.k.a “instance types”
https://aws.amazon.com/ec2/instance-types/
Algorithm model training jobs run well on memory
optimized instances of R3 type
Typical services run well on balanced compute
instances of M4 type
Using multiple instance types

34. How do we use multiple EC2 instance
types in the same Mesos agent cluster?
Using multiple instance types

35. Using multiple EC2 instance types
m4.4xlarge agent ASG r3.8xlarge agent ASG
Titus
Grouping agents by instance type let’s us autoscale them independently

36. Using multiple EC2 instance types
m4.4xlarge agent ASG r3.8xlarge agent ASG
Titus
User job: 2 CPUs,
5GB memory
User job: 8 CPUs,
8GB memory User job: 1 CPUs,
1GB memory

37. Continuous deployment of agents

38. Continuous deployment of agents
m4.4xlarge agent ASG v1
A new version of agent introduces a new ASG

39. Continuous deployment of agents
m4.4xlarge agent ASG v1 m4.4xlarge agent ASG v2
A new version of agent introduces a new ASG

40. Continuous deployment of agents
m4.4xlarge agent ASG v1 m4.4xlarge agent ASG v2
Disable
A new version of agent introduces a new ASG

41. Continuous deployment of agents
m4.4xlarge agent ASG v1 m4.4xlarge agent ASG v2
Disable
Migrate
tasks
A new version of agent introduces a new ASG

42. Continuous deployment of agents
m4.4xlarge agent ASG v1 m4.4xlarge agent ASG v2
Disable
A new version of agent introduces a new ASG

43. Continuous deployment of agents
m4.4xlarge agent ASG v2
Old agent
ASG removed
A new version of agent introduces a new ASG

44. Bringing it all together...
m4.4xlarge agent ASG r3.8xlarge agent ASG
Titus
v2
v1
v2
v1

45. Capacity guarantees for varied
applications

46. The capacity guarantee challenge
Demand
for
resources
Supply>

47. New batch of tasks
Running #tasks
Tasks launched
An execution sample from a cluster

48. New batch of tasks
Running #tasks
Tasks launched
An execution sample from a cluster
Waiting for agents
to free up…
Or, for new agents
from scale up

49. New batch of tasks
Running #tasks
Tasks launched
Scale up and freed
agents satisfy all
new pending tasks
An execution sample from a cluster

50. New batch of tasks
Running #tasks
Tasks launched
What if a service was
launched at this time?
Waiting for agents
to free up…
Or, new agents
from scale up
An execution sample from a cluster

51. Capacity guarantees
Guarantee capacity for timely job starts
Mesos support for quotas, etc. evolving
^
Agreed upon

52. Capacity guarantees
Guarantee capacity for timely job starts
Mesos support for quotas, etc. evolving
^
Agreed upon
Generally, optimize throughput for batch
jobs and start latency for service jobs

53. Capacity guarantees
Some service style jobs may be less
important
Categorize by expected behavior instead

54. Capacity guarantees
Some service style jobs may be less
important
Categorize by expected behavior instead
Critical versus Flex (flexible) scheduling
requirements

55. Capacity guarantees
Critical
Flex
Quotas

56. Capacity guarantees
Critical
Flex
Critical
Flex
Resource
Allocation
Order
Quotas Prioritiesvs.

57. AppC1
AppC2
AppC3
AppCN
AppF1
AppF2
AppFN
AppF3
Resource
Allocation
Order
Capacity guarantees: hybrid view
Critical
Flex

58. Capacity guarantees via Fenzo
Fenzo supports multi-tiered task
queues
Multiple “buckets” per tier with
“fair sharing” by dominant
resource usage
Tier 0
Tier 1

59. Translating application capacity to
EC2 instances
● Define per application capacity guarantees
● Define per tier capacity guarantees
● Translate to number of EC2 instances

60. Defining application capacity
App1-cap = num_app_instances *
app_instance_dimensions
app_instance_dimensions:
{ #cpus, memory, disk, network}
Agnostic to EC2 instance types

61. Defining application capacity
Applications specify resource needs, not EC2
instance types
● Can manage capacity guarantees using a variety of
instance types
● Eases migration to new instance types, thereby helps
capacity procurement teams

62. Tier Capacity =
SUM (App1-cap + App2-cap + … + AppN-cap)
+ BUFFER
BUFFER:
● Accommodate some new or ad hoc jobs with no guarantees
● Red-black pushes of services temporarily double capacity
Defining Tier capacity

63. #EC2_instances = Tier_capacity /
EC2_instance_dimensions
A tier may use multiple instance types
Translate to number of instances
Critical
Flex
= { m4.4xlarge, m3.2xlarge }
= { r3.8xlarge, g2.8xlarge }

64. Network resource and security groups

65. Container executor
+ <
Augment missing pieces:
IP per container
Security - Security Groups, IAM roles
Isolation for networking b/w, disk I/O
MULTI-TENANT

66. Elastic Network Interfaces (ENI)
AWS EC2 Instance
ENI0
IP0
IP1
IP2
IP3
ENI1
IP4
IP5
IP6
IP7
ENI2
IP8
IP9
IP10
IP11
ENI0
IP0
IP1
IP2
IP3
● Each EC2 instance
in VPC has 2 or
more ENIs
● Each ENI can have 2
or more IPs
● Security Groups are
set on the ENI

67. ENI+IP resource allocation model
A two level resource modeled in Fenzo
Each agent reports #ENIs and #IPs per ENI via custom attribute
Fenzo does allocation and usage tracking
ENI 1
Assigned Security Group: SG1 Used IPs Count: 2 of 7
ENI 2
Assigned Security Group: SG1,SG2 Used IPs Count: 1 of 7
ENI 3
Assigned Security Group: SG3 Used IPs Count: 7 of 7

68. Plumbing VPC Networking into Docker
No IP, SecGrp A
Task 0
SecGrp Y,Z
Task 1 Task 2 Task 3
Titus EC2 Host VMeth1
ENI1
SecGrp=A
eth2
ENI2
SecGrp=X
eth3
ENI3
SecGrp=Y,Z
IP 1
IP 2
IP 3
pod root
veth<id>
app
SecGrp X
pod root
veth<id>
app
SecGrp X
pod root
veth<id>
appapp
veth<id>
Linux Policy Based
Routing + Traffic Control
Titus
EC2
Metadata
Proxy
169.254.169.254
IPTables NAT (*)
* **
169.254.169.254
Non-routable IP
*

69. Network bandwidth isolation
Each container gets an IP on one of the ENIs
Linux tc policies used on virtual Ethernet
For both incoming and outgoing traffic
Bandwidth limited to the requested value
No borrowing of unused bandwidth
Easy to reason about

70. Ongoing and future work

71. Current and future work
● Fine grain capacity guarantees
○ Hierarchical sharing policies
○ Preemptions to satisfy priority tiers and sharing policies
● Execution environment security hardening
● Onboarding new applications
● Looking forward to working with the
community

72. In Summary...

73. Mesos and Fenzo help us run lots of containers
● In an elastic fashion
● With guaranteed capacity for varied
applications
● Custom AWS integration gives us network
resource isolation and security groups
In summary...

74. Questions?
Elastic Efficient Execution of
Varied Containers
Sharma Podila spodila @ netflix . com
@podila linkedin . com / in / spodila

75. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-containers-2016