This presentation covers the benefits of deploying to ECS, as we explore techniques and open source tools to help manage multiple ECS clusters. First we discuss why developing as containers and deploying to ECS offers a more cost-effective, faster, and safer workflow from dev to production than VMs on EC2. Next we present techniques Convox uses to manage countless clusters and apps. We will see a CloudFormation template that reliably provisions, updates, and horizontally and vertically scales ECS clusters, and another template that configures application containers inside the cluster. Finally we survey features that are "missing" on ECS and demonstrate simple tools and techniques Convox has developed to to fill in these gaps. For example: * Running a private build and registry service inside the ECS cluster itself * Splitting up an app's processes (i.e. web and worker) into their own task definitions for independent horizontal scaling * Running an agent on every cluster instance for forwarding all container logs to Kinesis, Lambda and beyond * Running an public API on a cluster that facilitates a `docker exec` debugging session inside application containers

1. ╔══════════════════════════════════════════╗
║ OPEN SOURCE TOOLING FOR ECS ║
║ ║
║ Noah Zoschke ║
║ noah@convox.com ║
║ @nzoschke ║
║ ║
║ 12/8/2015 ║
╚══════════════════════════════════════════╝
CONVOX

2. EC2 CONTAINER SERVICE
(BATTERIES NOT INCLUDED)
API
• Clusters
• TaskDefinitions
• Tasks and
Services
Bring Your Own
• Instances
• ecs-agent
• Load Balancers
• Logging
• Builds / Images
• Tools...

3. CONVOX
OPEN SOURCE PAAS ⟷ IAAS
Racks ⟷ ASG, CF, Dynamo, EC2, ECS, IAM, VPC
Apps ⟷ CF, ECS, ELB, Kinesis
Scale ⟷ ASG, CF, ECS
Environments ⟷ KMS, S3
Builds ⟷ S3
Logs ⟷ Kinesis, Lambda
Metrics ⟷ CloudWatch, Kinesis
Notifications ⟷ SNS

4. CONVOX REST API
$ convox api get /apps
[
{
"balancer": "myapp-1749418666.us-east-1.elb.amazonaws.com",
"name": "myapp",
"release": "REXIQURVKXE",
"status": "running"
},
{
"balancer": "myapp2-689551992.us-east-1.elb.amazonaws.com",
"name": "myapp2",
"release": "RNEFHNIUSKF",
"status": "running"
},
{
"balancer": "myapp3-435098803.us-east-1.elb.amazonaws.com",
"name": "myapp3",
"release": "RORJKBNVKDD",
"status": "running"
}
]

5. CONVOX REST API
$ convox api get /apps/myapp/processes
[
{
"app": "myapp",
"command": "bin/web",
"cpu": 0.0329,
"host": "10.0.3.135",
"id": "13254981d20",
"image": "registry.internal:5000/myapp-web:BHLRYHSMXNM",
"memory": 0.2068,
"name": "web",
"ports": [
"80:3000",
"443:3001"
],
"release": "REXIQURVKXE",
"started": "2015-01-01T00:00:00Z"
}
]

6. BEHIND THE SCENES
aws && docker || convox

7. RACK INSTALL
CLOUDFORMATION PARAMETERIZED INFRASTRUCTURE
→ Ami ami-c5fa5aae
→ InstanceCount 3
→ InstanceType t2.small
→ Password PuDpyqGTmxBN8ziGJ9UiMfrfGZfHDG
→ Tenancy default
→ Version 20151204013151
→ VolumeSize 30
→ VPCCIDR 10.0.0.0/16
↑ Balancer convox AWS::ElasticLoadBalancing::LoadBalancer
↑ Cluster convox-Cluster-1JI343QBLSMYJ AWS::ECS::Cluster
↑ DynamoBuilds convox-builds AWS::DynamoDB::Table
↑ DynamoReleases convox-releases AWS::DynamoDB::Table
↑ EncryptionKey arn:aws:kms:...:key/d40c0153... Custom::KMSKey
↑ IamRole convox-IamRole-M1YZSNXNS1F7 AWS::IAM::Role
↑ Instances convox-Instances-PCWRQ6OWDWTT AWS::AutoScaling::AutoScalingGroup
↑ Kinesis convox-Kinesis-C09RDWFR8NOE AWS::Kinesis::Stream
↑ NotificationTopic arn:aws:sns:...:convox-notifications AWS::SNS::Topic
↑ Settings convox-settings-13c91daqrj90z AWS::S3::Bucket
↑ Vpc vpc-b27ff8d6 AWS::EC2::VPC
← Dashboard convox-820546104.us-east-1.elb.amazonaws.com
← Kinesis convox-Kinesis-C09RDWFR8NOE

8. CLOUDFORMATION
CUSTOM RESOURCES
┌─────────────────────────────────────┐
│POST arn:aws:lambda:... │
│{ │
│ ResourceProperties: { │
│ Description: "Master Encryption",│ ┌─────────────────────────────────────┐ ┌───────────────────────────┐
│ KeyUsage: "ENCRYPT_DECRYPT" │ │aws kms create-key │ │200: OK │
│ } │ │ --description "Master Encryption" │ │400: LimitExceededException│
│} │ │ --key-usage ENCRYPT_DECRYPT │ │500: KMSInternalException │
└─────────────────────────────────────┘ └─────────────────────────────────────┘ └───────────────────────────┘
┌────────────────┐ ┌──────────────┐──────────────────────▶┌───────────┐
│ CloudFormation │──────────────────────▶│ Lambda │ │AWS KMS API│
└────────────────┘ CREATE_IN_PROGRESS └──────────────┘◀──────────────────────└───────────┘
▲ │
│ │
│ CREATE_COMPLETE │
│ OR ▼
│ CREATE_FAILED ┌─────────────┐
└────────────────────────────────│ S3 │
└─────────────┘

9. APP CREATE
CLOUDFORMATION PARAMETERIZED CONTAINERS
→ Cluster convox-Cluster-1JI343QBLSMYJ
→ Cpu 200
→ Environment https://httpd-settings-1e3ej4u01z4bv.s3.amazonaws.com/releases/RSAQCOYHGPV/env
→ Key arn:aws:kms:us-east-1:901416387788:key/d40c0153-4a57-4d50-9ca0-99a974daca11
→ Release RSAQCOYHGPV
→ VPC vpc-b27ff8d6
→ WebCommand
→ WebDesiredCount 1
→ WebImage convox-820546104.us-east-1.elb.amazonaws.com:5000/httpd-web:BQIWNCMIYZG
→ WebMemory 256
→ WebPort80Balancer 80
→ WebPort80Certificate
→ WebPort80Host 42563
→ WebPort80Secure No
↑ Balancer httpd AWS::ElasticLoadBalancing::LoadBalancer
↑ Kinesis httpd-Kinesis-FO32SUUFLX24 AWS::Kinesis::Stream
↑ LogsAccess AKIAIFI65IDSEURPK62Q AWS::IAM::AccessKey
↑ LogsUser httpd-LogsUser-96BAE2EL9TNL AWS::IAM::User
↑ ServiceRole httpd-ServiceRole-19LN8R18BIVRW AWS::IAM::Role
↑ Settings httpd-settings-1e3ej4u01z4bv AWS::S3::Bucket
↑ WebECSService arn:aws:ecs:...:service/httpd-web-SATOEEBOQNF Custom::ECSService
↑ WebECSTaskDefinition arn:aws:ecs:...:task-definition/httpd-web:6 Custom::ECSTaskDefinition
← BalancerWebHost httpd-908645489.us-east-1.elb.amazonaws.com
← Kinesis httpd-Kinesis-FO32SUUFLX24
← Settings httpd-settings-1e3ej4u01z4bv
← WebPort80Balancer 80

10. APP DEPLOY
DOCKER APPLICATION MANIFEST ⟷ AWS
┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
│web: Task Definition httpd-web:6 │
│ command: bin/web Service httpd-web-SATOEEBOQNF │
│ build: . Docker Image httpd-web:BQIWNCMIYZG │
│ ports: │
│ - 80:80 ELB 80 : 52452 : 80 │
│ - 443:80 ELB (SSL) 443 : 52452 : 80 │
│ │
│worker: Task Definition httpd-worker:6 │
│ command: bin/worker Service httpd-worker-SHAOPEQONEF │
│ build: . Docker Image httpd-worker:BQIWNCMIYZG │
│ links: │
│ - redis REDIS_URL=rer45wxl0uj8jn6.1qae5u.ng.0001.usw2.cache.amazonaws.com:6379│
│ - rabbit RABBIT_URL=httpd-1222973998.us-west-2.elb.amazonaws.com:5672 │
│ │
│rabbit: Task Definition httpd-rabbit:6 │
│ command: rabbitmq-server Service httpd-rabbit-SPNFHGMWNUU │
│ image: rabbitmq Docker Image httpd-rabbit:BQUWNCMIYZG │
│ ports: │
│ - 5672 ELB (Internal) 5672 : 24324 : 5672 │
│ │
│redis: │
│ image: convox/redis AWS::ElastiCache::CacheCluster │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘

11. APP BUILD
PRIVATE BUILD AND REGISTRY SERVICE
┌─────────────────────────────────────────┐
│ convox API │
│ POST /apps/httpd/build │
┌──────────────────┐ │ │
│ │ │ ┌─────────────────────────────┐ │ ┌──────────────────┐
│ │ │ │ pull httpd-web:BLASTBUILD │◀───┼───────┤ Convox Registry │
│ │ │ └─────────────────────────────┘ │ │ │
│ │ │ ┌─────────────────────────────┐ │ │ │
│ ├─────┼─────▶│ pull rabbitmq │ │ │ │
│ │ │ └─────────────────────────────┘ │ │ │
│ │ │ ┌─────────────────────────────┐ │ │ │
│ │ │ │tag httpd-rabbit:BQUWNCMIYZG │ │ │ │
│ │ │ └─────────────────────────────┘ │ │ │ ┌────────────────┐
│ DockerHub │ │ ┌─────────────────────────────┐ │ │ ├───────────▶│ S3 │
│ and │ │ │ push $REGISTRY_HOST ├────┼──────▶│ │◀───────────┤ RegistryBucket │
│Private Registries│ │ └─────────────────────────────┘ │ │ │ └────────────────┘
│ │ │ ┌─────────────────────────────┐ │ │ │
│ ├─────┼─────▶│ pull debian │ │ │ │
│ │ │ └─────────────────────────────┘ │ │ │
│ │ │ ┌─────────────────────────────┐ │ │ │
│ │ │ │ build Dockerfile │ │ │ │
│ │ │ └─────────────────────────────┘ │ │ │
│ │ │ ┌─────────────────────────────┐ │ │ │
│ │ │ │ tag httpd-web:BQUWNCMIYZG │ │ │ │
│ │ │ └─────────────────────────────┘ │ │ │
│ │ │ ┌─────────────────────────────┐ │ │ │
└──────────────────┘ │ │ push $REGISTRY_HOST ├────┼──────▶└──────────────────┘
│ └─────────────────────────────┘ │
│ ┌─────────────────────────────┐ │
│ │tag httpd-worker:BQUWNCMIYZG │ │
│ └─────────────────────────────┘ │
│ ┌─────────────────────────────┐ │
│ │ ... │ │
│ └─────────────────────────────┘ │
│ │
│ │
└─────────────────────────────────────────┘

12. …UNTIL EC2 CONTAINER
REGISTRY IS GA

13. APP LOGS
AGENT, DOCKER APIS, KINESIS, LAMBDA
┌──────────────────────────────────────────────────────────┐ ┌──────────────────┐
│ EC2 Instance in ECS Cluster │ │ app1 Kinesis │
│ │ │ ┌────────┐ │ ┌───────────────────────────────────────────┐
│ ┌──────────────┐ ┌──────────────────────────────────┐ │ ┌─┼───▶│shard 1 │ │──┐ │ Lambda w/ EventSourceMapping │
│ │ │ │ │ │ │ │ └────────┘ │ │ │ ┌──────────────────────────────────────┐ │
│ │ │ │ │ │ │ └──────────────────┘ │ │ │function(event, context) { │ │
│ │ app1 │ │ app2 │ │ │ │ │ │ event.records.forEach(function(r) { │ │
│ │ web.1 │ │ worker.1 │ │ │ │ │ │ winston.info(r.kinesis.data) │ │
│ │ │ │ │ │ │ └─┼▶│ }) │──┼────────▶┌───────────────┐
│ │ │ │ │ │ │ │ │ context.done() │ │ │ │
│ └──────────────┘ └──────────────────────────────────┘ │ │ ┌──────────────────┐ │ │} │ │ │ │
│ │ │ │ │ │ app2 Kinesis │ │ │ │ │ │ │
│ │ ┌─────────────────────┘ │ │ │ ┌────────┐ │ │ └──────────────────────────────────────┘ │ │ Syslog Server │
│ ▼ ▼ │ │ │ ┌─▶│shard 1 │ │ │ ┌────────────────────────────────┐ │ │ │
│ ┌────────────┐ ┌────────────┐─────────────┼───┘ │ │ └────────┘ │ │ │function(event, context) { ... }│──┼────────▶│ │
│ │ dockerd │◀─────────────│convox/agent│─────────────┼─────┼─┘ ┌────────┐ │ │ └────────────────────────────────┘ │ │ │
│ └────────────┘ └────────────┘─────────────┼─────┼───▶│shard 2 │ │ │ ┌────────────────────────────────┐ │ │ │
│ ▲ ┌────────────────────────────────────┐ │ │ └────────┘ │────┼─▶│function(event, context) { ... }│───────┼────────▶└───────────────┘
│ │ │GET docker /events (create) │ │ │ . │ │ └────────────────────────────────┘ │
│ ▼ │ GET ENV "Kinesis", "Process"│ │ │ . │ │ │
│ ┌────────────┐ │ GET Docker /logs?follow=1 │ │ │ . │ └───────────────────────────────────────────┘
│ │ ecs-agent │ │ PUT Kinesis /records │ │ │ ┌────────┐ │
│ └────────────┘ └────────────────────────────────────┘ │ │ │shard N │ │
│ │ │ └────────┘ │
└──────────────────────────────────────────────────────────┘ └──────────────────┘

14. DEBUGGING
RUN, EXEC, SSH OVER WEB SOCKETS
$ convox run web bash
root@3e4160f0c4d0:/app#
$ convox ps
ID NAME RELEASE CPU MEM STARTED COMMAND
551967b75abd web RHQZEJZFCSD 0.39% 21.04% 2 hours ago rails server -b 0.0.0.0
f5ec95c38f58 worker RHQZEJZFCSD 0.00% 30.35% 2 hours ago sidekiq
$ convox exec 551967b75abd bash
root@281d0a9c33a:/app#
$ convox exec 551967b75abd ps ax
PID USER TIME COMMAND
1 root 0:00 sh -c bin/web
6 root 0:00 {web} /bin/sh bin/web
9 root 0:00 unicorn master -c unicorn.rb
11 root 0:00 unicorn worker[0] -c unicorn.rb

15. SCALING
ONE APP ⟶ MANY SERVICES
Service Name Task Definition Desired Running
═══════════════════════════════════════════════════════════════════════════
myapp-clock-SVQQEUPGZPS myapp-clock:106 1 1
myapp-scheduler-SSMOCJRAGOM myapp-scheduler:183 1 1
myapp-web-SLHARAVBAWZ myapp-web:119 2 2
myapp-runner-SEGBMHLWREH myapp-runner:163 4 4

16. CAPACITY PLANNING
FITTING MANY APPS IN ONE CLUSTER
| |
| |
| |
| |
| |
| |
| |
| |
| +---+ |
| | . | |
| | . | |
| | . | |
| | . | |
| | . | |
| | . | |
| | . | |
| +---+ |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| ,---, +---+ |
| | . | | . | |
+---' . +---, ,---' . +---+---+--+
| . . . | . | | . . . | . . . | |
+ . +---+ . '---, | . ,---+ . . . | |
| . | . | . . . | | . | | . . . | |
+---+ . +---+ . | +---+---+---+---+ |
| . . . | . | . | | . . . | . . . | |
+ . +---+ . +---+ | . . . + . . . | |
| . | . . . . . | | . . . | . . . | |
+---+---+---+---+---+---+---+---+---+--+
• rolling app deploys
need slack capacity
• process type with
load balancer needs
entire instance
• scheduler can fail
to place tasks
behind the
scenes...

17. noah@convox.com
@nzoschke
Discuss these techniques and get involved
GitHub https://github.com/convox
Slack http://invite.convox.com/
_ _ _ _
| |_| |__ __ _ _ __ | | _____| |
| __| '_ / _` | '_ | |/ / __| |
| |_| | | | (_| | | | | <__ _|
__|_| |_|__,_|_| |_|_|____(_)