This session will introduce you to Empire, a new self-hosted PaaS built on top of Amazon’s EC2 Container Service (ECS). Empire is a recently open-sourced project that provides a mostly Heroku-compatible API. It allows engineering teams to deploy and manage applications in a method similar to Heroku, but with the added flexibility and control of running your own ECS container instances. We'll talk about why Remind decided to move its infrastructure from Heroku to AWS, introduce you to ECS and the open source platform we built on top of it to make migration easier, and then we'll demo Empire to show you how you can try it today.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Eric Holmes & Michael Barrett, Remind
October 2015
DVO308
Docker & ECS in Production
How We Migrated Our Infrastructure from Heroku to AWS

2. What to Expect from the Session
• A brief introduction about why we decided to build an
internal platform at Remind, and the lessons we learned
along the way
• An introduction to the open source PaaS we built called
Empire, and how we’re leveraging Amazon ECS
• Demo
• Q&A

3. About Us
• Eric Holmes & Michael Barrett
• Infrastructure engineers at Remind
• We build things for developers
• You can find our open source stuff at:
• https://github.com/ejholmes
• https://github.com/phobologic

4. Remind
• A messaging platform for teachers.
• Chat/announcements/files
• Over 30 million users
• Used actively in ~50% of U.S. public schools
• Over 2 billion messages delivered
• ~50 employees. ~30 engineers.

5. A Little History

6. A Little History
• Started as a “monorail”
• Scaling challenges during BTS
• Migrated to an SOA/micro-service architecture

7. Heroku was great, but…
• Every app on Heroku is publicly accessible
• Databases need to be exposed to Internet traffic
• Limited visibility and control

8. What we want from a PaaS
• AWS
• Flexibility
• Shared patterns for deployment
• Easy service operation
• Containers/Docker

9. Why Containers?
• Fast build + deploy iteration cycles
• Isolate dependencies
• Better dev/prod parity
• Immutable images
• Better resource utilization

10. Building an Empire

11. Design Goals
• Easy to operate
• Open source
• Support 12-factor stateless apps (12factor.net)
• Swappable scheduling back-ends
• Stability!
• Docker images as a unit of deployment

12. Components of a PaaS
Scheduler Router Control Plane

13. Scheduler :: Cluster Management
Join Leave

14. Scheduler :: Task Placement
Find Host Run Job
CPU/
Memory
Container
Cluster

15. Scheduler :: Task Placement
type App []Process
type Scheduler interface {
Run(App)
Remove(App)
Scale(Process)
Tasks(App) []Task
Stop(Task)
}

16. Empire :: V1
Scheduler Router Control Plane
+ Fleet
etcd + registrator + confd
Heroku Platform API
Spec + hk CLI

17. Amazon EC2 Container Service
• Managed cluster manager and scheduler
• Supports Docker
• Built-in service scheduler
• Integrates with Elastic Load Balancing

18. Amazon EC2 Container Service :: Resources
• Clusters
• Task definitions
• Tasks
• Services

19. Scheduler Interface
Run(App)
Remove(App)
Scale(Process)
Tasks(App) []Task
Stop(Task)
Amazon ECS Scheduler Implementation
Amazon ECS API
RegisterTaskDefinition ->
CreateService/UpdateService
DeleteService
UpdateService
ListTasks
StopTask

20. Empire :: V2
Scheduler Router Control Plane
ECS ELB
Heroku Platform API
Spec + emp CLI

21. Empire :: V2
An open-source, self-hosted PaaS for running
twelve-factor Docker apps backed by AWS
services

22. Twelve-Factor

23. Twelve-Factor Tenants
I. Codebase
II. Dependencies
III. Config
IV. Backing Services
V. Build, release, run
VI. Processes
VII. Port binding
VIII.Concurrency
IX. Disposability
X. Dev/prod parity
XI. Logs
XII. Admin processes

24. 12factor :: Dependencies
“Explicitly declare and isolate dependencies”
FROM ruby
RUN apt-get install imagemagick
RUN bundle install

25. 12factor :: Build, release, run
“Strictly separate build and run stages”
Empire

26. 12factor :: Build
$ git push

27. 12factor :: Release, Run
Config
{}
Release
Amazon ECS

28. 12factor :: Release, Run
$ cat Procfile
web: ./bin/web
worker: ./bin/worker
$ aws ecs list-services
arn:aws:ecs:us-east-1:***:service/api--web
arn:aws:ecs:us-east-1:***:service/api--worker
$ emp deploy org/api:latest
Status: Created v1 release.

29. Service Discovery
$ aws ecs describe-services --service api--web
"loadBalancers": [{
"containerName": "web”,
"containerPort": 9001,
"loadBalancerName”: "2888...a31d4c”
}]
$ curl http://api
Ok

30. 12factor :: Concurrency
“Scale out via the process model”
$ emp scale web=10
$ aws ecs describe-service --service api--web
“desired-count”: 10

31. 12factor :: Dev/prod parity
“Keep development, staging, and production as similar as
possible”
$ docker run --env-file <(emp env -a api) org/api

32. 12factor :: Logs
“Treat logs as event streams”
$ emp log
“GET / HTTP/1.1” 200
STDOUT Amazon Kinesis

33. 12factor :: Admin processes
“Run admin/management tasks as one-off processes”
$ emp run rake db:migrate
Migrated

34. Demo

35. Pain Points & Lessons Learned

36. Container Instance Rollout
1. Update AMI in AWS CloudFormation stack.
2. Kill 1 host
3. Wait for new Amazon ECS services to start running on
new host
4. Rinse and repeat

37. Logging
Logspout

38. Logging
SOURCE=<app>.<process>.<version>
SYSLOG_STRUCTURED_DATA=app={{ .Container.Config.Env "SOURCE" }}

39. Docker Monolith
= Bad Times

40. Docker Performance
• Early versions of Docker had abysmal push/pull
performance
• Use Docker >= 1.8.1
• Make your Dockerfile’s use the layer cache efficiently
• https://github.com/remind101/conveyor

41. This space moves fast!
• Containers have been around, but Docker made them
accessible
• New tools coming out ever day
• AWS’s offerings have been incredibly stable and feature
rich

42. Remember to complete
your evaluations!

43. Thank you!