Containers allow you to easily package an application's code, configurations, and dependencies into easy to use building blocks that deliver environmental consistency, operational efficiency, developer productivity, and version control. But how can developers leverage containers to drive innovation for their applications, their team, and organization? In this session, Asif Khan Technical Business Manager for AWS will discuss how containers are becoming a new cloud native compute primitive, and how your organization can use containers as a building block to accelerate innovation. WeWork's Christopher Tava, Joshua Davis, and OpsLine's Radek Wierzbicki will show how they adopted containers as discipline in code development, and how they refactored their production architecture into containers running on Amazon ECS in under 8 months.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A s i f K h a n , T B D M , A m a z o n W e b S e r v i c e s
C h r i s T a v a , V P – E n g i n e e r i n g , W e W o r k
J o s h D a v i s , P r i n c i p a l E n g i n e e r , W e W o r k
R a d e k W i e r z b i c k i , S o l u t i o n s A r c h i t e c t , O p s L i n e
CON203
AWS re:INVENT
Driving Innovation with Containers

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
T r a d i t i o n a l S o f t w a r e D e v e l o p m e n t a n d D e p l o y m e n t
releasetestbuild
delivery pipeline
app
(aka the“monolith”)developers

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges with the traditional software
development lifecycle
Long Build/Test/Release
Cycles
(who broke the build?)
Operations is a nightmare
(module X is failing, who’s
the owner?)
Difficult to scale
New releases
take months
Long time to add
new features
Architecture is hard to
maintain and evolve
Lack of innovation
Frustrated customers
Lack of agility
Infrastructure Application Business

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
- Adrian Cockcroft (VP, Cloud Architecture Strategy at AWS)

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Services communicate
with each other over the
network

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
You can update the services
independently; updating one
service doesn’t require
changing any other services.

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Self-contained; you can
update the code without
knowing anything about
the internals of other
microservices

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Portability
Flexibility
Efficiency
Speed
Same immutable images. Run anywhere.
Create modular environment. Decompose Apps.
Speeds up build and release cycles.
Optimize resource allocation.
Agility.

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
= 50 million deployments a year
Thousands of teams
× Microservice architecture
× Continuous delivery
× Multiple environments
5708 per hour – one every 0.63 seconds!

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
BUILD SHIP RUN

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Server
Guest OS
Bins/Libs Bins/Libs
App2App1
Going from a single host…

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
…to this is hard!

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Container Management Platforms

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon ECS
EC2 INSTANCES
ECS
AGENT
TASK
Container
TASK
Container
ECS
AGENT
TASK
Container
TASK
Container AGENT COMMUNICATION
SERVICE
Amazon
ECS
API
CLUSTER MANAGEMENT
ENGINE
KEY/VALUE STORE
ECS
AGENT
TASK
Container
TASK
Container
Internet LOAD
BALANCER
LOAD
BALANCER

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Remember…

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WeWork
Chris Tava
Josh Davis

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Startups are chaos
● We had three ways to Deploy
● Security setup was super light and too open
● Lots of variation in infrastructure

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
After 810 days of uptime...

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
On 6/20/17 - network card failure

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Time for a big change
● Needed to be able to quickly recreate our
environment
● Get it ready for [more] scale
● Secure the platform
● Adopting a devops culture
● KISS

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Fieldlens - scale, consumer use
● 80K users, 110K projects
● 1.6M posts, 2.2M comments, 4.1M media
● Mobile / web clients, background jobs
● Enterprise users expect reliability!

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What we started with
Docker Locally in development: docker-compose
Some micro-services
Jenkins - Build and Test, CI/CD
Environments - dev, qa, prod

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Where we want to be
Docker images for everything
Uniform deployment / management
New micro-services are easy to create
Enhanced security / scalability / reliability

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Desired Topology

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How many weeks to get here?
ECS Dev, QA, Prod

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How do we get there?
First step - Start building and publishing Docker images
Jenkins pipelines
Build can happen outside of a container
Docker images can be deployed locally with docker-compose
Image repository, naming convention, local environment config

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Developer using
docker-compose
‘Local infra’ git repo
docker-compose.yml
our-repo.jfrog.io/elasticmq:0.13.2
our-repo.jfrog.io/otherservice-
develop:latest
Docker
app/service git
repo
app/service git
repo
Database
server,
Other services

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Developer using docker-compose
our-repo.jfrog.io/custom-
tomcat:7
Docker
service git repo
docker-compose.yml
app/service git
repo
app/service git
repo
Database
server,
Other services
Service dev
container

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
First milestone
Services work in a container (locally with docker-compose)
Basic configuration established
Dockerfile / Jenkinsfile for all repos
Dependencies connected via docker network
Ready to go to the cloud!

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deploying to ECS
Cloud Formation - Infrastructure as Code
Environment Templates for the ECS clusters and other
resources (ops)
Service Templates in each repo (devs and ops)
Common shell scripts to handle basic tasks

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Environment Templates
Environment git repo
Env templates
dev
qa
prod

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service Templates
service git repo
Service
Template
Env Parameters
ECS Cluster
Servi
ce
app/service git
repo
app/service git
repo
Tasks

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Database Schema Updates
Task Definition - update image with CF
Run task synchronously - Ephemeral!
Begin deployment of new code
Database
ECS Cluster
Schema Update
Task
(single run)
app/service git
repo
app/service git
repo
Prev app
version

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Secrets
Encrypt sensitive config files with KMS
Developers decrypt and store encrypted configs in the
service repos
Encrypt / Decrypt privileges are controlled via IAM
Docker image contains encrypted configs
In ECS, decrypt in the entrypoint

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hazelcast Cluster
Register host IP in Consul via ContainerPilot
Hazelcast start-up queries Consul for initial hosts
ECS Cluster
ECS Instance
task
Consul
ECS Instance
task
Node1 ->
10.10.37.101:5701
Node2 ->
10.10.37.117:5701

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deployment in Action
Deployment scripts run CloudFormation updates
CloudFormation references the docker images
ECS Scheduler rolls out the new code with zero downtime
deployment
script
Jenkins
pipeline
Env params

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Other Details
Entrypoint - ContainerPilot
Task start-up logging and application logging
Singleton Services
Promoting builds to environments

41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It’s not all rainbows and unicorns...
Discovering configuration
Logentries
Datadog monitoring
Monolith needed higher ulimits
APM - TraceView
Deployment is different!
Using shell to troubleshoot - no longer a thing
Scheduler ‘flapping’ or ‘cycling’ - Rollback
Hosts constantly change - “Where are the logs?”

42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
OpsLine
Radek Wierzbicki

43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cloud Transformation and DevOps
Automation Experts

44. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Working with WeWork

45. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Motivation

46. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cloud Native approach
Automate everything using CloudFormation
• VPC, security groups
• IAM and KMS
• RDS
• ECS clusters (scalable EC2 nodes)
• ECS services
• Lambda
Version control code and configuration

47. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why ECS?
Provided by AWS
Simple
Scalable
Highly-available
Easy and fast to provision
Provides practical encapsulation of containers
Works well with CloudFormation

48. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How did we do it?

49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cluster management
Provisioning - CloudFormation
Configuration management - cfn-init
Monitoring - DataDog

50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ECS cluster autoscaling

51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ECS cluster autoscaling
With only one server, new tasks
will be pending.

52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ECS cluster autoscaling

53. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ECS cluster autoscaling
● 88% CPU utilization
● 56% Memory
utilization
● Scale out when CPU
is above 85%
● Scale in when CPU is
below 45%

54. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ECS cluster autoscaling
● 92% CPU utilization
● 58% Memory utilization

55. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application management
Parametrized to run in different environments
ECS service, ECS task definition
IAM task role (access to S3, SQS, KMS, etc)

56. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Ap p lication monitoring
Load balancer health checks
Sidecar containers
DataDog

57. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Appl i cati on d epl oyment
CloudFormation
Parameter
Update
New ECS task ECS deployment

58. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application deployment
v1 v1
Initial state before deployment.

59. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application deployment
v2
v1 v1
One new task added to the cluster.

60. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application deployment
v2
v1
v2
Old task removed, another new task added.

61. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Application deployment
v2 v2
Final state.

62. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The DevOps Way
Simplicity
Visibility
• Troubleshooting
• Deployments

63. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The DevOps Way
Education
• Documentation
• Training
Empowerment
• Documentation and best practices
• Shell repository with templates and scripts
• Jenkins shared libraries
• Standardized Jenkins project
Collaboration

64. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
WE ARE HIRING
For more information, please contact:
ctava@wework.com
THANK YOU!
For more information, please contact:
connect@opsline.com
DevOps On-Demand

65. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!