At Netflix, we continue to improve upon our continuous delivery process. We thrive in a hybrid environment, where every developer is able to deploy code, and with that freedom comes the responsibility for ensuring that our customers are not negatively impacted. We have constructed Open Source tools toward a Continuous Delivery solution. In this presentation, from QConSF 2013, you will learn about our tool chain so that you can determine which make sense in your environment.

1. BUILD
BAKE
DEPLOY
Continuous Delivery at Netflix: From
Code to the Monkeys
Dianne Marsh
QCon San Francisco, November 2013
http://www.linkedin.com/in/diannemarsh

2. BUILD
BAKE
DEPLOY

3. Netflix Goals
High Availability
But … Move fast
Tools encourage Best Practices
But … Freedom to do the right thing

4. Teams Deploy Their Own Code
Run What You Wrote
 Rapid Innovation
 Rapid Detection
 Rapid Response
= Freedom + Responsibility
http://www.slideshare.net/garethbowles/self-servicebuilddeploymentagile2013

5. BUILD
Jenkins Job DSL
Configuration as Code
Groovy Script
Scripts go in Version Control
http://www.slideshare.net/quidryan/configuration-as-code

6. BUILD
BAKE

7. BAKE
Aminator
• Create AMI from Base AMI
• Image contains service and
everything needed to run it
• Unit of Deployment for Test and
Prod
• Abstracts Cloud Details
http://techblog.netflix.com/2013/03/ami-creation-with-aminator.html

8. BUILD
BAKE
DEPLOY

9. DEPLOY
Asgard: AWS Deployment
Tool
Deploys Netflix to the Cloud
Red/Black push
http://www.infoq.com/presentations/asgard

10. CANARY ANALYSIS
Test, Int, Prod
Choose where to deploy
Run canary analysis
Scale up new instances
Turn on traffic to new ASG
Turn off traffic to old ASG
Wait … analyze … continue

11. Asgard Developer Portal

12. GLISTEN
Extending Asgard’s Workflow
Automated Red/Black Push
Test, Int, Prod stacks
Run canary/analysis
Scale up new instances
Turn on traffic to new ASG
Run more tests
Turn off traffic to old ASG
Wait … analyze … continue
http://techblog.netflix.com/2013/09/glisten-groovy-way-to-use-amazons.html

13. BUILD
BAKE
DEPLOY

14. Simian Army
•
•
•
•
•
Chaos Monkey
Latency Monkey
Janitor Monkey
Conformity Monkey
(and more!)
Test resiliency at runtime
http://www.infoq.com/presentations/netflix-resiliency-failure-cloud

15. One Button Deployment?

16. Regional Isolation
Limit Impact of Human
Error
 Stagger deployments
 Canary testing per
region

17. Multi-Region Consistency
Build Tooling to:
 Schedule
Deployments
 Prefer off peak
 Choose next available
region automatically
 Provide high visibility
per region

18. Send in the Conformity Monkey
Have deployments diverged?
 Balance regional consistency
with regional isolation
 Provide meaningful
thresholds
 Build best practices into
tooling and reporting
http://techblog.netflix.com/2013/05/conformity-monkey-keeping-yourcloud.html

19. Clean up with Janitor Monkey
 Disassociate unused EIPs
 Delete unassociated Amazon
EBS volumes
 Delete older Amazon EBS
snapshots
 Leverage Amazon S3 Object
Expiration
https://github.com/Netflix/SimianArmy/wiki/Janitor-Home

20. Key Elements for Netflix
 Value Self-service
 Test Everywhere
 Build Awareness of Multiple Regions
 Avoid peak times
 Roll back quickly and easily
 Be Cloud Native

21. Put NetflixOSS to Work for You
Netflix Platform
AMINATOR
** And 30+ more projects at http://netflix.github.io/

22. Keep the Conversation Going
Continuous Delivery
Open Space
Ballroom B/C (here!)
1:35-2:25, immediately
following lunch

23. Thanks!
We’re always hiring!
Dianne Marsh (@dmarsh)
dmarsh@netflix.com