This document discusses Netflix's continuous delivery pipeline which allows for frequent code deployments. It describes how Netflix uses tools like Jenkins, Chef, and Asgard to automate building, testing, and deploying code. It also covers how Netflix tests deployments, monitors services, and has mechanisms like red/black deployments and the Simian Army to improve resiliency.

1. Continuous Delivery at
Netflix
Speed at Scale
Monday, December 23, 13

2. Rob Spieldenner
Senior Software Engineer on Engineering Tools at
Netflix
@robspieldenner -> Twitter
Monday, December 23, 13

3. Netflix Culture
Monday, December 23, 13

4. Culture Deck
http://jobs.netflix.com/
Monday, December 23, 13

5. Freedom & Responsibility
Monday, December 23, 13

6. Context, not Control
Monday, December 23, 13

7. Self Service
Monday, December 23, 13

8. Developers on call for their
services
Monday, December 23, 13

9. Netflix OSS
http://netflix.github.io/
Monday, December 23, 13

10. OODA Loop - Observe,
Orient, Decide, Act
Monday, December 23, 13

11. Continuous Delivery Pipeline
Commit
Monday, December 23, 13
Build
Unit Test
Bake
Deploy To
Test
Integration &
Acceptance
Tests
Red/Black
Push

12. Commit and Code Review
Monday, December 23, 13

13. Build
simplify build file with plugins
add metadata
calculate version
output jar/war, sources, javadoc, rpm/deb
Monday, December 23, 13

14. Ant
build.xml
<project name="example">
<property environment="env"/>
<import file="${env.CBF_HOME}/library.xml"/>
</project>
ivy.xml
list dependencies
list publications
Monday, December 23, 13

15. Gradle
build.gradle
apply plugin: ‘nebula’
publications {
JAR
JAVADOC
SOURCES
}
dependencies {
testCompile ‘junit:junit:4.+’
}
Monday, December 23, 13

16. Jenkins
Monday, December 23, 13

17. Job-DSL
https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL
+Plugin
Monday, December 23, 13

18. Job DSL Example
example
def project = 'quidryan/aws-sdk-test'
def branchApi = new URL(“https://api.github.com/repos/${project}/branches”)
def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader())
branches.each {
def branchName = it.name
job {
name “${project}-${branchName}”.replaceAll(‘/’, ‘-’)
scm {
git(“git://github.com/${project}.git”, branchName)
}
steps {
gradle(‘test’)
}
}
}
Monday, December 23, 13

19. Bake
Monday, December 23, 13

20. aminator
https://github.com/Netflix/aminator
Monday, December 23, 13

21. chef for base image
Monday, December 23, 13

22. Apply deb/rpm, snapshot
Monday, December 23, 13

23. Unit of deployment
Monday, December 23, 13

24. Deploy
Monday, December 23, 13

25. Asgard
https://github.com/Netflix/asgard
Monday, December 23, 13

26. Monday, December 23, 13

27. UI or REST API
Monday, December 23, 13

28. Canary Deploys
Monday, December 23, 13

29. Red / Black Push
Monday, December 23, 13

30. Testing
Webdriver
Geb
Betamax
and manual
Monday, December 23, 13

31. Monitoring
How many people are watching?
Can the new version handle load?
Errors and logging
suro, jmx
Monday, December 23, 13

32. Alert on issues
Monday, December 23, 13

33. Maybe automatic rollback
Monday, December 23, 13

34. Resiliency / Anti-Fragility
Monday, December 23, 13

35. Simian Army
https://github.com/Netflix/SimianArmy
Chaos Monkey
Chaos Gorilla
Chaos Kong
Latency Monkey
Janitor Monkey
Conformity Monkey
Monday, December 23, 13

36. Assume services will break,
disappear, slow
Monday, December 23, 13

37. Books
Monday, December 23, 13

38. Other Presentations
Adrian Cockcroft Flowcon Keynote - Velocity and
Volume (or Speed Wins) - http://www.youtube.com/
watch?v=wyWI3gLpB8o
Roy Rapoport - Cloud Operations at Netflix - http://
www.youtube.com/watch?v=7779Wrun5fo
AWS re:Invent - http://reinvent.awsevents.com/
recap2013.html
Monday, December 23, 13

39. Netflix is Hiring
http://jobs.netflix.com/
Monday, December 23, 13