PayPal moved from a single large Jenkins instance to using containers to improve scalability and efficiency of their continuous integration and delivery (CI/CD) pipelines. They deployed Jenkins masters and slaves as Docker containers on a Mesos cluster. This reduced resource usage by 10x, saving over $2.7 million annually. Issues with inconsistent environments and slow tooling updates were also addressed by using containers to standardize environments.

1. Running CI at scale with containers at PayPal
Sathiya Narayanan

2. © 2015 PayPal Inc. All rights reserved. Confidential and proprietary. 2
At PayPal, we take CI seriously

3. ©2015 PayPal Inc. Confidential and proprietary. 3
o One Ginormous Jenkins Instance, customized for
PayPal
o 40,000 runnable jobs
o 24,000 builds run each day on Single Instance
o Stack: Java, Node, Python, C++
o CD : UnitTest, Static analysis and code coverage, &
deployments
o One Jenkins master with a pool of a hundred
Jenkins slaves
o Custom build system for 50 MLOC in <1 Min
Single Jenkins instance
One Giant Jenkins
Butler!

4. ©2015 PayPal Inc. Confidential and proprietary. 4
o SPOF
o Scalability
o Change Management
o No freedom for users
o Inefficient resource management
o Plugins don’t scale
Limitations with single Jenkins instance
Too much load!

5. ©2015 PayPal Inc. Confidential and proprietary. 5
o 1:1 model
o Individual VMs running Jenkins for each Application
o 2,500+VMs in use
o Each Jenkins instance would have 2 executor on
master to perform builds
o Freedom for users
Users loved it!
Dedicated VM for each Jenkins Instance

6. Jenkins on Mesos
o Light weight
o Long running task
o Marathon scheduler
©2015 PayPal Inc. Confidential and proprietary. 6
Jenkins Master
o Provisioned on demand with Jenkins
Mesos plugin
o Flavors of slaves based on the application
stack
Jenkins slave

7. ©2015 PayPal Inc. Confidential and proprietary. 7
Optimized Resource Utilization with Mesos

8. ©2015 PayPal Inc. Confidential and proprietary. 8
Operating cost reduced 10X times
After Mesos
o ~2400 CPU’s,
o 18 TB,
o 7.2 TB RAM
Before Mesos
o ~ 24,000 CPU’s,
o 180 TB Disk,
o 54 TB RAM

9. ©2015 PayPal Inc. Confidential and proprietary. 9
After:
~2400 CPU’s, 18 TB, 7.2 TB RAM
~300K
2.7 million saving could buy:
- A personal Jet for GPI
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Recovery was slow without Mesos

10. ©2015 PayPal Inc. Confidential and proprietary. 10
1
0
After:
~2400 CPU’s, 18 TB, 7.2 TB RAM
~300K
2.7 million saving could buy:
- A personal Jet for GPI
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Instant Recovery with Mesos

11. ©2015 PayPal Inc. Confidential and proprietary. 11
After:
~2400 CPU’s, 18 TB, 7.2 TB RAM
~300K
2.7 million saving could buy:
- A personal Jet for GPI
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Our workload was too much for Marathon

12. ©2015 PayPal Inc. Confidential and proprietary. 12
o Zero outages and no customer impact.
o Aurora was more stable
o Aurora was able to scale to more than 5,000 jobs
Replaced Marathon with Aurora

13. ©2015 PayPal Inc. Confidential and proprietary. 13
After:
~2400 CPU’s, 18 TB, 7.2 TB RAM
~300K
2.7 million saving could buy:
- A personal Jet for GPI
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
13
CIAPI
Aurora
Mesos Master
Mesos Cluster
Swift / Object
Storage
Zookeeper
Jenkins Master Jenkins Slave
Nginx
Highly available setup
Devxdash

14. Challenges
©2015 PayPal Inc. Confidential and proprietary. 14
o Inconsistent build environments between user
workstation and CI build system
o Tasks stepping onto each other
o Providing conflicting tool set for different
technology stacks
o Different DevOps teams/applications had their own
Mesos cluster
o Slower rollout of new tools to the build environment
o Inconsistent build environments within the CI build
system

15. Why Docker?
o Task isolation
o Cloud Native
o Eliminates the dependency on the host
o Reproducibility
o Portability
o Immutability
o Homogenous cluster
©2015 PayPal Inc. Confidential and proprietary. 15

16. ©2015 PayPal Inc. Confidential and proprietary. 16
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Dockerized Jenkins Masters
o Immutable containers for
consistency
o Eliminates the need to have
tools installed on slave
o Provides task isolation
o Other work loads can co-exist in
Mesos cluster
o Easy to cater to conflicting OS / toolset
requirements in a polyglot environment
o Easy to roll out updates
o Other work loads can co-exist in Mesos cluster
o Users can bake their own build environment
Dockerized Jenkins Slaves
Jenkins with Docker

17. ©2015 PayPal Inc. Confidential and proprietary. 17
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Adoption to Docker containers
BC [ Before Containers ] AD [ After Docker ]
SSH key / Tokens /
Maven credentials
Stored in local disk Injected only during build
time and rotating them
Toolsets like JDK, Maven
etc.
Installed in host Installed within Docker
container

18. ©2015 PayPal Inc. Confidential and proprietary. 18
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
What is state for Jenkins master ?
o Jenkins war
o Plugins
o Global configuration
o Job configuration
o Build history
o Console output
o Reports like code coverage, test result trend etc.
o Archived artifacts

19. ©2015 PayPal Inc. Confidential and proprietary. 19
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Backing up Jenkins state
Superviso
rd
Archive
the
directory
Encrypt
the data
Store in
cloud storage

20. ©2015 PayPal Inc. Confidential and proprietary. 20
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
When Jenkins Master container moves to another VM
Untar the
files into
Jenkins
home
Decrypt
the
backup
Check if a backup
exists in cloud
storage
Provisioning
script [Entry
point for Docker]

21. ©2015 PayPal Inc. Confidential and proprietary. 21
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
Use case - Maven build
Total build time - 4 minutes
Persisting workspace for subsequent builds
Decreased build time
Source code
[Clone]
Artifacts
Time taken 0m27.618s 2m56.000s
Number of files 10570 4456
Total Size 158 M 173 M

22. ©2015 PayPal Inc. Confidential and proprietary. 22
10X Savings
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million
CIAAS with Docker Architecture
Public cloud
Private cloud

23. ©2015 PayPal Inc. Confidential and proprietary. 23
After:
~2400 CPU’s, 18 TB, 7.2 TB RAM
~300K
2.7 million saving could buy:
- A personal Jet for GPI
Before:
~ 24,000 CPU’s, 180 TB Disk, 54 TB RAM
~3 Million