The document discusses Camunda's transition from a traditional Jenkins setup with virtual machines to a containerized continuous integration infrastructure using Docker and Jenkins. Some of the key problems with the previous setup included a lack of isolation between jobs, limited scalability, and difficulties maintaining the infrastructure. The new system achieves isolated and reproducible jobs through one-off Docker containers, scalability through Docker Swarm on commodity hardware, and infrastructure maintenance through immutable Docker images and infrastructure as code definitions. Lessons learned include automating as much as possible, designing for scale, testing all aspects of the new system, and controlling dependencies.
5. #jenkinsconf
Introduction - What is Camunda BPM
• Camunda BPM is an open source platform
for workflow and business process automation
• Integrates with:
• 7 Application Server (11 different versions)
• 6 Databases (17 different versions)
• 1 Development & 4 Maintained Versions
6. #jenkinsconf
• Every Camunda BPM version is tested against:
• 187 combinations of DBs and App Servers
• 11 JDKs
• ~ 400 Jobs per version
• Bi-Annual release of a new Camunda BPM version
• Support for Enterprise Customer (24/7 + Fix Time)
Introduction - Why CI is Important
8. #jenkinsconf
The Dark Age - The Numbers
• 1 Jenkins Master with alot of plugins
• 8 Jenkins Slaves VMs
• ~1000 Jobs total configured / manually managed:
• 4 Camunda Versions
• Community Projects
• Websites
• Maintenance
9. #jenkinsconf
The Dark Age - Isolation Problem
• Unit and Integration Tests need a
database/application server
• Only 1 Instance per Database
• All Jobs use the same Databases
• Every half-year a new Version (~400 Jobs) using
same Databases
10. #jenkinsconf
The Dark Age - No Scalability
• Total Executors: 12
• 1 Jenkins -> 4 Executors
• 8 static heterogenous slave VMs, each with 1
Executor
• Jobs tied to slaves through labels
• Slaves “restrict” database access by allowing no
other build to run
11. #jenkinsconf
The Dark Age - Maintenance Problems
• Upgrading Jenkins or any plugin
• Supporting a new Database vendor / version
• Supporting a new App Server version
• Creating jobs for new Camunda BPM version
• Disaster recovery
12. #jenkinsconf
The Dark Age - The Other Problems
• Slow feedback cycle for developers
• Developers cannot reproduce CI environments
• QA engineers use and maintain separate bloated
test build setup
15. #jenkinsconf
The Present - What we achieved
• Configuration & Infrastructure as Code
• Isolated and Reproducible Jobs
• Scalable CI Infrastructure
16. #jenkinsconf
The Present - Infrastructure as Code
1. Every configuration is checked into SCM
2. Every application/test runs in a Docker
Container
3. Every Docker image is build
automatically
17. #jenkinsconf
The Present - Infrastructure as Code
1. Every Configuration is checked into SCM
• Docker for
• Applications
• Test Environments
• JobDSL for
• Jenkins Jobs
18. #jenkinsconf
The Present - Infrastructure as Code
2. Every application/test runs in a Docker Container
Images:
• Application (Jenkins, Nexus …)
• Test Env. Images (DB + SSH)
• Build Env. Images
• DIND, QEMU + Packer.io
19. #jenkinsconf
The Present - Infrastructure as Code
3. Every Docker Container is build automatically
• Own Jenkins for Docker/KVM Images
• KVM Images build in Docker Container
with Packer + QEMU
• KVM Images bundled in Docker Image
20. #jenkinsconf
The Present - The Current Flow
camunda-ci
camunda
Camunda BPM
Platform
Infrastructure
Jenkins
CI Jenkins
21. #jenkinsconf
The Present - Isolation
One Jenkins per Concern:
• CI
• Release
• Infrastructure
• Community and other Projects
• Marketing
22. #jenkinsconf
The Present - Isolation & Reproducibility
• Every Jobs runs in an One-Shot Docker Container
• No Interference between Jobs
• The Database Settings are well documented
• Every Docker Image is stored in a private registry
• Developers/QA can use the Docker Images for local
testing
23. #jenkinsconf
The Present - Scalability
• Jenkins uses Docker-Plugin with one Docker Cloud
running on Docker Swarm
• Docker images are added through Groovy scripting
• Running on Commodity Hardware
• 1 Infrastructure Host (Jenkins, Nexus, …)
• 6 Docker Hosts as 1 Swarm
25. #jenkinsconf
The Present - Advantages
• Easy to add new Databases/Test Environments
• New Release = New Branch of JobDSL Repository
• Fully parallelized Job Execution
• Accountable Configuration History
• Testable Infrastructure
• Minimize Administration Overhead
26. #jenkinsconf
The Present - Conclusion
• 2 People + 3 Months of Work
• A fully scalable, isolated and reproducible CI
Infrastructure
• Faster Feedback
• Happy Developers and Product Owner
28. #jenkinsconf
• Automate as much as you can
• Jenkins config
• Jobs config
• Environment creation
• Design to scale to support the business agility
Lessons learned - Architecture
30. #jenkinsconf
Lessons learned - Job DSL
• Unit-test the job generation
• Write JobGenerator classes to abstract the
common build logic of most jobs out of the box
• Use XML diffing to compare previously generated
jobs with new ones
32. #jenkinsconf
• Pin your plugin versions
• Be prepared to contribute to plugin development or
maintain a branch yourself
• Choose the right plugin for the job
Our Top 3 plugins:
JobDSL, Docker-Plugin, Build-Failure-Analyzer
Lessons learned - Plugins
33. #jenkinsconf
Lessons learned - Control
• Control as much as possible
• Third party binaries vs package manager
• explicit versions
• own mirrors for important packages
35. #jenkinsconf
The Future
• Public Community Jenkins
• Internal Webapp for Developers and QA to start
Environments
• Jenkins is deploying Jenkins
• Back to the Datacenter
• Centralized Logging and Monitoring (ELK)
40. #jenkinsconf
Please Share Your Feedback
• Did you find this session valuable?
• Please share your thoughts in the
Jenkins User Conference Mobile App.
• Find the session in the app and click
on the feedback area.
40