From Virtual Machines to Containers
Achieving Continuous Integration, Build Reproducibility, Isolation and Scalability
London, UK, June 2015
1

#jenkinsconf
Agenda
• Introduction
• The Dark Age
• The Promising Present
• Lessons Learned
• The Bright Future

#jenkinsconf
2014201320122011201020092008
Incorporation
Camunda Services GmbH
Berlin, Germany
Incorporation
Camunda Inc.
San Francisco, US
BPM Consulting BPM Software Vendor
● 30 Fulltime Employees ● No external funding ● Avg. 50% Turnover Growth per Year
Introduction - About Camunda
@camundaBPM
Introduced
Hudson
Adapted
Jenkins

#jenkinsconf
Introduction - Camunda BPM

#jenkinsconf
Introduction - What is Camunda BPM
• Camunda BPM is an open source platform
for workflow and business process automation
• Integrates with:
• 7 Application Server (11 different versions)
• 6 Databases (17 different versions)
• 1 Development & 4 Maintained Versions

#jenkinsconf
• Every Camunda BPM version is tested against:
• 187 combinations of DBs and App Servers
• 11 JDKs
• ~ 400 Jobs per version
• Bi-Annual release of a new Camunda BPM version
• Support for Enterprise Customer (24/7 + Fix Time)
Introduction - Why CI is Important

#jenkinsconf
The Dark Age
Why our CI had to change
7

#jenkinsconf
The Dark Age - The Numbers
• 1 Jenkins Master with alot of plugins
• 8 Jenkins Slaves VMs
• ~1000 Jobs total configured / manually managed:
• 4 Camunda Versions
• Community Projects
• Websites
• Maintenance

#jenkinsconf
The Dark Age - Isolation Problem
• Unit and Integration Tests need a
database/application server
• Only 1 Instance per Database
• All Jobs use the same Databases
• Every half-year a new Version (~400 Jobs) using
same Databases

#jenkinsconf
The Dark Age - No Scalability
• Total Executors: 12
• 1 Jenkins -> 4 Executors
• 8 static heterogenous slave VMs, each with 1
Executor
• Jobs tied to slaves through labels
• Slaves “restrict” database access by allowing no
other build to run

#jenkinsconf
The Dark Age - Maintenance Problems
• Upgrading Jenkins or any plugin
• Supporting a new Database vendor / version
• Supporting a new App Server version
• Creating jobs for new Camunda BPM version
• Disaster recovery

#jenkinsconf
The Dark Age - The Other Problems
• Slow feedback cycle for developers
• Developers cannot reproduce CI environments
• QA engineers use and maintain separate bloated
test build setup

#jenkinsconf
So ...

#jenkinsconf
The Promising Present
How we solved our Problem
14

#jenkinsconf
The Present - What we achieved
• Configuration & Infrastructure as Code
• Isolated and Reproducible Jobs
• Scalable CI Infrastructure

#jenkinsconf
The Present - Infrastructure as Code
1. Every configuration is checked into SCM
2. Every application/test runs in a Docker
Container
3. Every Docker image is build
automatically

#jenkinsconf
The Present - Infrastructure as Code
1. Every Configuration is checked into SCM
• Docker for
• Applications
• Test Environments
• JobDSL for
• Jenkins Jobs

#jenkinsconf
The Present - Infrastructure as Code
2. Every application/test runs in a Docker Container
Images:
• Application (Jenkins, Nexus …)
• Test Env. Images (DB + SSH)
• Build Env. Images
• DIND, QEMU + Packer.io

#jenkinsconf
The Present - Infrastructure as Code
3. Every Docker Container is build automatically
• Own Jenkins for Docker/KVM Images
• KVM Images build in Docker Container
with Packer + QEMU
• KVM Images bundled in Docker Image

#jenkinsconf
The Present - The Current Flow
camunda-ci
camunda
Camunda BPM
Platform
Infrastructure
Jenkins
CI Jenkins

#jenkinsconf
The Present - Isolation
One Jenkins per Concern:
• CI
• Release
• Infrastructure
• Community and other Projects
• Marketing

#jenkinsconf
The Present - Isolation & Reproducibility
• Every Jobs runs in an One-Shot Docker Container
• No Interference between Jobs
• The Database Settings are well documented
• Every Docker Image is stored in a private registry
• Developers/QA can use the Docker Images for local
testing

#jenkinsconf
The Present - Scalability
• Jenkins uses Docker-Plugin with one Docker Cloud
running on Docker Swarm
• Docker images are added through Groovy scripting
• Running on Commodity Hardware
• 1 Infrastructure Host (Jenkins, Nexus, …)
• 6 Docker Hosts as 1 Swarm

#jenkinsconf
The Present - Scalability

#jenkinsconf
The Present - Advantages
• Easy to add new Databases/Test Environments
• New Release = New Branch of JobDSL Repository
• Fully parallelized Job Execution
• Accountable Configuration History
• Testable Infrastructure
• Minimize Administration Overhead

#jenkinsconf
The Present - Conclusion
• 2 People + 3 Months of Work
• A fully scalable, isolated and reproducible CI
Infrastructure
• Faster Feedback
• Happy Developers and Product Owner

#jenkinsconf
Lessons learned
27
Some advice from us to you

#jenkinsconf
• Automate as much as you can
• Jenkins config
• Jobs config
• Environment creation
• Design to scale to support the business agility
Lessons learned - Architecture

#jenkinsconf
Lessons learned - Testing
Test everything:
• Jenkins configuration
• required plugin features
• Job generation
• Docker images
• Scalability
• Disaster recovery

#jenkinsconf
Lessons learned - Job DSL
• Unit-test the job generation
• Write JobGenerator classes to abstract the
common build logic of most jobs out of the box
• Use XML diffing to compare previously generated
jobs with new ones

#jenkinsconf
Lessons learned - Docker
https://github.com/rancherio/vm/

#jenkinsconf
• Pin your plugin versions
• Be prepared to contribute to plugin development or
maintain a branch yourself
• Choose the right plugin for the job
Our Top 3 plugins:
JobDSL, Docker-Plugin, Build-Failure-Analyzer
Lessons learned - Plugins

#jenkinsconf
Lessons learned - Control
• Control as much as possible
• Third party binaries vs package manager
• explicit versions
• own mirrors for important packages

#jenkinsconf
The Bright Future
What comes next
34

#jenkinsconf
The Future
• Public Community Jenkins
• Internal Webapp for Developers and QA to start
Environments
• Jenkins is deploying Jenkins
• Back to the Datacenter
• Centralized Logging and Monitoring (ELK)

#jenkinsconf

#jenkinsconf
37

#jenkinsconf
Resources
• Camunda CI repository
https://github.com/camunda-ci
• Job DSL gradle example
https://github.com/sheehan/job-dsl-gradle-example
• Camunda BPM Docker image
https://registry.hub.docker.
com/u/camunda/camunda-bpm-platform/

#jenkinsconf
We Are Hiring
Come Join Us
@camundaBPM
http://camunda.com/jobs

#jenkinsconf
Please Share Your Feedback
• Did you find this session valuable?
• Please share your thoughts in the
Jenkins User Conference Mobile App.
• Find the session in the app and click
on the feedback area.
40