The document discusses Nuxeo's use of Jenkins for continuous integration over several years. Nuxeo started with Hudson in 2007 and migrated projects to Jenkins, growing the system from a few jobs to hundreds that test code across various environments. It helped improve quality, catch errors early and speed release cycles. Current Jenkins setups run over 450 jobs across 25 servers to test Nuxeo applications and platforms. Future plans include integrating with tools like Sonar and migrating Jenkins to the cloud.

1. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
Julien Carsique, DevOps
https://qa.nuxeo.org/jenkins

2. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The Nuxeo company — www.nuxeo.com
●
Founded in 2000
●
~ 50 employees (mainly France & USA)
●
Open Source Enterprise Content Management
●
No-charge software licensing with LGPL
●
Nuxeo Connect subscription offering for support,
maintenance, consulting, training and services:
●
●
Nuxeo Marketplace
●
17/04/2012
Nuxeo Studio
Nuxeo Cloud
2/25

3. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The Nuxeo products
Document and Case Management
17/04/2012
3/25

4. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The Nuxeo products
Digital Assets Management
17/04/2012
4/25

5. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The Nuxeo products
Web sites
17/04/2012
5/25

6. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The Nuxeo products
Public facing applications, RIA and much more …
17/04/2012
6/25

7. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The Nuxeo products
Open Source ECM platform, applications and services
●
Nuxeo Platform
●
Java and standards based
●
Component based architecture
●
Extensible and configurable ECM services
●
UI building blocks and High level frameworks
●
A lot of integration levels and APIs
●
Testing, benchmarking and packaging tools
●
Development tools (Nuxeo Studio, Nuxeo IDE)
●
Ready to use pre-packaged applications
17/04/2012
7/25

8. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The challenge of
The source code
●
200+ SubVersion, Mercurial and Git repositories
●
1M+ LoC, 500+ Maven modules
●
5 application modules and 60+ addons
●
500+ components in the Platform
●
2500+ unit tests, 2000+ Selenium and WebDriver tests
●
10+ languages (Java, XML, JavaScript, HTML, CSS,
Python, Shell, SQL, Groovy, …) and tools (Maven, Ant,
Shell, …)
●
30~50 active committers and contributors
●
~1000 commits/month
●
1 development branch, 3~5 maintenance branches
17/04/2012
8/25

9. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The challenge of
The human constraints
●
Follow and maintain quality
●
Save developers time
●
●
Providing a safety net for their changes
●
●
Doing more checks than they can
No need to build everything themselves
Save testers time
●
●
Warn on regression
●
●
Automate tests in various environments
Easy to download latest deliverables
Save integrators time
●
●
17/04/2012
Work on Nuxeo development branch with more confidence
Get guarantees on code quality
9/25

10. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The challenge of
The deployment constraints
●
Multiple deployment target
●
●
OS (Linux, Unix, Windows, MacOS)
●
DB (PostgreSQL, Oracle, SQLServer, MySQL, …)
●
●
WAR, JEE or bare Java
BlobStore backend (FS, NAS, S3, SQLDB, ...)
Scalability and performance
●
Cluster support
●
Multi-DB scale-out (several repositories and/or DB)
●
●
Raw performances (100M docs benchmark on a single server; 200
pages/s output on complex business apps)
Easy maintenance
●
Backup, restore, upgrade
●
export/import
17/04/2012
10/25

11. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
The beginning
●
Before 2007: BuildBot
●
2007 — Nuxeo 5 switch to Java
●
●
Trying CruiseControl, TeamCity, Continuum, … Hudson!
Goals for Nuxeo 5.1
●
Code analysis (Quality Assurance)
●
Build and unit test Nuxeo against default environment
–
Linux, JBoss, Sun JDK 5, H2
–
Smoke test at JBoss start
–
Run Selenium suite
●
●
Automate generation and test of documentation
●
17/04/2012
Automate Nuxeo release process
Involve developers
11/25

12. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.1 — CI architecture
Nexus Preprod
UPLOAD
UPLOAD
UPLOAD
Private Internal Nexus
DOWNLOAD
17/04/2012
RE
LEA
SE
S
DOWNLOAD
DOWNLOAD
Public Jenkins Master
UPLOAD RELEASES & SNAPSHOTS
SE
EA
RE
L
Public Internal Nexus
Maven.nuxeo.org
ES
AS
LE
RE
S
RELEASES and SNAPSHOTS
from outside
Private Jenkins Master
12/25

13. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.1 — CI architecture and usage
●
Speed up build time
●
Organize teams resources
●
Ensure publicly available artifacts quality
●
Isolate internal and external build chains
●
Enforce rules for developers
●
●
Early use of CI
Think about what kind of build (Maven, Shell, Ant, …) and what
must be checked for validation (unit tests, errors in logs, …)?
●
●
Make the build self-testing
●
Stay tuned, be aware of CI activity
●
17/04/2012
Automated build: nothing specific or contextual to the slave
Never let a project in an unstable status
13/25

14. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.1 to 5.3 — making heavy use of CI
●
Release process improvement
●
●
Nightly / On demand
●
●
Automated tasks
Build promotion
More and more tests and environments
●
●
Tests with Jetty, JBoss and Tomcat
●
Matrix testing jobs with OS and databases
●
●
More performance and functional tests
Testing development and administration tools
Seriously increasing number of builds
●
New packages and installers (Debian, VM, Windows, Multi-OS, …)
●
New applications (Android, Nuxeo CMF, Nuxeo DAM, …)
●
New plugins for Firefox, IE, OpenOffice/LibreOffice
●
Accumulating versions to maintain
17/04/2012
14/25

15. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.3 to 5.4 (2010-2011) — growing CI issues
●
CI performance issues
●
●
Slave usage strategy
●
●
CPU, network and disk usage
Full cycle duration
CI usage issues
●
●
Feedback delay
●
Feedback accuracy
●
17/04/2012
Build waiting time
Jobs' dependencies
15/25

16. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.4 (2011) — CI improvements, Jenkins analysis
●
●
Build cycles, workflows and durations
Use of Jenkviz (homemade tool), Dependency Graph View
and (enhanced) Build Timeout plugins
17/04/2012
16/25

17. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.4 (2011) — CI improvements, solutions
●
Work on source code
●
●
●
GWT optimizations (reduce build time)
Review testing suites,
prefer use of WebDriver over Selenium
Work on CI
●
●
Give priority to jobs with low duration (earlier feedback)
●
New short and permanent cycle
●
Create independent jobs for specific purposes
●
Create new views to help developers
●
Add CPU and/or memory on some slaves
●
Improve Nexus infrastructure (Maven build time)
●
17/04/2012
Improve slaves usage (using labels), local vs remote
Fix jobs triggering issue
17/25

18. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.5 — the current status
17/04/2012
18/25

19. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.5 — the current status
●
Great coverage and good delays
●
4 kinds of CI cycles per Nuxeo version maintained
–
Triggered by commit (300 jobs)
–
Full build and tests per day (10 jobs)
–
Independent cycle per 3 hours (11 jobs)
–
Nightly build (40 jobs)
●
Less than 10 minutes to 3 hours max for main feedback
●
Code and applications tested on multi-DB/OS/JDK/servers
●
6 hours for the longest build cycle
●
About 450 jobs on 25 servers (2 masters), 200~?K builds/day
●
6 database engines tested on 5 operating systems
●
7 applications tested in all environments
●
50K~100K tests ran including all products and versions under CI
17/04/2012
19/25

20. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.5 — the current status, major CI cycles
Job trigger
Commit
Mercurial/GitHub
SCM
trigger
Sources
checkout
Performance tests multi-DB
Build &
Unit tests
Packaging
7 apps, 2 servers
Functional tests multi-OS/DB
Various addons functional tests
Full build & tests multi-JDK
daily
trigger
Sources
checkout
Full build & tests multi-DB Linux
Packaging multi-OS Smoke tests
Full build & tests multi-DB Win
3h
trigger
Sources
checkout
Full build (no tests) Packaging all applications Functional tests
Merge « stable »
branch (GitHub)
Permanent cycle
Nightly
Manual
Sources
checkout
Full build (no tests)
Prepare release
Packaging all applications
Functional tests
multi-DB/OS/JDK/servers, cluster
Batch task
Perform release
Performance tests (référence)
Build installers (Debian,
Win, multi-OS, ...)
Merge «5.6-SNAPSHOT»
branch (GitHub)
Test and upload installers
Deploy «5.6-SNAPSHOT»
Maven artifacts
Push release tag (GitHub)
Deploy release artifacts
17/04/2012
20/25

21. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.5 — main Jenkins plugins used
●
Required
●
●
Useful for Jenkins management
●
●
●
Priority Sorter, Parameterized Trigger, Retry Failed Builds, Rebuilder,
Hudson Post build task, Matrix Reloaded, next-executions, Bulk Builder,
Join, Promoted Builds, batch task
Change Log History, Maven Deployment Linker, SCM Sync Configuration, Multiple
SCMs, Static Analysis Utilities, Timestamper, build timeout, Hudson Setenv,
java.io.tmpdir cleaner, Hudson global-build-stats, Node and Label parameter,
Monitoring, SSH Slaves, Dependency Graph Viewer, Performance, Disk Usage
Useful reports
●
●
Subversion, Mercurial, MSBuild, GIT, GitHub, Android Emulator
Checkstyle, FindBugs, Task Scanner, PMD, Warnings, Hudson Seleniumhq, Selenium
HTML report, Cobertura, TextFinder, JUnit Attachments, Duplicate Code Scanner
For end-users
●
17/04/2012
Dashboard View, Claim, Radiator View, Hudson iPhoneView, Continuous
Integration game, instant-messaging, Jabber notifier, built-on-column, Compact
Columns, Version Column, Email-ext, View Job Filters, Extra Columns, Simple Theme
21/25

22. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.6 — the near future
●
Better quality testing with Sonar
●
Integration with Jira
●
●
Automated merges from QA dedicated branches
to the development branch
Automated Jenkins infrastructure deployment on
cloud
●
Custom slaves on demand
(Chef / Puppet, chroot / LXC, …)
●
●
●
Work on Nuxeo Cloud & Compatible One R&D
CloudBees
Nuxeo CI (partial or full) migration to the cloud
17/04/2012
22/25

23. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
The history of
Nuxeo 5.6 — the near future
●
Jenkins enhancements (plugins)
●
●
Multiple SCM & Git
●
SCM change log feeder
●
Matrix jobs improvements
●
Node Label & Parameterized trigger
●
Dependency Graph
●
Infrastructure failures (Check job prerequisites++)
●
Jobs workflow management
●
17/04/2012
More abstraction to avoid plugins compliance
…
23/25

24. Jenkins User Conference
Paris, 17 April 2012
#jenkinsconf
Conclusion
●
Jenkins
●
●
Good scalability
●
Great community and plugins catalog
●
Powerful extensibility
●
Efficient contribution process
●
●
Easy maintenance
Cloud capabilities
Nuxeo
●
●
Stable APIs with backward and forward compatibility
●
Stable applications with good performances
●
17/04/2012
CI & QA on Nuxeo Platform frameworks
Automated processes with very few manual interventions
24/25

25. 25
17/04/2012
Jenkins User Conference Paris
17 April 2012
Thank You To Our Sponsors
Platinum Sponsor
Gold Sponsor
Silver Sponsor
#jenkinsconf