Learn how Nuxeo has been using Jenkins for the past five years.

1. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
Julien Carsique, DevOps
https://qa.nuxeo.org/jenkins

2. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The Nuxeo company — www.nuxeo.com
●
Founded in 2000
●
~ 50 employees (mainly France & USA)
●
Open Source Enterprise Content Management
●
No-charge software licensing with LGPL
●
Nuxeo Connect subscription offering for support,
maintenance, consulting, training and services:
●
Nuxeo Studio
●
Nuxeo Marketplace
●
Nuxeo Cloud
17/04/2012 2/25

3. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The Nuxeo products
Document and Case Management
17/04/2012 3/25

4. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The Nuxeo products
Digital Assets Management
17/04/2012 4/25

5. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The Nuxeo products
Web sites
17/04/2012 5/25

6. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The Nuxeo products
Public facing applications, RIA and much more …
17/04/2012 6/25

7. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The Nuxeo products
Open Source ECM platform, applications and services
●
Nuxeo Platform
●
Java and standards based
●
Component based architecture
●
Extensible and configurable ECM services
●
UI building blocks and High level frameworks
●
A lot of integration levels and APIs
●
Testing, benchmarking and packaging tools
●
Development tools (Nuxeo Studio, Nuxeo IDE)
●
Ready to use pre-packaged applications
17/04/2012 7/25

8. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The challenge of
The source code
●
200+ SubVersion, Mercurial and Git repositories
●
1M+ LoC, 500+ Maven modules
●
5 application modules and 60+ addons
●
500+ components in the Platform
●
2500+ unit tests, 2000+ Selenium and WebDriver tests
●
10+ languages (Java, XML, JavaScript, HTML, CSS,
Python, Shell, SQL, Groovy, …) and tools (Maven, Ant,
Shell, …)
●
30~50 active committers and contributors
●
~1000 commits/month
●
1 development branch, 3~5 maintenance branches
17/04/2012 8/25

9. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The challenge of
The human constraints
●
Follow and maintain quality
●
Save developers time
●
Doing more checks than they can
●
Providing a safety net for their changes
●
No need to build everything themselves
●
Save testers time
●
Automate tests in various environments
●
Warn on regression
●
Easy to download latest deliverables
●
Save integrators time
●
Work on Nuxeo development branch with more confidence
●
Get guarantees on code quality
17/04/2012 9/25

10. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The challenge of
The deployment constraints
●
Multiple deployment target
●
WAR, JEE or bare Java
●
OS (Linux, Unix, Windows, MacOS)
●
DB (PostgreSQL, Oracle, SQLServer, MySQL, …)
●
BlobStore backend (FS, NAS, S3, SQLDB, ...)
●
Scalability and performance
●
Cluster support
●
Multi-DB scale-out (several repositories and/or DB)
●
Raw performances (100M docs benchmark on a single server; 200
pages/s output on complex business apps)
●
Easy maintenance
●
Backup, restore, upgrade
●
export/import
17/04/2012 10/25

11. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
The beginning
●
Before 2007: BuildBot
●
2007 — Nuxeo 5 switch to Java
●
Trying CruiseControl, TeamCity, Continuum, … Hudson!
●
Goals for Nuxeo 5.1
●
Code analysis (Quality Assurance)
●
Build and unit test Nuxeo against default environment
– Linux, JBoss, Sun JDK 5, H2
– Smoke test at JBoss start
– Run Selenium suite
●
Automate Nuxeo release process
●
Automate generation and test of documentation
●
Involve developers
17/04/2012 11/25

12. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.1 — CI architecture
RELEASES and SNAPSHOTS Maven.nuxeo.org
from outside
RE
UPLOAD RELEASES & SNAPSHOTS
RE
S
LEA
LE
SE
SE
S
AS
EA
L
ES
RE
Public Internal Nexus Private Internal Nexus Nexus Preprod
DOWNLOAD
DOWNLOAD
UPLOAD
UPLOAD
DOWNLOAD
UPLOAD
Public Jenkins Master Private Jenkins Master
17/04/2012 12/25

13. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.1 — CI architecture and usage
●
Speed up build time
●
Organize teams resources
●
Ensure publicly available artifacts quality
●
Isolate internal and external build chains
●
Enforce rules for developers
●
Early use of CI
●
Think about what kind of build (Maven, Shell, Ant, …) and what
must be checked for validation (unit tests, errors in logs, …)?
●
Automated build: nothing specific or contextual to the slave
●
Make the build self-testing
●
Stay tuned, be aware of CI activity
●
Never let a project in an unstable status
17/04/2012 13/25

14. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.1 to 5.3 — making heavy use of CI
●
Release process improvement
●
Automated tasks
●
Nightly / On demand
●
Build promotion
●
More and more tests and environments
●
More performance and functional tests
●
Tests with Jetty, JBoss and Tomcat
●
Matrix testing jobs with OS and databases
●
Testing development and administration tools
●
Seriously increasing number of builds
●
New packages and installers (Debian, VM, Windows, Multi-OS, …)
●
New applications (Android, Nuxeo CMF, Nuxeo DAM, …)
●
New plugins for Firefox, IE, OpenOffice/LibreOffice
●
Accumulating versions to maintain
17/04/2012 14/25

15. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.3 to 5.4 (2010-2011) — growing CI issues
●
CI performance issues
●
CPU, network and disk usage
●
Slave usage strategy
●
Full cycle duration
●
CI usage issues
●
Build waiting time
●
Feedback delay
●
Feedback accuracy
●
Jobs' dependencies
17/04/2012 15/25

16. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.4 (2011) — CI improvements, Jenkins analysis
●
Build cycles, workflows and durations
●
Use of Jenkviz (homemade tool), Dependency Graph View
and (enhanced) Build Timeout plugins
17/04/2012 16/25

17. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.4 (2011) — CI improvements, solutions
●
Work on source code
●
GWT optimizations (reduce build time)
●
Review testing suites,
prefer use of WebDriver over Selenium
●
Work on CI
●
Improve slaves usage (using labels), local vs remote
●
Give priority to jobs with low duration (earlier feedback)
●
New short and permanent cycle
●
Create independent jobs for specific purposes
●
Create new views to help developers
●
Add CPU and/or memory on some slaves
●
Improve Nexus infrastructure (Maven build time)
●
Fix jobs triggering issue
17/04/2012 17/25

18. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.5 — the current status
17/04/2012 18/25

19. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.5 — the current status
●
Great coverage and good delays
●
4 kinds of CI cycles per Nuxeo version maintained
– Triggered by commit (300 jobs)
– Full build and tests per day (10 jobs)
– Independent cycle per 3 hours (11 jobs)
– Nightly build (40 jobs)
●
Less than 10 minutes to 3 hours max for main feedback
●
Code and applications tested on multi-DB/OS/JDK/servers
●
6 hours for the longest build cycle
●
About 450 jobs on 25 servers (2 masters), 200~?K builds/day
●
6 database engines tested on 5 operating systems
●
7 applications tested in all environments
●
50K~100K tests ran including all products and versions under CI
17/04/2012 19/25

20. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.5 — the current status, major CI cycles
Job trigger
Performance tests multi-DB
Commit SCM Sources Build & Packaging
Functional tests multi-OS/DB
Mercurial/GitHub trigger checkout Unit tests 7 apps, 2 servers
Various addons functional tests
Full build & tests multi-JDK
daily Sources Packaging multi-OS Smoke tests
trigger Full build & tests multi-DB Linux
checkout
Full build & tests multi-DB Win
3h Sources Merge « stable »
trigger Full build (no tests) Packaging all applications Functional tests
checkout branch (GitHub)
Permanent cycle
Nightly
Sources Full build (no tests) Functional tests
Packaging all applications multi-DB/OS/JDK/servers, cluster
Manual checkout Prepare release
Batch task
Performance tests (référence)
Perform release Build installers (Debian, Merge «5.6-SNAPSHOT»
Win, multi-OS, ...) branch (GitHub)
Push release tag (GitHub)
Deploy «5.6-SNAPSHOT»
Test and upload installers
Deploy release artifacts Maven artifacts
17/04/2012 20/25

21. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.5 — main Jenkins plugins used
●
Required
●
Subversion, Mercurial, MSBuild, GIT, GitHub, Android Emulator
●
Useful for Jenkins management
●
Priority Sorter, Parameterized Trigger, Retry Failed Builds, Rebuilder,
Hudson Post build task, Matrix Reloaded, next-executions, Bulk Builder,
Join, Promoted Builds, batch task
●
Change Log History, Maven Deployment Linker, SCM Sync Configuration, Multiple
SCMs, Static Analysis Utilities, Timestamper, build timeout, Hudson Setenv,
java.io.tmpdir cleaner, Hudson global-build-stats, Node and Label parameter,
Monitoring, SSH Slaves, Dependency Graph Viewer, Performance, Disk Usage
●
Useful reports
●
Checkstyle, FindBugs, Task Scanner, PMD, Warnings, Hudson Seleniumhq, Selenium
HTML report, Cobertura, TextFinder, JUnit Attachments, Duplicate Code Scanner
●
For end-users
●
Dashboard View, Claim, Radiator View, Hudson iPhoneView, Continuous
Integration game, instant-messaging, Jabber notifier, built-on-column, Compact
Columns, Version Column, Email-ext, View Job Filters, Extra Columns, Simple Theme
17/04/2012 21/25

22. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.6 — the near future
●
Better quality testing with Sonar
●
Integration with Jira
●
Automated merges from QA dedicated branches
to the development branch
●
Automated Jenkins infrastructure deployment on
cloud
●
Custom slaves on demand
(Chef / Puppet, chroot / LXC, …)
●
Work on Nuxeo Cloud & Compatible One R&D
●
CloudBees
●
Nuxeo CI (partial or full) migration to the cloud
17/04/2012 22/25

23. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
The history of
Nuxeo 5.6 — the near future
●
Jenkins enhancements (plugins)
●
More abstraction to avoid plugins compliance
●
Multiple SCM & Git
●
SCM change log feeder
●
Matrix jobs improvements
●
Node Label & Parameterized trigger
●
Dependency Graph
●
Infrastructure failures (Check job prerequisites++)
●
Jobs workflow management
●
…
17/04/2012 23/25

24. Jenkins User Conference Paris, 17 April 2012 #jenkinsconf
Conclusion
●
Jenkins
●
Easy maintenance
●
Good scalability
●
Great community and plugins catalog
●
Powerful extensibility
●
Efficient contribution process
●
Cloud capabilities
●
Nuxeo
●
CI & QA on Nuxeo Platform frameworks
●
Stable APIs with backward and forward compatibility
●
Stable applications with good performances
●
Automated processes with very few manual interventions
17/04/2012 24/25

25. 25
17/04/2012
Jenkins User Conference Paris 17 April 2012 #jenkinsconf
Thank You To Our Sponsors
Platinum Sponsor
Gold Sponsor
Silver Sponsor