1.
CI is dead, long live CI!
Frédéric Lepied
Senior Engineering Manager
Member of Technical Staff
Nov 2015

2.
POSS 20152
AGENDA
● Staging CI
● Gating CI
● DIY

3.
Staging CI

4.
POSS 2015
Definition
4
Staging CI
Pipeline of jobs triggered on a new commit in a VCS
Review Change Commit Change Build Artifacts Run Tests

5.
POSS 2015
What do we do when a job fails?
5
Staging CI
According to Jez Humble and David Farley in the reference book
Continuous Delivery:
● “Don’t check in on a broken Build”
● To fix the issue:
○ “Never Go Home on a Broken Build”
○ “Always Be Prepared to Revert to the Previous Revision”

6.
POSS 2015
Mitigation Strategies
6
Staging CI
● Use an integration branch before merging to master
● Run all the tests on the developer workstation before submitting

7.
POSS 2015
Next?
7
Staging CI
So you need discipline to avoid a broken CI.
● Difficult at scale
● Difficult on Free Software projects as the level of implication is not
equal between participants

8.
Gating CI

9.
POSS 2015
Introduction to next gen CI
9
Gating CI
CI is now centered on the code review system:
Review Change Merge ChangeBuild Artifacts Run Tests
● No more broken builds
● Ease the reviewing process by filtering with the tests
● Promote parallel work

10.
POSS 2015
Example of OpenStack
10
Gating CI
This CI process has been implemented by the OpenStack project
successfully. Every day:
● 1,000 proposed patchsets
● 7,500 posted comment and votes on the code review system
● 16,000 test environments spawn
● 250 changes merged

11.
POSS 2015
Components used by the OpenStack CI
11
Gating CI
Local changes
submitted with
git-review
Gerrit code
review system
git.openstack.org
mirror
github
mirror
Zuul Gearman
Jenkins
servers
Nodepool
Jenkins
slaves

12.
POSS 2015
Benefits of the OpenStack CI
12
Gating CI
● No broken master
● Parallel testing using speculative merging
● Cross project dependencies at the changeset level
● Optimized costs by provisioning the slaves on-demand

13.
DIY

14.
POSS 2015
Components
14
DIY
All the components are Free Software projects
● Download from source or pre-built packages
● Puppet modules for easy configuration

15.
POSS 2015
Software Factory
15
DIY
Alternatively, you can use the pre-built integration from softwarefactory-project.io
● ready to use VM with the OpenStack CI components: Gerrit, Zuul, Jenkins and
Nodepool
● Extras:
○ prebuilt integration and delivery pipelines in Zuul
○ SSO
○ web menu and dashboard
○ configuration API and CLI
○ bug tracking system (Redmine)
○ collaboration tools: paste and etherpad

16.
POSS 2015

17.
POSS 2015

18.
POSS 2015
Conclusion
18
DIY
No more broken master!
● re-use OpenStack CI components for your own needs
● help us improve Software Factory

19.
Questions?

20.
Contact: frederic.lepied@redhat.com
@flepied

21.
THANK YOU
plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews