We discuss things to be taken into account when deciding on a policy for your CI/CD pipelines. This might include Git workflows, testing approaches, and shipping strategies.
1. How to plan and define
your CI/CD pipelines
Francisco (aka Patxi) Gortázar
francisco.gortazar@urjc.es
@fgortazar
Funded by the
European Union
2. @fgortazar
Bio
Project Coordinator at @elastestio EU project
Devops @ Kurento/OpenVidu
TeachingTesting & Distributed Systems @ URJC
@fgortazar
https://es.linkedin.com/in/franciscogortazar
3. @fgortazar
Bio
Project Coordinator at @elastestio EU project
Devops @ Kurento/OpenVidu
TeachingTesting & Distributed Systems @ URJC
@fgortazar
https://es.linkedin.com/in/franciscogortazar
4. @fgortazar
How to plan and design your pipelines
● Some preliminary concepts
● Background
● Planning
● CI/CD environment
● Git workflow
● Pipelines
● Code review
● Merging
● Releasing
● Deploying
5. @fgortazar
Some preliminary concepts
● Software is usually built by teams
● People within the team share the code as they build it
by means of source control management tools
SCM
Tools / Services offering git repositories
6. @fgortazar
Some preliminary concepts
● Each time someone shares its changes through the
repository, we should check that:
● The project is still compiling
● All the tests still pass (they’re green)
● Compiling and running tests on a trusted
environment is Continuous Integration’s task
● When a problem arises, it is notified immediately
to the team
7. Some preliminary concepts
● Integration must happen continuously (at least once
per day)
● Fast feedback is key, so that we know our changes
are working when integrated with the rest of the
system
● Problems detected in our CI environment must be
solved immediately
● Status of the project in terms of builds, test, and
deployments is visible at any moment
13. Background
● Some numbers
● 30 code repositories
● ~400 Jenkins jobs
● +1,000 tests
● +20 different environments to test
● +80 artifacts to be deployed at release time
14. @fgortazar
How to plan and design your pipelines
● Some preliminary concepts
● Background
● Planning
● CI/CD environment
● Git workflow
● Pipelines
● Code review
● Merging
● Releasing
● Deploying
15. Planning
● What’s your Git workflow
● Branches are to be integrated as well?
● Depend-on components are to be considered?
● Infrastructure planning
● Snapshot versions
● Release versions
● Deployments
16. @fgortazar
CI/CD environment
● Feasible infrastructures and their consequences
● Public cloud vs private cloud (on premises)
● Controlling public cloud costs (e.g. spot instances, stopping
instances not in use...)
● Virtual Machines vs Docker
● 20 environments = 20 differentVMs
● Cost increases with each new environment!!
● Effort increases with each new environment to configure
(ops)!!
● Time-to-market increases also!!
18. @fgortazar
CI/CD environment
● Developers are pushing hard towards devops to
include changes in infrastructure
● Changes can hardly be reverted (possible, but…)
● Hard to test locally
● Works in my machine effect
● Wasted resources & insufficient resources at the same
time
● Can we do better?
22. CI/CD environment
● Developers own the infrastructure on top of which
their tests will run
● Updates can be done safely
● CI environment much more stable
● Docker images can be updated quickly and easily
● Infrastructure testing much easier
23. @fgortazar
Git workflow
● When working together in a shared repository,
some rules apply as to how changes from the
different people involved in the team are
integrated, and which is the source of truth when
releasing software versions
● We’ll briefly see some common models of git
workflows that people is using
● When necessary, I’ll point you towards specificities
for some models regading the definition of your CI/
CD pipeline
24. @fgortazar
Git workflows
● Basic / Master development /Trunk development
● Everything happens in the master branch
● Each feature is developed in a local repository
● When features are not small enough some problems arise
● Members of the team might push commits to master for
uncompleted features
● Impossible to release a new version before all features has been
completed and integrated
● Most of the time master is broken, and unplanned releases are a
risk
25. @fgortazar
Git workflows
● Branch per feature
● Each features is developed in its own branch (branch’ed
from master)
● The feature is integrated only when complete
● Master is stable, and can be released at any moment
● The branch might be local or it can be shared thus
enabling CI for the features under development
● If so you need a multi-branch pipeline job in Jenkins jargon
26. @fgortazar
Git workflows
● Branch per feature with merge requests
● Similar to branch per feature, however there’s a
hierarchical permission model
● Some people can integrate directly in master
● Others must request a merge, that will be reviewed by
some members of the time prior to merge
● The CI server might help in integrating changes
27. @fgortazar
Git workflows
● Gitflow
● Probably the most complex one
● Useful in big projects, with many features and several
release versions to maintain
● There are two branches
● Master → production branch. Integration into master only
happens if acceptance tests pass. Ready to be deployed.
● Develop → integration branch. Features, developed in branches,
are integrated here first, then tests are run.
http://nvie.com/posts/a-successful-git-branching-model/
28. @fgortazar
Git workflows
● Fork workflow
● One of the most used in GitHub
● Developers fork the repository into their accounts
● This is called “forking” the repo
● Then they clone their own copy into their machines
● Changes are integrated into the developer’s repo first
● Changes are requested to be integrated into the main repo by
means of “pull-requests” (PRs)
● Someone with the necessary permissions accepts or rejects
the PR, probably with some aid from the CI server
https://github.com/elastest/elastest-platform-manager/pulls?q=is%3Apr+is%3Aclosed
29. @fgortazar
How to plan and design your pipelines
● Some preliminary concepts
● Background
● Planning
● CI/CD environment
● Git workflow
● Pipelines
● Code review
● Merging
● Releasing
● Deploying
30. @fgortazar
Pipelines
● What is a pipeline?
● Two meanings
● Set of jobs that are run in sequence
● In Jenkins a Pipeline job is a job that leverages the
Pipeline syntax based on the Groovy language
● There are two different syntax for Pipeline jobs
● Declarative Pipeline (DSL)
● Scripted Pipeline (Groovy)
31. @fgortazar
Code review jobs
● Needed only when doing code review
● Pull Requests / Merge Requests
● Run per change request before the commit is push to
master
● Fast: unit tests mainly
● Fix the commit before getting too deep into any
other task
● CI votes the change
● +1 can be accepted & integrated→
● -1 cannot be integrated, amendment needed→
34. @fgortazar
Merge jobs
● Run once a something has been integrated into the
master branch
● Usually integration tests
● This might take longer
● Development versions are usually deployed to the
artifact repository if tests pass
● Living testing env updated if tests pass
38. @fgortazar
Merge jobs
●
We need tools to manage logs
●
Put all logs in a single place
●
Test logs and SUT logs
●
Remote service to be used from SUT deployed in
any place
●
Advanced Log analysis capabilities
●
Filtering
●
Searching
●
Comparing
46. @fgortazar
Release jobs
● Building binaries
● Tagging the repository
● Publishing binaries in the artifact repository
● Smoke tests in staging environment
● Performance tests
● These can be integrated into the merge jobs
● Based on the version number the development/release
behavior is triggered
47. @fgortazar
Deployment jobs
● Shipping binaries to production environment
● Migrating the database if necessary
● Deploy based on the chosen strategy
● Blue/green
● Canary
● ...
● Run smoke tests to assess the critical paths
48. @fgortazar
Nightly jobs
● Run during the night or any moment where the CI
infra is not under heavy use
● Some of these tests need to be run in isolation
● End-to-end tests
● Performance tests
● Might take several hours to complete
● Verify the critical paths
● New software version needs to be deployed first
50. @fgortazar
Final thoughts
● There might be dependencies…
● Consider if you’d like to run other projects that depend on this
one at every change /every release
● Increases the usage of the CI environment plan in advance→
● Which information is more important to you?
● Gather that information in your job
● Archive it for later inspection
● Additional tools might be needed
● ElasticSearch
● Kibana
● ElasTest
51. @fgortazar
Final thoughts
● Jobs can get really big…
● Separate them into different jobs
● Add a new job to orchestrate these new jobs
● Jobs can be called from other jobs
● Better visibility
stage('Update ci environments') {
steps {
build job: 'Development/run_ansible',
propagate: false
build job: 'Development/kurento_ci_build',
propagate: false,
parameters: [string(name: 'GERRIT_REFSPEC', value: 'master')]
}
}
53. @fgortazar
Final thoughts
● Don’t repeat yourself
● Shared libraries is a great way of factoring out
common parts in your pipelines into a library that can
be reused across jobs
● Groovy libraries in a code repository
● Need to be added into Jenkins global configuration
● Just import them and enjoy!
54. How to plan and define
your CI/CD pipelines
Francisco (aka Patxi) Gortázar
francisco.gortazar@urjc.es
@fgortazar
Funded by the
European Union
Thanks!