Marco Brandizi <marco.brandizi@rothamsted.ac.uk>
30/4/2021
Continuos Integration
@Knetminer
Find this presentation on SlidesShare
Background source: https://tinyurl.com/ydq2e6xn

What's Continuous Integration, by the
way?
Image source: https://pt.slideshare.net/thyagk/continuous-integration-system/3

Typical CI build workflow
1. Is build needed (no changes in a scheduled job or "[ci skip]")? => STOP
2. Get and set variables (eg, github and Nexus credentials)
3. Are we releasing? => change version in Maven POMs (eg, 2.0, parameter sent by the user)
4. "mvn deploy", or "mvn install"
1. depending on conditions (eg, if we're in the master branch and not in a pull request)
5. Are we releasing?
1. Tag github with new version and git-commit
2. Change POMs with next snapshot version (eg, 2.0.1-SNAPSHOT, parameter sent by the user)
6. Local git commits (eg, because of POM's new versions and tagging)? => git push

Our (and not only)
Issues
1. Pre-check if build is needed
2. Settings
3. Pre-build Release ops
4. Maven (includes deployment)
5. Post-build release ops
6. Git Push
• Pretty much the same for all
Java/Maven projects
• Copy-pasting is hard to maintain
• Can't we factorise?

Our (and not only)
Issues
• CIs in my past five years:
• Until 2016: Bamboo
• 2017-2021: Jenkins
• 2018-2020: Travis
• 2020-2021: GitHub Actions
• 2022 Yet another one?!
• Every time:
• New system to learn
• New syntax to describe CI jobs to learn
• Migrating previous jobs and test
• Migrating settings (eg, credentials)
• Can't we make it portable?
• (At least a little more?)

My Approach

Just plug-in in your CI
1. Pre-check if build is needed
2. Settings
3. Pre-build Release ops
4. Maven (includes deployment)
5. Post-build release ops
6. Git Push
Your CI job (see archetype-resources/.github/workflows/build.yml)
mkdir -p ci-build
<Download install.sh>
./ci-build/install.sh
# was downloaded together with others (eg, maven-settings.xml)
./ci-build/build.sh

Need an example?
https://github.com/Rothamsted/rdf2pg/blob/master/.github/workflows/build.yml

Customisation
1. Pre-check if build is needed
2. Settings
3. Pre-build Release ops
4. Maven (includes deployment)
5. Post-build release ops
6. Git Push
• Uhm... aren't you throwing your CI
workflow at me?
• I've had the problem myself
• Solution: script hooks

Customisation with script hooks
1. Is build needed (no changes in a scheduled job or "[ci skip]")? => STOP
2. Get and set variables (eg, github and Nexus credentials)
build-begin.sh
3. Are we releasing? => change version in Maven POMs (eg, 2.0, parameter sent by the user)
4. build-before.sh
( "mvn deploy", or "mvn install" ) OR build-body.sh
build-after.sh
5. Are we releasing?
1. Tag github with new version and git-commit
2. Change POMs with next snapshot version (eg, 2.0.1-SNAPSHOT, parameter sent by the user)
6. Local git commits (eg, because of POM's new versions and tagging)? => git push
build-end.sh

Customisation with script
hooks
Your CI job
1. Download
2. Run
1. Pre-check if build is needed
2. Settings
build-begin.sh
3. Pre-build Release ops
4. build-before.sh
Maven (includes deployment) OR build-body.sh
build-after.sh
5. Post-build release ops
6. Git Push
build-end.sh
Just add your hooks as needed
1. to <your-repo>/ci-build/
2. Only the ones you need
3. They see build variables (eg, IS_RELEASE,
NEW_RELEASE_VER)
4. They can expose their own variables (eg, for the next hook)
1. Or change existing ones (be careful)

Need an example?
https://github.com/Rothamsted/knetminer/tree/master/ci-build
• You'll see customisations for:
• Settings custom variables (docker_tag)
• Build on top of our own Docker-based environment, instead of GH Actions environment
• Much more portable
• post-build actions to push new Docker images (including release images)
• Invocation of Jenkins API to trigger internal deployment jobs
(which redeploy from the new Docker images)

Final notes
• build.sh isn't the bible, we can change/improve as we need, eg
• more hooks
• See if it can be adapted to other languages (eg, Python) or a new one is needed
• DO NOT copy it locally just because it doesn't fit your needs, try to reuse stuff!
• Please let me know about proposed/made changes
• Are there alternatives? => https://github.com/rosineygp/mkdkr
• + Entirely based on Docker + Make, more portable
• - Not sure using Docker even for the simplest builds is worth
• - I know Make very little, not sure I want to depend on it
• ~ Requires some CI configuration anyway (eg, to set up credentials and alike, should be solved with a
credential manager)
• - Presumably, would require volume mapping to manage the cache (and cache configuration at CI level)