By Robert Fach, TechniSat Digital GmbH Attendees will learn about a system which solves a code reproducibility problem in the automotive systems area. The code reproducibility problem is defined as being able to rebuild a binary identical software release for a specific amount of time (up to 15-20 years after initial release) and to rebuild a software application with only small changes, keeping the unchanged parts binary identical with the related release. The problem complexity grows due to: external dependencies in a changing environment, a requirement to perform cross-platform builds, scaling issues (hundreds of gigabytes of source code) and more. The Technisat Dresden team achieved their goals by combining several tools in the right way: Jenkins + custom Jenkins plugin, SaltStack, KVM and SVN/Git.

1. Jenkins World Tour 2015
London, UK, June 2015
1

2. #jenkinsconf
A Reproducible Build
Environment with Jenkins
Robert Fach, TechniSat Digital GmbH
2

3. #jenkinsconf
Footer
Agenda
• Some words about me
• Some words about TechniSat
• What is binary code reproducibility?
• Why do we need it?
• Why is it so difficult to achieve?
• Reproducible Build System
• Questions?

4. #jenkinsconf
Footer
Some words about me
• Degree in computer science
• Research background in H/A, Scalability, Big Data
• Today Software Integrator, DevOps and Systems
Engineer at TechniSat
• Contributing to Salt Stack + Salt Cookbook Reviewer
• High technical affinity
– was contributing in 3D printing area
– currently building fpv drones

5. #jenkinsconf
Footer
Some words about TechniSat
• TechniSat group develops and produces consumer and information
technology products
• Founded in 1987, today locations in Germany, Hungary, Poland, USA, China
• In-House hardware & software development and production
• Tier-1 supplier for well known car manufacturers
• Solutions for navigation, tuners, telematics systems

6. #jenkinsconf
Footer
Some words about TechniSat
• Develop digital receivers and digital television receivers
• Develop customer-specific OEM concepts for electronics and automotive
industry
Technisat Digital GmbH Dresden
Research & Development

7. #jenkinsconf
Footer
What is binary reproducibility?
• Same “inputs” produce same binary outputs!
– today, tomorrow, next month, in 15 years
– Difficulty: Identify & reproduce everything which has
influence on the output
Sources
(C,C+
+,Java,
…)
Target
BinaryTransform
Compiler,
Scripts, …

8. #jenkinsconf
Footer
Why do we need it?
• Customer requests a limited change in a software for a specific software
release
• Software bugfixes
• Reduce or avoid side effects in build environment
• Toolchain & build system validation
• Achieve a new level of trust to our customers (changes are proof-able)
• Independent verification of binaries
• Validation of cross-builds vs. native builds
• Finding bugs in software modules
• Good for automotive spice (e.g., for tracking of changes/requirements)

9. #jenkinsconf
Footer
Why do we need it?
Release XY Release XYZ
Rebuild
with
changes

10. #jenkinsconf
Footer
Why is it so difficult to achieve?
Sources
(C,C+
+,Java,
…)
Target
BinaryTransform
Compiler,
Scripts, …

11. #jenkinsconf
Footer
Why is it so difficult to achieve?
Sources
(C,C+
+,Java,
…)
Target
BinaryTransform
Compiler,
Scripts, …
Stable
Sources
SVN,GIT

12. #jenkinsconf
Footer
Why is it so difficult to achieve?
Sources
(C,C+
+,Java,
…)
Target
BinaryTransform
Compiler,
Scripts, …
Stable
Sources
SVN,GIT
Version
Issues

13. #jenkinsconf
Footer
Why is it so difficult to achieve?
Sources
(C,C+
+,Java,
…)
Target
BinaryTransform
Compiler,
Scripts, …
Stable
Sources
SVN,GIT
Version
Issues
Operating System
Filesystem
Issues

14. #jenkinsconf
Footer
Why is it so difficult to achieve?
Sources
(C,C+
+,Java,
…)
Target
BinaryTransform
Compiler,
Scripts, …
Stable
Sources
SVN,GIT
Version
Issues
Operating System
Filesystem
Issues

15. #jenkinsconf
Footer
Why is it so difficult to achieve?
Sources
(C,C+
+,Java,
…)
Target
BinaryTransform
Compiler,
Scripts, …
Stable
Sources
Stable
Compiler
& Scripts
Random
Decisions
SVN,GIT
Version
Issues
Operating System
Filesystem
Issues
Sorting
Issues

16. #jenkinsconf
Footer
Why is it so difficult to achieve?
Sources
(C,C+
+,Java,
…)
Target
BinaryTransform
Compiler,
Scripts, …
Stable
Sources
Stable
Compiler
& Scripts
Random
Decisions
SVN,GIT
Version
Issues
Operating System
Filesystem
Issues
Encoding
Issues
Date & Time
Issues
Random
Values,
UUIDs
Sorting
Issues

17. #jenkinsconf
Footer
Why is it so difficult to achieve?
• Workspace paths compiled into the binary
• Random compiler decisions for optimizations, for namespace mangling etc.
– frandom-seed
• Timestamps, UUIDs in object files and libraries
– deterministic mode
• Timestamps, dates generated by macros (__DATE__, __TIME__,
__TIMESTAMP__)
– Faketime/libfaketime (submillis not covered)
– Overriding these macros with defined value
• File processing order depends on inode ordering
• Locales can have influence on sorting order

18. #jenkinsconf
Footer
Why is it so difficult to achieve?
• Dealing with complexity & scale
– Multiple integration stages
– Modular software
– Load distribution (virtualization & build slaves)

19. #jenkinsconf
Footer
Why is it so difficult to achieve?
• Dealing with complexity & scale
– Multiple integration stages
– Modular software
– Load distribution (virtualization & build slaves)
Release
Management
extra external
dependencies

20. #jenkinsconf
Footer
Why is it so difficult to achieve?
Sources
(C,C+
+,Java,
…)
Target
BinaryTransform
Compiler,
Scripts, …
Sources
&
Binaries
Target
BinaryTransform
Compiler,
Scripts
Sources
&
Binaries
Sources
&
Binaries
Sources
&
Binaries
Stage 0
Stage >0
SVN,GIT
Artefact
Repository

21. #jenkinsconf
Footer
Why is it so difficult to achieve?
Sources
(C,C+
+,Java,
…)
Target
BinaryTransform
Compiler,
Scripts, …
Sources
&
Binaries
Target
BinaryTransform
Compiler,
Scripts
Sources
&
Binaries
Sources
&
Binaries
Sources
&
Binaries
Stage 0
Stage >0
SVN,GIT
Artefact
Repository
Artefact & Dependency
Management

22. #jenkinsconf
Footer
Reproducible Build System
Release
Manager
Release Mgmt
System
Release XY
Software
Container
Module
M1
Library
L1
Library
L2
Library
L3
Library
L4
Module
M2
Module
M2
specify
defines
Software Release
Configuration

23. #jenkinsconf
Footer
Reproducible Build System
Release XY
Jenkins Job
source references
script references
parameters, etc.
Jenkins Job
source references
script references
parameters, etc.
Jenkins Job
source references
script references
parameters, etc.
Jenkins Job
source references
script references
parameters, etc.
Release Mgmt
System
SCM (SVN,GIT)
Artefact
Repository
define
map

24. #jenkinsconf
Footer
Reproducible Build System
Jenkins Job
source references
script references
parameters, etc.
Jenkins Job
source references
script references
parameters, etc.
Jenkins Job
source references
script references
parameters, etc.
Jenkins Job
source references
script references
parameters, etc.
Release XY Infrastructure
Jenkins Master XY
Jenkins Slave XY
Jenkins Slave XY
Jenkins Slave XY
Release Mgmt
System
map

25. #jenkinsconf
Footer
Reproducible Build System
Release XYZ Infrastructure
Salt(manageseverything)
KVM Host
KVM Cluster (libvirt, OpenStack planned)
KVM Host … KVM Host
Release Mgmt
System
SCM
Artefact
Repository
DNS
Q/A Systems
Jenkins Master XY
Jenkins Slave XY
Jenkins Slave XY
Jenkins Slave XY
Release XY Infrastructure
Jenkins Master XY
Jenkins Slave XY
Jenkins Slave XY
Jenkins Slave XY

26. #jenkinsconf
Footer
Reproducible Build System
• Each release is built in a new bootstrapped vm infrastructure completely
described by SALT and Release Mgmt system
• Archive complete infrastructure including os, tools, jenkins, etc.
• Restore or recreate systems from specification or backup) for rebuild (1st, 2nd
and 3rd LOD)
• Infrastructure Bootstrapping
– node spec. + Salt conf. + release spec. defines release infrastructure
– images connect to salt cm to finalize bootstrap according their tasks
– jenkins instance is a “clone” from release predecessor
– Release Jenkins jobs include a “copy” of the job history of predecessor
• Finally map release configuration to Jenkins jobs

27. #jenkinsconf
Footer
Reproducible Build System
• Optimizing VM Performance
– Use tmpfs inside VMs for fast random I/O filesystems
– Try to use nfs read-only cache for saving network
bandwidth
– Put Jenkins workspace on a dedicated lvm volume on
the host instead of on a filesystem based disk
– Kvm Optimizations

28. #jenkinsconf
Any Questions?
28