1.
The Famous Cows of Cambridge
A non-standard use-case for Jenkins
Sarah Woodall, NXP Semiconductors

2.
#jenkinsconf
Footer
What this talk is about
• How (and why) we use Jenkins to test our software
automatically using a farm of custom hardware
• The plugins and features of Jenkins that help
• Improvements we would love to see
• Why we think the Workflow Plugin will be important
2

3.
#jenkinsconf
Setting the scene

4.
#jenkinsconf
Some of the world’s most photographed cows
4
Photo by Alex Brown
(originally posted to Flickr
as Cows and King's)
[CC BY 2.0 (http://
creativecommons.org/
licenses/by/2.0)], via
Wikimedia Commons

5.
#jenkinsconf
The product that our Jenkins builds and tests
A software
development
toolchain based
on Eclipse and
GCC/GDB
• Designed for ease of use
with NXP’s ARM-based
LPC microcontrollers
• Hosted on Windows, Mac,
Linux (separate installers)

6.
#jenkinsconf
LPCXpresso in normal use
6
Host computer
running
LPCXpresso
Debug probe
USB
Development board
Target MCU
Stub

7.
#jenkinsconf
LPCXpresso’s components
Open-source IDE built and tested by the Eclipse Foundation
Public releases three times a year
GNU tools (GCC compiler, GDB debugger, ...) built and tested by ARM
Public releases four times a year
Project wizards; Debug perspective; MCU definitions;
support for advanced features such as Trace, etc ...
Written locally: our responsibility to build and test
Debug stubs: let GDB talk to NXP MCUs on target boards via USB
Many target/probe/host platform combinations => many stubs!
Written locally: our responsibility to build and test
Target software: C Libraries; LPC-Open related components; ...
Written locally: our responsibility to build and test

8.
#jenkinsconf
Testing LPCXpresso’s
debug stubs
8

9.
#jenkinsconf
Manual testing will not get us very far
9

10.
#jenkinsconf
Debug stubs: the scale of the testing problem
This matrix is only a subset
• New targets and debug probes
are constantly being added
• We have to run the same tests
over again for each host platform

11.
#jenkinsconf
It’s not enough to run the tests just once
Why test frequently?
• It’s much easier to find problems straight away
• Developers see what is wrong while their work
is still fresh in their minds
Why test everything, everywhere?
• Many of the stubs share common code:
change one – break the others!
• Stubs work with the host OS, so they behave
differently on each platform
11

12.
#jenkinsconf
The solution: automate testing with Jenkins
• Jenkins builds the three LPCXpresso installers
– (How it does that is a whole talk for another day)
• A set of host machines are set up as Jenkins slaves
– Windows 7, Windows 8.1, Ubuntu, Fedora, Mac, …
• TestInstall (a matrix job)
– Uses the Copy Artifact plugin to get the right installer
– Runs it silently and checks the result
12

13.
#jenkinsconf
Test a wide range of target/probe combinations
• Using a big matrix job, each testing slave can
– use every kind of debug probe to communicate
with target boards
– execute a set of standard images compiled for
each kind of target hardware
13

14.
#jenkinsconf
The snag: how to power-cycle the test boards?
• In normal use, nobody would expect to have multiple
boards and debug probes attached to one computer
– They are not designed to work like that!
• Many of them have to be switched off and on again
as well as being booted before use
• How can we simulate this from Jenkins?
14

15.
#jenkinsconf
The answer: our test farm has COWs
15
COW =
“Collection
of Wires”

16.
#jenkinsconf
The test farm (much simplified)
16
Jenkins master
TestWindows TestMac TestUbuntu ... many similar Jenkins slaves
Ermintrude Daisy Poppy Willow
USB
probe
+
target
... many similar cows
... and very many more
probe
+
target
probe
+
target
probe
+
target
probe
+
target
probe
+
target
probe
+
target
probe
+
target
probe
+
target
USB

17.
#jenkinsconf
Part of the Cambridge herd
17

18.
#jenkinsconf
An expatriate cow
18
Our Jenkins master drives this laptop in California as a (rather slow) slave

19.
#jenkinsconf
How are our cows connected today?
• Cows are physically
plugged in to particular
test machines
• We run a special job to
tell Jenkins whenever a
cow has been moved
• All the test jobs begin by
copying over the artifact
that this job created
19

20.
#jenkinsconf
Summary Display shows test coverage available
The result of the
ConfigureTestFarm
job shows us
where the gaps in
our testing are

21.
#jenkinsconf
Test results overview shown in Jenkins
Results table
created with the
Jenkins Summary
Display Plugin

22.
#jenkinsconf
Tests are defined using Robot Framework
• Open source; easily extended using Python
• Works on all our supported platforms
• Ready-made output display formatted in HTML
• Easy to integrate with Jenkins
22
robotframework.org

23.
#jenkinsconf
Links in our results table lead to Robot’s reports

24.
#jenkinsconf
Drilling down further shows more detail

25.
#jenkinsconf
A cow in action
25

26.
#jenkinsconf
A cow driving a development board
26

27.
#jenkinsconf
More automation
Testing other aspects of LPCXpresso

28.
#jenkinsconf
Importing and compiling 2000 templates
28

29.
#jenkinsconf
Uploading, downloading, …
• Jenkins uploads our build artifacts to a cloud server
for easy access by colleagues in remote locations
• A shared cloud slave enables communication with a
distant group’s Jenkins instance
– Their Jenkins uploads zipped packages as artifacts
– Our Jenkins fetches them, unpacks them and runs
them through our test farm, reporting results by email
29

30.
#jenkinsconf
The plugins and features of
Jenkins that help us most

31.
#jenkinsconf
Our favourite plugins
• Exclusion
• Conditional buildstep
• EnvInject
• Parameterized trigger
• Matrix combinations
• Promoted builds
• Copy artifact
• Node/label parameters
• Summary display
• Description setter
• Build timeout
• Green balls
31

32.
#jenkinsconf
Things we do with build parameters
• Check out from a branch of the source
• Name a separate exclusion resource on each slave
• Allow interactive input to Groovy scripts
• Select matrix combinations dynamically
32

33.
#jenkinsconf
Selecting matrix combinations dynamically
33
This is part of the
configuration for a job
called LaunchTestMatrix

34.
#jenkinsconf
Matrix combinations example
This is generated automatically by a script
COMBINATIONS=(PROBE=="LPC-Link"&&(TARGET=="LPC2138"||
TARGET=="LPC11U14_201"||TARGET=="LPC1114_301"||
TARGET=="LPC1549"))||(PROBE=="RedProbe
+"&&(TARGET=="LPC2368"||TARGET=="LPC1768"))||(PROBE=="LPC-
Link2"&&(TARGET=="LPC4337"||TARGET=="LPC1768"||
TARGET=="LPC4330_SPIFI_1M_64K"||TARGET=="LPC1227_301"||
TARGET=="LPC1830_NGX_SPIFI"||TARGET=="LPC4370_SPIFI"||
TARGET=="LPC1549"||TARGET=="LPC11U68"||
TARGET=="LPC54102J512"||TARGET=="LPC810"))||(PROBE=="RDB-
Link"&&(TARGET=="LPC1768"))||(PROBE=="CMSIS-
DAP"&&(TARGET=="LPC11U68"||TARGET=="LPC824"||
TARGET=="LPC1549"||TARGET=="LPC4370_SPIFI"||
TARGET=="LPC54102J512"||TARGET=="LPC1768"))
34

35.
#jenkinsconf
Exclusion to manage shared resources
The whole point of the cows is to have only one target
board/probe combination switched on at one time.
• What if more than one job is running on the slave?
Our solution: use the Jenkins Exclusion Plugin
• Define a resource called ${SLAVE}_BOARDS
• Claim it before running each test
35

36.
#jenkinsconf
Our Jenkins wish-list

37.
#jenkinsconf
We do love Jenkins, but it would be nice if …
• You could select one configuration in a matrix job
when copying artifacts from a permalink
• There was proper support for source-code branches
• The Cross-platform shell plugin worked better
• Matrix jobs were first-class citizens
37

38.
#jenkinsconf
Automated is good,
but scripted is better!

39.
#jenkinsconf
Our job structure is quite complicated
Visualization created with the
Jenkins Dependency Graph Plugin
39

40.
#jenkinsconf
We think the Workflow plugin will help
• Allows complicated job structures to be expressed
concisely as scripts
• Easier to read, to write and to manage
But
• We shall need good support for matrix jobs
40

41.
#jenkinsconf
What is special about our
use-case for Jenkins

42.
#jenkinsconf
Our world is a matrix world
• Our test farm is a sparse multi-dimensional matrix
• The hardware is reconfigurable, so the Jenkins
model of it has to be flexible, too
• We need build parameters and combination filters to
manage this complexity
• The proper functioning of matrix jobs is essential
42

43.
#jenkinsconf
• www.lpcware.com/lpcxpresso
• sarah.woodall@nxp.com
• sarah@woodall.org.uk
Questions?

44.
London, UK 23-24 June