This document discusses continuous integration (CI) for open source software on OpenPOWER systems. It provides background on CI, OpenPOWER systems, and the Cloud Foundry platform. It then describes using the Concourse CI tool to continuously build a Concourse project from a GitHub repository. Key steps involve deploying OpenStack, setting up a Docker registry, installing BOSH and Concourse, defining a Concourse pipeline, and updating the pipeline to demonstrate the CI process in action. The document emphasizes the importance of CI for open source projects and how it benefits development on OpenPOWER systems.
2. #ibmedge
About Indrajit (a.k.a. I.P)
Expertise:
• Accelerated Cloud Data Services, Machine
Learning and Deep Learning
• Apache Spark, TensorFlow… with GPUs
• Distributed Computing (scale out and up)
• Cloud Foundry, Spectrum Conductor, Mesos,
Kubernetes, Docker, OpenStack, WebSphere
• Cloud computing on High Performance Systems
• OpenPOWER, IBM POWER
1
Indrajit Poddar
Senior Technical Staff Member,
Master Inventor, IBM Systems
ipoddar@us.ibm.com
Twitter: @ipoddar
3. #ibmedge
About Yulia
2
Yulia Gaponenko
Software Engineer, IBM Systems
yulia.gaponenko1@de.ibm.com
github: barthy1
Expertise:
• Cloud Foundry committer (BOSH project)
• Distributed Computing (scale out and up)
• Cloud Foundry, Docker, OpenStack, WebSphere
• Open Source
• Linux, Eclipse
• Cloud computing on High Performance Systems
• OpenPOWER, IBM POWER
4. #ibmedge
Please Note:
• IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice
and at IBM’s sole discretion.
• Information regarding potential future products is intended to outline our general product direction and it
should not be relied on in making a purchasing decision.
• The information mentioned regarding potential future products is not a commitment, promise, or legal
obligation to deliver any material, code or functionality. Information about potential future products may not be
incorporated into any contract.
• The development, release, and timing of any future features or functionality described for our products
remains at our sole discretion.
• Performance is based on measurements and projections using standard IBM benchmarks in a controlled
environment. The actual throughput or performance that any user will experience will vary depending upon
many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the
I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be
given that an individual user will achieve results similar to those stated here.
3
5. #ibmedge
Agenda
• Continuous Integration (CI) : What is it?
• OpenPOWER Systems: What are these?
• CI for Open Source Software on OpenPOWER: Why is it important?
• Cloud Foundry: What is it?
• Concourse: What is it?
• Scenario: Continuously build a Concourse project from GitHub
4
6. #ibmedge
Change is the only constant…
5
Collaborative Open Source Development has to embrace change
e.g. Linux has 2,000 new developers and
gets 10,000 patches for each version
http://arstechnica.com/information-technology/2015/02/linux-has-2000-new-
developers-and-gets-10000-patches-for-each-version/
7. #ibmedge
What is the elephant like?
6
Ubuntu
CentOS
Fedora
Intel - X86_64
Debian
POWER – ppc64le
ARM – arm64
https://en.wikipedia.org/wiki/Blind_men_and_an_elephant
Open Source Development has to embrace variations
Microprocessor Architectures
Linux OS Distros
8. #ibmedge
Continuous Integration for Open Source Development
• Merge developer copies into shared
mainline several times a day
https://en.wikipedia.org/wiki/Continuous_integration
• Best Practices
• Continuous Build
• Continuous Test
• Continuous Deployment
• Tools:
• Jenkins
• Travis
• Concourse
• Many more…
7
9. #ibmedge
IBM POWER Systems Cloud Performance on Open
Source
• Runs popular Linux Distros
• Runs Docker containers at a high density
per system
https://g01acxwass069.ahe.pok.ibm.com/cms/developerworks/linux/perfcol/cloud.html
8
10. #ibmedge
OpenPOWER: Open Hardware for High Performance
9
Systems designed for
big data analytics
and superior cloud economics
Upto:
12 cores per cpu
96 hardware threads per cpu
1 TB RAM
7.6Tb/s combined I/O Bandwidth
GPUs and FPGAs coming…
OpenPOWER
Traditional
Intel x86
http://www.softlayer.com/bare-metal-search?processorModel[]=9
11. #ibmedge
OpenPOWER Worldwide Development Community
10
continents
20+ technologies revealed
innovations under way100+
members200+
24 6countries
OpenPOWER is an open development community,
using the POWER Architecture
12. #ibmedge
IBM OpenPOWER Systems with Open Source Software
11
Open Source Software works great on POWER
GoLang
And many more …
13. #ibmedge
Continuous Integration for Platform-as-a-Service (PaaS)
• Develop, run, manage applications
without worrying about the complexity of
the infrastructure
• Often runs on top of a Infrastructure-as-a-
Service e.g. OpenStack
• Examples:
• Apache Mesos, Kubernetes with Docker
e.g. docker run –t ppc64le/ubuntu
• Cloud Foundry
e.g. cf push helloworld.js
• CI and PaaS naturally go hand in hand
12
14. #ibmedge
Continuous Integration for OpenPOWER
How to ensure that latest versions of open source
software packages always run on OpenPOWER?
• Source code changes from multiple developers do not cause
breakages
• Particularly when development is done on x86
• Changes are automatically tested on POWER architecture
• Binary packages for POWER are always available at the
most recent level
• in repository as binary blob
• as Docker image
• Automatic versioning
13
Docker
image
Source
code
Github
Building
next version
Testing on
Power
Repository
16. #ibmedge
15
Cloud Foundry. What is it?
• Open source cloud computing Platform-as-a-Service (PaaS)
• Faster and easier to build, test, deploy, and scale applications
• Choice of clouds, developer frameworks, and application services.
Pivotal Cloud Foundry, HP Helion, Predix
GoLang
Commercial PaaS products based on open source Cloud Foundry:
17. #ibmedge
Scenario: Continuously Build a Concourse project from GitHub
• https://github.com/barthy1/git-resource
• tracks commits in a branch of a Git repository
• Get:
• Github project (git-resource)
• Docker image with git (git-image)
• Put:
• New Docker image with git-resource to Docker registry (git-resource-image)
• Action:
• Build and test (git-resource)
16
18. #ibmedge
Scenario: CI Tools
• Docker Registry: to distribute container images
• Open Stack: to create connected virtual machines on multiple servers
• BOSH: to deploy Cloud Foundry releases on Open Stack
• Github: to collaboratively develop Open Source Software
• Concourse: to continuously integrate open source development and test
– Concourse cli called fly
• Concourse CI descriptor, called a pipeline:
to design and execute a workflow for continuous integration
17
19. #ibmedge
Scenario: CI Tools Interaction
18
Bare metals
VM VM
Concourse
Server
Docker Containers
VM VM
BOSH
Image Registry
Object Store
Concourse
fly client or
browser client
DJ1. Sets up CI infrastructure
DJ2. Adds CI pipeline
DJ3. Commit code
CS1. Monitor
CS2. Build
CS3. Test
4. Output
Developer Jane
20. #ibmedge
1. Deploy your own or use a managed OpenStack
• Oregon State University Open Source Lab OpenStack
on OpenPOWER systems
• SoftLayer bare metals with Ubuntu OpenStack on OpenPOWER
19
21. #ibmedge
2. Setup a Docker Private Registry
or use DockerHub
Docker Registry: Docker Image storage
DockerHub: Public Docker registry
• Docker private registry
• Install Docker
• Clone https://github.com/barthy1/distribution
• Build Docker image
• Configure TLS and basic auth
• Run Docker private registry
20
22. #ibmedge
3. Install BOSH
BOSH: Tool to simplify deployment of other projects (Concourse, CF…)
• Deploy project using release packages
• Update deployment with new versions of the release packages
• Monitor state of the deployment
• Fix problems with VMs or connectivity
• Scale out with additional VMs
• Bosh installation:
• Binaries available from:
ftp://ftp.unicamp.br/pub/ppc64el/ubuntu/14_04/cloud-foundry/
– bosh-init - BOSH installer
– BOSH release
– bosh cli – interact with BOSH server
• Sample deployment manifest:
https://www.cloudfoundry.org/bosh-on-openpower-systems/
– update only OpenStack connection details
and links to binaries
21
24. #ibmedge
5. Concourse pipelines
Pipeline: Set of steps to run the tests and build the result
• Various input and output data types
– Git - pull and push to git repositories
– Time - start on schedule or timestamp output
– Docker-image - fetch, build and push Docker images
– Semver – set or bump version numbers
– Amazon S3 - fetch from and upload to s3 buckets
– ….
– Custom resources
• Color of the box shows type and state
– Black – resources
– Green – successfully finished most recent build
– Red – test/build failed
• All the tests and the logs are available
• Group by tests for every commit
• History of the Github project updates
23
25. #ibmedge
5. Examine an existing pipeline
• Pipeline sample:
• Input resources:
– Github project
– Docker image
• Test/build stage
– Clone/update Github project
– Pull/refresh Docker image
– Run the test
– Build new Docker image
– Push Docker image to Docker registry
• Output resources:
– New Docker image
• Logs are available for public/private access
• New test can be started via Web
• Access to the container/s with tests
• Information about current commit
24
27. #ibmedge
6. Update the existing pipeline
• Existing pipeline to show live how it works
• Update the pipeline descriptor with new parameters
• Upload new pipeline to the server with Concourse cli
• Run new pipeline
• Show execution steps and logs
• Login to the running container with tests
• Run latest Docker image for POWER
26
29. #ibmedge
Resources
• Pointers to blogs on how to deploy BOSH, Cloud Foundry, Concourse on
OpenPOWER
http://blog.altoros.com/cloud-foundry-on-ibm-power.html
https://www.cloudfoundry.org/bosh-on-openpower-systems/
https://www.cloudfoundry.org/cf-release-for-openpower/
https://www.cloudfoundry.org/concourse-release-for-openpower
• How to setup Docker private registry:
https://gist.github.com/barthy1/246cd6892d5db2cec9b6f866de70c089
• Latest Docker binaries for POWER:
https://master.dockerproject.org/
• Latest Cloud Foundry binaries for POWER:
ftp://ftp.unicamp.br/pub/ppc64el/ubuntu/14_04/cloud-foundry/
• Oregon State University Open Source Lab with OpenStack:
http://osuosl.org/services/powerdev
28
31. #ibmedge
Notices and Disclaimers Con’t.
30
Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not
tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products.
Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the
ability of any such third-party products to interoperate with IBM’s products. IBM EXPRESSLY DISCLAIMS ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING BUT
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
The provision of the information contained h erein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual
property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®,
FileNet®, Global Business Services ®, Global Technology Services ®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG,
Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®,
PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®,
StoredIQ, Tealeaf®, Tivoli®, Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business
Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM
trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.