Building Your
Development
Pipeline

Olly Pomeroy
Docker
Laura Tacho
CloudBees

What is a pipeline?
A set of processes to make software development more efficient,
secure, and high-quality.
We want to deploy a containerized image. Our pipeline includes:
● Building the image
● Automated testing
● Security scanning
● Promoting the image and deploying to production

Build Test
Secure Deploy!
21
3 4

Why use containers for
your pipeline?

Containers make it easy to create consistent, reproducible
environments because your environment is declared in a Dockerfile.
You know exactly what’s running, where, and can modify and
reproduce environments easily
It also allows for efficiency by sharing some artifacts between dev,
test, and prod.
It’s about what’s INSIDE the container...

Since containers are lightweight, isolated, and fast to boot, they
enable different workflows that are a great fit for your pipelines
● Fanning out to run large tasks across multiple containers
● Parallelizing workflows
...And what goes on OUTSIDE in systems and workflows

Certain things will be made easier, but Docker can’t do the work for you.
It’s still up to you to:
● Follow 12-factor app guidelines like pinning dependencies
● Pay attention to size of images and understand what’s in them
● Perform security and vulnerability scans
But there are still no shortcuts!

Building images in a pipeline

We size our build agents for that 1 job that requires a lot of CPU.
The rest of the time they are pretty idle.
We don’t standardize our tools, so I need everything on every
build agent.
Optimizing Build Agents

We size our build agents for that 1 job that requires a lot of CPU.
The rest of the time they are pretty idle.
We don’t standardize our tools, so I need everything on every
build agent.
On Demand Build Agents!
A Build Agent Image for Everyone!
But is this secure?
Optimizing Build Agents

Containerized Build Agents: Docker in Docker
$ docker run --privileged --name dind -d docker:18.09.0-dind
$ docker run -it --rm --link dind:docker docker:18.09.0 sh
/ # docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
/ # docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

$ docker run -it --rm -v /var/run/docker.sock:/var/run/docker.sock docker:18.09.0 sh
/ # docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
olly/jenkins-slave 1 fae1591c6584 3 hours ago 628MB
nginx latest 62f816a209e6 6 days ago 109MB
openjdk 8-stretch 954739b8bdfb 2 weeks ago 624MB
golang latest 45e48f60e268 4 weeks ago 777MB
/ # docker ps
CONTAINER ID IMAGE COMMAND STATUS NAMES
b2e07edbd47e jenkins/slave:3.27-1 "sh" Up 2 hours optimistic_chandrasekhar
86f77c2b67f0 openjdk:8-stretch "sh" Up 3 hours distracted_mcnulty
Containerized Build Agents: Mounted Socket

$ docker run -it --rm -v .pipedocker_engine:.pipedocker_engine docker:18.09.0 cmd
C:>docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
openjdk 1803 2f91c65915d9 2 weeks ago 5.2GB
dtr.az.olly.dtcntr.net/openjdk 1803 2f91c65915d9 2 weeks ago 5.2GB
docker 18.09.0 629e0258a222 2 weeks ago 5.11GB
microsoft/aspnet 4.7.2-wind..cbdbd42e5a14 7 weeks ago 5.46GB
windows/servercore 1803 1a4a9d0fd8af 7 weeks ago 4.93GB
C:>docker ps
CONTAINER ID IMAGE COMMAND STATUS NAMES
a7911d0ff315 docker:18.09.0 "cmd" Up 9 seconds priceless_franklin
Containerized Build Agents: Mounted Pipe (Windows)

Docker in Docker
docker run dind, --privileged
The container can do almost everything the host can do :(
Mounted socket or pipe
-v /var/run/docker.sock:/var/run/docker.sock
-v .pipedocker_engine:.pipedocker_engine
The container controls your Docker host. Any security applied to the socket
has just been bypassed. :(
Containerized Build Agents: Is this Secure?

Docker in Docker
docker run dind, --privileged
Rootless Docker? (Coming soon! Docker CE / EE 19.03!)
Rootless building daemon? (Buildkit, Img, Kaniko)
Mounted socket or pipe
-v /var/run/docker.sock:/var/run/docker.sock
-v .pipedocker_engine:.pipedocker_engine
Dedicated build host?
Dedicated build cluster?
Containerized Build Agents: Is this Secure?

Building Container Images

Testing with containers

Build once, use everywhere
Treat your Dockerfile as a shared artifact that can before different
types of testing during all phases in your development process
● local testing during development
● standalone unit tests
● browser and integration tests
● promotion to staging or QA environments

Maintain flexibility in each environment
● Dev and test environments can be very different
○ Reuse the Dockerfile (or a shared base image, if that makes sense)
○ Create a specific docker-compose.yml file with your testing
environment
○ Long(er)-running dev environment can coexist alongside ephemeral
test environments

In practice: integration testing patterns
docker-compose.yml docker-compose.test.yml
version: '3'
services:
vote:
build: ../vote/
ports: ["80"]
depends_on:
- redis
- db
networks:
- front-tier
- back-tier
result:
...
worker:
...
redis:
...
db:
...
version: '3'
services:
test:
build: ./tests/
depends_on:
- vote
- result
- worker
vote:
...
result:
...
worker:
...
redis:
...
db:
...
Create new one-off application
environment
Create service to run integration tests

Configurations can be reused with many tools
OSS
Jenkins
Jenkins X
Hosted SaaS
Circle CI
CodeShip
Travis CI
Azure DevOps
Supported on-prem
CloudBees
CloudBees Jenkins Distribution
Circle CI
Bamboo
TeamCity
GitLab
and plenty more!

Parallel Testing with Docker
● Theory: employ task parallelism to split work across parallel
computers (containers)
○ Think of your container as just one process, and split testing
loads across processes
○ Improve performance on-demand by adding more containers
○ Manage environments simply with Docker ecosystem tools

Parallel Testing with Docker
● In practice: most CI tools do this for you i.e. declarative pipelines in
Jenkinsfile, CodeShip steps, GitLab
● Use cases
○ Test against a matrix of versions
○ Cross-compile on Linux and Windows
○ Run integration tests against different browsers
● Caution: parallelism is great for testing, but not deploying.

Example: Windows & Linux
Builds in Jenkins
pipeline {
agent none
stages {
stage("build and deploy on Windows and Linux") {
parallel {
stage("windows") {
agent {
label "windows"
}
stages {
stage("build") {}
stage("deploy") {}
}
}
stage("linux") {
agent {
label "linux"
}
stages {
stage("build") {}
stage("deploy") {}
}
}
}
}
}
}

Example: Selenium Grid
Selenium Hub
Firefox
Ubuntu
Chrome
MacOS
Safari
iOS
Safari
MacOS
Chrome
Windows

Securing
Container
Images

Security question
Q. I’ve downloaded all of these container images, how do I know what's inside?
What happens if there is an out of date package in there? How do I know which
vulnerabilities are exposed?
A. It's fine. They came from the DockerHub and the Dockerfile looks ok…

The Old World
Host Operating System, Kernel…
Devs
Ops
App1 App 2 App3
Java Python .Net

The New World
Host Operating System, Kernel….
Devs
Ops
Who’s giving this TLC?
App1 App 2 App3
Java Python .Net

Maybe we can check the Dockerfile?
# Pull base image
FROM oracle/serverjre:8
# Maintainer
LABEL
MAINTAINER=”bruno.borges@oracle.com”
ENV ORACLE_HOME=/u01/oracle
USER_MEM_ARGS="-Djava.security.egd=file:/
dev/./urandom"
PATH=$PATH:/usr/java/default/bin:/u01/ora
cle/oracle_common/common/bin
RUN mkdir -p /u01 &&
...
Oracle Weblogic 12.1.3 Image
FROM oraclelinux:7-slim
LABEL
MAINTAINER=”bruno.borges@oracle.com”
ENV
JAVA_PKG=server-jre-8u*-linux-x64.tar.
gz
JAVA_HOME=/usr/java/default
...
oracle/serverjre:8
FROM scratch
LABEL
MAINTAINER=”ol-ovm-info_ww@oracle.com”
ADD oraclelinux-7-slim-rootfs.tar.xz /
oraclelinux:7-slim

Maybe we can check the Dockerfile?
# Pull base image
FROM oracle/serverjre:8
# Maintainer
LABEL
MAINTAINER=”bruno.borges@oracle.com”
ENV ORACLE_HOME=/u01/oracle
USER_MEM_ARGS="-Djava.security.egd=file:/
dev/./urandom"
PATH=$PATH:/usr/java/default/bin:/u01/ora
cle/oracle_common/common/bin
RUN mkdir -p /u01 &&
...
FROM scratch
LABEL
MAINTAINER=”ol-ovm-info_ww@oracle.com”
ADD oraclelinux-7-slim-rootfs.tar.xz /
FROM oraclelinux:7-slim
LABEL
MAINTAINER=”bruno.borges@oracle.com”
ENV
JAVA_PKG=server-jre-8u*-linux-x64.tar.
gz
JAVA_HOME=/usr/java/default
...
Oracle Weblogic 12.1.3 Image oracle/serverjre:8 oraclelinux:7-slim

Image Vulnerability Scanning
Docker Trusted Registry

How do I control what runs in my cluster??
Enforcing “Corporate Standards” or Best Practises to all of the container
images within your environment:
○ Everyone builds from “myco/base:1” or “myco/base:2”
○ Container Images shouldn’t run as root by default
You can add stages in to your Pipeline to make this happen!
○ Linting Dockerfiles (fromlatest, hadolint + many more)
○ Policy Engines (Anchore Engine, used within SysDig)

How do I control what runs in my cluster??
Enforcing that the “production” images are the only images that can run on my cluster.
$ kubectl apply -f exampleapp.yaml
Error from server (Forbidden): error when creating "exampleapp.yaml": pods
"nginx" is forbidden: one or more container images do not meet the required
signing policy: [nginx: image did not meet required signing policy]
$ docker run nginx:latest
docker: Error response from daemon: image did not meet required signing
policy.

Automating your Pipeline with a Private Registry
Automated Image Promotion
dev/example qa/example
Developer Pushes
an Image to DTR
Promotion Policy
verifies that the
image has no
vulnerabilities.
Webhooks at every step

Deploying in
Containers

The Software Supply Chain
New Code
Lands in
SCM
Jenkins -
Builds new
Image from
SCM
Image
Uploaded
to
Registry
If Image has
no
vulnerabilities.
Move to
testing.
Jenkins
Pipeline
runs QA on
the Image.
Security
Team, sign
off on
image.
Moves to
Production.
New
Image
lands in
Production
Jenkins Pipeline
Creates the
Deployment
from Templates
App
Successfully
Deployed to
Docker EE
Cluster.
Continuous Integration
Continuous
Deployment

Out-of-the-Box Features
Modern orchestration systems come pre-baked with many
deployment features. The work is already done for you!
This means that your pipeline needs to monitor for status updates,
but not implement the functionality.

Deployment Strategies
Rolling Update
Update containers one-by-one (or in groups),
so that the application has no downtime. It’s
possible for two versions of the software to be
deployed at the same time.

Deployment Strategies
Rolling Update

Deployment Strategies
Rolling Update

Deployment Strategies
Rolling Update

Deployment Strategies
Rolling Update

In practice: Rolling Updates and Auto-Rollbacks
Swarm and Kubernetes handle this for you -- no need to custom build
service:
build: myapp/myservice
image: ${REGISTRY-127.0.0.1:5000}/myservice:${TAG-latest}
deploy:
replicas: 7
update_config:
delay: 5s
failure_action: rollback
max_failure_ratio: .5
monitor: 5s
parallelism: 1
docker stack

In practice: Rolling Updates and Auto-Rollbacks
...
strategy:
type: RollingUpdate #this is the default
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
Swarm and Kubernetes handle this for you -- no need to custom build
kubernetes deployment

The Software Supply Chain

The Software Supply Chain
New Code
Lands in
SCM
Jenkins -
Builds new
Image from
SCM
Image
Uploaded
to
Registry
If Image has
no
vulnerabilities.
Move to
testing.
Jenkins
Pipeline
runs QA on
the Image.
Security
Team, sign
off on
image.
Moves to
Production.
New
Image
lands in
Production
Jenkins Pipeline
Creates the
Deployment
from Templates
App
Successfully
Deployed to
Docker EE
Cluster.
Continuous Integration
Continuous
Deployment

Thank You!

Rate & Share
Rate this session in the DockerCon App
Follow Laura @rhein_wein
Follow Olly @ollypom
Tweet #DockerCon