This presentation covers a basic idea of how we could apply Docker while building a CI / CD environment

1. Introduction The pipeline The environment The images The problems Finish
Using Docker in CI process
Applied Docker
G. Godlewski
March 16, 2016

2. Introduction The pipeline The environment The images The problems Finish
Author
About me
Programming, karate, music and cheese making.
Currently working for SMT Software Services

3. Introduction The pipeline The environment The images The problems Finish
The background
Long time ago, in a galaxy far, far away...

4. Introduction The pipeline The environment The images The problems Finish
The background

5. Introduction The pipeline The environment The images The problems Finish
The background
Scope of automation
Unit tests
Functional tests
Integration tests
Quality checks (nightly build, code coverage, phpmd, phpcs,
jshint, jslint, ...)
Deployments
One should constantly search for spots where time could be saved

6. Introduction The pipeline The environment The images The problems Finish
Definitions
Continuous Integration
Martin Fowler:
Continuous Integration is a software development
practice where members of a team integrate their
work frequently, usually each person integrates at least
daily - leading to multiple integrations per day. Each
integration is verified by an automated build (including
test) to detect integration errors as quickly as possible.

7. Introduction The pipeline The environment The images The problems Finish
Definitions
Continuous Delivery
Martin Fowler:
A common question we hear is “what is the difference
between Continuous Delivery and Continuous
Deployment?” Both terms are similar and were coined
around the same time. I see the difference as a business
decision about frequency of deployment into production.
Continuous Delivery is about keeping your application
in a state where it is always able to deploy into
production. Continuous Deployment is actually
deploying every change into production, every day or
more frequently.

8. Introduction The pipeline The environment The images The problems Finish
Product
The point of view

9. Introduction The pipeline The environment The images The problems Finish
Process
The process
Building - parts put together, diagnostics and verification,
removal of unnecessary parts
Packaging - a ready product is being put into a package
(container)
Delivery - the whole package is shipped to the client in a
clearly defined manner
Each of the stages has a clear I/O definition

10. Introduction The pipeline The environment The images The problems Finish
Process
Building up a complete solution
In case of a product composed from several components (eg.
complex system) we could use the analogy of buying kitchen
furniture.
All packages contain components that fulfill a defined set of
criteria
We know how the components should be connected
We know how the components should be placed within
client’s infrastructure (kitchen)
Don’t dismiss the power of analogy!

11. Introduction The pipeline The environment The images The problems Finish
Warming up!
What will we use Docker for
Performing CI builds within isolated containers (docker in
docker, sic!)
Providing the package for the product in which will it be
further distributed
Running our applications

12. Introduction The pipeline The environment The images The problems Finish
Warming up!

13. Introduction The pipeline The environment The images The problems Finish
Warming up!
Getting ready
1 Ensure the right kernel version (starting from 3.10!)
2 Ensure you have all required kernel modules (aufs,
devicemapper etc - depends on the distribution)
3 Latest docker-engine installed
4 Latest docker-compose installed

14. Introduction The pipeline The environment The images The problems Finish
Additional tools
Ansible
What for?
Describe how the application should be delivered (deployment
to server)
Configuration management
Why?
Great documentation
Huge amount of modules - I didn’t have to build my own yet!

15. Introduction The pipeline The environment The images The problems Finish
Own Docker Hub
Docker Hub
hub:
restart: always
image: registry:2
ports:
- "5000:5000"
environment:
TERM: linux
REGISTRY_HTTP_TLS_CERTIFICATE: :)
REGISTRY_HTTP_TLS_KEY: :)
REGISTRY_AUTH: htpasswd
REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
volumes:
- "/home/docker-distro/registry/data:/var/lib/registry"
- "/etc/ssl/certs:/certs"
- "/etc/ssl/private:/keys"
- "/home/docker-distro/registry/auth:/auth"
Hint
Keep security in mind!

16. Introduction The pipeline The environment The images The problems Finish
The CI Server
TeamCity
Free version provides:
3 build agents
20 build configurations
Broad configuration capabilities (build parameters,
configuration templates, defining dependencies etc)

17. Introduction The pipeline The environment The images The problems Finish
Understanding CI
Build Server and Build Agents

18. Introduction The pipeline The environment The images The problems Finish
Customs
Special forces

19. Introduction The pipeline The environment The images The problems Finish
Customs
Custom Agent
FROM sjoerdmulder/teamcity-agent:latest
MAINTAINER Grzegorz Godlewski <grzegorz@sorcerystudio.com>
RUN apt-get -y install software-properties-common
RUN apt-add-repository ppa:ansible/ansible
RUN apt-get update
RUN apt-get -y install ansible
COPY ./keys/id_rsa /home/teamcity/.ssh/id_rsa
RUN chown -R teamcity:teamcity /home/teamcity/.ssh
ADD docker-entrypoint.sh /docker-entrypoint.sh
RUN chmod a+x docker-entrypoint.sh

20. Introduction The pipeline The environment The images The problems Finish
Customs
Extending the images

21. Introduction The pipeline The environment The images The problems Finish
Implementation
TeamCity Server
server:
image: "sjoerdmulder/teamcity:latest"
ports:
- "8111:8111"
volumes:
- "/home/teamcity/server/data:/var/lib/teamcity"
environment:
TERM: linux

22. Introduction The pipeline The environment The images The problems Finish
Implementation
TeamCity Agent
agent:
image: "my-own-hub/teamcity-agent:latest"
ports:
- "9090:9090"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock" # docker in docker
- "/usr/bin/docker:/usr/bin/docker"
- "/usr/bin/docker:/usr/local/bin/docker"
- "/home/teamcity/agent/work:/opt/buildAgent/work" # work directories
- "/home/teamcity/agent/composer:/opt/composer/cache" # composer cache
- "/home/teamcity/agent/docker:/home/teamcity/.docker" # hub auth keys
environment:
TERM: linux
TEAMCITY_SERVER: "http://teamcity_server:8111" # your server
TEAMCITY_AGENT_NAME: "Alpha" # readable name
AGENT_HOME_ON_HOST: "/home/teamcity/agent" # custom ENV for runtime

23. Introduction The pipeline The environment The images The problems Finish
Implementation
The flow
1 Run build in dev image
2 Package using dist image
3 Distribute using Ansible and docker-compose

24. Introduction The pipeline The environment The images The problems Finish
It’s never that easy
Not so easy...
Cleaning up
CI process performance
Proper kernel modules
Dynamically linked modules
Race conditions

25. Introduction The pipeline The environment The images The problems Finish
It’s never that easy
https://github.com/docker/docker/issues/4036

26. Introduction The pipeline The environment The images The problems Finish
It’s never that easy
https://github.com/docker/docker/issues/4036
If you’re using the devicemapper diver, make sure that Udev Sync
Supported is set to true.

27. Introduction The pipeline The environment The images The problems Finish
Questions?

28. Introduction The pipeline The environment The images The problems Finish
Thank you!
http://linkedin.com/in/ggodlewski
grzegorz@sorcerystudio.com
@GGodlewski