The document summarizes how Puppet can be used to enable lightweight virtualized containers by configuring applications and their dependencies into immutable container images during the build process. It compares deploying a Jenkins application with LDAP authentication on virtual machines versus containers. It discusses challenges with service resources in containers and provides solutions like overriding service resources or using multi-process images with systemd to build immutable Puppet-configured application images.

1. 2014
Presented by
How Puppet Enables
the Use of Lightweight
Virtualized Containers
Jeff McCune
Software Developer | Puppet Labs
@0xEFF

2. What we’ll cover
Compare and contrasts a Jenkins+LDAP deployed onto
virtual machines and onto lightweight containers.
Presented by
!
Jenkins and LDAP Puppet Forge Modules
Migrating from a shared VM to containers
Common Problems with Service Management
Summary of Lessons Learned

3. Presented by
Starting Point
https://github.com/jeffmccune/puppetconf2014
Traditional VM
Puppet deploys Jenkins with LDAP security
http://sharedvm:18080

4. Presented by
Shared VM
Vagrant VM
Puppet
Jenkins OpenLDAP
Puppet configures Jenkins
and OpenLDAP
Vagrant Puppet
provisioner

5. Presented by
Migrating to Containers
Vagrant VM
Puppet at runtime
Jenkins OpenLDAP
Docker Host
!
!
OpenLDAP
Container
Puppet at
build !
time
!
Jenkins
Container
Puppet at
build time

6. Puppet Forge module re-use
It is faster to deploy Jenkins with LDAP using forge modules:
rtyler/jenkins module
90% code re-use: 831 lines from the Forge, 88 lines added
camptocamp/openldap module
85% code re-use: 584 lines from the Forge, 108 lines added
Presented by

7. Presented by
Site Specific Customizations
Initialize LDAP for dc=jeffmccune,dc=net
Configure LDAP admin account and password
Load initial schema and base OU’s into LDAP
Configure Jenkins to use LDAP security

8. Moving from a VM to a Container
Challenges in Puppet:
Service resources fail and must be overridden
Downstream resources require the service resource
LDAP initial schema load requires Service[slapd]
Presented by

9. Why do service resources fail?
Two different models of containers:
Those with and without a service management framework.
In either case there is no service management framework
running during the image build phase.
Presented by

10. Presented by
Containers
Two Models of Containers:
OS Virtualization Model - Heavy
(Solaris Zones, FreeBSD Jails, IBM WPAR’s)
Microservice Model - Light
(Docker, CoreOS, Kubernetes)

11. False Dichotomy over “Light”
Weight is a range, not a boolean.
Single-process image
Multi-process image
Full-os image
Runtime manageability decreases as processes decrease
Presented by

12. Single-process Microservices
In general:
Executes the application process directly, no init system
startup time comparable to normal process startup
Puppet does not operate at runtime
Difficult to manage with volumes and sidekick processes
Presented by

13. Multi-process Microservices
In general:
Start an init system (SysV init, systemd, upstart)
The service management framework manages services
Runs ~5 or more processes per container
Puppet works at runtime without modification
Presented by

14. Full OS Virtualization Containers
Run everything a traditional OS would run
Dozens of running processes in the container
Long startup time (> 30 seconds)
Easier to manage at runtime
Better for some situations, e.g. mimicking a full running OS
Presented by

15. Container Images
Microservice containers have a distinct build phase
The result of the build step is an image, ideally immutable
Instances execute from a built base image
The Dockerfile is an example of a build script
Presented by

16. Dockerfile
FROM centos:centos7
ADD puppet.tar.gz /
RUN puppet apply -v --modulepath=/puppet/modules
/puppet/manifests/ldap.pp
EXPOSE 389
CMD ["/usr/sbin/init"]
Presented by

17. Dockerfile
FROM centos:centos7
ADD puppet.tar.gz /
RUN puppet apply -v --modulepath=/puppet/modules
/puppet/manifests/ldap.pp
EXPOSE 389
CMD ["/usr/sbin/init"]
Presented by

18. Dockerfile
FROM centos:centos7
ADD puppet.tar.gz /
RUN puppet apply -v --modulepath=/puppet/modules
/puppet/manifests/ldap.pp
EXPOSE 389
CMD ["/usr/sbin/init"]
Presented by

19. Dockerfile
FROM centos:centos7
ADD puppet.tar.gz /
RUN puppet apply -v --modulepath=/puppet/modules
/puppet/manifests/ldap.pp
EXPOSE 389
CMD ["/usr/sbin/init"]
Presented by

20. Presented by
docker build
$ docker build --tag base_image .
Sending build context to Docker daemon 4.718 MB
Step 0 : FROM centos:centos7
---> 3236320e3562
Step 1 : ADD puppet.tar.gz /
---> bc9aa11de092
Step 2 : RUN puppet apply …
---> Running in efdda7633bfc

21. docker build
$ docker build --tag base_image .
Sending build context to Docker daemon 4.718 MB
Step 0 : FROM centos:centos7
---> 3236320e3562
Step 1 : ADD puppet.tar.gz /
---> bc9aa11de092
Step 2 : RUN puppet apply …
---> Running in efdda7633bfc
Presented by

22. Presented by
docker build
$ docker build --tag base_image .
Sending build context to Docker daemon 4.718 MB
Step 0 : FROM centos:centos7
---> 3236320e3562
Step 1 : ADD puppet.tar.gz /
---> bc9aa11de092
Step 2 : RUN puppet apply …
---> Running in efdda7633bfc

23. Presented by
docker build
$ docker build --tag base_image .
Sending build context to Docker daemon 4.718 MB
Step 0 : FROM centos:centos7
---> 3236320e3562
Step 1 : ADD puppet.tar.gz /
---> bc9aa11de092
Step 2 : RUN puppet apply …
---> Running in efdda7633bfc

24. Docker Build with Puppet Try #1
Error: /Service[slapd]: Could not evaluate:
Could not find init script for 'slapd'
Warning: Openldap_database[dc=jeffmccune,dc=net]:
Skipping because of failed dependencies
Warning: Exec[inetorgperson schema]:
Skipping because of failed dependencies
And so on for all dependent resources…
Presented by

25. systemd is not installed
The error, Could not find init script for ‘slapd’ is caused by
a fake systemd in the base centos image
$ docker run -i -t centos rpm -qa | grep systemd
fakesystemd-1-15.el7.centos.noarch
systemd-libs-208-11.el7_0.2.x86_64
Presented by

26. Fix #1 Install the real systemd
Replace fakesystemd with the real deal
microservice => virtualized os model
Start a minimum number of services with systemd
New base image: centos7vps with systemd
Presented by

27. Presented by
Fix #1 Dockerfile
FROM centos:centos7
RUN yum -y swap
-- remove fakesystemd
-- install systemd systemd-libs
# RUN rm unit files in /{lib,etc}/systemd/system
CMD ["/usr/sbin/init"]

28. Presented by
Build new base image
$ docker build -t centos7vps .
This base image has the real systemd installed.

29. New Dockerfile
FROM centos7vps # <= was centos:centos7
ADD puppet.tar.gz /
RUN puppet apply -v --modulepath=/puppet/modules
/puppet/manifests/ldap.pp
EXPOSE 389
CMD ["/usr/sbin/init"]
Presented by

30. Presented by
Working for docker run…
Puppet works inside a running docker container
systemd is running

31. Presented by
Working for docker run…
Start the systemd container
$ CID=$(docker run --privileged -d centos7vps)
Enter the container
$ sudo /usr/local/bin/docker-enter $CID

32. Presented by
Working for docker run…
Run Puppet:
bash-4.2# puppet apply /puppet/manifests/ldap.pp
Compiled catalog for ldap in 1.05 seconds
Service[slapd]/ensure: 'stopped' => 'running'
Finished catalog run in 30.15 seconds

33. Not working for docker build
systemd is present, but not running during the build phase
$ docker build -t ldap:vps .
…
Error: Could not start Service[slapd]:
Execution of '/usr/bin/systemctl start slapd'
returned 1: Failed to get D-Bus connection:
No connection to service manager.
Presented by

34. Moving ahead
We could stop now and simply run puppet every time the
container starts prior to the application starting.
This would increase service start time.
Application images become mutable, managed by Puppet.
Doesn’t fit the microservice model very well.
Presented by

35. Goal: Puppet configured images
Puppet configures immutable pre-configured base
application images
Immutable images provide known good state
Closer to the microservice model
Compatible with the os virtualization model
Presented by

36. Docker build with Puppet Try #2
Override the Service[slapd] to avoid systemd at build time
class ldap_override inherits openldap::server::service {
Service[slapd] {
Presented by
ensure => running,
enable => undef,
start => '/usr/sbin/slapd -u ldap -h "ldapi:/// ldap:///"
}
}

37. Success!
We’re able to build an immutable application image fully
configured by Puppet.
Presented by

38. Presented by
LDAP running in systemd
/usr/bin/docker -d --selinux-enabled
_ /usr/sbin/init
_ /usr/lib/systemd/systemd-journald
_ /bin/dbus-daemon --system
_ /usr/sbin/slapd -u ldap -h ldapi:/// ldap:///

39. Moving further ahead
We could stop here, but there are a number of trade-offs
systemd requires the container to run in privileged mode
More processes: systemd, journald, dbus, application
Slower startup / tear-down time (~5 seconds)
Still faster than a VM (~30+ seconds)
Presented by

40. Presented by
Final Dockerfile
FROM centos7vps
ADD puppet.tar.gz /
RUN puppet apply -v /puppet/manifests/ldap.pp
EXPOSE 389
CMD ["/usr/sbin/slapd", "-u", "ldap",
"-h", "ldapi:/// ldap:///", "-d", "0"]

41. Presented by
Final Dockerfile
FROM centos7vps
ADD puppet.tar.gz /
RUN puppet apply -v /puppet/manifests/ldap.pp
EXPOSE 389
CMD ["/usr/sbin/slapd", "-u", "ldap",
"-h", "ldapi:/// ldap:///", "-d", "0"]

42. docker build ldap:microservice
Build is required for deployment
$ docker build -t ldap:microservice .
Presented by

43. Presented by
Single-process Microservice
Only one process running in the container:
/usr/bin/docker -d --selinux-enabled
_ /usr/sbin/slapd -u ldap -h ldapi:/// ldap:/// -d 0
Start / Stop time: ~1 second

44. Puppetized Microservice
The application image is immutable
Changes to the application require a new image build
Centralized reporting via Puppet
version control via Puppet code
Re-usable modules from the Puppet forge
Presented by

45. Putting it together
Link the jenkins microservice with the ldap microservice
$ docker run -d --name ldap ldap:microservice
$ docker run -d --name jenkins
--link ldap:ldap
--publish 18080:8080
jenkins:microservice
Presented by

46. Presented by
Results
The application (jenkins) is now isolated
With isolated dependencies included (ldap)
The applications are immutable
Multiple copies can easily be deployed
Difficult to change running instances

47. Making a change to LDAP
Need to modify the LDAP indexes as an optimization
1: Update Puppet Configuration
2: Rebuild LDAP image
3: Re-deploy LDAP
4: Re-deploy Jenkins (The link to LDAP is static)
Presented by

48. Advantages and Disadvantages
Immutable known good state
Deployment is highly repeatable and consistent
App and Dependency are tightly coupled
Ambassador Pattern decouples services (with tedium)
Presented by
!

49. Summary
Forge modules are re-usable in Docker containers
Service resources pose a challenge
Override the service resource to build immutable images
Multi-process images with systemd are an option
Presented by

50. Presented by
Thank you!
Code:
github.com/jeffmccune/puppetconf2014
David Lutterkort’s Dockercon talk:
http://links.puppetlabs.com/lutter-docker
!