This document summarizes the evolution of Docker infrastructure at Pipedrive to meet performance requirements. Key optimizations included reducing Docker build/deploy times to under 5 minutes by optimizing Dockerfiles and switching from Devicemapper to AUFS storage. Health checks and unique check paths were implemented to ensure only healthy services were accessed. The infrastructure was tuned through Linux kernel upgrades, memory limits, load spreading, and security measures to support 10,000 connections and constant high load.

1. Docker Tallinn @Pipedrive
Renno Reinurm
2016

2. How application
performance
requirements impacted
the (r)evolution of the
Docker infrastructure at
Pipedrive.

3. Visibility
Management
Interface
http://www.slideshare.net/rreinurm/living-with-mi
croservices-at-pipedrive

4. Requirement 1
You shall not build/test/deploy
Docker container
over 5 minutes
Based on: xkcd.com

5. Optimized Dockerfiles
First iteration, 14 layers:
FROM node
ENV SERVICE_NAME=company-statistics
ENV SERVICE_DESC="Company statistics"
ENV SERVICE_TAGS=statistics
ENV NODE_ENV=live
ENV SERVICE_CHECK_HTTP=/health
ENV SERVICE_CHECK_INTERVAL=10s
ENV SERVICE_CHECK_TIMEOUT=5s
ENV DOCKER=1
EXPOSE 8000
WORKDIR /src
USER nobody
COPY . /src/
CMD ["node", "."]
Optimized, 8 layers and ordered:
FROM node:6-slim
ENV SERVICE_NAME=company-statistics
SERVICE_DESC="Company statistics"
SERVICE_TAGS=statistics
NODE_ENV=live
SERVICE_CHECK_HTTP=/v1/companyStatistics/health
SERVICE_CHECK_INTERVAL=10s
SERVICE_CHECK_TIMEOUT=5s
DOCKER=1
EXPOSE 8000
WORKDIR /src
USER nobody
CMD ["node", "."]
COPY libraries/ /src/
COPY src/ /src/

6. https://youtu.be/X_q2l8hotAc?t=365

7. Deployment process optimizations
NB! https://docs.docker.com/engine/userguide/storagedriver/selectadriver/
Replacement of Devicemapper to AUFS reduced deployment process time 10x.
There are still improvements possible:
● Handle Linux signals
● Parallel rolling updates

8. Requirement 2
Consumers shall connect
only to healthy services

9. Beware the service discovery corruption
● Always enable health checks
● Use unique health checks
SERVICE_CHECK_HTTP=/health
vs
SERVICE_CHECK_HTTP=/v1/companyStatistics/health

10. Requirement 3
Container shall handle 10 000
connections and constant
high load.

11. https://youtu.be/PivpCKEiQOQ

13. Issues
● Linux kernel 3.13
● Fluentd logging agent
● Graylog logging driver
● Kernel sysctl parameters
● Swap usage
● PEBKAC
○ "net.ipv4.ip_forward" => 0
● WARNING: No memory limit support
● WARNING: No swap limit support
● WARNING: No kernel memory limit support
● WARNING: No oom kill disable support
● WARNING: No cpu cfs quota support
● WARNING: No cpu cfs period support

14. Docker hosts tuning
● Linux kernel 4.x
● Removed Swap
● Dropped fluentd driver
● Docker Engine upgrade/patches
● Kernel performance tuning
http://www.devconf.cz/files/slides2015/friday/Performance%20Tuning%20of%2
0Docker%20and%20RHEL%20Atomic.pdf

15. Service risk mitigation
● Number of nodes in cluster
○ If in doubt increase the number
● Spreading policies
● Multiple instances
● Memory limitations
● Healing policies
○ Autorestart
○ Reschedule

16. Security mitigation
● TLS authentication
● Minimal & official Docker images
● Signed Docker images
● User Namespace Remapping
● PID limits
● Capabilities
● Seccomp profiles
● AppArmor/SELinux profiles
https://github.com/riyazdf/dockercon-workshop

17. Recommendations for going
Live with Docker
● You still need to take care of OS
● Read Github issues
● Read from the source
● Keep it up to date
● (Performance) Test it

18. Thank you!
Give me your feedback @rreinurm