Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
The pain and gains running Docker in live @Pipedrive
1. The pain and gains running
Docker in live @Pipedrive
Renno Reinurm
17.01.17
2. ● Pipedrive helps small businesses control the complex
selling process
● Founded in 2010
● 30,000 paying customers worldwide
● 200+ employees
● Offices in Tallinn and Tartu
New York, NY
4. Why to use Docker?
● Growth pains with Chef
● New language + new tools = entry barrier
● You write recipes seldom enough and forget how it’s done
● But it runs fine in test!
5. Early docker platform started with
evaluating running docker inside
Vagrant box.
Instead we started to use custom built
docker-machine.
Lately moved to Docker4Mac
6. First use case for containers
Provision on demand test
environments per branch.
Was implemented only for test
coverage-suite execution
environment.
Lot of custom hacks to make it
work.
7. Docker infrastructure v1
The first Docker builds using Codeship Docker CI beta
The first usage of Tutum (Docker Cloud) as orchestration service
8. Yeah we were using Docker, but
CI processes with Codeship was slow, Docker build itself took ~15minutes
Deployment in Docker Tutum cluster took another ~10minutes
Sometimes it was so slow we wondered if it still works
Stability issues - we experienced “data loss” and “service downtime”
9. The Birth of Docker Infrastructure v2.0
Requirements:
Improve the speed of CI processes
Improve the reliability of Docker Infrastructure
14. Deployment process optimizations
NB! https://docs.docker.com/engine/userguide/storagedriver/selectadriver/
Replacement of Devicemapper to AUFS reduced deployment process time 10x.
There are still improvements possible:
● Handle Linux signals
● Parallel rolling updates
https://teespring.com/sigkill
16. Beware the service discovery corruption
● Always enable health checks
● Use unique health checks or validate output
SERVICE_CHECK_HTTP=/health
vs
SERVICE_CHECK_HTTP=/statistics-health
17. Pain 3 - Every day maintenance of Jenkins jobs
21. Issues
● Linux kernel 3.13
● Fluentd logging agent
● Graylog logging driver
● Kernel sysctl parameters
● Swap usage
● PEBKAC
○ "net.ipv4.ip_forward" => 0
● WARNING: No memory limit support
● WARNING: No swap limit support
● WARNING: No kernel memory limit support
● WARNING: No oom kill disable support
● WARNING: No cpu cfs quota support
● WARNING: No cpu cfs period support
22. Service risk mitigation
● Number of nodes in cluster
● Spreading policies
● Multiple instances
● Memory limitations
● Healing policies
○ Autorestart
○ Reschedule
23. Gains
Evolution of applications
generic enough to run in multiple regions, environments
Delivery time from idea to live
From 2 weeks to 1 day
Servers vs Services
those be managed asynchronously
24. Statistics
~ 70 inhouse built Dockerized services
~ 90 Docker images
~ 500 containers running
3200 container deploys since October
25.
26. Remember - Every Day
1 new container borns to stay @Pipedrive
30 container deployments
27. Recommendations for going
Live with Docker
● You still need to take care of OS
● Read Github issues
● Read from the source
● Keep it up to date
● (Performance) Test it