Brandon Mitchell, Boxboat + Docker Captain - How do you make your builds more performant? This talk looks at options to configure caching and mirroring of images that you need to save on bandwidth costs and to keep running even if something goes down upstream.

1. Caching
Caching Demos
Caching Limitations
Caching Other Registries
Caching Gitlab Demo
Mirroring
Manual Mirroring
Summary
Agenda
@sudo_bmitch1 / 35

2.
How to Use Mirroring and
Caching to Optimize Your
Image Registry
Brandon Mitchell
Twitter: @sudo_bmitch
GitHub: sudo-bmitch
2 / 35

3. Ephemeral Build Server?
@sudo_bmitch3 / 35

4. Cluster Pulling Remote Images?
@sudo_bmitch4 / 35

5. Worry About Upstream Image Changes?
@sudo_bmitch5 / 35

6. Build and Deploy Infrastructure Tolerant of
Upstream Outages?
@sudo_bmitch6 / 35

7. Production Resilience
@sudo_bmitch7 / 35

8. Build Infrastructure
@sudo_bmitch8 / 35

9. Build Outage
@sudo_bmitch9 / 35

10. Build Resilience
@sudo_bmitch10 / 35

11. Faster Builds and Less Bandwidth
@sudo_bmitch11 / 35

12. Caching
@sudo_bmitch12 / 35

13. Caching is the Easy Button
https://commons.wikimedia.org/wiki/File:Easy_button.JPG
@sudo_bmitch13 / 35

14. Cache Architecture
@sudo_bmitch14 / 35

15. Cache Implementation
Either the dockerd CLI:
dockerd --registry-mirror <cache-url>
Or /etc/docker/daemon.json
{ "registry-mirrors": [ "<cache-url>" ] }
Plus a registry:
docker run -e REGISTRY_PROXY_REMOTEURL=<upstream-url> registry:2
@sudo_bmitch15 / 35

16. @sudo_bmitch
00:00
16 / 35

17. @sudo_bmitch
00:00
17 / 35

18. @sudo_bmitch
00:00
18 / 35

19. @sudo_bmitch
00:00
19 / 35

20. @sudo_bmitch
00:00
20 / 35

21. So What's the Catch?
@sudo_bmitch21 / 35

22. Cache Limitations
The "registry-mirror" setting only applies to Docker Hub
Only caches pulls not pushes
Pulls still check the image manifest on Hub
Credentials are in the cache server
Docker implementation only supports one authentication method
@sudo_bmitch22 / 35

23. Options to Cache Other Registries
Con gure a squid HTTP caching proxy
Pull directly from the cache
Use DNS and TLS certs to send pulls to the proxy
@sudo_bmitch23 / 35

24. @sudo_bmitch
00:00
24 / 35

25. I Want More
@sudo_bmitch25 / 35

26. Mirroring
@sudo_bmitch26 / 35

27. Mirror Architecture
@sudo_bmitch27 / 35

28. Running a Registry
Docker image
docker container run -p 5000:5000 registry:2
Harbor
Many Artifact Repositories
@sudo_bmitch28 / 35

29. Manually Mirroring
docker image pull ${image}
docker image tag ${image} local-mirror:5000/${image}
docker image push local-mirror:5000/${image}
@sudo_bmitch29 / 35

30. Manual Mirror Script
docker image pull "$localimg"
docker image pull "$remoteimg"
remoteid=$(docker image inspect "$remoteimg" --format '{.Id}')
localid=$(docker image inspect "$localimg" --format '{.Id}')
if [ "$remoteid" != "$localid" ]; then
docker image tag "$localimg" "$localimg.$datestamp"
docker image tag "$remoteimg" "$localimg"
docker image push "$localimg.$datestamp"
docker image push "$localimg"
fi
@sudo_bmitch30 / 35

31. Why All the Complication?
@sudo_bmitch31 / 35

32. Advantages of Manually Mirroring
Over Automatically Syncing Repos:
Changes to images happen on your schedule
Backout option exists with breaking changes
Over Pull Through Cache
Those reasons plus...
Pushing locally built images to the registry
Upstream outage doesn't stop local builds/deploys
@sudo_bmitch32 / 35

33. Risks of Manually Mirroring
Images go stale if you do not automate the script
Adding new images is an added process
Recovering from a mirror outage requires populating images
FROM line in images needs to point to mirror
ARG REGISTRY=docker.io
FROM ${REGISTRY}/alpine:3.9
...
docker build --build-arg REGISTRY=local-mirror:5000 .
@sudo_bmitch33 / 35

34. Summary
Both
Saves bandwidth
Faster builds
Pull Through Cache
Easy to create
Little maintenance
Managed Mirror
Control changes
Tolerate upstream outages
@sudo_bmitch34 / 35

35. Brandon Mitchell
Twitter: @sudo_bmitch
GitHub: sudo-bmitch
Thank You
github.com/sudo-bmitch/presentations
35 / 35