Continuous delivery for containerized applications presents both opportunities and challenges. Container images become the single artifact representing the application build, so metadata must be carefully added to images. Validating non-functional requirements of containerized applications is also important. Developing "mechanical sympathy" by understanding how applications will run in containers can help address issues around resources, security and other operational concerns.

1. Continuous Delivery for Containerized Applications:
The Trials and Tribulations
Daniel Bryant
@danielbryantuk

2. Setting the scene…
• Continuous delivery is a large topic
• Focusing on the process and tooling
• No live coding today
• Mini-book contains more details
• “Building a CD pipeline” by Adrian and Kevin
20/12/2016 @danielbryantuk

3. TL;DR – Containers and CD
• Container image becomes the build pipeline ‘single binary’
• Adding metadata to containers images is vital
• Must validate container constraints (NFRs)
• Cultivate containerised ‘mechanical sympathy’
20/12/2016 @danielbryantuk

4. @danielbryantuk
• Chief Scientist at OpenCredo, CTO at SpectoLabs
• Agile, architecture, CI/CD, DevOps
• Java, Go, JS, microservices, cloud, containers
• Leading change through the application of technology and teams
20/12/2016 @danielbryantuk

5. Continuous Delivery
20/12/2016 @danielbryantuk

6. Continuous Delivery
• Produce valuable and robust software in short cycles
• Optimising for feedback and learning
• Not (necessarily) Continuous Deployment
20/12/2016 @danielbryantuk

7. Creation of a build pipeline is mandatory for continuous delivery
20/12/2016 @danielbryantuk

8. 20/12/2016 @danielbryantuk

9. The Impact of containers on CD
20/12/2016 @danielbryantuk

10. Container technology (and CD)
• OS-level virtualisation
• cgroups, namespaces, rootfs
• Package and execute software
• Container image == ‘single binary’
20/12/2016 @danielbryantuk

11. 20/12/2016 @danielbryantuk

12. 20/12/2016 @danielbryantuk

13. Creating a pipeline for containers
20/12/2016 @danielbryantuk

14. 20/12/2016 @danielbryantuk

15. Make your dev environment like production
• Develop locally or copy/code in container
• Use base images from production
• Must build/test containers locally
• Perform (at least) happy path tests
• All tests should be runnable locally
20/12/2016 @danielbryantuk

16. Lesson learned: Dockerfile content is super important
• OS choice
• Configuration
• Build artifacts
• Exposing ports
• Java
• JDK vs JRE and Oracle vs OpenJDK
• Golang
• Statically compiled binary
• Python
• Virtualenv
20/12/2016 @danielbryantuk

17. Please talk to the sysadmin people:
Their operational knowledge is invaluable
20/12/2016 @danielbryantuk

18. Different prod and test containers?
• Create “test” version of container
• Full OS (e.g. Ubuntu)
• Test tools and data
• Easy to see app/configuration drift
• Use test sidecar containers instead
• ONTEST proposal by Alexi Ledenev
20/12/2016 @danielbryantuk
http://blog.terranillius.com/post/docker_testing/

19. 20/12/2016 @danielbryantuk

20. Building images with Jenkins
• My report covers this
• Build as usual…
• Build Docker Image
• Cloudbees Docker Build and Publish Plugin
• Push image to registry
20/12/2016 @danielbryantuk

21. Storing in an image registry (DockerHub)
20/12/2016 @danielbryantuk

22. Lesson learned: Metadata is valuable
• Application metadata
• Version / GIT SHA
• Build metadata
• Build date
• Image name
• Vendor
• Quality metadata
• QA control
• Security audited etc
20/12/2016 @danielbryantuk

23. Metadata – Beware of “latest” Docker Tag
• Beware of the ‘latest’ Docker tag
• “Latest” simply means
• the last build/tag that ran without
a specific tag/version specified
• Ignore “latest” tag
• Version your tags, every time
• Danielbryantuk/test:2.4.1
20/12/2016 @danielbryantuk

24. Metadata - Adding Labels at build time
• Docker Labels
• Add key/value data to image
20/12/2016 @danielbryantuk

25. Metadata - Adding Labels at build time
• Microscaling Systems’ Makefile
• Labelling automated builds on
DockerHub (h/t Ross Fairbanks)
• Create file /hooks/build
• label-schema.org
• microbadger.com
20/12/2016 @danielbryantuk

26. Metadata - Adding Labels at runtime
20/12/2016 @danielbryantuk
$ docker run -d --label
uk.co.danielbryant.lbname=frontdoor nginx
• Can ’docker commit’, but creates new image
• Not possible to update running container
• Docker Proposal: Update labels #21721

27. 20/12/2016 @danielbryantuk

28. Component testing
20/12/2016 @danielbryantuk

29. Testing: Jenkins Pipeline (as code)
20/12/2016 @danielbryantuk

30. 20/12/2016 @danielbryantuk

31. Testing individual containers
20/12/2016 @danielbryantuk

32. Integration testing
20/12/2016 @danielbryantuk

33. Introducing Docker Compose
20/12/2016 @danielbryantuk

34. Docker Compose & Jenkins Pipeline
20/12/2016 @danielbryantuk

35. Mechanical sympathy: Docker and Java
• Watch for cgroup limits (and cgroup awareness)
• getAvailableProcessors issue (bugs.openjdk.java.net/browse/JDK-8140793)
• Default fork/join thread pool sizes (based from host CPU count)
• Set container memory appropriately
• JVM requirements = Heap size (Xmx) + Metaspace + JVM overhead
• Account for native thread requirements e.g. thread stack size (Xss)
• Entropy
• Host entropy can soon be exhausted by crypto operations
20/12/2016 @danielbryantuk | @spoole167 35

36. Mechanical sympathy: Docker and security
20/12/2016 @danielbryantuk

37. Containers are not a silver bullet
20/12/2016 @danielbryantuk

38. Containers: Expectations versus reality
20/12/2016 @danielbryantuk
“DevOps”

39. Containerise an existing (monolithic) app?
• For
• We know the monolith well
• Allows homogenization of the
pipeline and deployment platform
• Can be a demonstrable win for
tech and the business
• Against
• Can be difficult (100+ line scripts)
• Often not designed for operation
within containers, nor cloud native
• Putting lipstick on a pig?
20/12/2016 @danielbryantuk

40. Whatever you decide…
push it through the pipeline ASAP!
20/12/2016 @danielbryantuk

41. Key lessons learned
• Conduct an architectural review
• Architecture for Developers, by Simon Brown
• Architecture Interview, by Susan Fowler
• Look for data ingress/egress
• File system access
• Support resource constraints/transience
• Optimise for quick startup and shutdown
• Evaluate approach to concurrency
• Store configuration (secrets) remotely
20/12/2016 @danielbryantuk

42. New design patterns
20/12/2016 @danielbryantuk
bit.ly/2efe0TP

43. Microservices…
Containers and microservices are
complementary
Testing and deployment change
20/12/2016 @danielbryantuk
https://specto.io/blog/recipe-for-designing-building-testing-microservices.html

44. 20/12/2016 @danielbryantuk

45. 20/12/2016 @danielbryantuk

46. 20/12/2016 @danielbryantuk

47. Microservice architectural impact on CD
• Application decomposition
• Bounded context
• Change cadence
• Risk
• Performance
• Scalability
• Team location
h/t Matthew Skelton, Adam Tornhill
• Worth knowing about:
• Consumer-based contracts
• Service virtualisation
• Synthetic transactions and
semantic monitoring
20/12/2016 @danielbryantuk

48. Using containers does not obviate the need for
good architectural practices
20/12/2016 @danielbryantuk

49. 20/12/2016 @danielbryantuk
https://speakerdeck.com/caseywest/containercon-north-america-cloud-anti-patterns

50. Summary
20/12/2016 @danielbryantuk

51. In summary
• Continuous delivery is vitally important in modern architectures/ops
• Container images must be the (single) source of truth within pipeline
• Mechanical sympathy is important (assert properties in the pipeline)
• We’re now bundling more responsibility into our artifact (e.g. an OS)
• Not all developers are operationally aware
• The tooling is now becoming stable/mature
• We need to re-apply old CD practices with new technologies/tooling
20/12/2016 @danielbryantuk

52. Bedtime reading
20/12/2016 @danielbryantuk

53. Thanks for listening
• Any questions?
• Feel free to contact me
• @danielbryantuk
• daniel.bryant@opencredo.com
20/12/2016 @danielbryantuk