O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

The future of opinionated cloud builds

175 visualizações

Publicada em

SpringOne Platform 2018
Cloud Foundry Buildpacks and the Future of Opinionated Cloud Builds
Speakers: Emily Casey Staff Software Engineer, Pivotal Stephen Levine Staff Software Engineer, Pivotal

Publicada em: Software
  • Seja o primeiro a comentar

  • Seja a primeira pessoa a gostar disto

The future of opinionated cloud builds

  1. 1. Buildpacks and the Future of Opinionated Cloud Builds Emily Casey Software Engineering Manager Stephen Levine Staff Software Engineer & Product Manager
  2. 2. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Roadmap Eventually… cool new buildpack API! But first… a story
  3. 3. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The world today App > cf push
  4. 4. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Droplet The Staging Container Buildpack App Node.js npm packages
  5. 5. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ App Node.js npm packages App Node.js npm packages OS Packages App Node.js npm packages OS Packages App Node.js npm packages OS Packages Run Containers
  6. 6. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ App #1 Node.js npm packages OS Packages (stack) App #2 Node.js npm packages OS Packages (stemcell) Stack Updates: Roll Diego Cells
  7. 7. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ App #1 Node.js npm packages Outdated OS Packages (stack) App #2 Node.js npm packages App #3 Node.js npm packages Outdated OS Packages (stack) App #4 Node.js npm packages Outdated OS Packages (stemcell) Outdated OS Packages (stemcell)
  8. 8. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ App #1 Node.js npm packages Outdated OS Packages (stack) App #2 Node.js npm packages App #3 Node.js npm packages Outdated OS Packages (stack) App #4 Node.js npm packages Outdated OS Packages (stemcell) Outdated OS Packages (stemcell) App #1 Node.js npm packages Updated OS Packages (stack) App #2 Node.js npm packages Updated OS Packages (stemcell)
  9. 9. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ App #3 Node.js npm packages Outdated OS Packages (stack) App #4 Node.js npm packages Outdated OS Packages (stemcell) App #1 Node.js npm packages Updated OS Packages (stack) App #2 Node.js npm packages Updated OS Packages (stemcell)
  10. 10. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ App #3 Node.js npm packages Outdated OS Packages (stack) App #4 Node.js npm packages Outdated OS Packages (stemcell) App #1 Node.js npm packages Updated OS Packages (stack) App #2 Node.js npm packages Updated OS Packages (stemcell) App #3 Node.js npm packages Updated OS Packages (stack) App #4 Node.js npm packages Updated OS Packages (stemcell)
  11. 11. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ App #1 Node.js npm packages Updated OS Packages (stack) App #2 Node.js npm packages Updated OS Packages (stemcell) App #3 Node.js npm packages Updated OS Packages (stack) App #4 Node.js npm packages Updated OS Packages (stemcell)
  12. 12. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why not images? App Node.js npm packages OS Packages
  13. 13. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Docker v1 image format Layer JSON Layer JSON Layer JSON Layer JSON Changeset Changeset Changeset Changeset ● Layer oriented ● Each layer references it’s parent ● Layers contain runtime configuration ● Model is similar to a git tree
  14. 14. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ App 2 Droplet OS PackagesOS Packages App 1 Droplet If we used v1 images...
  15. 15. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ OS Packages App 1 Droplet Layer App 2 Droplet Lyaer OS Packages And there is a vulnerability in the base layer
  16. 16. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Updated OS Packages App 1 Droplet App 2 Droplet Updated OS Packages Updating a base layer requires re-converting the droplet into a layer & re-uploading it to a registry
  17. 17. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ If we used a Docker v1 registry to store our images... OS Packages App 1 Droplet OS Packages App 3 Droplet OS Packages App 2 Droplet OS Packages App 4 Droplet OS Packages App 5 Droplet OS Packages App 6 Droplet ● Changes to the base layer would require us to regenerate and re-upload every droplet layer ● For n different app images we need to re-upload n copies of their base layers ● Rolling upgrades would be slow and network intensive
  18. 18. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Docker v1 layers are identified by ID Layer JSON Layer JSON Layer JSON Layer JSON Changeset Changeset Changeset Changeset 8f52818719ad48a0af... 73bc3b5b6d6cdda83... 6046484f00f289fcd95... 8f52818719ad48a0af... ● No guarantee that content has not changed ● Hard to audit
  19. 19. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ A better image spec... In 2016, Docker Registry 2.3 introduced support for the Docker Image Manifest Version 2, Schema 2, based off the OCI image specification
  20. 20. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ OCI Images Image JSON Layer 1 Layer 2 Layer 3 Layer 4 ● Image-oriented ● Layers only consist of a filesystem changeset, no metadata ● Layers do not reference their parent ● Image’s config JSON records the order of the layers and image metadata
  21. 21. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ OCI layers are identified by a digest of their contents Changeset Changeset Changeset Changeset sha256:73046094... sha256:f60d2e9a681... sha256:3ee6c77dbff... sha256:eb115160df81... Content addressable layers provide confidence that layer contents have no changes.
  22. 22. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cross Repository Blob Mounting POST /v2/<name>/blobs/uploads/?mount=<digest>&from=<repository name> Content-Length: 0 Layer 4 Layer 1 Layer 2 Layer 3 Layer 1 Image 1 Image 2 Introduced in Docker Registry v2.3
  23. 23. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ we reimagined buildpacks to take advantage of these improvement In collaboration with the community...
  24. 24. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ app source (read-only) APM buildpack detect(opt) Node buildpack detect Ruby buildpack detect build plan Metadata describing dependencies to provide during build (overwritable) Detection process for a buildpack group consisting of an optional APM buildpack, a Node.js buildpack, and a Ruby buildpack. Many groups may be candidates for detection. If all non-optional buildpacks in a group pass detection, the group is selected and detection ends. #1: Detect
  25. 25. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Analysis. If an existing image is present, metadata about the last build is recovered from the image config. During the build step, buildpacks may choose to keep, replace, or delete individual directories based on the metadata. These directories are stored as layers. app layer agent layer nodejs layer modules layer mri layer gems layer configuration Remote image from last build my.buildpack.apm/ agent.toml my.buildpack.node/ nodejs.toml modules.toml my.buildpack.ruby/ mri.toml gems.toml app/ ubuntu:18.04 #2: Analyze
  26. 26. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ app source (writable) APM buildpack build Node buildpack build Ruby buildpack build build plan + metadata Build. Each buildpack examines layer metadata from the previous build, re-generates any layer directories that need to be replaced, and updates metadata (if necessary). Buildpacks may choose to modify the app source directory. A transparent, inter-build cache may be used to improve performance and provide other buildpacks with build-time dependencies. APM cache compiled app Node cache Ruby cache agent layer dir [replaces layer] nodejs layer [keep] modules layer dir [replaces layer] mri layer [keep] gems layer dir [replaces layer] #3: Build
  27. 27. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ my.buildpack.apm/ agent.toml agent/ my.buildpack.node/ nodejs.toml modules.toml modules/ my.buildpack.ruby/ mri.toml gems.toml gems/ app/ Export. The remote app layer and any re-generated buildpack layers are replaced. The image config is updated with new metadata. This process uses image layer rebasing, which can be accomplished remotely against a Docker v2 registry. app layer agent layer nodejs layer modules layer mri layer gems layer configuration Remote layers replaced ubuntu:18.04 #4: Export
  28. 28. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ app layer agent layer nodejs layer modules layer mri layer gems layer configuration ubuntu:18.04 ubuntu:18.04 By rewriting OCI image manifests to point to new base layers, CVEs in dependencies with strong compatibility guarantees (e.g., patches to OpenSSL, patches to the JVM) can be mitigated rapidly for many apps. Day 2: containerless-patching
  29. 29. Why do I care? Fast Builds! Portability! Open Standards! Operator-friendly!
  30. 30. Unless otherwise indicated, these slides are © 2013-2018 Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ A demo...

×