O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

Instant developer onboarding with self contained repositories

Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Próximos SlideShares
Running AWS Locally
Running AWS Locally
Carregando em…3
×

Confira estes a seguir

1 de 53 Anúncio

Instant developer onboarding with self contained repositories

Baixar para ler offline

Slide from my talk on "Instant developer onboarding with self-contained repositories".
https://sched.co/l9yG

Code examples on:
https://github.com/Yshayy/self-contained-repositories

Conference Recordings will be added once it will be public

Slide from my talk on "Instant developer onboarding with self-contained repositories".
https://sched.co/l9yG

Code examples on:
https://github.com/Yshayy/self-contained-repositories

Conference Recordings will be added once it will be public

Anúncio
Anúncio

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a Instant developer onboarding with self contained repositories (20)

Anúncio

Mais recentes (20)

Instant developer onboarding with self contained repositories

  1. 1. Instant Self-contained Development Environments for Everyone Yshay Yaacobi CTO @ Livecycle yshay@livecycle.io yshayy
  2. 2. TWEEKATHON
  3. 3. @yshayy About me ● CTO & co-founder of livecycle ● Full-stack developer for quite a time ● Passionate about cloud development, backend architecture, UX/DX, functional programming and Docker ● Creator & maintainer of Tweek - an open-source, “cloud- native” feature management solution ● Things I care about in code: simplicity, consistency and elegance
  4. 4. @yshayy About Livecycle ● Next generation collaboration tools for development teams ● Continuous playground environments ● Bridging the gap between coders & non-coders ● Soon in public beta
  5. 5. How does it feel like to start working on a new complex codebase?
  6. 6. @yshayy Try to build & run ● Wrong OS ● Missing or conflicting SDKs and/or PL runtimes ● Package managers throw “random” errors
  7. 7. @yshayy Read README.md and try to make it work ● Run some magic scripts (and watch them fails) ● Change hosts file ● Install required tools & dependencies on the environment ● Install Root CA?
  8. 8. @yshayy Try to develop ● Debugging doesn’t work ● IDE have problem with autocomplete or dependencies ● Code watch & build doesn’t work and we need watchman ● HMR doesn’t work because of websocket issues ● Problems with external dependencies ● CORS
  9. 9. Integration tests?
  10. 10. Do it all over again after few months of working on something else....
  11. 11. @yshayy Why is it so difficult? ● Different OSes (or versions) ● Lots of versions and fragmentation of sdk & runtimes ● Work on my machine syndrome ● Vast amount of different toolchains, IDEs, extensions ● Complex development flows that are difficult to setup and easily break: ○ Debugging, watching+building, hot-reloading, docker mounts ● Developers’ machines are polluted and overloaded with tools ● Environments and tools tend to change rapidly in active repositories ● ----> Waste of time and frustration!
  12. 12. The dream Development environments that are...
  13. 13. Consistent Provide the same predictable experience...
  14. 14. Reproducible It’s possible to destroy & rebuild them
  15. 15. Isolated Don’t or get affected by other environments
  16. 16. Self-Contained All dependencies and tools needed for development are defined & packaged inside
  17. 17. UNBREAKABLE We won’t struggle countless hours to get things working again
  18. 18. @yshayy Running our environments in containers
  19. 19. @yshayy Before we begin ● All examples & slides are available on Github ● All tools used in this presentation are OSS or free-to-use projects ● Most examples here are far from bullet-proof and some of them use tools that can be considered experimental
  20. 20. @yshayy Example #1 - qeesung/image2ascii CLI tool for creating ASCII art from images ● Project written in Go ● Before Go modules
  21. 21. @yshayy The Development Container ● Integration with SCM ● Remote code editing ● Remote terminal
  22. 22. @yshayy Configuring our environment ● Setting Runtime/SDKS/CLIs ● Setting environment variables and path ● Configure our shell ● Defining extensions
  23. 23. @yshayy Example #2 - yshayy/email-sender Simple Flask app to send email based on SendGrid example New challenges ● Running & interacting with a server ● Managing secrets ● Debugging
  24. 24. @yshayy Secrets Encryption ● Secrets sit inside the repository ● Using Mozilla Sops for encrypting secrets ● GPG keys are nice for start, but it can also integrate with cloud encryptions-as- service solution such as KMS, KeyVault ● MetaData is saved unencrypted which make it easy to do diffing and check history ● Practice usually used in GitOps context ● Other solutions - git-secret, git-crypt, Kamus, SealedSecrets, etc...
  25. 25. @yshayy IDE settings ● Launch settings - launch.json ● Port forwarding - forwarding port on the localhost
  26. 26. @yshayy #3 HabitRPG/Habitica OSS Web-based RPG for organizing your life New Challenges ● Huge project ● Front end ● Backend ● DB
  27. 27. @yshayy Full-stack application ● Docker-compose ● DB Image ● Additional tools
  28. 28. @yshayy Data seeding ● Basic scripts ● Alternatively, we can clone data from staging/production
  29. 29. @yshayy Use reverse proxy ● Route services to several subdomains instead of ports ● Wildcard “localhost” dns-es (localtest.me, xip) ● Traefik - a very simple and developer friendly reverse proxy
  30. 30. The next one is personal...
  31. 31. @yshayy #4 - Soluto/Tweek Cloud-native open-source feature flags and configuration management New challenges ● Several microservices ● Several DBs/Messaging systems ● Cross service communications ● Polyglot environment
  32. 32. @yshayy Complex architecture
  33. 33. @yshayy “Nested” containers ● Docker-in-docker vs docker-from-docker ● Development in nested containers with Tilt + Docker-Compose ○ Watching & rebuilding on every code commit ○ Remote debugging ○ “Hot” code reloading if possible ● Things can get slower...
  34. 34. @yshayy “Mock” cloud dependencies ● Docker images of databases (redis, mongo, postgres, etc…) ● Wire-compatible solutions (Minio, OIDC mock server) ● Manual mocks ● Full frameworks (localstack) ● Encrypted credentials with dedicated tenants. (or dynamic provisioning)
  35. 35. @yshayy #5 - kubecost/cost-model Tool for managing kubernetes costs New challenges ● We need kubernetes ● Metrics server ● Prometheus
  36. 36. @yshayy What can we do with Kubernetes? ● Kubernetes local development is already difficult ○ Fragmentation - Mini-kube, Docker for Desktop k8s, micro-k8s, kind, k3s, etc… ○ Versioning ○ Upgrading ● Using a single kubernetes distro+version can make life easy
  37. 37. @yshayy Kubernetes in dev-container ● K3S - minimal kubernetes distribution ● K3D - Make it easy to run k3s and a dedicated registry inside Docker ● Stable and cluster can be re-created ● Helm controller for installing helm charts declaratively ● Tilt facilitate building/pushing/running
  38. 38. @yshayy Docker Host Dev Container IDE Tilt Docker-in-Docker Registry K3S Node ContainerD App To put it “simply”....
  39. 39. @yshayy
  40. 40. No more demos…
  41. 41. @yshayy Containerized development environments ● Development environment configuration is also source-controlled and correspond to the application code. ● Developer machine stays clean ● Can scale well to multiple environments without conflicts ● Can run locally or remotely
  42. 42. @yshayy Our setup ● ~10 microservices in Golang & typescript/js ● Frontend with HMR ● Our own Kubernetes CR and controllers ● External dependencies (GH api, sendgrid, auth0, etc…) ● DB + Graphql engine ● A full blown CI system ● Container Registry ● Dynamic dns subdomains ● SSL Certs for local development ● Multiple CLI/SDKs for kubernetes and code-generation
  43. 43. @yshayy Results ● Time to teardown and download/rebuild all cluster and dependencies <15m ● Time to build/run/test code changes < 10s ● Time to onboard new developer < 3h (including remote provisioning of a dedicated host on AWS) ● Time to introduce new tool and update dev-environments if needed <5m) ● No “works on my machine” occurrences ● No strain on developer machines ● Developers need to deal less with secrets ● Our team work on both M1 and Intel Macs
  44. 44. @yshayy Future optimizations ● Shared build cache ● Snapshots (for reducing time or sharing state) ● Using a cloud provider optimized for dev-machines (cost, location, hibernation, cpu/ram, etc...)
  45. 45. @yshayy Drawbacks ● Creating the initial setup can take some time ● Many tools, some are bleeding-edge ● Additional code to manage ● Dev-Environments are not standardized yet ● Coupling to VS-Code, Docker, Git and Linux ● Some performance issues ● Security challenges between development and production context
  46. 46. @yshayy Alternatives to VS Code? ● It’s possible to use terminal based code-editors ● GitPod.io & Theia have similar features with gitpod.yaml ● Jetbrains Projector ● Run local IDE with Docker mounts
  47. 47. @yshayy What about serverless? ● Should work although some pieces might be missing ● FaaS frameworks and sdks usually can run locally ● For the other PaaS features - cloud mocking frameworks, emulators or wire compatible solutions. (localstack, minio, etc…) ● If necessary, throw IaC tools to the mix (pulumi/terraform) for dynamic provisioning
  48. 48. @yshayy What about native mobile? ● The problem still exists, developers can engage in epic battles with mobile IDEs, workspace, build and debug. ● It might be possible to stream applications with VNC/WebRTC during development, but experience is not optimal ● Mobile-emulators are heavy and can require nested virtualization to perform ● Container ecosystem is optimized for Linux ● IDEs for mobile are very tailored for mobile development ● Might be easier/possible with more cross-platform frameworks such as react- native/flutter
  49. 49. @yshayy Part of a larger trend to put more stuff in the repository ● Linting, style guidelines ● Declarative application dependencies ● Documentation ● OpenAPI Specification ● CI Pipelines definition ● Workflows (PR) ● Design systems ● Infrastructure-As-Code ● Secrets (encrypted) ● Dashboards/alerts/SLA configuration ● Notebooks ● ...
  50. 50. @yshayy Self-Contained Repositories ● All code, tools, knowledge, definitions and processes related to project resides in the repository. ● Git as the single source of truth ● Code is more accessible, lowering the barrier of entry ● Applications are portable ● Fine-grained developer experience ● Emerging tools ecosystem
  51. 51. @yshayy Patterns & Cheatsheet Challenge Solution Exampletools Basic SDK/Runtime dependencies Development in container VSCode+Docker, GitPod Cloud dependencies Compatible dockerized implementations Minio, redis Initial application data Data seeding Scripts, replicating from cloud More hardware Remote environment Docker-machine, codespaces Multi-Container Apps Nested Containers Dind, compose, tilt Kubernetes Apps Nested Kubernetes Kind, k3d, tilt, skaffold Exposing network dependencies Reverse proxy, wild card development dns Traefik, nginx, *.xip, *.localtest.me Serverless Mock cloud frameworks Localstack
  52. 52. Thank you @yshayy https://github.com/yshayy/self-contained-repositories
  53. 53. Questions

×