O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Kubernetes at Reddit: An Origin Story
Greg Taylor – EM, Reddit Infrastructure
/u/gctaylor
What is Reddit?
Something for everyone
5th/20th Alexa Rank (US/World)
400M+ Monthly active users
140K+ Communities
12M+ Posts per month
2B...
Example: /r/kubernetes
Get on with it!
Welcome to early 2016
2016 - The Infrastructure Team
● Provisioned and configured all infrastructure
● Operated most of our systems
● Responded ...
Mid-2016 and onward: The Great Embiggening
Determining the path forward
Service-oriented Architecture (SOA)
Growing pains: Infra team as a bottleneck
● Problem: Eng teams too dependent on Infra team
○ Service provisioning
○ Ongoin...
One size fits all some
Not all teams want to operate the full stack for their service
Don’t make me
infrastructure!
My toa...
What do the engineering teams REALLY want?
Service ownership
A service owner is empowered and expected to:
● Develop their service from start to finish
● Deploy thei...
Enter: InfraRedd
A service owner should be able to
develop, deploy, and operate their service.
Regardless of engineering background
Develop: Consistency in services
Regardless of language/toolset, the “shape” of
each service should be consistent:
● RPC p...
Develop: Service creation
Auto-generate starter material:
Service sources
● Python/Go/Node service stub
● Dockerfile
● CI ...
Develop: “Local” development
Development is facilitated by Skaffold.
Major considerations:
● Accessible to those without d...
A service owner should be able to
develop, deploy, and operate their service.
Regardless of engineering background
Deploy: Tests, builds, deploys
● CI runs through Drone
a. Tests
b. Artifact builds
● Spinnaker handles our deploys
a. Stan...
Deploy: Standard staging/production flow
Staging and production deploy flow:
1. Developer pushes to canonical repo
2. Test...
A service owner should be able to
develop, deploy, and operate their service.
Regardless of engineering background
Operate: Explicitly defined expectations
Service owners
● Learn some Kubernetes basics
● Deploy and operate own services
I...
Operate: Paint-by-numbers
Enabling service ownership for all backgrounds:
● Take the guesswork out
● Document all the thin...
● Service owners auth via
OpenID Connect
● RBAC policies are group-based
● Namespace per service
● Service owners have ful...
Operate: Guardrails
Things that prevent or minimize damage
● Resource limits and Network Policies
○ Built into Kubernetes
...
Operate: Oh no!
Something exploded!
Service owner:
1. Paged for service incident
2. Diagnoses + resolves issue
3. Can summ...
Observability by default:
● Metrics
○ Wavefront
● Alerting
○ PagerDuty
● Tracing
○ Zipkin
● Exception/error tracking
○ Sen...
Operate: Recap
Service owners have:
● Explicitly defined responsibilities
● Enough access to own their services
● Guardrai...
What does all of this buy us?
A service owner should be is able to
develop, deploy, and operate their service.
Regardless of engineering background
Infra team: From Operators to Enablers
Kubernetes at Reddit
7 Kubernetes clusters
~30% Of our Engineering teams
~20 Production services
10-20 Deploys a day
New s...
Closing Remarks
reddit.com/jobs
Presenter Info + Resources
● Greg Taylor - Reddit Infrastructure
● /u/gctaylor
● @gctaylor
● github.com/gtaylor
● reddit.c...
Kubernetes at Reddit: An Origin Story - KubeCon NA 2018
Próximos SlideShares
Carregando em…5
×

de

Vídeos do YouTube não são mais aceitos pelo SlideShare

Visualizar original no YouTube

Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 2 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 3 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 4 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 5 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 6 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 7 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 8 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 9 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 10 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 11 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 12 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 13 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 14 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 15 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 16 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 17 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 18 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 19 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 20 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 21 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 22 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 23 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 24 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 25 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 26 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 27 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 28 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 29 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 30 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 31 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 32 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 33 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 34 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 35 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 36 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 37 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 38 Kubernetes at Reddit: An Origin Story - KubeCon NA 2018 Slide 39
Próximos SlideShares
What to Upload to SlideShare
Avançar
Transfira para ler offline e ver em ecrã inteiro.

5 gostaram

Compartilhar

Baixar para ler offline

Kubernetes at Reddit: An Origin Story - KubeCon NA 2018

Baixar para ler offline

This session takes the audience through the motivations, the story, and the results of rebasing our infrastructure on Kubernetes. Today, Kubernetes forms the core of our internal Infrastructure "product", which is a paint-by-numbers experience that engineers of all backgrounds make use of.

Presented at KubeCon North America in 2018

Audiolivros relacionados

Gratuito durante 30 dias do Scribd

Ver tudo

Kubernetes at Reddit: An Origin Story - KubeCon NA 2018

  1. 1. Kubernetes at Reddit: An Origin Story Greg Taylor – EM, Reddit Infrastructure /u/gctaylor
  2. 2. What is Reddit?
  3. 3. Something for everyone 5th/20th Alexa Rank (US/World) 400M+ Monthly active users 140K+ Communities 12M+ Posts per month 2B+ Votes per month
  4. 4. Example: /r/kubernetes
  5. 5. Get on with it!
  6. 6. Welcome to early 2016
  7. 7. 2016 - The Infrastructure Team ● Provisioned and configured all infrastructure ● Operated most of our systems ● Responded to most incidents
  8. 8. Mid-2016 and onward: The Great Embiggening
  9. 9. Determining the path forward
  10. 10. Service-oriented Architecture (SOA)
  11. 11. Growing pains: Infra team as a bottleneck ● Problem: Eng teams too dependent on Infra team ○ Service provisioning ○ Ongoing operation ○ Debugging and performance work ● Short-term “solution”: Train and deputize infrastructure-oriented teams ○ Allows for more self-sufficiency ○ Only possible for some teams!
  12. 12. One size fits all some Not all teams want to operate the full stack for their service Don’t make me infrastructure! My toaster runs Docker
  13. 13. What do the engineering teams REALLY want?
  14. 14. Service ownership A service owner is empowered and expected to: ● Develop their service from start to finish ● Deploy their service early and often ● Operate their service
  15. 15. Enter: InfraRedd
  16. 16. A service owner should be able to develop, deploy, and operate their service. Regardless of engineering background
  17. 17. Develop: Consistency in services Regardless of language/toolset, the “shape” of each service should be consistent: ● RPC protocol ● Secrets fetching ● Metrics ● Tracing ● Log output format Baseplate: https://baseplate.readthedocs.io
  18. 18. Develop: Service creation Auto-generate starter material: Service sources ● Python/Go/Node service stub ● Dockerfile ● CI configs Helm Charts ● Friends don’t let friends write YAML!
  19. 19. Develop: “Local” development Development is facilitated by Skaffold. Major considerations: ● Accessible to those without deep Kubernetes experience ● As similar to production as possible ● Re-use our standard Charts + images ● Must not exhaust standard dev laptop’s resources
  20. 20. A service owner should be able to develop, deploy, and operate their service. Regardless of engineering background
  21. 21. Deploy: Tests, builds, deploys ● CI runs through Drone a. Tests b. Artifact builds ● Spinnaker handles our deploys a. Standardized pipeline templates b. Renders Helm Charts c. Applies rendered YAML d. Uses V2 Kubernetes provider
  22. 22. Deploy: Standard staging/production flow Staging and production deploy flow: 1. Developer pushes to canonical repo 2. Tests and builds run in CI 3. One of two flows are offered: a. CI triggers a deploy b. Eng manually triggers a deploy
  23. 23. A service owner should be able to develop, deploy, and operate their service. Regardless of engineering background
  24. 24. Operate: Explicitly defined expectations Service owners ● Learn some Kubernetes basics ● Deploy and operate own services Infrastructure team ● Keep the Kubernetes clusters running ● Provision AWS resources, caches, DBs ● Support and advise Product Users
  25. 25. Operate: Paint-by-numbers Enabling service ownership for all backgrounds: ● Take the guesswork out ● Document all the things ● You want to do X? Here’s a guide for that ● Must be supported by training!
  26. 26. ● Service owners auth via OpenID Connect ● RBAC policies are group-based ● Namespace per service ● Service owners have full access to their namespace(s) Operate: Service owner permissions
  27. 27. Operate: Guardrails Things that prevent or minimize damage ● Resource limits and Network Policies ○ Built into Kubernetes ● Throttling and circuit breaking ○ Envoy + Istio ● Object and Image policies ○ Open Policy Agent ● Finely scoped RBAC ○ Open Policy Agent
  28. 28. Operate: Oh no! Something exploded! Service owner: 1. Paged for service incident 2. Diagnoses + resolves issue 3. Can summon Infra if needed Infrastructure team: 1. Paged for cluster issues 2. Those never happen. Yep.
  29. 29. Observability by default: ● Metrics ○ Wavefront ● Alerting ○ PagerDuty ● Tracing ○ Zipkin ● Exception/error tracking ○ Sentry ● Central logging and analysis Operate: Observe, Diagnose, Resolve
  30. 30. Operate: Recap Service owners have: ● Explicitly defined responsibilities ● Enough access to own their services ● Guardrails to prevent+limit damage ● Tools needed to respond to and diagnose issues
  31. 31. What does all of this buy us?
  32. 32. A service owner should be is able to develop, deploy, and operate their service. Regardless of engineering background
  33. 33. Infra team: From Operators to Enablers
  34. 34. Kubernetes at Reddit 7 Kubernetes clusters ~30% Of our Engineering teams ~20 Production services 10-20 Deploys a day New services on Kubernetes by default in Q1!
  35. 35. Closing Remarks
  36. 36. reddit.com/jobs
  37. 37. Presenter Info + Resources ● Greg Taylor - Reddit Infrastructure ● /u/gctaylor ● @gctaylor ● github.com/gtaylor ● reddit.com/r/kubernetes ● redditblog.com/topic/technology
  • zhixingchou1

    Sep. 18, 2019
  • uzy_exe

    Feb. 17, 2019
  • gh02

    Jan. 3, 2019
  • RaviSingh139

    Dec. 13, 2018
  • PhilippeScorsolini

    Dec. 12, 2018

This session takes the audience through the motivations, the story, and the results of rebasing our infrastructure on Kubernetes. Today, Kubernetes forms the core of our internal Infrastructure "product", which is a paint-by-numbers experience that engineers of all backgrounds make use of. Presented at KubeCon North America in 2018

Vistos

Vistos totais

2.265

No Slideshare

0

De incorporações

0

Número de incorporações

131

Ações

Baixados

15

Compartilhados

0

Comentários

0

Curtir

5

×