O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.
Develop, deploy, and operate services
at Reddit scale
Greg Taylor – EM, Reddit Infrastructure
/u/gctaylor
What is Reddit?
/r/CrappyDesign
/r/theocho
/r/news
/r/daddit
/r/IAMA
Reddit by the numbers
5th/8th Alexa Rank (US/World)
330M+ Monthly active users
138K Active communities
12M Posts per month...
Rapid Recent Growth
Engineers in early 2016
Engineers today
Increase in less than three years
25
185
7X
Reddit Engineering in 2016
Monolith
2016 - The Infrastructure Team
● Provisioned and configured all infrastructure
● Operated most of our systems
● Handled mo...
2016 - The Stack
Mid 2016 - Rapid growth
Diminishing returns
Determining the path forward
Service-oriented Architecture (SOA)
The first service
Monolith
Service
Growing pains: Automated testing
● Problem: Unit/integration testing as an afterthought
○ “We know we should do better, bu...
Growing pains: Something to build on
● Problem: Artisanal, hand-crafted services
● Solution: Reddit service framework
○ Na...
Hey! That was pretty cool. Let’s do it again!
Service
Monolith
Service
Growing pains: Artisanal Infrastructure
● Problem: Artisanal, hand-crafted Infrastructure
● Solution: Infrastructure-as-co...
Once more, with feeling!
Service
Service
Monolith
Service
Service
Growing pains: Staging/integration woes
● Problem: Inadequate staging solution for SOA
● Outlandish idea: Give Kubernetes ...
Reddit Engineering in 2017
Service
Service
Service
Service
Monolith
Growing pains: Infra team as a bottleneck
● Problem: Development is too dependent on Infra
○ Service provisioning
○ Ongoin...
One size fits all some
Not all teams want to operate the full stack for their service
Don’t make me
infrastructure!
My toa...
You want me to do WHAT?!?
What do we really really want?
Service ownership!
A service owner is empowered to:
● Dev and test their service in a prod-like env
● Do most of the work ...
Service ownership challenges
The learning curve looms large
Service ownership challenges
With great power comes great responsibility
Service ownership challenges
Mistakes are going to happen… often!
Early 2018 - How to enable Service Ownership?
Reddit Infrastructure as a Product
Selecting the right foundation
How to build an Infrastructure Product
Limit the surface area
The user/operator contract
Product Users (Service Owners)
● Learn some Kubernetes basics
● Deploy and operate own services...
Batteries-included
Provide as much as we can out-of-the-box:
● Test, build, and deploy pipelines
● Observability
● Service...
Paint-by-numbers
Enabling service ownership for all backgrounds:
● Limit learning expectations
● You want to do X? Here’s ...
Consistency throughout
As a developer takes an application through its development cycle, the
operating environment should...
Deploy with confidence
A developer should have tools and processes that will help them build
confidence as work approaches...
Guardrails and Safeties
Things that prevent or minimize damage
● Resource limits
● Throttling
● Network policy and Access ...
Reddit Infrastructure Product Core Tenets
● Limit the surface area
● The user/operator contract
● Batteries-included
● Pai...
What does all of this buy us?
Infra team: From Operators to Enablers
Infra team: From Operators to Enablers
Infra provisions all infrastructure Infra provides infra as a product
Infra deploys...
Organizational scalability
Closing Remarks
reddit.com/jobs
Presenter Info + Resources
● Greg Taylor - Reddit Infrastructure
● /u/gctaylor
● @gctaylor
● github.com/gtaylor
● reddit.c...
Develop, deploy, and operate services at reddit scale   oscon 2018
Próximos SlideShares
Carregando em…5
×

de

Vídeos do YouTube não são mais aceitos pelo SlideShare

Visualizar original no YouTube

Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 2 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 3 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 4 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 5 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 6 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 7 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 8 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 9 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 10 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 11 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 12 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 13 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 14 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 15 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 16 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 17 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 18 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 19 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 20 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 21 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 22 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 23 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 24 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 25 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 26 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 27 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 28 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 29 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 30 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 31 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 32 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 33 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 34 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 35 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 36 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 37 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 38 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 39 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 40 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 41 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 42 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 43 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 44 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 45 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 46 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 47 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 48 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 49 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 50 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 51 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 52 Develop, deploy, and operate services at reddit scale   oscon 2018 Slide 53
Próximos SlideShares
What to Upload to SlideShare
Avançar
Transfira para ler offline e ver em ecrã inteiro.

0 gostaram

Compartilhar

Baixar para ler offline

Develop, deploy, and operate services at reddit scale oscon 2018

Baixar para ler offline

The last few years have been a period of tremendous growth for Reddit. Process, tooling, and culture have all had to adapt to an organization that has tripled in size and ambition. Greg Taylor discusses Reddit's evolution and explains how one of the world’s busiest sites develops, deploys, and operates services at significant scale.

Presented at OSCON 2018 in Portland, Oregon

  • Seja a primeira pessoa a gostar disto

Develop, deploy, and operate services at reddit scale oscon 2018

  1. 1. Develop, deploy, and operate services at Reddit scale Greg Taylor – EM, Reddit Infrastructure /u/gctaylor
  2. 2. What is Reddit?
  3. 3. /r/CrappyDesign
  4. 4. /r/theocho
  5. 5. /r/news
  6. 6. /r/daddit
  7. 7. /r/IAMA
  8. 8. Reddit by the numbers 5th/8th Alexa Rank (US/World) 330M+ Monthly active users 138K Active communities 12M Posts per month 2B Votes per month
  9. 9. Rapid Recent Growth Engineers in early 2016 Engineers today Increase in less than three years 25 185 7X
  10. 10. Reddit Engineering in 2016 Monolith
  11. 11. 2016 - The Infrastructure Team ● Provisioned and configured all infrastructure ● Operated most of our systems ● Handled most non-trivial debugging
  12. 12. 2016 - The Stack
  13. 13. Mid 2016 - Rapid growth
  14. 14. Diminishing returns
  15. 15. Determining the path forward
  16. 16. Service-oriented Architecture (SOA)
  17. 17. The first service Monolith Service
  18. 18. Growing pains: Automated testing ● Problem: Unit/integration testing as an afterthought ○ “We know we should do better, buut….” ○ We needed CI for mere mortals ● Solution: Cultural shift ○ Git hooks are not enough! ○ Testing, linting, integration, oh my! ○ Green master branches ● https://drone.io/
  19. 19. Growing pains: Something to build on ● Problem: Artisanal, hand-crafted services ● Solution: Reddit service framework ○ Named “baseplate” ○ Consistency between services ○ Batteries-included ● https://baseplate.readthedocs.io
  20. 20. Hey! That was pretty cool. Let’s do it again! Service Monolith Service
  21. 21. Growing pains: Artisanal Infrastructure ● Problem: Artisanal, hand-crafted Infrastructure ● Solution: Infrastructure-as-code ○ Declarative, repeatable ○ Code review… for infrastructure ○ Reusable modules ● https://www.terraform.io/
  22. 22. Once more, with feeling! Service Service Monolith Service Service
  23. 23. Growing pains: Staging/integration woes ● Problem: Inadequate staging solution for SOA ● Outlandish idea: Give Kubernetes a shot ○ Let’s see what we can MVP in 3 weeks! ○ Pile of manifests and bash scripts ○ MVP very positively received ● https://kubernetes.io/
  24. 24. Reddit Engineering in 2017 Service Service Service Service Monolith
  25. 25. Growing pains: Infra team as a bottleneck ● Problem: Development is too dependent on Infra ○ Service provisioning ○ Ongoing operation ○ Debugging and performance work ● Short-term “solution”: Train and deputize infrastructure-oriented teams ○ Allows for more self-sufficiency ○ Results in accelerated dev velocity
  26. 26. One size fits all some Not all teams want to operate the full stack for their service Don’t make me infrastructure! My toaster runs Docker
  27. 27. You want me to do WHAT?!?
  28. 28. What do we really really want?
  29. 29. Service ownership! A service owner is empowered to: ● Dev and test their service in a prod-like env ● Do most of the work to get to production ● Own the health of their service ● Diagnose issues
  30. 30. Service ownership challenges The learning curve looms large
  31. 31. Service ownership challenges With great power comes great responsibility
  32. 32. Service ownership challenges Mistakes are going to happen… often!
  33. 33. Early 2018 - How to enable Service Ownership?
  34. 34. Reddit Infrastructure as a Product
  35. 35. Selecting the right foundation
  36. 36. How to build an Infrastructure Product
  37. 37. Limit the surface area
  38. 38. The user/operator contract Product Users (Service Owners) ● Learn some Kubernetes basics ● Deploy and operate own services Product Operators (Reddit Infrastructure) ● Keep the Kubernetes clusters running ● Provision AWS resources ● Support and advise Product Users
  39. 39. Batteries-included Provide as much as we can out-of-the-box: ● Test, build, and deploy pipelines ● Observability ● Service discovery ● Baseline security
  40. 40. Paint-by-numbers Enabling service ownership for all backgrounds: ● Limit learning expectations ● You want to do X? Here’s a guide for that ● Training and extensive documentation ● Auto-generate all the things An engineer should not require deep infra experience in order to be productive!
  41. 41. Consistency throughout As a developer takes an application through its development cycle, the operating environment should be as consistent as possible Local dev Staging Production
  42. 42. Deploy with confidence A developer should have tools and processes that will help them build confidence as work approaches production Code Review Unit Testing Integration Testing Some
  43. 43. Guardrails and Safeties Things that prevent or minimize damage ● Resource limits ● Throttling ● Network policy and Access controls ● Scanning for common mistakes ● Docker Image policies
  44. 44. Reddit Infrastructure Product Core Tenets ● Limit the surface area ● The user/operator contract ● Batteries-included ● Paint-by-numbers ● Consistency throughout ● Deploy with confidence ● Guardrails and Safeties
  45. 45. What does all of this buy us?
  46. 46. Infra team: From Operators to Enablers
  47. 47. Infra team: From Operators to Enablers Infra provisions all infrastructure Infra provides infra as a product Infra deploys new services Service owners deploy new services Infra operates most services Service owners operate services Infra is a blocking dependency Infra is an adviser and enabler
  48. 48. Organizational scalability
  49. 49. Closing Remarks
  50. 50. reddit.com/jobs
  51. 51. Presenter Info + Resources ● Greg Taylor - Reddit Infrastructure ● /u/gctaylor ● @gctaylor ● github.com/gtaylor ● reddit.com/r/kubernetes ● redditblog.com/topic/technology

The last few years have been a period of tremendous growth for Reddit. Process, tooling, and culture have all had to adapt to an organization that has tripled in size and ambition. Greg Taylor discusses Reddit's evolution and explains how one of the world’s busiest sites develops, deploys, and operates services at significant scale. Presented at OSCON 2018 in Portland, Oregon

Vistos

Vistos totais

1.614

No Slideshare

0

De incorporações

0

Número de incorporações

7

Ações

Baixados

12

Compartilhados

0

Comentários

0

Curtir

0

×