O slideshow foi denunciado.
Utilizamos seu perfil e dados de atividades no LinkedIn para personalizar e exibir anúncios mais relevantes. Altere suas preferências de anúncios quando desejar.

My past-3 yeas-developer-journey-at-linkedin-by-iantsai

Ian Tsai shared his past 3years developer journey at Linkedin. it was about migrate monolith into microservices 3 years ago, he faced so diffcult challenges and need to have effective tools to support the change.

Livros relacionados

Gratuito durante 30 dias do Scribd

Ver tudo
  • Seja o primeiro a comentar

My past-3 yeas-developer-journey-at-linkedin-by-iantsai

  1. 1. 我的LinkedIn 三年開發遊歷 Microservice practice in Linkedin
  2. 2. Table of Content LinkedIn 的 Service infra 我的心得:從Microsercvice Architecture 看 LinkedIn Service Infra LinkedIn 的 CI/CD process 我的心得:從Microservice Architecture 看CI/CD 結論
  3. 3. Welcome to LinkedIn Engineering! LinkedIn’s Service infra
  4. 4. Codebase Archive 程式庫管理格式 Multi-product 1 MP : 1 Git Repo Dependency & Container Configuration: product-spec.json ● Depends:{...} ● Includes:{...} ● Application:{...} Centralized configuration management: Fabric + svn
  5. 5. 原始碼控管與審查 Git + Review board iantsai$ git review create iantsai$ git review update iantsai$ git submit Other ReviewBoard Solution: Gerrit
  6. 6. 原始碼搜尋系統(1) Linedin 的repositories:10,000+ Share libs 在哪裡被用到? Configuration files 裡誰呼叫了誰? 今天要被EU給GDPR了,該怎麼找出來哪裡要改要修? 今天要被MS給Bing了,該怎麼找出來哪裡要改要修? JARVIS go/codeSearch ● Indexing ● Querying
  7. 7. 原始碼搜尋系統(2) LinkedIn 的 Search Engine 平台 Galene (Lucene based) 以上都是LinkedIn的,我們可以考慮的Enterprise Code Search Engine Solution: ● OpenGrok ● Zoekt
  8. 8. LinkedIn 的系統監控 Application Measurement Lib(Java): Metrics 自訂圖表與警戒服務InGraphs : 提供json 、yaml格式定義監控數據來源 輸出各式圖表、可搜尋、可集成 搭配 iris + Oncall 制定管理escalation plan (sms, email)
  9. 9. Unified Data Exchange Format Avro Schema (*.avsc) Schema Validation in CI process: Backard compatible enforcement ● rename existing field is not allowed ● newly added field must be optional Restli direct support ● Pegasus Object Serialization and Deserialization ● New -> Old -> New problem
  10. 10. Restful Service Container: Restli(1) Restli Pegasus Data Schema (*.pdsc) Code Generator for Client side Request Builder Library ● *.pdsc ● Gradle based R2D2 Distributed Request handling and distributed request routing
  11. 11. R2: Request Response Framework LinkedIn ways of Request to Response handling pipeline framework(On top of Servlet, similar to JSR 311) Define interfaces to impl your own interceptors, filters for request: ● Call Tree Id issuing and tracking ● Authentication, ● Authorization, ● Context aware Access logging, ● integration testing(Record & Replay)
  12. 12. D2: Dynamic Discovery Context Aware: Cluster sticking on memberId, ContractId, etc Request Distributed Routing: d2://MyService/ContextPath -> http://myServiceApplicationInstance.linkedin.com:9520/ContextPath
  13. 13. Event Queue: Kafka Biz layer model synchronization Near real-time Each event got consumed by one Consumer When to Use? Derived Biz Model update (New Role Assignment, New Email or Phone Number) Topic: A Event 1 Consumer X Consumer Y Kafka Topic: A Event 2
  14. 14. Streaming Pipeline: Brooklin Data layer synchronization Near real-time Each Dataset operation got processed by all pipeline topic subscribers Like Trigger in RDBMS, but able to adapt to different DBs(RDBMS <-> NoSQL) Can Start from epoch to latest(Replaying all actions) When to Use? ● Migrate DB ● Table Trigger DB_A:Table_A: Record_001: update Biz Model B Brooklin Update DB_B:Table_B: Record_010
  15. 15. Tracing System: Call Tree Call Tree Id: ● Only support Restli ● Cookie based Other solution: Zipkin
  16. 16. Microservice in LinkedIn My Opinion: The Service infra part
  17. 17. 個人心得:Reviewboard 很好的產品 使用流程非常順 Diff 可以跟Code Search Engine 整合起來會更好
  18. 18. 個人心得:R2D2 SOA 架構 Sidecar內建在 Framework中 Java Based Shared library required 有很好的開發生產力(CodeGen), 但限制語言平台(Java)
  19. 19. 個人心得:Kafka 用起來簡單,一但大了,管起來很難 LinkedIn: Kafka Cruise Control Yahoo: Kafka-manager confluent:confluent-control-center
  20. 20. 個人心得:Kafka & Brooklin 技術上非常好的framework & platform Monitoring Tool 有點散亂,而且缺乏有結構的文件去整體介紹 積極透過程式碼一致性提高生產力:Java、Avro 但:Samza Code Quality 不夠好(Java 1.4 style)、Review 成本太高
  21. 21. LinkedIn 在技術架構上 SOA Compliant DTO share libs 造成系統嚴重綁定Java -> 與歷史、生產力的妥協 轉Service Mesh 架構時才會遇到一些挑戰
  22. 22. CI/CD in LinkedIn
  23. 23. After you wrote a piece of code Send your patch to Review board, get 2 reviews from colleagues If this contains DB or Service API changes: Data Model Review Committee If this has new API exposed to Internet: Security review - Penetration test If this is a new function impl: ACL approval to all downstream services Verifying GDPR compliance
  24. 24. After you get all Approval Make sure shared Staging ENV is ready Using web UI tools to lunch it to Staging (an environment with all NEWEST Services) Test it in staging semi-auto push it to Canary: go/EKG release to Production env
  25. 25. Start your own Multiproduct? (1) In Your Team: ● Design Doc ● Product team Design Review meeting ● Product SRE team acceptance ● Create new Multi-product(new Git Repo) ○ Declaring ACL Owners ○ Declaring profiles of this MP(Multiproduct) in Fabric ○ Defining Restli Service API (if this is a Rest.li Application
  26. 26. Start your own Multiproduct? (2) DMRC & GDPR: ● Service API DMRC Review ● ACL Access approvals ● Create DB and table Schemas ○ Dataset Schema DMRC Review ○ Dataset's GDPR Compliance validation ○ DB Hadoop snapshot and incremental ETL setup ○ Hadoop SRE approval ○ Data integration SRE approval
  27. 27. Start your own Multiproduct? (3) Security & Product SRE: ● Security Review ● DB & Service inGraph configuration ○ DB Query Per Second (each Table) ○ DB Byte Per Second (each Table) ○ DB HotSpot Detection ○ Service API QPS ○ CallTree Response Time ● Wiki Runbook for Weekly OnCall & Release Owner
  28. 28. Microservice in LinkedIn My Opinion: The CI/CD part
  29. 29. 個人心得:Data Model Review Committee When you MUST go through? ● Service API Schema change ● DB Schema change It’s like: 高速公路收費站 收費員只能對設計合理性做粗略審查
  30. 30. 個人心得:DMRC of Service API Schema DMRC should be like CHP Microservice Restful API 事後補救容易 可以做更進一步的設計審查
  31. 31. 個人心得:DMRC of Service API Schema(2) 更進一步的設計審查: ● Synchronized & Long Operation(Async) API Design ● Scheduled Process & User Request Process ● Error Handling & Inter Services status propagation
  32. 32. 個人心得:DMRC of DB schema Wrong Design of DB Table Schema is very expensive New Table 不但得全面攔查,而且是做Design Review: ● NoSQL DB ○ Partition Key Design ○ Key Length ○ encryption ○ Usage: Getter or Finder ○ Mapping & Reverse mapping ○ Brooklin as Trigger ● Bounded Context check
  33. 33. 個人心得:Linkedin Staging Environment 缺乏實驗組/控制組的概念 測試失敗無法100%確定成因 Integration Test Results 不可靠 Newest A Newest B PROD D PROD C Newest E CI/CD Pipeline Staging Cluster
  34. 34. 個人心得:Linkedin Staging Environment 任一個Codebase 有兩個Instance: ● PROD ● Newest CI/CD 生成針對PR的Request Routing Plan 保證 Integration Test Results 可靠 CI/CD Pipeline Newest A PROD B PROD D PROD C PROD E Staging Cluster Newest D Newest E Newest C Newest BPROD A
  35. 35. 從Microservice 看產品
  36. 36. 一個產品的Repositories Old: ● XXX-frontend ● XXX-backend New: ● XXX-web ● XXX-api ● XXX-mt ● ooo-backend ● XXX-samza ● XXX-common
  37. 37. This is Distributed Monolith, not Microservice! Distributed Monolith Syndromes: ● Service name is same as team name or product name! ● Service name has no High-level Business conceptual terms in it ● Services names prefix or suffix consistent with 3 layer architecture
  38. 38. 個人心得:Distributed Monolith 的成因 如果 Conway's Law : ● Repository、Service 無法照業務切割 如果開始一個Repository 成本太高: ● Developer 會想偷懶 ● Codebase 容易爛 如果Staging 對Integration Test不夠友善: ● SRE 會癱瘓() ● Release Process 無法全自動
  39. 39. 總結
  40. 40. 關於大編制的Web Application產品開發 對任何已經走到SOA架構的團隊,要能適應Microservice Architecture: 審查流程需要瘦身:CHP is better than Checkpoint Staging 需要改善:PROD/Newest instances for every Repos Knowledge Management: Forum is better than Email, Jira and Office hour Knowledge Management:Omni Search Engine(Code & text) is very important
  41. 41. Knowledge Management For any Product Development team 50+ Code Review Platform : Review board, Gerrit Code Search Platform : go/codeSearch, OpenGrok, Zoekt Wiki, Forum: StackOverflow Ticket System: Jira Omni Search Engine: Omni Search Engine Code Review Platform Code Search Platform Wiki Forum Ticket System
  42. 42. Q & A

    Seja o primeiro a comentar

    Entre para ver os comentários

  • FongXuanLiou

    May. 8, 2019
  • mshang5

    May. 8, 2019
  • williamyeh

    May. 8, 2019
  • SilberLee

    May. 8, 2019
  • RiverLin4

    May. 8, 2019
  • gostlike

    May. 8, 2019
  • HoYinWong7

    May. 12, 2019
  • HYDN

    Jul. 9, 2019
  • ShaochengZhang

    Feb. 15, 2020

Ian Tsai shared his past 3years developer journey at Linkedin. it was about migrate monolith into microservices 3 years ago, he faced so diffcult challenges and need to have effective tools to support the change.


Vistos totais


No Slideshare


De incorporações


Número de incorporações