O slideshow foi denunciado.
Seu SlideShare está sendo baixado. ×

20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management

Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio
Anúncio

Confira estes a seguir

1 de 120 Anúncio

20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management

Baixar para ler offline

隨著公司的業務發展,我們開始注意到要快速的部署一個新的環境、市場越來越沒有效率,而 CM(Configuration Management)在我們解決這些問題中扮演著重要角色。本演講將專注於解決下列問題:

1. 為什麼我們需要 CM?
2. 如何管理不同種類的 Configuration: Realtime, Service Discovery and Provision?
3. 如何降低團隊的導入 CM 複雜度?

隨著公司的業務發展,我們開始注意到要快速的部署一個新的環境、市場越來越沒有效率,而 CM(Configuration Management)在我們解決這些問題中扮演著重要角色。本演講將專注於解決下列問題:

1. 為什麼我們需要 CM?
2. 如何管理不同種類的 Configuration: Realtime, Service Discovery and Provision?
3. 如何降低團隊的導入 CM 複雜度?

Anúncio
Anúncio

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a 20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management (20)

Anúncio

Mais recentes (20)

20191018 DevOpsDays Taipei 2019 從零開始的 Configuration Management

  1. 1. 從零開始的 Configuration Management Levi Chen @ DevOpsDays Taipei 2019
  2. 2. About Me 陳俊毅 (Levi Chen) Engineer in Common Service & Infra Team @ 91APP Contact me ● FB: https://fb.me/chenjiunyi ● Blog: https://blog.levichen.tw 2
  3. 3. Responsibilities @ 91APP ● Develop 3 common services ○ Product, Notification and Rate Limiter ● Help RDs to use cloud technologies ○ Cloud Native, Serverless ● Help architecture team to implement new policies About Me 3
  4. 4. 4
  5. 5. 5 https://paulhammant.com/2014/08/27/provisioning-deployment-and-app-config-cycles/
  6. 6. Agenda ● Why configuration management? ● How to manage configuration? ● How to go live? 6
  7. 7. Agenda ● Why configuration management? ● How to manage configuration? ● How to go live? 7
  8. 8. 台灣最大 & 成長最快 品牌新零售解決方案公司 - 2013 年成立 - 總部在台北,馬來西亞/香港分公司 - 員工人數 400 人 8
  9. 9. 9
  10. 10. 品牌客戶超過 10,000 家 10
  11. 11. Challenge #1 Deploying to Mutiple Environments 11
  12. 12. TW HK MY Expand to New Markets 2013 TW 2018 MY 2019 HK 12
  13. 13. Load Test https://www.monitis.com/blog/load-test-and-be-sure-your-website-is-ready/13
  14. 14. RD + PD Team: 150 people, 9 function teams Multiple Test Environments 14
  15. 15. Team 1 15 Team 2 Team 3 Team 4 Team 5 SIT Staging Production
  16. 16. Team 1 16 Team 2 Team 3 Team 4 Team 5 SIT Staging Production Env 1 Env 2 Env 3 Env 4 Env 5
  17. 17. The Complexity of Deployment = Number of Environments x Number of Services x Number of Instances 17
  18. 18. Environment 1 Service CD Server 1. Build 2. Deploy 18
  19. 19. Environment 1 Service CD Server 1. Build 2. Deploy Environment 2 Service CD Server 1. Build 2. Deploy 19
  20. 20. Challenge #2 Multiple Tenancies 20
  21. 21. Different configurations between tenancies https://sodelicious.recipes/in-the-kitchen/christmas-food-around-the-world/ 21
  22. 22. The Complexity of Deployment = Number of Environments x Number of Services x Number of Instances x Number of Configurations22
  23. 23. If we have 10 environments, 40 services, 2 instances, 20 configuration… 10 environments x 40 services x 2 instances x 20 configuration = 16,000 configurations 23
  24. 24. Challenge #3 Changing Configurations in Realtime 24
  25. 25. Change configurations != Change systems http://ellenhorn.com/blog/the-ten-reasons-not-to-change/25
  26. 26. Missing configuration after deployment https://www.express.co.uk/life-style/health/431422/Busy-Then-forget-it-Hectic-lifestyle-affects-our-memory26
  27. 27. Challenge #4 Too Many Domain Names 27
  28. 28. 28 Instance #1 Instance #2 Instance #3 LBClient Server-Side Service Discovery https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/
  29. 29. Environment 1 A B C D E a.environment1.com b.environment1.com c.environment1.com d.environment1.com e.environment1.com Environment 2 A B C D E a.environment2.com b.environment2.com c.environment2.com d.environment2.com e.environment2.com29
  30. 30. 40 services = 40 domains 30
  31. 31. If we have 10 environments, 40 services… 10 environments x 40 services = 400 domain names 31
  32. 32. ● Deploying to multiple environments ● Multiple tenancies ● Changing configuration in realtime ● Too many domain names Recap: Why configuration management? 32
  33. 33. Agenda ● Why configuration management? ● How to manage configuration? ● How to go live? 33
  34. 34. Step #1 Make Service Artifacts Stateless 34
  35. 35. Environment 1 Service CD Server 1. Build 2. Deploy Environment 2 Service CD Server 1. Build 2. Deploy 35
  36. 36. Responsibility of CD Server ● Build Artifacts for a specific environment ● Deployment ○ Permission ○ Network Topology ○ Deployment Strategy 36
  37. 37. number of artifacts = number of environments = number of CD server 37
  38. 38. What is your performance? https://www.humanresourcesonline.net/how-to-effectively-set-and-communicate-performance-expectations/ 38
  39. 39. How does open source deal with it? 39
  40. 40. We should …. Single Codebase Multiple Deployment Deployment = Artifact + Configurations 40
  41. 41. Environment 1 AM CI Server CMService Environment 2 CI Server CMService 41
  42. 42. Environment 1 AM CI Server CMService Environment 2 CI Server CM 1. Build Config 1. Build Code Service 42
  43. 43. Environment 1 AM CI Server CMService Environment 2 CI Server 2. Get Artifact 2. Get Artifact CM 1. Build Config 1. Build Code Service 43
  44. 44. Environment 1 AM CI Server CMService Environment 2 CI Server 3. Get Config 2. Get Artifact 2. Get Artifact CM 1. Build Config 1. Build Code 3. Get ConfigService 44
  45. 45. Everyone could use artifacts to build services everywhere, by different configurations https://www.sc.qa/en/news/2022-fifa-world-cup-stadium-workers-given-5m-repayment45
  46. 46. Step 2. Configuration Hierarchy 46
  47. 47. Global Region Market Shop Default Settings Schema Comments Regional Settings ap-northeast-1 ap-southeast-1 tw hk my Shop Id: 9527 Shop Id: 9528 /{module}/global.json /{module}/{region}/global.json /{module}/{region}/{market}.json /{module}/{region}/{market}-{shop}.json Configuration Hierarchy (Source) 47
  48. 48. 48
  49. 49. 49
  50. 50. 50
  51. 51. 51
  52. 52. global region market shop 52
  53. 53. Global Region Market Shop Default Settings Schema Comments Regional Settings ap-northeast-1 ap-southeast-1 tw hk my Shop Id: 9527 Shop Id: 9528 Configuration Hierarchy (Build) /{module}/index.json /{module}/{market}.json /{module}/{market}-{shop}.json 53
  54. 54. 54
  55. 55. $ docker run -v ${PWD}:/build $DOCKER_IMG cfgctl build_local_file --config ./config --artifacts ./artifacts --module MsaDemo --datacenter dc-01 --tag 1.1.0 55
  56. 56. shop 8888 market 56
  57. 57. 57 $ docker run -v ${PWD}:/build $DOCKER_IMG cfgctl build_local_file --config ./config --artifacts ./artifacts --module MsaDemo --datacenter dc-01 --tag 1.1.0 --map true
  58. 58. 58
  59. 59. Benefits of Configuration Hierarchy ● Tenancies level configuration ● No copy paste (remove redundant configuration) 59
  60. 60. Step #3 Configuration Build Tools: Cfgctl 60
  61. 61. ● Command line interface for building configuration hierarchy ● Test for Configuration ● JSON -> Key-Value pairs ● Build and deploy configuration to 3 types of storages ○ Local files ○ Consul ○ S3 Cfgctl 61
  62. 62. Play with cfgctl: For developer in local environment Local Files Configuration Local Consul Cfgctl build for local files Cfgctl build for consul CodeSDK 62
  63. 63. Play with cfgctl: For OOO Environment S3 Configuration Consul Cfgctl build for S3 Cfgctl build for consul ServiceASDK ServiceBSDK 63
  64. 64. Step #4 Configuration SDK 64
  65. 65. Configuration SDK ● Languages Supported: C#, Node.js ● Features ○ Interface ○ Supported multiple configuration storages ○ Caching ○ Hot reload 65
  66. 66. Service SDK Consul S3 Local Files 66
  67. 67. Environment 1 AM CI Server CMService Environment 2 CI Server 3. Get Config 2. Get Artifact 2. Get Artifact CM 1. Build Config 1. Build Code 3. Get ConfigService 67
  68. 68. Environment 1 AM CI Server CMService Environment 2 CI Server 3. Get Config 2. Get Artifact 2. Get Artifact CM 1. Build Config By Cfgctl 1. Build Code 3. Get ConfigService S D K S D K 68
  69. 69. Step #5 Service Discovery 69
  70. 70. 70 Instance #1 Instance #2 Instance #3 LBClient Server-Side Service Discovery https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/
  71. 71. 71 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2
  72. 72. 72 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register)
  73. 73. 73 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat)
  74. 74. 74 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat) I want to call Service B
  75. 75. 75 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat) I want to call Service B Service B has a Instance #1
  76. 76. 76 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat) I want to call Service B Service B has a Instance #1 Call Service B Instance #1
  77. 77. 77 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat) I want to call Service B Service B has a Instance #1 Call Service B Instance #1 Hi, I am Service B (register)
  78. 78. 78 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 Hi, I am Service B (register) Hi, I still alive (send heartbeat) I want to call Service B Service B has a Instance #1 Call Service B Instance #1 Hi, I am Service B (register) Hi, I still alive (send heartbeat)
  79. 79. 79 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B
  80. 80. 80 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2
  81. 81. 81 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2 Call Service B Instance #2 (50%)
  82. 82. 82 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2 Call Service B Instance #2 (50%) I wanna to leave (deregister)
  83. 83. 83 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2 Call Service B Instance #2 (50%) I wanna to leave (deregister) I want to call Service B
  84. 84. 84 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2 Call Service B Instance #2 (50%) I wanna to leave (deregister) I want to call Service B Service B has a Instance #2
  85. 85. 85 Service A Service Registery Service B #1 Service B #2 Service A Service Registery Service B #1 Service B #2 I want to call Service B Service B has a Instance #1, #2 Call Service B Instance #2 (50%) I wanna to leave (deregister) I want to call Service B Service B has a Instance #2 Call Service B Instance #2
  86. 86. Environment 1 A B C D E cm.environment1.com cm.environment2.com CM Environment 2 A B C D ECM 86
  87. 87. Multiple tenancies infrastructure in client-side service discovery https://www.nginx.com/blog/service-discovery-in-a-microservices-architecture/ VIP VIP Normal 87
  88. 88. VIP Normal Cleint VIP.client.service.com Normal.client.service.com Multiple tenancies infrastructure in server-side service discovery 88
  89. 89. Step #6 Mininum Configuration (How to Discover Service Discovery?) 89
  90. 90. 90 https://www.slideshare.net/AmazonWebServices/leveraging-elastic-webscale-computing-with-aws
  91. 91. 91 Put bootstring configuration on AWS EC2 tags
  92. 92. Convert to Environment Variables It Could Cover Container, VM base Application 92
  93. 93. Environment 1 AM CI Server CMService Environment 2 CI Server CM 0. Build Config By Cfgctl 0. Build Code Service S D K S D K 93
  94. 94. Environment 1 AM CI Server CMService Environment 2 CI Server CM 0. Build Config By Cfgctl 0. Build Code Service S D K S D K 94 1. Get Metadata 1. Get Metadata Service Name, Version, AM, CM, etc.. Service Name, Version, AM, CM, etc..
  95. 95. Environment 1 AM CI Server CMService Environment 2 CI Server 2. Get Artifact 2. Get Artifact CM 0. Build Config By Cfgctl 0. Build Code Service S D K S D K 95 1. Get Metadata 1. Get Metadata Service Name, Version, AM, CM, etc.. Service Name, Version, AM, CM, etc..
  96. 96. Environment 1 AM CI Server CMService Environment 2 CI Server 3. Get Config 2. Get Artifact 2. Get Artifact CM 0. Build Config By Cfgctl 0. Build Code 3. Get ConfigService S D K S D K 96 1. Get Metadata 1. Get Metadata Service Name, Version, AM, CM, etc.. Service Name, Version, AM, CM, etc..
  97. 97. Service Provisioning ● Build code and configuration ● Get Metadata ● Get Artifacts ● Get Confugration ● Run Service 97
  98. 98. Step #7 Types of Configuration 98
  99. 99. 3 Levels of Configuration ● Realtime ○ Configuration SDK + Configuration Storages (Consul or S3) ● Service Discovery ○ Configuration SDK + Advance Configuration Services (Consul) ● Provision ○ Configuration Storages (Consul or S3) 99
  100. 100. Agenda ● Why configuration management? ● How to manage configuration? ● How to go live? 100
  101. 101. Rule #1. Consider what you can contribute to teams and minimize the impact on them 101
  102. 102. 102 https://www.fabiaoqing.com/search/search/keyword/我委屈但是我不哭
  103. 103. ● Arch Team ○ Design ○ Configuration SDK ■ Easy to change configuration storages ■ Cache ■ Hotreload ■ Service Registration Consider what you can contribute to teams and minimize the impact on them 103
  104. 104. Consider what you can contribute to teams and minimize the impact on them 104 ● Infra Team & Common Service Team ○ PoC / Provision / Maintain Configuration Storage ○ Consider network topology / security ○ Build tools: Cfgctl ■ Resolve level configuration issue ○ Use it in our own services first
  105. 105. Rule #2. Prove it works 105
  106. 106. 106
  107. 107. Rule #3. Use it in small project or your services 107
  108. 108. Rule #4. Prove it is stable 108
  109. 109. Service SDK Consul S3 Local Files 109
  110. 110. 110 Performance across the S3 Storage Classes https://aws.amazon.com/s3/storage-classes/?nc1=h_ls
  111. 111. Server (Follower) Server (Leader) Server (Follower) GOSSIP Replication Leader Forwarding Replication Client Client Client RPC Leader Forwarding GOSSIP RPCGOSSIP Consul Cluster https://learn.hashicorp.com/consul/datacenter-deploy/deployment-guide111
  112. 112. Server (Follower) Server (Leader) Server (Follower) GOSSIP Replication Leader Forwarding Replication Client Client Client RPC Leader Forwarding GOSSIP RPCGOSSIP Consul Cluster Availability 112
  113. 113. Server (Follower) Server (Follower) GOSSIP Leader Forwarding Replication Client Client Client RPC GOSSIP RPCGOSSIP Consul Cluster Availability, leader election 113
  114. 114. Consul Cluster Availability: Multiple AZ deployment https://aws.amazon.com/quickstart/architecture/consul/114
  115. 115. Performance Tuning https://www.consul.io/docs/install/performance.html115
  116. 116. 116
  117. 117. 117 Design for Operation http://www.fgsbmc.org.tw/news_latestnews_c.aspx?News_Id=201801084
  118. 118. Questions
  119. 119. 119
  120. 120. Thanks

×