DevOpsDays Taipei 2017 從打鐵到雲端
•
3 gostaram•295 visualizações
從打鐵到雲端 - 架構適應與變遷 歷經了 2 年創業時間、3 次主機遷移、4 代架構改造，技術變革與容器編排工具的快速演進，影響了每一次架構改進的切入角度。 從 Bare metal 到 Public Cloud；從單機 Docker、Docker Swarm 到 Kubernetes。每一代複雜的業務需求考量、解決方案的選用與抉擇、資源不全的過渡時期，都是滿滿的辛酸血淚。 而當漸漸面對全球流量時，隻身一人的後台，又該如何以小搏大來面對？讓我們看下去。
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Semelhante a DevOpsDays Taipei 2017 從打鐵到雲端
Semelhante a DevOpsDays Taipei 2017 從打鐵到雲端 (20)
Último
Último (20)
DevOpsDays Taipei 2017 從打鐵到雲端
- 2. Co-founder from kidscrape A backend engineer 陳宏彥 (Zz Chen) chy168@gmail.com Hello! 2
- 3. Designed by freestockcenter / Freepik ● Point of view as a Startup ● Hotest Technology VS. Your Business ● Cost ● Mistakes 3
- 5. speed (PM) cost (BOSS) solutions (RD) Created by Asierromero - Freepik.com Created by Luis_molinero - Freepik.comCreated by Luis_molinero - Freepik.com 5
- 6. A journey 6
- 7. 孟母三遷 Created by Vectorarte - Freepik.com 7
- 10. Touch Lock Prevent user touching screen accidently 10
- 11. parse.com 11 Choose app’s backend Build apps fast, without managing infrastructure
- 12. 12
- 13. Problems ● App update adoption rate ● Traffic ○ 60 request / sec ○ ~ 10K NTD/month ● Unbalanced peak time 13
- 14. 14
- 15. 15
- 16. Build a platform by ourself 16 what do we need?
- 17. ● Features ○ DAU (Daily active users) ○ Retention ○ New installation ● Platforms ○ Google Play ○ Google Analytics ○ Parse.com App Statistics 17 Created by Balintseby - Freepik.com
- 18. Challenges ● Data collection and calculation ● Availability to Global ● Cost ● Retire parse.com 18
- 19. Self Hosting Bare Metal in IDC 19
- 20. Created by Jcomp - Freepik.com Friend’s support ● A dedicated Server ● Unlimited Bandwidth ● 72GB Memory ● Public IPs 20
- 21. 21
- 22. Collect Data 22
- 24. 24
- 25. 25
- 27. 27 a fast and general engine for large-scale data processing tool
- 29. thakho app statistic platform by ourself 29
- 30. thakho ● Build our own SDK ● App statistic ● Reports 內部專案，沒有logo，省下設計的時間 .. (被揍) 30
- 31. Deployment 31
- 32. 1 32
- 34. Dockerize Applications Server ● Spark ● ElasticSearch ● Logstash ● Kibana ● Go Application ● nginx 34 Docker NGINX Elasticsearch Spark Cluster API (Golang) kibana 34 logstash
- 35. Dockerize Spark Cluster ● Spark-1.4~1.6 ● Necessary ports expose ● Set environment variables ○ $SPARK_PUBLIC_DNS ○ $SPARK_{MASTER|LOCAL}_IP ● Set spark.broadcast.factory ● Handle docker aliases ● 3rd-party librarys 35
- 36. spark-defaults.conf spark-env.conf Reference: Running Spark standalone cluster in Docker 36
- 37. Problems ● HA / SPOF problem ● Cluster so-called cluster ○ Disk I/O pressure 37 真 的 要 爆 炸 了！！
- 40. Bare Metal in IDC Self Host Multiple Servers 2 ..... 40 Multiple =
- 44. 時空背景 44 Very Early Stage of Docker Swarm
- 45. ● docker 1.9 ○ Support Network plugin ● docker swarm 1.1.2 ○ Run as container ○ Just announce production ready ● docker-compose 1.5.1 ● leak documents Early Stage 45
- 46. consul, docker engine, swarm master / node Components to Install 46
- 47. 47 Docker Daemon Swarm Node Docker Daemon Swarm Node Docker Daemon Swarm Node Docker Daemon Swarm Master Swarm Node Docker API consul
- 48. 48 刻骨銘心的筆記
- 49. Reference: Tips for using Docker Swarm mode in production OK, infras done 49 Created by Kjpargeter - Freepik.com
- 50. 50
- 51. Challenges ● Load balancer ● Service discover ● Overlay Network ● Build & Deployment 51 IDC Server 1 Server 2 DockerDocker Overlay Network NGINX ES-02 ES-03ES-01 C*-01 C*-02C*-03 Spark/M Spark/WSpark/W API API NGINX logstash Rediskibana
- 52. Service Discovery & Load Balancer 52 Reference: 关于负载均衡和服务发现 https://goo.gl/2DYoZs
- 53. IDC Server 1 Server 2 DockerDocker Overlay Network NGINX ES-02 ES-03ES-01 C*-01 C*-02C*-03 Spark/M Spark/WSpark/W API API API NGINX logstash Rediskibana VPN 53 Service Discovery
- 54. ● IP + Port = Service ● Container link (deprecated) ● gliderlabs/registrator ○ Registers and deregisters automatically ○ Communicate with Docker engine ○ Works with consul/etcd Service Discovery in Swarm 54 registrator: image: gliderlabs/registrator volumes: - /var/run/docker.sock:/tmp/docker.sock networks: - YEAH command: "-cleanup -resync 30 consul://consul:8500" depends_on: - consul
- 55. Load balancer in Swarm ● fabiolb/fabio ○ Zero-conf ○ Work with consul ■ Already setup consul ○ HTTPS off-load ○ Dynamic loading without restart ○ Metrics Support 55 LBS Svc2 Svc3 Svc1 Svc4
- 56. 56
- 57. Designed by evening_tao / Freepik ... XD Time’s ticking Switch back to NGINX & Container link 57
- 59. Build & Deployment ● docker-compose.yml ● `docker-compose build` ● `docker-compose push` ● `docker-compose up` ○ Stateful service ■ es-0, es-1, spark[worker], cassandra ... etc ○ Node constraint 59 No pipeline No multiple projects No complexed build
- 60. Challenges with Bare Metal Servers 60
- 61. Chanllenges in IDC ● No Firewall ● No LoadBalancer ● Only Host disk ○ Hard to scale-out & distribute loading ● No Object storage ○ Backup files ○ Archive logs 61
- 62. IDC Server 1 Server 2 DockerDocker Overlay Network NGINX ES-02 ES-03ES-01 C*-01 C*-02C*-03 Spark/M Spark/WSpark/W API API API NGINX logstash Rediskibana Disk Disk hostPath: /data/es-01 hostPath: /data/csx-01 NFS VPN AdHoc 62
- 63. Not perfect, but it works. Designed by Molostock / Freepik 63
- 65. Startup Incubation Program ● 10K USD per month ● 12 months Designed by kues1 / Freepik A Cloud Provider 65
- 66. IaaS from Cloud Provider ● Virtual / Bare metal server ● Order by hourly, monthly ● Basic Network 66
- 67. Rebuild infrastructure Bare Metal in IaaS 67
- 68. Refactor ● Load balancer ● Deployment (why??) 68
- 69. ● docker (1.12) ● docker swarm ● docker compose (1.8) ○ bundle (experimental) ■ https://docs.docker.com/compose/bundles/ ○ stack (experimental) ■ https://docs.docker.com/engine/swarm/stack-deploy/ 時空背景 69
- 70. Designed by Kues / Freepik 70
- 71. Designed by tirachard / Freepik 71 Kubenetes
- 72. kubernetes ● Application patterns ● Controller to manage application pattern ● Service discovery ● Scalability ● ...etc 72
- 73. Swarm To Kubernetes ● Remove container link ● Remove consul 73
- 74. 詳情請見「小城故事不多 -科技部」部落格： Kubernetes 純手作部署在 Ubuntu 16.04 74
- 75. Migrate kubernetes in 2 wks 因為「一卡皮箱即可入住」的設計 75
- 76. ● Move data ● End-user ○ via DNS load balancing ○ Split n% traffic to new data center Switch to New DC 76
- 77. Designed by Asierromero / Freepik 77
- 78. nginx, I am back! 還好，東西都還在皮箱裏準備好好的 78
- 79. 79
- 80. ● Monitor ○ Prometheus ○ End-to-end monitor ○ User as Monitor (誤) ■ Fabric ■ Firebase Problems 80
- 81. Problems - Storage Server ● Heavy I/O ● Scalability in kubernetes ○ stateful service ○ pod arrangement Docker NGINX ES-02ES-01 C*-01C*-03 Spark/W API API kibana Disk 81
- 82. Summary of IaaS / Bare Metal ● Assemble your own components ● High cost (budget & communication) ● Storage still a big problem 82
- 83. After 12 months Your program will be ended on ... 83
- 85. Designed by skadyfernix / Freepik 這次，可以好好的上雲端了吧！ 85
- 86. 86
- 87. KubenetesGoogle Cloud Platform Google Container Engine 87 Just moving data, update DNS, migration are DONE
- 88. Refactor ● Data Analysis ○ Retire our Spark ■ Dataproc ■ BigQuery ● Integrated Network ○ Load Balancer ○ Ingress API Container Engine Kubernetes Services Cloud Load Balancing Multiple Clients API Requests 88
- 89. ● Storage ○ Persistent disk Refactor (cont.) 89 Server Docker NGINX ES-02ES-01 C*-01C*-03 Spark/W API API kibana Disk Server Docker NGINX ES-02ES-01 C*-01C*-03 Spark/W API API kibana PD PD PD PD
- 90. 90
- 91. Conclusions ● Lower Cost (~10X) ● Lower Operation ● Speed-up Business ● But More Dependency Created by Bamdewanto - Freepik.com 91
- 92. Summary 92 ● Compose components from scratch ● Container orchestration tool ● Keep refactoring ● Cost optimization ● Mistakes
- 93. 93 Bare Metal x1 Bare Metal x2 IaaS Public Cloud Server Specs 難懂 難懂 難懂 秒懂 Load Balancer 自建 自建 混搭 附贈 Storage 自建 自建 自建 附贈 CI / CD `docker build` Packer Jenkins + Packer Jenkins + Packer Orchestration tool Docker Docker Swarm kubernetes kubernetes Cost (Budget) 中 中 高 低 Cost (Human) 高 高 高 中 Dependency 低 低 中 高 水深程度 高 高 中 中 * 後端設計一定有風險，技術投資有賺有賠，跳水前應一起來聊聊。 解決方案比較與心得
- 94. THANK YOU! 94