2024: Domino Containers - The Next Step. News from the Domino Container commu...
Automate CI/CD with Rancher
1. CI/CD with Rancher
How we automate container orchestration in our
environments at Kloeckner-i using Gitlab, Docker and
Rancher
Original google slides version: https://goo.gl/YiMQkJ
4. Topics
I. Introduction
II. Container Orchestration
III. CI/CD with Rancher
IV. Setup storage management in
Rancher
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
source:medium.com
5. Kloeckner-i
● Kloeckner-i, a corporate-startup
hybrid based in Berlin
● We’re building the next
generation of tools for the steel
industry
● To spend as much time as
possible developing, we need to
automate everything else
source:pdssheetmetal.co.uk
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
6. Automate all the things!
What we want to automate:
● Checking our code style
● Testing our code
● Creating QA environments and
deploying our code to them
● Running integration tests
● Deploying our code to production
systems
● Scaling our systems if we need
to
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
7. What we’re building - Microservices
Platform
Frontend
frontend.DOMAIN
...
API Proxy
api.DOMAIN
Backend Magic
(Services)
ERP (SAP)
IDP
IdentityProvider
idp.DOMAIN
Account Producer
Contract Producer
Order Producer
ConnectorAPI
Central processing system
(Multiple queues)
Consumer
Consumer
Web
frontend.DOMAIN
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
8. Container Orchestration
● Tools that organizations use to manage
containers at scale (and in production)
● May include features such as container:
○ Upgrades
○ Scaling
○ Multi Host Networking
○ Storage management
○ Health checks
○ ...
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
source:quora.com
9. So why Rancher?
Candidates
● Amazon ECS
● Google Container Engine
● Azure Container Service
● Apache Mesos
● Kubernetes
● Docker Swarm
● CF Diego
● CoreOS Fleet
● Puppet, Bash, ...
Reasons I chose Rancher
● Wanted no vendor lock-in (self hostable
and open-source)
● Something developers knowing
docker-compose understand
● Docker Swarm (1.12) not ready
● Mesos and Kubernetes more complicated
● Github: 4K, Issues: open 1K, closed 4K
● Powerful GUI
● Just feels right
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
10. CI/CD with Gitlab, Docker, Rancher
Current stack
● Hosting: Google Compute Engine
● Automation: Gitlab
○ Run tests: docker runner
○ Host docker images: docker registry
● Orchestration: Rancher
source:lakehub.co.ke
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
11. Docker development workflow
1. Build Dockerfile
2. Run container
3. Build docker-compose.yml
4. Run container with dependencies
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
12. Gitlab Dockerception
1. Build .gitlab-ci.yml
a. Lint
b. Test
c. Build docker image
2. Push to repository
3. Gitlab runs your jobs
4. Build docker containers within
docker containers with docker
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
18. SHIA - Just do it!
● Connecting Gitlab and Rancher
● Rubygem SHIA
● Deploy stack to environment
● Deploy full stack to a “QA” environment
● Don’t let your dreams be dreams …
●
SHIA stands for Seamless Hive Integration Agent (it’s really hard to find buzzwords for that acronym )
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
19. SHIA - Deploying Services
● Create Environment
● Add hosts on GCE via docker-machine
● Add users to environment
● Process stack config
● Create description (branch:commit-hash)
● Create or update stack
● One command: shia -e linuxcon deploy
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
20. SHIA - Deploying QA Environments
● Convention “@env_xx” in git branch name
● Deploys stack that is to be tested with current branch
● Deploys the rest of the stack with master branch
● Multiple stacks can be deployed to the same environment with different
branches
● One command: shia -b ${CI_BUILD_REF_NAME} deploy_all
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
21. SHIA - Deploying Services - Demo
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
22. SHIA - Deploying QA Environments - Demo
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
23. SHIA - What we’re working on
● Automate GCE networking configuration
○ Create GCE LB
○ Configure DNS (currently using myra cloud)
● Seed data efficiently in QA environments
● Run full stack integration tests in QA environments
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
25. Storage options with Docker
● Default : No Data Persistence
● Data Volumes : Container Persistence
● Data Only Container : Container
Persistence
● Host Mapped Volume : Container
Persistence
● Host Mapped Volume, backed by Shared
Storage : Host Persistence
● Convoy Volume Plugin : Host Persistence
(of Storage Host)
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
26. Convoy-NFS features
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher
● Docker Volume Driver Plugin
● Backend for different storage options
○ Device Mapper
○ NFS/EFS/VFS
● Snapshots, Backups, Restore
● Single Go binary
● Can be deployed independently of Rancher
29. Where to get SHIA
Kloeckner-i Github
https://github.com/kloeckner-i/shia
Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher / Fin
