Have you ever thought about migrating your Kubernetes clusters to Google Cloud to get your services closer to your customers? Yes? We too! Join us on an interactive journey to discover the main challenges of live migration at scale of etcd's, traffic routing and application workloads from your on-premise platform to GCP. The talk will discuss the current state of the technical concept, known problems and insides of the already proven migration steps for stateless workload. As part of the journey, we'll see the differences between migrating one or one hundred clusters with productive workloads; What parts can be automated? What steps may need to be manual? Let's see how an automated solution could look like in the future and what steps are missing.

1. A Future Journey:
How to Migrate 100
Clusters to GCP without
Downtime?

2. Tobias Schneck
Head of Professional Service
toschneck
@toschneck
tobi@kubermatic.com
Manuel Stößel
Systems Architect / Tech Lead
@ManuStoessel
@Manuel_Stoessel
manuel@kubermatic.com
What Else?
• Part of Professional Services @
Kubermatic
• Supporting customers on their
cloud-native journey
• Geeking out over Kubernetes and
adjacent technologies

3. Why Migrate Clusters?

4. Reasons for Cluster Migration Scenarios
● Business Reasons
● Better contract/conditions at another cloud provider ⇒ cost saving
● Data center migration to/from (cloud) providers
● Multi cloud strategy ⇒ decrease dependency to existing provider
● Technical Reasons
● Location migration of data centers
● Migrate to other network segments
● Adaption of on-prem / cloud improvements at new data center provider
● Data location of cloud offered service e.g. machine learning data

5. What Are the Main
Challenges?

6. Kubernetes Abstracts Infrastructure, But:
● Consummation of infrastructure resources
○ (Virtual) Machines
○ Network:
■ Network IP Address Spaces
■ Routing, Firewall
■ Ingress / Egress Traffic
○ DNS
○ External Storage Systems
● Cloud dependent Kubernetes components
○ Cloud Controller Manager
■ Node controller - responsible for updating kubernetes nodes
■ Service controller - responsible for services of type LoadBalancer
■ Route controller - responsible for setting up network routes
○ Storage Classes
○ (sometimes) Overlay Networking

7. K8s Master
API Server
Dashboard
Scheduler
kubelet kubelet kubelet kubelet
Container
Registry
etcd
Controllers
web browsers
kubectl
web browsers
Config
file
Image
CNI - Plugin Network (e.g. Flannel, Calico)
Developer
Developer

8. ⇒ Application workload has the highest priority!
● Ensure fundamental networking rules at any time
○ All containers within a pod can communicate (L4) with each other unimpeded.
○ All pods can communicate with all other pods without NAT.
○ All nodes can communicate with all pods (and vice-versa) without NAT.
○ The IP that a pod sees itself as is the same IP that others see it as.
● External dependencies need to be reachable
○ External routed IPs for Load Balancers / Node Port Service
○ DNS Names need to be reachable
● Storage
○ State needs to migrated without data loss
Migration Without Downtime

9. Scale Level of 100 Clusters
● Larger organizations running a lot of clusters
⇒ different locations, org units, time zones
● Cluster users are only consumers
⇒ following the cluster as a service approach
● Cluster connection and secrets needs to be stable
=> no change of interface

10. Solution Approach

11. Status Quo
● Multi Cloud Setup with Kubermatic Kubernetes Platform (KKP)
○ Seed cluster hold containerized control plane of user clusters
○ Worker nodes provisioned by Cluster API conform Kubermatic machine-controller
○ Canal as default overlay network
● Target
○ Migrate user and seed cluster control planes and worker to different cloud
○ Keep external Cluster Endpoints stable
■ Control Plan: Kubernetes API Server endpoints
■ Application: DNS, Ingress
○ Out-of-Scope (for now): Storage replication
■ Assumption: Application Layer manages storage replication, e.g. etcd

12. Kubermatic
Kubermatic Master Cluster
KubeOne Seed Cluster - Region EU
User k8s
Worker
User k8s
Worker
User k8s Master
API Controller
Scheduler etcd
User k8s
Worker
User k8s
Worker
User k8s Master
API Controller
Scheduler etcd

13. Recommended Prerequisites
● Announce maintenance window and block cluster updates
● Ensure backups and recovery procedure for
○ Seed and user clusters
○ Application workload
● Create target cloud cluster as reference
● Ensure control of DNS entries

14. Migrate User Clusters
Solution Approach

15. 1) Migrate User Cluster Workers
● Create new worker nodes in target cloud
⇒ Machine controller with new Machine Deployment at target cloud
● User worker nodes and Pods need to talk to each other at any time
⇒ Strap a VPN overlay by DaemonSets across current and target cloud
⇒ Route overlay CNI traffic through VPN network
● Ensure reachability
=> Keep old and create new cluster Ingress endpoints
=> Transfer workload to new cloud
=> Delete after workload / connectivity is ensured

16. KubeOne Seed Cluster - Region EU
User k8s Master
API Controller
Scheduler etcd
User k8s
Worker
VPN Server Machine
Controller
Application
User k8s
Worker
Application
*.cluster-1.example.com
K8s API Server tunnel
Canal Overlay
(eth0)
MetalLB
Migrate User Cluster Worker Nodes:

17. KubeOne Seed Cluster - Region EU
User k8s Master
API Controller
Scheduler etcd
User k8s
Worker
VPN Server Machine
Controller
User k8s
Worker
Application Application
*.cluster-1.example.com
K8s API Server tunnel
Canal Overlay
(kube)
Migrate User Cluster Worker Nodes:
1. VPN Daemon Set with client-to-client
communication
2. Route Overlay Traffic over VPN interface
3. Pause existing Cluster & Machine Deployment
VPN
Client
VPN
Client
MetalLB

18. KubeOne Seed Cluster - Region EU
User k8s Master
API Controller
Scheduler etcd
User k8s
Worker
Machine
Controller
VPN Server
User k8s
Worker
Application Application
*.cluster-1.example.com
K8s API Server tunnel
Canal Overlay
(kube)
Migrate User Cluster Worker Nodes:
1. VPN Daemon Set with client-to-client
communication
2. Route Overlay Traffic over VPN interface
3. Pause existing Cluster & Machine Deployment
4. Update Cluster Spec & Cloud Credentials
5. Unpause Cluster with new Cloud Provider
6. Apply new Machine Deployment
VPN
Client
VPN
Client
User k8s
Worker
VPN
Client
MetalLB
User k8s
Worker
VPN
Client
GCP LB

19. KubeOne Seed Cluster - Region EU
User k8s Master
API Controller
Scheduler etcd
User k8s
Worker
Machine
Controller
User k8s
Worker
*.cluster-1.example.com
K8s API Server tunnel
Canal Overlay
(kube)
Migrate User Cluster Worker Nodes:
1. VPN Daemon Set with client-to-client
communication
2. Route Overlay Traffic over VPN interface
3. Pause existing Cluster & Machine Deployment
4. Update Cluster Spec & Cloud Credentials
5. Unpause Cluster with new Cloud Provider
6. Apply new Machine Deployment
7. Test new cluster ingress entrypoint
8. Migrate Workload and update DNS
VPN
Client
VPN
Client
User k8s
Worker
VPN
Client
MetalLB GCP LB
User k8s
Worker
VPN
Client
Application Application

20. KubeOne Seed Cluster - Region EU
User k8s Master
API Controller
Scheduler etcd
VPN Server Machine
Controller
*.cluster-1.example.com
K8s API Server tunnel
Migrate User Cluster Worker Nodes:
1. VPN Daemon Set with client-to-client
communication
2. Route Overlay Traffic over VPN interface
3. Pause existing Cluster & Machine Deployment
4. Update Cluster Spec & Cloud Credentials
5. Unpause Cluster with new Cloud Provider
6. Apply new Machine Deployment
7. Test new cluster ingress entrypoint
8. Migrate Workload and update DNS
9. Cleanup old cloud resource
User k8s
Worker
GCP LB
User k8s
Worker
Application Application
Canal Overlay
(eth0)

21. github.com/kubermatic-labs/cluster-migration
DEMO Time 🎉

22. Migrate Seed Cluster
Solution Approach

23. Kubermatic
KubeOne Master Cluster
migrated
KubeOne Seed Cluster - Region EU
User k8s
Worker
User k8s
Worker
User k8s Master
API Controller
Scheduler etcd
migrated
User k8s
Worker
User k8s
Worker
User k8s Master
API Controller
Scheduler etcd

24. 2) Migrate Seed Cluster
● Create new seed master nodes at new cloud
=> New Kubernetes API Load Balancer
=> API Endpoint needs to be updated by DNS
=> Block seed cluster upgrades to ensure worst case recovery
● Migrate user cluster control plane
=> Handle migration the same way (like user cluster workload)
=> Ensure etcd quorum and migration by data replication
=> Block user cluster upgrades to ensure worst case recovery

25. User k8s Master
API Scheduler Controller etcd
KubeOne Seed Cluster - Region EU
Seed k8s
Master
Seed k8s
Master
Seed k8s
Master
seed-k8s-api.example.com
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Migrate Seed Master Nodes:
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

26. KubeOne Seed Cluster - Region EU
Seed k8s
Master
Seed k8s
Master
Seed k8s
Master
seed-k8s-api.example.com
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Migrate Seed Master Nodes:
1. Setup VPN Overlay
2. Pause existing Cluster & Machine
Deployment
3. Create and join new 2 Master Nodes
Seed k8s
Master
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

27. KubeOne Seed Cluster - Region EU
seed-k8s-api.example.com
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Migrate Seed Master Nodes:
1. Setup VPN Overlay
2. Pause existing Cluster & Machine
Deployment
3. Create and join new 2 Master Nodes
4. Add new LB Service & Update DNS
5. Remove 2 old Master Nodes and move
etcd quorum to new cloud
Seed k8s
Master
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Seed k8s
Master
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

28. KubeOne Seed Cluster - Region EU
Seed k8s
Master
seed-k8s-api.example.com
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Migrate Seed Master Nodes:
1. Setup VPN Overlay
2. Pause existing Cluster & Machine
Deployment
3. Create and join new 2 Master Nodes
4. Add new LB Service & Update DNS
5. Remove 2 old Master Nodes and move
etcd quorum to new cloud
6. Create 3rd Master Node at new cloud
and remove last old Master Node
Seed k8s
Master
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

29. Migrate Seed Cluster Workers
Solution Approach

30. KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
VPN Server
Canal Overlay
(kube)
Migrate Seed Worker Nodes:
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

31. KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

32. KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
3. Taint existing workers as non-schedule
4. Scale up etcd count of user cluster to 5
⇒ data replicated by etcd
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

33. Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
3. Taint existing workers as non-schedule
4. Scale up etcd count of user cluster to 5
⇒ data replicated by etcd
5. Create new LB for NodePort Proxy and update DNS
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

34. Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
3. Taint existing workers as non-schedule
4. Scale up etcd count of user cluster to 5
⇒ data replicated by etcd
5. Create new LB for NodePort Proxy and update DNS
6. Add 1 new worker and drain 1 old workers
⇒ etcd quorum migrated to new cloud
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Worker
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

35. Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
3. Taint existing workers as non-schedule
4. Scale up etcd count of user cluster to 5
⇒ data replicated by etcd
5. Create new LB for NodePort Proxy and update DNS
6. Add 1 new worker and drain 1 old workers
⇒ etcd quorum migrated to new cloud
7. Drain missing worker nodes, cleanup old cloud
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Master
VPN Server
Canal Overlay
(kube)
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

36. Migrate Seed Worker Nodes:
1. VPN Overlay, Pause existing Cluster, Machine
Deployment
2. Create 2 new Workers (migration steps similar to
user cluster)
3. Taint existing workers as non-schedule
4. Scale up etcd count of user cluster to 5
⇒ data replicated by etcd
5. Create new LB for NodePort Proxy and update DNS
6. Add 1 new worker and drain 1 old workers
⇒ etcd quorum migrated to new cloud
7. Drain missing worker nodes, cleanup old cloud
8. Scale down etcd count of user cluster to 3
9. Remove VPN Overlay
KubeOne Seed Cluster - Region EU
K8s API Server tunnels
Seed k8s
Master
Seed k8s
Master
*.seed.example.com
User k8s
Worker
User k8s
Worker
User k8s
Worker
[cluster-id]
NodePort Proxy
Service
Seed k8s
Worker
Seed k8s
Master
Canal Overlay
(eth0)
Seed k8s
Worker
Seed k8s
Worker
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd
User k8s Master
API Scheduler Controller etcd

37. Outlook
● Automate clean up procedure
○ Idea: switch back cloud provider / machine controller for clean up
● Manage migration by Operator
○ Health checks
○ Wait conditions for migration steps
● Stabilize VPN connection
○ Multiple VPN servers
○ Soft switchover between VPN / Host network overlay
○ Evaluate Wireguard usage
● Automate Load Balancer and DNS management

38. Any Questions?

39. Join Us For Our Next Webinar!
February 23 at 8 AM CET / 6 PM AEST
Empower Your DevOps
Organization With Kubernetes

40. We’re Hiring!
Check Out Our Job Openings:
https://careers.smartrecruiters.com/KubermaticGmbH
kubermatic/kubermatic
@kubermatic
marketing@kubermatic.com