Challenges of Kubernetes On-premise Deployment

1. Challenges of Kubernetes On-
premise Deployment
Sa Pham - RnD Department BizFly Cloud

2. Challenges
● Expose service type Load Balancer
● Persistent Volume
● Deployment tools

3. Common Apps deployment patterns on
Kubernetes
With stateless application, we use kind
Deployment to create multiple pods
and Service to expose the application
We have 3 types of Service
● NodePort
● ClusterIP
● Load Balancer

4. Service type Load Balancer?
On Cloud Environment, With Load
Balancer Service type, Kubernetes
will create a new Load Balancer
for that service.
For example: Open port 443 on
Load Balancer and forward to port
30000 on worker nodes. Kube-
proxy will forward request from
port 30000 to correct pod.

5. How Load balancers are provisioned?
Before, cloud providers have
to contribute to kubernetes
source code to work with their
cloud -> In-tree Providers

6. How Load balancers are provisioned?
Later, Cloud Controller Manager
(CCM) is decoupled from
kubernetes source code.
When service Load Balancer is
created, CCM call to Cloud to
create Load Balancer and Pool,
Listener of Load Balancer.

7. How Load balancers are provisioned?
On bare-metal environment ?

8. Load Balancer on Bare Metal Environment
Fortunately, most of our applications using HTTP to communicate from
outside. We use Nginx-Ingress for Layer 7 Load Balancer and Haproxy For
Layer 4 Load Balancer.

9. Our architecture
- Use host-based routing and
path based routing at ingress

10. Common Apps deployment patterns on
Kubernetes
Some applications (database,
cache, ...) require Persistent
Volume, then we use kind
StatefulSet for these
applications.
Requirements: Persistent
Volume with Storage Backend
(Storage class)

11. How Persistent Volumes are provisioned?
On Cloud Environment, Cloud providers have
Block Storage Service and File System
Service. We can use them to provide
Persistent Volume for Kubernetes
Container Storage Interface (CSI) was born.

12. How Persistent Volumes are provisioned?What is Container Storage Interface?

13. How Persistent Volumes are provisioned?
CSI on Cloud Environment and Bare metal Environment

14. How Persistent Volumes are provisioned?
CSI Driver List: Drivers - Kubernetes CSI Developer
Documentation

15. CSI Drivers
What will we do if there is no existing CSI driver for our Storage?
- Use Ember CSI - embercsi/ember-csi: Multi-vendor CSI plugin supporting over 80 storage drivers
- Write a new one. - How to write a Container Storage Interface (CSI) plugin · Fatih Arslan
Ember CSI using Cinder drivers to work with storage backend
A lot of storage backends are supported by Cinder community

16. CSI Drivers
We are using Ember CSI as
CSI Driver for our SAN
Storage.
Cons:
- Multiple pools, multiple
deployments

17. Storage backend for Kubernetes on-premise
Some applications with old
architectures,
- Store data on local disk
- Use NFS for shared storage
How did we run on kubernetes ?

18. Storage backend for Kubernetes on-premise
Cephfs performance on our cluster
We used tocdo.net script to test on a kubernetes pod.

19. Deployment tools and manage kubernetes
With managed kubernetes service (such as GKE, AKS, EKS, ..), we don't have to
manage kubernetes master node.
And easy to add a new node.

20. On Premise
We need a tool which can
- quick deploy a new cluster
- easy to control the cluster
- easy to add a new node to cluster
- Infrastructure as code (IaC)
- Separate etcd cluster

21. Deployment tools and manage kubernetes
Too many deployment tools for on-premise kubernetes.
- Kubeadm
- RKE
- Kubespray
- Kubernetes the hard way
- ...

22. RKE
Cluster.yml example
Then, rke up