1.
Multi-Tenancy
on
Jirayut Nimsaeng (Dear)
CEO & Founder, Opsta (Thailand) Co.,Ltd.
TechTalkThai Conference: Kubernetes Trends
September 16, 2021 https://bit.ly/opsta-ttt-k8s-tenancy

2.
Multi-Tenancy on Kubernetes
#whoami
Jirayut Nimsaeng (Dear)
Jirayut has been involved in DevSecOps, Container, Cloud Technology
and Open Source for over 10 years. He has experienced and
succeeded in transforming several companies to deliver greater
values and be more agile.
● He is Founder and CEO of Opsta (Thailand) Co.,Ltd.
● He is Cloud/DevSecOps Transformation Consultant and
Solution Architecture
● He is the first Certified Kubernetes Security Specialist (CKS)
and Certified Kubernetes Administrator (CKA) in Thailand

3.
Multi-Tenancy on Kubernetes
Agenda
● What is Multi-Tenancy?
● Multi-Tenancy Primitive
● Access Control
● Fair Sharing
● Isolation
● Future

4.
Multi-Tenancy on Kubernetes
What is Multi-Tenancy?

5.
Multi-Tenancy on Kubernetes
Single vs Multi-Tenant
https://dev.to/sciencebae/multi-tenant-architecture-and-it-s-issues-h06

6.
Multi-Tenancy on Kubernetes
Multi-Tenancy on Kubernetes
https://cloud.google.com/kubernetes-engine/docs/concepts/multitenancy-overview

7.
Multi-Tenancy on Kubernetes
Why Kubernetes Multi-Tenancy?
● Reduced management overhead
● Reduced resource fragmentation
● Cost efficiency

8.
Multi-Tenancy on Kubernetes
Kubernetes Multi-Tenancy Types
Soft Multi-tenancy
● trust tenants
● may has relation between
tenants
● we believe that they are not
trying to harm other tenants
● focus on preventing accidents
Hard Multi-tenancy
● zero trust tenants
● each tenant has no relation to
each other
● we believe that they are trying
to exploit the system
● focus on securing and
isolating each tenant

9.
Multi-Tenancy on Kubernetes
Kubernetes Multi-Tenancy Models
● Namespaces as a Service
allows sharing clusters and hence enables resource efficiencies with
cluster-wide resources limitation
● Clusters as a Service
better isolation with higher management and resource overhead.
● Control Planes as a Service
virtual cluster where each tenant gets their own dedicated Kubernetes
control plane but share worker node resources

10.
Multi-Tenancy on Kubernetes
Kubernetes
Multi-Tenancy Primitive

11.
Multi-Tenancy on Kubernetes
Multi-Tenancy on Kubernetes Overview
https://www.vamsitalkstech.com/architecture/a-deepdive-into-kubernetes-multitenancy-1-2/

12.
Multi-Tenancy on Kubernetes
Kubernetes Multi-Tenancy Primitive
● Access Control
Use policies to ensure that
tenants can access only what
they should have access to
○ RBAC
● Fair Sharing
Enforce limits per tenant
○ Resource Quota
○ Pod Priority
○ Quality of Service
○ Taints & Tolerations
○ Pod Affinity / Anti-affinity
● Isolation
Ensure tenants cannot access
each others’ workloads,
secrets, etc.
○ Namespace
○ Pod Security Policy
○ Network Policy
○ Sandbox

13.
Multi-Tenancy on Kubernetes
Access Control

14.
Multi-Tenancy on Kubernetes
RBAC
Role-based access control (RBAC) is a method of regulating access to computer
or network resources based on the roles of individual users within your organization
https://www.cncf.io/blog/2020/08/28/kubernetes-rbac-101-authorization/

15.
Multi-Tenancy on Kubernetes
Multi-Tenancy Roles
● [ClusterRoleBinding] Cluster Administrator
● A cluster administrator has access to all cluster resources and can configure
new tenant namespaces
● [ClusterRoleBinding] Cluster view
Read privileges for all resources in the cluster
● [RoleBinding] Tenant Administrator
● A tenant administrator manages namespaces that belong to the tenant
● [RoleBinding] Tenant User
● Read/write privileges for all resources scoped to that tenant

16.
Multi-Tenancy on Kubernetes
Sample Role
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tenant-a-role
namespace: tenant-a
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["batch"]
resources:
- jobs
- cronjobs
verbs: ["*"]

17.
Multi-Tenancy on Kubernetes
Fair Sharing

18.
Multi-Tenancy on Kubernetes
Resource Quota
A resource quota, defined by a ResourceQuota
object, provides constraints that limit aggregate
resource consumption per namespace.
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu-demo
spec:
hard:
requests.cpu : "1"
limits.cpu : "2"
apiVersion: "v1"
kind: "LimitRange"
metadata:
name: "resource-limits"
spec:
limits:
- type: "Container"
max:
cpu: "2"
memory: "1Gi"
min:
cpu: "100m"
memory: "4Mi"
default:
cpu: "300m"
memory: "200Mi"
defaultRequest:
cpu: "200m"
memory: "100Mi"
maxLimitRequestRatio:
cpu: "10"

19.
Multi-Tenancy on Kubernetes
Pod Priority
Priority indicates the importance of a Pod relative to other Pods. If a Pod cannot be
scheduled, the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the
pending Pod possible. An administrator can use ResourceQuota to prevent users from creating
pods at high priorities.
Priority Class Purpose Value/Priority
Cluster Core Essential services to operate the cluster itself, such as Dex for
authentication
100000
Tenant Critical Services which are business-critical to your tenant, and cannot
easily move to a new machine, such as database back-ends
70000
Administrative Services Dashboards that are important for managing the cluster, but could
tolerate a short outage while moving to a new machine
50000
Best Effort (default) No priority assigned; allows termination in favor of higher priority
workloads
100

20.
Multi-Tenancy on Kubernetes
Sample Pod Priority
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: tenant-critical
value: 70000
globalDefault: false
description: "This priority class should be used for s ervices
which are business-critical to your tenant, and cannot easily
move to a new machine, such as database back-ends "

21.
Multi-Tenancy on Kubernetes
Quality of Service for Pods
Quality of Service (QoS) class is a Kubernetes concept that the scheduler uses
for deciding the scheduling and eviction priority of the pods.
https://medium.com/blutv/qos-classes-of-k8s-pods-722238a61c93

22.
Multi-Tenancy on Kubernetes
Taints & Tolerations
● Taints are applied to node, to repel a set of pods.
● Tolerations are applied to pods, and allow (but do not require) the pods to
schedule onto nodes with matching taints.
● Use cases: Dedicated Nodes and Nodes with Special Hardware

23.
Multi-Tenancy on Kubernetes
Pod Disruption Budgets
A PDB limits the number of Pods of a replicated application that are down
simultaneously from voluntary disruptions
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
maxUnavailable : 1
selector:
matchLabels :
app: zookeeper

24.
Multi-Tenancy on Kubernetes
Pod Affinity / Anti-affinity
You can use Pod anti-affinity to prevent Pods
from different tenants from being scheduled on the
same node. Anti-affinity constraints are based on
Pod labels. For example high workload shouldn’t
stay on the same node.
apiVersion: v1
kind: Pod
metadata:
name: bar
labels:
team: "billing"
spec:
affinity:
podAntiAffinity :
requiredDuringSchedulingIgnoredD
uringExecution :
- topologyKey :
"kubernetes.io/hostname"
labelSelector :
matchExpressions :
- key : "team"
operator : NotIn
values : ["billing"]

25.
Multi-Tenancy on Kubernetes
Isolation

26.
Multi-Tenancy on Kubernetes
Namespace
● System namespaces
Exclusively for system pods. Usually kube-system namespace and manage by
cluster administrator
● Service namespaces
These namespaces should run services or applications that need to be accessed
by services in other namespaces. Usually manage by cluster administrator
● Tenant Namespaces
Tenant namespaces should be spun up to run applications that do not need to
be accessed from other namespaces in the cluster. Usually manage by tenant
administrator

27.
Multi-Tenancy on Kubernetes
Pod Security Policy
A PodSecurityPolicy is an admission controller resource you create that validates requests to
create and update Pods on your cluster. The PodSecurityPolicy defines a set of conditions that Pods
must meet to be accepted by the cluster. when a request to create or update a Pod does not meet the
conditions in the PodSecurityPolicy, that request is rejected and an error is returned.
https://rancher.com/blog/2020/pod-security-policies-part-2

28.
Multi-Tenancy on Kubernetes
Sample PSP
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
spec:
privileged: false
# Required to prevent escalations to root.
allowPrivilegeEscalation
: false
# The following is redundant with non-root + disallow
privilege
# escalation, but we can provide it for defense in depth.
requiredDropCapabilities
:
- ALL
# Allow core volume types.
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
# Assume that persistentVolumes set up by the cluster
admin
# are safe to use.
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
# Require the container to run without
root privileges.
rule: 'MustRunAsNonRoot'
seLinux:
# Assumes the nodes are using AppArmor
rather than SELinux.
rule: 'RunAsAny'
supplementalGroups
:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535

29.
Multi-Tenancy on Kubernetes
PodSecurityPolicy Deprecation
Kubernetes 1.21 starts the deprecation process for PodSecurityPolicy.
The current plan is to remove PSP from Kubernetes in the 1.25 release.
Kubernetes v1.22 as an Alpha feature, Kubernetes offers a built-in Pod
Security admission controller, the successor to PodSecurityPolicies.
● https://github.com/kubernetes/enhancements/issues/2579
● https://kubernetes.io/docs/concepts/security/pod-security-admission/

30.
Multi-Tenancy on Kubernetes
Open Policy Agent (OPA)
package kubernetes. admission
deny[msg] {
input.request.kind.kind == "Pod"
some i
image := input.request.object.spec.containers[i].image
not startswith (image, "hooli.com/")
msg := sprintf("image '%v' comes from untrusted registry" ,
[image])
}

31.
Multi-Tenancy on Kubernetes
Network Policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
namespace: tenant-a
spec:
podSelector:
matchLabels :
ingress:
- from:
- podSelector : {}
you should block traffic between
namespaces that host different
tenants' applications

32.
Multi-Tenancy on Kubernetes
Sandbox
VM Container gVisor

33.
Multi-Tenancy on Kubernetes
Future

34.
Multi-Tenancy on Kubernetes
k8s-sig-multi-tenancy
https://github.com/kubernetes-sigs/multi-tenancy

35.
Multi-Tenancy on Kubernetes
Benchmarks
https://github.com/kubernetes-sigs/multi-tenancy/blob/master/benchmarks/kubectl-mtb/README.md

36.
Multi-Tenancy on Kubernetes
The Hierarchical Namespace Controller
$ kubectl hns create my-service -n my-team
$ kubectl hns tree my-team
my-team
└── my-service
https://github.com/kubernetes-sigs/hierarchical-namespaces

37.
Multi-Tenancy on Kubernetes
VirtualCluster
https://www.cncf.io/blog/2019/06/20/virtual-cluster-extending-namespace-based-multi-tenancy-with-a-cluster-view/

38.
Multi-Tenancy on Kubernetes
Wrap-up

39.
Multi-Tenancy on Kubernetes
Key Take-aways
● Use multi-tenancy for improved resource efficiency,
cost, and operations
● Multi-tenancy is unavoidable in the future
● Choose your multi-tenancy type and model
● To do multi-tenancy, you need to config kubernetes
access control, fair sharing, and isolation

40.
Multi-Tenancy on Kubernetes
More questions?
jirayut@opsta.co.th
Jirayut Nimsaeng
CEO & Founder
Opsta (Thailand)
086-069-4042
Facebook