Something About . . .
Operation/Maintenance
Kubernetes

Hello!
I am Tom Tsai aka smalltown
- MaiCoin Site Reliability Engineer
- Taipei HashiCorp User Group
Organizer
- AWS User Group Taiwan Staff
2

“
3
http://bit.ly/taipei-hug

“ Infrastructure as Code
Terraform ❤ AWS EKS
4

“ Have You Used Kubernetes
in Production Environment?
5

Outline
◂ Where is Your Kubernetes?
◂ High Availability/Scalability
◂ Manage Stateful Service
◂ Log/Monitoring
◂ Continuous Delivery
◂ Your Best Friend: kubectl
◂ Troubleshooting Guide
◂ Q&A
6

1.
Where is Your
Kubernetes?
AWS, Azure, GEK, IDC?

Select Where Your Kubernetes is!
8
Public Cloud
No
Yes
Completed
Network Infra.
General Usage
Yes
Yes
No
Shutdown
Suddenly
No Yes
No

Secure Your Kubernetes Cluster
No Matter Where it is
9
Private Network
- Kubernetes
- Database
- NAT Gateway
- ...
Public Network
- Load Balancer
- Linux Gateway
- ...
Non-Employee Only
Access Product Service
Through Load Balancer

When Should I Self-Hosted Kubernetes
◂ You Should Avoid It! Why?
◂ Unless Company Policy Not Allow
◂ And XKS Really Not Fulfill Your
Requirements (X= A, E, G...)
10

1.
High
Availability/Scalability
Failure is Normal and
Inevitable

High Level Design
12
Ref

High Availability: Kubernetes Itself
13
- Key/Value Storage - apiserver
- scheduler
- controller-manager
- kubelet
- kube-proxy
- container runtime- HA: 1, 3, 5 …
- Clustering Guide
- General HA Mode

What About K8S
Inside?
14

15
Ref

16
Ref

High Availability: Application in K8S
◂ Kubernetes Deployments/StatefulSets
Handle it
17
~$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
Nginx 3 3 3 3 18s

Before Talk About
Scalability . . .
18

K8S Resource Request & Limit
19
Ref

Scalability: Horizontal Pod Autoscaler
20
Ref

Scalability: Vertical Pods Autoscaler
21
Ref

Scalability: Cluster Autoscaler
22
Ref

1.
Manage
Stateful Service
Running Stateful Service in
Kubernetes Easily

Stateful Service
◂ Replication Setup, e.g. MongoDB, MySQL
◂ Cluster Management, e.g. Etcd, Kafka,
ElasticSearch
◂ Scale Out, Scale Up
◂ Upgrade
◂ Data Backup
◂ Other Operation Job...
24

Manage Production Service
25
Stateless Stateful
Setup
Scale Out/Up
Backup
Upgrade
Operation

CoreOS Brings
K8S Operator . . .
26

What is Kubernetes Operator?
◂ An Operator is a Software with Specific
Application Knowledge
◂ Extends the Kubernetes through Custom
Controller/Resource
◂ Enable Users to Create, Configure, and
Manage Applications
27

What is Kubernetes Operator?
28
An Operator is a Domain Specific Controller

Operation of Stateless Service
29
kubectl
ReplicaSet
app=web, env=prod
Pod
app=web
env=prod
scale up
desired=3 count=1
kubectl
ReplicaSet
app=web, env=prod
Pod
app=web
env=prod
desired=3 count=3

Operation of Stateful Service
(Operator)
30
backup
kubectl
etcd Operator
app=3.0.13
size=3
Pod
app=web
env=prod
backup-tool
interval=300s

Exist Operators
◂ There is Several Operators Currently
◂ Kubernetes Summit Sharing (Demo)
31

1.
Log/Monitoring
Know Service Health at Any
Time

Log Management
. . .
33

Log Management: Yes, ELKF
◂ EK: Deploy By Operator
◂ Use Docker Image Without X-Pack
◂ docker.elastic.co/elasticsearch/elastics
earch-oss:6.3.1
34

Log Collect
Centralised V.S. Decentralised
◂ Official Suggestion
◂ Ship Log By Each Node
◂ Ship Log By Each Pod
35

System/Application
Metric Collect . . .
36

Metrics: Yes, Prometheus
◂ Deploy by Operator
37

Get Alert When
Service . . .
38

Monitoring: Internal
◂ Kubernetes Liveness and Readiness
Probes Cover Auto-Healing
◂ Prometheus Operator Include All
Kubernetes Resource Monitor
39
Ref

Monitor: External
◂ Who Can Monitor Prometheus?
◂ Suggest Find Solution to Monitor From
Outside
◂ Access From Different Country
◂ Example: Site24x7, Monitis, ...etc
40

1.
Continuous Delivery
Huge Benefits, But
Challenges Too

Common Resource View of K8S Application
42
Deployment YAML
ConfigMap YAML Secret YAML/VaultDocker Image Tag
Service YAML

Deploy With kubectl
43
Ref

The Key Man: Helm
◂ The Package Manager for Kubernetes
◂ Template Engine + Dependency
Management
◂ Jenkins X and GitOps also Integrate With
It
44

Deploy With Helm
45
Ref

Good Old Days V.S. Kubernetes
46
GNU/Linux
ELF Binaries Config in /etc
apt, yum, etc
Chef, Puppet, Ansible
Kubernetes
Images K8S Object
Helm
Operator Framework

CI/CD Total Solution For K8S
◂ Jenkins X
◂ GitLab
◂ GitOps
◂ Spinnaker
◂ ...
47

GitOps Patterns
48
Ref

1.
TroubleShooting
Guide
Where’s Wally?

Official TroubleShooting Guide
◂ For Application Debug
◂ Pod, Replication Controller, Service
◂ For Cluster Debug
◂ Master, Node
50

Just My Two Cents: Pod
~$ kubectl describe pod #{Pod Name}
◂ Pending: No Node Available, Persistent
Volume Issue
◂ Terminating: Container Cannot be Killed
Normally, Login Node to Check Docker
◂ CrashLoopBackOff: No Docker Image
Exist, or Find Why Container Start Fail
51

Just My Two Cents: Service
~$ kubectl describe svc #{SVC Name}
◂ Endpoint is Empty: No Pod is Selected
For This Service
◂ NodePort Conflict With Other Service
◂ Some Modification Need To Delete Then
Re-Create
◂ Ingress Problem
52

Just My Two Cents: Node Register
◂ Overlay Network Issue Lead to Register
Fail
◂ Firewall Rule Between Node and Master
◂ ...
◂ Anyway, Login Node to Find Which
Component Encounter Trouble
53

1.
Your Best
Friend: kubectl
Good Craftsmanship Depends
on Use of The Right Tools

When You Manage N Clusters N
Namespaces
◂ You Want to Perform Operation For B
Pod in D Cluster U Namespace
◂ Switch Kubeconfig to D Cluster
◂ Switch to U Namspace
◂ Start to Perform Operation For B Pod
◂ Takes Many Steps, Don’t Mention How To
Know In The Right Place?
55

There Are Several Helper Tool For You
◂ Setup Kubeconfig For Multiple Cluster
◂ Enable Shell Completion For kubectl
◂ Install kubectx to Switch Cluster or
Namespace
◂ Install kube-ps1 to Know Current Cluster
and Namespace From Terminal
◂ More Helper Tools
56

57
Thanks!
Any questions?
You can find me at
◂ facebook.com/smalltown0110