Self-healing does not equal self-healing. There are multiple layers to it, whether a self-healing infrastructure, cluster, pods, or Kubernetes. Kubernetes itself ensures self-healing pods. But how do you ensure your applications, whose reliability depends on every single layer, are truly reliable? In this presentation we discuss aspects of reliability and self-healing in the different layers of a comprehensive container management stack; what Kubernetes does and doesn't do (at least not by default), and what you should look out for to ensure true reliable applications.

1. How Self-Healing Nodes and Infrastructure
Management Impact Reliability
Oleg Chunikhin |CTO

2. Introductions
Oleg Chunikhin
CTO, Kublr
✔ 20+ years in software architecture & development
✔ Working w/ Kubernetes since its release in 2015
✔ CTO at Kublr—an enterprise ready container
management platform
✔ Twitter @olgch; @kublr
Like what you hear? Tweet at us!

3. Automation
Ingress
Custom
Clusters
Infrastructure
Logging Monitoring
Observability
API
Usage
Reporting
RBAC IAM
Air Gap TLS
Certificate
Rotation
Audit
Storage Networking
Container
Registry
CI / CD App Mgmt
Infrastructure
Container Runtime Kubernetes
OPERATIONS SECURITY &
GOVERNANCE
What’s Kublr?
@olgch, @kublr

4. Building a Reliable System with Kubernetes
• Day 1: trivial, K8S will restart my pod!
• Day 30: this is not so simple...
• What Kubernetes does, can, and doesn’t do
• Full stack reliability: tools and approaches
@olgch, @kublr

5. Kubernetes Cluster
K8S Architecture Refresher: Components
The Master, agent, etcd, API, overlay network, and DNS
Master
API Server
etcd data
controller
manager
scheduler etcd
kubectl
Worker
kubelet
container
runtime
overlay
network
cluster
DNS
kube-proxy
@olgch, @kublr

6. Cluster
K8S Architecture Refresher: API Objects
Nodes, pods, services, and persistent volumes
Node 1 Node 2
Pod A-1
10.0.0.3
Cnt1
Cnt2
Pod A-2
10.0.0.5
Cnt1
Cnt2
Pod B-1
10.0.0.8
Cnt3
SrvA
10.7.0.1
SrvB
10.7.0.3
Persistent
Volume
@olgch, @kublr

7. Reliability Tools: Pod Probes and Controllers
Probes
Liveness and readiness check
• TCP, HTTP(S), exec
Controllers
ReplicaSet
• Maintain specific number of identical replicas
Deployment
• ReplicaSet + update strategy, rolling update
StatefulSet
• Deployment + replica identity, persistent volume stability
DaemonSet
• Maintain identical replicas on each node (of a specified set)
Operators
@olgch, @kublr

8. • Resource framework
• Standard: CPU, memory, disk
• Custom: GPU, FPGA, etc.
• Requests and limits
• Kube and system reservations
• no swap
• Pod eviction and disruption budget (resource starving)
• Pod priority and preemption (critical pods)
• Affinity, anti-affinity, node selectors & matchers
Reliability Tools: Resource & Location Mgmt
@olgch, @kublr

9. Reliability Tools: Autoscaling
Horizontal pod autoscaler (HPA)
Vertical pod autoscaler (VPA)
• In-place updates - WIP (issue #5774)
Cluster Autoscaler
• Depends on infrastructure provider - uses node groups
AWS ASG, Azure ScaleSets, ...
• Supports AWS, Azure, GCE, GKE, Openstack, Alibaba Cloud
@olgch, @kublr

10. Applications/pods/containers
“Middleware”
• Operations: monitoring, log collection, alerting, etc.
• Lifecycle: CI/CD, SCM, binary repo, etc.
• Container management: registry, scanning, governance, etc.
Container Persistence: cloud native storage, DB, messaging
Container Orchestrator: Kubernetes
• “Essentials”: overlay network, DNS, autoscaler, etc.
• Core: K8S etcd, master, worker components
• Container engine: Docker, CRI-O, etc.
OS: kernel, network, devices, services, etc.
Infrastructure: “raw” compute, network, storage
Full Stack Reliability
@olgch, @kublr

11. Architecture 101
• Layers are separate and independent
• Disposable/restartable components
• Reattachable dependencies (including data)
• Persistent state is separate from disposable
processes
Pets vs cattle - only data is allowed to be pets
(ideally)
@olgch, @kublr

12. If a node has a problem...
• Try to fix it (pet)
• Replace or reset it (cattle)
Infrastructure and OS
Tools
• In-cluster: npd, weaveworks kured, ...
• hardware, kernel, servicer, container
runtime issues
• reboot
• Infrastructure provider automation
• AWS ASG, Azure Scale Set, ...
• External auto node recovery logic
• Custom + infrastructure provider API
• Cluster management solution
• (Future) cluster API
@olgch, @kublr

13. Components
• etcd
• Master: API server, controller
manager, scheduler
• Worker: kubelet, kube-proxy
• Container runtime: Docker,
CRI-O
Already 12 factor
Kubernetes Components: Auto-Recovery
Monitor liveliness, automate
restart
• Run as services
• Run as static pods
Dependencies to care about
• etcd data
• K8S keys and certificates
• Configuration
@olgch, @kublr

14. API Server
controller
manager
scheduler
API Server
controller
manager
scheduler
etcd data
etcd
etcd data
etcd
K8S multi-master
• Pros: HA, scaling
• Cons: need LB (server or client)
etcd cluster
• Pros: HA, data replication
• Cons: latency, ops complexity
etcd data
• Local ephemeral
• Local persistent (survives node failure)
• Remote persistent (survives node replacement)
Kubernetes Components: Multi-Master
API Server
etcd data
controller
manager
scheduler
etcd
kubectl
kubelet
@olgch, @kublr

15. • Persistent volumes
• Volume provisioning
• Storage categories
• Native block storage: AWS EBS, Azure Disk, vSphere volume,
attached block device, etc.
• HostPath
• Managed network storage: NFS, iSCSI, NAS/SAN, AWS EFS, etc.
• Some of the idiosyncrasies
• Topology sensitivity (e.g. AZ-local, host-local)
• Cloud provider limitations (e.g. number of attached disks)
• Kubernetes integration (e.g. provisioning and snapshots)
Container Persistence
@olgch, @kublr

16. • Integrates with Kubernetes
• CSI, FlexVolume, or native
• Volume provisioners
• Snapshots support
• Runs in cluster or externally
• Approach
• Flexible storage on top of backing storage
• Augmenting and extending backing storage
• Backing storage: local, managed, Kubernetes PV
• Examples: Rook/Ceph, Portworx, Nuvoloso, GlusterFS, Linstor,
OpenEBS, etc.
Cloud Native Storage
@olgch, @kublr

17. Data pool
mon
config
data
config
data
monmon
config
data
Cloud Native Storage: Rook/Ceph
raw data
osd
raw data
osd
raw data
mdsosd
Data pool
Image Image
Ceph
Filesystem
Components
Abstractions
Ceph
rgw
S3/Swift
Object Store
mgr
Rook
Operator
CSI plugins
osdosdganesha
NFS
CephCluster
Block Pool
Object Store
Filesystem
NFS
Object Store User
Provisioners
rbd-mirror
@olgch, @kublr

18. Operations: monitoring, log collection, alerting, etc.
Lifecycle: CI/CD, SCM, binary repo, etc.
Container management: registry, scanning, governance, etc.
Deployment options:
• Managed service
• In Kubernetes
• Deploy separately
Middleware
@olgch, @kublr

19. • Region to region; cloud to cloud; cloud to on-prem (hybrid)
• One cluster (⚠) vs cluster per location (✔)
Tasks
• Physical network connectivity: VPN, direct
• Overlay network connectivity: Calico BGP peering, Native routing
• Cross-cluster DNS: CoreDNS
• Cross-cluster deployment: K8S federation
• Cross-cluster ingress, load balancing: K8S federation, DNS, CDN
• Cross-cluster data replication
• native: e.g. AWS EBS, Snapshots inter-region transfer
• CNS level: e.g. Ceph geo-replication
• database level: e.g. Yugabyte geo-replication, sharding, ...
• application level
Something Missing? Multi-Site
@olgch, @kublr

20. To Recap...
• Kubernetes provides robust tools for application reliability
• Underlying infrastructure and Kubernetes components
recovery is responsibility of the cluster operator
• Kubernetes is just one of the layers
• Remember Architecture 101 and assess all layers accordingly
• Middleware, and even CNS, can run in Kubernetes and be
treated as regular applications to benefit from K8S capabilities
• Multi-site HA, balancing, failover is much easier with K8S and
the cloud native ecosystem. Still requires careful planning!
@olgch, @kublr

21. Q&A
@olgch, @kublr

22. Oleg Chunikhin
CTO | Kublr
oleg@kublr.com
Thank you!
Take Kublr for a test drive!
kublr.com/deploy
Free non-production license
@olgch, @kublr