More Related Content Similar to Kubernetes Architecture - beyond a black box - Part 1 (20) Kubernetes Architecture - beyond a black box - Part 11. Kubernetes - Beyond a Black Box
A humble peek into one level below your running application in production
2. Copyright Ā© 2017 Hao Zhang All rights reserved.
About The Author
ā¢ Hao (Harry) Zhang
ā¢ Currently: Software Engineer, LinkedIn, Team Helix @ Data
Infrastructure
ā¢ Previously: Member of Technical Staff, Applatix, Worked with
Kubernetes, Docker and AWS
ā¢ Previous blog series: āMaking Kubernetes Production Readyā
ā¢ Part 1, Part 2, Part 3
ā¢ Or just Google āKubernetes productionā, āKubernetes in productionā, or similar
ā¢ Connect with me on LinkedIn
3. Copyright Ā© 2017 Hao Zhang All rights reserved.
Motivation
ā¢ Itās one of the hottest cluster management project on Github
ā¢ It has THE largest developer community
ā¢ State-of-art architecture designed mainly by people from Google
who have been working on container orchestration for 10 years
ā¢ Collaboration of hundreds of engineers from tens of companies
4. Copyright Ā© 2017 Hao Zhang All rights reserved.
Introduction
This set of slides is a humble dig into one level below your running
application in production, revealing how different components of
Kubernetes work together to orchestrate containers and present your
applications to the rest of the world.
The slides contains 80+ external links to Kubernetes documentations, blog
posts, Github issues, discussions, design proposals, pull requests,
papers, source code ļ¬les I went through when I was working with
Kubernetes - which I think are valuable for people to understand how
Kubernetes works, Kubernetes design philosophies and why these design
came into places.
5. Copyright Ā© 2017 Hao Zhang All rights reserved.
Outline - Part I
ā¢ A high level idea about what is Kubernetes
ā¢ Components, functionalities, and design choice
ā¢ API Server
ā¢ Controller Manager
ā¢ Scheduler
ā¢ Kubelet
ā¢ Kube-proxy
ā¢ Put it together, what happens when you do `kubectl create`
6. Copyright Ā© 2017 Hao Zhang All rights reserved.
Outline - Part II
ā¢ An analysis of controlling framework design
ā¢ Micro-service Choreography
ā¢ Generalized Workload and Centralized Controller
ā¢ Interfaces for production environments
ā¢ High level workload abstractions
ā¢ Strength and limitations
ā¢ Conclusion
9. ā Kubernetes Ofļ¬cial Website
āKubernetes is an open-source system for
automating deployment, scaling, and
management of containerized
applications.ā
10. Copyright Ā© 2017 Hao Zhang All rights reserved.
Manage Apps, Not Machines
Manages Your Apps
ā¢ App Image Management
ā¢ Where to Run
ā¢ When to Get, when to Run, and when to Discard
ā¢ App Image Usage
ā¢ Image Service: Image lifecycle management
ā¢ Runtime: Run image as application
11. Copyright Ā© 2017 Hao Zhang All rights reserved.
Manage Apps, Not Machines
Manages Things Around Your Apps
ā¢ High level workload abstractions
ā¢ Pod, Job, Deployment, StatefulSet, etc.
ā¢ āIf container resembles atom, Kubernetes provides moleculesā - Tim Hockin
ā¢ Storages
ā¢ Data volumes
ā¢ Network
ā¢ IP, Port mapping, DNS, ā¦
ā¢ Monitoring, scaling, communicating, ā¦
12. Copyright Ā© 2017 Hao Zhang All rights reserved.
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
Persistent Data Volume
KubeletKube-proxy
Kubernetes Control Plane
KubeletKube-proxy KubeletKube-proxy
13. Copyright Ā© 2017 Hao Zhang All rights reserved.
A Simpliļ¬ed Typical Web Service
Load Balancer
Web Server
MemCache
Object Store
(BLOB)
Data Volumes
Database
(SQL / NoSQL)
Backend Sync
Read Services
Backend Write
Sync Services
Backend Write
Async Services
Task Queue
Worker Pool
Browser Client
14. Copyright Ā© 2017 Hao Zhang All rights reserved.
Web Service On Kube (AWS)
VPC (Virtual Private Cloud)
Subnet
EC2 Instance
EBS etcd
apiserver
Sched
Kubelet
CtlMgr
Docker
EBS
Auto Scaling Group
EC2 Instance
EBS etcd
apiserver
Sched
Kubelet
CtlMgr
Docker
EBS
ElasticLoadBalancer
Subnet
Elastic Load Balancer
EC2 Instance
KubeletDocker
Kube-proxy
EBS
SyncWrite AsyncW
Worker Pool MemCache
EC2 Instance
KubeletDocker
Kube-proxy
EBS
Web SvrSyncRead
AsyncW TaskQ EBS
MemCache
EC2 Instance
KubeletDocker
Kube-proxy
EBS
Web Svr Worker Pool
MemCache DB EBS
EC2 Instance
KubeletDocker
Kube-proxy
EBS
SyncRead SyncWrite
Worker PoolMemCache
DB EBS
EC2 Instance
KubeletDocker
Kube-proxy
EBS
SyncWrite TaskQ EBS
MemCache
DB EBS
EC2 Instance
KubeletDocker
Kube-proxy
EBS
Web SvrAsyncW
TaskQ EBSWorker Pool
DBEBS
Auto Scaling Group
VPC Routing Table
S3 (External Blob, Kubernetes Binaries, Docker Registry)
VPC Gateway
Note: This chart is a rough illustration about how different production components might sit in cloud.
It does not reļ¬ect any strict production conļ¬gurations such as high-availability, fault tolerance, machine load balancing etc.
16. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes REST API
ā¢ Every piece of information in Kubernetes is an API object (a.k.a
Cluster metadata)
ā¢ Node, Pod, Job, Deployment, Endpoint, Service, Event,
PersistentVolume, ResourceQuota, Secrets, Conļ¬gMap, ā¦ā¦
ā¢ Schema and data model explicitly deļ¬ned and enforced
ā¢ Every API object has a UID and version number
ā¢ Micro-services can perform CRUD on API objects through REST
ā¢ Desired state in cluster is achieved by collaborations of separate
autonomous entities reacting on changes of one or more API
object(s) they are interested in
17. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes REST API
ā¢ The API Space
ā¢ Different API Groups
ā¢ Usually a group represents a set of features, i.e. batch, apps, rbac, etc.
ā¢ Each group has its own version control
/apis/batch/v1/namespaces/default/jobs/myjob
Root Group
Name
Namespaced
Object
Namespace
Name
API Resource
Kind
API Object
Instance Name
Version
More Readings:
ā¤ Extending APIs design spec: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/extending-api.md
ā¤ Customer Resource Documentation: https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-deļ¬nitions/
ā¤ Extending core API with aggregation layer: https://kubernetes.io/docs/concepts/api-extension/apiserver-aggregation/
ā¤ Kubernetes API Spec: https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.7/api/openapi-spec/swagger.json
ā¤ Kubernetes API Server Deep Dives:
ā¤ https://blog.openshift.com/kubernetes-deep-dive-api-server-part-1/
ā¤ https://blog.openshift.com/kubernetes-deep-dive-api-server-part-2/
18. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes REST API
ā¢ The API Object Deļ¬nition
ā¢ 5 major ļ¬elds: apiVersion, kind, metadata, spec and status
ā¢ A few exceptions such as v1Event, v1Binding, etc
ā¢ Metadata includes name, namespace, label, annotation, version, etc
ā¢ Spec describes desired state, while status describes current state
$ kubectl get jobs myjob --namespace default --output yaml
apiVersion: batch/v1
kind: job
metadata:
name: myjob
namespace: default
spec:
(Describing desired ājobā object, i.e. Pod template, how many runs, etc.)
status:
(ājobā object current status, i.e. ran # times already, etc.)
19. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Control Plane
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
Persistent Data Volume (Usually SSD)
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node
20. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-apiserver & Etcd
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
Persistent Data Volume (Usually SSD)
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node
21. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-apiserver & Etcd
Etcd as metadata store:
ā¢ Watchable distributed K-V store
ā¢ Transaction and locking support
ā¢ Scalable and low memory consumption
ā¢ RBAC support
ā¢ Uses Raft for leader election
ā¢ R/W on all nodes of Etcd cluster
ā¢ ā¦
General Functionalities:
ā¢ Authentication
ā¢ Admission control (RBAC)
ā¢ Rate limiting
ā¢ Connection pooling
ā¢ Feature grouping
ā¢ Pluggable storage backend
ā¢ Pluggable customer API deļ¬nition
ā¢ Client tracing and API call stats
For API Objects:
ā¢ CRUD Operations
ā¢ Defaulting
ā¢ Validation
ā¢ Versioning
ā¢ Caching
For Containers:
ā¢ Application Proxying
ā¢ Stream Container Logs
ā¢ Execute Command in Container
ā¢ Metrics
WRITE API Call Process Pipeline:
1. Route through mux to the routine
2. Version conversions
3. Validation
4. Encode
5. Storage backend communication
READ API calls are process roughly
opposite of WRITE API calls.
etcd3
Kube-apiserver
gRPC + protobuf
REST APIs:
HTTP/HTTPS
Compute Node
Usually Co-locate
on same host
SSD
22. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-apiserver & Etcd
ā¢ API Server caching
ā¢ Decoding object from Etcd is a huge burden
ā¢ Without caching, 50% CPU spent on decoding objects read from etcd,
70% of which is on convert K/V (as of Etcd2)
ā¢ Also adds pressure on Golangās GC (Lots of unused K/V)
ā¢ In-memory Cache for READs (Watch, Get and List)
ā¢ Decode only when etcd has more up-to-date version of the object
23. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-apiserver & Etcd
Citation and More Readings:
ā¤ Etcd! Etcd3!
ā¤ Discussion about pluggable storage backend (ZK, Consul, Redis, etc): https://github.com/kubernetes/kubernetes/issues/1957
ā¤ More on āWhy etcd?ā (Could be biased): https://github.com/coreos/etcd/blob/master/Documentation/learning/why.md
ā¤ Upgrade to etcd3: https://github.com/kubernetes/kubernetes/issues/22448
ā¤ Etcd3 Features: https://coreos.com/blog/etcd3-a-new-etcd.html
ā¤ Kubernetes & Etcd
ā¤ Kubernetesā Etcd usage: http://cloud-mechanic.blogspot.com/2014/09/kubernetes-under-hood-etcd.html
ā¤ Raft - the consensus protocol behind etcd: https://speakerdeck.com/benbjohnson/raft-the-understandable-distributed-consensus-protocol/
ā¤ Protocol Buffer: https://developers.google.com/protocol-buffers/
ā¤ Setting up HA Kubernetes cluster: https://kubernetes.io/docs/admin/high-availability/#clustering-etcd
ā¤ Kubernetes API
ā¤ Kubernetes API Convention: https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md
ā¤ API Server Optimizations and Improvements
ā¤ Discussion about cache: https://github.com/kubernetes/kubernetes/issues/6678, Watch Cache Design Proposal
ā¤ Discussion about optimizing etcd connection: https://github.com/kubernetes/kubernetes/issues/7451
ā¤ Optimizing GET, LIST, and WATCH: https://github.com/kubernetes/kubernetes/issues/4817, https://github.com/kubernetes/kubernetes/issues/6564
ā¤ Serve LIST from memory: https://github.com/kubernetes/kubernetes/issues/15945
ā¤ Optimizing get many pods: https://github.com/kubernetes/kubernetes/issues/6514
24. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-apiserver & Etcd
ā¢ Kubernetesā Performance SLO
ā¢ API Responsiveness: 99% API calls < 1s
ā¢ Pod Startup Time: 99% Pods and their containers start < 5s (With pre-pulled images)
ā¢ 5000 Nodes, 150,000 Pods, ~300,000 Containers (1~1.5M API objects)
ā¢ Resource (Single master set up for 1000 nodes as of v1.2): 32 Cores, 120G Memory
Citation and More Readings:
ā¤ Kubernetes 1.2 Scalability: http://blog.kubernetes.io/2016/03/1000-nodes-and-beyond-updates-to-Kubernetes-performance-and-scalability-in-12.html
ā¤ Kubernetes 1.6 Scalability: http://blog.kubernetes.io/2017/03/scalability-updates-in-kubernetes-1.6.html
25. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-controller-manager
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
Persistent Data Volume (Usually SSD)
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node
26. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-controller-manager
ā¢ Framework: Reļ¬ector, Store, SharedInformer, and Controller
ā¢ One of the optimizations in 1.6 to make it scale from 2k to 5k nodes
ā¢ ListAndWatch on a particular resource
ā¢ All instance of a resource
ā¢ List upon startup and connection broken
ā¢ List: get current states
ā¢ Watch: get updates
ā¢ Push ListAndWatch results into store
ā¢ Store is always up-to-date
ā¢ In memory read-only cache
ā¢ Shared among informers
ā¢ Optimizes GET and LIST
ā¢ Cache Update Broadcast
ā¢ Events: OnAdd, OnUpdate, OnDelete
ā¢ Store is at least as fresh as notiļ¬cation
Controller
Reļ¬ector
Store
SharedInformer
Event Handlers
Task Queue
ā¢ Blocking, MT Safe FIFO Queue
ā¢ Additional feature such as delayed
scheduling, failure counts, etc.
ā¢ Controller registers event handlers
to the shared informer of the
resource it is interested in
ā¢ A single SharedInformer has a pool
of event handlers registered by
different controllers
ā¢ Worker pool of a particular resource
ā¢ Reconciliation controlling loops
ā¢ Action is computed based on
DesiredState and CurrentState
27. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-controller-manager
ā¢ Reconciliation Controlling Loop: Things will ļ¬nally go right
func (c *Controller) Run(poolSize int, stopCh chan struct{}) {
// start worker pool
for i := 0; i < poolSize; i++ {
// runWorker will loop until "something bad" happens. The .Until will
// then rekick the worker after one second
go wait.Until(c.runWorker, 1 * time.Second, stopCh)
}
// wait until we're told to stop
<-stopCh
}
func (c *Controller) runWorker() {
// hot loop until we're told to stop.
for c.processNextWorkItem() {}
}
func (c *Controller) processNextWorkItem() {
obj = c.queue.Get()
// Reconciliation between current and desired state
err = c.makeChange(obj.CurrentState, obj.DesiredState)
if !err {
return
}
// Cool down and retry using delayed scheduling upon error
c.queue.AddRateLimited(obj)
return
}
28. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-controller-manager
Shared Resource Informer Factory
Shared Cache
Pod Reļ¬ Job Reļ¬ Dep Reļ¬ Svc Reļ¬ Node Reļ¬ Ds Reļ¬ Rs Reļ¬ ā¦ā¦
Pod EventHdl Job EventHdl Dep EventHdl Svc EventHdl Node EventHdl Ds EventHdl Rs EventHdl ā¦ā¦
Pod Informer Job Informer Dep Informer Svc Informer Node Informer Ds Informer Dep Informer Other Informers
Shared Kubernetes Client
Handler Event Dispatching Mesh
Dep Worker Pool
Dep Controller
Queue
Job Worker Pool
Job Controller
Queue
Svc Worker Pool
Svc Controller
Queue
Rs Worker Pool
Rs Controller
Queue
Ds Worker Pool
Ds Controller
Queue
Node Worker Pool
Node Controller
Queue
XYZ Worker Pool
XYZ Controller
Queue
29. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-controller-manager
ā¢ 28 Controllers as of 1.8, usually deployed as a Pod
ā¢ All autonomous micro-services: easily turn on/off
ā¢ HA via leader election
ā¢ Shared Kubernetes client
ā¢ Limit QPS, manage session/HTTPConnection, auth, retry with jittered backoff
ā¢ InformerFactory (SharedInformer)
ā¢ Reduce API Call, ensure cache consistency, lighter GET/LIST
ā¢ Reļ¬ector / Store / Informer / Controller framework
ā¢ Easy, separated development of different controllers as micro services
ā¢ Worker Pool
ā¢ Individually conļ¬gurable agility and resource consumption
30. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-controller-manager
More Readings:
ā¤ Controller Dev Guide: https://github.com/kubernetes/community/blob/master/contributors/devel/controllers.md
ā¤ Controller Manager Main: https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-controller-manager/app/controllermanager.go
ā¤ Shared Informer: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/shared_informer.go
ā¤ Controller Conļ¬g: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/controller.go
ā¤ Reļ¬ector: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/reļ¬ector.go
ā¤ List and Watch: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/listwatch.go
ā¤ All Controller Logics: https://github.com/kubernetes/kubernetes/tree/master/pkg/controller
31. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-scheduler
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
Persistent Data Volume (Usually SSD)
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node
32. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-scheduler
ā¢ Scheduling Algorithm
ā¢ Filter Out Impossible Node
ā¢ NodeIsReady: Node should be in Ready state
ā¢ CheckNodeMemoryPressure: Node can be over-provisioned,
ļ¬lter out node with memory pressure
ā¢ CheckNodeDiskPressure: Node reporting host disk pressure
should not be considered
ā¢ MaxNodeEBSVolumeCount: If a Pod needs a volume, ļ¬lter out
nodes reach max volume count
ā¢ MatchNodeSelector: If Pod has a nodeSelector, it should only
consider nodes with that label
ā¢ PodFitsResources: Node should have resource quota
ā¢ PodFitsHostPort: Podās HostPort should be available
ā¢ NoDiskConļ¬ict: We should have volume this Pod wants
ā¢ NoVolumeZoneConļ¬ict: Filter out nodes with zone conļ¬ict
ā¢ ā¦ā¦
ā¢ Give Every Node a Score
ā¢ LeastRequestedPriority: Nodes āmore idleā is preferred
ā¢ BalancedResourceAllocation: Balance node utilization
ā¢ SelectorSpreadPriority: Replicas should be spread out
ā¢ Afļ¬nity/AntiAfļ¬nityPriority: āI prefer to stay away / co-locate
from some other categories of Podsā
ā¢ ImageLocalityPriority: Node is preferred if it has the image
ā¢ ā¦ā¦
Kubernetes Client
Node Filter
Node Ranker
Decision
Watch Unscheduled Pods
Node Candidates
Nodes Sorted By Ranking
POST/v1/bindings
Node with highest ranking
33. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-scheduler
ā¢ Default scheduler
ā¢ Sequential scheduler
ā¢ Some application / topology awareness
ā¢ User can deļ¬ne afļ¬nity, anti-afļ¬nity, node-selector, etc
ā¢ Schedule 30,000 Pods onto 1,000 Nodes within 587 Seconds (In v1.2)
ā¢ HA via leader election
ā¢ Supports multiple schedulers
ā¢ Pod can choose the scheduler that schedules it
ā¢ Schedulers share same pool of nodes, race condition is possible
ā¢ Kubelet can reject Pods in case of conļ¬ict, and triggers reschedule
ā¢ Easy to write userās own scheduler or just extend algorithm provider for default
scheduler
34. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-scheduler
More Readings:
ā¤ Documentations:
ā¤ Scheduler Overview: https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler.md
ā¤ Guaranteed scheduling through re-scheduling: https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
ā¤ Default Scheduling Algorithm: https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler_algorithm.md
ā¤ Blogs
ā¤ Scheduler Optimization by CoreOS: https://coreos.com/blog/improving-kubernetes-scheduler-performance.html
ā¤ Advanced Scheduling and Customized Scheduler After 1.6: http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kubernetes.html
ā¤ Critical Code Snippets
ā¤ Scheduling Predicates: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go
ā¤ Scheduling Defaults: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go
ā¤ Designs, Specs, Discussions
ā¤ Scheduling Feature Proposals: https://github.com/kubernetes/community/tree/master/contributors/design-proposals/scheduling
35. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubelet
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
Persistent Data Volume (Usually SSD)
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node
36. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubelet
ā¢ Kubernetesā node worker
ā¢ Ensure Pods run as described by spec
ā¢ Report readiness and liveness
ā¢ Node-level admission control
ā¢ Reserve resource, limite # of Pods, etc.
ā¢ Report node status to API server (v1Node)
ā¢ Machine info, resource usage, health, images
ā¢ Host stats (Cgroup metrics via cAdvisor)
ā¢ Communicate with container through runtime interface
ā¢ Stream logs, execute commands, port-forward, etc
ā¢ Usually not run as a container, but a service under Systemd
37. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubelet
ContainerRuntime(i.e.Docker)
ā¦
Pod Lifecycle Event Generator (PLEG)
(Pending, Running, Succeeded, Failed, Unknown)
Runtime Status
(RPC)
Pod Cache
Kubernetes
ClientWatched
PodsiNotifyPod Spec
Source
Pod Worker UID1
Pod Worker UID2
Pod Worker UIDN
Runtime Op
(RPC)
Pod Worker Pool
External Pod Event
Sources:
i.e. Container runtime
event stream, cAdvisor
events (Cgroup)
Pod Status
Report
Volume
Management
Container
Prober
Main Sync Loop
Pod Conļ¬g Channel
Pod Re-sync Channel
(Clock Tick)
Housekeeping Channel
(Clock Tick)
Cleanup unwanted
Pods, pod workers,
release volumes, etc
PLEG
Channel
38. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubelet
ā¢ Main Controller Loop
ā¢ Listen on events from channels and react
ā¢ PLEG (Pod Lifecycle Event Generation)
ā¢ Listen and List on external sources, i.e. container runtime, cAdvisor
ā¢ Translate container status changes to Pod Events
ā¢ Pod Worker
ā¢ Interact with container runtime
ā¢ Move Pod from current state (v1Pod.status) to desired state
(v1Pod.spec)
39. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubelet
ā¢ Container Prober
ā¢ Probe container liveness and healthiness using methods deļ¬ned by
spec
ā¢ Publish event (v1Event) and put container in CrashLoopBackoff upon
probing failures
ā¢ Volume Manager
ā¢ Local directory management, device mount/unmount, device
initialization, etc
ā¢ cAdvisor
ā¢ Container matrix collection and reporting, events about kernel Cgroups
40. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubelet
ā¢ How Pod is set up
ā¢ Sandbox: An isolated environment with resource constraints
ā¢ Impl. Based on runtime
ā¢ Could be VM, Linux Cgroup, etc
ā¢ Kubelet sets the following for docker impl.
ā¢ Cgroup, Port mapping, Linux security context
ā¢ Holds resolv.conf that is shared by all containers in Pod
ā¢ dockerService.makeSandboxDockerConfig()
ā¢ App Containers
ā¢ Use Sandbox as Cgroup parent
ā¢ Share same ālocalhostā
ā¢ dockerService.CreateContainer()
ā¢ kubeGenericRuntimeManager. startContainer()
Pod App
Container
Pod
Pod App
Container
Port: 80
curl localhost:80
Pod Sandbox
(Infra Container)
PodIP: 192.168.12.3
PortMapping: 80::33580
curl 192.168.12.3:33580
41. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubelet
More Readings:
ā¤ Kubelet Main: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go
ā¤ Kubelet main function: Kubelet.Run()
ā¤ Kubelet sync Loop channel event processing: Kubelet.syncLoopIteration()
ā¤ Kubelet sync Pod: Kubelet.syncPod()
ā¤ Pod Worker: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/pod_workers.go
ā¤ Container Runtime Sync Pod: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_manager.go
ā¤ Sync Pod using CRI: kubeGenericRuntimeManager.SyncPod()
ā¤ Container Runtime Start Container: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_container.go
ā¤ How kubelet setup container: kubeGenericRuntimeManager.startContainer()
ā¤ Implementation of PodSandbox using Docker: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_sandbox.go
ā¤ Create Pod sandbox: dockerService.RunPodSandbox()
ā¤ PLEG Design Proposal: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/pod-lifecycle-event-generator.md
ā¤ Pod Cgroup Hierarchy: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/pod-resource-
management.md#cgroup-hierachy
ā¤ All Kubelet related design docs: https://github.com/kubernetes/community/tree/master/contributors/design-proposals/node
42. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-proxy
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
Persistent Data Volume (Usually SSD)
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node
43. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kube-proxy
Kubernetes
Client
Kube-proxy
WATCH /api/v1/service
WATCH /api/v1/endpoint
Kernel iptables
Manages iptable rules
ā¢ Daemon on every node
ā¢ Watch on changes of Service and
Endpoint
ā¢ Manipulate iptable rules on host to
achieve service load balancing (one of
the implementations)
More Readings:
ā¤ v1Service object and different types of proxy implementations: https://kubernetes.io/
docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
ā¤ iptable proxy implementation: https://github.com/kubernetes/kubernetes/blob/
master/pkg/proxy/iptables/proxier.go
ā¤ Proxier.syncProxyRules()
ā¤ Other built-in proxy implementations: https://github.com/kubernetes/kubernetes/
tree/master/pkg/proxy
ā¤ Some early discussions about Service object and implementations: https://
github.com/kubernetes/kubernetes/issues/1107
45. Copyright Ā© 2017 Hao Zhang All rights reserved.
Put it together
$ kubectl create -f myapp.yaml
Deployment nginx-deployment created
$
$ cat myapp.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 9376
ā¢ What will happen when you do the
following?
46. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Reaction Chain
Kubenetes API Server & Etcd
Kubectl
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
Scheduler
WATCH v1Pod,
v1Node
Kubelet-node1
WATCH v1Pod on node1
CRI
CNI
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior
47. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Reaction Chain
Kubenetes API Server & Etcd
1.POSTv1Deployment
Kubectl
v1Deployment
2. Persist v1Deployment API object
desiredReplica: 3
availableReplica: 0
3.Reļ¬ectorupdatesDep
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
Scheduler
WATCH v1Pod,
v1Node
Kubelet-node1
WATCH v1Pod on node1
CRI
CNI
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior
48. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Reaction Chain
Kubenetes API Server & Etcd
1.POSTv1Deployment
Kubectl
v1Deployment
2. Persist v1Deployment API object
desiredReplica: 3
availableReplica: 0
3.Reļ¬ectorupdatesDep
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
4. Deployment reconciliation loop
action: create ReplicaSet
5.POSTv1ReplicaSet
v1ReplicaSet
6. Persist v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
7.Reļ¬ectorupdatesRS
Scheduler
WATCH v1Pod,
v1Node
Kubelet-node1
WATCH v1Pod on node1
CRI
CNI
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior
49. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Reaction Chain
Kubenetes API Server & Etcd
1.POSTv1Deployment
Kubectl
v1Deployment
2. Persist v1Deployment API object
desiredReplica: 3
availableReplica: 0
3.Reļ¬ectorupdatesDep
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
4. Deployment reconciliation loop
action: create ReplicaSet
5.POSTv1ReplicaSet
v1ReplicaSet
6. Persist v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
7.Reļ¬ectorupdatesRS
8. ReplicaSet reconciliation loop
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
10. Persist 3 v1Pod API objects
status: Pending
nodeName: none
11.Reļ¬ectorupdatesPod
Kubelet-node1
WATCH v1Pod on node1
CRI
CNI
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior
50. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Reaction Chain
Kubenetes API Server & Etcd
1.POSTv1Deployment
Kubectl
v1Deployment
2. Persist v1Deployment API object
desiredReplica: 3
availableReplica: 0
3.Reļ¬ectorupdatesDep
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
4. Deployment reconciliation loop
action: create ReplicaSet
5.POSTv1ReplicaSet
v1ReplicaSet
6. Persist v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
7.Reļ¬ectorupdatesRS
8. ReplicaSet reconciliation loop
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
10. Persist 3 v1Pod API objects
status: Pending
nodeName: none
11.Reļ¬ectorupdatesPod
12. Scheduler assign
node to Pod
13.POSTv1Binding*3
14. Persist 3 v1Binding API objects
name: PodName
targetNodeName: node1
Kubelet-node1
WATCH v1Pod on node1
15.Podspecpushedtokubelet
CRI
16. Pull Image, Run Container,
PLEG, etc. Create v1Event and
update Pod status
CNI
v1Binding * 3
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior
51. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Reaction Chain
Kubenetes API Server & Etcd
1.POSTv1Deployment
Kubectl
v1Deployment
2. Persist v1Deployment API object
desiredReplica: 3
availableReplica: 0
3.Reļ¬ectorupdatesDep
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
4. Deployment reconciliation loop
action: create ReplicaSet
5.POSTv1ReplicaSet
v1ReplicaSet
6. Persist v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
7.Reļ¬ectorupdatesRS
8. ReplicaSet reconciliation loop
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
10. Persist 3 v1Pod API objects
status: Pending
nodeName: none
11.Reļ¬ectorupdatesPod
12. Scheduler assign
node to Pod
13.POSTv1Binding*3
14. Persist 3 v1Binding API objects
name: PodName
targetNodeName: node1
Kubelet-node1
WATCH v1Pod on node1
15.Podspecpushedtokubelet
CRI
16. Pull Image, Run Container,
PLEG, etc. Create v1Event and
update Pod status
17.PATCHv1Pod*3
17-1.PUTv1Event
17. Update 3 v1Pod API objects
status: PodInitializing -> Running
nodeName: node1
v1Event * N
17-1. Several v1Event objects created
e.g. ImagePulled, CreatingContainer,
StartedContainer, etc., controllers, scheduler
can also create events throughout the entire
reaction chain
CNI
v1Binding * 3
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior
52. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Reaction Chain
Kubenetes API Server & Etcd
1.POSTv1Deployment
Kubectl
v1Deployment
2. Persist v1Deployment API object
desiredReplica: 3
availableReplica: 0
3.Reļ¬ectorupdatesDep
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
4. Deployment reconciliation loop
action: create ReplicaSet
5.POSTv1ReplicaSet
v1ReplicaSet
6. Persist v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
7.Reļ¬ectorupdatesRS
8. ReplicaSet reconciliation loop
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
10. Persist 3 v1Pod API objects
status: Pending
nodeName: none
11.Reļ¬ectorupdatesPod
12. Scheduler assign
node to Pod
13.POSTv1Binding*3
14. Persist 3 v1Binding API objects
name: PodName
targetNodeName: node1
Kubelet-node1
WATCH v1Pod on node1
15.Podspecpushedtokubelet
CRI
16. Pull Image, Run Container,
PLEG, etc. Create v1Event and
update Pod status
17.PATCHv1Pod*3
17-1.PUTv1Event
17. Update 3 v1Pod API objects
status: PodInitializing -> Running
nodeName: node1
v1Event * N
17-1. Several v1Event objects created
e.g. ImagePulled, CreatingContainer,
StartedContainer, etc., controllers, scheduler
can also create events throughout the entire
reaction chain
CNI
18.Reļ¬ectorupdatesPods
19. ReplicaSet reconciliation loop
Update RS when ever a Pod enters
āRunningā state
v1Binding * 3
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior
53. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Reaction Chain
Kubenetes API Server & Etcd
1.POSTv1Deployment
Kubectl
v1Deployment
2. Persist v1Deployment API object
desiredReplica: 3
availableReplica: 0
3.Reļ¬ectorupdatesDep
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
4. Deployment reconciliation loop
action: create ReplicaSet
5.POSTv1ReplicaSet
v1ReplicaSet
6. Persist v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
7.Reļ¬ectorupdatesRS
8. ReplicaSet reconciliation loop
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
10. Persist 3 v1Pod API objects
status: Pending
nodeName: none
11.Reļ¬ectorupdatesPod
12. Scheduler assign
node to Pod
13.POSTv1Binding*3
14. Persist 3 v1Binding API objects
name: PodName
targetNodeName: node1
Kubelet-node1
WATCH v1Pod on node1
15.Podspecpushedtokubelet
CRI
16. Pull Image, Run Container,
PLEG, etc. Create v1Event and
update Pod status
17.PATCHv1Pod*3
17-1.PUTv1Event
17. Update 3 v1Pod API objects
status: PodInitializing -> Running
nodeName: node1
v1Event * N
17-1. Several v1Event objects created
e.g. ImagePulled, CreatingContainer,
StartedContainer, etc., controllers, scheduler
can also create events throughout the entire
reaction chain
CNI
18.Reļ¬ectorupdatesPods
19. ReplicaSet reconciliation loop
Update RS when ever a Pod enters
āRunningā state
20.PATCHv1ReplicaSet
21. Update v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0 -> 3
v1Binding * 3
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior
54. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Reaction Chain
Kubenetes API Server & Etcd
1.POSTv1Deployment
Kubectl
v1Deployment
2. Persist v1Deployment API object
desiredReplica: 3
availableReplica: 0
3.Reļ¬ectorupdatesDep
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
4. Deployment reconciliation loop
action: create ReplicaSet
5.POSTv1ReplicaSet
v1ReplicaSet
6. Persist v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
7.Reļ¬ectorupdatesRS
8. ReplicaSet reconciliation loop
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
10. Persist 3 v1Pod API objects
status: Pending
nodeName: none
11.Reļ¬ectorupdatesPod
12. Scheduler assign
node to Pod
13.POSTv1Binding*3
14. Persist 3 v1Binding API objects
name: PodName
targetNodeName: node1
Kubelet-node1
WATCH v1Pod on node1
15.Podspecpushedtokubelet
CRI
16. Pull Image, Run Container,
PLEG, etc. Create v1Event and
update Pod status
17.PATCHv1Pod*3
17-1.PUTv1Event
17. Update 3 v1Pod API objects
status: PodInitializing -> Running
nodeName: node1
v1Event * N
17-1. Several v1Event objects created
e.g. ImagePulled, CreatingContainer,
StartedContainer, etc., controllers, scheduler
can also create events throughout the entire
reaction chain
CNI
18.Reļ¬ectorupdatesPods
19. ReplicaSet reconciliation loop
Update RS when ever a Pod enters
āRunningā state
20.PATCHv1ReplicaSet
21. Update v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0 -> 3
22.Reļ¬ectorupdatesRS
23. Deployment reconciliation loop
Update Dev status with RS status changes
v1Binding * 3
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior
55. Copyright Ā© 2017 Hao Zhang All rights reserved.
Kubernetes Reaction Chain
Kubenetes API Server & Etcd
1.POSTv1Deployment
Kubectl
v1Deployment
2. Persist v1Deployment API object
desiredReplica: 3
availableReplica: 0
3.Reļ¬ectorupdatesDep
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
4. Deployment reconciliation loop
action: create ReplicaSet
5.POSTv1ReplicaSet
v1ReplicaSet
6. Persist v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
7.Reļ¬ectorupdatesRS
8. ReplicaSet reconciliation loop
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
10. Persist 3 v1Pod API objects
status: Pending
nodeName: none
11.Reļ¬ectorupdatesPod
12. Scheduler assign
node to Pod
13.POSTv1Binding*3
14. Persist 3 v1Binding API objects
name: PodName
targetNodeName: node1
Kubelet-node1
WATCH v1Pod on node1
15.Podspecpushedtokubelet
CRI
16. Pull Image, Run Container,
PLEG, etc. Create v1Event and
update Pod status
17.PATCHv1Pod*3
17-1.PUTv1Event
17. Update 3 v1Pod API objects
status: PodInitializing -> Running
nodeName: node1
v1Event * N
17-1. Several v1Event objects created
e.g. ImagePulled, CreatingContainer,
StartedContainer, etc., controllers, scheduler
can also create events throughout the entire
reaction chain
CNI
18.Reļ¬ectorupdatesPods
19. ReplicaSet reconciliation loop
Update RS when ever a Pod enters
āRunningā state
20.PATCHv1ReplicaSet
21. Update v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0 -> 3
22.Reļ¬ectorupdatesRS
24.PATCHv1Deployment
23. Deployment reconciliation loop
Update Dev status with RS status changes
25. Update v1Deployment API object
desiredReplica: 3
availableReplica: 0 -> 3
v1Binding * 3
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior