Let’s unbox Rancher 2.0 <v2.0.0>

Let’s unbox Rancher 2.0
<v2.0.0>
1
LINE Corporation
Verda2 Yuki Nishiwaki

1. Rancher 2.0 Overview
2. Deep dive Rancher Server
2.1. Rancher API
2.2. Rancher Controllers
2.3. Controller/Context
3. Dependent Library
3.1. Norman Framework
3.2. Kontainer-Engine
3.3. RKE
2
Agenda

Casts in Rancher 2.0
rancher
server
Node1 Node2 Node3
rancher
node-agent
rancher
node-agent
rancher
node-agent
rancher
cluster-agent1. Rancher Server
2. Rancher Cluster Agent
3. Rancher Node Agent
Parent Kubernetes
Child Kubernetes
deployed by Rancher
Child Kubernetes
deployed by Rancher
Parent k8s: k8s working with rancher
Child k8s: k8s deployed by rancher
3

rancher
server
Node1 Node2 Node3
rancher
node-agent
rancher
node-agent
rancher
node-agent
rancher
Parent Kubernetes
Child Kubernetes
deployed by Rancher
Child Kubernetes
deployed by Rancher
Parent k8s: k8s working with rancher
Child k8s: k8s deployed by rancher
4

About Rancher Server
➢ Provide the user with GUI/API
➢ Rancher Server depend on Kubernetes Cluster (Need to be deployed
before start to run Rancher Server)
○ If you try to run Rancher Server without Kubernetes, Rancher Server automatically try to
build all in one Kubernetes on same host
○ All data related to Rancher will be stored in Kubernetes as a CRD
○ Rancher run custom controllers to deploy/maintain multiple Kubernetes
➢ Everytime Rancher Server need to talk to deployed Kubernetes, Use
Rancher Cluster/Node Agent as a TCP Proxy Server via Websocket
➢ Multiple Rancher Server deployment is not available
○ HA Configuration is available but This is just run 1 Rancher Server on multi-node
Kubernetes environment behind Loadbalancer
5

About Rancher Server Implementation
➢ One binary cover all of following features….
○ Rancher API
○ Various Controller
○ gRPC Server of Kontainer-Engine
○ GUI of Rancher
➢ Depend on some other Rancher’s library/middleware
○ Norman
■ Used as a framework by API, Controller implementation
○ Kontainer Engine
■ Used by Controller to Deploy/Update k8s cluster on various environment
like GKE, EKS, Any Server with RKE….
○ RKE
■ Used by Kontainer Engine, API(Nodeconfigserver) to Deploy/Update k8s cluster
on Server
6

What Server does?
Server
API Controllers
CRD
Kind: Cluster
Node1 Node2 Node3
rancher
node-agent
rancher
node-agent
rancher
node-agent
rancher
cluster-agent
Child Kubernetes
deployed by Rancher
Child Kubernetes
deployed by Rancher
CRD
Kind: Node
All data stored
as CRD in k8s
Watch CRD
Deploy
Monitor Cluster/Sync Data
Call docker/k8s API via websocket, If need.
Don’t access to docker/k8s api directly from rancher server
Websocket session
Point 2 Point 3
Point 4
Point 5
Point 1
Provide API
7

What Server does?
Server
API Controllers
CRD
Kind: Cluster
Node1 Node2 Node3
rancher
node-agent
rancher
node-agent
rancher
node-agent
rancher
cluster-agent
Child Kubernetes
deployed by Rancher
Child Kubernetes
deployed by Rancher
CRD
Kind: Node
Provide unified access to multiple k8s cluster
Point 6
8

rancher
server
Node1 Node2 Node3
rancher
node-agent
rancher
node-agent
rancher
node-agent
rancher
Parent Kubernetes
Child Kubernetes
deployed by Rancher
Child Kubernetes
deployed by Rancher
9

About Rancher Agent
➢ All nodes need to run
➢ Periodically call /v3/connect/config
API and Check If node need to run
any container or create any file
➢ Provide TCP Proxy via websocket
(/v3/connect)
Rancher Node Agent Rancher Cluster Agent
➢ Cluster need to run 1 agent
➢ Provide TCP Proxy via websocket
(/v3/connect)
Use same binary and switch agent type by
environment variable(CATTLE_CLUSTER)
There are 2 types of Agent to run on Kubernetes deployed by Rancher
10

What Agent does?
Node Agent
Node A
Cluster Agent
Child Kubernetes
Node Agent
Node B
Parent Kubernetes
Server
Dialer API
(pkg/dialer)
RkeNodeConfig API
(pkg/rkenodeconfigserver)
Controllers
websocket session
(/v3/connect)
/v3/connect/config
Use session
For access
(k8s, docker)
Rancher Agent is basically to establish websocket to provide TCP Proxy
and just check NodeConfig periodically. Almost all configurations will be
done/triggered by controllers through websocket
Point 2
Establish websocket session
Point 1 Provide TCP Proxy
via websocket
Point 3
Check If file,container need to
create/run or not periodically
11

Rancher 2.0 overview summary
Almost all logics are in Rancher Server and Agent is just sitting as a TCP Proxy
Server in k8s deployed so that Rancher Server can use
➢ Rancher Server
○ All data for Rancher is stored as CRD in Kubernetes (translating Rancher’s resource into CRD)
○ Rancher’s API is kind of wrapper for Kubernetes API
○ Rancher have various controllers to watch CRD resources in parent k8s to deploy k8s
(Management Controllers)
○ Rancher have various controllers to watch resources including CRD in parent k8s to inject
some data to k8s deployed (User Controllers)
➢ Rancher Agent
○ Establish websocket to provide TCP Proxy
■ This is used when Rancher Server want to talk to Child Kubernetes
■ This is used when user want to call Kubernetes API
○ Check periodically if node need to create something file or run something container 12

Rancher 2.0 overview summary
Almost all logics are in Rancher Server and Agent is just sitting as a TCP Proxy
Server in k8s deployed for Rancher Server
● Rancher Server
a. All data for Rancher stored as CRD in Kubernetes (translating Rancher’s resource into CRD)
b. Rancher’s API is kind of proxy to Kubernetes API
c. Rancher have various controllers to watch CRD resources in parent k8s to deploy k8s
(Management Controllers)
d. Rancher have various controllers to watch CRD resources in parent k8s to inject some data
to k8s deployed (User Controllers)
e. Use websocket session to access deployed Node or K8s Cluster.
● Rancher Agent
a. Establish websocket to provide TCP Proxy
b. Check periodically if node need to create something file or run something container
If we want to know more about How Rancher maintain
Kubernetes Cluster, It’s enough to see just Rancher Server.
Because Agent is just to provide proxy.
13

2.1. Rancher API
3.3. RKE
14
Agenda

2.1. Rancher API
Server
API Controllers
CRD
Kind: Cluster
CRD
Kind: Node
Node1 Node2 Node3
rancher
node-agent
rancher
node-agent
rancher
node-agent
rancher
cluster-agent
Child Kubernetes
deployed by Rancher
All data stored
as CRD in k8s
Point 2
Point 1
Provide API
15

Rancher API Overview
➢ All data used/created by Rancher will be stored as a Kubernetes Resource
○ Which Kubernetes is used is depending on API Path
■ Parent Kubernetes
■ Child Kubernetes
➢ What you can do in Rancher API(Management API) can be done by calling
directly k8s API as well
➢ Rancher API allow you to create almost all resources type in K8s, not only CRD
○ Proxy request to k8s after some manipulation like adding annotation, label
➢ API can be classified into 5 types (It’s not officially classified)
16
2. Deep dive Rancher Server 2.1. Rancher API

5 types of API
Server
Controllers
API
Parent Kubernetes
➢ API can be classified into 5 types
➢ Some API is for only Agent
○ API for user
■ Management
■ Auth
■ K8s Proxy
○ API for agent
■ Dialer
■ RKE Node Config
Auth API
Management API
K8s Proxy API
Dialer API
RKE Node Config API
Main
/v3-public
/v3/token
/v3/
/k8s/clusters
/v3/connect
/v3/connect/register
/v3/connect/config Agent
User
17

Management API
Server
Controllers
API
Parent Kubernetes
Auth API
Management API
K8s Proxy API
Dialer API
RKE Node Config API
Main
/v3/
Child Kubernetes
deployed by Rancher
Create/Update/Get
Resource
Create/Update/Get
Resource
POST
/v3/cluster
POST
/v3/project/
<cluster-id>:<project-id>/pods
CRD
Cluster
PodAgent
depending on Path
Use TCP Proxy
Cluster Agent provide
18

Management API
➢ This is Main API for Rancher These resources are created by
Management API
19

Management API
➢ Provide CRUD for Almost All CRD(like cluster) and k8s resources(like pod)
➢ Use Norman Framework
○ Schema definition can be seen in following path
■ types/apis/management.cattle.io/v3/schema/schema.go
■ types/apis/cluster.cattle.io/v3/schema/schema.go
■ types/apis/project.cattle.io/v3/schema/schema.go
➢ According to resource type, Management API use proper data store
○ Use Parent Kubernetes for CRD like Cluster
○ Use Child Kubernetes for Kubernetes Core resource like Pod
➢ It doesn’t try to create actual resource and provisioning
○ This API just create cluster CRD in Parent Kubernetes
○ Provisioning is a responsibility of Controller
20

Auth API
Server
Controllers
API
Parent Kubernetes
Management API
K8s Proxy API
Dialer API
RKE Node Config API
Main
CRD
User
Auth API
/v3-public
/v3/token
Authenticate with User CRD
resource for Rancher API
Get Token
21

K8s Proxy API
Server
API
Parent Kubernetes
Management API
Dialer API
RKE Node Config API
Main
Child Kubernetes
deployed by Rancher
CRD
Token
Auth API
Authenticate with User CRD
resource for Rancher API
K8s Proxy API
Controllers
Websocket
Sessions
Agent
Call Child K8s API via TCP Proxy via Websocket
GET /k8s/clusters/<cluster>
/api/v1/componentstatuses
/k8s/clusters
GET
22

K8s Proxy API
➢ All request to Kubernetes deployed by Rancher is *authenticated* by this API
➢ If authentication succeeded, Proxy the request to k8s via websocket session to
Cluster Agent
➢ Use Impersonate-User, Impersonate-Group HTTP Headers to propagate User
information to k8s and always use same ServiceAccountToken to be created
when cluster deployed
➢ Authorization will be done by k8s deployed not K8s Proxy
○ Although Rancher’s Controller inject role,clusterrole and these binding information into k8s
deployed according to the value of roletemplate(CRD)
23

K8s Proxy Implementation
Node Agent
Node A ….
/k8s/clusters/<id of clusterA>
Cluster Agent
Cluster A
As you can see here, somehow
User access to k8s will be done
through Websocket session to
Cluster Agent
24
We use always same Service Account which is
created for each cluster.
Impersonate-User, Impersonate-Group HTTP
Headers are used to tell the user by k8s cluster

Dialer API
Server
API
Parent Kubernetes
Management API
RKE Node Config API
Main
Child Kubernetes
deployed by Rancher
Auth API
K8s Proxy API
Controllers
Websocket
Sessions
Agent
Dialer API
/v3/connect
/v3/connect/register
wss://<rancher-server>/v3/connect
CRD
ClusterRegisterToken
Start Provide
TCP Proxy via websocket
Check which cluster
Does agent belong to
Add websocket session for “K8s Proxy” and
Controllers to use TCP Proxy
25

Dialer API
➢ This is most important API from Agent perspective
➢ Dialer API(TunnelServer) provide endpoint of websocket and maintain
sessions for the controllers in Rancher
➢ All Cluster/Node Agent establish 1 websocket session and Provide TCP Proxy
➢ The websocket sessions established here will be used by Controllers and K8s
Proxy API
○ If agent failed to establish the websocket session, Any controllers in Rancher can not do
anything.
○ Keep it in mind Rancher also fail to proxy k8s access because k8s proxy API use websocket
session to Cluster Agent
26

Dialer API Implementation
Rancher
Controllers
Agent
wss://<rancher-server>/v3/connect
Node A
Provide
TCP Proxy
Interface to
- lookup proper websocket session
- start/maintain connection over websocket
TCP 127.0.0.1:443
Rancher
K8s proxy API
Various components of Rancher Server use websocket session to
access docker/k8s running on the target node
Cluster A
27

RKE Node Config API
Server
API
Parent Kubernetes
Management API
Main
Child Kubernetes
deployed by Rancher
Auth API
K8s Proxy API
Controllers
Agent
Dialer API
RKE Node Config API/v3/connect/config
CRD
Cluster
RKE
library
Check Config
Generate
NodeConfig
According to NodeConfig
- Create File
- Create container via docker
28

RKE Node Config API
➢ This is the API only for Node Agent on k8s deployed by RKE
○ If Node Agent running on GKE, EKS, This API always return HTTP Status Code 404
➢ This API return NodeConfig object which include followings
○ Files to create
○ Process to run (Container)
○ Certificate to use
➢ This Node Config object is generated based on one of attributes in Cluster
Object which is rancherKubernetesEngineConfig
29

RKE Node Config API Implementation
Node Agent
Node A
/v3/connect/config
Create file/Run container
based NodeConfig
Node
Config
● What process need to run
● What file need to create
● What certificate need to use
Get NodeConfig for Node A
rke generate
NodeConfig based on
rancherKubernetesEngineConfig which
is one of the attributes in Cluster CRD
Cluster A 30

2.1. Rancher API
3.3. RKE
31
Agenda

Server
API Controllers
CRD
Kind: Cluster
Node1 Node2 Node3
rancher
node-agent
rancher
node-agent
rancher
node-agent
rancher
cluster-agent
Child Kubernetes
deployed by Rancher
CRD
Kind: Node
Watch CRD
Deploy
Monitor Cluster/Sync Data
Call docker/k8s API via websocket, If need.
Don’t access to docker/k8s api directly from rancher server
Websocket session
Point 3
Point 4
Point 5
32

2.2.1 Rancher Controllers Overview
➢ Rancher API just create CRD resource in k8s
➢ Actual provisioning/configuration is done by Controller when detect
something change in k8s
➢ Rancher Controller watch resource@Child K8s not only resource@Parent K8s
➢ Many Controllers Rancher would run, altogether more than 40 Controllers
➢ Rancher Controller Implementation actively use Norman Framework
33
2. Deep dive Rancher Server 2.2. Rancher Controllers

API Controllers
Management
Controllers
Cluster(User)
Controllers
Workload Controllers
4 types of Controllers
Server
API
Controllers
Parent Kubernetes
Create
Resource
Watch
Resource
➢ Rancher Controllers can be classified
into 4 types of group
➢ Each group have own trigger to start
➢ Triggered when Server start
○ API Controllers
○ Management Controllers
➢ Triggered when new Cluster detected
○ Cluster(User) Controllers
○ Workload Controllers
34
2. Deep dive Rancher Server 2.2. Rancher Controllers 2.2.1 Overview

API Controllers
Management
Controllers
Cluster(User)
Controllers
API Controllers
Server
API
Controllers
Parent Kubernetes
Create
Resource
Watch
Resource
Configure
➢ Watch CRD resource related to API
Server Configuration
○ settings
○ dynamicschemas
○ nodedrivers
➢ Configure API server according to
the change of resource
35

API Controllers
Management
Controllers
Cluster(User)
Controllers
Management Controllers
Server
API
Controllers
Parent Kubernetes
Create
Resource
Watch
Resource
Provisioning/Update Cluster
Start Cluster(User),
Child Kubernetes
deployed by Rancher
➢ Watch Cluster/Node related CRD
➢ Provision/Update Cluster according
to the change of resource
➢ After Provision, Start Cluster(User),
Workload Controllers to start data
sync and monitor
36

API Controllers
Management
Controllers
Cluster(User)
Controllers
Cluster(User) Controllers
Child Kubernetes
deployed by Rancher
Server
Controllers
Parent Kubernetes
Create
Resource
Watch
ResourceCreate
Resource
Watch
Resource
Update/Create CRD
According to Child K8s
Update/Create
Resource including Pod
According to Parent K8s CRD
37
Cluster CRD
Secret
Alerts CRD
Status
Spec
Node
For updating CRD in Parent K8s
Resource Sync between Parent and Child K8s

API Controllers
Management
Controllers
Cluster(User)
Controllers
Child Kubernetes
deployed by Rancher
Server
API
Controllers
Parent Kubernetes
Create
Resource
Watch
ResourceCreate
Resource
Watch
Resource
The Simple Custom Controller to extend K8s
➢ Watch only resource@Child K8s
➢ Create/Update/Delete related
resource
➢ These Controller are more like
enhancing K8s feature itself
38

2.2.2. Important Management Controllers
Controllers
39
API Controllers
Management
Controllers
Cluster(User)
Controllers
Server
API
Create
Resource
Watch
Resource
Parent Kubernetes

Cluster Controller (pkg/controllers/management)
Overview
➢ Deploy actual Kubernetes Cluster by Kontainer-Engine
➢ After deployed Child Kubernetes, Do followings
○ Make sure cluster-agent/node-agent run on Child Kubernetes
○ Make sure Cluster(User Controllers) start against Child Kubernetes
➢ If the attributes of cluster has been changed, Update cluster by
Kontainer-Engine
➢ Update Status of Cluster based on Node’s(nodes.management.cattle.io)
information which is synced with Node’s(core.v1.nodes) information
40
2. Deep dive Rancher Server 2.2. Rancher Controllers 2.2.2. Management Controllers

Cluster Controller Implement (pkg/controllers/management)
Parent k8s
Server
Cluster Controller
(one of management controllers)
handlers
lifecycles
cluster-provisioner-controller
cluster-agent-controller
cluster-scoped-gc
cluster-deploy
cluster-stats
CRD
Cluster A
Informer
Child k8s
Cluster A
watch
Execute
deploy
Node Agent Cluster Agent
deploy
Alerts ingress ...
Run Cluster Controllers for Cluster A
CRD
Node A
CRD
Node B
Update Cluster Collect status
41

Cluster Controller Implement (pkg/controllers/management)
➢ [lifecycle] cluster-provisioner-controller (clusterprovisioner/provisioner.go)
○ Create: Initialize condition object of cluster. Call Create RPC of proper driver in Kontainer-Engine (rke, gke…)
○ Update: Call Update RPC of proper driver in Kontainer-Engine if RancherKubernetesEngineConfig(this is
generated by translating each driver config like rancherKubernetesEngineConfig) got changed from before
○ Remove: Call Remove RPC of proper driver in Kontainer-Engine
➢ [lifecycle] cluster-agent-controller (usercontrollers/controller.go)
○ Update: Re/Start All Controllers related to UserContext, User Only Context
○ Remove: Stop All Controllers related to UserContext, UserOnlyContext
➢ [lifecycle] cluster-scoped-gc (clustergc/cluster_scoped_gc.go)
○ Remove: Remove cluster-name finalizer from the objects cluster depending on roletemplate, project…
➢ [handler] cluster-deploy (clusterdeploy/clusterdeploy.go)
○ Have a responsibility to run cluster-agent/node-agent on child kubernetes
➢ [handler] cluster-stats (clusterstats/statsaggregator.go)
○ Update cluster status by collecting all machine’s status like the number of active pod, the memory consumed
Watch clusters.management.cattle.io(CRD) resources. Trigger followings
42
2. Deep dive Rancher Server 2.2. Rancher Controllers2. Deep dive Rancher Server 2.2. Rancher Controllers 2.2.2. Management Controllers

Node Controller (pkg/controllers/management)
Overview
➢ Deploy VM by docker-machine command if need
➢ Run rancher node-agent after deployed VM
➢ Rancher node-agent established websocket session and register own node
to the cluster
○ This will trigger Cluster Controller to provision Kubernetes related process
43

Node Controller Implement (pkg/controllers/management)
Parent k8s
CRD
Node A
CRD
Node B
Server
Node Controller
(one of management controllers)
handlers
lifecycles
node-controller
cluster-provisioner-controller
cluster-stats
nodepool-provisioner
Informer
watch
Execute
VM
Node Agent
Managements
Controllers
Cluster
Controller
NodePool
Controller
Just trigger handlers
Run Node Agent
Create VM If
doesn’t exist
docker-machine
trigger handlers
Create VM
44
Call wss://<server>/v3/connect/register
To register node into specific cluster

Node Controller Implement (pkg/controllers/management)
➢ [lifecycle] node-controller (node/controller.go)
○ Create: Initialize nodeConfig object and store it as a secret resource in k8s
○ Update: If Node doesn’t have the condition “Provisioned is True”, Try to create new VM on Public/Private
Cloud by docker-machine command
○ Remove: Just delete nodeConfig (secret) in k8s
➢ [handler] cluster-provisioner-controller (clusterprovisioner/provisioner.go)
○ Enqueue job to Cluster Controller
➢ [handler] cluster-stats (clusterstats/statsaggregator.go)
○ Enqueue job to Cluster Controller
➢ [handler] nodepool-provisioner (nodepool/nodepool.go)
○ Enqueue job to NodePool Controller
Watch nodes.management.cattle.io(CRD) resource. Trigger followings
45

Node Pool Controller (pkg/controllers/management)
Overview
➢ This Controller allow user to create multiple Nodes as a group
➢ If we specify the group having 3 nodes, This controller automatically create
Node CRD for each and Node Controller will do actual provisioning as usual
46

Node Pool Controller Implement (pkg/controllers/management)
Parent k8s
CRD(Kind: Node)
Node A
CRD(Kind NodePool)
Nodes For Cluster A
CRD
Node B
Server
Nodes For Cluster A
- Node A
- Role: Etcd
- Node B
- Role: Control
- Node C
- Role: Worker
Note:
Just image about
what nodepool defined
NodePool Controller
lifecycles
nodepool-provisionerInformer
CRD
Node C
Create If missing
Check
all Nodes exist
Watch
Execute
Confirmed exist
47

Node Pool Controller Implement (pkg/controllers/management)
➢ [lifecycle] nodepool-provisioner (nodepool/nodepool.go)
○ Update: Check If all node CRD in the nodepool are created or not. if not, create node
CRD in k8s
○ Remove: Delete node CRD described in the deleted nodepool in k8s
Watch nodepools.management.cattle.io(CRD) resources. Trigger followings
48

2.2.3. Important User Controllers
49
API Controllers
Management
Controllers
Cluster(User)
Controllers
Server
API
Controllers
Parent Kubernetes
Create
Resource
Watch
Resource
2. Deep dive Rancher Server 2.2. Rancher Controllers2. Deep dive Rancher Server 2.2. Rancher Controllers

Alerts Controller (pkg/controllers/user/alerts)
Overview
➢ User can define Alerts against Child K8s to check
○ Event, Resource status, Node mem/cpu…
➢ User can define notifiers like slack, email, webhook
○ This notifier definition is used when alert fired
➢ Prometheus Alertmanager is deployed @Child K8s for the part to send
notification
○ You can easily support any other notification system as long as Prometheus support
50
2. Deep dive Rancher Server 2.2. Rancher Controllers 2.2.3. User Controllers

Alerts Controller Implement (pkg/controllers/user/alerts)
Parent k8s
CRD
ClusterAlerts
CRD
Node
Server
ClusterAlerts Controller
handlers
cluster-config-syncer
cluster-alert-deployer
Informer
watch
Execute
Watchers
EventWatcherNodeWatcher
ProjectAlerts
Controller
Child k8s
alertmanager
Config.yaml
notification.tmpl
secret
mount
Pod
Maintain config files
According to ClusterAlerts CRD
Deploy alertmanager if need
….
Check Node violate
ClusterAlerts
Check Event
violate ClusterAlerts
Send Notify
Send Alert via email, slack
Same as
ClusterAlerts
51
2. Deep dive Rancher Server 2.2. Rancher Controllers2. Deep dive Rancher Server 2.2. Rancher Controllers 2.2.3. User Controllers

Alerts Controller Implement(1/2) (pkg/controllers/user/alerts)
➢ Watch clusteralerts.management.cattle.io resource @Parent K8s
○ [handler] cluster-alert-deployer
■ Make sure alert manager(prom/alertmanager/) is running with alert-manager
helper(rancher/alertmanager-helper) on Child k8s when notifier and alerts
resources are created
● Alertmanager-helper is just to watch config and reload alertmanager by API
● Alertmanager refer config file which is secret resource via secret mount and
have a responsibility to notify alert to notifiers(slack, webhook…)
○ [handler] cluster-config-syncer
■ Maintain config file on secret resource according to the change of ClusterAlerts
➢ Watch projectalerts.management.cattle.io resource @Parent K8s
○ [handler] project-alert-deployer
■ Same as “cluster-alerts-deployer”
○ [handler] project-alert-syncer
■ Same as “cluster-config-syncer” 52

Alerts Controller Implement(2/2) (pkg/controllers/user/alerts)
➢ Watch notifiers.management.cattle.io resource @Parent K8s
○ [handler] notifier-config-syncer
■ Same as “cluster-config-syncer”
➢ Run Alert Evaluation threads(called as Watcher) for Events/Pod/Node…..
○ One thread(watcher) for One resource type
○ Periodically evaluate the condition alert described with current status and If current
status broke the condition of alerts, Call http://<alert-manager service ip>/api/alerts to let
alert manager know new alert fired via websocket session to cluster-agent
53

NodeEndpointsController (pkg/controllers/user/endpoints)
Overview
➢ In Kubernete, It’s not easy to tell which address, port, protocol are exposed to
outside cluster on the spot
➢ This NodeEndpointsController check all resources possibly exposed to outside
and store all exposed address/port/protocol information into annotation
$kubectl get nodes rancher003 -o=custom-columns=NAME:.metadata.annotations
NAME
map[field.cattle.io/publicEndpoints:[
{"nodeName":"local:machine-z6qww","addresses":["rancher003"],"port":30080,"protocol":"TCP",
"serviceName":"cattle-system:cattle-service","allNodes":true},
{"nodeName":"local:machine-z6qww","addresses":["rancher003"],"port":30443,"protocol":"TCP",
"serviceName":"cattle-system:cattle-service","allNodes":true}
] 54

NodeEndpointsController Implement
(pkg/controllers/user/endpoints)
Parent k8s
CRD(Kind: Node)
Node A
Server core.v1.Node Controller
handlers
nodeEndpointControllerInformerWatch
Execute
Child k8s
Core.v1.Node
Node A
Service A
Lookup Node Name
Corresponding to core.v1.node Node A
Get All Service exposed outside
Update field.cattle.io/publicEndpoints
to store all exposed endpoint
55
1.
2.
3.

NodeEndpointsController Implement
(pkg/controllers/user/endpoints)
➢ Watch core.v1.nodes resource @ Child k8s
○ [handler] nodesEndpointController
■ Check all endpoints exposed to outside child k8s cluster by nodePort, LB service or
hostport pod and Store these endpoints with “field.cattle.io/publicEndpoints”
annotation in core.v1.nodes resource in child k8s
56

EventSyncer (pkg/controllers/user/eventssyncer)
Overview
➢ This is watching Event resource@Child K8s and Store all of events into
Parent K8s as a ClusterEvent CRD
○ This doesn’t update event status/message and always create new event
➢ This ClusterEvent CRD is deleted after 24 hours passed from creation by
/pkg/controllers/management/clusterevents/clustereventscleanup.go
➢ This Controller is not enabled at default now because of scaling issue
○ https://github.com/rancher/rancher/issues/11771
○ If you want to enable this, it’s better to modify this logic to store external database
57

EventSyncer Implement (pkg/controllers/user/eventssyncer)
Parent k8s
CRD
ClusterEvents
Server
core.v1.Event Controller
handlers
events-syncerInformer
Execute
Child k8s Event
Watch
Translate event into
clusterevents and Create
clustereventscleanup
(pkg/controllers/management/)
Goroutine
Watch
Delete clusterevent after 24
hours passed from create
58

EventSyncer Implement (pkg/controllers/user/eventssyncer)
➢ Watch core.Event resource @ Child K8s
○ [handler] events-syncer
■ Create/Update ClusterEvents corresponding to event object in child k8s.
This ClusterEvents CRD resource are deleted after 24 hours passed by
pkg/controllers/management/clusterevents/clustereventscleanup.go
○ This Controller is not enabled by default now because of scaling issue
■ https://github.com/rancher/rancher/issues/11771
59

HealthSyncer (pkg/controllers/user/healthsyncer)
Overview
➢ Start thread to check component status for Child k8s periodically
○ Call <Child K8s Endpoint>/v1/componentstatuses periodically
○ Store components status into Status.ComponentStatuses in v3.Cluster object @Parent K8s
○ If This controller failed to get component status from Child k8s, Stop All UserControllers and
periodically check if Cluster got back to alive or not.
60

HealthSyncer Implement (pkg/controllers/user/healthsyncer)
Parent k8s
CRD
Cluster A
Server
healthSyncer
Goroutine
Child k8s
ClusterAlert Controller Notifier Controller ...
GET
Stop if failed to
get status
Re-start
If status got alive
Update
Status.ComponentStatuses
61

Cluster(Project)Logging Controller
(pkg/controller/user/logging/)
Overview
➢ User can specify where all container logs should be sent to
➢ The logs will be sent by fluentd which is deployed by this controller
➢ If user specify the embedded Elasticsearch, Elasticsearch will be deployed
onto Child K8s as well
62

Cluster(Project)Logging Controller Implement
Parent k8s CRD
ClusterLogging
Server
Child k8s
ClusterLogging Controller
lifecycle
cluster-logging-controllerInformer
Execute
ProjectLogging Controller
Almost same as
ClusterLogging Controller
Watch
Daemonset
cluster.conf
ConfigMap
project.conf
ConfigMap
/var/lib/docker/containers/
/var/log/containers/
/var/log/pods
/var/lib/rancher/rke/log
HostPath
Mount
Mount
Mount
Deploy
Update
Out of Scope
Send logs
63

Cluster(Project)Logging Controller Implement
➢ Watch clusterloggings.cattle.io resource @Parent K8s
○ [lifecycle] cluster-logging-controller
■ Create: Deploy fluentd daemon set and initial config
● Create 2 ConfigMap @Child K8s. One is for to store cluster.conf, the other is for to
store project.conf.
● Deploy daemonset @Child K8s. rancher/fluentd docker image is used. This
daemonset have following hostpath volume
○ /var/lib/docker/containers (created by docker)
○ /var/log/containers, /var/log/pods (created by k8s)
○ /var/lib/rancher/rke/log (created by rke)
■ Updated: Update ConfigMap (cluster.conf) according to spec of clusterlogging resource.
Configuration template can be seen in
pkg/controllers/user/logging/generator/cluster_template.go
■ Remove: Delete the namespace including all logging related resources
ProjectLogging Controller
have very similar behaviour
64

User Node Remove (pkg/controller/user/noderemove/)
Overview
➢ This controller is in charge of deleting core.v1.Node when
nodes.management.cattle.io resource is deleted
Implementation
➢ Watch nodes.management.cattle.io@ Parent K8s
○ [lifecycle] user-node-remove
■ Remove: delete core.Node resource @ Child K8s corresponding to Node CRD
65

NodeSyncer (pkg/controller/user/nodesyncer/)
Overview
➢ This controller is in charge of updating nodes.management.cattle.io CRD
resource so that it has same status as core.v1.Node
core.v1.Node@Child K8s Nodes.management.cattle.io CRD@Parent K8s
Spec Spec.InternalNodeSpec
Status Status.InternalNodeStatus
Annotations Status.NodeAnnotation
Labels Status.NodeLabels
Name Status.Nodename
Translate
66

NodeSyncer Implement (pkg/controller/user/nodesyncer/)
Parent k8s
CRD
Node A
Server
Child k8s
Core.v1
Node A
core.v1.Node Controller
handlers
Informer
nodeSyncer
management.cattle.io.v3.Node Controller
Informer
handlers
machinesSyncer
machineLabelSycner
Watch
Execute Execute
Watch
Execute
Get Information
Store information
into Node CRD Update Label,
Annotation
Check Spec.
Annotation and Label
67

NodeSyncer Implement(1/2) (pkg/controller/user/nodesyncer/)
➢ Watch core.Node @Child K8s
○ [handler] nodeSycner
■ Just enqueue job to NodeController
➢ Watch nodes.managements.cattle.io @ Parent K8s
○ [handler] machinesSyncer
■ Check all core.Node resources @Child K8s and If there is missing
nodes.management.cattle.io resource corresponding to core.Node @Parent K8s, Create
new nodes.management.cattle.io. If there is nodes.management.cattle.io resource
corresponding to no core.Node, Try to delete that nodes.management.cattle.io resource
■ Update nodes.management.cattle.io resource
● Spec (core.Node) -> Spec.InternalNodeSpec (nodes.management.cattle.io)
● Status (core.Node) -> Status.InternalNodeStatus (nodes.management.cattle.io)
● Calculate requested resource by pod -> Status.Request (nodes.manageme…)
● Annotations (core.Node) -> Status.NodeAnnotation (nodes.manageme…
● Labels (core.Node) -> Status.NodeLabels (node.manageme…
● Name (core.Node) -> Status.NodeName (node.manageme… 68

NodeSyncer Implement(2/2) (pkg/controller/user/nodesyncer/)
➢ Watch nodes.managements.cattle.io @ Parent K8s
○ [handler] machinesLabelSyncer
■ If nodes.managements.cattle.io resource@Parent K8s have Spec.DesiredNodeLabels
or Spec.DesiredNodeAnnotation, Try to make sure core.Node@Child K8s have
completely same Annotation or Label
69

RBAC Controllers (pkg/controller/user/rbac)
RoleTemplate
Overview
➢ This controller allow user to create roletemplate which is corresponding
to clusterrole in K8s
➢ Roletemplate can be assigned to User in Rancher and this information will
propagate to all clusters so that all cluster have same RBAC information
70

RBAC Controllers (pkg/controller/user/rbac)
You can choose roletemplate
Choose user
to assign roletemplate
Adding Cluster Member
= Assign specific roletemplate to user
71

RBAC Controllers Implement (pkg/controller/user/rbac)
Parent k8s
CRD
ClusterRoleTemplateBinding
Server
Child k8s
ClusterRole
RoleTemplate Controller
Informer
lifecycle
cluster-roletemplate-sync
ClusterRoleTemplateBinding Controller
lifecycle
cluster-crtb-syncInformer
CRD
RoleTemplate
Watch
Watch
Execute
Execute
ClusterRoleBindings
Check if there is binding resource
to refer RoleTemplate
Update
Create If need
Create If need
Get
Get
Get
72

RBAC: RoleTemplate Controller Implement
(pkg/controller/user/rbac)
➢ Watch roletemplate@Parent K8s
○ [lifecycle] cluster-roletemplate-sync
■ Update:
● Check If there is project role template binding referring to the roletemplate
● Check If there is cluster role template binding referring to the roletemplate
● If there is, Make sure roletemplate’s rule is same as clusterrole corresponding
to the roletemplate @Child K8s
■ Remove:
● Remove clusterrole @Child K8s corresponding to deleted roletemplate
73

RBAC: ClusterRoleTemplateBinding Controller
Implement (pkg/controller/user/rbac)
➢ Watch clusterroletemplatebindings.management.cattle.io @Parent K8s
○ [lifecycle] cluster-crtb-sync
■ Create: Make sure followings
● There is clusterrole corresponding to roletemplate
clusterroletemplatebindings refer to @Child K8s
● There is clusterrolebindings corresponding to clusterroletemplatebindings
@Child K8s
■ Update: same as Create
■ Remove: Make sure delete clusterrolebindings corresponding to
clusterroletemplatebindings @Child K8s
74

Secret Controller (pkg/controller/user/secret)
Overview
➢ User can create secret resource for all namespaces thanks to this Controller
➢ If user create secret with all namespaces, this controller watch the
namespaces resource@Child K8s and create secret for new namespace
75

Secret Controller Implement (pkg/controller/user/secret)
Parent k8s Secret
Server
Child k8s Namespace
Namespace Controller
handlers
Informer
SecretsController
Secret Controller
Informer
handlers
secretsController
Watch
Execute
Watch
Execute
CreateCheck If the secret have
field.cattle.io/projectId annotation
Secret
Create
Get All secret in project
Check If the namespace have
field.cattle.io/projectId annotation
76

Secret Controller Implement (pkg/controller/user/rbac)
➢ Watch core.Namespaces @Child K8s
○ [handler] secretsController
■ If the namspace have annotation (field.cattle.io/projectId), Create all secrets under
<project-name> namespace@Parent K8s into Child K8s
➢ Watch core.Secret @Parent K8s
○ [lifecycle] secretsController
■ Create: If the secret have annotation (field.cattle.io/projectId), Create the secret into
specified namespace in Child K8s.
■ Update: Same as Create
■ Remove: Delete secret in Child K8s as well
77

2.2.4. Important Workload Controllers
78
API Controllers
Management
Controllers
Cluster(User)
Controllers
Server
API
Controllers
Parent Kubernetes
Create
Resource
Watch
Resource

ExternalIPServiceController (pkg/controllers/user/externalservice)
Overview
➢ Headless Service(Cluster-IP: None) is usually used to balance traffic
between Pods by selector
➢ This is to extend Headless Service to support external IP which is
maintained manually in usual k8s
Endpoint having all ip addresses as a
subset are created 79
2. Deep dive Rancher Server 2.2. Rancher Controllers2. Deep dive Rancher Server 2.2. Rancher Controllers 2.2.4. Workload Controllers

ExternalIPServiceController Implement
(pkg/controllers/user/externalservice)
Parent k8s
Server
Child k8s Service
Service Controller
Informer
handlers
externalIPServiceController
Watch
Execute
Endpoint
Update subset so at to
have all ip address described in annotation
“field.cattle.io/ipAddresses”
Check
“Filed.cattle.io/ipAddresses” annotation
80
2. Deep dive Rancher Server 2.2. Rancher Controllers2. Deep dive Rancher Server 2.2. Rancher Controllers2. Deep dive Rancher Server 2.2. Rancher Controllers 2.2.4. Workload Controllers

ExternalIPServiceController Implement
(pkg/controllers/user/externalservice)
➢ Watch core.v1.Service resource @ Child k8s
○ [handler] externalIpServiceController
■ Check “field.cattle.io/ipAddresses” annotation on the service resource and create
endpoint object having ip addresses described in annotation
81

Rancher Ingress Controller (pkg/controllers/user/ingress)
Overview
➢ Kubernetes Ingress rule allow you to specify only service as a backend
➢ But Rancher allow you to use Deployment/Pods as a backend for Ingress rule
○ By letting this controller automatically create service for these deployments/Pods
Workload is actually Pod, Deployment...
82

Rancher Ingress Controller Implement
(pkg/controllers/user/ingress)
Parent k8s
Server
Child k8s
Ingress
Ingress Controller
Informer
handlers
ingressWorkloadController
Watch
Execute
Check Ingress definition created by Rancher by
annotation “field.cattle.io.ingress/state” including
info about target workload
Service
Create Service with NodePort having
selector workload_ingress_***
Workload Controller (pkg/controllers/user/workload/workload_common.go)
Informer
handlers
syncDeployments
syncReplicasets
….
Deployments
Pod
ingressEndpointController
Add workload label for ingress selector
Watch
Execute
Get info to identify pod
Execute
83

Rancher Ingress Controller Implement
(pkg/controllers/user/ingress)
➢ Watch extensions.ingress resources @ Child K8s
○ [handler] ingressWorkloadController
■ Create service for deployment(seen as workload in Rancher) specified by ingress
resource
● Only ingress resource created by Rancher API
(/v3/project/<cluster-id>/<cluster-id>:<project-name>/ingress) is the target
resource this controller deal with.
● We can tell ingress resource is created by Rancher API or not with annotation
“field.cattle.io.ingress/state”
84

2.2.5. Controllers We didn’t cover this time
Management Controllers:
* pkg/controllers/management/auth
* pks/controllers/management/catalog
* pkg/controllers/management/compose
* pkg/controllers/management/nodedriver
* pkg/controllers/management/podsecuritypolicy
Cluster(User) Controllers:
* pkg/controllers/user/helm
* pkg/controllers/user/namespacecompose
* pkg/controllers/user/networkpolicy
* pkg/controllers/user/nslabel
* pkg/controllers/user/pipeline
* pkg/controllers/user/usercompose
Workload Controllers:
* pkg/controllers/user/dnsrecord
* pkg/controllers/user/targetworkloadservice
* pkg/controllers/user/workload
API Controllers:
* pkg/api/controllers/dynamicschema
* pkg/api/controllers/settings
* pkg/api/controllers/whitelistproxy
85

2.1. Rancher API
3.3. RKE
86
Agenda

Context represents Controller groups
Contexts
ScaledContext
ManagementContext
UserContext
UserOnlyContext
Credential
For parent
Credential
For parent
Credential
For child
Credential
For child
➢ Each Context have a group of controllers to run
➢ Each Context have kubernetes config for a certain
environment
○ ScaledContext for Parent k8s
○ ManagementContext for Parent k8s
○ UserContext for Child k8s and Parent k8s
○ UserOnlyContext for Child k8s
➢ UserContext and UserOnlyContext are created for
each Child k8s
➢ All contexts except for UserOnlyContext have
dialer to “Node/Cluster Agent’s TCP Proxy Server”
and Use it to access Child k8s
UserContext
UserOnlyContext
Credential
For child
Credential
For child
87
2. Deep dive Rancher Server 2.3. Controller/Context

Context/Controllers Relation
Controllers
API Controllers
Management
Controllers
Cluster Controllers
Contexts
ScaledContext
ManagementContext
UserContext
UserOnlyContext
Clientconfig
For parent
Clientconfig
For parent
Clientconfig
For child
Clientconfig
For child
They are watching api related CRD in parent
k8s and do whatever action like replacing
certificates.
They are started when rancher server start.
All controllers can be seen in
pkg/api/server/server.go.
They are watching deployed cluster/node
related CRD in parent k8s and do whatever
action like actual provisioning cluster, adding
node...
They are started when rancher server start.
pkg/controllers/management/controller.go
After deployed cluster, These controllers have a
responsibility to monitor and sync data for the
cluster and nodes, They are watching many
objects in both of k8s cluster parent and child.
They are re/started when cluster change is
detected by usercontroller which is one of the
management controllers, which means we have
active controller for each cluster
pkg/controllers/user/controllers.go
Cluster Controllers
(User Controllers)
UserContext
UserOnlyContext
Clientconfig
For child
Clientconfig
For child They are just watching resource in child k8s.
They don’t require access to Parent k8s and
aim for extending existing in child k8s. Trigger
to start is same as Cluster(User) controllers
pkg/controllers/user/controllers.go
88

ManagementContext(Context)
Start
How/When Rancher Start Controller?
ScaledContext (Context)
Clientconfig
For parent
Start
➢ Context have a responsibility to run Controllers
➢ Context.Start is the function start controllers be in charged of
Clientconfig
For parent
Is it same?
vendor/github.com/rancher/types/config/context.go
Execute Start
Execute Start
89

ManagementClient B
ManagementContext(Context)
Start
Register Concept
ScaledContext (Context)
Clientconfig
For parent
Start
Clientconfig
For parent
ManagementClient A
starters
nodeController …
Management Controllers
nodeController
userController
…
…
Register
Start only controllers in starters
Execute Start
Execute Start
…ControllerA
different
90

Flow to start Controllers
Actual Example: app/app.go
Generate Context Register Controllers
Context Start
(Start Controllers)
➢ Always create context when start controllers in Rancher
➢ For management, scaled controllers are started when
start rancher server. See bellow code
91

2.1. Rancher API
3.3. RKE
92
Agenda

3.1. Norman (https://github.com/rancher/norman)
➢ Rancher actively use this framework
➢ Provide API Framework working with Kubernetes as a backend storage
○ All Data will be stored in k8s as a CRD resource (technically we can configure other storage as well)
➢ Provide easy way to build Controller on Kubernetes (Wrapper for client-go)
Something App
Norman
API Part
Controller Part
Create Resource as CRD
CRD (Test)
Resource A
CRD (Test)
Resource B
Watch the changesReturn If app stored resource
information into k8s or not
Do whatever you want
asynchronously
93

3.1.1. Norman API Part
Something App
Norman
API Part
Controller Part
CRD (Test)
Resource A
CRD (Test)
Resource B
Create Resource as CRD
Return If app stored resource
information into k8s or not
94
3. Dependent Library 3.1 Norman Framework

API Server
➢ This is Simple API Server provide ServeHTTP function
➢ API Server need API Schema(Concept in Norman) having actual logic
➢ We can register multiple API Schemas
➢ API Server lookup proper schema from requested URL and HTTP Method
95
3. Dependent Library 3.1 Norman Framework 3.1.1 API Part

➢ Each API schema can define
○ How/Where the data need to be stored
■ Rancher use CRD in k8s as a datastore
○ The actual logic for each action
(CREATE, DELETE, UPDATE, LIST)
○ Which HTTP Method this schema can accept
API Schema
https://github.com/rancher/norman/blob/master/types/types.go
96

API Schema Generation
➢ 1 Struct represent Rancher’s resource can be used for…
○ Generating API Schema be used by norman.APIServer (norman/api/server.go)
■ https://github.com/rancher/norman/blob/aecae32b4ae6b73b9945cdedef5a5b0dafa11973/types/r
eflection.go#L75
○ Generating CRD definition (technically from API Schema)
■ https://github.com/rancher/norman/blob/aecae32b4ae6b73b9945cdedef5a5b0dafa11973/store/c
rd/init.go#L122
rancher/vendor/github.com/rancher/types/apis/
management.cattle.io/v3/cluster_types.go
API Schema
For Cluster
CRD
For Cluster
Generate
Use as Store
97

How APIServer lookup schema?
Norman API Server
ServeHTTP
/v3/
/v3/project
Schema.Version.Path
cluster
node
reflect.TypeOf(<type A>).Name()
Node Schema
A Schema
POST
/v3/cluster
CreateHandler DeleteHandler
…..
Cluster Schema
2. Check if method is allowed or not
3. Execute proper Handler
according to HTTP method, URL Parameter
CollectionMethods:
GET POST
1. Choose correct schema
98

Rancher Use Norman in
Management API
Use norman framework.
Difficult to know how
management API implemented
without understanding Norman
1 API Server instance have multiple
schemas which have different data
store.
- Use parent k8s
(managementstored)
- Use child k8s
(userstored)
All custom handlers other than
default handler Norman defined are
stored in this directory.
About handler of Norman, Plz see
explanation page of Norman
99

3.1.2. Norman Controller Part
Something App
Norman
API Part
Controller Part
CRD (Test)
Resource A
CRD (Test)
Resource B
Watch the changes
Do whatever you want
asynchronously
100

The concepts around Controller
➢ Generic Controller
○ Implement basic Controller with client-go
○ Allow us to build Custom Controller by just following
■ Passing K8s Client for specific resource to controller
■ Passing the function(handler) to controller. This handler is executed when specific
etcd key changed
● Do whatever you want
➢ 2 Types of Handler
○ Normal Handler
○ LifeCycle Handler
101
3. Dependent Library 3.1 Norman Framework 3.1.2 Controller Part

Generic Controller
etcd key changed
○ Normal Handler
102

Generic Controller Internal
Generic Controller
(norman/controller/generic_controller.go)
cache.SharedIndexInformer
(k8s.io/client-go/tools/cache)
cache.ListWatch
workqueue.RateLimitingInterface
(k8s.io/client-go/util/workqueue)
Client
handler
handler
handler
handler
handlers
handler
handler
handler
Updated key
Deleted key
Added key
Watch specific key
By using Client
When adde/updated/created
Evaluate all registered
handlers With key
You can make your own controller by
defining Client and handlers
Name1
Name2
Name3
103

How Rancher Use: Generic Controller
vendors/github.com/rancher/
types/apis/management.cattle.io/v3/zz_generated_node_controller.go
type nodeController struct {
controller.GenericController
}
~~omit~~ Each Controller embed GenericController struct and override few function (e.g.
AddHandler)
This Controller is created when nodeClient.Controller method executed in the first
time like “management.Management.Clusters("").Controller()” (see
pkg/controllers/management/clusterprovisioner/provisioner.go). Rancher usually
named the function generating Controller and pushing controller to starter list as
“Register” function
This is started by Context.Start (see app/app.go in rancher repository) 104
<apiGroup>/<version>/<resource_name>
3.1.2 Controller Part

How Use:
Prepare XXXController for each Resource Type
nodeController
Generic Controller
(Norman)
Handlers
clusterController
Generic Controller
(Norman)
Handlers
XXXController
Generic Controller
(Norman)
Handlers
types/apis/management.cattle.io/v3/zz_generated_XXX_controller.go
・・・
pkg/controllers/ managements/ user/
XX.go ZZ.go
Just Add Handlers to
the controller to watch 1
resource type
105

How Ranche Use:
Prepare XXXController for each Resource Type
nodeController
Generic Controller
(Norman)
Handlers
clusterController
Generic Controller
(Norman)
Handlers
XXXController
Generic Controller
(Norman)
Handlers
types/apis/management.cattle.io/v3/zz_generated_XXX_controller.go
・・・
pkg/controllers/ managements/ user/
XX.go ZZ.go
Just Add Handler or
LifeCycle to the
controller watching 1
resource type
There is directory seems defining controllers (pkg/controllers/) in
rancher repository.
But the code under this directory is just to add handler or lifecycle
to the proper Controller to watch 1 resource type.
And some handler/lifecycle have controller-ish name like
cluster-provisioner-controller but this is actually just name of
handler. Keep it mind.
106

How Rancher Use: Override AddHandler function
types/apis/management.cattle.io/v3/zz_generated_node_controller.go
func (c *nodeController) AddHandler(name string, handler NodeHandlerFunc) {
c.GenericController.AddHandler(name, func(key string) error {
obj, exists, err := c.Informer().GetStore().GetByKey(key)
~~omit~~
return handler(key, obj.(*Node))
})
}
Override function so as to pass object data itself as well as key to
handler function. That’s why all handler registered by Rancher
can get object not only just etcd key
This function is actually
called by Generic
Controller(Norman)
This handler is actual
function rancher would
register
107

Normal Handler
etcd key changed
○ Normal Handler
108

What’s Normal Handler
➢ Handler Function is executed by GenericController when changes detected
in Kubernetes
➢ Handler Function need to have a argument
(key name of the data which is changed in etcd)
○ Rancher Override AddHandler so that handler function can have 2 argument (key name,
object data)
➢ The registered handlers are executed when following event happened
○ Monitored Keys is “Created”
○ Monitored Keys is “Deleted”
○ Monitored Keys is “Updated”
=> We can not know which kind of event triggers handler.
By Created or Deleted or Updated? 109

How Rancher Use: Register Handler
pkg/controllers/management/clusterprovisioner/provisioner.go
func Register(management *config.ManagementContext) {
~~ omit ~~
mngNodes := management.Management.Nodes("")
mngNodes.AddHandler("cluster-provisioner-controller", p.machineChanged)
~~ omit ~~
}
~~ omit ~~
func (p *Provisioner) machineChanged(key string, machine *v3.Node) error {
Register “p.machineChanged” function with the name of “cluster-provisioner-controller”
as a handler
Thanks to override of AddHandler, callback function will get object itself not only just key
110

LifeCycle Handler
etcd key changed
○ Normal Handler
111

What’s LifeCycle Handler
➢ Kind of framework of handler function for Controller
➢ Need to Implement following functions for Lifecycle struct
○ Create, Remove, Updated
➢ Generic Controller expect Lifecycle struct to be wrapped by
ObjectLifecycleAdapter(norman/lifecycle/object.go) and developer to
register ObjectLifecycleAdapter.sync function as a handler
➢ The check for object is already created(initialized) or not is judged by extra
annotation (lifecycle.cattle.io/create.<lifecycle name>)
Generic Controller
Handlers
ObjectLifecycleAdapter
sync
Lifecycle
Create
Updated
Removed
Register sync function as
Normal Handler
Execute 112

Inside ObjectLifecycleAdapter.sync function
func (o *objectLifecycleAdapter) sync(key string, obj runtime.Object) error {
~~ omit ~~
metadata, err := meta.Accessor(obj)
~~ omit ~~
if cont, err := o.finalize(metadata, obj); err != nil || !cont {
return err
}
if cont, err := o.create(metadata, obj); err != nil || !cont {
return err
}
copyObj := obj.DeepCopyObject()
newObj, err := o.lifecycle.Updated(copyObj)
if newObj != nil {
o.update(metadata.GetName(), obj, newObj)
}
return err
}
Get Metadata from object
Check this object is deleted or not.
If deleted, call o.lifecycle.Remove.
Finalize and return
Check if object is already created or not by annotation
“lifecycle.cattle.io/create.<lifecycle name>” == true.
If not, add finalizer “controller.cattle.io/<lifecycle name>”.
And Call o.lifecycle.Create and then return
Simply just execute o.lifecycle.Updated
vendors/github.com/rancher/norman/lifecycle/object.go
113

How Rancher Use: Register Add LifeCycle
func Register(management *config.ManagementContext) {
p := &Provisioner{~~ omit ~~}
p.Clusters.AddLifecycle("cluster-provisioner-controller", p)
}
func (p *Provisioner) Remove(cluster *v3.Cluster) (*v3.Cluster, error) {
~~ omit ~~}
func (p *Provisioner) Updated(cluster *v3.Cluster) (*v3.Cluster, error) {
~~ omit ~~}
func (p *Provisioner) Create(cluster *v3.Cluster) (*v3.Cluster, error) {
~~ omit ~~}
pkg/controllers/management/clusterprovisioner/provisioner.go
Add Lifecycle object as handler
Remove function is executed
only when object is deleted
Updated function is executed
only when object is updated
Create function is executed
only when object is created 114

2.1. Rancher API
3.3. RKE
115
Agenda

(https://github.com/rancher/kontainer-engine)
Coming Soon...
116

2.1. Rancher API
3.3. RKE
117
Agenda

3.3. RKE (https://github.com/rancher/rke)
Coming Soon...
118

Concern about Rancher 2.0
In the context of backend for Verda k8s as a Service
119

1 Rancher Server binary have ton of features
➢ It’s tough to operate
➢ We want to use only features we need and trace the changelog.
➢ Difficult performance tuning
○ We can not run multiple instances for specific feature even if that feature allow us to run
multiple instances
■ E.g. Run 5 process for K8s Proxy API
■ E.g. Run 10 process for rkenodeconfig API and so on
=> We will separate some of feature from a server binary and Run separately
=> We will disable unneeded Controllers
120

Scalability Problem to manage k8s deployed
➢ Current design won’t work with Active-Active HA
○ There is no scheduling logic in Rancher
■ Currently only 1 Rancher Server can manage all of k8s cluster
■ Need a extra layer before actual managing cluster to scale
● Like Pod Scheduler and Kubelet relations
○ Controller depend on websocket session to Rancher Server
■ Even If we can redesign Rancher Server with scheduling concept and Active-Active,
We still need to think about how scheduled controller talk to Agent
=> We will have simple scheduling concept in front of Rancher Server
121

K8s Proxy API depend on Websocket to Agent
➢ Even If Rancher Server can access to k8s deployed, Currently Rancher
Server proxy k8s API request via websocket session to Cluster Agent
➢ This design prevent us from running multiple K8s Proxy, even if we
succeeded in separating this process from Rancher Server binary
=> The load of K8s Proxy API is most difficult to predict
because this is really depending on How the user use k8s API
=> We separate this feature from a server binary and run multiple process
by modifying current way which is using websocket session
122

How I feel overall in Rancher 2.0
➢ Less document and Reading Code is only way to know well
➢ I can see some effort to make it easy to deploy Rancher Server
○ Less precondition to run Rancher Server
■ Rancher Server depend on k8s but doesn’t define it as precondition (automatically
detect missing k8s and deploy all in one k8s working with)
○ Less precondition to deploy k8s
■ Even for the environment having NAT
○ But this effort prevent from scaling
➢ If we need scalability like maintaining 1000 clusters, we need additional
consideration
○ Separate the feature from 1 massive binary. Run multiple instances.
○ Use other data store from Kubernetes for some resources.
○ Ask yourself if you need all features or not and If not, let’s disable after understood impact
123

How I feel overall in Rancher 2.0
➢ There are some interesting Controllers like alert, logging, eventsync...
➢ Need to know Norman Framework If you want to know more Rancher
➢ Easy to modify/add Rancher behaviour thanks to Norman Framework
○ Change datastore for specific API resource thanks to Norman API Schema
○ Add Custom Controller thanks to Norman Generic Controller
124

Enhancement Plan for Rancher 2.0
125

Server
Server
K8s Proxy
K8s Proxy
XXX API
After start service, see the
performance and consider
to separate/scaleWithout touching anything
If we can not scale
Rancher Server anymore,
we will add one more
cluster.
Phase 1 Phase 2
Rancher Scalability Improvement
Scheduling
Other
Datastore
Use other datastore
for some data
Extra Monitoring
Enhance
Monitoring
Point 2
Point 1
Point 3
Point 4
126
Custom
API
Custom
API

Appendix: Kind of Code Structure
I straighten my understandings as a diagram.
It’s available in (https://github.com/ukinau/rancher-analyse)
127

Let’s unbox Rancher 2.0 <v2.0.0>

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Let’s unbox Rancher 2.0 <v2.0.0>

Semelhante a Let’s unbox Rancher 2.0 <v2.0.0> (20)

Mais de LINE Corporation

Mais de LINE Corporation (20)

Último

Último (20)

Let’s unbox Rancher 2.0 <v2.0.0>