Watch the recording here: https://youtu.be/doren6cnvac ------------------------------------------- Wksctl supports the creation and GitOps management of Kubernetes clusters on any set of machines with an SSH key. Jerry Jackson from the wksctl engineering team will discuss various aspects of wksctl including: - its architecture and implementation - how to use it to create clusters - its relationship to flux and how to work with it using a GitOps model - how to combine it with footloose for easy use in development environments - how it enables consistent environments across the spectrum from development to production ------------------------------------------- Speaker: Jerry Jackson, Software Engineer, Weaveworks Bio: Jerry is an expert developer and architect of infrastructure and backend systems. For the last 15 years, his main work has been in the areas of system and cluster management, for which he has received over a dozen patents. Most recently, Jerry has worked on the automated construction of Kubernetes clusters and generation of Kubernetes application definitions from graphical models and Docker compose files. Over his 30+ year career, he has developed firmware, compilers, database mapping tools, and a cluster management expert system. Jerry is the author of “Java by Example”, one of the first books available about Java. When he isn’t working, Jerry loves to run and read science fiction.

1. wksctl: GitOps
Management of
k8s Clusters
Jerry Jackson, Software Engineer, Weaveworks
Tamao Nakahara, Head of DX, Weaveworks

5. Weaveworks is a company founded on open source:
● Weave Net: Fast, Encrypted, Cloud-Native Mesh Networking
● Flux (in CNCF Sandbox!): GitOps for k8s
● Cortex (in the CNCF): Distributed, Long-term-storage TSDB compatible
with Prometheus
● Weave Flagger: Declarative Progressive Delivery for Service Meshes
● EKSctl: Create an Amazon EKS cluster with one command
● Weave Ignite: VMs with container UX & built-in GitOps management
● Weave Scope: Network/Process Observability for Container Clusters
● WKSctl: k8s configuration management with GitOps
● & More (jkcfg, footloose, kured, ...)

6. Weaveworks
You can pay us for these things :)
● Weave Cloud: SaaS product for K8S management,
monitoring, and automated deployments (Hosted
Prometheus/Cortex, Scope, and Flux)
● Weave Kubernetes Platform: GitOps-aware Enterprise
Kubernetes for Production
● Consulting / Training / Support

7. weave.works

8. Speakers Help/Support
Duration
30-45 Minutes
Jerry Jackson
Software Engineer
Weaveworks
Tamao Nakahara
Head of DX
Weaveworks
Browser
Safari copy/paste
shortcuts may not work
wksctl: GitOps Management of Kubernetes Clusters
Using Zoom
Questions?
• Use chat (button: top
left corner of screen)
• Escape to exit full
screen
• “To Everyone” or “To
all panelists and
attendees”
Support:
https://support.zoom.us/hc/
en-us/articles/206175806-T
op-Questions
Troubleshooting
Use chat
If the issue is not easily resolved,
we ask that you follow along as
we demo the sample app.

9. ● What is it?
● What can you do with it?
● Demo
● Under the Hood
● Q&A
9
Overview

10. ● A tool to easily build and manage GitOps Kubernetes Clusters
● Requires only:
○ Cluster description
■ Subnet definitions for services and pods
■ Path to SSH key with access to all machines
■ Username of SSH user
■ Boilerplate configuration of yum repositories and docker
○ Machine descriptions (IP addresses, ports, roles (master/worker))
○ Git repository
● Currently based on v1 of Cluster API
10
What is it?

11. ● Construct Kubernetes Clusters based on configurations in Git
○ Currently CentOS 7
○ Ubuntu under development
● Manage clusters via Git commits
○ Upgrade clusters
○ Add / remove nodes
11
What can you do with it?

12. ● Single Source of Truth
○ Definition of workloads is always accurate and available
● Changes are recorded
○ Can be reviewed or audited via standard tools
● Previous states can be easily restored
○ Failed deployments can be rolled back
● See: https://www.weave.works/blog/what-is-gitops-really for a complete discussion
12
Why manage clusters with Git(Ops)?

13. ● Manage clusters from within
● Defines CRDs that represent machines and clusters
● Specifies goal-seeking controller to maintain desired cluster state
● Works well with GitOps
○ Cluster and Machine manifests managed just like user manifests
13
Cluster API Project

14. ● Set up ssh connectivity to a set of machines
● Define cluster with simple manifests in Git
● Run wksctl apply command to start processing
● Run wksctl kubeconfig to get a kubeconfig file providing cluster access
● That’s it! -- Cluster is created and can then be managed by Git updates
14
How to set up and manage a GitOps cluster with
WKSctl

15. ● Need single private ssh key that can access all cluster machines
● Can use any user with sudo permissions
○ User specified in cluster.yaml (defaults to “root”)
○ Key in
■ cluster.yaml (release 0.8.1)
■ Command argument (release 0.8.2+)
15
Set up ssh connectivity

16. ● cluster.yaml
○ Ancillary files
■ docker-config.yaml
■ repo-config.yaml
● machines.yaml
● cluster.yaml and machines.yaml specific to installation
● docker-config.yaml and repo-config.yaml are boilerplate
● All committed and pushed to GitHub
16
Define cluster with simple manifests

17. 17
cluster.yaml
apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
metadata:
name: example
namespace: weavek8sops
spec:
clusterNetwork:
pods:
cidrBlocks:[192.168.0.0/16]
serviceDomain: cluster.local
services:
cidrBlocks:[10.96.0.0/12]
providerSpec:
value:
apiVersion: baremetalproviderspec/v1alpha1
kind: BareMetalClusterProviderSpec
cri:
kind: docker
package: docker-ce
version: 18.09.7
user: root
os:
files:
- destination: /etc/yum.repos.d/kubernetes.repo
source:
configmap: repo
key: kubernetes.repo
- destination: /etc/yum.repos.d/docker-ce.repo
source:
configmap: repo
key: docker-ce.repo
- destination: /etc/docker/daemon.json
source:
configmap: docker
key: daemon.json

18. 18
docker-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: docker
namespace: system
data:
daemon.json: |
{
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"exec-opts": [
"native.cgroupdriver=cgroupfs"
]
}

19. 19
repo-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: repo
namespace: system
data:
kubernetes.repo: |
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kube*
docker-ce.repo: |
[docker-ce-stable]
name=Docker CE Stable - $basearch
baseurl=https://download.docker.com/linux/centos/7/$basearch/stable
enabled=1
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg
[docker-ce-stable-debuginfo]
name=Docker CE Stable - Debuginfo $basearch
baseurl=https://download.docker.com/linux/centos/7/debug-$basearch/stable
enabled=0
gpgcheck=1
gpgkey=https://download.docker.com/linux/centos/gpg
….

20. ● Specify
○ Destination
○ Config map name
○ Key
● Create
○ Local <config map name>-config.yaml
● Add
○ Data to deploy under Key
20
“os.files” is a general file deployment mechanism

21. 21
machines.yaml
apiVersion: v1
items:
- apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
labels:
set: master
name: master-0
namespace: weavek8sops
spec:
providerSpec:
value:
apiVersion: baremetalproviderspec/v1alpha1
kind: BareMetalMachineProviderSpec
private:
address: 172.17.0.2
port: 22
public:
address: 127.0.0.1
port: 2222
versions:
kubelet: 1.14.1
- apiVersion: cluster.k8s.io/v1alpha1
kind: Machine
metadata:
labels:
set: worker
name: worker-0
namespace: weavek8sops
… etc. ...
kind: List

22. wksctl apply --help
Create or update a Kubernetes cluster
Usage:
wksctl apply [flags]
Flags:
--cluster string Location of cluster manifest (default "cluster.yaml")
--config-directory string Directory containing configuration information for the cluster (default ".")
--git-branch string Git branch WKS should use to sync with your cluster (default "master")
--git-deploy-key string Path to the Git deploy key
--git-path string Relative path to files in Git (default ".")
--git-url string Git repo containing your cluster and machine information
-h, --help help for apply
--machines string Location of machines manifest (default "machines.yaml")
--namespace string namespace override for WKS components (default "weavek8sops")
--sealed-secret-cert string Path to a certificate used to encrypt sealed secrets
--sealed-secret-key string Path to a key used to decrypt sealed secrets
--ssh-key string Path to a key authorized to log in to machines by SSH (default "./cluster-key")
--use-manifest-namespace use namespaces from supplied manifests (overriding any --namespace argument)
22
wksctl apply

23. ● For the demo
○ wksctl apply --git-url=<path to GitHub repo>
--git-deploy-key <path to private key for repo access>
23
Run “wksctl apply” command

24. ● Weaveworks tool for creating containers that look like VMs
○ Can work with docker containers or ignite/firecracker
microVMs
● “Vagrant, but with containers”
○ Extremely fast startup
● Demo will run on footloose “machines”
● More info: https://github.com/weaveworks/footloose
24
Footloose

25. ● Steps
○ Create GitHub repo and clone locally
○ Create and install a deploy key
○ Run wksctl apply
○ Run wksctl kubeconfig
25
Demo

26. ● Can also create cluster using “quickstart”
○ Easiest way to get started
○ Can experiment with GitOps
○ Useful for local testing clusters
■ Like “Minikube” but can run multi-node clusters
○ Steps
■ Fork and clone wks-quickstart-firekube weaveworks repository
■ Change directory to the clone
■ Run ./setup.sh
26
Note

27. ● Add load balancer across control plane nodes
● See Chanwit Kawasaki’s excellent blog post:
https://www.weave.works/blog/fork-clone-run-a-gitops-model-for-
provisioning-multi-machine-ha-clusters-with-rolling-upgrades
27
Highly Available Clusters

28. ● Initial Master Node installed by wksctl via commands over SSH
● wks-controller running on initial master node installs other nodes
● All Installation performed via “Plans” and “Resources”
○ Resources represent individual tasks
■ Execute a command or script
■ Install a package
■ Install a file
■ Etc.
○ Plans are resources that group other resources
28
Under the Hood

29. 29
Components of Running WKSCtl System

30. ● Periodically checks for git updates and applies them to the cluster
● Configured with information about git repository
○ Git URL
○ Git branch
○ Git path (can look at a subset of a git repository by setting a path)
○ And others (poll interval, readonly, etc.)
● See: https://fluxcd.io/ for details
30
Flux makes WKSctl into a GitOps System

31. 31
Basic WKSctl Cluster Creation Flow

32. ● Responsible for node:
○ Creation (except for initial master), Update (including Upgrade), Deletion
● Notified of changes to machine objects
● Processes one machine at a time
● Ordering of operations performed via error returns
○ If not ready to operate on a particular machine, error out
○ Upgrades masters before workers by erroring out on a worker if there are
non-upgraded masters
● Stores no machine state (except for footloose scaling prototype)
32
Machine Actuator

33. ● Create:
○ Generates a Node Plan
○ Executes it
○ Stores json version of the plan on the node if successful
● Update:
○ Generates a new Plan
○ Compares it to stored Plan
○ Updates the node if Plans differ
○ Tears node down and rebuilds it to ensure idempotency
33
Machine Actuator (cont.)

34. ● Upgrade handled specially
○ Doesn’t rebuild machine
○ Uses kubeadm
○ Upgrades masters before workers
■ “Initial master” first (works even without load-balancer)
● Does not currently support downgrade
34
Machine Actuator (cont.)

35. ● Currently unused :-)
35
Cluster Actuator

36. ● Resources
● Plans
36
Deep Dive

37. ● Implement all atomic operations performed by wksctl
○ Except for:
■ Modifying node labels
■ Modifying node annotations
■ Draining nodes
■ Uncordoning nodes
37
Resources

38. ● Directory (install, remove directories)
● File (install, remove files)
● RPM (install, remove RPMs)
● Kubeadm (init, join)
● Kubectl (apply, wait)
● Secret (write contents of secret to host file)
● OS (query OS parameters)
● Service (manipulate systemd services)
● Run (execute an inline command/script)
● RunScript (execute a script given a path)
● Plan (group other resources in a dependency graph and execute them)
38
Resource Types

39. ● // Runner is something that can realise a step.
type Runner interface {
// RunCommand runs a command in a shell. This means cmd can be more than one
// single command, it can be a full bourne shell script.
RunCommand(cmd string, stdin io.Reader) (stdouterr string, err error)
}
● // Resource is an atomic step of the plan.
type Resource interface {
// State returns the state that this step will realize when applied.
State() State
// QueryState returns the current state of this step. For instance, if the step
// describes the installation of a package, QueryState will return if the
// package is actually installed and its version.
QueryState(runner Runner) (State, error)
// Apply this step and indicate whether downstream resources should be re-applied
Apply(runner Runner, diff Diff) (propagate bool, err error)
// Undo this step.
Undo(runner Runner, current State) error
}
39
Resources (cont.)

40. ● Group resources recursively
● “Apply” invokes resources in dependency order
● “Undo” invokes resource undos in reverse dependency order
● Constructed via “Builder”:
b := plan.NewBuilder()
b.AddResource(
"upgrade:node-unlock-kubernetes",
&resource.Run{Script: object.String("yum versionlock delete 'kube*' || true")})
b.AddResource(
"upgrade:node-install-kubeadm",
&resource.RPM{Name: "kubeadm", Version: version, DisableExcludes: "kubernetes"},
plan.DependOn("upgrade:node-unlock-kubernetes"))
40
Plans

41. ● Seed Node Plan (to create initial master)
● Node Plan (to create all other nodes)
41
Two Main Plans

42. ● Each node is annotated with a json representation of its plan
○ When a machine is processed by the machine actuator, the plan that
corresponds to its new state is compared with its old plan from the
corresponding node
○ When the machine actuator is first invoked with any machine, it retroactively
annotates the seed node with a standard node plan for future comparisons
● The seed node plan can be viewed
○ wksctl plan view is a hidden command (not needed for using wksctl)
○ View as a graph or json
42
Plans (cont.)

43. 43
Example Seed Node Plan

44. Weave Online User Group
Tuesdays, 10:00 am Pacific Time / 18:00 UK time
Format: talks or discussions
Schedule (topics subject to change based on demand):
• Mar 24: Image Is Everything. (Let’s Keep it Secure!) with Jason Epstein
• April 7: What’s New in Flagger 1.0 with Stefan Prodan
• April 8: Denver DevOps: GitOps Hands-On with Leigh Capili (Denver, CO)

45. Next Steps
• Questions? Email tamao@weave.works
• The Practical Guide to GitOps: eBook: http://bit.ly/gitops_guide
•
• GitOps Hands-On Challenge: http://bit.ly/GitOps_HandsOn_EKS
• Join us on Slack if you have more questions: https://slack.weave.works
• Join the Weave User Group:
https://www.meetup.com/Weave-User-Group/

46. THANK YOU!