A presentation where I spin up a working Kubernetes cluster from scratch in AWS.

1. Kubernetes from the ground up
Through the looking glass of the Kubernetes
internals
Skyworkz - https://skyworkz.nl
Sander Knape - https://sanderknape.com

2. Hello, I’m Sander Knape
Cloud Engineer @ Skyworkz

3. Quick history
Hardware → Virtualization → Containerization

4. Container use cases
● Run X amount of containers
○ Autoscaling
○ Distributed on different nodes / AZs / Regions
● Rolling updates
● Schedule containers (cron)
● ...

6. Kubernetes is big
● Ninth place in commits at GitHub on March 6, 2018 https://en.wikipedia.org/wiki/Kubernetes
● First CNCF graduated project https://www.cncf.io/blog/2018/03/06/kubernetes-first-cncf-project-graduate/

7. Why build it yourself?
● Learn more about the Kubernetes components
● Gives you more knowledge, helpful when debugging Kubernetes
● It’s fun!

8. Disclaimer
● What we’ll build today is not production-ready

9. Kubernetes the hard way
https://github.com/kelseyhightower/kubernetes-the-hard-way

10. Building Kubernetes from the ground up
1. Infrastructure
2. Certificates
3. Control plane
4. Workers
5. Networking
6. Test!

11. Infrastructure

12. VPC / Subnet
Infrastructure
AWS
Load balancer
Controller 1 Controller 2 Controller 3
Autoscaling group
Worker NWorker NWorker N
S3
Bucket

13. Demo

14. Certificates

15. Controller
Infrastructure
kube-apiserver
kube-controller-
manager
kube-scheduler
etcd
Worker
kubelet
kube-proxy
1
2
3
4
5
6
7

16. Demo

17. Control plane

18. VPC / Subnet
Infrastructure
AWS
Load balancer
Autoscaling group
Worker NWorker NWorker N
S3
Bucket
Controller 1 Controller 2 Controller 3

19. Controller 1
Control plane
kube-apiserver
kube-controller-
manager
kube-scheduler
etcd
1
2
3
4
Controller 2
kube-apiserver
kube-controller-
manager
kube-scheduler
etcd
1
2
3
4
Controller 3
kube-apiserver
kube-controller-
manager
kube-scheduler
etcd

20. API server
● Gateway into the cluster
● Handles authentication, authorization

21. Kubernetes changes
https://kubernetes.io/blog/2019/03/21/a-guide-to-kubernetes-admission-controllers/

22. while(true) {
if(currentState() != desiredState() {
updateState();
}
}
Controller manager

23. Controller manager: manages controllers
1. Node Controller: Responsible for noticing and responding when nodes go down.
2. Replication Controller: Responsible for maintaining the correct number of pods for every
replication controller object in the system.
3. Endpoints Controller: Populates the Endpoints object (that is, joins Services & Pods).
4. Service Account & Token Controllers: Create default accounts and API access tokens for
new namespaces.
https://kubernetes.io/docs/concepts/overview/components/#kube-controller-manager

24. Scheduler
https://medium.com/@dominik.tornow/the-kubernetes-scheduler-cd429abac02f

25. etcd

26. Controller 1
Control plane
kube-apiserver
kube-controller-
manager
kube-scheduler
etcd
1
2
3
4
Controller 2
kube-apiserver
kube-controller-
manager
kube-scheduler
etcd
1
2
3
4
Controller 3
kube-apiserver
kube-controller-
manager
kube-scheduler
etcd

27. Demo

28. Workers

29. VPC / Subnet
Infrastructure
AWS
Load balancer
Controller 1 Controller 2 Controller 3
S3
Bucket
Autoscaling group
Worker NWorker NWorker N

30. Infrastructure
Worker
kubelet
kube-proxy

31. Demo

32. Networking

33. Kubernetes Networking Model
1. All Pods can communicate with all other Pods without using network address
translation (NAT).
2. All Nodes can communicate with all Pods without NAT.
3. The IP that a Pod sees itself as is the same IP that others see it as.
https://sookocheff.com/post/kubernetes/understanding-kubernetes-networking-model/

34. Kubernetes Networking Model
● Same network: https://github.com/aws/amazon-vpc-cni-k8s
● Overlay network: https://github.com/coreos/flannel
https://sookocheff.com/post/kubernetes/understanding-kubernetes-networking-model/

35. VPC / Subnet
Same network
EC2 InstanceEC2 Instance
EC2 Instance

36. VPC / Subnet
Overlay network
EC2 InstanceEC2 Instance
flannel
(10.200.0.0/24)
flannel
(10.200.1.0/24)

37. Container Network Interface
● Pluggable networking layer
https://kubedex.com/kubernetes-network-plugins/

38. Infrastructure
Worker
kubelet
kube-proxy

39. Demo

40. Let’s test!

41. Deployment
Rolling updates
Rollbacks
Containers in Kubernetes
Replicaset
Number of replicas
Restarts failed containers
Pod
At least one container
Annotations
Metadata
Container

42. Deployment
Accessing containers in Kubernetes
Replicaset
Pod
Service Container
Pod
Container

43. Demo

44. Building Kubernetes from the ground up
● Infrastructure
● Certificates
● Control plane
● Workers
● Networking
● Test!

45. What’s next? (1)
● Improve certificate management
● High availability
○ Multi-AZ / Multi-Region / Multi-Cloud
○ Cluster Autoscaler / Pod Autoscaler
● DNS
○ CoreDNS (https://coredns.io)
● Cloud Controller Manager
● Networking plugin

46. What’s next? (2)
● Service mesh
○ Istio (https://istio.io) / Consul Connect (https://www.consul.io)
● Secret management
○ https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data / Vault https://www.vaultproject.io
● Application deployments
○ E.g. Helm (https://helm.sh)
● Observability
● ...

47. “Managed” Kubernetes
● Google Kubernetes Engine (GKE) - Google Cloud
● Elastic Container Service for Kubernetes (EKS) - Amazon Web Services
● Azure Kubernetes Service (AKS) - Azure
● DigitalOcean Kubernetes - DigitalOcean
● ...

48. Thank you!
Questions?
Skyworkz - https://skyworkz.nl
Sander Knape - https://sanderknape.com - @SanderKnape