Production-ready Kubernetes cluster deployments in AWS in Pixel Federation.

1. Running Kubernetes
Adam Hamsik
DevOps/Cloud Engineer

2. www.pixelfederation.com
Kubernetes deployment on AWS
TL;DR Summary
- Kubernetes deployment tools
- Kubernetes infrastructure components
- Pods, Services, Endpoints
- Workloads
• Deployments, Stateful sets, Replica Sets
- Demo

3. www.pixelfederation.com
1. Running Kubernetes version 1.9 on AWS
2. Deployed with Kops and Terraform
3. If possible we use managed services (RDS, Elasticache)
a. Better performance, easier to manage
4. Prometheus/Grafana used for monitoring
5. Elk stack for log gathering
Kubernetes deployment on AWS
Kubernetes @ Pixel

4. www.pixelfederation.com
Kubernetes deployment on AWS
Kubernetes deployment tools
Multiple available alternatives for kubernetes
deployment
1. Kops (AWS, GCE, digitalocean)
a. Automated deployment (Go binary orchestrating deployment)
2. Kube-up
a. Shell script
3. Kubespray
a. Automated deployment using ansible
4. Kubeadm

5. www.pixelfederation.com
Kubernetes deployment on AWS
Using Kops for Kubernetes Deployment
1. Cloud Architecture
a. Kops can create cloud components or it can generate Terraform templates
b. Cluster nodes are managed as cattle(immutable deployment)
2. Highly available deployment
a. Masters/Nodes running in multiple AZs
b. ASGs used to make sure we can detect/heal from server failure
3. Cluster Updates
a. Rolling update for cluster without outage
i. User-data script is used for kubernetes installation

6. www.pixelfederation.com
Kubernetes deployment on AWS
Kubernetes AWS architecture

7. www.pixelfederation.com
Kubernetes deployment on AWS
Kubernetes Components
1. Kubernetes master
a. Kubernetes-api
b. Kubernetes-scheduler
i. Scheduling pods in cluster, finds proper node for placement
(affinity/anti-affinity)
c. Etcd
i. Kubernetes datastore
d. networking CNI
2. Node
a. Docker/RKT/Container engine
b. Kubelet
i. Monitors Running Containers (health-checks)
c. Kube-proxy
i. Makes sure we can reach services/pods with traffic

8. www.pixelfederation.com
Kubernetes deployment on AWS
Kubernetes Components

9. www.pixelfederation.com
Kubernetes deployment on AWS
Kubernetes Components

10. www.pixelfederation.com
Addons
extend core Kubernetes functionality
1. External-dns
a. Create route53 dns entries based on kubernetes services
2. Cluster-autoscaler
a. Scale cluster nodes based on scheduler events
3. Prometheus
a. Monitor kubernetes cluster and applications running in it
Kubernetes deployment on AWS
Kubernetes Addons

11. www.pixelfederation.com
1. Trust your health-checks
a. liveness / readiness
2. Stateless applications
a. Sharing data between pods complicates cluster architecture
b. Sticky sessions possible, but problematic
3. One process per container
a. Use sidecar containers if needed
4. Scaling
a. Scale apps with HPA (Add pods based on cpu/mem metrics)
b. Scale cluster with cluster-autoscaler (Add nodes based on scheduler events)
Kubernetes deployment on AWS
Running applications on Kubernetes

12. www.pixelfederation.com
1. Use ingress controllers (traefik, nginx, haproxy, AWS ALB)
a. Cost effective solution for traffic proxy
b. Can forward different paths to different services
c. Use multiple ingress controllers for different types of traffic
2. Plan your updates native support for
a. Recreate
b. Rolling Update
c. Blue/Green,Canary,A/B needs to be scripted
3. Package your manifests with HELM
Kubernetes deployment on AWS
Running applications on Kubernetes vol. 2

13. www.pixelfederation.com
Kubernetes deployment on AWS
Application Pod with sidecar container

14. www.pixelfederation.com
1. Package manager for Kubernetes manifests
a. Client/Server deployment with server running inside Kubernetes
2. Use templates to develop reusable charts
a. Sharing charts is simple
3. Supports rollback to previous application versions
a. You can rollback to any application version installed before
4. Use hooks to manage/orchestrate complicated scenarios
Kubernetes deployment on AWS
Kubernetes Helm Package manager

15. www.pixelfederation.com
1. Pods
a. group of one or more containers running on a same physical host
2. Services
a. A Kubernetes Service is an abstraction which defines a logical set of
Pods and a policy by which to access them
b. Use selectors/labels to select right pods
3. Deployments, Stateful sets, Daemon Sets
a. Abstractions with different properties managing lifecycle and deployment of
pods
Kubernetes deployment on AWS
Kubernetes Service objects

16. www.pixelfederation.com
Kubernetes deployment on AWS
Kubernetes application architecture

17. www.pixelfederation.com
Kubernetes deployment on AWS
Questions ?

18. www.pixelfederation.com
Kubernetes deployment on AWS
Thanks !
ahamsik@pixelfederation.com

19. www.pixelfederation.com
Encapsulation based CNI
1. Flannel
2. Weave Net
Routing based CNI
1. Project Calico
2. Kube-router
3. amazon-vpc-cni-k8s
Kubernetes deployment on AWS
Kubernetes networking

20. www.pixelfederation.com
Kubernetes deployment on AWS
Kubernetes networking flannel

21. www.pixelfederation.com
1. Routing based CNI
a. Using BGP to share IP routes to pods/services
b. No traffic encapsulation/decapsulation
c. Scalable
d. Simple
e. Fine grained security policies
Kubernetes deployment on AWS
Kubernetes networking calico