Ihor Borodin, Lead DevOps Engineer at Intellias.  Topic: "Træfik as Kubernetes Ingress controller".

1. Træfik as Kubernetes Ingress Controller
by me

2. # whoami
Ihor Borodin
https://www.linkedin.com/in/ihor-borodin-903706106/
➢ some kind of engineer at Intellias
➢ working A LOT with Kubernetes
➢ still knowing almost nothing about Fancynetes
➢ fan of https://martinfowler.com/
➢ active member of ukrops Slack channel
➢ posting rocket science news at https://t.me/UkropsDigest
➢ co-author of https://github.com/Mykolaichenko/devopsfactors

3. At the beginning there was a Service…
https://kubernetes.io/docs/concepts/services-networking/service/
➢ A Kubernetes Service is an abstraction which defines a logical set of Pods
and a policy by which to access them.
➢ In Kubernetes v1.0, Services are a “layer 4” (TCP/UDP over IP) construct, the
proxy was purely in userspace. In Kubernetes v1.1, iptables proxy was added
and become the default operating mode since Kubernetes v1.2. In
Kubernetes v1.8.0-beta.0, ipvs proxy was added.

4. Heavy lifting to implement Service
https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/
➢ kube-proxy - responsible for implementing a form of virtual IP for Services
and can do simple TCP and UDP stream forwarding or round robin TCP and
UDP forwarding across a set of backends
➢ kube-dns - watches the Kubernetes master for changes in Services and
Endpoints, and maintains in-memory lookup structures to serve DNS requests
➢ dnsmasq - adds DNS caching to improve performance
★ kube-router - brave new world of IPVS/LVS kernel routing and L3 load
balancing

6. Service types
➢ ClusterIP (default) - exposes service only within cluster
➢ NodePort - creates ClusterIP and exposes the same port on every node
➢ LoadBalancer - creates LB in cloud provider and points to respective ports
(NodePort)
➢ ExternalName - creates an alias to a DNS record of service residing outside
the cluster

7. LoadBalancer service type downsides
➢ 1000 services = 1000 cloud LB’s
➢ Gets pretty expensive over time (~20$/month in AWS)
➢ Hard to monitor, impossible to fully utilize
➢ Doesn’t have any magical logic in chain of cloud LB, Kubernetes NodePort
and iptables
➢ Eventually doesn’t comply with “API Gateway” microservice paradigm

8. Here it comes Ingress…
https://kubernetes.io/docs/concepts/services-networking/ingress/
➢ Was introduced in Kubernetes 1.2 (pretty mature)
➢ Gives you a way to route requests to services based on the request host or
path (L7), centralizing a number of services into a single entrypoint.
➢ Ecosystem consists of Ingress Controllers and Ingress resources.
➢ Ingress Controllers can technically be any system capable of reverse
proxying.
➢ Provides stuff like (depending on Ingress Controller implementation) load
balancing, SSL termination and name-based virtual hosting out of the box

9. How does Ingress work

10. Mainstream Ingress Controller implementations
➢ Nginx (most popular one?)
➢ GLBC (default in Google Kubernetes Engine)
➢ Voyager (based on HAProxy)
➢ Traefik
➢ Contour (based on Envoy)
➢ ...You name it

11. Why Nginx is a mess
➢ Nginx Ingress Controller = Nginx + config generator for Nginx
➢ Automatically generated config is not always good (almost always bad)
➢ Doesn’t have any dashboard, requires to set up proper observability from day
one
➢ Strange behaviour when working with WebSockets
➢ Doesn’t support gRPC
➢ Development is being done by two different parties in two repositories -
“nginxinc/kubernetes-ingress“ versus “kubernetes/ingress-nginx“
➢ A lot of regression between versions
http://danielfm.me/posts/painless-nginx-ingress.html

12. Why GLBC is not what you want to use in GKE
➢ No Load Balancing Algorithm settings
➢ Takes ~1 min to allocate LB and ~5-6
➢ “GLBC is not built for performance”
➢ In Beta since forever
➢ IP is allocated for every Ingress object (ephemeral for non-SSL, static for
SSL)

13. Why Traefik is so fancy
➢ Written in golang, single tiny binary, can be launched from “scratch” image
➢ Almost as fast as Nginx
➢ Has Rest API endpoint
➢ Real hot configuration reload
➢ Circuit breakers, retries out of the box
➢ rr, wrr load-balancers

14. A fly in the ointment
➢ A little bit slow with SSL termination
➢ All Kubernetes-related features will become available through annotations in
1.6
➢ No TCP Load Balancing support yet
➢ Some Ingress features are missing
➢ Plugin support is in early stages of WIP

15. Traefik basics

16. Production blueprints with Traefik Ingress Controller
on AWS Kubernetes deployment

17. Deployment options: Deployment vs DaemonSet
Choose DaemonSet because of simplicity - it scales automatically to all nodes
that meets a specific selector and guarantees to fill nodes one at a time
Choose Deployment when you need controlled scalability and automatic
scalability

18. Always set resource requests and limits
Always try to measure your resource consumption and tune resource requests
and limits
● too strict and Traefik will be throttled while serving requests
● too loose and Traefik may waste resources not available for other containers

19. Use Horizontal Pod Autoscalers in case of dynamic loads

20. Use rate limits to protect from flood and from pod termination
*Available only via config, will become configurable through annotations in Traefik
1.6

21. Use basic auth to easily protect non-public Ingresses

22. Use HTTPS enforcement
*Available only via config, will become configurable through annotations in Traefik
1.6

23. SSL termination best options
● Terminate on ELBClassic LB (TCP listener with SSL certificate attached from
AWS Certificate Manager via ARN)
● Terminate on Traefik (request certificates dynamically upon Ingress creation
from Let’s Enсrypt via ACME protocol)

24. Better tastes with External-DNS

25. Always use Circuit Breaker and Retry because Martin Fowler tells you to
(and it’s free)
*Retry available only via config, will become configurable through annotations in
Traefik 1.6

26. Monitor network usage on hosts
Never run Traefik Ingress, Kuberntes cluster, or anything serious in production
without monitoring

27. Some tricky host metrics
● Conntrack memory usage (by kube-proxy)
● TCP listen backlog
● Other Kernel network buffers

28. Be careful with liveness probes
One liveness probe on overloaded pod will cause Kubernetes to restart it, close
existing connections and transfer load to other pods and eventually start chain of
pod restarts

29. Use labelSelector and namespace separator to run multiple Traefik Ingress
Controllers for different purposes

30. Mark your critical addons as critical
● Critical pod annotations with Rescheduler in Kubernetes >= 1.7
https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-
addon-pods/
● Pod priority in Kubernetes 1.8+
https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/

31. Know your tooling
https://docs.traefik.io/user-guide/kubernetes/
https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers
/aws/aws.go#L72

32. Live demo time