1.
The Journey to the
Kubernetes Networking

2.
TOC
• Network setting
• Pod IP routing (same node)
• Pod IP routing (different node)
• Service IP routing
• DNS lookup

3.
Prerequisite
• Container and Kubernetes fundamental
• Basic network routing and TCP/IP

4.
NOT include
• CNI detail
• Calico detail
• Ingress
• Network policy
• Source code tracing

5.
Environment
• RHEL 7.8
• Kernel 3.10.0-1127
• Kubernetes v1.18.2
• Docker 19.3.13
• Containerd 1.3.7-3.1
• Etcd v3.3.12
• Calico v3.14.0

6.
Scenario
• 1 namespace, 3 pods
• 2 pods are nginx server,
the alpine pod is client

7.
Network setting

8.
Container basis
• In Linux, container = namespace + cgroup
• Namespace: environment isolation (E.g., network & pid namespace)
• Cgroup: resource constraint (E.g., CPU & memory limit)

9.
Container network
• In early years, container or pod IP is managed by container runtime
• Recently, pod IP is managed by CNI plugin in k8s.
• Different pods have different network namespace, different container
share the same namespace in same pod.

10.
Kubernetes Network Model
• Pods can communicate with all other pods on any other node without
NAT
• Agents on a node (e.g. system daemons, kubelet) can communicate
with all pods on that node
https://kubernetes.io/docs/concepts/services-networking/

11.
CNI and CNI plugins
• The CNI protocol is based on execution of binaries invoked by the
container runtime. CNI defines the protocol between the plugin
binary and the runtime.
• A CNI plugin is responsible for configuring a container’s network
interface in some manner.
https://www.cni.dev/docs/spec/#section-2-execution-protocol

12.
K8s pod & service IP range
CIDR Range for Pods in cluster
CIDR Range for Services in cluster
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/

13.
CNI
IPAM: IP Address Management

14.
Log
Kubelet log
Calico log

15.
IP address in pod & host
IP interface in pod
IP interface on host

16.
Network namespace of container
container PID

17.
Pod IP Routing (same node)

18.
on w05t query host 10.233.112.167

19.
Determine the routing in namespace

20.
Determine the routing by enabling proxy_arp
https://wiki.debian.org/BridgeNetworkConnectionsProxyArp
https://tldp.org/HOWTO/Adv-Routing-HOWTO/lartc.bridging.proxy-arp.html

21.
Determine the routing on host

22.
Pod IP Routing (different node)

23.
on w05t query host 10.233.112.167

24.
IP in IP tunnel

25.
on w04t query host 10.240.205.39 and not port 53

26.
on w04t query host 10.240.205.39 and not port 53

27.
Service IP routing

28.
Service IP allocation
• Kubernetes assigns this Service an IP address which is used by the
Service proxies (E.g., kube-proxy or CNI plugin).
• Kube-proxy in iptables mode install the iptables rule and continuously
update the corresponding endpoints on every worker node.
https://kubernetes.io/docs/concepts/services-networking/service/

29.
Scenario
• Create 1 deployment with 3 replicas and expose it as a service

30.
NAT and conntrack when query service IP
on w05t query host 10.233.112.167
DNAT
SNAT & conntrack

31.
Iptables load-balance probability
• 1st endpoint:
1
3
• 2nd endpoint:
2
3
×
1
2
=
1
3
• 3rd endpoint:
2
3
×
1
2
× 1 =
1
3
https://github.com/kubernetes/kubernetes/issues/37932

32.
DNS lookup

33.
DNS resolution for service and pod
• DNS queries may be expanded using the Pod's /etc/resolv.conf.
Kubelet sets this file for each Pod and passes DNS resolver (CoreDNS)
information to each container.
• CoreDNS will reply to DNS queries based on IP of the services and
pods of Kubernetes.
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
CoreDNS config

34.
on w05t query host 10.233.99.52 or host 10.233.88.44

35.
DNS resolution for others
• Any queries that are not within the cluster domain of Kubernetes will
be forwarded to predefined resolvers (/etc/resolv.conf).
https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/
CoreDNS config

36.
DNS cache
• NodeLocal DNSCache improves Cluster DNS performance by running
a DNS caching agent on cluster nodes as a DaemonSet.
• Pods will reach out to the DNS caching agent running on the same
node, thereby avoiding iptables DNAT rules and connection tracking.
https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/
nodelocaldns config

37.
on w05t query host 169.254.25.10 and port 53

38.
Proxy ARP Reference
• https://www.tkng.io/cni/calico/
• https://projectcalico.docs.tigera.io/reference/faq#why-does-my-
container-have-a-route-to-16925411
• https://www.sobyte.net/post/2022-06/proxy-arp-calico/
• https://developpaper.com/disclosure-of-calico-network-
communication-principle/
• https://dramasamy.medium.com/life-of-a-packet-in-kubernetes-part-
2-a07f5bf0ff14