Testing applications is important, as shown by the rise of continuous integration and automated testing. In this talk, I will focus on one area of testing that is difficult to automate: poor network connectivity. Developers usually work within reliable networking conditions so they might not notice issues that arise in other networking conditions. I will give examples of software that would benefit from test scenarios with varying connectivity. I will explain how traffic control on Linux can help to simulate various network connectivity. Finally, I will run a demo showing how an application running in Kubernetes behaves when changing network parameters.
Sched Link: http://sched.co/6Bb3
2. Alban Crequy
∘ Worked on rkt the last 14 months
∘ Currently tech lead on rkt
∘ In 2014, worked on traffic control for multimedia
applications in cars (tcmmd)
https://github.com/alban
3. ∘ What is traffic control and how does it work on Linux
∘ Using TC in containers for tests
∘ Demo
∘ In Kubernetes
∘ Demo with pings
∘ Demo with guestbook
∘ Integration in a testing framework
∘ Demo with guestbook
Plan
5. Traffic control, why?
web server client
client
client
THE
INTERNET
∘ fair distribution
of bandwidth
∘ reserve
bandwidth to
specific
applications
∘ avoid
bufferbloat
6. ∘ Network scheduling algorithm
∘ which packet to emit next?
∘ when?
∘ Configurable at run-time:
∘ /sbin/tc
∘ Netlink
∘ Default on new network interfaces: sysctl net.core.default_qdisc
Queuing disciplines (qdisc)
eth0 THE INTERNETqdisc
7. ∘ First In, First Out
∘ But with 3 bands, based on IP header’s ToS field (type of service)
Linux’ default qdisc: pfifo_fast
eth0 THE INTERNETFIFO 1
FIFO 2
FIFO 0
9. ∘ drop packets to avoid buffer bloat
∘ similar to Random Early Detection (red) but based on delays rather than
the size of the buffer
∘ set as default by systemd since 2014
Fair Queuing Controlled Delay
(fq_codel)
eth0 THE INTERNET
X
19. Testing with traffic control in
Kubernetes
Kubernetes
minion 1
pod
pod
Kubernetes
minion 2
pod
pod
Testing framework
∘ configure network
simulator
∘ play scenarios
20. Testing with traffic control in
Kubernetes
Kubernetes
minion 1
pod
pod
Kubernetes
minion 2
pod
pod
tcd tcd
gRPC or D-Bus methods:
∘ Install()
∘ ConfigureEgress()
https://github.com/kinvolk/tcd
29. Using filters in Kubernetes
Kubernetes
minion 1
pod
pod
Kubernetes
minion 2
pod
pod
Testing framework
drop
100%
latency
100ms
latency
100ms
configuring tc filter
based on IPs
(type=u32)
31. ∘ 1 network namespace per pod
∘ rktnetes: apps started as
systemd units
∘ How to filter by app?
systemd.resource-control(5):
NetClass=auto
∘ added in v227, 2015-10-07
∘ removed in v229 :(
Filtering by app
Kubernetes
minion 1
pod
app app
pod
app app
32. cgroup “net_cls”: filter by app
∘ Classifying based on cgroups with “net_cls”
∘ Previously exposed by systemd
∘ Then, tc filter “cgroup”
∘ But not available in cgroup unified hierarchy, to ensure delegation
∘ netfilter/iptables being replaced by nftables
∘ New xt_cgroup just added to match on cgroup full path, then could
mark it and use net_cls