Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what can I do with it?

1. Fun with PBR, VRFs, and NetNS on Linux What is it, how does it work, and what can I do with it? Maximilian Wilhelm max@sdn.clinic 1 / 45

2. Agenda 1. Motivation 2. Routing basics 3. Policy-based routing 4. VRFs 5. NetNS 2 / 45

3. Who's who Maximilian Wilhelm Networker OpenSource Hacker Fanboy of (Debian) Linux (Linux) networking Occupation: By day: Network Engineer at Cloudflare By night: Infrastructure Archmage, Freifunk Hochstift In between: Freelance Infrastructre Architect for hire Contact @BarbarossaTM max@sdn.clinic 3 / 45

4. Who's who Motivation Motivation 4 / 45

5. Who's who Motivation Example 1 - University in need of magic routing (PBR) 5 / 45

6. Who's who Motivation Example 2 - Freifunk Hochstift Backbone 6 / 45

7. Who's who Motivation Example 3 - A small ISP 7 / 45

8. Who's who Motivation Example 3 - A small ISP 8 / 45

9. Who's who Motivation Use cases Policy-based routing Route IPv4 traffic leaving the network to CGN boxes Route non-interactive traffic across cheaper link VRFs Keep Internet and internal routing domains seperated Provide LB/proxy to internal services but don't expose hosts completely Provide overlays for customers / different routing domains NetNS Full-blown separation for applications (-> containers) vEth + NetNS for debugging purposes 9 / 45

10. Who's who Motivation Routing Routing (on Linux) 10 / 45

11. Who's who Motivation Routing Routing Every device speaking IP has a routing table German translation according to IBM: "Leitwegtabelle" Packets are forwarded according to longest prefix match Default Gateway or Gateway of last resort used if no entry matches Hot Potato principle Packets forwarded to next hop w/o knowledge of their routing table Asymmetric routing Path to destination and return path don't have to be identical 11 / 45

12. Who's who Motivation Routing Routing table Possible routing table of your laptop when using company VPN: Prefix Iface Next-hop 10.0.0.0/8 tun0 10.23.42.1 10.23.42.0/25 tun0 192.168.178.0/24 wlan0 0.0.0.0/0 wlan0 192.168.178.1 12 / 45

13. Who's who Motivation Routing Source address selection With every routing decision for a locally originated connection a source address is selected based on the routing table. Usually the (primary) IP configured on the outgoing interface May be explicitly set to any IP For example IP on loopback interface Prefix Iface Next-hop Src address 10.0.0.0/8 tun0 10.23.42.1 10.23.42.0/25 tun0 10.23.42.8 192.168.178.0/24 wlan0 192.168.178.5 0.0.0.0/0 wlan0 192.168.178.1 13 / 45

14. Who's who Motivation Routing Source address selection - ICMP erros What IP will answer on errors? # icmp_errors_use_inbound_ifaddr - BOOLEAN # # If zero, icmp error messages are sent with the primary address of # the exiting interface. # # If non-zero, the message will be sent with the primary address of # the interface that received the packet that caused the icmp error. # This is the behaviour many network administrators will expect from # a router. And it can make debugging complicated network layouts # much easier. # # Note that if no primary address exists for the interface selected, # then the primary address of the first non-loopback interface that # has one will be used regardless of this setting. # # Default: 0 net.ipv4.icmp_errors_use_inbound_ifaddr = 1 https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst IPv6: It's complicated, see RFC6724 14 / 45

15. Who's who Motivation Routing Routing tables Every Linux box has a number of routing tables $ ip route help Usage: ip route { list | flush } SELECTOR ... SELECTOR := ... [ table TABLE_ID ] ... TABLE_ID := [ local | main | default | all | NUMBER ] By default routing table main is used So ip route show and ip route show table main show the same thing 15 / 45

16. Who's who Motivation Routing Default Routing Tables on Linux Table local Contains all routes to Locally connected IPs Broadcast addresses Table main Contains "usual" routes Locally connected subnets Routes to remote subnets Table default Usually empty 16 / 45

17. Who's who Motivation Routing Default Routing Tables on Linux Table local $ ip route show table local broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 broadcast 192.168.178.0 dev wlan0 proto kernel scope link src 192.168.178.42 local 192.168.178.42 dev wlan0 proto kernel scope host src 192.168.178.42 broadcast 192.168.178.255 dev wlan0 proto kernel scope link src 192.168.178.42 Table main $ ip route show [table main] default via 192.168.178.1 dev wlan0 proto dhcp metric 600 192.168.178.0/24 dev wlan0 proto kernel scope link src 192.168.178.42 metric 600 Table default $ ip route show table default $ 17 / 45

18. Who's who Motivation Routing What happens on link-down? By default Linux will try to use routes with link down Behaviour can be controlled via sysctl # ip r default via 192.168.178.1 dev eth2 192.168.178.0/24 dev eth2 proto kernel scope link src 192.168.178.42 # echo 1 > /proc/sys/net/ipv4/conf/eth2/ignore_routes_with_linkdown # ip r default via 192.168.178.1 dev eth2 dead linkdown 192.168.178.0/24 dev eth2 proto kernel scope link src 192.168.178.42 dead linkdown # ping 1.1.1.1 connect: Network is unreachable 18 / 45

19. Who's who Motivation Routing PBR Policy-based routing 19 / 45

20. Who's who Motivation Routing PBR Policy-based routing Available since Linux 2.2 (1999) Allows to influence routing decision depending on (e.g.) Ingress interface Source address Source/destination port Something netfilter can match Drawbacks Beware to close loopholes Rule for IPv4 Rule for IPv6 Rule for incoming interface ICMP errors might still get routed by main table 20 / 45

21. Who's who Motivation Routing PBR Defaut routing policy on every Linux box Remember the routing tables from before? $ ip rule 0: from all lookup local 32766: from all lookup main 32767: from all lookup default 21 / 45

22. Who's who Motivation Routing PBR PBR rules $ ip rule help Usage: ip rule { add | del } SELECTOR ACTION ip rule { flush | save | restore } ip rule [ list [ SELECTOR ]] SELECTOR := [ not ] [ from PREFIX ] [ to PREFIX ] [ tos TOS ] [ fwmark FWMARK[/MASK] ] [ iif STRING ] [ oif STRING ] [ pref NUMBER ] [ l3mdev ] [ uidrange NUMBER-NUMBER ] [ ipproto PROTOCOL ] [ sport [ NUMBER | NUMBER-NUMBER ] [ dport [ NUMBER | NUMBER-NUMBER ] ] ACTION := [ table TABLE_ID ] [ protocol PROTO ] [ nat ADDRESS ] [ realms [SRCREALM/]DSTREALM ] [ goto NUMBER ] SUPPRESSOR SUPPRESSOR := [ suppress_prefixlength NUMBER ] [ suppress_ifgroup DEVGROUP ] TABLE_ID := [ local | main | default | NUMBER ] 22 / 45

23. Who's who Motivation Routing PBR PBR rules - examples Half our users are special # ip rule add from 192.168.178.0/25 table 178 Web traffic is special # ip rule add dport 80 table 80 # ip rule add dport 443 table 80 Packets arriving at eth0 are special # ip rule add iif eth0 table 23 23 / 45

24. Who's who Motivation Routing PBR VRFs Virtual Routing and Forwarding 24 / 45

25. Who's who Motivation Routing PBR VRFs Virtual Routing and Forwarding (VRFs) Independent routing instances, provides Layer 3 separation Commonly used for (OOB) mgmt access L3-VPNs, usually in combination with MPLS VRFs on Linux VRF interface is master for “real” (member) interfaces Maps to a (numeric) routing table Netfilter rules shared across VRFs Introduced in Kernel 4.[345] (use >= 4.9) 25 / 45

26. Who's who Motivation Routing PBR VRFs Con guring VRFs By foot ip link add vrf_external type vrf table 1023 ip link set eth0 master vrf_external # Option 1: generic ip link set eth0 vrf vrf_external # Option 2: VRF specific ifupdown2 / ifupdown-ng auto eth0 iface eth0 address 2002:db8:23:42::2/64 gateway 2001:db8:23:42::1/64 vrf vrf_external auto vrf_external iface vrf_external vrf-table 1023 Device routes move from table main and local to table 1023 26 / 45

27. Who's who Motivation Routing PBR VRFs VRFs: Under the hood - IPv4 A VRF is like a routing table with benefits: $ ip r s vrf vrf_external default via 192.0.2.1 dev eth0 metric 1 192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.42 $ ip r s table 1023 default via 192.0.2.1 dev eth0 metric 1 broadcast 192.0.2.255 dev eth0 proto kernel scope link src 192.0.2.42 192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.42 local 192.0.2.42 dev eth0 proto kernel scope host src 192.0.2.42 broadcast 192.0.2.255 dev eth0 proto kernel scope link src 192.0.2.42 27 / 45

28. Who's who Motivation Routing PBR VRFs VRFs: Under the hood - IPv6 A VRF is like a routing table with benefits: $ ip -6 r s vrf vrf_external anycast 2001:db8:23:42:: dev eth0 proto kernel metric 0 pref medium 2002:db8:23:42::/64 dev eth0 proto kernel metric 256 linkdown pref medium anycast fe80:: dev eth0 proto kernel metric 0 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium multicast ff00::/8 dev eth0 proto kernel metric 256 pref medium default via 2001:db8:23:42::1 dev eth0 metric 1 pref medium $ ip -6 r s table 1023 anycast 2001:db8:23:42:: dev eth0 proto kernel metric 0 pref medium local 2001:db8:23:42::2 dev eth0 proto kernel metric 0 pref medium 2002:db8:23:42::/64 dev eth0 proto kernel metric 256 linkdown pref medium anycast fe80:: dev eth0 proto kernel metric 0 pref medium local fe80::222:19ff:fe65:b835 dev eth0 proto kernel metric 0 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium multicast ff00::/8 dev eth0 proto kernel metric 256 pref medium default via 2001:db8:23:42::1 dev eth0 metric 1 pref medium 28 / 45

29. Who's who Motivation Routing PBR VRFs VRFs: Under the hood - Plumbing Remember PBR? Setting up a VRF adds a global VRF rule: $ ip rule 0: from all lookup local 1000: from all lookup [l3mdev-table] 32766: from all lookup main 32767: from all lookup default 29 / 45

30. Who's who Motivation Routing PBR VRFs Connecting VRFs Requires vEth pair Like a virtual network cable within the box A end in main VRF, Z end in VRF “foo” Usual routing Static Bird talking BGP to itself Drawback: ARP didn't work recently (didn't debug :)) Static entries helped ND worked though 30 / 45

31. Who's who Motivation Routing PBR VRFs Connecting VRFs By foot # ip link add VETH_END1 type veth peer name VETH_END2 ifupdown2* / ifupdown-ng iface veth_ext2int link-type veth veth-peer-name veth_int2ext vrf vrf_external iface veth_int2ext link-type veth veth-peer-name veth_ext2int * Merged with PR25, unsure if still works 31 / 45

32. Who's who Motivation Routing PBR VRFs Leaking Routes Similar to vendor boxes Leaking VRF -> GRT (eth2 part of GRT): # ip route add default via 192.0.2.1 dev eth2 vrf vrf_foo Leaking GRT -> VRF # ip route add 198.51.100.0/24 dev vrf_foo 32 / 45

33. Who's who Motivation Routing PBR VRFs VRF awareness for applications By default applications only use main table Packets received in VRF table reach application Reply sent out via main table There's help: # tcp_l3mdev_accept - BOOLEAN # # Enables child sockets to inherit the L3 master device index. # Enabling this option allows a "global" listen socket to work # across L3 master domains (e.g., VRFs) with connected sockets # derived from the listen socket to be bound to the L3 domain in # which the packets originated. Only valid when the kernel was # compiled with CONFIG_NET_L3_MASTER_DEV. # # Default: 0 (disabled) net.ipv4.tcp_l3mdev_accept = 1 This switch has influence on IPv6, too! 33 / 45

34. Who's who Motivation Routing PBR VRFs Real World Applications for VRFs LB / Web proxy/frontend External interface is part of vrf_external tcp_l3mdev_accept set to 1 nginx as reverse proxy Listens on ip in GRT + IP in vrf_external Uses main table for connections to internal services Can serve queries from external + internal clients External interface in VRF External interface is part of vrf_external GRE / OpenVPN tunnel sent / receive encapsulated packets over VRF Local tunnel endpoint is in GRT No risk of leaking stuff from GRT by accident 34 / 45

35. Who's who Motivation Routing PBR VRFs Real World Applications - Tunnels / GRE Outer and/or inner side of tunnel can be part of a VRF Outer side in VRF # ip link add DEVICE type gre remote ADDR local ADDR dev PHYS_DEV If PHYS_DEV is within a VRF, all encapsulated packets are send/received in VRF Inner side in VRF Pushing the inner side of a tunnel into a VRF is equally simple: # ip link set DEVICE master VRF 35 / 45

36. Who's who Motivation Routing PBR VRFs Real World Applications - Tunnel / OpenVPN Pushing the inner side of an OpenVPN tunnel into a VRF is as simple as before. Sending/receiving encapsulated packets into/from a VRF needs application support. My patch from October 2016 finally made it into OpenVPN 2.5 :) # openvpn --config your_config.cfg --bind-dev VRF This is used to glue remote POPs of Freifunk Hochstift together 36 / 45

37. Who's who Motivation Routing PBR VRFs Real World Applications - VRFs + MPLS In need of L3VPN? It's in the cards, too! 37 / 45

38. Who's who Motivation Routing PBR VRFs Real World Applications - VRFs + MPLS The plumbing: # modprobe mpls_iptunnel # Active MPLS # sysctl -w net.mpls.platform_label=1000000 # Set max. label # sysctl -w net.mpls.conf.ethX.input=1 # Active MPLS decap on ethX Encap traffic with MPLS label 2342, send it to neighbor on ethX (in GRT) # ip route add 192.0.2.0/24 encap mpls 2342 via inet6 2001:d8:42::1 dev ethX vrf vrf_x Decap traffic with label 4223 and send it to VRF vrf_x # ip -M route add 4223 dev vrf_x Swap labels on the path (100 -> 200) # ip -M route add 100 as 200 via inet6 2001:db8:4711::1 38 / 45

39. Who's who Motivation Routing PBR VRFs NetNS Network Namespaces (NetNS) Layer 1 separation An interface is part of exactly one NetNS Similar to VRFs on vendor gear Own set of routing tables VRFs and PBR available within NetNS Own set of netfilter rules Processes can be bound to a NetNS Introduced in Kernel 2.6.29 39 / 45

40. Who's who Motivation Routing PBR VRFs NetNS Network Namespaces and net lter There may be side effects when traffic enters NetNS via vEth/macvlan Already went through conntrack NAT might not work as expected 40 / 45

41. Who's who Motivation Routing PBR VRFs NetNS Con guring Network Namespaces $ ip netns help Usage: ip netns list ip netns add NAME ip netns set NAME NETNSID ip [-all] netns delete [NAME] ip netns identify [PID] ip netns pids NAME ip [-all] netns exec [NAME] cmd ... ip netns monitor ip netns list-id NETNSID := auto | POSITIVE-INT 41 / 45

42. Who's who Motivation Routing PBR VRFs NetNS Connecting Network Namespaces You guessed it, vEth to the rescue # ip link add veth_grt type veth peer name veth_client # ip netns add "${netns}" # ip link set veth_client netns "${netns}" 42 / 45

43. Who's who Motivation Routing PBR VRFs NetNS Real world applications All your containers Testing networking stuff 43 / 45

44. Who's who Motivation Routing PBR VRFs NetNS Summary 44 / 45

45. Who's who Motivation Routing PBR VRFs NetNS Links Further Reading Contemporary Linux Networking - DENOG9 (2017) https://www.slideshare.net/BarbarossaTM/contemporary-linux-networking VRFs https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/ Documentation/networking/vrf.rst https://de.slideshare.net/CumulusNetworks/operationalizing-vrf-in-the-data-center OpenVPN and VRFs https://blog.sdn.clinic/2018/12/openvpn-and-vrfs/ MPLS Lab – Playing with static LSPs and VRFs on Linux https://blog.sdn.clinic/2022/01/mpls-lab-playing-with-static-lsps-and-vrfs-on-linux/ 45 / 45

Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what can I do with it?

Esté a ler uma amostra.

Confira estes a seguir

Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what can I do with it?

Mais Conteúdo rRelacionado

Diapositivos para si (20)

Semelhante a Fun with PRB, VRFs and NetNS on Linux - What is it, how does it work, what can I do with it? (20)

Mais de Maximilan Wilhelm (16)

Mais recentes (20)