This talks shows how to implement the Application-Based Routing in the common Linux Distribution. We use the NDPI to execute the DPI function to category the packet first, use the linux kernel build-it mark to pass the information from user-space to kernel space and then the policy routing system use that mark to route the packet by different destination or interface.

1. SDN X CLOUD NATIVE
MEETUP #10

2. APPLICATION-BASED ROUTING
Hung-Wei Chiu(hwchiu)
hwchiu@thundertoken.tw

3. WHO AM I
Hung Wei Chiu (Hwchiu)
https://blog.hwchiu.com
DevOps Engineer @ ThunderToken
Interest in Networking/DevOps/Kubernetes/SDN/Programming

4. Software Defined Wide Area Network (SD-WAN)

5. SD-WAN
Create a virtual-overlay to abstract underlying private/public WAN connections
LTE
MPLS
Wifi
Fiber
Route WAN traffic along the best route
Latency
QoS

6. SD-WAN
Managed by a centralized controller
Remotely program edge devices and reduce provisioning times.
Minimizing the need to manually configure network devices
Security
IPSec
Firewall

7. SD-WAN
Traffic
Improve performance through a combination WAN connections
Fail-over
Simplifies the network
Deployments
Configurations
Operations

8. DSL
MPLS
LTE
DSL
MPLS
LTE
DSL
MPLS
LTE
Controller
Internet

9. Application Based Routing

10. APPLICATION-BASED ROUTING
Assume there’re multiple WAN Connections
Different Latency
Route the traffic based on application
Priority
Game/VoIP/…etc

11. Public Internet
10 sec
2 sec
3 sec500 ms

12. Public Internet
10 sec
2 sec
3 sec500 ms
File Transfer
Web Browse

13. How Can We Implement it In a Linux Host

14. CHALLENGES
Which Application
Policy
Route Traffic

15. DPI
Deep Packet Inspection
Well-known tuple (L3 + L4)
TCP/UDP
Port (53/67/80/443)

16. DPI
L7 application
Organized format
Guess by pattern
SSL Termination
User should import a trusted CA from DPI devices

17. NDPI
Open Source Software (OSS)
Based on C++ language
Support 185+ protocols

18. NDPI
Support capture packets by pcap
Decision Tree
Example (Skype)

19. nDPI Engine
Packet PacketPacket Packet Packet
Packet Packet Packet
Packet Packet Packet
Packet = skype
Connections Results
Packet = Quic
Packet = Facebook

20. APPROACHES
With Linux Kernel
Without Linux Kernel
OpenvSwitch/Openflow

21. SOURCE (WITH KERNEL)
Use the pcap to catch packets from a
specific interface
By iptables and our implemented daemon

22. PCAP APPROACH
Implement a daemon
Capture packets by libpcap
Feed packets to nDPI engine to get its
type

23. Application
nDPI Engine
User-space
Kernel-space
Physical
libpacp
enp0s1 enp0s2 wlan1
BPF BPF BPF

24. IPTABLES APPROACH
Implement a daemon
Capture packets by iptables module
(NFQueue)
Feed packets to nDPI engine to get its
type

25. Network Stack
Application
nDPI Engine
User-space
Kernel-space
Physical
libnetfilter_queue
enp0s1 enp0s2 wlan1
netfilter
nfqueue

26. IPTABLES APPROACH
Use netlink to pass packets from/to user-
kernel
Flexible than pcap approach
Iptables rules

27. IPTABLES APPROACH
Use netlink to pass packets from/to user-
kernel
Flexible than pcap approach
Iptables rules

28. Iptables -A INPUT -j NFQUEUE --queue-num 0
Iptables -A FORWARD -j NFQUEUE --queue-num 0
Iptables -A FORWARD -i br0 -j NFQUEUE --queue-
num 0

29. Next Challenge..
How route packet by nDPI result?

30. ROUTE
Tag the packets and route by tag
Open the RAW socket to transmit that
packet(?)

31. NFQUEUE
We can send packets back to linux kern
el
Via netlink
It’s based on sk_buff structure

32. NFQUEUE
We can use the mark to represent the
application ID.
And then we can use that mark in the
iptables to accept/drop that packet

33. http://lt.netfilter.org/projects/libnetfilter_queue/doxygen/
group__Queue.html

37. Network Stack
Application
nDPI Engine
User-space
Kernel-space
Physical
libnetfilter_queue
enp0s1 enp0s2 wlan1
netfilter
nfqueue
- Receive Packets
- Detect Packet
- Mark Packet and send back
to Kernel
Routing
- Accept/Drop by mark
- Route ??

39. Iptables -i FORWARD -m mark --mark 0x0003 -j DROP
Iptables -i FORWARD -m mark --mark 0x0003 -j ACCEPT

40. But how to use that mark to route via different interfaces?

41. LINUX ROUTINGTABLE
Route packets by destination IP address in default.
Policy Route
Source IP address
L3/L4 protocols
Tos/Mark

42. LINUX ROUTINGTABLE
Multiple Routing table
Number from 0 - 32767 (high to low)
Build-in
0(local)
32766(main)
32767(default)
Custom

43. LINUX ROUTINGTABLE
echo 201 hwchiu.test >> /etc/iproute2/rt_tables
Use ip rule to manipulate the lookup order of tables.
ip rule add fmmark 10 table 201
Ip rule add from 140.113.235.234 fwmark 25 table 202
ip rule show

44. LINUX ROUTINGTABLE
Use ip route add to add the routing rule into table.
ip route add default via 10.0.2.2 dev enp0s3 table 201

45. Network Stack
Application
nDPI Engine
User-space
Kernel-space
Physical
libnetfilter_queue
enp0s1 enp0s2 wlan1
netfilter
nfqueue
- Receive Packets
- Detect Packet
- Mark Packet and send back
to Kernel
PRDB
main local default
custom

46. nfqueue
User-space
Kernel-Space
Physical Interface
enp0s1
Driver
netfilter_system
kernel_system
Thread
nfqueue_verdict
netfilter_system
enp0s2
Driver
call nDPI

47. How About Performance ?

48. PERFORMANCE
All packets have same l3/l4 tuple belong to
same connection (mostly)
We don’t need to detect all packets to
know its application.
Just pass the unknown connection packets
to nDPI engine

49. PERFORMANCE
Use the connmark to set the mark to
connection tracking entry.
Save the mark based on its skb_buff
Iptables -t mangle -j CONNMARK --save-
mark

50. nfqueue
User-space
Kernel-Space
Physical Interface
enp0s1
Driver
netfilter_system
kernel_system
Thread
nfqueue_verdict
netfilter_system
enp0s2
Driver
call nDPI

51. PERFORMANCE
You can also add those connection
tuple(l3/l4) to hardware to get the high
performance
Remember, only few packets need to be
detected.

52. Thanks!