1. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Network packet capture in Linux kernelspace
An overview of the network stack in the Linux kernel
Beraldo Leal
beraldo@ime.usp.br
http://www.ime.usp.br/~beraldo/
Institute of Mathematics and Statistics - IME
University of Sao Paulo - USP
25th October 2011
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 1 / 25

2. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Outline
Introduction
Network stack
Packet ingress flow
Methods to capture packets
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 2 / 25

3. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Introduction
• Sniffers;
• Improvements in packet reception;
• Linux kernel network subsystem;
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 3 / 25

4. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Sniffers
• tcpdump, wireshark, snort, etc;
• Using the well-known library libpcap;
• Not suitable for > 10 Gbps;
• Packet loss;
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 4 / 25

5. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Improvements in packet reception
• Commodity hardware for packet capture;
• 3COM
• Intel
• endace, ...
• Many Interruptions
• NEW API or NAPI (interruption coalescence)
• zero-copy
• Direct Memory Access - DMA
• mmap()
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 5 / 25

6. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Linux kernel network subsystem
• Kernel number of files: 36.680 1 2
• net/ number of files: 1.293 ( 3.5% )
• drivers/net/ number of files: 1.935 ( 5.27% )
• Kernel SLOC: 9.723.525
• net/ SLOC: 480.928 ( 5% )
• drivers/net/ SLOC: 1.155.317 ( 12% )
1
kernel 3.0.0
2
source: wc, find, cat, etc..
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 6 / 25

8. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Network stack
L5: Application
http, ftp, ssh, telnet, ... (message)
L4: Transport
tcp, udp, ... (segment)
L3: Network
ipv4, ipv6, ... (datagram/packet)
L1/2: Link / host-to-network
ethernet, token ring, ... (frame)
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 8 / 25

9. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Important data structs:
• net device
• include/linux/netdevice.h
• sk buff
• include/linux/skbuff.h
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 9 / 25

10. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Important data structs:
• net device (include/linux/netdevice.h)
• unsigned int mtu
• unsigned int flags
• unsigned char dev addr[MAX ADDR LEN]
• int promiscuity
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 10 / 25

11. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Important data structs:
• sk buff (include/linux/skbuff.h)
• struct sk buff *next;
• struct sk buff *prev;
• ktime t tstamp;
• struct net device *dev;
• unsigned int len;
• unsigned int data len;
• u16 mac len;
• u8 pkt type;
• be16 protocol;
• sk buff data t transport header; (old h)
• sk buff data t network header; (old nh)
• sk buff data t mac header; (old mac)
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 11 / 25

12. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Important sk buff routines
• alloc skb();
• dev alloc skb();
• kfree skb();
• dev kfree skb();
• skb clone();
• skb network header(skb);
• skb transport header(skb);
• skb mac header(skb);
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 12 / 25

13. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Packet ingress flow
• When working in interrupt driven model, the nic registers an
interrupt handler;
• This interrupt handler will be called when a frame is received;
• Typically in the handler, we allocate sk buff by calling
dev alloc skb();
• Copies data from nic’s buffer to this struct just created;
• nic call generic reception routine netif rx();
• netif rx() put frame in per cpu queue;
• if queue is full, drop!
• net rx action() decision based on skb->protocol;
• This function basically dequeues the frame and delivery a copy
for every protocol handler;
• ptype all and ptype base queues
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 13 / 25

14. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Packet ingress flow
• ip v4 rcv() will receive the ip datagram (if is a ipv4 packet);
• ip checksum, check ip headers, ....
• ip rcv finish() makes route decision (ip forward() or
ip local delivery())
• ip local delivery() defrag fragmented packets, and call
ip local deliver finish()
• ip local deliver finish() find protocol handler again;
• tcp v4 rcv(), udp rcv(), or other L4 protocol handler
• ...
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 14 / 25

15. ip_local_deliver_finish() <continue>
(net/ipv4/ip_input.c) NF_IP_FORWARD
find protocol handler or ip_forward()
send icmp_dst_unreach Layer 3
(net/ipv4/ip_forward.c)
Network
handle route alert;
ip_local_deliver() send redirect if
(net/ipv4/ip_input.c) necessary;
NF_IP_LOCAL_IN
defrag fragmented decrease TTL;
packets verify if frag is
possible (mtu)
ip_rcv_finish()
ip_error()
NF_IP_PRE_ROUTING (net/ipv4/ip_input.c)
(net/ipv4/route.c)
find route and handle
routing error, send
IP options
icmp pkt
ip_rcv() packet_rcv()
arp_rcv()
(net/ipv4/ip_input.c) <tcpdump_process> <...>
(handle arp requests
verify skb, IP headers <dhcpd process>
and replies)
and IP checksum <...>
net_rx_action()
netif_rx() input_queue (net/core/dev.c)
(net/core/dev.c) [cpu] decision based on
skb->protocol field
Layer 1/2
Physical/Link
Network Drivers
(drivers/net/*)

16. Application
userspace
kernelspace
Socket Layer
(net/core/sock.c)
__tcp_v4_lookup()
tcp_v4_do_rcv()
(net/ipv4/tcp_ipv4.c) generate ICMP
(net/ipv4/tcp_ipv4.c)
check for socket in error
check for socket state
LISTEN, with dst_port
tcp_v4_rcv() udp_rcv()
(net/ipv4/tcp_ipv4.c) (net/ipv4/udp.c) <...> Layer 4
check for tcp headers check for udp headers Transport
ip_local_deliver_finish() <continue>
(net/ipv4/ip_input.c) NF_IP_FORWARD
find protocol handler or ip_forward()
send icmp_dst_unreach Layer 3
(net/ipv4/ip_forward.c)
Network
handle route alert;
ip_local_deliver() send redirect if
(net/ipv4/ip_input.c) necessary;
NF_IP_LOCAL_IN
defrag fragmented decrease TTL;
packets verify if frag is
possible (mtu)
ip_rcv_finish()
ip_error()
NF_IP_PRE_ROUTING (net/ipv4/ip_input.c)
(net/ipv4/route.c)
find route and handle
routing error, send
IP options
icmp pkt

17. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Methods to capture packets
• protocol handler
• register a function to handler packets with dev add pack()
• netfilter hooks
• userspace tools;
• socket AF PACKET, libpcap, ...
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 17 / 25

18. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
1 struct packet type my proto;
2
3 int my packet rcv(struct sk buff ∗skb, struct net device ∗dev,
4 struct packet type ∗pt, struct net device ∗orig dev) {
5
6 printk(KERN ERR ”+ 1!n”);
7
8 kfree skb(skb);
9 return 0;
10 }
11
12 static int hello init(void) {
13 printk(”<1> Hello world!n”);
14
15 my proto.type = htons(ETH P ALL);
16 my proto.dev = NULL;
17 my proto.func = my packet rcv;
18
19 dev add pack(&my proto);
20 return 0;
21 }
22
23 static void hello exit(void) {
24 dev remove pack(&my proto);
25 printk(”<1> Bye, cruel worldn”);
26 }
27 module init(hello init);
28 module exit(hello exit);
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 18 / 25

19. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
1 int my packet rcv(struct sk buff ∗skb, struct net device ∗dev, struct packet type ∗pt, struct net device
∗orig dev)
2 {
3 switch (skb−>pkt type) {
4 case PACKET HOST:
5 printk(”PACKET HOST − ”);
6 break;
7 case PACKET BROADCAST:
8 printk(”PACKET BROADCAST − ”);
9 break;
10 case PACKET MULTICAST:
11 printk(”PACKET MULTICAST − ”);
12 break;
13 case PACKET OTHERHOST:
14 printk(”PACKET OTHERHOST − ”);
15 break;
16 case PACKET OUTGOING:
17 printk(”PACKET OUTGOING − ”);
18 break;
19 case PACKET LOOPBACK:
20 printk(”PACKET LOOPBACK − ”);
21 break;
22 case PACKET FASTROUTE:
23 printk(”PACKET FASTROUTE − ”);
24 break;
25 }
26 printk(”%s 0x%.4X 0x%.4X n”, skb−>dev−>name, ntohs(skb−>protocol), ip hdr(skb)−>protocol)
27
28 kfree skb(skb);
29 return 0;
30 }
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 19 / 25

20. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Netfilter hooks
• iptables = userspace;
• netfilter = kernelspace;
• Netfilter is merely a series of hooks in various points in a
protocol stack;
• packet filtering, network address [and port] translation
(NA[P]T) and other packet mangling;
• www.netfilter.org
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 20 / 25

23. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
References
• br.kernelnewbies.org/node/150 has many links
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 23 / 25

24. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Thankyou! Question?
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 24 / 25

25. Introduction
Network stack
Packet ingress flow University of Sao Paulo - USP
Methods to capture packets
Network packet capture in Linux kernelspace
An overview of the network stack in the Linux kernel
Beraldo Leal
beraldo@ime.usp.br
http://www.ime.usp.br/~beraldo/
Institute of Mathematics and Statistics - IME
University of Sao Paulo - USP
25th October 2011
Beraldo Leal 25th October 2011 Network packet capture in Linux kernelspace 25 / 25