The introduction of network

1. HIGH PERFORMANCE
NETWORK
HUNG WEI CHIU

2. WHO AM I
• Hung-Wei Chiu (邱宏瑋)
• hwchiu@linkernetworks.com
• hwchiu.com
• Experience
• Software Engineer at Linker Netowrks
• Software Engineer at Synology (2014~2017)
• Co-Found of SDNDS-TW
• Open Source experience
• SDN related projects (mininet, ONOS, Floodlight, awesome-sdn)

3. WHAT WE DISCUSS TODAY
 The Drawback of Current Network Stack.
 High Performance Network Model
 DPDK
 RDMA
 Case Study

4. DRAWBACK OF CURRENT NETWORK STACK
• Linux Kernel Stack
• TCP Stack
• Packets Processing in Linux Kernel

5. LINUX KERNEL TCP/IP NETWORK STACK
• Have you imaged how applications communicate by network?

6. Linux Linux
www-server
Chrome
Network
PACKET

7. IN YOUR APPLICATION (CHROME).
• Create a Socket
• Connect to Aurora-Server (we use TCP)
• Send/Receives Packets. User-Space
Kernel-Space Copy data from the user-space
 Handle TCP
 Handle IPv4
 Handle Ethernet
 Handle Physical
 Handle Driver/NIC

8. Had you wrote a socket programming before ?

9. FOR GO LANGUAGE

10. FOR PYTHON

11. FOR C LANGUAGE

12. Did you image how kernel handle those operations ?

13. HOW ABOUT THE KERNEL ?
SEND MESSAGE
• User Space -> send(data….)
• SYSCALL_DEFINE3(….)  kernel space.
• vfs_write
• do_sync_write
• sock_aio_write
• do_sock_write
• __sock_sendmsg
• security_socket_sendmsg(…)

14. • inet_sendmsg
• tcp_sendmsg  finally
TCP …
• __tcp_push_pending_frames
• Tcp_push_one
• tcp_write_xmit
• tcp_transmit_skb
• ip_queue_xmit ---> finally
IP
• ip_route_output_ports
• ip_route_output_flow ->
routing
• xfrm_lookup -> routing
• Ip_local_out
• dst_output
• ip_output
• …...

16. HOW ABOUT THE KERNEL ?
RECEIVE MESSAGE
• User Space -> read(data….)
• SYSCALL_DEFINE3(….)  Kernel Space
• …..

18. WHAT IS THE PROBLEM
• TCP
• Linux Kernel Network Stack
• How Linux process packets.

19. THE PROBLEM OF TCP
• Designed for WAN network environment
• Different hardware between now and then.
• Modify the implementation of TCP to improve its performance
• DCTCP (Data Center TCP)
• MPTCP (Multi Path TCP)
• Google BBR (Modify Congestion Control Algorithm)
• New Protocol
• [論文導讀]
• Re-architecting datacenter networks and stacks for low latency and high performance

20. THE PROBLEM OF LINUX NETWORK STACK
• Increasing network speeds: 10G  40G  100G
• Time between packets get smaller
• For 1538 bytes.
• 10 Gbis == 1230.4 ns
• 40 Gbis == 307.6 ns
• 100 Gbits == 123.0 ns
• Refer to
http://people.netfilter.org/hawk/presentations/LCA2015/net_stack_challenges_100G_L
CA2015.pdf
• Network stack challenges at increasing speeds The 100Gbit/s challenge

21. THE PROBLEM OF LINUX NETWORK STACK
• For smallest frame size 84 bytes.
• At 10Gbit/s == 67.2 ns, (14.88 Mpps) (packet per second)
• For 3GHz CPU, 201 CPU cycles for each packet.
• System call overhead
• 75.34 ns (Intel CPU E5-2630 )
• Spinlock + unlock
• 16.1ns

22. THE PROBLEM OF LINUX NETWORK STACK
• A single cache-miss:
• 32 ns
• Atomic operations
• 8.25 ns
• Basic sync mechanisms
• Spin (16ns)
• IRQ (2 ~ 14 ns)

23. SO..
• For smallest frame size 84 bytes.
• At 10Gbit/s == 67.2 ns, (14.88 Mpps) (packet per second)
• 75.34+16.1+32+8.25+14 = 145.69

24. PACKET PROCESSING
• Let we watch the graph again

27. PACKET PROCESSING
• When a network card receives a packet.
• Sends the packet to its receive queue (RX)
• System (kernel) needs to know the packet is coming and pass the data to a
allocated buffer.
• Polling/Interrupt
• Allocate skb_buff for packet
• Copy the data to user-space
• Free the skb_buff

28. PACKETS PROCESSING IN LINUX
User Space
Kernel Space
NIC TX/RX Queue
Application
Socket Driver Ring Buffer

29. PROCESSING MODE
• Polling Mode
• Busy Looping
• CPU overloading
• High Network Performance/Throughput

30. PROCESSING MODE
• Interrupt Mode
• Read the packet when receives the interrupt
• Reduce CPU overhead.
• We don’t have too many CPU before.
• Worse network performance than polling mode.

31. MIX MODE
• Polling + Interrupt mode (NAPI) (New API)
• Interrupt first and then polling to fetch packets
• Combine the advantage of both mode.

32. SUMMARY
• Linux Kernel Overhead (System calls, locking, cache)
• Context switching on blocking I/O
• Interrupt handling in kernel
• Data copy between user space and kernel space.
• Too many unused network stack feature.
• Additional overhead for each packets

33. HOW TO SOLVE THE PROBLEM
• Out-of-tree network stack bypass solutions
• Netmap
• PF_RING
• DPDK
• RDMA

34. HOW TO SOLVE THE PROBLEM
• How did those models handle the packet in 62.7ns?
• Batching, preallocation, prefetching,
• Staying cpu/numa local, avoid locking.
• Reduce syscalls,
• Faster cache-optimal data structures

35. HOW TO SOLVE THE PROBLEM
• How did those models handle the packet in 62.7ns?
• Batching, preallocation, prefetching,
• Staying cpu/numa local, avoid locking.
• Reduce syscalls,
• Faster cache-optimal data structures

36. HOW TO SOLVE.
• Now. There’re more and more CPU in server.
• We can dedicated some CPU to handle network packets.
• Polling mode
• Zero-Copy
• Copy to the user-space iff the application needs to modify it.
• Sendfile(…)
• UIO (User Space I/O)
• mmap (memory mapping)

37. HIGH PERFORMANCE NETWORKING
• DPDK (Data Plane Development Kit)
• RDMA (Remote Directly Memory Access)

38. DPDK
• Supported by Intel
• Only the intel NIC support at first.
• Processor affinity / NUMA
• UIO
• Polling Mode
• Batch packet handling
• Kernel Bypass
• …etc

39. PACKETS PROCESSING IN DPDK
User Space
Kernel Space
NIC TX/RX Queue
Application DPDK
UIO (User Space IO)
Driver
Ring Buffer

40. COMPARE
Network Interface Card
Linux Kernel
Network Stack
Network Driver
Application
Network Interface Card
Linux Kernel
Network Stack
Network Driver
Application
Kernel
Space
User Space

41. WHAT’S THE PROBLEM.
• Without the Linux Kernel Network Stack
• How do we know what kind of the packets we received.
• Layer2 (MAC/Vlan)
• Layer3 (IPv4, IPv6)
• Layer4 (TCP,UDP,ICMP)

42. USER SPACE NETWORK STACK
• We need to build the user space network stack
• For each applications, we need to handle following issues.
• Parse packets
• Mac/Vlan
• IPv4/IPv6
• TCP/UDP/ICMP
• For TCP, we need to handle three-way handshake

43. FOR ALL EXISTING NETWORK APPLICATIONS
• Rewrite all socket related API to DPDK API
• DIY
• Find some OSS to help you
• dpdk-ans (c )
• mTCP (c )
• yanff (go)
• Those projects provide BSD-like interface for using.

44. SUPPORT DPDK?
• Storage
• Ceph
• Software Switch
• BSS
• FD.IO
• Open vSwitch
• ..etc

45. A USE CASE
• Software switch
• Application
• Combine both of above (Run Application as VM or Container)

46. Kernel
User
Open vSwitch(DPDK)
NIC(DPDK) NIC(DPDK)
Kernel
User
My Application
NIC(DPDK)

47. Kernel
User
Open vSwitch(DPDK)
NIC(DPDK) NIC(DPDK)
Container
1
Container
2
How container
connect to the
OpenvSwitch?

48. PROBLEMS OF CONNECTION
• Use VETH
• Kernel space again.
• Performance downgrade
• Virtio_user

49. RDMA
• Remote Direct Memory Access
• Original from DMA (Direct Memory Access)
• Access memory without interrupting CPU.

50. ADVANTAGES
• Zero-Copy
• Kernel bypass
• No CPU involvement
• Message based transactions
• Scatter/Gather entries support.

51. WHAT IT PROVIDES
• Low CPU usage
• High throughput
• Low-latency
• You can’t have those features in the same time.
• Refer to :Tips and tricks to optimize your RDMA code

56. SUPPORT RDMA
• Storage
• Ceph
• DRBD (Distributed Replicated Block Device)
• Tensorflow
• Case Study - Towards Zero Copy Dataflows using RDMA

57. CASE STUDY
• Towards Zero Copy Dataflows using RDMA
• 2017 SICCOM Poster
• Introduction
• What problem?
• How to solve ?
• How to implement ?
• Evaluation

58. INTRODUCTION
• Based on Tensorflow
• Distributed
• Based on RDMA
• Zero Copy
• Copy problem
• Contribute to Tensorflow (merged)

59. WHAT PROBLEMS
• Dataflow
• Directed Acyclic Graph
• Large data
• Hundred of MB
• Some data is unmodified.
• Too many copies operation
• User Space <-> User Space
• User Space <-> Kernel Space
• Kernel Space -> Physical devices

60. WHY DATA COPY IS BOTTLENECK
• Data buffer is bigger than the system L1/L2/L3 cache
• Too many cache miss (increate latency)
• A Single Application unlikely can congest the network bandwidth.
• Authors says.
• 20-30 GBs for data buffer 4KB
• 2-4 GBs for data buffer > 4MB
• Too many cache miss.

61. HOW TO SOLVE
• Too many data copies operations.
• Same device.
• Use DMA to pass data.
• Different device
• Use RDMA
• In order to read/write the remote GPU
• GPUDirect RDMA (published by Nvidia)

63. HOW TO IMPLEMENT
• Implement a memory allocator
• Parse the computational graph/distributed graph partition
• Register the memory with RDMA/DMA by the node’s type.
• In Tensorflow
• Replace the original gRPC format by RDMA

64. EVALUATION (TARGET)
• Tensorflow v1.2
• Basd on gRPC
• RDMA zero copy Tensorflow
• Yahoo open RDMA Tensorflow (still some copy operat Software ions)

65. EVALUATION (RESULT)
• RDMA (zero copy) v.s gRPC
• 2.43x
• RDMA (zero copy) v.s Yahoo version
• 1.21x
• Number of GPU, 16 v.s 1
• 13.8x

66. Q&A?

67. EVALUATION (HARDWARE)
• Server * 4
• DUal6-core Intel Xeon E5-2603v4 CPU
• 4 Nvidia Tesla K40m GPUs
• 256 GB DDR4-2400MHz
• Mellanox MT27500 40GbE NIC
• Switch
• 40Gbe RoCE Switch
• Priority Flow Control

68. EVALUATION (SOFTWARE)
• VGG16 CNN Model
• Model parameter size is 528 MB
• Synchronous
• Number of PS == Number of Workers
• Workers
• Use CPU+GPU
• Parameter Server
• Only CPU