40GbE Server Inter-Connect. / ENOG17 Meeting in Echigo. Author: SAKURA Internet Research Center / Naoto MATSUMOTO

1. 23 Oct, 2012
SAKURA Internet Research Center
Naoto MATSUMOTO
(C)Copyright 1996-2010 SAKURA Internet Inc.

2. Bulk Transfer(RX) Benchmark
1/10/40GbE/FDR10 Bulk Transfer(RX) Benchmark result
High is faster Low is better
Mellanox ConnectX3 FDR10-HCA (IPoIB)
29.56 Gbit/s** 61Kpps (MTU 65,520)
Mellanox ConnectX3 40GbE-NIC (IPoEth)
27.45 Gbit/s* 2.28Mpps (MTU 1,500)
Mellanox ConnectX3 10GbE-NIC (IPoEth)
9.93 Gbit/s* 827Kpps (MTU 1,500)
Intel 82599EB 10GbE-NIC (IPoEth)
9.98 Gbit/s* 820Kpps (MTU 1,500)
Realtek RTL8168B 1GbE-NIC (IPoEth)
0.98 Gbit/s* 81Kpps (MTU 1,500)
(Gbit/sec) (Packet/sec)
[System: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz / 32GB DDR3-DIMM]
[Traffic Generater: *pktgen, **iperf. Traffic Monitor: vnstat] Sourece: SAKURA Internet Research Center 07/2012

3. Packet Processing(RX) Graph
(Packet/sec)
1/10/40GbE/FDR10 Packet Processing(RX) Graph chart
(MTU Size)
** *
*
*
[System: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz / 32GB DDR3-DIMM]
[Traffic Generater: *pktgen, **ib_write_bw. Traffic Monitor: *vnstat, **ib_write_bw] Sourece: SAKURA Internet Research Center 07/2012

4. Small Packet(RX) Benchmark
1/10/40GbE/FDR10 Small Packet Processing(RX) Benchmark result
High is better
Mellanox ConnectX3 FDR10-HCA (ib_write_bw) 9.97Mpps** (MTU 64)
Mellanox ConnectX3 40GbE-NIC (IPoEth) 4.56Mpps* (MTU 64)
Mellanox ConnectX3 10GbE-NIC (IPoEth) 3.47Mpps* (MTU 64)
Intel 82599EB 10GbE-NIC (IPoEth) 2.12Mpps* (MTU 64) Copyright(c)
Japan Road Traffic
Information Center
Realtek RTL8168B 1GbE-NIC (IPoEth) 0.32Mpps* (MTU 64) All rights reserved.
Sourece: SAKURA Internet Research Center 07/2012
(Million Packet/sec by MTU 64)
[System: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz / 32GB DDR3-DIMM]
[Traffic Generater: *pktgen, **ib_write_bw. Traffic Monitor: *vnstat, **ib_write_bw]
Copyright© 2008-2011 MLIT Japan.
All Rights Reserved.

5. Eth/IB MTU Size Overview
IP over Infiniband (MTU 65,520)
IP over Ethernet (MTU 16,000)
IP over Ethernet (MTU 4,000)
IP over Ethernet (MTU 9,000)
IP over Ethernet (MTU 1,500)
Sourece: SAKURA Internet Research Center 07/2012

6. World Fastest Interface
Mellanx Connect-IB 100Gb/s HCA
PCI Express 3.0 x16
Copyright© 2008-2011 MLIT Japan. All Rights Reserved.

7. Next-gen Server type
Copyright(c)
Japan Road Traffic
Information Center
All rights reserved.

8. Future Technology(ARM)
Hyper Scale Server Packet Processing (Future Technology) Overview
2.14Mpps (MTU 64)
204Kpps (MTU 64)
(Packet/sec) Copyright(c)
Japan Road Traffic
Information Center
All rights reserved.
[System: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz / 32GB DDR3-DIMM]
[Traffic Generater: *pktgen, Traffic Monitor: vnstat, ifconfig] Sourece: SAKURA Internet Research Center 07/2012

9. Predictions of 2014E
Source: The Mobile Internet Report . December 2009 /© 2009-2012 Morgan Stanley. All rights reserved.

10. Thanks for your interest!