Howto createOpenFlow Switchusing FPGA (at FPGAX#6)

How to create
OpenFlow Switch
using FPGA
Presented at FPGA Extreme Computing #6
2015/02/01
Kentaro Ebisawa | 海老澤健太郎 | Twitter: @ebiken

Disclaimer
•The objective of this presentation is to “share information”
related to OpenFlow Switch development and increase people
interested in OpenFlow and developing Network Equipments.
•Thus, it is not directly related to specification or design of any
existing product.
How to create OpenFlow Switch using FPGA | FPGA Extreeme Computing #6 | FPGAX 2014/02/01 | Twitter: @ebiken 2

Self-introduction
Current
• Dev manager of OpenFlow Switch
@ start-up company
(Riava Networks, Inc.)
Past
• Several Technical role at multiple startup
companies to help booting up team or
company, growing presence in Asia market.
• Head of support team
• Sales Engineer (SE)
• Solution Architect designing OSS/BSS.
• Product design and research.
• Development management.
• Twitter: @ebiken
• Technical Area
• ATM / Ethernet / PPPoE / L2TP / RADIUS
• Streaming / Web Proxy
• Virtualization
• OpenVZ / Parallels Cloud Server
• Hosting Service. OSS/BSS.
• Operation / Business Support System
• IPv6/v4 Translation (nat64)
• OpenFlow

List of Contents
1. What is “OpenFlow Switch” ?
(Comparing with Layer 2 / 3 switch)
2. Why use FPGA to build OpenFlow Switch?
3. OpenFlow Switch components
4. OpenFlow Switch design trade-offs

What is “OpenFlow Switch” ?
Comparing with Layer 2 / 3 switch.

How traditional network switch works (Layer 2)
MAC Address Port
00-00-5E-00-53-01 1
00-00-5E-00-53-02 2
00-00-5E-00-53-03 3
00-00-5E-00-53-04 4
Layer 2 forwarding (MAC learning) table
1 2 3 4 5
-01 -02 -03 -04 -05
Table Entry
• Self learning (Source MAC)
Action (match)
• Forward to Port
Action (no-match)
• Flooding
VLAN 1 VLAN 2

How traditional network switch works (Layer 3)
Subnet (LPM) Gateway Port
192.168.10.0 /24 10.0.0.5 5
192.168.20.0 /24 172.16.0.1 1
... ... ...
0.0.0.0/0 172.16.0.1 1
Layer 3 forwarding table
1 2 3 4 5
172.16.0.0/24 10.0.0.0/24
Table Entry
• Routing Protocol etc.
Action (match)
• Decrement TTL
• Modify Destination MAC
• Forward to Port
Action (no-match)
• Drop
• Send “ICMP Dest untreachable”

How OpenFlow switch works
Multiple Flow (Lookup) Tables
1 2 3 4 5
Match Field Instruction / Action
MPLS Label pop MPLS label, meter 10, goto-table 4
VLAN output port 2, ...
IPv4 Address, TCP port set IPv4 Address, Queue 10, goto-table 3
... ...
Table Entry
• Inject from Controller
... ...
... ...
VLAN output port 2
... ...
OpenFlow Controller

Which packet? (OpenFlow Match Field : 40 Fields, 1261 bits)
Field Name bits
OXM_OF_IN_PORT 32
OXM_OF_IN_PHY_PORT 32
OXM_OF_METADATA 64
OXM_OF_ETH_DST 48
OXM_OF_ETH_SRC 48
OXM_OF_ETH_TYPE 16
OXM_OF_VLAN_VID 12+1
OXM_OF_VLAN_PCP 3
OXM_OF_IP_DSCP 6
OXM_OF_IP_ECN 2
OXM_OF_IP_PROTO 8
OXM_OF_IPV4_SRC 32
OXM_OF_IPV4_DST 32
OXM_OF_TCP_SRC 16
Field Name bits
OXM_OF_TCP_DST 16
OXM_OF_UDP_SRC 16
OXM_OF_UDP_DST 16
OXM_OF_SCTP_SRC 16
OXM_OF_SCTP_DST 16
OXM_OF_ICMPV4_TYPE 8
OXM_OF_ICMPV4_CODE 8
OXM_OF_ARP_OP 16
OXM_OF_ARP_SPA 32
OXM_OF_ARP_TPA 32
OXM_OF_ARP_SHA 48
OXM_OF_ARP_THA 48
OXM_OF_IPV6_SRC 128
Field Name bits
OXM_OF_IPV6_DST 128
OXM_OF_IPV6_FLABEL 20
OXM_OF_ICMPV6_TYPE 8
OXM_OF_ICMPV6_CODE 8
OXM_OF_IPV6_ND_TARGET 128
OXM_OF_IPV6_ND_SLL 48
OXM_OF_IPV6_ND_TLL 48
OXM_OF_MPLS_LABEL 20
OXM_OF_MPLS_TC 3
OXM_OF_MPLS_BOS 1
OXM_OF_PBB_ISID 24
OXM_OF_TUNNEL_ID 64
OXM_OF_IPV6_EXTHDR 9

How to process it? (OpenFlow Instruction / Action)
Instruction
Meter Policing（Rate-Limit）
Apply-Actions Apply Imidiately
Clear-Actions Initialize Action Set
Write-Actions Add to Action Set
Write-Metadata Stamp metadata
Goto-Table <ID>
Go through another
table
Action （アクション）
Output <port_no> Forward packet
Group <group_id> Group processing
Set-Queue <queue_id> Assign Queue
Push-Tag <ethertype>
MPLS,PBB,VLAN
Pop-Tag <ethertype>
Set-Field
<field_type> <value>
Set value to arbitrary
fields.
Set TTL <ttl>
MPLS, IP
TTL (Time To Live)
Decrement TTL
Copy TTL out/inwards

OpenFlow Pipeline
OpenFlow Pipeline
Flow Table #0
Flow Entry
Flow Entry
Flow Table #n
Flow Entry
Flow Entry
Match
Flow Table #1
Flow Entry
Action Set
(associated with each packet)
Flow Entry
Action
Action
Action
Packet
Ingress
Match
Match
Goto-Table Goto-Table Packet
Output
Execute
Action
Set(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)

Comparing Layer 2/3 Switch and OpenFlow Switch
Which packet? How to process it?
Layer 2 MAC Address
Forward to port where
Host with Dest MAC exists
Layer 3 IP Address
Set Dest MAC Address
Forward to Next Hop
OpenFlow
Set from Controller
(Based on Packet header)
Set from Controller
(Set header fields,
forward, QoS etc.)

Which packet? How to process it?
Layer 2 MAC Address
Forward to port where
Host with Dest MAC exists
Layer 3 IP Address
Set Dest MAC Address
Forward to Next Hop
OpenFlow
Set from Controller
(Based on Packet header)
Set from Controller
(Set header fields,
forward, QoS etc.)
Comparing Layer 2/3 Switch and OpenFlow Switch
How to Match/Process is pre-defined.
How to Match/Process is programmable.

Why use FPGA to build
OpenFlow Switch?

3 reasons for using FPGA to build OpenFlow Switch
Avoid restriction of existing Network Chip
Keep up with new version of standard.
Continuous improvements based on new use cases.

Avoid restriction of existing Network Chip
• Difficult to implement Full OpenFlow Pipeline using existing
network chip aimed for Layer 2/3 switch.
• TCAM is small
• Mostly used for ACL only.
• Re-use of L2/L3 Forwarding Table
• Limited match fields available on each table.
• Less Instruction / Action supported.
• Some tables cannot use most Instructions / Actions.

Continuous improvements based on new use cases.
• OpenFlow will make network programmable.
• Programmable = expected behavior / feature depends more on
Business requirements. (Compared to L2 / L3 switch)
• Sizing balance (where to put more HW resource) depends on
business model.
• Group, Meter, QoS, Packet-in/out … which one is more important??
• Datacenter CLOS Network ?
• Carrier / Provider Edge Gateway ?
• Security Gateway ?
• Wireless ?

Keep up with new version of standard.
2010 2011 2012 2013 2014
(Dec. 31, 2009)
1.0.0
(Jun. 7, 2012)
1.0.1 Errata
(Nov. 1, 2013)
1.0.2 Errata
(Apr. 13, 2012)
1.3.0
1.3.1
(Sept. 6, 2012)
(Apr. 25, 2013)
1.3.2
(Mar. 27, 2014)
1.3.4
1.3.3
(Dec. 18, 2013)
(Feb. 28, 2011)
1.1.0
(Dec. 5, 2011)
1.2
(Oct. 15, 2013)
1.4.0
0x01
0x02
0x04
0x05
0x03
Wire Protocol #
(Jan. 9, 2015)
1.5.0
0x06History of OpenFlow Standard
2015
https://www.opennetworking.org/sdn-resources/technical-library#tech-spec

OpenFlow Switch components
Using Hardware Datapath

OpenFlow Switch components | Software
Hardware Abstraction Layer
Hardware Specific Driver
Datapath
PCIe
Open Flow Agent
Protocol Interface
Admin Interface
• Switch/Controller IP Address configuration
• Showing Status / Statistics
• Debug, Log
OF Protocol <-> Datapath data format conversion.
Managing Flow Entry Layout in Datapath (Stored in
TCAM/RAM）
Packet-in/out processing.
PCIe Driver.
FPGA packet processing board (Datapath) will be
recognized as PCI express device from OS.
Admin Interface
(CLI, syslog, SNMP)
CPU Board
FPGA / TCAM / RAM(*)
Physical Network Ports
(*) RAM = SRAM/DRAM/BlockRAM etc.
Protocol level interface between controller.
Managing Flow Table data coupled with OpenFlow
protocol.

OpenFlow Switch ~ Hardware Datapath Implementation Types
Datapath (FPGA etc.)
Flow
Table #0
Flow
Table #x
Flow
Table #N
Software (CPU)
Flow
Table #0
Flow
Table #x
Flow
Table #N
Insert Flow Table
OpenFlow Agent
Packet
Type 1: Flow Installation

Flow Cache
Table
Software (CPU)
Flow
Table #0
Flow
Table #x
Flow
Table #N
OpenFlow Agent
Packet
Type 2: Flow Cache (No Cache Entry)
Insert Flow CacheRequest

Flow Cache
Table
Software (CPU)
Flow
Table #0
Flow
Table #x
Flow
Table #N
OpenFlow Agent
Packet
Type 2: Flow Cache (With Cache Entry)

Type 1: Flow Installation Type 2: Flow Cache
 Packet in same flow has same latency
 Large delay processing initial packet of
each flow
 (Relatively) Easy to expect performance.
 Bottleneck between Datapath/CPU
 Significant performance impact when
cache over flows
 (Relatively) easy to add / mod Flow Entry
 Require Flushing Cache when add / mod
Flow Entry (Multi Table)
(Could be avoided with smart algorism but
complex)
 Consume more HW resource (for non-used
Flow Entry)
 Consume less HW resource (only active
flow)

Type 1: Flow Installation Type 2: Flow Cache
 Packet in same flow has same latency
 Large delay processing initial packet of
each flow
 (Relatively) Easy to expect performance.
 Bottleneck between Datapath/CPU
 Significant performance impact when
cache over flows
 (Relatively) easy to add / mod Flow Entry
 Require Flushing Cache when add / mod
Flow Entry (Multi Table)
(Could be avoided with smart algorism but
complex)
 Consume more HW resource (for non-used
Flow Entry)
 Consume less HW resource (only active
flow)
Today we will discuss
Type 1 only.

OpenFlow Switch components | Hardware Datapath
Hardware Abstraction Layer
Hardware Specific Driver
Datapath
PCIe
Open Flow Agent
Protocol Interface
Admin Interface
• Switch/Controller IP Address configuration
• Showing Status / Statistics
• Debug, Log
OF Protocol <-> Datapath data format conversion.
Managing Flow Entry Layout in Datapath (Stored in
TCAM/RAM）
Packet-in/out processing.
PCIe Driver.
FPGA packet processing board (Datapath) will be
recognized as PCI express device from OS.
Admin Interface
(CLI, syslog, SNMP)
CPU Board
FPGA / TCAM / RAM(*)
Physical Network Ports
(*) RAM = SRAM/DRAM/BlockRAM etc.
Protocol level interface between controller.
Managing Flow Table data coupled with OpenFlow
protocol. Now Moving to
Datapath.

CPU Board
(OpenFlow Agent)
Physical Ports (10Gbps / 1Gbps)
PCIe Core
DMA Control
Host Interface
MAC (XAUI / QSGMII)
TCAM
(Match Field)
SRAM | DRAM
(Instruction, Action)
(Packet Buffer)
Packet FIFO
OpenFlow Pipeline Processor
Flow Table ProviderFPGA Switch / Queue
PCIe
Insert Flow Entry

CPU Board
(OpenFlow Agent)
Physical Ports (10Gbps / 1Gbps)
PCIe Core
DMA Control
Host Interface
MAC (XAUI / QSGMII)
TCAM
(Match Field)
SRAM | DRAM
(Instruction, Action)
(Packet Buffer)
Packet FIFO
OpenFlow Pipeline Processor
Flow Table ProviderFPGA Switch / Queue
PCIe
Packet Input

OpenFlow Switch design trade-offs
a. Long Search / TCAM Entry Key Length
b. Update considering Priority
c. A lot of packet fields to process

Layer 2 Forwarding (MAC learning) Table Search (Using CAM)
MAC Address (48 bit) Port
00-00-5E-00-53-FE 5
00-15-17-ED-9E-5B 3
6C-F0-49-7F-79-AA 1
00-00-5E-00-53-01 1
00-15-17-ED-9E-01 2
… …
00-00-5E-00-53-01
Search Key
Forward to Port #1
CAM (Exact Match)

Layer 2 Forwarding (MAC learning) Table Search (Hash Table)
MAC Address Port MAC Address Port MAC Address Port
0 ⇒ 00-00-5E-00-53-FE 5
1 ⇒ 00-15-17-ED-9E-00 3 00-15-17-ED-9E-2F 2 00-15-17-ED-9E-8B 4
2 ⇒ 6C-F0-49-7F-79-AA 1
3 ⇒ 00-00-5E-00-53-08 3 00-00-5E-00-53-01 1
4 ⇒ 00-15-17-ED-9E-01 2 00-00-5E-00-53-F0 4 00-15-17-ED-9E-32 8
… ⇒ … … … … … …
00-00-5E-00-53-01
Search Key
Forward to Port #1
HASH
Hash Table (on SRAM)

Layer 3 Forwarding Table Search (TCAM - Longest Prefix Match)
TCAM (with Don’t care bit)
IPv4 Address (32 bit) Gateway
192 168 10 * 10.0.0.5
10 128 0 * 172.16.0.1
192 168 20 * 172.16.0.2
192 168 * * 10.0.0.100
... ... ... ... ...
* * * * 172.16.0.1
192.168.20.100
Search Key
• TCAM is required for LPM
• Search Key has fixed length
• Only 64bit even for IPv6
※ Actually masked per bit

OpenFlow Flow Table (TCAM – Simple design)
IN_PORT METADATA ETH_DST ETH_SRC VLAN_VID MPLS_LBL IP_PROTO IPV4_SRC …
10 * * * 1 * * 10.1.1.* …
* * * * * * 6 10.1.1.1 …
* * * * 10 * * * …
5 * * * 5 * * * …
* 0xABAB0010 * * * * 17 * …
… … … … … … … … …
PORT, METADATA, ETH_DST …
Search Key (1248 bit)
Instruction/ Action
pop VLAN, …
push vlan, goto-table 1
Output 5, …
set-metadata, …
set-field …
…
TCAM (40 Fields, 1261 bits)
• Inefficient use of TCAM
• 1261 bit per TCAM Entry => about 40Mbit per 32K Entry
• Only a few fields used per Flow Entry
• Entry has Priority => Could require shifting when insert.

How to deal with Long Search Key (example 1)
参考文献：”OF-DPA Abstract Switch Specification ver1.0” by Broadcom Corporation.
Simplified “Figure 2: OpenFlow OF-DPA Abstract Switch Pipeline“
https://github.com/Broadcom-Switch/of-dpa/blob/master/doc/OFDPA_OASS-ETP101-R.PDF
Ingress
Port
Table 1
MAC
Table 3
Unicast
Routing
Table 4a
Multicast
Routing
Table 4b
Bridge
Table 4c
ACL Policy
Table 5
VLAN
Table 2
Match limited field per Table
 Some table does not require TCAM
 Cannot match entry using AND operation
(ex： MAC_DST==XXX && VLAN_ID==YY)
Search
Process
CAM or
SRAM
TCAM

Update considering Priority (using TCAM)
Strict Priority for all entries  Loose Priority (*1)
Resource efficiency (BAD  GOOD) | Management Logic (Simple  Complex)
Priority PORT VLAN IP Addr TCP Port
255 3 * * 80
255 1 20 10.0.0.* *
50 * * * 80
10 1 * * 443
0 * * * *
255 3 * * 80
255 1 20 10.0.0.* *
50 * * * 80
10 1 * * 443
0 * * * *
10 * 10 10.0.0.* 22
(*1) Sort in priority order for each set of rules which can match same packet.

Trade-off depending on how to process packet fields.
Layer 3 Switch
• Modify MAC DA
• DA: Destination Address
• Decrement TTL
• Layer 2 Processing
MAC DA MAC SA VLAN EtherType MPLS IPv4 / IPv6 / ARP TCP / UDP / ICMP Payload
Layer 2 Switch
• VLAN Tag Push/Pop
Layer 2 / 3 Switch modifies only a few packet fields.

OpenFlow Switch
• VLAN Tag Push/Pop
• MPLS Push / Pop
• Modification of almost any header fields

parallel processing
 Delay: small (1 ~ a few cycles)
 Logic size: Huge
Sequentially processing each field
 Delay: big (10s ~ 100+ cycles)
 Logic size: Small

For people who wants to try it…
Packet processing using FPGA is FUN!!

• Can buy at Digilent Inc.
• Part# 410-301P-KIT
• Academic：$1,695
• List price: $24,500
How to try?
• NetFPGA SUME
• NIC type FPGA board for research
and experimentation.
• FPGA: Virtex-7 690T
• Network:
• 4 x 10Gbps (SFP+)
• Memory:
• 2 x 4GB DDR3 SoDIMM
• 3 x 72Mbits QDR II SRAM
• PCIe x8 Gen. 3 (8Gbps x8)
• 2 x SATA-III
Photo from Digilent Inc.
http://digilentinc.com/Products/Detail.cfm?NavPath=2,1301,1311&Prod=NETFPGA-10G-SUME

How to try?
• OpenFlow implementation using
FPGA available on GitHub
• A bit old but good start.
•  Design Document
• Block Diagram
• State Machine
•  Source Code
• GitHub
•  OpenFlow 1.0 only
• Single Table (No multi-table)
https://github.com/NetFPGA/NetFPGA-public/wiki/NetFPGA-10G-OpenFlow-Switch

!! Let’s have fun together !!
FPGA + OpenFlow
海老澤健太郎｜ Kentaro Ebisawa
Twitter: @ebiken

Howto createOpenFlow Switchusing FPGA (at FPGAX#6)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to Howto createOpenFlow Switchusing FPGA (at FPGAX#6)

Similar to Howto createOpenFlow Switchusing FPGA (at FPGAX#6) (20)

More from Kentaro Ebisawa

More from Kentaro Ebisawa (20)

Recently uploaded

Recently uploaded (20)

Howto createOpenFlow Switchusing FPGA (at FPGAX#6)