1.
v
Operationalizing VRF in the Data Center
Ivan Pepelnjak, David Ahern, Dinesh G Dutt
June 23, 2016

2.
Agenda
Why Do We Need VRF in the DC
VRF in the Linux Kernel
Operational Workflow of VRF
June 29, 2016 cumulusnetworks.com 2

3.
Key Takeaways
VRF is a first class citizen in Linux
Uniform model across hosts and routers
 Pushing routing all the way to the edge
Cumulus Linux 3.0 release supports all the
familiar constructs around VRF usage
June 29, 2016 cumulsnetworks.com 3

4.
Agenda
Why Do We Need VRF in the DC
VRF in the Linux Kernel
Operational Workflow of VRF
June 29, 2016 cumulusnetworks.com 4

5.
5 © ipSpace.net 2016 VRF 101
What Is a Virtual Routing and Forwarding (VRF) Table?
VRF = independent routing instance
• Routing protocols, routing and forwarding table
• Management plane is common
• Exposure to DoS is shared
• Inter-device connectivity with trunking interfaces, overlays or MPLS
VRF-capable router
Management
Trunking interface or MPLS
OSPF topology BGP table
OSPF process BGP process
IP routing table
IP forwarding table

6.
6 © ipSpace.net 2016 VRF 101
Typical Use Case: Management VRF
• Management plane processes attached to management VRF
• User VRF does not have a route to management IP address
• End-users cannot reach management plane even though they have IP connectivity to the switch
• Simpler than protecting every switch port with ACLs
VRF-capable switch
Management plane
Fabric
Management
VLAN

7.
7 © ipSpace.net 2016 VRF 101
Layer-3 Path Isolation (VRFs) Can Replace VLANs
Facts:
• VLANs are used in data center networks as a security zone mechanism
• Layer-3 path isolation (L3VPN) provides the same (or better) isolation
• L3VPN scales better than VLANs
Use cases:
• Retain separation while connecting L2 pods over L3 core (DC or WAN)
• Combine ACLs with firewalls
• Complex traffic forwarding policies

8.
8 © ipSpace.net 2016 VRF 101
Example: Multi-Tier Application with VRFs
Each segment needs a different default gateway
• Layer-2-only solution or multiple routing domains
You might need routing within a segment
(example: divert backup traffic directly to backup server)
• Multiple layer-3 forwarding domains are the only option
Implementations:
• MPLS/VPN
• VRF Lite (or Cisco EVN)
• EVPN with VXLAN Outside
Web servers App servers DB servers
Default gateway
Default gateway Default gateway

9.
9 © ipSpace.net 2016 VRF 101
Layer-3 Path Isolation in DCI Environment
Replace stretched VLANs with L3VPN
• Retain separation of security zones
• Cannot solve hot VM mobility or stretched L2 clusters
• Terminate L2 domains within data center
• Use parallel VRFs, MPLS/VPN or EVPN (over MPLS or VXLAN)
Parallel VRFs also known as: VRF-Lite, Multi VRF, EVN (Cisco), MCE (HP)
DWDM, ME, IP
Data Center A
DC core
Data Center B
DC core

10.
Agenda
Why Do We Need VRF in the DC
VRF in the Linux Kernel
Operational Workflow of VRF
June 29, 2016 cumulusnetworks.com 10

11.
cumulusnetworks.com
Linux Implementation for VRF
Linux Implementation
 Kernel v4.3 and forward
Developed by Cumulus Networks for Linux
 Consistent API across all Linux Devices - switches
and hosts

12.
cumulusnetworks.com 12
Distribution
Kernel
Version
VRF
status
Capabilities
Cumulus Linux 3.0 4.1 enabled All + Mgmt VRF
Debian - stretch 4.6 enabled
IPv4, IPv6 global,
“VRF All” TCP sockets
Ubuntu 16.04 4.4 enabled IPv4, IPv6 global
Fedora 24
(May beta)
4.5 disabled none
Fedora 23 4.5 disabled none
Linux OS Distributions (June 2016)

13.
Feature by Linux Kernel Version
13
Version Feature
4.3 Basic IPv4 support, FIB tracepoint
4.4 Basic IPv6 support (global addresses only)
4.5 “VRF-all” sockets, FIB6 tracepoint
4.6 Bug fixes, IPv6 global addresses on admin down
4.7 Ingress device for IP_PKTINFO, IP6_PKTINFO
4.8 IPv6 linklocal and multicast, local VRF traffic, FIB rule
cumulusnetworks.com
v4.8 == Cumulus Linux 3.1

14.
cumulusnetworks.com
What about Network Namespaces?
Wrong Model for VRFs
 VRF is a Layer 3 separation
 Network Namespace is a device layer separation
14
Default namespace mgmt namespace
swp4 swp5 swp6
“red” namespace
neigh
swp2 swp3swp1 eth0
FIBsockets neigh FIBsockets
neigh FIB
sockets
lldpd ipquaggaptmd
collectd

15.
cumulusnetworks.com
Overview of VRF Design for Linux
VRF represented as a virtual (pseudo)
network device
 VRF device correlates to routing table
 FIB rules for VRF device direct lookups to VRF table
 Network interfaces are associated with VRF device
 Host and connected routes moved to VRF table
L3-only concept
 Impacts only IPv4 and IPv6 route and socket
lookups

16.
cumulusnetworks.com
Advantages of Device Model
Network interface-to-VRF association similar
to bridges
 familiar paradigm for Linux networking
16

17.
cumulusnetworks.com
Advantages of Device Model
Network interface-to-VRF association similar
to bridges
netdevice is a fundamental construct in
Linux networking stack
 Network addresses on VRF device
VRF-local loopback (127.0.0.1 and unnumbered
deployments)
 netfilter, tc, tcpdump on VRF device
17

18.
cumulusnetworks.com
Advantages of Device Model
Network interface-to-VRF association similar
to bridges
netdevice is a fundamental construct in
Linux networking stack
VRFs can exist inside network namespaces
 Follows existing paradigms for network interfaces
and namespaces
18

19.
cumulusnetworks.com
Advantages of Device Model
Network interface-to-VRF association similar
to bridges
netdevice is a fundamental construct in
Linux networking stack
VRFs can exist inside network namespaces
Applications use existing socket APIs
 SO_BINDTODEVICE, IP_PKTINFO
19

20.
cumulusnetworks.com
Advantages of Device Model
Network interface-to-VRF association similar to
bridges
netdevice is a fundamental construct in Linux
networking stack
VRFs can exist inside network namespaces
Applications use existing socket APIs
Existing frameworks for monitoring,
serviceability
 iproute2 commands, netlink 20

21.
Application Behavior With VRF
 Applications by default run in the default VRF context
 Applications that support binding to a device, can run within a
specified VRF
 Examples are ping, traceroute
 Applications that don’t support binding to a device need a
helper to set context to correct VRF
 Cumulus Linux has “vrf exec” to do this, in the process of upstreaming
 LD_PRELOAD is another option
 Applications that work in multiple VRFs concurrently need to
be modified
 Example is BGP
June 29, 2016 Cumulus Networks Confidential 21

22.
Specifying VRF in Applications
Use existing POSIX APIs
 VRF context by binding sockets to device or cmsg
setsockopt(SO_BINDTODEVICE), IP_PKTINFO
22cumulusnetworks.com

23.
Agenda
Why Do We Need VRF in the DC
VRF in the Linux Kernel
Operational Workflow of VRF
 Interface configuration
 Management VRF in Cumulus Linux
 Routing
June 29, 2016 cumulusnetworks.com 23

24.
cumulusnetworks.com
Cumulus Linux 3.0 Workflow: VRF Creation & Slave Port Attach
Define configuration in
/etc/network/interfaces
24
red FIB
table 1001
swp1 swp2
OR
auto all
iface red
address 127.0.0.1/8
vrf-table auto
iface swp1
vrf red
iface swp2
vrf red
auto all
iface red
address 127.0.0.1/8
vrf-table auto
%for i in range(1,4):
iface swp${i}
vrf red
%endfor

25.
cumulusnetworks.com
Simplified Linux Workflow (Ubuntu 16.04)
25
Create VRF device
with table id
Setup FIB rules
pointing to device
Add default route for
VRF
Enslave interfaces
ip link add red type vrf table 123
ip link set red up
ip rule add iif red table 123
ip rule add oif red table 123
ip -6 rule add iif red table 123
ip -6 rule add oif red table 123
ip route add table 123 unreachable default
metric 8192
ip -6 route add table 123 unreachable default
metric 8192
ip link set eth1 master red
Ip link set eth2 master red

26.
VRF Implementation in Cumulus Linux
VRF name is purely an operator's choice.
 Exception is Management VRF; must be “mgmt”
Table id’s limited to 1001-1255
 Hardware considerations + future growth in number of VRFs
Table-id to name mapping is saved
 allows for name-based reference in iproute2 commands
June 29, 2016 cumulusnetworks.com 26

27.
Agenda
Why Do We Need VRF in the DC
VRF in the Linux Kernel
Operational Workflow of VRF
 Interface configuration
 Management VRF in Cumulus Linux
 Routing
June 29, 2016 cumulusnetworks.com 27

28.
Management VRF
June 29, 2016 cumulusnetworks.com 28
Management VRF
 Uses non-main table
 Name must be “mgmt”
 Loopback address on
mgmt device
Front panel ports
default to “Default VRF”
(main table)
auto all
iface mgmt
address 127.0.0.1/8
vrf-table auto
iface eth0 inet dhcp
vrf mgmt

29.
Management VRF and Commands
Commands need to specify VRF to use – in general
 ping and traceroute wrappers default these commands to front
panel ports and default VRF
• Use ‘ping –I <vrf> …’ and ‘traceroute –i <vrf> …’ to specify VRF of
interest
 Helper for commands that do not support bind to device or
interface options: vrf exec <vrf> <command>
• Remember, affects IPv4 and IPv6 sockets only!
June 29, 2016 cumulusnetworks.com 29

30.
Management VRF: Default VRF “Context”
Login context defaults to Management VRF
 IPv4/IPv6 sockets opened by processes automatically bound
to Management VRF device
 Do not need to specify VRF for commands unless a different
VRF is wanted (except ping and traceroute because of the
wrappers)
 Default PS1 bash prompt provides VRF context
cumulus@cel-redxp-99:mgmt-vrf:~$
June 29, 2016 cumulusnetworks.com 30

31.
Enabling Services in VRFs
systemd based services can be run in a VRF context
 systemctl <action> <service>@<vrf>
• systemctl enable ntp@mgmt.
• systemctl start ntp@mgmt
 Possible collision with port if service is running in default VRF
• Process in default VRF with wildcard bind owns port
across all VRFs
Loopback address on Management VRF allows ntpq
to work
Cumulus Linux specific, upstream WIP
June 29, 2016 cumulusnetworks.com 31

32.
Agenda
Why Do We Need VRF in the DC
VRF in the Linux Kernel
Operational Workflow of VRF
 Interface configuration
 Management VRF in Cumulus Linux
 Routing
June 29, 2016 cumulusnetworks.com 32

33.
Routing Support for VRF
Routing protocol constructs (routes,
nexthops, router-id etc.) and processing are
VRF-aware.
 Cumulus Quagga routing protocol suite
Static routes (IPv4 and IPv6) supported in a
VRF
June 29, 2016 cumulusnetworks.com 33

34.
BGP Support for VRF
Per-VRF neighbors
 EBGP/IBGP
 Numbered & Unnumbered
Per-VRF parameters (address-families,
redistribution etc.)
BFD support (single / multihop)
June 29, 2016 cumulusnetworks.com 34

35.
VRF provisioning - Quagga
‘bgp multiple-instance’ is ON by
default
3/13/16 cumulusnetworks.com 35
router bgp 64900 vrf red
bgp router-id 6.0.3.7
network 20.7.2.0/24
network 20.7.3.0/24
neighbor 10.1.1.2 remote-as 20
neighbor 20.1.1.2 remote-as 30
!
!
router bgp 64900 vrf blue
bgp router-id 18.0.4.7
neighbor CUSTPEER peer-group
neighbor CUSTPEER remote-as external
neighbor swp1.10 peer-group CUSTPEER
!
address-family ipv4 unicast
redistribute connected
exit-address-family
!

36.
VRF Verification and Troubleshooting – Quagga
Various existing commands are VRF-aware
 show ip route [vrf <all | name>]
 show ip bgp [vrf <name>] summary
 show interface [vrf <name>]
Couple of new commands
 show vrf
 show bgp vrfs
Existing Quagga ‘debug’ commands cover
VRF too
June 29, 2016 cumulusnetworks.com 36

37.
VRF Hardware Acceleration
Supported on all switching silicon chipsets:
 Broadcom chipsets – T+, T2, Tomahawk
 Mellanox Spectrum
Kernel table id to VRF id mapping
 VRF_id = table_id – 1000
 Except for default VRF which uses VRF 0
June 29, 2016 cumulusnetworks.com 37

38.
Scale Numbers in Cumulus Linux 3.0
Maximum VRF supported is 64
No per-VRF limit on interfaces, neighbors,
routes etc.
June 29, 2016 cumulusnetworks.com 38

39.
Routing Protocol Caveats in CL 3.0
Only BGP supported
 OSPFv2 and OSPFv3 are not VRF-aware
Each VRF table in the kernel needs a default
route
 keeps lookups from dropping to the next table
 high metric allows overrides
June 29, 2016 cumulusnetworks.com 39

40.
Demo Topology
June 29, 2016 cumulusnetworks.com 40
 Show ping and traceroute
 Traffic takes the shortest
path
 Node L1 is real switch
 Rest are VMs
 VRF Loopbacks are
assigned as:
 10.253.<vlan>.<nodeid>
 Non-VRF loopbacks are
assigned as:
 10.252.<>.<>
S1 S2 S3
L1 L2
NL1 NL2
FW1 FW2
H1 H2 H3 H4
10.7.x.3 10.7.x.4 10.8.x.5 10.8.x.6
10.252.0.210.252.0.1
10.253.2.9
10.253.3.9
10.253.4.9
10.253.2.10
10.253.3.10
10.253.4.10
10.253.2.11
10.253.3.11
10.253.4.11
10.253.2.13
10.253.3.13
10.253.4.13
10.253.2.7
10.253.3.7
10.253.4.7
10.253.2.8
10.253.3.8
10.253.4.8

41.
cumulusnetworks.com
References
ifupdown2 support for Debian/Ubuntu
 ifupdown2 with VRF support to be released soon
 https://github.com/CumulusNetworks/ifupdown2
 https://github.com/CumulusNetworks/ifupdown2/blob/master/a
ddons/vrf.py
41

42.
Summary
VRF is a first class citizen in Linux now
 For example, distributions such as Ubuntu 16.04
support it
VRF design is such that it takes advantage
of existing socket APIs
Routing support is via BGP and static routes
 Familiar configuration model
June 29, 2016 Cumulus Networks Confidential 42

43.
June 29, 2016 cumulusnetworks.com 43
•Routing on the Host: Concepts &
Customer Case Studies
•Guest Speaker: Ivan Pepelnjak
•When: July 21
Next Month’s Webinar

44.
© 2016 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in
the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI, the exclusive
licensee of Linus Torvalds, owner of the mark on a world-wide basis.
ThankYou!
cumulusnetworks.com 44June 29, 2016