VyOS now supports VXLAN interfaces which allow multiple L2 segments to be multiplexed over a single physical network. VXLAN uses encapsulation to transport Ethernet frames over IP. The VNI field in VXLAN headers maps frames to different L2 segments. VyOS VXLAN interfaces can be configured and used like physical interfaces for routing, bridging, and protocols like OSPF. However, attributes like the VNI and multicast group cannot be changed after interface creation without deleting and recreating the interface.

1. VyOS VXLAN and Linux Device Driver
VyOSのVXLANとLinuxのデバドラの話
2014/11/2
VyOS users meeting #2
Ryo Nakamura
upa@haeena.net

2. Virtual eXtensible LAN
• An Ethernet over IP overlay. RFC7348.
– Ethernet frame is encapsulated in IP + UDP + VXLAN headers.
– VXLAN header contains 24bit Virtual Network Identifier (VNI) field. 2^24 L2
segments can be multiplexed in one VXLAN overlay network domain.
– Unicast traffic is encapsulated in IP Unicast.
– BUM traffic is encapsulated in IP Multicast.
• Multicast based VTEP learning is described in RFC, Sec 4.
– Many vendors propose and use their original control planes.
– Of course, I know that Multicast is difficult in actual environments, but they don’t
have INTEROPERBILITY :(
2

3. Multicast based VTEP learning
OuterIP Src A
OuterIP Dst M
SrcMAC : 1
DstMAC : FF
VTEP:A
VTEP:B
Node:1 VTEP:D
VTEP:C
Node:3
Node:4
Node:2
Node 1 send arp request Node 4
3
Node 1 is in
VTEP A !!

4. Multicast based VTEP learning
VTEP:A
VTEP:B
Node:1 VTEP:D
VTEP:C
Node:4
Node:2
OuterIP Src D
OuterIP Dst A
SrcMAC : 4
DstMAC : 1
Node 4 send arp reply to Node 1
4 Node:3
Node 4 is in
VTEP D !!
Node 1 is in
VTEP A !!

5. VyOS VXLAN support
• 2014/9/20, merged.
5

6. Linux kernel version issue
• Linux VXLAN Driver is appeared in kernel 3.7
– 2012/9/24, first patch was contributed to netdev.
– I was really looking forward to Vyatta Core with
kernel 3.7 and later.
• Kernel version of VyOS Helium is 3.13.11 !!
– HooooooooOOOO!!! WrrrrryyyyyyYYYYYYYYYY !!!!!!!!
– Hydrogen is kernel 3.3
6

7. VyOS VXLAN CLI
• Under the interfaces section
– set interfaces vxlan vxlan0
– set interfaces vxlan vxlan0 group 239.0.0.1
– set interfaces vxlan vxlan0 vni 0
– and basic interface operations
• IPv4/v6 routing
• bridge-group
• policy
interfaces {
vxlan vxlan0 {
group 239.0.0.1
vni 0
}
} 7

8. Operation example
interfaces {
vxlan vxlan0 {
address 172.16.0.1/24
group 239.0.0.10
ip {
ospf {
cost 10
}
}
vni 0
}
}
protocols {
ospf {
area 0 {
network 172.16.0.0/24
}
}
}
8

9. Operation example
vyos@vyos:~$ show interfaces vxlan vxlan0
vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UNKNOWN group default
link/ether b2:74:c9:fa:1d:fd brd ff:ff:ff:ff:ff:ff
inet 172.16.0.1/24 brd 172.16.0.255 scope global vxlan0
valid_lft forever preferred_lft forever
inet6 fe80::b074:c9ff:fefa:1dfd/64 scope link
valid_lft forever preferred_lft forever
RX: bytes packets errors dropped overrun mcast
0 0 0 0 0 0
TX: bytes packets errors dropped carrier collisions
2446 25 0 0 0 0
9

10. Operation example
vyos@vyos:~$ show ip ospf interface vxlan0
vxlan0 is up
ifindex 3, MTU 1500 bytes, BW 0 Kbit <UP,BROADCAST,RUNNING,MULTICAST>
Internet Address 172.16.0.1/24, Broadcast 172.16.0.255, Area 0.0.0.0
MTU mismatch detection:enabled
Router ID 10.10.20.189, Network Type BROADCAST, Cost: 10
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) 10.10.20.189, Interface Address 172.16.0.1
No backup designated router on this network
Multicast group memberships: OSPFAllRouters OSPFDesignatedRouters
Timer intervals configured, Hello 10s, Dead 40s, Wait 40s, Retransmit 5
Hello due in 7.900s
Neighbor Count is 0, Adjacent neighbor count is 0
10

11. node.def
• VXLAN interface name
– Different number from VNI can be used to an interface
name. But, I think it is really confusing :(
val_help: <vxlanN>; VXLAN interface name
syntax:expression: pattern $VAR(@) "vxlan[0-9]+$"
11

12. node.def (cont’d)
• REQUIRED
– A vxlan overlay network is identified by VNI.
– Multicast Group Address is required to encapsulate BUM Traffic
in IP Multicast.
Group Address can be reused for other VNI.
commit:expression: $VAR(./group/) != "";
"Must configure vxlan group for $VAR(@)"
commit:expression: $VAR(./vni/) != "";
"Must configure vxlan vni for $VAR(@)“
12

13. node.def (cont’d)
• create interface
VXLAN_VNI="id $VAR(./vni/@)"
VXLAN_GROUP="group $VAR(./group/@)"
VXLAN_TTL="ttl 16"
if [ ! $VAR(./link/) == "" ]; then
VXLAN_DEV="dev $VAR(./link/@)"
fi
ip link add name $VAR(@) type vxlan
$VXLAN_VNI $VXLAN_GROUP $VXLAN_TTL $VXLAN_DEV
ip link set $VAR(@) up
touch /tmp/vxlan-$VAR(@)-create
skimped
work...
underlay
device
And,
execute
iproute2
13

14. Change vni or group of existing
vxlan interfaces
• Sorry, it is not supported.
• Changing group or vni requires delete and
re-create the vxlan interface.
14

15. VXLAN in Linux
• ip link add type vxlan
– Pseudo ethernet interface : vxlanX
– Interfaces are connected to each vxlan overlay network corresponding to a VNI
(vxlan_dev and FDB / VNI)
– Namespace is supported
struct net_device
Linux Kernel
netif_rx(skb)
vxlan1
FDB
vxlan0
FDB
kernel udp socket
udp_sk(sk)->encap_rcv =
vxlan_udp_encap_recv
iptunnel_xmit()
15

16. How to specify attributes
• ip link add type vxlan id 0 group X
– Netlink API : An API to communicate to Kernel
– NETLINK_ROUTE, NETLINK_NETFILTER and more
Linux Kernel
Userland Application
Netlink Socket
socket(AF_NETLINK, SOCK_RAW, netlink_family)
Inter
face
routing
table
Netfilter
struct nlmsghdr
and rtattr etc
16

17. How to specify attributes (cont’d)
• ip link add type vxlan id 0 group X
– RTNETLINK : routing socket
• RTM_NEWLINK message is sent with attributes related to
VXLAN (see man ip-link)
int do_iplink(int argc, char **argv)
{
if (argc > 0) { if (iplink_have_newlink()) {
if (matches(*argv, "add") == 0)
return iplink_modify(RTM_NEWLINK,
NLM_F_CREATE|NLM_F_EXCL,
argc-1, argv+1);
iproute2 package is a
good text book of
Netlink !!
17

18. Attributes of vlxan interface
• id : Virtual Network Identifier
• dev : Uunderlay device (in VyOS, link)
• group : Multicast group address
• remote : An unicast IP address of VTEP for BUM traffic
• local : Source IP address of encapsulated packet
• ttl : TTL of encapsulated packet
• port : Source port range of encapsulated packet
But, these attributes can be only specified
when a pseudo interface is created !!
18

19. How to specify attributes (cont’d)
• VXLAN driver kernel-source/drivers/net/vxlan.c
– RTM messages are processed by rtnl_link_ops
static struct rtnl_link_ops vxlan_link_ops __read_mostly = {
.kind = "vxlan",
.maxtype = IFLA_VXLAN_MAX,
.policy = vxlan_policy,
.priv_size = sizeof(struct vxlan_dev),
.setup = vxlan_setup,
.validate = vxlan_validate,
.newlink = vxlan_newlink,
.dellink = vxlan_dellink,
.get_size = vxlan_get_size,
.fill_info = vxlan_fill_info,
};
vxlan_newlink () is called
when RTM_NEWLINK
is received
19

20. vxlan_newlink ()
• Codes can not be pasted... too long...
1. Parse attributes
2. Set up parsed parameters to vxlan_dev
3. register_netdeivce
20

21. And, you can see vxlan0
asano2:/home/upa % ifconfig vxlan0
vxlan0 Link encap:Ethernet HWaddr 02:0a:1e:ad:7f:31
inet6 addr: fe80::a:1eff:fead:7f31/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:690 (690.0 B)
asano2:/home/upa % ip -d link show dev vxlan0
9: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UNKNOWN mode DEFAULT group default
link/ether 02:0a:1e:ad:7f:31 brd ff:ff:ff:ff:ff:ff promiscuity 0
vxlan id 0 group 239.0.0.1 srcport 32768 61000 dstport 8472 ageing 300
asano2:/home/upa % bridge fdb show dev vxlan0
00:00:00:00:00:00 dst 239.0.0.1 self permanent
21

22. As a result
• vxlan parameters can not be changed after
pseudo interface is created.
• Do you have good ideas ?
– I have just only one idea.
• Use Generic Netlink like l2tp driver
• Generic Netlink is a mechanism to add user defined
netlink family dynamically.
• It requires patches to vxlan driver and iproute2...
22

23. Future work ?
• Change destination port ?
– Default is 8472 (OTV). 4789 is assigned for VXLAN by IANA
– It can be changed through module_param.
But it requires rmmod/insmod when port is changed.
Of course, all pseudo interfaces are removed...
• Support “remote” attribute
– Easy. Is it needed for the community ?
23

24. Overlay is the Only Way!!
Thanks!
upa@haeena.net
24