VXLAN is a point to point, UDP-based "tunneling" protocol, that enables L2 encapsulation over an L3 "undernet", while also allowing up to 16 million Virtual Networks. One challenge with deploying VXLAN is that by default VXLAN requires multicast support for Broadcast, Unknown and Multi-cast packets. Often this is not possible in customer networks. An alternative approach is to use the Service Node concept where dedicated node(s)/process(es) are responsible for flooding Broadcast, Unknown, and Multicast packets throughout a network. This removes the need for multi-cast, and greatly simplifies network configuration. However, it does require a scalable, and highly available implementation.

1. David Lapsley (@devlaps), Chet Burgess (@cfbIV), Kahou Lei (@kahou82)
May 20, 2015
OpenStack Vancouver Summit
VXLAN Distributed Service Node

2. Virtualization in the data
center has changed network
requirements

3. Number of end hosts 
Number of networks 
Bandwidth requirements 

4. This is a problem for
traditional data center
networks

5. • L2 Access with L3 Aggregation
• Wasted capacity: STP blocks ports to prevent loops
• VLAN Exhaustion: only 4K with 802.1Q label
• ToR Scalability: hw tables need to scale with endpoints
Traditional Data Centers

6. L3 to the edge can help

7. • L3 is Scalable
• Well known and supported
• Equal Cost Multi-Path (ECMP) Routing
• Each link active at all times
L3

8. How do we scope
tenants/projects?

9. • MAC over UDP/IP overlay
• Re-uses existing IP core (L3 ECMP, No STP)
• Reduces pressure on ToR L2 tables
• Supports over 16M+ VLANs
• Maintains L2 bridging semantics
VXLAN

10. VXLAN Encapsulation

11. • Virtual Network Identifier
• 24 bits  16+ million
• VXLAN Tunnel End Point (VTEP)
• Encapsulation, Decapsulation
• Listen on UDP port 4789 (IANA), 8472 (Linux default) for incoming VXLAN
packets
• VNI to VTEP IP mapping
VXLAN Components

12. VXLAN Example Deployment
Hypervisor 1
VM1 VM2
VTEP (vxlan100)
Tenant bridge (br101)
VM1 VM2
VTEP (vxlan101)
L3 Network
eth0
Hypervisor 2
Tenant bridge (br100)
VM3 VM4
VTEP (vxlan100)
Tenant bridge (br101)
VM3 VM4
VTEP (vxlan101)
eth0
VXLAN
100
VXLAN
101
DMAC SMAC 802.1Q EType Payload CRC
Outer
MAC
Outer
IP
Outer
UDP
VXLAN CRCPayload
VXLAN
Network Identifier
(24 bits)
VXLAN
Flags
(8 bits)
Reserved
(24 bits)
Reserved
(8 bits)
Tenant bridge (br100)

13. • Broadcast, Unknown, and Multicast packets (e.g. ARP,
DHCP, multi-cast, etc.) are flooded to all VTEPs for the
given VNI
• Two mechanisms used:
• Multicast
• Multi-cast address and VNI configured for each VXLAN segment
• VTEP sends IGMP join/leave as VMs spin up/down
• Broadcast domain implemented using multicast
• Service Node:
• Use a “central” service node to maintain mapping of VNIs to VTEP IPs
Broadcast, Unknown and Multicast Packets

14. Service Node
Hypervisor 1
VM1 VM2
vxlan100 (1.1.1.1)
Tenant bridge (br101)
VM1 VM2
vxlan101 (3.3.3.3)
L3 Network
eth0
Hypervisor 2
Tenant bridge (br100)
VM3 VM4
vxlan100 (2.2.2.2)
Tenant bridge (br101)
VM3 VM4
vxlan101 (4.4.4.4)
eth0
VXLAN
100
VXLAN
101
Tenant bridge (br100)
VNI VTEPs
100
1.1.1.1
2.2.2.2
101
3.3.3.3
4.4.4.4
Remote
Service
Node

15. Service Node

16. Central Service Node

17. Central Service Node

18. Distributed Service Node

19. Distributed Service Node

20. Distributed Service Node

21. Distributed Service Node

22. VXLAN Distributed Service
Node

23. Design

24. Design

25. Design
Controller 1 Controller 2 Controller 3
L3 Network
Hypervisor 1
Tenant bridge (br100)
VM1 VM2
Tenant bridge (br101)
VM1 VM2
VTEP (vxlan101)
eth0
Hypervisor 500
Tenant bridge (br100)
VM1 VM2
VTEP (vxlan100)
Tenant bridge (br101)
VM1 VM2
VTEP (vxlan101)
eth0
eth0
VTEP (vxlan100)
eth0 eth0
Distributed
VXLAN
Service Node
Distributed
VXLAN
Service Node
mcrouter
memcache
mcrouter
memcache
mcrouter
memcache

26. Design
Controller 1 Controller 2 Controller 3
L3 Network
Hypervisor 1
Tenant bridge (br100)
VM1 VM2
Tenant bridge (br101)
VM1 VM2
VTEP (vxlan101)
eth0
Hypervisor 500
Tenant bridge (br100)
VM1 VM2
VTEP (vxlan100)
Tenant bridge (br101)
VM1 VM2
VTEP (vxlan101)
eth0
eth0
VTEP (vxlan100)
eth0 eth0
Distributed
VXLAN
Service Node
Distributed
VXLAN
Service Node
mcrouter
memcache
mcrouter
memcache
mcrouter
memcache

27. Design
Controller 1 Controller 2 Controller 3
L3 Network
Hypervisor 1
Tenant bridge (br100)
VM1 VM2
Tenant bridge (br101)
VM1 VM2
VTEP (vxlan101)
eth0
Hypervisor 500
Tenant bridge (br100)
VM1 VM2
VTEP (vxlan100)
Tenant bridge (br101)
VM1 VM2
VTEP (vxlan101)
eth0
eth0
VTEP (vxlan100)
eth0 eth0
Distributed
VXLAN
Service Node
Distributed
VXLAN
Service Node
mcrouter
memcache
mcrouter
memcache
mcrouter
memcache

28. • Multi-threaded python program (multiprocessing module)
• Runs on every hypervisor
• Shares state using Distributed Cache
• FB Mcrouter – memcached protocol router (5B requests /second @ peak!)
• Listens for new VTEP registrations
• Forwards new mappings to Distributed Cache
• Listens for Broadcast, Unknown, Multicast packets
• Floods to all VTEPs in the Virtual Network
VXLAN Distributed Service Node

29. Service Node

30. Service Node

31. Configuring VXLAN

32. ip link add vxlan1 type vxlan id 1 remote 169.254.1.1 dev
eth0
ip addr add 172.16.1.1 dev vxlan1
ip link set dev vxlan1 mtu 1450
ip link set dev vxlan1 up
Creating VXLAN interfaces

33. root@mhv2:~# ip addr show vxlan1
4: vxlan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc
noqueue state UNKNOWN group default
link/ether f2:af:3f:62:cf:65 brd ff:ff:ff:ff:ff:ff
inet 172.16.1.5/24 scope global vxlan1
valid_lft forever preferred_lft forever
inet6 fe80::f0af:3fff:fe62:cf65/64 scope link
valid_lft forever preferred_lft forever
Configured VXLAN Interface

34. iptables –t nat -A OUTPUT -d 169.254.1.1/32 -p udp -m udp -
-dport 8472 -j DNAT --to-destination 127.0.0.1:8473
The @cfbIV rule

35. -t nat -A OUTPUT -d 169.254.1.1/32 -p udp -m udp --dport 8472 -j DNAT
--to-destination 127.0.0.1:8473
The @cfbIV rule

36. -t nat -A OUTPUT -d 169.254.1.1/32 -p udp -m udp --dport 8472 -j DNAT --to-destination 127.0.0.1:8473
The @cfbIV rule

37. -t nat -A OUTPUT -d 169.254.1.1/32 -p udp -m udp --dport 8472 -j DNAT
--to-destination 127.0.0.1:8473
The @cfbIV rule

38. -t nat -A OUTPUT -d 169.254.1.1/32 -p udp -m udp --dport 8472 -j DNAT
--to-destination 127.0.0.1:8473
The @cfbIV rule

39. Demo

40. Demo Setup
Controller 1 Controller 2 Controller 3
L3 Network
Hypervisor 1
VTEP (172.16.3.4)
192.168.225.231
Hypervisor 500
192.168.225.232
192.168.225.226
VTEP1 (172.16.1.4)
192.168.225.227 192.168.225.228
VTEP1 (172.16.1.4) VTEP (172.16.3.6)VTEP1 (172.16.1.5) VTEP1 (172.16.1.5)
VXLAN
Distributed
Service Node
VXLAN
Distributed
Service Node
mcrouter
memcache
mcrouter
memcache
mcrouter
memcache

43. • Open source VDSN source code
• Integration with Neutron (if community interest)
• Performance and scalability testing
Future work

44. References

45. • Presentation slides: http://bit.ly/vdsn-presentation
• VDSN Source Code and Ansible playbooks:
• Simple, accessible model, horizontal scaling
• http://bit.ly/vdsn-ansible
• VDSN code coming soon (@devlaps, #devlaps)
• Production Code:
• Multi-area VXLAN! Highly optimized, requires expertise to
configure/troubleshoot
• http://bit.ly/multi-area-vxlan
References

46. • C. Burgess, N. Leake, L3 + VXLAN Made Practical,
OpenStack Summit Spring 2014.
• M. Mahalingam, et. Al, Virtual eXtensible Local Area
Network (VXLAN): A Framework for Overlaying
Virtualized Layer 2 Networks over Layer 3 Networks,
https://tools.ietf.org/html/rfc7348
References

47. • Sanjay K. Hooda, Shyam Kapadia, Padmanabhan
Krishnan, Using TRILL, FabricPath, and VXLAN:
Designing Massively Scalable Data Centers (MSDC) with
Overlays, Cisco Press, 2014.
• Introducing McRouter, http://bit.ly/introducing-mcrouter
References

48. • McRouter on github,
https://github.com/facebook/mcrouter
• Pyroute2, https://pypi.python.org/pypi/pyroute2
• Maintaining a set in Memcached, http://bit.ly/memcache-
sets
• Ansible, http://docs.ansible.com
References

49. @devlaps, dlapsley@cisco.com
Thank You