Audience: Beginner About: This session details the design and implementation of an L3 network underlay, routing to the host, and a hardware VXLAN gateway used with an enterprise OpenStack distribution. Speaker Bio: Gerard Chami – Technical Support Engineer, Cumulus Networks Gerard is a Technical Support Engineer for Cumulus Networks and a founding members of the Australian support team. Since joining Cumulus Gerard has enjoyed working with Open Source and DevOps tools to help bring web-scale architectures and efficiency to enterprise networking. Prior to joining Cumulus Networks, Gerard worked at Cisco Systems where focused on emerging data centre solutions including UCS, Nexus Switching and ACI. Speaker Bio: Scott Laffer – Technical Support Engineer, Cumulus Networks Scott works at Cumulus Networks as a Technical Support Engineer. Always a fan of networking, while at Cumulus, Scott has enjoyed being a part of the Linux networking evolution. He is passionate about using NetDevOps tools to build, maintain and troubleshoot new generations networking architectures, all utilising the power of Linux. Scott started his career as a network administrator, before joining Cisco Systems to work with their high end Nexus switching range. OpenStack Australia Day - Sydney 2016 https://events.aptira.com/openstack-australia-day-sydney-2016/

1. v
Simplifying OpenStack Networks with
Routing on the Host
Scott Laffer and Gerard Chami
5th of May, 2016

2. cumulusnetworks.com 2
• Overview
• Who are we?
• Why care about the plumbing?
• What options are there?
• Demo
Agenda

3. Who are we?
cumulusnetworks.com 3
Scott Laffer
Technical Support Engineer
slaffer@cumulusnetworks.com
@slaffah
Gerard Chami
Technical Support Engineer
gchami@cumulusnetworks.com
@gerardchami

4. Transformation: First Servers, Now Networking
cumulusnetworks.com 4
First:
Compute
Transformed
LOCKED
Now:
Networking
Transforms
OPEN
Open Networking Enables Platform Choice and Affordable Capacity
cumulusnetworks.com
Applications, OS and Hardware
Open Ecosystem
Agile, open, scalable with unprecedented cost savings
4

5. “NetDevOps” – using existing
DevOps tools for networking
Operational efficiencies, increased
deployment speed
OpenStack + Cumulus – Own the Rack with Linux
cumulusnetworks.com 5
.
Why OpenStack?
.
Why Cumulus Linux?
Open source and associated
ability to innovate
No vendor lock-in
Affordable
Commoditized hardware
Bridge the gap between your
sysadmins and network engineers
Treat your switch like a server
OpEx and CapEx savings
Disaggregated HW and SW
Linux throughout your entire rack!

6. Why care about the plumbing?
cumulusnetworks.com 6

7. ML2 Type Driver Choices – “The What”
Flat Type Driver
All subnets assigned are placed in
the same Layer-2 broadcast domain.
Commonly used for defining a single
provider network (single pool of
external IP addresses).
cumulusnetworks.com 7
VLAN Type Driver
Each OpenStack subnet is assigned
to a different VLAN. Discussed in
detail in the Cumulus OpenStack
Validated Design Guide.
VxLAN Type Driver
Each OpenStack subnet is assigned to a
different VxLAN.
Looks similar to a typical Cumulus VxLAN
design except VTEP can be in the host.
Other
Linux
Bridge
OvS VendorOther GRE VLAN VxLAN
Core Plugin (ML2)
Type Manager
Type Driver
Mechanism Manager
Mechanism Driver

8. Other
Linux
Bridge
OvS VendorOther GRE VLAN VxLAN
ML2 Mechanism Driver Choices – “The How”
Linux Bridge
Provides Layer-2 and Layer-3
connectivity on a compute node
using traditional bridging constructs.
cumulusnetworks.com 8
OpenVswitch (OVS)
Provides Layer-2 and Layer-3
connectivity on a compute node
using networking stack that sits on
top of the Linux Kernel. It does not
use the Linux Kernel API.
Cumulus Linux
Instantiates/Destroys VLANs on a
Cumulus Switch after a tenant
network is created/deleted on the
OpenStack Compute Nodes
Core Plugin (ML2)
Type Manager
Type Driver
Mechanism Manager
Mechanism Driver
ML2
Framework providing a way to configure L2/L3 connectivity
on any networking platform such as the linux kernel
(linuxbridge) or OpenVSwitch

9. Design 1: ML2 + VLAN: MLAG Between Host/Leaf and Leaf/Spine
§ Overall: A well known and common design using MLAG at the spine layer, MLAG at leaf layer,
but least scalable and least flexible. An “old school” but proven network design.
cumulusnetworks.com 9
§ Considerations:
§ VLANs statically assigned but doesn’t scale well
§ STP heavy between Leaf/Spine and Leaf/Host
§ MLAG difficult to manage at scale.
§ Using Cumulus ML2 Mechanism driver to
dynamically add/remove VLANs doesn’t make
sense. How do you add/remove VLANs from
spines consistently?
§ Cumulus “Stickiness”:
§ Better automation story
§ Better operational story
• Common tools for operation switch and server
§ Validated Design Guide certified
L2
ML2 Pair

10. Design 2: MLAG at Top-of-rack, IP Fabric Between Leaf/Spine
Overall: Uses less MLAG, more Layer-3, VxLAN, and is therefore more scalable.
Caveat: Utilizes third-party SDN overlays, which could add to overall complexity.
cumulusnetworks.com 10
§ Considerations:
§ Scales better than L2 + MLAG
§ SDN Overlays dynamically provision VxLAN on the switch
• SDN overlay – Midokura, Nuage, PLUMgrid
• Hierarchical Port Binding with Cumulus Mechanism Driver (alpha)
§ Future “Upsides”:
§ Scales better than L2 + MLAG
§ SDN Overlays dynamically provision VxLAN on
the switch
• SDN overlay – Midokura, Nuage, PLUMgrid
• Hierarchical Port Binding with Cumulus Mechanism Driver (alpha)
§ Cumulus “Stickiness”:
§ Simple Layer-3 config for IP fabric
§ BGP/OSPF unnumbered
§ HPB + Cumulus ML2 in production
ML2 Pair
L2
L3
ECMP

11. Design 3– Layer-3 to the Host: Single Attach
Overall: “Good Enough” for single links from hosts to switches, and recommended by Openstack.org
Caveat: Not unique/novel - other networking vendors can accomplish this
cumulusnetworks.com 11
§ Considerations:
§ Application need to be distributed
§ Not recommended for those who believe in dual
attaching host
§ VTEP on the host
§ VXLAN offload NICs recommended
§ Cumulus “Stickiness”:
§ Eliminated STP (Spanning Tree)
L3
ECMP

12. L3
Design 4: Layer-3 to the Host: Multiple Attach (Quagga on the Host)
§ Overall: The best overall networking solution with OpenStack and Cumulus Networks in
large configurations. 100% simple and flexible architecture with Layer-3 networking using
Linux quagga package extendable to other software solutions.
cumulusnetworks.com 12
§ Considerations:
§ VXLAN offload network interfaces recommended
§ Succeeds in docker container environments
§ Supports more than two links from hosts to
switches for load balancing
§ Cumulus “Stickiness”:
§ Simplified infrastructure config
§ Server/switch/rack mobility
§ Major reduction in IPv4 addressing
§ Requires Cumulus Quagga package
ECMP
ECMP

13. OpenStack Network Design Decision Tree
cumulusnetworks.com 13
Tenant
Separation
method?
VLAN Type
Driver on host
VxLAN Type
Driver on host
IP Fabric Between
Leaf/Spine
Number of Host to
Switch Links?
1 host to
switch link
2 or more host
to switch links
All L2/MLAG - Leaf/Spine MLAG,
Host/Leaf MLAG, applies Cumulus
Validated Design Guide
All L3 - Assign L3 address on host
interfaces. Unnumbered for IP fabric
switch interfaces.
All L3 - L3 BGP/OSPF unnumbered
config all the way to the host.
Install Linux Quagga package from
Cumulus on each host.
“past”
“present”
“future”

14. Cumulus Networks
Demo Time
14

15. Not just a party trick…
cumulusnetworks.com 15

16. © 2016 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its
affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux® is used pursuant to a sublicense from LMI,
the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.
§ Thank You!
cumulusnetworks.com 16
Unleashing the Power of Open Networking

17. v
Backup Slides
cumulusnetworks.com 17

18. Getting Started with the Validated Design Guide
cumulusnetworks.com 18
§ Detailed 54-page HOWTO deployment
guide - two spine and four leaf switches
§ Install and configure switches and
compute nodes
§ Closely mimics production architecture
VMware®
vSphere and Cumulus Networks®
Validated Solution Guide
Deploying VMware
®
vSphere with Cumulus
®
Linux
®
Switches
Big Data Hadoop and Cumulus Networks
®
Validated Solution Guide
Deploying Apache Hadoop with Cumulus
®
Linux
®
Switches
0
0
00
0
0
0
00
0
0
01
1
0
1
11
1 1 1
11
1
1 11
1

19. Demo: Cumulus VX "Rack-on-a-Laptop" Part I (L2 + MLAG)
cumulusnetworks.com 19
VirtualBox Appliance Contains:
§ Two Cumulus VX leaf nodes +
Two RDO compute nodes
§ Custom tenant creation and
tear-down script
§ Command line input via any
local Web browser
§ Cumulus ML2 mechanism driver
enabled – create 1 or 2 tenants
http://tinyurl.com/RackOnALaptop
OpenStack Controller
Compute Node (Nova)
Network Node (Neutron)
Dashboatd Node (Horizon)
Compute Node
192.168.100.4/24192.168.100.3/24
192.168.100.2/24192.168.100.1/24
Mgmt
Bridge
Leaf 1 Leaf 2802.1q bond
Virtual Experience
Cumulus VX
Virtual Experience
Cumulus VX
swp18
swp17
swp18
swp17
swp32s0
ens0p9
swp32s0
ens0p9
host1 host2

20. Demo: Cumulus VX "Rack-on-a-Laptop" Part II (L3 to the Host)
cumulusnetworks.com 20
VirtualBox Appliance Contains:
§ One Cumulus VX spine node +
Two Cumulus VX leaf nodes +
Two RDO compute nodes +
One Debian external router
§ Custom tenant creation and
tear-down script
§ Command line input via any
local Web browser
§ Quagga packages on each compute node for
Layer-3 to the host with BGP unnumbered http://tinyurl.com/RackOnALaptop-2

21. OpenStack Network Design: Layer 2 vs. Layer 3
cumulusnetworks.com 21
VMVM
bridge - <>bridge - <>
subinterface
taptap
subinterface
802.1q trunk 802.1q trunk
802.1q bond
VMVM
bridge - <>bridge - <>
subinterface
taptap
tap tap
taptap
subinterface
vRouter
L3 Agent
DHCP AgentDHCP Agent
3
4
VXLAN –> Tunnel IP
Server1 Network Node
172.16.1.1
172.16.1.2
192.168.40.2192.168.40.3/24
VM
br-<random> br-<random> br-external
TAP
VXLAN-2061
eth0
eth0 eth0 eth0
swp1 swp8
swp47
vRouter
VXLAN-2061
Mgmt Network
1
2 5
6
203.0.113.1/24
203.0.113.2/24
Layer 2 + VLAN Layer 3 + VXLAN