Scaling an Openstack deployment requires serious thought and consideration, commonly it is achieved by multiplying the constrained resource. A notable exception is Neutron, in particular the L2 subsystem. The L2 network needs to scale up while most connected components are scaling out. A common approach to scaling the number of available L2 network segments in a region is the introduction of advanced overlay networks, with the drawback that all connected devices need to implement the overlay protocol. Installations requiring multiple hypervisor types or bare metal have a limited number of overlay protocol options. Neutron provides an alternative to a single overlay protocol; separating L2 in the network fabric from the protocol used on the network edge via Hierarchical Port Binding. This talk introduces a Neutron HPB architecture and ML2 implementation currently in use at SAP. We will discuss the issues that HPB solved and the challenges faced during our HPB deployment.

1. Scaling Neutron
In Heterogeneous Environments
Martin Klein, SAP
June 21, 2016

2. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 2
Agenda
Where do we come from
 Overview of the SAP Monsoon Converged Cloud
Where do we want to go
 Transitioning a private IAAS Platform to OpenStack
How do we want to get there
 Architecting OpenStack Neutron to enable growth

3. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 3
SAP Monsoon Converged Cloud
SAP Internal IAAS/PAAS Platform
Currently running In-House SW Stack with some OpenStack Components. Offering
Custom API and Amazon EC2 compatible API’s.
IAAS
 Global footprint with currently 6 Regions on 4 Continents
 Provide a unified global platform for SAP’s Cloud offerings
 Offers Compute, Block Storage and limited Networking
PAAS
 Focus on automation and Continuous Delivery using OpsCode Chef
 Optional for customers which don’t bring their own PAAS service

4. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 4
SAP Monsoon Converged Cloud
Current Scale
Platform is offered in 6 Regions extending to 15 In 2016
Absolute Size
 CPU 17.000 Cores
 Memory 600 TB
 Storage 6.2 PB
 Instances 27.000
 Volumes 58.000
Operations
 Instance Operations 3.000/Day
 Instance Growth 250/Day

5. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 5
SAP Monsoon Converged Cloud
OpenStack Transition
Replace all infrastructure controllers with OpenStack implementation
Existing Implementations
 Replace with standard OpenStack implementations
 Transition custom services to an OpenStack like schema
 Introduce a thin layer on top of Keystone to to reflect special requirements
New Features
 Swift - Object Storage
 Neutron – Networking including L2/L3, LBaaS, VPNaaS and FWaaS
 Designate - DNSaaS
 Manila - NFSaaS
 Monasca - Monitoring as a Service

6. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 6
SAP Monsoon Converged Cloud
Key Challenges
Running OpenStack on an Enterprise Hardware Stack
OpenStack Challenges
 Running Nova in a Multi Hypervisor Environment
 Scaling Neutron beyond the limits of VLAN
Infrastructure Challenges
 Scaling a network Fabric beyond 4k L2 Networks
 Attaching arbitrary devices to the Fabric without additional requirements on connected
devices
 Finding a universally available overlay protocol

7. Scaling in OpenStack

8. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 8
OpenStack Scaling I
Neutron
Nova
Cinder
Manila
No
Ne Ci
Ma
No
NoNe
Ma

9. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 9
OpenStack Scaling III
Neutron
Nova
Cinder
Manila
No
Ne Ci
Ma
No
NoNe
Ma
No No
NoNo
Ne
NeCi
Ci
Ma
No
Ma
No
Neutron – L2
• OpenStack Services are designed to scale out
• Exception: Neutron L2 needs to scale up

10. Neutron ML2 Network Model

11. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 11
Neutron ML2 Network Model
Network Type
 Defines Network Protocol
 Is type ID of Neutron networks
Mechanism Driver
 Is responsible for implementing
the network types on a specific
Device
 Is responsible for binding the
ports
Core Plugin
Type Manager Mechanism Manager
Linux
Bridge
Type Driver Mechanism Driver
Vendor
Linux
OVS
GRE VxLANVLAN

12. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 12
Neutron ML2 VLAN Networks
Neutron VLAN Networks
Network Level
 All VLAN ID’s available
everywhere
 All Host ports in trunk mode
Host Level
 Trunk port as uplink for OVS
 OVS Handles tag for vnic
VM Level
 Interface in access mode
VLAN ID 1-4096
VLAN ID 1-4096 VLAN ID 1-4096
VM
vnic0
OVS
eth0
Trunk Trunk
Trunk
Trunk
Access
VM
vnic0
OVS
eth0
Trunk
Trunk
Access
Network Level
Host Level
VM Level

13. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 13
Neutron ML2 VxLAN Networks
Neutron VxLAN Networks
Network Level
 Provides L3 Underlay Network
 All Host have access
Host Level
 Has access to undelay
 OVS Handles Encap-Decap
VM Level
 Interface in access mode
VLAN ID 10
VLAN ID 10 VLAN ID 10
VM
vnic0
OVS
eth0
Trunk Trunk
Access
Encap
Access
VM
vnic0
OVS
eth0
Access
Encap
Access
Network Level
Host Level
VM Level

14. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 14
Neutron VxLAN vs VLAN Type Networks
VLAN Type Networks
 PRO: Available Everywhere
 PRO: Low overhead on the Host
 CON: Very limited scale (4K theoretically, 2k realistically)
 CON: Entire Fabric needs to be aware of all available VLAN’s
VxLAN Type Networks
 PRO: Large ID Space (16M theoretically, 20K realistically with Flood and Learn )
 PRO: Fabric needs only be L3 and not aware of VxLAN
 CON: Not available on every OS / Device (Only Flood and Learn commonly available)
 CON: High CPU Overhead for Encap-Decap
 CON: Large number of VTEP if termination on every device.

15. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 15
OpenStack Scaling III
So what’s the problem again
No Common Overlay protocol available in the Enterprise environment
No
Ne Ci
Ma
No
NoNe
Ma
No No
NoNo
Ne
NeCi
Ci
Ma
No
Ma
No
Neutron – L2

16. Neutron Hierarchical Port Binding

17. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 17
Neutron Flat Port Binding
Neutron Network
Network Details (Global)
 network_type: vxlan
 segmentation_id: 14410
Network requirements
 All devices use one protocol
 All network layers are protocol
aware
Core Network
Edge Switch Edge Switch
HV
Storage
Device
Network
Device
HV
VM VM
Port Port
VNI:14410 VNI:14410
VNI:14411 VNI:14411
Port Port

18. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 18
Neutron Hierarchical Port Binding
Neutron Network
Network Details (Global)
 network_type: vxlan
 segmentation_id: 14410/11
Network Local (local)
 network_type: vlan
 segmentation_id: local
Network requirements
 Core/Edge devices share
protocol
 Connected devices are not
overlay aware
HV
Storage
Device
Network
Device
HV
VM VM
Port Port
VLAN: 40 VLAN:10
VLAN:10 VLAN:15
Core Network
Edge Switch
VNI:14410-14411
VLAN:15 VLAN:10
Edge Switch
VNI:14410-14411
VLAN: 40 VLAN:10
Port Port

19. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 19
Neutron Hierarchical Port Binding II
Neutron HPB Networks
Network Core Level
 Provides Underlay for VXLAN
 Can be routed networks
Network Edge Level
 Trunk port as uplink for OVS
 Translation local VLAN  VxLAN
Host Level
 Trunk port as uplink for OVS
 OVS Handles tag for vnic
VM Level
 Interface in access mode
VxLAN ID 1-16M
Local VLAN ID 1-4096 Local VLAN ID 1-4096
VM
vnic0
OVS
eth0
L3 L3
Trunk
Trunk
Access
VM
vnic0
OVS
eth0
Trunk
Trunk
Access
Network Level
Host Level
VM Level

20. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 20
Neutron Hierarchical Port Binding IV
OpenStack Configuration
Without Hierarchical Port Binding
With Hierarchical Port Binding

21. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 21
Neutron Hierarchical Port Binding IV
OpenStack Binding Levels
> net-show 4f904b00-9705-4098-85ff-023eb57ec29e
….....
| segments | {“network_type": "vxlan", “physical_network": null, “segmentation_id": 10059} |
| | {“network_type": "vlan", “physical_network": "bb98-fi", “segmentation_id": 3927}|
Each Network has multiple segments
neutron=# select * from ml2_port_binding_levels
port_id | host | level | driver | segment_id
-------------+---------------------------+-------+-------------+-------------------------
55f29793-... | l3-asr1.labx.mo.sap.corp | 0 | aci | d27f202d-...
55f29793-... | l3-asr1.labx.mo.sap.corp | 1 | openvswitch | de7b0947-...
Each Port can be bound against mutilple segments,
dependant on the binding host
Neutron HPB Ports
Network Edge Level
 Performs Partial Binding
 Creates Lower Level segment
 Assigns lower segment ID
Host Level
 OVS Driver completes Binding
 OVS Handles tag for vnic

22. © 2016 SAP SE or an SAP affiliate company. All rights reserved. 22
Scaling Neutron
Summary
VLAN Type Networks
• Very efficient on the Host Level
• Very limited Scale
VxLAN Type Networks
 Very large overhead on the Host Level
 Not commonly available
 Provide large ID space
HPB Networks
 Provide overlay type scale with VLAN type host complexity
 Encapsulation done on network edge in wire speed

23. © 2016 SAP SE or an SAP affiliate company. All rights reserved.
Thank you
Contact information:
Martin Klein
Cloud Infrastructure Architect
martin.klein01@sap.com