June 20–23, 2016 | Berlin, Germany

Composing a New OPNFV
Solution Stack
Wojciech Dec, Cisco

Abstract
This session showcases how a new OPNFV solution stack
(a.k.a. “scenario") is composed and stood up. We'll use a
new solution stack framed around a new software
forwarder ("VPP") provided by the FD.io project as
example for this session. The session discusses how an
evolution/change of upstream components from
OpenStack, OpenDaylight and FD.io are put in place for
the scenario, how installers and tests need to be evolved to
allow for integration into OPNFV's continuous integration,
deployment and test pipeline.

Introduction
• What is an OPNFV Scenario?
A selection of software components that can be automatically installed and
tested via the OPNFV CI/CD pipeline.
• The Realization of a scenario is a key release vehicle for OPNFV
• Drives evolution/development of components
• Drives development of system test
• Necessitates an installer and installer platform
• Translates into an continous integration & test deployment test pipeline

New OPNFV FastDataStacks (FDS)
Scenarios
• Components:
• Openstack
• Opendaylight w/ Neutron & GBP
• Fd.io: VPP
• OPNFV Installer & Test
• Diverse set of contributors:
• Scenario variations covered: L2,
L3, HA
Install Tools
VM Control
Network Control
Apex
OpenStack
OpenDaylight
L2
Hypervisor KVM
Forwarder VPP
Apex
OpenStack
OpenDaylight
L3
KVM
VPP
Apex
OpenStack
KVM
VPP

New OPNFV FastDataStacks (FDS)
Scenarios
• Components:
• Openstack
• Opendaylight
• Fd.io: VPP
• OPNFV Installer & Test
• Diverse set of contributors:
• Scenario variations covered: L2,
L3, HA
Install Tools
VM Control
Network Control
Apex
OpenStack
OpenDaylight
L2
Hypervisor KVM
Forwarder VPP
Apex
OpenStack
OpenDaylight
L3
KVM
VPP
Apex
OpenStack
KVM
VPP

Why FD.io - The Vector Packet Processor
Dataplane?
Existing Dataplanes Fd.io VPP
Performance, Scalability &
Stability issues
Highly performant, modular and
designed for scale, no drops and
minimal delay
„In house” Architectures; hard
to evolve, slow to innovate
A Linux Foundation Collaborative open
source project. Multi vendor support &
project governance
Difficult to upgrade & operate.
Deep kernel dependencies
Standard based protocol and
configuration model driven
management agents.
Activation of new plugins at run-time
Processor specific Support for multiple processor
architectures (x86, ARM)
Limited features L2 and L3 feature rich
Testing? Automated testing for all projects
Management Agent
VPP
(DPDK)
vSwitch
vFirewall
vRouter vFirewall
Custom
App

Highly performant? You bet…
VPP technology in a nutshell
› VPP data plane throughput not impacted by large FIB size
› OVSDPDK data plane throughput heavily impacted by FIB size
› VPP and OVSDPDK tested on Haswell x86 platform with E5-2698v3
2x16C 2.3GHz (Ubuntu 14.04 trusty)
OVSDPDK
VPP
0
5
10
15
20
2 MACs
2k MACs
20k MACs
NDR rates for 2p10GE, 1 core, L2 NIC-to-NIC
[IMIX Gbps]
OVSDPDK
VPP
0.0
20.0
40.0
60.0
80.0
100.0
120.0
12 routes
1k routes 100k
routes
500k
routes
1M
routes
2M
routes
NDR rates for 12 port 10GE, 12 cores, IPv4
[IMIX Gbps]

VPP Feature Summary
14+ MPPS, single core
Multimillion entry FIBs
Source RPF
Thousands of VRFs
Controlled X-VRF lookups
Multipath – ECMP & Unequal Cost
Multiple million Classifiers –
Arbitrary N-tuple
VLAN Support – Single/Double tag
Counters for everything
Mandatory Input Checks:
TTL expiration
header checksum
L2 length < IP length
ARP resolution/snooping
ARP proxy
IPv4/IPv6 IPv4
GRE, MPLS-GRE, NSH-GRE,
VXLAN
IPSEC
DHCP client/proxy
IPv6
Neighbor discovery
Router Advertisement
DHCPv6 Proxy
L2TPv3
Segment Routing
MAP/LW46 – IPv4aas
iOAM
MPLS
MPLS-o-Ethernet –
Deep label stacks
MPLS-o-GRE
L2
VLAN Support
Single/ Double tag
L2 forwarding with
EFP/BridgeDomain concepts
VTR – push/pop/Translate
(1:1,1:2, 2:1,2:2)
Mac Learning – default limit of
50k addresses
Bridging – Split-horizon group
support/EFP Filtering
Proxy ARP
ARP termination
IRB – BVI Support with
RouterMac assignment
Flooding
Input ACLs
Interface cross-connect

Why OpenDaylight?
• To offer a Software Defined Networking (SDN) platform, covering a broad set of
virtual and physical network devices
• OpenDaylight is an Open Source Software project under the Linux Foundation
with chartered with the development of an open source SDN platform
Code Acceptance Community
To create a robust, extensible,
open source code base that
covers the major common
components required to build
an SDN solution
To get broad industry acceptance amongst
vendors and users
• Using OpenDaylight code directly or through
vendor products
• Vendors using OpenDaylight code as part of
commercial products
To have a thriving and growing
technical community contributing
to the code base, using the code
in commercial products, and
adding value above, below and
around.

ODL Software Architecture
SAL/Core
Netconf
Client
Network DevicesNetwork DevicesNetwork Devices
Protocol
Plugin
... Netconf
ServerRESTCONFApplication Application
REST
ApplicationsApplicationsOSS/BSS, External Apps
Data Store Messaging Core
Apps/Services
Yang Model
Data RPCs, Notifications

What is Group Based Policy?
● An intent driven policy framework
model intended to describe
network application requirements
independent of underlying
infrastructure.
● Concepts
○ Group Endpoints (EPs) into
Endpoint Groups (EPGs)
○ Apply Policy (Contracts) to
traffic between groups
○ Contracts apply directionally
EPG:Hosts
EPG:
WebServers
Contracts
web
Match:
destport:80
Action:
Allow
ssh
Match:
destport:22
Action:
Allow
any
Match: *
Action:
Allow
Contract:
web, ssh
Contract:
any
EP:1
EP:2
EP:3
EP:4

Endpoints live in a network context
● Network Context
○ Can be an L2-Bridge Domain
(L2BD)
○ Can be an L3-domain (L3D)
(think VRF)
● Network Contexts can have
Subnets
● Endpoint Groups can specify a
default network context for
members
EP:1
EP:6
EP:2
Bridge:1
Bridge:2
EP:5
Subnet:1
Subnet:2
Subnet:3
EPG:Hosts
EP:1
EP:2

Compute
VPP
SDN Controller w/ GBP
IPSec
IWAN
Operator Portal
FW &
Security
Tenant Portal
NFV Service Packs
L2 L3 FW LB
PublicPrivate
MPLS
VPN
VM Resource Orchestrator - Openstack
CPE1
Cust-A
CPE2
Cust-A
A Broader Picture on FDS
Service Orchestrator
SP Data Centre Infra
● Data Services… = Need for network workload
speed in a virtualized context (NFV, NFVI)
● VPP
● … with network application policies…
● Group Based Policy
● … to offer programmatic services on a
combination of virtualized and physical devices
against a common GBP policy: SDN Controller
● Opendaylight
Policy: End-points @ CPE 1 not allowed to communicate with EP’s @CPE2 unless over a private network

The Fast Data Stack w/ VPP Architecture
● Openstack VM control
● Opendaylight
○ Neutron Server
○ Group Based Policies
○ L2 & L3 Topology
○ Virtual Bridge Domain
Manager
○ VPP configuration Rendering
○ Netconf client
● Fd.io VPP:
● Honeycomb Netconf
configuration server
● Data Plane
Controller Node
OpenStack Neutron
ODL ML2 neutron plugin
ODL Neutron Service
Neutron to GBP mapper
GBP Manager
VPP renderer
Netconf
Honeycomb
neutron API REST calls
Policies
Legend
- In OpenStack
- In OpenDaylight
VDB
VPP
- In Fd.io
DPDK
Compute Node
Detailed Architecture at:
https://wiki.opnfv.org/display/fds/OpenStack-ODL-VPP+integration+design+and+architecture
Netconf
Topology
Store
Neutron to VPP mapper

Neutron to GBP mapperNeutron to VPP mapper
The Fast Data Stack w/ VPP Architecture
Controller Node
OpenStack Neutron
ODL ML2 neutron plugin
ODL Neutron Service
GBP Manager
VPP renderer
Netconf
Honeycomb
Legend
- In OpenStack
- In OpenDaylight
VDB
VPP
- In Fd.io
DPDK
Compute Nodes
POST PORT
(id=uuid, host_id=vpp, vif_type=vhostuser)
Update Port
Map Port to GBP Endpoint + relay policies
(Neutron specifics to Generic Endpoint mapping)
Update/Create GBP Endpoint (L2 context, MAC,...)
Apply Policy
Update node(s), bridge-domain
Netconf Commit
(bridge config, VxLAN tunnel config)
Topology
Store

Some differences to what you may be used to
1/2
• Per Tenant Bridge and an interface centric model
• No Flows, VLAN re-mapping “tricks”, etc
• *Significantly* simpler to trace/figure-out
VM_A
Tenant1 Bridge
Virtual
Eth0/0
Eth0
VirtIO/
VhostSrv
Vhost
client
VLAN
101
GE0/0.102
VM_Z
Tenant2 Bridge
Virtual
Eth0/0
Eth0
VirtIO/
VhostSrv
Vhost
client
VLAN
102
Socket
/tmp/uui
dA
Socket
/tmp/uui
dB
Configured by Nova Compute
Configured by ODL
VPP
GE0/0.101
Ostack with OVS Ostack with VPP

Some differences to what you may be used to
2/2
• Full network control & visibility
is in Opendaylight
• No network configuration
“hidden” into compute node
configuration (ovs-agent)
• Allows and visualization and
computation of *all* of the
topology in ODL
• Reconfigurable from the
controller
Compute Node 2
Honeycomb
VPP
VM3
(NFV3)
Port
Port
Port
Port
Port
VM4
(NFV4)
mgmt
int
Port internet
int
public
mgmt
int
private
int
public
int
Port
Port
private
int
Edge
networks
VxLAN/Vlan
s
Port
MGMT
Nova Agent
private
Bridge
ODL
Controller
Physical Toplogy 1:
Compute1 –> eth0 -> physical switch eth0/0
Compute2 ->eth1 -> physical switch eth1/0
Neutron Logical Topology: Net1
NFV1 -> private net (VLAN 100
NFV3 -> private net (VLAN 100)
Subtended by: Physical topology 1
Compute Node 1
Honeycomb
VPP
VM1
(NFV1)
Port
Port
Port
Port
Port
VM2
(NFV2)
mgmt
int
Portinternet
int
public
mgmt
int
private
int
public
int
Port
Port
private
int
Edge
networks
VxLAN/Vlan
s
Port
MGMT
Nova Agent
private
Bridge

Active Code Development realized
• OPNFV:
• Apex Installer: Automated install of Openstack w/ ODL, VPP and Honeycomb
on compute nodes
• Functests: Initial automated testing of
• Openstack (Mitaka):
• Neutron ODL ML2 parser extended to handle VPP elements and vhostuser
binding in picked up in topology
• Opendaylight (pre Boron):
• Group Based Policy Neutron & VPP data mapping
• Virtual Bridge Domain Manager (builds L2 bridges using VxLAN or native
VLAN interfaces) – new ODL project
• Extended topology models
• VPP Configuration rendering
• Fd.io
• Honeycomb and VPP model supporting VxLAN, VLAN, vhostuser

Adjustments Planned/Needed
• Apex
• Parameterization…. Parameterization … & Intelligent defaults...
• VPP’s netconf “mounting in ODL” after install.
• Install paramaters need to be user configurable, eg domain name, networks, etc.
• Testing
• Covered on next slide…
• Openstack
• ML2 ODL driver robustness and state cleanup
• ODL
• HA for Neutron, GBP
• L3 config: Router agent equivalent w/ VxLAN attachment.
• L2 ACL rendering
• DHCP TAP Interfaces
• Fd.io
• ACL support
• NAT support (for floating IP)
• Fix Interface “acquisition” issues on RHEL

Test Coverage Needs
• Automated set-up of Ostack, ODL and
multi-node HC
• Test from Openstack Port creation:
• Verify HC configuration
• Test from Openstack VM creation:
• Verify multi VM creation
• Verify VM-VM same node and “other node”
connectivity
• At least some scale Testing…
• HA Testing...
Honeycomb
(Dataplane Agent)
VPP
DPDK
OpenDaylight
Neutron NorthBound
GBP & VPP Neutron
Mapper
Topology
Mgr vBD
VPP
renderer
GBP Renderer
Manager
Lightweight-
ODL
Neutron
No VPP
coverage
No Full
ODL
coverage
Only
simulated
use tests

FDS Project Schedule – Near Term
CiscoLive Las Vegas (July 2016)
• Base O/S-ODL-VPP stack
(Infra complete: Neutron – GBP Mapper – GBP Renderer – Topology
Mgr – Honeycomb – VPP)
• Automatic Install
• Basic system-level testing
• Basic L2 Networking (no NAT/floating IPs, no Security Groups)
• Overlays: VXLAN, VLAN
OPNFV Colorado Release (September 2016)
• O/S-ODL-VPP stack
(Infra complete: Neutron – GBP Mapper – GBP Renderer – Topology Mgr – Honeycomb – VPP)
• Automatic Install
• Ongoing OPNFV system-level testing (FuncTest, Yardstick testsuites) – part of OPNFV CI/CD pipeline
• Complete L2 Networking (NAT/floating IPs, Security Groups)
• HA
• Overlays: VXLAN, VLAN, NSH
Detailed development plan: https://wiki.opnfv.org/display/fds/FastDataStacks+Work+Areas#FastDataStacksWorkAreas-Plan

Summary
• New OPNFV Fast Data Stack aimed at providing a high
performance NFV services platform
• VPP provides a highly performant & modular data plane
• Opendaylight with Group Based Policy provides SDN
device and a consistent network application policy
framework
• The stack’s functionality is growing.
• Lot’s of features that can never have enough tests
• Join us in building the FDS!