White paper submitted to the Society of Cable Telecommunications Engineers (SCTE) by Mazen Khaddem of Cox Communications and Dr. Loukas Paraschis of Cisco Systems. Paper covers technical reference design in SDN including the role of open source, orchestration and control, and the importance of a hybrid control plane for legacy, multivendor networks.

1. SDN and NFV value in Business Services.
Innovations in Network Monetization and Optimization.
A Technical Paper prepared for the Society of Cable Telecommunications Engineers
By
Mazen Khaddam
Network Architect
Cox
Atlanta
Mazen.Khaddam@cox.com
Loukas Paraschis
Technology Solution Architect
cisco,
loukas@cisco.com

2. Overview
The increasingly maturing SDN and NFV innovations offer an important opportunity for
service providers to better monetize their networks, by improving the time-to-market,
and SLAs guarantees for premium services, as well as the network utilization of volume-based
service delivery. This paper outlines the main benefits SDN and NFV can bring
to network service delivery, especially for business services, motivating the adoption of
SDN and NFV in the network architecture.
Most notably, SDN and NFV enhancements to the already ubiquitous cloud model can
improve time-to-market, add new functionality, and ensure customer loyalty for
applications such as dynamic capacity business VPNs, or policy-based service delivery,
at the edge of the network. They also remove many of the legacy constraints among
the access, WAN, and the data center, and enable advanced demand engineering, and
capacity optimization at the core.
To this end, an evolutionary approach to the adoption of SDN is being proposed, based
on a “hybrid” control plane architecture that combines the current distributed control-plane
routing infrastructure, with a unified “controller” platform that provides new
significant network visibility and programmability. The SDN controller capabilities are
enhanced by innovations in network protocols, APIs, and most notably new user-defined
network applications. The SDN evolution is complemented by the NFV
capabilities. NFV services provide the real-time network resource management needed
to support new applications to be deployed on-demand, and with the ability to choose
where each service may be placed. Equally important, SDN and NFV are enhanced by
cross-domain orchestration that can manage service chains across hybrid cloud and
data-center (DC) architectures to deliver seamless connectivity between compute
services in the enterprise and the cloud.
Contents
Growing the revenue of network and cloud services, especially for business customers,
is arguably among the highest priorities for network operators in general, and cable
service providers in particular. Currently, such services are very often challenged by
lengthy provisioning and complicated operations, which usually limit significantly the
operators’ ability to fully monetize their network infrastructures, and to compete with
over-the-top (OTT) providers for cloud based services. The recent, increasingly
maturing SDN and NFV innovations allow service providers to improve their network
monetization by improving the time-to-market, and SLAs guarantees for premium
services, as well as to optimize their network utilization for volume-based service
delivery. This paper outlines the main values that SDN and NFV innovations can bring

3. to servic
motivate
As the n
cloud se
time ser
services
the tradi
(Figure
Network
new netw
ce provider
e the fast ad
new era of a
ervice delive
rvice deliver
s such as dy
itional netw
1). Two ne
k Function V
working era
Figur
SDN allo
requirem
enable a
services
applicati
Of cours
brings a
of transi
cloud or
use of n
leverage
multiven
analytics
other ap
capabilit
architect
enterpris
networks, e
doption of S
agile servic
ery models
ry are enha
ynamic cap
work service
ew key tech
Virtualizatio
a.
re 1: Service
ows networ
ments. At th
applications
s can be pla
ions, thus e
se, the com
lso new ch
ent network
rchestration
etwork, com
e the SDN i
ndor, virtual
s, mobility,
pplications.
ties to man
tures, thus
se and the
especially i
SDN and N
e creation w
, new requi
ancing the a
pacity busin
e domains o
hnologies, S
on (NFV) [2
es innovatio
rks to beco
he same tim
s to be depl
aced anywh
enhancing t
mmercial be
allenges in
k and servi
n solutions n
mpute, and
innovations
l and physic
collaboratio
The enhan
age “servic
delivering s
cloud.
n their bus
FV in the n
with virtuali
irements fo
already ubiq
ness VPNs,
of access, e
Software-De
], enable se
on based o
me much m
me, NFV an
loyed on-de
here within
the way net
nefits of eff
customer S
ces resourc
need to ens
storage re
s in data ce
cal, infrastr
on, manage
nced comm
ce chains” a
seamless c
iness servic
network arc
ized IT infra
or on-deman
quitous clo
or policy-b
edge and co
efined Netw
ervice prov
n SDN, NFV
more agile a
nd its real-ti
emand. Wi
the networ
tworks are
ficient, elas
SLA expect
ces. There
sure service
esources. T
nter, WAN,
ructure, and
ed services
mon orchest
across hybr
connectivity
ce offerings
chitecture.
astructure h
nd connect
ud model w
based mobi
ore WAN, a
working (SD
viders to bes
V, and cross
and respon
ime networ
ith SDN an
rk, accordin
designed, o
stic provisio
tations, esp
efore, advan
e automatio
This cross-
, and acces
d support re
s, video, ge
tration platf
rid cloud an
y between c
s, which ind
deed
has prolifera
tivity and re
with network
ility, mergin
and data ce
DN) [1], and
st evolve to
s-domain o
sive to serv
rk resource
d NFV wor
ng to the ne
operated, a
oning of netw
pecially in a
ncements in
on, with rap
-domain orc
ss (Figure 1
eal-time sec
neral conte
forms also a
nd data-cen
compute se
ated
eal-k
ng
enter
d
o this
orchestratio
vice
manageme
king in con
eeds of spec
and monetiz
work servic
an environm
n conventio
pid and sec
chestration
1) to contro
curity,
ent, and ma
allow NFV
nter
ervices in th
n.
ent
cert,
cific
zed.
ces
ment
onal
cure
can
l
any
he

4. This new architecture offers significant benefits for network service providers in terms of
enhanced service provisioning and extending the virtualization innovations in compute
and storage, to networking. More specifically, the two most immediate use-cases are:
 SDN innovations combined with new network provisioning functionality, most notably
those achieved through the innovations of NETCONF protocol and YANG models,
can significantly advance the automation of network provisioning, and reduce the
time-to-market for new services, allowing substantial operational simplification.
 At the same time, substantial benefits arise from the NFV ability to create “virtual”
Provider Edge (vPE) and Customer Provider Edge (vCPE) functionality which can be
customized to the specific needs of each application. NFV applications can execute
in a virtual environment, running over a mix of physical and virtual infrastructure
components, and using service chaining (or “forwarding graphs” in the NFV
terminology) to link functional blocks together to provide sophisticated service sets
tailored to specific users. This is particularly important for business services. It also
offers faster time-to-market for new services, and lowers infrastructure costs (both
CapEx and OpEx).
In this evolved service architecture, compute can take many forms, ranging from large
data centers environments, to distributed compute instantiation around the network.
When combined with the fast, automated service provisioning of SDN, NFV and cross-domain
orchestration (Figure 1), this new architecture gives rise to very interesting
monetization and optimization opportunities, allowing network service providers to
leverage the network as a key service differentiation advantage, for managed and cloud
services.
One key such use-case is the ability for optimal placement of cloud services, which is
also referred to as “demand engineering”. In this advanced network optimization
scenario, a service instance is placed, or content is located, using “global” network
awareness (e.g. topology, traffic, etc.) to determine optimal SLA, or network utilization
[3]. Demand engineering has been reported to increase the network infrastructure
utilization by around 30% in most cases [3].
Until now, slow service provisioning has demotivated most types of fast, let alone
dynamic, bandwidth provisioning. Hence, off-line planning, occasionally coupled with
some traffic engineering, has addressed sufficiently the traffic management needs of
most IP/MPLS networks. As the deployment of cloud services proliferates and is
enhanced by faster NFV and SDN provisioning, advanced network control capabilities
can optimize the trade-offs between SLA performance and network utilization, and offer
some new exciting use-cases for network monetization, as described later in Figure 4.
Even before the implementation of such advanced network control capabilities, like
traffic placement and demand engineering, SDN offers immediate substantial

5. operational (OpEx) benefits for network operators. In particular, for cable operators
SDN can enable improved business services workflow automation based on a unified
control and operating model, common to all network elements. Cable business services
can therefore converge with residential services, much more readily than today,
allowing for significant OpEx reduction.
In the rest of this paper, we outline the key SDN architecture and technology
innovations that advance network operations, and business service offerings, and are
important in the adoption of a robust SDN architecture.
SDN typically refers to a network architecture vision that has been championed by the
Open Networking Foundation (ONF) [1]. In this vision, data networking equipment and
software can separate and abstract the application, control and data plane. The control
plane resides centrally, decoupled from the forwarding components which remain
distributed. The central controller(s) can enhance network operations by introducing the
abilities to:
 Maintain full view of the network
 Program the network equipment
 Provide an abstraction of the network for higher-level applications.
Central to SDN evolution are the openness, network simplification, programmability, and
abstraction capabilities. This ability for programmatic interaction of the control plane
with applications and network elements is indeed the key innovation of the SDN
architecture. In the “northbound” direction, the control plane provides a common
abstracted view of the network to higher-level applications using APIs. In the
“southbound” direction, the controller programs the (physical or virtual) network
elements using new or existing network protocols, or APIs. Particularly in service
provider environments, an evolutionary architecture needs to accommodate also the
existing pre-SDN infrastructure, and hence to extend well beyond the ONF vision.
Service Providers have large operational networks and significant investment in
OSS/BSS infrastructure. For the SDN evolution to succeed, its adoption cannot
compromise existing functionality, the current carrier-class reliability, and the support for
the available standardized technologies, and multivendor systems. At the same time, it
is also important to enable network differentiated quality of experience to the end-user.
A new SDN hybrid control plane (Figure 2), combining the current distributed control
plane components residing within network elements, with centralized controllers, offers
the best SDN evolution to a network able to enhance customer experience, and allow
for service abstraction and capacity optimization.

6. Figure 2
SDN d
2: The SDN “
distributed c
More sp
consistin
layer. So
when dis
from a c
distribut
offers th
default (
control t
and also
capabilit
• Service Pro
• End User A
• External IS
hybrid” co
control-plan
well as appli
networks ar
ributed con
ol plane func
On the othe
al view of e
plane with c
lution towa
egacy) conf
zes the netw
orchestratio
work servic
ovider Applica ons – OSS/BSS, O
Applica ons
Ps / Content Providers
• Augments
• Control app
• Elementary
Controller NB APIs
Orchestra on etc
distributed control plane
plica on – func on specific
y Infrastructure Func ons– com
• Simplified
• Augmente
• Data plane
Device Level APIs
distributed control plane
d by central controllers
e forwarding
Figure 3
protocol
applicat
Key to a
architect
users, to
w
pecifically, n
ng of a distr
ome contro
stributed. O
central glob
ed control p
he best evol
including le
that optimiz
o improve o
ties for netw
Applica ons
SDN Controller
Network
mon pla orm
ontrol-plane
ne (left) thro
ications, an
re currently
ntrol plane,
ctions, such
er hand, op
end-to-end
centralized
rds network
figuration c
work perfor
on and prov
e differentia
itecture for
, and servic
: SDN arch
ls and APIs
ions.
achieve bett
ture that all
o interact w
e evolution
ough the int
nd network-built
advanced n
ce and netw
ter network
lows user a
with the netw
k monetizat
applications
work and im
augments t
troduction
programma
routing/sw
r no interac
IGP conver
c placement
onditions [4]
e and a pro
ion. This h
to be comb
specific app
us offering
using
with little or
h as rapid I
timal traffic
network co
intelligence
k optimizati
apabilities t
mance for s
isioning, th
ation.
network co
work optimiz
the function
of centraliz
atic APIs.
witching plat
ction with th
rgence, wor
t can certai
]. Combinin
ogrammable
ybrid appro
bined with c
plications, f
significant
ntrol throug
zation throu
ion and opt
s, operated
mprove perf
nality of the
zed control,
e pre-as
tforms
he applicatio
rk indeed b
inly benefit
ng the
e infrastruc
oach allows
centralized
flows, or us
new
gh “southb
ugh “northb
timization is
by service
formance, e
bound”
bound”
s a program
provider, o
ease mana
on
best
cture,
s
sers,
mmable, op
or even end
gement, or
pen,
d-r

7. increase infrastructure utilization. A complete such SDN architecture shall include the
following four basic building blocks (Figure 3):
 SDN Controller(s): the set of software tools, and technologies that offer
centralized intelligence, network abstraction (northbound), and programmatic
network control (southbound).
 Infrastructure: physical and virtual network elements, which in the case of the
WAN can also include multiple layers; e.g. extend to optical transport [5].
 Application Programming Interfaces: APIs and protocols that enable
programmability at multiple levels of the SDN infrastructure. At the lowest level,
device level programmatic interfaces and protocols enable SDN control of network
elements. Separate, northbound APIs in the SDN architecture allow end-user
applications to communicate with the controller layers.
 Applications: the most important and novel aspect of SDN that enables network
operators or end users to program the network through controller(s).
These “top” layer software applications can utilize APIs exposed by the controller to
request specific behavior from the network, or gather information about the network.
These APIs enable business processes to be programmed and become part of the
network operations, and should also facilitate graceful migration and integration with the
existing BSS/OSS. Representational State Transfer (REST) APIs have emerged as the
de-facto standard framework for the interaction between these applications and the
controller layers [6].
Currently, the available, first generation, SDN controllers are application specific and
typically designed to interact directly with network, each one independently. For simple
network designs, such applications controllers may be acceptable. However, in large
networks, and particularly in highly heterogeneous WAN, where the control functions
need to interface with many devices using a multitude of protocols, such first generation
designs would result in significant additional development effort, and limit scale, as
network devices are touched for data retrieval and programming by many different
functions. Therefore, the most scalable SDN WAN architecture could benefit from a
unified single infrastructure controller that in turn enables all the different higher layer
application specific controllers to interact with the network in a common framework. The
unified infrastructure controller can then provide a common view of the network, gather
and hold network information, provide centralized control functions, and program each
network element using the appropriate device level APIs and/or network protocols. This
functional separation between “application controllers” and the “infrastructure controller”
allows for: 1) A unified infrastructure that provides a single point of contact to the
network, both for information retrieval and programming, and 2) Each application
controller to not be concerned with the precise mechanisms for interacting with the
network, like the device specific API, or protocol applicable in each network element.

8. Open Da
Foundat
SDN thr
enables
ODL infr
network
basic ne
The SDN
operatio
NETCO
“southbo
include:
Element
its Path
An impo
leveragi
multiven
for conv
protocol
architect
innovatio
aylight (OD
tion, has be
rough the cr
both eleme
rastructure
service fun
etwork func
N “southbo
ons, and ha
NF and YA
ound” progr
Open Flow
t Protocol (
Computatio
ortant value
ng open so
ndor networ
verged mult
details from
ture mature
ons of each
Figur
There ar
innovatio
applicati
 A pa
e 4: Examp
re multiple
ons describ
ions are as
rticular netw
ting innovat
Rout
DL), an indu
een formed
reation of a
entary infra
platform co
nctions, and
tions for en
und” APIs a
ve also offe
ANG may be
rammatic A
w, the exten
PCEP) to a
on Clients (
of these S
ource imple
rks. Even m
ivendor net
m the netw
es, the netw
h protocol,
ustry-wide, o
with the go
common m
astructure fu
onsists of m
d also a Se
nd-user app
and protoco
ered signific
e the most
APIs. Other
nsions of the
allow for sta
PCC), and
DN protoco
mentations
more impor
twork progr
ork users.
work applica
become the
ple of an SD
SDN contro
bed above,
sociated ei
work techn
tions that le
open sourc
oal of furthe
multivendor
unctions an
many southb
ervice Abstr
plications.
ols are impo
cant innova
notable pro
r notable “s
e IETF stan
ate-aware p
the link-sta
ols is that th
s - standard
rtant, a sop
rammability
Therefore,
ations, rath
e important
N applicatio
oller applica
to advance
ther with:
ology, like f
everage SD
ce, project o
ering the ad
r controller
nd controlle
bound APIs
raction Laye
operating u
doption and
platform [7
r applicatio
s and “plug
er (SAL) tha
ortant in en
ations. As a
otocol innov
southbound
ndard for th
provisioning
ate extensio
hey are typ
ds that allow
histicated S
y that essen
as the SD
her than the
t focus.
on for netw
ations that
e the WAN
for example
DN with sou
nder the Li
d innovation
7]. ODL
ons [7]. The
-ins”, basic
at abstracts
nabling imp
already me
vation exam
d” innovation
he Path Com
g, and activ
ons for BG
ically open
w for hetero
SDN infrast
ntially “abst
N evolution
e specific de
work moneti
can leverag
service del
e the recen
urce routing
nux
n of
e
c
s the
proved netw
entioned,
mple of
n examples
mputation
e operation
P (BGP-LS
- often
ogeneous,
tructure allo
tracts away
n of the WA
etails and
zation and
ge the SDN
livery. Typ
nt, importan
g to simplify
work
s
n on
S).
ows
y” the
AN
optimizatio
N infrastruct
ically, these
nt Segment
y network an
on.
ture
e
nd

9. service engineering by removing most complex state from the distributed control-plane
[8],
 Workflow automation of services or network functions like analytics, policy,
optimization, or orchestration [9].
Figure 4 describes the basic implementation of a significant such SDN optimization
application specifically for services with well-defined profiles, which include most
business services. For such “controllable” services, when the provider does not need to
guarantee the exact timing of delivery, e.g. asynchronous bulk data transfers, an
intelligent WAN SDN controller can time shift these services away from times of high
next utilization (“peaks”) to times of otherwise low network utilization (“valleys”).
Alternatively, for services that are controllable but with timing well-defined and inflexible
scheduling (e.g. synchronous business data backups), an SDN WAN controller
application can instead leverage network information (gathered potentially real-time) on
utilization, or failures, or other performance attributes, to identify the optimal routing for
this traffic given its specific SLA requirements. The same unified SDN controller then
can potentially also program the required LSPs in the network, perhaps by using PCEP
as the southbound protocol. A very good example of an actual WAN deployment that
leverages such an intelligent SDN implementation to optimize the network delivery of
controllable services has been extensively analyzed in [10].
As mentioned in the beginning of this paper, a sophisticated SDN infrastructure enables
a network vision where user-defined applications allow optimal placement of each new
service instance according to user-defined SLA or network utilization constraints, based
on advanced network optimization that leverages “global” network awareness of
topology, traffic, or location of the required content, compute, or storage resources, as
for example described in [3]. Until now, slow service provisioning has allowed off-line
planning, occasionally coupled with traffic engineering, to address sufficiently the traffic
management needs of WAN networks. As cloud services deployment proliferates,
faster NFV and SDN service provisioning can be significantly enhanced by advanced
network control capabilities that optimize the trade-offs between SLA performance and
network utilization, allowing for better network monetization. This programmability
becomes important for the overall evolution of WAN architectures to the network cloud
era of cable and telecom operators [11].
In summary, this paper outlines the main benefits SDN and NFV can bring to network
service delivery, especially for business services. Most notably, SDN and NFV
enhancements to the already ubiquitous cloud model can add new functionality, and
ensure customer loyalty for applications such as dynamic business VPNs, or policy-based
service delivery, at the edge of the network. They also remove many of the
legacy constraints among the access, WAN, and the data center, and enable more
advanced demand engineering, and capacity optimization at the core. An evolutionary
approach to the adoption of SDN is being proposed, based on a “hybrid” control plane

10. architecture that combines the current distributed control-plane routing infrastructure,
with a unified SDN “controller” platform that provides new significant network visibility
and programmability. The SDN controller capabilities are enhanced by innovations in
network protocols, APIs, and most notably new user-defined applications. The SDN
evolution is complemented by the NFV capabilities. NFV services provide the real-time
network resource management needed to support new applications, deployed on-demand,
and with the ability to choose where each service may be placed. Equally
important, SDN and NFV are enhanced by common platforms for orchestration that can
manage service chains across hybrid cloud and data-center architectures to help deliver
seamless connectivity between compute services in the enterprise and the cloud.
These increasingly maturing SDN and NFV innovations offer an important opportunity
for service providers to better monetize their networks, by improving the time-to-market,
and SLAs guarantees for premium services, as well as the network utilization of volume-based
service delivery.
Bibliography
The authors would like to acknowledge many insightful discussions with colleagues at
Cox and Cisco, including Jeff Finkelstein, Simon Spraggs, and Alon Bernstein.
1. https://www.opennetworking.org/sdn-resources/sdn-library/whitepapers
2. http://portal.etsi.org/home.aspx
3. J. Evans, et al “SDN-based traffic management…”, MPLS World Congress 2014
4. M. Horneffer, “IGP Tuning in an MPLS Network”, NANOG 33, February 2005
5. M. Khaddam, et al, Multilayer Network Optimization, invited paper IEEE OFC 2015
6. http://en.wikipedia.org/wiki/Representational_state_transfer
7. http://www.opendaylight.org/project/technical-overview
8. http://tools.ietf.org/html/draft-martin-spring-segment-routing-ipv6-use-cases-00
9. D. Ward, “Simplifying the WAN...”, Plenary (d1-08), MPLS World Congress 2014
10. http://cseweb.ucsd.edu/~vahdat/papers/b4-sigcomm13.pdf
11. L. Paraschis “Advancements in Network Architectures…”, pp. 793–817 in Op. Fib.
Telecom. VI B, Elsevier 2013. (ISBN 978-0123969606)