1. 2012 13th ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed
Computing
Experiments on Multi-Layer Network Virtualization
towards the Software Defined Transport Network
Akeo Masuda, Akinori Isogai, Daisaku Shimazaki, Yoshihiko Uematsu and Atsushi Hiramatsu
NTT Network Service Systems Laboratories, NTT Corporation
Musashino-shi, Tokyo, Japan
Email: masuda.akeo@lab.ntt.co.jp
Abstract—This paper proposes a novel architecture which change the route of the existing flows. In this case, they will
enables software defined networking not only at the routing be able to achieve better performance by acquiring additional
layer but also at the transport layer. Proposed architecture network resources, or optimizing the topology of tunnel paths.
provides multiple SDTNs with wide range of controllability
level, in spite that the SDTNs coexist upon a shared multi- For another example, they can achieve high availability if
layered network infrastructure. We have conducted a nation- they can prepare a SRLG(Shared Risk Link Group)-aware
wide experiment where we have provided SDTNs to practical protection path at the transport layer, designed in accordance
users such as broadcasting studios. Through the experiments, we to the design of server redundancy.
have successfully verified the resource management mechanism
and network control functionalities. On the other hand, network carriers do not devote their net-
work resources to a single service or single user. They logically
I. I NTRODUCTION slice their network resources to launch a new service including
inter-cloud connection, and provide the portion of the slice to
Recently the main players and the drivers of the devel-
each user. Sharing the infrastructure by multiple usage of the
opment of networking technologies seem to be shifting to
network is usually done in most of the network providers to
operators and users of datacenter. Software developers of cloud
keep their competitiveness by the cost efficient operation of
operators are eager to totally program the operation of not
their infrastructure. This can be seen as a virtualization of the
only their computing equipments, but also the network. Inside
network infrastructure. The difficulty of network virtualization
and between the datacenters, there are numerous dataflows
is to offer the programmability at the same time.
between virtual machines (VMs) running upon numerous com-
puters, and they keep being generated and changed dynami- Speaking generally, network providers do not desire to allow
cally. The concept of software defined network is expected users to freely configure the network equipments. It may
to release the network operation from time-consuming tasks cause serious problem if a certain user of the network directly
of manual configuration of each network equipment along configures the functionalities of the routers and switches.
which the flow traverses. This enables programmed control It will prevent the fair use of the network among multiple
of the dataflow routing in order to achieve optimization, users and services, and causes conflict between the controls
scalability and resiliency of the network, similar to the way of from multiple users. In order to offer programmability of the
management where the cloud operators program the usage of transport layer, we need a new technology to overcome this
computing resources. OpenFlow[1] is expected to be the main problem.
enabler of SDN (Software Defined Network). This technology Several works had addressed the architecture of total control
lets cloud operators to explicitly designate the route at flow of the network including the SDN layer and transport layer [2],
level granularity, and slice the network capacity to multiple [3]. However, previous researches only focus on the integrated
independent tenants. control of both layers by unification of the control plane.
However, at least in the past, SDN had been seen to be only Software defined control of the transport layer where multiple
the enabler of control function at the flow routing level. We users share the infrastructure is still an open issue at this
believe that controllability should be enhanced deeply to the moment.
transport level for full utilization of network resources. The
main contribution of this paper is to address an architecture We propose the SDTN architecture that enables network
of SDTN (Software Defined Transport Network) that enables virtualization in the transport layer, which provides secure
virtualization and programmability of the transport layer. shared use and programmability at the same time to multiple
users.
Virtualization and programmability is the major require-
ments for future network operation. From the user’s point This paper is organized as follows. Next section explains
of view, they can be able to achieve much flexibility, high the architecture of SDTN. In section III, we illustrate the
performance and resiliency if they can also program the design of the experimental network. Section IV discuses about
transport layer of the inter-cloud network. For example, they what we confirmed through the experiments. Finally, section
may lack of bandwidth in case they newly generate a flow or V concludes the paper.
978-0-7695-4761-9/12 $26.00 © 2012 IEEE 661
DOI 10.1109/SNPD.2012.134

2. II. T HE S OFTWARE D EFINED T RANSPORT N ETWORK Virtual Network (VN) #1 Virtual Network (VN) #2
A. SDTN Architecture VN Topology
SDTN architecture is designed to enable network virtualiza-
tion in the transport layer, that provides secure shared use and (4) Setup Paths
programmability the same time to multiple users. Key concept SDTN
(Allocated
is that we provide multiple SDTNs upon a shared multi- Resources)
layered network infrastructure for users. Note that “users”
SDTN Optical Paths
could be the cloud tenants, cloud service providers and other Controller
operators of network services. (3) Allocate resources
SDTN is made of set of network resources such as links,
wavelengths, unit of bandwidth and switching capabilities.
Each unit of resources is assigned permission to users. Users (2) Configure VN#1 VN#2
Shared Private
permission to VNs Dedicated Dedicated
are allowed to setup optical and packet transport paths mak-
Resource Link
ing use of the resources that are assigned permission to Router
themselves. This ensures the portion of the network to be ᾉ
independently controlled without any contention.
OXC
The key component of the architecture is the Physical Net- PN Manager
work Manager (PN Manager), which is the unified controller (1) Collect Resource Info L2 switch,
(OSPF-TE/LLDP) Physical network(PN) Router
of the optical network. It provides functions for the users
to invoke network control such as resource allocation and
Fig. 1. Construction of VNTs using the resources allocated from the PN
path setup in order to program their own software of network Manager.
topology designing (Fig. 1). PN Manager provides API [4] for
SDTN operators to develop a software to control their SDTN.
Network providers are able to optimize the operation of their
network infrastructure. For example, optimization of resource layer-2 link will be provided by connecting a pair of layer-2
allocation to each slice according to the traffic demands will switches by an optical path through layer-1 nodes using the
provide statistical multiplexing effect. Furthermore, sharing layer-1 resources (e.g. GMPLS TE-links). Then, those layer-2
redundant resources prepared for forecasted future demand and links can be seen as resources to setup a path in the layer-2,
detour routes in case of failures will provide high efficiency by which the layer-3 IP routers can be connected in order
of capital expenditure. Since the SDTN is logically formed to form an IP link. In this manner, VNT in a certain layer
by set of circuits that can be provisioned automatically, it also can be provided dynamically and recursively. SDTNs are
enables fast launch of new services by making use of available provided to the users as the VNTs at the desired layers.
network resources. It may provide survivability of services in Consequently, layer-2 and 3 SDTNs are provided by uti-
case of disaster, by letting the slices to share a small portion lizing network resources of layer-1 and 2. Resources used
of the remained part of the network. to setup layer-1 optical paths are routers, OXCs, fibers and
On the other hand, SDTN benefits the users in terms of the wavelengths. They are handled in a unit called TE-link which
programmability of the transport network. As mentioned in defined in the GMPLS technology. We can describe and
the previous section, users can optimize the transport layer as utilize the resource to setup optical paths because proper-
well as the flow routing layer. Cloud operators can be able to ties of TE-link provides sufficient information of the link
program the total system including the computing resources, such as connected node address, link address, maximum and
flow routing, underlying circuits, and the amount of allocated minimum reservable bandwidth, switching capability (fiber,
network resource to configure the circuits. lambda, TDM and packet), SRLG, and so on. Information of
B. Recursive VNT construction upon multi-layer network in- the existing resources are automatically collected by listening
frastructure to OSPF-TE[6] advertisement in GMPLS.
For the physical network infrastructure, we assume a multi- Resources used to setup layer-2 paths can be handled by
layer network which is consisted of layer-1, 2, 3 nodes such L2SC TE-links that are also defined in GMPLS. However, as
as optical cross-connects (OXCs), L2 switches and IP routers. L2SC is not actually popular in the market, we can make use of
This can be prepared with ordinary products that are already ethernet related technologies such as LLDP (IEEE 802.1AB).
available in the market. To be precise, there are no exact technologies to be named
In each layer, resources are defined in order to as layer-2 path. What is needed here is actually a technology
setup a path. Here we incorporate the notion of to setup a packet transport path to slice the huge bandwidth
V irtualN etworkT opology(V N T )[5]. When a pair of provided by the layer-1 path that is too much to offer to users.
nodes in a certain layer is connected with a path in the lower Here we can employ MPLS-TP LSPs, or S-VLANs defined in
layer, it will form a link in the upper layer. For example, PBB (Provider Backbone Bridge) configured with rate limits.
662

3. IP Link Allocated exclusively
Allocation to each SDTN #A #B #C
Layer-3 Return
Resource VN Operator
Path setup
Path Release Obtain/Release
Equivalent Path
PN Operator
#A #A #B #B #C
L2 path between Permission Dedi- Shared Dedi- Dedi-
L3 Router cated cated cated
Layer-2
VN Operator
Assignment of Permission
PN Operator Resource detection Initially permitted only to PN
(OSPF-TE/LLDP) administrator
L1 path between L2 switches
Layer-1 Fig. 3. Resource access control model.
VN Operator
L1 path between L2 switches
PN Operator
to design the SDTN at that level. Therefore, abstraction of
Fig. 2. Multi-layer network resource state machine. network resources may provide much usefulness to the users.
We assume following three types of abstraction: type-T , a
topology which contains links and nodes, type-P , a set of
C. Multi-layer resource state machine point-to-point paths, and type-S, a virtual switch.
For each unit of resource, the administrator of the physical In type-T , users are provided with links and nodes in order
network will apply permission for SDTNs to obtain them. to setup transport paths by their own. Users are provided a
Using the obtained resources, SDTN operators are allowed to large range of freedom to control the network, such as de-
setup paths in order to form their own VNT. Fig.2 shows the signing multi-layer topology optimization or capacity planning
state machine we have designed for the multi-layer resource according to the traffic demands, and provisioning protection
management model. Users are permitted to obtain layer-1 paths. This type can be seen as an abstraction at the most
resources. Using the layer-1 resources, users can setup layer-1 lower level.
paths between layer-2 or 3 node pairs in order to form links In type-P , users are provided with a set of point-to-point
at layer-2 or 3. In addition, resources can also be assigned to paths. Users only request the paths that connect the desired
the administrator of the total physical network infrastructure, endpoint in order to connect the nodes owned by the users.
which we call the PN (Physical Network) operator. PN oper- Users do not have the level of controllability as much as type-
ator can setup layer-1 paths to produce layer-2 resources, and T , but still it is their work to design the topology formed by
then assign permission to users. By this, users are also able to the provided paths and their nodes.
start from obtaining layer-2 resources in order to form layer-3 In type-S, the provided SDTN is seen as a single switch.
VNT by connecting IP routers by layer-2 paths. Users are provided with connection points, as if they are
Resources are permitted as either dedicated or shared. provided with several ports of a big switch. Users only need to
Shared resources can be noticed by multiple VNs, but it will connect their equipments to those ports, and the packets will
be allocated to only one of that VNs. Sharing the unallocated be forwarded to any of the points they have connected. This
resources enables capital cost reduction of the physical net- type can be seen as an abstraction at the most higher level.
work infrastructure, by sharing the redundant resource that
III. NATION - WIDE EXPERIMENTAL NETWORK
should have been prepared for each of the network service if
no virtualization is adopted. Fig.3 shows the resource access As shown in Fig.4, we have implemented a network in-
control model. frastructure for experiments, upon a national R&E network
Balance of the amount of resources allocated to each virtual in Japan, called JGN-X[7]. Through June 2011 to February
network can be modified flexibly by changing the permission 2012, we have connected four OXCs, ten Layer-2 switches,
of each resource. This enables efficient utilization of the and six IP routers upon JGN-X. Scale of the network in-
resources in accordance to the change in traffic demands. frastructure changed at each experiment event. At most the
number of nodes was 14. Network spanned over the nation,
D. Resource abstraction and variety of controllability level from Hokkaido to Okinawa, which are the north and south
Here we discuss on abstracting the network resources. end of Japan. Some of the links had 10 Gbps capacity, and
Previously we explained that users form SDTN for them by others had 1 Gbps.
themselves, utilizing the resources obtained at the granularity We have implemented an SDTN controller software with
of links. However, we should be aware that not all of the GUI that invoke the PN Manager API in order to let SDTN
users of the network require controllability at that level. Some users to obtain resources and setup paths.
of them don’t need to, some of them don’t want to, and For some of the users, layer-1 resources were directly
some of them are not the network experts skilled enough allocated. Those users formed IP links by connecting IP router
663

4. 2) Dynamic resource allocation: In the experiment event in
Sapporo
February 2012, we have provided layer-2 SDTNs to four TV
broadcasting studio groups. As bandwidth capacity of most of
Koganei the links was 1 Gbps, we sliced the network to provide SDTNs
Otemachi with limited capacity of 150 Mbps each. As the topologies of
Fukuoka SDTNs were different according to the required access point
among users, reserved and residual capacity at each physical
links were different. Residual capacity was maintained as a
bandwidth pool that can be allocated dynamically according
to user’s requests.
Two of the broadcasting studios turned out to require larger
Okinawa Osaka Musashino
amount of bandwidth capacity for their video transmission. In
one case, they needed to simultaneously transmit video file
IP Router Layer-2 Switch OXC for remote TV program editorial and live streaming for news
program. Total bandwidth usage exceeded the default alloca-
Fig. 4. Experimental network infrastructure implemented upon JGN-X. tion of 150 Mbps, so we additional capacity was allocated to
them to enhance the limit to be 200 Mbps. In another case,
a broadcasting studio desired to try a new video encoder that
pairs with GMPLS optical paths. There was another case that consumes bandwidth of 150 Mbps. Also in this case, we added
the PN operator connected layer-2 Ethernet switches with allocation to let it enhance to 200 Mbps. These operations of
optical paths in order to produce layer-2 resources. These resource allocation was also done during the time when other
resources were divided by setting up point-to-point S-VLANs SDTN users were transmitting commercial video stream.
with upper rate limit. SDTNs consisted of set of S-VLANs 3) Abstraction variety: Through the experiment, we were
were provided to users. Users setup C-VLANs between the able to test the usage of SDTNs with all three variations of
desired access points in order to transmit their data flows. abstraction level which mentioned above in section II.
Although we haven’t completed the evaluation from the SDTN for a research project that tested their proposal of
performance point of view, we report that time needed to high-efficiency layer-4 protocol was provided in the manner
setup a single optical path was about 15 seconds, and that of type-T , a topology which contains links and nodes, We
of a single point-to-point S-VLAN connectivity was around also provided measurement functions that the user were able
10-15 seconds. Note that these results may differ according to to check the precise performance in terms of data rate,
conditions. These are expected to be shortened by additional jitter at multiple measurement point implemented inside the
tuning efforts. network. By analysis of the performance degradation point,
they were able to optimize the transmission path. As a result,
IV. R ESULTS
this user was successful in achieving their highest record of
Through the network operation in the experiment which was performance. This experiment can be seen as a successful case
close to practical use, we successfully confirmed the feasibility that the high level of controllability of SDTN had provided
and the benefits provided by our control architecture. benefit to the user.
A. Multiple SDTN operation Another experiment that we provided a SDTN for users to
demonstrate their OpenFlow enabled equipments. As the user
Totally we had provided 11 SDTNs to users including
side nodes were capable of controlling the flow route with
experiment project of new generation network technologies,
OpenFlow technology, they only needed a path to connect their
demonstrations for international conference, and live video
nodes. This experiment can be seen as a use case of type-P ,
transmission for commercial TV program broadcasting. At
a set of point-to-point paths. In addition, we have successfully
most, five SDTNs were operated simultaneously.
1) Independent control of multiple SDTNs: All of the users tested path switchover in the transport layer. As the transport
of 11 SDTNs were able to completely carry out their event path was provided by Ether-over-MPLS circuit, the switchover
of such as experiment, demonstration, and broadcasting. This did not cause any packet losses, and we confirmed the isolated
means that, we confirmed that user traffic was successfully control in independent layers.
isolated in terms that no user experienced any trouble caused Finally, SDTNs provided to most of the broadcasting stu-
by network control or data traffic of other users that share the dios, except the ones that operated the topology change
network infrastructure. Indeed, two SDTNs used by broad- described above, was a case of resource abstraction type-S,
casting studios had changed the topology of their SDTN in which the network can be seen as a virtual switch. Most of
advance of a planned construction work that was known to the broadcasting studios do not care for the inner topology
force outage of the connection at a certain physical link. of the network. They only desire to connect the camera crew
Even in this case, network control to change topology had sites and editorial facilities, and broadcasting stations to the
carried out while broadcasting studios using other SDTNs access points of the network. In this experiment, topology of
were transmitting their commercial video stream. the SDTN was designed and operated by physical network
664

5. operator. However, ideally the network should automatically
design and setup an SDTN with the optimal topology with
optimal bandwidth capacity according to the connectivity
requirements submitted from the users. This case had implied
many future issues for us of the value-adding functions that
the transport network can provide.
V. C ONCLUSION
SDTN is a slice of a physical network that can be controlled
independently by the user of it. As mentioned in this paper,
we believe there should be many variety of how the SDTN
is provided to the users, in terms of abstraction level and
controllability level. The way to provide the SDTN should be
different according to the user’s requirements. For example,
advanced users will be able to totally program the network at
each layer of the network, by making use of SDTN functions
in addition to the SDN functions at the flow routing layer.
On the other hand, for users that do not care about the inner
networking technologies, it may be beneficial for them if the
network can offer useful functions to users such as automatic
capacity designing and topology optimization. The experiment
results shown in this paper are valuable findings derived from
practical use cases that suggests us of the future research
topics. Further discussions are expected to be focused on
defining the total architecture and the interfaces between user
systems and the SDTN controllers such as our PN Manager.
ACKNOWLEDGEMENTS
The authors would like to thank Dr. Kazumasa Kobayashi,
Yoshihiko Kanaumi and all of the JGN-X related researchers
and engineers in NICT for strongly supporting us on the
experiments.
R EFERENCES
[1] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson,
J. Rexford, S. Shenker, and J. Turner, “OpenFlow: enabling innovation
in campus networks,” SIGCOMM Comput. Commun. Rev., vol. 38, no. 2,
pp. 69–74, Mar. 2008.
[2] S. Das, G. Parulkar, N. McKeown, P. Singh, D. Getachew, and L. Ong,
“Packet and circuit network convergence with OpenFlow,” in Optical
Fiber Communication Conference. Optical Society of America, 2010,
p. OTuG1.
[3] S. Azodolmolky, R. Nejabati, E. Escalona, R. Jayakumar, N. Efstathiou,
and D. Simeonidou, “Integrated OpenFlow–GMPLS control plane: an
overlay model for software defined packet over optical networks,” Opt.
Express, vol. 19, no. 26, pp. B421–B428, Dec 2011.
[4] A. Masuda, A. Isogai, T. Miyamura, K. Shiomoto, and A. Hiramatsu,
“Application-defined control of virtual networks over IP-optical net-
works,” in CNSM. IEEE, 2011, pp. 1–6.
[5] K. Shiomoto, D. Papadimitriou, J. L. Roux, M. Vigoureux, and D. Brun-
gard, “Requirements for GMPLS-Based Multi-Region and Multi-Layer
Networks (MRN/MLN),” RFC 5212 (Informational), Internet Engineering
Task Force, Jul. 2008.
[6] K. Kompella and Y. Rekhter, “OSPF Extensions in Support of Gener-
alized Multi-Protocol Label Switching (GMPLS),” RFC 4203 (Proposed
Standard), Internet Engineering Task Force, Oct. 2005.
[7] “New generation network testbed JGN-X,” http://www.jgn.nict.go.jp/.
665