1. D E C E M B E R 2 0 1 2 / V O L . 3 / N 0 . 6
evolution
NETWORK
BUILDING TH E INFRASTRUCTURE TO ENABLE THE CHANGING FACE OF IT
LET THE
40 GIGABIT
ETHERNET
MIGRATION
BEGIN
Early adopters
of 40 GbE share
successes …
and pitfalls PLUS:
ULTRA-FAST
ETHERNET
AND THE
MONITORING
CHALLENGE
400 GIGABIT
ETHERNET
IN THE
ENTERPRISE …
REALLY?

2. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 2
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
EDITOR’S DESK
A FEW YEARS ago, a group of network
engineers laughed in my face when
I brought up an article I was writ-
ing about 10 GbE migration. All of
them were still using 1 GbE, and
only a few were even testing 10 GbE
switches. The transition, they said,
would be long and slow. It was the
classic gap between a user’s reality
and a journalist’s research.
So it may seem even stranger
this month that we have produced
an entire issue of Network Evolution
about 40 and 100 GbE migration
(we even have a piece pondering
the future of 400 and Terabit GbE).
After all, we are only at the very
beginning of widespread 10 GbE
implementation.
But things are different now than
they were a few years ago. The need
for capacity and speed is explod-
ing as network teams build private
and hybrid clouds and deliver video
throughout their campuses and
branch offices. So even if they don’t
need 40 GbE today, they’ll need to
build networks that can be transi-
tioned in the future. And that won’t
be easy, as Shamus McGillicuddy
reports in his feature “40 Gigabit
Ethernet: The migration begins.”
The transition will mean testing
new equipment as vendors race
to release 40 and 100 GbE-ready
switching. It will also require consid-
eration of new cabling and optics.
But the challenges don’t stop
there. Existing monitoring, trouble-
shooting and management tools
aren’t ready to scale to new ultra-
fast Ethernet networks, reports
contributor Jim Frey in his feature
“The 100 Gigabit Ethernet manage-
ment challenge.” The same problem
exists in scaling firewalls and intru-
sion detection devices for 40 GbE
networks. While vendors are work-
ing to upgrade monitoring and secu-
rity appliances, engineers testing or
newly implementing 40 GbE find
themselves patching the problem
by load balancing traffic across mul-
tiple 10 GbE-ready monitoring and
security devices.
The purpose of this issue isn’t to
urge readers to invest in 40 and 100
GbE. Instead, it’s to be sure that you
are aware of the challenges that lie
ahead so that when you invest in
your current 10 GbE infrastructure,
you do so with future migration to
40 and 100 GbE in mind. n
Rivka Gewirtz Little
Executive Editor
It’s Not Too Early To Worry
About 40 and 100 GbE Migration

3. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 3
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
COVER STORY
THE NEED FOR more speed and capac-
ity never goes away. Most enter-
prises are still rolling out 10 Gigabit
Ethernet links in their networks,
but already network architects and
engineers are preparing for the
future because data traffic never
stops growing. Early adopters are
testing and deploying the first
generation of 40 Gigabit Ethernet
(GbE) switches and routers to get
a step ahead of this continuous
onslaught of traffic.
“We are seeing incidental peaks
of traffic nearing 10 Gbps,” said
Jeroen van Ingen, a network engi-
neer at the University of Twente in
the Dutch city of Enschede. “Given
the traffic growth over the years, we
expect to need more capacity within
12 to 24 months. That’s why we
decided that new core equipment
should support 40 GbE.”
University of Twente’s van Ingen
is not alone in predicting the need
for 40 GbE and beyond. Research
from the Dell’Oro Group forecasts
the overall Layer 2 and Layer 3 Eth-
ernet switch market will reach $25
billion in 2016, with 40 and 100
GbE technology approaching $3 bil-
lion in sales. Meanwhile, Infonetics
Research shows sales of 40 GbE
technology sales grew 50% in both
Q1 and Q2 of this year. The expan-
sion is largely driven by bigger data
centers and the move to the cloud,
according to researchers.
The university recently upgraded
the core of the residential portion
40 GIGABIT
ETHERNET: THE
MIGRATION BEGINS
Expertsandearlyadoptersof40GigabitEthernet
sharesuccessesandpitfalls. BY SHAMUS MCGILLICUDDY
InfoneticsResearch
showssalesof40GbE
technologysalesgrew
50%inbothQ1and
Q2ofthisyear.

4. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 4
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
MIGRATING TO 40 GBE
of its campus network with two
of Cisco Systems’ new Catalyst
6500-E switches. Today, those core
switches are interconnected with
multiple 10 GbE links, but traffic is
spiking upward and the university
will need 40 GbE interconnections
soon. Van Ingen is testing a 40 GbE
interconnect between the two Cata-
lyst 6500s and plans to put two 40
GbE interconnections into produc-
tion soon.
The demand for this bandwidth
is driven by three factors: The uni-
versity is increasing its use of cloud
services and cloud storage, video
traffic to the student dormitories is
growing, and various open source
projects that are mirrored on the
university’s campus are extremely
popular.
“If we don’t keep up with
upgrades, the network experience
will get worse and worse,” van
Ingen said. “If the links actually get
saturated before we upgrade, the
increased packet loss will lead to
error messages in their applications,
actions that can’t be completed and
lost productivity in general.”
Verisign Inc., the Reston, Va.-
based Internet infrastructure pro-
vider, which manages two of the
Internet’s 13 root servers, will likely
transition to 40 GbE within two
years, according to Mike Gibbs,
Verisign’s network architect. Gibbs
and his team beta-tested the 40
GbE capabilities on Cisco’s Catalyst
6500 Supervisor Engine 2T. “The
first and most obvious place that I
suspect a lot of people will initially
install 40 gigabit is in their QA
[quality assurance] and lab environ-
ments,” he said. “It’s more conve-
nient to have testing tools and load
generation tools and interconnects
that are single links of large capac-
ity, as opposed to many aggregated
links.”
Verisign will soon have to replace
aggregated 10 GbE links with 40
GbE links, particularly to network
service devices like load balancers
and firewalls. Today, GbE-capable
load balancers and firewall appli-
ances are scarce on the market,
but the migration will happen. “As
capacity for our infrastructure has
to grow, as DNS grows, as our
denial-of-service product grows,
we’re going to need greater capacity
between devices,” Gibbs said.
“
Ifthelinksactually
getsaturatedbefore
weupgrade,the
increasedpacket
losswillleadtoerror
messagesintheir
applications.”
—JEROEN VAN INGEN,
networkengi­
neer,UniversityofTwente

5. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 5
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
MIGRATING TO 40 GBE
40 GIGABIT ETHERNET:
WHEN LINK AGGREGATION
IS NO LONGER SUFFICIENT
Link aggregation has been an essen-
tial tool for network engineers for
years, whether aggregating multiple
Gigabit Ethernet links or 10 GbE
links. There always comes a time
when aggregation is no longer prac-
tical, however, and a migration to
the next generation of Ethernet is
necessary. Enterprises whose busi-
nesses are based on network per-
formance and reliability will reach a
point where link aggregation breaks
down, and they will have no choice
but to upgrade to 40 GbE.
“There’s a limit to how many
single 10-gigabit links we’ll be able
to wrap together into an aggregate
link,” Gibbs said. “With link aggre-
gation, you have a hashing mecha-
nism to send traffic across them as
equally as possible. Not every ven-
dor does that equally well. Because
of that, when you start having ven-
dors interoperate with each other,
you can’t guarantee an equal spread
across all those links. So, a single
40-gigabit link ends up being much
more preferred.”
Link aggregation is based on
the IEEE 802.1ax standard, but the
hashing algorithms each vendor
uses to load balance traffic across
multiple links is unique, according to
Dhritiman Dasgupta, senior direc-
tor of product marketing at Juniper
Networks.
“With 40 Gigabit Ethernet you
get guaranteed and dedicated
40-gigabit bandwidth, so hashing
algorithms don’t play a part any-
more,” Dasgupta said. “[When]
predictability is their business,
[networking pros] are moving to 40
gigabit. They can’t take a chance on
hashing algorithms playing a part in
how long transactions take.”
TRANSITIONING TO
40 GIGABIT ETHERNET
As is the case with every next-gen-
eration Ethernet standard, the tran-
sition from 10 GbE to 40 GbE will
not be disruptive, Dasgupta said.
From a protocol standpoint, TCP,
UDP and Layer 2 and Layer 3 proto-
cols remain unchanged.
“The things that need to change
are the cabling and the optics at
both ends,” he said.
Specifically, enterprises will
need to deploy new fiber—OM3 or
OM4—for link lengths of up to 100
meters and duplex single-mode
fiber (SMF) for links up to 10 kilo-
meters. Of course, new transceivers
will also become necessary, wheth-
er SR4 or LR4.
Many enterprises are already
thinking about the path to 40 GbE
as they install 10 GbE today, and are
rolling out ribbon fiber that can sup-
port the higher bandwidth.
While many enterprises install
40 GbE-ready infrastructure dur-

6. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 6
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
MIGRATING TO 40 GBE
ing a 10 GbE upgrade to save time
and money, those savings will only
happen if the networking team
maintains the cabling plant prop-
erly. Signal loss tolerance for a 40
GbE network is extremely low in
comparison to Gigabit Ethernet and
10 GbE, and it can be challenging to
achieve an acceptable rate of loss.
“People will connect fiber up and
never have cleaned it and never
inspected it,” said Ed Gastle, prod-
uct line manager for JDS Uniphase
Corp. (JDSU), a Milpitas, Calif.-
based optical test and measure-
ment company. “The [40 GbE] links
will come up and they will work, but
you’ll be taking errors because your
data is getting corrupted.”
Poor fiber maintenance became
an issue during Interop Las Vegas
in 2011, he added. JDSU provided
the testing and measurement for
InteropNet that year. That year, the
InteropNet network upgraded from
Gigabit Ethernet to 10 GbE.
“They had a whole bunch of fiber
out, but they hadn’t treated it very
well,” he said. “They hadn’t been
cleaning and inspecting it. And now
they were trying to run 10 giga-
bit. Well, guess what? It doesn’t
work. Now you have to go through
a recovery effort to get your end
faces to good enough loss [levels]
to support 10 gigabit. We were able
to get a few of them back by aggres-
sively cleaning them, but it takes a
lot of time. And if you have to re-
terminate [the fiber], it takes a lot of
money.”
The migration to 40 GbE is even
more sensitive to poorly maintained
fiber. Enterprises need to inspect
and clean the end faces on those
fiber links because the next genera-
tion of Ethernet will have a much
lower loss budget.
“We hear more and more that as
these speeds go up, systems are not
coming up at all or are coming up
and having problems,” Gastle said.
“Then we start to troubleshoot and
we find that the problem is caused
by a poorly maintained end face.
They’ve been connected up so many
times that dirt and debris has been
ground into it and you … have to re-
terminate that fiber.”
Connecting dirty fiber to active
equipment can also spread damage,
he said. Anything short of pristine
fiber can pose a risk of damage to
the optics, for instance.
SECURING AND MONITORING
40 GIGABIT ETHERNET LINKS
As network engineers deploy 40
GbE switches and routers, eventual-
ly they’ll need Layer 4-7 appliances,
such as application delivery control-
lers, firewalls or WAN optimization
controllers, that can handle 40 GbE
traffic. They also need evaluate how
these devices monitor the network,
whether it is for security or perfor-
mance.

7. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 7
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
MIGRATING TO 40 GBE
“It’s not the link speed that you
have to be concerned about,” said
Verisign’s Gibbs, who is evaluating
40 Gbps firewalls. “It’s whether or
not the systems can actually do the
processing of the traffic between
them, especially firewalls, where
every packet is going to be at least
briefly reviewed at a minimum of
Layer 3, if not Layer 4, all the way
up to Layer 7. That takes a huge
amount of computational power.”
Some organizations will want line
rate visibility right at the 40 GbE
link. The University of Texas at Aus-
tin has deployed high-availability
(HA) pairs of Cisco Nexus 7000s
switches in two data centers, with
40 GbE interlinks between the HA
pairs and between the data centers,
according to Brent Boggan, regional
sales manager at Ixia, a network
monitoring vendor based in Cala-
basas, Calif., which has helped the
school instrument the network for
visibility.
Boggan said the university has
40 GbE links in each of two Austin
data centers. Ixia put a physical,
passive optical 40 Gbps tap on each
of those connections. The monitor-
ing ports coming out of those taps
then flow into a network monitoring
switch, the Anue Net Tool Optimiz-
er (NTO) 5288, which can process
up to 40 Gbps of data at full duplex.
The NTO 5288 then filters, rep-
licates and load balances that 40
Gbps flow across multiple 10 Gbps
tools, including multiple intrusion
detection appliances like Source-
Fire’s Real-time Network Aware-
ness tool.
Still, early adopters will have to
deal with limitations of existing
technology. Many organizations
will rely on NetFlow and SNMP for
network visibility because line-rate
packet analysis will remain a chal-
lenge.
The University of Twente will use
open source SNMP counters and
NetFlow analyzers to maintain vis-
ibility on its 40 GbE links, van Ingen
said. For packet visibility, he per-
forms full packet captures on the 10
GbE links that are aggregated by the
Catalyst 6500.
“If we really need to do full packet
analysis on a 40 Gbps link, we’ll
probably configure a high-end serv-
er with a multiport 10 GbE card and
see how far that gets us,” he said.
Verisign relies on NetFlow and
SNMP to monitor its 10 GbE net-
work today and will probably con-
tinue that approach as it migrates to
Manyorganizations
willrelyonNetFlow
andSNMPfornetwork
visibilitybecauseline-
ratepacketanalysis
willremainachal­
lenge.

8. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 8
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
MIGRATING TO 40 GBE
40 GbE, Gibbs said.
But the process is imperfect,
Gibbs acknowledged. Some events
will always slip through. These pro-
tocols won’t always reveal bursts of
traffic going up and down. On the
security side, something malicious
can slip through very easily.
“You’re looking at patterns to
figure things out instead of looking
for something inside of a packet,”
he said. “It also means any [probing
attack] with a single or couple of
packets you might miss because it’s
outside of your sampling point. It’s
a known risk everyone has to deal
with, but it is the side effect of more
bandwidth and more speed.”
Mitigating that risk involves a
multi-tiered solution for monitoring
and security, Gibbs said. Network
engineers need to continue watch-
ing what’s going on in NetFlow and
other sampling technologies, but
then have another tier of defense
with inspection devices down-
stream in the network.
“Don’t just look at these [core]
routers,” he said. “If you miss it at
the core routers, you should catch
it in a net further down in the
chain.”n

9. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 9
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
DO 100 GBE MANAGEMENT AND MONITORING TOOLS EXIST?
IT MAY BE early days for 100 Gigabit
Ethernet, but many service provid-
ers have deployed the technology,
and now a small but growing num-
ber of enterprises are proceeding
with it as well. As soon as 100 GbE
is in live production, network and
security pros will need to manage
and monitor these networks. Herein
lies the challenge—some manage-
ment tools are ready for this next
level of ultra-speed, but some clear-
ly are not.
Of 15 network management and
monitoring vendors contacted for
recent research, very few had con-
crete products in place or even a
near-term roadmap. Yet most are
aware of the emerging need for 100
GbE monitoring and management
and are exploring how to accommo-
date customers.
The difficulty of monitoring 100
GbE networks varies depending on
the type of network management
tool or strategy. Some network
management strategies lend them-
selves more easily to adapting to
ultra-high speed networks, while
others must be heavily transformed.
The following is an assessment of
existing network management strat-
egies and whether they’re ready to
handle 100 GbE networks.
n Fault/availability management:
Since 100 GbE is delivered via net-
work devices, the first objective is to
make sure that tools for the network
operations center (NOC) are ready
to discover and recognize these
devices. That will mean recogniz-
ing the new interfaces they employ,
their topology placement, and their
DO 100 GBE
MANAGEMENT
AND MONITORING
TOOLS EXIST?
Vendorsareracingtorelease100GbEswitching,but
networkmonitoringtoolsmaynotbeabletokeeppace.
BY JIM FREY

10. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 10
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
DO 100 GBE MANAGEMENT AND MONITORING TOOLS EXIST?
ongoing health. These basic capa-
bilities are already present in most
fault/availability monitoring plat-
forms today, particularly among
products that have been deployed
in service provider environments
(i.e. ISPs, mobile operators, hosting
providers) where 100G is already in
place today. These tools may need
to be scaled down for an enterprise
network environment.
n Statistics-based performance
management: Things get a bit stick-
ier when you look at performance
management platforms that gather
statistics by regular polling of net-
work devices. While many metrics
will not be directly affected, traffic
polling counters may be insufficient
for tracking total volume of activity
unless those counters are harvested
on a much more rapid basis than
polling engines are typically con-
figured to support. This will not be
a problem in the early days of 100
GbE, as total traffic volumes will ini-
tially be far less than total capacity,
but eventually these tools will need
to scale.
n Flow-based security and perfor-
mance management: Tools that use
technologies such as NetFlow or
IPv6 to gather data regarding com-
position volume of traffic should
be essentially agnostic to 100 GbE,
and will be stressed only if traffic
volumes rise to a level that results in
flow record volumes beyond a tool’s
capacity to collect and analyze.
Again, that’s not likely to occur in
the near term, as flow-based tools
designed for today’s enterprise or
service provider environments are
already equipped to handle very
high volumes of flow record data.
n Packet-based security and perfor-
mance management: Here’s where
we run into a major problem. Packet
inspection tools are the heavy lift-
ers of the management tools sector,
and they face the greatest technical
challenge when trying to accom-
modate increased network speeds.
There are a few essential parts to
this challenge—getting to the pack-
ets, distributing the packets for
analysis, and analyzing the packets.
There are no packet analysis tools
available today that can directly
support 40 GbE, let alone 100 GbE,
at full line rates. Many have certified
full 10 GbE support and a few offer
40 GbE interfaces, but none have
addressed 100 GbE at this time.
SCALING PACKET INSPECTION TOOLS
TO MEET 100 GBE NETWORKS
The only viable approach to 100
GbE network management today is
adapt existing tools that are today
only rated to 10 GbE. This requires a
two-part process.
The first step is gathering packets
at 100 GbE. There are currently two

11. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 11
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
DO 100 GBE MANAGEMENT AND MONITORING TOOLS EXIST?
techniques primarily used for gath-
ering packets for analysis – TAP and
SPAN (a.k.a. port mirroring.) SPAN
is a function of the network device
itself, but taps must be upgraded
to support 100 GbE. The good
news here is that taps are already
available for 100 GbE. NetOptics
released the 100G Flex Tap back in
September, and the company claims
deployments are already underway
in mobile telco, ISP and hosting
provider shops. ONPATH (which is
being acquired by NetScout) is also
shipping 100G-rated, Layer 1 access
devices in the form of Optical Blade
for its 3900 switch family, mostly
to testing environments. Other L1
vendors have this on their roadmaps,
so expect more options coming in
2013.
The second element of deep
packet inspection is distributing
packets at 100G. This is the job
of network monitoring switches
(a.k.a. network packet brokers),
which adapt traffic from 100 GbE
interfaces over to 10 GbE interfaces
to manage the torrent of packets
via load balancing, slicing, filtering,
etc. The idea is to prevent analy-
sis tools from being overwhelmed.
While 100 GbE is on the roadmap
of some vendors (some say they’ll
have product in early 2013), Endace
was the first to productize and ship
a solution. The EndaceAccess 100
provides L1 adaptation and load
balancing so that 10G-rated packet
analysis tools can be deployed to
monitor 100 GbE links. The solution
is deployed and being used in pro-
duction by Endace customers today.
Other vendors are working on 100
GbE network monitoring switching
as well. For example, on Novem-
ber 1, 2012, Gigamon announced
it would be demonstrating 100G
aggregation and connections later
in the month. Expect to see others
making announcements late this
year and early next.
For many, 100 GbE is far off in
the future, so there is no immediate
need to worry about how to man-
age it. But for those at this bleeding
edge, core fault/availability tools
are ready now, and recent introduc-
tions by companies such as NetOp-
tics, ONPATH, and Endace, place
packet-based security and perfor-
mance monitoring within reach. n
MORE ON NETWORK
MONITORING TOOLS
n
Deep packet inspection tools:
Proxy vs. stream-based
n
Cloud monitoring tools:
Using Wireshark in the cloud
n
Network diagnostics that
see through virtualization
n
Network performance
testing for a 40 GbE upgrade

12. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 12
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
WILL WE EVER NEED 400 GIGABIT ETHERNET ENTERPRISE NETWORKS?
THOUGH 10 GIGABIT ETHERNET (GbE) is
just at the beginning of widespread
uptake, there’s already plenty of talk
in the market about the need for 40
and 100 GbE in the enterprise. In
the middle of all this, do we need to
consider the potential for 400 Giga-
bit Ethernet in the WAN and LAN?
The answer is, plainly, yes.
The IEEE has just formed a work-
ing group to determine whether
there will be need for 400 GbE
or Terabit Ethernet development.
But we are already some looming
use cases—even in the enterprise
LAN—that will drive the need for
400 Gigabit Ethernet.
In general, high-speed standards
like 400 GbE or even Terabit Eth-
ernet are applied deeper in the
network where traffic has been
aggregated. For now, computer and
storage interface standards define
transfer rates well below 100 GbE,
so that won’t cause demand. But
since the edge of the network is
getting faster, it would be sensible
to assume that this higher edge
traffic would force up speeds in the
metro/core aggregation network as
well.
Modern networks, whether they
span the globe or only the data
center, are built on a hierarchy of
devices, with “deeper” switches or
routers networking the edge devices
together for full connectivity. This
has encouraged the use of succes-
sively faster trunks as port speeds
have increased. The trend makes it
logical to assume that future LAN
switches or WAN switch/routers
will need 400 GbE or faster. More
traffic, more capacity. But traffic
WILL WE EVER
NEED 400 GIGABIT
ETHERNET ENTER-
PRISE NETWORKS?
With40and100GigabitEthernetsonew,whatcould
possiblydrivetheneedfor400GigabitEthernetinthe
enterprise?Hint:It’snotjusttheWAN. BY TOM NOLLE

13. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 13
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
WILL WE EVER NEED 400 GIGABIT ETHERNET ENTERPRISE NETWORKS?
doesn’t impact the WAN or LAN
uniformly—and therefore the needs
may be vastly different in the two
network types.
IN THE WAN, THINK FASTER
ETHERNET FOR OPTICAL SWITCHING
In the WAN, the largest source of
additional traffic in the future net-
work will likely be content. That
content is increasingly cached in a
content delivery networks (CDNs)
or in content farms in each metro
area. This means that while the
growth of access bandwidth is likely
to continue, and this increased edge
capacity will demand greater capac-
ity in the metro area for aggrega-
tion, the majority of content traffic
will stop in the metro cache and
never reach the core at all. In that
case, within a metro there’s less
traffic to aggregate, which means
there’s less pressure on Ethernet
performance.
It’s likely that there will be greater
need for faster Ethernet interfaces
on optical switches and optical
point-to-point paths in metro net-
working. Almost all traffic in a metro
network goes one place—the metro
concentration point, where it’s con-
nected to a cache, a data center or
the core network. Ethernet routing
or switching based on destination
address is hardly useful when all the
traffic is headed in the same direction.
Even when traffic does reach the
core, which would be the case for
business traffic and especially video
conferencing, that traffic may not
drive the kind of changes everyone
expects at the equipment level.
Operators have been looking for
alternatives to adding more lay-
ers of switch/router aggregation,
including the optical transport net-
work (OTN) or the routerless core.
In both of these alternative network
architectures, metro areas are con-
nected to each other in a mesh
rather than to deeper-level electri-
cal devices like routers (hence the
term routerless). Because traffic is
limited to each metro area partner,
it’s less likely this connection would
demand a radical increase in Eth-
ernet speeds. However, even when
traffic growth does drive faster Eth-
ernet interfaces, they’re again more
likely to be used on optical devices
than on Ethernet switches or routers.
MORE ON NEXT-GENERATION
ETHERNET
n
IEEE explores new standards
for ultra-fast Ethernet
n
40 Gigabit Ethernet in the
data center: Migration best
practices
n
Research institute deploys
100 Gigabit Ethernet from
core to closet

14. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 14
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
WILL WE EVER NEED 400 GIGABIT ETHERNET ENTERPRISE NETWORKS?
SURPRISE, SURPRISE: 400 GIGABIT
ETHERNET NEEDED FOR THE LAN
In the LAN, it’s harder to discard the
notion that uniform connectivity is
needed. Cloud computing encour-
ages the creation of dense, highly
connected, data centers. Cloud
applications are often more compo-
nentized and horizontally integrated
than traditional applications, which
makes traditional multi-tiered LAN
switching performance more prob-
lematic. In a cloud data center, even
10 GbE server/storage interfaces
connected in a four- or five-layer
structure might drive switch inter-
face speeds to 400 GbE or more in
the deeper layers.
While intra-data-center con-
nectivity seems to be migrating to
fabric switching, and away from
traditional stacks of LAN switches,
that may not preclude the need for
faster, 400 GbEd. Fabric switches
today either have no internal trunk
connections, or base their connec-
tions on standards other than Eth-
ernet. But when clouds are created
by linking multiple data centers over
fiber, a super-Ethernet connection is
almost inevitable.
The bottom line is that we need
faster Ethernet. Faster Ethernet will
connect cloud data centers and will
support optical metro aggregation
and OTN-based cloud core net-
works. While these missions are
more limited than that of 1 GbE, for
example, they will be of paramount
importance to network design in the
future, and so it is very likely that
the race for faster Ethernet will con-
tinue. n
FasterEthernetwill
connectclouddatacen-
tersandwillsup­
port
opticalmetroaggrega-
tionandOTN-based
cloudcorenetworks.

15. NETWORK EVOLUTION E-ZINE • DECEMBER 2012 15
HOME
EDITOR’S DESK
LET THE 40
GIGABIT ETHERNET
MIGRATION BEGIN
ULTRA-FAST
ETHERNET AND
THE MONITORING
CHALLENGE
400 GIGABIT
ETHERNET IN
THE ENTERPRISE …
REALLY?
ABOUT THE AUTHORS
Networking Evolution is a
SearchNetworking.com e-publication.
Kate Gerwig
Editorial Director
Rivka Gewirtz Little
Executive Editor
Shamus McGillicuddy
Director of News and Features
Kara Gattine
Senior Managing Editor
Linda Koury
Director of Online Design
Scott Kelly
Associate Publisher
skelly@techtarget.com
TechTarget
275 Grove Street, Newton, MA 02466
www.techtarget.com
©2012 TechTarget Inc. No part of this publication
may be transmitted or reproduced in any form or
by any means without written permission from the
publisher. TechTarget reprints are available through
The YGS Group.
AboutTechTarget:TechTargetpublishesmediaforinfor-
mation technology professionals. More than 100 focused
websitesenablequickaccesstoadeepstoreofnews,advice
andanalysisaboutthetechnologies,productsandprocesses
crucialtoyourjob.Ourliveandvirtualeventsgiveyoudirect
access to independent expert commentary and advice. At
IT Knowledge Exchange, our social community, you can get
advice and share solutions with peers and experts.
Shamus McGillicuddy is the director
of news and features for TechTarget
Networking Media. He writes about
networking, security, data centers, net-
work management and other topics for
SearchNetworking.com. He also man-
ages overall news coverage for TechTar-
get’s other networking sites, including
SearchUnifiedCommunications.com,
SearchEnterpriseWAN.com and Search-
CloudProvider.com. Before joining Tech-
Target in 2006, Shamus was an award-
winning reporter for The Patriot Ledger, a
daily newspaper in Massachusetts.
Jim Frey is the managing research direc-
tor at Enterprise Management Associ-
ates. Frey has 24 years of experience
in the computing industry developing,
deploying, managing, and market-
ing software and hardware products,
with the last 18 of those years spent in
network management, straddling both
enterprise and service provider sectors.
Tom Nolle is a strategic egghead—
someone who first wants to know the
truth, no matter what it is, and then
wants to explain it in a way that reaches
everyone who cares to know it. He’s an
analyst in telecommunications, media
and technology, and a former software
architect who now works to blend tech-
nology detail and business reality. He has
more than 30 years of experience as an
author and analyst, covering the most
complicated and important develop-
ments in the industry. Nolle is president
of CIMI Corp., a strategic consulting firm
specializing in telecom and data commu-
nications since 1982. He is the publisher
of Netwatcher, a journal addressing
advanced telecom strategy issues.