The Internet backbone consists of just over 6000 independent networks that exchange traffic in fashions that are not well understood outside of the backbone networking community. We explain how it works, how it has evolved and how it is continuing to evolve today. This is a revised and annotated version of material most recently given as an invited presentation at OFC 2014, the optical fiber conference in San Francisco, in March 2014. To provide higher resolution, I've also uploaded a version w/o annotations, i.e. just the graphics.

1. Why
care?
Peering
and
Transit
are
li3le
understood
and
yet
this
is
the
heart
of
Internet
infrastructure
economics
and
it
is
the
growth
of
Internet
infrastructure
that
ul:mately
funds
most
of
us
here
at
OFC.
I
going
to
describe
how
thousands
of
independent
organiza:ons
compete
but
also
exchange
traffic
and
con:nually
grow
an
ever
more
distributed
Internet
infrastructure.
1

2. There
are
more
than
3
billion
Internet
users
and,
with
the
advent
of
very
low
cost
Android
smart
phones
($24
in
India
in
Feb
2014),
it’s
likely
that
number
will
double
in
just
a
few
years.
There
are
also
tens
of
millions
of
local
networks
(perhaps
more
than
100
million
based
just
on
the
number
of
WiFi
routers
that
have
been
sold).
They
connect
to
the
Internet
through
tens
of
thousands
of
ISPs
which
may
be
classified
as
Access,
Aggrega:on
or
Backbone,
or
as
Local,
Regional,
Na:onal
or
Interna:onal
but,
of
course,
many
ISPs
cross
these
boundaries.
For
today,
I’m
going
to
focus
on
the
6000
or
so
major
ISPs
that
form
today’s
Internet
backbone.
But
to
understand
today’s
complex
environment,
it’s
useful
to
see
how
it
emerged.
2

3. 25
years
ago,
there
was
only
one
backbone.
It
was
run
by
the
Na:onal
Science
Founda:on
for
the
benefit
of
various
researchers
and
government
agencies.
Regional
networks
connected
to
the
NSF
backbone
but,
with
only
one
backbone,
there
was
only
one
source
of
addressing
and
of
ul:mate
rou:ng
decisions.
3

4. As
other,
commercial
networks
grew
up,
they
interconnected
with
the
NSFNET
to
exchange
email
and
data
files.
They
also
found
other
ways
to
exchange
data
among
themselves,
but
s:ll
relied
on
the
NSFnet
as
the
ul:mate
authority
on
addressing
and
rou:ng.
4

5. With
the
advent
of
the
World
Wide
Web
and
the
Mosaic
browser,
Internet
growth
accelerated
and
the
NSF
sought
a
way
to
get
out
of
the
backbone
business.
Part
of
this
required
development
of
a
new
rou:ng
protocol
(BGP,
on
which
more
later)
which
went
on
within
the
IETF
between
1991-­‐1994.
Part
of
this
required
establishing
four
Network
Access
Points
(NAPs)
where
backbone
providers
would
exchange
traffic
des:ned
for
other
backbones.
Of
course,
each
backbone
provider
had
its
own
network
that
enabled
all
connected
users
and
content
providers
to
communicate
with
one
another.
However,
users
were
not
interested
in
communica:ng
just
with
just
those
other
users
connected
to
the
same
backbone
provider.
They
wanted
to
communicate
with
any
user
and
any
content
provider,
regardless
of
backbone
provider.
To
offer
universal
connec:vity,
backbone
providers
interconnected
at
NAPs
(and
elsewhere)
to
exchange
traffic
des:ned
for
each
other’s
users.
It
is
these
interconnec:ons
that
make
the
Internet
the
“network
of
networks”
that
it
is
today.
Finally,
in
April
1995,
the
NSF
stopped
providing
backbone
services
and
the
commercial
Internet
was
born.
5

6. In
order
to
provide
complete
Internet
access,
the
backbone
providers
had
to
exchange
traffic
with
each
other.
What’s
more,
the
NSFNET
backbone
had
facilitated
open
traffic
exchange
at
many
levels,
so
there
were
many
peering
agreements
at
first.
But
the
Internet
was
also
growing
rapidly,
requiring
significant
capital
investments.
At
a
minimum,
investors
wanted
to
see
a
path
to
a
return
on
their
investment.
6

7. And,
with
just
six-­‐seven
full
backbones
networks
in
existence,
the
backbone
ISPs
began
to
realize
they
had
the
makings
of
a
cartel.
7

8. As
a
cartel,
none
of
the
backbone
operators
had
to
provide
free
peering
to
regional,
local
or
other
smaller
networks.
Instead
they
could
sell
them
“Internet
Transit”
service
–
a
service
that
delivers
packets
to
the
rest
of
the
Internet.
Gradually
(and
some:mes
abruptly),
peering
rules
became
quite
exclusive.
To
peer
with
the
backbone,
you
had
to
be
present
at
all
major
NAPs,
you
had
to
have
a
significant
amount
of
traffic
and
that
traffic
had
to
be
roughly
symmetric.
This
had
an
immediate
impact
on
many
Tier
2
operators,
some
of
which
were
growing
more
rapidly
than
the
backbones.
8

9. De-­‐peering
also
impacted
cable
companies,
several
major
content
hos:ng
networks
and
some
large
savvy
content
providers.
These
folks
realized
that,
even
if
they
had
to
buy
“Internet
Transit”
from
a
backbone
provider,
they
could
reduce
what
they
paid
the
backbone
providers
by
exchanging
traffic
among
themselves.
9

10. By
2002,
donut
peering
had
emerged.
The
Tier
2
ISPs,
cable
companies
and
content
providers
had
built
a
ring
around
the
cartel,
largely
rendering
the
original
cartel
irrelevant.
10

11. Indeed,
many
Tier
2
providers
now
had
interna:onal
networks
and
offered
lower
latency
&/or
be3er
pricing.
By
the
early
2000s,
the
Internet
was
substan:ally
more
distributed.
11

12. The
third
wave,
which
started
in
the
early
2000s
and
is
s:ll
evolving
today,
was
the
advent
of
Content
Distribu:on
Networks
(CDNs).
CDNs
may
have
limited,
private
or
no
communica:ons
infrastructure
of
their
own,
instead
they
distribute
content
servers
in
what
is
effec:vely
an
overlay
network.
Akamai
and
Limelight
created
early
CDNs.
Today,
Google,
Amazon
and
Level
3
also
run
content
distribu:on
networks
and
Nejlix
has
begun
deploying
their
own
CDN.
12

13. Typically,
major
CDNs
supply
their
servers
and
remotely
manage
them,
but
local
ISPs
install
them
and
pay
for
electricity
and
rack
space.
This
is
good
business
for
the
local
ISP
as
it
reduces
latency
for
their
customers
and
reduces
the
amount
of
upstream
Internet
transit
service
they
must
pay
for.
13

14. The
past
20
years
have
seen
enormous
turbulence
among
those
providing
the
core
of
the
Internet.
The
original
backbone
networks
have
survived,
but
their
ownership
has
gone
through
a
series
of
bankruptcies,
mergers
and
acquisi:ons.
Meanwhile
the
number
of
networks
par:cipa:ng
in
the
Internet
backbone
has
grown
from
6
to
over
6000.
14

15. I’ve
been
bandying
around
the
terms
“Peering”
and
“Internet
Transit.”
Let
me
explain
exactly
how
they
differ.
Internet
Transit
is
a
service
where
the
upstream
ISP
commits
to
deliver
traffic
to
any
valid
Internet
address.
It’s
typically
priced
in
$/Mbps/Month
and
the
Mbps
of
traffic
is
determined
by
measuring
traffic
levels
every
five
minutes
and
then
compu:ng
the
95th
percen:le
of
all
those
measurements
during
the
month.
Now
suppose
I’m
ISP1.
I
have
a
router
in
a
regional
data
center
where
I
buy
Internet
Transit
services,
but
I
no:ce
that
4%
of
my
traffic
is
to
my
compe:tor,
ISP2,
and
he
happens
to
have
a
router
in
the
same
regional
data
center
just
a
few
hundred
feet
away
from
mine.
He’s
my
compe:tor,
but
we
could
each
save
4%
of
our
monthly
bills
for
Internet
transit
if
we
agree
to
locally
exchange
the
traffic
that’s
des:ned
for
each
other’s
networks.
15

16. No:ce
that
we’re
only
exchanging
traffic
that
originates
with
a
customer
of
one
ISP
and
terminates
with
a
customer
of
the
other
peered
ISP.
16

17. ISP2
may
have
other
connec:ons
to
other
ISPs,
but
these
are
not
involved
(or
even
visible)
to
the
peering
arrangement
with
ISP1.
That’s
the
key
difference.
Peering
is
traffic
exchange
involving
only
those
addresses
that
are
served
by
the
two
peers.
Transit
involves
handling
packets
that
will
be
passed
off
to
one
or
more
addi:onal
networks.
17

18. But
whether
it’s
peering
or
transit,
what
is
actually
exchanged
and
how
does
it
work?
Here
things
are
remarkably
stable.
Operators
may
exchange
other
kinds
of
traffic
(MPLS,
Carrier
Ethernet)
for
other
services,
but
for
Internet
traffic,
they
exchange
IP
packets
(mostly
IPv4)
and
they
nego:ate
routes
using
Border
Gateway
Protocol
(BGP).
IPv4
is
essen:ally
unchanged
for
over
30
years
and
the
current
version
of
BGP
has
had
only
minor
tweaks
since
it
was
deployed
20
years
ago.
Business
arrangements
have
been
turbulent,
but
the
technology
has
been
remarkably
stable.
18

19. To
get
a
be3er
understanding
of
BGP,
suppose
I’m
running
BGP
on
my
edge
router
there
on
the
lel.
There
are
two
ISPs
I
wish
to
exchange
traffic
with
(either
peering
or
transit).
In
par:cular,
I’m
interested
in
gemng
traffic
to
address
blocks
A,
B
&
C.
My
router
starts
by
establishing
BGP
sessions
with
the
edge
routers
at
each
ISP.
19

20. Once
the
sessions
are
up,
I
get
an
announcement
from
the
edge
router
at
ISP1
saying
it’s
prepared
to
deliver
traffic
to
address
block
A
over
a
route
that
has
three
hops
and
traffic
for
address
block
B
over
a
route
that
has
one
hop.
20

21. This
is
followed
by
an
announcement
from
ISP2
saying
they
can
deliver
traffic
to
address
block
B
in
two
hops
or
to
address
block
C
in
two
hops.
Now,
I
have
to
make
some
decisions.
21

22. First
these
announcements
come
from
other
organiza:ons
who
may
or
may
not
be
competent.
Should
I
believe
ISP1
when
he
says
he
can
deliver
traffic
to
address
block
B
in
just
one
hop?
A
classic
example
of
mistakes
that
can
happen
occurred
in
Feb
2008
when
the
government
of
Pakistan
told
Pakistan
Telecom
to
block
traffic
to
YouTube
because
YouTube
was
hos:ng
blasphemous
videos.
The
engineers
at
Pakistan
Telecom
complied
by
crea:ng
a
very
specific
route
for
just
the
YouTube
addresses
(part
of
a
larger
Google
address
block).
Request
packets
that
matched
this
specific
route
were
sent
to
a
“black
hole
server,”
i.e.
a
server
that
dropped
each
packet
it
received.
Unfortunately,
this
black
hole
route
leaked
out
to
the
large
interna:onal
carrier,
Hong
Kong-­‐based
PCCW.
PCCW
didn’t
have
route
filtering
in
place
on
this
par:cular
link
and
they
passed
the
black
hole
route
around
the
world.
Over
90
major
ISPs
erroneously
accepted
this
route
and
for
more
than
two
hours
YouTube
was
dark
while
almost
all
the
world’s
YouTube
requests
went
to
the
black
hole
server
in
Pakistan.
So
you
can’t
always
trust
your
neighbor,
however
competent
they
may
have
seemed
in
the
past.
There
are
many
addi:onal
considera:ons.
For
example,
certain
routes
may
have
preferen:al
pricing
up
to
a
certain
commitment
level
but
become
expensive
at
higher
traffic
levels.
So
the
choice
of
which
adver:sed
route
to
use
can
involve
some
quite
complex
considera:ons.
22

23. To
give
you
a
sense
of
the
business
trade
offs
that
go
on,
I
have
two
examples.
The
first
is
a
friend
of
mine
who
formed
a
fixed
wireless
ISP
in
southeastern
Illinois
a
few
years
ago.
Because
he
was
located
in
farm
country,
the
only
way
he
could
get
an
Internet
connec:on
was
by
buying
Internet
Transit
service
(called
Direct
Internet
Access
or
DIA)
from
Ameritech
(now
AT&T)
the
local
telephone
monopoly.
His
price
was
more
than
100x
what
Internet
Transit
would
have
cost
him
in
Chicago,
but
there
were
no
compe:ng
fiber
routes
through
his
area
and
even
if
he’d
been
close
to
a
long
distance
fiber
route
(say
between
Chicago
and
St
Louis),
local
connec:ons
to
long
distance
fiber
are
extremely
expensive
or,
more
olen,
just
not
available.
Once
his
business
was
up
and
running,
my
friend
spent
many
days
driving
to
and
from
Chicago
looking
for
tall
buildings
and
talking
to
building
owners.
Eventually
he
build
a
series
of
four
wireless
links
(totaling
more
than
70
miles)
which
connected
him
to
Chicago.
In
Chicago,
he
signed
up
for
a
monthly
recurring
charge
for
rack
space,
for
roof
rights
on
the
Chicago
data
center
and
for
a
cable
from
his
rack
to
their
“meet
me
room.”
He’d
also
promised
free
high
speed
Internet
service
to
three
building
owners,
downstate,
who
gave
him
roof
access
on
the
route
to
Chicago.
But
now
that
he
was
connected
in
Chicago,
he
could
purchase
Internet
transit
from
any
of
a
dozen
compe:ng
carriers
(at
a
:ny
frac:on
of
what
he
was
paying
AT&T).
Although
he
had
spent
nearly
$100K
(and
untold
man
hours)
pumng
this
wireless
route
together,
he
figured
his
payback
was
9
weeks.
Loca:on
ma3ers!
The
second
thing
that
happened
was,
as
his
total
traffic
grew
he
began
to
qualify
for
peering
with
major
content
providers
like
Google
and
Akamai.
This
cut
further
cut
his
costs
for
Internet
transit.
23

24. The
second
example
is
only
approximate,
but
representa:ve.
I
don’t
have
the
actual
numbers
on
YouTube’s
traffic
or
their
costs
during
the
20
months
between
their
founding
in
Feb
2005
and
their
purchase
by
Google
in
Oct-­‐Nov
2006,
but
I
can
tell
you
that
one
of
their
early
employees
was
a
“peering
coordinator”
who
showed
up
at
NANOG
mee:ngs
early
in
2006.
In
early
2006,
there
was
already
a
great
interest
in
peering
with
YouTube.
By
the
summer
of
2006,
YouTube
was
the
5th
most
traffic’d
website
in
the
world.
They
were
s:ll
only
peering
in
Palo
Alto,
but
anyone
with
a
router
in
Palo
Alto
was
interested
in
offloading
their
YouTube
traffic.
And
any
Tier
one
carrier
that
didn’t
peer
with
YouTube
would
quickly
find
traffic
ra:os
going
unbalanced
on
links
where
they
handed
off
YouTube
traffic
to
someone
who
was
peering
with
YouTube.
I’m
not
showing
YouTube’s
costs
going
to
zero,
but
they
clearly
did
not
increase
(and
likely
went
down)
as
YouTube’s
traffic
grew!
24

25. Bill
Woodcock
and
Vijay
Adhikari
of
Packet
Clearing
House
did
a
very
comprehensive
survey
of
backbone
ISPs
in
2011
gemng
a
remarkable
86%
response
rate.
All
the
internal
indica:ons
are
this
survey
yielded
very
high
quality
data.
Several
interes:ng
things
emerged
from
this
data.
Most
notably,
many
operators
publish
a
set
of
peering
requirements,
and
these
typically
include
an
NDA.
But
if
you
meet
the
requirements,
there
are
no
formal
contracts!
These
are
handshake
agreements.
25

26. One
interes:ng
thing
was,
to
the
extent
there
are
contracts
between
operators
in
different
countries,
for
example
the
NDAs,
the
choice
of
governing
law
always
favors
the
country
with
stable
ins:tu:ons,
minimum
corrup:on
and
a
func:oning
judiciary.
26

27. In
terms
of
how
the
Internet
backbone
is
evolving,
the
most
interes:ng
thing
to
emerge
was
the
rise
of
mul:-­‐lateral
peering.
These
are
arrangements
that
started
in
Asia
and
selected
loca:ons
in
Europe.
We
haven’t
seen
this
in
the
US
yet,
but
there
is
an
organiza:on,
“open-­‐ix.org,”
backed
by
Google
and
Amazon
among
others,
that
is
trying
to
foster
the
spread
of
mul:-­‐lateral
peering.
27

28. Mul:-­‐lateral
peering
drama:cally
reduces
the
number
of
BGP
sessions
one
must
configure
and
manage,
thus
facilita:ng
more
peering.
With
bi-­‐lateral
peering,
there
is
a
separate
BGP
session
for
every
peer.
28

29. In
mul:-­‐lateral
peering,
one
organiza:on
–
perhaps
a
co-­‐op
or
a
vendor
–
provides
a
single
route
server.
Each
par:cipant
establishes
a
single
BGP
session
to
this
server.
Typically,
the
route
server
includes
session-­‐specific
configura:on
which
allows
you
some
of
the
flexibility
you
would
have
had
with
N
bi-­‐lateral
peering
sessions
but,
to
get
started,
you
can
ignore
all
that
and
just
establish
one
simple
BGP
session
that
reaches
hundreds
of
peers.
29

30. This
graph
shows
the
number
of
IP
addresses
handled
by
various
carriers
as
a
func:on
of
how
many
peering
agreements
those
carriers
have.
You
can
see
that
some
of
the
original
Tier
1s
are
s:ll
visible
in
the
upper
lel,
but
otherwise,
the
Internet
backbone
is
very
distributed.
And
this
graph
is
based
on
addresses
handled,
not
on
traffic
carried.
30

31. When
we
look
at
traffic,
the
top
ISPs
are
quite
distributed.
Also,
we
can
see
what
happens
when
the
large
carrier
(Level
3
at
the
top)
buys
the
second
largest
carrier
(Global
Crossing
in
grey
just
below)
as
happened
in
April
2012.
Both
networks
immediately
saw
a
drop
in
traffic
as
customers
who
wanted
redundant
connec:ons
dropped
one
of
their
connec:ons
to
the
now
merged
business.
Then
over
:me,
both
networks
see
further
drops
in
traffic
as
the
rest
of
the
players
rearrange
their
networks.
Also,
note
that
this
traffic
diagram
only
deals
with
ISPs
that
offer
Internet
transit
services.
The
second
largest
network
in
the
world,
by
traffic,
is
Google.
If
Google
were
shown
on
this
graph,
it
would
appear
between
the
black
and
grey
lines.
So
the
Internet
is
very
distributed
and,
as
Rensys
notes
in
their
report,
the
rela:ve
market
share
of
the
backbone
carriers
as
a
group
has
been
falling
over
the
past
decade.
In
~20
years
of
the
commercial
Internet,
no
one
has
been
able
to
gain
control
of
the
Internet
backbone.
In
the
1990s,
the
original
gang
of
~six
backbone
providers
thought
they
had
an
oligopoly
(a
cartel),
but
by
2002,
second
:er
backbones
used
"donut
peering”
to
eliminate
the
original
:er
ones’
leverage.
Since
2000,
we've
seen
the
emergence
of
mul:ple
CDNs
(Akamai,
Level
3,
Google,
Limelight,
plus
Amazon,
Nejlix,
and
others
in
the
making)
which
have
further
diluted
any
a3empt
to
monopolize
the
backbone.
Also
over
the
past
20+
years,
we've
seen
an
explosion
in
the
number
of
buildings
where
some
kind
of
peering
takes
place.
In
short,
no
one
has
been
able
to
monopolize
the
Internet
backbone.
Now
we’re
seeing
the
emergence
of
mul:-­‐lateral
peering
and
even
more
backbone
par:cipants.
31

32. The
Internet
backbone
is
a
very
interes:ng
phenomenon.
It’s
essen:ally
unregulated.
IANA
(the
body
that
supervises
the
assignment
of
addresses
and
other
protocol
number
assignments)
provides
only
coordina:on.
If
IANA
withheld
or
manipulated
assignments,
their
func:on
could
be
quickly
and
informally
bypassed.
Recently
we’ve
heard
a
lot
about
“regula:ng
the
Internet”
especially
since
the
revela:ons
of
NSA
spying.
But
most
such
discussion
is
happening
without
any
understanding
of
how
the
Internet
backbone
actually
works.
Forecasts
are
iffy,
but
the
current
system
is
extremely
successful
and
extremely
robust,
so
I
am
op:mis:c
the
Internet
will
con:nue
to
grow,
indefinitely.
32

33. 33