What's Next for the Net? Security, Reliability, Capability, Performance, Flexibility and More

1. Routing/Interconnection: An Introduction
“The Internet works by interconnecting networks”
“This is the IETF RFCs in action”
University of Cyprus
Sepember 2018
Martin J. Levy, Network Strategy
Cloudflare

2. Introduction & Agenda
The word “Internet”
Interconnected networks
The relationship between individual networks
IP transit
IP peering
Three key topics
An introduction to peering
The creation of IXPs (Internet Peering Points)
Visualizing interconnected networks

3. The word “Internet” - How routing works
Technically – a very complex subject
The Internet is a collection of networks
No network stands alone
Interconnections are required
Robustness can be created
Multi-homing (more than one transit)
Peering between “like” networks
Diversity (physical and logical)
Nothing is static!

4. The relationship between individual networks
Perfect“Tier1”routing
Prettygoodrouting(withmanypeers;onetransit)
Classicbackbonewithtwotransits(wellbalanced)

5. Reality … The Internet is complex (but
works!)
“ISP”
“Transit Provider”
“Transit Provider”
“Transit Provider”
“Transit Provider”
“Transit Provider”
“Transit Provider”
Thisismessy!
“Transit Provider”
“Transit Provider”

6. Reality … The Internet is complex (but
works!)

7. An introduction to peering

8. An introduction to peering
Internet Exchange Points (IXPs) are a good idea
Peering is a good idea
A countries self-reliance is a good idea
Critical services (DNS, NTP, etc) are a good idea
The Internet is not going away; in fact it’s growing
Peering helps the Internet grow

9. Peering/Transit - defining the terms
Peering
The exchange of traffic between two
networks
Only for traffic associated with the
network, or its customers
Can be via an IXP (or NAP) or a private
interconnect
Subject to a bilateral or unilateral contract
Either zero-cost, charged-for service or as
part of other contracts
Majority of negotiations done on a bilateral
basis
Transit
A network providing another network with
a full connection to the Internet
Sometimes provided as a “default route” or
as full routes
Partial Transit
A network providing access to only some
of the global Internet
Sometimes sold as access to a specific
provider or destination

10. Is the Internet growing?
Internet growth is continuously happening
More growth within IPv6 vs. IPv4; which makes sense
More networks joining the global Internet routing tables each-and-every day
This is just a measure of destinations on the Internet
It’s not an exact measure of peering or transit relationships
http://bgp.he.net/report/prefixes
Network PrefixesASNs
~3,300 new networks in IPv4 table per year
~1,400 new networks in IPv6 table per year
IPv4
IPv6

11. Who are the big Internet peering players?
This is a measure of BGP adjacencies
Customers are adjacencies
Peers are adjacencies
A high adjacencies means there’s a short path
Short as in “AS PATH” vs. physical path
http://bgp.he.net/report/peers
IPv4 IPv6

12. Internet Peering Points

13. The creation of IXPs (Internet Peering Points)
IXPs provide a common facility to interconnect networks
Normally defined as an IXP when three or more networks interconnect
IXPs provide a common facility for traffic to flow between networks
All networks are considered “equal”
Peering is not transit
Peering can be multilateral or bilateral
IXPs can be membership based, commercial or informal
IXPs provide a location for additional services

14. The creation of IXPs (Internet Peering Points)
Initial idea was very simple (Early 1993)
NSF created four “NAPs” (Network Access Points)
Opened the Internet to commercial use
Additional “NAP” sites (CIX & MAE-West in California)
Peering Points and BGP go together perfectly
City Name
Pennsauken, NJ Sprint NAP
San Francisco, CA PacBell NAP
Chicago, IL Ameritech AADS NAP
Vienna, VA MFS MAE-East

15. IXPs (Internet peering points) globally
Tokyo
JPNAP
JPIX
BBIX
EQUINIX
Los Angeles
EQUINIX
ANY-2
LAIIX
Seattle
SIX
EQUINIX
San Jose
EQUINIX
ANY-2
AMES
São Paulo
NAP do Brasil
PTTMetro
Buenos Aires
CABASE
Miami
EQUINIX
NOTA Virginia area
EQUINIX
ANY2
Atlanta
TIE TELX
EQUINIX
Chicago
EQUINIX
Chicago IX
Toronto
EQUINIX
TORIX
Dublin
INEX
NJ & NY
EQUINIX
TIE TELX
NYIIX
BigApe
London
LINX
LONAP
Paris
EQUINIX
FRANCEIX
SFINX
FreeIXBrussels
BNIX
FreeBiXAmsterdam
AMSIX
NL-IX
Oslo
NIX
Boston
MXP
Johannesburg
NAP Africa
JINX
Nairobi
KIXP
Singapore
EQUINIX
SGIX
Auckland
APE
V6IX
Beijing
National IX
Lisbon
GigaPix
Wellington
WIX
Sydney
EQUINIX
PIPE
Santiago
NAP Chile
Madrid
ESPANIX
Geneva
CERN
Moscow
MSK-IX
Frankfurt
DE-CIX
KleyReX
Stockholm
NetNod
STHIX
SOL-IX
Jakarta
IIX
Zurich
SwissIX
EQUINIX
Prague
NIX CZ
Vienna
VIX
Milan
MIX
MINAP
Hong Kong
HKIX
EQUINIX
Seoul
KIDC
KINX
Taipei
TWIX
Mumbai
NIXI
New Delhi
NIXI
Kuala Lumpur
MyIX
Dallas
EQUINIX
TIE TELX
Guangzhou
National IX
Shanghai
National IX
Cairo
CAIX
MajorIXs/NAPsrepresented;plentymoreexist

16. Why peering helps grow the local IP market
Two points of view – each can work together
Peering improves traffic flows and reduces costs
Traffic flowing with lower latency (less hops?), hence is more efficient
Networks that peer can reduce transit expenditure
Networks that peer see local content or local eyeballs easier
Peering reduces transit revenue from major players
Reduced revenue is a short term effect (but it’s there)
As customer experience improves; network dependence grows
Peering never replaces 100% of transit needs

17. Visualizing Networking

18. Acknowledgment for Hurricane Electric
I would like to thank my previous employer (Hurricane Electric)
for all the work presented here.
The public data is available at http://bgp.he.net/
The internal data (mainly the country graphs) is not available.

19. Why is BGP such a powerful protocol?
The IETF specified a protocol (BGP4) that could handle:
Massive routing tables
CIDR routing (the ability to specify an IP network address plus a network size
IPv4 & IPv6
Rules for routing internally within a network vs. routing to an external network
Much more
BGP in real-life is used by every network on the Internet
Every destination on the globe exists within the BGP global routing tables
Everything is public, visible, exposed, recorded

20. http://bgp.he.net/ – Route propagation
graphs
Peers or transits
that see routes
ASNoriginatingroutes
Routesseedownstreamofpeers
Select tab for more information
Transit provider
Transit provider

21. Example listings
http://bgp.he.net/country/BH
A listing of all active ASNs within a country
http://bgp.he.net/AS5416#_peers
Alistofall“peers”ofanetworks
A listing for a specific ASN

22. Looking at specific networks
BancoNacionalde
CostaRica(BNCR)
TherestoftheInternet(onlymajornetworksshown)
One transit provider
Note: Transits for the transit

23. Looking at specific networks
LibertyBroadcasting
Network
TherestoftheInternet(onlymajornetworksshown)
Transit provider
Transit provider

24. Collections of networks

25. The Democratic Republic of the Congo
2013
2014

26. Liberia
2013
2014

27. Seychelles

28. Lebanon

29. Country by country diagrams
Jamaica
Turks and Caicos Islands
Chile
Azerbaijan
Gabon
Sierra Leone
Barbados

30. Tanzania
2013
2014

31. Kenya

32. Belgium

33. France – 1,125 active ASNs (254 with v6)

34. Germany

35. United Kingdom (Great Britain)

36. Regional interconnections visualized
LatinAmerica&CentralAmerica

37. Regional interconnections visualized
Russia, Ukraine and surrounding countries

38. Regional interconnections visualized
Africa
Caribbean

39. Summary

40. Q&A
Contact:
Martin J. Levy
Network Strategy
Cloudflare
San Francisco, CA, USA & London, UK
http://cloudflare.com/
martin at cloudflare dot com
+1 (408) 499 3801
?

41. Additional slides - Peering Agreement review

42. Peering Agreement – The review
What follows is a sentence-by-sentence review of the
Comcast Settlement-Free Interconnect (SFI) Policy
Can be found at: http://www.comcast.com/peering/
Text dated Jan/2008 June/2009 July/2011 Oct/2013
Anything with strikethrough is from an earlier version
This peering language can change at any time
Not actually called “peering” but “SFI”

43. Sample peering agreement (public information)
Comcast Communications, Inc., Comcast Cable Communications Management, LLC, through its
operating subsidiaries ("Comcast"), seeks to interconnect its IP network with other Internet backbone
providers on a settlement-free basis when such interconnection provides tangible benefits to Comcast
and its customers.
For purposes of this SFI policy, an Internet Network must be a single Autonomous System (“ASN”).
The criteria involving traffic measurement below applies to IPv4 IPv4 and IPv6 and must be met by the
Applicant for three consecutive months. To apply for IPv4 or IPv6 settlement-free interconnection with
Comcast an email must be sent to peering@comcast.com containing evidence of meeting all criteria
listed in the policy. Applicants will be responded to within a reasonable timeframe to discuss their
request.
Networks that do not meet the requirements of the settlement-free interconnection policy, but remain
interested in Comcast dedicated internet access services may also address their requests to
peering@comcast.com. In this case the subject line of the e-mail should indicate a need for sales
assistance. The body of the message should include as many details as possible, such as location
and telephone number, to expedite the redirect for assistance.

44. Sample peering agreement (public information)
• IPv4 IPv4 and IPv6 Settlement-Free Interconnection Requirements for Comcast AS7922
• Applicant must operate a US-wide IP backbone whose links are primarily 10 Gbps or greater.
• Applicant must meet Comcast at a minimum of four mutually agreeable geographically diverse
points in the US. Interconnection points must include at least one city on the US east coast, one in
the central region, and one on the US west coast, and must currently be chosen from Comcast
peering points in the following list of metropolitan areas: New York City/Newark NJ, Ashburn,
Atlanta, Miami, Chicago, Denver, Dallas, Los Angeles, Palo Alto/San Jose, and Seattle.
• Applicant’s traffic to/from the Comcast network must be on-net only and must amount to at least 3
Gbps peak 7 Gpbs peak 10 Gbps average 20 Gbps average in the dominant direction.
Interconnection bandwidth must be at least 1 Gbps 10 Gbps at each interconnection point.
• A network (ASN) that is a customer of a Comcast network for any dedicated IP services may not
simultaneously be a settlement-free network peer.
• Applicant must have a professionally managed 24x7 NOC and agree to repair or otherwise remedy
any problems within a reasonable timeframe. Applicant must also agree to actively cooperate to
resolve security incidents, denial of service attacks, and other operational problems.

45. Sample peering agreement (public information)
• Applicant must maintain responsive abuse contacts for reporting and dealing with UCE (Unsolicited
Commercial Email), technical contact information for capacity planning and provisioning and
administrative contacts for all legal notices.
• Applicant must agree to participate in joint capacity reviews at pre-set intervals and work towards
timely augments as identified.
• Applicant must maintain a traffic scale between its network and Comcast that enables a general
balance of inbound versus outbound traffic. The end-to-end costs of network cost burden for
carrying traffic between networks shall be similar to justify SFI.
• Applicant must abide by the following routing policy:
• Applicant must use the same peering AS at each US interconnection point and must
announce a consistent set of routes at each point, unless otherwise mutually agreed.
• No transit or third party routes are to be announced; all routes exchanged must be
Applicant's and Applicant's customers' routes.
• Applicant must filter route announcements from their customers by prefix.
• Neither party shall abuse the SFI network peering relationship by engaging in activities such
as, but not limited to: pointing a default route at the other or otherwise forwarding traffic for
destinations not explicitly advertised, resetting next-hop, selling or giving next-hop to others.

46. Sample peering agreement (public information)
• Applicant should be willing to enter into an NDA before formal discussions begin.
• Applicant should be advised that the SFI processes will start with a 90 day trial. On successful
completion of that trial, a formal interconnect agreement will be processed. This agreement will
renew annually, subject to the then current SFI Policy. During the year if there is a violation of the
policy, the agreement and interconnections may be terminated upon written notice to the contacts
specified in the agreement.
• Applicant shall not be permitted to offer or sell any IP transit services providing only AS7922.
• Applicant must be financially stable.
• Comcast requires that Applicants seeking SFI in the United States agree to provide reciprocal SFI
arrangement with Comcast in the Applicant’s home market.

47. Sample peering agreement (public information)
• Ongoing SFI Reviews with Comcast AS7922
• Meeting the SFI network peering guidelines set forth herein is not a guarantee that an SFI network
peering relationship with Comcast will be established. Comcast shall evaluate a number of
business factors and reserves the right not to enter into a SFI network peering agreement with an
otherwise qualified applicant. Applicants may apply for settlement-free interconnection once every
six (6) months.
• Existing SFI network peers of Comcast will have their status reviewed periodically to ensure that
joint capacity planning intervals are sufficient for growth, contacts are refreshed for operations
purposes, and all criteria continue to be met.
• Comcast reserves the right to terminate SFI network peering, upon a notice period as determined
by the parties' agreement, with peers who do not meet the criteria described above.
• Periodic review of the policies contained here will be conducted to ensure that the criteria and
eligibility requirements are consistent with Comcast's business needs. Comcast may modify this
SFI policy at any time.

48. Additional slides about OECD report

49. The OECD report
OECD
An International Treaty Organization
Authored
2012 edition by:
Dennis Weller – Navigant Economics
Bill Woodcock – PCH
Published
October 2012
Part of a series, published every five years
Audience
Policy Makers, Regulators, Lawmakers, Economists, etc
Atotalof99pages,108references.Plentyof
Internetpeeringstats
sights.org/2012/10/22/internet-traffic-exchange-2-billion-users-and-its-done-on-a-h
Weller, D. and B. Woodcock (2012), “Internet Traffic
Exchange: Market Developments and Policy Challenges”,
OECD Digital Economy Papers, No. 207, OECD Publishing.
http://dx.doi.org/10.1787/5k918gpt130q-en

50. The OECD report (key takeaways - globally)
High handshake % for peering agreements
Ratio of traffic/ecosystem cost vs. voice cost
Peering and IXs work (well obviously)
Very good results with little regulation

51. The reports peering numbers (#1)
Key peering numbers:
“… this report gathered responses from 4,331 ISPs … representing 86% of … in 96
countries”
“… survey of 142,000 peering agreements …”
“… 99.5% of interconnection agreements … without a written contract”
“… less than 1% of a full mesh”
“market participants will self-organize efficient Internet exchange points”

52. The reports peering numbers (#2)
Performance:
“… traffic grow at an average rate of about 50% per year”
Costs:
“… rates for Internet transit have declined dramatically over the last fifteen years …”
“… the rate of change has leveled off in the most recent period …”
“… transit … five orders of magnitude lower than typical voice”

53. The real audience – legal and policy
Every market exists within a legal and policy framework established by
government.
Markets for network communications in OECD countries have benefitted
from liberalisation policies which have sought to open markets and promote
competition.
These have included specific measures to ensure availability of leased
lines and access to rights of way, for example.
The success of the Internet IP market for traffic exchange would not be
possible without this broad policy framework of liberalisation. Indeed, in
regions where the development of Internet IP traffic exchange has been
less satisfactory, the cause has generally been a lack of sufficient
liberalisation within the country or region, rather than a lack of performance
by the global Internet market as a whole.
Legal and Policy Frameworks

54. Challenges for the future – four points
Internet Architecture
IPv4 to IPv6
Technical Innovation
Speed of links need to keep up with Internet growth (10GbE to 100GbE links)
Optoelectronic physics research & Internet exponential growth
Internet model of traffic exchange
Tromboning, peering (bilateral & multilateral), CDNs
… evolution of relationships …
Policy
… challenges for policy makers …
“bright line” between TDM world and IP world

55. Policy (#1)
The “bright line” proposal
TDM mainly lives within a regulatory framework
IP traffic mainly lives without regulation
VoIP
(this was not reviewed – but there’s good text there)
TDM reform
Several OECD countries have undertaken reform

56. Policy (#2)
Disagreements in the IP peering world
“For a commercial agreement to be voluntary, parties to it must be able to decline or terminate it.”
Peering disputes have been fought-out in the press and in front of potential regulators
“there must be some limit to the amount of disruption a private dispute can be allowed to case”
Pricing model
“One allegation is that, when CDNs establish peering arrangements with local access networks, they deprive them of
transit revenue. This may be true, …”
Go read: A viable future model for the Internet, AT Kearney, 2011 www.atkearney.com/index.php/Publications/a-viable-
future-model-for-the-internet.html
Traffic exchange and network neutrality
Growth of CDNs & commercial agreements & potential exclusions in the marketplace
2005 – Cogent & Level3 – fought in the press
2010 – Level3 & Comcast – all solved commercially (vs. legal or regulatory)

57. Regional survey (#1)
“The state of development of the Internet economy differs greatly by region”
“In nearly every region of the world, regulatory decisions constrain Internet development far more than
technology, geography, economy, or demand. These regulatory decisions are rarely intended to
benefit Internet users; they are nearly always defined by other communities of interest, at the expense
of Internet users.”
Differences exist regionally
Africa - Latin America - Caribbean - Middle East - Europe - Canada & US - Pacific - Australia & NZ - South Asia - East Asia
Success stories exist regionally
Egypt, Nigeria, Ghana, Kenya & Tanzania. Brazil (CGI) & Argentina (CABASE)
“Europe has more IXPs than any other region, and seven of the top ten exchanges both by number of participants and volume …”
Less than successful story
India and NIXI – “… current domestic exchange capacity … less than 10% of the needs of a single national broadband provider”

58. Regional survey (#2)
Less than successful stories (an example) - Mexico
“Mexico continues to lag, being by far the largest nation in the world, and the only OECD member
country, to continue without any domestic Internet exchange capacity”
“This situation may be on the brink of reform, as COFETEL, the Mexican regulator, has opened
access to competitive long-haul circuits, has licensed a second national carrier, and is investigating
the establishment of an IXP.”
Source:
Telegeography
Figure 6: Retail price of enterprise Internet transit, mbps/month, in USD

59. The major takeaway
Lack of Regulation has helped the Internet