RouteFlow & IXPs This talk will discuss the architecture of RouteFlow which is a leading OpenFlow based virtual router. It will focus on the new projects based upon RouteFlow which are finding traction in Internet eXchange Points (IXPs) - Cardigan being one of the most popular one. Some common aspects of IXPS will be shown. The talk will conclude with a list of future projects and vision of SDN routing. About Raphael Vincent Rosa Raphael is a Communications Network Engineer. He finished his MS in Computer Science working with intra datacenter routing, contributing to open source SDN projects such as Ryu network controller and RouteFlow platform. Currently he is pursuing PhD research under the guidance of Dr. Christian Esteve Rothenburg with main interests in SDN and Distributed-NFV topics.

1. ROUTEFLOW &
SOFTWARE-DEFINED EXCHANGES
Raphael Vicente Rosa
Christian Esteve Rothenberg
University of Campinas (UNICAMP), Brazil

2. Agenda
• Background:
• Software-Defined Networking (SDN)
• Separating Routing from Routers (BGP-based pre-SDN)
• RouteFlow: Software-Defined IP Routing
• Basics
• Cardigan deployment at live IXP (>1 year)
• Software-Defined Exchanges
• Research perspectives and open innovation
2

3. Background on SDN [5]
3
[5] "Software-Defined Networking: A Comprehensive Survey."

4. SDNAND OPEN SOURCEARE EVOLVINGAND
ACCELERATING THE PATH OF STANDARDIZATION
Components of the New Frontier

5. SDN & NFV: The Frontier of Networking
Existing
• CLIs
• Closed Source
• Vendor Lead
• Classic Network
Appliances
New
• APIs
• Open Source
• Customer Lead
• Network Function
Virtualization (NFV)
Adapted from: Kyle Mestery, Next Generation Network Developer Skills

6. Separating Routing From Routers
• Routing Control Platform (2004) [1]
• Path computation based on a consistent view of network state
• Controlled interactions between routing protocol layers
• Expressive specification of routing policies
6
Figure 1 – Routing Control Platforms (RCPs) [1]

7. Redefinition of Inter-AS Routing
• Inter-AS Routing Issues
• Routing tables scalability (route entries)
• BGP policy definitions (communities, filter tags)
• Dynamic negotiation of network resources (e.g., bandwidth prices)
• …
• Complexity can be better managed
• Simpler routing configuration
• Network-wide traffic engineering
• Better network diagnostics and troubleshooting
• New interdomain routing protocols
7

8. RCP Challenges
• Robustness
• Consistent view: loop-free paths
• Scalability
• Session establishment, path computation, management
• Convergence speed
• Intra and Inter AS route computation
• Configuration languages
• Correctness and security
• Troubleshooting and diagnostics
• Detach control and data planes
8

9. RouteFlow: Basics
Data Plane
Control Plane

10. Linux
RouteFlow: Basics

11. RouteFlow [2]
• Background
• Glue of IP routing stacks with OpenFlow
• Controller-centric hybrid IP networks
• Migration path to SDN (e.g., Cardigan)
• Benefits:
• Cheap network gear with minimal embedded software
• Provide interoperability with legacy network elements
• Avoid re-writing legacy protocols in a centralized fashion
• Innovation power to stakeholders
• network operators, service providers
11
[2] “Revisiting routing control platforms with the eyes and muscles of software-defined networking”

12. RouteFlow: High Level Architecture
• Modular
• 3 components
• Hierarchical
• Distributed
• Multi-controller
• POX, Ryu, ODL
• Routing stacks
• Quagga, BIRD, XORP
12
Figure 2 – RouteFlow Architecture [2]

13. RouteFlow
13
Figure 3 – RouteFlow General Architecture

14. RouteFlow: SDX for Layer 3
• RouteFlow – an approach to build BGP in the controller
• Create a virtual model of the router in the controller
• Forward BGP update packets to the virtual interface
• Compute BGP route tables (RIB and FIB) based on virtual model
• Translate FIB into Flow Tables based on real switch model
• Install Flow Tables in the data plane
• Build a SDN-based Router
• vandevecken code fork on RouteFlow (higher performance, simpler, 1.3 capable)
• Build peering fabric based on dynamic SDN policies
• rather than static network constructs
• Use SDN rules to scrub bad behavior*
• Deal with Router hygiene issues (default deny instead of default allow).
• Deal with “Capacity Theft” issues (as well as “noise” traffic)
(Adapted from – Inder Monga NVS'14)
14

15. Abstracting SDX:
what are the common functions?
• A place where peering domains come together
• Peering domains need not be physically connected
• Supports establishment of inter-domain connectivity/routing
• A broker of inter-domain and inter-SRP trust
• Each SDX defines its rules of engagement
• Enforcer of individual domain policies
• Marketplace for Software Defined Services (SDS) can be built
as a SDX overlay
• For example, an application-specific exchange (a video marketplace)
• Resources may be virtual or physical
(Adapted from – Inder Monga NVS'14)
15

16. Internet eXchange Points (IXPs)
• It's a network fabric that participants connect to
• They advertise their networks
• Any packet you place into the exchange finds the appropriate
destination
• It is not a: router, switch, layer 3 switch…
• It's just a place you poke two things...
• Advertisements for things you'll accept
• and packets you want to get rid of
16

17. IXPs
17
Figure 4 – Typical IXP Architecture [3]

18. Anatomy of a large European IXP [3]
• A particular Internet ecosystem (2012)
• Agnostic of any tier structure (flatenning 400 members)
• Peer-Peer links (more than 50,000)
• High-densely connected (visible / invisible / cannot tell P-P links)
• Prefix exchange ratio
• Traffic matrix (10 PB of IP traffic daily)
• Misconceptions reassessment
• Tier-1 ISPs do not peer at IXP (they do)
• IXPs are not used for transit (they are)
• The number of peer-peer links on the internet is small (larger than
assumed)
• IXP links are mostly used for back-up (they are not)
18

19. Cardigan [4]
• SDN-based distributed router in a live IXP
• Reduces operational complexity
• Hybrid SDN-IP network side-by-side
• New approach to the router abstraction model
• RouteFlow Vandervecken branch
19
Figure 5 – Cardigan initial 2-switch deployment [4]

20. Initial Limitations and Easy Fixes
• Extensible message formats
• RFProtocol flexibility
• Set of matches, actions, options...
• IPv6, MPLS on Ethernet
• Inneficient gateway resolution
• RFClient: Reception of its Netlink announcement and discovery of the
associated gateway MAC address
• Cache IPv4 and IPv6 routes until ARP resolutions
• Scalable router abstraction
• Router abstraction dependent on the physical topology
• No arbitrary paths inside the ISP network
• Traffic classification using VLAN tags per static inter-switch links (ISLs)
configuration
• Fine granular traffic control introducing MPLS paths
20

21. Cardigan 2.0
• MPLS Label Switched Path (LSP)
• A prefix to a path (set of paths)
• Ingress node through a set of transit nodes
• Operator freely defines packet circuits over the network (arbitrary
level of detail)
• Forwarding Path Manager (FPM) component On RouteFlow
architecture
• Feeds all calculated routes (even if initially discarded)
21

22. Cardigan 2.0
22
Figure 6 – Cardigan with MPLS design [4]

23. Deployment
• Pronto Switches (PicOs) - 1G SFPs
• Out-of-band VM controller by layer 2 VLANs
• Traffic forwarded directly by OpenFlow switches
• In production for 9 months
• 90 organizations - forwarding customer traffic and sharing routes
• 1134 flows on each swicth (1028 layer 3 routes)
23

24. Discussion Itens (1/2)
• Protocol compliance
• OF 1.0 TTL decrement
• MAC addressing
• Scalability of flow tables
• OF agent implementation
• Vendor switches memory leaks and flow counters
• Encapsulation Hazards
• MTU size for Ethernet, VLAN, MPLS, etc
24

25. Discussion Itens (2/2)
• Gateway Address Resolution - Increased performance
• Separation of gateway resolution and route processing
• Scalability
• Distribution of the FIB accross multiple devices, different data plane
technologies (NPU/FPGA)
• Resilience
• High-available non-stop forwarding solution and systematical SDN
troubleshooting
• Policy enforcements at IXPs – tedious tasks
• Manual time-of-the-day routing, dynamic traffic engineering, route
preferences, etc)’
25

26. Related Work
• IXP: an interesting networking landscape
• SDX: A Software Defined Internet Exchange
• Arpit Gupta (Georgia Institute of Technology), Laurent Vanbever (Princeton
University), Muhammad Shahbaz (Georgia Institute of Technology), Sean P.
Donovan (Georgia Institute of Technology), Brandon Schlinker (University of
Southern California), Nick Feamster (Georgia Institute of Technology),
Jennifer Rexford (Princeton University), Scott Shenker (UC Berkeley), Russ
Clark (Georgia Institute of Technology), Ethan Katz-Bassett (University of
Southern California)
• ACM SIGCOMM, Chicago, IL. August 2014.
• Enforcing RPKI-Based Routing Policy on the Data Plane at an
Internet Exchange
• HotSDN ’14
26

27. Future Work
• Rethink peering between SDN domains
• Strictly enforce routing and security policy
• Prevent crafted BGP packets from exploiting policy
• Prevent misdirected traffic
• Simplify operations
• Increase reliability
27

28. Research & Open Innovation
• Open Innovation
• Google Summer of Code [ 2013, 2014, … new proposals]
• Github Community [ forks, pull requests ]
• Code Contributions [5K – 10K LOC, testing hours, bug reports ]
• Mailing Lists [ +200 Members, 160 Topics ]
• Community contributions & collaborations
• Web-based UI & Internet 2 HW pilot [C. Small, Indiana]
• Aggregated BGP Routing Service [C. Corrêa, Unirio]
• SNMP plugin [J. Stringer, Google]
• IPv6, MPLS, IXP [REANZ, Google]
• Optimal BGP best path reflection [R. Raszuk, NTT-MCL]
• Intra data center BGP routing [R. V. Rosa, Unicamp]
• MPLS LDP support [Vikram Dham]
• ….
28

29. RouteFlow
• Ongoing Work
• OpenFlow hardware upfront: Pica8, NoviFlow, Corsa (OF 1.3)
• Open Issues
• Replicate data plane events in the virtual environment
• Extensive evaluation on realistic networking conditions
• OpenFlow table abstractions
• Towards a routing control PaaS
• IXP Perspective
• Scale to large networks
• Future implementation of policies
• Load balancing, closest exit usage, complex setups
https://github.com/routeflow/RouteFlow/wiki/Projects-and-ideas-for-RouteFlow
29

30. Summary
• Routing Control Platforms possible through SDN
• IXP is a particular microcosm of the Internet ecosystem
• SDN may solve IXP challenges
• Software Defined Exchanges
• RouteFlow as a SDX layer 3
• Rethink peering between IP and SDN domains (and SDN-SDN)
• More related work at below L3 and at application-layer
• Cardigan RouteFlow pilot deployment is a real SDN-
based legacy IXP implementation
30

31. References
• [1] Nick Feamster, Hari Balakrishnan, Jennifer Rexford, Aman Shaikh, and Jacobus van der
Merwe. 2004. The case for separating routing from routers. In Proceedings of the ACM
SIGCOMM workshop on Future directions in network architecture (FDNA '04). ACM, New
York, NY, USA, 5-12.
• [2] Christian Esteve Rothenberg, Marcelo Ribeiro Nascimento, Marcos Rogerio Salvador,
Carlos Nilton Araujo Corrêa, Sidney Cunha de Lucena, and Robert Raszuk. 2012. Revisiting
routing control platforms with the eyes and muscles of software-defined networking. In
Proceedings of the first workshop on Hot topics in software defined networks (HotSDN '12).
ACM, New York, NY, USA, 13-18.
• [3] Bernhard Ager, Nikolaos Chatzis, Anja Feldmann, Nadi Sarrar, Steve Uhlig, and Walter
Willinger. 2012. Anatomy of a large european IXP. SIGCOMM Comput. Commun. Rev. 42, 4
(August 2012).
• [4] Jonathan Philip Stringer, Qiang Fu, Christopher Lorier, Richard Nelson, and Christian
Esteve Rothenberg. 2013. Cardigan: deploying a distributed routing fabric. In Proceedings of
the second ACM SIGCOMM workshop on Hot topics in software defined networking
(HotSDN '13). ACM, New York, NY, USA, 169-170
• [5] Diego Kreutz, Fernando M. V. Ramos, Paulo Verissimo, Christian Esteve Rothenberg,
Siamak Azodolmolky, Steve Uhlig. "Software-Defined Networking: A Comprehensive Survey."
To appear in Proceedings of the IEEE, 2015.
• https://github.com/SDN-Survey/latex/wiki
• http://arxiv.org/abs/1406.0440
31

32. 32
Thank you!
Questions?
• Raphael Vicente Rosa
• raphaelvrosa [at] gmail [dot] com
Ask and contribute!
routeflow-discuss@googlegroups.com
Get the code!
https://github.com/routeflow/RouteFlow/

33. BACKUP
33

34. RouteFlow Project History
34
• Start Msc. Thesis
work by Marcelo
N.
• First Prototype
• QuagFlow Poster
@ SIGCOMM
• Open-Source
Release
• Demos @ ONS11
• Demo @
SuperComputing 11
• Tutorial & Demo @
OFELIA/CHANGE SS• First Short-Paper
@ WPEIF
• Evaluation on
NetFPGA testbed
• Indiana University
- Pronto OF
switches + BGP
peering with
Juniper MX • Demos @ ONS12
• Running on FIBRE /
OFELIA testbed
• HotSDN Paper
• Collaboraion with
NTT

35. RouteFlow 2013 / 2014
• Significant contributions from CARDIGAN and the
VANDERWECKEN fork
• Live in production network in New Zealand
• Inter-VLAN routing at UNESP, Brazil.
• Google Summer of Code (GSOC 2013 / 2014)
• …
35