This document provides tips and common mistakes to avoid when developing an OpenFlow controller using the Trema framework. It discusses key things to know about OpenFlow and Trema, such as Trema being a programming framework rather than a full controller itself. It then presents a use case of developing a production OpenFlow controller to manage virtual networks across thousands of switches and hosts. The design uses a load balancer and three-tier architecture. An evaluation showed the controller could manage over 400 switches and 16,000 virtual networks per controller instance.
2. Agenda
• Provide tips and typical mistakes when
developing OpenFlow controller with Trema C
library
• Explain a production OpenFlow controller
development use case
Copyright (C) 2013 NEC Corporation
3. THINGS TO KNOW ABOUT OPENFLOW
AND TREMA BEFORE DEVELOPING
OPENFLOW CONTROLLER
Copyright (C) 2013 NEC Corporation
4. Trema is not an OpenFlow controller
• Trema is a programming framework and just a
building block for implementing OpenFlow
controllers
• You may use any other building blocks as well
as Trema
– For high availability, scalability, network protocol
support, and etc.
Copyright (C) 2013 NEC Corporation
5. OpenFlow switch is not a database
• OpenFlow switch is not a database and does not
have ACID properties
• Error messages are returned from switch
asynchronously
• Send a Barrier Request to complete a previously
sent Flow Mod (or any other) operation
• See and reuse Transaction Manager:
– https://github.com/trema/apps/tree/master/transacti
on_manager
Copyright (C) 2013 NEC Corporation
6. Asynchronous events happen before
features request/reply exchange
• You may receive asynchronous OpenFlow
events such as Packet-In from unknown
switches that features reply has not been
received
• In Trema, Switch Ready event is triggered
when a features reply is received
Copyright (C) 2013 NEC Corporation
7. send_openflow_message() does not
send message actually
• send_openflow_message() just pushes a given
message to an appropriate send queue
• The send queue is automatically flushed inside
the main loop
• Message ordering is NOT guaranteed among
switches
Copyright (C) 2013 NEC Corporation
8. libtrema is not fully thread-safe
• The followings may be thread-safe but not
guaranteed for all functions
– buffer, doubly linked list, hash, linked list, log,
match table, OpenFlow message, packet info,
packet parser, stat, utility, wrapper
• OpenFlow application interface is NOT thread-
safe
• Call send_openflow_message() from the main
thread that trema_run() is running
Copyright (C) 2013 NEC Corporation
9. Length of send queue is limited
• The maximum length of send queue (for
sending OpenFlow messages) is limited and
fixed
• send_openflow_message() may return false if
the send queue is full
Copyright (C) 2013 NEC Corporation
10. Do not call flush_messenger()
• flush_messenger() flushes all send queues and
receive queues immediately and
synchronously
• Calling it in an OpenFlow event handler may
recursively call the event handler!
• Call only if you known how it works very well
Copyright (C) 2013 NEC Corporation
11. start_trema() may call fork(2)
• start_trema() calls fork(2) if your controller is
to be daemonized
• Use caution if you call functions that may be
affected by fork(2)
Copyright (C) 2013 NEC Corporation
13. Requirements
• Functional Requirements
– Manage association among virtual networks (based
on non-OpenFlow technology) and switch ports
– Associate a switch port with MAC addresses located
on the switch port
– All operations above can be done via REST interface
• Non-functional Requirements
– 1K+ switches must be handled
– 10K+ active virtual networks must be managed
– 10K+ end hosts must be connected to virtual networks
Copyright (C) 2013 NEC Corporation
18. Evaluation items and results
• # of switches that can be managed
– 410 - 412 switches per a single Virtual Network
Manager were connected and initialized properly
• Switch daemons were not able to run due to
insufficient memory (system memory was 2 GB)
– 1024 switches were connected and initialized with
three Virtual Network Managers
Copyright (C) 2013 NEC Corporation
19. Evaluation items and results
• # of virtual networks that can be managed
– 16384 virtual networks that have 8 ports (hosts)
each were successfully created with 1024 switches
and three Virtual Network Managers
• Virtual network setup time
– Setup time did not increase even if we have a
number of virtual networks
– Database access time was constant and a minor
factor
Copyright (C) 2013 NEC Corporation