Taking Security Groups to Ludicrous Speed with OVS (OpenStack Summit 2015)

1. Taking Security Groups to Ludicrous Speed with Open vSwitch OpenStack Summit Vancouver, 2015 Miguel Angel Ajo @mangel_ajo Ivar Lazzaro @ivarlazzaro Thomas Graf @tgraf__ Justin Pettit @Justin_D_Pettit

2. Agenda Problem Statement – Status Quo – a.k.a “The Bridge Mess” Possible Solution – OVS + Stateful services (+ OVN) Results – Performance Numbers Q&A

3. Status Quo

4. Mess of Bridges. br-eth1 (Open vSwitch) OpenFlow table OVS bridge br-int (Open vSwitch)

5. Mess of Bridges. br-eth1 (Open vSwitch) veth OpenFlow table OVS bridge br-int (Open vSwitch)

6. Mess of Bridges. br-eth1 (Open vSwitch) qbr (Linux Bridge) qbr (Linux Bridge) qbr (Linux Bridge) veth iptables rules OpenFlow tableOVS bridge Linux bridge br-int (Open vSwitch)

7. Mess of Bridges. Why? VM br-eth1 (Open vSwitch) qbr (Linux Bridge) qbr (Linux Bridge) qbr (Linux Bridge) VM lxc tap veth iptables rules OpenFlow table OVS bridge Linux bridge 4-5 network devices per guest in host! br-int (Open vSwitch)

8. Possible Solution Stacking Things Properly (c) Karen Sagovac

9. Can we have a pure OVS Model? br-int (Open vSwitch) VM br-eth1 (Open vSwitch) VM lxc Tap, veth, or internal port OpenFlow table with security groups OVS bridge 1 network device per guest in host! Makes VMs and containers equally happy.

10. Some Background (OVS, OVN, Kernel CT)

11. ● Highly scaleable multi layer virtual switch for hypervisors – Apache License (User Space), GPL (Kernel) ● Extensive flow table programming capabilities – OpenFlow 1.0 – 1.5 (some partial) – Vendor Extensions ● Designed to manage overlay networks – VXLAN (+ extensions), GRE, Geneve, LISP, STT, VLAN, ... ● Remote management protocol (OVSDB) ● Monitoring capabilities Open vSwitch

12. ● Virtual Networking for OVS – Developed by same team that made OVS – Works on same platorms (Linux, Containers, Hyper-V) ● Provides L2/L3 virtual networking – Logical switches and routers – Conntrack-based security groups – L2/L3/L4 ACLs – Physical and DPDK-based logical-physical gateways ● Integrated with OpenStack and other CMSs OVN

13. Implementing a Firewall with OVS ● OVS has traditionally only supported stateless matches ● As an example, currently, two ways to implement a firewall in OVS – Match on TCP flags (Enforce policy on SYN, allow ACK|RST) ● Pro: Fast ● Con: Allows non-established flow through with ACK or RST set, only TCP – Use “learn” action to setup new flow in reverse direction ● Pro: More “correct” ● Con: Forces every new flow to OVS userspace, reducing flow setup by orders of magnitude – Neither approach supports “related” flows or TCP window enforcement

14. Connection Tracking ● We are adding the ability to use the conntrack module from Linux – Stateful tracking of flows – Supports ALGs to punch holes for related “data” channels ● FTP, TFTP, SIP ● Implement a distributed firewall with enforcement at the edge – Better performance – Better visibility ● Introduce new OpenFlow extensions: – Action to send to conntrack – Match fields on state of connection ● Have prototype working. Expect to ship as part of OVS in next release.

15. Netfilter Conntrack Integration OVS Flow Table Netfilter Connection Tracker CT Table Userspace Netlink API Create & Update CT entries Connection State (conn_state=) conntrack() Recirculation 1 2 3 4

16. Zone 1 Connection Tracking Zones OVS Flow Table CT Table Zone 2 CT Table Netfilter Connection Tracker

17. OVSFirewallDriver

18. OVSFirewallDriver ● Original proposal from Amir Sadoughi – https://review.openstack.org/#/c/89712 ● Stable/kilo (just a POC) – https://review.openstack.org/#/c/183725/

19. Example HTTP Request VM 1 VM 2 HTTP req response GLOSARY of OF actions NORMAL = “do like a normal switch” ct(commit) = “push this packet to CT” ct(recirc) = “grab any ct info we have, set +trk, and send to T0”

20. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL VM1 VM2

21. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk-trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL VM1

22. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL

23. SG OpenFlow Table structure +trk(+est or +rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL

24. SG OpenFlow Table structure +trk(+est/+rel) → NORMAL ARP (with filters) → NORMAL (…) SG rules in OF: ip Egress T1Input T0 ct(commit, recirc) (…) SG rules: tp_dst=80 Ingress T2 From VM(n) (MAC+in_port -trk) -trk → ct(recirc) ToVM(n)(MAC) match ct(commit),NORMAL VM2

29. openvswitch_firewall.py ● update_security_group_{rules, members} ● prepare_port_filter ● update_port_filter ● remove_port_filter ● filter_defer_apply_{on,off} neutron.agent.linux.firewall.FirewallDriver neutron.agent.linux.openvswitch_firewall.OVSFirewallDriver

30. Performance Numbers

31. Test Setup Explained System: 2 Socket, 24 core, IvyBridge CPU: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz Kernel: 3.10.0-229.1.2.el7.x86_64 Test: Netperf with TCP_STREAM and TCP_RR Notes: Virt overhead eliminated, netperf/netserver runs baremetal Compute 1 netperf Compute 2 10GiB Link netserver netserver local Multi node

32. TCP Stream, Local, 1 netperf thread 64 128 512 1024 9000 64000 0 50 100 150 200 250 300 350 400 450 500 0 5,000 10,000 15,000 20,000 25,000 TCP stream Local, 1 netperf threads sub-title iptables throughput OVS throughput iptables cycles OVS cycles Packet Size CPUMegacyclesperMbit Mbit

33. TCP Stream, Local, 16 netperf threads 64 128 512 1024 9000 64000 0 50 100 150 200 250 300 350 400 0 50,000 100,000 150,000 200,000 250,000 TCP stream Local, 16 netperf threads sub-title iptables throughput OVS throughput iptables cycles OVS cycles Packet Size CPUMegacyclesperMbit Mbit

34. TCP Stream, Multi Node, 8 netperf threads 64 128 512 1024 9000 64000 0 50 100 150 200 250 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 10,000 TCP stream node-to-node, 8 netperf threads iptables throughput OVS throughput iptables cycles OVS cycles Packet Size CPUMegacyclesperMbit Mbit

35. TCP Requests, Local, 1 netperf thread 64 128 512 1024 9000 64000 0 50 100 150 200 250 300 350 400 450 500 0 5,000 10,000 15,000 20,000 25,000 TCP stream Local, 1 netperf threads sub-title iptables throughput OVS throughput iptables cycles OVS cycles Packet Size CPUMegacyclesperMbit Mbit

36. TCP Requests, Local, 64K packets 1 4 8 16 0 10 20 30 40 50 60 70 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 160,000 TCP Requests/s Local, 64K packets sub-title iptables requests/s OVS requests/s iptables cycles OVS cycles Number netperf threads CPUMegacyclesperMbit Requests/s

37. TCP Requests, Multi Node, 1 netperf thread 64 128 512 1024 9000 64000 0 2 4 6 8 10 12 14 16 18 20 0 5,000 10,000 15,000 20,000 25,000 30,000 TCP Requests/s node-to-node, 1 netperf threads sub-title iptables requests/s OVS requests/s iptables cycles OVS cycles Packet Size CPUMegacyclesperMbit Requests/s

38. TCP Requests, Multi Node, 64K packets 1 4 8 16 0 5 10 15 20 25 30 35 40 0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 18,000 20,000 TCP Requests/s node-to-node, 64K packets sub-title iptables requests/s OVS requests/s iptables cycles OVS cycles Number netperf threads CPUMegacyclesperMbit Requests/s

39. Conclusion ● Both throughput and latency are considerably improve (Up to 6x in some situations). ● If limited by wire speed, pure OVS approach generally consumes less CPU cycles for the same result, leaving more resources for actual workload. ● Issue for specific packet sizes to be investigated and resolved before merge.

40. Next Steps ● Convert ML2 PoC to a patch that can be merged – Write functional tests – Optimize OF rules/manipulation ● Complete upstream merge of connection tracking support in Open vSwitch in the Linux kernel ● Consider and realize OVN integration of this work ● Hopefully ready for Liberty

41. Q&A ● OVS w/ CT Neutron ML2 plugin – https://github.com/mangelajo/vagrant-rdo-juno-ovs-ct -firewall ● Open vSwitch – http://openvswitch.org/ ● Conntrack code on GitHub – https://github.com/justinpettit/ovs/tree/conntrack ● Stateful Connection Tracking & Stateful NAT (OVS conference) – http://www.openvswitch.org/support/ovscon2014/17 /1030-conntrack_nat.pdf

42. Thank You!