Docker Multi Host Networking
Rachit Arora , IBM
BigInsights on Cloud

Agenda

Docker networking – overview

Native Multi Host networking (Experimental)

Software defined networks SDNs

Multi Host networking options
− OpenVSwtich
− Weave
− Flannel , SocketPlane Etc..

Goals

Make "network" a first class object

Provide a Pluggable networking stack

Span networks across multiple hosts

Support multiple platforms

libnetwork
Updated Networking Stack in Docker
Create Networks using the Docker CLI
Multi-host Networking
Services UI

SDNs value proposal
SDNs proposes network solutions embracing cloud paradigms

Massively multi-tenant

Thousands tenants, massively scalable

Easy & fast (de)provisioning

Infra as code, API centric

Infrastructure agnostic

L3, does not stick with lower levels (physical designs, vlans & co)

Decouple infrastructure & tenants lifecycles

Cross technology, vendor agnostic


Full-mesh, network agnostic and encapsulated approach

Relies on L3 networks and GRE, VXLAN… tunnels for inter-hosts communication (avoid using L2)

Network agnostic, til hosts can route trafic

SDN Routers must route traffic between an inner virtual net and the ext. world
SDN Properties
Host #3
Host #1
Host #2
SDN
network #1
SDN
network #1
SDN
network #1
SDN
network #1
SDN
network #1
Router
Physical net.
Flow encapsulation
L3 network

OpenVSwtich

Open vSwitch is a production quality, multilayer virtual switch
licensed under the open source Apache 2.0 license.

It is designed to enable massive network automation through
programmatic extension, while still supporting standard
management interfaces and protocols (e.g. NetFlow, sFlow,
IPFIX, RSPAN, CLI, LACP, 802.1ag).

In addition, it is designed to support distribution across multiple
physical servers similar to VMware's vNetwork distributed
vswitch

Host #1 Host #3Host #2
Getting Started
Container ‘app’
Container ‘db’
Bridgedocker0
Container ‘app’
Container ‘db’
Bridgedocker0
Container ‘app’
Container ‘db’
Container ‘app’
Bridgedocker0
Container ‘app’
TCP ports of
services running
in containers are
mapped

Get rid of actual Docker bridges implementation ! (Turn ICC False)

If not, all containers will talk together across a same host

SetUp - OVS

Setup Dev Tools
− yum -y groupinstall "Development Tools"
− yum -y install wget openssl-devel
− yum -y install wget openssl-devel gcc make python-devel openssl-devel kernel-devel graphviz kernel-debug-
devel autoconf automake rpm-build redhat-rpm-config libtool

Set up Bridge Utils
− yum -y install bridge-utils

Prepare OVS rpm
− cd ~
− mkdir -p rpmbuild/SOURCES
− wget http://openvswitch.org/releases/openvswitch-2.3.1.tar.gz
− tar xvfz openvswitch-2.3.1.tar.gz
− cd openvswitch-2.3.1/
− cp ../openvswitch-2.3.1.tar.gz ~/rpmbuild/SOURCES/
− sed 's/openvswitch-kmod, //g' openvswitch-2.3.1/rhel/openvswitch.spec > openvswitch-
2.3.1/rhel/openvswitch_no_kmod.spec
− rpmbuild -bb --nocheck ~/openvswitch-2.3.1/rhel/openvswitch_no_kmod.spec

You can find the files in ~/rpmbuild/RPMS/x86_64/.

yum -y localinstall openvswitch-2.3.1-1.x86_64.rpm


Use OpenVSwitch to create bridges on each host, for each tenant’s subnet. For
instance, on host #1:
»
ovs-vsctl add-br tech-br
»
ovs-vsctl add-port tech-br tep0 -- set interface tep0 type=internal
»
ifconfig tep0 192.168.1.1 netmask 255.255.255.0
»
ovs-vsctl add-br sdn-br0
»
ovs-vsctl set bridge sdn-br0 stp_enable=true
Create SDN compliant bridges
Host #1
sdn-br0
Host #3
sdn-br0
sdn-br1
Host #2
sdn-br0
sdn-br1
Simplified view. Detailed insight exposed in later slides
Once per host: common plumbing
For each bridge: create and protect
against ethernet loops using
Spanning Tree Protocol (beware, in
complex/large deployments, it may
consumes a lot of CPU!)

Link SDN bridges
Host #1
sdn-br0
Host #3
sdn-br0
sdn-br1
Host #2
sdn-br0
sdn-br1

Use OpenVSwitch to link corresponding bridges accross hosts

In this example, we decided to use the full-mesh approach. On host #1:

ovs-vsctl add-port sdn-br0 gre0 --set interface gre0 type=gre options:remote_ip:1.2.3.2

ovs-vsctl add-port sdn-br0 gre1 --set interface gre1 type=gre options:remote_ip:1.2.3.3
gre0
gre1
1.2.3.1 1.2.3.2 1.2.3.3
Simplified view. Detailed insight exposed in later slides

U
Start containers and map them to the bridges
Host #1
sdn-br0
Host #3
sdn-br0
sdn-br1
Host #2
sdn-br0
sdn-br1
Container ‘app’
Container ‘db’
Container ‘app’
Container ‘db’
Container ‘app’
Container ‘app’
Container ‘db’
Container ‘app’
Container ‘db’

Instanciate containers without pre-built interfaces to avoid
plugging containers to native docker0 bridge

Use “docker run … ip = ?? -net=none” switch in “docker run” calls

Weave

Weave

Weave can traverse firewalls and operate in partially connected
networks.

Traffic can be encrypted, allowing hosts to be connected across
an untrusted network.

With weave you can easily construct applications consisting of
multiple containers, running anywhere.

Weave works alongside Docker's existing (single host)
networking capabilities, so these can continue to be used by
containers.

Weave Installation
Ensure you are running Linux (kernel 3.8 or later) and have
Docker (version 1.3.1 or later) installed

Weave Setup

Performance Comparison – self test

Used qperf to test performance

Physical Host
− tcp_bw:
− bw = 117 MB/sec

OVS with gre tunnel (Default)
− tcp_bw:
− bw = 81.9 KB/sec

OVS with gre tunnel (ethtool -K ovsBridge tx off sg off tso off) ( technique for increasing
outbound throughput of high-bandwidth network connections by reducing CPU overhead , not
recommended for production)
− tcp_bw:
− bw = 104 MB/sec

Weave(Latest version) based on VXLAN (Older versions based on pcap had lower
performance)
− tcp_bw:
− bw = 92.3 MB/sec

Bonus: Alternate design - OpenStack Neutron paradigms

Alternate design, based on OpenStack Neutron paradigms

All VMs/containers of a same tenant network are segregated
inside a dedicated, local VLAN of a shared unique bridge

Full-mesh of GRE tunnels between all hosts

On each host, local mapping between a local tenant network
VLAN and its GRE identifier shared across all hosts