SlideShare uma empresa Scribd logo
1 de 22
Baixar para ler offline
How VXLAN works on Linux
Basic mechanism and
Application to OpenStack and Docker
]中井悦司 / Etsuji Nakai
Senior Solution Architect
and Cloud Evangelist
Red Hat K.K
v1.1 2015/07/09
2
How VXLAN works on Linux
$ who am i
 中井悦司 / Etsuji Nakai
– Twitter @enakai00
– Senior Solution Architect and
Cloud Evangelist at Red Hat.
– The author of some OpenStack books.
3
How VXLAN works on Linux
Contents
 VXLAN basics
 OpenStack Neutron OVS Plugin
 VTEP implementation with Flannel
 References
VXLAN basics
5
How VXLAN works on Linux
The objective of VXLAN
 Creating virtual L2 network over physical L3 network.
VXLAN
Switch
VXLAN
Switch
VXLAN
Switch
Tokyo Osaka Fukuoka
10.1.0.0/16
10.1.1.0 10.1.2.0 10.1.3.0
Physical view
Logical view
from servers
6
How VXLAN works on Linux
Packet encapsulation with VXLAN header
 VXLAN encapsulates L2 packet inside L3 packet.
VXLAN
Switch
VXLAN
Switch
Tokyo Osaka
Dest Address
yy.yy.yy.yy
Original
Packet
Source Address
xx.xx.xx.xx
Original
Packet
VXLAN Header
xx.xx.xx.xx yy.yy.yy.yy
Original
Packet
7
How VXLAN works on Linux
8
How VXLAN works on Linux
The fundamental problem of L2 over L3
 How to find the correct location of packet destination?
How did you know that
the destination is in Osaka!?
VXLAN
Switch
VXLAN
Switch
Tokyo Osaka
Dest Address
yy.yy.yy.yy
Original
Packet
Source Address
xx.xx.xx.xx
Original
Packet
VXLAN Header
xx.xx.xx.xx yy.yy.yy.yy
Original
Packet
9
How VXLAN works on Linux
ARP resolution on L2 layer
 VXLAN switches need to emulate the ARP resolution
mechanism.
IP  10.1.2.0
MAC zz:zz:zz:zz:zz:zz
① ARP Request
"What's the MAC
for IP 10.1.2.0?"
② ARP Reply
"zz:zz:zz:zz:zz:zz"
Dest IP
10.1.2.0
Source IP
10.1.1.0
Payload
Dest MAC
zz:zz:zz:...
Source MAC
xx:xx:xx:...
L3 headerL2 header
IP  10.1.1.0
MAC xx:xx:xx:xx:xx:xx
④ Send L2 packet to
"zz:zz:zz:zz:zz:zz"
③ Port <-> MAC association
is recorded in MAC table
10
How VXLAN works on Linux
Additional features for L2 over L3
 Packet encapsulation is not enough for L2 over L3. VXLAN switches need to
implement the following features.
– ARP resolution: Need to reply to ARP request from local servers without
broadcasting the ARP packet.
– Destination search : Need to find the destination location corresponding to the
destination MAC.
 The VXLAN endpoint providing these features is referred as "VTEP".
ARP Reply
「zz:zz:zz:zz:zz:zz」
Dest "zz:zz:zz:zz:zz:zz" is
located in Osaka.VXLAN
Switch
Tokyo
xx.xx.xx.xx
① ARP Request
"What's the MAC
for IP 10.1.2.0?"
④ Send L2 packet to
"zz:zz:zz:zz:zz:zz"
11
How VXLAN works on Linux
12
How VXLAN works on Linux
Variations of VTEP implementation
 To implement VTEP features, there must be some mechanism to share the
tuple (MAC, IP Address, Location) of all servers.
 The followings are some variations of VTEP implementation.
– Exchange MAC/IP information using L3 multicasting among switches.
– Use SDN controller as a central MAC/IP database.
– Use local agent and virtual VXLAN switch running on Linux servers.
OpenStack Neutron OVS Plugin
14
How VXLAN works on Linux
ML2 l2population driver
 In the case of OpenStack Neutron OVS plugin, VXLAN encapsulation is done on
the local Open vSwitch on compute nodes.
– MAC/IP information is sent by L2 agent and populated by l2population ML2 driver.
– The l2population driver populates the following entries in OVS.
• FDB (forwarding database): a lookup table to find a destination node
corresponding to the dest MAC address.
• Flowtable entries for replying to ARP requests from local VMs.
VM
OVS (br-int)
VM
l2population
driver
Messaging server
(RabbitMQ)
VM
OVS (br-int)
VM
l2population
driver
L2 Agent L2 Agent
① Attaching
new VM
② Send MAC/IP
information
③ Populate flow
table in OVS
15
How VXLAN works on Linux
 Reference : ML2 – Address Population
– http://assafmuller.com/2014/02/23/ml2-address-population/
VTEP implementation
with Flannel
17
How VXLAN works on Linux
Overlay network with Flannel
 Flannel is a opensource tool to create overlay network for Docker containers.
It's often used with Kubernetes.
– It uses Linux kernel's native VXLAN devices for packet encapsulation.
– Flannel daemon dynamically populates FDB and ARP table according to the
kernel requests via the "L2/L3 MISS" notification mechanism.
• The mechanism is originally named as "DOVE extensions"
• https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
id=e4f67addf158f98f8197e08974966b18480dc751
– The IP/MAC information is shared with the backend KVS (etcd).
etcd
Physical network
192.168.122.0/24
Minion
flannel.1
Minion
flannel.1
Internal network for container communication
10.1.0.0/16
Minion
flannel.1
VXLAN device
18
How VXLAN works on Linux
Kernel's DOVE extensions
 You can use the native VXLAN device with the current Linux kernel.
– You don't necessarily need OVS for using VXLAN.
– It's just like using the traditional VLAN device with Linux :)
 VTEP features are implemented with a userland agent via "L2/L3 MISS"
notification mechanism. (The notification is sent via netlink.)
– L3MISS
• The kernel asks the agent to populate the local ARP table when necessary
instead of broadcasting the ARP request packet.
– L2MISS
• The kernel asks the agent to populate FDB when necessary.
# ip -d l show flannel.1
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 82:ce:d5:09:06:2c brd ff:ff:ff:ff:ff:ff promiscuity 0
vxlan id 1 local 192.168.122.101 dev eth0 srcport 0 0 dstport 8472 proxy l2miss ageing 300
# bridge fdb show dev flannel.1
56:e1:c1:d6:b7:51 dst 192.168.122.102 self
# cat /proc/sys/net/ipv4/neigh/flannel.1/app_solicit
3
19
How VXLAN works on Linux
 Reference: Kernel patch - add DOVE extensions for VXLAN
– https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
id=e4f67addf158f98f8197e08974966b18480dc751
References
21
How VXLAN works on Linux
References
 ML2 – Address Population
– http://assafmuller.com/2014/02/23/ml2-address-population/
 Kernel patch: add DOVE extensions for VXLAN
– https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
id=e4f67addf158f98f8197e08974966b18480dc751
 FlannelのVXLANバックエンドの仕組み
– http://enakai00.hatenablog.com/entry/2015/04/02/173739
EMPOWER PEOPLE,
EMPOWER ENTERPRISE,
OPEN INNOVATION.

Mais conteúdo relacionado

Mais procurados

The Basic Introduction of Open vSwitch
The Basic Introduction of Open vSwitchThe Basic Introduction of Open vSwitch
The Basic Introduction of Open vSwitchTe-Yen Liu
 
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking ShapeBlue
 
Open vSwitch Offload: Conntrack and the Upstream Kernel
Open vSwitch Offload: Conntrack and the Upstream KernelOpen vSwitch Offload: Conntrack and the Upstream Kernel
Open vSwitch Offload: Conntrack and the Upstream KernelNetronome
 
EBPF and Linux Networking
EBPF and Linux NetworkingEBPF and Linux Networking
EBPF and Linux NetworkingPLUMgrid
 
OVN operationalization at scale at eBay
OVN operationalization at scale at eBayOVN operationalization at scale at eBay
OVN operationalization at scale at eBayAliasgar Ginwala
 
Tutorial: Using GoBGP as an IXP connecting router
Tutorial: Using GoBGP as an IXP connecting routerTutorial: Using GoBGP as an IXP connecting router
Tutorial: Using GoBGP as an IXP connecting routerShu Sugimoto
 
IP Virtual Server(IPVS) 101
IP Virtual Server(IPVS) 101IP Virtual Server(IPVS) 101
IP Virtual Server(IPVS) 101HungWei Chiu
 
[KubeCon EU 2022] Running containerd and k3s on macOS
[KubeCon EU 2022] Running containerd and k3s on macOS[KubeCon EU 2022] Running containerd and k3s on macOS
[KubeCon EU 2022] Running containerd and k3s on macOSAkihiro Suda
 
Open vSwitch - Stateful Connection Tracking & Stateful NAT
Open vSwitch - Stateful Connection Tracking & Stateful NATOpen vSwitch - Stateful Connection Tracking & Stateful NAT
Open vSwitch - Stateful Connection Tracking & Stateful NATThomas Graf
 
OVN 設定サンプル | OVN config example 2015/12/27
OVN 設定サンプル | OVN config example 2015/12/27OVN 設定サンプル | OVN config example 2015/12/27
OVN 設定サンプル | OVN config example 2015/12/27Kentaro Ebisawa
 
OVN - Basics and deep dive
OVN - Basics and deep diveOVN - Basics and deep dive
OVN - Basics and deep diveTrinath Somanchi
 
OpenStack Networking
OpenStack NetworkingOpenStack Networking
OpenStack NetworkingIlya Shakhat
 
ML2/OVN アーキテクチャ概観
ML2/OVN アーキテクチャ概観ML2/OVN アーキテクチャ概観
ML2/OVN アーキテクチャ概観Yamato Tanaka
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringScyllaDB
 
Kubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveKubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveMichal Rostecki
 
Docker Networking with New Ipvlan and Macvlan Drivers
Docker Networking with New Ipvlan and Macvlan DriversDocker Networking with New Ipvlan and Macvlan Drivers
Docker Networking with New Ipvlan and Macvlan DriversBrent Salisbury
 

Mais procurados (20)

The Basic Introduction of Open vSwitch
The Basic Introduction of Open vSwitchThe Basic Introduction of Open vSwitch
The Basic Introduction of Open vSwitch
 
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
Deploying CloudStack and Ceph with flexible VXLAN and BGP networking
 
Open vSwitch Offload: Conntrack and the Upstream Kernel
Open vSwitch Offload: Conntrack and the Upstream KernelOpen vSwitch Offload: Conntrack and the Upstream Kernel
Open vSwitch Offload: Conntrack and the Upstream Kernel
 
Linux Network Stack
Linux Network StackLinux Network Stack
Linux Network Stack
 
EBPF and Linux Networking
EBPF and Linux NetworkingEBPF and Linux Networking
EBPF and Linux Networking
 
OVN operationalization at scale at eBay
OVN operationalization at scale at eBayOVN operationalization at scale at eBay
OVN operationalization at scale at eBay
 
Tutorial: Using GoBGP as an IXP connecting router
Tutorial: Using GoBGP as an IXP connecting routerTutorial: Using GoBGP as an IXP connecting router
Tutorial: Using GoBGP as an IXP connecting router
 
IP Virtual Server(IPVS) 101
IP Virtual Server(IPVS) 101IP Virtual Server(IPVS) 101
IP Virtual Server(IPVS) 101
 
macvlan and ipvlan
macvlan and ipvlanmacvlan and ipvlan
macvlan and ipvlan
 
[KubeCon EU 2022] Running containerd and k3s on macOS
[KubeCon EU 2022] Running containerd and k3s on macOS[KubeCon EU 2022] Running containerd and k3s on macOS
[KubeCon EU 2022] Running containerd and k3s on macOS
 
Deploying IPv6 on OpenStack
Deploying IPv6 on OpenStackDeploying IPv6 on OpenStack
Deploying IPv6 on OpenStack
 
Meetup 23 - 02 - OVN - The future of networking in OpenStack
Meetup 23 - 02 - OVN - The future of networking in OpenStackMeetup 23 - 02 - OVN - The future of networking in OpenStack
Meetup 23 - 02 - OVN - The future of networking in OpenStack
 
Open vSwitch - Stateful Connection Tracking & Stateful NAT
Open vSwitch - Stateful Connection Tracking & Stateful NATOpen vSwitch - Stateful Connection Tracking & Stateful NAT
Open vSwitch - Stateful Connection Tracking & Stateful NAT
 
OVN 設定サンプル | OVN config example 2015/12/27
OVN 設定サンプル | OVN config example 2015/12/27OVN 設定サンプル | OVN config example 2015/12/27
OVN 設定サンプル | OVN config example 2015/12/27
 
OVN - Basics and deep dive
OVN - Basics and deep diveOVN - Basics and deep dive
OVN - Basics and deep dive
 
OpenStack Networking
OpenStack NetworkingOpenStack Networking
OpenStack Networking
 
ML2/OVN アーキテクチャ概観
ML2/OVN アーキテクチャ概観ML2/OVN アーキテクチャ概観
ML2/OVN アーキテクチャ概観
 
High-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uringHigh-Performance Networking Using eBPF, XDP, and io_uring
High-Performance Networking Using eBPF, XDP, and io_uring
 
Kubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep DiveKubernetes Networking with Cilium - Deep Dive
Kubernetes Networking with Cilium - Deep Dive
 
Docker Networking with New Ipvlan and Macvlan Drivers
Docker Networking with New Ipvlan and Macvlan DriversDocker Networking with New Ipvlan and Macvlan Drivers
Docker Networking with New Ipvlan and Macvlan Drivers
 

Semelhante a How VXLAN works on Linux

Ovn vancouver
Ovn vancouverOvn vancouver
Ovn vancouverMason Mei
 
Openstack Neutron Insights
Openstack Neutron InsightsOpenstack Neutron Insights
Openstack Neutron InsightsAtul Pandey
 
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...Jim St. Leger
 
VXLAN Distributed Service Node
VXLAN Distributed Service NodeVXLAN Distributed Service Node
VXLAN Distributed Service NodeDavid Lapsley
 
Network stack personality in Android phone - netdev 2.2
Network stack personality in Android phone - netdev 2.2Network stack personality in Android phone - netdev 2.2
Network stack personality in Android phone - netdev 2.2Hajime Tazaki
 
LinuxCon 2015 Stateful NAT with OVS
LinuxCon 2015 Stateful NAT with OVSLinuxCon 2015 Stateful NAT with OVS
LinuxCon 2015 Stateful NAT with OVSThomas Graf
 
Scaling the Container Dataplane
Scaling the Container Dataplane Scaling the Container Dataplane
Scaling the Container Dataplane Michelle Holley
 
neutron_icehouse_update
neutron_icehouse_updateneutron_icehouse_update
neutron_icehouse_updateAkihiro Motoki
 
Scaling OpenStack Networking Beyond 4000 Nodes with Dragonflow - Eshed Gal-Or...
Scaling OpenStack Networking Beyond 4000 Nodes with Dragonflow - Eshed Gal-Or...Scaling OpenStack Networking Beyond 4000 Nodes with Dragonflow - Eshed Gal-Or...
Scaling OpenStack Networking Beyond 4000 Nodes with Dragonflow - Eshed Gal-Or...Cloud Native Day Tel Aviv
 
Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1
Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1
Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1Etsuji Nakai
 
PLNOG15: Is there something less complicated than connecting two LAN networks...
PLNOG15: Is there something less complicated than connecting two LAN networks...PLNOG15: Is there something less complicated than connecting two LAN networks...
PLNOG15: Is there something less complicated than connecting two LAN networks...PROIDEA
 
Calico with open stack
Calico with open stackCalico with open stack
Calico with open stackD.Rajesh Kumar
 
PLNOG16: VXLAN Gateway, efektywny sposób połączenia świata wirtualnego z fizy...
PLNOG16: VXLAN Gateway, efektywny sposób połączenia świata wirtualnego z fizy...PLNOG16: VXLAN Gateway, efektywny sposób połączenia świata wirtualnego z fizy...
PLNOG16: VXLAN Gateway, efektywny sposób połączenia świata wirtualnego z fizy...PROIDEA
 
Sdn dell lab report v2
Sdn dell lab report v2Sdn dell lab report v2
Sdn dell lab report v2Oded Rotter
 
Dreamhost deploying dreamcompute at scale
Dreamhost deploying dreamcompute at scaleDreamhost deploying dreamcompute at scale
Dreamhost deploying dreamcompute at scaleCumulus Networks
 
Quantum - Virtual networks for Openstack
Quantum - Virtual networks for OpenstackQuantum - Virtual networks for Openstack
Quantum - Virtual networks for Openstacksalv_orlando
 

Semelhante a How VXLAN works on Linux (20)

rtnetlink
rtnetlinkrtnetlink
rtnetlink
 
Ovn vancouver
Ovn vancouverOvn vancouver
Ovn vancouver
 
Openstack Neutron Insights
Openstack Neutron InsightsOpenstack Neutron Insights
Openstack Neutron Insights
 
NSX-MH
NSX-MHNSX-MH
NSX-MH
 
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
DPDK Summit - 08 Sept 2014 - Futurewei - Jun Xu - Revisit the IP Stack in Lin...
 
VXLAN Distributed Service Node
VXLAN Distributed Service NodeVXLAN Distributed Service Node
VXLAN Distributed Service Node
 
Network stack personality in Android phone - netdev 2.2
Network stack personality in Android phone - netdev 2.2Network stack personality in Android phone - netdev 2.2
Network stack personality in Android phone - netdev 2.2
 
LinuxCon 2015 Stateful NAT with OVS
LinuxCon 2015 Stateful NAT with OVSLinuxCon 2015 Stateful NAT with OVS
LinuxCon 2015 Stateful NAT with OVS
 
Scaling the Container Dataplane
Scaling the Container Dataplane Scaling the Container Dataplane
Scaling the Container Dataplane
 
neutron_icehouse_update
neutron_icehouse_updateneutron_icehouse_update
neutron_icehouse_update
 
Networking in Openstack - Neutron 101
Networking in Openstack - Neutron 101Networking in Openstack - Neutron 101
Networking in Openstack - Neutron 101
 
Scaling OpenStack Networking Beyond 4000 Nodes with Dragonflow - Eshed Gal-Or...
Scaling OpenStack Networking Beyond 4000 Nodes with Dragonflow - Eshed Gal-Or...Scaling OpenStack Networking Beyond 4000 Nodes with Dragonflow - Eshed Gal-Or...
Scaling OpenStack Networking Beyond 4000 Nodes with Dragonflow - Eshed Gal-Or...
 
Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1
Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1
Architecture Overview: Kubernetes with Red Hat Enterprise Linux 7.1
 
The State of Linux Containers
The State of Linux ContainersThe State of Linux Containers
The State of Linux Containers
 
PLNOG15: Is there something less complicated than connecting two LAN networks...
PLNOG15: Is there something less complicated than connecting two LAN networks...PLNOG15: Is there something less complicated than connecting two LAN networks...
PLNOG15: Is there something less complicated than connecting two LAN networks...
 
Calico with open stack
Calico with open stackCalico with open stack
Calico with open stack
 
PLNOG16: VXLAN Gateway, efektywny sposób połączenia świata wirtualnego z fizy...
PLNOG16: VXLAN Gateway, efektywny sposób połączenia świata wirtualnego z fizy...PLNOG16: VXLAN Gateway, efektywny sposób połączenia świata wirtualnego z fizy...
PLNOG16: VXLAN Gateway, efektywny sposób połączenia świata wirtualnego z fizy...
 
Sdn dell lab report v2
Sdn dell lab report v2Sdn dell lab report v2
Sdn dell lab report v2
 
Dreamhost deploying dreamcompute at scale
Dreamhost deploying dreamcompute at scaleDreamhost deploying dreamcompute at scale
Dreamhost deploying dreamcompute at scale
 
Quantum - Virtual networks for Openstack
Quantum - Virtual networks for OpenstackQuantum - Virtual networks for Openstack
Quantum - Virtual networks for Openstack
 

Mais de Etsuji Nakai

「ITエンジニアリングの本質」を考える
「ITエンジニアリングの本質」を考える「ITエンジニアリングの本質」を考える
「ITエンジニアリングの本質」を考えるEtsuji Nakai
 
Googleのインフラ技術に見る基盤標準化とDevOpsの真実
Googleのインフラ技術に見る基盤標準化とDevOpsの真実Googleのインフラ技術に見る基盤標準化とDevOpsの真実
Googleのインフラ技術に見る基盤標準化とDevOpsの真実Etsuji Nakai
 
Introducton to Convolutional Nerural Network with TensorFlow
Introducton to Convolutional Nerural Network with TensorFlowIntroducton to Convolutional Nerural Network with TensorFlow
Introducton to Convolutional Nerural Network with TensorFlowEtsuji Nakai
 
Googleにおける機械学習の活用とクラウドサービス
Googleにおける機械学習の活用とクラウドサービスGoogleにおける機械学習の活用とクラウドサービス
Googleにおける機械学習の活用とクラウドサービスEtsuji Nakai
 
Spannerに関する技術メモ
Spannerに関する技術メモSpannerに関する技術メモ
Spannerに関する技術メモEtsuji Nakai
 
Googleのインフラ技術から考える理想のDevOps
Googleのインフラ技術から考える理想のDevOpsGoogleのインフラ技術から考える理想のDevOps
Googleのインフラ技術から考える理想のDevOpsEtsuji Nakai
 
A Brief History of My English Learning
A Brief History of My English LearningA Brief History of My English Learning
A Brief History of My English LearningEtsuji Nakai
 
TensorFlowプログラミングと分類アルゴリズムの基礎
TensorFlowプログラミングと分類アルゴリズムの基礎TensorFlowプログラミングと分類アルゴリズムの基礎
TensorFlowプログラミングと分類アルゴリズムの基礎Etsuji Nakai
 
TensorFlowによるニューラルネットワーク入門
TensorFlowによるニューラルネットワーク入門TensorFlowによるニューラルネットワーク入門
TensorFlowによるニューラルネットワーク入門Etsuji Nakai
 
Using Kubernetes on Google Container Engine
Using Kubernetes on Google Container EngineUsing Kubernetes on Google Container Engine
Using Kubernetes on Google Container EngineEtsuji Nakai
 
Lecture note on PRML 8.2
Lecture note on PRML 8.2Lecture note on PRML 8.2
Lecture note on PRML 8.2Etsuji Nakai
 
Machine Learning Basics for Web Application Developers
Machine Learning Basics for Web Application DevelopersMachine Learning Basics for Web Application Developers
Machine Learning Basics for Web Application DevelopersEtsuji Nakai
 
Your first TensorFlow programming with Jupyter
Your first TensorFlow programming with JupyterYour first TensorFlow programming with Jupyter
Your first TensorFlow programming with JupyterEtsuji Nakai
 
Deep Q-Network for beginners
Deep Q-Network for beginnersDeep Q-Network for beginners
Deep Q-Network for beginnersEtsuji Nakai
 
TensorFlowで学ぶDQN
TensorFlowで学ぶDQNTensorFlowで学ぶDQN
TensorFlowで学ぶDQNEtsuji Nakai
 
DevOpsにおける組織に固有の事情を どのように整理するべきか
DevOpsにおける組織に固有の事情を どのように整理するべきかDevOpsにおける組織に固有の事情を どのように整理するべきか
DevOpsにおける組織に固有の事情を どのように整理するべきかEtsuji Nakai
 
インタークラウドを実現する技術 〜 デファクトスタンダードからの視点 〜
インタークラウドを実現する技術 〜 デファクトスタンダードからの視点 〜インタークラウドを実現する技術 〜 デファクトスタンダードからの視点 〜
インタークラウドを実現する技術 〜 デファクトスタンダードからの視点 〜Etsuji Nakai
 

Mais de Etsuji Nakai (20)

PRML11.2-11.3
PRML11.2-11.3PRML11.2-11.3
PRML11.2-11.3
 
「ITエンジニアリングの本質」を考える
「ITエンジニアリングの本質」を考える「ITエンジニアリングの本質」を考える
「ITエンジニアリングの本質」を考える
 
Googleのインフラ技術に見る基盤標準化とDevOpsの真実
Googleのインフラ技術に見る基盤標準化とDevOpsの真実Googleのインフラ技術に見る基盤標準化とDevOpsの真実
Googleのインフラ技術に見る基盤標準化とDevOpsの真実
 
Introducton to Convolutional Nerural Network with TensorFlow
Introducton to Convolutional Nerural Network with TensorFlowIntroducton to Convolutional Nerural Network with TensorFlow
Introducton to Convolutional Nerural Network with TensorFlow
 
Googleにおける機械学習の活用とクラウドサービス
Googleにおける機械学習の活用とクラウドサービスGoogleにおける機械学習の活用とクラウドサービス
Googleにおける機械学習の活用とクラウドサービス
 
Spannerに関する技術メモ
Spannerに関する技術メモSpannerに関する技術メモ
Spannerに関する技術メモ
 
Googleのインフラ技術から考える理想のDevOps
Googleのインフラ技術から考える理想のDevOpsGoogleのインフラ技術から考える理想のDevOps
Googleのインフラ技術から考える理想のDevOps
 
A Brief History of My English Learning
A Brief History of My English LearningA Brief History of My English Learning
A Brief History of My English Learning
 
TensorFlowプログラミングと分類アルゴリズムの基礎
TensorFlowプログラミングと分類アルゴリズムの基礎TensorFlowプログラミングと分類アルゴリズムの基礎
TensorFlowプログラミングと分類アルゴリズムの基礎
 
TensorFlowによるニューラルネットワーク入門
TensorFlowによるニューラルネットワーク入門TensorFlowによるニューラルネットワーク入門
TensorFlowによるニューラルネットワーク入門
 
Using Kubernetes on Google Container Engine
Using Kubernetes on Google Container EngineUsing Kubernetes on Google Container Engine
Using Kubernetes on Google Container Engine
 
Lecture note on PRML 8.2
Lecture note on PRML 8.2Lecture note on PRML 8.2
Lecture note on PRML 8.2
 
Machine Learning Basics for Web Application Developers
Machine Learning Basics for Web Application DevelopersMachine Learning Basics for Web Application Developers
Machine Learning Basics for Web Application Developers
 
Your first TensorFlow programming with Jupyter
Your first TensorFlow programming with JupyterYour first TensorFlow programming with Jupyter
Your first TensorFlow programming with Jupyter
 
Deep Q-Network for beginners
Deep Q-Network for beginnersDeep Q-Network for beginners
Deep Q-Network for beginners
 
Life with jupyter
Life with jupyterLife with jupyter
Life with jupyter
 
TensorFlowで学ぶDQN
TensorFlowで学ぶDQNTensorFlowで学ぶDQN
TensorFlowで学ぶDQN
 
DevOpsにおける組織に固有の事情を どのように整理するべきか
DevOpsにおける組織に固有の事情を どのように整理するべきかDevOpsにおける組織に固有の事情を どのように整理するべきか
DevOpsにおける組織に固有の事情を どのように整理するべきか
 
PRML7.2
PRML7.2PRML7.2
PRML7.2
 
インタークラウドを実現する技術 〜 デファクトスタンダードからの視点 〜
インタークラウドを実現する技術 〜 デファクトスタンダードからの視点 〜インタークラウドを実現する技術 〜 デファクトスタンダードからの視点 〜
インタークラウドを実現する技術 〜 デファクトスタンダードからの視点 〜
 

Último

Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdfJamie (Taka) Wang
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.francesco barbera
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 

Último (20)

Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.Digital magic. A small project for controlling smart light bulbs.
Digital magic. A small project for controlling smart light bulbs.
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 

How VXLAN works on Linux

  • 1. How VXLAN works on Linux Basic mechanism and Application to OpenStack and Docker ]中井悦司 / Etsuji Nakai Senior Solution Architect and Cloud Evangelist Red Hat K.K v1.1 2015/07/09
  • 2. 2 How VXLAN works on Linux $ who am i  中井悦司 / Etsuji Nakai – Twitter @enakai00 – Senior Solution Architect and Cloud Evangelist at Red Hat. – The author of some OpenStack books.
  • 3. 3 How VXLAN works on Linux Contents  VXLAN basics  OpenStack Neutron OVS Plugin  VTEP implementation with Flannel  References
  • 5. 5 How VXLAN works on Linux The objective of VXLAN  Creating virtual L2 network over physical L3 network. VXLAN Switch VXLAN Switch VXLAN Switch Tokyo Osaka Fukuoka 10.1.0.0/16 10.1.1.0 10.1.2.0 10.1.3.0 Physical view Logical view from servers
  • 6. 6 How VXLAN works on Linux Packet encapsulation with VXLAN header  VXLAN encapsulates L2 packet inside L3 packet. VXLAN Switch VXLAN Switch Tokyo Osaka Dest Address yy.yy.yy.yy Original Packet Source Address xx.xx.xx.xx Original Packet VXLAN Header xx.xx.xx.xx yy.yy.yy.yy Original Packet
  • 7. 7 How VXLAN works on Linux
  • 8. 8 How VXLAN works on Linux The fundamental problem of L2 over L3  How to find the correct location of packet destination? How did you know that the destination is in Osaka!? VXLAN Switch VXLAN Switch Tokyo Osaka Dest Address yy.yy.yy.yy Original Packet Source Address xx.xx.xx.xx Original Packet VXLAN Header xx.xx.xx.xx yy.yy.yy.yy Original Packet
  • 9. 9 How VXLAN works on Linux ARP resolution on L2 layer  VXLAN switches need to emulate the ARP resolution mechanism. IP  10.1.2.0 MAC zz:zz:zz:zz:zz:zz ① ARP Request "What's the MAC for IP 10.1.2.0?" ② ARP Reply "zz:zz:zz:zz:zz:zz" Dest IP 10.1.2.0 Source IP 10.1.1.0 Payload Dest MAC zz:zz:zz:... Source MAC xx:xx:xx:... L3 headerL2 header IP  10.1.1.0 MAC xx:xx:xx:xx:xx:xx ④ Send L2 packet to "zz:zz:zz:zz:zz:zz" ③ Port <-> MAC association is recorded in MAC table
  • 10. 10 How VXLAN works on Linux Additional features for L2 over L3  Packet encapsulation is not enough for L2 over L3. VXLAN switches need to implement the following features. – ARP resolution: Need to reply to ARP request from local servers without broadcasting the ARP packet. – Destination search : Need to find the destination location corresponding to the destination MAC.  The VXLAN endpoint providing these features is referred as "VTEP". ARP Reply 「zz:zz:zz:zz:zz:zz」 Dest "zz:zz:zz:zz:zz:zz" is located in Osaka.VXLAN Switch Tokyo xx.xx.xx.xx ① ARP Request "What's the MAC for IP 10.1.2.0?" ④ Send L2 packet to "zz:zz:zz:zz:zz:zz"
  • 11. 11 How VXLAN works on Linux
  • 12. 12 How VXLAN works on Linux Variations of VTEP implementation  To implement VTEP features, there must be some mechanism to share the tuple (MAC, IP Address, Location) of all servers.  The followings are some variations of VTEP implementation. – Exchange MAC/IP information using L3 multicasting among switches. – Use SDN controller as a central MAC/IP database. – Use local agent and virtual VXLAN switch running on Linux servers.
  • 14. 14 How VXLAN works on Linux ML2 l2population driver  In the case of OpenStack Neutron OVS plugin, VXLAN encapsulation is done on the local Open vSwitch on compute nodes. – MAC/IP information is sent by L2 agent and populated by l2population ML2 driver. – The l2population driver populates the following entries in OVS. • FDB (forwarding database): a lookup table to find a destination node corresponding to the dest MAC address. • Flowtable entries for replying to ARP requests from local VMs. VM OVS (br-int) VM l2population driver Messaging server (RabbitMQ) VM OVS (br-int) VM l2population driver L2 Agent L2 Agent ① Attaching new VM ② Send MAC/IP information ③ Populate flow table in OVS
  • 15. 15 How VXLAN works on Linux  Reference : ML2 – Address Population – http://assafmuller.com/2014/02/23/ml2-address-population/
  • 17. 17 How VXLAN works on Linux Overlay network with Flannel  Flannel is a opensource tool to create overlay network for Docker containers. It's often used with Kubernetes. – It uses Linux kernel's native VXLAN devices for packet encapsulation. – Flannel daemon dynamically populates FDB and ARP table according to the kernel requests via the "L2/L3 MISS" notification mechanism. • The mechanism is originally named as "DOVE extensions" • https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/? id=e4f67addf158f98f8197e08974966b18480dc751 – The IP/MAC information is shared with the backend KVS (etcd). etcd Physical network 192.168.122.0/24 Minion flannel.1 Minion flannel.1 Internal network for container communication 10.1.0.0/16 Minion flannel.1 VXLAN device
  • 18. 18 How VXLAN works on Linux Kernel's DOVE extensions  You can use the native VXLAN device with the current Linux kernel. – You don't necessarily need OVS for using VXLAN. – It's just like using the traditional VLAN device with Linux :)  VTEP features are implemented with a userland agent via "L2/L3 MISS" notification mechanism. (The notification is sent via netlink.) – L3MISS • The kernel asks the agent to populate the local ARP table when necessary instead of broadcasting the ARP request packet. – L2MISS • The kernel asks the agent to populate FDB when necessary. # ip -d l show flannel.1 3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT link/ether 82:ce:d5:09:06:2c brd ff:ff:ff:ff:ff:ff promiscuity 0 vxlan id 1 local 192.168.122.101 dev eth0 srcport 0 0 dstport 8472 proxy l2miss ageing 300 # bridge fdb show dev flannel.1 56:e1:c1:d6:b7:51 dst 192.168.122.102 self # cat /proc/sys/net/ipv4/neigh/flannel.1/app_solicit 3
  • 19. 19 How VXLAN works on Linux  Reference: Kernel patch - add DOVE extensions for VXLAN – https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/? id=e4f67addf158f98f8197e08974966b18480dc751
  • 21. 21 How VXLAN works on Linux References  ML2 – Address Population – http://assafmuller.com/2014/02/23/ml2-address-population/  Kernel patch: add DOVE extensions for VXLAN – https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/? id=e4f67addf158f98f8197e08974966b18480dc751  FlannelのVXLANバックエンドの仕組み – http://enakai00.hatenablog.com/entry/2015/04/02/173739