What is OpenStack DVR?

1. What is OpenStack DVR
(Distribute Virtual Router)
내것인듯 내것아닌 내것같은 Neutron/DVR
Yongyoon SHIN. SDN section, ETRI

2. Let’s Talk
• OpenStack and Distribute Virtual Router
– OpenStack
– What is DVR
• DVR Case
– East/West Traffic
– SNAT
– Floating IP

3. OpenStack Naming
• OpenStack Version Naming
– Austin: Austin, Texas
– Bexar: San Antonio, Texas
– Cactus: Cactus, Texas
– Diablo: Santa Clara, CA
– Essex: Boston, Massachusetts
– Folsom: San Francisco, CA
– Grizzly: San Diego, CA
– Havana: Portland, Oregon
– Icehouse: Hong Kong
– Juno: Atlanta, Georgia
– Kilo: Paris, France
– Liberty: Vancouver, British Columbia
– M*: Tokyo, Japan (https://wiki.openstack.org/wiki/Release_Naming/M_Proposals)
– N*: Austin, Texas
– O*: Barcelona, Spain
Quantum
Neutron
Nova Network
* 한국은 Vancouver summit에서 2019년 summit 신청 (Naming 순서는 ‘W’)
.. 2019년에도 OpenStack이 지금처럼 번성할까? ..

4. OpenStack Release
• Main Schedule (Neutron)

5. OpenStack Source
• github 사용
• Neutron
– 현재까지 업데이트 중
• 2015.06.11. 현재 8일전 update

6. 3 Main Services in Neutron
• East-West Communication
– Data Center 내부, VM 사이의 IP Traffic 처리
• Floating IP (aka DNAT: Destination NAT)
– Public Network을 통하여 직접 VM에 접근 가능할 수 있도록 VM에
Public IP를 제공
• Shard IP (aka SNAT: Source NAT)
– Data Center 내부 VM들이 외부 IP를 공용으로 사용하여 public Network
에 접속할 수 있도록 함

7. Main Idea of DVR
• Network Node의 병목현상 해소
– L3 Agent 기능을 모든 Compute Node에 분산
• 네트워크 노드의 단일 장애점 및 트래픽 집중문제 해결
Compute Node A
VM VM VM
VM VM VM
10.0.10.0/24
129.254.172.0/24
129.254.172.0/24
20.0.20.0/24
DVR
Public Network
(129.254.172.0/24)
Compute Node B
VM VM VM
VM VM VM
10.0.10.0/24
129.254.172.0/24
129.254.172.0/24
20.0.20.0/24
DVR

8. DVR vs without DVR
• Without DVR
• With DVR
Compute Node A
VM
Network Node Compute Node B
br-int
br-tun
VM
br-int
br-tun
br-int
br-tun
eth eth eth
VR
Compute Node A
VM
Compute Node B
br-int
br-tun
VM
br-int
br-tun
eth eth
VR VR

9. OpenStack DVR
• OpenStack major work
– Juno version release (~ing)
• 리눅스의 Name Space 사용

10. OpenStack DVR
• Network/Compute Node NameSpace
Network Node Compute Node
VM01
(Network A)
VM02
(Network B)
br-int
br-ex br-tun
dnsmasq dnsmasq
br-int
br-ex br-tun
dhcp NameSpacedhcp NameSpace
Router NameSpace Router NameSpace
SNAT NameSpace
patch-tun
patch-int
patch-tun
patch-int
eth0 eth0 vxlan-xxxx vxlan-yyyyvxlan-xxxx vxlan-yyyy
FIP NameSpace
qvoVM1 qvoVM2
qrouter-xxx
fip-xxx
qr-aaa qr-bbb
rfp-xxx
fpr-xxx
fg-yyy
qdhcp-kkk qdhcp-qqq
tap123abc tap456def
int-br-ex
phy-br-ex
qrouter-xxx
snat-xxx
qr-aaa qr-bbbsg-123aaa sg-456bbb
qg-789abc
External

11. Inter subnet routing East-West
• ML2 plugin
• l2pop mechanism driver enabled
• Openvswitch mechanism driver enavled
• ovs agent installed on all of compute nodes
• Tunneling (only use VXLAN / GRE)
– Kilo version support VLAN

12. Inter subnet routing East-West

14. Inter subnet routing East-West
• Compute Node Router
– Compute Node A
– Compute Node B
• Controller Node
- IP: 192.168.56.20
- DevStack All_in_one
• Compute Node
- IP: 192.168.56.25
- DevStack Compute
• Network Node
- IP: 192.168.56.30
- DevStack Network

15. Inter subnet routing East-West
• Router namespace created on both nodes
– Same interfaces, MAC, IP

16. Inter subnet routing East-West
• ARP Table
– Compute Node A
• vm01(100.0.0.2) / vm03(100.0.20.2)
• ARP Table: vm01/vm02/vm03
– Compute Node B
• vm02(100.0.10.2)
• ARP Table: vm01/vm02/vm03

17. Inter subnet routing East-West
• Flows
Compute Node A
VM
Compute Node B
br-int
br-tun
VM
br-int
br-tun
eth eth
VR VR
① ②
③
④
⑤
⑥
⑦
⑧
src MAC dst MAC src IP dest IP
Blue VM Blue Router Interface Blue VM Red VM

18. Inter subnet routing East-West
• Flows
– ARP Tables
Compute Node A
VM
Compute Node B
br-int
br-tun
VM
br-int
br-tun
eth eth
VR VR
①
②
③
④
⑤
⑥
⑦
⑧
src MAC dst MAC src IP dest IP
Red Router Interface Red VM Blue VM Red VM

19. Inter subnet routing East-West
• Flows
– Matched Red VM MAC
Compute Node A
VM
Compute Node B
br-int
br-tun
VM
br-int
br-tun
eth eth
VR VR
① ②
③
④
⑤
⑥
⑦
⑧
src MAC dst MAC src IP dest IP
Red Router Interface Red VM Blue VM Red VM

20. Inter subnet routing East-West
• Flows
Compute Node A
VM
Compute Node B
br-int
br-tun
VM
br-int
br-tun
eth eth
VR VR
① ②
③
④
⑤
⑥
⑦
⑧
src MAC dst MAC src IP dest IP
Compute Node A MAC Red VM Blue VM Red VM

21. Inter subnet routing East-West
• Flows
– Encapsulation via VXLAN/GRE tunnel
Compute Node A
VM
Compute Node B
br-int
br-tun
VM
br-int
br-tun
eth eth
VR VR
① ②
③
④
⑤ VXLAN / GRE
⑥
⑦
⑧
src MAC dst MAC src IP dest IP
Compute Node A MAC Red VM Blue VM Red VM

22. Inter subnet routing East-West
• Flows
– Tunnel encapsulation is stripped, tagged with a local VLAN
Compute Node A
VM
Compute Node B
br-int
br-tun
VM
br-int
br-tun
eth eth
VR VR
① ②
③
④
⑤
⑥
⑦
⑧
src MAC dst MAC src IP dest IP
Compute Node A MAC Red VM Blue VM Red VM

23. Inter subnet routing East-West
• Flows
– Traffic matched remote host MAC
Compute Node A
VM
Compute Node B
br-int
br-tun
VM
br-int
br-tun
eth eth
VR VR
① ②
③
④
⑤
⑥
⑦
⑧
src MAC dst MAC src IP dest IP
Compute Node A MAC Red VM Blue VM Red VM

24. Inter subnet routing East-West
• Flows
– Traffic matched remote host MAC, Red Network VLAN tag
– VLAN tag stripped
Compute Node A
VM
Compute Node B
br-int
br-tun
VM
br-int
br-tun
eth eth
VR VR
① ②
③
④
⑤
⑥
⑦
⑧
src MAC dst MAC src IP dest IP
Red Router Interface Red VM Blue VM Red VM

25. Inter subnet routing East-West
• East/West Traffic Flow summary
Compute Node B
VM03
(Network A)
br-int
br-ex br-tun
Router NameSpace
patch-tun
patch-int
eth0 vxlan-xxxx vxlan-yyyy
qvoVM3
qrouter-xxx
qr-aaa qr-bbb
Compute Node A
VM01
(Network A)
VM02
(Network B)
br-int
br-ex br-tun
Router NameSpace
patch-tun
patch-int
eth0 vxlan-xxxx vxlan-yyyy
FIP NameSpace
qvoVM1 qvoVM2
qrouter-xxx
fip-xxx
qr-aaa qr-bbb
rfp-xxx
fpr-xxx
fg-yyy

26. Distribute Virtual Routing - SNAT
• SNAT : Source NAT
– Floating IP를 설정하지 않은 VM이 외부 통신
– 라우터는 외부 네트워크에서 단일 IP를 할당 받음 (VM은 IP 공유)
• Still Centralized
– 현재 완전 분산 형태로 진행 중 (neutron blueprint)
• SNAT를 사용하지 않아도 되는 경우 발생 (VM이 외부망을 사용할 필요 없음)
Compute Node A
VM
Network Node
br-int
br-tun
br-int
br-tun
eth1 eth0
VR VR(SNAT)
eth1
br-ex
qr1 qr2 sg1 sg2 qg

27. Distribute Virtual Routing - SNAT

29. Distribute Virtual Routing - SNAT
• Network node
– Neutron l3_agent.ini
– Network node Interface

30. Distribute Virtual Routing - SNAT
• Network Node IP Rule

31. Distribute Virtual Routing - SNAT
• Compute Node A – IP Rule
• Compute Node B – IP Rule
모든 컴퓨트 노드의 라우팅 룰은
라우터와 연결된 모든 서브넷
에 대한 내용
dvr_snat 노드의 SNAT
네임스페이스에 대한 내용

32. Distribute Virtual Routing - SNAT
• Flows
Compute Node A
VM
Network Node
br-int
br-tun
br-int
br-tun
eth1 eth0
VR VR(SNAT)
① ②
③
④
⑤
⑥
⑦
src MAC dst MAC src IP dest IP
Blue VM Blue qr Interface Blue VM ex) Google IP
eth1
br-ex
qr1 qr2 sg1 sg2 qg
⑧

33. Distribute Virtual Routing - SNAT
• Flows
Compute Node A
VM
Network Node
br-int
br-tun
br-int
br-tun
eth1 eth0
VR VR(SNAT)
①
②
③
④
⑤
⑥
⑦
src MAC dst MAC src IP dest IP
Blue VM Blue qr Interface Blue VM ex) Google IP
eth1
br-ex
qr1 qr2 sg1 sg2 qg
⑧

34. Distribute Virtual Routing - SNAT
• Flows
Compute Node A
VM
Network Node
br-int
br-tun
br-int
br-tun
eth1 eth0
VR VR(SNAT)
① ②
③
④
⑤
⑥
⑦
src MAC dst MAC src IP dest IP
Blue qr Interface Blue sg Interface Blue VM ex) Google IP
eth1
br-ex
qr1 qr2 sg1 sg2 qg
⑧

35. Distribute Virtual Routing - SNAT
• Flows
Compute Node A
VM
Network Node
br-int
br-tun
br-int
br-tun
eth1 eth0
VR VR(SNAT)
① ②
③
④
⑤
⑥
⑦
src MAC dst MAC src IP dest IP
Compute Node A MAC Blue sg Interface Blue VM ex) Google IP
eth1
br-ex
qr1 qr2 sg1 sg2 qg
⑧

36. Distribute Virtual Routing - SNAT
• Flows
Compute Node A
VM
Network Node
br-int
br-tun
br-int
br-tun
eth1 eth0
VR VR(SNAT)
① ②
③
④
⑤
⑥
⑦
src MAC dst MAC src IP dest IP
Blue qr Interface Blue sg Interface Blue VM ex) Google IP
eth1
br-ex
qr1 qr2 sg1 sg2 qg
⑧

37. Distribute Virtual Routing - SNAT
• Flows
Compute Node A
VM
Network Node
br-int
br-tun
br-int
br-tun
eth1 eth0
VR VR(SNAT)
① ②
③
④
⑤
⑥
⑦
src MAC dst MAC src IP dest IP
qg Interface Default GW MAC qg Interface ex) Google IP
eth1
br-ex
qr1 qr2 sg1 sg2 qg
⑧

38. Distribute Virtual Routing - SNAT
• SNAT Traffic Flow summary
Network Node Compute Node
VM01
(Network A)
VM02
(Network B)
br-int
br-ex br-tun
dnsmasq dnsmasq
br-int
br-ex br-tun
dhcp NameSpacedhcp NameSpace
Router NameSpace Router NameSpace
SNAT NameSpace
patch-tun
patch-int
patch-tun
patch-int
eth0 eth0 vxlan-xxxx vxlan-yyyyvxlan-xxxx vxlan-yyyy
FIP NameSpace
qvoVM1 qvoVM2
qrouter-xxx
fip-xxx
qr-aaa qr-bbb
rfp-xxx
fpr-xxx
fg-yyy
qdhcp-kkk qdhcp-qqq
tap123abc tap456def
int-br-ex
phy-br-ex
qrouter-xxx
snat-xxx
qr-aaa qr-bbbsg-123aaa sg-456bbb
qg-789abc

39. Distribute Virtual Routing – Floating IP
• Compute Node의 FIP(Floating IP) namespace
– rfp: Router NameSpace에 생성
– fpr: FIP NameSpace에 생성
Compute Node
VM01
(Network A)
VM02
(Network B)
br-int
br-ex br-tun
Router NameSpace
patch-tun
patch-int
eth0 vxlan-xxxx vxlan-yyyy
FIP NameSpace
qvoVM1 qvoVM2
qrouter-xxx
fip-xxx
qr-aaa qr-bbb
rfp-xxx
fpr-xxx
fg-yyy
Veth pair
Compute Node A
VM
br-int
VR
qr
FIP
br-ex
eth0
rfp - 129.254.171.0/24
fpr - 129.254.171.0/24
fg

40. Distribute Virtual Routing – Floating IP

42. Distribute Virtual Routing – Floating IP
• FIP NameSpace 생성 및 External Network 확인
– Compute Node A
– Compute Node B

43. Distribute Virtual Routing – Floating IP
• veth pare
– Compute Node A
• Router NameSpace
– Compute Node B
• Router NameSpace

44. Distribute Virtual Routing – Floating IP
• Veth pair
– Compute Node A
• FIP NameSpace
• External Device: fg
– Compute Node B
• FIP NameSpace
• External Device: fg

45. Distribute Virtual Routing – Floating IP
• Routing Rule
– Compute Node A
– Compute Node B

46. Distribute Virtual Routing – Floating IP
• Floating IP Packet Tracking
– Compute Node A
– Compute Node B

47. Distribute Virtual Routing – Floating IP
Compute Node A
VM
br-int
VR
① qr
FIP
br-ex
eth0
rfp - 129.254.171.0/24
fpr - 129.254.171.0/24
fg
src MAC dst MAC src IP dest IP
Blue VM Blue qr Interface Blue VM fixed IP ex) Google IP
④
⑤
②
③

48. Distribute Virtual Routing – Floating IP
Compute Node A
VM
br-int
VR
① qr
FIP
br-ex
eth0
rfp - 129.254.171.0/24
fpr - 129.254.171.0/24
fg
src MAC dst MAC src IP dest IP
Blue VM Blue qr Interface Blue VM fixed IP ex) Google IP
④
⑤
②
③

49. Distribute Virtual Routing – Floating IP
Compute Node A
VM
br-int
VR
①
②
qr
FIP
br-ex
eth0
rfp - 129.254.171.0/24
fpr - 129.254.171.0/24
fg
src MAC dst MAC src IP dest IP
Blue VM Blue qr Interface Blue VM fixed IP ex) Google IP
③
④
⑤

50. Distribute Virtual Routing – Floating IP
Compute Node A
VM
br-int
VR
① qr
FIP
br-ex
eth0
rfp - 129.254.171.0/24
fpr - 129.254.171.0/24
fg
src MAC dst MAC src IP dest IP
rfp fpr Blue VM Floating IP ex) Google IP
④
⑤
②
③

51. Distribute Virtual Routing – Floating IP
Compute Node A
VM
br-int
VR
① qr
FIP
br-ex
eth0
rfp - 129.254.171.0/24
fpr - 129.254.171.0/24
fg
src MAC dst MAC src IP dest IP
fg default GW Blue VM Floating IP ex) Google IP
⑤
④
②
③

52. Distribute Virtual Routing - Floating IP
• Floating-IP Traffic Flow summary
Network Node Compute Node
VM01
(Network A)
VM02
(Network B)
br-int
br-ex br-tun
dnsmasq dnsmasq
br-int
br-ex br-tun
dhcp NameSpacedhcp NameSpace
Router NameSpace Router NameSpace
SNAT NameSpace
patch-tun
patch-int
patch-tun
patch-int
eth0 eth0 vxlan-xxxx vxlan-yyyyvxlan-xxxx vxlan-yyyy
FIP NameSpace
qvoVM1 qvoVM2
qrouter-xxx
fip-xxx
qr-aaa qr-bbb
rfp-xxx
fpr-xxx
fg-yyy
qdhcp-kkk qdhcp-qqq
tap123abc tap456def
int-br-ex
phy-br-ex
qrouter-xxx
snat-xxx
qr-aaa qr-bbbsg-123aaa sg-456bbb
qg-789abc

53. OpenStack DVR Summary
특성 Forwarding Mechanism
Local
• Source/Destination IP는 동일한 Subnet에 속함
• VM들은 동일한 노드에 배열
br-int는 관련된 MAC Address를 학습하고 로컬 트래픽을 전달
Remote
• Source/Destination IP는 동일한 Subnet에 속함
• VM들은 서로 다른 노드에 배열
Segmentation type(VXLAN/GRE)에 따라 전달
East/West • Source/Destination IP는 다른 Subnet에 속함 컴퓨트 노드의 라우터 인터페이스에 의해 전달
SNAT
(Source NAT)
• Destination IP는 컴퓨트 노드가 알고 있는 Subnet
에 속하지 않음
• VM에 Floating IP 설정이 되어 있지 않음
dvr_snat 노드 상의 SNAT 네임스페이스에 존재하는 IP 매칭에
따라 전달
FIP
(Floating IP)
• Destination IP는 컴퓨트 노드가 알고 있는 Subnet
에 속하지 않음
• VM에 Floating IP 설정
컴퓨트 노드의 FIP 네임스페이스와 연결된 컴퓨트 노드 라우터에 전
달

54. OpenStack에서는 앞으로…
• Juno 배포 이후 업데이트 예정
– L3 고가용성 지원
• HA 지원 / Cascading 관련 내용 Candidate
– Security Groups 개선
• VM의 증가로 인하여 iptables의 필터링 규칙 폭증 -> 패킷 필터링 성능 저하
• Kilo 배포 이후 추가 사항
– VPNaaS (Virtual Private Network as a Service) 지원
– VLAN 지원
– IPv6 지원
– 분산 DHCP 서비스
– 분산 SNAT 서비스

55. Yongyoon. SHIN (SDN Research Section, ETRI)
Email: uni2u@etri.re.kr
http://uni2u.meximas.com
Thank you