Introduction of OpenStack cascading solution

HUAWEI TECHNOLOGIES CO., LTD. Page 1Huawei Confidential
Introduction of
OpenStack cascading solution
Chaoyi Huang ( joehuang@huawei.com )
Hongning Wu ( wuhongning@huawei.com )
Last edited Apr. 27 2015
Last update Jan.12, 2015

• Caution
• All design in this document is the PoC design, the formal open source project of
OpenStack cascading solution is Tricircle (https://github.com/openstack/tricircle,
https://wiki.openstack.org/wiki/Tricircle), where stateless design is applied, and
the stateless design is different what’s described in here. It’s useful to learn how
the design is evolved from the state-ful to stateless.

• Requirement and driving forces
• The answer – OpenStack cascading solution
• Technical diving – OpenStack cascading solution

Motivation
The requirement and driving forces for multi-site/multi-dc clouds integration is as following:
1. Multi-site/multi-dc cloud has co-existence requirement of multi-vendor OpenStack distribution, multi-
OpenStack instance, multi-OpenStack version
Multi-vendors’ OpenStack distribution: anti-vendor lock in business policy.
Multi OpenStack instance: each vendor has his own OpenStack solution distribution, different site/dc built with
different OpenStack instance.
Multi OpenStack version: step-wise cloud construction, upgrade gradually
2. Ecosystem friendly global open API for tenant resources in multi-site/multi-dc cloud
Cross DC orchestration: tenant virtual resources will be distributed in multi-site/multi-dc but inter-connected by
L2/L3 networking.
Ecosystem friendly open API : It takes almost 4 years for OpenStack to grow the eco-system, the OpenStack API
must be retained for distributed multi-site/multi-dc cloud.
3. Restful open API /CLI for each site/dc
each site/dc installation/upgrade/maintenance decoupled , make the cloud always workable and manageable
standalone in each site. Any part upgrade/update will not lead to the whole cloud upgrade/update at the same
time.
OpenStack API in each site: Open, de facto standard API

Motivation
Issues of multiple independent OpenStack instances in multi-site:
•Missing global view and control of tenant quota
•Missing global view and control of tenant total resource and utilization
•Missing global view for tenant level metering data
•No global view for tenant level IP address space management
•Issue of clone security groups across OpenStack instances
•Issue of establishing tenant level L2/L3 networking across region for application high availability across OpenStack
instances
•Lack of ssh keys clone from one region to another
•Lack of image replication across region
•Lack the function like volume replication across regions for Geo-site Redundancy/Disaster Recovery
•…

• Requirement and driving forces for multi-site cloud

DC 1
OpenStack
(Vendor1 /
Version 2.0)
DC3
DC 2
OpenStack
(Vendor2 /
Version 2.1)
OpenStack
(Vendor3 /
Version 2.1)
OpenStack cascading is “OpenStack orchestrate OpenStacks” solution for multi-site cloud
with unified global OpenStack API exposed
OpenStack API
OpenStack
OpenStack cascading solution
OpenStack API
OpenStack API
OpenStack API

DC 1
OpenStack
(Vendor1 /
Version 2.0)
DC3
DC 2
OpenStack
(Vendor2 /
Version 2.1)
OpenStack
(Vendor3 /
Version 2.1)
with unified OpenStack API exposed
OpenStack API
OpenStack
Value to cloud admin
OpenStack API
OpenStack API
OpenStack API
1 One global single OpenStack API, rich APP ecosystem
3
co-existence of Multi-Vendor/Multi-Version
OpenStack
2 Geo-distributed multi-site/multi-DC cloud
clearly separated
autonomous sub-cloud
inside a large scale
distributed cloud

DC 1
OpenStack
(Vendor1 /
Version 2.0)
DC3
OpenStack
(Vendor3 /
Version 2.1)
OpenStack API
OpenStack
OpenStack API
OpenStack API
OpenStack API
5
4
Plug & Play fast integration for multi-site
cloud through OpenStack API
DC 2
OpenStack
(Vendor2 /
Version 2.1)
Fault isolation. Any fault can only affect part of
the cloud
clearly separated
distributed cloud

DC 1
OpenStack
(Vendor1 /
Version 2.0)
DC3
DC 2
OpenStack
(Vendor2 /
Version 2.1)
OpenStack
(Vendor3 /
Version 2.1)
OpenStack API
OpenStack
OpenStack API
OpenStack API
OpenStack API
6
Clear OAM boundary. Reduce
upgrade/troubleshooting/etc challenge
Bug Fix
6
Clear OAM boundary. Reduce
upgrade/troubleshooting/etc challenge
clearly separated
distributed cloud

Value to end user
• Tenant has global view for resources in multi-clouds
The tenant's resources VM, Volume may be distributed in multi-OpenStacks which using shared KeyStone or
KeyStone federation, and also these resources are inter-connected through L2/L3 networking with advanced
service like FW,LB,VPN. The tenant's distributed resources can be managed through the cascading
OpenStack, like the tenant has one virtual OpenStack allocated to him, the tenant has global view for his
resources like image, metering data, VM/volume/network, etc. The tenant also has global quota control and
resources utilization through the cascading OpenStack.
• Tenant level global IP address management.
The cascading OpenStack can work as the global IP address management for the tenant across multiple
cascaded OpenStack.
• High availability application across different physical data center.
With the aid of overlay virtual L2/L3 networking across data centers and image Replication function,
application backup/disaster recovery/load balance is easy to implement in the distributed cloud.
• Virtual machine / volume migration / vApp migration from one data center to another:
With the help of OpenStack cascading, VM/Volume migration from one DC to another one is feasible.

Value to
scalability
OpenStack …
…
1 2 100
1 2 1000
OpenStack
…
1 2 1000
OpenStack
…
1 2 1000
OpenStack
Max. 100 data centers
Max. 100k physical server nodes
Max. 1 million VMs
OpenStack API OpenStack API OpenStack API
OpenStack API
Scalability inside one data center, multi-
data centers or multi-sites
How large scale is up to
the scalability of the
cascading OpenStack
Test report: http://www.slideshare.net/JoeHuang7/test-report-for-open-stack-cascading-solution-to-support-1-million-v-ms-in-100-data-centers

Stop and thinking, why build a very large cascading OpenStack for all tenants?
In fact, each tenant only cares about his own resources.
Why not just dynamically allocate or assign one virtual OpenStack to serve the
tenant for his resources which is distributed in many OpenStacks, no matter
these OpenStacks in one data center or multi-data centers?
Cascading OpenStack can work as tenant level virtual OpenStack service.
Evolve to unlimited scalability
*Here the OpenStack mainly indicates service to provide IaaS function like Nova,Cinder,Neutron, Glance, Ceilometer,KeyStone

Evolve to unlimited scalability
Cascading
OpenStack
(Tenant 1)
…
Cascading
OpenStack
(Tenant 2)
Cascading
OpenStack
(Tenant x)
…
Tenant 1
Virtual Resources
Tenant 2
Virtual Resources
Tenant x
Virtual Resources
Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y
https://tenant1.OpenStack/ https://tenant2.OpenStack/ https://tenantx.OpenStack/
OpenStack API
OpenStack API
OpenStack API
OpenStack API
OpenStack API
OpenStack API
OpenStack API
OpenStack API
If the tenant wants resources in multi-OpenStacks, allocate one cascading OpenStack
for him (or share one cascading OpenStack with others to reduce management
resource, but make it as small as possible, to ease disaster recovery, backup, …)
If the tenant wants
resource only in one
OpenStack, no
cascading required.
1. Fully distributed, no central
point at all, no scalability
bottleneck.
2. Unlimited OpenStack
instances in one cloud or
federated clouds.
3. Provide tenant with seamless
one OpenStack experience no
matter how many OpenStack
instances behind

• Requirement and driving forces for multi-site cloud

Neutron Server
OVS
Agent
Cinder API
Rabbit-MQ
Cinder
Volume
Cinder
Volume
Nova API
RabbitMQ
Nova
Compute
Nova
Compute
Libvirt
Driver
Nova
Driver
LVM
Driver
Cinder
Driver
KVM Nova LVM Cinder
RabbitMQ
OVS
L3
Agent
Linux
Router
Scheduler
Scheduler
Neutron
Agent
Neutron
Magic happens by just considering OpenStack as its own backend !
Glance Image1:
Loc1: NFS
Loc2: Glance1
Loc3:Glance2
Glance1 Glance2
Image1:
Loc1: Ceph
Image1:
Loc1: Ceph
Ceilometer
API
hBase
Store
Store
CeilometerCeilometer
Nova as hypervisor Cinder as block storage Neutron as networking device Glance as Img. Location Ceilometer as store
** Architecture simplified for illustration only

Neutron
ServerCinder API
Rabbit-MQ
Cinder
Proxy
Nova API
RabbitMQ
Nova
Proxy
Nova Cinder
RabbitMQ
Scheduler
Scheduler
Neutron
Proxy
(L2/L3/LB/V
PN/FW)
Neutron
Components introduced for cascading
Glance
Glance1 Glance2
Ceilometer
API
Ceilometer-
Proxy
CeilometerCeilometer
Replic-
Manager
Cascading
Layer
Cascaded
Layer
Components introduced for cascading
*KeyStone is global service shared or federated by cascading and cascaded layer
* Heat will use OpenStack API to do orchestration, no cascading required.

Design Pattern – fractal (self similar)
Fractal: Recursive self-similar, growth to scale
http://ethshar.wikia.com/wiki/File:Tricircle_fractal.gif
Cascading: Also recursive self-similar, growth to scale.
Just treat the cascaded OpenStack as a huge compute-node

Nova Cascading – how it works
Nova-API
Nova-Scheduler
RbbitMQ
Nova-Proxy
(host1/AZ1)
Nova
Nova-Api
Nova-Proxy
(host2/AZ2)
Nova
Nova-Api
…
Cascading OpenStack
Cascaded OpenStack
1.Launch VM1 (AZ1)
2.Host1 scheduled for it belongs to AZ1
3.Translate the Launch VM 1 request to
configured Nova
4.Launch VM1 in AZ1
5.Reboot VM1 @ host1
6. Translate the Reboot VM1
request to configured Nova
7. Reboot VM1
Solved Challenges:
• UUID mapping
• VM status replication
• Host aggregate
• Flavor
• Metadata modification
• Password / Key pair update
• Image / Cinder / Neutron orchestration
• Cold migration cross Nova
AZ1 AZ2
AZ: availability zone
*Only AvailabilityZone(AZ) and HostAggregate related filter will be configured in the cascading layer nova-scheduler
8.Launch VM2 (AZ2)
9. Host2 scheduled for it belongs to AZ2
10Translate the Launch VM2
request to configured Nova
11.Launch VM2 in AZ2

Cinder Cascading – how it works
Cinder-API
Cinder-Scheduler
RbbitMQ
Cinder-Proxy
(host1/AZ1)
Cinder
Cinder-Api
Cinder-Proxy
(host2/AZ2)
Cinder
Cinder-Api
…
Cascading OpenStack
Cascaded OpenStack
1.Create Volume1 (AZ1)
3.Translate the Create Volume1
request to configured Cinder
4.Create Volume1 in AZ1
5.Resize Volume1 @ host1
6. Translate the Resize
volume request to configured
Cinder
7. Resize Volume
Solved Challenges:
• UUID maping
• Timestamp based query
• Volume type / QoS
• Volume migration cross Cinder
AZ1 AZ2
AZ: availability zone
*Only AvailabilityZone(AZ) filter will be configured in the cascading layer cinder-scheduler
8.Create Volume2 (AZ2)
10.Translate the Create Volume2
request to configured Cinder
11.Create Volume2 in AZ2

Neutron Cascading – L2 networking (VxLAN Point2Point)
Neuton-API
L2/L3-Proxy
L2/L3-Proxy
Neutron Neutron
VM1 VM2
VxLAN0
VM3 VM4
AZ1 AZ2
VM1 VM2
VxLAN0 DVR
VM3 VM4
VxLAN0
DVR
DVR
2.Periodic polling port
status( for example
VM2 port)
3. VM2 Port
status up
4. L2 population
5. fdb_add ( Port for VM2 IP / VM 2 mac / Host IP )
6. Create virtual remote Port for VM2
(with VM2 IP / VM2 mac / VM2 host IP)
VM2
7. Internal L2 population for
virtual remote port for VM2
Virtual remote port
Cascading OpenStack
Cascaded OpenStack
1. Internal L2 population for VM2
If VM2 was added to VxLAN0…
Self similar L2 population happened
in the cascading levelAZ1 AZ2

Neutron Cascading – L2 networking (VxLAN L2GW)
Neuton-API
L2/L3-Proxy
L2/L3-Proxy
Neutron Neutron
VM1 VM2
VxLAN0
VM3 VM4
AZ1 AZ2
VM1 VM2
VxLAN0
DVR
VM3 VM4
VxLAN0
DVR
DVR
2.Periodic polling port
status( for example
VM2 port)
3. VM2 Port
status up
4. L2 population
5. fdb_add ( Port for VM2 IP / VM 2 mac / Host IP )
6. Create virtual remote Port for VM2
(with VM2 IP / VM2 mac / VM2 host IP)
Cascading OpenStack
Cascaded OpenStack
1. Internal L2
population for VM2
If VM2 was added to VxLAN0…
Self similar L2 population happened
in the cascading level
VM2
1. Virtual remote port attached
to L2GW
2. ARP responder proxy
3. Reduce L2 population here
L2GW

Neutron Cascading – L2 networking (VxLAN L2GW - continue)
Neuton-API
L2/L3-Proxy
L2/L3-Proxy
Neutron Neutron
VM1 VM2
VxLAN0
VM3 VM4
AZ1 AZ2
VM1 VM2
VxLAN0
DVR
VM3 VM4
VxLAN0
DVR
DVR Cascading OpenStack
Cascaded OpenStack
L2GWL2GW
Solved Challenges:
1. Not all compute nodes need to be
routable, only GW nodes to be
routable, reduce physical networking
complexity
2. Reduce L2 population traffic in the
cascaded layer
3. Easy to build L3 networking routing
rule for cross OpenStack networking,
for example, VLAN 2 VxLAN, VLAN 2
VLAN, VxLAN 2 VxLAN
4. L2GW mentioned and implemented
in the current BPs can be leveraged

Neutron Cascading – L2 networking (VLAN)
Neuton-API
L2/L3-Proxy L2/L3-Proxy
Neutron Neutron
VM1 VM2
VLAN0
VM3 VM4
AZ1
AZ2
VM1 VM2
VLAN0
Router
VM3 VM4
VLAN1
DVR
Cascading OpenStack
Cascaded OpenStack
VLAN1
Router
192.168.1.0/24
192.168.2.0/24
DVR
network_vlan_ranges
=AZ2:300:2000
network_vlan_ranges =AZ1:100:4000
network_vlan_ranges =AZ1:100:4000
tenant_network_types = vlan,vxlan
network_vlan_ranges =AZ1:100:4000, AZ2:300:2000
vni_ranges=4096:100000
network_vlan_ranges
=AZ2:300:2000
1. neutron net-create net01
--provider:network_type vlan
--provider:physical_network AZ1
Nova-Proxy
2. LaunchVM ( net01 )
3. CreatePort (Net01),
get Port: IP/mac
4. Neutron net-create net01
--provider:network_type vlan
--provider:physical_network AZ1
--segmentation_id = 101
5. port01 = CreatePort(net01, IP/mac)
6. Launch VM ( net01, port01 )
***the network / subnet /
port creation will be
moved to L2 proxy after
uuid -mapping refectory
finished

Neutron Cascading – L3 networking (E-W simplified)
Neuton-API
L2/L3-Proxy L2/L3-Proxy
Neutron Neutron
VM1 VM2
VLAN0
VM3 VM4
AZ1 AZ2
VM1 VM2
VLAN0
Router
VM3 VM4
VLAN1
DVR
Cascading OpenStack
Cascaded OpenStack
VLAN1
Router
192.168.1.0/24
192.168.2.0/24
1. Router-add-interface(net2)
2. Router-add-update(net2)
2. Router-add-update(net2)
3. Update extra-route next hop
“ 100.64.20.20“, destination
“ 192.168.2.0/24 ”
100.64.0.0/10
100.64.10.10 100.64.20.20
3. Update extra-route (next
hop “ 100.64.10.10“,
destination “ 192.168.1.0/24 ”)
GRE tunneling over extra-route is
used for VLAN2VLAN L3 networking
across data centers. Other tunneling
like VPN support later
DVR population

Neuton-API
L2/L3-Proxy
L2/L3-Proxy
Neutron Neutron
VM1 VM2
VLAN0
VM3 VM4
AZ1 AZ2
VM1 VM2
VLAN0
Router
VM3 VM4
VLAN1
DVR
Cascading OpenStack
Cascaded OpenStack
VLAN1
Router
192.168.1.0/24
192.168.2.0/24
1. Router-add-interface(net2)
Neutron Cascading – multi-SDN controller co-exist
OpenDayLight OpenContrail
DCI Controller
METRO
OTN/SDH
OpemFlow
data center interconnection ( DCI )
overlay L2 network
1. One cascaded Neutron one SDN controller,
multi-SDN controllers co-exist(each SDN
controller almost covers from L2-L7)
2. provide cross data center Network as a Service,
one DCI controller (data center
interconnection) under cascading Neutron to
orchestrate cross DC networking, provide cross
DC overlay L2 network.

Neutron Cascading – L3 networking (N-S simplified)
DVR
(Centralized
Node)
N-S
Router
100.64.0.0/10
Internet
VM1 VM2
VLAN0 DVR
1. DVR ( external network: 100.64.10.10 )
3. DVR ( next hop “ 100.64.30.30“,
destination “ 0.0.0.0/0 ” )
192.168.1.0/24
100.64.10.10
100.64.30.30
AZ1
1. remote subnet addressing
Through extra route to next hop for remote
subnet mapping
2. Piggy network introduced.
Using this space 100.64.0.0/10 for cross
OpenStack physical networking
3. Onlink routes introduced.
N-S routers with onlink route
N-S cross OpenStacks
GRE tunneling over extra-route is
used for VLAN2VLAN L3 networking
across data centers. Other tunneling
like VPN support later
AZ2

Glance Cascading
Glance cascading solution:
Just use cascaded Glance as location backend of cascading Glance.
DB
Glance-API
Repli-Manager
Repli-Driver
DB
Glance-API
Storage
Image-
Store
StorageImage-
Store
DB
Glance-API
Storage Image-
Store
Customized image uploading and distribution:
Policy1: Replication when first used
When the image is used in the cascaded OpenStack for the first time, replicate the
image metadata and image data to the cascaded OpenStack. Bad user experience for
the first time, the gain is image replication is much more simpler
Policy2: Replication when upload
---------------------------------------------------------------------------------------
There are 3 scenarios will trigger the replication of Image
1. Upload image data
2. Patch location
3. Create VM snapshot/Volume image
If one the above 3 scenarios happened, the replication manager will check the replication
policy and the image owner, to see if the image should be replication to other
cascaded Glance. If yes, call the Replication-driver to replication the image to
cascaded Glance according to replication-region-list:
1) replication the image metadata to the specified glance
2) replication the image data to the specified region image storage (by copy-from or
import/export or direct copy between storage…)
3) Register the image location to the image in the just replication cascaded Glance
4) Register the new image location( the image URL in the cascaded Glance) to the
cascading image
Better user experience, but complex replication mechanism
1 2 3
Nova-Proxy
Cinder-Proxy
Image
Client
Image
Client
Image registration:
If image has been already distributed in multi-OpenStack instances, just register the image link in
cascaded Glance as one of the image location in cascading Glance

Ceilometer Cascading
Ceilometer
Heat
Cascading
AutoScaling
Alarm request
Ceilometer-Proxy
Ceilometer Ceilometer
Ceilometer
API calling
class Ceilometer-Proxy(base.StorageEngine)
{
UUID mapping injected by
Nova/Cinder/Neutron/Glance
Resource UUID translation
Resource UUID and Ceilometer Location
addressing
Proxy the request to proper Ceilometer
}
The webhook setting ( callback
to HEAT ) for alarm action will
be sent to cascaded Ceilometer
transparently
The webhook
(callback to
HEAT)
Ceilometer cascading solution:
Just use cascaded Ceilometers as StogradeEngine of Cascading
OpenStack. All requests from cascading ceilometer will be proxy to
proper Ceilometer
Cascading OpenStack

OpenStack cascading
DB
Message Bus
Nova-API
Nova-Scheduler
Nova-Conductor
DB
Message Bus
Cinder-API
Cinder-Scheduler
DB
Message Bus
Neutron-API
Neutron-Plug-in
DB
Message Bus
Nova-API
Nova-Scheduler
Nova-Conductor
DB
Message Bus
Cinder-API
Cinder-Scheduler
DB
Message Bus
Neutron-API
Neutron-Plug-in
Cascaded OpenStack 1 Cascaded OpenStack x
…
Controller Node
Compute 1 Compute n
…
Compute 1 Compute n
DB
Message Bus
Nova-API
Nova-Scheduler
Nova-Conductor
DB
Message Bus
Cinder-API
Cinder-Scheduler
DB
Message Bus
Neutron-API
Neutron-Plug-in
Cascading OpenStack
Nova-API Cinder-API Neutron-API Nova-API Cinder-API Neutron-API
Controller Node Compute x
Nova-API Cinder-API Neutron-API
Nova-Proxy
Cinder-Proxy
L2-Proxy
L3-Proxy
LB-Proxy
VPN-Proxy
Nova-Proxy
Cinder-Proxy
L2-Proxy
L3-Proxy
LB-Proxy
VPN-Proxy
FW-Proxy
FW-Proxy
Compute 1
VM
VM
VM
VM
Controller Node
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
Cascading OpenStack is normal OpenStack except that
the process running in compute-node as proxy which
treat the cascaded OpenStack as its own backend

Evolution of OpenStack cascading to hybrid cloud
Cascading
OpenStack
(Tenant 1)
…Cascading
OpenStack
(Tenant 2)
Cascading
OpenStack
(Tenant x)
…
Tenant 1
Virtual Resources
Tenant 2
Virtual Resources
Tenant x
Virtual Resources
Cascaded OpenStack 2Cascaded OpenStack 1 Cascaded OpenStack y
https://tenant1.OpenStack/ https://tenant2.OpenStack/ https://tenantx.OpenStack/
OpenStack API
OpenStack API
OpenStack API
OpenStack API
OpenStack APIOpenStack API
AWS API
Azure API

More information:
Wiki: https://wiki.openstack.org/wiki/OpenStack_cascading_solution
Wiki: https://wiki.openstack.org/wiki/Tricircle
Source Code: https://github.com/openstack/tricircle

Introduction of OpenStack cascading solution

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a Introduction of OpenStack cascading solution

Semelhante a Introduction of OpenStack cascading solution (20)

Último

Último (20)

Introduction of OpenStack cascading solution