CloudStack is an open source cloud orchestration platform that allows users to provision infrastructure as a service (IaaS) clouds. It supports multiple hypervisors and cloud deployment strategies. Key features include self-service VM provisioning, monitoring of consumed resources, volume and snapshot management, and network services like load balancing and firewall rules. CloudStack uses a multi-tenant architecture with logical abstractions like zones, pods, clusters, and hosts to manage the underlying physical infrastructure.

1. CloudStack Overview
Alan Chu@ TCloud Computing

2. Outline
• Overview of CloudStack
• Problem Definition
• Feature set overview
• Network
• Storage

3. What is CloudStack
• Multi-tenant cloud
orchestration platform
– Turnkey solution for delivering
IaaS clouds
– Hypervisor agnostic
Build your cloud the way the – Scalable and flexible
world’s most successful clouds – Open source, open standards
are built
• Deliver cloud services faster
and cheaper

4. History of Apache CloudStack
• 2009: Cloud.com, 100% proprietary
• 2010: Cloud.com, open core, GPL v3
• 2011: Citrix Systems, 100% open, GPL v3
• 2012: ASF, 100% open, Apache License 2

6. CloudStack Supports Multiple Cloud Strategies
Private Clouds Public Clouds
On-premise Hosted Multi-tenant
Enterprise Cloud Enterprise Cloud Public Cloud
• Dedicated • Dedicated • Mix of shared and
resources resources dedicated
• Security & total • Security resources
control • SLA bound • Elastic scaling
• Internal network • 3rd party owned • Pay as you go
• Managed by and operated • Public internet,
Enterprise or 3rd VPN access
party

7. CloudStack Provides On-demand Access
Org A Org B
Users
Admin Admin
End User Users Users
Compute Network Storage
Admin

8. Problem Definition
• Offer a scalable, flexible, manageable IaaS platform that
follows established cloud computing paradigms
• IaaS
– Orchestrate physical and virtual resources to offer self-service
infrastructure provisioning and monitoring
• Scalable
– 1 -> N hypervisors / VMs / virtual resources
– 1 -> N end users
• Flexible
– Handle new physical resource types
• Hypervisors, storage, networking
– Add new APIs
– Add new services
– Add new network models

9. Problem Definition (cntd)
• Manageable
– Hide complexity of underlying resources
– Rich functional end-user and admin UI
– Admin API to automate operations
– Easy install, upgrade for small -> large clouds
– Simple scaling, automated resilience

10. Feature Set Overview

11. Open Flexible Platform
Compute Hypervisor
XenServer VMware KVM Bare metal
Storage Block & Object
Fiber
Local Disk iSCSI NFS Swift
Channel
Primary Storage Secondary Storage
Network Network & Network Services
Connection Load
Isolation Firewall VPN
Type balancer

12. Service Offering

13. Create Custom Virtual Machines via Service Offerings
Select Operating System
• Windows, Linux
Select Compute Offering
• CPU & RAM
Select Disk Offering
• Volume Size
Select Network Offering
• Network & Services
Create VM

14. Dashboard Provides Overview of Consumed Resources
• Running, Stopped &
Total VMs
• Public IPs
• Private networks
• Latest Events

15. Virtual Machine Management
Users
Change
VM Operations VM Access VM Status
Service Offering
Start
• CPU Utilized 2 CPUs 4 CPUs
Stop 1 GB RAM 4 GB RAM
• Network Read
20 GB 200 GB
Restart • Network Writes
20 Mbps 100 Mbps
Destroy

16. Volume & Snaphost Management
VM 1
Add / Delete
Volumes Volume
Create Templates Volume Template
from Volumes
Hourly Weekly
Schedule Now
Snapshots Daily Monthly
….
View Snapshot
History

17. Network & Network Services
• Create Networks
• Acquire public IP address for NAT &
load balancing
• Control traffic to VM using ingress
and egress firewall rules
• Set up rules to load balance traffic
between VMs

18. CloudStack Deployment Architecture
Internet  Hypervisor is the basic unit of
CloudStack
Management scale.
Server
Zone 1  Cluster consists of one ore
more hosts of same hypervisor
L3 core
 All hosts in cluster have access
to shared (primary) storage
Pod 1 Access Layer Pod N
Secondary
 Pod is one or more clusters,
…. Storage usually with L2 switches.
Cluster N
 Availability Zone has one or
more pods, has access to
…. secondary storage.
 One or more zones represent
Cluster 1
cloud
Host 1
Primary
Storage
Host 2

19. Management Server Managing Multiple Zones
Cloud
Data Center 1 Data Center 2
Data Center 2
Data Center 3
Zone 2
Zone 2
Zone1 Zone 3
Zone 4 3
Zone
CloudStack Cloud can have
one or more Availability
Zones (AZ).
Data Center 2
Data Center 2
Data Center 2
Zone 2
Zone 2
Zone 2 3
Zone
Zone 3
Zone 3

20. Management Server Managing Multiple Zones
Cloud
Data Center 1 Data Center 2  Single Management Server can
Data Center 2
Mgmt Data Center 3 manage multiple zones
Server
Zone 2  Zones can be geographically
Zone 2 distributed but low latency links are
Zone 3 expected for better performance
Zone1
Zone 4 3
Zone
 Single MS node can manage up to
10K hosts.
 Multiple MS nodes can be deployed
Data Center 2 as cluster for scale or redundancy
Data Center 2
Data Center 2
Zone 2
Zone 2
Zone Zone 3
2
Zone 3
Zone 3

21. Management Server Deployment Architecture
Single-node Multi-node
Deployment Deployment
MS
User API User API
MS MySQL Load
MS
DB Balancer
Admin API Admin API
MySQL
MS DB
Back Up
Replication DB
 MS is stateless. MS can be deployed
as physical server or VM
Infrastructure Infrastructure
Resources  Single MS node can manage up to Resources
10K hosts. Multiple nodes can be
deployed for scale or redundancy

22. Core CloudStack Components
• Hosts
• Servers onto which services will be provisioned
VM
• Primary Storage Host
• VM storage
VM
• Cluster Host
• A grouping of hosts and their associated storage
• Pod Primary
Storage
• Collection of clusters
• Network Cluster
• Within the switch
Secondary Storage
Secondary
• Storage Network Cluster
• Template, snapshot and ISO storage
• Zone CloudStack Pod
• Collection of pods, network offerings and secondary
storage
• Management Server Farm CloudStack Pod
• Responsible for all management and provisioning
tasks Zone

23. CloudStack Storage
Primary Storage
• Configured at Cluster-level. Close to hosts for better
performance
L3 switch
• Stores all disk volumes for VMs in a cluster
• Cluster can have one or more primary storages
Pod 1 L2 switch
• Local disk, iSCSI, FC or NFS Secondary
Cluster 1 Storage
Host 1
Primary
Secondary Storage Storage
Host 2
• Configured at Zone-level
• Stores all Templates, ISOs and Snapshots
• Zone can have one or more secondary storages
• NFS, OpenStack Swift

24. Provisioning Process
1. User Requests Instance VM
2. Provision Optional Network Host
Services
Host
3. Copy instance template from Primary Storage
secondary storage to primary
Cluster
storage on appropriate cluster
4. Create any requested data Pod
volumes on primary storage for the
cluster Template
5. Create instance
Secondary Storage
6. Start instance Zone

25. Citrix XenServer
CloudStack
• Integrates directly with XenServer Pool Manager
Master
• Snapshots at host level XenServer Pool
Master Host
• System VM control channel at host level
• Network management is host level XenServer Host
XenServer Host
XenServer Host
XenServer Host
XenServer
Resource Pool

26. RedHat Enterprise Linux (KVM)
• Integrates with libvirt using Cloud
Agent CloudStack
Manager
• Snapshots at host level
• System VM control channel at host Cloud Agent
level
Libvirt
• Network management is host level
KVM Host
Cloud Agent
Libvirt
KVM Host

27. VMware vSphere
CloudStack
• Integration through vCenter Manager
• System VM control channel via vSphere Host
CloudStack private network vCenter
vSphere Host
• Snapshot and volume management
via Secondary Storage VM vSphere Cluster
• Networking via vSphere vSwitch
vSphere Host
vSphere Host
vSphere Host
vSphere Cluster
Data Center

28. Management Server Interaction with Hypervisors
Management
Server
XAPI HTTPS
vCenter Agent
XenServer
KVM
ESX
• XS 5.6, 5.6FP1, 5.6 SP2, • ESX 4.1, 5.0 • RHEL 6.0, 6.1, 6.2
6.0.2
• Full Snapshots • Full Snapshots (not live)
• Incremental Snapshots
• VMDK • QCOW2
• VHD
• NFS, iSCSI, FC & Local disk • NFS, iSCSI & FC
• NFS, iSCSI, FC & Local disk
• Storage over-provisioning: • Storage over-provisioning:
• Storage over-provisioning: NFS, iSCSI NFS
NFS

29. Multi-tenancy & Account Management
Cloud
Resources
Domain
VMs, IPs, Snapshots…
• Domain is a unit of
Org A isolation that represents
Admin a customer org, business
unit or a reseller
Domain
Reseller A
• Domain can have
Admin Resources arbitrary levels of sub-
Sub-Domain
Org C
VMs, IPs, Snapshots… domains
Admin
• A Domain can have one
Account
or more accounts
Group A
• An Account represents
Account one or more users and is
Group B the basic unit of isolation
User 1 • Admin can limit
resources at the Account
User 2
or Domain levels

30. CloudStack Network

31. CloudStack Network Features
The Service of
Virtual Network
The Management of
Physical Network

32. Physical Network
Operations
Users
Admin and
Cloud API
CloudStack MS
Cluster
Router
MySQL
Load Balancer
Availability Zone
L3 Core Switch
Access
Layer
Switches
Secondary
Servers
… … … … … Storage
Pod 1 Pod 2 Pod 3 Pod N

33. CloudStack Network Traffic Type
Network Traffic type:
Public Network:
Public traffic is generated when VMs in the cloud
access the internet, e.g Virtual Router
Guest Network:
The tenant network to which instances are attached.
Storage Network:
The physical network which connects the hypervisor to
the storages.
Management Network:
Control Plane traffic between CloudStack management
server and hypervisor clusters

34. CloudStack Network Mode
Basic Network Advanced Network
• AWS-style networking • Account’s VM Isolation by VLAN
• Account’s VM Isolation by • VR can provide more services :
Security Group NAT, Firewall, PF, LB, VPN
• VR provides service: DHCP, DNS • Guest Network supports Isolated
• Each VM has only one NICs and Shared Network types
(Network) • Each VM can have more NICs
(Network)

35. CloudStack Basic Network
• VR provides service: DHCP, DNS
• VMs Isolation by Security Group

36. Security Group Isolation

37. CloudStack Advanced Network
Public Network Guest Network 1
VLAN 101
Public IP Guest 1 10.1.1.2
Gateway
address VM 1
address
65.37.141.11
10.1.1.1
Guest 1 Virtual Guest 1 10.1.1.3
Public Router VM 2
Internet
NAT
Guest 1 10.1.1.4
DHCP
VM 3
Load Balancing
Port Forwarding
Firewall
VPN
Guest Network 2
Public IP VLAN 102
address Gateway Guest 2 10.1.1.2
65.37.141.24 address VM 1
10.1.1.1
Guest 2 Virtual Guest 2 10.1.1.3
Router VM 2
Guest 2 10.1.1.4
VM 3

38. VLAN Isolation

39. CloudStack Advanced Network Service
• Firewall
• Source NAT
• Static NAT
• Load Balancing
• Port Forwarding
• VPN

40. Advanced Network - NAT
• Default Deny all the

41. Advanced Network - Load Balancing

42. Advanced Network - Port Forwarding

43. Guest Virtual Network With Physical Device
CS Virtual Router provides Network Services External Devices provide Network Services
Guest Virtual Network 10.1.1.1/8 Guest Virtual Network 10.1.1.1/8
VLAN 100 VLAN 100
Public Public
Network/Internet Network/Internet
Guest Private IP Guest
10.1.1.2 VM 1 Public IP 10.1.1.1 VM 1
65.37.141.111 Juniper 10.1.1.111
Gateway
Public IP SRX
address
CS Firewall
65.37.141.11 10.1.1.1 Guest Guest
Virtual
10.1.1.3 VM 2 10.1.1.3 VM 2
Router
Public IP Private IP
DHCP, DNS NetScaler 10.1.1.112
65.37.141.112
NAT Guest Load Guest
Load Balancing 10.1.1.4 VM 3 Blancer VM 3
10.1.1.4
VPN
Guest Guest
10.1.1.5 VM 4 10.1.1.5 VM 4
CS
Virtual DHCP, DNS
Router

44. Network Offerings
• Cloud provider defines the
feature set for guest networks
• Toggle features or service levels
– Security groups on/off
– Load balancer on/off
– Load balancer software/hardware
– VPN, firewall, port forwarding
• User chooses network offering
when creating network
• Enables upgrade between
network offerings
• Default offerings built-in
– For classic CloudStack networking

45. Physical Network – Guest Network Mapping
VM Instance
• Choose the instantiated guest network
Guest Network
• Instance of Network Offering
• Shared: created by Admin
• Isolated: Created and owned by user
• One virtual router for one network
• Cross pod, within Zone
• VLAN id picked from the pool
Physical Network Network Offering
• Zone level • Only for Guest traffic
• Defined by NIC • Guest network type: Shared or Isolated
• Assigned with traffic type (P, G, M, S) • Defined a set of network services,
• Associated by label/vswitch name such as DHCP, Firewall, VPN, NAT…
• Attached with device as service • Bandwidth
provider
Tag

46. Advanced Network – Multi-tier Network
Public Network Guest Network 1 Guest Network 2 Guest Network 3
Internet 10.1.1.0/24 10.1.2.0/24 10.1.3.0/24
VLAN 100 VLAN 101 VLAN 102
App VM
10.1.2.31 1 10.1.3.21
Web VM
10.1.1.2 1 10.1.2.21
Public IP
65.37.141.111
App VM
10.1.2.24 2 10.1.3.45
Private IP Web VM
CS 10.1.1.1 10.1.1.3 2 10.1.2.18
Virtual
Router
Web VM
DHCP, DNS 3 10.1.2.38 10.1.3.24 DB VM 1
10.1.1.4
NAT, Firewall
LB, VPN, Port
Forwarding Web VM
10.1.1.5 4 10.1.2.39
CS
Virtual CS
DHCP, DNS Router DHCP, DNS Virtual
Router

47. Advanced Network – Virtual Private Network
Internet Internal VLAN
CS
Loadbalancer Virtual Router / IPSec site-to-site VPN
Other Data
Center
Virtual Router Services
App VM
• DNS 1
10.1.2.31
• LB Web VM
1
• Site-to-Site VPN 10.1.1.1
• Static Routes App VM
• Network ACLs Web VM 10.1.2.24 2
• NAT, PF 10.1.1.3 2
• FW [ingress & egress]
Web VM
3 DB VM 1
10.1.1.4 10.1.3.24
Web VM
10.1.1.5 4
Guest Network Guest Network Guest Network
10.1.1.0/24 10.1.2.0/24 10.1.3.0/24
VLAN 100 VLAN 101 VLAN 102

48. CloudStack Storage

49. Storage
• Primary Storage
Zone-Level Layer 3 Switch
Private Network – Block device to the VM
– IOPs intensive
– Accessible from host or
Pod 1 Pod Pod cluster wide
• WORM Storage
Pod-Level Layer-2 2 N
Switch
Scale-Out
NFS – Secondary Storage or Object
Computing
Server 1
Primary Store for templates, ISO, and
Storage
snapshot archiving
Cluster 2
Computing
Primary
– High capacity
Server 2
Storage
• CloudStack manages the
Computing
Scale-Out storage between the two to
Server 3
NFS
achieve maximum benefit and
Cluster 1
Primary
Storage resiliency
Computing
Server 4

50. Primary Storage Support Matrix
Type XenServer VMWare KVM
Local Disk Supported Supported Supported
iSCSI Supported Supported Not Supported
Fiber Channel Supported Supported Not Supported
NFS Supported Supported Supported

51. Storage Tagging
• Supported via storage tags for primary storage
• Specify a tag when adding a storage pool
• Specify a tag when adding a disk offering
• Only storage pools with the tag will be
allocated for the volume

52. WORM Storage
• Write Once Read Many storage pattern is
supported by two different storage types
– Secondary Storage (NFS Server within an availability
zone)
– Object Store (Swift implementation for cross-zone)
• Objective for WORM storage
– High capacity, cheap storage
– Easy to increase capacity
• Used to store templates, ISOs, and snapshots

53. CloudStack System VM

54. CloudStack System VMs
• System VMs optimize and scale the data path on behalf of CloudStack
– Stateless, can be destroyed and recreated from database state
– Highly Available
– Communicates with Management Server over management network
– Usually have 3 interfaces: control(linked-local), mgmt and public
• Console Proxy VM
– Provides AJAX-style HTTP-only console viewer
– Grabs VNC output from hypervisor
– Scales out (more spawned) as load increases
– Java-based server Communicates with MS
• Secondary Storage VM
– Provides image (template) management services
– Download from HTTP file share or Swift
– Copy between zones
– Scale out to handle multiple NFS mounts
– Java-based server communicates with MS

55. CloudStack System VMs
• Virtual Router VM
– Provides multiple network services
– IPAM (DHCP), DNS, NAT, Source NAT, Firewall, Port
Forwarding, VPN
– User-data, Meta-data, guest SSH keys and password change
server
– Redundancy via VRRP
– MS configures VR over SSH
• Proxied via the hypervisor on XS and KVM

56. System VM spec
• Debian 6.0 ("Squeeze"), 2.6.32 kernel with the latest security patches from the Debian security
APT repository. No extraneous accounts
• 32-bit for enhanced performance on Xen/VMWare
• Only essential software packages are installed. Services such as, printing, ftp, telnet, X, kudzu,
dns, sendmail are not installed.
• SSHd only listens on the private/link-local interface. SSH port has been changed to a non-
standard port (3922). SSH logins only using keys (keys are generated at install time and are
unique for every customer)
• pvops kernel with Xen paravirt drivers + KVM virtio drivers + VMware tools for optimum
performance on all hypervisors. Xen tools inclusion allows performance monitoring
• Template is built from scratch and is not polluted with any old logs or history
• Latest versions of haproxy, iptables, ipsec, apache from debian repository ensures improved
security and speed
• Latest version of jre from Sun/Oracle ensures improved security and speed

57. MS Internals
• Architecture
• Workflow
• High Availability
• Scalability

58. Inside a Management Server
Plugins
cmd.execute() Plugins
Cmds Plugins
Async
CS API API Job
Services
Servlet Queue
API
Mgr Kernel
Responses
Agent API
(Commands) Agent Resources
Manager Local
Or
Remote
Hypervisor Network
Native Device
APIs API
MySQL

59. Old Architecture
API Layer
Pros
EC2 CloudStack
Access Control • Agile development for
existing developers
Virtual Machine Manager
Console Proxy Manager
• Scales well horizontally
Async Job Manager
Snapshot Manager
Template Manager
Network Manager
Storage Manager
Cons
… • Monolithic
• Difficult to educate
new and third-party
Agent Manager
XenServ KVM SRX F5
NetScal Other
developers
er er
• Easy to introduce bugs
Resour Resour Resour Resourc
Resourc ce ce Resour es
ce
e ce
59

60. New Deployment Architecture
• Scales horizontally to
different pressure points
• Automatically scales
service VMs in zones to
facilitate most efficient
data path transfers
• Fault isolation between
API servers and
Execution Servers and
resources within zones

61. New Architecture – API Server
UI
Cloud
CLI
Other
Clients
• API Server isolates
Portal
integration code from
REST
Execution Server
API Server • API Server can
OAM&P API
Pluggable API Engine
End User EC2 Other
horizontally scale to
Management Services
API API
ACL &
APIs
Integration
handle traffic
- Resource
management
- Configuration
Authentication
- Accounts,
Domains, and
• Easily adds other API
- Additional
operations added
Projects
- ACL, limits
compatibility
• Easily exposes API
by third party checking
Framework
- Job Queue
-
-
Database Access Layer
OSGi
needed by third party
vendors

62. New Architecture – Execution
Server
Execution Server • Execution Server protected by
job queue
• Kernel kept small for stability. It
Services API
Kernel Plugins
• Drives long running VM operations • Storage only drives processes.
• Syncs between resources managed
and DB
Handling
• Network • Plugins provide mappings of
• Generates events Handling
• Deployment
virtual entities to physical
planning
• Hypervisor
resources
Handling
• Third party plugins to provide
vendor differentiation in
CloudStack
•
Framework
Cluster Management • Component Framework • Communicates with resources
•
•
Job Management
•
Alert & Event Management
(OSGi)
Transaction Management
within data center over message
•
•
Database Access Layer
Messaging Layer
bus

63. New Architecture – Resources
Agent
• Resources are carried in
Hypervisor Resources
service VMs to be in close
network proximity to the
Network Resources
physical resources it
Storage Resources manages
Image & Template Resources
• Easily scales to utilize the
most abundant resource in
Snapshot Resources
data center (CPU & RAM)
• Communicates with
Execution Server over
message bus (JSON)
• Can be replicated for fault
tolerance

64. Cloud Other
UI CLI Clients
Portal
Management Server
REST API
OAM&P API End User API EC2 API Other APIs Pluggable Service API Engine
Console Proxy ACL & Authentication Security Adapters
Management - Accounts, Domains, and Projects
- ACL, limits checking Account Management
Connectors
Template Services API
Access
Deployment Planning
Plugin API
HA
Kernel
- Drives long running VM
Services API
Network Configurations
Usage operations
Calculations - Syncs between resources
managed and DB Network Elements
Additional - Generates events
Services
Hypervisor Gurus
Cluster Resource Job Alert & Event Database
Management Management Management Management Access
Event Bus
Message Bus
Hypervisor Network Storage Image Snapshot
Resources Resources Resources Resources Resources

65. Kernel Module
• Understands how to orchestrate long running
processes (i.e. VM starts, Snapshot copies,
Template propagation)
• Well defined process steps
• Calls Plugin API to execute functionalities that
it needs

66. Plugins
• Various ways to add more capability to
CloudStack
• Implements clearly defined interfaces
• All operations must be idempotent
• All calls are at transaction boundaries
• Compiles only against the Plugin API module

67. Anatomy of a Plugin
Rest API
- Optional. Required only if needs to expose configuration API to admin. ServerResource
- Optional. Required if
Plugin needs to be co-
located with the
resource
- Implements translation
layer to talk to resource
- Communicates with
Plugin API
Implmentation server component via
JSON
Data Access Layer

68. Anatomy of a Plugin
• Can be two jars: server component to be
deployed on management server and an optional
ServerResource component to be deployed co-
located with the resource
• Server component can implement multiple Plugin
APIs to affect its feature
• Can expose its own API through Pluggable Service
so administrators can configure the plugin
• As an example, OVS plugin actually implements
both NetworkGuru and NetworkElement

69. Plugin Interfaces Available
• NetworkGuru – Implements various network isolation technologies
and ip address technologies
• NetworkElement – Facilitate network services on network elements
to support a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)
• DeploymentPlanner – Different algorithms to place a VM and
volumes.
• Investigator – Ways to find out if a host is down or VM is down.
• Fencer – Ways to fence off a VM if the state is unknown
• UserAuthenticator – Methods of authenticating a user
• SecurityChecker – ACL access
• HostAllocator – Provides different ways to allocate host
• StoragePoolAllocator – Provides different ways to allocate volumes

70. Adding a Plugin to CloudStack
• Components are configured through
components.xml
• Supports DAO, Manager, and Adapter patterns
• Open to other component frameworks (OSGi a
possibility)

71. High Availability

72. High Availability
• Service Offering contains a flag for whether
HA should be supported for the VM
• Does not use the native HA capability of
hypervisors for XenServer and KVM
• Uses adapters to fine tune HA process

73. Triggering High Availability
VM HA are triggered via the following methods:
• VM Sync detects out of band VM changes
• Resource Management detects that a resource is
unreachable and its state can not be determined.
• VM start/stop has been sent to the resource but
resource does not return
• Details of how high availability is done is at
http://docs.cloudstack.org/CloudStack_Documentation/Design_Documents/CloudStack_High_Availability_-
_Developer's_Guide

74. Scalability

75. Current Status
• 10k resources managed per management server
node
• Scales out horizontally (must disable stats
collector)
• Real production deployment of tens of thousands
of resources
• Internal testing with software simulators up to
30k physical resources with 300k VMs managed
by 4 management server nodes
• We believe we can at least double that scale per
management server node

76. Balancing Incoming Requests
• Each management server has two worker thread pools for incoming
requests: effectively two servers in one.
– Executor threads provided by tomcat
– Job threads waiting on job queue
• All incoming requests that requires mostly DB operations are short
in duration and are executed by executor threads because incoming
requests are already load balanced by the load balancer
• All incoming requests needing resources, which often have long
running durations, are checked against ACL by the executor threads
and then queued and picked up by job threads.
• # of job threads are scaled to the # of DB connections available to
the management server
• Requests may take a long time depending on the constraint of the
resources but they don’t fail.

77. Comparison of two Approaches
• Stats Collector – collects capacity statistics
– Fires every five minutes to collect stats about host CPU and
memory capacity
– Smart server and dumb client model: Resource only
collects info and management server processes
– Runs the same way on every management server
• VM Sync
– Fires every minute
– Peer to peer model: Resource does a full sync on
connection and delta syncs thereafter. Management
server trusts on resource for correct information.
– Only runs against resources connected to the management
server node

78. Resource Load Balancing
• As management server is added into the cluster, resources are rebalanced
seamlessly.
– MS2 signals to MS1 to hand over a resource
– MS1 wait for the commands on the resources to finish
– MS1 holds further commands in a queue
– MS1 signals to MS2 to take over
– MS2 connects
– MS2 signals to MS1 to complete transfer
– MS1 discards its resource and flows the commands being held to MS2
• Listeners are provided to business logic to listen on connection status and
adjusts work based on who’s connected.
• By only working on resources that are connected to the management
server the process is on, work is auto-balanced between management
servers.
• Also reduces the message routing between the management servers.

79. Interactions
OVM Cluster Primary
Storage
vcenter
Monitoring Primary
CS API vSphere Cluster
Storage
End
User UI
Primary
XS Cluster Storage
Admin
UI
Clustered
CloudStack XAPI
Domain CS Admin & CloudStack
CloudStack
Admin End-user API Primary
UI
Management JSON KVM Cluster Storage
Server
NetConf
Juniper SRX
Cloud user Nitro API
{API client (Fog/etc)} VNC
JSON
ec2 API JSON Netscaler
Cloud user Console
Console
{ec2 API client } Proxy VM
Proxy VM NFS
MySQL Server
{Proxied} SSH Sec. Storage NFS NFS
Sec. Storage
VM
Ajax HTTPS VM
Console
Router VM HTTP (Template Download)
Router VM HTTP (Template Copy)
Router VM
Cloud user HTTP (Swift)

80. CloudStack vs. OpenStack vs.
Eucalyptus

81. CloudStack
• Mainly written in Java
• ASL2.0 license
• Has more than 100 production clouds (Around May, 2012)
• Support private/hybrid/public cloud
• Scale to 30K physical host in commercial environment
• Support XenServer/Vsphere/KVM/OVM/Baremetal as
hypervisor
• Multiple geographically distributed datacenters management
• Flexible and rich network functionality
• Easy installation and management
• Amazon EC2 API compatible
• Well documented
• Active community

82. OpenStack
• Mainly written in Python
• ASL2.0 license
• Support private/hybrid/public cloud
• Immature for commercial usage
• Support XenServer/Vsphere/KVM/Xen/Hyper-V as hypervisor
• Network is single point of failure
• Weak VPN support for enterprise hybrid cloud
• All inter-module communication are based on MQ
• Not well documented
• A bit hard to install
• Amazon EC2 API partially compatible

83. Eucalyptus (Open Source edition)
• Mainly written in Java
• GPLv3 license
• Focus on private cloud
• Support KVM/Xen as hypervisor
• Fully compatible with Amazon EC2
• Fully compatible with Amazon S3 via Walrus
• EBS support via AoE and iSCSI
• Both web UI and command line tools for cloud administration
• Well documented
• Difficult to getting started