1. Developer Day – 7/21/2012
Will Chan – Director of Engineering
2. Agenda
• Welcome and Introduction
• What is CloudStack?
• CloudStack Basics
• Cloudstack Deployment Architecture
• Networking Deep Dive
• Software Architecture
• Current Architecture
• Future Architecture
• Q&A
• Lunch
3. Agenda (cont.)
• CloudStack Integration
• UI Customization
• API Deep Dive
• Future UI Plugin Framework
• Q&A/Break
• Apache Community
• Why Apache and the Apache Server Foundation (ASF)?
• How to contribute to CloudStack
• Closing Remarks
5. Welcome and Introduction
• Will Chan
• Founding Engineer/Director of Engineering @ Cloud.com since 2008
• Director of Engineering @ Citrix Systems since 2011
• PPMC member @ ASF CloudStack since 2012
• Committer @ ASF CloudStack since 2012
• Sheng Liang
• Cloud Visionary and Founder of Cloud.com
• CTO, CloudPlatforms Group at Citrix Systems
7. Apache CloudStack
• Secure, multi-tenant cloud orchestration
platform
– Turnkey platform for delivering IaaS clouds
– Over 100 commercial deployments: private and
public
Build your cloud the way the world’s – Full featured GUI, end-user API and admin API
most successful clouds are built
8. Apache CloudStack
• Open Source
• Apache License
• Incubating in the Apache Software
Foundation since April 2012
• Open Source since May 2010
Build your cloud the way the world’s
most successful clouds are built • In production since 2009
9. Apache CloudStack
• Flexibility and scale
• Hypervisor agnostic
• Flexible network topologies
• Multiple storage options
• Proven to scale to tens of thousands of
Build your cloud the way the world’s
most successful clouds are built hypervisors
11. 146 Companies 238 Developers Global User Groups
Service Providers 100’s of Production Clouds
32,000 Community Members
Enterprises Universities
12. Server Virtualization++ Cloud
Built for traditional enterprise apps Designed around big data, massive
& client-server compute scale & next-gen apps
• Enterprise arch for 100s of hosts • Cloud architecture for 1000s of hosts
• Scale-up (server clusters) • Scale-out (multi-site server farms)
• Apps assume reliability • Apps assume failure
• VLAN (or no) isolation • L3 isolation or overlays
• Bonding, multi-link, multi-path, • Generally do not support multicast or
redundant networks, STP broadcast
• Proprietary vendor stack • Open, value-added stack
Think: vCloud Director Think: AWS, RAX, zCloud, eBay, etc.
13. CloudStack Supports Multiple Cloud Strategies
Private Clouds Public Clouds
On-premise Hosted Multi-tenant
Enterprise Cloud Enterprise Cloud Public Cloud
• Dedicated resources • Dedicated resources • Mix of shared and
• Security & total control • Security dedicated resources
• Internal network • SLA bound • Elastic scaling
• Managed by • 3rd party owned and • Pay as you go
Enterprise or 3rd party operated • Public internet, VPN
access
14. Designing a zone for a traditional workload
Hypervisor
Traditional-Style Availability Zone
vSphere or XenServer Enterprise
vCenter/XenCenter
Storage
Enterprise Networking (e.g., VLAN) SAN
Networking
Hyperviso Hyperviso Hyperviso L2 VLANs
r Cluster r Cluster r Cluster
Network Services
Enterprise Storage (e.g., SAN) Load Balancing VPN
Multi-tier Apps
Multi-tier VLANs OVF
15. Designing a zone for an Amazon-style workload
Amazon-Style Availability Zone
Software Defined Networks Hypervisor
(e.g., Security Groups, EIP, ELB,...) XenServer
Advanced
Server Server Server Server
Storage
Racks Racks Racks Racks
Local EBS Object store
Server Server Server Server
Racks Racks Racks Racks Networking
L3 SDN based L2 Elastic IP
Server Server Server Server
Racks Racks Racks Racks Network Services
Security Groups ELB GSLB
Elastic Block Storage Multi-tier Apps
3rd Party Tools (e.g.,
RightScale, enStratus) CloudFormation
17. Deployment Architecture
Zone 1
Host is the basic unit of scale. Runs a
hypervisor or is bare metal
Load Balancer Firewall
Cluster consists of one ore more hosts
of same hypervisor
L3 switch
All hosts in cluster have access to
shared (primary) storage
Pod 1 L2 switch Pod N
Secondary
Pod is one or more clusters, usually
…. Storage with a L2 switch. Represents a rack
Cluster N
Availability Zone has one or more
pods, has access to secondary
…. storage.
Firewall and Load balancers separate
Cluster 1
public and private networks
Host 1
Primary
One or more zones represent cloud
Storage
Host 2
18. Deployment Architecture (Storage)
Primary Storage
• Configured at Cluster-level. Close to hosts for better performance
• Stores all disk volumes for VMs in a cluster L3 switch
• Cluster can have one or more primary storages
Pod 1 L2 switch
• Local disk, iSCSI, FC or NFS
Secondary
Cluster 1 Storage
Host 1
Primary
Secondary Storage Storage
Host 2
• Configured at Zone-level
• Stores all Templates, ISOs and Snapshots
• Zone can have one or more secondary storages
• NFS, OpenStack Swift
19. Deployment Architecture
Data Center 1
Data Center 3
Zone1
Zone 4 CloudStack Clouds can have one
or more Availability Zones.
Data Center 2
Zone 2
Zone 3
20. Management Server Managing Multiple Zones
Cloud
Data Center 1 Data Center 2 Single Management Server can manage
Data Center 2
Management Data Center 3 multiple zones
Server
Zone 2 Zones can be geographically distributed but
Zone 2 low latency links are expected for better
Zone 3 performance
Zone1
Zone 4 3
Zone
Single MS node can manage up to 10K hosts.
Multiple MS nodes can be deployed as cluster
for scale or redundancy
Data Center 2
Data Center 2
Data Center 2
Zone 2
Zone 2
Zone 2Zone 3
Zone 3
Zone 3
21. Management Server Deployment Architecture
Single-node Deployment Multi-node Deployment
Management
User API User API Server
Management Load
MySQL
Server DB Balancer
Admin API Admin API
Management
Server MySQL
DB
Back Up
Replication DB
MS is stateless. MS can be deployed as
physical server or VM
Single MS node can manage up to 5K hosts. Infrastructure
Infrastructure
Multiple nodes can be deployed for scale or
Resources Resources
redundancy
Commercial: RHEL 5.4+; FOSS: Ubuntu
10.0.4, Fedora 16
Citrix Confidential - Do Not Distribute
22. Management Server Interaction with Hypervisors
Management
Server
XAPI HTTP
vCenter Agent Agent
XenServer
KVM OVM
ESX
• XS 5.6, 5.6FP1, 5.6 SP2, 6.0 • ESX 4.1, 5.0 (coming) • RHEL 6.0, 6.1, 6.2 (coming) • OVM 2.2
• Incremental Snapshots • Full Snapshots • Full Snapshots (not live) • No Snapshots
• VHD • VMDK • QCOW2 • RAW
• NFS, iSCSI, FC & Local disk • NFS, iSCSI, FC & Local disk • NFS, iSCSI & FC • NFS & iSCSi
• Storage over-provisioning: NFS • Storage over-provisioning: • Storage over-provisioning: NFS • No storage over-provisioning
NFS, iSCSI
25. Layer-3 Guest Network
Network Services Managed Externally Network Services Managed by CS
Public Network
65.11.0.0/16
Security Group 1
Security Group 1
Public
Network/Internet 65.11.1.2
65.11.1.2
Guest VM 1
Guest VM 1
65.11.1.3
65.11.1.3 NetScaler Guest VM 2
Guest VM 2 Load Blancer
EIP, ELB
65.11.1.4
65.11.1.4 Guest VM 3
Guest VM 3
65.11.1.5
65.11.1.5
Guest VM 4
Guest VM 4
CS
CS
DHCP, Virtual Security Group 2
DHCP, Virtual Security Group 2 Router
Router DNS
DNS
26. Layer-2 Guest Virtual Network
CS Virtual Router provides Network Services External Devices provide Network Services
Guest Virtual Network 10.0.0.0/8 Guest Virtual Network 10.0.0.0/8
VLAN 100 VLAN 100
Public Public
Network/Internet Network/Internet
Guest VM 1 Public IP Private IP Guest VM 1
10.1.1.1 10.1.1.111 10.1.1.1
65.37.141.11 Juniper SRX
Public IP Gateway address 1 Firewall
65.37.141.11 10.1.1.1
CS
Guest VM 2 Guest VM 2
Virtual Router 10.1.1.3 10.1.1.3
Public IP Private IP
DHCP, DNS 65.37.141.11 10.1.1.112
NAT NetScaler
Guest VM 3 2 Load Blancer Guest VM 3
Load Balancing 10.1.1.4 10.1.1.4
VPN
Guest VM 4 Guest VM 4
10.1.1.5 10.1.1.5
CS
DHCP, Virtual
Router
DNS
27. Network Offerings
• Same concept with disk and service offerings
• What can you control?
• Name
• Enable Redundant Router
• Control Network Rate
• Specify Network Services (Firewall, Loadbalancer, etc…)
• Specify Network Provider (VR, SRX, Netscaler, F5, etc…)
• Specify access (All, Domain, Account)
• Allow upgrade and downgrade across offerings.
28. Multi-tier virtual networking
Guest Virtual Network 10.1.1.0/24 Guest Virtual Network 10.1.2.0/24 Guest Virtual Network 10.1.3.0/24
VLAN 100 VLAN 101 VLAN 102
Public
Network/Internet
Web VM 1 App VM 1 DB VM 1
Gateway address 10.1.1.2 10.1.2.2 10.1.3.2
Public IP 10.1.1.1 Gateway address
65.37.141.11 CS 10.1.2.1 Gateway address
Virtual Router 10.1.3.1
DHCP, DNS
NAT
Web VM 3 App VM 2 DB VM 2
Load Balancing 10.1.1.3 10.1.2.3 10.1.3.3
VPN
Web VM 4 App VM 3
10.1.1.4 10.1.2.4
30. UI Cloud Portal CLI Other Clients
Management Server
REST API
OAM&P API End User API EC2 API Other APIs Pluggable Service API Engine
Console Proxy ACL & Authentication Security Adapters
Management - Accounts, Domains, and Projects
- ACL, limits checking
Account Management Connectors
Services API
Template Access
DB
Plugin API
Deployment Planning
HA
Orchestration Engine
Services API
- Drives long running VM operations Network Gurus
- Syncs between resources managed and DB
Usage Calculations - Generates events
Network Elements
Additional Services
Hypervisor Gurus
Cluster Resource Job Alert & Event Database
Management Management Management Management Access
Event Bus
Message Bus Usage Server
Resource API
Hypervisor Network Storage Image Snapshot
Resources Resources Resources Resources Resources
31. Orchestration Engine
• Understands how to orchestrate long running processes (i.e.
VM starts, Snapshot copies, Template propagation)
• Well defined process steps
• Calls Plugin API to execute functionalities that it needs
32. Plugins
• Various ways to add more capability to CloudStack
• Implements clearly defined interfaces
• All calls are at transaction boundaries
• Compiles only against the Plugin API module
33. Anatomy of a Plugin
• Can be two jars: server component to
be deployed on management server
and an optional ServerResource
Rest API
- Optional. Required only if needs to expose component to be deployed co-located
configuration API to admin.
with the resource
• Server component can implement
Plugin API
Implementation
multiple Plugin APIs to add its feature
Data Access Layer
• Can expose its own API through
Pluggable Service so administrators
can configure the plugin
-
ServerResource
Optional. Required if Plugin needs to be co-located with
• As an example, OVS plugin actually
-
the resource
Implements translation layer to talk to resource
implements both NetworkGuru and
- Communicates with server component via JSON
NetworkElement
34. Plugin Interfaces Available
• NetworkGuru – Implements various network isolation and ip address
technologies
• NetworkElement – Facilitate network services on network elements to support
a VM (i.e. DNS, DHCP, LB, VPN, Port Forwarding, etc)
• DeploymentPlanner – Different algorithms to place a VM and volumes.
• Investigator – Ways to find out if a host is down or VM is down.
• Fencer – Ways to fence off a VM if the state is unknown
• UserAuthenticator – Methods of authenticating a user
• SecurityChecker – ACL access
• HostAllocator – Provides different ways to allocate host
• StoragePoolAllocator – Provides different ways to allocate volumes
37. What you will learn
• How to customize the CloudStack 3.0.x user interface
• Showcase changes specific in the CSS to alter the look and feel of CloudStack
• Showcase an example of how to add your own side navigation
• Dealing with Cross Site Request Forgery (CSRF)
• Simple Single Signon
• Localization
38. What you will learn
• Working with the API
• Session Based Auth vs API Key Auth
• How to sign a request with apiKey/secretKey
• Asynchronous commands
• Response Format
• Pagination
41. CloudStack UI
• Reference implementation of the CloudStack API
• Built on HTML 4.0, CSS, and jQuery
• Uses Java Server Pages for localization only
• Three types of customizations
• Minor customizations – logo changes, minor CSS changes
• Major customizations – Changing tabs, adding additional links
• Complete rewrite – user UI is completely offloaded to a portal
47. Adding navigation buttons
1. Go to /ui/scripts/cloudStack.js
2. Add a new section to the array:
sections: {
dashboard: {},
instances: {},
storage: {},
network: {},
templates: {},
events: {},
accounts: {},
domains: {},
system: {},
projects: {},
'global-settings': {},
configuration: {},
// New section
testSection: {}
}
48. Adding navigation buttons (cont.)
3. Open /ui/index.jsp. Create HTML 4. Enclose a function in
somewhere in the 'template' div to 'testSection', which returns a
contain your HTML content, which will jQuery object
be drawn in the browser pane: containing your template code,
and whatever other content you
<!-- Templates --> wish to
<div id="template"> be shown:
<div class="testSection-tmpl"> sections: {
<h1>Test section</h1> dashboard: {},
instances: {},
</div> storage: {},
</div> network: {},
templates: {},
events: {},
accounts: {},
domains: {},
system: {},
projects: {},
'global-settings': {},
configuration: {},
// New section
testSection: {
title: 'Title for section',
show: function(args) {
return $('#template .testSection-
tmpl').clone();
}
}
}
49. Adding navigation buttons (cont.)
5. Add the section to the pre-filter, so that it isn't filtered out for
the admin account:
--
sectionPreFilter: function(args) {
if(isAdmin()) {
return ["dashboard", "instances", "storage", "network", "templates",
"accounts", "domains", "events", "system", "global-settings", "configuration",
"projects"];
},
sectionPreFilter: function(args) {
if(isAdmin()) {
return ["dashboard", "instances", "storage", "network", "templates",
"accounts", "domains", "events", "system", "global-settings", "configuration",
"projects",
// New section
"testSection"];
},
...
50. Adding navigation buttons (cont.)
7. (optional) Add an icon for your new section in the CSS, either at
the bottom of /ui/css/cloudstack3.css or in your own CSS file under
/ui/css folder. Make sure the size of the icon is ~32x32 pixels:
#navigation ul li.testSection span.icon {
background: url('../images/testSection-icon.png') no-repeat 0px 0px;
}
51.
52.
53. Cross Site Request Forgery (CSRF)
• Type of malicious exploit of a website whereby unauthorized
commands are transmitted from a user that the website
trusts. Unlike cross-site scripting (XSS), which exploits the
trust a user has for a particular site, CSRF exploits the trust
that a site has in a user's browse
• What does CS do to prevent this?
• After execution of the login command you will get two session variables
• JSESSIONID – default cookie
• SESSIONKEY – random token that is passed along every API request
• http://<API URL>?sessionkey=<SESSIONKEY>&…
55. Localization
• Support for Japanese and Simplified Chinese
• Takes advantage of the Java ResourceBundle to do localization
• Simply create a /WEB-INF/classes/resources/messages_<language
code>.properties
• Server side vs Client side processing
57. Session-based Auth vs API Key Auth
• CloudStack supports two ways of authenticating via the API.
• Session-based Auth
• Uses default Java Servlet cookie based sessions
• Use the “login” API to get a JSESSIONID cookie and a SESSIONKEY token
• All API commands require both cookie and token to authenticate
• Has a timeout as configured within Tomcat
• API Key Auth
• Works similarly to AWS API
• Requires a bit more coding to generate the signature
• All API commands require a signature hash
58. SIGNING REQUEST WITH API KEY / SECRET KEY
Step 1:
commandString = command name + parameters + api key
URL encode each field-value pair within the commandstring
Step 2:
Lower case the entire commandString and sort it alphabetically via the field for each field-value pair.
sortedCommandString :
apiKey=vmwijj…&command=createvolume&diskofferingid=1&name=smallvolume=zoneid=1
Step 3:
Take the sortedCommandString and run it through the HMAC SHA-1 hashing algorithm (most
programming languages offer a utility method to do this) with the user’s Secret Key. Base64 encode
the resulting byte array in UTF-8 so that it can be safely transmitted via HTTP. The final string
produced after Base64 encoding should be SyjAz5bggPk08I1DE34lnH9x%2f4%3D
59. Asynchronous Commands
• Starting with 3.0, in your standard CRUD (Create, Read, Update, Delete) of any
first class objects in CloudStack, CUD are automatically asynchronous. R is
synchronous.
• Rather than returning a response object, it will return a job ID.
• If it is a “Create” command, it will also return the object ID.
• With the job ID, you can query the async job status via the
queryAsyncJobResult command.
• The queryAsyncJobResult response will return the following possible job status
code:
• 0 - Job is still in progress. Continue to periodically poll for any status changes.
• 1 - Job has successfully completed. The job will return any successful response values associated with
command that was originally executed.
• 2 - Job has failed to complete. Please check the <jobresultcode> tag for failure reason code and
<jobresult> for the failure reason.
60. Response Formats
• CloudStack supports two formats as the response to an API
call.
• The default response is XML. If you would like the response
to be in JSON, add &response=json to the Command String.
62. Pagination
• Using the page and pagesize parameter
• page defines the current cursor to the list
• pagesize defines the number of items per request
• Pagesize is limited by the administrator
• Sample:
• listVirtualMachines&page=1&pagesize=500
• listVirtualMachines&page=2&pagesize=500
63. UI Plugin Framework
• Problems today?
• Any major customizations require modification of CloudStack UI code.
• Modifications require deep knowledge of CloudStack UI code.
• Versioning becomes difficult.
• Future Plugin Framework
• Creating UI widgets that are re-usable
• A JS configuration file that will allow partners/developers to specify how to
include their UI into the CloudStack UI without having to modify core
CloudStack UI code.
• Example could be left navigation link or possibly new actions. These are TBD.
66. Why Apache Software Foundation?
• Best governance
• 15+ years, 100+
projects
• 2500+ Developers
67. The Road to an Apache “Top Level Project”
• April: convert source code to Apache License
• April: announce intent to donate
• April: proposal for donation; get accepted to Incubator
• May: donation, mailing lists, enter Incubation
• Sept : Apache CloudStack 4.0 release
• 2012: work in the “Apache Way”
• Graduate to Top Level Project, contingent on:
• Community involvement
• Follow legal requirements and Apache standards
68. Implications for Partners and Customers
• CloudStack awareness increased
• CloudStack on path to be #1 orchestration software
• Apache license provides more options for enhancements
• More direct influence possible
• Better visibility into CloudStack development
69. Citrix CloudPlatform
• Citrix released CloudPlatform 3.0.3 mid June.
• Citrix plans to contribute 100% of development back into CloudStack
• Monetization remains the same before and after Apache.
• We expect Apache CloudStack to be 3 months ahead of CloudPlatform
• Citrix CloudPlatform will have a release schedule separate from CloudStack and
will be determined by business needs.
71. Apache Roles
• User
• A user is someone that uses our software. They contribute to the Apache
projects by providing feedback to developers in the form of bug reports and
feature suggestions. Users participate in the Apache community by helping
other users on mailing lists and user support forums.
• Developer
• A developer is a user who contributes to a project in the form of code or
documentation. They take extra steps to participate in a project, are active on
the developer mailing list, participate in discussions, provide patches,
documentation, suggestions, and criticism. Developers are also known
ascontributors .
72. Apache Roles (cont.)
• Committer
• A committer is a developer that was given write access to the code repository
and has a signed Contributor License Agreement (CLA) on file. They have
an apache.org mail address. Not needing to depend on other people for the
patches, they are actually making short-term decisions for the projectDeveloper
• PMC Member
• A PMC member is a developer or a committer that was elected due to merit for
the evolution of the project and demonstration of commitment. They have write
access to the code repository, an apache.org mail address, the right to vote for
the community-related decisions and the right to propose an active user for
committership. The PMC as a whole is the entity that controls the project,
nobody else.
73. Development Environment
• Development Machine
• Apache Tomcat, version 6.0.33. Set environment variable CATALINA_HOME to
point to your apache install directory.
• Mysql, version 5.1.58
• Git, the latest version
• Java, the latest version
• Ant, the latest version
74. Development Environment (cont.)
• To setup a Windows environment:
• http://wiki.cloudstack.org/display/dev/Setting+up+Cloudstack+dev+environment
+on+Windows
• To setup a Mac OS environment:
• http://wiki.cloudstack.org/display/dev/Setting+up+a+CloudStack+development+
environment+on+Mac+OSX
75. Development Environment (cont.)
• To get the CloudStack source code
• git clone https://git-wip-us.apache.org/repos/asf/incubator-cloudstack.git
• git checkout master
• To build CloudStack
• ant clean-all build-all deploy-server deploydb
• To start the Management Server
• ant debug
77. CloudStack Developer Mailing List
• This is where all CloudStack development discussion are
mostly held.
• All new features should be discussed on this mailing list.
• If you want to contribute to CloudStack, you are highly
encouraged to subscribe to the cloudstack-dev list if you
haven’t done so.
• To subscribe, email to cloudstack-dev-subscribe@incubator.apache.org
• You can also subscribe to the users list (cloudstack-users-
subscribe@incubator.apache.org)
• And to the commit list (cloudstack-commits-subscribe@incubator.apache.org)
78. How to Contribute
• Clone ASF cloudstack repo:
• git clone https://git-wip-us.apache.org/repos/asf/incubator-cloudstack.git
• Checkout master branch:
• git checkout master
• Write code, make sure it's properly unit-tested. Unit-tests
have to be submitted as a part of the patch
• Create the patch for review:
• git format-patch -o <dir of patch> --signoff master^
79. How to Contribute (cont.)
• Create Jira ticket (or use existing ticket) and attach the
patch:
• http://bugs.cloudstack.org/secure/Dashboard.jspa
• Submit the patch for review on Reviewboard for repository
"cloudstack-git":
• https://reviews.apache.org/r/new/
80. How to Contribute (cont.)
• Post on developer mailing list for review. Either the patch
will be directly merged into the master branch or a topic
branch will be created if it’s a large feature.
• If you contribute a lot of good patches to CloudStack, a PMC
member may decide to initiate a vote on your behalf to
become a full-time committer.
81. Resources
• CloudStack docs and knowledge base:
• http://docs.cloudstack.org/
• http://wiki.cloudstack.org/
• CloudStack architecture review:
• http://wiki.cloudstack.org/display/dev/CloudStack+Presentations
• CloudStack packages and dependencies:
• http://wiki.cloudstack.org/display/dev/CloudStack+Packages+and+Dependencie
s
82. Resources (Cont.)
• Exceptions handling in CloudStack:
• http://wiki.cloudstack.org/display/dev/Cloudstack+Error+Codes+and+Exception
+handling
• DB upgrade development for CloudStack:
• http://wiki.cloudstack.org/display/dev/DB+upgrade+in+CloudStack
• Git workflow and coding standards in CloudStack:
• http://wiki.cloudstack.org/display/dev/Git+workflow+in+the+brave+new+world#G
itworkflowinthebravenewworld-Creatingpatches
83. devCloud
• What is devCloud?
• DevCloud is a VirtualBox image, on which CloudStack management server +
Xen hypervisor are installed. CloudStack management server is running on
Ubuntu 12.04 dom0, can also add dom0 itself as a Xen hypervisor host and
create Linux virtual machines on it.
• As a developer, you can push your modified CloudStack code into DevCloud,
then deploy and run the CloudStack management server in DevCloud.
• As an user, you can access CloudStack management server running inside
DevCloud through web UI, large part of functionality of CloudStack are
supported in DevCloud, such as creating VM, taking snapshot, creating
template, console proxy, etc.
• http://wiki.cloudstack.org/display/COMM/DevCloud