CloudFoundry Elastic Architecture on OpenStack

CloudFoundry: Elastic Architecture
Design and Deploy Based on
OpenStack
郭峰 (Golfen Guo)
@教授也是浮云
golfen.guo@emc.com

Cloud Platform and Application Lab, EMC Labs China

© Copyright 2012 EMC Corporation. All rights reserved. 1

EMC Labs China

Advanced Technology
Research and Development
University
Collaboration
Vision
Big Data Lab Become an elite
research and
advanced
technology institute
Industry
Cloud Infrastructure in China
Standards Office
and System Lab -
Become the model
for future EMC Labs
Cloud Platform and worldwide
Applications Lab IP Portfolio
Development


Agenda
• What is PaaS
• CloudFoundry Overview
• Elastic Architecture in CloudFoundry
• Introduce to BOSH
• CPI and OpenStack
• Deploy PaaS with BOSH


What is PaaS? (from Search Cloud
Computing)
• Platform as a Service (PaaS) is a way to rent hardware, operating
systems, storage and network capacity over the Internet.

• The service delivery model allows the customer to rent virtualized
servers and associated services for running existing applications or
developing and testing new ones.

RENT, not BUILD one!

Virtual Server and Associated Service


How PaaS Benefit Us?

Before: Now:


How PaaS Benefit Us? (Cont.)
From Accelerating your Journey to Application Transformation, EMC World
2012
PaaS powered

You Manage
Traditional IT IaaS PaaS

You Manage
Applications Applications Applications
Agility and Data
Data Data
Cost Savings

Managed by Platform

Managed by Platform
Runtime
You Manage

Runtime Runtime
Middleware Middleware Middleware
O/S O/S O/S
Virtualization Virtualization Virtualization
Servers Servers Servers
Storage Storage Storage
Networking Networking Networking


Cloud Foundry – The Open PaaS


PaaS in Action…
Another analogy from Mark Lucovsky,

Every one knows, “Hello World”:

$ cat hw.c
#include <stdio.h>
main() {
printf(“Hello Worldn”);
}

$ cc hw.c; ./a.out


PaaS in Action... (Cont. )
“Hello World” in the Cloud:

$ cat hw.rb
require 'rubygems'
require 'sinatra'

$hits = 0
get '/' do
$hits = $hits + 1
"Hello World - #{$hits}"
end

$ vmc push hw


Scale App On Demand
Implement a Scalability “Hello World” in Cloud:

Step 1: Scale instance to 10:

$ vmc instances hw 10

Step 2: Change the codes:

get '/' do
$hits = $redis.incr(‘hits’)
"Hello World - #{$hits}" Redis
end

Step 3: bind Redis service to App:

$ vmc create-service redis –bind hw


Scale App On Demand(Cont.)
From Cloud Foundry Launch Event, April, 2011 PaaS powered


Simplify Deployment
[mysqld]
user = foobar
port = 3306
Before: basedir = /usr
bind-address = 172.58.77.101
key_buffer = 16M
thread_stack = 128K
thread_cache_size = 8
…

[nginx]
http.include mime.types;
default_type: application/octet-stream;
Web App log_format: main ‘$remote_addr - $remote_user []…’
keepalive_timeout 65;

[tomcat]
<Connector redirectPort=‚8443‛ emptySessionPath…/>
<bean id=‚sessionFactory‛ class=‚org.springframework…/>

[frontend]
dependencies:
- mysqlclient
- ruby
files:
- core/app/fe/**/*
- core/common/**/*

[blah]
- blah blah blah


Simplify Deployment (Cont.)

Now:
# to target and login to cloud foundry
vmc target http://api.cloudfoundry.com
vmc login

Web App # to create and boot the app for the first time
vmc push myapp –instances 2 –mem 64M –path ../code

# to create the database and bind it to the app
vmc create-service mysql –name mydb –bind myapp

# update live app with new code
vmc update myapp –path ../code


First Touch CloudFoundry
prompt> gem install vmc
prompt> vmc target api.cloudfoundry.com
prompt> vmc login
prompt> vmc push

Would you like to deploy from the current directory? [Yn] Yes
Application Name: hello
Application Deployed URL: 'hello.cloudfoundry.com'? hello-bob.cloudfoundry.com
Detected a Sinatra Application, is this correct? [Yn] Yes
Memory Reservation [Default:128M] (64M, 128M, 256M, 512M or 1G) (Press Enter to take
default)
Would you like to bind any services to 'hello'? [yN]: No
Uploading Application:
Checking for available resources: OK
Packing application: OK
Uploading (0K): OK
Push Status: OK
Staging Application: OK
Starting Application: OK


First Touch CloudFoundry(Cont.)
Upload your app
Memory of each instance

Change served instances
Your apps deployed
Start, stop, update, restart

Services of current app

Your all created services Information of app


PaaS Architecture Pattern

MySQL
RabbitMQ
MongoDB

Redis


PaaS Architecture Pattern (Cont. )


Conclusion
Abstract to three layers:
• Routers for finding right endpoint of Apps
• Runtime for Apps
• Services provided by platform, consumed by
Apps


The Keys of Design…
• Elastic
• Failover/System Robust
• Resource Recycling


Elastic Architecture in CloudFoundry
Sends droplet

Design principals:
heart beats and
exit messages

1. Each components can be run standalone;
Router

2. Each components can be scale-out, and notify the peersand with
Router
Registers
message;
Registers and
unregisters unregisters

3. The components communicate only with message or RESTful API.
Routes droplet
Routes REST API
requests
requests
Droplet change
notiﬁcations

Self-government and LooseController Cloud Couples : Stop, Find)
Droplet
Cloud Orchestrates
Health Manager start/stop Controller (Start, Droplet Execution
requests

• Easy to add new components. eg Stager, UAA, ACM…; Agent (DEA)

• Easy to evolve each component. eg. CC_ng, Router v2…
Periodically scans Persists droplets
for consistency and provisioned Guest applications
Advertise
services consume
Cloud Service
Controller
Provision and
Database
unprovision

Service "A"
Provision Service
and "A"
Provisioning Agent
unprovision


Elastic Architecture in CloudFoundry
Sends droplet A request comes…
heart beats and
exit messages

Router
Router

Registers and Registers and

Routes droplet
Routes REST API
requests
requests
Droplet change
notiﬁcations

Droplet
Cloud Controller Orchestrates
Health Manager start/stop Cloud Controller (Start, Stop, Find) Droplet Execution
requests
Agent (DEA)

Advertise
services consume
Cloud Service
Controller
Provision and
Database
unprovision

Service "A"
Provision Service
and "A"
Provisioning Agent
unprovision


Pluggable Runtime Support
Refers to https://github.com/cloudfoundry/vcap-staging

• stager -> vcap-staging klass = StagingPlugin.load_plugin_for(plugin_name)
plugin = klass.from_file(config_path)
plugin.stage_application

App
Stager StagingPlugin Start Stop
Script Script Server

PHP Rails3 Java_Web Play …

extends StagingPlugin


Pluggable Runtime Support (Cont.)
So what we need to do is…
• Extends Class StagingPlugin in
Common.rb

Line3 ~ Line 62, 59 lines of
codes to support PHP. 


Pluggable Runtime Support (Cont.)
Key methods to rewrite:
• stage_application
• start_command
• startup_script
• stop_command
• stop_script


Pluggable Services Support
Refers to a nice presentation by Nicholas Kushmerick
Cloud Foundry Services in last forum:
• Service advertisement
• Service Gateway -> Cloud Controller
• POST /services/v1/offerings

• DELETE /services/v1/offerings/:label
• Instance management
• Cloud Controller -> Service Gateway
• Provision: POST /gateway/v1/configurations

• Bind: POST /gateway/v1/configurations/:id/handles

• Unbind: DELETE /gateway/v1/configurations/:id/handles/:handle

• Unprovision: DELETE /gateway/v1/configurations/:id


But …
• Modular Design
• Version Tolerance
• Flexible Runtime/Service
• Elastic Architecture

Trade Off • Many kinds of nodes
• Many nodes each kind
Complex deployment process like other
distribution system


Practical problem at CloudFoundry.com
40+ unique node types
75+ unique software packages
500-5,000 VMs
2x/week cf.com updates
24*7*365 non-stop operation
No-downtime deployments
Reliable ,robust ,repeatable
deployments ,updates , capacity
adjustments
Small teams manage many
instances


How to deploy CloudFoundry?


CloudFoundry BOSH
CloudFoundry BOSH is an open source tool-chain for release
engineering, deployment, and lifecycle management of large scale distributed
services
– Prescriptive way of creating releases and managing systems and services
– It is not a collection of shell scripts, not a pile of Perl
Built to deploy and manage production-class, large scale clusters
– Production grade Cloud Foundry clusters: 500+ VMs, 40+ jobs, 75+ packages
– Multi-node, multi-tier, complex clusters
Built for DevOps usage and scale by a crack team of veterans
– A project, not a product: command line interface, YAML, etc.
– Continuous improvement, iterative development, rough edges

https://github.com/cloudfoundry/bosh


CloudFoundry BOSH (Cont.)
Built from the need to operate cloudfoundry.com
– Over 40 different types of VMs
– Hundreds of running instances of VMs
– Pushing at least 2 changes per week
End-to-end management
– Release engineering
– Deployment
– Lifecycle management
Generic solution
– Any service
– Any cloud
– Any operating system


BOSH Architecture
1. Upload Stemcell
1. CLI -> Director
2. Director -> Blobstore
2. Upload Release
1. CLI -> Director
2. Director -> Blobstore
3. Deploy
1. CLI -> Director
2. Director -> A
3. Agent -> B
4. Agent -> C


BOSH Concepts
Stemcell
– VM template
– BOSH Agent
– IaaS Plugin

Release
– Jobs

Job
– Packages
– Templates (scripts, confs)
– Monitoring

Package
– Source/blobs
– Dependencies
– Packaging (scripts)


IaaS Neutral

vSphere: battle tested implement AWS: code complete

Cloud Foundry BOSH

Cloud Provider Interface(CPI)

OpenStack: testable release https://github.com/piston/
openstack-bosh-cpi


Cloud Provider Interface
Stemcell
– create_stemcell (image, cloud_properties)
– delete_stemcell (stemcell)
VM
– create_vm (agent_id, stemcell, resource_pool, networks, disk_locality, env)
– delete_vm (vm)
– reboot_vm (vm)
– configure_networks (vm, networks)
Disk
– create_disk (size, vm_locality)
– delete_disk (disk)
– attach_disk (vm, disk)
– detach_disk (vm, disk)


Cloud Provider Interface (Cont.)
For OpenStack
Stemcell
– create_stemcell
…
– delete_stemcell
Cloud Provider Interface
VM
– create_vm
– delete_vm
– reboot_vm OpenStack VM
VM
Image Service VM
– configure_networks VM
VM
Disk
– create_disk OpenStack
– delete_disk Compute
– attach_disk
– detach_disk


Deploy PaaS with BOSH
Sends droplet
heart beats and
exit messages

Router
Router

Registers and Registers and

CLI Routes REST API
Routes droplet
requests
requests
Droplet change
notiﬁcations

Droplet
Cloud Controller Orchestrates
Health Manager start/stop Cloud Controller (Start, Stop, Find) Droplet Execution
requests
Agent (DEA)

Advertise
services consume
Cloud Service
Controller
Provision and
Database
unprovision

Service "A"
Provision Service
and "A"
Provisioning Agent
unprovision

BOSH

IaaS


Deployments
Release
Network
Resource pools
Jobs
Properties
Update concurrency
Compilation workers
Cloud properties


Deployments for CloudFoundy
Cloudfoundry.yml
name: cloudfoundry

release:
name: cloudfoundry
version: 89.1-dev

compilation:
workers: 4
network: default
cloud_properties:
ram: 1024
disk: 2048
cpu: 2

update:
canaries: 1
canary_watch_time: 3000-90000
update_watch_time: 3000-90000
max_in_flight: 2
max_errors: 1


Deployments for CloudFoundy (Cont.)
Cloudfoundry.yml
networks:
- name: default
subnets:
- static:
- 192.168.2.50 - 192.168.2.89
range: 192.168.2.0/24
gateway: 192.168.2.1
dns:
- 10.254.174.10
cloud_properties:
name: PrivateNetwork
- name: lb
subnets:
- static:
- 192.168.2.90 - 192.168.2.99
range: 192.168.2.0/24
gateway: 192.168.2.1
dns:
- 10.254.174.10
cloud_properties:
name: PrivateNetwork


Cloudfoundry.yml
resource_pools:

- name: infrastructure
network: default
size: 29
stemcell:
name: bosh-stemcell
version: 0.6.2
cloud_properties:
ram: 256
disk: 2048
cpu: 1
env:
bosh:
password:


Cloudfoundry.yml
jobs:

- name: cloud_controller
template: cloud_controller
instances: 1
resource_pool: infrastructure
networks:
- name: default
static_ips:
- 192.168.2.60

- name: nats
template: nats
instances: 1
resource_pool: infrastructure
networks:
- name: default
static_ips:
- 192.168.2.52


Deployments(CloudFoundy)
Cloudfoundry.yml

properties:
domain: cflocal.com

env: {}

networks:
apps: default
management: default

nats:
user: nats
password: aaa3ij3122
address: 192.168.2.52
port: 4222


Use Case
bosh deployment
bosh target cf
cf.yml

Bug report to Dev git pull

Run bosh upload release

Done! bosh deploy


Acknowledgments

VMware China R&D Center Network & Information Center,
Shanghai Jiao Tong Univ.

CloudFoundry Community Piston Community
http://www.cloudfoundry.org https://github.com/piston/open
stack-bosh-cpi


Q&A


Backup Slides


Management Path– vmc info

db
cloud controller

nats health manager
router

dea

service app app


Deployment Path– vmc push

db
cloud controller

nats health manager
router

dea

service app app


Data Path– Client Request

db
cloud controller

nats health manager
router

dea

service app app


Evolving Cloud Foundry - Kernel
uaa stager

db
Package cloud controller
cache v2
v2

nats health manager
v2 v2
v2
router v2
v2

warden dea

caldecott
service app app


How to write a custom service?
• Public API for advertising services
and managing service instances Cloud
• Endpoints: Controll
– Cloud Controller er
– Service Gateway
(one per service)
• REST / JSON / HTTP(S)
MySQL
Gatewa MongoD
y B
Gatewa
y


How to write a custom service? (Cont.)
Step 1: Service advertisement

Service Gateway -> Cloud Controller, two methods:
 POST /services/v1/offerings
 DELETE /services/v1/offerings/:label
:label => 'mysql-5.1'
:url => 'http://12.34.56.78:4321',
:plans => ['free', 'bronze', 'silver', 'gold']
:tags => ['mysql', 'mysql-5.1', 'database']
:active => true,
:description => 'MySQL database service'
:acls => {
:wildcards => '*@vmware.com',
:plans => {
:bronze => {
:users => ['joe@vmware.com', 'jane@vmware.com']
}
}
}


How to write a custom service? (Cont.)
Step 2: Instance management

Cloud Controller -> Service Gateway, four methods:

 Provision: POST /gateway/v1/configs
 Bind: POST /gateway/v1/configs/:id/handles
 Unbind: DELETE
/gateway/v1/configs/:id/handles/:handle
 Unprovision: DELETE /gateway/v1/configs/:id


Reference implementation
Develope A
vmc create-service postgresql
r
vmc, STS
B
POST /services/v1/configurations

Cloud Controller
C G H
{host: 12.34.56.78, port: 3456,
POST /gateway/v1/configurations
VCAP_SERVICES = [postgres:
db: abc123, user: pqr456}
{host: 12.34.56.78, port: 3456,
db: abc123, user: pqr456}]
Service Gateway
F
D
Private API
{host: 12.34.56.78, port: 3456, App
NATS: provision SELECT *
db: abc123, user: pqr456}
E FROM FOO

CREATE DATABASE abc123
CREATE ROLE pqr456
I GET /foo/bar
Service Node
(eg, PostgreSQL)
Service
(eg, PostgreSQL) host 12.34.56.78
End
Users


For legacy service – Service Broker
• Cloud Foundry users may already have existing services
• Legacy database that hasn’t / won’t / can’t be moved
CloudFoundry.com
• Services not yet supported
• Service Broker exposes an arbitrary legacy service instance to Cloud
Foundry so that it can be used just like any other instance


For legacy service – Service Broker
(Cont.) B
D register hrdb {host: 87.65.43.21,
port: 6543, db: def123, user: stu456}
vmc create-service hrdb

vmc, STS Cloud Controller Service Broker
Admin Tools
E H
POST /services/v1/configurations VCAP_SERVICES = [hrdb:
POST /gateway/v1/configurations G
{host: 87.65.43.21, port:6543,
db: def123, user: stu456}] port: 6543,
{host: 87.65.43.21,
C
F
POST /broker/v1/configuration
db: def123, user: stu456}

Service Gateway App Connecti
on
Details
Service Broker
A
Core Service
CREATE DATABASE def123
(eg, PostgreSQL) CREATE USER stu456
SELECT *
FROM FOO Brokered Service
host 12.34.56.78
(eg, Oracle DB)
GET /foo/bar I


CloudFoundry Elastic Architecture on OpenStack

CloudFoundry Elastic Architecture on OpenStack

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a CloudFoundry Elastic Architecture on OpenStack

Semelhante a CloudFoundry Elastic Architecture on OpenStack (20)

Mais de OpenCity Community

Mais de OpenCity Community (20)

Último

Último (20)

CloudFoundry Elastic Architecture on OpenStack

Notas do Editor