Autoscaling OpenStack Natively with Heat, Ceilometer and LBaaS workshop I delivered at OpenStack Vancouver Summit (May, 2015) jointly with Jason and Sharmin from Cisco System. More details can be found at https://github.com/grimmtheory/autoscale

1. 201 Walkthrough
Autoscaling OpenStack
natively with Heat,
Ceilometer and LBaaS

2. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Speakers
Jason Grimm
Consulting Systems Engineer
Cisco OpenStack Private Cloud
Sharmin Choksey
Technical Leader
Cisco Cloud Services
Shixiong Shang
Chief Technology Officer
Nephos6
Agenda
Introductions <
About The Workshop <
Environment Setup <
Background & Use Cases <
Services Enabling Autoscale <
Hands-On <

3. About The Workshop Tips and Logistics
Content and Schedule

4. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
▪ We all work for or with Cisco, but this is not a marketing workshop
▪ Highly technical and all content is 100% FOSS (Free and Open-Source Software)
▪ All Materials available at https://github.com/grimmtheory/autoscale
▪ Please connect to http://cisco.webex.com/rooms/jasgrimm
▪ Chat, voting, questions, recording, etc.
▪ Allows us to see when everyone is ready to proceed
▪ Session recording will be made available after the summit
▪ Please work with your peers at your table!
▪ Bandwidth constraints and technical issues mean might not get a test environment up in time
▪ Some of you may have more experience with this technology and can assist others
▪ General collaboration and discussion
Workshop Tips and Logistics

5. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Content and Schedule
Introductions
About The Workshop
** Environment Setup **
Autoscale Background and Use Case
Services Enabling Autoscale
Hands-On Workshop
Lab 1
Lab 2.1
Lab 2.2
Lab 3
Q & A
Open Discussion
Close
30 Minutes
(or less)
45 Minutes
(or more)
15 Minutes
(or less, as needed)

6. Environment Setup
Virtual Box, Ubuntu, DevStack
Install test environment
Install content location and examples
Test environment logical architecture
Test environment configuration
Test environment build process

7. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Install Test Environment…
Instructions
Before we can begin working with LBaaS,
autoscale, etc. We need an environment.
For this purpose we’ve chosen a locally
installed environment due to past
experiences with WIFI coverage and
bandwidth at OpenStack summits.
1) Mount media
2) Copy data folder over locally
3) Install VirtualBox
4) Install Vagrant (Optional)
5) Boot Virtual Machine
DVD Local Repo USB Key Internet
HTTP HTTPMOUNTMOUNT
Step 1
Mount Data
Step 2
Copy Data
Step 3
Install Apps
Step 4
Boot VM

8. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Install content location and examples
Also available at:
• My VM - http://192.168.1.100/autoscale
• My Cloud Server - http://38.84.67.227/autoscale
• VirtualBox - https://www.virtualbox.org/wiki/Downloads
Vagrantfile
Git clone example
Wget example

9. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Test environment logical architecture

10. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Test environment configuration
Disabled
Trove
Sahara
Cinder
Nova Network
Tempest
Swift
Enabled
MySQL
RabbitMQ
Keystone
Horizon
Glance
Neutron
Ceilometer
Heat
OS & OpenStack Configuration
$HOME = /home/vagrant/devstack
Operating System = Ubuntu 14.04.x
OpenStack version = “stable/kilo”
eth0 = NAT, default GW
eth1 = Management / API network
eth2 = Neutron / Bridge network
Open vSwitch
L3 / DHCP / Floating IP
“Flat” networking also available
No tenant VLANs, FWaaS or VPNaaS
Vagrant / VirtualBox Configuration
Vagrant v.1.7.2
VirtualBox 4.3.28
eth0 = NAT
eth1 = Host Only 192.168.33.2 /24
eth2 = Host Only 0.0.0.0
VT-x off, APIC-IO off, promiscuous on eth2

11. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Test environment build process
Methodology
• Tools-based install and config of the OS,
DevStack and post-install tasks.
• Automate difficult configuration of services
and pre-requisites (Neutron, Heat,
Ceilometer, OVS, custom images, etc.)
Build Options
• Base - OS install & DevStack configured
• Staged - Base + DevStack install completed
and basic configuration (security groups,
networks and subnets created, etc.)
• Complete - Staged + all objects created
(LBs, VIPs, Ceilometer alarms, Heat
templates, etc.)
CREATE MACHINE
2 vCPU, 4 GB RAM, VT-x, APIC-IO
3 NICs - eth0 NAT, eth1 API, eth3 VMs
Promiscuous mode for Neutron Bridge
DEVSTACK PRE INSTALL
Install pre-requisites
Clone git repositories
Configure local.conf, bridges,
routes users, groups,iptables,
ovs, sysctl, permissions, etc.
INSTALL OS
Base Install
Patch / Update
Install VBox
extensions
DEVSTACK INSTALL
Install and monitor
DevStack Install
DEVSTACK POST
INSTALL BASIC
Add DNS to subnets
Edit security groups
Create / install SSH keys
DEVSTACK POST
INSTALL ADV.
Create load balancers
Add members to pool
Create VIPs
COMPLETE
Snapshot the
virtual machine

12. Background
and Use Cases
What is
Autoscaling and
when / how is it
used?

13. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Autoscale Background and Use Cases
What is Autoscaling…?
“…a method used in cloud computing, whereby the
amount of resources in a server farm scales
automatically based on the load on the farm. It is closely
related to, and builds upon, the idea of load balancing.”
Who uses Autoscaling…?
Any workload that benefits from programmatic scaling up or down
as load dictates.
Particularly true from a cost perspective when high workloads are
very infrequent.
Some examples are:
• Retail - Black Friday, holidays, video game / movie launch, etc.
• Academic / Research - HPCC
• Media - Video / Audio rendering
• Analytics - Big data, data mining, data modeling
• Security - Federal or private data heuristic, cyber defense, etc.
How does Autoscaling work…?
Server
Load
Stress
Meter
Alarm Action
Server
Result+ + + +

14. Services Enabling
Autoscaling
An overview of:
Heat
Ceilometer
Neutron w/ LBaaS

15. Heat Overview

16. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
HEAT
OpenStack Orchestration Service
Heat makes “clouds” rise
heat
The heat tool is a CLI which communicates with
the heat-api to execute AWS CloudFormation
APIs. End developers could also use the heat
REST API directly.
heat-api
The heat-api component provides an OpenStack-
native REST API that processes API requests by
sending them to the heat-engine over RPC.
heat-api-cfn
The heat-api-cfn component provides an AWS
Query API that is compatible with AWS
CloudFormation and processes API requests by
sending them to the heat-engine over RPC.
heat-engine
The heat engine’s main responsibility is to
orchestrate the launching of templates and
provide events back to the API consumer.
Infrastructure resource abstraction and orchestration:
• Physical compute Power
• Physical switches
• Physical block Storage
• …
OpenStack resource abstraction and orchestration:
• VM Instances
• Virtual Networks
• Virtual Routers
• …
“Typically” Virtual
(See Ironic Project)

17. Ceilometer & Neutron LBaaS Overview
Ceilometer & Heat Overview

18. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Ceilometer Overview
1"
Polling"agents"
Meter" Transformer(s
)"
New"
Meter"
Publishing"Pipeline"
API" Database"
AMQP"
Receiver"
Collector"
Other"
OpenStack""
Service"
alarm"
evaluator"
Publisher(s)"
alarm"
evaluator"
Alarm"

19. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Ceilometer and Neutron LBaaS
2"
Polling"agents"
Meter"
Neutron"
(LBaaS)"
Neutron"
API"
LBaaS"Default"Meters" Type" Unit"
network.services.lb.ac0ve.connec0ons1 gauge1 connec0on1
network.services.lb.incoming.bytes1 cumula0ve1 B1
network.services.lb.total.connec0ons1 cumula0ve1 connec0on1
network.services.lb.vip1 gauge1 vip1

20. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Ceilometer and Neutron LBaaS
3"
Transformer(s
)"
New"
Meter"
Publishing"Pipeline"
Publisher(s)"
sinks:&
&&&&'&name:&lb_total_conn&
&&&&&&transformers:&
&&&&&&&&&&'&name:&"rate_of_change"&
&&&&&&&&&&&&parameters:&
&&&&&&&&&&&&&&&&source:&
&&&&&&&&&&&&&&&&&&&map_from:&
&&&&&&&&&&&&&&&&&&&&&&&name:&"network.services.lb.total.connec;ons"&
&&&&&&&&&&&&&&&&&&&&&&&unit:&"connec;on"&
&&&&&&&&&&&&&&&&target:&
&&&&&&&&&&&&&&&&&&&&map_to:&
&&&&&&&&&&&&&&&&&&&&&&&&name:&"network.services.lb.total.connec;ons.rate"&
&&&&&&&&&&&&&&&&&&&&&&&&unit:&"connec;ons/s"&
&&&&&&&&&&&&&&&&&&&&type:&"gauge"&
&&&&&&publishers:&
&&&&&&&&&&'&no;fier://&
sources:&
&&&&'&name:&lb_total_conn&
&&&&&&interval:&60&
&&&&&&meters:&
&&&&&&&&&&'&"network.services.lb.total.connec;ons"&
&&&&&&sinks:&
&&&&&&&&&&'&lb_total_conn&
Configuration file: /etc/ceilometer/pipeline.yaml

21. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Ceilometer and Neutron LBaaS
4"
API" Database"
Receiver"
Collector"
ceilometer sample-list -m network.services.lb.total.connections.rate!
+--------------------------------------+--------------------------------------------+-------+--------+---------------+---------------------+!
| Resource ID | Name | Type | Volume | Unit | Timestamp |!
+--------------------------------------+--------------------------------------------+-------+--------+---------------+---------------------+!
| 9d944aec-7d66-46b5-a71b-434e4e9fa98f | network.services.lb.total.connections.rate | gauge | 0.0 | connections/s | 2015-05-13T02:45:02 |!
| 9d944aec-7d66-46b5-a71b-434e4e9fa98f | network.services.lb.total.connections.rate | gauge | 0.0 | connections/s | 2015-05-13T02:44:02 |!
| 9d944aec-7d66-46b5-a71b-434e4e9fa98f | network.services.lb.total.connections.rate | gauge | 0.0 | connections/s | 2015-05-13T02:43:02 |!
+--------------------------------------+--------------------------------------------+-------+--------+---------------+---------------------+!
!
!
!
ceilometer statistics -m network.services.lb.total.connections.rate!
+--------+---------------------+---------------------+-----+-----+-----+-----+-------+----------+---------------------+---------------------+!
| Period | Period Start | Period End | Max | Min | Avg | Sum | Count | Duration | Duration Start | Duration End |!
+--------+---------------------+---------------------+-----+-----+-----+-----+-------+----------+---------------------+---------------------+!
| 0 | 2015-05-13T02:16:02 | 2015-05-13T02:47:02 | 0.0 | 0.0 | 0.0 | 0.0 | 32 | 1860.0 | 2015-05-13T02:16:02 | 2015-05-13T02:47:02 |!
+--------+---------------------+---------------------+-----+-----+-----+-----+-------+----------+---------------------+---------------------+!

22. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Ceilometer and Neutron LBaaS
5"
API"
alarm"
evaluator"
alarm"
evaluator"
Alarm"
ceilometer alarm-threshold-create --name lb_conn_rate_hi !
--description 'lb connection rate is too high' --severity critical --enabled True !
--meter-name network.services.lb.total.connections.rate !
--period 60 --evaluation-periods 3 --statistic avg --comparison-operator gt --threshold 2.0 !
--repeat-actions True!
ceilometer alarm-list!
+----------+-----------------+--------+----------+---------+------------+-----------------------------------------------------------------+!
| Alarm ID | Name | State | Severity | Enabled | Continuous | Alarm condition |!
+----------+-----------------+--------+----------+---------+------------+-----------------------------------------------------------------+!
| uuid | lb_conn_rate_hi | ok | critical | True | True | network.services.lb.total.connections.rate > 2.0 during 3 x 60s |!
+----------+-----------------+--------+----------+---------+------------+-----------------------------------------------------------------+!

23. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Ceilometer and Heat
6"
Polling"agents"
Meter" Transformer(s
)"
New"
Meter"
Publishing"Pipeline"
API" Database"
AMQP"
Receiver"
Collector"
alarm"
evaluator"
Publisher(s)"
alarm"
evaluator"
Alarm"
Heat"

24. Hands-On
Workshop
Heat feature progression
Basic Heat Operations
Advanced Heat Operations
Autoscale

25. ‹#›
Auth Model
• Icehouse
• Security Isolation
• Keystone Domain “heat”
• Stack Domain Users managed by
heat_admin_domain
• Pre-configured in devstack
• Requires Keystone v3
Resource Groups
• Icehouse
• De-coupling hierarchical nested
stacks
Stack Snapshot/Rollback
Provider Resources
• Icehouse
• Useful in Template composition
TripleO Heat Templates
• Leverages Resource Groups,
Provider Resources and
SoftwareConfig Resource
Key enhancements in Icehouse and
• Kilo
SoftwareConfig Resources
• Icehouse
• Useful in Template composition

26. ‹#›
26
Heat Autoscaling - Lab Exercise - 1 (Simple-Server)
Spawn a VM
Injects an ssh key
Security groups
VM port fixed ip + security
group
Floating ip associated it with the
fixed ip
Installs a simulated http server
via the -init user-data script
heat stack-create simple-stack -f simple-server.yaml
--parameters
"key_name=<key_name>
;node_name=<node_name>
;node_server_flavor=<node_server_flavor>
;node_image_name=<node_image_name>;
;floating_net_id=<floating_net_id>;
;private_net_id=<private_net_id>;
;private_subnet_id=<private_subnet_id>"
User-defined
Constrained and/or
default values
Type/Description
Fundamental
blocks
Consumed by the
Heat engine to
orchestrate

27. ‹#›
27
Heat Autoscaling - Lab Exercise - 2 Part 1 (Load balancer)
lb vip :: floating ip
curl -X GET http://128.107.2.111
503 Service Not Available

28. ‹#›
28
Heat Autoscaling - Lab Ex - 2 Part 2 (Adding Members)
web-server.yaml environment.yaml
heat stack-create lb-members-stack -f lb-members.yaml
-e environment.yaml --parameters
"key_name=<key_name>
;node_name=<node_name>
;node_server_flavor=<node_server_flavor>
;node_image_name=<node_image_name>;
;floating_net_id=<floating_net_id>;
;private_net_id=<private_net_id>;
;private_subnet_id=<private_subnet_id>;
;pool_id=<pool_id>
;capacity_count=<capacity_count>"
curl -X GET http://128.107.2.111
Welcome to 192.168.5.116
curl -X GET http://128.107.2.111
Welcome to 192.168.5.117
Nested
Resource
Identically
Configured
lb-members.yaml
Provider
Resource

29. ‹#›
Scales arbitrary #
of resources
Resource to
be scaled
29
Heat Autoscaling - Lab Ex - 3 (Autoscaling)
lb-autoscale-members.yaml
Creates
an Alarm
on a
Action to be
taken upon Alarm
Raise
Action to be
executed against
resource
Connection rate crosses 3 per second for a
consecutive period of 60 seconds each, the
lb_connection_rate_high will be raised and a
scaleup_policy will get executed

30. References
Web Links
(Content we used as reference
when building this lab)
Extra slides

31. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
Summit Lab / Workshop
▪ Materials for the presentation are available for download at https://github.com/grimmtheory/autoscale
DevStack
▪ DevStack Configuration Guide – http://docs.openstack.org/developer/devstack/configuration.html
Ceilometer
Heat
▪ Heat main page - https://wiki.openstack.org/wiki/Heat
▪ Heat developer documentation -http://docs.openstack.org/developer/heat/
Heat operations documentation
Heat main github repository
Heat templates github respository
Heat user template guide
Heat developer template guide
▪ Heat Orchestration Template (HOT) guide – http://docs.openstack.org/developer/heat/template_guide/hot_guide.html
▪ Heat on DevStack guide – http://docs.openstack.org/developer/heat/getting_started/on_devstack.html
References

32. Cisco Confidential ‹#›© 2013-2014 Cisco and/or its affiliates. All rights reserved.
References
Title Link
Autoscale workshop repository https://github.com/grimmtheory/
Heat: main page https://wiki.openstack.org/wiki/Heat
Heat: developer documentation http://docs.openstack.org/developer/heat/
Heat: operations documentation
Heat main github repository