Operating your open stack private cloud rackspace

Operating your
OpenStack Private

Ryan Richard
OpenStack Engineer
ryan.richard@rackspace.com
@rackninja

October 12, 2012

Monitoring and Reporting
Where we were - April 2012

Basic CDM

RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Monitoring and Reporting
Where we were - April 2012

Basic CDM
Now


Tools

There is no good way to get the following info:

I need a list of instances on a host and their IPs

I need to gracefully start/stop all instances on a host

Some tools needs hostname, some need id (decimal or
hex), some need uuid


Tools

There is no good way to get the following info:

I need a list of instances on a host and their IPs

I need to gracefully start/stop all instances on a host

Some tools needs hostname, some need id (decimal or
hex), some need uuid

SELECT
instances.id,instances.hostname,instances.project_id,fixed_ips.address
as fixed_address,floating_ips.address as floating_address FROM instances
LEFT JOIN fixed_ips ON instances.id=fixed_ips.instance_id LEFT JOIN
floating_ips ON floating_ips.fixed_ip_id=fixed_ips.id WHERE
instances.deleted="NULL" AND instances.host="<hostname of physical
machine>" ORDER BY instances.id;


Tools
WE NEED BETTER OPS TOOLS!

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4

Tools
WE NEED BETTER OPS TOOLS!

Pulsar

https://github.com/
rsoprivatecloud/pulsar

“nova swiss army knife”

requires direct nova
database access

RACKSPACE® HOSTING | WWW.RACKSPACE.COM 4

Tools
Holland (opensource database backup framework)

Written by Rackspace DBAs

http://wiki.hollandbackup.org/


Tools
dsh

dsh -Mcg compute uname-a


Tools
dsh

dsh -Mcg compute uname-a

bashfoo

for i in `knife node list | grep cpu`; do knife node run_list
add $i "role[single-compute]"; done

for k in `seq 1 20`; do for i in {compute,network}; do nova-
manage service disable computevm0$k nova-$i; done; done


Performance and Scale Considerations
Disk IO

For high performance
use remote block storage

For “local” disk IO, raw
image type is only
slightly faster than
qcow2

IO will degrade while
Glance copies images
between machines

scheduler=cfq, KVM
cache=none


Disk IO Async&Random&IO&

rs/speed/test12"(cfq,"host"deadline,"cache=none)"

For high performance
Rs/speed/test13"(noop,"cache=writeback)"

use remote block storage rs/speed/test13"(cfq,"cache=writeback)"

Rs/speed/test12"(noop"cache=none)"

randW"(direct)"
Rs/speed/test12"(cfq"cache=none)"
randR"(direct)"

For “local” disk IO, raw Rs/speed/test13"(cfq,"cache=none,"no"ht)"
randW"

image type is only
randR"

Rs/speed/test13"(deadline"cache=none)"

slightly faster than compute/host"(deadline)"

qcow2 compute/host"(no"ht)"

compute/host"

0" 200" 400" 600" 800" 1000" 1200" 1400" 1600"

IO will degrade while Host&vs.&Instance&

Glance copies images 14000"

between machines 12000"

10000"

scheduler=cfq, KVM
8000"

compute/host"

cache=none 6000" Rs/speed/test12"(cfq"cache=none)"

4000"

2000"

0"
randR" randW" randR" randW" seqR" seqW"RACKSPACE® HOSTING
seqR" seqw" | WWW.RACKSPACE.COM
(direct)" (direct)" (direct)" (direct)"

Glance chunk Size

200Mb chunk
size

1GB chunk size

5GB chunk size


Swift disk usage with different chunk sizes

5 zones - 4 x 1TB disks per zone
20TB raw - 6.67TB usable RACKSPACE® HOSTING | WWW.RACKSPACE.COM

Swift disk usage with different chunk sizes



Glance chunk size

Too high and swift can become unbalanced

What are the downsides to being too low?


Glance

Disk Tuning (swift)

read ahead on your block device(s) - no noticeable
gain

deadline scheduler - no noticeable gain

Best thing for glance performance - Caching


Glance

Disk Tuning (swift)

gain



Image Size Not Cached Cached

1.4GB 20secs 1sec

16.4GB 2min 21secs 1sec


Glance

Disk Tuning (swift)

gain



Image Size Not Cached Cached *times from
“creating image” to
1.4GB 20secs 1sec “qemu-img create”

16.4GB 2min 21secs 1sec



Scheduler

What we use by default:

scheduler tasks are not processed in parallel

Adding additional schedulers helps provide HA but they don’t speed up
overall times to complete requests


Automated Conﬁg Management

Chef: http://github.com/rcbops/chef-
cookbooks

time to stand up

controller - less than 20 minutes

compute node - less than 2 min


Day to Day tasks

Dealing with new issues

resize - all nova-compute processes need to be able to log
into all other compute nodes via ssh keys

Hardware failures

We’re still managing infrastructure, failures happen


Lessons Learned

We need better Operations tools!

Network Design can be confusing for people used to “the old way”

OpenStack is still relatively new, help your organization understand it.

It’s easy to forget we’re working with Linux machines

It’s not you, it’s a bug :)


But....

But this is a design summit also

Open to discussions/thoughts/questions


Rackspace is hiring
www.rackertalent.com

RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM

RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM

Operating your open stack private cloud rackspace

Recomendados

Recomendados

Mais conteúdo relacionado

Destaque

Destaque (20)

Operating your open stack private cloud rackspace

Notas do Editor