1.
An introduction
Scale in the Cloud
Created by: Simone Soldateschi
Modified Date: 2014-06-02
Classification: Public Conference

2.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Who am I?
Simone Soldateschi
• Java, C/C++, PHP, Python developer
• More than 8 years experience as SysAdm/SysEng
• Developer Support Engineer at Rackspace
• Task automation enthusiast
• MTB’ing, triathlon, photo, manga
@soldasimo
simonesoldateschi
2

3.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Who are Rackspace?
Founded in 1998 in San Antonio, TX by three guys that
wanted to create a hosting company
Home of Fanatical Support /o/
Second biggest Public Cloud in the world
OpenStack Project co-founder

4.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
To be recognized as one of the
world’s great service companies.
“
”
Rackspace Vision
4

5.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Python SDK, Cloud
• Auto Scaling
• Management System
• Control law
• Garçon, all together now!
Roadmap
5

6.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Install SDK
$ mkvirtualenv pyconru
New python executable in pyconru/bin/python
Installing setuptools, pip...done.
6
(pyconru)$ pip install pyrax ipython
Downloading/unpacking pyrax
Downloading pyrax-1.8.1-py2.py3-none-any.whl (316kB): 316kB downloaded
Downloading/unpacking ipython
Downloading ipython-2.1.0-py2-none-any.whl (2.8MB): 2.8MB downloaded
…
Successfully installed pyrax ipython …
Cleaning up…
See: https://github.com/rackspace/pyrax

7.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Authentication
7
# authenticate
pyrax.set_setting('identity_type', 'rackspace')
pyrax.set_credentials(os.getenv('OS_AUTH_USER'),
os.getenv('OS_AUTH_APIKEY'),
region=os.getenv('OS_AUTH_REGION'))
print “authenticated: %s” % pyrax.identity.authenticated

8.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Authentication
Check credentials:
(pyconru)$ python -m pyconru.basic
(DEBUG) OS_AUTH_USER: foo
(DEBUG) OS_AUTH_APIKEY: ****
(WARNING) OS_AUTH_REGION undefined, using default 'LON'
(DEBUG) authenticated: True
(INFO) identity token: cfe6d60f070947bf****************
8
Define environment variables:
(pyconru)$ export OS_AUTH_USER=foo
(pyconru)$ export OS_AUTH_KEY=bar
(pyconru)$ export OS_AUTH_REGION=LON

9.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Cloud components
9

10.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Cloud components
10

11.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Python SDK, Cloud
• Auto Scaling
• Management System
• Control law
• Garçon, all together now!
Roadmap
11

12.
simone.soldateschi@rackspace.co.uk
Vertical scaling
2 GB
2 CORES
8 GB
8 CORES

13.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
What is Autoscaling?

14.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
What is Autoscaling?
WASTED $$$

15.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
What is Autoscaling?

16.
simone.soldateschi@rackspace.co.uk
New Usage Models
CLOUDSMART
16
Dedicated Servers are Pets
• Great thought to their acquisition
• Name them and know each one
• Willing to pay big money for their care
Cloud Servers are Livestock
• Use them as long as they provide value
• Acquire more of them when needed
• Dispose of any that aren’t needed
• Get rid of them if they become ill

17.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
http://www.flickr.com/photos/fischerfotos/7419253200/

18.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Traffic Patterns

19.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Traffic Patterns
ON & OFF
Analytics
Banks/Tax Agencies
Test environments

20.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Traffic Patterns
FAST GROWTH
Events
Business Growth
Slashdot Effect

21.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Traffic Patterns
VARIABLE
News & Media
Event Registrations
Rapid fire sales

22.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Traffic Patterns
CONSISTENT
HR Application
Accounting/Finance
E-mail

23.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
http://www.flickr.com/photos/maximalideal/3356408693/

24.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Autoscaling Methodologies
Time Based
Reactive
Predictive

25.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Time Based Autoscaling
Load
Balancer
Server Server

26.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Time Based Autoscaling
Load
Balancer
Server Server
9:00am

27.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Time Based Autoscaling
Load
Balancer
Server Server
Nov 1st

28.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Time Based Autoscaling
Load
Balancer
Server Server Server

29.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Time Based Autoscaling
GOOD FOR
On & Off
Consistent

30.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Reactive Autoscaling
Load
Balancer
Server
60%
Server
60%

31.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Reactive Autoscaling
Load
Balancer
Server
80%
Server
80%

32.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Reactive Autoscaling
Load
Balancer
Server
60%
Server
60%
Server
40%

33.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Reactive Autoscaling
Load
Balancer
Server
30%
Server
30%
Server
30%

34.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Reactive Autoscaling
Load
Balancer
Server
45%
Server
45%

35.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Reactive Autoscaling
GOOD FOR
Fast Growth
Variable

36.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Predictive Autoscaling
Load
Balancer
Server Server

37.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Predictive Autoscaling
Load
Balancer
Server Server
Forecasted
Traffic
+30%

38.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Predictive Autoscaling
Load
Balancer
Server Server Server

39.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Predictive Autoscaling
GOOD FOR
Fast Growth
Variable

40.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto scaling - Schedule-based scaling

41.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto scaling - Schedule-based scaling

42.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto scaling - Event-based scaling

43.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto scaling - Event-based scaling
SCALE UP

44.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto scaling - Event-based scaling
COOL DOWN

45.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
45
Cooldown

46.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto scaling - Event-based scaling
COOL DOWN

47.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto scaling - Event-based scaling
SCALE DOWN

48.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto Scale – Use case
FRONT END

49.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto Scale – Use case
LB
FRONT END
Share nothing
Stateless nodes

50.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto Scale – Use case
LB
FRONT END
LB
API
BOSS WORKER

51.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
http://www.flickr.com/photos/samuraislice/3309481048/

52.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Python SDK, Cloud
• Auto Scaling
• Management System
• Control law
• Garçon, all together now!
Roadmap
52

53.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
The basics
That’s it!
$ pip install ansible
Installation on management host

54.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
The basics
Install agent on managed hosts:

55.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Why use ansible?
Desired state
Go live!

56.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Desired State
Write code to tell the
computer
how to set up itself!
56RACKSPACE® HOSTING | WWW.RACKSPACE.COM

57.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Python SDK, Cloud
• Auto Scaling
• Management System
• Control law
• Garçon, all together now!
Roadmap
57

58.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Closed-Loop Control Law
58

59.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Event-based Auto Scale

60.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Auto Scale

61.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Closed-loop Control Law – Garçon implementation
61
?

62.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Python SDK, Cloud
• Auto Scaling
• Management System
• Control law
• Garçon, all together now!
Roadmap
62

63.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon
How to integrate
Cloud Monitoring
and
Auto Scale
?
63

64.
simone.soldateschi@rackspace.co.uk
Garçon - How?
Garçon

65.
simone.soldateschi@rackspace.co.uk
Garçon - Overview
Garçon
cm2asd
cfgmgmtd

66.
simone.soldateschi@rackspace.co.uk
Garçon - Go Live!
cfgmgmtd
Go Live!

67.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Closed-Loop Control Law – Garçon implementation
67
Garçon

68.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon in-depth – cm2asd
68
# fetch current list of servers
l_current_servers = scaling_group_servers(scaling_group_id)
Garçon

69.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon in-depth – cm2asd
69
for i in range(len(l_current_servers)-1, -1, -1):
server_id = l_current_servers[i]
s = get_server(server_id)
Garçon
if s.status != 'ACTIVE':
# server not active
l_current_servers.pop(i)
continue
m = get_server_metadata(s.id)
try:
if (m['aspoc.server_status'] != 'configured'):
server not configured
l_current_servers.pop(i)
continue
except KeyError:
# server not configured
l_current_servers.pop(i)
continue

70.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon in-depth – cm2asd
70
Garçon

71.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon in-depth – cm2asd
71
# compute average system load for scaling group
servers_avg_load = servers_average_load(l_checks, samples,
sample_time)
# compare current load against configured threshold
if servers_avg_load >= threshold_high:
# trigger scale_up_webhook
r = requests.post(scale_up_webhook)
if r.status_code != 202:
logger.error('scale_up_webhook (%s) returned HTTP %d' %
(scale_up_webhook, r.status_code))
Garçon

72.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon in-depth – cm2asd
72
if servers_avg_load <= threshold_low:
# trigger scale_down_webhook
r = requests.post(scale_down_webhook)
if r.status_code != 202:
logger.error('scale_down_webhook (%s) returned HTTP %d' %
(scale_down_webhook, r.status_code))
Garçon

73.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon in-depth – cfgmgmtd
73
for s_id in l_current_servers:
...
# server exists?
try:
cs = pyrax.cloudservers
cs.servers.get(s_id)
except:
logging.warning('Auto Scale server (%s, %s) missing '
'(maybe deleted manually?)' % ('-', s_id))
continue
...
try:
# read server metadata
m = get_server_metadata(s_id)
...

74.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon in-depth – cfgmgmtd
74
Use metadata
try:
if (server_status != 'configured' and
server_status != 'configuring'):
...
# run thread to configure server
threading.Thread(target=configure_server, args=(s_id,)).start()
No metadata?
except KeyError:
# CONFIGURE server (KeyError, no metadata) in thread
threading.Thread(target=configure_server,
args=(s_id, ansible_timeout,)).start()
?
X

75.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon in-depth – cfgmgmtd, Ansible
75
Reset server’s password:
# set server password
password = generate_password(10, punctuation=False)
set_server_password(server_id, password)
Server’s info (e.g. IP address):
# fetch server info
ip = get_server_ipv4(server_id, MGMT_NETWORK)

76.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon in-depth – cfgmgmtd, Ansible
76
Leverage Configuration Management System:
for playbook in list_ansible_playbooks():
...
cmd = ['ansible-playbook', playbook_filename,
# '-vvvv',
'-c', 'paramiko', '-i', inventory_file]
errcode = run_cmd(cmd, logfilename=playbook_logfilename,
timeout=ansible_timeout)
playbook_logfilename = (os.path.join(LOG_DIR, '%s-%s' %
(s.name, playbook)))
playbook_logfile = open(playbook_logfilename, 'w')
playbook_filename = (os.path.join((playbooks_base_dir),
'%s/main.yml' % playbook))

77.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
Garçon in-depth – cfgmgmtd, Monitoring System
77
Create checks for new server to be managed
# cloud monitoring (agent_id := server_uuid)
add_cm_cpu_check(server_id))
# good, set 'aspoc_server.status=configured' in metadata
set_server_metadata(server_id, 'aspoc.server_status', 'configured')

78.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
• Python SDK, Cloud
• Auto Scaling
• Management System
• Control law
• Garçon, all together now!
RECAP
78

79.
RACKSPACE® HOSTING | WWW.RACKSPACE.COM
79
Q&A
@soldasimo
simonesoldateschi

80.
RACKSPACE® HOSTING | 5000 WALZEM ROAD | SAN ANTONIO, TX 78218
US SALES: 1-800-961-2888 | US SUPPORT: 1-800-961-4454 | WWW.RACKSPACE.COM
RACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COMRACKSPACE® HOSTING | © RACKSPACE US, INC. | RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN TH E UNITED STATES AND OTHER COUNTRIES. | WWW.RACKSPACE.COM
RACKSPACE® HOSTING | 5 MILLINGTON ROAD | HAYES, UNITED KINGDOM UB3 4AZ
UK SALES: +44 (0)20 8712 6507 | UK SUPPORT: 0800 988 0300 | WWW.RACKSPACE.CO.UK
@soldasimo
simonesoldateschi