"The Cloud" is a buzzword these days, but in this talk Nate Aune and Sally Kleinfeldt describe what you need to know about public and private clouds, why it's important, and why you might want to consider it for your company/organization. We'll discuss the pros/cons of public cloud providers such as Amazon Web Services and Rackspace Cloud, and how you can build your own private cloud using OpenStack or Eucalpytus, or even a hybrid that leverages both public and private clouds. Lastly, we'll look at how Plone can be deployed onto the cloud, and best practices for setting up your Plone site in a flexible and scalable way to take full advantage of all that these cloud providers have to offer. In particular, we'll examine the Connexions/Rhaptos project as a case study, and how they benefited from an Amazon-backed infrastructure. Video of the presentation is available here: https://streaming.psu.edu/media/?movieId=13392

1. Moving
to the
Cloud
Nate Aune
(Presented by Sally Kleinfeldt)
Jazkarta.com
Plone Symposium East 2011
Penn State University

2. Agenda
• Benefits of cloud computing
• What services does Amazon provide?
• 5 minute launch of Plone on EC2
• Case study: Rhaptos hosting on EC2
• Questions?

3. What makes the Cloud
so attractive?

4. Abstract resources
Focus on your needs, not on hardware specs.
As your needs change, so should your resources.

5. On-demand provisioning
Ask for what you need, exactly when you need it.
Get rid of it when you don’t need it.

6. Scalability in minutes
Scale out or in depending on usage needs.

7. Pay per consumption
No contracts or long-term commitments.
Pay only for what you use.

8. Efficiency of Experts
Utilize the skills, knowledge and resources of experts.

9. Most applications need:
• Compute
• Storage
• Messaging
• Payment
• Distribution
• Scale
• Analytics

10. Amazon delivers this

11. Characteristics of Truly
Scalable Service
• Increasing resources results in a
proportional increase in performance
• capable of handling heterogeneity, loosely
coupled
• operationally efficient
• resilient
• becomes more cost effective when it grows

15. Flexible infrastructure
• Elastic Computing Cloud (EC2)
• Amazon Machine Images (AMI)
• Elastic IPs
• Elastic Block Storage (EBS)
• EBS bootable AMI
• Auto-scaling
• Elastic Load Balancing (ELB)

16. Other services
• Simple Storage Solution (S3)
• CloudFront (CDN)
• CloudWatch (monitoring)

17. Compared to Rackspace
• Rackspace is still catching up to AWS (just
announced load balancing a few weeks ago)
• Rackspace is a bit easier to get started with
• sends you the root password
• machines already have a static IP
• the disk storage is persisted
• Prices are comparable to AWS
• Cheapest Rackspace VM is $14/mo

18. Private clouds
• Provision virtual machines on your own
hardware
• Pros: you control your own machines.
• Cons: you have to manage your own
machines
• Examples: Eucalyptus, OpenStack
http://open.eucalyptus.com
http://openstack.org/

19. http://uec-images.ubuntu.com/
Text

20. mr.awsome
Buildout recipe to manage EC2 instances

21. Add an AWS part to buildout
[buildout]
parts = aws
[aws]
recipe = zc.recipe.egg
eggs = mr.awsome
entry-points =
aws=mr.awsome:aws
assh=mr.awsome:aws_ssh
arguments = configpath="${buildout:directory}/etc"

22. Make a etc/aws.conf file
[securitygroup:demo-server]
description = Bristol Demo Server
connections =
tcp 22 22 0.0.0.0/0
tcp 80 80 0.0.0.0/0
tcp 8080 8080 0.0.0.0/0
[instance:demo-server]
keypair = bristol-keypair
securitygroups = demo-server
region = us-east-1
placement = us-east-1a
instance_type = t1.micro
image = ami-480df921
startup_script = startup-demo-server
fabfile = fabfile.py

23. Make a bash script:
etc/startup-demo-server
#!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive
wget http://www.enfoldsystems.com/pubkey.gpg
apt-key add ./pubkey.gpg
echo "deb http://dist.clients.enfoldsystems.com/ubuntu lucid
universe" >> /etc/apt/sources.list
apt-get update && apt-get upgrade -y
apt-get install plone-default -y
/etc/init.d/plone-default

24. Set your access keys
File: setkeys.sh
#!/bin/bash
export AWS_ACCESS_KEY_ID="<your_access_key_id>"
export AWS_SECRET_ACCESS_KEY="<your_secret_access_key>"
Set these environment variables.
$ source setkeys.sh
$ env

25. Start the demo server
$ ./bin/aws start demo-server
INFO: Instance 'demo-server' unavailable
INFO: Creating instance 'demo-server'
INFO: Instance created, waiting until it's available
....
INFO: Instance 'demo-server' available
INFO: Instance running.
INFO: Instances DNS name ec2-50-16-25-92.compute-1.amazonaws.com
INFO: Instances public DNS name ec2-50-16-25-92.compute-1.amazonaws.com

26. Benefits of EC2
• Seamless disaster recovery with repeatable
deployments using buildout and mr.awsome
• Easy launching of testing, staging and QA
servers that are isolated from your
production environment (stop when not
used to avoid paying for them)
• Pay-for-what-you-use metered pricing to
launch servers on-demand during peak
periods
• Auto-scaling to launch new Zeo client
machines and shut them down when no
longer needed

27. Connexions & Rhaptos
A case study in deploying OSS to the cloud

28. Log In Contact Us Report a Bug
Search
Home Content Lenses About Us Help MyCNX
You are here: Home
MY ACCOUNT
Connexions is: FIND CONTENT
a place to view and share educational material made of Username
16547 reusable modules woven into 1007 collections.
small knowledge chunks called modules that can be Password
organized as courses, books, reports, etc. Anyone may view Search Content Go
Log in
or contribute: or browse by ...
Arts Get an account
authors create and collaborate Subject
Business Forgot your password?
s
instructors rapidly build and share custom collections Language Humanities
Mathematics and Statistics
learners find and explore content Popularity
Science and Technology
More about us ... New features Title, author, etc. Social Sciences
FEATURED CONTENT
CREATE CONTENT
Advanced Algebra II
Creating content in Connexions is as easy as 1, 2, 3:
The world of second-year SPOTLIGHT
algebra comes alive in Kenny
Felder's Advanced Algebra II. User feedback
Felder, a high school math
"I enjoy using
teacher in North Carolina,
the
designed his course using a Connexions
nontraditional approach to a very site and the
traditional subject. Centered
around a Homework and 1 Get an account
and log in to 2 Make a module
from scratch or 3 Publish your
works, sharing
philosophy
behind it
Activities book, along with your workspace. convert it from a them with the world. encourages
accompanying Conceptual Word doc. one to learn
Explanations and Teacher's Guide collections, new fields and
Advanced Algebra II tackles topics by focusing on to network with others. Your
Jump right in Guides and tutorials
comprehension rather than straight lecture delivery. site is my new homeschool on
Get an account New author guide the internet. It is helping me
Advanced Algebra II was selected by California's Free to grow as a writer of
Digital Textbook Initiative to be included in a list of How to create a module in Connexions Tutorial and academic materials as I read
free textbooks available to California schools starting minutes Reference what others have written so as
in Fall 2009. How to create a collection to shape my own. I am also
amazed at the range of
with existing modules institutions where your writers
Collaborative Statistics come from. Thanks for the use
of these great facilities."
Collaborative Statistics was
written by two faculty members

29. Search
Login Preferences Help/Guide About Trac
Wiki Timeline Roadmap Browse Source View Tickets New Ticket Search
Start Page Index History Last Change
Welcome to Connexions and Rhaptos Software Development
Connexions ( cnx.org) is legos for education. It is a world-wide repository and publishing system to build textbooks, courses, lessons, and
journals. Individual authors, major content publishers, and educational organizations all contribute. Lenses provide a way for high-quality material
to be endorsed, recommended, categorized, and aligned to standards. Rhaptos is the open source software that runs the Connexions site and
Enterprise Rhaptos is the software for running private content installations. Read more about Connexions and Rhaptos mission and architecture
here.
Enterprise Rhaptos Connexions and Rhaptos Development Table of Contents
Welcome to Connexions and Rhaptos Software Development
Enterprise Rhaptos
API's for Accessing Connexions Data
Accessing content, feeds, and statistics
API's for Accessing Data Quality Assurance and Testing Downloading, transforming and branding content
Software Architecture and Application Programming Interfaces
Connexions Rhaptos Development
Developer Resources (Getting involved)
Roadmap, Designs, Desires
Starting Points for Connexions and Rhaptos Software Development
Quality Assurance and Testing
Test, Release, and Configuration Docs
Using the Connexions Rhaptos Trac Bug System
Release and Configuration Details for Connexions and Rhaptos
Enterprise Rhaptos More about TRAC
Enterprise Rhaptos enables the creation, sharing, modification, and vetting of educational materials accessible to anyone, anywhere, anytime via
the World Wide Web. Rhaptos' unique features of permanent versions of all materials, a simple, semantic document language, and a powerful
lensing system for post-publication quality control, customized tagging, and community-based search and discovery are the engine behind a truly
reusable repository of knowledge and learning. Enterprise Rhaptos allows companies and individual organizations to host their own private
authoring platform and publishing repository.
Installation Instructions for Enterprise Rhaptos
API's for Accessing Connexions Data

30. Virtualization of Rhaptos
Two primary reasons to move to the cloud:
1) Hurricane territory (disaster recovery)
2) Increase adoption by other universities
Cost was also a consideration but not a
primary reason for carrying out the project.

31. Deploy to Amazon EC2
with Python scripts
• Define server profiles with OS dependencies
and turn into an AMI (Amazon Machine Image)
• Install & configure application on EBS volume
• Take snapshot of volume
• Launch new AMIs and attach EBS volume
created from saved snapshot.

33. Distributed Architecture
Internet
s2 frontend1
Webserver
Cache Sys s3 frontend2
Load Balancer
backend
s1 proxy s4 frontend3 s5

34. aws.conf - define security groups
[macro:base-securitygroup]
connections =
tcp 22 22 0.0.0.0/0
tcp 80 80 0.0.0.0/0
[securitygroup:backend]
<= macro:base-securitygroup
description = Bristol backend machine running Zeo server
[securitygroup:frontend1]
<= macro:base-securitygroup
description = Bristol frontend machine running Zeo client 1
connections =
tcp 8080 8080 0.0.0.0/0
[securitygroup:frontend2]
<= macro:base-securitygroup
description = Bristol frontend machine running Zeo client 2
connections =
tcp 8080 8080 0.0.0.0/0
[securitygroup:proxy]
<= macro:base-securitygroup
description = Bristol proxy machine running Apache, Varnish, HAProxy

35. [macro:base-instance]
image = ami-480df921
keypair = bristol-keypair
region = us-east-1
placement = us-east-1c
instance_type = t1.micro
fabfile = fabfile.py
[instance:backend]
<= macro:base-instance
securitygroups = backend
startup_script = backend-startup.sh
ip = xxx.xxx.xxx.xxx
[instance:frontend1]
<= macro:base-instance
securitygroups = frontend1
startup_script = frontend-startup.sh
ip = xxx.xxx.xxx.xxx
[instance:frontend2]
<= macro:base-instance
securitygroups = frontend2
startup_script = frontend-startup.sh
ip = xxx.xxx.xxx.xxx
[instance:proxy]
<= macro:base-instance
securitygroups = proxy
startup_script = proxy-startup.sh
ip = xxx.xxx.xxx.xxx

36. [buildout]
parts =
backend-script
frontend-script
proxy-script
[backend-script]
recipe = collective.recipe.template
input = deployment/templates/zope-startup-script.sh
output = deployment/backend-startup.sh
svn_url = http://svn.jazkarta.com/.../bristol-buildout/trunk
buildout_config = backend.cfg
[frontend-script]
recipe = collective.recipe.template
input = deployment/templates/zope-startup-script.sh
output = deployment/frontend-startup.sh
svn_url = http://svn.jazkarta.com/.../bristol-buildout/trunk
buildout_config = frontend.cfg
[proxy-script]
recipe = collective.recipe.template
input = deployment/templates/proxy-startup-script.sh
output = deployment/proxy-startup.sh
svn_url = http://svn.jazkarta.com/.../bristol-buildout/trunk
buildout_config = proxy.cfg

37. #!/bin/bash
set -e -x
export DEBIAN_FRONTEND=noninteractive
apt-get update && apt-get upgrade -y
echo "----- Install system packages for Plone"
apt-get -y install python2.6-dev subversion build-essential
echo "----- Create an unprivileged user zope with password disabled"
adduser zope --disabled-password --gecos ""
echo "----- Check out the buildout"
cd /home/zope
echo 'pn' | svn co http://svn.jazkarta.com/.../.../trunk buildout
echo "----- Make the zope user the owner of this directory"
chown -R zope:zope buildout
echo "----- Run the buildout as user zope"
cd buildout
sudo -u zope python2.6 bootstrap.py -c backend.cfg
sudo -u zope ./bin/buildout -v -c backend.cfg
echo "----- Start Zeo processes"
sudo -u zope ./bin/supervisord
sudo -u zope ./bin/supervisorctl status
echo "----- Make sure that Supervisor starts on reboot"
ln -s /home/zope/buildout/etc/supervisor /etc/init.d/supervisor
chmod +x /etc/init.d/supervisor
update-rc.d supervisor defaults

38. Must define the FQDN
of the Zeo clients and
Zeo servers
[buildout]
backend-dns-name = ec2-174-129-192-63.compute-1.amazonaws.com
frontend1-dns-name = ec2-174-129-116-33.compute-1.amazonaws.com
frontend2-dns-name = ec2-184-73-240-104.compute-1.amazonaws.com
When these names are resolved *on* the server, the DNS
server returns the private IP, which means you avoid paying
bandwidth charges for outgoing/incoming traffic.

39. Start up the servers
with one command
$ ./bin/aws start backend
$ ./bin/aws start frontend1
$ ./bin/aws start frontend2
$ ./bin/aws start proxy

40. Fault tolerant
Eliminating single points of failure.

41. Distributed Architecture
Internet
s2
Webserver
Cache Sys s3
Load Balancer
s1 s4 s5

42. Heartbeat
monitors Zeo
server and moves to
backup in case of
failure
DRDB
Distributed storage
system similar in
principle in RAID,
but which works
over the network.

43. Improvements to the
deployment process
Before After
• time to launch a new site
was days
• brittle non-repeatable
installation process
• delay in procuring
hardware and installing
OS dependencies

44. Improvements to the
deployment process
Before After
• time to launch a new site • time to launch a new site
was days was minutes
• brittle non-repeatable • streamlined and
installation process automated installation
process
• delay in procuring
hardware and installing • virtual machines are
OS dependencies launched instantly and
OS dependencies are
already installed on the disk
image.

45. Fabric
Scripts to deploy to remote hosts.
http://www.fabfile.org

46. Fabric
set(fab_user='plone',
fab_hosts=['localhost'],
local='/home/plone/prod/',
remote='/opt/zope/prod/',
site='budapest.buildout')
def deploy():
local('cd $(local)$(site)')
local('tar cvfz $(site).tgz buildout --exclude=.svn --exclude=*.pyc')
run('cd $(remote)$(site); ./bin/supervisorctl shutdown')
run('rm -rf $(remote)$(site)')
put('$(site).tar.gz', '$(remote)$(site).tar.gz')
run('cd $(root)$(site) && tar zxf $(site).tar.gz')
start()
def start():
run('cd $(remote)$(site)')
run('./bin/supervisord')

47. Deploy
$ fab deploy
What does this one command do?
• Provides a local packaging of the source into
a tarball
• Removes the old remotely deployed code
• Puts the source on the remote host
• Restarts the site

48. Benefits of Fabric
• Chain local-command, remote-command,
and remote-copy commands
• Substitute variables, and the splitting of
distinct operations (like in the case of the
'restart' command) so you run them
separately, with a fab restart, for example.

49. Example Fabric file
• See Alex Clark’s Fabric file:
• http://github.com/aclark4life/Debian-Deploy-
Plone/blob/master/fabfile.py

50. Boto
• Python library to script AWS services
• Programmatically:
• Launch new EC2 instances
• Attach EBS volumes
• Create S3 buckets
http://code.google.com/p/boto/

51. collective.hostout
Buildout-based recipe for deploying to remote host
http://plone.org/products/collective.hostout

52. [buildout]
extends = buildout.cfg
parts += hostout prod
auto-checkout += collective.hostout
[sources]
collective.hostout = svn https://svn.plone.org/svn/collective/collective.hostout/trunk
[hostout]
recipe = collective.hostout
user = root
#password = blah
identity-file = /Users/nateaune/id_rsa-gsg-keypair
pre-commands =
# ${buildout:directory}/bin/supervisorctl shutdown || echo 'Unable to shutdown'
# post-commands =
# ${buildout:directory}/bin/supervisord
#effective-user = zope
[prod]
recipe = collective.hostout
extends = hostout
host = ec2-75-101-211-135.compute-1.amazonaws.com
buildout = buildout.cfg
path = /opt/zope/prod

53. Run the deploy
commands
$ ./bin/buildout -N
Installing prod.
Generated script '/bin/hostout'.
$ bin/hostout deploy
Invalid hostout hostouts are: prod
$ bin/hostout deploy prod
...

54. Chef & Puppet
• Configuration management tools to prepare
machines to behave according to their role.
• Unlike Fabric which “pushes” config to the
server, with Chef/Puppet, the server “pulls” it’s
configuration from a master node.
• Usually overkill for a simple deployment, but
with a multi-server deployment can be useful,
especially to keep machines up-to-date.
http://opscode.com/chef/
http://www.puppetlabs.com/

55. Questions?
• http://rhaptos.org
• http://pypi.python.org/pypi/mr.awsome
• http://fabfile.org
• http://code.google.com/p/boto/
• http://pypi.python.org/pypi/collective.hostout