1. Real-world Django Deployment with Chef
DjangoCon 2011
Noah Kantrowitz
noah@opscode.com
@kantrn

2. Who am I?
• Django hacker
• Ruby user
• Developer
• Sys admin

3. Who are you?
• System administrators?
• Designers?
• Developers?
• “Business” People?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/

4. What are we talking
about?
http://www.flickr.com/photos/peterkaminski/2174679908/

5. Agenda
• How’s and Why’s
• Getting Started with Chef
• Python-specific Tools
• Case Study: Packaginator
http://www.flickr.com/photos/koalazymonkey/3590953001/

6. Infrastructure as Code

7. A technical domain
revolving around
building and
managing
infrastructure
programmatically

8. Enable the reconstruction
of the business from
nothing but a source code
repository, an application
data backup, and bare
metal resources.

9. Configuration
Management

10. System Integration
http://www.flickr.com/photos/opalsson/3773629074/

11. n-Tier Infrastructure
Load Balancer
App Server { { App Server
•
•
•
Provision
Configure
Integrate
Database Master

13. The Chef Framework
With thanks (and apologies) to Stephen Nelson-Smith

14. The Chef Framework
• Reasonability
• Flexibility
• Library & Primitives

15. The Chef Tool(s)
With thanks (and apologies) to Stephen Nelson-Smith

16. The Chef Tool(s)
• ohai
• chef-client
• chef-server
• knife

17. The Chef API
With thanks (and apologies) to Stephen Nelson-Smith

18. The Chef API
• Client/Server
• RESTful API w/ JSON
• Search Service
• Derivative Services

19. The Chef Community
With thanks (and apologies) to Stephen Nelson-Smith

20. The Chef Community
• Apache License, Version 2.0
• 400+ Individual contributors
• 90+ Corporate contributors
• Dell, Rackspace,VMware, RightScale,
Heroku, and more
• 240+ cookbooks
• http://community.opscode.com/

21. Chef Enables Infrastructure as Code
package "haproxy" do
action :install
end
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
• Resources owner "root"
group "root"
• Recipes mode 0644
•
notifies :restart, "service[haproxy]"
Roles end
• Source Code service "haproxy" do
supports :restart => true
action [:enable, :start]
end

22. Chef Resources
package "haproxy" do
action :install
end
• Have a type. template "/etc/haproxy/haproxy.cfg" do
•
source "haproxy.cfg.erb"
Have a name. owner "root"
• Have parameters. group "root"
mode 0644
• Take action to put the resource notifies :restart, "service[haproxy]"
end
in the declared state.
• Can send notifications to other
service "haproxy" do
supports :restart => true
resources. action [:enable, :start]
end

23. Resources take action
through Providers

24. Chef Providers
package “haproxy”
{ yum install haproxy
apt-get install haproxy
pacman sync haproxy
pkg_add -r haproxy

25. Common Resources
package "apache2"
template "/etc/apache2/httpd.conf"
cookbook_file "/etc/apache2/key.pem"
user "www-data"
execute "/etc/init.d/apache2 restart"

26. Idempotence
execute "createdb myapp" do
not_if "psql -c list | grep myapp"
end
• Convergence
execute "createdb myapp" do
• Guard clauses only_if do
db.query("SELECT ...").first == 0
end
end

27. Recipes are collections
of Resources

28. Chef Recipes
package "haproxy" do
action :install
end
template "/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
• Recipes are evaluated for owner "root"
resources in the order they group "root"
mode 0644
appear. notifies :restart, "service[haproxy]"
• Each resource object is added end
to the Resource Collection. service "haproxy" do
supports :restart => true
action [:enable, :start]
end

29. Chef Recipes
• Recipes can include other include_recipe
include_recipe
"apache2"
"apache2::mod_rewrite"
recipes. include_recipe "apache2::mod_deflate"
• Included recipes are
include_recipe
include_recipe
"apache2::mod_headers"
"apache2::mod_php5"
processed in order.

30. Chef Recipes
pool_members = search("node", "role:mediawiki")
%w{ template "/etc/haproxy/haproxy.cfg" do
• Extend recipes with template php5 php5-dev php5-cgi }.each do |pkg|
"/etc/haproxy/haproxy.cfg" do
source "haproxy.cfg.erb"
source "haproxy.cfg.erb"
Ruby. owner "root"
package pkg do
owner "root"
group "root"
•
action :install
group "root"
Dynamic configuration end mode 0644
mode 0644
notifies :restart, "service[haproxy]"
through search. variables :pool_members => pool_members
end end:restart, "service[haproxy]"
notifies
end

31. Chef Roles
name "mediawiki"
description "mediawiki app server"
run_list(
"recipe[mysql::client]",
"recipe[application]",
"recipe[mediawiki::status]"
)
• Roles describe nodes.
name "mediawiki_load_balancer"
• Roles have a run list. description "mediawiki load balancer"
run_list(
• Roles can have attributes.
)
"recipe[haproxy::app_lb]"
override_attributes(
"haproxy" => {
"app_server_role" => "mediawiki"
}
)

32. Track it like source code...
% git log
commit d640a8c6b370134d7043991894107d806595cc35
Author: jtimberman <joshua@opscode.com>
Import nagios version 1.0.0
commit c40c818498710e78cf73c7f71e722e971fa574e7
Author: jtimberman <joshua@opscode.com>
installation and usage instruction docs
commit 99d0efb024314de17888f6b359c14414fda7bb91
Author: jtimberman <joshua@opscode.com>
Import haproxy version 1.0.1
commit c89d0975ad3f4b152426df219fee0bfb8eafb7e4
Author: jtimberman <joshua@opscode.com>
add mediawiki cookbook
commit 89c0545cc03b9be26f1db246c9ba4ce9d58a6700
Author: jtimberman <joshua@opscode.com>
multiple environments in data bag for mediawiki

33. Other Chef Terms
• Cookbooks are collections of recipes
• Environments are pegged cookbook versions
• Servers are Nodes
• Data bags are JSON blobs

34. Python Cookbook
• python::package, python::source — Install Python
• python::pip, python::virtualenv — Make it dance
• python::default — All of the above!
• python_pip, python_virtualenv

35. Installing a package
python_virtualenv "/srv/myapp/env" do
action :create
end
python_pip "django" do
action :upgrade
version "1.3"
virtualenv "/srv/myapp/env"
end

36. Gunicorn Cookbook
• gunicorn::default
• gunicorn_config

37. Supervisor Cookbook
• supervisor::default
• Debian-style for now
• supervisor_service

38. Packaginator

39. Role: base
name "base"
description "Base role applied to all nodes."
run_list(
• Install users "recipe[users::sysadmins]",
"recipe[sudo]",
•
"recipe[apt]",
Configure sudo )
"recipe[build-essential]"
•
override_attributes(
apt-get update :authorization => {
:sudo => {
• Install gcc
}
:users => ["ubuntu"],
:passwordless => true
}
)

40. Recipe: packaginator
application "packaginator" do
path "/srv/packaginator"
owner "nobody"
group "nogroup"
repository "https://github.com/coderanger/packaginator.git"
revision "master"
migrate true
packages ["libpq-dev", "git-core", "mercurial"]
django do
packages ["redis"]
requirements "requirements/mkii.txt"
settings_template "settings.py.erb"
debug true
database do
database "packaginator"
•
engine "postgresql_psycopg2"
The good stuff
username "packaginator"
password "awesome_password"
end
database_master_role "packaginator_database_master"
•
collectstatic "build_static --noinput"
end
General parameters gunicorn do
only_if { node['roles'].include? 'packaginator_application_server' }
•
app_module :django
port 8080
Sub-resources end
celery do
only_if { node['roles'].include? 'packaginator_application_server' }
config "celery_settings.py"
django true
celerybeat true
celerycam true
broker do
transport "redis"
end
end
nginx_load_balancer do
only_if { node['roles'].include? 'packaginator_load_balancer' }
application_port 8080
static_files "/site_media" => "site_media"
end
end

41. Application Resource
application "packaginator" do
• Who are we? path "/srv/packaginator"
• Where are we going? owner "nobody"
group "nogroup"

42. Application Resource
directory "/srv/packaginator"
directory "/srv/packaginator/shared"
deploy_revision "packaginator" do
deploy_to "/srv/packaginator"
owner "nobody"
group "nogroup"

43. Application Resource
repository "https://github.com/..."
• Source code revision "master"
• Migration status migrate true
packages ["libpq-dev",
• Base packages
"git-core","mercurial"]

44. Django Resource
• Python packages
• pip requirements
django do
packages ["redis"]
• Settings file
requirements "requirements/mkii.txt"
settings_template "settings.py.erb"
• Debug mode
debug true
collectstatic "build_static --noinput"
• Static files

45. Django Resource
python_pip "redis"
execute "pip -E ... -r ..."
template "shared/local_settings.py"
execute "manage.py build_static"

46. Folder Layout
• /srv/myapp
• /srv/myapp/shared
• /srv/myapp/shared/releases
• /srv/myapp/shared/releases/4faedb2ee...
• /srv/myapp/shared/releases/9457be295...
• ...
• /srv/myapp/current

47. Folder Layout
• /srv/myapp
• ...9457be295/local_settings.py -> ../shared/local_settings.py
• /srv/myapp/current -> .../releases/9457be295...

48. Django Resource
database do
database "packaginator"
• Database engine "postgresql_psycopg2"
username "packaginator"
• Where? password "awesome_password"
end
database_master_role "packaginator_database_master"

49. Django Resource
if node["roles"].include? ...
• Check local first node
else
• Cloud IP! search(:node, "roles:...").first
end

50. Gunicorn Resource
gunicorn do
only_if do
• Only on frontends node["roles"].include? ...
• Use Django mode end
app_module :django
• Internal port
port 8080
end

51. Gunicorn Resource
• Write out config gunicorn_config ...
• Symlink it in
supervisor_service ...
• Load service

52. Celery Resource
celery do
only_if ...
config "celery_settings.py"
• Where?
django true
• Django-mode celerybeat true
• Enable all three celerycam true
broker do
• Broker and results transport "redis"
end
end

53. Nginx Resource
nginx_load_balancer do
only_if ...
application_port 8080
• Internal port again static_files({
• Static map "/site_media" =>
"site_media"
• URL => path
})
end

54. https://github.com/coderanger/djangocon2011

55. Noah Kantrowitz
@kantrn
noah@opscode.com