My KRUG (Kraków Ruby Users Group) presentation about automating boring tasks with Opscode's Chef.

1. CHEF
or how to make computers do the work for us
Marcin Kulik, Lunar Logic Polska
KRUG 2011/11/08

2. Everyday we're dealing with mechanical, repetitive tasks... we
can automate.

3. What is Chef?

4. Automation tool
written in Ruby

5. DSL

6. Created by Opscode

7. "Chef is an open source systems integration framework built
to bring the benefits of configuration management to your
entire infrastructure."
"You write source code to describe how you want each part of
your infrastructure to be built, then apply those descriptions
to your servers."
"The result is a fully automated infrastructure: when a new
server comes on line, the only thing you have to do is tell Chef
what role it should play in your architecture."

8. Why do you need it?

9. Economics + Efficiency + Scalability

10. Terms

11. Node
remote server, local machine...

12. Role
web server, database server, ruby dev workstation...

13. Cookbook
mysql, ssh-access, dotfiles...

14. Recipe
install mysql-server, create database, add user...

15. Resource
file, dir, user, package, service, gem, virtual host...

16. Run list
list of recipes to run in order

17. {
"run_list": [
"recipe[mysql]",
"recipe[git]",
"recipe[ruby19]"
]
}

18. Cookbook structure
|-- config
| |-- node.json
| `-- solo.rb
|-- cookbooks
| |-- book1
| | |-- attributes
| | |-- files
| | |-- metadata.rb
| | |-- recipes
| | | |-- default.rb
| | | `-- source.rb
| | `-- templates
| |-- book2
| | |-- attributes
| | | `-- default.rb
| | |-- files
| | |-- recipes
| | | `-- default.rb
| | `-- templates

19. | | `-- templates
| | `-- default
| | `-- authorized_keys.erb
| |-- book3
| | |-- attributes
| | |-- files
| | | `-- default
| | | `-- secret-key
| | |-- recipes
| | | `-- default.rb
Installation
| | `-- templates
|-- config
| |-- node.json
| `-- solo.rb
|-- cookbooks
| |-- book1
| | |-- attributes
| | |-- files
| | |-- metadata.rb
| | |-- recipes
| | | |-- default.rb
| | | `-- libs.rb
| | `-- templates

20. $ gem install chef

21. Modes of operation

22. Cookbooks stored
in central repository
(free cookbooks hosting by Opscode:
https://manage.opscode.com/)

23. $ sudo chef-client

24. Cookbooks stored
on the node

25. $ sudo chef-solo -c /path/to/cfg.rb
-j /path/to/node-data.json

26. Use cases

27. Configure new machine
(in the cloud with Knife)
Amazon EC2, Engine Yard, Linode, BrightBox...

28. Manage config of existing
company servers
Client demo apps (directory, vhost, god config), developers' ssh
keys...

29. Bootstrap workstation!
rvm + ruby 1.9, git, mysql, vim/emacs...

30. Enough with theory!

31. Lunar Station
https://github.com/LunarLogicPolska/lunar-station

32. Lunar Station is a set of Chef cookbooks and a bash script (???)
for bootstrapping developers machines at Lunar Logic Polska.

33. You need ruby to run Chef

34. (We assume) you use RVM
No need for system ruby for ruby devs nowadays

35. bootstrap.sh

36. detects platform (Ubuntu, Fedora, OSX)
installs compilers and other RVM
dependencies
installs RVM & ruby 1.9 & chef gem
downloads latest Lunar Station
cookbooks
runs chef-solo

37. $ curl -skL http://bit.ly/lunar-station | bash
Initializing Lunar Workstation...
>> Fedora Linux detected.
>> Checking for RVM...
>> Fetching latest version of Lunar Station cookbooks...
>> Starting chef-solo run...
[Mon, 07 Nov 2011 22:19:54 +0100] INFO: *** Chef 0.10.4 ***
[Mon, 07 Nov 2011 22:19:54 +0100] INFO: Setting the run_list to
...

38. Nodes

39. # linux-rubydev.json
{
"run_list": [ "role[rubydev]" ]
}

40. # osx-rubydev.json
{
"run_list": [ "role[osx]", "role[rubydev]" ]
}

41. Roles

42. # base.rb
run_list 'recipe[repos]', 'recipe[curl]',
'recipe[wget]', 'recipe[git]',
'recipe[libxml2]', 'recipe[ack]',
'recipe[vim]', 'recipe[ctags]',
'recipe[skype]', 'recipe[firefox]' ,
'recipe[google-chrome]'

43. # rubydev.rb
run_list 'role[base]', 'recipe[mysql]'

44. # osx.rb
run_list "recipe[homebrew]"

45. Cookbooks

46. repos cookbook

47. # cookbooks/repos/recipes/default.rb
case node[:platform]
when 'fedora'
path = "/tmp/rpmfusion-free-release-stable.noarch.rpm"
bash "download rpmfusion free package" do
code "wget http://download1.rpmfusion.org/.../" +
"rpmfusion-free-release-stable.noarch.rpm -O #{path}"
not_if { File.exist?(path) }
end
package "rpmfusion-free-release-stable" do
source path
options "--nogpgcheck"
end
when 'ubuntu'
...
end

48. end
# cookbooks/repos/recipes/default.rb
case node[:platform]
when 'fedora'
...
when 'ubuntu'
bash "enable multiverse repo" do
code "head -n 1 /etc/apt/sources.list | " +
"sed 's/main universe/multiverse/' " +
">> /etc/apt/sources.list"
not_if "egrep '^deb.+multiverse' /etc/apt/sources.list"
end
end

49. vim cookbook

50. # cookbooks/vim/recipes/default.rb
case node[:platform]
when "ubuntu"
package "vim"
package "vim-gnome"
when "fedora"
package "vim-enhanced"
package "vim-X11"
when 'mac_os_x'
package "macvim"
end

51. skype cookbook

52. # cookbooks/skype/recipes/default.rb
case node[:platform]
when 'ubuntu'
include_recipe 'init::ubuntu' # for partner repo
package 'skype'
when 'mac_os_x'
dmg_package "Skype" do
source "http://www.skype.com/go/getskype-macosx.dmg"
action :install
end
when 'fedora'
...
end

53. Lunar Kitchen

54. Source of LLP servers configuration data and a set of Chef
cookbooks

55. chef-solo invoked on
remote machines
no chef server

56. Each server we configure has its corresponding node
configuration file in nodes/ directory of kitchen project that
specifies run_list and few other settings

57. # nodes/deneb.json
{
"run_list": [ "recipe[ssh_access]" ],
"ssh_access": [ "marcin.kulik", "anna.lesniak", ...],
"opened_ports": {
"tcp": [80, 443, 22, 8080],
"udp": []
},
...

58. How do we run chef-solo
on remote machine?

59. Capistrano!

60. # See the list of configured servers:
$ cap -T
# Make the changes happen on the server:
$ cap configure:deneb

61. How does Capfile look like?

62. set :user, 'chef'
NODE_LIST = Dir["nodes/*.json"].map do |nodefile|
File.basename(nodefile, '.json')
end
NODE_LIST.each do |node|
role node.to_sym, node
end
NODE_CONFIG = <<-EOS
file_cache_path '/tmp/chef-solo'
cookbook_path '/tmp/chef-solo/cookbooks'
role_path '/tmp/chef-solo/roles'
EOS
...

63. ...
namespace :configure do
NODE_LIST.each do |node|
desc "Configure #{node}"
task node.to_sym, :roles => node.to_sym do
run "if [ ! -e /tmp/chef-solo ]; then mkdir /tmp/chef-sol
upload("cookbooks", "/tmp/chef-solo/", :via => :scp, :rec
upload("roles", "/tmp/chef-solo/", :via => :scp, :recursi
upload("nodes/#{node}.json", "/tmp/chef-solo/node.json",
put(NODE_CONFIG, "/tmp/chef-solo/solo.rb")
run "rvmsudo chef-solo " +
"-c /tmp/chef-solo/solo.rb " +
"-j /tmp/chef-solo/node.json"
end
end
end

64. SSH access

65. ├── Capfile
├── config
├── cookbooks
├── nodes
├── README.md
├── roles
└── ssh_keys
├── anna.lesniak
├── artur.bilski
├── ...
└── marcin.kulik

66. # cookbooks/access/recipes/default.rb
username = 'dev'
ssh_keys = node[:ssh_access].map do |f|
File.read("/tmp/chef-solo/ssh_keys/#{f}")
end
template "/home/#{username}/.ssh/authorized_keys" do
source "authorized_keys.erb"
owner username
group 'users'
mode "0600"
variables :ssh_keys => ssh_keys
end

67. # cookbooks/access/templates/authorized_keys.erb
# Generated by Chef, do not edit!
<%= @ssh_keys.join("n") %>

68. Tips

69. Learn step by step
EC2 + Chef + Knife + Opscode... = Fuuuuuuuuuuuuuuuuuuuuu

70. Start with chef-solo

71. Run on local machine
Easy to troubleshoot problems

72. Use Vagrant
http://vagrantup.com/
Great for testing cookbooks - doesn't pollute your system

73. Q?

74. Thanks!
marcin.kulik@llp.pl | @sickill | https://github.com/sickill