1. What Big Data Folks Need
to Know About DevOps
Speaker:
Matt Ray Technical Evangelist
‣ matt@opscode.com
‣ @mattray
Copyright © 2011 Opscode, Inc - All Rights Reserved 1

2. Copyright © 2011 Opscode, Inc - All Rights Reserved
http://www.flickr.com/photos/anotherphotograph/2100904507/sizes/o/ 2

3. Developer, SysAdmin, Hacker,
Community Manager
Copyright © 2011 Opscode, Inc - All Rights Reserved
http://www.flickr.com/photos/anotherphotograph/2100904507/sizes/o/ 2

4. Developer, SysAdmin, Hacker,
Community Manager
Many biz & dev environments
Copyright © 2011 Opscode, Inc - All Rights Reserved
http://www.flickr.com/photos/anotherphotograph/2100904507/sizes/o/ 2

5. Developer, SysAdmin, Hacker,
Community Manager
Many biz & dev environments
Opscode: Training, Services &
Evangelism
Copyright © 2011 Opscode, Inc - All Rights Reserved
http://www.flickr.com/photos/anotherphotograph/2100904507/sizes/o/ 2

6. http://www.flickr.com/photos/timyates/2854357446/sizes/l/
Copyright © 2011 Opscode, Inc - All Rights Reserved 3

7. Developers?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/
Copyright © 2011 Opscode, Inc - All Rights Reserved 3

8. Developers?
Systems Administrators?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/
Copyright © 2011 Opscode, Inc - All Rights Reserved 3

9. Developers?
Systems Administrators?
“BigData” Hacker?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/
Copyright © 2011 Opscode, Inc - All Rights Reserved 3

10. Developers?
Systems Administrators?
“BigData” Hacker?
“Business” People?
http://www.flickr.com/photos/timyates/2854357446/sizes/l/
Copyright © 2011 Opscode, Inc - All Rights Reserved 3

11. DevOps
Copyright © 2011 Opscode, Inc - All Rights Reserved 4

12. DevOps
tools + culture
Copyright © 2011 Opscode, Inc - All Rights Reserved 5

13. Culture
Copyright © 2011 Opscode, Inc - All Rights Reserved 6

14. Trust
Copyright © 2011 Opscode, Inc - All Rights Reserved 7

15. Trust
(but verify)
Copyright © 2011 Opscode, Inc - All Rights Reserved 8

16. Automation
Copyright © 2011 Opscode, Inc - All Rights Reserved 9

17. Infrastructure as
Code
Copyright © 2011 Opscode, Inc - All Rights Reserved 10

18. Copyright © 2011 Opscode, Inc - All Rights Reserved 11

19. Chef is an API for
your Infrastructure
Copyright © 2011 Opscode, Inc - All Rights Reserved 12

20. Principles
Copyright © 2011 Opscode, Inc - All Rights Reserved 13

21. Principles
Idempotent
Copyright © 2011 Opscode, Inc - All Rights Reserved 13

22. Principles
Idempotent
Data-driven
Copyright © 2011 Opscode, Inc - All Rights Reserved 13

23. Principles
Idempotent
Data-driven
Sane defaults
Copyright © 2011 Opscode, Inc - All Rights Reserved 13

24. Principles
Idempotent
Data-driven
Sane defaults
Hackability
Copyright © 2011 Opscode, Inc - All Rights Reserved 13

25. Principles
Idempotent
Data-driven
Sane defaults
Hackability
TMTOWTDI
Copyright © 2011 Opscode, Inc - All Rights Reserved 13

26. Multiple applications of
an operation do not
change the result
Copyright © 2011 Opscode, Inc - All Rights Reserved 14

27. We start with APIs, you
supply data
Copyright © 2011 Opscode, Inc - All Rights Reserved 15

28. option :json_attribs,
:short => "-j JSON_ATTRIBS",
:long => "--json-attributes JSON_ATTRIBS",
:description => "Load attributes from a JSON file or
URL",
:proc => nil
option :node_name,
:short => "-N NODE_NAME",
:long => "--node-name NODE_NAME",
:description => "The node name for this client",
Defaults are sane, but
:proc => nil
easily changed
Copyright © 2011 Opscode, Inc - All Rights Reserved 16

29. Open source and
community
Copyright © 2011 Opscode, Inc - All Rights Reserved 17

30. Copyright © 2011 Opscode, Inc - All Rights Reserved 18

31. TMTOWTDI
Copyright © 2011 Opscode, Inc - All Rights Reserved 19

32. Copyright © 2011 Opscode, Inc - All Rights Reserved 20
http://www.brooklynstreetart.com/theBlog/wp-content/uploads/2008/12/swedish_chef_bork-sleeper-cell.jpg

33. Chef Client runs on your
systems
Copyright © 2011 Opscode, Inc - All Rights Reserved 21

34. Clients talk to a Chef
Server
Copyright © 2011 Opscode, Inc - All Rights Reserved 22

35. RESTful API w/ JSON
responses
Copyright © 2011 Opscode, Inc - All Rights Reserved 23

36. Opscode Platform
the central, highly scalable, multi-tenant
configuration service from Opscode...
a hosted Chef Server
Copyright © 2010 Opscode, Inc. – Confidential – Do Not Redistribute 24

37. We call each system
you configure a Node
Copyright © 2011 Opscode, Inc - All Rights Reserved http://www.flickr.com/photos/peterrosbjerg/3913766224/ 25

38. Nodes have Attributes
{
"kernel": {
Kernel info!
"machine": "x86_64",
"name": "Darwin",
"os": "Darwin",
"version": "Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386",
"release": "10.4.0"
},
"platform_version": "10.6.4",
"platform": "mac_os_x",
"platform_build": "10F569",
"domain": "local", Platform info!
"os": "darwin",
"current_user": "jtimberman",
"ohai_time": 1278602661.60043,
"os_version": "10.4.0",
"uptime": "18 days 17 hours 49 minutes 18 seconds",
"ipaddress": "10.13.37.116",
"hostname": "cider",
"fqdn": "cider.local",
"uptime_seconds": 1619358
Hostname and IP!
}
Copyright © 2011 Opscode, Inc - All Rights Reserved 26

39. Nodes have a Run List
What Roles or Recipes to apply
in Order
Copyright © 2011 Opscode, Inc - All Rights Reserved 27

40. Nodes have Roles
Copyright © 2011 Opscode, Inc - All Rights Reserved http://www.flickr.com/photos/laenulfean/374398044/ 28

41. Roles have a Run List
What Roles or Recipes to apply
in Order
Copyright © 2011 Opscode, Inc - All Rights Reserved 29

42. remote_file
link
cookbook_file
service
ruby_block
template
execute
Chef manages
Resources on Nodes
package bash git log
deploy
user http_request
Copyright © 2011 Opscode, Inc - All Rights Reserved 30

43. Resources...
Declare a description of the state a part of the node should be in
http://www.flickr.com/photos/xiaming/382205902/sizes/l/

44. Resources...
Declare a description of the state a part of the node should be in
package "apache2" do
version "2.2.11-2ubuntu2.6"
action :install
end
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end
http://www.flickr.com/photos/xiaming/382205902/sizes/l/

45. Resources...
Declare a description of the state a part of the node should be in
‣ Have a type package "apache2" do
version "2.2.11-2ubuntu2.6"
action :install
end
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end
http://www.flickr.com/photos/xiaming/382205902/sizes/l/

46. Resources...
Declare a description of the state a part of the node should be in
‣ Have a type package "apache2" do
version "2.2.11-2ubuntu2.6"
action :install
‣ Have a name end
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end
http://www.flickr.com/photos/xiaming/382205902/sizes/l/

47. Resources...
Declare a description of the state a part of the node should be in
‣ Have a type package "apache2" do
version "2.2.11-2ubuntu2.6"
action :install
‣ Have a name end
template "/etc/apache2/apache2.conf" do
‣ Have parameters source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end
http://www.flickr.com/photos/xiaming/382205902/sizes/l/

48. Resources...
Declare a description of the state a part of the node should be in
‣ Have a type package "apache2" do
version "2.2.11-2ubuntu2.6"
action :install
‣ Have a name end
template "/etc/apache2/apache2.conf" do
‣ Have parameters source "apache2.conf.erb"
owner "root"
‣ Take action to put the group "root"
mode 0644
resource in the action :create
declared state end
http://www.flickr.com/photos/xiaming/382205902/sizes/l/

49. Resources take action
through Providers
Copyright © 2011 Opscode, Inc - All Rights Reserved 32

50. Providers...
Know how to actually perform the actions specified by a resource.
http://www.flickr.com/photos/affableslinky/562950216/

51. Providers...
Know how to actually perform the actions specified by a resource.
Apt, Yum, Rubygems,
Multiple providers
Portage, Macports,
per resource type.
FreeBSD Ports, etc.
http://www.flickr.com/photos/affableslinky/562950216/

52. http://www.flickr.com/photos/acurbelo/2628837104/sizes/o/

53. Resources
http://www.flickr.com/photos/acurbelo/2628837104/sizes/o/

54. Resources
Platform
http://www.flickr.com/photos/acurbelo/2628837104/sizes/o/

55. Resources
Platform
Provider
http://www.flickr.com/photos/acurbelo/2628837104/sizes/o/

56. Recipes are lists of
Resources
http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/
Copyright © 2011 Opscode, Inc - All Rights Reserved 35

57. Recipes...
Apply resources in the order they are specified
http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/

58. Recipes...
Apply resources in the order they are specified
package "apache2" do
version "2.2.11-2ubuntu2.6"
action :install
end
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
end
http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/

59. Recipes...
Apply resources in the order they are specified
package "apache2" do
version "2.2.11-2ubuntu2.6"
action :install
1
‣ Evaluates resources in
end
the order they appear
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "root"
group "root"
mode 0644
action :create
2
end
http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/

60. Recipes...
Apply resources in the order they are specified
‣ Evaluates resources in
[
the order they appear "package[apache2]",
"template[/etc/apache2/apache2.conf]"
‣ Adds each resource to ]
the Resource
Collection
http://www.flickr.com/photos/roadsidepictures/2478953342/sizes/o/

61. Order Matters
Copyright © 2011 Opscode, Inc - All Rights Reserved 37

62. Order Matters
http://www.infrastructures.org/papers/turing/turing.html
Copyright © 2011 Opscode, Inc - All Rights Reserved 37

63. Cookbooks are
packages for Recipes
Copyright © 2011 Opscode, Inc - All Rights Reserved 38

64. Common Cookbook Components
Copyright © 2011 Opscode, Inc - All Rights Reserved 39

65. Common Cookbook Components
recipes/
default.rb
Copyright © 2011 Opscode, Inc - All Rights Reserved 39

66. Common Cookbook Components
recipes/
default.rb
files/
Copyright © 2011 Opscode, Inc - All Rights Reserved 39

67. Common Cookbook Components
recipes/
default.rb
files/
templates/
Copyright © 2011 Opscode, Inc - All Rights Reserved 39

68. Common Cookbook Components
recipes/
default.rb
files/
templates/
attributes/
default.rb
Copyright © 2011 Opscode, Inc - All Rights Reserved 39

69. Common Cookbook Components
recipes/
default.rb
files/
templates/
attributes/
default.rb
metadata.rb
Copyright © 2011 Opscode, Inc - All Rights Reserved 39

70. Cookbooks are
shareable!
cookbooks.opscode.com
Copyright © 2011 Opscode, Inc - All Rights Reserved 40

71. Data bags store
arbitrary data
Copyright © 2011 Opscode, Inc - All Rights Reserved 41

72. A user data bag item...
% knife data bag show users jtimberman
{
"comment": "Joshua Timberman",
"groups": "sysadmin",
"ssh_keys": "ssh-rsa SUPERSEKRATS jtimberman@cider",
"files": {
".zshrc": {
"mode": "0644",
"source": "dot-zshrc"
},
".vimrc": {
"mode": "0644",
"source": "dot-vimrc"
}
},
"id": "jtimberman",
"uid": 7004,
"shell": "/usr/bin/zsh",
"openid": "http://jtimberman.myopenid.com/"
}
Copyright © 2011 Opscode, Inc - All Rights Reserved

73. Data bags make recipes
awesome-r (that’s
totally a word)
Copyright © 2011 Opscode, Inc - All Rights Reserved 43

74. sysadmins = search(:users, 'groups:sysadmin')
sysadminss.each do |u|
user u['id'] do
uid u['id']
shell u['shell']
comment u['comment']
supports :manage_home => true
home "/home/#{u['id']}"
end
directory "/home/#{u['id']}/.ssh" do
owner u['id']
group u['id']
mode 0700
end
template "/home/#{u['id']}/.ssh/authorized_keys" do
source "authorized_keys.erb"
owner u['id']
group u['id']
mode 0600
variables :ssh_keys => u['ssh_keys']
end
end
Copyright © 2011 Opscode, Inc - All Rights Reserved 43

75. sysadmins = search(:users, 'groups:sysadmin')
sysadminss.each do |u|
user u['id'] do
uid u['id']
shell u['shell']
comment u['comment']
supports :manage_home => true
home "/home/#{u['id']}"
end
directory "/home/#{u['id']}/.ssh" do
owner u['id']
group u['id']
mode 0700
end
template "/home/#{u['id']}/.ssh/authorized_keys" do
source "authorized_keys.erb"
owner u['id']
group u['id']
mode 0600
variables :ssh_keys => u['ssh_keys']
end
end
Copyright © 2011 Opscode, Inc - All Rights Reserved 43

76. sysadmins = search(:users, 'groups:sysadmin')
sysadminss.each do |u|
user u['id'] do
uid u['id']
shell u['shell']
comment u['comment']
supports :manage_home => true
home "/home/#{u['id']}"
end
directory "/home/#{u['id']}/.ssh" do
owner u['id']
group u['id']
mode 0700
end
template "/home/#{u['id']}/.ssh/authorized_keys" do
source "authorized_keys.erb"
owner u['id']
group u['id']
mode 0600
variables :ssh_keys => u['ssh_keys']
end
end
Copyright © 2011 Opscode, Inc - All Rights Reserved 43

77. Command-line API
utility, Knife
http://www.flickr.com/photos/myklroventine/3474391066/
Copyright © 2011 Opscode, Inc - All Rights Reserved 44

78. Nodes, Roles, Data
Bags are Searchable
% knife search node “role:webserver”
search(:users, “group:sysadmins”)
Copyright © 2011 Opscode, Inc - All Rights Reserved 45

79. Nodes, Roles, Data
Bags are Searchable
% knife search node “role:webserver”
search(:users, “group:sysadmins”)
Copyright © 2011 Opscode, Inc - All Rights Reserved 45

80. Nodes, Roles, Data
Bags are Searchable
% knife search node “role:webserver”
search(:users, “group:sysadmins”)
Copyright © 2011 Opscode, Inc - All Rights Reserved 45

81. Cluster Chef
Copyright © 2011 Opscode, Inc - All Rights Reserved 46

82. Hadoop
HDFS
NameNode
Secondary NN*
DataNode(s)
Copyright © 2011 Opscode, Inc - All Rights Reserved 47

83. Hadoop
MapReduce
JobTracker
TaskTracker(s)
Copyright © 2011 Opscode, Inc - All Rights Reserved 48

84. Let’s Get Cooking
Prerequisites are already in place right?
http://bit.ly/dda-chef
Copyright © 2011 Opscode, Inc - All Rights Reserved 49

85. Push the Cookbooks
$ cd $CLUSTER_CHEF_PATH
$ knife cookbook upload --all
These run as root, kids.
Let’s not blindly trust the upstream too much!
Copyright © 2011 Opscode, Inc - All Rights Reserved 50

86. Cookbooks
Recipes
datanode.rb
jobtracker.rb
namenode.rb
secondarynamenode.rb
tasktracker.rb
more!
Copyright © 2011 Opscode, Inc - All Rights Reserved 51

87. Push the Roles
$ for foo in roles/*.rb ; do knife role from file $foo &
sleep 1 ; done
Copyright © 2011 Opscode, Inc - All Rights Reserved 52

88. Cluster Chef’s Facets
Roles
hadoop_master
hadoop_namenode
hadoop_secondarynamenod
e
hadoop_jobtracker
hadoop_worker
hadoop_datanode
hadoop_tasktracker
Copyright © 2011 Opscode, Inc - All Rights Reserved 53

89. Provisioning
Nodes
demohadoop-master-i-77f2661b
demohadoop-worker-i-e390148f
demohadoop-worker-i-ff901493
Copyright © 2011 Opscode, Inc - All Rights Reserved 54

90. Is this thing on?
$ knife cluster
Available cluster subcommands: (for details, knife SUB-COMMAND --help)
** CLUSTER COMMANDS **
knife cluster launch CLUSTER_NAME FACET_NAME (options)
knife cluster show CLUSTER_NAME FACET_NAME (options)
knife cluster bootstrap CLUSTER_NAME FACET_NAME SERVER_FQDN (options)
Copyright © 2011 Opscode, Inc - All Rights Reserved 55

91. Let’s launch our
Hadoop Cluster!
$ knife cluster launch demohadoop master --bootstrap
Copyright © 2011 Opscode, Inc - All Rights Reserved 56

92. knife ec2 server create
Copyright © 2011 Opscode, Inc - All Rights Reserved 57

93. knife ec2 server create
Creates EC2 instance via API
Copyright © 2011 Opscode, Inc - All Rights Reserved 57

94. knife ec2 server create
Creates EC2 instance via API
Retrieves local configuration
Copyright © 2011 Opscode, Inc - All Rights Reserved 57

95. knife ec2 server create
Creates EC2 instance via API
Retrieves local configuration
SSH to instance
Copyright © 2011 Opscode, Inc - All Rights Reserved 57

96. knife ec2 server create
Creates EC2 instance via API
Retrieves local configuration
SSH to instance
‣ Cluster Chef extends this
security groups
picks the AMI
builds the number of specified nodes
‣ Writes chef configuration and authentication
‣ Installs Ruby and Chef
‣ Runs Chef with specified run list
Copyright © 2011 Opscode, Inc - All Rights Reserved 57

97. knife ec2 server create
Creates EC2 instance via API
Retrieves local configuration
SSH to instance
‣ Cluster Chef extends this
security groups
picks the AMI
builds the number of specified nodes
‣ Writes chef configuration and authentication
‣ Installs Ruby and Chef
‣ Runs Chef with specified run list
Copyright © 2011 Opscode, Inc - All Rights Reserved 57

98. Still a bit of tweaking
$ knife ssh "role:demohadoop_master" "sudo service hadoop-0.20-datanode
stop; sudo service hadoop-0.20-namenode stop; sudo service hadoop-0.20-
tasktracker stop; sudo service hadoop-0.20-jobtracker stop; sudo service
hadoop-0.20-secondarynamenode stop; sudo -u hdfs hadoop fs -chown -R
hbase:hbase /hadoop/hbase; sudo chef-client" -x ubuntu -a
ec2.public_hostname -i ~/.chef/keypairs/demohadoop.pem
Copyright © 2011 Opscode, Inc - All Rights Reserved 58

99. Hadoop Workers!
$ knife cluster launch demohadoop worker --bootstrap
Copyright © 2011 Opscode, Inc - All Rights Reserved 59

100. Is it really on?
‣ Configure your network settings to use a SOCKS proxy
‣ http://ec2-public-ip-
address.compute-1.amazonaws.com
‣ copy & paste the SSH command
‣ Profit!
Copyright © 2011 Opscode, Inc - All Rights Reserved 60

101. Our Hadoop Cluster
is Operational...
Copyright © 2011 Opscode, Inc - All Rights Reserved 61

102. Resources/Questions
www.opscode.com/chef
IRC and Mailing lists
‣ irc.freenode.net #chef
‣ lists.opscode.com
Twitter:
‣ @opscode, #opschef
‣ @mattray
Questions?
Copyright © 2011 Opscode, Inc - All Rights Reserved 62