6. Infrastructure as code
Building and managing infrastructure programmatically
Enable the reconstruction of the business from:
@gbagnoli Introduzione a Chef 12/09/2012 5 / 50
7. Infrastructure as code
Building and managing infrastructure programmatically
Enable the reconstruction of the business from:
• computing resources
@gbagnoli Introduzione a Chef 12/09/2012 5 / 50
8. Infrastructure as code
Building and managing infrastructure programmatically
Enable the reconstruction of the business from:
• computing resources
• a source code repository
@gbagnoli Introduzione a Chef 12/09/2012 5 / 50
9. Infrastructure as code
Building and managing infrastructure programmatically
Enable the reconstruction of the business from:
• computing resources
• a source code repository
• data backups
@gbagnoli Introduzione a Chef 12/09/2012 5 / 50
10. Infrastructure as code (2)
source: Chef wiki
Provisioning Get new computing resources
Configuration Management Keeps track of all steps required to take bare
metal resources to doing their job
System Integration Takes all configured systems and make them work
together.
@gbagnoli Introduzione a Chef 12/09/2012 6 / 50
12. What is Chef
• A library/framework for configuration management
@gbagnoli Introduzione a Chef 12/09/2012 7 / 50
13. What is Chef
• A library/framework for configuration management
• A configuration management system
@gbagnoli Introduzione a Chef 12/09/2012 7 / 50
14. What is Chef
• A library/framework for configuration management
• A configuration management system
• A system integration platform
@gbagnoli Introduzione a Chef 12/09/2012 7 / 50
15. What is Chef
• A library/framework for configuration management
• A configuration management system
• A system integration platform
• An API for the infrastructure
@gbagnoli Introduzione a Chef 12/09/2012 7 / 50
16. What is Chef
• A library/framework for configuration management
• A configuration management system
• A system integration platform
• An API for the infrastructure
• Open Source! (Apache License, version 2.0)
@gbagnoli Introduzione a Chef 12/09/2012 7 / 50
22. Chef Principles
Idempotent Describes states
Extensible Uses ruby as the DSL
Order It matters
Client-Server Thick Clients, thin Server
@gbagnoli Introduzione a Chef 12/09/2012 9 / 50
23. Chef Principles
Idempotent Describes states
Extensible Uses ruby as the DSL
Order It matters
Client-Server Thick Clients, thin Server
Various Flavours:
@gbagnoli Introduzione a Chef 12/09/2012 9 / 50
24. Chef Principles
Idempotent Describes states
Extensible Uses ruby as the DSL
Order It matters
Client-Server Thick Clients, thin Server
Various Flavours:
• Hosted on Opscode platform (chef-hosted)
@gbagnoli Introduzione a Chef 12/09/2012 9 / 50
25. Chef Principles
Idempotent Describes states
Extensible Uses ruby as the DSL
Order It matters
Client-Server Thick Clients, thin Server
Various Flavours:
• Hosted on Opscode platform (chef-hosted)
• Open source Chef Server
@gbagnoli Introduzione a Chef 12/09/2012 9 / 50
26. Chef Principles
Idempotent Describes states
Extensible Uses ruby as the DSL
Order It matters
Client-Server Thick Clients, thin Server
Various Flavours:
• Hosted on Opscode platform (chef-hosted)
• Open source Chef Server
• Serverless - chef-solo
@gbagnoli Introduzione a Chef 12/09/2012 9 / 50
27. Chef Principles
Idempotent Describes states
Extensible Uses ruby as the DSL
Order It matters
Client-Server Thick Clients, thin Server
Various Flavours:
• Hosted on Opscode platform (chef-hosted)
• Open source Chef Server
• Serverless - chef-solo
• Private Chef (opscode-supported behind-the-firewall installation)
@gbagnoli Introduzione a Chef 12/09/2012 9 / 50
28. Chef API
• A RESTful service with JSON responses
• RSA key authentication with Signed Headers
• Search Service
• Derivative (easy to integrate with other tools)
• . . . i.e pychef :-)
@gbagnoli Introduzione a Chef 12/09/2012 10 / 50
29. Chef API
• A RESTful service with JSON responses
• RSA key authentication with Signed Headers
• Search Service
• Derivative (easy to integrate with other tools)
• . . . i.e pychef :-)
chef-client connects to the server consuming the API.
CLI management tool knife and the webUI use the API too.
@gbagnoli Introduzione a Chef 12/09/2012 10 / 50
30. Chef Architecture
Solr
CouchDB
Indexer
RabbitMQ
Chef API Server
Chef WebUI
Client Client Client
Knife Knife
@gbagnoli Introduzione a Chef 12/09/2012 11 / 50
31. API Client
In chef, an API client provides the identity used to authenticate
requests to the API server.
@gbagnoli Introduzione a Chef 12/09/2012 12 / 50
32. API Client
In chef, an API client provides the identity used to authenticate
requests to the API server.
The public half of the public/private of a key pair is stored in the db on
the server, while the private part is local to the client.
@gbagnoli Introduzione a Chef 12/09/2012 12 / 50
33. API Client
In chef, an API client provides the identity used to authenticate
requests to the API server.
The public half of the public/private of a key pair is stored in the db on
the server, while the private part is local to the client.
Each request to the API contains a request signature in the HTTP
headers.
The request signature is computed by the hash of the request content and
encrypted with the client private key, so it’s possible to verify the identity
of the user/machine making the request.
@gbagnoli Introduzione a Chef 12/09/2012 12 / 50
34. Nodes
A Node is a host that runs the chef-client.
• Has attributes
• Has a run list
• Has 0+ roles
• Belongs to an environment
In the common case, 1 host ⇔ 1 node ⇔ 1 client
@gbagnoli Introduzione a Chef 12/09/2012 13 / 50
35. run list
"run_list": {
"role[python_hosting]",
"recipe[postgresql::client]",
"recipe[chishop]"
}
@gbagnoli Introduzione a Chef 12/09/2012 14 / 50
36. Roles
• Have attributes
• Have a run list
• Declared in JSON or . . .
• Declared with the ruby DSL (automatically compiled to JSON)
@gbagnoli Introduzione a Chef 12/09/2012 15 / 50
37. Roles
• Have attributes
• Have a run list
• Declared in JSON or . . .
• Declared with the ruby DSL (automatically compiled to JSON)
If 1+ roles are in the node run list, the node run list is expanded
@gbagnoli Introduzione a Chef 12/09/2012 15 / 50
38. Roles (2)
An example role (in ruby):
name "python_hosting"
description "Python App hosting"
default_attributes(
"nginx" => {
"default_site_enabled" => false
}
)
run_list(
"recipe[python::virtualenv]",
"recipe[uwsgi]",
"recipe[nginx]"
)
@gbagnoli Introduzione a Chef 12/09/2012 16 / 50
39. Attributes
Store node data (i.e. ip address, hostname, fqdn, database host address,
etc.)
There are four types of attributes (in order of precedence, lowest to
highest):
• default
• normal
• override
• automatic
@gbagnoli Introduzione a Chef 12/09/2012 17 / 50
40. Attributes
Store node data (i.e. ip address, hostname, fqdn, database host address,
etc.)
There are four types of attributes (in order of precedence, lowest to
highest):
• default
• normal
• override
• automatic
Attributes can be set in:
• cookbooks
• environments
• roles
• nodes
@gbagnoli Introduzione a Chef 12/09/2012 17 / 50
41. Attributes (2)
So, in the end, as attributes are deep-merged, the following precedence
applies:
@gbagnoli Introduzione a Chef 12/09/2012 18 / 50
42. Attributes (2)
So, in the end, as attributes are deep-merged, the following precedence
applies:
• default attributes applied in an cookbook
• default attributes applied in an environment
• default attributes applied in a role
• default attributes applied on a node directly in a recipe
• normal attributes applied in a cookbook
• normal attributes applied on a node directly in a recipe
• override attributes applied in an cookbook
• override attributes applied in an environment
• override attributes applied in a role
• override attributes applied on a node directly in a recipe
• automatic attributes generated by Ohai
@gbagnoli Introduzione a Chef 12/09/2012 18 / 50
43. Attributes (3)
Automatic, ovverride and default are reset at the beginning of every run.
Normal attributes persist between runs.
@gbagnoli Introduzione a Chef 12/09/2012 19 / 50
44. Attributes (3)
Automatic, ovverride and default are reset at the beginning of every run.
Normal attributes persist between runs.
Attributes are searchable:
search(:node, ’platform:ubuntu’)
or
knife search node "platform:ubuntu"
@gbagnoli Introduzione a Chef 12/09/2012 19 / 50
46. Attributes (4)
Summary:
• (sane) defaults in cookbooks
@gbagnoli Introduzione a Chef 12/09/2012 20 / 50
47. Attributes (4)
Summary:
• (sane) defaults in cookbooks
• . . . overridden in roles
@gbagnoli Introduzione a Chef 12/09/2012 20 / 50
48. Attributes (4)
Summary:
• (sane) defaults in cookbooks
• . . . overridden in roles
• . . . and node-specific data as normal attributes on the node.
@gbagnoli Introduzione a Chef 12/09/2012 20 / 50
49. Attributes (4)
Summary:
• (sane) defaults in cookbooks
• . . . overridden in roles
• . . . and node-specific data as normal attributes on the node.
• override and node.set can be used to force values
@gbagnoli Introduzione a Chef 12/09/2012 20 / 50
51. Resources
Chef manages resources on a node.
Resources are specified in recipes, recipes stored in cookbooks.
@gbagnoli Introduzione a Chef 12/09/2012 21 / 50
52. Resources
Chef manages resources on a node.
Resources are specified in recipes, recipes stored in cookbooks.
The expanded run list specifies all the recipes (and thus the resources) to
manage on a given node.
@gbagnoli Introduzione a Chef 12/09/2012 21 / 50
53. Resources (2)
A resource
• has a type package "tar" do
• has a name version "1.16.1-1"
action :install
• has parameters
end
• takes actions
@gbagnoli Introduzione a Chef 12/09/2012 22 / 50
54. Resources (2)
A resource
• has a type package "tar" do
• has a name version "1.16.1-1"
action :install
• has parameters
end
• takes actions
Actions are taken using providers, providers are chosen based on the node
platform.
(i.e. the package resource installs packages using apt on debian/ubuntu
and using yum on centos/RHEL)!
@gbagnoli Introduzione a Chef 12/09/2012 22 / 50
55. Recipes
Recipes evaluate resources in the order they appear
package "pdns-recursor" do
action :install
end
template "#{node[:pdns][:confd]}/recursor.conf" do
source "recursor.cfg.erb"
owner "root"
group "root"
mode 0644
notifies :restart, "service[pdns-recursor]"
end
service "pdns-recursor" do
action [:enable, :start]
end
@gbagnoli Introduzione a Chef 12/09/2012 23 / 50
56. Recipes (2)
Recipes can include other resources, and are just ruby code
include_recipe "apache2"
...
%w{config logs files}.each do |dir|
directory "#{node[:myrecipe][:base_dir]}/#{dir}" do
recursive true
owner "myuser"
group "mygroup"
mode 02775
end
end
@gbagnoli Introduzione a Chef 12/09/2012 24 / 50
57. Cookbooks
Cookbooks are (shareable) packages for recipes.
Cookbooks for chef ⇔ gems for ruby
cookbooks / python /
| - - attributes
| ‘-- default . rb
| - - files
| ‘-- default
| - - LICENSE
| - - metadata . rb
| - - providers
| | - - pip . rb
| ‘-- virtualenv . rb
| - - README . md
| - - recipes
| | - - default . rb
| | - - package . rb
| | - - pip . rb
| | - - source . rb
| ‘-- virtualenv . rb
| - - resources
| | - - pip . rb
| ‘-- virtualenv . rb
‘-- templates
‘-- default
‘-- p r o f i l e _ v i r t u a l e n v w r a p p e r . sh . erb
@gbagnoli Introduzione a Chef 12/09/2012 25 / 50
58. Cookbooks (2)
Currently 133 cookbooks available on opscode-cookbooks GitHub org.
too many to list them all!
https://github.com/opscode-cookbooks
More cookbooks on the community site at
http://community.opscode.com/cookbooks
@gbagnoli Introduzione a Chef 12/09/2012 26 / 50
59. Metadata
maintainer "Opscode, Inc."
maintainer_email "cookbooks@opscode.com"
license "Apache 2.0"
description "Installs and configures mysql for client or server"
long_description IO.read(File.join(File.dirname(__FILE__), ’README.md’))
version "1.2.5"
recipe "mysql", "Includes the client recipe to configure a client"
recipe "mysql::client", "Installs packages required for mysql clients using run_action magic"
recipe "mysql::server", "Installs packages required for mysql servers w/o manual intervention"
recipe "mysql::server_ec2", "Performs EC2-specific mountpoint manipulation"
%w{ debian ubuntu centos suse fedora redhat scientific amazon }.each do |os|
supports os
end
@gbagnoli Introduzione a Chef 12/09/2012 27 / 50
60. Environments
Environments can be used to manage different environments (production,
test, etc) in a single Chef setup.
Roles can have different run list on different environments
@gbagnoli Introduzione a Chef 12/09/2012 28 / 50
61. Environments
Environments can be used to manage different environments (production,
test, etc) in a single Chef setup.
Roles can have different run list on different environments
name "production"
description "The production environment"
cookbook_versions(
"mysql" => "= 1.2.5", # use version 1.2.5 only
"apache2" => "~> 1.1" # anything 1.1.0 < x < 1.2.0
)
# default attributes for this environment
attributes(
"apache2" => {
"listen_ports" => ["80", "443"]
}
)
@gbagnoli Introduzione a Chef 12/09/2012 28 / 50
62. Environments
Environments can be used to manage different environments (production,
test, etc) in a single Chef setup.
Roles can have different run list on different environments
name "production"
description "The production environment"
cookbook_versions(
"mysql" => "= 1.2.5", # use version 1.2.5 only
"apache2" => "~> 1.1" # anything 1.1.0 < x < 1.2.0
)
# default attributes for this environment
attributes(
"apache2" => {
"listen_ports" => ["80", "443"]
}
)
As with role, the ruby DSL gets compiled to JSON when uploading to server
@gbagnoli Introduzione a Chef 12/09/2012 28 / 50
63. Environments (2)
$ knife environment list
production
$ knife environment show production -F json
{
"name": "production",
"description": "The production environment",
"cookbook_versions": {
"mysql": "= 1.2.5",
"apache2": "~> 1.1"
},
"json_class": "Chef::Environment",
"chef_type": "environment",
"default_attributes": {
"apache2": {
"listen_ports": [
"80",
"443"
]
}
},
"override_attributes": {
}
}
@gbagnoli Introduzione a Chef 12/09/2012 29 / 50
64. Environments (3)
Cookbooks can be frozen, so that following uploads with the same version
will fail.
$ # -E automatically sets a requirement for the specified environment
$ knife cookbook upload redis -- freeze -E production
Uploading redis ...
upload complete
$ knife cookbook show redis 1.0.2 | grep " frozen "
frozen ?: true
$ knife cookbook upload redis
Uploading redis ...
ERROR : Version 1.0.2 of cookbook redis is frozen . Use -- force to override .
ERROR : Failed to upload 1 cookbook .
$ knife environment show production | grep redis
users : = 1.0.2
@gbagnoli Introduzione a Chef 12/09/2012 30 / 50
65. Data Bags
Data bags provide an arbitrary store of globally available JSON data.
@gbagnoli Introduzione a Chef 12/09/2012 31 / 50
66. Data Bags
Data bags provide an arbitrary store of globally available JSON data.
Data bags can be encrypted (but then cannot be searched, except for id)
@gbagnoli Introduzione a Chef 12/09/2012 31 / 50
67. Data Bags (Encrypted)
(warn: fake data ahead)
$ knife data bag show accounts gbagnoli
comment : tenoh > dieliSh ’ i7eexeijeiSh ^ u9phaeGhuu4chaa *=
email : A h r 8 i s 3 a h C h o h m 6 a e n e i c ( aef " a h1e eree Voh hie 6Up =
group : Waix8Pa # iniy # oh6eem$eij =
groups : xa i7on g7a ihi u1n eH & ah3ier3Goh } rae7nik$einaeb =
id : gbagnoli
shadow : yi e@ja h0v e$g 2Ae Gh } ido6koobuew | aebeenaequeRo ( xaiYei8eizi + f
7 ohqu < i@enequ & oh7ef - ahdae8dia [ c h a h 7 e e 4 y i e $ N 4 E e B i c h e e 5 e i r o
h2JaGhae ^ k 6a e ph o hj ah s h6 Ae j a ^ cheew } o ) i 0 w o 5 i e s i s h 3 d i g h i e w o h
tohoh0eegho7eik =
shell : aeshi2ohy , ai6ai h2Ahquu =
ssh_keys : [.. cut ..]
uid : thahvo2IGhoh3osho8Ees /a=
username : poh5WiuZ2Er : it ! ee1ahf { u =
@gbagnoli Introduzione a Chef 12/09/2012 32 / 50
68. Data Bags (Decrypted)
$ knife data bag show accounts gbagnoli -- secret - file ~/. chef / enc_db_secret
comment : Giacomo Bagnoli
email : g . bagnoli@asidev . com
group : wheel
groups : [ asidev , users ]
id : gbagnoli
organization : asidev
shadow : $6$ [... cut ...]
shell : / bin / bash
ssh_keys : [" ssh - rsa [... cut ...] Giacomo Bagnoli "]
uid : 3000
username : g . bagnoli
@gbagnoli Introduzione a Chef 12/09/2012 33 / 50
69. Anatomy of a Chef Run
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
70. Anatomy of a Chef Run
• chef-client starts
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
71. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
72. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
73. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
• Cookbook sync
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
74. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
• Cookbook sync
• Compiles resource collection, loading:
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
75. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
• Cookbook sync
• Compiles resource collection, loading:
• libraries
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
76. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
• Cookbook sync
• Compiles resource collection, loading:
• libraries
• resources / definitions
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
77. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
• Cookbook sync
• Compiles resource collection, loading:
• libraries
• resources / definitions
• attributes
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
78. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
• Cookbook sync
• Compiles resource collection, loading:
• libraries
• resources / definitions
• attributes
• recipes
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
79. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
• Cookbook sync
• Compiles resource collection, loading:
• libraries
• resources / definitions
• attributes
• recipes
• Executes - Configure Node
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
80. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
• Cookbook sync
• Compiles resource collection, loading:
• libraries
• resources / definitions
• attributes
• recipes
• Executes - Configure Node
• Converge: each resource is mapped to a provider and which takes
action on it
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
81. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
• Cookbook sync
• Compiles resource collection, loading:
• libraries
• resources / definitions
• attributes
• recipes
• Executes - Configure Node
• Converge: each resource is mapped to a provider and which takes
action on it
• Saves Node
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
82. Anatomy of a Chef Run
• chef-client starts
• Builds node (runs ohai, perform deep-merge of attrs)
• chef-client registers with the server
• Cookbook sync
• Compiles resource collection, loading:
• libraries
• resources / definitions
• attributes
• recipes
• Executes - Configure Node
• Converge: each resource is mapped to a provider and which takes
action on it
• Saves Node
• Runs notification Handlers
On errors, exception handlers are run.
@gbagnoli Introduzione a Chef 12/09/2012 34 / 50
85. Chef development workflow
• Write cookbooks/recipe
• Upload the modified cookbook to the chef server
@gbagnoli Introduzione a Chef 12/09/2012 35 / 50
86. Chef development workflow
• Write cookbooks/recipe
• Upload the modified cookbook to the chef server
• Add the cookbook to a run list (in a node or in a role)
@gbagnoli Introduzione a Chef 12/09/2012 35 / 50
87. Chef development workflow
• Write cookbooks/recipe
• Upload the modified cookbook to the chef server
• Add the cookbook to a run list (in a node or in a role)
• Wait for chef-client to run on nodes
@gbagnoli Introduzione a Chef 12/09/2012 35 / 50
88. Chef development workflow
• Write cookbooks/recipe
• Upload the modified cookbook to the chef server
• Add the cookbook to a run list (in a node or in a role)
• Wait for chef-client to run on nodes
• Commit changes in git
@gbagnoli Introduzione a Chef 12/09/2012 35 / 50
89. Search
Full-text query engine based on Apache Solr.
Searches can be performed from knife and in recipes.
Almost any object is indexed by the chef server, like roles, nodes, api
clients and environments.
$ knife search node " recipes : apache2 "
7 items found
...
$ knife search node " recipes : apache2 AND chef_environment : production "
5 items found
...
$ knife search node " roles : lxc_guest "
9 items found
...
$ knife search client " admin : true "
4 items found
...
$ knife search role " name : lxc *"
2 items found
...
@gbagnoli Introduzione a Chef 12/09/2012 36 / 50
90. Bootstrapping
Bootstrapping is installing chef on new nodes . . . using chef.
First, create the node:
$ knife node create mynewnode . example . com
# .. fires up $EDITOR
# .. set run_list / attributes / etc / environment
@gbagnoli Introduzione a Chef 12/09/2012 37 / 50
91. Bootstrapping
Bootstrapping is installing chef on new nodes . . . using chef.
First, create the node:
$ knife node create mynewnode . example . com
# .. fires up $EDITOR
# .. set run_list / attributes / etc / environment
Assuming that the new node is a bare ubuntu install, bootstrap the node
$ knife boostrap -N mynewnode . example . com -d ubuntu $NODE_IP -- sudo -V -x ubuntu
@gbagnoli Introduzione a Chef 12/09/2012 37 / 50
92. Bootstrapping
Bootstrapping is installing chef on new nodes . . . using chef.
First, create the node:
$ knife node create mynewnode . example . com
# .. fires up $EDITOR
# .. set run_list / attributes / etc / environment
Assuming that the new node is a bare ubuntu install, bootstrap the node
$ knife boostrap -N mynewnode . example . com -d ubuntu $NODE_IP -- sudo -V -x ubuntu
Or, combine with provisioning (i.e. Amazon AWS)
knife ec2 server create -I ami - db595faf -- flavor t1 . micro -- region eu - west -1
-G default -x ubuntu -N newnode . example . com -d ubuntu -Z eu - west -1 a
Chef Omnibus bootstrap template
@gbagnoli Introduzione a Chef 12/09/2012 37 / 50
94. Monitoring
We use CheckMK to configure Icinga for monitoring our infrastructure.
@gbagnoli Introduzione a Chef 12/09/2012 39 / 50
95. Monitoring
We use CheckMK to configure Icinga for monitoring our infrastructure.
CheckMK is a general purpose nagios plugin to retrive data from hosts.
@gbagnoli Introduzione a Chef 12/09/2012 39 / 50
96. Monitoring
We use CheckMK to configure Icinga for monitoring our infrastructure.
CheckMK is a general purpose nagios plugin to retrive data from hosts.
1. One active check per host per check interval (calling check mk as a
plugin).
@gbagnoli Introduzione a Chef 12/09/2012 39 / 50
97. Monitoring
We use CheckMK to configure Icinga for monitoring our infrastructure.
CheckMK is a general purpose nagios plugin to retrive data from hosts.
1. One active check per host per check interval (calling check mk as a
plugin).
2. The connection is done via TCP to the check mk agent on the target
host
@gbagnoli Introduzione a Chef 12/09/2012 39 / 50
98. Monitoring
We use CheckMK to configure Icinga for monitoring our infrastructure.
CheckMK is a general purpose nagios plugin to retrive data from hosts.
1. One active check per host per check interval (calling check mk as a
plugin).
2. The connection is done via TCP to the check mk agent on the target
host (All host data is sent back at once as ASCII text.)
@gbagnoli Introduzione a Chef 12/09/2012 39 / 50
99. Monitoring
We use CheckMK to configure Icinga for monitoring our infrastructure.
CheckMK is a general purpose nagios plugin to retrive data from hosts.
1. One active check per host per check interval (calling check mk as a
plugin).
2. The connection is done via TCP to the check mk agent on the target
host (All host data is sent back at once as ASCII text.)
3. check mk extracts performance data.
@gbagnoli Introduzione a Chef 12/09/2012 39 / 50
100. Monitoring
We use CheckMK to configure Icinga for monitoring our infrastructure.
CheckMK is a general purpose nagios plugin to retrive data from hosts.
1. One active check per host per check interval (calling check mk as a
plugin).
2. The connection is done via TCP to the check mk agent on the target
host (All host data is sent back at once as ASCII text.)
3. check mk extracts performance data.
4. check mk checks warn/crit thresholds and submits results to Icinga as
passive checks.
@gbagnoli Introduzione a Chef 12/09/2012 39 / 50
101. Monitoring - Nodes
On nodes, the check mk::agent recipe, included in all nodes via the base
role, installs the check mk agent.
@gbagnoli Introduzione a Chef 12/09/2012 40 / 50
102. Monitoring - Nodes
On nodes, the check mk::agent recipe, included in all nodes via the base
role, installs the check mk agent.
It also sets up xinetd and the firewall so that connections to the agent
are allowed only from the monitoring host(s).
@gbagnoli Introduzione a Chef 12/09/2012 40 / 50
103. Monitoring - Nodes
On nodes, the check mk::agent recipe, included in all nodes via the base
role, installs the check mk agent.
It also sets up xinetd and the firewall so that connections to the agent
are allowed only from the monitoring host(s).
Monitoring hosts are specified as attributes in the base role.
@gbagnoli Introduzione a Chef 12/09/2012 40 / 50
104. Monitoring - Nodes (2)
(almost) Every cookbook pushes a MRPE or check mk plugin check to
the node, so the check mk agent returns data for all configured services.
i.e. , in the mysql:: server recipe
mrpe_check "mysql" do
script "check_mysql"
variables(
:passwd => node["mysql"]["server_root_password"],
:checks => checks,
:tunables => node["mysql"]["tunable"]
)
end
@gbagnoli Introduzione a Chef 12/09/2012 41 / 50
105. Monitoring - Nodes (2)
(almost) Every cookbook pushes a MRPE or check mk plugin check to
the node, so the check mk agent returns data for all configured services.
i.e. , in the mysql:: server recipe
mrpe_check "mysql" do
script "check_mysql"
variables(
:passwd => node["mysql"]["server_root_password"],
:checks => checks,
:tunables => node["mysql"]["tunable"]
)
end
> telnet mysql-server.example.com 6556
[...]
<<<mrpe>>>
(check_mysql) mysql_idx 0 OK - index usage 53.60% | index_usage=53.60%;0:;0:
(check_mysql) mysql_running 0 OK - 0 long running processes | long_running_procs=0;10;20
(check_mysql) mysql_threads 0 OK - 18 client connection threads | threads_connected=18;80;95
[...]
@gbagnoli Introduzione a Chef 12/09/2012 41 / 50
106. Monitoring - Server
On the server, the check mk:: server recipe installs and configures icinga,
nsca, nagvis, pnp4nagios smokeping and check mk.
@gbagnoli Introduzione a Chef 12/09/2012 42 / 50
107. Monitoring - Server
On the server, the check mk:: server recipe installs and configures icinga,
nsca, nagvis, pnp4nagios smokeping and check mk.
The check mk configuration file (which is used by check mk to configure
active and passive checks in icinga) is managed as a template by the
recipe. The recipe use the search API to discover nodes using a
configurable query.
@gbagnoli Introduzione a Chef 12/09/2012 42 / 50
108. Monitoring - Server
On the server, the check mk:: server recipe installs and configures icinga,
nsca, nagvis, pnp4nagios smokeping and check mk.
The check mk configuration file (which is used by check mk to configure
active and passive checks in icinga) is managed as a template by the
recipe. The recipe use the search API to discover nodes using a
configurable query.
nodes = search(:node, node["check_mk"]["search_query"])
default query:
default [”check mk”][”search query”] = ”chef environment:production”
@gbagnoli Introduzione a Chef 12/09/2012 42 / 50
109. Monitoring - Server (2)
So, when a node is promoted to production, as soon as chef runs on the
monitoring server,
@gbagnoli Introduzione a Chef 12/09/2012 43 / 50
110. Monitoring - Server (2)
So, when a node is promoted to production, as soon as chef runs on the
monitoring server,
• It adds the node to the check mk conf file (the template is the same,
but data has changed)
@gbagnoli Introduzione a Chef 12/09/2012 43 / 50
111. Monitoring - Server (2)
So, when a node is promoted to production, as soon as chef runs on the
monitoring server,
• It adds the node to the check mk conf file (the template is the same,
but data has changed)
• . . . the modified templates notifies the check mk write conf
resource, which is queued
@gbagnoli Introduzione a Chef 12/09/2012 43 / 50
112. Monitoring - Server (2)
So, when a node is promoted to production, as soon as chef runs on the
monitoring server,
• It adds the node to the check mk conf file (the template is the same,
but data has changed)
• . . . the modified templates notifies the check mk write conf
resource, which is queued
template "#{node[’check_mk’][’conf_dir’]}/main.mk" do
source "check_mk_main.mk.erb"
variables(
:nodes => nodes,
:hostgroups => hostgroups,
:params => node[’check_mk’][’params’]
)
mode 0644
owner "root"
group "root"
notifies :run, "execute[check_mk_write_conf]"
end
@gbagnoli Introduzione a Chef 12/09/2012 43 / 50
113. Monitoring - Server (3)
The check mk write conf resource regenerates icinga conf
@gbagnoli Introduzione a Chef 12/09/2012 44 / 50
114. Monitoring - Server (3)
The check mk write conf resource regenerates icinga conf
execute "check_mk_write_conf" do
command "#{node[’check_mk’][’prefix’]}/bin/check_mk -O"
action :nothing
end
@gbagnoli Introduzione a Chef 12/09/2012 44 / 50
115. Monitoring - Server (3)
The check mk write conf resource regenerates icinga conf
execute "check_mk_write_conf" do
command "#{node[’check_mk’][’prefix’]}/bin/check_mk -O"
action :nothing
end
Then the recipe scan nodes for services (using check mk inventory)
This is done only the first time (no reinventory)
nodes.each do |n|
if not n[’tags’] or not n[’tags’].include? "noagent"
check_mk_inventory n[’fqdn’]
end
end
check mk inventory is an LWRP defined in the check mk cookbook
@gbagnoli Introduzione a Chef 12/09/2012 44 / 50
116. Monitoring - Server (4)
That way new nodes are automatically added to the monitoring when they
show up as result of the search query.
Since cookbooks push and configure check mk plugins, everything gets
monitored and
all the monitoring logic is in the cookbook itself.
@gbagnoli Introduzione a Chef 12/09/2012 45 / 50
117. Monitoring Chef Clients
Once you start relying on chef, you want to know if chef-client is
correctly running on nodes and if/when it fails.
@gbagnoli Introduzione a Chef 12/09/2012 46 / 50
118. Monitoring Chef Clients
Once you start relying on chef, you want to know if chef-client is
correctly running on nodes and if/when it fails.
Instead on relying on the active model of check mk, we use a completely
passive approach. We use NSCA for this.
@gbagnoli Introduzione a Chef 12/09/2012 46 / 50
119. Monitoring Chef Clients
Once you start relying on chef, you want to know if chef-client is
correctly running on nodes and if/when it fails.
Instead on relying on the active model of check mk, we use a completely
passive approach. We use NSCA for this.
When chef-client runs on the node, it pushes a report/exception
handler called NSCAHandler.
This handler runs at the end of a chef-client run, and submits the
result to the NSCA server on the monitoring host, so that:
@gbagnoli Introduzione a Chef 12/09/2012 46 / 50
120. Monitoring Chef Clients
Once you start relying on chef, you want to know if chef-client is
correctly running on nodes and if/when it fails.
Instead on relying on the active model of check mk, we use a completely
passive approach. We use NSCA for this.
When chef-client runs on the node, it pushes a report/exception
handler called NSCAHandler.
This handler runs at the end of a chef-client run, and submits the
result to the NSCA server on the monitoring host, so that:
• check is in critical state if chef run failed.
• check is in warning/critical state if time elapsed is above thresholds
• check is in warning/critical state if the number of modified resources
is above thresholds
@gbagnoli Introduzione a Chef 12/09/2012 46 / 50
121. Monitoring Chef Clients (2)
Passive checks are created on the server for every node which runs the
chef-client
These checks have a freshness threshold of 1 day, so that if no data
arrives from the node the check will enter the UNKOWN state, meaning the
chef-client is not running on that node.
@gbagnoli Introduzione a Chef 12/09/2012 47 / 50
122. Monitoring - Final Words
The check mk:: server recipe also configures:
• host parents
• host groups
• service groups
• notification periods for services
• contact groups and administrators (data is in the accounts data bag)
• uses smokeping to perfom hosts checks (instead of using check ping)
• . . . and integrates smokeping web ui with check mk multisite.
• host icon for the status map :-)
Behavior can be changed by setting nodes tag with knife. i.e, this changes
the notification period for a host.
knife tag create myhost.example.com workhours
@gbagnoli Introduzione a Chef 12/09/2012 48 / 50