During this hands-on tutorial you will learn how to quickly provision local test/development/demo environments using Vagrant and Virtualbox. We will cover provisioning and configuring machines quickly using Vagrant and CFEngine. You will learn how Vagrant and Virtualbox can be used to bring up local development/test/demo environments. You will also learn how CFEngine can be leveraged to automate configuration of the environment after it has been initialized. You will take away a multi-vm test environment managed by CFEngine.
This tutorial targets technical people who need repeatable test environments and are comfortable using the Linux command-line. These environments can speed developer on-boarding, play a role in continuous integration, or just provide quick sandboxes for experimentation. No previous knowledge of Vagrant or CFEngine is required.
2. Before we get started
Is everyone in the right place?
Has everyone installed Virtualbox 4.2.16 or later?
VirtualBox --help | grep VirtualBox
Has everyone installed Vagrant 1.2.7 or later?
vagrant –version
I have USB keys with Installers and content for the tutorial,
if you don't have it yet please let me know.
3. Get to know each other
Hi, I'm Nick
SysAdmin > 10 Years
Work at CFEngine
Live in Lawrence, KS
Twitter: @cmdln_
IRC: nickanderson
Blog: http://www.cmdln.org
Who are you?
What do you do?
Have you used Vagrant?
Have you used CFEngine?
Why did you choose this
session and what do you
hope to get from it?
4. 9/17/13
What is Vagrant?
Tool to make working with
development environments
easy.
Create, configure, destroy
lightweight, reproducible,
and portable environments.
● Created by Mitchell Hashimoto
● @mitchelh
● http://www.vagrantup.com
5. 9/17/13
How can it help?
Developer on-boarding
Quickly provision/decommission test environments
in repeatable fashion
Bug Validation
Continuous Integration
Ad-hoc Demos
10. 9/17/13
Boxes
● Base operating system image
● Provider specific
● http://www.vagrantbox.es
● Use veewee or packer.io (build your own automatically)
– Kickstart/preseed, postinstall scripts
12. 9/17/13
Useful Plugin
● vagrant-vbguest automatically installs
the host's VirtualBox Guest Additions
on the guest system.
● vagrant plugin install vagrant-vbguest
● https://github.com/dotless-de/vagrant
-vbguest
13. 9/17/13
CFEngine Provisioner: Currently Undocumented
● am_policy_hub
●
extra_agent_args
– Extra arguments to pass to cf-agent executions
●
classes
– Additional classes to define when running cf-agent
● deb_repo_file
– The apt repository configuration file to use for configuring
the repository containing the CFEngine packages
● deb_repo_line
– The line that specifys the repository to use for CFEngine
packages
●
files_path
– Directory to copy on top of the default masterfiles
● force_bootstrap
– If true, bootstrap the host even if it has been bootstrapped
before
● install
– Install CFEngine package from repository
● mode
– “bootstrap” or “single_run”, determines whether CFEngine will
be bootstrapped or just executed once on the host
●
policy_server_address
●
repo_gpg_key_url
– http location of GPG key used for checking package signatures
●
run_file
– Standalone CFEngine policy file to upload and execute
●
upload_path
– Path to upload run_file
●
yum_repo_file
– The yum repository file to use when configuring the repository
containing CFEngine packages
●
yum_repo_url
– The url of the repository containing the CFEngine packages
●
package_name
– The cfengine package name to install
22. 9/17/13
CFEngine
● IT infrastructure automation, compliance, and
knowledge management framework
● Opensource and Commercial Software
● Originally written by Mark Burgess
● @markburgess_osl
● http://www.cfengine.com
23. 9/17/13
CFEngine History
● First released in 1993
● CFEngine 2 released in 1998, self healing
computer immunology. Added machine
learning and anomaly detection.
● 2003 Promise Theory work began
● 2008 CFEngine 3 released. Integrates
knowledge management and discovery
mechanisms.
31. 9/17/13
Get Going
● Import vagrant basebox
– cd resources/veewee
– vagrant box add CFEngine_Training CFEngine_Training.box
● Bring up environment
– vagrant status
– vagrant up
– vagrant status
34. 9/17/13
Using cf-sketch to configure infrastructure
● Log in to your policy hub, locate the
design center repository and access
the cf-sketch shell
– vagrant ssh hub
– sudo -i
– cd /vagrant/resources/design-center/tools/cf-sketch
– ./cf-sketch.pl
45. Before we get started
Is everyone in the right place?
Has everyone installed Virtualbox 4.2.16 or later?
VirtualBox --help | grep VirtualBox
Has everyone installed Vagrant 1.2.7 or later?
vagrant –version
I have USB keys with Installers and content for the tutorial,
if you don't have it yet please let me know.
46. Get to know each other
Hi, I'm Nick
SysAdmin > 10 Years
Work at CFEngine
Live in Lawrence, KS
Twitter: @cmdln_
IRC: nickanderson
Blog: http://www.cmdln.org
Who are you?
What do you do?
Have you used Vagrant?
Have you used CFEngine?
Why did you choose this
session and what do you
hope to get from it?
47. 9/17/13
What is Vagrant?
Tool to make working with
development environments
easy.
Create, configure, destroy
lightweight, reproducible,
and portable environments.
● Created by Mitchell Hashimoto
● @mitchelh
● http://www.vagrantup.com
Questions? Stop me
48. 9/17/13
How can it help?
Developer on-boarding
Quickly provision/decommission test environments
in repeatable fashion
Bug Validation
Continuous Integration
Ad-hoc Demos
50. 9/17/13
Portable
● VirtualBox
● AWS
● VMware
● More
● https://github.com/mitchellh/vagrant/wiki/Available-Vagrant-Plugins
Take it with you (offline)
VirtualBox
Vmware
Use someone else's infrastructure
AWS
Rackspace
53. 9/17/13
Boxes
● Base operating system image
● Provider specific
● http://www.vagrantbox.es
● Use veewee or packer.io (build your own automatically)
– Kickstart/preseed, postinstall scripts
Veewee written by Patrick Debois
Packer.io written by Mitchell Hashimoto
Leverage your existing infrastructure. Use the same
(or very similar) kickstart/preseed and postinstall
scripts that you use in your production env.
54. 9/17/13
Magic
● Ssh port forwards
● Shared project folder /vagrant
Automatically forwards Local port to 22
on the host
Automatically mounts the directory that
the Vagrantfile lives in (Vagrant Project
Dir) on each host.
Add your own custom
Problems with automagic? Check the
tools version. Not required to match but
it helps
55. 9/17/13
Useful Plugin
● vagrant-vbguest automatically installs
the host's VirtualBox Guest Additions
on the guest system.
● vagrant plugin install vagrant-vbguest
● https://github.com/dotless-de/vagrant
-vbguest
● Sometimes there are issues if the
version of virtualbox tools does not
match the currently running version.
● This plugin will detect if the guest tools
are outdated, download build, install,
restart the guest
● If you're lucky, vagrant-vbguest does
not require any configurations.
56. 9/17/13
CFEngine Provisioner: Currently Undocumented
● am_policy_hub
● extra_agent_args
– Extra arguments to pass to cf-agent executions
●
classes
– Additional classes to define when running cf-agent
●
deb_repo_file
– The apt repository configuration file to use for configuring
the repository containing the CFEngine packages
● deb_repo_line
– The line that specifys the repository to use for CFEngine
packages
● files_path
– Directory to copy on top of the default masterfiles
● force_bootstrap
– If true, bootstrap the host even if it has been bootstrapped
before
●
install
– Install CFEngine package from repository
●
mode
– “bootstrap” or “single_run”, determines whether CFEngine will
be bootstrapped or just executed once on the host
● policy_server_address
● repo_gpg_key_url
– http location of GPG key used for checking package signatures
● run_file
– Standalone CFEngine policy file to upload and execute
●
upload_path
– Path to upload run_file
●
yum_repo_file
– The yum repository file to use when configuring the repository
containing CFEngine packages
● yum_repo_url
– The url of the repository containing the CFEngine packages
● package_name
– The cfengine package name to install
The CFEngine provisioner is currently
undocumented. This is a great
opportunity for someone to contribute. I
already did part of the work right here in
this slide ;)
Options for
Package Source/install
Extra Arguments
Bootstrap or Standalone oneshot policy
58. 9/17/13
Getting started
● vagrant box
● vagrant init
● vagrant status
● vagrant up
● vagrant ssh
– vagrant ssh node
● vagrant destroy
59. 9/17/13
Daily Use
vagrant up
vagrant {destroy, halt, suspend}
!-2
There are more commands, but you use vagrant up
to bring up an environment, and vagrant destroy to
delete the vms.
63. Black Hole
There are so many places where configurations can
hide. I think of Vms as kind of a black hole of
knowledge.
64. 9/17/13
Automating Vagrant Provisioning
● Ansible
● CFEngine
● Chef
● Puppet
● Salt Stack
● Shell Scripts
● MixnMatch!
Automate configuration on top of base
image.
There are valid reasons for baking
config into a basebox, usually for speed
of deployment. Not a replacement for
good configuration management.
65. 9/17/13
CFEngine
● IT infrastructure automation, compliance, and
knowledge management framework
● Opensource and Commercial Software
● Originally written by Mark Burgess
● @markburgess_osl
● http://www.cfengine.com
66. 9/17/13
CFEngine History
● First released in 1993
● CFEngine 2 released in 1998, self healing
computer immunology. Added machine
learning and anomaly detection.
● 2003 Promise Theory work began
● 2008 CFEngine 3 released. Integrates
knowledge management and discovery
mechanisms.
CFEngine has a solid history. Its been around for 20 years.
Runs on over 10 million servers in over 10 thousand
companies
Promises are a declaration of intent
67. 9/17/13
CFEngine Properties
● Small
CFEngine is written in C
~ 100k lines of code (remember its a 20
year old project)
~ 5M single package install
~ 15-25M memory consumption
(depends on your policy of course)
68. 9/17/13
CFEngine Properties
● Small
● Secure (http://web.nvd.nist.gov/view/vuln/search)
Security is a core focus. The voluntary
cooperation principal of Promise Theory
and the pull model are important for this.
Great track record
CFEngine hasn't had a published
security vulnerability since 2005
(CFEngine 2)
0 since CFEngine 3 was released in
2009
69. 9/17/13
CFEngine Properties
● Small
● Secure (http://web.nvd.nist.gov/view/vuln/search)
● Portable
Because its written in C it runs on just about
anything.
Linux, BSDs, AIX, HPUX, Solaris
even Windows
Storage Devices (Qnap)
Switches (Cisco, Arista, Juniper)
Embedded Devices
Raspburry Pi
Robot bottom of the ocean
Water testing devices in fields with cows
Laser cutters (that make puppets)
70. 9/17/13
CFEngine Properties
● Small
● Secure (http://web.nvd.nist.gov/view/vuln/search)
● Portable
● Resilient
CFEngine works when other things are broken.
CFEngine tries to fix itself – failsafe.cf
Decisions are made by the agents running on individual
hosts.
If the network is down they continue to apply the policy
they have. These policies can be extremely dynamic since
all decisions are made by the individual agent. They can
use external sources of information if desired or required.
Convergence – continual repair of system state toward
desired specification
If something can't be fixed track it and move on (usually)
If installing httpd fails, it could continue on and ensure that
SSH is hardened. Or if you desire all execution could stop
at that point.
71. 9/17/13
CFEngine Properties
● Small
● Secure (http://web.nvd.nist.gov/view/vuln/search)
● Portable
● Resilient
● Declarative
CFEngines policy language is declarative in nature.
This allows you to focus on the goals of how things should
be and converge towards this desired state.
It works kind of like a GPS. It doesn't matter where you
start, it will continually re-route to reach the destination.
For example
Apache promises to be installed on webservers. Not install
apache on host x,y,z
Httpd process promises to be running in production during
non maintenance hours
Sshd process promises to not be running, and completely
fire-walled off when the number of SSH sessions into or out
of a host are 3 standard deviations higher than normal
72. 9/17/13
CFEngine Properties
● Small
● Secure (http://web.nvd.nist.gov/view/vuln/search)
● Portable
● Resilient
● Declarative
CFEngines policy language is declarative in nature.
This allows you to focus on the goals of how things should
be and converge towards this desired state.
It works kind of like a GPS. It doesn't matter where you
start, it will continually re-route to reach the destination.
For example
Httpd config file promises to have this configuration for
hosts running application x
Httpd process promises to be running on web servers
Sshd process promises to not be running when the number
of SSH sessions into or out of a host are 3 standard
deviations higher than normal
73. 9/17/13
Bootstrap a test environment
Examine Vagrantfile
Shell provisioner to prep the environment for
offline use.
Dynamic multi-vm configuration
Host only network for vms to communicaate on
Forward ports
Synced files for hub masterfiles (normally, you
would update your masterfiles from a version
control repository)
CFEngine policy
Splay set to 0
Runs every minute (body executor control)
Emails root@localhost
Pre-written demo policy
74. 9/17/13
Get Going
● Import vagrant basebox
– cd resources/veewee
– vagrant box add CFEngine_Training CFEngine_Training.box
● Bring up environment
– vagrant status
– vagrant up
– vagrant status
We need to add it manually because
conference INTERNET
75. 9/17/13
More Nodes!
● Increase nodes in Vagrantfile
● vagrant up
● vagrant ssh node00{1,2}
Increase nodes to 1 or 2 (dependent on
resources)
Verify that you can ssh to them
Check out shared directory support
Look in /vagrant
Update a file from inside the vm, check
from workstation, vice versa
76. 9/17/13
CFEngine Design Center
● Community contributed reusable
policy
● Curated Repository
● Cli and GUI (enterprise) clients
When I am talking about paths to cfengine
configuration files in these examples they are relative
to masterfiles
So the synced vagrant directory is
resources/synced_masterfiles
Edit there and the policy will get synchronized to the
hubs masterfiles directory
77. 9/17/13
Using cf-sketch to configure infrastructure
● Log in to your policy hub, locate the
design center repository and access
the cf-sketch shell
– vagrant ssh hub
– sudo -i
– cd /vagrant/resources/design-center/tools/cf-sketch
– ./cf-sketch.pl
When I am talking about paths to cfengine
configuration files in these examples they are relative
to masterfiles
So the synced vagrant directory is
resources/overlay_var_cfengine/masterfiles
Edit there and the policy will get synchronized to the
hubs masterfiles directory
78. 9/17/13
Configure Timezones
● search time
● info -v tzconfig
● install System::tzconfig
● define paramset System::tzconfig
– Name: NO_Oslo_TZ
– Timezone: Europe/Oslo
– Zoneinfo: /usr/share/zoneinfo
You may want to have terminal open
79. 9/17/13
Configure Timezones
● search time
● install System::tzconfig
● define paramset System::tzconfig
– Name: NO_Oslo_TZ
– Timezone: Europe/Oslo
– Zoneinfo: /usr/share/zoneinfo
When I am talking about paths to cfengine
configuration files in these examples they are relative
to masterfiles
So the synced vagrant directory is
resources/overlay_var_cfengine/masterfiles
Edit there and the policy will get synchronized to the
hubs masterfiles directory
80. 9/17/13
Configure Timezones Cont.
● define paramset System::tzconfig
– Name: US_Central_TZ
– Timezone: US/Central
– Zoneinfo: /usr/share/zoneinfo
When I am talking about paths to cfengine
configuration files in these examples they are relative
to masterfiles
So the synced vagrant directory is
resources/overlay_var_cfengine/masterfiles
Edit there and the policy will get synchronized to the
hubs masterfiles directory
81. 9/17/13
Activate and Deploy Timezone Configuration
● activate System::tzconfig NO_Oslo_TZ hub
● activate System::tzconfig US_Central_TZ node001
● deploy
When I am talking about paths to cfengine
configuration files in these examples they are relative
to masterfiles
So the synced vagrant directory is
resources/overlay_var_cfengine/masterfiles
Edit there and the policy will get synchronized to the
hubs masterfiles directory
82. 9/17/13
Editor War!
● Which side are you on?
● services/editor_war.cf
Stop here, take a look at the file.
Who can tell what the policy is doing without having it
explained to them first?
83. 9/17/13
Wage War
● Remove Disallowed Packages
– vagrant ssh hub
– watch rpm -q emacs-nox
– Uncomment disallowed_packages to activate policy.
Watch it get fixed.
● Install Required Packages
– watch rpm -q vim-enhanced
– Uncomment required_packages to activate policy
How was this policy executed?
See body common control inputs and
bundlesequence.
Try playing around and manually
removing packages