This document discusses automated testing of Puppet code. It describes using RSpec-puppet for unit testing Puppet manifests and ensuring resources are defined correctly. It also discusses performing "after testing" by executing tests on systems after Puppet runs to check configurations are applied as expected. Post Provision Checks (PPC) is introduced as a framework for these user space tests that can integrate with monitoring tools and be extended with plugins. The document emphasizes the importance of testing beyond just Puppet reports and provides examples of issues user space tests can find that manifest tests may miss.

1. Automated Puppet Testing
“All code is guilty, until proven
innocent.” – Anonymous
Scott Nottingham July 23rd, 2012

2. Before and After Testing
• Before Testing
– Unit tests
• Cucumber-puppet
• Rspec-puppet
• After Testing
– User Space tests
• Did everything that was supposed to get done actually
get done?

3. Unit Tests - frameworks
• Cucumber-puppet
– Tests the outcome of the entire module as
opposed to just the manifests
– Overly complicated by trying to be simple (imho)
• Rspec-puppet
– Unit tests for the manifests
– Truly simple (and effective)

4. RSpec-puppet
• Written by Tim Sharpe (rodjek)
• Available on github
– https://github.com/rodjek/rspec-puppet/

5. RSpec-puppet
• PROs
– Simple language and structure
– Easy to maintain
– Fast to execute
– Execution easily automated
• CONs
– Only tests what you tell it to
• Easy to make changes to manifests and forget to update
RSpec tests

6. Writing RSpec Tests
• Directory Tree
module
|
+-- manifests
|
+-- files
|
+-- templates
|
+-- lib
|
+-- spec
|
+-- spec_helper.rb
|
+-- classes
|
+-- <class_name>_spec.rb

7. Writing RSpec Tests
• spec_helper file
– Required for RSpec to know where to find the module
path
– Resides within: <module_name>/spec/spec_helper.rb
– contents:
require 'puppet'
require 'rubygems'
require 'rspec-puppet'
RSpec.configure do |c|
c.module_path = File.join(File.dirname(__FILE__), '../../')
c.manifest = '../../../../manifests/site.pp'
end

8. Writing RSpec Tests
• Spec file
– Contains test logic
– Naming convention:
• <classname>_<pp_filename>_spec.rb
• Ex. nagios_init_spec.rb
– Contents
• Requires
– require ‘spec_helper’
• Tests

9. Writing RSpec Tests
• Defining Tests
– Tests are created using the ‘describe’ method.
– The first test must describe the class defined in
your manifest.
describe ‘nagios’ do
…
…
end

10. Writing RSpec Tests
• Accounting for class dependencies
– Some modules define class dependencies
– Example:
• Class*‘yum’+ -> Class*‘nagios’+
– These are handled in RSpec test with
‘:pre_condition’
let(:pre_condition) , 'class ,”yum":-' -

11. Writing RSpec Tests
• Setting parameters used by puppet manifests
– Optional unless manifest needs them to compile
– Handled with ‘:params’
– Example (nested parameters):
let(:params) { {:key1 => 'value1', :key2 => 'value2', :key3 => {'keyA' => 'valA'},
:key4 => {'keyB' => {'keyX' => 'valX'} } } }

12. Writing RSpec Tests
• Setting a node name for the tests (optional)
– Might use if your manifests has node name
dependent logic
let(:node) { host.example.com }

13. Writing RSpec Tests
• Setting custom fact values
– Probably the most commonly used
– Example:
let(:facts) { {:fact1 => 'val1', :fact2 => 'val2'} }

14. Writing RSpec Tests
• Sub-tests
– These are the tests for each condition that you want
to test for a given manifest.
describe 'nagios' do
describe 'with valid parameters passed' do
....
end
describe 'with invalid parameters passed' do
....
end
end

15. Writing RSpec Tests
• Testing the resources
• Generally speaking, the tests you will be writing will be to confirm that the
class contains some set of resources. This is done with the
‘contain_<resource_type>’
• Each resource will generally have attributes that you will want to test are
present. This is done with the ‘.with_<attribute>(‘value’)’ method.
• Example:
it { should contain_file('foo').with_ensure('present').with_owner('root').with_group('root') }

16. Writing RSpec Tests
• Cascading .with_ methods can get ugly!
• Cascading should only be used for resources with 1 or 2
attributes defined.
• Alternatively, you can use block notation for resources with
many attributes
it do should contain_file('/var/spool/squid/cache').with(
'require' => 'Package[squid]',
'ensure' => 'directory',
'owner' => 'squid',
'group' => 'squid'
) end

17. Writing RSpec Tests
• Sometimes multiple values are assigned to a
single attribute
• These are handled like so:
it do should contain_mount('/var/spool/squid/cache').with(
'require' => ['Logical_volume[lv_cache]', 'Package[xfsprogs]' ],
....
....
) end

18. Writing RSpec Tests
• Templates
– If your file content is generated by a template, rspec can test for this
too. Here’s how:
• Test that a template generates any content
it 'should generate valid content for squid.conf' do
content = catalogue.resource('file', '/etc/squid/squid.conf').send(:paramters)[:content]
content.should_not be_empty
End
• Test that a template generates specific content
it 'should generate valid content for squid.conf' do
content = catalogue.resource('file', '/etc/squid/squid.conf').send(:paramters)[:content]
content.should match('some_string')
end

19. Writing RSpec Tests
• Test for an expected error
• Sometimes, you want to test that an error is
raised.
– For example, if not passing a parameter when one
is needed, or passing an invalid parameter.
– Example test
it { expect { should contain_class('nagios::server') }.to raise_error(Puppet::Error) }

20. require 'spec_helper'
describe 'sendmail' do
let(:params) { {:mail_server => 'foobarbaz'} }
it { should contain_package('sendmail').with_ensure('installed') }
it do should contain_file('/etc/mail/sendmail.cf').with(
'require' => 'Package[sendmail]',
'notify' => 'Exec[makemap]'
) end
it 'should generate /etc/mail/sendmail.cf from a template' do
content = catalogue.resource('file', '/etc/mail/sendmail.cf').send(:parameters)[:content]
content.should match('foobarbaz')
end
it 'should generate /etc/mail/submit.cf from a template' do
content = catalogue.resource('file', '/etc/mail/submit.cf').send(:parameters)[:content]
content.should match('foobarbaz')
end
it do should contain_file('/etc/mail/submit.cf').with(
'require' => 'Package[sendmail]',
'notify' => 'Exec[makemap]'
) end
it { should contain_file('/etc/sysconfig/sendmail').with_source('puppet:///sendmail/sendmail') }
it do should contain_service('sendmail').with(
'require' => 'Package[sendmail]',
'enable' => 'true',
'ensure' => 'running'
) end
end

21. Running RSpec Tests
• rspec –color –format documentation
/etc/puppet/modules/sendmail/spec/classes/sendmail_init_spec.rb
sendmail
should contain Package[sendmail] with ensure => "installed"
should contain File[/etc/mail/sendmail.cf] with notify => "Exec[makemap]" and require =>
"Package[sendmail]"
should generate /etc/mail/sendmail.cf from a template
should generate /etc/mail/submit.cf from a template
should contain File[/etc/mail/submit.cf] with notify => "Exec[makemap]" and require =>
"Package[sendmail]"
should contain File[/etc/sysconfig/sendmail] with source => "puppet:///sendmail/sendmail"
should contain Service[sendmail] with enable => "true", ensure => "running" and require =>
"Package[sendmail]"
Finished in 1.98 seconds
7 examples, 0 failures

22. Running RSpec Tests
• Oops! Someone put a comma where they meant to put a semi-colon!
Sendmail
Failures:
1) sendmail
Failure/Error: it { should contain_package('sendmail').with_ensure('installed') }
Puppet::Error:
Syntax error at '/etc/sysconfig/sendmail'; expected '}' at
/etc/puppetmaster/modules/sendmail/spec/../../sendmail/manifests/init.pp:19 on node neodymium.trading.imc.intra
# ./sendmail_init_spec.rb:6
2) sendmail
Failure/Error: it { should contain_package('sendmail').with_ensure('installed') }
Puppet::Error:
Syntax error at '/etc/sysconfig/sendmail'; expected '}' at
/etc/puppetmaster/modules/sendmail/spec/../../sendmail/manifests/init.pp:19 on node neodymium.trading.imc.intra
# ./sendmail_init_spec.rb:6
Finished in 0.19808 seconds
2 examples, 2 failures

23. Automating RSpec Tests
• Tests can be run by anything capable of executing a
system command
• We integrate ours into Team City
– Much like Jenkins
– Allows for event driven execution of tests
• Any time a commit occurs in version control, tests
are executed
– If tests pass, manifests are wrapped inside an RPM and
pushed to puppet masters
– If tests fail, we are notified

24. Automating RSpec Tests
• RSpec tests do a great job of making sure manifests
are syntactically correct and, generally speaking,
function as expected.
• But….Things aren’t
always as they seem!

25. ‘After’ Tests
• To ensure our environment really is clean we
must test in user space land.
• Probably many solutions out there
– We wrote our own

27. User Space Testing
• Post Provision Checks (PPC) is a framework for
testing stuff in user space.
– Originally written as health checks for testing a
machine immediately after provisioning it and as
verification testing after maintenance.
– Quickly grew to an extensible framework capable of
integrating with Nagios/Icinga and Teamcity, or
running as stand alone application.
– Contains libraries of functions for commonly used
tasks.
– Easily customize runtime options with profiles
– Can be extended with plugins

28. PPC functionality

29. puppet_ust plugin
• The puppet user space tests plugin was
written to extend the framework for
specifically testing puppet applied
configurations.
– Auto detects the puppet classes that are applied
to a given box
– Determines the parameters defined for the host
being tested and utilizes them within tests.
• Also uses facter facts in the same manner

30. That’s great, but puppet reports will tell you
what worked and what didn’t…

31. misses
• You wouldn’t believe the things we have seen go wrong
with applying configurations (that the puppet logs/reports
didn’t indicate).
– PPC tests can be written to catch all of these!
• Mistakes only happen once
• Winning!
– A few examples
• Version of package specified not what got installed – not puppet’s
fault, but still a problem.
• Package got installed, but some key files were zero length
• Drivers upgraded, but not properly loaded
• Machines joined to wrong domain
• Services running (per status) but not functioning properly
• Yum repos configured but not functioning
• Sysctl settings still not loaded after sysctl –p

32. PPC sample output
• Stand Alone mode

33. PPC Sample Output
• Team City Mode

34. Automating PPC Tests
• Team City + multiplexed SSH (for parallelization)
• Each version control commit results in a puppet
run + user space test run in our dev and qa
environments.
• Over 20,000 tests performed in ~20 minutes
– This means every 20 minutes new, fully tested
changes are pushed from the development
environment to QA
– Each stage of the pipeline is tested in a similar fashion