1. CONTINUOUS
DEPLOYMENT OF PUPPET
MODULES
HOW WE DO IT AT MAILCHIMP

2. BILL O'NEILL
@WONEILL

4. EMAIL SERVICE PROVIDER
Deliver ~500 million emails daily
723 million emails delivered on Cyber Monday
Sender Score of 97
http://delivery.mailchimp.com/

5. HISTORY

6. Image source: https://blog.engineyard.com/2014/configure-before-
you-boot

7. CONFIGURATION MANAGEMENT IS HARD
"With Chef, Puppet, and CFEngine we found a
not-insignificant learning curve on setting up
the different server daemons and learning the
DSL. This was particularly challenging when
we were configuring unique software not yet
given recipes by the existing community.
Given our cluster sizes, we also didn't really
need any of the advanced features those
systems provided."
- README from internally built tool

8. MOVE TO COLOCATION
Buy vs. Lease analogy
Grow our Operations team
Needed a tool with dry-run mode

9. PEOPLE MAKE MISTAKES

10. HOW DO WE CATCH THESE MISTAKES AS EARLY
AS POSSIBLE?
AUTONOMATION
"automation with a human touch"
1. Detect the abnormality
2. Stop
3. Fix or correct the immediate condition

11. HOW DO WE CATCH THESE MISTAKES AS EARLY
AS POSSIBLE?
DSL tools
Editor Support
Source Code Management
Continuous Integration

12. DSL TOOLS
Puppet
ERB
YAML
Puppet Style Guide

13. PUPPET
puppet parser validate mymanifest.pp

14. package { 'openssh-server':
ensure => installed,
}
file { '/etc/ssh/sshd_config':
source => 'puppet:///modules/sshd/sshd_config',
owner => 'root',
group => 'root',
mode => '640',
notify => Service['sshd'] /* sshd will restart whenever you
edit this file. */
require => Package['openssh-server'],
}
service { 'sshd':
ensure => running,
enable => 'true',
hasstatus => 'true',
hasrestart => 'true',
}

15. $ puppet parser validate validate_1.pp
Error: Could not parse for environment production:
Syntax error at 'require'; expected '}' at validate_1.pp:12

16. ERB
erb -P -x -T '-' mytemplate.erb | ruby -c

17. restrict default kod nomodify notrap nopeer<% unless @service %> noquery
restrict 127.0.0.1
restrict -6 ::1
driftfile /var/lib/ntp/drift
<% @serverlist.sort.each do |server| -%>
server <%= server %> iburst maxpoll 6
restrict <%= server %> mask 255.255.255.255 nomodify notrap noquery
<% end -%>

18. $ erb -P -x -T '-' broken-ntp.conf.erb | ruby -c
-:11: syntax error, unexpected $end, expecting kEND

19. YAML
npm install -g js-yaml; js-yaml hiera.yaml
ruby -e "require 'yaml'; YAML.load_file('hiera.yaml')"

20. ---
ntp::servers:
- 0.us.pool.ntp.org
- 1.us.pool.ntp.org
- 2.us.pool.ntp.org
- 3.us.pool.ntp.org
hp::ilo::settings:
ssh_status
type: global
value: true
ssh_port
type: global
value: '22'
http_port
type: global
value: '80'
https_port
type: global
value: '443'

21. $ js-yaml hiera.yaml
JS-YAML: bad indentation of a mapping entry at line 9, column 13:
type: global
^
$ ruby -e "require 'yaml'; YAML.load_file('hiera.yaml')"
yaml.rb:133:in `load':
syntax error on line 9, col 14: ` value: true' (ArgumentError)
from yaml.rb:133:in `load'
from yaml.rb:144:in `load_file'
from yaml.rb:143:in `open'
from yaml.rb:143:in `load_file'
from -e:1

22. PUPPET STYLE GUIDE
https://docs.puppetlabs.com/guides/style_guide.html
gem install puppet-lint
puppet-lint --fix /my/puppet/code

23. package { 'openssh-server':
ensure => installed,
}
file { '/etc/ssh/sshd_config':
source => 'puppet:///modules/sshd/sshd_config',
owner => 'root',
group => 'root',
mode => '640',
notify => Service['sshd'], /* sshd will restart whenever you
edit this file. */
require => Package['openssh-server'],
}
service { 'sshd':
ensure => running,
enable => 'true',
hasstatus => 'true',
hasrestart => 'true',
}

24. $ puppet-lint validate_2.pp
WARNING: quoted boolean value found on line 16
WARNING: quoted boolean value found on line 17
WARNING: quoted boolean value found on line 18
WARNING: indentation of => is not properly aligned on line 6
WARNING: indentation of => is not properly aligned on line 7
WARNING: indentation of => is not properly aligned on line 8
WARNING: indentation of => is not properly aligned on line 9
WARNING: indentation of => is not properly aligned on line 10
WARNING: mode should be represented as a 4 digit octal value
or symbolic mode on line 9
WARNING: /* */ comment found on line 10

25. $ puppet-lint --fix validate_2.pp
FIXED: quoted boolean value found on line 16
FIXED: quoted boolean value found on line 17
FIXED: quoted boolean value found on line 18
FIXED: indentation of => is not properly aligned on line 6
FIXED: indentation of => is not properly aligned on line 7
FIXED: indentation of => is not properly aligned on line 8
FIXED: indentation of => is not properly aligned on line 9
FIXED: indentation of => is not properly aligned on line 10
FIXED: mode should be represented as a 4 digit octal value
or symbolic mode on line 9
FIXED: /* */ comment found on line 10

26. package { 'openssh-server':
ensure => installed,
}
file { '/etc/ssh/sshd_config':
source => 'puppet:///modules/sshd/sshd_config',
owner => 'root',
group => 'root',
mode => '0640',
notify => Service['sshd'], # sshd will restart whenever you
# edit this file.
require => Package['openssh-server'],
}
service { 'sshd':
ensure => running,
enable => true,
hasstatus => true,
hasrestart => true,
}

27. --- validate_2.pp 2014-12-08 09:43:38.000000000 -0500
+++ validate_2.pp-fixed 2014-12-08 09:50:51.000000000 -0500
@@ -3,18 +3,18 @@
}
file { '/etc/ssh/sshd_config':
- source => 'puppet:///modules/sshd/sshd_config',
- owner => 'root',
- group => 'root',
- mode => '640',
- notify => Service['sshd'], /* sshd will restart whenever you
- edit this file. */
+ source => 'puppet:///modules/sshd/sshd_config',
+ owner => 'root',
+ group => 'root',
+ mode => '0640',
+ notify => Service['sshd'], # sshd will restart whenever you
+ # edit this file.
require => Package['openssh-server'],
}
service { 'sshd':
ensure => running,
- enable => 'true',
- hasstatus => 'true',
- hasrestart => 'true',
+ enable => true,
+ hasstatus => true,
+ hasrestart => true,
}

28. EDITOR SUPPORT

29. VIM
PLUGINS FTW
Syntastic
vim-puppet
UltiSnips with
vim-snippets

30. EMACS
http://www.emacswiki.org/emacs/PuppetProgramming

31. GEPPETTO
http://puppetlabs.github.io/geppetto/index.html

32. SOURCE CODE
MANAGEMENT

33. COMMIT HOOKS
SCRIPT RUNNING THE DSL TOOLS AGAINST NEW FILES

34. PEER REVIEW

36. TRUNK BASED
DEPLOYMENT

37. CONTINUOUS
INTEGRATION

38. JENKINS
HTTPS://GITHUB.COM/VSTONE/JENKINS-PUPPET-
SCRIPTS

39. WHY NOT RSPEC OR
BEAKER?

40. CONTINUOUS
DEPLOYMENT

41. REMEMBER TRUNK BASED
DEPLOYMENT?
# Keep environment up-to-date
vcsrepo { '/etc/puppet/environments/production':
ensure => latest,
provider => hg,
source => 'https://localhost/mercurial/puppet-modules',
}

42. REVIEW TIME!
Catch mistakes early
Automation with a human touch
Trunk Based Deployments

43. QUESTIONS?

44. THANKS!
BILL O'NEILL
WONEILL@POBOX.COM
@WONEILL
Slide sources at
http://github.com/woneill/puppetcamp_atlanta_2014