Keith Ferguson, Australia & New Zealand Banking Group

1. Classification: Public
Puppet and A DevOps Journey
Keith Ferguson
Puppet Camp Melbourne 2014

2. TODAY
Who am I?
What is ANZ GCIS?
Our challenges
Our approach
Lessons Learnt (a.k.a. Things I Wish We Knew at the Beginning)
Storing sensitive information in Hiera
Classification: Public
2

3. WHAT IS ANZ GCIS
Classification: Public
3
Global Customer Integration Solution

4. WHAT IS ANZ GCIS
$30.00
Billion
$25.00
$20.00
$15.00
$10.00
$5.00
Classification: Public
4
350%
300%
250%
200%
150%
100%
50%
0%
Customer Growth
Oct-12 Oct-13
$-
Value Processed
Oct-12 Oct-13

5. GCIS DEVELOPMENT – MARCH 2014
4 teams, ~40 people
Windows Server / .NET / IIS / Microsoft SQL
~50 servers across 5 environments (Prod/DR + 4 non-prod environments)
Agile (scrum)
Supporting Tools:
Git
Atlassian: Bamboo, Stash, Jira, Confluence
Powershell
RDP & Elbow Grease
Classification: Public
5

6. GCIS DEVELOPMENT – MARCH 2014
Things We Did Well
CI & Deployment Automation – ~8k builds, 2.5k non-prod deploys / month
Test Automation – On commit & nightly for all channels
Delivering change to our existing channels
Classification: Public
6
9
8
7
6
5
4
3
2
1
0
Production Releases
Jan Feb Mar Apr May Jun Jul Aug Sep Oct

7. GCIS DEVELOPMENT – MARCH 2014
Things We Needed To Do Better
Management of complexity — Dependencies and change
Delivery of new channels — Infrastructure & configuration of application
containers
3rd level support & ops
Classification: Public
7

8. CHALLENGE 1 — DEPENDENCIES & CHANGE
Classification: Public
UAT ENVIRONMENT
8
3 2
1
We Had:

9. CHALLENGE 1 — DEPENDENCIES & CHANGE
PROJECT 1 PROJECT 2 PROJECT 3
Classification: Public
9
We Needed:

10. CHALLENGE 2 — GROWTH IN CHANNELS
10
5
Classification: Public
10
0
2012 2013 2014 2015

11. CHALLENGE 3 — 3RD LEVEL OPS
Little to no production access
No broad knowledge of production infrastructure
Heavy reliance on key team members
Far removed from incidents
Classification: Public
11

12. Classification: Public
12
APPROACH
APPLICATION INSIGHT CENTRALIZED LOGGING CONFIGURATION MANAGEMENT

13. OUR APPROACH
Puppet Enterprise vs. Open Source
Classification: Public

14. OUR APPROACH
1. Dedicate somebody
2. Spike it, try to understand it
Classification: Public
14

15. OUR APPROACH
Classification: Public
15
https://docs.puppetlabs.com/pe/latest/images/puppet/pe-configuration-data.png

16. OUR APPROACH
1. Dedicate somebody
2. Spike it, try to understand it
3. Identify the right questions
4. Training for a core group – Puppet Fundamentals
5. Implement the framework
6. Expand usage across team
Classification: Public
16

17. ROLES & PROFILES
Puppet Is Code. Abstractions Matter
A node has one role
A role is composed of one or more profiles
Profiles are composed of module declarations
Craig Dunn:
https://puppetlabs.com/presentations/designing-puppet-rolesprofiles-pattern
Classification: Public
17

18. ROLES & PROFILES
Classification: Public
18
site.pp wordpress.pp
db.pp
node wp {
include role::wordpress
}
class role::wordpress {
include profile::db
include profile::db::php
include profile::wordpress
}
class profile::db {
class { 'mysql::server':
Adapted from https://github.com/hunner/roles_and_profiles
config_hash =>
{ 'root_password' =>
'8ZcJZFHs...' }
}
}
class role::wordpress_web {
include profile::wordpress
}
class role::wordpress_db {
include profile::db
include profile::db::php
}
site.pp
node wp_web {
include role::wordpress_web
}
node wp_db {
include role::wordpress_db
}
roles

19. CLASSIFICATION
Site.pp
Classification: Public
19
node 'www1.example.com’, 'www1.example.com' {
include profile::common
include profile::apache
}
node /^(foo|bar)d+.example.com$/ {
include profile::common
}

20. CLASSIFICATION AT THE NODE
Define a role Fact
2 site.pp
Classification: Public
20
1
node default {
} include $::role
On the node:
puppet config set role ‘role::some_role’
Or remotely:
Invoke-Command –ComputerName c1,c2 –ScriptBlock {puppet config set role ‘role::some_role’}
3
CON
> Not secure for multi-tenant environments
> Node classification is not version controlled
PRO
> Simple & flexible
> Great for dynamic environments

21. PACKAGING ON WINDOWS
Like Yum/Apt-Get for Windows, Powered by NuGet & Powershell
Classification: Public
21
https://chocolatey.org

22. PACKAGING ON WINDOWS
$chocolatey_package_list = [
]
package { $chocolatey_package_list:
}
Classification: Public
22
Chocolatey Package Provider for Puppet:
'powershell4',
'DotNet4.5',
'git.install',
ensure => installed,
provider => 'chocolatey',
https://forge.puppetlabs.com/rismoney/chocolatey

23. PACKAGING ON WINDOWS
Classification: Public
23
Constraints
No internet access
Trust
Package availability

24. PACKAGING ON WINDOWS
Package1
|--- Package1.nuspec
|--- ChocolateyInstall.ps1
|--- ChocolateyUninstall.ps1
Package2
|--- Package2.nuspec
|--- ChocolateyInstall.ps1
|--- ChocolateyUninstall.ps1
Classification: Public
24
Binaries Archive
IIS
nuget feed
Install-ChocolateyPackage ... -checksum 'EE01FC4110C73A8E5EFC7CABDA0F5FF7'

25. R10K IS KEY TO WORKFLOW
Do you want?
Multiple environments from a single puppet master
Dynamic environments eg from feature branches
Declarative management of puppet modules: Puppetfile
If so you want r10k: https://github.com/adrienthebo/r10k
‘Smarter Puppet deployment, powered by killer robots’
Classification: Public
25
forge "https://forge.puppetlabs.com"
mod 'puppetlabs-ntp', "0.0.3”
mod 'puppetlabs-apt',
:git => "git://github.com/puppetlabs/puppetlabs-apt.git"

26. R10K DEPLOYMENT
Classification: Public
1. Repository Trigger 2. Webhook to r10k
3. git: fetch branch to puppet master

27. GOVERNANCE
Trust, but verify: Code
Minimum approvers policy for PRs
Pull request please plugin:
Workzone plugin:
- PR reviewer addition via policy (Users + Groups)
- Automated un-approval on PR change
Classification: Public
27
____ _ _ ____ _ ___ _
| _ _ _ | || | | _ ___ __ _ _ _ ___ ___ | |_ |__ | |
| |_) || | | || || | | |_) | / _ / _` || | | | / _ / __|| __| / /| |
| __/ | |_| || || | | _ < | __/| (_| || |_| || __/__ | |_ |_| |_|
|_| __,_||_||_| |_| _ ___| __, | __,_| ___||___/ __| (_) (_)
|_|
Please raise a pull request to merge code to this branch.

28. GOVERNANCE
Trust, but verify: Deployment
Classification: Public
28

29. R10K & PUPPETFILE
Challenge:
Puppetfile functionality runs on the master
Master needs to download modules from somewhere
Master has no internet access
Trust
Classification: Public
29

30. R10K MODULE WORKFLOW
acidprime/r10k
* master
Classification: Public
devops/r10k
Puppetfile
mod 'r10k',
* gcis-master
/
* master
:git => 'ssh://git@stash/r10k.git',
:ref => 'gcis-master'

31. WHEN IS AN ENVIRONMENT NOT AN ENVIRONMENT
Do you want to stage the rollout of puppet code across Production?
If yes: production is not a puppet environment, it’s an application environment (tier)
Puppet environments exist to apply different revisions of code to different nodes
Application environments are how you want to configure (hiera) your code
Classification: Public
31
|-- prod_a.yaml
|-- prod_b.yaml
|-- dr_a.yaml
|-- dr_b.yaml
Problem: Hiera: "%{environment}”
Solution: Custom fact ‘tier’: puppet config set tier ‘production’
Hiera: "%{::tier}"
|-- production.yaml
|-- uat.yaml
|-- sit.yaml
|-- st.yaml
http://garylarizza.com/blog/2014/03/26/random-r10k-workflow-ideas/

32. USE PUPPET TO MANAGE PUPPET
Example: Hiera.yaml
Classification: Public
32
---
:backends:
- yaml
:logger: console
:hierarchy:
- "nodes/%{::clientcert}"
- "%{environment}"
- global
:yaml:
:datadir: /etc/puppetlabs/puppet/hiera/%{environment}/

33. USE PUPPET TO MANAGE PUPPET
Example: Hiera.yaml via Puppet – https://forge.puppetlabs.com/hunner/hiera
Classification: Public
33
class { '::hiera':
backends => ['yaml’],
datadir => '/etc/puppetlabs/puppet/hiera/%{environment}/',
hierarchy => [
'servers/%{::clientcert}’,
'%{environment}’,
'global',
],
}

34. USE PUPPET TO MANAGE PUPPET
Classification: Public
34
Example: Install & configure r10k - https://forge.puppetlabs.com/zack/r10k
class profile::puppet::master {
class { 'r10k':
version => '1.3.1',
sources => {
'puppet' => {
'remote' => 'ssh://git@stash/puppet.git',
'basedir' => "${::settings::confdir}/environments",
'prefix' => false,
},
'hiera' => {
'remote' => 'ssh://git@stash/hiera.git',
'basedir' => "${::settings::confdir}/hiera",
'prefix' => false,
},
},
purgedirs => ["${::settings::confdir}/environments", "${::settings::confdir}/hiera" ],
manage_modulepath => true,
modulepath => "${::settings::confdir}/environments/$environment/modules:${::settings::…
mcollective => true,
}
}

35. STORING SENSITIVE INFORMATION IN HIERA
Hiera-eyaml: https://github.com/TomPoulton/hiera-eyaml
/etc/puppetlabs/puppet/hiera.yaml
Classification: Public
35
---
:backends:
- eyaml
:eyaml:
:datadir: /etc/puppet...
:extension: "yaml"
:pkcs7_private_key: "/etc/..."
:pkcs7_public_key: "/etc/..."
/etc/puppetlabs/puppet/hiera/prod.yaml
---
profile::my_app::app_secret: >
ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoII
BejCCAXYCAQAxggEhMIIBHQIBADAFMAAC
AQEwDQYJKoZIhvcNAQEBBQAEggEAWLZRf
TW8YxvCOf4akyxYdBV5m2AdF+HF85Cz
lqw8qQNKW9oMqktgyaoNzOMGufr0rp2BO
zrdQ+/z3bmQ9HPK2qqJ1k/n/jX9
D5lf5ReEKj5VnhnLblbLyI5HCfLbR...]

36. STORING SENSITIVE INFORMATION IN HIERA
/etc/puppetlabs/puppet/hiera.yaml
Classification: Public
36
0000 +
---
:backends:
- eyaml
:eyaml:
:datadir: /etc/puppet...
:extension: "yaml"
:pkcs7_private_key: "/etc/..."
:pkcs7_public_key: "/etc/..."
/etc/puppetlabs/puppet/hiera/prod.yaml
---
profile::my_app::app_secret: >
ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoII
BejCCAXYCAQAxggEhMIIBHQIBADAFMAAC
AQEwDQYJKoZIhvcNAQEBBQAEggEAWLZRf
TW8YxvCOf4akyxYdBV5m2AdF+HF85Cz
lqw8qQNKW9oMqktgyaoNzOMGufr0rp2BO
zrdQ+/z3bmQ9HPK2qqJ1k/n/jX9
D5lf5ReEKj5VnhnLblbLyI5HCfLbR...]

37. STORING SENSITIVE INFORMATION IN HIERA
Classification: Public
37
Hardware Security Module (HSM)
Provides network based FIPS 140-2 Level 3 secure cryptographic services
Puppet Integration:
Custom hiera eyaml encryptor: https://github.com/acidprime/hiera-eyaml-pkcs11
Operates in two modes: Native pkcs11 or OpenSSL CHIL
RSA encryption – Private key is protected by HSM, public key is committed to
version control
Native mode will not work in PE >= 3.4 due to jruby
Currently limited by RSA block size

38. hiera-eyaml-pkcs11
Classification: Public
38

39. SUMMARY
Roles & Profiles Pattern
Classification at the node
Chocolatey for packaging on Windows
Use r10k & build a good workflow, governance included
Puppet environments aren’t application environments
Use Puppet to manage puppet
The community is excellent, use it
Storing sensitive information in hiera – This is a risk weighted decision
Classification: Public
39

40. WHERE ARE WE NOW
> Started in March
> Appdynamics, Splunk & Puppet all operational
> Deployed & configured AD + Splunk agents via Puppet & Chocolatey (and
upgraded since)
> Functional vagrant workflow integrated with vmware through the vagrant-vsphere
plugin
> Puppet framework is stable, expanding use to broader team
> On-demand testing environments are a WIP
Classification: Public
40

41. THANKS
My Team
— Dylan Ratcliffe
— William Gaunt
— Laith Murad
GCIS Infrastructure Team
— Lee Murphy
— Sathish Kannan
Brett Gray & Zack Smith
Gary Larizza & Craig Dunn
Rob Reynolds
Classification: Public
41

42. LINKS & INFORMATION
Contact Info:
keith.ferguson@anz.com; linkedin.com/in/keithferguson
Tools
r10k - https://github.com/adrienthebo/r10k – read the docs not just the readme
r10k configuration module - https://github.com/acidprime/r10k
hiera eyaml - https://github.com/TomPoulton/hiera-eyaml
hiera eyaml pkcs11 backend - https://github.com/acidprime/hiera-eyaml-pkcs11
Reading / Learning
Gary Larizza – http://garylarizza.com/ – Building a functional puppet workflow series is great
Rob Nelson Puppet for vSphere Admins – http://rnelson0.com/puppet-for-vsphere-admins/
Crag Dunn’s Roles & Profiles Pattern Talk – https://puppetlabs.com/presentations/
designing-puppet-rolesprofiles-pattern
Puppet Conf & Puppet Camp talks in general
Classification: Public
42

43. QUESTIONS
Classification: Public
43