1. CONTINUOUSLY INTEGRATED PUPPET IN A DYNAMIC ENVIRONMENT
SAM BASHTON, BASHTON LTD

2. CONTINUOUSLY INTEGRATED PUPPET IN A DYNAMIC ENVIRONMENT
MASTERLESS PUPPET: WHY AND HOW
SAM BASHTON, BASHTON LTD

3. ABOUT ME
Linux guy since Slackware, floppy disks and root + boot
Using Puppet since 2007
Run a company in Manchester, North West England
We provide outsourced ops for other companies

4. OUR FULLY MANAGED ENVIRONMENTS
Primarily transactional websites (e­commerce)
Majority (70%+) on Amazon Web Services (AWS)
Majority using CentOS

5. HOW WE WORK
Simple is better than complex
Complexity is worth adding only if it provides obvious functional benefits
Re­usability
Resilience

6. WHY DID WE PICK AWS?
Featureset and toolset massively in advance of any other cloud provider, public or private
#1 customer reason for switching to AWS? The ability to scale on demand

8. TOOLS WE USE FOR BUILDING AND MANAGING
Do one thing and do it well
CloudFormation ­ Amazon tool to manage infrastructure
Puppet ­ Manage system configuration
Pulp ­ centralised repository, manages package revisions
Jenkins

9. HOW WE USE PUPPET
No Puppetmaster
Puppet manifests, hieradata and modules distributed to all machines via RPM
All machines boot with a common, blank image and get configured at first boot

10. WHAT'S WRONG WITH MASTER BASED PUPPET?
Pets vs Cattle
Puppet designed for a world of servers as pets
We do not live in that world

11. PUPPET DESIGNED FOR PETS
Many assumptions in Puppet presume that your servers are pets
Some of these work against us when managing a herd

12. MANUAL CERTIFICATE SIGNING
Clearly unsuitable when machines are automatically provisioned

13. POTENTIAL WORKAROUNDS:
Autosign
Use/write another automated certificate generation mechanism
Possibly tied in with autoscaling

14. NO MECHANISM FOR CLEANING OLD HOSTS
Likely to have host­names reused, causing machines to fail to configure
Puppetmaster will fill with certificates for machines that ran for a few hours and went away again

15. POTENTIAL WORKAROUNDS:
Use UUID certificates
Agree not to look in the certificate directory
Write mechanism for cleaning up old certificates

16. HOSTS CONFIGURED BASED ON HOSTNAME
Our machines have names like ip­172­26­5­123
How does Puppet know what type of machine this is?

17. POTENTIAL WORKAROUNDS
Use an external node classifier
Use some mechanism for giving a better hostname, eg web­172­26­5­123 and use regex for node
names

18. PUPPETMASTER IS A SINGLE POINT OF FAILURE
If the Puppetmaster fails, we can no longer autoscale up
In particular, this could be a problem if there is availability zone failure

19. POTENTIAL WORKAROUNDS
Clustered Puppetmasters

20. WORKAROUND RECAP
Use/write alternative certificate management software
Write an external node classifier / mechanism for setting hostname appropriately
Cluster multiple Puppetmasters

21. WHAT WE DID INSTEAD
Decided using a Puppetmaster was trying to fit a square peg into a round hole
Instead, decided to run Puppet without a master

22. APPLYING LOCAL PUPPET MANIFESTS
puppet apply --modulepath=/etc/puppet/modules example.pp

23. DISTRIBUTING MANIFESTS
Use RPM
Distribute full set of manifests/modules to each machine
Apply only the manifest relevant to that machine

24. PACKING PUPPET MANIFESTS IN RPM
Build an RPM containing everything under /etc/puppet
Make files readable only by root

25. APPLY PUPPET MANIFESTS
Have an RPM %postinst command apply the Puppet config
This isn't as straightforward as running the puppet apply from %postinst
Puppet needs to install packages via yum, but yum is running installing the Puppet package
Instead, we work around with a dirty hack: have the %postinst create an at script which
checks if yum has finished and then runs the puppet apply

26. RPM INSTALLATION AND MANAGEMENT
How do we get these RPMs on our machines?

27. PULP
We were already using Pulp
Provides yum repository management
Used for managing security updates and deploying application code
http://pulpproject.org/

28. WHAT IS PULP
Repository manager
Allows us to easily audit what packages and versions are installed where
Allows us to push package installations
Uses qpid message queue
Has concept of 'content distrubtion servers' for easy replication and clustering

29. HOW WE USE PULP
Puppet contains details of what packages should be installed
Pulp manages which version of the package should be installed
Pulp allows us to clone repos and copy packages between them for easy qa­>stage­>live environment
management

30. DEPLOYING CONFIGURATION AS CODE
Allows us to reuse our existing code deployment infrastructure
Manage configuration deployment from Jenkins

31. HOW WE DEPLOY CODE
Everything managed via the Jenkins continuous integration server
Jenkins uses Pulp to install code on remote machines

32. DETAILS ON HOW WE DEPLOY CODE
Jenkins fetches code from source control (git)
An RPM is built
Tests are run
If tests pass, the RPM is added to the relevant Pulp repository RPM installed on the target machine(s)

33. DEPLOYMENT LIFE­CYCLE
Jenkins also manages deployment life­cycle
RPMs are installed on staging
Promoted Builds plugin then used to install the same RPMs on live once testing is complete

34. PUPPET DEPLOYMENT PROCESS
Puppet manifests are checked into git
Lint tests via
Jenkins pulls in modules with librarian­puppet, then builds an RPM
Deployment to test environments, functional tests for wider code­base run
Jenkins Warnings plugin

35. PUTTING IT INTO PRODUCTION
Once suitable tests (automated and manual) have been carried out, we promote Puppet config into
production
We use the Jenkins 'Promoted Builds' plugin for this

36. JENKINS: PROMOTION

37. EXCEPT..
How does a machine get from a bare image to the state where we can push packages to it from Pulp?
How does a machine know what type of machine it is?
How do we find other resources, eg database hostname?

38. CLOUDFORMATION
Amazon tool for specifying infrastructure
Everything* we provision inside AWS is provisioned via CloudFormation
JSON templates
* Everything except for the things Amazon doesn't expose via CloudFormation..

39. CLOUD­INIT
Works with multiple cloud types
Sorts out things like SSH keys, allows us to configure host names
Also allows us to provide a bash script to run on startup

40. PROVISIONING A BARE INSTANCE
cloud­init automatically manually adds the pulp repo which contains Pulp, Puppet and our Puppet
manifests/modules
Installs appropriate RPMs
Puppet runs, subscribing the machine to the relevant Pulp repos, and installing packages in the usual
Puppet way

41. HOW DOES IT KNOW WHAT TYPE OF MACHINE IT IS?
We tell it!
Use an environmental variable $HOSTTYPE
Simply run
puppet apply
--modulepath=/etc/puppet/modules ${HOSTTYPE}.pp

42. EXTRA FACTS
Custom facter facts
Also specified in an environmental variable
Data comes from within the CloudFormation template
On our list of things to look at:
FACTER_HOSTENVIRONMENT=live
FACTER_STACKNAME=customer-web-live
https://github.com/fanduel/hiera­cloudformation

43. OTHER RESOURCES
We either:
Provide details as a facter fact
`FACTER_DBHOST=xyz
Also use this approach to limit distribution of secure details, eg DB passwords
Discover via the EC2 API
Eg Varnish servers discover web backends by calling API and finding hosts tagged
appropriately

44. FREE WINS!

46. FREE WINS!
Greater control over the timing of Puppet runs
Improved visibility ­ for ops and devs
Configuration changes now have to be deployed to testing/staging first

47. MORE FREE WINS!
Puppet configs now have a version
Easy to find config version on the machine itself
Config changelogs accessible on every machine
(Git changelog added to RPM)

48. THE DOWNSIDES
Puppet manifests and modules on all machines
Potentially a security issue?
Mitigated by CloudFormation holding most sensitive data

49. ALTERNATIVE IMPLEMENTATIONS
Don't want to use Pulp?
Could do basically the same thing with yum s3 plugin
Use mcollective to push package updates
https://github.com/jbraeuer/yum­s3­plugin

50. FUTURE IMPROVEMENTS
Build AMIs using Packer instead of configuring at boot time
Decrease time to autoscale
Would probably still need to run Puppet at first boot to configure machine specific settings

51. QUESTIONS? COMMENTS?
Sam Bashton
Twitter:
(Psst.. )
sam@bashton.com
@bashtoni
http://www.bashton.com/jobs/