Experiences from Running Masterless Puppet - Erik Dalén, Spotify

1. September 24, 2014
Experiences from
running masterless
Puppet
Erik Dalén

2. whoami
• System engineer at Spotify since almost 3 years.
• Community contributor to Puppet.
• Author of Puppet Explorer web UI.
• Author of the puppetdbquery, dnsquery and puppet ls modules.
2

3. Agenda
• Background about our puppet setup
• How we implemented masterless
• Why we did the switch to masterless
• New workflows and future improvements
3

4. Background
• Puppet users since 4 years
• Two large Puppet installations, 20+ Puppet masters.
• A single repository with all modules
• Git branches mapped to Puppet environments
• Code review in Gerrit to merge into production branch
• Tests using AWS virtual machines on every code review
4

5. Section name
Implementation
Started with just
switching to the
same workflow
but masterless.
Using a custom Ruby
“wrapper” around
puppet apply.
5
Gerrit
SITE X
SITE X
git mirror git mirror
SITE Y
git mirror git mirror
PuppetDB
Node
Node
Node
● ServerDB
● LDAP
Gerrit replication Gerrit replication
git pull git pull
failover if
pod mirrors
are
unavailable
Queries
* Queries
* Store facts
* Store catalog
* Store report
hiera
secrets
hiera
secrets
HTTPS
GET
auth by
cert
Puppet CA
request
certificate

6. Benefits of this setup
• Debug logging of compilation and hiera lookups
• Ability to get traceback of custom functions directly on the node
• Deprecation warnings from compilation visible on the node
$ sppuppet hiera resolvconf::servers
8.8.8.8
8.8.4.4
6

7. Drawbacks
• Requires more configuration, and can’t use puppet to manage it.
• Quite specialised.
7

8. Secret management
• Small web service where nodes authenticates using their SSL cert
• Calculates the facts for the node from the certificate
• Sends the secret hiera data that applies to that particular node
8

9. PuppetDB access control
• Very binary per default, allow everyone or a whitelist
9
Node Apache PuppetDB
mod_ext_filter
jq

10. Why switch to masterless?
• Scaling the Puppet masters not an issue
• But scaling the workflow is an issue
10

11. Growing amount of committers
11
Number of monthly committers
220
165
110
55
0
2 years ago 1 year ago now

12. Growing amount of modules
12
Number of modules
600
450
300
150
0
2 years ago 1 year ago now

13. More complex dependencies
• 126 modules using the apache module
• 91 modules using the nginx module
• 92 modules using the postgresl module
Backwards incompatible changes to shared modules almost
impossible to do.
13

14. Flexible workflow needed
• Allow one set of modules per Puppet run determined by hiera
• Easier ways to do continuous delivery of both application and
configuration
• Gradual rollout of new module versions
14

15. Why not r10k or librarian-puppet?
Both great utilities that allow building a environment dynamically.
But it is still a fixed environment for many nodes.
production_new_apache
production_new_apache_new_postgres
production_old_apache_new_postgres
production_new_apache_old_postgres
15

16. Our solution to the problem
At each puppet
run we look in
hiera data
which modules
to install.
16
hiera/common.yaml:
modules:
spotify-hostbase: latest
hiera/role/spotify_web.yaml
modules:
spotify-apache: latest
hiera/role/puppetexplorer.yaml
modules:
puppetlabs-apache: latest

17. Internal forge
• A simple forge implementation using the new V3 forge API
• Mirrors the upstream forge
• Will be used for distributing internal modules as well
github.com/jhaals/puppet-anvil
17

18. Questions?