Scalable Cloud-Native Masterless Puppet, with PuppetDB and Bolt, Craig Watson, ForgeRock

Copyright © 2020 ForgeRock. All rights reserved
Craig Watson
Senior Systems Engineer - ForgeRock IT
Virtual Puppet Camp Germany - 5th May, 2020
Scalable Cloud-Native Masterless
Puppet, with PuppetDB and Bolt

Who Am I?
Senior Systems Engineer, ForgeRock IT - Bristol, UK
Puppet user since 2011, community member since 2012
AWS: 2013, Google Cloud: 2017
Background: Systems Engineering, Public Cloud consultancy and systems design
Dad, heavy metal, Liverpool FC and Doctor Who fan
AWS Certified SysOps Associate & DevOps Professional
Puppet Certified Professional (2016 & 2017)
Presenter at Puppetize PDX 2019
2

Copyright © 2020 ForgeRock. All rights reservedCopyright © 2020 ForgeRock. All rights reserved
A Little History
3

Master of Puppets - Somewhere Back in Time
4
Over time, Puppet Masters become monoliths
Servers are “long-lived cattle”
Lift-and-shift cloud migrations become problematic
Hybrid infrastructure?
Use on-premise masters for cloud?
Solutions exist (auto-signing, compile-masters)
Most of the time, results in a compromise!
Scalability and manageability most often sacrificed

Cloud-Native, Scalable Puppet
7

Masterless/Agentless Puppet - Summary
8
Puppet runs locally via puppet apply
Puppet codebase distributed to every node
Exact mechanism can vary (RPM/DEB, tar-ball, Git …)
Decentralised - no/few outside dependencies
Packages can be downloaded from object storage (S3/GCS)
Scalable - no single point of failure for new nodes
Bootstrap/user-data scripts take care of all provisioning
Testable - Allows easy development via Vagrant
Everything is local!
First step to immutable infrastructure
As Puppet runs locally, images can be taken post-run

Secrets Management
Secrets are encrypted at-rest in Git with EYAML and SaaS KMS
AWS - https://github.com/adenot/hiera-eyaml-kms
GCP - https://github.com/craigwatson/hiera-eyaml-gkms
We wrote a helper script to interface with KMS
9
---
profiles::confluence::db_password: ENC[GKMS,CiQAPPX7KHnvqMjmxXUsaIJZil55rm1oBbs=]
/etc/puppetlabs/code/data/env/prod/confluence.yaml
$ ./eyaml.sh -e prod -a encrypt -v correcthorsebatterystaple
ENC[GKMS,CiQAPPX7KHnvqMjmxXUsaIJZil55rm1oBbs=]

Hiera and Instance Metadata
Scripts can enumerate metadata and store as static facts for cross-cloud portability
10
for DATA in $(curl http://169.254.169.254/computeMetadata/v1/instance/attributes/); do
KEY=$(echo "${DATA}" | sed 's/-/_/g')
VALUE=$(curl "http://169.254.169.254/computeMetadata/v1/instance/attributes/${DATA}")
echo "${KEY}=${VALUE}" >> /etc/facter/facts.d/metadata.txt
done
resource "aws_instance" "pdb" {
instance_type = "c3.xlarge"
availability_zone = "eu-west2a"
...
tags = {
Name = "puppetdb"
role = "puppetdb"
}
}
resource "google_compute_instance" "pdb" {
name = "puppetdb"
machine_type = "n1-standard-2"
region = "europe-west1"
...
metadata = {
role = "puppetdb"
}
}

PuppetDB
11

PuppetDB Overview
Central “node database” for Puppet
Puppet sends facts, catalog and report for each run
Data exposed via Puppetboard UI - thanks to Vox Pupuli!
App - https://github.com/voxpupuli/puppetboard
Puppet module - https://github.com/voxpupuli/puppet-puppetboard
Deployed standalone as a standard “three-tiered” web-application
Puppet module - https://forge.puppet.com/puppetlabs/puppetdb
Two PuppetDB servers, behind and SSL-terminating load balancer
We use Google CloudSQL to provide a SaaS PostgreSQL database
12

PuppetDB Installation
Add classes to role via Hiera
Configure
puppetdb::server::disable_ssl: true
puppetdb::server::gc_interval: 1
puppetdb::server::node_ttl: '32m'
puppetdb::server::node_purge_ttl: '1s'
profiles::nginx_proxy::upstream_port: 8080
13
---
classes:
- puppetdb::server
- profiles::cloud_sql_proxy
- profiles::nginx_proxy

Sending Node Data to PuppetDB (1)
Install puppetdb-termini package
Configure Puppet’s routes.yaml (YMMV at this point!)
14
---
apply:
catalog:
terminus: compiler
cache: puppetdb
resource:
terminus: ral
cache: puppetdb
facts:
terminus: facter
cache: puppetdb_apply
/etc/puppetlabs/puppet/routes.yaml

Sending Node Data to PuppetDB (2)
Configure Puppet
15
[main]
server_urls = https://puppetdb.example.com:443
soft_write_failure = true
verify_client_certificate = false
/etc/puppetlabs/puppet/puppetdb.conf
[main]
report = true
reports = puppetdb
localcacert = /etc/pki/tls/certs/ca-bundle.crt
certificate_revocation = false
/etc/puppetlabs/puppet/puppet.conf

Bolt
16

Bolt Overview
Executes tasks over SSH, can use PuppetDB for inventory
Handles rich scripts/plans in Puppet SDL, and also allows arbitrary CLI commands
We use bolt command run to:
Update Puppet code via yum (we package our codebase as an RPM and host on GCS)
Run Puppet via puppet apply
We deploy a bolt user on each host, and use Jenkins as our Bolt “control node”
17

Connecting Bolt to PuppetDB
We use Jenkins as a Bolt control node
18
/var/lib/jenkins/.puppetlabs/bolt/bolt.yaml
---
modulepath: '/etc/puppetlabs/code/modules'
ssh:
host-key-check: false
run-as: root
user: bolt
puppetdb:
server_urls: ["https://puppetdb.example.com:443"]
cacert: /etc/pki/tls/certs/ca-bundle.crt

Bolt PuppetDB Inventory Template
Within the wrapper script, an “inventory template” file is copied to /tmp, edited via sed
and passed to bolt
19
version: 2
groups:
- name: dynamic
targets:
- _plugin: puppetdb
query: "inventory[certname] {PQL_QUERY_PLACEHOLDER}"
target_mapping:
name: facts.networking.fqdn
uri: facts.networking.ip
cp /path/to/template.yaml /tmp/inventory.yaml
sed -i "s/PQL_QUERY_PLACEHOLDER/facts.role = 'confluence'/" /tmp/inventory.yaml
bolt command run "hostname" --inventory /tmp/inventory.yaml --targets dynamic

Build Pipeline & Summary
20

Full Orchestration Pipeline
21
Install Modules
librarian-puppet
Build RPM
fpm
Download Repo
gsutil rsync
Add Package
createrepo
Upload Repo
gsutil rsync
Run PQL Query
bolt-wrapper.sh
Return Nodes
PuppetDB
SSH to each node
Bolt
Run Command
Bolt
Install Puppet Code to target instance / Run Puppet
Build Package

Final Thoughts
Masterless Puppet allows us to scale our Puppet deployment with little overhead
Secrets are encrypted at-rest with per-environment KMS keys, decrypted via EYAML
Our nodes send facts, catalogs and reports to PuppetDB
PuppetDB is deployed as a standard three-tier web-application with LB and SaaS DB
As part of our deployment pipeline, Bolt queries PuppetDB for inventory
Bolt then connects to each node via SSH and runs the required commands
22

Thank You!
craigwatson1987
craigwatson

Scalable Cloud-Native Masterless Puppet, with PuppetDB and Bolt, Craig Watson, ForgeRock

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Scalable Cloud-Native Masterless Puppet, with PuppetDB and Bolt, Craig Watson, ForgeRock

Similar to Scalable Cloud-Native Masterless Puppet, with PuppetDB and Bolt, Craig Watson, ForgeRock (20)

More from Puppet

More from Puppet (20)

Recently uploaded

Recently uploaded (20)

Scalable Cloud-Native Masterless Puppet, with PuppetDB and Bolt, Craig Watson, ForgeRock