This session will be an overview of highly available components that can be deployed with Puppet Enterprise. It will focus on some of the current Beta support in PuppetDB as well as tips and tricks from the professional services department. The session will cover field solutions ( both supported and unsupported ) that allow architectures to be designed that align with different levels of high availability across the services that support running puppet on agent nodes during an outage of your primary puppet infrastructure.

1. High Availability
for Puppet
Russ Mull - @mullr
Senior Software Engineer
Zack Smith - @acidprime
Principal Professional Services Engineer

2. Puppet Services related to High Availability
The important bits of Puppet to make highly available

3. Enterprise Readiness: High Availability
Auto scaling
Active/ Active
Robust Backup and
Restore tooling
Disaster recovery
in Multi datacenter/
Geo diverse
environments
Eliminate Single
Points of Failure
Building capabilities that matter
Puppet
Runs continue

4. High Availability for Puppet - Puppetconf 2016
Building a new catalog
Classification, Exported Records, Hiera Data and puppet code being synced from version control
4
run
Puppet Code
Classifier

5. High Availability for Puppet - Puppetconf 2016
Two of Everything!
don’t forget about your external services like git, ldap etc
5
Cl
as
sifi
er
Pup
pet
Cod
e
Classifier
Pup
pet
Cod
e Classifier
Balancer
“Load”
check
check backup

6. Lets start with the basics

7. Certificate Authority Files
Puppets SSL implementation
7

8. High Availability for Puppet - Puppetconf 2016
Puppet CA Replication Components
CA private key and cert
Signed Directory
Serial file
Certificate Revocation List (CRL)
8
serial
3E8
crl.pem
signed
101
ca_crt

9. High Availability for Puppet - Puppetconf 2016
CA Private key
ca
If you don’t care about revocation (security/revocation) this is the only file needed to replicate
9
ca_key.pem
ssl
ca_crt.pem

10. High Availability for Puppet - Puppetconf 2016
Signed Directory
signed
Used when checking for duplicate CN ( certs with the same name)
10
host1.company.com.pem
ca

11. High Availability for Puppet - Puppetconf 2016
Serial file
Tracking the next numeric serial to be issued to new agent
12
serial
ca
3E8
decimal: 1000
decimal: 1001
3E9

12. High Availability for Puppet - Puppetconf 2016
Certificate Revocation List
Tracking revoked certificates
13
crl.pem
ca
decimal: 1000
decimal: 1001
1000
1001
serial
3E9

13. High Availability for Puppet - Puppetconf 2016
Simply copy your ssldir ahead of the second installation
14
$ssldir
scp -r
$ssldir
installer
CA
When using old versions of PE delete the pe-internal* certs post transfer , pre install
DR site
1.
2.

14. PostgreSQL Replication
Database level synchronization
17

15. High Availability for Puppet - Puppetconf 2016
Streaming Replication
This happens as the postgres database layer
18
PostgreSQL PostgreSQL
PuppetDBPDB PuppetDBPDB
22
Read (Standby)Write
5432
write ahead logs
R
W1
2
16MB

16. High Availability for Puppet - Puppetconf 2016
Split Reads and writes
Can survive temporary failures of the write master
19
PostgreSQL PostgreSQL
PuppetDBPDB
Read (Standby)Write
5432
W R
WWW
queue

17. High Availability for Puppet - Puppetconf 2016
Promote Standby to Writable
This happens as the postgres database layer
20
PostgreSQL
Write
Read (standby)
PostgreSQL
PostgreSQL
Write
5432
PostgreSQL
Read (standby)
5432

18. Multi master PuppetDB Beta
Puppet Enterprise Only
21

19. High Availability for Puppet - Puppetconf 2016
Master side Failover
This is know as “terminus” failover as its handled in the puppetdb terminus package code
22
PuppetDBPDB
[main]
server_urls = https://primary:8081, https://replica:8081
Primary
PuppetDBPDB
Replica1 2

20. High Availability for Puppet - Puppetconf 2016
Command Broadcast
command_broadcast = true in puppetdb.conf
23
PuppetDBPDB
PuppetDBPDB

21. High Availability for Puppet - Puppetconf 2016
PuppetDB Replication Reconciliation
Reconciliation happens on an interval
24
PostgreSQL
8081
Write
PostgreSQL
PuppetDBPDB
Write
PuppetDBPDB
Sync Interval

22. Puppet Enterprise HA
Coming soon…
25

23. High Availability for Puppet - Puppetconf 2016
Simple HA
Monolithic master + Replica
26
P Primary R Replica

24. High Availability for Puppet - Puppetconf 2016
Large Environment Installation
Monolithic master + Compile masters + Replica
27
R Replica
Balancer
Load
P Primary
Balancer
Load

25. High Availability for Puppet - Puppetconf 2016
New: Agent Side Failover!
Shipping in Puppet 4.6 and higher, PE 2016.4+
28
Primary Replica
1 2

26. High Availability for Puppet - Puppetconf 2016
Use Cases
● Puppet runs keep working
● Promote replica to master
29
When master is unreachable

27. High Availability for Puppet - Puppetconf 2016
When the master is unreachable
● Run puppet
● Promote replica to master
30
You can:
You can’t
● Change classification
● Deploy new puppet code
● Issue new certs
● Use the Puppet Enterprise Console
● Use Application Orchestrator

28. Provisioning Replica
(monitoring replication)
31

29. High Availability for Puppet - Puppetconf 2016
Command Line Interface
32
puppet infra provision replica <hostname>
replica.mycorp.net
puppet infra enable replica
replica.mycorp.net
puppet infra status
. . .

30. High Availability for Puppet - Puppetconf 2016
1. Provision Replica
33
puppet infra provision replica <hostname>
replica.mycorp.net

31. High Availability for Puppet - Puppetconf 2016
2. Monitor status of replication
34
puppet infra status
> Per-service ‘alerts’
> Visible in the UI as well

32. High Availability for Puppet - Puppetconf 2016
3. Enable replica
35
puppet infra enable replica
replica.mycorp.net

33. Replica Services
Services on a provisioned replica
36

34. High Availability for Puppet - Puppetconf 2016
What’s a replica?
● Compile Master
● PuppetDB (r/w)
● RBAC, classifier, activity (r/o)
● Orchestrator data (not running)
● CA data (r/o using a proxy)
37
R Replica
PuppetDBPDB
Console

35. File Sync Replication
Replicate Code Directory and Certificate Authority Data
38

36. High Availability for Puppet - Puppetconf 2016
File Sync - Compile Masters
M
Master of Masters
MOM
Compile Master
COMC
Compile Master
COMC
Compile Master
COMC

37. High Availability for Puppet - Puppetconf 2016
File Sync - CA Replica data
Primary Master Replica Master
R ReplicaP Primary
8140
ssl ssl

38. High Availability for Puppet - Puppetconf 2016
Puppet Enterprise CA Proxy
Primary Master Replica Master
R ReplicaP Primary
ssl
CSR

39. Database Replication
Replicate the data used in your PE deployment
42

40. High Availability for Puppet - Puppetconf 2016
PGLogical Replication
PostgreSQL
RBAC
NC
Classifier
5432
5432
PostgreSQL
RBAC
NC
Classifier
Write Read (Standby)
PuppetDB
PDB
PuppetDB
PDBNot synced

41. High Availability for Puppet - Puppetconf 2016
PE HA - Replication
PuppetDB
PuppetDBPDB PuppetDBPDB
PostgreSQL
PostgreSQL
PGlogical
PGlogical
FileSync
Primary Replica
RBAC
NC
Classifier
W
W
RBAC
NC
Classifier
R
R

42. High Availability for Puppet - Puppetconf 2016
Puppet Enterprise HA - Beta Signup
Interested in what you heard?
Please signup for our HA beta program through the Puppet Enterprise Support portal
45

43. High Availability for Puppet - Puppetconf 2016
46
https://goo.gl/Z85HLS
PE HA Beta Signup
Support
Knowledge base
Z 8 5 H L S