PuppetDB: Higher-order Puppet

deepak@puppetlabs.com
@grim_radical

persistent data long term
data ephemeral data mach
local data meticulously str
aapuppet generates data fr
d lots of data free form da
human readable data mach
data resource data depend
data ssl certificate data ho

ﬁle { “/tmp/foo”:
content => “This is a test”
}

{"exported": false,
"ﬁle": "/puppet/site.pp",
"line": 1,
"parameters": {
"content": "This is a test"
},
"tags": [
"ﬁle",
"node",
“default”
],
"title": "/tmp/foo",
"type": "File"}

File “/tmp/foo/bar”
User “deepak”
Dir “/tmp/foo”
Dir “/tmp”

Dir “/tmp” User “deepak”

Dir “/tmp/foo”

File “/tmp/foo/bar”

resource

catalog

cow

herd

resource

catalog

cow crow

herd murder

resource

catalog

cow superhero crow

herd avengers murder

Group[peadmin]

User[peadmin]

Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin]

Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile]

peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/sha

Relationships

Group[peadmin] Group[puppet-dashboard] Class[Pe_accounts::Data]

User[peadmin] User[puppet-dashboard]

File[/opt/puppet/libexec/mcollective/mcollective/agent] File[/opt/puppet/libexec/mcollective/mcollective/security] Exec[mcollective-server-cert] File[/etc/puppetlabs/mcollective/ssl] Pe_accounts::User[peadmin] File[/var/lib/peadmin] Pe_accounts::Home_dir[/var/lib/peadmin] Pe_accounts::User[puppet-dashboard] File[/opt/puppet/share/puppet-dashboard] Pe_accounts::Home_dir[/opt/puppet/share/puppet-dashboard]

File[/opt/puppet/libexec/mcollective/mcollective/util] File[/opt/puppet/libexec/mcollective/mcollective/application/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration] File[/opt/puppet/libexec/mcollective/mcollective/application/puppetd.rb] File[mcollective-cert.pem] File[mcollective-private.pem] File[mcollective-public.pem] File[/etc/puppetlabs/mcollective/ssl/clients] Exec[mcollective-client-cert] File[/var/lib/peadmin/.mcollective.d] File[/var/lib/peadmin/.mcollective] File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc] File[/var/lib/peadmin/.ssh] File[/var/lib/peadmin/.bash_profile] Exec[puppet-dashboard-client-cert] File[/opt/puppet/share/puppet-dashboard/.mcollective.d] File[/opt/puppet/share/puppet-dashboard/.mcollective] File[/opt/puppet/share/puppet-dashboard/.bashrc.custom] File[/opt/puppet/share/puppet-dashboard/.bashrc] File[/opt/puppet/share/puppet-dashboard/.bash_profile] File[/opt/puppet/share/puppet-dashboard/.vim] File[/opt/puppet/share/puppet-dashbo

/mcollective/mcollective/agent/puppetral.rb] File[/etc/puppetlabs/mcollective/server.cfg] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/package.rb] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetd.ddl] File[/opt/puppet/libexec/mcollective/mcollective/agent/puppetral.ddl] File[/opt/puppet/libexec/mcollective/mcollective/util/actionpolicy.rb] File[/opt/puppet/libexec/mcollective/mcollective/application/service.rb] File[/opt/puppet/libexec/mcollective/mcollective/registration/meta.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/aespe_security.rb] File[/opt/puppet/libexec/mcollective/mcollective/security/sshkey.rb] File[/etc/puppetlabs/mcollective/ssl/clients/mcollective-public.pem] File[peadmin-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-private.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-public.pem] File[puppet-dashboard-public.pem] File[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem] File[/var/lib/peadmin/.ssh/authorized_keys] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-cert.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-public.pem] File[/opt/puppet/share/puppet-dashboard/.mcollective.d/puppet-dashboard-private.pem] File[/opt/puppet/share/puppet-dashboard/.ssh/

Service[mcollective]

Relationships

Group[peadmin]

User[peadmin]

File[/var/lib/peadmin]

File[/var/lib/peadmin/.bashrc.custom] File[/var/lib/peadmin/.vim] File[/var/lib/peadmin/.bashrc]

le[/var/lib/peadmin/.mcollective.d/peadmin-cert.pem]

Catalog:

all the things we
manage on a node,
and how they relate
to each other

netmask_lo: 255.0.0.0 kernel: Linux
augeasversion: 0.10.0 kernelrelease: 2.6.32-5-686
fqdn: pe-debian6.localdomain ipaddress: 172.16.245.128
manufacturer: "VMware, Inc." processor0: Intel(R) Core(TM)
processorcount: "1" i7-2635QM CPU @ 2.00GHz
productname: VMware Virtual lsbdistrelease: 6.0.2
Platform uniqueid: 007f0101
physicalprocessorcount: 1 hardwaremodel: i686
facterversion: 1.6.7 kernelversion: 2.6.32
boardproductname: 440BX operatingsystem: Debian
Desktop Reference Platform architecture: i386
kernelmajversion: "2.6" lsbdistdescription: Debian GNU/
hardwareisa: unknown Linux 6.0.2 (squeeze)
timezone: PDT lsbmajdistrelease: "6"
puppetversion: 2.7.12 (Puppet interfaces: "eth0,lo"
Enterprise 2.5.1) ipaddress_lo: 127.0.0.1
lsbdistcodename: squeeze uptime_days: 0
is_virtual: "true" lsbdistid: Debian
operatingsystemrelease: 6.0.2 rubysitedir: /opt/puppet/lib/
virtual: vmware site_ruby/1.8
type: Other rubyversion: 1.8.7
domain: localdomain osfamily: Debian
hostname: pe-debian6 memorytotal: &id001 502.57 MB
selinux: "false" memorysize: *id001

Catalogs:
what we tell puppet
about a node

Facts:
what a node tells
puppet about itself

It’s about who controls
the information.

“There's a war out there,
old friend. A world war.
And it's not about who's
got the most bullets.
It’s about who controls
the information.

-- Sneakers (1992)
What we see and hear,
how we work, what we
think... it's all about the
information!”

every resource
every parameter
every relationship
every class
every fact
for every node

Query this data, for
use in scripts or
other tools

Integration with
other tools is great,
but can we feed that
data back into
puppet itself?

Configure a node
using resources
from other nodes

class ssh {

@@sshkey { $hostname:
type => dsa,
key => $sshdsakey
}

Sshkey <<| |>>

}

Every host exports
its public key, and
imports the public
keys of every other
node, automatically!

class nagios_target {

@@nagios_host { $fqdn:
ensure => present,
alias => $hostname,
address => $ipaddress,
use => "generic-host",
}

@@nagios_service { "check_ping_${hostname}":
check_command => "check_ping!100.0,20%!
500.0,60%",
use => "generic-service",
host_name => "$fqdn",
notiﬁcation_period => "24x7",
service_description => "${hostname}_check_ping"
}

}

class nagios-monitor {

# collect resources and
# populate /etc/nagios/nagios_*.cfg
Nagios_host <<||>>
Nagios_service <<||>>

}

Thus, you can
automatically create
checks for things
you’re managing

key distribution
monitoring
clustered services
master/slave replication
load balancers
shared filesystems
firewall rules
...

Using Puppet’s
knowledge to
improve Puppet’s
knowledge

Using Puppet’s
knowledge to
improve Puppet’s
knowledge
Achievement unlocked
YO DAWG

Reading from the Puppet Data Library
Nick Lewis
3:50P @ Meeting Room 2

Why aren’t we doing
stuff like this all the
damn time?

Every node,
on every puppet run,
generates data

We have customers
generating over
750GB of data a day!
even storing a small subset of
that much information adds up...

When data storage is
slow, the whole
system slows down
and it makes baby Deepak cry! :(

Current APIs are
limited!
Hard to get at the data, and
performance concerns discourage
use

We demand:

Store as much data as we can!
Much better queryability!

Oh yeah, but:

Don’t slow down the system!
Don’t compromise reliability!

PuppetDB
Definitely Better!

Fast storage
of catalogs & facts
like, *way* faster!

Compatible
with storeconfigs and
inventory service
you don’t have to change
your Puppet code!

HTTP APIs
for resource, fact, and
node retrieval
plenty of data, just
a “curl” away!

Secured
using SSL client and
server certificates
the same certificate infrastructure
you’re already using!

science
&
secret alien
technology

PuppetDB Server DLO

DB Workers

HTTP MQ

Agent Master
Facts Catalo Resrc
g

PuppetDB Server DLO

DB Workers

HTTP MQ

Agent Master
Facts Catalo Resrc
F g

PuppetDB Server DLO

DB Workers

HTTP MQ

Agent Master
Facts Catalo Resrc
g
F

PuppetDB Server DLO

DB Workers

HTTP MQ
F

Agent Master
Facts Catalo Resrc
g
F

PuppetDB Server DLO

DB Workers
F

HTTP MQ

Agent Master
Facts Catalo Resrc
g

PuppetDB Server DLO

DB Workers

HTTP MQ
F

Agent Master
Facts Catalo Resrc
g
F ?

PuppetDB Server DLO

DB Workers

HTTP MQ
? F

Agent Master
Facts Catalo Resrc
F g

PuppetDB Server DLO

DB Workers
? F

HTTP MQ

Agent Master
Facts Catalo Resrc
F g

PuppetDB Server DLO

DB Workers
F

HTTP MQ
?

Agent Master
Facts Catalo Resrc
F g

PuppetDB Server DLO

DB Workers
F

HTTP MQ

Agent Master
Facts Catalo Resrc
F g
?

PuppetDB Server DLO

DB Workers
F

HTTP MQ

Agent Master
Facts Catalo Resrc
F g

PuppetDB Server DLO

DB Workers

HTTP MQ
F

Agent Master
Facts Catalo Resrc
g

PuppetDB Server DLO
F

DB Workers

HTTP MQ

Agent Master
Facts Catalo Resrc
g

PuppetDB Server DLO

DB Workers

HTTP MQ

PuppetDB Server
Workers DLO
DB

HTTP MQ

PuppetDB Server
Workers DLO
HTTP DB
Proxy
(SSL)
HTTP MQ

We work very hard to
persist everything we
accept

Acknowledgements with UUIDS,
Checksums,
Queueing,
Automatic retry,
Automatic reconnect,
and the Dead Letter Office if all else fails!

Anything Puppet
does with PuppetDB,
you can do, too
we don’t cheat!

Query your own resources,
Upload new fact sets,
Create catalogs,
Inspect facts,

all open and documented!

#> curl
-H "Accept: application/json"
"http://puppetdb/metrics/mbean/
com.puppetlabs.puppetdb.command:type=global,name=processing-time"

{
"50thPercentile": 209.05,
"95thPercentile": 428.3065999999959,
"999thPercentile": 1246.722744999993,
"99thPercentile": 818.9180600000001,
"Count": 3322,
"EventType": "calls",
"FifteenMinuteRate": 1.1500295609205015e-06,
"FiveMinuteRate": 1.387569444096042e-18,
"LatencyUnit": "MILLISECONDS",
"Max": 26514.032,
"Mean": 314.1111032510536,
"MeanRate": 0.21577717049577358,
"Min": 185.53,
"OneMinuteRate": 3.390107448865515e-90,
"RateUnit": "SECONDS",
"StdDev": 833.6079354075728
}

#> curl
"http://puppetdb/metrics/mbean/
com.puppetlabs.puppetdb.command:type=global,name=processing-time"

{
"95thPercentile": 428.3065999999959,
"999thPercentile": 1246.722744999993,
"99thPercentile": 818.9180600000001,
"Count": 3322,
"EventType": "calls",
"FifteenMinuteRate": 1.1500295609205015e-06,
"FiveMinuteRate": 1.387569444096042e-18,
"LatencyUnit": "MILLISECONDS",
"Max": 26514.032,
"Mean": 314.1111032510536,
"MeanRate": 0.21577717049577358,
"Min": 185.53,
"OneMinuteRate": 3.390107448865515e-90,
"RateUnit": "SECONDS",
"StdDev": 833.6079354075728
}
WALL OF TEXT

curl
"http://puppetdb/facts/host.my.net"

curl
"http://puppetdb/resources?query=..."

https://github.com/dalen/
puppet-puppetdbquery

Ships with a real-time dashboard,
Dozens of metrics and gauges,
Correlate-able logs,
Easy to monitor

we care about operational visibility!

https://github.com/
jasonhancock/nagios-puppetdb

We’ve seen huge reductions in compile times,
resource collection times, time to persist
catalogs and facts, etc.

O_o

ONE DOES NOT SIMPLY

SPEED UP PUPPET

Posit:
Hosts are not entirely
unique snowflakes

Therefore:
A resource often
exists across multiple
hosts

Feature:
Single-instance
resource storage

Resource dedupe
Compute unique hashes for resources

We quickly hash all the resources in a catalog,
and use bulk operations to compare them to
hashes stored.

Resource dedupe
Significant speed improvement!

Internal to Puppet Labs, we see ~83% resource
duplication; this number is consistent with what
we’ve seen in most customer environments.

Posit:
Puppet runs frequently,
but catalogs change
infrequently

Therefore:
We’ll often receive the
same catalog for a
host

Feature:
Single-instance
catalog storage

Catalog dedupe
Compute unique hashes for catalogs

Puppet Labs sees ~88% catalog duplication, rest
of the planet sees even bigger numbers

Big savings!

Posit:
You have more than one
core, though storeconfigs
is single-threaded

Therefore:
Throughput is not
maximized

Feature:
Massively parallel
operation

Parallel
We can pat our heads and rub our tummies at
the same time

Database operations don’t block MQ operations
don’t block HTTP operations don’t block hash
computation operations don’t block metric
calculations don’t block...

Dozens of threads, zero locks

Documented at
http://
docs.puppetlabs.com
/puppetdb
install, config, upkeep, specs,
the works!

Packaged
as deb and rpm for
both open source and
Puppet Enterprise
available in the Puppet Labs
package repositories

Puppetized
using the
puppetlabs/puppetdb
module
available now, on the
Module Forge!

> puppet module install
puppetlabs/puppetdb
> vim site.pp
node puppetmaster {
include puppetdb
include puppetdb::master::config
}

Open source

http://github.com/
puppetlabs/puppetdb
same license as Puppet itself!

Many production
deployments
Small shops with a dozen hosts,
large shops with thousands of
hosts, intercontinental
deployments...

over a billion resources served!

Report storage
Historical data
Grand Unified Query
and of course, keep it fast!

Use it!

and send us more
dashboard screenshots! :)

deepak
giridharagopal
deepak@puppetlabs.com
@grim_radical [github twitter freenode]

7 5 3

11 8

2 9
Let’s get TOPOLOGICAL! 10

PuppetDB: Higher-order Puppet

Recomendados

Recomendados

Mais conteúdo relacionado

Último

Último (20)

Destaque

Destaque (20)

PuppetDB: Higher-order Puppet

Notas do Editor