Here are the slides from Thomas Uphill's presentation called Puppet Troubleshooting. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa
4. When you have eliminated all which is impossible,
then whatever remains, however improbable, must
be the truth
Sir Arthur Conan Doyle
4
5. https://goo.gl/8LyZzN
Troubleshooting 101
● document current state
● discover recent changes
○ audit everything, even things you don't touch
○ never assume it's magic
● change one thing at a time
○ if it doesn't fix the problem, revert it
5
1 une uno um unis odin jeden jedward
yksi 一 एक אחד واﺣد
37. https://goo.gl/8LyZzN
Modulus
n = pq
OpenSSL
# openssl rsa -noout -modulus -in ca_key.pem |sha256sum
69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38
# openssl x509 -noout -modulus -in ca_crt.pem |sha256sum
69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38
BIG *RSE PRIME
37
38. https://goo.gl/8LyZzN
Modulus
n = pq
OpenSSL
# openssl rsa -noout -modulus -in ca_key.pem |sha256sum
69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38
# openssl x509 -noout -modulus -in ca_crt.pem |sha256sum
69578e29c08c130d37c7c0141134f1cc4778445c7b7d1d96d253b86d6bf4ca38
BIG *RSE PRIME
$ puppet agent -t
Error: Could not request certificate: The certificate retrieved from the master
does not match the agent's private key.
Certificate fingerprint:
D4:D3:76:F1:6B:51:83:3C:4B:72:69:BF:BC:B0:80:94:79:75:1A:3B:D8:29:F5:EF:81:2C:44:3
5:21:93:CE:FD
To fix this, remove the certificate from both the master and the agent and then
start a puppet run, which will automatically regenerate a certificate.
On the master:
puppet cert clean cottage
On the agent:
1a. On most platforms: find /home/thomas/.puppetlabs/etc/puppet/ssl -name
cottage.pem -delete
1b. On Windows: del "homethomas.puppetlabsetcpuppetsslcertscottage.pem"
/f
2. puppet agent -t
38
The certificate retrieved from the master does not match the agent's
private key.
39. https://goo.gl/8LyZzN
OpenSSL recap
● x509
view certificate
check expiration
check serial number
● crl
revoked cert serial#
● verify
verify cert with CA and/or CRL
● modulus
how the cert was encrypted
39
47. catalog
fails to compile
duplicate resource
modulepath/bad module
name
fails to apply
unpredictable exec
bad/broken service
bad/missing variable
47
55. https://goo.gl/8LyZzN
Bad/Missing Variable
$one = "1"
file {"pcone":
path => "/tmp/pc$one",
ensure => 'directory',
}
file {"pc1":
path => "/tmp/pc1",
ensure => 'file',
}
Info: Caching catalog for puppet.example.com
Error: Evaluation Error: Error while evaluating a Resource
Statement, Cannot alias File[pc1] to ["/tmp/pc1"] at
/root/pc.pp:6; resource ["File", "/tmp/pc1"] already
declared at /root/pc.pp:2 at /root/pc.pp:6:3 on node
puppet.example.com
55
56. https://goo.gl/8LyZzN
Bad/Missing Variable
$PC = 'puppetconf'
pc {'one':
place => "/tmp/$PC",
type => "directory",
}
pc {'two':
place => "/tmp/$PC",
type => "file",
}
define pc (
String $place,
String $type,
) {
file {"$title":
path => $place,
ensure => $type,
}
}
Info: Caching catalog for puppet.example.com
Error: Evaluation Error: Error while evaluating a Resource
Statement, Evaluation Error: Error while evaluating a
Resource Statement, Cannot alias File[two] to
["/tmp/puppetconf"] at /root/define.pp:5; resource ["File",
"/tmp/puppetconf"] already declared at /root/define.pp:5 at
/root/define.pp:5:2 at /root/define.pp:15 on node
puppet.example.com
56
60. https://goo.gl/8LyZzN
Debug Script… just an example
#!/bin/bash
LOG=$(mktemp /tmp/puppet-debug.XXXXXX)
echo Puppet Debug -- $@ -- $(date) | tee $LOG
echo "-- Disk --" | tee -a $LOG
df -h |tee -a $LOG
df -i |tee -a $LOG
echo "-- Mem --" | tee -a $LOG
free | tee -a $LOG
echo "-- Files --" | tee -a $LOG
PUPPET=$(pgrep puppet)
for proc in $PUPPET
do
lsof -p $proc |tee -a $LOG
done
Puppet Debug -- before resolv.conf -- Fri Oct 24 01:13:34 EDT 2014
-- Disk --
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
6.7G 2.5G 3.9G 39% /
tmpfs 246M 0 246M 0% /dev/shm
/dev/vda1 485M 80M 380M 18% /boot
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/VolGroup-lv_root
440640 79253 361387 18% /
tmpfs 62783 1 62782 1% /dev/shm
/dev/vda1 128016 50 127966 1% /boot
-- Mem --
total used free shared buffers cached
Mem: 502268 415488 86780 0 22176 172036
-/+ buffers/cache: 221276 280992
Swap: 835580 0 835580
-- Files --
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
puppet 2058 root cwd DIR 253,0 4096 14 /root
puppet 2058 root rtd DIR 253,0 4096 2 /
puppet 2058 root txt REG 253,0 10600 36617 /usr/bin/ruby
puppet 2058 root mem REG 253,0 156928 4134 /lib64/ld-2.12.so
puppet 2058 root mem REG 253,0 1926680 6282 /lib64/libc-2.12.so
60
61. https://goo.gl/8LyZzN
Printing - Template
- scope.to_hash
- reject a few
- sort
- print, one per line
file { "/tmp/puppet-debug.txt":
content => inline_template("<% vars =
scope.to_hash.reject { |k,v| !( k.is_a?(String) &&
v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%=
v %>n<% end %>"),
}
vars = scope.to_hash.reject
{ |k,v| !( k.is_a?(String) &&
v.is_a?(String) ) };
vars.sort.each do |k,v|
k=vn
end
61
69. https://goo.gl/8LyZzN
pry demo
#
69
#
…
From:
/etc/puppetlabs/code/environments/production/modules/pry
/lib/puppet/parser/functions/pry.rb @ line 4
#<Module:0xfb588d1>#real_function_pry:
2: newfunction(:pry) do |args|
3: require 'pry'
=> 4: binding.pry
5: end
[1] pry(#<Puppet::Parser::Scope>)>
puppet agent -tpuppetserver foreground
…
Puppet Server has successfully started and is now ready
to handle requests
exit
…
Puppet Compiled Catalog for xxx.example.com in y.z
seconds
Info: Caching catalog for xxx.example.com
Info: Applying configuration version 'XXX'
70. https://goo.gl/8LyZzN
where to go for help
● IRC #puppet / #puppet-dev
● slack puppetcommunity.slack.com
#pug
#puppet
● google group / mail list
https://groups.google.com/forum/#!forum/puppet-users
● PUG
https://www.meetup.com/Seattle-Puppet-Meetup/
70
71. https://goo.gl/8LyZzN
Summary
Puppet is an HTTPS service
End-to-end (gethostbyname, nc mtr)
OpenSSL is your friend (x509,crl, verify, s_client
make a debug class
remember scope
basic UNIX permissions
71