Puppet Camp LA 2015 talk covering: packages, package managers, puppet, and tips, tricks, and puppet modules for setting up secure package repositories.
3. hi, I’m joe!
• i think these things are cool:
• computer programs
• reproducible builds / infrastructure
• automation
• configuration management
• tahdig*
* an rice food
4. packagecloud.io
• I work on packagecloud.io
• packagecloud makes it easy to upload,
download, store, and delete software packages
• you should use it, it’s cool.
7. Why?
• Central to maintaining, building, and testing
infrastructure.
• Packages are a primitive in Puppet.
• Understanding where packages come from, and how
to store them properly is a requirement for
infrastructure of any size.
• Packages and packaging are much trickier than they
seem!
8. Overview
• what is a package?
• what is a package manager?
• ./configure && make && make install pattern
• open source tools for package repositories
• HOWTO manage repos in your infra with puppet
9. What is a package?
Beck Gusler, https://flic.kr/p/4A15jm
10. What is a package?
• A package generally consists of:
• metadata (version, architecture, deps, etc)
• files to be written to the filesystem (/usr/sbin/
nginx, etc)
12. Common package types
• RPM packages
• Used on CentOS, RHEL, Scientific Linux, Fedora, …
• files typically have the “.rpm” file extension
• can be inspected, installed, and removed with rpm
• are actually a:
• header structure (binary data)
• CPIO archive
15. Common package types
• Deb packages:
• Used on Ubuntu, Debian, Knoppix, …
• files typically have the “.deb” file extension
• can be inspected, installed, and removed with
dpkg
16. Common package types
• Deb packages:
• are actually an AR archive with:
• version file: the debian format version
• data.tar.gz: the actual files to write to the filesystem
• control.tar.gz: the package metadata
• Can be GPG signed, but signatures are never checked!
18. Common package types
• There are lots more! (ruby gems, npm, java,
python, …)
• Some packaging systems also have source
packages.
19. What is a source package?
• A source package consists of:
• metadata (version, architecture(s), build deps,
etc).
• source files (C source, C++ source, py scripts,
etc).
• Allows you to rebuild a binary package easily.
20. Install packages with puppet
Use the resource type ‘package’ to install packages:
package { 'pygpgme':
ensure => latest,
}
21. Install packages with puppet
package { 'pygpgme':
ensure => ‘0.3-11’,
}
Specify the version you want by setting ensure:
22. Summary
• Packages are a collection of files with metadata.
• The metadata usually has info like:
• architecture
• version
• dependency info
• and more.
• Installation is easy if you don’t have dependencies.
24. Dependencies
• Installing 1 package is as easy as:
• dpkg -i filename.deb
• rpm -ivh filename.rpm
• Of course, you should use puppet instead :D
• But what if your program needs other programs?
• For example: nginx depends on libssl, zlib, …
27. Package manager
• A package manager is a collection of software
that allows you to:
• install, upgrade, remove packages
• query package info from local system or repos
• Some tools include more advanced features like
mirroring or more advanced caching features.
29. • yum (Yellowdog Updater, Modified)
• Common on RHEL, CentOS, Fedora, …
• Used for installing, removing, configuring, and
querying RPM packages and dependencies.
Common package managers
31. Common package managers
• APT (Advanced Package Tool)
• Common on Debian, Ubuntu, KNOPPIX, …
• Used for installing, removing, configuring, and
querying Debian packages and dependencies.
32. Install packages with puppet
• When you install packages with puppet, puppet
will automatically detect which package
manager to use.
• You won’t need to worry about which command
to run, or what options to pass; puppet will take
care of that for you!
33. Summary
• package managers help you install software and
associated dependencies
• easily remove, upgrade, and query packages
• Puppet will automatically detect the system’s
package manager when you install a package.
35. A problem
• You run Ubuntu 10.04 LTS
• You want to install redis
• Ubuntu 10.04 comes with redis-server 1.2.0-1
• That’s too old! You need 2.8.19!
• So, now what?
36. Common (not great) solution
• A common solution to this sort of problem is
building redis (or ruby, or …) from source in your
puppet manifest
• Like this….
38. Why?
• It’s easy!
• ./configure && make && make install
• It works!
• I’m using puppet so it’s reproducible!
39. But…
• What happens if you need to:
• completely remove Redis?
• install a security update?
• install a new version?
• install the same exact Redis on 200 machines?
40. The not-so great side
• Not all Makefiles have uninstall targets, so you
have to remove files manually
• Leaving artifacts on the filesystem can cause
really, really hard to debug problems later
• If the build process changes version to version,
it can be painful to rollback
41. The not-so great side
• Rebuilding the same source does not necessarily
get you the same byte-for-byte binary
• If the binaries aren’t identical, you can end up
with bugs in some of the compiled binaries but
not others
• Painful to recreate source builds inside of puppet
• Makes writing tests for manifests painful
42. Make a package
• Install the same binary on every machine
• When the package is removed, all installed files are
removed
• Versioning of build process built in (with most tools)
• Keep your puppet manifests about config
management
• Your build steps are “factored out” into the package
43. Your new puppet manifest
package { 'redis':
ensure => latest,
}
44. Your package
• Your build steps get encapsulated in the package
itself
• Makes iterating on the build more straight forward
• Don’t need to apply (potentially) a bunch of
manifests to a machine every time you do a build
51. Tradeoffs
• Once you learn how to make packages you can
build reproducible infrastructure much more
easily
• You can use your prod environment in dev and
test
• You can more easily build tests for your
infrastructure with beaker/kitchen.ci
54. Package repositories
• Major linux distributions keep repositories of
packages for users:
• EPEL
• Ubuntu / Debian official repositories
• You can store a package and its dependencies to
make it easy to install them all on your infrastructure
56. Package repositories
• createrepo: creates yum repositories
• reprepro: creates apt repositories
• Many other free tools available!
• Read the documentation carefully. Lots of tricky
options.
• I’ll show some examples to get you started!
59. GPG is important
• Using GPG to sign the generated repository
guarantees that you generated the repository.
• This is important.
• This means that no one else modified, removed, or
inserted a package other than you.
• GPG signing the repository is not a very well known
security measure, but it is incredibly important!
• This is NOT the same as using rpmsign/rpm --sign.
60. Secure YUM repos
• Sign repository metadata with GPG
• Sign packages with GPG (use rpmsign)
• Serve repositories over SSL
• Enable all the right options for SSL verification,
repository GPG checking, AND package GPG
checking.
61. Wouldn’t it be cool to do all
that with Puppet instead?
Good news: you can!
64. You still need to GPG sign
the repository yourself :(
exec { “gpg_sign_yumrepo”:
command => “gpg --detach-sign --armor
/var/yumrepos/yumrepo/repodata/repomd.xml“,
}
66. Add YUM repos with puppet
yumrepo { 'my_repo':
baseurl => "http://myurl.com/repo",
gpgcheck => 1,
repo_gpgcheck => 1,
gpgkey => “http://myurl.com/gpg.pub.key”,
sslverify => 1,
sslcacert => “/etc/pki/tls/certs/ca-bundle.crt”,
enabled => 1,
}
most people never turn on repo_gpgcheck or
sslverify, or set the ssl certificate path, but you
should!!
67. But that’s not all!
• You MUST have the ‘pygpgme’ package
installed on the system that will verify the
signatures.
• Without pygpgme, yum will not be able to verify
signatures!
• Some versions of CentOS / RHEL do not
automatically install pygpgme with yum!!
68. Make sure to install pygpgme
package { 'pygpgme':
ensure => latest,
}
72. reprepro
• You can add more sections if you need more code
names (lucid, trusty, etc).
• SignWith specifies which GPG key to use for signing
repository metadata
• You can get your gpg key ID by looking at the output
of gpg —list-keys
• This is not the same as using debsigs/debsign !!!
80. But that’s not all!
• You MUST have the ‘apt-transport-https’ package
installed on the system if your repository is served
over HTTPS!
• Without apt-transport-https, you can’t install
packages over HTTPS.
• You definitely want this.
81. Make sure to install apt-transport-https
package { ‘apt-transport-https‘:
ensure => latest,
}
83. Success
• You can now use beaker/kitchen.ci/etc to test your
infrastructure.
• Determine if the packages you need are actually
installed after your manifests are applied.
• Determine if the repositories you added are
actually added after your manifests are applied.
• Don’t need to wait forever for Ruby, redis, et al to
build during a test run.
84. BEST OF ALL !!!!
• You can now run Puppet on your development
VM using the same manifests you use in
production
• The manifests are applied and you are running
the same exact binaries you run in production
• Won’t catch ALL production bugs, but getting
closer to production during development is
super useful
85. Summary
• Creating package repositories can be tricky. Make
sure to GPG sign repository metadata.
• 99% of package repositories get this wrong.
• Carefully read the documentation of createrepo and
reprepro.
• Make sure to install necessary libraries for verifying
signatures and accessing repositories via HTTPS.
• Always serve up your repositories over HTTPS.
