"Wrangling 3rd Party installers from Puppet" by Spencer Krum and Mike Kinney of UTI Worldwide at Puppet Camp Portland 2014.

1. Wrangling 3rd Party
Installers with Puppet
Puppet Camp Portland 2014
January 14, 2014

2. Overview
●
●
●
●
●
About us
Definitions
Our situation
Overall steps
What we get and
what is missing
from installers
● Requirements
●
●
●
●
●
●
●
●
Installer “state”
Initial scripting
v2.0 using Puppet
Our solution
Testing
Challenges
Future
Q&A

3. About us
Mike Kinney
Spencer Krum
mike.kinney@gmail.com
krum.spencer@gmail.com
github.com/mkinney
github.com/nibalizer
● UTi Worldwide
● Mentioned in the
“Book”
● UTi Worldwide
● PSU CAT
● Co-author
of the “Book”
2nd Edition

4. About us
Mike Kinney
● Automating
installations for few
years
● New to puppet
Spencer Krum
● New to these 3rd
party tools
● Puppet expert

5. A few words about vendors
● We use a few vendors.
● We will try to keep their names out of it,
though it might slip out once or twice.
● The focus of this talk is how we work around
not being able to use a package
management tool. (like apt/rpm)
● These patterns should be universal.

6. Definitions (1 of 4)
● 3rd Party software:
○ Software that is bought from a software vendor,
almost always closed source.
● Installer:
○ The installation method described by the vendor for
the 3rd party software. Often this is a binary file like
“installer.bin”.

7. Definitions (2 of 4)
● Quality Assurance (QA):
○ Process that validates functionality of a release.
● Silent installation template file:
○ File with the all installation options and values.
Sample file:
<?xml
version="1.0"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
<entry key="acceptLicense">true</entry>
</properties>

8. Definitions (3 of 4)
● Version upgrade: When either the x or y in
version x.y.z is higher than the currently
installed version.
When the product has bugs or errors, sometimes we will
get out of cycle patches from the vendor:
● Engineering build: (no applicable version)
○ quick (little or no) QA before releasing version
● Hotfix: (the “z” in version x.y.z)
○ full QA before releasing version
Note: Some vendors use x.y.z.a

9. Definitions (4 of 4)
● idempotent:
Only make changes if they are required.
● hiera:
Helps to separate data from code (see
Chapter 12 of the “Book”)

10. Our situation
● most of our stack uses Enterprise tools (lots
of 3rd party installers)
● mostly linux (OEL 5)
● lack of “root” has its challenges
● lots of environments
● each technology on a separate virtual
machine (means lots of VMs)
● many vendor/consultants/partners

11. Overall Installation steps
●
●
●
●
●
●
Download installation file (zip/tgz)
Review requirements
Pre-install steps
Run installer
Installation options
Post-install steps

12. 3rd Party Installers
Most have:
● sample silent
template file
● installation binary
● installation files
● installation
documentation
Most are missing:
● service script
(start/stop/status)
● any verification
methods (post
installation)

13. Pre-install Requirements
● Most require java (run-time)
○ cannot use openjdk!
○ some products bring own jre
● Most require another specific version of
another 3rd party installation (example:
product B v4.9.0 requires product A v8.3+)

14. Installation User Interfaces
● Interactive graphical user interface (xterm
window)
○ often the options are saved in a file and can be
converted into a silent installation file
● Interactive text mode
● Command line using “silent” installation
option:
./installer -silent options.txt

15. 3rd Party Installer “state”
● Different installers store installation status in
different places; typically in home dir and or
in the destination directory
● If vendor support wants to know about
system, they often ask for files/directories
showing this state (for what is installed)

16. Scripting options
A. Re-package:
○ install once - preserve files post-install
○ the internal “state” would have to be reverse
engineered
○ vendor would have an excuse “you did not install per
instructions”
B. Script using the “silent” installation

17. Initial installation scripting (v1.0)
●
●
●
●
●
●
Initial scripting (ant/shell scripts)
One configuration file per installation per VM
Lots of redundant config
Hard to maintain
Hard to train
Put the binary installers in version control
system (sounds bad - but it works for us)
● Shared /software source (home of tarballs)

18. v2.0 - Using puppet
●
●
●
●
<environment>/site.pp
rootless package
/data/env/<envname> (hiera)
Combine product installs into one class

19. v2.0 - Using puppet (continued)
● Vendor install class (defined type)
● Two stages to install:
1. extract software to /tmp and run
installation (see sufact hack)
2. configuration files, cron, aliases, etc.

20. Example of a Puppet Package
package { 'apache2':
ensure => '2.4',
}

21. Resource abstraction layer
Pretty Picture Here

22. Resource abstraction layer

23. Puppet is a Programming Language
● Contains and hides complexity (API)
● You do not really want to see how the
sausage gets made.
Note:
● We are creating “anti-patterns”.
● Need to ensure code is idempotent.

24. Example of API
class { 'ssh':
root_login => 'without-password',
}

25. Another API example
class { 'apache':
mods => ['index', 'rewrite'],
}

26. Example of Vendor::Software
class { 'vendorSoft::someSoftware':
version => 2.1,
}

27. Sample Vendor/Product
As an example, we are going to install Acme’s
product Foo version 4.9.0.
● In the <environment>/site.pp:
class { 'acme::foo490': }

28. Two types of Vendor Installs
1. 'untar' installs
Install consists of untarring or unziping files to the correct locations.
2. 'installer' installs
Install consists of running the vendor's installation binary with flags and a
silent file.
Note: Both of these will have the same
interface from puppet.

29. 'untar' installs: Puppet class
● Ensure all dependencies
● Unzip zipfile/tarball from network share to
installation location
● Set values in configuration files

30. 'install' installs: Puppet class
● Unzip zipfile/tarball from network share to
temporary location
● Set values in configuration files or “silent” file
● Run installer
● Clean up temporary installation
Note: We were not going for how many Puppet Style guide infractions we could
make, it just happened that way.

31. Idempotency
How do we make these idempotent?

32. Idempotency: untars
rootless::tardir {
'/opt/app/other_place/folder':
ensure => present,
tarfile => '/big/nfs/mount/tar.tar.gz'
}

33. Idempotency: untars
exec { 'untar-directory':
provider => shell,
command => '/bin/tar -xvzf /big/file.tgz
/the/location',
unless
=> 'stat /the/location/tardir',
}

34. Sample acme::acme_install (1 of 5)
define acme::acme_install(
$creates_file,
$installationFile,
$installationRoot,
$installer,
$product,
$tmpdir,
$version,
$installer_content,
$ensure
= true,
$group
= $::puppet_user,
$owner
= $::puppet_group,
$source_dir
= "/software/install/installers",
$create_tmpdir = true,
){

35. Sample acme::acme_install (2 of 5)
if $ensure {
# install
if $create_tmpdir {
file { "${tmpdir}-create":
path
=> $tmpdir,
ensure => directory,
}
}
anchor { "$name-unpack::begin": }
anchor { "$name-unpack::end": }
<snip - unzip/untar >
Anchor pattern

36. Sample acme::acme_install (3 of 5)
file { "${tmpdir}/silent":
ensure
=> file,
content => "$installer_content",
Silent file from template
require => Anchor["$name-unpack::end"],
}
file { "${tmpdir}/${installer}":
ensure
=> file,
mode
=> '0755',
require => Anchor["$name-unpack::end"],
}
Ensure installer is
executable

37. Question
What is the return value of a zero byte file that
has the “execute” bit set?
$ echo > /tmp/nobytes
$ ls -al /tmp/nobytes
-rw-r--r--
1 user
wheel
1 Jan 14 12:06 /tmp/nobytes
$ chmod +x /tmp/nobytes
$ ./nobytes
$ echo $?
What is the return value from “./nobytes”?

38. Sample acme::acme_install (4 of 5)
# verify that the file we are about to execute is *not* a 0 byte "executable"
exec { "verify-installer-${acme_product}":
command
=> "/usr/bin/[ -s ${tmpdir}/${installer} ]",
provider => 'shell',
cwd
=> $tmpdir,
require
Check
=> File["${tmpdir}/${installer}"],
}
exec { "install-${acme_product}":
command
=> "source ~/.bashrc && ${tmpdir}/${installer} -silent -V responseFile=${tmpdir}/silent",
provider => 'shell',
timeout
=> 30000,
cwd
=> $tmpdir,
Do installation
require => [File["${tmpdir}/silent"], File["${tmpdir}/${installer}"], Exec["verifyinstaller-${product}"], ],
creates
}
=> "${installationRoot}/${creates_file}",

39. Sample acme::acme_install (5 of 5)
# Check the logs for errors
exec { "${acme_product}-log-check":
command
=> "! /bin/grep -R -E "::ERROR::|err.X" ${::puppet_user_home}/.ACME",
provider
=> 'shell',
require
=> Exec["install-${acme_product}"]
Any errors?
}
# need to clean up tmpdir
if $clean_up_tmpdir {
exec { "${tmpdir}-remove":
Clean up
command
=> "/bin/rm -fr ${tmpdir}",
provider
=> 'shell',
require
=> [ Exec["install-${product}"], Exec["${product}-log-check"], ],
}
}
} else { # uninstall
<snip>

40. Problem is (still) idempotency
● Multiple resources cannot be chained to an
unless clause of an exec
● Anchors are a part of this
● Maybe 3.4 contains() will help fix this, we
have not had time to evaluate

41. Sufact (su fact)
● Puppet module
● Modest ambitions
● Turns out to be really awesome
https://github.com/TheDarren/fact

42. Sufact (su fact)
● fact { "su_support": value => "IDG" }
● Then:
○ $su_support == "IDG"
We use this to flag and pin versions of vendor
software.

43. Puppet
● Built each version of each installer as
separate module
acme::yyy123
where yyy is a short name of the product
and 123 is the “version”
Write facts to set these values, then check

44. Sample foo490 (1 of 3)
class acme::foo490(
$owner
= $::puppet_user,
$group
= $::puppet_group,
$install_root
= "/opt/app/${::puppet_user}",
$source_dir
= '/software/install/installers',
$tmpdir
= "/tmp/tmpfoo490-${::puppet_user}",
$installationFile
= 'foo_4.9.0_linux26gl23_x86_64.zip',
$installer
= 'Installer-lnx-x86.bin',
$version
= '4.9.0',
# these values are in the silent file
$silent_environmentName
$feature_acme
$ensure
= true,
Silent values
= hiera('acme_feature_xxx', “blargh”,
){
= 'FOO',
Hiera pattern

45. Sample foo490 (2 of 3)
$installationRoot="${install_root}/somedir"
$product = 'acme490'
anchor { "${product}_installed": }
if $ensure {
Anchor pattern
Ensure pattern
# Note: Config files/cron entries, etc. here
if $acme490_installed == undef {
$installer_content = template("acme/${product}/silent.erb")
acme::acme_install { $product:
installationFile
=> $installationFile,
installationRoot
=> $installationRoot,
installer
=> $installer,
product
=> $product,
tmpdir
=> $tmpdir,
Call our defined type

46. Sample foo490 (3 of 3)
version
=> $version,
creates_file
=> "foo${product}",
ensure
=> true,
group
=> $group,
owner
=> $owner,
source_dir
=> $source_dir,
installer_content => $installer_content,
before
=> Anchor["${product}_installed"],
Anchor pattern
}
fact { "${product}_installed":
value
=> $version,
require => Acme::Acme_install[$product],
}
}
} else { # uninstall
Fact pattern

47. Combining Vendor products
class acme::foo12 (
$installationRoot = "/opt/app/${::puppet_user}/somedir",
){
anchor { 'acme::foo12::begin': }
anchor { 'acme::foo12::end': }
class { 'acme::foo123':
#parameters
}
Anchor [‘acme::foo12::begin’] ->
Class [‘acme::bar123’] ->
Class [‘acme::baz124’] ->
Anchor [‘acme::foo12::end’
<snip>
Dependency
chaining pattern

48. Testing the Puppet installations
●
●
●
●
●
Similar to Rspec-system/Beaker
Totally home-grown
Written in shell
Integration testing
Linux Container (LXC) Based

49. Testing the Puppet installations
●
●
●
●
●
●
Fires up LXC
Builds ssh bridge to LXC
Run “puppet apply”
Runs suite of 'checks'
Teardown LXC
Repeat for ever suite of checks

50. Testing the Puppet installations
● Work in progress
● Plan to leverage this for refactoring and
beautification project
● Plan to use Jenkins for reporting

51. Testing the Puppet installations
#!/bin/bash
testname=foo610
if [ -d "/somedir/config" ];then
:
else
exit 1
fi

52. Challenges
●
●
●
●
●
●
●
Often used the “dot-uh-oh” versions
Large and many installers
Many different versions
Testing
Puppet development environment
Installation validation
Orchestration issues (queue creation, stop
commands before upgrading files, start &
wait, etc.)

53. Future
● Automation of pre-installation (checks)
● Automatically add the monitoring/syslog
configs (both are done manually now)
● Possibly pull post-processing into puppet
● md5sum checking of installed files
● Possibly look into app deployments using
puppet (need orchestration in a big way)

54. Questions
Any questions?