A presentation at PuppetConf 2013 about modules reusability, naming standards and stacks.

1. Anatomy of a
Reusable Module
Alessandro Franceschi
github.com/example42
PuppetConf 2013

2. Be

3. patient
please

4. How

5. do
we

6. use
Puppet
today?
Include classes
Set Params
Variables
Define
Business
Integration Logic
Provide
Configuration Files
Manage
Resources

7. How do we use Puppet today
Include classes
manifests/
site.pp
Set Parameters / Variables
Integration logic
Resources
ENC HIERA
SITE
MODULES
SHARED
MODULES
BAD
EDGE
Configuration files
manifests/
site.pp
ENC HIERA
manifests/
site.pp
BAD?
BAD?
BAD
SITE
MODULES
SITE
MODULES
SHARED
MODULES
SITE
MODULES
SHARED
MODULES
manifests/
site.pp
BAD?
ENC
BAD?

8. A

9. reusable

10.
module
is

11. all

12. about
CHOICE
Operating Systems
Infrastructures
Scales
Node classifiers
Installation methods
Alternative setups

13. The

14. cons
of

15. a
reusable
module
Harder Longer
Development
Complexity
Verbosity
Not Optimized for
performance

16. THE PARAMETERS
DILEMMA
Managed resources attributes
Application specific config options
Application logic and behaviour
Integration with other modules

17. Parameters: Resources attributes
Enough:
$package = $redis::params::package,
$service = $redis::params::service,
$service_ensure = 'running',
$service_enable = true,
$file = $redis::params::file,
$file_notify = Service['redis'],
$file_source = undef,
$file_content = undef,
Too much?
$package_provider = undef,
$file_owner = $redis::params::file_owner,
$file_group = $redis::params::file_group,
$file_mode = $redis::params::file_mode,
$file_replace = $redis::params::file_replace,
Benefits from: A standard naming convention

18. Parameters: Application options
Enough:
$puppet_server = “puppet.${::domain}”,
$syslog_server = “syslog.${::domain}”,
$munin_server = “munin.${::domain}”,
$dns_servers = [ '8.8.8.8' , '8.8.4.4' ],
Too much!
$anonymous_enable = true,
$anon_mkdir_write_enable = true,
$anon_upload_enable = false,
$chroot_list_enable = true,
$chroot_list_file = '/etc/vsftpd/chroot_list',
$resourcefile = $nagios::params::resourcefile,
$statusfile = $nagios::params::statusfile,
$commandfile = $nagios::params::commandfile,
$resultpath = $nagios::params::resultpath,
$retentionfile = $nagios::params::retentionfile,
$p1file = $nagios::params::p1file,
Benefits from: Template + Options Hash pattern

19. Parameters: Application logic
Examples:
$install_client = true,
$install_stomp_server = true,
$install_plugins = true,
$use_ssl = false,
$munin_autoconfigure = true,
$service_autorestart = true,
$manage_package_repo = true,
$run_initdb = undef,
Benefits from: A standard naming convention

20. Parameters: Modules Integrations
Examples:
$mongo_db_host = $graylog2::params::mongo_db_host,
$mongo_db_port = $graylog2::params::mongo_db_port,
$mongo_db_name = $graylog2::params::mongo_db_name,
$mongo_user = $graylog2::params::mongo_user,
$mongo_password = $graylog2::params::mongo_password,
$elasticsearch_template = $graylog2::params::elasticsearch_template,
$elasticsearch_path = $graylog2::params::elasticsearch_path,
$database = $puppetdb::params::database,
$manage_redhat_firewall = $puppetdb::params::manage_redhat_firewall,
$db_type = 'mysql',
Benefits from: Shared Stacks

21. PATTERNS
REUSABILITY

22. Managing

23.
files
Let user decide how
to manage
configuration files.
Alternatives:
source
content
concat
augeas
custom types

24. Managing files: source content
redis/manifests/init.pp
class redis (
$file = $redis::params::file,
$file_source = undef,
$file_template = undef,
$file_content = undef,
) {
[...]
$managed_file_content = $file_content ? {
undef = $file_template ? {
undef = undef,
default = template($file_template),
},
default = $file_content,
}
[...]
if $redis::file {
file { 'redis.conf':
path = $redis::file,
source = $redis::file_source,
content = $redis::managed_file_content,
}
}
}
Provide the Puppet path of an erb template
class { ‘redis’:
file_template = ‘site/redis/
redis.conf.erb’,
}
Provide directly the content attribute
class { ‘redis’:
file_content = “template(‘site/redis/
redis.conf.erb’)”,
}
Provide a fileserver source path
class { ‘redis’:
file_source = ‘puppet:///modules/site/
redis/redis.conf’,
}
Manage the configuration file with other methods
(augeas, concat...)
class { ‘redis’: }

25. Multiple

26.
config

27. files

28.
Add parameters
to main class
Use a generic
conf define
Manage the whole
configuration dir

29. Multiple files: Add parameters
elasticsearch/manifests/init.pp
class elasticsearch (
$file = $elasticsearch::params::file,
$file_source = undef,
$file_template = undef,
$file_content = undef,
[...]
$init_script_file = '/etc/init.d/elasticsearch',
$init_script_file_template = 'elasticsearch/init.erb',
$init_options_file = $elasticsearch::params::init_options_file,
$init_options_file_template = 'elasticsearch/init_options.erb',
Provide custom templates for the main file and the init script
class { ‘elasticsearch’:
file_template = ‘site/elasticsearch/elasticsearch.yml.erb’,
init_script_file_template = ‘site/elasticsearch/elasticsearch.init.erb’,
}

30. Multiple files: Generic conf define
nova/manifests/conf.pp
define nova::conf (
$source = undef,
$template = undef,
$content = undef,
$path = undef,
[...]
$options_hash = undef,
$ensure = present ) {
include nova
$managed_path = $path ? {
undef = ${nova::config_dir}/${name},
default = $path,
}
[...]
file { nova_conf_${name}:
ensure = $ensure,
source = $source,
content = $managed_content,
path = $managed_path,
mode = $managed_mode,
owner = $managed_owner,
group = $managed_group,
require = $managed_require,
notify = $managed_notify,
replace = $managed_replace,
}
}
Provide a custom template for an alternative config file in config_dir
nova::conf { ‘rootwrap.conf’:
template = ‘site/nova/rootwrap.conf.erb’,
}

31. Multiple files: Whole config dir
redis/manifests/init.pp
class redis (
$dir = $redis::params::dir,
$dir_source = undef,
$dir_purge = false,
$dir_recurse = true,
) {
[...]
$dir_ensure = $ensure ? {
'absent' = 'absent',
'present' = 'directory',
}
if $redis::dir_source {
file { 'redis.dir':
ensure = $redis::dir_ensure,
path = $redis::dir,
source = $redis::dir_source,
recurse = $redis::dir_recurse,
purge = $redis::dir_purge,
force = $redis::dir_purge,
notify = $redis::file_notify,
require = $redis::file_require,
}
}
}
Provide a custom source for the whole config_dir
class { ‘redis’:
dir_source = ‘puppet:///modules/site/redis/conf/’,
}
Provide a custom source for the whole config_dir and purge any
not managed config file
class { ‘redis’:
dir_source = ‘puppet:///modules/site/redis/conf/’,
dir_purge = true,
}

32. Managing

33.
Users
Everyone has his
own users...
Leave options to
decide
if, how and where to
manage the ones
the module requires.

34. Managing Users
elasticsearch/manifests/init.pp
class elasticsearch {
$ensure = 'present',
[...]
$user = 'elasticsearch',
$user_uid = undef,
$user_gid = undef,
$user_groups = undef,
$user_class = 'elasticsearch::user',
[...]
if $elasticsearch::user_class {
require $elasticsearch::user_class
}
elasticsearch/manifests/user.pp
class elasticsearch::user {
@user { $elasticsearch::user :
ensure = $elasticsearch::ensure,
comment = ${elasticsearch::user} user,
password = '!',
managehome = false,
uid = $elasticsearch::user_uid,
gid = $elasticsearch::user_gid,
groups = $elasticsearch::user_groups,
shell = '/bin/bash',
}
User | title == $elasticsearch::user |
}
Do not create the requested user
class { ‘elasticsearch’:
user_class = undef,
}
Provide the user in a different custom class
class { ‘elasticsearch’:
user_class = 'site::users',
}
Run elasticsearch with a different user
class { ‘elasticsearch’:
user = 'apache',
}

35. Managing
extra

36.
resources
Options to specify
custom classes
Options to pass
an hash to
create_resources

37. Extra Resources: Custom classes
elasticsearch/manifests/init.pp
class elasticsearch (
$dependency_class = 'elasticsearch::dependency',
$monitor_class = 'elasticsearch::monitor',
$firewall_class = 'elasticsearch::firewall',
$my_class = undef,
) {
[...]
if $elasticsearch::dependency_class {
include $elasticsearch::dependency_class
}
if $elasticsearch::monitor and $elasticsearch::monitor_class {
include $elasticsearch::monitor_class
}
if $elasticsearch::firewall and $elasticsearch::firewall_class {
include $elasticsearch::firewall_class
}
if $elasticsearch::my_class {
include $elasticsearch::my_class
}[...]
Provide the modules dependencies with a custom class
class { ‘elasticsearch’:
dependency_class = 'site::dep_elasticsearch',
}

38. Extra Resources: Resources Hash
elasticsearch/manifests/init.pp
class elasticsearch (
$create_resource = undef,
$resources_hash = undef,
) {
[...]
if $create_resource {
create_resources( $create_resource , $resources_hash )
}
Alternative: A single hash that includes resources and
resources_hash
Provide the modules dependencies with a custom class
class { ‘elasticsearch’:
create_resource = 'file',
resources_hash = {
path = '/etc/elasticsearch/my_file',
content = template('site/elasticsearch/my_file.erb),
mode = '0600',
},
}

39. Managing
Packages
and
Services
Names change
Custom packages
are common
Leave choice,
optionally

40. Managing packages
openssh/manifests/init.pp
class openssh (
$ensure = 'present',
$version = undef,
$package = $openssh::params::package,
[...]
) {
if $version and $ensure == 'present' {
$managed_package_ensure = $version
} else {
$managed_package_ensure = $ensure
}
if $openssh::package {
package { $openssh::package:
ensure = $openssh::managed_package_ensure,
}
}
openssh/manifests/params.pp
class openssh::params {
$package = $::osfamily ? {
Suse = 'openssh',
OpenBSD = '',
default = 'openssh-server',
}
Install a custom company-openssh package
class { ‘openssh’:
package = 'company-openssh',
}

41. Managing services
openssh/manifests/init.pp
class openssh (
$service = $openssh::params::service,
$service_ensure = 'running',
$service_enable = true,
[...]
) {
if $ensure == 'absent' {
$managed_service_enable = undef
$managed_service_ensure = stopped
} else {
$managed_service_enable = $service_enable
$managed_service_ensure = $service_ensure
}
if $openssh::service {
service { $openssh::service:
ensure = $openssh::managed_service_ensure,
enable = $openssh::managed_service_enable,
}
}
openssh/manifests/params.pp
class openssh::params {
$service = $::osfamily ? {
Debian = 'ssh',
default = 'sshd',
}
[...]
Manage a custom company-openssh service
class { ‘openssh’:
service = 'company-openssh',
}

42. Managing
Installation

43.
options
Let users decide:
OS Packages
Upstream tarballs
Provider

44. Installation options
elasticsearch/manifests/init.pp
class elasticsearch (
$package_provider = undef,
$install = 'package',
$install_base_url = $elasticsearch::params::install_base_url,
$install_source = undef,
$install_destination = '/opt',
) {
[...]
$managed_file = $elasticsearch::install ? {
package = $elasticsearch::file,
default = ${elasticsearch::home_dir}/config/elasticsearch.yml,
}
[...]
case $elasticsearch::install {
package: {
package { $elasticsearch::package:
ensure = $elasticsearch::managed_package_ensure,
provider = $elasticsearch::package_provider,
}
}
upstream: {
puppi::netinstall { 'netinstall_elasticsearch':
url = $elasticsearch::managed_install_source,
destination_dir = $elasticsearch::install_destination,
owner = $elasticsearch::user,
group = $elasticsearch::user,
}
[...]
Install elasticsearch from upstream source
class { ‘elasticsearch’:
install = 'upstream',
install_source = 'https://download.elasticsearch.org/
elasticsearch/elasticsearch/elasticsearch-0.90.3.zip',
}

45. Templates
and

46. hashes
Managing specific
application configs
parameters may get
out of control
A single config hash
to show them all
A custom template
to use them
Application specific configs
THE PARAMETERS
DILEMMA

47. Options Hash: Setup
openssh/manifests/init.pp
class openssh (
[...]
$file_template = undef,
$options_hash = undef,
site/templates/openssh/sshd_config.erb
# File Managed by Puppet
[...]
Port %= scope.function_options_lookup(['Port','22']) %
PermitRootLogin %= scope.function_options_lookup(['PermitRootLogin','yes']) %
UsePAM %= scope.function_options_lookup(['UsePAM','yes']) %
[...]
* Function options_lookup currently in Example42's Puppi module
Alternative site/templates/openssh/sshd_config.erb
Port %= scope.lookupvar('openssh::options_hash')['Port'] ||='22' %
PermitRootLogin %= scope.lookupvar('openssh::options_hash')['PermitRootLogin'] ||='yes' %
UsePAM %= scope.lookupvar('openssh::options_hash')['UsePAM'] ||='yes' %
[...]

48. Options Hash: Usage
Usage (with Hiera):
include openssh
/etc/puppet/hieradata/global.yml:
---
openssh::file_template: 'site/openssh/sshd_config.erb'
openssh::file_options_hash:
Port: '22222'
PermitRootLogin: 'no'
Usage (with parametrized class):
class { 'openssh':
file_template = 'site/openssh/sshd_config.erb'
file_options_hash = {
Port = '22222',
PermitRootLogin = 'no',
}

49. STANDARDS
NAMING
Managed resources attributes
THE PARAMETERS
DILEMMA

50. The
Handy

51. Grail
of

52. Modules

53.
Standards
A blog post*
Some discussions on
Puppet-Users
github.com/stdmod
Naming standards
for modules
parameters
Community driven
(draft 0.0.2)
* http://www.example42.com/?q=The_handy_Grail_of_Modules_Standards

54. Benefits

55. of

56.
suggested
(and

57. shared)
naming

58.
conventions
Saner User
Experience
Better modules
Interoperability
Reusability Patterns
Predictability in
usage and
development

59. Stdmod Params: Main resources
### General parameters
ensure (enable?)
version (package_version?)
### Package - Service - Main configuration file
package (package_name?)
package_ensure
package_provider
package_* [any relevant package type attribute]
service (service_name?)
service_ensure
service_enable
service_subscribe
service_*
file (file_path? config_file? config?)
file_source (source? config_file_source? config_source?)
file_template (template? config_file_template? config_template?)
file_content (content? config_file_content? config_content?)
file_* (config_file_*? config_*?)
file_options_hash (options? options_hash? file_options?)

60. Stdmod Params: Extra resources
other_package
other_package_*
client_package
client_package_*
server_package
server_package_*
other_service
other_service_*
log_file
log_file_*
pid_file
pid_file_*
init_script_file
init_script_file_*
init_config_file
init_config_file_*

61. Stdmod Params: Installation
### Parameter related parameters
install
install_url
install_base_url
install_source
install_destination
install_pre_exec
install_pre_exec_*
install_post_exec
install_post_exec_*
install_script_file
install_script_file_*
install_response_file
install_response_file_*

62. Stdmod Params: Monitor
monitor
monitor_tool
monitor_host
monitor_port
monitor_protocol
monitor_url
monitor_process
monitor_service
monitor_config_hash

63. Stdmod Params: Firewall
firewall
firewall_src
firewall_dst
firewall_port
firewall_protocol

64. STACKS
MODULES
Integrations of modules
THE PARAMETERS
DILEMMA

65. Why

66.
reinventing

67.
the

68. stack?
We always use
stacks.
We need them to
make something
useful with modules.
What about:
Sharing?
Best practices?
Standardization?

69. Stacks - A Simple Sample
class stack::logs (
$ensure = 'present',
$syslog_server = false,
$syslog_server_port = '5544',
$elasticsearch_server = false,
$elasticsearch_server_port = '9200',
$elasticsearch_cluster = 'logs',
$elasticsearch_java_opts = '-Xmx2g -Xms1g',
$install_logstash = false,
$install_elasticsearch = false,
$install_kibana = false,
$install_graylog2 = false,
$install_graylog2_webinterface = false,
$syslog_config_template = 'stack/logs/syslog.conf.erb',
$logstash_config_template = 'stack/logs/logstash.conf.erb',
$elasticsearch_config_template = 'stack/logs/elasticsearch.yml.erb',
$kibana_config_template = 'stack/logs/config.js.erb',
$graylog2_config_template = 'stack/logs/graylog2.conf.erb',
) {
[... TO BE CONTINUED ...]

70. Stacks - A Simple Sample
[...]
if $syslog_server {
rsyslog::config { 'logstash_stack':
content = template($syslog_config_template),
}
}
if $install_logstash {
class { 'logstash':
template = $logstash_config_template,
}
}
if $install_elasticsearch {
class { 'elasticsearch':
java_opts = $elasticsearch_java_opts,
template = $elasticsearch_config_template,
}
}
[...]

71. Stacks - Usage
On any host:
stack::logs { 'central':
syslog_server = 'syslog.example42.com',
}
On the Logstash (syslog) server:
stack::logs { 'central':
syslog_server = 'syslog.example42.com',
install_logstash = true,
elasticsearch_server = 'el.example42.com',
}
On the Elasticsearch server(s), with a custom configuration file:
stack::logs { 'central':
syslog_server = 'syslog.example42.com',
install_elasticsearch = true,
elasticsearch_server = 'el.example42.com',
elasticsearch_config_template = 'site/logs/elasticsearch.yml.erb',
}
On the Kibana server:
stack::logs { 'central':
syslog_server = 'syslog.example42.com',
install_kibana = true,
elasticsearch_server = 'el.example42.com',
}

72. The
Stacks
Logic
Stacks are local
Modules are shared
Higher level
interface
Integrate different
set of modules
Preserve modules
local change

73. How do we use Puppet today
Include classes
manifests/
site.pp
Set Parameters / Variables
Integration logic
Resources
ENC HIERA
SITE
MODULES
SHARED
MODULES
BAD
EDGE
Configuration files
manifests/
site.pp
ENC HIERA
manifests/
site.pp
BAD?
BAD?
BAD
SITE
MODULES
SITE
MODULES
SHARED
MODULES
SITE
MODULES
SHARED
MODULES
manifests/
site.pp
BAD?
ENC
BAD?
STACKS
STACKS
STACKS

74. Standards
for
Stacks?
Usual benefits:
User Experience
Interoperability
Higher level API
exposure
Possible GUI
Integrations

75. Steps
Define stdmod
naming conventions
Explore Stacks
design and approach
Create templates
for stacks and
modules
Explore
GUI integrations

76. SO

77. Long and thanks
for all the fish!
Graphics: www.tatlin.net
@alvagante

78. SO

79. Long and thanks
for all the fish!
Graphics: www.tatlin.net
@alvagante