R.I.Pienaar gave a presentation on external data in Puppet 4. The presentation covered hierarchical data lookup using Hiera and the new lookup() function. It discussed automatic parameter lookup, CLI usage of lookup(), data in modules, and tips/tricks like using lookup() to include classes. The presentation positioned external data as configuration for code and emphasized the pluggable and hierarchical nature of lookup in Puppet 4.

1. R.I.Pienaar
PuppetConf 2016
External Data in Puppet 4

2. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Who am I?
• Puppet User since 0.22
• Really into external data
• Author facts.d, extlookup, Hiera, MCollective
• Consultant - DevOps, Automation,
Architect, Development
• Open Source @ github.com/ripienaar
• Volcane on IRC

3. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
External Data
Is Configuration For
Your Code

4. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Hierarchical Data
class monitor (
Array[Monitor::Plugin] $plugins = []
) {
…
}

5. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Hierarchical Data
class{“monitor”:
plugins => [
“cpu",
“write_graphite”,
“df”, …
]
}

6. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Automatic Param Lookup
include monitor
# hieradb/nodes/host1.yaml
monitor::plugins:
- my_app
class{“monitor”:
plugins => [
“my_app”, “cpu”, “df”, …
]
}
+
=

7. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Hierarchical Data
host1 host2 host3 host4 host5 host6
prod stage dev
dc1 dc2
Site wide common data

8. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Hierarchical Data
host1 host2 host3 host4 host5 host6
prod stage dev
dc1 dc2
Site wide common data

9. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Hierarchical Data
host1 host2 host3 host4 host5 host6
prod stage dev
dc1 dc2
Site wide common data

10. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Hierarchical Data
host1 host2 host3 host4 host5 host6
prod stage dev
dc1 dc2
Site wide common data
$trusted[“certname”]
$environment
$facts[“dc”]

11. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Automatic Param Lookup
include monitor
# hieradb/nodes/host1.yaml
monitor::plugins:
- my_app
class{“monitor”:
plugins => [
“my_app”, “cpu”, “df”, …
]
}
+
=

12. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup
• lookup() function replaces hiera*()
• Automatic Param Lookup more useful
• environment and module aware
• Great CLI
• YAML/JSON out of the box, more to come
• Data compatible with Hiera, cohabits with
Hiera
• Pluggable

13. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - CLI
$ puppet lookup monitor::plugins —merge unique
——
- my_app
- write_graphite
- cpu

14. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
$ puppet lookup monitor::plugins —merge unique —explain
Merge strategy unique
…
Data Provider "Nodes"
Path “/etc/…/production/data/nodes/dev3.devco.net.yaml”
Original path: "nodes/%{trusted.certname}"
Found key: "monitor::plugins" value: [
"my_app"
]
…
Data Provider "common"
Path “/etc/…/data/common.yaml”
Original path: "common"
Found key: "monitor::plugins" value: [
"cpu",
"write_graphite"
]
…
Merged result: [
"my_app",
"cpu",
"write_graphite"
]

15. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
$ puppet lookup monitor::plugins —merge unique —explain
Merge strategy unique
…
Data Provider "Nodes"
Path “/etc/…/production/data/nodes/dev3.devco.net.yaml”
Original path: "nodes/%{trusted.certname}"
Found key: "monitor::plugins" value: [
"my_app"
]
…
Data Provider "common"
Path “/etc/…/data/common.yaml”
Original path: "common"
Found key: "monitor::plugins" value: [
"cpu",
"write_graphite"
]
…
Merged result: [
"my_app",
"cpu",
"write_graphite"
]
Every tier
Final result

16. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - Config
# environments/production/hiera.yaml
---
version: 4
datadir: data
hierarchy:
- name: "Nodes"
backend: yaml
path: "nodes/%{trusted.certname}"
- name: "common"
backend: yaml

17. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
$plugins = lookup(“monitor::plugins”)

18. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
$plugins = lookup(“monitor::plugins”, {
“value_type” => Array[Monitor::Plugin],
“default_value” => [“cpu”, …]
})

19. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
function monitor::params() {
$result = {
“monitor::plugins” => [
“cpu”, …
]
}
}
$plugins = lookup(“monitor::plugins”, {
“default_value_hash” => monitor::params()
})

20. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
$servers = lookup(“monitor::plugins”, {
“merge” => “first”
})
Default behaviour

21. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
$servers = lookup(“monitor::plugins”, {
“merge” => “unique”
})
Array Merge (hiera_array())

22. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
$servers = lookup(“monitor::plugins”, {
“merge” => “hash”
})
Shallow Hash Merge (hiera_hash())

23. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
$servers = lookup(“monitor::plugins”, {
“merge” => “deep”,
})
Deep Hash Merge

24. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
# common.yaml
users::local:
rip:
email: “rip@devco.net"
comment: “R.I.Pienaar”
sudoer: true
shell: “/bin/zsh”

25. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
# nodes/dev3.devco.net
users::local:
rip:
email: “arri.pienaar@client.com”

26. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
$ puppet lookup users::local —merge deep
---
rip:
email: arri.pienaar@client.com
comment: R.I.Pienaar
sudoer: true
shell: "/bin/zsh"

27. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Automatic Param Lookup
include monitor
# hieradb/nodes/host1.yaml
monitor::plugins:
- my_app
class{“monitor”:
plugins => [
“my_app”, “cpu”, “df”, …
]
}
+
=

28. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
$lookup_options = {
“merge” => {
“strategy” => “unique”
}
}
$plugins = lookup(“monitor::plugins”, $lookup_options)

29. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - function
$plugins = lookup(“monitor::plugins”)
# hieradata/common.yaml
lookup_options:
monitor::plugins:
merge:
strategy: unique

30. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
include monitor
# …/production/hieradata/common.yaml
lookup_options:
monitor::plugins:
strategy: unique
monitor::plugins:
- cpu
- …
class monitor (
Array[Monitor::Plugin] $plugins = []
) {}
Lookup - function

31. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Lookup - Data In Modules
# modules/monitor/hiera.yaml
---
version: 4
datadir: "data"
hierarchy:
- name: "common"
backend: "yaml"
# modules/monitor/data/common.yaml
lookup_options:
monitor::plugins:
strategy: unique

32. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
include monitor
# hieradata/common.yaml
monitor::plugins:
- cpu
- write_graphite
Lookup - Data In Modules
# hieradata/nodes/node1.yaml
monitor::plugins:
- my_app

33. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Tips and Tricks

34. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
lookup(“classes”, {“merge” => “unique”}).include
hiera_include()

35. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
hiera_include()
# common.yaml
lookup_options:
classes:
merge:
strategy: “deep”
knockout_prefix: “-“
classes:
- nagios
# nodes/node1.yaml
classes:
- -nagios
- sensu

36. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
lookup(“classes”).include
hiera_include()
# common.yaml
lookup_options:
classes:
merge:
strategy: “deep”
knockout_prefix: “-“

37. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
params.pp
# modules/ntp/hiera.yaml
---
version: 4
datadir: data
hierarchy:
- name: "Operating System Family"
backend: yaml
path: "%{facts.os.family}"
- name: "common"
backend: yaml

38. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
params.pp
# modules/ntp/data/AIX.yaml
---
ntp::driftfile: '/etc/ntp.drift'
ntp::package_name:
- 'bos.net.tcp.client'
ntp::restrict:
- 'default nomodify notrap nopeer noquery'
- '127.0.0.1'
ntp::service_name: 'xntpd'

39. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
params.pp
# modules/ntp/data/Debian.yaml
ntp::restrict:
- '-4 default kod nomodify notrap nopeer noquery'
- '-6 default kod nomodify notrap nopeer noquery'
- '127.0.0.1'
- '::1'
ntp::service_name: 'ntp'
ntp::servers:
- '0.debian.pool.ntp.org'
- '1.debian.pool.ntp.org'

40. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Future
• Hiera deprecated
• No more site wide hiera, only environment
and modules
• Pluggable via functions in modules - native or
ruby, much much simpler than old backends
• lookup.yaml or lookup.conf

41. R.I.Pienaar | rip@devco.net | http://devco.net | @ripienaar
Questions?
twitter: @ripienaar
email: rip@devco.net
blog: www.devco.net
github: ripienaar
freenode: Volcane
slack.puppet.com: ripienaar
https://www.devco.net/