"Puppeting in a Highly Regulated Industry" by Marinus Damm of PGE at Puppet Camp Portland 2014.

1. Puppe%ng
in
a
Highly
Regulated
Industry
Marinus
Damm
marinus.damm@pgn.com

2. Every
business
is
regulated…
• Labor
regs
minimum
wage,
paid
sick
leave,
hours
and
breaks
• Money
regs
income
tax
withholding,
accoun%ng
prac%ces
(SOX)
• Safety
regs
protec%ve
equipment,
training,
repor%ng
accidents
• Licensing
regs
business
license,
HAZMAT,
serving
liquor

3. “Highly
Regulated”
as
it
pertains
to
system
administra#on
Ac#ve
Monitoring
Level
of
Detail
of
regs

4. Ac#ve
Monitoring
Level
of
Detail
of
regs
Is
a
Policy
in
place?
Are
Procedures
to
implement
that
in
place?
Do
employees
receive
Training
on
P&P?
Can
you
Prove
that
P&P
are
followed?

5. Ac#ve
Monitoring
Level
of
Detail
of
regs
•
•
•
•
•
Separa%on
of
du%es
Data
access
System
access
%meouts
Least
privilege
Passwords
•
“Passwords
shall
be
at
least
eight
characters
in
length,
and
shall
include
at
least
one
uppercase
character,
one
lowercase
character,
one
numeral,
and
one
special
character.”

6. Who
Is
The
Boss?
FERC:
Federal
Energy
Regulatory
Commission
and
its
designee
NERC:
North
American
Electric
Reliability
Corp.

7. Power
Flow
to
Congress
to
FERC
to
NERC
United States
Constitution
Art. 1, Sec. 8
“to regulate
commerce among
the several
states”
Do
this,
or
else.
Or
else
what?
$$
Fines,
baby…
fines.

8. Power
Surge
• Used
to
be
that
NERC
made
sugges%ons
only
• As
electric
power
suppliers
were
deregulated,
the
need
for
predictable
delivery
increased
• In
2006,
FERC
designated
NERC
as
the
na%onal
‘Electric
Reliability
Organiza%on’
• NERC’s
sugges%ons
are
now
Standards.

9. How
Can
Companies
Get
On
Track?
Obviously
all
these
NERC
P&Ps
will
massively
increase
produc%vity….
or
not
So
how
do
we
deal
with
the
new
strictures?
à
We
need
a
framework!
Anybody
got
one?

10. Coincidentally,
on
a
Parallel
Track…
Aber
the
…
excesses
…
of
the
dot-­‐com
era,
the
business
side
wanted
to
rein
in
IT
Information
Technology
Infrastructure
Library (ITIL)

11. Two
Tracks
Align
• The
FERC
Reliability
Standards,
plus
• The
MBAs’
counteradack
on
Techies
gave
us
*
CHANGE
MANAGEMENT

12. Change
Management
The objective of change management ... is to
ensure that standardized methods and
procedures are used for efficient and prompt
handling of all changes to control IT infrastructure,
in order to minimize the number and impact of
any related incidents upon service.
from
the
wikipedia
ar#cle

13. Simplified
Example
of
Change
Flow
1.
2.
3.
4.
5.
6.
7.
8.
9.
Sysadmin
writes
proposal
for
new
sehng
Different
sysadmin
or
manager
agrees
Sysdmin
becomes
Change
Owner,
engages
CM
tool:
Describe
business
effects
of
doing/not
doing
Iden%fy
systems/services/apps/users
affected
Design
the
procedure
(including
verifica%on
and
backout
plan)
Design
and
execute
a
test
plan
Change
Owner
schedules
%me
for
change
Every
affected
IT
group
assesses
change
impact
Every
affected
system/service/app/user
reviews
change
and
authorizes
Change
Board
considers
all
imminent
changes,
weighs
risks
and
conflicts,
approves
change
for
implementa%on
Change
Owner
executes
procedure
at
scheduled
%me
Change
Owner
completes
change
record

14. And
that's
just
for
the
kiddie
systems
The
systems
handling
the
power
grid
proper
are
a
whole
'nuther
animal.

15. Ques%ons?
Correc%ons?

16. Why
do
we
puppet
the
way
we
do?
Power
History
Accountability

17. Why
do
we
puppet
the
way
we
do?
Power
History
Accountability

18. Why
do
we
puppet
the
way
we
do?
Power
History
Accountability

19. PGE
–
PEC
(Puppet
Enterprise
Components)
puppet
master
hdp
unk
puppet
console
DB
webservice
dev
puppet
db
tst
prd
webservice
database
All
three
are
VMs,
2
cores/8GB
ram,
RHEL

20. PGE
–
Puppet
Environments
Every
node
is
in
one
and
only
one
environment.
The
puppetmaster
has
three
parallel
directory
structures:
/etc/puppetlabs/puppet/environments/[dev|tst|prd]
unk
dev
The
directories
are
all
clones
of
a
single
git
repo,
and
pull
from
that
remote
repo
for
manifest
and
module
updates.
tst
prd

21. PGE
-­‐
Promo%on
dev
–
80
or
so
systems
deploy,
then
watch
puppet
reports
to
verify
what’s
changing...
and
that
things
don’t
keep
changing.
tst
–
around
100
systems
Collect
‘test
results’
for
inclusion
in
the
Change
dev
tst
change
management
bar
prd
–
around
120
systems
Several
new
or
revised
modules
are
promoted
as
a
single
change
–
an
‘OS
Release’
prd

22. PGE
-­‐
Keeping
Tabs
With
Custom
Facts
• third-­‐party
sobware
• locate
inconsistency
• feed
our
manifests
and
templates

23. PGE
-­‐
Custom
Facts
Defined
# synergy_status.rb
Facter.add("synergy_installed") do
setcode do
File.executable?("/usr/bin/syninfo")
end
end
Facter.add("synergy_joined") do
confine :synergy_installed => true
setcode do
domain = Facter::Util::Resolution.exec(‘syninfo --domain')
domain.eql?(“it.pgn.com")
end
end
Facter.add("synergy_status") do
setcode do
if Facter.value(:synergy_installed)
if Facter.value(:synergy_joined)
Facter::Util::Resolution.exec(‘syninfo --mode')
else
"Not_Joined"
end
else
"Not_Installed"
end
end
end

24. PGE
-­‐
Custom
Facts
Realized
synergy_installed => true
synergy_joined => true
synergy_status => connected
These
are
reportable/searchable
via
PuppetDB.

25. PGE
-­‐
Custom
Facts
Available
#!/bin/bash
FACT=$1
VALUE=$2
curl -X GET -H "Accept: application/json"
--cacert /home/marinus/puppetInventory/ca.pem
--cert
/home/marinus/puppetInventory/cert.pem
--key
/home/marinus/puppetInventory/private.pem
'https://puppetdb:8081/v2/facts/'${FACT}
--data-urlencode 'query=["not", ["=", "value", "'${VALUE}'"]]'
Just
show
me
systems
where
Synergy
is
not
‘connected’:
/facts_without_value.sh
synergy_status
connected

26. PGE
-­‐
Really
Simple
Modules
• A
few
module-­‐level
variables
probably
set
from
facts
or
literals,
not
computed
• A
File
resource
usually
a
.conf
file
• A
Service
resource
subscribed
to
the
file
resource

27. PGE
-­‐
Really
Similar
Modules
• If
you’ve
seen
one,
you’ve
seen
‘em
all
• Every
file’s
content
comes
from
a
template
even
if
there’s
no
variability
• puppet-­‐lint
helps
us
enforce
textual
appearance

28. PGE
–
Common
Module
Layout
class synergy {
if $::synergy_installed != 'true' {
warning('This node does not have Synergy installed')
} else {
$os = $::operatingsystem
$filegroup = $os ? {
/AIX/
=> 'system',
/RedHat/ => 'root',
default => 'unk',
}
File {
ensure => file,
mode
=> '0644',
owner => 'root',
group => $filegroup,
}
file { '/etc/synergy/gid.ignore':
content => template ("synergy/gid.ignore.${os}.erb"),
}
file { '/etc/synergy/synergy.conf':
content => template ("synergy/synergy.conf.${os}.erb"),
}
service { 'synergy':
ensure
=> running,
enable
=> true,
subscribe => File['/etc/synergy/synergy.conf'],
}
}
}

29. coda
Puppet
Enterprise
gives
us
Power
lets
us
deal
with
our
History
eases
Accountability
Marinus
Damm
marinus.damm@pgn.com

30. PGE
Service
Territory
St. Helens
PGE SERVICE TERRITORY
30
Scapoose
26
Banks
North Plains
5
30
Fairview
47
84
26
84
10
Milwaukie
Tigard
King City
Scholls
YAMHILL CO
Tualatin
Carlton
47
99W
Dayton
212
Sandy
219
26
Eagle Creek
26
Zigzag
211
99E
Government Camp
Estacada
Canby
Barlow
26
HOOD RIVER CO
WASCO CO
Aurora
St. Paul
224
Mulino
99E
Hubbard
18
99W
Brightwood
211
Oregon City
Willsonville
26
224
213
MARION CO
YAMHILL CO
Lafayette
McMinnville
YAMHILL CO
Newberg
Dundee
Carver
West Linn
5
99W
MULTNOMAH CO
CLACKAMAS CO
Johnson City
43
205
Yamhill
VE
R CO
MA
H CO
Boring
Lake Oswego
Rivergrove
47
219
Happy Valley
205
217
WASHINGTON CO
NO
99E
Beaverton
210
T
UL
219
84
30
26
10
Gaston
Troutdale
Gresham
RI
Portland
8
Wood Village
M
405
OD
Hillsboro
8
Cornelius
HO
Forest Grove
221
Colton
211
Woodburn
211
5
AC
CO
A
M
KA O N C
MA
18
Willamina
Molalla
CL
Amity
Sheridan
RI
213
Gervais
S
O
Mt. Angel
Grand Ronde
99E
214
213
Marquam
Scotts Mills
Keizer
22
Silverton
221
213
99W
22
Salem
5
214
WASHINGTON COUNTY
214
MULTNOMAH COUNTY
YAMHILL COUNTY
Counties
Columbia
CLACKAMAS COUNTY
Washington
Multnomah
POLK COUNTY
MARION COUNTY
Yamhill
Clackamas
Marion
Polk
Turner
• About
a
million
points
of
delivery
• 1400
servers
(Windows
&
UNIX)
• Sixty
people
in
IT
Infrastructure
…
and
nice
benefits