At the end of 2012 ESPN undertook an effort to modernize its deployment and maintenance of linux based platform services. ESPN faced a challenging problem in that hundreds of servers needed to be puppetized yet the largest cluster of identical servers was only eight servers. Therefore having a puppet environment that was flexible, consistent, simple to understand and data driven was critical to success. This session looks at the architectural decisions made by ESPN while performing a green field implementation of Puppet 3.0 and reflects on the resulting good and bad of those decisions. Ben Schofield Senior Application Architect, ESPN Ben Schofield is the middleware architect for ESPN. With 11 years of IT experience working for Fortune 200 companies in the retail, insurance, financial and media industries, Ben has seen the good, bad and ugly of IT operations and management. He brings a unique perspective on how a well designed devops team with the right mind set can help large IT departments reduce costs and decrease time to market.

1. PuppetConf 2013

2. CHALLENGES

3. • Rapid growth
– Number of servers being provisioned
– New services being provided
• Manual configurations
• Drift
• Silos
• Traditional clouds not reasonable

4. Objectives

5. • Push button configuration
• Self service for customers
• Visibility into state of configurations
– Reports
• Simplify knowledge transfer
• Data driven
• Elastic
• Build/rebuild a node at anytime

6. • Support workstations/dev/test/prod
• Allow adhoc manual changes*
• Do not slow down development teams
• Include legacy servers

7. Application
Middleware
OS
Server
Storage
Network
IAAS
PAAS
SAAS
Puppet

8. Puppet Versions
• Puppet 3.0.0
• Puppet Dashboard 1.2.12
• PuppetDB 1.0.1
• MCollective 2.2.1
• ActiveMQ 5.5

9. The Data

10. • Inherent Facts
– Facter
– Exists simply because the node exists.
• Derived Facts
– Business rules applied to inherent facts.
– Puppet custom facts
• Assigned Facts
– Exist because we deem it to be true.
– Top or node scoped variables

11. Assigned Facts
• espn_role
– Identifes the role class to be applied
• espn_cluster
– Identifies nodes in the cluster
• espn_env
– workstation, dev, test, qa, prod
• espn_owner
– change notifications
– security delegation
– licensing

12. What to manage? Everything!

13. Role
• Aligns with business and or IT needs
• Defines everything on a node
• Used for classification of nodes
• Exactly one role per node
• Includes profiles and releases
• espn_role

14. Group everything by roles
AA BB CC

15. Release Environment
• Not puppet environments
• Determined by application development
and release cycle
• espn_env

16. Group by release environment
AA BB CC
ProdProd
QAQA
DevDev
WorkWork

17. Clusters
• Always identical
• Configured to interact with one another
• espn_cluster

18. Group by clusters
AA BB CC
ProdProd
QAQA
DevDev
WorkWork

19. Hiera hierarchy in theory
hostname/%{hostname}
cluster/%{espn_cluster_name}
role_env/%{espn_role}_%{espn_env}
role/%{espn_role}
env/%{espn_env}
network/%{espn_network}
os/%{operationsystem}
default

20. Hiera hierarchy in practice
role_env/%{espn_role}_%{espn_env}
role/%{espn_role}
env/%{espn_env}
network/%{espn_network}
defaults

21. Classification

22. Application
Middleware
OS
Server
Storage
Network
IAAS
PAAS
SAAS
Role
Profiles
Releases

23. Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy

24. Resources
• The building blocks
• Everything managed is a resource
• Defined in modules, used by profiles
• Two resources never manage the same
configuration

25. Modules
• Isolate resources within the module
• Never reference another module
• No organizational specific logic
• init.pp is a minimal installer
• Reusability is key

26. R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
Resources by module

27. R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
R RR RR RR R
Never cross modules

28. Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy

29. Profile
• Defines the platform
• Cross module references
• Enforces dependencies between modules
• Class parameters prevent hiera overrides

30. Example Profile
class profile::jboss_eap_6(
$java_version=‘jdk_1_7_u10’,
$mod_cluster_version=‘1.2.3’,
$httpd_version=‘2.2.22’,
){
include “java::${java_version}”
include directories::middleware
class{‘mod_cluster’:
version => $mod_cluster_version,}
class{‘httpd’:
version => $httpd_version,}
class{‘jboss_eap_6’:
java_home => getvar(“java::${java_version}::home”),}
}

31. Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy

32. Release
• Special type of profile
• Knows how to install on top of a profile
• Deploys resources from an “artifact hash”
• Cleans up removed artifacts
• Driven by versioned release id
– espn_release_id

33. Artifact Hash
• Defines abstract resources in a release
• Contract between developers and
operational groups
• Profile agnostic

34. Example Artifact Hash
applications
‘jee-app.ear’:
version: 1.2.3
espn.war:
version: 1.0.0
datasources:
datasource1:
driver_name: ”oracle”
libraries:
utility.jar:
version: 1.0.0

35. Example Release
Class release::studio_record{
artifact_hash = hiera(‘artifacts’,undef,”release/${espn_release_id}”)
#modify the artifact hash so it can be used with create_resources
#set organization specific parameters such URL’s to the artifact repo
resources {‘jboss7_datasource: purge => true,}
create_resources(jboss7_datasource, $artifact_hash[‘datasources’])
resources {‘jboss7_deployment: purge => true,}
create_resources(jboss7_deployment, $artifact_hash[‘applications’])
}

36. Role
Profiles
Releases Resources
Resources
Classes Resources
Resource Hierarchy

37. Example Role
class role::studio_record{
include profile::base
include profile::jboss_eap_6
include profile::mysql_5
include release::studio_record
}

38. site.pp
# assigned facts retrieved from external datasources
$espn_cluster_nodes = espn_cluster_nodes()
$espn_release_id = hiera(‘espn_release_id’)
# single assigned fact drives 100% of classification
node default {
include “role::${espn_role}”
}

39. Defining a Node

40. MCollective

41. Colonol John Boyd
• Military Strategist
• OODA Loop

42. Observations
Decision
(Hypothesis)
Action
(Test)
Observe Orient Decide Act
Feedback
Feedback
Unfolding
Circumstances
Outside
Information
Unfolding
Interaction
With
Environment
Cultural
Traditions
Analysis &
Synthesis
Previous
Experience
New
Information
Genetic
Heritage
Feedback
Implicit
Guidance
& Control
Implicit
Guidance
& Control

43. OODA Loop
"Time is the dominant parameter. The
pilot who goes through the OODA cycle in
the shortest time prevails because his
opponent is caught responding to
situations that have already changed.“
Harry Hillaker (chief designer of the F-16)

44. Observations
Decision
(Hypothesis)
Action
(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet environments
Feedback
Unfolding
Circumstances
Outside
Information
Unfolding
Interaction
With
Environment
Cultural
Traditions
Analysis &
Synthesis
Previous
Experience
New
Information
Genetic
Heritage
Feedback
Implicit
Guidance
& Control
Implicit
Guidance
& Control

45. Observations
Decision
(Hypothesis)
Action
(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet environments
Feedback: (Test) Noop puppet run on production
Unfolding
Circumstances
Outside
Information
Unfolding
Interaction
With
Environment
Cultural
Traditions
Analysis &
Synthesis
Previous
Experience
New
Information
Genetic
Heritage
Feedback
Implicit
Guidance
& Control
Implicit
Guidance
& Control

46. Noop Puppet Runs
• Hourly cron job
– “mco puppet runall 20 –noop”
• Dashboard displays unresponsive nodes
– no_longer_reporting_cutoff

47. Observations
Decision
(Hypothesis)
Action
(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet environments
Feedback: (Test) Noop puppet run on production
Unfolding
Circumstances
Outside
Information
Unfolding
Interaction
With
Environment
Puppet
Dashboard
Cultural
Traditions
Analysis &
Synthesis
Previous
Experience
New
Information
Genetic
Heritage
Feedback: (Action) No-noop puppet run on production
Implicit
Guidance
& Control
Implicit
Guidance
& Control

48. Configuration Changes
• Always noop first
• Always target nodes with filters
• Always use tags
• Validate changes then apply –no-noop
mco puppet runonce
--no-noop
-C jboss-eap-6
--tag initscript --tag jboss-eap-6

49. Observations
Decision
(Hypothesis)
Action
(Test)
Observe Orient Decide Act
Feedback: Puppet runs on dev/test puppet environments
Feedback: (Test) Noop puppet run on production
Unfolding
Circumstances
Outside
Information
Cultural
Traditions
Analysis &
Synthesis
Previous
Experience
New
Information
Genetic
Heritage
Feedback: (Action) No-noop puppet run on production
Implicit Guidance & Control:
MCollective Agent Plugins
Implicit
Guidance
& Control
Unfolding
Interaction
With
Environment
Puppet
Dashboard

50. 20/20 Hindsight

51. Do Differently?
• Implement MCollective first
– Security
– Sub-collectives
– Availability
• Plan for developer dashboards
• Implement Custom ENC

52. Nice to have?
• Puppet runs that span multiple nodes
– Allocate disk on node A, create shared
filesystem on node B
• Resources automatically tagged with
catalog unique identifiers
• Role based access control for
dashboard

53. Questions?