Scaling Puppet Usage to a Global Organization

Case Study

Scaling Puppet Usage to a Global Organization

PuppetCamp Feb 7, 2013

The Presenter

Tor-Åke
– Hands-on System Architect
– Linux Pro since 2000
– DevOps / Operation Automation since 2005
– Puppet since 0.25
RedBridge
– Open Source Infrastructure Builders
– Consulting, Operations and Open Source
Products Since 2003
– Partners with PuppetLabs, Redhat, Amazon,
Zimbra, Others


TOC

Part 1: Case description
– The Customer and the Challenge
Part 2 : Way of Working
– How to Develop Code
Part 3 : Technical Platform
– Puppet Masters etc


Part I : Case Description

A Swedish telecommunications company
About 10 sites around the world
Thousands of users
Thousands of systems


System Types (in scope)

Virtual and metal servers
SuSE, RedHat, CentOS, Ubuntu and Solaris
Mostly OSS and Third Party Software
Divisions choose from predef:d system types
– Some unique applications
Supporting infrastructure
– Network flesystems for applications and user
data
– OS-native deployment systems (Satellite, Ops
Center etc)


The Challenge

Bring Home R&D IT From Outsourcing

... and in the process:

Shorten Lead Time
– Automate Deployment and (Change)
Management
Increase Cost Efficiency
– Solve each problem once
– Share the solution globally
– A scalable technical platform


Additional Requirements

Leverage existing expertise
Maintain site-local freedom to solve unique problems
Keep site freedom to plan and execute code updates


Puppet?

Puppet is naturally only a part of the solution...

...but an important focal point as it touches all
services!


Part II : Way of Working

Why and when is code developed?
How and by whom?
How is code shared?


The Facilitator

A Global Team, coordinating Puppet Development
Knowledge identifcation and sharing
Keeper of the code standard
Develop and support a Puppet architecture


Example Why and When

Site users need a service e.g. ”Hosted Jenkins”
Site team calls for Puppet code
Global team fnds a Jenkins expert
Global team helps Jenkins expert write modules
Modules are delivered to requesting site team and
users


Another example

Site users need a service e.g. Hosted Tomcat
Site has a Tomcat expert who can write Puppet
module
Global Team is notifed that Tomcat module exists
If another site requests same service, existing code
is ”globalized” with assistance from Global Team


Need
Need Development Sync Flowchart
code
code

Global
Global Ask
Ask
available? No
available? around
around

Yes

Fetch
Fetch Any
Any
and test
and test available? No
available?
Yes

Code
Code Modify
Modify Develop
Develop
OK?
OK? No and test
and test and test
and test
Yes

Done.
Done. Post global
Post global
Deploy!
Deploy! suggestion
suggestion


Code Sharing

Global Git repository (actually per module)
Each site pulls code to site-local repository
Test locally, and deploy (ITIL Change)
Global team is notifed of any local changes
If changes are to be globalized, GT pulls code from
site-local repo


Code Standard

Code structure optimized for sharing some parts,
while keeping others site-private
Readability and documentation built in
Unit test


Code Structure

Parameters module local to the site
params::jenkins in
moduleroot/params/manifests/jenkins.pp
All parameters can be overridden per node
Priority:
1) Node defnition (class params)
2) Params module
3) Module default (in init.pp!)


README

What is the scope of this module
What site and what OS:es has it been tested on
Example params fle for params module
No description of params!
– Those go in init.pp


Predictable Results

Else-clause with a fail()
– e.g.
If $::operatingsystem == Solaris {
…
} else {
fail ( ”we have not tested this OS yet” )
}


Code Review

Members of the Global Team send code for review
”Please look at this code and test it on your site”
– Code deemed unreadable = FAIL
– Code breaks other modules unit test = FAIL


Adherance to Standard

Lots of code contributors
– Varying experience with Puppet
Not always developers
– Unfamiliar with peer review, Scrum, XP, Unit
tests etc.
Global Team must fll the gaps
– But we are not subject experts!
– Educate eachother


Boilerplate

A module with all elements
Well commented
Copy and fll out the blanks


Manifest Patch Strategy

Many small increments?
OR
Take a big hit when needed?

+ 10 sites with slight differences, ever-evolving
+ 7 different OS:es
+ System experts distributed on the sites
= Regression testing must also be distributed!


Part III : Puppet Platform

Serve thousands of clients
Deployable by Puppet (apply)
Support Way of Working


Deploying Puppet

A global network flesystem (rsync+nfs)
Git repos with puppet code and packages
Clone it
Change parameters
Bootstrap a frst Puppet Master


Adding More Masters

DNS alternate names in the RR certifcate
Add server
Mount shared storage
Bootstrap server from another server
puppet agent –server=... --ca_server=...


Puppet Masters

3-10 Masters
1 CA Server
Shared (NFS) storage
– Manifests
– Certifcates
Apache plus Passenger
Round Robin DNS Records


Foreman

Just reports (for now...)
Masters store Yaml report on disk
Spool to foreman db periodically
– Foreman server can be ofine indefnitly w/o
losing reports


Agents

Deployed with Kick/Jumpstart
Version 2.7
Installed under /opt
Run by cron, hourly with splay


puppet:/// fles

We try to avoid them. Why?
– NAS is faster than Passenger
Packages are installed from OS native channels
– Available in Global NFS
What to do with Solaris?
– Packages directly from Global NFS


Example: installing Solaris pkg

With puppet fle transfer With NFS Mount
if $::custom_fact == install { file package { ‘VNDRpkg’ : source =>
{ “local.pkg”: source => ‘/net/nfsserver/remote.pkg’, adminfile
puppet:///remote.pkg => ‘/net/nfsserver/remote.adm’,
} file { “local.adm”: source => }
puppet:///remote.adm # We’re done!
} File[“local.pkg”] ->
Package[‘VNDRpkg’]}package
{ ‘VNDRPkg’: source => “local.pkg”,
adminfile => “local.adm”,
}
# + the custom fact ruby code!


Orchestration

We don't have it (yet!)
Generous ITIL Change Windows
Sprawling networks
– Firewall red tape


Lessons Learned

Modules should not depend on modules
– 10 sites with prod, dev and test environments
– Slightly different module version mix
Puppet is not for Everything!
– Template shellscripts are powerful
– So is Rpm/Deb/Pkg
The biggest issues are with people
– Aligning expectations
– Consensus about everything from way-of-
working to variable naming


Scaling Puppet Usage to a Global Organization

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (8)

Semelhante a Scaling Puppet Usage to a Global Organization

Semelhante a Scaling Puppet Usage to a Global Organization (20)

Mais de Puppet

Mais de Puppet (20)

Último

Último (20)

Scaling Puppet Usage to a Global Organization