SlideShare a Scribd company logo
1 of 57
Download to read offline
Sock Puppets: Growing your
puppet codebase

Fabio Lessa & Andrew Myers
Part 1
Empty server room
Project teams




          teams
OPS team for app
    support
The
                                   codebase
                                   we've
                                   started with
Something that represents a mess
Standard
Puppet Master setup
Code was split by environments
Workflow:

1. Push change

2. Wait for it to propagate to that
environment

3. Check it worked

If you need that applied to another
environment, go to 1
The real Workflow:
1. Push change

1.1. Stop the puppet daemon in all the critical nodes for that
environment

1.2. Run puppetd manually in one host
to ensure it works

1.3. Start the daemon again (on all the hosts you remember)

2. Wait for it to propagate to that environment

3. Check it worked (meh)

If you need that applied to another environment, go to 1 (if you
remember, that is)
Production




UAT
                   System Testing




           CI
Making changes
was painful
Part 2
"don't duplicate your manifests in your tests" Focus
  on the catalog problems he described earlier and
 test your logic. Don't test if puppet is doing it's job,
         test that your logic it's doing it's job.

                                        - Nikolay Sturm
The goal...
Before          After
So how do we test/refactor
Compile Puppet Catalogues
Compare Versions
Application specific tests
Tying it all together
Part 3
Continuous Delivery
We were moving application deploys, why
not do the same with infrastructure?
MCollective
The new setup
Picture of the pipelines
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/
puppet/modules/hostinfo/lib/facter/hostinfo.rb
info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/
puppet/modules/concat/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb
info: Loading facts in /var/lib/puppet/facts/zone.rb
info: Loading facts in /var/lib/puppet/facts/read_facts.rb
info: Loading facts in /var/lib/puppet/facts/stomp_host.rb
info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb
info: Caching catalog for somehost.nbndc.local
err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for
Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet-
environments/puppet/modules/puppetmaster/manifests/repos.pp:36
info: Retrieving plugin
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb
info: Loading facts in /var/lib/puppet/facts/zone.rb
info: Loading facts in /var/lib/puppet/facts/read_facts.rb
info: Loading facts in /var/lib/puppet/facts/stomp_host.rb
info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb
info: Caching catalog for somehost.nbndc.local
err: Failed to apply catalog: You cannot specify more than one of
content, source, target at /usr/share/puppet-recipes/noop/puppet-
environments/puppet/modules/sbs/manifests/init.pp:73
notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/
Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message:
current_value absent, should be Setting java the following java as default:
1.6.0.31 sun 1.6.0 (noop)
notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh'
from 1 events
notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered
'refresh' from 1 events
notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value
absent, should be directory (noop)
err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not
evaluate: Could not retrieve information from environment noop source(s)
puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/
share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/
manifests/init.pp:30
notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events
notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value
directory, should be absent (noop)
notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure:
current_value file, should be absent (noop)
We were moving application deploys, why
not do the same with infrastructure?
Our MCollective agent
The new action
At every push, all nodes connect to the noop
     environment and run in noop mode
Picture of a report
Change
management
process
  ?
What changed?
What did you test?
Who will do it?
Part 4
New data centre is ready!
Should be easy to migrate....
Okay so our manifests are incomplete,
but we need to be able to rebuild machines...




        Virtualisation to the rescue...
Magnets?
What happens
Wrap Up
120 pushes to
production in the
last 8 months: 2
failures
MTTR FTW!



       5 minutes!
Thank you!

More Related Content

Similar to Sock puppets

Scalable Systems Management with Puppet
Scalable Systems Management with PuppetScalable Systems Management with Puppet
Scalable Systems Management with PuppetPuppet
 
Scalable systems management with puppet
Scalable systems management with puppetScalable systems management with puppet
Scalable systems management with puppetPuppet
 
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...Puppet
 
Testing your infallibleness
Testing your infalliblenessTesting your infallibleness
Testing your infalliblenessCorey Osman
 
Puppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitPuppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitAlessandro Franceschi
 
A Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy SystemA Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy Systemadrian_nye
 
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docxCNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docxmary772
 
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEINTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEIPutuAdiPratama
 
How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)Jose Hernandez
 
K8s Quick Start
K8s Quick StartK8s Quick Start
K8s Quick StartGanesh Pol
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?Julien Pivotto
 
Puppet slides for intelligrape
Puppet slides for intelligrapePuppet slides for intelligrape
Puppet slides for intelligrapeSharad Aggarwal
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...Yury Bushmelev
 
Managing-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfManaging-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfssusera181ef
 
Bamboo Hands on training 2016
Bamboo Hands on training 2016Bamboo Hands on training 2016
Bamboo Hands on training 2016Takahiro Yamaki
 
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)Artem Zhurbila
 
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...Puppet
 
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxCreating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxwilliejgrant41084
 
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneWhen Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneDavid Glick
 

Similar to Sock puppets (20)

Scalable Systems Management with Puppet
Scalable Systems Management with PuppetScalable Systems Management with Puppet
Scalable Systems Management with Puppet
 
Scalable systems management with puppet
Scalable systems management with puppetScalable systems management with puppet
Scalable systems management with puppet
 
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...
 
Puppet quick start guide
Puppet quick start guidePuppet quick start guide
Puppet quick start guide
 
Testing your infallibleness
Testing your infalliblenessTesting your infallibleness
Testing your infallibleness
 
Puppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction KitPuppet Systems Infrastructure Construction Kit
Puppet Systems Infrastructure Construction Kit
 
A Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy SystemA Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy System
 
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docxCNS440 – Lab Assignment Week 2   Complete Lab 1 Perfo.docx
CNS440 – Lab Assignment Week 2 Complete Lab 1 Perfo.docx
 
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCEINTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
INTRODUCTION TO MACHINE LEARNING FOR MATERIALS SCIENCE
 
How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)How to Make a Honeypot Stickier (SSH*)
How to Make a Honeypot Stickier (SSH*)
 
K8s Quick Start
K8s Quick StartK8s Quick Start
K8s Quick Start
 
Do you know all of Puppet?
Do you know all of Puppet?Do you know all of Puppet?
Do you know all of Puppet?
 
Puppet slides for intelligrape
Puppet slides for intelligrapePuppet slides for intelligrape
Puppet slides for intelligrape
 
From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...From SaltStack to Puppet and beyond...
From SaltStack to Puppet and beyond...
 
Managing-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdfManaging-Splunk-with-Puppet 31-January-2022.pdf
Managing-Splunk-with-Puppet 31-January-2022.pdf
 
Bamboo Hands on training 2016
Bamboo Hands on training 2016Bamboo Hands on training 2016
Bamboo Hands on training 2016
 
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)Artem zhurbila  the story of rebuilding puppet (devops meetup 29.10.2015)
Artem zhurbila the story of rebuilding puppet (devops meetup 29.10.2015)
 
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
PuppetConf 2016: Getting to the Latest Puppet – Nate McCurdy & Elizabeth Witt...
 
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docxCreating a Scheduled Backup and Replicating System Folders Introduct.docx
Creating a Scheduled Backup and Replicating System Folders Introduct.docx
 
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting PloneWhen Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
When Good Code Goes Bad: Tools and Techniques for Troubleshooting Plone
 

Recently uploaded

Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsMiki Katsuragi
 
Training state-of-the-art general text embedding
Training state-of-the-art general text embeddingTraining state-of-the-art general text embedding
Training state-of-the-art general text embeddingZilliz
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Scott Keck-Warren
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Mattias Andersson
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst
 

Recently uploaded (20)

Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationBeyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
Vertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering TipsVertex AI Gemini Prompt Engineering Tips
Vertex AI Gemini Prompt Engineering Tips
 
Training state-of-the-art general text embedding
Training state-of-the-art general text embeddingTraining state-of-the-art general text embedding
Training state-of-the-art general text embedding
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024Advanced Test Driven-Development @ php[tek] 2024
Advanced Test Driven-Development @ php[tek] 2024
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?Are Multi-Cloud and Serverless Good or Bad?
Are Multi-Cloud and Serverless Good or Bad?
 
Human Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR SystemsHuman Factors of XR: Using Human Factors to Design XR Systems
Human Factors of XR: Using Human Factors to Design XR Systems
 

Sock puppets

  • 1. Sock Puppets: Growing your puppet codebase Fabio Lessa & Andrew Myers
  • 2.
  • 6.
  • 7. OPS team for app support
  • 8. The codebase we've started with Something that represents a mess
  • 10. Code was split by environments
  • 11.
  • 12. Workflow: 1. Push change 2. Wait for it to propagate to that environment 3. Check it worked If you need that applied to another environment, go to 1
  • 13. The real Workflow: 1. Push change 1.1. Stop the puppet daemon in all the critical nodes for that environment 1.2. Run puppetd manually in one host to ensure it works 1.3. Start the daemon again (on all the hosts you remember) 2. Wait for it to propagate to that environment 3. Check it worked (meh) If you need that applied to another environment, go to 1 (if you remember, that is)
  • 14.
  • 15. Production UAT System Testing CI
  • 18.
  • 19. "don't duplicate your manifests in your tests" Focus on the catalog problems he described earlier and test your logic. Don't test if puppet is doing it's job, test that your logic it's doing it's job. - Nikolay Sturm
  • 21. So how do we test/refactor
  • 24.
  • 26. Tying it all together
  • 29. We were moving application deploys, why not do the same with infrastructure?
  • 32. Picture of the pipelines
  • 33. info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/ puppet/modules/hostinfo/lib/facter/hostinfo.rb info: Loading facts in /usr/share/puppet-recipes/noop/puppet-environments/ puppet/modules/concat/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: Could not find dependency File[/root/.hgrc] for Mercurial::Repo::Clone[proddc2-repo] at /usr/share/puppet-recipes/noop/puppet- environments/puppet/modules/puppetmaster/manifests/repos.pp:36
  • 34. info: Retrieving plugin info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb info: Loading facts in /var/lib/puppet/lib/facter/hostinfo.rb info: Loading facts in /var/lib/puppet/facts/zone.rb info: Loading facts in /var/lib/puppet/facts/read_facts.rb info: Loading facts in /var/lib/puppet/facts/stomp_host.rb info: Loading facts in /var/lib/puppet/facts/puppetmaster_host.rb info: Caching catalog for somehost.nbndc.local err: Failed to apply catalog: You cannot specify more than one of content, source, target at /usr/share/puppet-recipes/noop/puppet- environments/puppet/modules/sbs/manifests/init.pp:73
  • 35. notice: Class[Hostinfo]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Java::Sun/Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]/ Java::Jdk::Default[1.6.0.31-sun]/Notify[java::jdk::default:sun]/message: current_value absent, should be Setting java the following java as default: 1.6.0.31 sun 1.6.0 (noop) notice: Java::Jdk::Default[1.6.0.31-sun]: Would have triggered 'refresh' from 1 events notice: Java::Jdk::Base[jdk-1.6.0.31-1jpp.1.el5]: Would have triggered 'refresh' from 1 events notice: Class[Java::Sun]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/etc/greenmail]/ensure: current_value absent, should be directory (noop) err: /Stage[main]/Greenmail/File[/etc/greenmail/greenmail.conf]: Could not evaluate: Could not retrieve information from environment noop source(s) puppet://puppet/modules/greenmail/etc/greenmail/greenmail.conf at /usr/ share/puppet-recipes/noop/puppet-environments/puppet/modules/greenmail/ manifests/init.pp:30 notice: Class[Puppetagent]: Would have triggered 'refresh' from 1 events notice: /Stage[main]/Greenmail/File[/usr/greenmail]/ensure: current_value directory, should be absent (noop) notice: /Stage[main]/Greenmail/File[/usr/greenmail/startgm.sh]/ensure: current_value file, should be absent (noop)
  • 36. We were moving application deploys, why not do the same with infrastructure?
  • 37.
  • 40. At every push, all nodes connect to the noop environment and run in noop mode
  • 41.
  • 42. Picture of a report
  • 45. What did you test?
  • 46. Who will do it?
  • 48. New data centre is ready!
  • 49. Should be easy to migrate....
  • 50. Okay so our manifests are incomplete, but we need to be able to rebuild machines... Virtualisation to the rescue...
  • 54. 120 pushes to production in the last 8 months: 2 failures
  • 55.
  • 56. MTTR FTW! 5 minutes!