Talk at Icinga Camp Amsterdam 2016 by Kris Buytaert

1. On theimportanceof
InfrastructureasCode
Kris Buytaert
@krisbuytaert

2. Kris BuytaertKris Buytaert● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source
Consultant @Consultant @inuits.euinuits.eu
● Everything is an effing DNS ProblemEverything is an effing DNS Problem
● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore
● Some books, some papers, some blogsSome books, some papers, some blogs
● Evangelizing devopsEvangelizing devops
● Organiser of #devopsdays, #cfgmgmtcamp,Organiser of #devopsdays, #cfgmgmtcamp,
#loadays, ….#loadays, ….
● Part of the travelling geek circusPart of the travelling geek circus

3. What's this devopsWhat's this devops
thing anyhow ?thing anyhow ?

4. C(L)AMSC(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● MeasurementMeasurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim

5. Why automate ?Why automate ?

6. Common ProblemsCommon Problems
● Many manual changes to systems
● Many undocumented changes
● Emergency Administration only
● Disaster Recovery site is a Disaster
● Time to deliver a box is to slow
● All boxen are different
● Computers don’t work hard enough for us

7. More ProblemsMore Problems
● How long does it take to reinstall a machine from 0
● To the exact same point as before ?
● With different Hardware ? In a different cloud ?
● What about your (customer/personal data )

8. Security ?Security ?
● Monitoring that your platform hasn't changed.Monitoring that your platform hasn't changed.
•
Why is selinux disabled ?Why is selinux disabled ?
•
Who added / dropped that firewall ?Who added / dropped that firewall ?
•
What did this originally look like ?What did this originally look like ?
•
Is this file really what Bernd meant it to be ?Is this file really what Bernd meant it to be ?

9. #monitoringsucks#monitoringsucks
● Monitoring is out of sync with realityMonitoring is out of sync with reality
● Managed manuallyManaged manually
● Can't keep up..Can't keep up..

10. Do you want to ?Do you want to ?
● Install these racks manuallyInstall these racks manually
● Over and over again ?Over and over again ?
● And can you guarantee that installs areAnd can you guarantee that installs are
identical ?identical ?
● ““No simple admin taks is fun more thanNo simple admin taks is fun more than
twice”twice”
● s/twice/once/g;s/twice/once/g;
● Repeating installs are boring and prone toRepeating installs are boring and prone to
errorserrors
● Each installation is unintentionally UniqueEach installation is unintentionally Unique
● Manual installs DO NOT scaleManual installs DO NOT scale

11. ChallengesChallenges
● ReproducabilityReproducability
● SpeedSpeed
● AuditingAuditing
● Keeping stuff in syncKeeping stuff in sync
•
MonitoringMonitoring
•
SecuritySecurity
•
BackupBackup

12. The 10The 10thth
floor testfloor test
● Grab a random machine (don’t take a backup before)
● Throw it out a 10th
floor window
● Can you recover it in 10 minutes ?

13. Facts!Facts!
● Data Backup is only a part
● Sysadmin backup needs to be done
also
● Manual Installations = bad
● Bad installations = unusable
infrastructure
● Bad installations = unproductive users
● Bad installations = manual efforts
● Manual efforts = no time
● No time = no updates no patches no
security
● Manual work = high costs

14. Deploying an InfrastructureDeploying an Infrastructure
● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue2001 : Mondo rescue (reproducable single instances)(reproducable single instances)
● 2003 : SystemImager2003 : SystemImager
•
Reproducable Infrastructure , withReproducable Infrastructure , with
“OVERRIDES”“OVERRIDES”
•
Fast Multicast Image deploymentsFast Multicast Image deployments
•
Image Sprawl (thank you VMware)Image Sprawl (thank you VMware)

15. Deploying an InfrastructureDeploying an Infrastructure
● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue2001 : Mondo rescue
● 2003 : SystemImager2003 : SystemImager
● 2005 :2005 : Kickstart / FAIKickstart / FAI
•
Dreaming of Jeos + IAC (Cfengine)Dreaming of Jeos + IAC (Cfengine)

16. Deploying an InfrastructureDeploying an Infrastructure
● 1996 : Manual Installations1996 : Manual Installations
● 2001 : Mondo rescue2001 : Mondo rescue
● 2003 : SystemImager2003 : SystemImager
● 2005 : Dreaming of Jeos + IAC2005 : Dreaming of Jeos + IAC
● 2008 : Actual JeOS + IAC2008 : Actual JeOS + IAC
● 2010 : Vagrant for development2010 : Vagrant for development

17. Imagesprawl ANDImagesprawl AND
SnowflakesSnowflakes
● Image Sparwl :Image Sparwl :
•
Copy vm 3xCopy vm 3x
•
Modify 2xModify 2x
•
Copy 21xCopy 21x
•
How the Heck did we get here ?How the Heck did we get here ?
● SnowFlakes :SnowFlakes :
•
Don't touch this box it might breakDon't touch this box it might break
•
Look how nice it is !Look how nice it is !

18. You never deployYou never deploy
something “just” oncesomething “just” once
● Local test … experiment,Local test … experiment,
•
Vagrant box / local containersVagrant box / local containers
● Integration PlatformIntegration Platform
•
Same codebase,, different environmentSame codebase,, different environment
● Dev/ UAT/ Prod / DR …Dev/ UAT/ Prod / DR …
● Or your customer just forgot to renew the leaseOr your customer just forgot to renew the lease
on his VPS. #toldyousoon his VPS. #toldyouso

19. What's different in the cloud ?What's different in the cloud ?
● ScaleScale
● VelocityVelocity
● ChangeChange

20. Your machines as CattleYour machines as Cattle

21. Treat your people as petsTreat your people as pets

22. Configuration MgmtConfiguration Mgmt
● Configure 1000 nodes,Configure 1000 nodes,
● Modify 15000 files,Modify 15000 files,
● Think :Think :
•Cfengine,Puppet, Chef, SaltCfengine,Puppet, Chef, Salt
● Put configs under version controlPut configs under version control
● Please don't roll your own ...Please don't roll your own ...

23. Infrastructure as CodeInfrastructure as Code
● Treat configuration automation as codeTreat configuration automation as code
● Development best practicesDevelopment best practices
•
Model your infrastructureModel your infrastructure
•
Version your cookbooks / manifestsVersion your cookbooks / manifests
•
Test your cookbooks/ manifestsTest your cookbooks/ manifests
•
Dev/ test /uat / prod for your infraDev/ test /uat / prod for your infra
● Model your infrastructureModel your infrastructure
● A working service = automated ( Application Code + InfrastructureA working service = automated ( Application Code + Infrastructure
Code + Security + Monitoring )Code + Security + Monitoring )
● IAC -ne scripting (or translating bash to yaml)IAC -ne scripting (or translating bash to yaml)

24. IAC Is a TestingIAC Is a Testing
RequirementRequirement
● Stable reproducable starting pointStable reproducable starting point

25. AuditabilityAuditability
● git loggit log
● git blamegit blame
● Review,Review,
● authorizationauthorization

26. File monitoringFile monitoring

27. Fixing Monitoring FatigueFixing Monitoring Fatigue

28. Stored ConfigsStored Configs

29. Collection and ExportCollection and Export
Export :Export :
@@resource {@@resource {
... }... }
Collect:Collect:
Resource <<| query |Resource <<| query |
>>>>
Clean out nodes that dissapearClean out nodes that dissapear
puppet node cleanpuppet node clean

30. Use Cases:Use Cases:
● Ssh keysSsh keys
● Reverse proxy configsReverse proxy configs
● Monitoring resourcesMonitoring resources
● Measuring resourcesMeasuring resources

31. Puppetmaster Example:Puppetmaster Example:

32. Defining a ServiceDefining a Service
● profile that :profile that :
•
Configures service using a standardConfigures service using a standard
module call with hiera based parametersmodule call with hiera based parameters
•
Configures BackupConfigures Backup
•
Configures logrotationConfigures logrotation
•
Configures logshippingConfigures logshipping
•
Exports Monitoring NeedsExports Monitoring Needs

33. Chronicle of a failedChronicle of a failed
private cloudprivate cloud● Tool X provisions a VMTool X provisions a VM
•
3 weeks from the request / can only be done by 1 team3 weeks from the request / can only be done by 1 team
● Tool Y installs patchesTool Y installs patches
•
2 weeks2 weeks
● Team Z installs backupTeam Z installs backup
•
1 day1 day
● Team A installs monitoringTeam A installs monitoring
•
3 weeks3 weeks
● AppApp
•
Manual deploy on wrong JVM, return to senderManual deploy on wrong JVM, return to sender

34. Application IncludedApplication Included
● Application =Application =
•
PackagePackage
•
ConfigConfig
•
ServiceService
● No manual scriptingNo manual scripting
● Think about your bootstrapping / scaleoutThink about your bootstrapping / scaleout

35. Automation ofAutomation of
#monitoring#monitoring
brought backbrought back
thethe #love#love

36. ConclusionConclusion
● IAC solves a lot of problemsIAC solves a lot of problems
•
Improves SecurityImproves Security
•
Creates Monitoring LoveCreates Monitoring Love
•
Creates SpeedCreates Speed
● But it still is code, and needs to be treated likeBut it still is code, and needs to be treated like
code !code !

37. ContactContact
Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Essensteenweg 31Essensteenweg 31
BrasschaatBrasschaat
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221