My Talk for Continuous Lifecycle Confrence London 2019

1. Continuous Infrastructure
First
Kris Buytaert
@krisbuytaert

2. Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● CTO and Open Source Consultant @CTO and Open Source Consultant @inuits.euinuits.eu
● Everything is a freaking DNS ProblemEverything is a freaking DNS Problem
● Evangelizing devopsEvangelizing devops
● Organiser of #devopsdays, #cfgmgmtcamp,Organiser of #devopsdays, #cfgmgmtcamp,
#loadays, ….#loadays, ….

3. What hasWhat has
Ops ,Ops ,
automation,automation,
engineering ,engineering ,
got to do with it ?got to do with it ?

4. #devops=~C(L)AMS#devops=~C(L)AMS
● CultureCulture
● (Lean)(Lean)
● AutomationAutomation
● Monitoring and MeasurementMonitoring and Measurement
● SharingSharing
Damon Edwards and John WillisDamon Edwards and John Willis
Gene KimGene Kim

5. UmbrellaOpsUmbrellaOps

6. NoOps & YOLO OpsNoOps & YOLO Ops
● StartupStartup
● VCVC
● Exit StrategyExit Strategy
● 6-9 months6-9 months
● Actual BusinessActual Business
● Real CustomersReal Customers
● SurvivalSurvival
● 6-9 years6-9 years

7. HistoryHistory
● Europe :Europe :
•
Starting from Operations,Starting from Operations,
•
Improved Artifact Quality,Improved Artifact Quality,
•
Less pain / stabilityLess pain / stability
● US :US :
•
Push from devoplmentPush from devoplment
•
Faster PlatformsFaster Platforms
•
Faster changeFaster change

8. 4 Diferent4 Diferent TransitionTransition Cases:Cases:
Startups & MultinationalsStartups & Multinationals

9. Case 1: Chaotic OpsCase 1: Chaotic Ops
● Complete ChaosComplete Chaos
● 10% reproducability10% reproducability
● CI infra hides under a dev’s deskCI infra hides under a dev’s desk
● Ops in Debug ModeOps in Debug Mode
● No standardisationNo standardisation
● Apollo MomentApollo Moment

10. Case 1: T0+3monthsCase 1: T0+3months
● Build a Reproducable Jenkins + SlavesBuild a Reproducable Jenkins + Slaves
● CI for Puppet by OPSCI for Puppet by OPS
● Test your codeTest your code
● Promotion Stage for Infrastructure CodePromotion Stage for Infrastructure Code
● Split confg out of codeSplit confg out of code
● Keep delivering updatesKeep delivering updates

11. Case 1: T0+6 monthsCase 1: T0+6 months
● Stack AlignmentStack Alignment
•
1 jdk, 1 jboss , ...1 jdk, 1 jboss , ...
● Project Dolly :Project Dolly :
•
Puppet for everythingPuppet for everything
● 90% reproducability90% reproducability
● Standardised BuildsStandardised Builds
● Increased Test CoverageIncreased Test Coverage
● Java Developers contribute to Infra TestsJava Developers contribute to Infra Tests

12. Case 1: ConclusionCase 1: Conclusion
● Started with preparing ops folks to automateStarted with preparing ops folks to automate
● Learned the same tools developers useLearned the same tools developers use
● Developers help the ops folks to improveDevelopers help the ops folks to improve
● Collaboration + Progress ++Collaboration + Progress ++

13. Case 2: CI by DevsCase 2: CI by Devs
● Some devs have testsSome devs have tests
● Some dev teams have “CI”Some dev teams have “CI”
● Deployments are ChaosDeployments are Chaos
● Ops nags about Artifact QualityOps nags about Artifact Quality

14. Case 2: T0+18 monthsCase 2: T0+18 months
● Found the frst ops skills in the orgFound the frst ops skills in the org
● Mostly overworked Brent’sMostly overworked Brent’s
● Move them out of their ofcesMove them out of their ofces
● Focus teamFocus team
● Teach AgileTeach Agile
● Adopt IAC (puppet)Adopt IAC (puppet)
● First SuccessesFirst Successes
● Move people back to teamsMove people back to teams

15. Case 2: T0+24 monthsCase 2: T0+24 months
● Grey Beard Ops person has converted to AgileGrey Beard Ops person has converted to Agile
EvangelistEvangelist
● Preaches Kanban (for ops) and ScrumPreaches Kanban (for ops) and Scrum
● Writes Test for his CodeWrites Test for his Code
● Coaches developers to achieve CI/CDCoaches developers to achieve CI/CD

16. Case 2: ConclusionsCase 2: Conclusions
● Starting with dev delayed the collaboration forStarting with dev delayed the collaboration for
1+ year1+ year
● Ops were fre fghting and not involvedOps were fre fghting and not involved
● Once ops resources were dedicatedOnce ops resources were dedicated
collaboration and quality improvement startedcollaboration and quality improvement started
to happento happen

17. Case 3: Countries FightCase 3: Countries Fight
● Country 1 has full IAC with CICountry 1 has full IAC with CI
● Country 2 has no IAC (but claims they do), lowCountry 2 has no IAC (but claims they do), low
test coverage and manual deploystest coverage and manual deploys
● Country 3 has to deploy in country 2 , extremeCountry 3 has to deploy in country 2 , extreme
pain is the result.pain is the result.
● Once it has been proven that Country 2 reallyOnce it has been proven that Country 2 really
doesn’t have IAC , Country 3 is allowed todoesn’t have IAC , Country 3 is allowed to
deploy in Country 1. Things smooth outdeploy in Country 1. Things smooth out

18. Case 3: ConclusionsCase 3: Conclusions
You might be able to fake automation tillYou might be able to fake automation till
someone who has automation shows upsomeone who has automation shows up

19. Case 4:Ops NOT involvedCase 4:Ops NOT involved
● Large TransformationLarge Transformation
● ““devops” team dictates tools they have neverdevops” team dictates tools they have never
used them selvesused them selves
● Tools they as a team don’t need themselvesTools they as a team don’t need themselves
● Developers complain about unusable toolsDevelopers complain about unusable tools
● Developers complain about broken toolsDevelopers complain about broken tools
● Tools enforce a manual processTools enforce a manual process

20. Case 4: 2 years laterCase 4: 2 years later
● Average “devops” role stays for 2 months ,Average “devops” role stays for 2 months ,
then leavesthen leaves
● Senior IT management has left (2x)Senior IT management has left (2x)
● Only In house analysts remainOnly In house analysts remain
● Mostly contract based developersMostly contract based developers
● Failing Cloud StrategyFailing Cloud Strategy
● Legacy Container EcosystemLegacy Container Ecosystem

21. Case ConclusionsCase Conclusions
Earlier involvement of the Ops Skills createsEarlier involvement of the Ops Skills creates
higher success rates , and smoother adoptionhigher success rates , and smoother adoption

22. Change Management &Change Management &
ResistanceResistance
● 20 – 60 – 2020 – 60 – 20
● 20% Early adopters , fnd your peers20% Early adopters , fnd your peers
● 60% Will wait , but adopt slowly60% Will wait , but adopt slowly
● 20%20%

23. Why ops frst ?Why ops frst ?
● You can’t support / understand what you don’tYou can’t support / understand what you don’t
do yourselve.do yourselve.
● Code = CodeCode = Code
● Unblock deliveryUnblock delivery
● Unblock provisioningUnblock provisioning
● Metrics & Monitoring Build inMetrics & Monitoring Build in

24. On Prem vs CloudnativeOn Prem vs Cloudnative
● Puppet, Chef, AnsiblePuppet, Chef, Ansible
● VPN, FirewallsVPN, Firewalls
● Pipelines & PipelinesPipelines & Pipelines
● StandardisationStandardisation
● SecuritySecurity
● MonitoringMonitoring
● Terraform, Cloudinit,Terraform, Cloudinit,
● VPC, SGVPC, SG
● Pipelines & PipelinesPipelines & Pipelines
● StandardisationStandardisation
● SecuritySecurity
● MonitoringMonitoring

25. Container NaiveContainer Naive

26. Dev OoopsDev Ooops #container edition#container edition
● ““Put this Code Live, here's a DockerPut this Code Live, here's a Docker
Image ”Image ”
● No machines available ?No machines available ?
● What database ? Where to store theWhat database ? Where to store the
data ?data ?
● Security ? What distro is this even ?Security ? What distro is this even ?
Bad Cows ?Bad Cows ?
● How do we monitor his ?How do we monitor his ?
● Backups ?Backups ?
● How did you build this ?How did you build this ?
● Has DockerHub been hacked again ?Has DockerHub been hacked again ?

27. Culture vs ToolsCulture vs Tools

28. Waterfall & ITIL vs AgileWaterfall & ITIL vs Agile
● Ops has been doing waterfall projects forOps has been doing waterfall projects for
decadesdecades
● They need time to adaptThey need time to adapt
● Time to UnlearnTime to Unlearn

29. A CI EcosystemA CI Ecosystem
● Version ControlVersion Control
● DeploymentDeployment
● Build ToolingBuild Tooling
● Artifcact RepositoryArtifcact Repository
● Code Coverage ToolingCode Coverage Tooling
● Testing ToTesting To
How many of those tools is your average opsHow many of those tools is your average ops
person used to use ?person used to use ?

30. Understanding CIUnderstanding CI
● Understanding Dependency Management,Understanding Dependency Management,
Testing, Packaging , Deploying =>Testing, Packaging , Deploying =>
•
Same LanguageSame Language
•
Same ToolsSame Tools
•
Same PainSame Pain

31. People Argument against CIPeople Argument against CI
•Setting up the stack costs timeSetting up the stack costs time
•U don’t have testsU don’t have tests
•Thats why you are still runningThats why you are still running
vulnerable security publically !vulnerable security publically !
•Operations and development areOperations and development are
diferent budgetsdiferent budgets
•One shot projects , fre and forgetOne shot projects , fre and forget

32. Typical Dev FirstTypical Dev First
● Dev lacks environments or uses wrong stackDev lacks environments or uses wrong stack
● Dev can’t deploy yetDev can’t deploy yet
● No PipelinesNo Pipelines
•
Technical debt is createdTechnical debt is created
● Ops is catching up with envsOps is catching up with envs
● NFR get ignoredNFR get ignored
● Brent stays a heroBrent stays a hero

33. This is not Continuous DeploymentThis is not Continuous Deployment
•@stahnma@stahnma
@#devopsdays Ohio@#devopsdays Ohio

34. Culture Hack:Culture Hack:
Set up CI/CD for your CI/CD infrastructureSet up CI/CD for your CI/CD infrastructure
frst, If the people running your infra don'tfrst, If the people running your infra don't
know how CI/CD works , how do youknow how CI/CD works , how do you
expect them to support / teach yourexpect them to support / teach your
application teams ?application teams ?

35. Sprint 0Sprint 0
● Onboard the teamOnboard the team
● Hello World from source to prodHello World from source to prod
● Keep the pipeline greenKeep the pipeline green
● Create basic monitoring / healthCreate basic monitoring / health
● Keep it healthyKeep it healthy

36. Security ?Security ?

37. How do we get fromHow do we get from
We don’t dare to patchWe don’t dare to patch
ToTo
All systems green , lets goAll systems green , lets go

38. Continuous Delivery is aContinuous Delivery is a
Security RequirementSecurity Requirement

39. MTTR ~> 0MTTR ~> 0

40. It's not about the toolsIt's not about the tools
It's about changeIt's about change
It's about the peopleIt's about the people

41. Devopsdays Gent 2019Devopsdays Gent 2019
29-30 october 201929-30 october 2019
Vooruit GhentVooruit Ghent
https://devopsdays.org/https://devopsdays.org/
events/2019-ghent/events/2019-ghent/
Early Bird = OPENEarly Bird = OPEN
CFP = OPENCFP = OPEN

42. ContactContact
Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be
Further ReadingFurther Reading
@krisbuytaert@krisbuytaert
http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/
http://www.inuits.be/http://www.inuits.be/
InuitsInuits
Essensteenweg 31Essensteenweg 31
BrasschaatBrasschaat
BelgiumBelgium
891.514.231891.514.231
+32 475 961221+32 475 961221