Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitchen sink full of scanners

Richard Bullington-McGuire
Richard Bullington-McGuirePrincipal Architect em Modus Create, Inc
Extensible DevSecOps
pipelines with Jenkins,
Docker, Terraform, and a
kitchen sink full of scanners
http://bit.ly/ext-devsecops-pipelines
November 9, 2019
Richard Bullington-McGuire
Principal Architect, Modus Create
richard@moduscreate.com
@obscurerichard
Demo
Terraform Tightrope
(environment setup kickoff)
The old way:
Swirling Chaotic Snowflake Hell
Checklists and POAMs and DIACAP, Oh My!
The new way: Pets vs Cattle
Infrastructure as Code to the Rescue!
What about security?
Bake that into the IaC stack too - you get DevSecOps
Why DevSecOps?
A security classic!
Defense in Depth
Works
Layering Security Measures
● In this presentation:
○ Check when you build
○ Check when you first deploy
● Not in this presentation but
also a good idea:
○ Check on schedule
○ Check on demand
About Me: My DevSecOps Experience
1995-2019: Continuously operated and defended obscure.org from attackers
2014-2017 : Used AWS and Infrastructure as Code
2014-2017: Applied DevOps approach to improve performance 10x in
hospitality system - saving the client’s reputation - .NET, SQL Server, Windows,
VMWare, JMeter load tests, New Relic monitoring
2017: Used Terraform & New Relic to migrate 14 critical systems to AWS for a
large education company - with a mandate to not make security worse.
2017: Taught real estate information software firm how to do cloud migration
right with on AWS with Terraform and CodeDeploy
2018-2019: Built out devops-infra-demo Terraform / Codedeploy
Terraform + CodeDeploy for DevSecOps
Code name for stack:
Corporately Deformed
(the only 2 word anagram in English for “Terraform Codedeploy”)
Case Study:
Corporately Deformed Stack in Education
Driven by Jenkins CI, with CIS Baseline
Case Study:
Terraform & New Relic & JMeter at work
Driven by Jenkins CI
● Education company cloud migration (4mo -> prod)
● Apps w/> 30,000 RPM at peak measured with New Relic
● Production with 80+ sizeable EC2 instance baseline
● Auto Scaling to 200+ instances under heavy load
● Multiple environments & accounts: dev, qa, staging, prod
● Terabyte-scale MySQL Aurora cluster, 50+ TB in S3
● Jenkins, Terraform, Ansible, Packer, CodeDeploy, JMeter
load tests, New Relic monitoring
Everyone is using Docker for just about everything!
Google and Netflix use containerized microservices
Great benefits: self-healing, auto-scaling,
BUT: at the cost of complexity and major refactoring (12 factor
refactoring can be a lot of factors)
It can be super hard to stuff legacy apps into containers
Containerized Microservices
(or, Fully Automated Luxury Space Communism)
Local Development
Jenkins as Orchestrator
Jenkins
Elastic Load
Balancer
EC2 Auto Scaling
Group - Web App
Terraform Provisioning
CodeDeploy Builds to S3
CodeDeploy Deployments from S3
Packer
Provision
S3
● Use Packer to create machine images for the cloud
● Leverage tools such as Ansible to reduce boilerplate
● Use an image bakery pattern - consider immutable
infrastructure or a hybrid pattern.
● Use Jenkins or another CI process to drive the bakery
● During the bakery process, run security scans
THIS is how you get to DevSecOps!
Cloud Image Bakery with
Infrastructure as Code tools for repeatability
Run baseline scans during the image bakery process
For example:
● OpenSCAP
● Gauntlt
Make sure you have a good baseline before deployment
Scans run during baking process
● Security testing framework
● Uses the Gherkin language from Cucumber
● Written in Ruby for high interop with testing tools
● Wide variety of attack adapters pre-written
● Infinitely extensible
Gauntlt
● Baked into Red Hat derived systems
● Scanner is Free - though some templates are restrictively
licensed
● Pretty output
● Claims to produce remediations - but scripts are of varying
quality
● UGH - C2S profile no longer ships with CentOS! Complicates
auditing vs. CIS Baseline http://bit.ly/cisbakery :(
OpenSCAP
Demo
Bakery Scans:
Gauntlt & OpenSCAP
● There Be Dragons In This Forest
● Some software will only install correctly before hardening
○ tmp lockdown woes
○ Selinux smackdown
● Do you want to fix all the upstream bugs in all your vendor’s
software? Maybe not!
Hardening: Before or After software install?
Classically, if any test fails, you fail your build
BUT….
Your tests might start out failing, especially expanding suites of
compliance tests.
Consider failing soft to start, or adding a failure count threshold
Failing soft or hard in CI: tradeoffs
Terraform
● Cloud-agnostic tool - not a silver bullet
● Run Terraform through Docker
● Run it via CI and you get a very powerful, auditable IaC
system
● Make sure you review the plan output before applying!
● Manual review & approval step in the CI pipeline is critical
CodeDeploy: packaging
● Consider using Docker as part of the build solution for your
package even if it can’t be deployed as a 12 factor app
● It’s just a zip file and a manifest and some housekeeping
scripts
● A bit of a learning curve
● A good fit for legacy apps with lots of installation and
deployment scripts
CodeDeploy: deploying
● Reliable lifecycle that is the same for all apps
● Some quirks you have to watch out for: heartbeat timer
● Multiple options built in for how to deploy
● Tradeoffs between fast and safe options
● Hook scripts give almost infinite flexibility on what you have
to do to deploy and validate the install before marking it
healthy
● Relies on Mutable processes, which is a weakness
Re-Validation in Deployment Cycle
● Often once scans get done they don’t ever get repeated
● Break this cycle by validating security essentials on every
deploy
● Challenge: preserve the scan reports if your deploy fails
○ This issue is not resolved in this repository yet
Demo
Deployment Scans
Have even more tools hooked up to the scan process
Having all scanning tools stay on the nodes after baking is not
ideal, find way to run at least some from a remote host, or install
and remove them as part of the CodeDeploy process
Get CIS baseline remediation scripts working with CodeDeploy
again, get CIS baseline pre-baked image working at all
Future Directions: better, stronger, faster
Conclusion
http://bit.ly/ext-devsecops-pipelines
Audience Questions
http://bit.ly/ext-devsecops-pipelines
Credit where Credit is Due
● Andy Dennis wrote the first cut of the Gauntlt integration
I could not have done this without his help!
Thank You!
http://bit.ly/ext-devsecops-pipelines
richard@moduscreate.com
@obscurerichard
1 de 28

Recomendados

infrastructure as codeinfrastructure as code
infrastructure as codeAmazon Web Services
4.3K visualizações72 slides
Container SecurityContainer Security
Container SecurityAmazon Web Services
608 visualizações36 slides

Mais conteúdo relacionado

Mais procurados(20)

Devops as a serviceDevops as a service
Devops as a service
Saravanan Subburayal1.1K visualizações
Introduction to MicroservicesIntroduction to Microservices
Introduction to Microservices
Amazon Web Services66.8K visualizações
Migrating to the CloudMigrating to the Cloud
Migrating to the Cloud
Amazon Web Services3.6K visualizações
DevOps, Common use cases, Architectures, Best PracticesDevOps, Common use cases, Architectures, Best Practices
DevOps, Common use cases, Architectures, Best Practices
Shiva Narayanaswamy17.4K visualizações
Docker & kubernetesDocker & kubernetes
Docker & kubernetes
NexThoughts Technologies6.3K visualizações
Cloud native principlesCloud native principles
Cloud native principles
Diego Pacheco2.2K visualizações
Microsoft Azure Technical OverviewMicrosoft Azure Technical Overview
Microsoft Azure Technical Overview
gjuljo12.8K visualizações
DevSecOps reference architectures 2018DevSecOps reference architectures 2018
DevSecOps reference architectures 2018
Sonatype 10.1K visualizações
AWS Tagging StrategyAWS Tagging Strategy
AWS Tagging Strategy
Shiva Narayanaswamy7.4K visualizações
DevOps on AWSDevOps on AWS
DevOps on AWS
Amazon Web Services3.3K visualizações
CAF presentation 09 16-2020CAF presentation 09 16-2020
CAF presentation 09 16-2020
Michael Nichols1.7K visualizações
TerraformTerraform
Terraform
Otto Jongerius775 visualizações
Infrastructure as CodeInfrastructure as Code
Infrastructure as Code
Robert Greiner4.8K visualizações
Successfully Implementing DEV-SEC-OPS in the CloudSuccessfully Implementing DEV-SEC-OPS in the Cloud
Successfully Implementing DEV-SEC-OPS in the Cloud
Amazon Web Services934 visualizações

Similar a Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitchen sink full of scanners

Docker for devDocker for dev
Docker for devErik Talboom
184 visualizações45 slides
DockerCon 15 Keynote - Day 2DockerCon 15 Keynote - Day 2
DockerCon 15 Keynote - Day 2Docker, Inc.
20.2K visualizações97 slides
CI/CD on Google Cloud PlatformCI/CD on Google Cloud Platform
CI/CD on Google Cloud PlatformDevOps Indonesia
4.5K visualizações21 slides

Similar a Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitchen sink full of scanners(20)

Docker for devDocker for dev
Docker for dev
Erik Talboom184 visualizações
Successful DevOps implementation for small teams  a true storySuccessful DevOps implementation for small teams  a true story
Successful DevOps implementation for small teams a true story
Jakub Paweł Głazik189 visualizações
DockerCon 15 Keynote - Day 2DockerCon 15 Keynote - Day 2
DockerCon 15 Keynote - Day 2
Docker, Inc.20.2K visualizações
CI/CD on Google Cloud PlatformCI/CD on Google Cloud Platform
CI/CD on Google Cloud Platform
DevOps Indonesia4.5K visualizações
An Introduction to DockerAn Introduction to Docker
An Introduction to Docker
bwinterton537 visualizações
Cloud Native Dünyada CI/CDCloud Native Dünyada CI/CD
Cloud Native Dünyada CI/CD
Mustafa AKIN354 visualizações
Deploying software at ScaleDeploying software at Scale
Deploying software at Scale
Kris Buytaert2.5K visualizações
ContainerCon - Test Driven InfrastructureContainerCon - Test Driven Infrastructure
ContainerCon - Test Driven Infrastructure
Yury Tsarev2.6K visualizações
Achieving Full Stack DevOps at Colonial Life Achieving Full Stack DevOps at Colonial Life
Achieving Full Stack DevOps at Colonial Life
DevOps.com227 visualizações
Run automated tests in DockerRun automated tests in Docker
Run automated tests in Docker
Oleksandr Metelytsia131 visualizações
Truemotion Adventures in ContainerizationTruemotion Adventures in Containerization
Truemotion Adventures in Containerization
Ryan Hunter200 visualizações

Último(20)

PyCon ID 2023 - Ridwan Fadjar Septian.pdfPyCon ID 2023 - Ridwan Fadjar Septian.pdf
PyCon ID 2023 - Ridwan Fadjar Septian.pdf
Ridwan Fadjar161 visualizações
Micron CXL product and architecture updateMicron CXL product and architecture update
Micron CXL product and architecture update
CXL Forum21 visualizações
AMD: 4th Generation EPYC CXL DemoAMD: 4th Generation EPYC CXL Demo
AMD: 4th Generation EPYC CXL Demo
CXL Forum113 visualizações
ThroughputThroughput
Throughput
Moisés Armani Ramírez25 visualizações
Liqid: Composable CXL PreviewLiqid: Composable CXL Preview
Liqid: Composable CXL Preview
CXL Forum114 visualizações
.conf Go 2023 - SIEM project @ SNF.conf Go 2023 - SIEM project @ SNF
.conf Go 2023 - SIEM project @ SNF
Splunk134 visualizações
Spesifikasi Lengkap ASUS Vivobook Go 14Spesifikasi Lengkap ASUS Vivobook Go 14
Spesifikasi Lengkap ASUS Vivobook Go 14
Dot Semarang30 visualizações
ChatGPT and AI for Web DevelopersChatGPT and AI for Web Developers
ChatGPT and AI for Web Developers
Maximiliano Firtman143 visualizações
Java 21 and Beyond- A Roadmap of Innovations  .pdfJava 21 and Beyond- A Roadmap of Innovations  .pdf
Java 21 and Beyond- A Roadmap of Innovations .pdf
Ana-Maria Mihalceanu49 visualizações
[2023] Putting the R! in R&D.pdf[2023] Putting the R! in R&D.pdf
[2023] Putting the R! in R&D.pdf
Eleanor McHugh31 visualizações

Extensible dev secops pipelines with Jenkins, Docker, Terraform, and a kitchen sink full of scanners

  • 1. Extensible DevSecOps pipelines with Jenkins, Docker, Terraform, and a kitchen sink full of scanners http://bit.ly/ext-devsecops-pipelines November 9, 2019 Richard Bullington-McGuire Principal Architect, Modus Create richard@moduscreate.com @obscurerichard
  • 3. The old way: Swirling Chaotic Snowflake Hell Checklists and POAMs and DIACAP, Oh My! The new way: Pets vs Cattle Infrastructure as Code to the Rescue! What about security? Bake that into the IaC stack too - you get DevSecOps Why DevSecOps?
  • 4. A security classic! Defense in Depth Works Layering Security Measures ● In this presentation: ○ Check when you build ○ Check when you first deploy ● Not in this presentation but also a good idea: ○ Check on schedule ○ Check on demand
  • 5. About Me: My DevSecOps Experience 1995-2019: Continuously operated and defended obscure.org from attackers 2014-2017 : Used AWS and Infrastructure as Code 2014-2017: Applied DevOps approach to improve performance 10x in hospitality system - saving the client’s reputation - .NET, SQL Server, Windows, VMWare, JMeter load tests, New Relic monitoring 2017: Used Terraform & New Relic to migrate 14 critical systems to AWS for a large education company - with a mandate to not make security worse. 2017: Taught real estate information software firm how to do cloud migration right with on AWS with Terraform and CodeDeploy 2018-2019: Built out devops-infra-demo Terraform / Codedeploy
  • 6. Terraform + CodeDeploy for DevSecOps Code name for stack: Corporately Deformed (the only 2 word anagram in English for “Terraform Codedeploy”)
  • 7. Case Study: Corporately Deformed Stack in Education Driven by Jenkins CI, with CIS Baseline
  • 8. Case Study: Terraform & New Relic & JMeter at work Driven by Jenkins CI ● Education company cloud migration (4mo -> prod) ● Apps w/> 30,000 RPM at peak measured with New Relic ● Production with 80+ sizeable EC2 instance baseline ● Auto Scaling to 200+ instances under heavy load ● Multiple environments & accounts: dev, qa, staging, prod ● Terabyte-scale MySQL Aurora cluster, 50+ TB in S3 ● Jenkins, Terraform, Ansible, Packer, CodeDeploy, JMeter load tests, New Relic monitoring
  • 9. Everyone is using Docker for just about everything! Google and Netflix use containerized microservices Great benefits: self-healing, auto-scaling, BUT: at the cost of complexity and major refactoring (12 factor refactoring can be a lot of factors) It can be super hard to stuff legacy apps into containers Containerized Microservices (or, Fully Automated Luxury Space Communism)
  • 11. Jenkins as Orchestrator Jenkins Elastic Load Balancer EC2 Auto Scaling Group - Web App Terraform Provisioning CodeDeploy Builds to S3 CodeDeploy Deployments from S3 Packer Provision S3
  • 12. ● Use Packer to create machine images for the cloud ● Leverage tools such as Ansible to reduce boilerplate ● Use an image bakery pattern - consider immutable infrastructure or a hybrid pattern. ● Use Jenkins or another CI process to drive the bakery ● During the bakery process, run security scans THIS is how you get to DevSecOps! Cloud Image Bakery with Infrastructure as Code tools for repeatability
  • 13. Run baseline scans during the image bakery process For example: ● OpenSCAP ● Gauntlt Make sure you have a good baseline before deployment Scans run during baking process
  • 14. ● Security testing framework ● Uses the Gherkin language from Cucumber ● Written in Ruby for high interop with testing tools ● Wide variety of attack adapters pre-written ● Infinitely extensible Gauntlt
  • 15. ● Baked into Red Hat derived systems ● Scanner is Free - though some templates are restrictively licensed ● Pretty output ● Claims to produce remediations - but scripts are of varying quality ● UGH - C2S profile no longer ships with CentOS! Complicates auditing vs. CIS Baseline http://bit.ly/cisbakery :( OpenSCAP
  • 17. ● There Be Dragons In This Forest ● Some software will only install correctly before hardening ○ tmp lockdown woes ○ Selinux smackdown ● Do you want to fix all the upstream bugs in all your vendor’s software? Maybe not! Hardening: Before or After software install?
  • 18. Classically, if any test fails, you fail your build BUT…. Your tests might start out failing, especially expanding suites of compliance tests. Consider failing soft to start, or adding a failure count threshold Failing soft or hard in CI: tradeoffs
  • 19. Terraform ● Cloud-agnostic tool - not a silver bullet ● Run Terraform through Docker ● Run it via CI and you get a very powerful, auditable IaC system ● Make sure you review the plan output before applying! ● Manual review & approval step in the CI pipeline is critical
  • 20. CodeDeploy: packaging ● Consider using Docker as part of the build solution for your package even if it can’t be deployed as a 12 factor app ● It’s just a zip file and a manifest and some housekeeping scripts ● A bit of a learning curve ● A good fit for legacy apps with lots of installation and deployment scripts
  • 21. CodeDeploy: deploying ● Reliable lifecycle that is the same for all apps ● Some quirks you have to watch out for: heartbeat timer ● Multiple options built in for how to deploy ● Tradeoffs between fast and safe options ● Hook scripts give almost infinite flexibility on what you have to do to deploy and validate the install before marking it healthy ● Relies on Mutable processes, which is a weakness
  • 22. Re-Validation in Deployment Cycle ● Often once scans get done they don’t ever get repeated ● Break this cycle by validating security essentials on every deploy ● Challenge: preserve the scan reports if your deploy fails ○ This issue is not resolved in this repository yet
  • 24. Have even more tools hooked up to the scan process Having all scanning tools stay on the nodes after baking is not ideal, find way to run at least some from a remote host, or install and remove them as part of the CodeDeploy process Get CIS baseline remediation scripts working with CodeDeploy again, get CIS baseline pre-baked image working at all Future Directions: better, stronger, faster
  • 27. Credit where Credit is Due ● Andy Dennis wrote the first cut of the Gauntlt integration I could not have done this without his help!