1. Extensible DevSecOps
pipelines with Jenkins,
Docker, Terraform, and a
kitchen sink full of scanners
http://bit.ly/ext-devsecops-pipelines
November 9, 2019
Richard Bullington-McGuire
Principal Architect, Modus Create
richard@moduscreate.com
@obscurerichard

2. Demo
Terraform Tightrope
(environment setup kickoff)

3. The old way:
Swirling Chaotic Snowflake Hell
Checklists and POAMs and DIACAP, Oh My!
The new way: Pets vs Cattle
Infrastructure as Code to the Rescue!
What about security?
Bake that into the IaC stack too - you get DevSecOps
Why DevSecOps?

4. A security classic!
Defense in Depth
Works
Layering Security Measures
● In this presentation:
○ Check when you build
○ Check when you first deploy
● Not in this presentation but
also a good idea:
○ Check on schedule
○ Check on demand

5. About Me: My DevSecOps Experience
1995-2019: Continuously operated and defended obscure.org from attackers
2014-2017 : Used AWS and Infrastructure as Code
2014-2017: Applied DevOps approach to improve performance 10x in
hospitality system - saving the client’s reputation - .NET, SQL Server, Windows,
VMWare, JMeter load tests, New Relic monitoring
2017: Used Terraform & New Relic to migrate 14 critical systems to AWS for a
large education company - with a mandate to not make security worse.
2017: Taught real estate information software firm how to do cloud migration
right with on AWS with Terraform and CodeDeploy
2018-2019: Built out devops-infra-demo Terraform / Codedeploy

6. Terraform + CodeDeploy for DevSecOps
Code name for stack:
Corporately Deformed
(the only 2 word anagram in English for “Terraform Codedeploy”)

7. Case Study:
Corporately Deformed Stack in Education
Driven by Jenkins CI, with CIS Baseline

8. Case Study:
Terraform & New Relic & JMeter at work
Driven by Jenkins CI
● Education company cloud migration (4mo -> prod)
● Apps w/> 30,000 RPM at peak measured with New Relic
● Production with 80+ sizeable EC2 instance baseline
● Auto Scaling to 200+ instances under heavy load
● Multiple environments & accounts: dev, qa, staging, prod
● Terabyte-scale MySQL Aurora cluster, 50+ TB in S3
● Jenkins, Terraform, Ansible, Packer, CodeDeploy, JMeter
load tests, New Relic monitoring

9. Everyone is using Docker for just about everything!
Google and Netflix use containerized microservices
Great benefits: self-healing, auto-scaling,
BUT: at the cost of complexity and major refactoring (12 factor
refactoring can be a lot of factors)
It can be super hard to stuff legacy apps into containers
Containerized Microservices
(or, Fully Automated Luxury Space Communism)

10. Local Development

11. Jenkins as Orchestrator
Jenkins
Elastic Load
Balancer
EC2 Auto Scaling
Group - Web App
Terraform Provisioning
CodeDeploy Builds to S3
CodeDeploy Deployments from S3
Packer
Provision
S3

12. ● Use Packer to create machine images for the cloud
● Leverage tools such as Ansible to reduce boilerplate
● Use an image bakery pattern - consider immutable
infrastructure or a hybrid pattern.
● Use Jenkins or another CI process to drive the bakery
● During the bakery process, run security scans
THIS is how you get to DevSecOps!
Cloud Image Bakery with
Infrastructure as Code tools for repeatability

13. Run baseline scans during the image bakery process
For example:
● OpenSCAP
● Gauntlt
Make sure you have a good baseline before deployment
Scans run during baking process

14. ● Security testing framework
● Uses the Gherkin language from Cucumber
● Written in Ruby for high interop with testing tools
● Wide variety of attack adapters pre-written
● Infinitely extensible
Gauntlt

15. ● Baked into Red Hat derived systems
● Scanner is Free - though some templates are restrictively
licensed
● Pretty output
● Claims to produce remediations - but scripts are of varying
quality
● UGH - C2S profile no longer ships with CentOS! Complicates
auditing vs. CIS Baseline http://bit.ly/cisbakery :(
OpenSCAP

16. Demo
Bakery Scans:
Gauntlt & OpenSCAP

17. ● There Be Dragons In This Forest
● Some software will only install correctly before hardening
○ tmp lockdown woes
○ Selinux smackdown
● Do you want to fix all the upstream bugs in all your vendor’s
software? Maybe not!
Hardening: Before or After software install?

18. Classically, if any test fails, you fail your build
BUT….
Your tests might start out failing, especially expanding suites of
compliance tests.
Consider failing soft to start, or adding a failure count threshold
Failing soft or hard in CI: tradeoffs

19. Terraform
● Cloud-agnostic tool - not a silver bullet
● Run Terraform through Docker
● Run it via CI and you get a very powerful, auditable IaC
system
● Make sure you review the plan output before applying!
● Manual review & approval step in the CI pipeline is critical

20. CodeDeploy: packaging
● Consider using Docker as part of the build solution for your
package even if it can’t be deployed as a 12 factor app
● It’s just a zip file and a manifest and some housekeeping
scripts
● A bit of a learning curve
● A good fit for legacy apps with lots of installation and
deployment scripts

21. CodeDeploy: deploying
● Reliable lifecycle that is the same for all apps
● Some quirks you have to watch out for: heartbeat timer
● Multiple options built in for how to deploy
● Tradeoffs between fast and safe options
● Hook scripts give almost infinite flexibility on what you have
to do to deploy and validate the install before marking it
healthy
● Relies on Mutable processes, which is a weakness

22. Re-Validation in Deployment Cycle
● Often once scans get done they don’t ever get repeated
● Break this cycle by validating security essentials on every
deploy
● Challenge: preserve the scan reports if your deploy fails
○ This issue is not resolved in this repository yet

23. Demo
Deployment Scans

24. Have even more tools hooked up to the scan process
Having all scanning tools stay on the nodes after baking is not
ideal, find way to run at least some from a remote host, or install
and remove them as part of the CodeDeploy process
Get CIS baseline remediation scripts working with CodeDeploy
again, get CIS baseline pre-baked image working at all
Future Directions: better, stronger, faster

25. Conclusion
http://bit.ly/ext-devsecops-pipelines

26. Audience Questions
http://bit.ly/ext-devsecops-pipelines

27. Credit where Credit is Due
● Andy Dennis wrote the first cut of the Gauntlt integration
I could not have done this without his help!

28. Thank You!
http://bit.ly/ext-devsecops-pipelines
richard@moduscreate.com
@obscurerichard