Development methodologies like Agile teach developers to make small and quick improvements to code to get features into production quickly. SAAS and web scale companies now push updated code to production using tools such as Jenkins and Chef more than ten times a day. This means that while we get updated features and software delivered faster, there is a side effect of security issues being introduced into production quicker than ever. In this talk we’ll explore why security professionals should embrace the very same DevOps tools like Jenkins, Chef, Puppet, etc. to automate security checks and remediation on code and automated infrastructure.
2. What are you talking about?
● What hell are Devops, Agile, and
$INSERT_BUZZWORD_HERE
● What kind of challenges do these methodologies present to
security folks
● How to use these tools to your advantage
○ Config management suites 101
○ Automated security scans when code is committed
● Cats
3. Who are you anyway?
● Have done the security analyst / security engineer thing
● Have done the security consulting thing
● MSS Architect leading a team of Security Engineers and Software Devs
● DevOps Security Engineer at:
We’re hiring!
7. Ship it!
1. Weeks writing code
2. Weeks “testing” code
3. Hand it over the ops team for deployment
Dev VS Ops
8. 1. Weeks writing code
2. Weeks “testing” code
3. Hand it over the ops team for deployment
Dev VS Ops
4. Get out of the office as quickly as possible
9. 1. Weeks writing code
2. Weeks “testing” code
3. Hand it over the ops team for deployment
4. Get out of the office as quickly as possible
Dev VS Ops
5. Deployment is a complete train wreck
10. Dev VS Ops
1. Weeks writing code
2. Weeks “testing” code
3. Hand it over the ops team for deployment
4. Get out of the office as quickly as possible
5. Deployment is a complete train wreck
6. Blame
11. Dev VS Ops
1. Weeks writing code
2. Weeks “testing” code
3. Hand it over the ops team for deployment
4. Get out of the office as quickly as possible
5. Deployment is a complete train wreck
6. Blame
7. Roll back….
14. Agile Development
Methodology
● Frequent changes to production
● New features (new code!!!) being introduced every sprint (often
weekly)
● Less time for QA / Security review
16. DevOps 101
DevOps is the practice of operations and development engineers participating
together in the entire service lifecycle, from inception to deployment.
Operations staff making use many of the same techniques & tools as
developers for their systems work.
26. “Infrastructure as code”
Version-able, Repeatable, Testable
Consistency - Ensure that your server builds meet security
requirements.
Makes life easier - Automate repeated tasks
Change Management - “Code” review and be less scared of
change! (patching anyone..? Bueller? Bueller?)
DR is easier!
27. How the big boys do it
Etsy: Average 25 deploys per day Amazon: Well….
30. Config Management for good
● Base security configs
● Centralized logging & auth
● Host based firewall rules
● Automated Vuln Scans
● Easier Patching
31. Taking a step back..Config Management
Roles:
● Web Server
● Database Server
● Mail Server
Environments:
● Production
● Staging
● Development
32. Automate security
Configure a base hardened config that is used everywhere
Configure centralized auth
Configure centralized logging FOR ALL THE THINGS
Configure host based firewalls per role & environment
Quickly mitigate insecure configurations
33. Example Time
Write a recipe to install and configure rsyslog to
send auth logs to centralized logging servers
34. Automatically logging all the things
1.Automatically install rsyslog
2.Automatically write a configuration file with custom
“destination” log server based on the “nodes”
environment
3.Start the service
4.Profit (from logs...yeah...logs)
37. 1.Write the rsyslog config file
2.Use the ruby template file as the source
3.Replace the “logserver” variable in the template with the proper
log server based on environment
Write a custom rsyslog config file
1
2
3
38. Enable and start the service
1
2
1.For the service named “rsyslog”:
2.Enable it during boot & start it immediately
44. Make it easier for later
1
2
3
1.List of packages to upgrade and their versions
saved as a “node” attribute
2.Upgrade the package:
3.To the specified version
54. 1.Find all nodes with a role of “web_server”
2.For each node in the “webnodes” variable:
3.Add a simple iptables rule to allow inbound to MySQL
4.Replace the source with the value for of that web servers IP Address
Automatically generate firewall rules
3
1
2
4
55. Let’s take a step back
Testing:
● Unit
● Integration
● Regression
● Security
CI System
56. How CI does its thing
Dev Cat
Code Repo
CI System
Did you commit yet?
Did you commit yet?
Did you commit yet?
57. How CI does its thing
CI System ohhh a
commit!
1.Compile the source (if needed)
I’m a
VM!
Build Agent
Run tests
2.Spin up VM / container
3.Deploy code
4.Run tests
5.Profit!
63. What we covered
● Intro to Agile & Devops
● How we NEED to automate to properly secure our
infrastructure & code
● Config management suites 101
○ Build a base server config that is used everywhere
● Automated security tests
64. THANKS FOR LISTENING
Q A&
I’ll try to answer questions now
Q&A Session
@Francisckrs Francisco Donoso Francisck