Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Overcoming Security Challenges in DevOps
1. Taking a DevOps Approach to Security
Alert Logic & Chef discuss overcoming security challenges in DevOps
Yvo Van Doorn
Manager, Solutions Architects
Chef Software
Justin Criswell
Cloud Solutions Architect
Alert Logic
2. Speaker Introduction
• Yvo Van Doorn
• Manager, Solutions Architects
• Chef Software, Inc.
• @yvov
• www.linkedin.com/in/yvovandoorn
• Justin Criswell
• Cloud Solutions Architect
• Alert Logic, Inc.
• @JustinCriswell5
• www.linkedin.com/in/justincriswell
3. Session Overview
More organizations are embracing DevOps to realize compelling
business benefits such as faster yet safer feature release cadence,
increased application stability, and rapid response to shifting market
conditions. However, security and compliance monitoring tools have not
kept up and often represent the single largest remaining hurdle to
Continuous Delivery.
Topics covered in this session:
•How DevOps Improves your Security Posture
•Overcoming Challenges in DevOps Environments
5. Mark Burgess, Father of Configuration Management
Author of “In Search of Certainty”
Configuration Management
“We suffer sometimes from the hubris of
believing that control is a matter of applying
sufficient force, or a sufficiently detailed set of
instructions.”
6. Automation and Convergent Infrastructure
“A system's desired configuration state can be said to be
defined by fixed points. Most configuration management
systems (e.g.: CFEngine, Chef, Puppet, PowerShell DSC) are
based on this idea: they provide means to declare what must
happen instead of requiring imperative workflows that
prescribe what to do.”
Mark Burgess, Father of Configuration Management
Author of “In Search of Certainty”
8. Driving toward Immutable Infrastructure
"This is what I call disposable computing. Throw away a
broken process rather than trying to fix it. Machines can be
made expendable as long as the total software is designed for
it. Not much of it is today, but we´re getting there. Nature
shows that this is a good way of scaling services."
Mark Burgess, Father of Configuration Management
Author of “In Search of Certainty”
9. Infrastructure as Code
• Converge on a regular interval
• Configuration Management is idempotent
• All persistent changes must be in source control
• Manual intervention discouraged
• Out-of-band changes will be lost
20. Traditional Security/Compliance is Slow
Mature DevOps Velocity is Fast
Security Practice does not Keep Up
Security at Odds with DevOps Velocity
21. InfoSec Ends Up Being Marginalized
“The problem for the security person who is used to turning
around security reviews in a month or two weeks is they're
just being shoved out of the game. There's no way with how
Infosec is currently configured that they can keep up with
that. So, Infosec gets all the complaints about being
marginalized and getting in the way of doing what needs
getting done.”
Gene Kim, former CTO of Tripwire
Author of “The Phoenix Project: A Novel About IT, DevOps”
& “Helping Your Business Win”
22. Integrating Security with DevOps
• Leveraging DevOps practice for better Security
- Prevent attack vectors with Immutable Systems
- Manage vulnerabilities with Base Images and
Configuration Management
- Robust Auditing and Centralized Log Collection
- Adopt strategy of Phoenix Upgrades
- Embrace end-to-end Continuous Deployment
23. Manage Vulnerabilities with Base Images
Manage Vulnerabilities
• Conduct normal vulnerability scanning
• Identify Vulnerabilities that exist in Base
Images versus Application specific packages
• Remediate at appropriate level as part of
Continuous Delivery process
• Start with Hardened “secure by default” base
Results
• Less work, done more reliably
• Patching fits naturally into Phoenix Upgrades
• Continuous Delivery allow frequent scanning
in test environments to have real value
• Fixes potential vulnerabilities systematically
24. Prevent Attacks with Immutable Systems
Build secure base images that are
representative of your infrastructure
system base
Design file system layout to separate
code from data, and lock down to
minimum required permissions.
Should expand to network as well
Leverage SANS Checklist and CIS
Benchmark resources for system
level security best practices and
guidance
Leverage configuration management
tools to standardized all software
versions and configurations
Design Secure
Immutable Infrastructure
25. Centralize Robust Auditing & Logging
# This file contains the auditctl rules that are loaded
# whenever the audit daemon is started via the initscripts.
# The rules are simply the parameters that would be passed
# to auditctl.
# First rule - delete all
-D
# Increase the buffers to survive stress events.
# Make this bigger for busy systems
-b 1024
-a exit,always -S unlink -S rmdir
-a exit,always -S stime.*
……
Implement Local Auditing
#Sample syslog-ng configuration
#Lots of configuration required
........
# Send *ALL* System Logs to Log Appliance
destination df_log_appliance_forward {
tcp("my-log-appliance" port(514));
};
log {
source(s_all);
destination(df_log_appliance_forward);
};
Collect Important Logs
Centralize Log Collection for Search and Filtering
26. Adopt Strategy of Phoenix Upgrades
Embrace Phoenix Upgrades
• Stand up new instances, don’t upgrade
• Route traffic between old and new instances
• Rich service metrics and automate rollback
• Advanced routing can enable selective rollout
Results
• Creates evergreen systems, avoiding
configuration drift and technical debt
• Enforces refresh of all system components as
complete artifact, tested as a holistic system
• Greatly reduces security risks when combine
with immutable instances and configuration
management
27. Embrace Complete Continuous Deployment
End-to-End Continuous Deployment
• Configuration Management (Chef),
• Standardized Environment Images (Packer)
• Environment/Subsystem Orchestration Layer
• Production-like environments in Dev & Test
must include all components
Results
• Hardened base images become part of the
standard development process
• Continuous validation of secure configuration
• Holistic system validation & testing
#Sample Alert Logic Chef NodeDef
{
"name": "cloud-api-node",
"versions": {
"1.6.0": {
"vm_type": "squeeze64",
],
"install_phase": {
"run_list": [
"app-version-install@4.1.0",
]
},
….
Leverage Configuration
Management
Leverage Standardized
Environment Images
Build an Orchestration Layer
#Sample Packer Configuration
{
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access`}}"
,
"secret_key": "{{user `aws_secret`}}"
,
"region": "us-east-1",
"source_ami": "ami-de0d9eb7",
"instance_type": "t1.micro",
"ssh_username": "ubuntu",
"ami_name": "packer-ex {{timestamp}}"
}]
28. Moving to Software Defined Security
• Significant Opportunity remains in front of us
- Security remains a lot of work
- Does not fit naturally into DevOps Roles
• AWS ready to Accelerate Security Technology
- Leverage end-to-end visibility available
- Protect automatically with real-time configuration
- Transform periodic assessment into real-time
automated responses
29. Leverage End-to-End Visibility
Leverage Detailed Visibility of AWS
• AWS describe endpoints reveals environment
• Cloud Trail shows incremental changes
• Long-term AWS Config unifies this Information
Results
• Enables visualization of entire environment
• Can reason directly about exploitability
• View of environment always up-to-date
• No need to coordinate application changes
with security infrastructure
30. Protect with Automatic Configuration
Automated Protection
• Add security infrastructure on-demand to
cover changes to environment
• Autoscale that infrastructure based on traffic
seen in that environment environment
Results
• Focus on application deployments and not
configuration of security infrastructure
• Implements least-privilege configurations
• Seamless experience that “just works” with
Security fitting naturally into DevOps practice
31. Transform Assessment to be Real-Time
Real-Time Protection
• Assessment activities should become
continuous
• Continuous Assessment “just happens” as
deployments occur
• Zero-day coverage delivered automatically
Results
• No more periodic assessment cycle
• Security posture of environment available
immediately and automatically after
deployments
• Security matches DevOps Velocity!
32. Contact Us
Justin Criswell
Cloud Solutions Architect
Alert Logic
jcriswell@alertlogic.com
@JustinCriswell5
Yvo Van Doorn
Manager, Solutions Architects
Chef Software, Inc.
yvo@chef.io
@yvov
Thank you!
Notas do Editor
As recommended by AWS, this slide will contain a visual of the AWS shared security model, and define the scope of the security practices and technology that the rest of the presentation will address.
Customer see security as the top pain point for Cloud deployments. This is because as challenging as security is in traditional environments, the adoption of Public Cloud infrastructure and DevOps practices have moved faster than security technology and practice, even farther away from meaningful effectiveness.
Traditionally security technology have always been challenged by the divergent activities of network, systems and application changes, where the majority of customers continue to operate with manually maintained network diagrams. Even when changes are known, the impacts to visibility between systems is not clear because the information is never available in a single place
Modern cloud environments present new unique challenges to understanding host identity and network visibility. No longer do hosts maintain the same IP address originally assigned when restarted, and with auto-scaling the identity of the individual host is no longer meaningful.
All of this makes security technology unable to have a complete understanding of deployment environments and the exposure these new environments present to applications that are running therein
- Current security people, process and tools are oriented to provide value on the order of years, quarters and even months. Even the best compliance practices in PCI proceeds on this cadence because they are simply difficult to keep up on in the mix of tools and practice that exist today.
- DevOps tools and techniques are moving faster and faster, providing mature DevOps organizations with the ability to deliver business value into production at an amazing rate.
- The result is that security is effective on one in every 10 or even 100 of system configurations
So let’s turn our attention away from the challenges that face security technology and practice to concrete things that can be done to help bring effective security to your environment by leveraging DevOps tools and practices. By fully embracing end-to-end configuration management and automated continuous deployment you can gain a greater control of the security posture of your environment. I’m going to cover three key techniques that should be built into all systems:
First key choice is to build systems in which most file systems are read-only, and writeable file systems are noexec/nosuid. This will severely limit the non-privileged attack surface of systems
Second is the corollary of immutable systems – that applying changes to should be done by replacing nodes in load balanced groups, rather than performing upgrades in-place. This enables the use of immutable systems.
Third is to configure strong auditing and centralize all logging in order to detect out-of-band changes that are made outside of the the continuous deployment systems
Finally ensure that you use common base images and system configurations, in order to allow vulnerabilities to be addressed on-mass, rather than a set of one-off systems
This slide is meant to be a pre-cursor to the Outcomes effort. Currently in today’s “Software Defined Infrastructure” nothing exists that allows organizations to conduct vulnerability scans in a real meaningful way that is in “lock step” with continuous deployment of systems.
Scanning today is infrequent at best and usually identifies numerous vulnerabilities that must be addressed
Addressing vulnerabilities in-mass introduces considerable risk to existing stability and performance of the infrastructure
A way needs to exist that will allow an environment to scan frequently and identify problems as they arise. Vulnerabilities (or configuration problems) can be quickly addressed before they “stack up” and introduce added risk
This slide will provide an example of constructing immutable system using config management and deployment tools
Apply those standards to systems that can then be imaged and used as “golden images” during automated deployments. Use tools that enable creating machine images (Packer) which allow standardizing system deployment across different infrastructures (VMWare, EC2, etc.)
Design systems so that executable code is cleanly separated from data at the file system level, and implement a permissions strategy for these file systems based on Least Permissions, including read-only and noexec/nosuid mount options.
Use resources like the SANS Linux Security Checklist or CIS Benchmark as you build golden base images that form the foundation of your deployments. These provide a clear definitive answer to how you can easily ensure security of the systems within your infrastructure, and CIS even offers hardened AMIs for EC2.
Leverage configuration management (Chef) to ensure that all software and configuration changes are standardized and repeatable.
Following these guidelines will ensure that your infrastructures are built to be as immutable fashion, and will enable maximum leverage out of your security and compliance hardening efforts.
All of this makes separation of duties easier to achieve when there are no longer environment specific configurations that are outside of configuration management tools being used by developers.
This slide will show how to integrate strong auditing (via linux auditd) and centralized logging as a basic attribute of all systems built using the configuration management tools.
Enable auditing (auditd) on local systems and ensure configuration is controlled through configuration management (Chef). This example shows a snippet of the CIS benchmark recommendations for auditd configuration for CentOS/Redhat systems.
Ensure syslog is offloaded
Alert Logic has a nice tool…
This slide will demonstrate how to affect change by replacing nodes in a load balancing group instead of performing in-place upgrades.
Embrace the concept of Phoenix Upgrades, systems should not be pulled along but reborn when new versions are available
Technical debt begins to accumulate and replacing over upgrading resets the clock
When combined with deployment of immutable systems and configuration management, it provides a path for releasing new technology (kernel upgrades, etc.) in complete package that can be fully tested prior to deployment
This slide will focus on how important it is to develop, test and deploy the same way in order to greatly reduce overall risk and introduced vulnerabilities
Configuration management ensures that “Infrastructure” becomes defined as code and introduces the capability to deploy in many different environments (Laptop, Test Data Centers, Production, etc). Tools like CHEF provide us the ability to define infrastructure as code
Standardizing Environment Images ensures that we are not overburdened with creating many different images that “might” work in all different environments. Tools like Packer allow us to create an image that can be build and deployed into many different environments (LXC, EC2, VMWare, etc..)
Orchestration (We use an internally developed tool called Environment Manager) allows us to ensure we can deploy our systems and ensure they are configured in a repeatable fashion. This ensures that systems are deployed, installed and configured in the proper order to maximize repeatable experience for development, testing and production
Combining all three with the ability to execute Phoenix Upgrades ensures that we can deploy infrastructure in a repeatable fashion AND we do so without incurring technical debt. This greatly reduces risk and overall introduction of new vulnerabilities due to misconfigurations
Starting from that foundation, we have the opportunity to go further to achieve end-to-end security in the AWS cloud infrastructure. DevOps tools and techniques are a start… and software defined infrastructure provides access to end-to-end visibility… but security tools really need to catch up. Security technologies need to advance in three very important ways:
AWS provides significant visibility into network and system deployments, which should be leveraged directly to identify the configuration of whole environments
Transform traditionally period assessment tasks normally performed on a weekly or monthly basis, into more focused change-driven responses that operate continuously
Automatically reconfigure security protection services based on the real-time change to the environment
Each of these represent important advances that Security Technologies need to adopt to truly begin providing security coverage for Cloud environments
The first challenge is for our security technologies to gain visibility to the complete end-to-end picture of deployed environments. What we need is for this level of visibility to be automatic and immediately react to changes.
This video segment shows the automated discovery of an entire complex environment, including EC2 instances, subnets, VPCs, and internet gateways.
<video segment starts, showing auto-discovery of an environment deployed in AWS>
This type of automatic discover and complete end-to-end visibility is necessary for security technologies to have up-to-date information about the environment in which it operates
<video stabilizes the view, and then begins reconfiguring additional VPC, subnets and some hosts>
Now when changes are made to the environment, via a deployment of additional instances in this case, the view updates to show the new adjusted environment
<video stabilizes after the addition of new instances>
The last challenge is to respond to these changes in the as-deployed environment to automatically reconfigure the security protection to match the environment. This is particularly important in dynamic auto-scaling environments where the size of the environment could grow 10 times or more in a matter of minutes.
This video segment picks up with a simpler web stack environment running just a few instances, plus the security infrastructure instances that are currently protecting the web stack.
<video segment starts, showing the web stack and security appliances protecting it>
Now as traffic grows to the web stack, the ELB will start new instances to handle the traffic.
<video continues to showing the increased traffic and auto-scaled application stack instances>
As the web stack grows, the security infrastructure protecting it expands to handle the additional traffic.
<video continues to show auto-scaled appliance deployments to handle the application traffic>
The last challenge is to respond to these changes in the as-deployed environment to automatically reconfigure the security protection to match the environment. This is particularly important in dynamic auto-scaling environments where the size of the environment could grow 10 times or more in a matter of minutes.
This video segment picks up with a simpler web stack environment running just a few instances, plus the security infrastructure instances that are currently protecting the web stack.
<video segment starts, showing the web stack and security appliances protecting it>
Now as traffic grows to the web stack, the ELB will start new instances to handle the traffic.
<video continues to showing the increased traffic and auto-scaled application stack instances>
As the web stack grows, the security infrastructure protecting it expands to handle the additional traffic.
<video continues to show auto-scaled appliance deployments to handle the application traffic>