The DevSecOps concept is widely accepted yet its implementation at enterprise-scale presents challenges. This session will describe three challenges and software solutions that address them: 1) dozens of autonomous teams using varied tooling to build applications; 2) use of many different scanning tools and security information sources; and 3) Identity and context aware security guidance to development teams.
4. #RSAC
What is DevSecOps?
4
DevSecOps is the integration of security into emerging agile IT and
DevOps development (and deployment) as seamlessly and as
transparently as possible (Gartner)
Seems straightforward enough, so why do we need this session?
Threat
Modeling
Static
Analysis
Component
Analysis
Dynamic
Analysis
Build
Integrity and
Authenticity
Configuration
Validity
Logging
Alerting
Runtime
Tracking
6. #RSAC
Problem Statement
6
• Many autonomous development groups (10s -100s), developing apps using several different technology stacks
• Some groups are working on well-established ”legacy” apps, others on next generation applications with
modern tools
• Different approaches to detecting and closing security vulnerabilities: with every commit or build or at sprint
boundaries or all of the above
6
JS + Java Stack | Jenkins CI | Data Center
Development Group A
DevSecOps Tools
JS + Node.js | Cloud Native CI | Cloud
Development Group B
DevSecOps Tools
Xamarin + Mobile Platform | Another CI | Consumer Devices
Development Group C
DevSecOps Tools
7. #RSAC
Challenge: Lack of Uniform View across Departments
7
Lack of a uniform view and understanding of security state of
applications
– Difficult to assess and compare security posture of various products
across departments
– Difficult to enforce a uniform level of compliance with enterprise
guidelines
Cost of maintaining separate departmental infrastructures
– Headcount/License/Training/Maintenance
8. #RSAC
Challenge: Managing Diversity of Scanners and
Security Information Sources
8
Numerous scanners and scanner types available
– Open Source, Existing Enterprise Licenses for on-prem and SaaS models
– Lightweight (take a few minutes) vs Exhaustive (may take hours to run)
Selection of appropriate scanners for comprehensive coverage
– SAST [Static Code Security] Scanner
– SCA [Software Composition Analysis] Scanner
– Embedded Secrets Scanner
– DAST [Dynamic App Security] Scanner
– Infrastructure mis-configurations (cloudformation, kube deploy, etc.)
Ability to process vulnerability feeds from various input sources
Coping with scanner noisiness/chattiness
9. #RSAC
Challenge: Developer Enablement
9
How to ensure that developers and development teams are tasked only with
security vulnerabilities relevant to them?
Rich set of application artifacts - git branches, repositories, build processes,
docker containers, application assemblies, cloud accounts - create difficulties in
linkage to devs or dev teams
Lack of information sharing between development teams
Actionability of remediation guidance by app developers
– App developers are not security experts!!
Process inefficiencies in consultation between development teams and CSO
(Security SMEs)
11. #RSAC
Solution: Enterprise-scale DevSecOps
11
Shared infrastructure “plugs-in” to the specific development tech stacks and CI+CD frameworks used by different
groups
Layered architecture accommodates new tech stacks, languages and target platforms
Ability to process and manage security vulnerabilities from diverse scanners and security information sources
11
JS + Java Stack | Jenkins CI | Data Center
Development Group A
JS + Node.js | Cloud Native CI | Cloud
Development Group B
Xamarin + Mobile Platform | Another CI | Consumer Devices
Development Group C
12. #RSAC
High Level Logical Architecture
12
Jenkins based Pipeline
SAST [Static Code
Security] Scanner
SCA [Composition
Analysis] Scanner
Embedded
Secrets Scanner
Layered Integration
Application
Security
Workbench
2
1
4
3
Security
Organization
Findings
Additional Security
Information Sources
JIRA
Code Repository
BU Leader
Architect / PM
App Lead
Developer
Business Unit
Build Applications
Security
Scan Reports
(Immediate)
View & Prioritize
Security
Vulnerabilities
Status Updates
Ingestion (at a cadence)
Create tickets for
Remediation
Updated status
post Remediation
DAST [Dynamic
App Security]
Scanner
Kubernetes
Checker
Scanner+
(Cloudformation,
nodeJS,
python,…)
Offline
Enterprise
Scanners
Team-specific
Scanners
L
a
y
e
r
e
d
I
n
t
e
g
r
a
t
i
o
n
13. #RSAC
Solution
Software framework built out of stock components and open source
– Scanner layer provides a uniform way to package scanners into docker images
We selected a standard set of open source scanners familiar to us
Can be replaced by alternatives or licensed versions as needed
– Dockerized scanners can be plugged into many different CI pipelines
Current focus is on Jenkins-based pipelines, but others are in our backlog
Linkage between repositories/projects to products and teams
– Based on machine readable meta-data added by teams to repositories/projects
– Ensures that code/artifacts/assemblies/docker images/cloud accounts can be
linked to products and teams
13
14. #RSAC
Solution (contd.)
Workbench provides a generic data model and uniform GUI for all security
vulnerabilities
– Allows new scanners or security information sources to be added as needed
Including off-line scanners and security information sources that are available asynchronously
– Standardized display of vulnerabilities
Severity, Remediation Guidance, False Positives, Acceptable Risk
Main focus is helping application developers to act on the information
• Including ways of sharing information between teams acting on similar vulnerabilities
Workbench provides division/product level rollups
– Fine grained vulnerability reports at repositories/artifact level
– Aggregate vulnerabilities at department/product level
– Exposes extent of security maturity across division/products
14
18. #RSAC
Apply #1 [Immediate]
18
Agree on an enterprise-wide approach with all stakeholders (Dev
teams, leadership, CSO office)
– Identify DevSecOps efforts that are on-going in different teams and capture their
requirements
Identify if there are any existing enterprise license agreements with
security vendors
– Supplement with use of well-respected open source systems
– Security scanners are a commodity, examine vendor claims of superiority with caution!
Agree on a base level set of security scanners and information sources
– Getting off the ground is key; don’t try to achieve nirvana!!
19. #RSAC
Apply #2 [30-60 days]
19
Ensure that the selected tools support varied team development
methodologies and different tech stacks
– This is why a pluggable framework is important
– This will also guide your BUY vs BUILD decision
Agree on how security vulnerabilities should be surfaced
– E.g., Webapp, webex, email, CI/CD links, bitbucket annotations
Identify and engage with early adopters / champions
– Establish regular feedback mechanism
20. #RSAC
Apply #3 [180 days]
20
Develop a process for remediation timelines and priority
– Who determines impact and risk and how will it be manifested by your
tools?
– Not every security issue can be fixed easily or quickly, important to have a
tracking process
Culture shift through office hours/trainings
– Developers to become familiar with security vulnerability patterns common
within the enterprise
– Train developers on becoming proficient at remediating these vulnerabilities
22. #RSAC
Closing Thoughts, Credits and Demerits
Enterprise-scale DevSecOps requires going beyond selecting or
creating a toolset
– Culture change for both Development Teams and Security SMEs
– Requires process changes in Development Teams and movement away from a
model of “security as punishment/shaming”
Credits
– OWASP organization and community esp. open source tools
– Vendors supporting open source tools (SonarQube, Anchore - Grype/Syft)
Demerits
– Vendors with unrealistic and unreasonable claims pushing proprietary solutions
22
Notas do Editor
Welcom to this session! I am xyz and with me is Gaurav Bhargava, we both work for ADP where we have been building a DevSecOps infrastructure across the company.
Today we will share some of our experiences and insights from this journey. We look forward to your questions - and for those who want to dig deeper - we are leading a BOF session on Wednesday afternoon.
During the last five years, we have seen an increasing focus on integrating security with agile development and deployment
So by DevSecOps we mean the processes and tools needed to achieve this integration
In this diagram, we show categories of security processes integrated with each stage of the DevOps lifecycle
-- call out each stage
So lets dig deeper to understand the challenges of enterprise-scale devsecops
Now if you have a couple of dozen or even a couple of hundred developers creating and maintaining a handful of products on a single technology platform - yes, you can pull together a DevSecOps toolset and program relatively easily.
Our focus is on large organizations with dozens of development groups creating 10s or 100s of products using varying technology stacks
And these products are at different stages of maturity - some are established ”legacy” products - others are newer apps - for example they may be completely cloud native
Finally, team cultures and development practices in teams can be quite different - some teams address security vulnerabilities within each sprint - others accumulate security debt and use a security sprint to address security vulnerabilities.
With teams owning and implementing independent devsecops programs, its difficult to have a uniform view of the security state of applications
…
Read slide
Application security requires a range of scanners and information sources for comprehensive coverage
Need to identify and make available scanners in each different category
Inline or lightweight scanners that can provide feedback within minutes
Offline scanners that may take hours to find a more exhaustive set of vulnerabilities
-- final 2 bullets --
There is no magic formula that will help in choosing the right tools - enterprise securlty policies and compliance needs will drive the selection of scanners
Developer and dev team enablement is key to the DevSecOps program
linkage of security vulnerabilities from all the different application artifacts - build processes, docker containers, cloud accounts - to devs and dev teams and products is a challenge that has to be solved
Application dev teams and developers aren’t security experts and we shouldn’t expect them to become security SMEs
As vulnerabilities are discovered, we need practical guidance for remediation
When different teams deal with similar vulnerabilities, we need to support information sharing and experiences across teams
We need defined workflows when the CSO office should be involved (consultations, exceptions, risk acceptance)
So now I am going to turn to Gaurav Bhargava to talk about our solution to these requirements
The first and most important step is to drive consensus around the value of an enterprise-wide approach
- The best way to achieve this is going to vary, it really depends how your organizational structure
Our approach was to create a POC and show some workflows that teams found helpful
----
Switch the last two bullets
The key to success here is to meet dev teams where they are
What this means is that you have to understand their development culture and tech stacks - and this is why your solution needs to be able to plug security into their processes with only a small investment in time and effort for them
- Bullet 2
How you communicate vulnerabilities has to be low impact and fit with team practices
Understanding when/if a vulnerability has been successfully remediated should be straightforward
- Bullet 3
