Security in DevOps is often laser-focused on automation. This talk looks at methods to incorporate manual tasks in a non-disruptive manner.

1. Blending the Automated
and the Manual:
Making Application
Vulnerability
Management Your Ally
DevOpsConnect | San Francisco | 2015

2. Who We Are
! Kris Curylo | Application Security Manager at Ally Financial
! Dan Cornell | CTO at Denim Group

3. Introduction
! Application security programs and in
particular, application testing has
traditionally been a fairly slow and
manual process.
! Development teams are moving faster
through the implementation of DevOps
processes.
! We need to keep up.

4. Why I’m The One Talking To You
! I have spent the past 2 years building the
application security program at Ally Financial
! I inherited a pile of tools and a few (unclear)
requirements
! I was told to “Make it work, make it work better,
make it provide value rather than just check the
box”
! Oh, and make sure you do it with existing resources
and budget.
! I’m guessing if you’re here, you’re probably in a
similar position.

5. How I Got Started
! Take Inventory (of EVERYTHING)
! Applications
! Processes
! Tools
! Requirements
! Complaints
! Organize
! Plan

6. Pain Points
! Too many “things”
! Too many tools
! Too many processes
! Too many interfaces for data
! Too many report formats
! Redundant decisioning
! This all leads to the biggest complaints:
! Everything takes too long and is inconsistent

7. Automate and Consolidate
! Need fewer manual processes
! Managing requirements
! Running scans
! Handling data
! But…can’t have no manual
processes
! Need better view into data
! Single TODO list of vulnerabilities to
address
! Slice and dice

8. Great…What Do I Do Now?
! I used SharePoint:
! Created my own application inventory
! Created test tracking process
! Automated “compliance calculation”
! Exposed it to stakeholders
! This reduced complexity and allowed stakeholders to
make informed decisions and prioritize security
requirements with other business objectives.

9. What About Vulnerability Management?
! We use lots of vendors & tools:
! HP WebInspect (DAST)
! Veracode (SAST)
! Trustwave/Cenzic Hailstorm (DAST)
! BurpSuite (DAST)
! OWASP Zap (DAST)
! HP Quality Center (Defect tracking)
! Leads to passing reports around or sending people to various
interfaces

10. Communication Patterns
! “Here’s a 300 page PDF with a color graph on the
front page”
! “Here’s another, different, 300 page PDF with a
different color graph on the front page”

11. Automate and Consolidate
Security
Services
Request
Security
Orchestration
Manual
Assessment
3rd Party Manual
Assessment
Testing Tools & Services
AppSec False
Positive Analysis
Defect Tracker
Reporting & Metrics
Developer
Remediation
IDE

12. ThreadFix Background
! Application vulnerability management platform
! ThreadFix allows teams to:
! Create a consolidated view of your applications and vulnerabilities
! Prioritize application risk decisions based on data
! Translate vulnerabilities to developers in the tools they are already using
! Extensive REST API for automation
! Allow application security teams to focus on high-value activities
! Open Source ThreadFix Community Edition:
! https://github.com/denimgroup/threadfix
! http://www.threadfix.org/

13. APIs Are the “Key”
! Today, we specifically require any new tool or process to
integrate with ThreadFix to be considered for use in the program
! We have worked through every testing tool we have to identify
APIs and individually review them for adding automation to the
process.

14. No API? No Problem...
! ThreadFix's RESTful API allows us to write our
own automation
! Using SharePoint and standard naming
conventions to upload test results via
workflow
! Create cron jobs to batch upload

15. Automate and Consolidate – Next Steps
Security
Services
Request
Security
Orchestration
Manual
Assessment
3rd Party Manual
Assessment
Testing Tools & Services
AppSec False
Positive Analysis
Defect Tracker
Reporting &
Metrics
Developer
Remediation
IDE
Web Application
Firewall
Training Plans
Build Servers
Attack Surface Seeding

16. Can’t Escape the Manual
! External test results from manual efforts are now
tracked along side our own test results
! For ASPs and external vendors, we can require
them to submit their own test results to us
! Standardized submissions have allowed us to gain
better insight to 3rd party security posture

17. Bring Everything Together
! Using ThreadFix, we:
! Give our management, development and support
teams one interface
! Expose the data that matters to the proper people
! Retain proper tracking of vulnerability meta data and
decisioning
! Reduce overall complexity while increasing value and
agility (pun intended...)
! Pull results from testing tools as they become
available

18. Speak to the Developers (In Their Own Language)
! HP Quality Center APIs allow us to push defects directly into
the defect tracker from ThreadFix
! ThreadFix then pulls info back when the developers update
the defect records
! Eclipse API shows results in the IDE along side the code

19. Unplanned Advantages
! With all data residing in one spot, we can identify trends
! What training should we offer to developers?
! When training was conducted, did it help?
! Are certain teams, languages, business units better or worse at
specific things?
! Do we have an opportunity to develop a pattern to address
certain flaws?
! Most complete view of application security posture we have
ever had to enable better decision making of risk and
priorities

20. We Found Lots of Places to Introduce Automation:
! Static testing execution
! Dynamic testing execution
! Results review
! Result tracking
! Compliance tracking
! Metrics

21. Advice From the Field
! Don’t let perfect be the enemy of good
! Small victories and incremental progress will keep your efforts in
front of management and dev teams
! Tackle a crowd pleaser early on
! If you address the loudest critic quick, you will gain credibility and
will be more apt to get help implementing automation
! Build it and they will come
! Get one build server integrated
! Get one application team using ThreadFix alone for all decisioning.
! Get one team to publish defects into your bug tracker through
ThreadFix

22. Lessons Learned
The Good
! Developers want to write good code. They
will use the tools made available if they are
not too intrusive
! Building in automation allows us identify
trends and systemic opportunity for
improvement regardless of developer
participation
! There are more opportunities for
automation than expected

23. Lessons Learned
The Bad
! Retrofitting an existing program is painful
! No matter how much you automate, it will never
be enough
! We learned some scary things about our
environment.
! Expect to be overwhelmed.

24. Where We Go Next
Push automation further:
! Integrate further with build servers
! Virtual Patching via WAF rules
! Automate sanity check scans through attack surface
mapping and API to dynamic tools
! Targeted training based on flaws present in applications

25. Questions / Contact
Kris Curylo
! Kristopher.Curylo@ally.com
Dan Cornell
! dan@denimgroup.com
! @danielcornell