Continuous Acceleration: Why Continuous Everything Needs A Supply Chain Approach - by @joshcorman
Today we write less code & consume more re-usable open source code. Innovation accelerates & complexity grows. Complexity is the enemy of quality. Poor quality creates break-fixes & slows dev. It’s a continuous loop.
Learn about software supply chain management strategies to accelerate delivery: scrutinize your 'suppliers,' minimize risk & bloat, and improve traceability, visibility & response times.
Draws upon work from:
@iamthecavalry https://www.iamthecavalry.org
@RuggedSoftware https://www.ruggedsoftware.org
@gauntlt
@RuggedDevOps
@RealGeneKim https://www.itrevolution.com The PhoenixProject
@JezHumble
2. @joshcorman
Conclusions / Apply!
Idea: A full embrace of Deming is a SW Supply Chain:
Fewer/Better Suppliers
Highest Quality Supply
Traceability/Visibility throughout Manufacturing / Prom & Agile Recall
Benefits: Such rigor enables:
Even FASTER: Fewer instances of Unplanned/Unscheduled Work
More EFFICIENT: Faster MTTD/MTTR
Better QUALITY/RISK: Avoid elective/avoidable complexity/risk
Urgency: It’s OpenSeason on OpenSource
And our dependence on connected tech is increasingly a public safety issue
Coming Actions: Known Vulnerabilities” Convergence
Lawmakers, Insurers, Lawyers, etc. are converging
19. @joshcorman
•The
The Cavalry isn’t coming… It falls to us
Problem Statement
Our society is adopting connected
technology faster than we are able to
secure it.
Mission Statement
To ensure connected technologies with
the potential to impact public safety
and human life are worthy of our trust.
Collecting existing research, researchers, and resources
Connecting researchers with each other, industry, media, policy, and legal
Collaborating across a broad range of backgrounds, interests, and skillsets
Catalyzing positive action sooner than it would have happened on its own
Why Trust, public safety, human life
How Education, outreach, research
Who Infosec research community
Who Global, grass roots initiative
WhatLong-term vision for cyber safety
Medical Automotive
Connected
Home
Public
Infrastructure
I Am The Cavalry
26. @joshcorman
Agile goats; not goat rodeo. “We need to be agile, but not fragile.”
@RuggedSoftware @joshcorman @mortman #RSAC #DevOps
27. @joshcorman
ON TIME.
Faster builds.
Fewer interruptions.
More innovation.
ON BUDGET.
More efficient.
More profitable.
More competitive.
ACCEPTABLE QUALITY/RISK.
Easier compliance.
Higher quality.
Built-in audit protection.
Agile / CI
28. @joshcorman
DevOps
It may feel like DevOps is Pandora’s Box, but it’s open… and hope remains. ;)
@joshcorman @mortman #RSAC #DevOps
29. @joshcorman
ON TIME.
Faster builds.
Fewer interruptions.
More innovation.
ON BUDGET.
More efficient.
More profitable.
More competitive.
ACCEPTABLE QUALITY/RISK.
Easier compliance.
Higher quality.
Built-in audit protection.
DevOps / CD
Agile / CI
31. @joshcorman
ON TIME.
Faster builds.
Fewer interruptions.
More innovation.
ON BUDGET.
More efficient.
More profitable.
More competitive.
ACCEPTABLE QUALITY/RISK.
Easier compliance.
Higher quality.
Built-in audit protection.
SW Supply Chain
DevOps / CD
Agile / CI
33. @joshcorman
Open source usage is
EXPLODING
Yesterday’s source
code is now replaced with
OPEN SOURCE
components
33 Source: Sonatype, Inc. analysis of (Maven) Central Repository component requests.
201320122011200920082007 2010
2B1B500M 4B 6B 8B 13B 17B
2014
38. @joshcorman
In 2013, 4,000
organizations downloaded
a version of Bouncy Castle
with a level 10 vulnerability
20,000 TIMES …
Into XXX,XXX Applications…
SEVEN YEARS
after the vulnerability was fixed
NATIONAL CYBER
AWARENESS SYSTEM
Original Notification Date:
03/30/2009
CVE-2007-6721
Bouncy Castle Java Cryptography API
CVSS v2 Base Score: 10.0 HIGH
Impact Subscore: 10.0
Exploitability Subscore: 10.0
BOUNCY CASTLE
39. @joshcorman
In December 2013,
6,916 DIFFERENT
organizations downloaded
a version of httpclient with broken
ssl validation (cve-2012-5783)
66,824 TIMES …
More than ONE YEAR
AFTER THE ALERT
NATIONAL CYBER
AWARENESS SYSTEM
Original Release Date:
11/04/2012
CVE-2012-5783
Apache Commons HttpClient 3.x
CVSS v2 Base Score: 5.8 MEDIUM
Impact Subscore: 4.9
Exploitability Subscore: 8.6
HTTPCLIENT 3.X
40. @joshcorman
40
Current approaches
AREN’T WORKING
TAKE COSTS OUT OF YOUR SUPPLY CHAIN
Component
Selection
DEVELOPMENT BUILD AND DEPLOY PRODUCTION
COMPONENT
SELECTION
228K
Unique components
downloaded per
company
!
75%
Lack meaningful
controls over
components in
apps
!
X
Average number of
suppliers per
company
!
48
Different versions
of the same
component
downloaded
!
41. @joshcorman
41 5/8/2015
X Axis: Time (Days) following initial HeartBleed disclosure and patch availability
Y Axis: Number of products included in the vendor vulnerability disclosure
Z Axis (circle size): Exposure as measured by the CVE CVSS score
COMMERCIAL RESPONSES TO OPENSSL
45. @joshcorman
H.R. 5793 “Cyber Supply Chain Management and Transparency Act of 2014”
Elegant Procurement Trio
1) Ingredients:
Anything sold to $PROCURING_ENTITY must provide a Bill of
Materials of 3rd Party and Open Source Components (along with
their Versions)
2) Hygiene & Avoidable Risk:
…and cannot use known vulnerable components for which a
less vulnerable component is available (without a written and
compelling justification accepted by $PROCURING_ENTITY)
3) Remediation:
…and must be patchable/updateable – as new vulnerabilities will
inevitably be revealed
46. @joshcorman
In 2013, 4,000
organizations downloaded
a version of Bouncy Castle
with a level 10 vulnerability
20,000 TIMES …
Into XXX,XXX Applications…
SEVEN YEARS
after the vulnerability was fixed
NATIONAL CYBER
AWARENESS SYSTEM
Original Notification Date:
03/30/2009
CVE-2007-6721
Bouncy Castle Java Cryptography API
CVSS v2 Base Score: 10.0 HIGH
Impact Subscore: 10.0
Exploitability Subscore: 10.0
PROCUREMENT TRIO + BOUNCY CASTLE
51. @joshcorman
Current approaches
AREN’T WORKING
Component
Selection
DEVELOPMENT BUILD AND DEPLOY PRODUCTION
COMPONENT
SELECTION
75%
Lack meaningful
controls over
components in
apps
27
Different versions
of the same
component
downloaded
95%
Inefficient sourcing:
Components are not
downloaded to caching
repositories
63%
Don’t track
components
used in
production
24
Critical or severe
vulnerabilities
per app
4
Avg of strong
copyleft licensed
components per
app
52. @joshcorman
Component
Selection
DEVELOPMENT BUILD AND DEPLOY PRODUCTION
COMPONENT
SELECTION
PUBLIC
REPOSITORIES
NEXUS LIFECYCLE
PRECIOUSLY
IDENTIFY
COMPONENTS
& RISKS
REMEDIATE
EARLY IN
DEVEOPMENT
AUTOMATE
POLICY ACROSS
THE SDLC
MANAGE RISK
WITH
CONSOLIDATED
DASHBOARD
CONTINUOUSLY
MONITOR
APPS FOR
NEW RISKS
54. @joshcorman
Conclusions / Apply!
Idea: A full embrace of Deming is a SW Supply Chain:
Fewer/Better Suppliers
Highest Quality Supply
Traceability/Visibility throughout Manufacturing / Prom & Agile Recall
Benefits: Such rigor enables:
Even FASTER: Fewer instances of Unplanned/Unscheduled Work
More EFFICIENT: Faster MTTD/MTTR
Better QUALITY/RISK: Avoid elective/avoidable complexity/risk
Urgency: It’s OpenSeason on OpenSource
And our dependence on connected tech is increasingly a public safety issue
Coming Actions: Known Vulnerabilities” Convergence
Lawmakers, Insurers, Lawyers, etc. are converging
Incentives Incentivize – Any strategy that requires human nature to change is likely to fail
The Eternal Essence of SW Developers is to be “On Time. On Budget. With Acceptable Quality/Risk”
Which translates into Go Faster,. Be more Efficient. Manage Quality.” In that order…
IMG SRC = https://www.flickr.com/photos/opensourceway/4862920379/in/photolist-8pHJNP-9YLxpV-bZwfxo-4cv2XJ-6u2Sii-6u2Sbv-6u2RPV-7TCwgh-7DhXvU-8bpC9J-f119g-6V7UHx-63Bo2R-bwS9ux-7svgys-755bHf-9YLxvr-4R4GZV-dhtwk7-6V69PL-8nuXRE-c8Hc9m-9RTeA4-5HhfEX-8Vnaei-aFf72q-pgL6BQ-6n9a5w-6n98H1-6n99vN-6n4YQe-751hbP-4PLgno-M3uTP-9YPru5-BAPs6-8JMvEe-6t2Jfa-k9ZQVz-eF7qWf-6VbWPN-4UKjC3-7z4qGQ-jAC6Ap-9YLuSK-9YLuLD-9YLwYr-5zUdBo-64ooGC-9YLvPx
Gene and I realized back in 2011 that DevOps was a game changer and that Security as we knew it was not going to survive. My Rugged Software Manifesto and movement and our shored beliefs of culture and incentives compelled us to marry the tribes for mutual benefit.
An in 2015 We now Merge with the broader DevOps leadership community… with a full day 700 person Rugged DevOps workshop… at this year’s RSAC
Gene is realizing the mad chemistry we’ve done… Jez is asking what the heck am I getting myself into ;)
NIST’s NVD (National Vulnerability Database_ http://web.nvd.nist.gov/view/vuln/search-results?query=OpenSSL&search_type=all&cves=on
SEIMENS was affected by 4 OpenSSL flaws beyond HeartBleed. One from 2010
http://www.scmagazine.com/siemens-industrial-products-impacted-by-four-openssl-vulnerabilities/article/361997/
“Several Siemens products used for process and network control and monitoring in critical infrastructure sectors are affected by four vulnerabilities in the company's OpenSSL cryptographic software library.
The vulnerabilities – CVE-2014-0224, CVE-2014-0198, CVE-2010-5298, and CVE-2014-3470 – can be exploited remotely, and fairly easily, to hijack a session as part of a man-in-the-middle attack or to crash the web server of the product, according to a Thursday ICS-CERT post.
Siemens has already issued updates for APE versions prior to version 2.0.2 and WinCC OA (PVSS), but has only issued temporary mitigations for CP1543-1, ROX 1, ROX 2, and S7-1500.
The products are typically used in the chemical, critical manufacturing, energy, food and agriculture, and water and wastewater systems sectors, according to the post.”
www.ruggedsoftware.org
https://www.ruggedsoftware.org/documents/
“I recognize that my code will be attacked by talented and persistent adversaries who threaten our physical, economic, and national security“
[ picture of messy data center ] Ten minutes into Bill’s first day on the job, he has to deal with a payroll run failure. Tomorrow is payday, and finance just found out that while all the salaried employees are going to get paid, none of the hourly factory employees will. All their records from the factory timekeeping systems were zeroed out. Was it a SAN failure? A database failure? An application failure? Interface failure? Cabling error?
Deming has sage advice for us… but he has more than that… Deming may be the key to changing our fate… …more on this later…
See also Josh’s RSA Europe Keynote Video:
Survival Isn’t Mandatory: Challenges and Opportunities of DevOps
http://youtu.be/m4Y_K7MXQxQ
Incentives Incentivize – Any strategy that requires human nature to change is likely to fail
The Eternal Essence of SW Developers is to be “On Time. On Budget. With Acceptable Quality/Risk”
Which translates into Go Faster,. Be more Efficient. Manage Quality.” In that order…
IMG SRC = https://www.flickr.com/photos/opensourceway/4862920379/in/photolist-8pHJNP-9YLxpV-bZwfxo-4cv2XJ-6u2Sii-6u2Sbv-6u2RPV-7TCwgh-7DhXvU-8bpC9J-f119g-6V7UHx-63Bo2R-bwS9ux-7svgys-755bHf-9YLxvr-4R4GZV-dhtwk7-6V69PL-8nuXRE-c8Hc9m-9RTeA4-5HhfEX-8Vnaei-aFf72q-pgL6BQ-6n9a5w-6n98H1-6n99vN-6n4YQe-751hbP-4PLgno-M3uTP-9YPru5-BAPs6-8JMvEe-6t2Jfa-k9ZQVz-eF7qWf-6VbWPN-4UKjC3-7z4qGQ-jAC6Ap-9YLuSK-9YLuLD-9YLwYr-5zUdBo-64ooGC-9YLvPx
Comparing Toyota and General Motors
JOSH: Bring up:
Healthcare.gov
81 versions of Spring vs 1
15% Innovation lift at Insurer
MTTD 6 minutes versus 6 weeks
In fact open source usage is exploding based on the number of downloads from the (Maven) Central Repository.
Looking at these numbers it is easy to understand why only 10% of a typical application is source code.
As the stewards of the Central repository, we have the unique insight into both the phenomenal growth – as well as the related risk.
…….
Way back in the dark ages of 2001, our founder named Jason van Zyl, who was very, very frustrated about how inefficient the leveraging of open source was in that time, created this thing called Maven. And Maven, I’m sure most of you know, but Maven is basically a recipe container and dependency resolver and it allowed for very efficient use of open source binaries. When he created Maven, he said he decided that it would be really convenient if Maven could just look to a default place. And so as an afterthought he created and became caretakers of the Maven Central. And it gives us an incredible amount of visibility into how the ecosystem works. We can see who is contributing what innovations, who is consuming what components, what trends there are in open source usage. We’ve seen is just an explosion of module software development. Last year we serviced 13 billion open source component requests. And as big as that number sounds, it’s understated because 25% of the requests came from cash and proxies like Nexus.
From an attacker eye view… it used to require finding a flaw in Bank XYZ and then exploiting that bankand only than bank… but now….
…if an attacker finds a flaw in Struts… it can attack EVERY bank who uses it – which is most of them.
Same reason Heartblled was so far reaching… shared depenedance == shared risk/attack surface
POINT: INCREASE in attacker interest/value – aka Blood is in the water…
Before the highly publicized OpenSSL Heartbleed
Last summer/fall… a worst case CVSS 10 flaw in the hugely popular Apache Struts Project was used to compromise most of the banks and other serious targets above.
This bug had been there for YEARS unnoticed.
Many had to be told by the FBI that they were compromised
This triggered the FS-ISAC to issue guidance on 3rd Party and OpenSource Supply Chain risk… out of necessity
The 3 letter agency SHOCKED me… but alas is true
The green is a Chinese attack tool out almost immediately after the CVE was announced.
I looked deeper into the Apache Struts Project.
A pattern I’ve recognized is that there is more vulnerability/attacker interest in the most depended upon OpenSource Projects.
Struts is one of the most depended upon – especially so in the Financial Services industries…
As previously stated, one of the CVSS lvl 10 (of 10) struts vulnerabilities wreaked havoc on
POINT: There are more vulnerabilities – and more serious ones…. in the recent year to two.
I may ask this gets dynamically autogenerated per-project by my teams.
NOTE: Many of these flaws were dormant a VERY long time – despite the “many eyeballs” false belief.
NOTE: I personally think this more speaks to attacker/aversary interest.
Here are just a few examples so you can see that this risk is real…
Bouncy Castle is a popular open source component… and even after critical security alerts were issued in 2009 (CVE issued earlier in 2007), 4000 companies still downloaded it 20,000 times.
And that was seven years after a better, safer replacement was issued.
This is a level 10 critical security risk. Imagine the exposed applications out there… maybe some of them store your personal credit card data or other personal information.
This example is even worse… a version of httpclient with a broken SSL validation downloaded by 6,916 organizations 66,824 times more than a year after the alert.
It wasn’t hard for us to find these examples… this just skims the surface.
Some of you may have heard about the FBI Warning last year about Struts… a vulnerable – and old – version of this framework was used to hack into a handful of large organizations.It mde a lot of news. But people are still using it today.
Bouncy Castle – CVSS 10 – 2009 -- Since then
11,236
organizations
downloaded it
214,484
times
httpClient
Since then
29,468
organizations
downloaded it
3,749,193
times
Qualitative takeaways:
Virtually all major (and not so major) software vendors are building on a stack of open source (including security vendors).
The breadth of use across some vendors, IBM most notably is remarkably high (open source is not just in a few rogue products).
New discoveries are getting more serious over time.
New discoveries are getting less vendor attention (fewer vendor disclosures) despite their being more serious.
Vendors are responding to new discoveries at a somewhat slower pace.
The significant increase in product disclosures after the later OpenSSL disclosures, which affect all versions of OpenSSL not just versions 1.0.1 or later, implies that many vendors and products were using old libraries (version 0.9.8 was first released in July, 2005).
Total disclosures: 227
Total product instances affected by disclosures: 2,513
Mean time to repair: 35.8
Median time to repair: 22.0
Here are just a few examples so you can see that this risk is real…
Bouncy Castle is a popular open source component… and even after critical security alerts were issued in 2009, 4000 companies still downloaded it 20,000 times.
And that was five years after a better, safer replacement was issued.
This is a level 10 critical security risk. Imagine the exposed applications out there… maybe some of them store your personal credit card data or other personal information.
Bouncy Castle – CVSS 10 – 2009 -- Since then
11,236
organizations
downloaded it
214,484
times
httpClient
Since then
29,468
organizations
downloaded it
3,749,193
times
Early and Ongoing Vulnerability Identification
Provide tools throughout the development lifecycle to identify potential issues as early as possible to build a secure software supply pipeline
Understand and Remediate
Monitor high risk franchise applications to determine vulnerabilities included in application components
Implement an *Application VTM* type process to look at internally consumed products like we do in the VTM cycle
