Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Software rotting - DevOpsCon Berlin
1. Software rotting
Why you need to change your approach to security…
@giulio_vian
https://www.getlatestversion.eu
http://blog.casavian.eu
https://www.slideshare.net/giuliov
https://github.com/giuliov
Giulio Vian
22 June 2022
8. High-level process
CVE / Security
triggers
Developer Teams
search
Developer Teams
fix
Release
Management
deploy
9. Affected by
Vulnerability
Application stack
Container images
Virtual Machine images
Application itself
Application code
Libraries
Internal
3rd party
Self-contained run-time
Application
Run-time
OS
libraries
Image
Self-
contained
search
10. Find code, easy?
Multiple production branches
release/* and hotfix/*
Untagged releases
SCA† tools pipeline-bound
Rarely built code
Pipeline does not work anymore
† Software Composition Analysis
search
13. What’s normal?
Many teams
Many repos
My company has 3,000 repos
across 100 teams, storing over
13 million lines of code, and
using 2,800 pipelines
A single vulnerability
may affect 10s teams and
100s of repos Image: The Crowd For DMB 1 by Moses
fix
16. Estimate Risk
Probability of an adverse
cyber event
Frequency of attack
Availability of 0-day exploit
Cost factors
Number of systems to patch
% of Components to build
and redeploy
Actuaries already have rich
models
search
Image source: WikiMedia
18. Good SCM
Practices
Standardize SCM to Git
Single management system
with rich API
Standard naming for
Production tags
For branches, also
Rich metadata via tagging
features
Repo owners
Hotfix pipelines
search
Image by David Iliff from Wikimedia
19. Breadth of
change
Fix impacting many
systems at once
Hundreds of concurrent pipelines
Can your build & deploy
tool auto-scale?
Can your approval process
scale?
How fast can you rebuild a
substantial portion of IT
systems?
fix
Image source: public domain
20. Tooling
Your SCA may generate
code changes
e.g. dependabot
Build scripts can be
massively edited
git-xargs, auto-pr, multi-gitter, …
Image: robotic arm in the Conrad Prebys Center for Chemical Genomics by Josh Baxt
fix
21. Good patching
practices
Scan all repos often
Fast-track automated
pipelines for all systems
Thorough automated
regression testing
Expedite approval
process
fix
Image source: public domain
22. Bill of Materials
on steroids
Reverse indexes
Library → Binaries [SCA tool]
O.S. API → Binaries [SAST tool]
Binary → Pipelines [artifact store]
Pipeline → Repo(s) [pipeline tool]
Pipeline
Binaries
Production
Library
Repo
deploy
23. Redeploy.
Every. Day.
Simplest pattern
Once automated
patching is in place
Zero-downtime deploy
in place
Consider pipeline
resources
Image: the gerbil wheel pose by dbgg1979
deploy
24. Expedite
pipelines
Separation of Duties
Regulation / audit requirement
Slows 0-day patching
Tightly controlled usage
Automated checks
Single commit with limited
churn
Additional approvers for
quick turnaround
Image courtesy of SpaceX
deploy
34. App Platform shift
Chrome 1 month patched after 14 days
Node.JS 30 months (LTS) patched every 25 days
6 months
Go 6 months patched every 26 days
Two major releases supported.
MongoDB 30 months patched every 5 weeks
.NET 3 years (LTS) patched every 6 weeks
18 months
Java 3 years (LTS) patched every 12 weeks
6 months
38. Technical
Inflation
Unintended reduction
in value of a software
product over time,
independent of source
code changes.
Depreciation does not
capture two elements:
Unintentionality
Value can be restored Image source: Max Pixel
39. 1974
Continuing Change law
«A[n E-type] system
must be continually
adapted or it becomes
progressively less
satisfactory.»
Image source: WikiMedia
40. Executive
Summary
Software decays rapidly,
and decay rate is speeding
up.
Security is the main force,
but not the only one.
We must improve tooling
and practices to cope with
this increased velocity.
Technical Inflation helps
Management understand
what is going on. Image source: Public Domain
43. References (2/5)
https://heartbleed.com/
Why Every Business Is a Software Business — Watts S. Humphrey Informit, Feb 22, 2002
http://www.informit.com/articles/article.aspx?p=25491
https://en.wikipedia.org/wiki/Watts_Humphrey
https://www.sonatype.com/resources/state-of-the-software-supply-chain-2021
https://www.shopify.com/enterprise/global-ecommerce-statistics
https://blog.cloudflare.com/popular-domains-year-in-review-2021/
https://radar.cloudflare.com/year-in-review-2021
https://snyk.io/blog/net-open-source-security-insights/
https://www.contrastsecurity.com/the-state-of-the-oss-report-2021
https://octoverse.github.com/static/github-octoverse-2020-security-report.pdf
…except that the next day, a new vulnerability has reached prime time, and … yes, Joe app is impacted!
How often this happened to you?
Is it happening more frequenty?
Software is not a problem if not deployed. The most secure computer is off and unplugged.
Who am I?
I work at Unum, a Fortune 500 company, with more than a thousand people in IT.
I studied DevOps for over 10 years and I speak at international conferences. Awarded by Microsoft as Most Valuable Professional on Azure DevOps category in the last few years.
If you want to discuss today’s ideas or other DevOps topics you can reach me at Twitter as giulio_vian or email me directly.
While I try to solve a new problem each day, some issues take years to go away.
How we run the process today?
Publication of a CVE triggers the Security team in the organization,Security team instructs Dev Teams to
fix application code as needed,
code must be deployed to Production under Release Management team supervision
A Release Management role may be required by SOX, Basilea, and similar regulation
Deploy where? Production! We don’t care about the rest (although…), so we need to…
Joe stops and thinks: I need to look at my pom.xml (build.gradle, *.csproj, Makefile, package.json, … name it) for references to Log4J (or whatever is vulnerable).
Oh, but I use SLF4J which in turns… indirect dependencies! I need a tool just to find all possible references recursively.
Oh Oh, our Tomcat configuration is using Log4J! I must check more than my JAR file, says Joe.
…and the next question is…
Here we discuss how to identify:1. the code that needs to be patched
2. the pipeline that release that code in Production
and some issues that one may face:
If more than one branch can reach prod, which one you choose?
How do you match the exact version of code?
Software Composition Analysis kicks in only through pipelines? Is triggered by the deploy pipeline?
The deploy pipeline hasn’t been used in months and doesn’t work anymore (e.g. a token expired, or there is no more an apt agent)
…are there tools to support me and detect vulnerabilities in the code I deliver?Yes, there are BLAH
The vulnerability could be a bad code pattern, use of an API, a vulnerable dependency; in any case we need to find the impacted code.
We must scan all repositories that contain production code. Non-production repositories should be included in the search but listed separately to remove noise.
Some patching can be easily automated, in particular library dependencies listed in project file (e.g. package.json, pom.xml,.csproj, …)
† Lack of blue/green, canary, rolling/progressive deployment
When I’ll be next
We had billions of attacks!? Mostly intercepted email, though, one day it will be successful
Ok but what might be the consequences of an attack?
First batch of crucial elements of a solution identifying the qualities required on Software Configuration Management (SCM).
Moving to Git is a prerequisite because any modern development tool shifted to it.
What about the rest?
Consolidate all projects into a single SCM platform – GitHub, Azure DevOps, BitBucket, GitLabThe one listed have rich API that enable automation and integration
Recommend set of branch names and mandatory tag names to identify code that goes/matches productionThis does not translate that all teams use the same process (e.g. GitFlow) but that they use the same conventions for branch names and, especially for tags
Modern systems allows to attach custom properties to Git repositories and Pipeline definitionsA Tag/Label/Property can identify repo in use, archived, or just with experimental codeSame for pipelines, there are builds with limited scope (quick CI) and builds that deploy to production: use Tag/Label/Property to distinguish
As mentioned, on a small scale, it is easy. Problems raise when you need to manage at scale: more than a few teams, repos, or pipeline.
Consider the scenario where a single vulnerability impacts most of your applications (which is probable when you the majority of you code use the same platform, e.g. Log4J impacting all Java-based applications).
You need to patch lots of repositories and deploy lots of components, each through a separate pipeline.
In such scenario, you need new capabilities:
Global editing tool
Launch most pipelines in parallel (consider batching)
Auto-scale build resources to sustain the spike
Single-approval for the set of pipeline runs
These aren’t offered by current systems.
Can be automated? <pause>To my knowledge there are some tools that do some of the work, like GitHub dependabot
It scans sources and proposes changes via a pull-request mechanism
It does not support all package manager, though, and some features requires GitHub
And clearly we need to input which is the correct version to use. We have seen toolchain attacks were the fix was to rollback, haven’t we?
A crucial pattern to implement is the fast-track (expedite) pipeline.
Every deployable component must have a pipeline that delivers just security fixes with as much regression testing as possible within a limited 2-hours timebox and as little manual steps (e.g. approvals) as possible.
Tip: maybe you need some kind of incremental build to minimize build, test and deploy times.
Current tooling may offer some information but a well-rounded process lot of cross-reference data.
Dependency management is a weak spot in general, SCA (Software Composition Analysis) can identify vulnerabilities in libraries.
Use of API may be caught by security scans
Artifact management tool can track the source (build) of binaries if properly used.
Pipeline knows which repositories they use, what we need here is ability to call a REST API that tell us the dependency.
If you can use such tools, great. Maybe you need to follow a bit of conventions and write some query tools.
In the worst scenario, you have to build and maintain your own database.
Deploy where? Production! We don’t care about the rest (although…), so we need to…
A Release Management role may be required by SOX, Basilea, and similar regulation
But you need speed when it is a 0-day exploit.
For example, you must be able to deploy a patch within hours of its release from a 3rd party (an OSS project or a vendor).
fast-track (expedite) pipelines are not for normal usage: there should be some kind of trigger, like a new CVE, a communication from the Security team or upper management.
What is the way to solve this burning problem?
…they are not decreasing, quite the opposite.
Increasing more than linearly!
…display the same pattern, even more.
Why?
Modern app development is not using just a few suppliers like in the past.
A study from Sonatype confirms that we use a lot more libraries, from all kind of sources, in particular Open Source libraries
and attacks leverage this trend.
The graphs illustrate Javascript scenario, but other languages…
Both graphs illustrate that we, as an industry, aren’t exactly great at reacting and fixing our applications.
The one on the left is data about OSS projects.
The one on the right is more interesting because based on telemetry data, a more significant insight on IT organizations.
.NET Core 3.1
3.1.0 December 3, 2019
3.1.22 December 14, 2021
got 22 patch releases in 3 years i.e. every 45 days/6 weeks
Node v14 (Fermium)
Active LTS start 2020-10-27 v14.15.0
2022-02-01, Version 14.19.0
total 19 releases in 463 days or 66 weeks i.e. every 24.4 days
JDK 11
Java SE 11 (LTS)September 25, 2018
11.0.13+8 (GA), October 19th 2021
total 13 releases(updates) in 1121 days i.e. every 12.3 weeks or 86.2 days
Go 1.16 released 2021-02-16
go1.16.14 (released 2022-02-10)total 14 updates in 360 days i.e. 26 days
go1 (released 2012-03-28) -> go1.17 (released 2021-08-16)
17 major releases in 3429 days or 490 weeks
MongoDB 5.0
5.0.0 - Jul 13, 2021
5.0.6 - January 31, 2022
total 6 releases in 203 days or 29 weeks i.e. every 4.8 weeks
What is the way to solve this burning problem?
Agile and DevOps focused on value-flow
«An E-program is written to perform some real-world activity; how it should behave is strongly linked to the environment in which it runs, and such a program needs to adapt to varying requirements and circumstances in that environment»
“On understanding laws, evolution, and conservation in the large-program life cycle” Lehman M.M. - Journal of Systems and Software Vol. 1, 1979–1980, pp. 213-221
Today, I hope to convince you that we have serious problems in the way we patch and deploy applications, problems that we must address as an industry. At the core a perfectly working application today, is a huge risk tomorrow.
That’s why I speak of decay and rotting, because it is not a slow process. Wear, erosion, rust… They do not convey the urgency and work required to preserve from decay.
#1 unless you put it in a fridge or in a can, it starts smelling very soon
#2 those other processes requires time, while rotting requires quick action to stop it
I am not sure big an effort is to fix processes and tool to cope with security-related problems – the one this audience is acquainted to --. Security is the main driver, although not the only one.
To change process and invest in tools, we have to speak to leadership/executive using a simple but effective vocabulary, so I suggest using the word inflation to convey the idea and start a discussion.
As you may guessed, this presentation is a bit visionary, high-level, I will talk about industry trends and process not technology. For those interested in technology details, I recommend the sessions of my friends Michael Kaufmann and Matteo Emili.
Now you have a couple of minutes to switch if you are not interested.