The Security, DevOps, and Chaos Playbook to Change the World

James Wicke!
— Sr. Sec Eng & Dev Advocate @
Verica
— Author, LinkedIn Learning
— Organizer, DevOps Days Austin
& DevSecOps Days Austin
verica.io // @wickett && @aaronrinehart

Get the slides
wicke!@verica.io

The Security, DevOps,
and Chaos Playbook
to Change the World

Audacious
Talk Title, Yoalso it's 178 slides, so there's that too

The World != Earth

The World == our day to day lives in IT

world as in worldview

worldviewour mental model and conception of reality

No one's worldview
is the same

Dev and Ops and Security do
not share a worldview

Disclaimer: we use DevOps and
DevSecOps interchangeably

Security has known
about the worldview
disconnect for some time

"many security teams
work with a worldview
where their goal is to
inhibit change as much as
possible"

[Security by risk
assessment] introduces a
dangerous fallacy: that
structured inadequacy is
almost as good as
adequacy and that
underfunded security
efforts plus risk
management are about as
good as properly funded
security work

Companies are spending a
great deal on security, but
we read of massive
computer-related attacks.
Clearly something is
wrong. The root of the
problem is twofold: we’re
protecting the wrong
things, and we’re hurting
productivity in the process.

While engineering teams are busy
deploying leading-edge
technologies, security teams are
still focused on ﬁghting yesterday’s
battles.
— SANS 2018 DevSecOps Survey

A Highly Desireable New Breed:
The DevSecOp

...not a tool
...not a CI/CD pipeline
...can’t be bought

An inclusive person
participating in the movement
of security into devops.

DevOps is the inevitable result of
needing to do efﬁcient operations
in a distributed computing and
cloud environment.
— Tom Limoncelli

DevSecOps is the inevitable
result of needing to do
efﬁcient IT in a new world

DevSecOps was needed to ﬁx
the inequitable distribution of
labor

100:10:1
DEV:OPS:SEC

Not just the people

Much
Science

Security has been
on the sidelines
too long

Complex Systems Traits
• Cascading Failures
• Di!cult to determine boundaries
• Difﬁcult to Model Behavior
• Dynamic network of multiplicity
• May produce emergent phenomena
• Relationships are non-linear
• Relationships contain feedback loops

Examples of Complex Systems
• Global Financial Markets
• Nation-State PoliicS
• Weather Patterns
• The Human Body
• Bird Patterns
• Distributed Computing Systems (aka your systems)

Fact: Outages & Breaches will
continue to get worse

DEVSECOPS

But, how?

MEASURE

Makers
Experimentation
Automation
Safety
Unrestrained Sharing
Rugged
Empathy

Makers

We are so!ware engineers who
specialize in a speciﬁc discipline:
security

Security must be able to
write code

Unfortunately, that triggers
some security professionals
I am sorry for that

How about

Security should participate in the
so!ware delivery practice and use the
language of their peers as they
ship so!ware together

With all the resources
available today...

Security is part of the
making

Why? Shi!ing Focus on what
ma"ers most.

Value Chain
As a security professional can you
explain where you ﬁt in your
company’s value chain?

Does the company exist to deliver
product and services or employee
desktops?

Everyone is responsible for the
engineering not just the security.

Outcomes:
— Empathy building
— Familiarity with tools
— Able to move up the pipeline

A bug is a bug is a bug

Defect Density studies range
from .5 to 10 defects per KLOC

Defect density
is never zero

With framework/deps,
500 LOC you write can
easily be 400,000 LOC

You cannot train developers
to write secure code
all of the time

Instead, focus on Methods Developers use
— TDD/BDD/ATDD
— Meaningful comments/commits
— Code Smells, Patterns, Refactoring
— Instrumentation, Observability

The goal should be to
come up with a set of
automated tests that
probe and check security
conﬁgurations and
runtime system behavior
for security features that
will execute every time
the system is built and
every time it is deployed.

Security is connected with
quality

Makers— See security as part of engineering
— View quality as a way to bring in security
— Use code to solve problems

Experimentation
and Learning

Beneﬁts to Experimentation
— Measured, Repeatable
— Results based on your needs
— Actionable Outcomes

Security incidents are not effective
measures of detection because at
that point it's already too late
— Aaron Rinehart

Know Most Likely A!acks and How
to Measure Abuse and Misuse

We can't cede home ﬁeld
advantage
— Zane Lackey

Experimenting necessitates
understanding steady state

Resources
— Shannon Lietz (@devsecops)
— DOES 2018 Talk: youtu.be/yuOuVC8xljw

Automation

Continuous Delivery is how little
you can deploy at one time
— Jez Humble & David Farley

Optimize total cycle time from
code commit to running in prod

15,000deploys in 3.5 years

Security in the Pipeline
— Software composition analysis
— Lang linters, git-hound, ...
— Scanners, gauntlt
— Monitoring and telemetry

[Deploys] can be
treated as standard
or routine changes
that have been pre-
approved by
management, and
that don’t require a
heavyweight change
review meeting.

Resources:

linkedin.com/learning/devsecops-building-a-secure-
continuous-delivery-pipeline

linkedin.com/learning/devsecops-automated-security-
testing

Safety

Safety and Security
have a lot in common

Fact: No system is secure on
its own, it requires humans to
create it

Safety Differently Origins
"Safety differently' is about relying
on people’s expertise, insights and
the dignity of work as actually done
to improve safety and efﬁciency. It is
about halting or pushing back on
the ever-expanding
bureaucratization and compliance
of work."
— Sydney Dekker

"Security differently’ is about relying
on people’s expertise, insights and the
dignity of work as actually done to
improve security and efﬁciency. It is
about halting or pushing back on the
ever-expanding bureaucratization and
compliance of work."

Security Currently vs. Security Differently
Security Currently Security Differently
People are the Source of
Problems
People are the Solution
Tell them what to do Ask them what they need
(Control & Compliance) Competency & Common Sense
Count absence of Negative
events
Count Presence of Positives

Outcomes are the ultimate
measurement of effectiveness

Why do outages and breaches
seem to be happening more o!en?

Flawed Understanding
Our understanding of our systems
has become fundamentally ﬂawed

Simple vs. Complex Systems

Simple Systems:
Linear in nature
Easy to Predict
Able to comprehend

Complex Systems:
Non-linear (bullwhip effect)
Unpredictable in nature
No mental model available

Question: How well do you
really understand your own
systems? I mean really.

System Engineering is a
Messy Affair

Complex Systems are
Challenging

Root Cause (in a complex system) is a Myth
— Lacks full picture
— Complex systems are not linear
— Result of blame culture
— Forgets organizational decisions
— Puts the focus on the event over situation

Humans aren’t the problem,
they are the solution

Field Guide to 'Human-Error'
Investigations by Sydney Dekker

Old View
— Human Error is a cause of trouble
— You need to ﬁnd people’s mistakes, bad judgements
and inaccurate assessments
— Complex Systems are basically safe
— Unreliable, erratic humans undermine systems safety
— Make systems safer by restricting the human
condition

New View
— Human error is a symptom of deeper system trouble
— Instead, understand how their assessment and actions
made sense at the time - context matters
— Complex systems are basically unsafe
— Complex systems are tradeoffs between competing
goals safety vs. efﬁciency
— People must create safety through practice at all
levels

Failure is an
inevitable by-
product of a
complex system's
normal functioning

Failures are a systems problem
because there is not enough safety
margin.
— @adrianco

Where Security Fits
— Add safety margin
— Telemetry and instrumentation
— Blameless retros
— ...more to explore in this area

Resources
— Drift into Failure by Dekker
— Safety Differently by Dekker
— Understanding Human Error Video Series youtu.be/
Fw3SwEXc3PU
— Richard Cook paper bit.ly/2ydDQS2

Unrestrained
Sharing

Culture is the most important
aspect to devops succeeding in
the enterprise
— Patrick DeBois

DevSecOps is the extension of
the DevOps culture for the
inclusion of Security

A security team who embraces
openness about what it does and
why, spreads understanding.
— Rich Smith

Unrestrained Sharing
affects culture

Unrestrained Sharing goes
against security's standard
operating procedure

It will feel uncomfortable

Sharing breaks
down silos

Four Keys to Culture
— Mutual Understanding
— Shared Language
— Shared Views
— Collaborative Tooling

20% of developers don't know
what security expects of them

Security Shares Through
— Making invisible as visible
— Security Observability
— APIs, webhooks, dev tooling

Creating a Just Culture

Blameless and
Just Culturelets get rid of the Name, Blame and
Shame Culture

Fear of punishment inhibits
learning

Sharing includes
auditors

Resources
— Phoenix Project
— Agile Application Security
— dearauditor.org

Rugged

So!ware Bill of Materials
Know what you have

Favor Short Lived Systems

DIE Framework
Distributed
Immutable
Ephemeral
source: @sounilyu

Rugged in 2020
Deception
Chaos Eng

Deception
— Honeypots, Tarpits, Mantraps
— Simple to get started (http headers)
— HoneyPy, DeceptionLogic

We’re moving from disaster
recovery to chaos engineering to
resiliency
— @adrianco

Resilience doesn’t mean
what you think it means

Resilience != DR/BCP

Our systems almost never fail
the way we think it will

Resilience is the ability of systems
to prevent or adapt to changing
conditions in order to maintain
control over a system property…to
ensure safety… and to avoid failure.
— Hollnagel, Woods, & Leveson

Failure is the
Normal Condition

Where is the Safety Margin in
Security?

How do you know when your
ge!ing close to falling off the
edge?
Do we have to fall off of the
edge to know where it is?

[Chaos Engineering is] empirical
rather than formal. We don’t use
models to understand what the
system should do. We run
experiments to learn what it does.
— Michael Nygard, Release It 2nd Ed.

Security LOVES Chaos Engineering

Chaos Engineering Flips the Model
Postmortems = Preparation

Create Objective Feedback
Loops about Security
Effectiveness

The security discipline of [chaos]
experimentation is done in order
to build conﬁdence in the system’s
ability to defend against malicious
conditions.

Chaos Engineering
— Experiments that span eng and security
— Manual opt-out
— Valuable Learning
— Controlled experiment blast radius

Security Problems that Plague Complex Systems
— Conﬁguration drift over time
— Regressions in code where previously solved vulnerabilities are
reintroduced
— Role and privilege changes for users and applications
— Additive code or microservice that introduces a downstream
vulnerability
— Security controls not placed in correct locations in the system
— Bullwhip effect in systems where small security change results
in widespread outage

Outside-In Approach to SCE
— SCE does not validate a conﬁg, it exercises it
— SCE does not check auth privileges, it attempts to
thwart them
— SCE does not trust network settings, it sends real trafﬁc
— SCE does not check app policy, it interacts with the
application
— SCE does not build a model from infrastructure
templates, it builds understanding from experimentation

4 Steps of Security Chaos Engineering
— Deﬁne expected behavior of a security defense
— Hypothesize that when security turbulence is
introduced it will be either prevented, remediated, or
detected.
— Introduce a variable that introduces security
turbulence.
— Try to disprove the hypothesis by looking for a
difference in expected behavior and actual behavior

Example: ChaoSlingr: Misconﬁgured Port

Resources
— Aaron Rinehart's talk at RSA youtu.be/wLlME4Ve1go
— principlesofchaos.org
— Release It! 2nd ed., Nygard
— Phillip Maddux's talk: youtu.be/k81xKjCEeqE
— Herb Todd's talk: youtu.be/Cf_XXmRLnRQ

Empathy

Empathy
is a two way street

Rewarding Empathy Experiment
Training 1000 Security Professionals
to Code Python

those stupid developers
— Security

you want a machine powered off
and unplugged
— Developer

Half of developers say that
don't have enough time to
spend on security

Don’t be a blocker
be an enabler

New O'Reilly Book on SCE

Next Week:
* Dive into 1-2 of the MEASURE areas
* Find a place to add value
In a Month:
* Present on MEASURE to your company
* Bring Dev/Ops/Sec together to ﬁnd a joint
improvement
In 6 months:
* Share externally your DevSecOps Journey

Get the slides
and book update info
wicke!@verica.io

The Security, DevOps, and Chaos Playbook to Change the World

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Semelhante a The Security, DevOps, and Chaos Playbook to Change the World

Semelhante a The Security, DevOps, and Chaos Playbook to Change the World (20)

Mais de James Wickett

Mais de James Wickett (11)

Último

Último (20)

The Security, DevOps, and Chaos Playbook to Change the World