1.
DEFENSE-ORIENTED DEVOPS FOR
MODERN SOFTWARE DEVELOPMENT
James Wickett, Signal Sciences
@wickett
1

2.
DEFENSE-ORIENTED DEVOPS
FOR MODERN SOFTWARE DEVELOPMENT
@WICKETT

3.
@WICKETT
Want the slides and
referenced links?
james@signalsciences.com

4.
@WICKETT
‣ HEAD OF RESEARCH AT SIGNAL SCIENCES
‣ ORGANIZER OF DEVOPS DAYS AUSTIN
‣ LYNDA.COM AUTHOR ON DEVOPS
‣ BLOG AT THEAGILEADMIN.COM
@WICKETT

5.
@WICKETT
‣ BUILT TO DEFEND WEB APPLICATIONS AND
MICROSERVICES AT CLOUD SCALE
‣ DEFENDING OWASP TOP TEN, ATO, APP DDOS,
AUTH ATTACKS, BOTS, SCRAPERS
‣ TRUSTED BY SOME OF THE LARGEST
COMPANIES ON THE INTERNET: ETSY, ADOBE,
VIMEO, CHEF, DATADOG
SIGNAL SCIENCES WEB
PROTECTION PLATFORM

6.
Agent

7.
@WICKETT
‣ DEVOPS IS CHANGING AND THERE IS A BIG RISK
TO LOSE OUR WAY.
‣ SECURITY IS IN CRISIS
‣ SECURITY AT FORWARD-LEANING SHOPS HAVE
FOUND THE NEW WAY.
‣ LET’S JUXTAPOSE THE OLD WAY AND THE NEW
WAY OF SECURITY IN DEVOPS.
SUMMARY

8.
@WICKETT
‣ WHY DO WE HAVE DEVOPS?
‣ DID WE BUILD DEVOPS PROPERLY?
‣ IS THE DEVOPS CULTURE LOST?
‣ CAN WE GET IT BACK?
‣ CAN WE PROTECT DEVOPS FROM FURTHER
DISTORTION?
QUESTIONS ON MY MIND

9.
@WICKETT
My Journey

10.
@WICKETT
‣ WEB AND ECOMM FOR $1B COMPANY
‣ BRUTAL ONCALL ROTATIONS
‣ +24HR DEPLOYMENTS
‣ WATERFALL, WATERFALL, WATERFALL
‣ FRIENDS ARE BORN FROM ADVERSITY
FIRST BIGCO JOB

11.
@WICKETT
‣ IN 2007 WENT STARTUP AND AWS CLOUD
‣ LEARNED A BIT ABOUT FAILURE AND
HAPPINESS
‣ REJOINED OLD TEAM IN 2010 FOR NEW CLOUD
VENTURE BACK IN BIGCO
CLOUDING FOR PROFIT

12.
@WICKETT
‣ DEVOPS AND INFRA AS CODE
‣ NOT CD, BUT DEPLOYS DAILY
‣ AT BIGCO DELIVERED 4 SAAS PRODUCTS IN 2
YEARS WITH DEVOPS AND CLOUD
ENTER DEVOPS

13.
@WICKETT
‣ FOUND RUGGED SOFTWARE
‣ MET GENE KIM IN 2012 IN A BAR IN AUSTIN
‣ CREATED GAUNTLT
‣ LATER, JOINED SIGNAL SCIENCES
DEVOPS AND SECURITY

14.
@WICKETT
DevOps is Friendship

15.
@WICKETT
Compassion for Ops

16.
@WICKETT
10:1
DEV:OPS

17.
@WICKETT
Labor Inequity
Permeates IT Ranks

18.
@WICKETT
100: 10: 1
DEV:OPS:SEC

19.
@WICKETT
Yet, I remained
optimistic
for DevOps + Security

20.
@WICKETT
ENTER DOUBTS

21.
@WICKETT
‣ DEVOPS ON A BUS AT RSA
‣ EXPO FLOOR AT DOCKER CON AND THE
DEVOPS TOOLCHAIN
TWO EVENTS

22.
@WICKETT
HAD WE ALLOWED DEVOPS TO BE
A NEW GIMMICK OR SLOGAN?

23.
@WICKETT
WHAT HAD DEVOPS BECOME?

24.
@WICKETT
‣ WHY DO WE HAVE DEVOPS?
‣ DID WE BUILD DEVOPS PROPERLY?
‣ IS THE DEVOPS CULTURE LOST?
‣ CAN WE GET IT BACK?
‣ CAN WE PROTECT DEVOPS FROM FURTHER
DISTORTION?
QUESTIONING DEVOPS

25.
@WICKETT
OUR ROOTS: FRIENDSHIP

26.
@WICKETT
There is irony in my
story…

27.
@WICKETT
‣ TEACH THREE DEVOPS CLASSES IN THE DEVOPS
FOUNDATIONS SERIES AT LYNDA / LINKEDIN
LEARNING
‣ WORK AT A POPULAR VENDOR OF DEVSECOPS
SOLUTIONS
‣ WRITE DEVOPS AND SECURITY ARTICLES AS
PART OF MY ROLE AT SIGNAL SCIENCES

28.
@WICKETT
Back to Our Roots

29.
@WICKETT
Culture is the most important
aspect to devops succeeding in
the enterprise
- Patrick DeBois

30.
@WICKETT

31.
@WICKETT
‣ MUTUAL UNDERSTANDING
‣ SHARED LANGUAGE
‣ SHARED VIEWS
‣ COLLABORATIVE TOOLING
4 KEYS TO CULTURE

32.
@WICKETT
FRIENDSHIP

33.
@WICKETT
Make a friend through
your journey today at
SpringOne Platform

34.
@WICKETT
Security is in Crisis

35.
@WICKETT
Companies are spending a great deal on
security, but we read of massive computer-
related attacks. Clearly something is wrong.
The root of the problem is twofold:
we’re protecting the wrong things,
and we’re hurting productivity in the process.
THINKING SECURITY, STEVEN M. BELLOVIN 2015

36.
@WICKETT

37.
[Security by risk assessment] introduces
a dangerous fallacy: that structured
inadequacy is almost as good as
adequacy and that underfunded security
efforts plus risk management are about
as good as properly funded security work

38.
@WICKETT
Security is often the
cultural outlier in an
organization

39.
@WICKETT
many security teams work
with a worldview where their
goal is to inhibit change as
much as possible

40.
“SECURITY PREFERS A SYSTEM POWERED
OFF AND UNPLUGGED”
- DEVELOPER

41.
“…THOSE STUPID DEVELOPERS”
- SECURITY PERSON

42.
@WICKETT
It is 30 times cheaper
to fix security defects
in dev vs. Prod
NIST, 2002, The Economic Impacts of Inadequate Infra for Software Testing

43.
@WICKETT
It is 30 times cheaper
to fix security defects
in dev vs. Prod
NIST, 2002, The Economic Impacts of Inadequate Infra for Software Testing

44.
@WICKETT
Security must
Change or Die

45.
@WICKETT
“EVERY ASPECT OF MANAGING WAFS IS AN ONGOING
PROCESS. THIS IS THE ANTITHESIS OF SET IT AND FORGET IT
TECHNOLOGY. THAT IS THE REAL POINT OF THIS RESEARCH.
TO MAXIMIZE VALUE FROM YOUR WAF YOU NEED TO GO IN
WITH EVERYONE’S EYES OPEN TO THE EFFORT REQUIRED TO
GET AND KEEP THE WAF RUNNING PRODUCTIVELY.”
- WHITEPAPER FROM AN UNDISCLOSED WAF VENDOR

46.
@WICKETT

47.
@WICKETT
Bottleneck Approach

48.
THE AVERAGE TIME TO DELIVER CORPORATE IT PROJECTS
HAS INCREASED FROM ~8.5 MONTHS TO OVER 10
MONTHS IN THE LAST 5 YEARS
Revving up your Corporate RPMs, Fortune Magazine, Feb 1, 2016
THE GROWTH OF [SECURITY] FUNCTIONS WHICH IS TOO
OFTEN POORLY COORDINATED… [RESULTING IN] A
PROLIFERATION OF NEW TASKS IN THE AREAS OF
COMPLIANCE, PRIVACY AND DATA PROTECTION.

49.
@WICKETT
Many security professionals
have a hard time adapting their
existing practices to a world
where requirements can change
every few weeks, or where they
are never written down at all.

50.
@WICKETT

51.
@WICKETT

52.
@WICKETT
Security didn’t get an
invite to the DevOps
party!
- John Willis (@botchagalupe)
“You Build It, You Secure It” DOES 2017

53.
@WICKETT
Read-only containers and
serverless shift the
security story to almost
100% application
security

54.
@WICKETT
DevOps
A New Traveling Companion
for Security
(…and probably the only way to survive)

55.
@WICKETT
High performers spend 50 percent less
time remediating security issues than
low performers.
By better integrating information
security objectives into daily work,
teams achieve higher levels of IT
performance and build more secure
systems.
2016 State of DevOps Report

56.
@WICKETT
High performing orgs achieve
quality by incorporating
security (and security teams)
into the delivery process
2016 State of DevOps Report

57.
@WICKETT
http://www.youtube.com/watch?v=jQblKuMuS0Y

58.
@WICKETT
The New Path

59.
@WICKETT
Embrace Secrecy Create Feedback Loops
Just Pass Audit! Compliance adds Value
Enforce Stability Create Chaos
Build a Wall Zero Trust Networks
Slow Validation Fast and Non-blocking
Certainty Testing Adversity Testing
Test when Done Shift Left
Process Driven The Paved Road
OLD PATH VS. NEW PATH

60.
@WICKETT
Embrace Secrecy Create Feedback Loops
Just Pass Audit! Compliance adds Value
Enforce Stability Create Chaos
Build a Wall Zero Trust Networks
Slow Validation Fast and Non-blocking
Certainty Testing Adversity Testing
Test when Done Shift Left
Process Driven The Paved Road
OLD PATH VS. NEW PATH

61.
@WICKETT
A security team who embraces
openness about what it does and
why, spreads understanding.
- Rich Smith

62.
@WICKETT
Runtime is arguably the
most important place to
create feedback loops

63.
@WICKETT
‣ ACCOUNT TAKEOVER ATTEMPTS
‣ AREAS OF THE SITE UNDER ATTACK
‣ MOST LIKELY VECTORS OF ATTACK
‣ BUSINESS LOGIC FLOWS
DETECT WHAT MATTERS

64.
@WICKETT
Are you under attack?

65.
@WICKETT
Where?

66.
@WICKETTWhich is a better feedback
loop?
Source: Zane Lackey, Signal Sciences

67.
@WICKETT
Options: RASP, NGWAF or
Web Protection Platform

68.
@WICKETT
‣ SURFACE LEVEL
‣ WHAT WENT WRONG? HOW DID IT BREAK? HOW
DO WE FIX IT?
‣ DEEPER LEVEL
‣ WHAT ARE THINGS THAT WENT INTO MAKING IT
NOT AS BAD AS IT COULD HAVE BEEN?
ALL INCIDENTS CAN BE WORSE
Source: John Allspaw, DOES 2017

69.
@WICKETT
Embrace Secrecy Create Feedback Loops
Just Pass Audit! Compliance adds Value
Enforce Stability Create Chaos
Build a Wall Zero Trust Networks
Slow Validation Fast and Non-blocking
Certainty Testing Adversity Testing
Test when Done Shift Left
Process Driven The Paved Road
OLD PATH VS. NEW PATH

70.
@WICKETT
‣ POLICIES AND PROCEDURES IN PLACE
‣ EFFECTIVE EXECUTION OF THOSE POLICIES TO
ALLOW YOU TO KEEP FUNCTIONING
‣ MOST OF PCI AND OTHER FRAMEWORKS
PROVIDE REASONABLY GOOD PRACTICES *IF*
YOU REMOVE ALL THE WATERFALL BITS
UNDERSTAND AUDITORS

71.
@WICKETT
[Deploys] can be treated as
standard or routine changes
that have been pre-approved
by management, and that
don’t require a heavyweight
change review meeting.

72.
SEPARATION OF DUTIES CONSIDERED HARMFUL

73.
PCI 6.4.2

74.
@WICKETT
In environments where one individual
performs multiple roles (for example,
administration and security operations), duties
may be assigned such that no single
individual has end-to-end control of a process
without an independent checkpoint.
(aka Auditable Delivery Pipeline)

75.
@WICKETT
Developers with Access
to Production, Oh My!!!
https://www.schellmanco.com/blog/2012/12/auditing-devops-developers-with-access-to-
production/

76.
@WICKETT
Check out DevOps Audit
Defense Toolkit
https://cdn2.hubspot.net/hubfs/228391/Corporate/
DevOps_Audit_Defense_Toolkit_v1.0.pdf

77.
@WICKETT
Embrace Secrecy Create Feedback Loops
Just Pass Audit! Compliance adds Value
Enforce Stability Create Chaos
Build a Wall Zero Trust Networks
Slow Validation Fast and Non-blocking
Certainty Testing Adversity Testing
Test when Done Shift Left
Process Driven The Paved Road
OLD PATH VS. NEW PATH

78.
@WICKETT
‣ ADD IN CHAOS TO YOUR SYSTEM AND
APPLICATION
‣ CHAOS MONKEY
‣ ANTI-FRAGILE
‣ RELEASE IT! BOOK
CHAOS ENGINEERING

79.
@WICKETT

80.
@WICKETT
‣ ADDS MISCONFIG TO THE STACK AND CHECKS
TO SEE IF IT GETS DETECTED
‣ NEW OPEN SOURCE TOOL!
‣ RUNS AS A LAMBDA
CHAOS SLINGR

81.
@WICKETT
‣ I AM BEING PEN TESTED ANYWAY, WHY NOT
FIND OUT WHAT THEY ARE FINDING?
‣ 24/7 PEN TESTING
‣ BUILDS DEVELOPER CONFIDENCE
‣ FINDS MIX OF LOW HANGING FRUIT AND
SOMETIMES MUCH MORE!
BUG BOUNTIES

82.
@WICKETT
‣ HACKERONE
‣ BUGCROWD
BUG BOUNTY OPTIONS

83.
@WICKETT
Embrace Secrecy Create Feedback Loops
Just Pass Audit! Compliance adds Value
Enforce Stability Create Chaos
Build a Wall Zero Trust Networks
Slow Validation Fast and Non-blocking
Certainty Testing Adversity Testing
Test when Done Shift Left
Process Driven The Paved Road
OLD PATH VS. NEW PATH

84.
@WICKETT

85.
@WICKETT
‣ NO PERIMETER SECURITY
‣ ASSUME COMPROMISE
‣ INSTRUMENT ALL LAYERS
‣ EXTENDS FROM LAPTOPS TO WEB
APPS TO CUSTOMER ACCOUNTS
ZERO TRUST NETWORKS

86.
@WICKETT
Embrace Secrecy Create Feedback Loops
Just Pass Audit! Compliance adds Value
Enforce Stability Create Chaos
Build a Wall Zero Trust Networks
Slow Validation Fast and Non-blocking
Certainty Testing Adversity Testing
Test when Done Shift Left
Process Driven The Paved Road
OLD PATH VS. NEW PATH

87.
@WICKETT
‣ DON’T SLOW DELIVERY
‣ CONTINUOUS TESTING AND VALIDATION
‣ TESTING ON THE SIDE OF THE PIPELINE
‣ PENETRATION TESTING OUTSIDE OF DELIVERY
FAST AND NON-BLOCKING

88.
@WICKETT
Currently, at Signal
Sciences we do about 15
deploys per day

89.
@WICKETT
Roughly 10,000 deploys
in the last 2.5 yrs

90.
@WICKETT

91.
@WICKETT
CD is how little you
can deploy at a time

92.
@WICKETT
We optimized for cycle
time—the time from code
commit to production

93.
GAVE POWER TO THE TEAM TO DEPLOY

94.
@WICKETT
Signal Sciences is a
software as a service
company and a security
company

95.
@WICKETT
Security is part of CI/
CD and the overall
delivery pipeline

96.
@WICKETT
‣DESIGN
‣INHERIT
‣BUILD
‣DEPLOY
‣OPERATE
PIPELINE PHASES

97.
@WICKETT
‣INHERIT
‣BUILD
‣OPERATE
SECURITY CONSIDERATIONS
What have I bundled into my
app that leaves me
vulnerable?
Do my build acceptance
tests and integration tests
catch security issues before
release?
Am I being attacked right
now? Is it working?

98.
@WICKETT
Embrace Secrecy Create Feedback Loops
Just Pass Audit! Compliance adds Value
Enforce Stability Create Chaos
Build a Wall Zero Trust Networks
Slow Validation Fast and Non-blocking
Certainty Testing Adversity Testing
Test when Done Shift Left
Process Driven The Paved Road

99.
@WICKETT
Be Mean to Your Code

100.
@WICKETT
The goal should be to come up with a
set of automated tests that probe and
check security configurations and
runtime system behavior for security
features that will execute every time
the system is built and every time it is
deployed.

101.
@WICKETT
Security tools are
intractably noisy and
difficult to use

102.
@WICKETT
A method of
collaboration was
needed for devs, ops
and security eng.

103.
@WICKETT
There needed to be a
new language to span
the parties

104.
@WICKETT
Started Gauntlt
4 years ago

105.
@WICKETT

106.
@WICKETT
Open source, MIT License
Gauntlt comes with pre-canned steps that
hook security testing tools
Gauntlt does not install tools
Gauntlt wants to be part of the CI/CD pipeline
Be a good citizen of exit status and stdout/
stderr

107.
@WICKETT
gauntlt.org

108.
@WICKETT

109.
@WICKETT

110.
@WICKETT

111.
@WICKETT
$ gem install gauntlt
# download example attacks from github
# customize the example attacks
# now you can run gauntlt
$ gauntlt

112.
@slow @final
Feature: Look for cross site scripting (xss) using arachni
against a URL
Scenario: Using arachni, look for cross site scripting and verify
no issues are found
Given "arachni" is installed
And the following profile:
| name | value |
| url | http://localhost:8008 |
When I launch an "arachni" attack with:
"""
arachni —check=xss* <url>
"""
Then the output should contain "0 issues were detected."
Given
When
Then
What?

113.
@WICKETT
“We have saved millions of
dollars using Gauntlt for
the largest healthcare
industry project.”
- Aaron Rinehart, UnitedHealthCare

114.
http://bit.ly/2s8P1Ll

115.
@WICKETT
‣ 8 LABS FOR GAUNTLT
‣ HOW TO USE GAUNTLT FOR NETWORK CHECKS
‣ GAUNTLT FOR XSS, SQLI, OTHER APSES
‣ HANDLING REPORTING
‣ USING ENV VARS
‣ CI SYSTEM SETUP
WORKSHOP INCLUDES:

116.
github.com/gauntlt/gauntlt-demo

117.
github.com/gauntlt/gauntlt-starter-kit

118.
SOURCE: THE
THREE WAYS OF
DEVOPS, GENE KIM

119.
@WICKETT
Most teams use Gauntlt
in Docker containers

120.
@WICKETT
https://github.com/
gauntlt/gauntlt-docker

121.
@WICKETT
Embrace Secrecy Create Feedback Loops
Just Pass Audit! Compliance adds Value
Enforce Stability Create Chaos
Build a Wall Zero Trust Networks
Slow Validation Fast and Non-blocking
Certainty Testing Adversity Testing
Test when Done Shift Left
Process Driven The Paved Road

122.
@WICKETT
Red Team Mondays
at Intuit

123.
@WICKETT
But, but, containers!

124.
@WICKETT
OVER 30% OF OFFICIAL IMAGES IN
DOCKER HUB CONTAIN HIGH PRIORITY
SECURITY VULNERABILITIES
https://banyanops.com/blog/analyzing-docker-hub/

125.
@WICKETT
Embrace Secrecy Create Feedback Loops
Just Pass Audit! Compliance adds Value
Enforce Stability Create Chaos
Build a Wall Zero Trust Networks
Slow Validation Fast and Non-blocking
Certainty Testing Adversity Testing
Test when Done Shift Left
Process Driven The Paved Road

126.
@WICKETT
‣ MAKE IT EASY FOR PEOPLE TO DO THE RIGHT
THING
‣ JASON CHAN, NETFLIX
‣ GOLD IMAGES
‣ BLESSED BUILDS AND DEPENDENCIES
THE PAVED ROAD

127.
@WICKETT
Don’t be a blocker, be
an enabler of the
business

128.
@WICKETT
Want the slides and
referenced links?
james@signalsciences.com

129.
LEARN MORE. STAY CONNECTED.
Free eBook:
https://info.signalsciences.com/book
134
#springone@s1p