First presented at Security BSidesLA, Hermosa Beach, California, August 16, 2012
Continuous deployment is characters by a small and frequent changes to production. Find out why it's my #1 security feature. It's not just about pushing fast!
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Continuous Deployment - The New #1 Security Feature, from BSildesLA 2012
1. Continuous Deployment
The New #1 Security Feature
Nick Galbreath Security BSides Los Angeles
http://www.client9.com/ Hermosa Beach
nickg@client.com Aug 16, 2012
@ngalbreath
3. whoami
Nick Galbreath www.client9.com @ngalbreath
• Director of Engineering, Etsy
• enterprise, fraud, security, fun
• But.. on very good terms.
In fact Etsy sponsored my trip here
• VP Engineering, “Company Confidential”
• Stay tuned for details
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
4. Context Alert!
• My background is software development
• Mostly in public, web-facing applications
• Everything from C to PHP
• Your mileage may vary if you are in different
industries, with different risk profiles
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
5. Problem Statement
• Web App vulnerabilities aren’t conceptually that
hard and should be easy to deal with.
• In spite (or because) of our efforts, security is an
“end of line” process or whack-a-mole
• Security education has been at best marginally
useful to developers (in the large, your organization
may be different).
• How can we get ever get ahead?
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
6. How did we get here?
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
8. High Distribution Cost
The Software Product Model is designed for applications
where the cost of distribution is high. Where “high” might be
measure by risk, money, time, resources, customer annoyance.
• Retail, CD/DVDs
• Embedded or Exotic Hardware
• Safety, Medical or Defense Systems
• Operating Systems (phone or desktop)
• Your Homework (1-time deploy)
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
9. SPM -SDLC
Release
QA Ops
QA
Commits
freeze
Specs Development Bug Fix / New Specs
Slush
time in weeks or months
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
10. SPM-Production
Changes to Production
a ng g
B igB B an
Big
Major Release Minor Releases Major Release
New Features going live are 100% correlated
with volume of changes to production.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
11. Nothing wrong with this
given the requirements.
Given high distribution costs, it makes sense to
front-load the development process
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
12. WebApps Y2K
• Mostly followed software product model since
that’s all we knew
• High barrier to entry
• Specialized Hardware, software, people to get
started
• Lots of engineering to keep things running and
scaling.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
13. True Story
• “Can’t push out the spelling error change
since it’s too risky”
• “That code has already been through QA, it’s
locked down.”
• “Product has to prioritize that, else we aren’t
touching it.”
his ls
T l
e
Sm
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
14. WebApps 2012
• Almost no barrier to entry
• Commodity hardware
• “Learn PHP in 24 hours”
• Scaling problems can be outsourced (sorta)
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
15. WebApps 2012 and
Cost of Distribution
• Moving a few megabytes from source control
to a few machines in production over a 1Gb
or 10Gb link.
• In other words... free!
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
16. Given this and
competitive /customer
expectations, it’s not
unreasonable to expect
an SDLC moves faster
than the Software
Product Model
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
17. On the other hand,
WebApps 2012 have very
different failure cases
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
18. The Nature of Failure
• WebApps 2012 are data-driven.
• and frequently have APIs, user-generated content,
social features (unexpected use cases, new problems)
• Failure might be due to algorithm problems, but...
• ...more likely it’s bad user input, bad data in database,
or operational load.
• This means data added in the past might cause
problems in the future. Complicated!
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
19. When SPM meets WebApp2012
• There is a long time between code-written to code-
deployed. This “non-value added” steps for what
should be low-cost changes.
• Might be weeks or months before code deployed.
• Feedback loop between code in dev and code in
production broken.
• When the bug/security report comes in, it’s likely the
engineer is on a different project.
• Any wonder that engineers don’t care for operations
or security?
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
20. Hypothesis
• It is impossible to simulate the production
environment in development, either to
operational differences or data differences.
• No amount of QA or Security Testing can prove
you don’t have bugs, vulnerabilities, or won’t
cause severe operational problems.
• You have bugs and vulnerabilities right now
your site.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
22. • Company wide push to move faster
• Being a bottleneck isn’t acceptable.
• Nor is giving up or saying “need more
resources”
• Engineers disengaged
• Looming security disaster awaits
• Whack-a-Mole doesn’t scale
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
23. If we want to fix
Security,
we have to fix
Development.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
24. Continuous
Deployment
A System of Software Production Characterized By
Numerous Small Changes to the Production Environment
or
That Crazy Shit That Etsy Does. And Google. And
Facebook. And Amazon. And Twitter. And NetFlix.
So maybe not that crazy.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
25. CD -Changes to
Production
new feature
new feature
New Features are not correlated with volume
of changes to production
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
27. What If You Had a Button that said
DEPLOY
•Pushes whatever is on HEAD/TRUNK to production.
•In about a minute.
•Anyone is allowed to push it.
This button logs who performed the change, and what
the change was, but no other rules or controls.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
28. Take 1: Fear
• Likely no one is going to
push it since they are afraid
they’ll break something.
• Meanwhile un-deployed
code keeps piling up.
ex. New hires are terrified of
deploying an... HTML change!
“but I don’t want to break Etsy!”
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
29. Take 2: First Attempt
• At some point, some brave sole will put their
code on TRUNK, and push the button.
• It’s likely someone else tells them that their
feature blew up the site or doesn’t work, and
to please role it back.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
30. Take 3: With Graphs
• The developer learns that they’d don’t know
how the code runs in production and they need
some way of understanding how it works.
• Enter Graphite/Ganglia/StatsD!
http://codeascraft.etsy.com/2011/02/15/
measure-anything-measure-everything/
• Make it free to monitor anything in the
application and expose that to everyone.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
31. Take 4: Push It
• Repushing out code with fix, still causes some
problem as witness by a graph falling off a
cliff, but the developer was aware of it and
was able to role back.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
32. Take 5: Isolation
• Hmmm, the developer in reviewing the code
notices that actually they are pushing a few
bugs fixes, and some other minor features.
• Maybe just pushing out a single bug fix one at
time to help isolate the problem.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
33. Take 6: Success!
• Yes! The developer pushed code and fixed a
bug and made the site just that much better.
• The secret about continuous deployment is
small deltas that you or anyone can
understand easily.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
34. Take 7: Dark Pushes
• Now that the developer got the bugs out of the
way, it time for the feature.
• Let’s push out all the supporting files. By
themselves they do nothing. By getting these
out first, you isolate them as being “unlikely to
cause a site problem”
• Also now that they are on the trunk, others can
look at them (easily).
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
35. Take 8: Rampups
• Now it’s time to get that feature live.
• Instead of a Big Bang, he’ll put a ‘rampup’ in the
code. This will control how many people on the
site will get the new feature.
• Maybe start with “employees only” so his team
can test in production.
• Start at 1%, 5%, 10% and make sure things work,
graphs are stable and work up to 100%.
• if problem, can rampdown or turn off.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
36. Take 8: Eliminate
• Along the way you’ll get burned by little things, so, we’ll
• A development environment that mimics prod as close as
possible (won’t be exact)
• Fast and stable unit and functional tests that are easy to run.
If they are slow and flakey, no one will use them
• Eliminate stupid bugs with commit or pre-commit static
analysis.
• Move QA/Security/Release checks as close as possible to
the developer.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
37. Take 9: Communicate
• As more people get use to it, you’ll need a way
of co-ordinating releases among people.
• IRC works well
• Need set of conventions that match your risk
levels.
• At least developers are talking about releases!
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
38. Take 10: Learn
• No doubt along the way, serious mistakes will be made.
Complex system failures will happen.
• Learn from them. Do Post-Mortems. Do Root-Cause Analysis.
• Recount what happened.
• 99.99999999% of problems are caused by mistakes not
maliciousness
• How can the environment be changed so it doesn’t happen
again?
• Publish the results.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
40. What About That Guy
Who Pushes at 3AM
• That Guy who pushes at 3AM, and something
goes wrong and wakes up all of operations
with pagers going off will quickly learn this
was a bad idea.
• It’s about courtesy and respect.
• Of course there are off-hours exception, in
which That Guy should pre-inform everyone.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
41. What about code
reviews?
• Yup, do them
• Nothing here precludes code reviews.
• In fact, it’s frequently easier to do since the
reviewer doesn’t have to dork around with
branches or tags.... they have all the dark code
already on Trunk/Head
• .. and the reviews are smaller and faster
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
42. What about security
reviews?
• Yup do ‘em.
• Nothing here eliminates your existing review
cycle.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
43. What about Agile
Methods?
• (everyone does “agile differently” so hard to
qualify this).
• Agile methods frequently work to improve the
spec-writing / development cycle
• Or the spec / dev / qa cycle
• But code still pools up waiting to go to
production.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
44. What about Customer Service?
Do they freak out with all the
changes?
• Remember, most changes either do nothing,
or are trivial or are minor.
• Feature launches always need to be co-
ordinated with customer service
(from audience question)
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
45. So Why Did I Tell You
All This?
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
46. That Engineer who previously
didn’t push code is now sensitized
that their code has consequences
and are responsive to fix it.
It’s amazing how interested engineers become in
security when you find problems with their code
when they are able to fix quickly themselves.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
47. Security Fixes can go
out quickly.
In addition, you know fixes can go out since they happen
every day.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
48. You can repurpose the QA stack,
graphing and log analysis for
attack detection and vulnerability
prevention.
Need ideas? Check out these other presentations on
fraud and security by Etsy:
http://slidesha.re/IMaavq
http://slidesha.re/JGaU2s
http://slidesha.re/KPvHYu
http://slidesha.re/Kw5zdV
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
49. While there is always whack-
a-mole, you can focus on
being a service organization
and work on engineering to
be secure by default.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
50. New Roles, Less Silos
• Developers: works with operations
• QA: works on making systems to empower people
to write tests, static analysis, in-house consultancy
on good test design
• Release: tools to push code to production, system
images.
• Security: in house consultancy, security
engineering, secure by default, detection
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
52. Google Chrome
• Really made updates painless for the consumer.
• Frequent changes “regularly” -- maybe not continuous
but way faster than normal software product
• Multiple channels of releases.
• Config flags can turn on or off experimental features.
• Works so well, many others are copying this
approach.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
53. Apps
• Due to cost of deployment being high
(e.g. due to approval from Apple)
• And due to diversity of destination (how many
different types of hardware will it run on), hard to
predict how well it work.
• Put as much as you can into the release
• Then read configs from internet to light up or turn
off features
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
54. Chip Design
• After this talk, I met an engineer who does
hardware design.
• All changes are tiny and then tested, then
committed.
• Any change too big is rejected.
• Learned the hard way that big changes are
impossible to understand and test.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
56. Security is in a Good
Position to Force Change
• Security bridges multiple disciplines: ops,
dev, qa, release, business. Unique position to
make change.
• When breach happens (in whatever the layer),
we need to patch fast.
• I hope that is not controversial.
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
57. Start with
the Deploy Button
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath
58. It will change your SDLC
more resilient
more secure
Nick Galbreath BSidesLA Aug16,2012 @ngalbreath