Keeping The Auditor Away: DevOps Audit Compliance Case Studies

@RealGeneKim
@jdeluccia
Session ID:
Gene Kim
James DeLuccia
Keeping The Auditor Away:
DevOps Audit Compliance Case
Studies

@RealGeneKim
@jdeluccia
OMG. Developers Deploying Code?!?

@RealGeneKim
@jdeluccia
Introductions
Gene Kim
▪ Co-author of "The Phoenix Project”
▪ Founder and CTO of Tripwire, Inc. for
13 years
▪ Worked with Jez Humble (co-author
of “Continuous Delivery book) to
benchmark 14K technology
organizations
▪ Co-chaired SOX-404 Scoping
Committee at the Institute of Internal
Auditors (2005)
James DeLuccia
▪ Author, “IT Compliance & Controls”
▪ Ernst & Young, leader for Americas
Certification & Compliance Services
▪ Focus: startups, technology,
governance, security
▪ Patent holder - crypto privacy
comparison system

@RealGeneKim
@jdeluccia
Golly, Why Are You Attending This Talk?
▪ How many people have to deal with compliance?
▪ On a scale of 1-10, how painful are your
interactions with auditors? (1=delightful,
10=awful beyond words?)

@RealGeneKim
@jdeluccia
Problem Statement
Gene ● DevOps and continuous delivery introduce problems with audit,
because the work patterns are so different than traditional SDLC
● Agile also had issues (e.g., testing at end of project, requirements
phase at the beginning), but is not as radical as DevOps
○ tens/hundreds of deploys/day (change is risk; can’t rely on
change approvals, separation of duty)
● No widespread agreement on what DevOps control requirements
should look like
James ● Auditors must work off a mature and testable environment
● They must stake their livelihood that what you say is correct,
completely
● A partnership is needed between you and them to ensure such an
environment exists (of course, it also needs to operate and be
amazing .. but that is another talk)

@RealGeneKim
@jdeluccia
Agenda
▪ The Top-Down, Risk Based Audit Process
▪ What Goes Wrong
▪Scoping
▪Control Testing
▪ Scenarios From The DevOps Audit Defense Toolkit
▪ Ask An Auditor Anything!

@RealGeneKim
@jdeluccia
The DevOps Audit Defense Toolkit
http://bit.ly/DevOpsAudit
James DeLuccia IV
Jeff Gallimore
Gene Kim
Byron Miller

@RealGeneKim
@jdeluccia
What Is Audit
▪ Management is defined as those who are there to achieve the goals of the
organizations, which includes the officers of the company (e.g., CEO, CFO,
etc.), executives and managers, as well as everyone who reports to them.
▪ Includes some board of directors, GRC departments
▪ Audit is defined to be the function inside the organization that resides
outside of management to serve as an independent, objective source of
assurance that the organization can achieve its goals.
▪ Includes internal auditors, external auditors (regulators, assessors,
etc.)

@RealGeneKim
@jdeluccia
Internal Controls
“a process, effected by an organization’s board of
directors, management, and other personnel,
designed to provide reasonable assurance
regarding the achievement of objectives related to
operations, reporting, and compliance.”
- Operations (effectiveness, efficiency)
- Financial Reporting (accuracy of account
balances and values)
- Compliance (with relevant laws and
regulations, contractual obligations: PCI DSS,
US Export Law, FEDRAMP, SOC-2)
Source: http://coso.org (Committee of Sponsoring Organizations for the Commission on Fraudulent Financial Reporting)

@RealGeneKim
@jdeluccia
How Audit Plans Are Built And Run
▪ Business objectives
▪ Risks
▪ Control objectives
▪ Control procedures
Unfortunately, most contact with auditors start with control procedures…
It’s totally appropriate to ask to show work and start from beginning...

@RealGeneKim
@jdeluccia
The Audit Cycle
▪ Planning
▪ Gaining an understanding of the organization
▪ Scoping
▪ Sampling, reporting period, types of evidence needed, recipient of
report
▪ Schedule
▪ Fieldwork
▪ Controls testing
▪ Substantive testing
▪ Reporting
▪ Management responses
▪ Attestation by auditor and delivered to regulator/clients

@RealGeneKim
@jdeluccia
When Scoping Goes Wrong

@RealGeneKim
@jdeluccia
▪ 2001: Enron fails ($63B market
cap), Arthur Andersen dissolution
▪ 2002: WorldCom (peak $117B
market cap)
▪ Leads to Sarbanes-Oxley Act of
2002

@RealGeneKim
@jdeluccia
Source: KPMG

@RealGeneKim
@jdeluccia
Problem: Bottom Up Auditing
Source: ISACA

@RealGeneKim
@jdeluccia
Analysis: Audit control testing work was scoped properly,
linking controls to compliance objectives and risk.
Control failures must result potentially undetected
material financial reporting errors
The Problem: Improperly Scoped Audits

@RealGeneKim
@jdeluccia
Financial Reporting Material Weakness
What happens when an audit generates a material weakness?

@RealGeneKim
@jdeluccia
Under-Scoping Operating Risk

@RealGeneKim
@jdeluccia
▪ When we don’t understand why we are being audited
▪ “Why are we doing this audit?” (customers, SOX, regulatory; who is it
for?)
▪ When we are asked for something we don’t have (e.g., “evidence of SoD or
change approvals)
▪ “What is the control objective? Can we rewrite the control procedure
for this asset?”
▪ Do this before the auditor shows up
When Auditors Attack Unexpectedly
These are delicate conversations, with potentially large
impacts on scope, cost, risk...

@RealGeneKim
@jdeluccia
▪ If we are reacting to these conversations before we’ve done any of our
homework, we may be trouble
▪ Extra work (average time to respond to audit is 40 hours; that’s one
Dev sprint)
▪ Audit cost and schedule overages: a 3 hour audit test just turned into a
16 hour audit project
▪ Reduced confidence from auditors, increased visibility from audit and
management
When Auditors Attack Unexpectedly
The DevOps Audit Defense Toolkit

@RealGeneKim
@jdeluccia
Practice: Enabling A Shared Understanding
Source: DevOps Audit Defense Toolkit

@RealGeneKim
@jdeluccia
Walk Through Of DevOps Risk And
Control Strategies
What does an effective DevOps
control environment look like?

@RealGeneKim
@jdeluccia
DevOps Orgs Actually Love Process
“Facebook values people, tools, and way, way
down the list is process.”
Jay Parikh
VP Infrastructure Engineering,
Facebook
Not true! They are conflating “process” and
“approvals!”

@RealGeneKim
@jdeluccia
High Performing DevOps Orgs
Source: 2014 Puppet Labs State Of DevOps
30xmore frequent
deployments
8,000xfaster lead times
than their peers

@RealGeneKim
@jdeluccia
2xhigher change
success rates
12xfaster mean time to
recover (MTTR)

@RealGeneKim
@jdeluccia
more likely to exceed
profitability,
market share &
productivity goals
2x higher market
capitalization growth
over 3 years*
50%

@RealGeneKim
@jdeluccia
Top Predictors Of Performance
▪ Version control of all production artifacts
▪ Continuous integration and deployment
▪ Automated acceptance testing
▪ Peer-review of production changes (vs. external change
approval)
▪ High trust culture
▪ Proactive monitoring of the production environment
▪ Win-win relationship between Dev and Ops

@RealGeneKim
@jdeluccia
DevOps Orgs Need Hardcopy
DevOps has higher automation and closer monitoring controls than
traditional deployment environments and therefore reduced points
for human failure
The documentation of ephemeral systems, tools, and deployment
processes into a hardcopy breakdown will communicate and
simplify this management long term.

@RealGeneKim
@jdeluccia
Practice: Document Risks & Control
Strategy

@RealGeneKim
@jdeluccia
Practice: Document Control Strategy

@RealGeneKim
@jdeluccia
▪ Gained an understanding of the organization and its
objectives
▪ Understood how our service fits in and where we jeopardize
those objectives
▪ Designed and documented our control environment so that
auditors can share our understanding
▪ Enable auditors to do their work effectively
What We Have Done

@RealGeneKim
@jdeluccia
▪ Save the date: October 21-23, 2014
▪ DevOps Enterprise is a conference for horses, by horses
▪ Macy’s, Disney, GE Capital, Blackboard, Telstra, US Citizen and Immigration Services, CSG,
Raytheon, Ticketmaster/LiveNation, Capital One, Nordstrom, Union Bank of California
▪ Leaders driving DevOps transformations will talk about
▪ The business problem they set out to solve
▪ The obstacles they had to overcome
▪ The business value they created
▪ Submit talks at: http://devopsenterprisesummit.com/
DevOps Enterprise Summit

@RealGeneKim
@jdeluccia
▪ We don’t need to wait for auditors to learn about DevOps -- by learning about audit, we
can successfully bridge the gap
▪ DevOps control environments can be even more secure than traditional control
environments
▪ The DevOps Audit Defense Toolkit might be able to help you! http://bit.ly/DevOpsAudit
▪ We’d love your scrutiny and case studies!
▪ DevOps Enterprise Summit: http://devopsenterprise.io
▪ Emailing us: genek@realgenekim.me, jdeluccia@gmail.com
Conclusion

@RealGeneKim
@jdeluccia
Ask An Auditor Anything!
▪ Ask the Auditor and the audience anything:
▪ Separation of Duties?
▪ Security beyond checkboxes and non-contextual
requirements?
▪ Governance effects of DevOps and/or Agile?
▪ Integration and dialogues and timing with Management,
Auditors, and the effect?
▪ Ask Gene on practical examples
▪ Questions for the audience:
▪ Are you using ISO 27034 as a reference architecture?

@RealGeneKim
@jdeluccia
Results Of Halving Deployment Interval

@RealGeneKim
@jdeluccia
And customers got the
feature in half the time!
Source: Scott Prugh, CSG

@RealGeneKim
@jdeluccia
Source: Scott Prugh, CSG

@RealGeneKim
@jdeluccia
Call to Action
● We're looking for case studies
○ Rough life lessons and smooth successes
○ Submit to:
■ DevOps Audit Defense Toolkit: Google+ Community:
● Look at the DevOps Audit Defense Toolkit
● DevOps Enterprise Summit
○ http://devopsenterprise.io/

Keeping The Auditor Away: DevOps Audit Compliance Case Studies

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Keeping The Auditor Away: DevOps Audit Compliance Case Studies

Similar to Keeping The Auditor Away: DevOps Audit Compliance Case Studies (20)

More from Gene Kim

More from Gene Kim (14)

Recently uploaded

Recently uploaded (20)

Keeping The Auditor Away: DevOps Audit Compliance Case Studies