Presented by: Jacob Kitchel, Industrial Defender Abstract: This presentation will review useful concepts and tools that can be applied by DevOps team with “Controlled Remediation”. We’ll demonstrate the application of non-security, system administration, deployment, monitoring and change tracking using tools to achieve controlled remediation. This will build a foundation through which security, compliance, and change management goals can be achieved in an automated fashion within control system environments. DevOps is a juxtaposition of the words “development” and “operations” and is meant to portray a tight relationship between the two traditionally separate roles which build and operate complex computer systems and software applications. DevOps groups work with a unified goal to rapidly and reliably deploy and manage the underlying systems which organizations rely upon to make a profit while balancing resource constraints. “Controlled Remediation” is a concept used to describe the use of automation to maintain acceptable configuration and settings on industrial cyber assets. Additionally, this presentation will discuss the variations of “Automated Remediation” and “Manual Remediation”.

1. 6 Tools for Improving IT Operations in ICS
Jacob Kitchel
Sr. Manager, Security & Compliance

2. 9/24/13 2
Before we begin, a little about me …
§ Serve as the internal expert on various regulatory compliance
requirements and frequently speaks on ICS security related topics.
§ Past experience includes: performed >100 risk assessments, pen
testing, vulnerability assessment, gap analysis, architecture review,
etc.
§ Participated in Project Basecamp
§ Also has a background in security operations and monitoring.
§ Endorsed for many hilarious skills on a well known business social
network

3. 9/24/13 3
“Amateurs practice until they get it right. Professionals
practice until they can’t get it wrong.”

4. 9/24/13 4
What’s this really about?
Reducing the Chance and Impact of Failure,
Increasing Reliability, and Improving System
Awareness though:
• Continuous Delivery
• DevOps: Development & Operations working together
• How you can use these principles and tools to improve your operations
and gain confidence in your environments

5. 9/24/13 5
Why is Continuous Delivery Important to ME (YOU)?
• What “it” is:
– Small, frequent changes to production
– Actively testing every change across development and test before push to
production
– Lowers risk of change
– Helps to plan change better
• That ‘thing’ everyone says is the ‘right’ way to do things but it’s really
hard gosh darnit!

6. 9/24/13 6
Let’s back up: Present Day
§ Develop, Test (QA), and Production in LARGE chunks
§ This is called the “waterfall” model
§ OR “throw it over the wall”
§ Like it or not, you are pushing CODE whether you develop it or
not
§ Push changes and wait around to see if anything breaks
What you do now
Where you are now
§ Failure means
§ HIGH cost
§ SLOW recovery time
§ DIFFICULT to recover from
§ Great deal of UNCERTAINTY when recovering from failure

7. 9/24/13 7
Where you want to be
Failure has a low cost
Failure has a quick recovery time
Failure is easy to recover from
You are agile when recovering from failure
You are confident when recovering from failure
Code updates, testing, and deployment are automated
Automation enables you to do more things

8. 9/24/13 8
Continuous Delivery means…
Every change to your
environment is proven to be
deployable to production with
predictable results

9. 9/24/13 9
Let’s talk about tools

10. 9/24/13 10
In your toolbox…
Continuo
us
Delivery
& DevOps
Version
Control &
Change
Review
Metrics
Configuration
Management
Orchestration
Dashboards
Virtualization

11. 9/24/13 11
q Takeaway: Every change must go through version
control and also be attributable to a person
q Version Control
§ Track versions of every change
q Change Review
§ Allows you to step through every change
q Available tools
q Git: http://git-scm.com/
Version Control &
Change Review
Metrics
Configuration Change
Management
Orchestration
Virtualization
Dashboards

12. 9/24/13 12
• A Holy Grail of Enterprise IT
• Enterprise: slow, tedious, high overhead, rarely
‘correct’ electronic paper shuffling exercise
• Now: Automation with an audit trail and reporting
• Important: Use the same configuration across
Dev, Test, and Production
• Free tools to use for practical application:
– Puppet
– Chef
– Ansible
– Salt Stack
Version Control &
Change Review
Metrics
Configuration Change
Management
Orchestration
Virtualization
Dashboards

13. 9/24/13 13
Configuration Management – 2 Approaches
Passive
q Always watching
q Never changing production
q “Oh, we see a change. Is it ok?
Click ‘Yes’ or ‘No’”
q Baseline gets updated after the
fact if ‘Yes’
q Production asset gets
manually reverted if ‘No’
Active
q Always watching
q Never changing production
q “Oh, we see a change.
Revert that change back to
the approved configuration
automatically.”
q No permanent changes to
production until approved
configuration change
q Baseline gets updated to
enable change

14. 9/24/13 14
q “to arrange or manipulate, especially by means of
clever or thorough planning or maneuvering”
q Rolling out applications and configuration changes
in a specific order
q Leverage automation to reduce human error and
scale
q Free Tools to enable Orchestration:
§ Puppet
§ Chef
§ Mcollective
§ Ansible
§ Capistrano
§ Fabric
§ WinRM
§ (Any automated, remote administration tool)
§ Your own home grown scripts
Version Control &
Change Review
Metrics
Configuration Change
Management
Orchestration
Virtualization
Dashboards

15. 9/24/13 15
q Vendor specific, but they probably use VMWare
§ Important to have Dev, Test, and Production
environments mirrored
§ Use configuration management and orchestration
tools to do this!
§ Bonus: “backup”/redundant assets
o Example: Server2 and Workstation3 go down? You
can spin up virtual instances until hardware
instances recover
o Everyone wants a “do over” or “What if?” button.
Get one.
Version Control &
Change Review
Metrics
Configuration Change
Management
Orchestration
Virtualization
Dashboards

16. 9/24/13 16
Version Control &
Change Review
Metrics
Configuration Change
Management
Orchestration
Virtualization
Dashboards
q Metrics are performance ‘things’ that are
measured
q Important because they help you understand
how you are performing
q Continually monitor your environment so you can
determine how to improve it
q Free Tools to use for Metrics:
§ Graphite
§ Logstash
§ Nagios
q NOTE: you must have a way to consume and
evaluate metrics like…

17. 9/24/13 17
q The *other* Enterprise IT Holy Grail
q Visual representation of your operating state
q Quick ‘hit’, good/bad, green/yellow/red, trending,
etc.
q What do you *really* need to know?
§ Development, testing, production roll-outs
§ Metrics
§ State
§ Performance
§ Some examples:
o Assets (groups, rules, policies, etc.)
o Events (all sorts of events in various metric categories,
security, compliance, changes, etc.)
o Configuration
o Workflow (newly discovered, promotion state)
Version Control &
Change Review
Metrics
Configuration Change
Management
Orchestration
Virtualization
Dashboards

18. 9/24/13 18
q Free Tools to enable Dashboard use:
§ Graphite
§ Logstash
§ Bamboo
§ Jenkins
§ Cactii
§ Nagios
Version Control &
Change Review
Metrics
Configuration Change
Management
Orchestration
Virtualization
Dashboards

19. 9/24/13 19
… and the not-so-free kindVersion Control &
Change Review
Metrics
Configuration Change
Management
Orchestration
Virtualization
Dashboards

20. 9/24/13 20
How does it all fit together?
Unit Tests
Platform
Tests
Deliver to
Staging
Application
Acceptance
Tests
Deploy to
Production
Post Deploy
Tests

21. 9/24/13 21
How We Can Help?
Version Control &
Change Review
Metrics
Configuration
Management
Orchestration
Dashboards
Virtualization

22. 9/24/13 22
Summary
Leverage tools which can help you improve your
operations and reliability
Use automation to glue the tools together
Have confidence in deployments to production
Know and See what is happening in your environment
across your systems and assets

23. 9/24/13 23
DevOps Novel
• Head over to the
Industrial Defender
booth – we have 25
copies to give away!

24. 9/24/13 24
web
industrialdefender.com
blog
blog.industrialdefender.com
twitter
@i_defender
More Information: