1. © 2015 Carnegie Mellon University
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Data-Driven Cybersecurity
Governance
Douglas Gray

2. 2
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Copyright 2015 Carnegie Mellon University
This material is based upon work funded and supported by the Department of Defense under Contract No.
FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a
federally funded research and development center.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s)
and do not necessarily reflect the views of the United States Department of Defense.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE
MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO
WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT
NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR
RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE
ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR
COPYRIGHT INFRINGEMENT.
[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please
see Copyright notice for non-US Government use and distribution.
This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic
form without requesting formal permission. Permission is required for any other use. Requests for permission
should be directed to the Software Engineering Institute at permission@sei.cmu.edu.
CERT® and OCTAVE® are registered marks of Carnegie Mellon University.
DM-0003094

3. 3
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Data-Driven Cybersecurity Governance
Introduction

4. 4
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
• The Software Engineering Institute (SEI)
is a U.S.-owned not-for-profit federally
funded research and development
center (FFRDC) operated by Carnegie
Mellon University to focus on software
and cybersecurity.
• The CERT Division of the SEI is a
trusted provider of operationally relevant
cybersecurity research and innovative
and timely solutions to our nation's
cybersecurity challenges.
• The CERT Division developed and
maintains the CERT Resilience
Management Model (CERT-RMM) and
OCTAVE Allegro Methodology.
Who We Are
Introduction
We work with the following
organizations:
• Carnegie Mellon University
• Discover Financial
• Highlands Union Bank
• Lockheed Martin Corporation
• Marshall & Ilsley Corporation
• PNC Corporation
• Pacific Gas and Electric
• University of Pittsburgh Medical
Center
• U.S. Dept. of Defense
• U.S. Dept. of Energy
• U.S. Dept. of Homeland Security
• U.S. Dept. of Health & Human
Services
• U.S. Environmental Protection
Agency
• U.S. National Security Agency
• U.S. Postal Inspection Service
• USBank

5. 5
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
To discuss a process to integrate data analytics into operational
cybersecurity governance decision making and execution in a way
that
• frames the problem quickly and accurately and that enables a
fast, effective Observe, Orient, Decide, Act Loop
• facilitates better data collection and synthesis, quantitative and
qualitative analysis, and visualization
• enables practical and repeatable analytical battle drills and
TTPs for leaders and enablers at all echelons
Purpose
Introduction

6. 6
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
W. Edwards Deming’s thoughts What it means to us
“If you do not know how to ask the right
question, you discover nothing.”
We must have a reason to analyze data
“If you don't understand how to run an
efficient operation, new machinery will
just give you new problems of operation
and maintenance. The sure way to increase
productivity is to better administrate man
and machine.”
We can’t “tool” our way out of
cybersecurity challenges
“People with targets and jobs dependent
upon meeting them will probably meet the
targets - even if they have to destroy the
enterprise to do it.”
Compliance is the beginning, not the
end
“Whenever there is fear, you will get
wrong figures.”
The use of data analytics must be
productive in the aggregate, punitive as
the exception
Improving People and Process
Introduction
Technology is useless without effective processes and trained people

7. 7
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Governance vs. Operations
Introduction
Operations Governance
Scope Individual
networks, systems,
users,
organizations
Multiple networks,
systems, user
bases,
organizations
Timescale Immediate to 6
months
3 to 36 months*
Level of
Abstraction
Transactional Trends,
aggregations
Management
Impact
Direct interaction Context setting
*Although maximum technology-related decision making is limited to approximately
three years due to rate of technological change, military organizations must program
their expected budget needs five years in advance.

8. 8
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Governance vs. Operations
Introduction
8
Operations
Weather – “It will snow.”
Tactical Cyber – “CVE 2015-xx-
xxxx is prevalent and is being
compromised.”
Governance
Climate – “Drought in the
southwest limits irrigation.”
Strategic/Operational Cyber –
“FedRAMP usage improves
asset management.”

9. 9
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Why Focus on Governance
Introduction
Know Prevent Detect
Respond
Recover
Reconnaissance Weaponization Delivery Exploitation Installation
Command and
Control
Actions on the
Objective
Threat Actor
Actions1
Friendly Actions2
Harden People, Information,
Information, Technology,
Facilities
Create Faster,
More Accurate
TTPs, Battle Drills
Source:
1Lockheed Martin Kill Chain
2NIST Cybersecurity Framework
Effective preparation creates the
context for effective response

10. 10
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Leveraging Situational Awareness to Enable
Cyber Mission Command
Introduction
Observe
Orient
Decide
Act
mutual
trust
shared
understanding
clear
leadership
intent
disciplined
initiative
mission-
oriented
directives
prudent risk
management

11. 11
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Data-Driven Cybersecurity Governance
Observe

12. 12
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Facets of Cybersecurity Governance
Observe

13. 13
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Data Fusion
Observe
Data
Fusion
Activities
Automated vulnerability sensor
information
•Hardware & Software
•Behavioral Observables (Insider Threat)
Threat Information
•Threat Actor Analysis
•Prevailing Attack Patterns
Management Information
•Budget Information
•Demographic Information
•Legal & Administrative Investigation
Statuses
•Mission Impact Analysis
Qualitative Assessment
• Inspections/Assessments
• Professional Sentiments Analysis
Orient
Unstructured Data
Machine Learning
Text Analysis
Trend Analysis
Correlation

14. 14
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Data-Driven Cybersecurity Governance
Orient

15. 15
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Decision Science vs. Dashboard
Orient
1
Dashboard
“It’s going to snow.”
Decision Science
“It’s going to snow.
Wear galoshes, gloves,
scarf, winter coat.”

16. 16
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Level 1
Perception of
the elements in
the environment
Level 2
Comprehension
of the current
situation
Level 3
Projection of
future status
Developing Situational Awareness
Orient
Source: Endsley, M. & Jones, D. Designing for Situation Awareness:
An Approach to User-Centered Design (2nd ed.). CRC Press. 2012.

17. 17
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Decomposing the Situation to Develop
Situational Awareness
Orient
Situational
Awareness
Voice of the
Environment
Socio-Political
Legal and Policy
Technological
Business
Physical
Voice of the
Organization
Voice of the
Mission
Voice of the
Service
Strategic
Objectives and
Supporting
Services
Organizational
Culture
Organizational
Assets
External
Dependencies
Voice of the
Threat Actor
Describe Threat
Actor
Develop Threat
Actor Use Cases
Indices
Probabilistic Models
Game Theory
Expert Opinion

18. 18
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Build and Update Targeted Metrics
Orient
Requirements
Identify
requirements
from mandates,
doctrine,
strategy
Group
requirements
into categories
Goals
Develop one or
more goals for
each category
Question
Develop one or
more questions
that, if
answered, help
determine if the
goal is met
Indicators
Identify the
information
requirements to
answer the
question
Metrics
Identify the
metrics that will
measure the
indicator to
answer the
question
Use new metrics
to mature
current metrics
What do we want to know? Why do we want to know it? What will
we do once we know it? Build and add to a metrics library.

19. 19
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Authoritative vs. Non-Authoritative Data
Orient
Authoritative Data
• Based on their ability to stand alone as a source for one or
more facets of cybersecurity governance
• Population
• Comprehensiveness
• Poor data quality does not make a source not authoritative; it
means the quality problems should be fixed
Non-Authoritative Data
• Source does not cover enough of the population or not
comprehensive enough to be authoritative
• Can speak to confidence level of an authoritative data
source
• Examples: reviews, assessments, inspections, surveys.

20. 20
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Using Behavioral Models to Target
Stakeholder Need
Orient
Executives:
• Elected leaders, appointees,
GOs, FOs, SESs
• Target data with eye toward
organizational mission and
constituents
Middle Management:
• Staff officers, analysts
• Target data with eye toward
routines, procedures
information
Source: Allison, G. T., & Zelikow, P. (1999). Essence of Decision: Explaining the Cuban Missile Crisis (2nd ed.) (Kindle
Edition). New York: Longman.
Results of data analysis must be impactful to the recipient.
Frame products according to organizational behavioral models.

21. 21
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Data-Driven Cybersecurity Governance
Decide

22. 22
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
• Determine confidence level in assessed data
• Low – analyze through subsequent OODA loop
• Medium to High – develop action plan to effect change
• Identify and prioritize governance-level risks; identify metric-
supported thresholds of acceptability and unacceptability
• Support solutions. Go beyond “name and shame.” Use metrics to
identify key trends and corrective governance-level actions
• Tie metrics to a resulting set of possible risk management outcomes
• Identify enablers such as SMEs, funding, contract vehicles
• Identify organizations that exceed expectations in certain areas and
their lessons learned
• Identify what expected changes in metric values should be and how
to avoid bias/gaming
• Prioritize and identify metric thresholds where costs will exceed
benefits
Key Planning and Decision-Making Factors
Decide

23. 23
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Data-Driven Cybersecurity Governance
Act

24. 24
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Leveraging Enablers to Achieve Desired
Effects
Act

25. 25
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
• Leverage enablers at the proper organizational level; avoid the
“3,000-mile screwdriver”
• Governance sets the direction through governance facets.
Operations executes through disciplined project management
• Avoid numerous, rapid changes that cause enterprise
turbulence
• Tie actions to expected outcomes and expected timeframes;
socialize and communicate expectations
• Set decision points to check progress against expectations
• Build knowledge base to make for faster and more effective
OODA loop
Success at the Point of Execution
Act

26. 26
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Data-Driven Cybersecurity Governance
Implementation

27. 27
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Identify success stories
• Lessons Learned
• Tie to data analysis
Identify cautionary tales
• Lessons Learned
• Tie to data analysis
Track event-driven events
• Identify trends that respond to events
• Resourcing, technology, incidents
Building a Cybersecurity Knowledge Base
Implementation
2
7

28. 28
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
• Inventory on-hand data
• Inventory metrics
• Develop data fusion capabilitiesObserve
• Refine metrics based on constraints and mandates
• Define stakeholders based on behavioral models
• Develop quantitative and qualitative analysis engines
• Develop visualization capabilities
Orient
• Inventory enablers and their capabilities
• Identify desired outcomes for metrics (i.e., thresholds)
• Develop decision support TTPs
• Develop decision-support systems
Decide
• Develop knowledge base
• Simulate and practice new decision-making TTPs
• Develop and refine process control mechanisms
• Develop, refine and leverage communications channels
Act
How to Implement
Implementation

29. 29
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Outcomes of Data Driven Governance
Implementation
• Faster, more accurate
decision making
• Better use of resources
• Better enterprise cohesion
and synchronization
• Data-driven outcomes
• Improved information
sharing
• Adaptable to change
Observe
Orient
Decide
Act

30. 30
Data Driven Cybersecurity Governance
Dec. 18 2015
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release;
Distribution is Unlimited
Data-Driven Cybersecurity Governance
Questions