For those who are truly interested in developing a Security Metrics program with VALUE

1. SECURITY METRICS
A presentation developed by Cydney Davis, Senior Technical Write

2. What are Metrics?
2
A method which facilitates decision-making and
improved performance and accountability through
collection, analysis and reporting of performance-
related data.
Information Security metrics must be:
• based on Information Security performance goals and
objectives
• useful for reduction and management of risks
• readily obtainable and replicable
• useful for tracking performance and directing resources
• able to yield quantifiable information

3. What is our Mission/Goal?
3
It is critical that we use metrics that are relevant to
our organization and to the mission we are
measuring.
But first, we have to determine:
• Where we are (Baseline)
• Where we are going (End Goals)
• Who/what relies on us? (Users/Management)
• What do they need/expect? (Reports/Assurance)
• What are we trying to prove?
• What are we trying to solve?
• What are we trying to improve?

4. How can we use Metrics?
4
 Communicate Performance
 Drive Performance Improvement
 Measure Effectiveness of Security Controls
 Help Diagnose Problems
 Provide Effective Decision-making Support
 Increase Accountability
 Guide Resource Allocation
 Demonstrate the state of compliance
 Facilitate Benchmark Comparisons

5. Metrics can help determine:
5
• the number of resources it takes to accomplish security
goals
• justifiability for financing new security measures
• If the company is getting its money’s worth
• If the company is managing risk appropriately*
• what Information Security needs to do to improve
Security
ˉ administration/processes/procedures/policies/person
nel/enhancements/technology/etc.)
• where we are with comparisons to peers regarding to
standards, best practices, execution and results of
security measures
*The residual risk that a company is willing to take based on; business needs,
budget limits, industry regulations/requirements and other criteria.

6. 6
Building the
Security Metrics Program

7. Executive Focus
7
“The heart of it is that if a business process
cannot be measured in one way or another,
we likely ought to cast it off as wasted effort.”
Comment from a CEO to an anonymous Information Security Profession
Translation:
Why do it if we can’t prove/justify its value?
(time, money, effort, results and actions)

8. Good Metrics Guidelines
8
• Consistently Measured
• apples to apples/same time same place
• Cheap to Produce (Time-wise)
• Yield Quantifiable Information
• Contextually Specific – who
• Expressed using at least 2 units of measure or data po

9. Metrics Program Success
9
Criteria
Identify incident trends important to key senior managers,
stakeholders and to the InfoSec Mission from a management
perspective.*
Provide consistent information that adds value and is actionable by:
• Tracking changes on a consistent basis.
• Focusing on what's important in our business
• Developing a few value indicators that we can track with a high
degree of reliability
• Doing some service benchmarking with our peers.
*This is the first and most important decision

10. Basic Information Security
10
Measures
Anti-malware Firewalls Asset
Management
Intrusion Anti-SPAM Patch
Detection Management
and Prevention
Vulnerability Unified Threat Application
Management Management Security
Scanners
Databases Website Statistics Network Access
Control
System Integrity Operating Data Leakage
Checking Systems Protection
Configuration Secure Web Web Application
Hardening Gateways Firewalls
Mobile Data Media Sanitation Storage
Protection Encryption

11. Formula for Deriving True
11
Meaning
WHY we need WHAT we need WHO we are
to measure it to measure measuring it for
• Financial DATA • C-Level
• Governance DATA • Board of Directors
• Legal DATA • Marketing Releases
• Regulatory DATA • Industry Report
• Directive DATA • General Staff
Determine how the information will be analyzed, interpreted and
used!

12. 12
“Good metrics facilitate discussion, insight
and analysis...”

13. Metrics Program - Components
13
Program Component
 Define the metrics program goal(s) and objectives
 Decide which metrics to generate
 Develop strategies for generating the metrics
 Establish benchmarks and targets
 Determine how the metrics will be reported
 Create an action plan and act on it
 Establish a formal program review/refinement cycle

14. High Level Process Steps
14
 Obtain management input, agreement and support for the implementation of a strong
metrics program.
 Review our organization’s mission statements, policies, plans, procedures, goals and
objectives, and assess them against legislative and regulatory requirements, as well as
against effectiveness goals.
 Describe how we will achieve company and department goals
 List milestones, dates and quantifiable objectives against which to map progress.
 Select appropriate, quantifiable effectiveness metrics to indicate baseline, interim and
final success.
 Gather the metrics.
 Analyze and present the results to management and key stakeholders.
 Recommend that management make decisions based on the metrics, and plan the
execution of these decisions. * Metrics are often referred to as “decision support.”
 Evaluate the outcome of decisions against goals. This should be done from a perspective of
*The real value of a metrics program

15. Project Plan Overview
15

16. 16 Metrics Versus Numbers

17. Good metrics are those that are
17
SMART;
• Specific
• Measurable
• Attainable
• Repeatable,
• Time-dependent
Truly useful metrics indicate the degree to which
security goals are being met – and they drive
actions that need to be taken to improve our
overall security goals.

18. Metrics? Or Just Numbers?
18
Exhibit A - This set of numbers can give us a sense of the overall health of anti-virus
defenses and can show trends over time; but the information is not actionable in
any way and will not serve as a meaningful diagnostic tool.
SO WHAT??? = False sense of security without more knowledge

19. Good Metrics = Numbers with
Relevance
19
Exhibit B displays the same measurements as Exhibit A. By drilling down into the data we can begin to
understand which locations are struggling with this activity. This in turn will help us choose where to focus in
order to improve the performance of our organization. This kind of actionable intelligence is valuable and it
can really drive performance improvement and provide information that is actionable to a productive end.
Example Metrics showing
RELEVANCE
Percentage of computers with current anti-virus definitions
City A 99.4 %
City B 94.7 %
City C 89.8 %
50 % 55 % 60 % 65 % 70 % 75 % 80 % 85 % 90 % 95 % 100 %

20. Good Metrics = Actionable
20
Percentage of computers with current anti-virus definitions
CITY A 99.4 %
City B 94.7 %
City C
89.8 %
50 % 55 % 60 % 65 % 70 % 75 % 80 % 85 % 90 % 95 % 100 %
Example Question: Why is one location so much farther behind in implementation?
Possible Reasons: Understaffed
Limited Bandwidth
More staff traveling that previous years
Possible Actions: Hire additional staff
Share resources if the implementation MUST be done by xxx date
Set different schedules for each location for future projects

21. Presenting and Interpreting Data
21
Reports
Visually Appealing Visually Appealing
Interpreted and Actionable
_______% improved
_______% improved
from _______ and that means _________ .
What we need is ______ based on
requirements for __________ . Going
forward we should consider doing
___________ .

22. 22
Measuring for value not
numbers
Examples to work with
Defining, refining and Interpreting data/results for the intended audience

23. EXAMPLE Metric : Baseline Defenses
Coverage
(Antivirus, Antispyware, Firewall, etc)
23
Measurement of how well we are protecting our enterprise against
the most basic information security threats.
Just Numbers: ________ %
What would an additional relevant value be that we can use to have
SMART data?
Metrics: ________ %
Increase since (prior month/inception/year over year/etc.)
Device Type
Location
Length of time it took to detect

24. EXAMPLE Metric : Legitimate E-Mail Traffic
Analysis
24
Legitimate e-mail traffic analysis is a family of metrics including incoming and outgoing
traffic volume, incoming and outgoing traffic size, and traffic flow between our company and
others.
By monitoring legitimate e-mail flow over time, we can learn where to set alarm points.
Numbers:
Compare the amount of good and junk e-mail that we are receiving
____ percent good
____ percent junk
What would an additional relevant value be that we can use to have SMART data?
Metrics
____ percent good
____ percent junk
Quarterly/Annually/Since inception/Current Month
Since adding the _________ criteria
Received from _________ types/organizations
Sent During ____________ (AM/PM – Holidays , etc.)
Junk Detected Quicker _______ (first time/second time)

25. Conclusion
25
By presenting information in a sufficiently granular way we can inject business relevance
into the exhibits. Producing a benchmark is also a powerful approach to performance
improvement.
Percentage of computers with current anti-virus definitions
City A 99.4 %
City B 94.7 %
City C
89.8 %
50 % 55 % 60 % 65 % 70 % 75 % 80 % 85 % 90 % 95 % 100 %
Frequently this level of visibility will spark a competitive fire in those being measured. Professional pride will
drive most people to make sure they are found among the high performers on your report.