So exactly how do you integrate information security metrics into action in an organization and actually achieve value from the effort. Learn what efforts are currently underway in the industry to create consensus metrics guides and what initial steps an organization can take to start measuring the effectiveness of their security program.

1. Information Security Metrics:
Practical Steps to Measurement
Jack Nichelson & James Tarala

2. I defend my companies competitive
advantage by helping solve business
problems through technology to
work faster and safer.
Who is Jack Nichelson?
 Director of Infrastructure & Security for Chart Industries.
 Recognized as one of the “People Who Made a Difference in Security” by the
SANS Institute and Received the CSO50 award.
 Adviser for Baldwin Wallace’s, State winner Collegiate Cyber Defense
Competition (CCDC) team. “Solving Problems, is my Passion”
Introduction

3. • How do you measure & report progress?
• Is your team focused on the right problems?
• How do you promote accountability &
transparency?
• How do you find waste, time and money?
• Are your projects improving the daily jobs of
your end users?
“Secure more with less, show continuous improvement and value”
Problem Statement
In an era of security breaches we tend to have only one metric
– Have my systems been compromised?

4. Why are Metrics Needed?
• Businesses use metrics to facilitate decision making
• Better data leads to better decisions
• Metrics allow organizations to set appropriate priorities
• Measurement allows comparison:
– Between our organization and industry benchmarks
– Between our organization and other organizations risk
levels
– Between levels of accepted risk over time
– Between business units within an organization

5. Metrics from the Business World
• The business world uses metrics all the time
• Consider the following examples:
– Price to Earnings Ratio
– Profit & Loss Statements
– Product Sales Quotas
– Number of Safety Incidents
– Unit Production
– Web Advertisement Click Counts
– Number of Facebook “Likes” per Post

6. Metrics in Technology
• Organizations even commonly use metrics to help measure
the performance of technology systems as well
• Consider the following examples:
– System uptime
– CPU Utilization Percentage
– Memory Use Percentage
– Average Email Mailbox Size
– Support Technician to Computer Node Ratio
– Help Desk Ticket Time to First Touch
– Help Desk Ticket Time to Resolution

7. IS Metrics: Too Broad?
• The first question we need to ask is, “What do we mean by
the term Information Security metrics?”
• IS Metrics is too broad of a term
• “Begin with the end in mind.” – Stephen Covey
• Measurement for measurement’s sake helps no one
• Organizations must be specific on what they are measuring
and the benefits they hope to achieve from it

8. Suggested Solution
Create an effective, sustainable security aware culture
that is results driven.
 Foundation
 Leading Change
 Gemba Board
• Security
• Quality
• Delivery
• Cost
• People
 Case Study Examples & Results

9. Begin With The End In Mind
Example of how some
simple goals that are
tracked as a team will
move security forward.

10. Primary Recommendation
1. Start small, excel at gathering a small number of metrics
2. Integrate these metrics into your business process
3. Grow the number of metrics you collect
• United States Department of State iPost began with only three
data sensors:
– Tenable Nessus
– Microsoft Active Directory
– Microsoft System Management Server (System Center)

11. iPost

12. Foundation
• Obtain a security charter from senior management
• Create an organization wide IS Steering Committee
• Document your organization’s overall security goals
• Create an asset inventory & Assign data owners to all of your systems
• Deploy a vulnerability scanner & scan your hosts on a regular basis
• Start with 4 data sources:
• Microsoft Active Directory
• Help Desk Ticketing System
• Microsoft System Center (SCCM)
• Tenable Nessus or Qualys
Recommended elements for getting started:

13. Leading Change
Step 1: Create Urgency - For change to happen, you
need to make the case why and be brutally honest.
Step 2: Form a Powerful Coalition – Get visible
support from key people and link metrics to
performance.
Step 3: Create a Vision for Change - Develop what
you "see" as the future that people can grasp easily
and remember.
Step 4: Communicate the Vision - Talk about it every
chance you get. Use the vision daily to make
decisions and solve problems.
Culture Eats Strategy - Make metrics part of your culture

14. Leading Change
Step 5: Remove Obstacles - Empower the people you need to execute
your vision, and help the change move forward.
Step 6: Create Short-Term Wins - Nothing motivates like success
Step 7: Build on the Change - change projects fail because victory is
declared too early.
Step 8: Anchor the Changes in Corporate Culture - Your culture
determines what gets done, so the values behind your vision must
show in day-to-day work.
You have to work hard to change a culture successfully. If
you're too impatient, and if you expect too many results
too soon, your plans for change are more likely to fail.

15. Gemba Board
Gemba (現場) is a Japanese term referring to the place where value
is created. The idea of Gemba is that the problems are visible, and
the best improvement ideas will come from going to the Gemba.

16. Gemba Board: Security
Example Metrics:
• # of systems not monitored & tracked in inventory by Location or LoB
• # Top Vulnerabilities by Location or LoB
• # of Legacy Systems by Location or LoB
• # of Users with Local Admin & Accounts with Domain Admin
• # of Total Security Incidences by Location or LoB
• # of Past Due Security Awareness Training by Location or LoB
Security - The current security posture at a glance

17. Gemba Board: Quality
Example Metrics:
• # of Servers & Workstation missing OS & App patches (30 day SLA)
• # of infections/Re-Images tickets (3 day SLA)
• # of Security Event tickets (5 day SLA)
• # of Security Request tickets (15 days SAL)
• Cause Mapping Analysis to find root cause of problems
Quality – Results for SLA goals of events & requests

18. Gemba Board: Delivery
Delivery – Active Projects & Audits at a glance
Example Metrics:
• Active Projects Status
• Active Audit Status
• Remediation Progress by Location or LoB
• On-Site Awareness Training by Location

19. Gemba Board: Cost
Cost – P&L at a glance
Example Metrics:
• Operating budget spending plan (OPEX & CAPEX)
• ROIC Qualitatively Rating of Perceived Value
• Support Agreements Costs & Renew dates
• Consultant Support Agreements Costs & Renew dates
• Running total of cost savings

20. Gemba Board: People
People – Skills matrix at a glance
Example Metrics:
• Skills Matrix of everyone in Security
• Training and development plans
• On-Call & Vacation Schedules
• Awards

21. Practical Steps: Base
• To create an effective, sustainable program to implement
metrics, don’t start by creating metrics
• Recommendation would be:
1. Obtain a security management charter from senior
management
2. Create an organization wide IS Steering Committee
3. Document your organization’s overall security goals
4. Create & approve appropriate security policies,
procedures, & standards
5. Educate your organization on those documents

22. Practical Steps: Phase I
Once a base or foundation for information assurance is laid, then
you can begin with metrics
• The next phase would be to:
1. Identify what information security sensors you have
already successfully deployed
2. Determine what meaningful metrics can be gleaned from
these sensors
3. Deploy a tool that can centrally aggregate, normalize, and
report on the data collected by the sensors
4. Create basic reports based on the metrics from strep #2
5. Work with business owners to remediate risk

23. Practical Steps: Phase II
Now you are ready for continuous process improvement
• The last steps are to refine your effort, gather more data, and
remediate more risk:
1. Deploy additional sensors & aggregate the results
2. Determine meaningful metrics that new sensors can
bring
3. Collaborate with business owners to make metrics more
meaningful
4. Remediate new risks as they are discovered
5. Automate the response to as many metrics as possible

24. Software Tools to Help
• Open Source Projects:
– Practical Threat Analysis (PTA) Professional
– OSSIM Open Source SIEM
• Commercial Tools:
– Archer Technologies SmartSuite
– OpenPages Enterprise GRC
– Bwise GRC
– MetricStream
– Methodware ERA
– Protiviti Governance Portal
– CCH TeamMate, Sword, & Axentis

25. Bare Minimum Response
1. Create an asset inventory
2. Assign data owners to all of your systems
3. Deploy a vulnerability scanner & scan all of your hosts on a
regular basis
4. Create overall CVSS risk scores, by business unit, and publish
those scores to key business owners
5. Remediate the risk you discover
• Focus on the basics, then improve your efforts
• Run a 5K first, then try a marathon

26. Further Questions
• Jack Nichelson
– E-mail: Jack@Nichelson.net
– Twitter: @Jack0Lope
– Website: http://www.linkedin.com/in/nichelson
• Resource for further study:
– Security Metrics: Replacing Fear, Uncertainty,
and Doubt by Andrew Jaquith