This document discusses metrics, measures, and myths related to security metrics. It begins by defining some key terms and presenting quotes emphasizing the importance of measurement. It then addresses five common myths about metrics and emphasizes that metrics should be used to identify opportunities and drive improvement, not to punish people. The document outlines characteristics of good metrics and provides examples of security metrics and key performance indicators. It discusses how metrics can be displayed in dashboards and how monitoring transforms data into useful security information and knowledge.
Let’s have a look to today’s main points in the agenda. First of all we are going to see the power of metrics and how important they are to know what is happening in a company and how the enterprise is doing regarding bottom-line impact. Metrics are the indicators that tell not only management but also people on day-to-day operations how well they are performing to already established goals and business objectives. As we will see later, there is way (and a deep need, in my opinion) to align security management with the business.
We will also make a quick overview of what are CSFs, KGIs and KPIs and the intimate relationship between them.
As a security practitioner and consultant, I will give you some real examples of KPIs and how they integrate in a balanced scorecard and also talk about a real implementation of a security dashboard on a customer.
Finally, to wrap up, we will see the SMART side of metrics and a quick summary. Let’s go
Objectives need to be defined
The course in charted
Risks are identified, evaluated and managed
Resources and their criticality and sensitivity are determined
Objectives are:
Strategic alignment
Risk Management
Business process assurance
Value delivery
Resource Management
Performance measurement
It is said that you cannot manage what you cannot measure (and I fully agree with that vision) and my colleague Krag Brotby will later on the day do a presentation about it.
It has to be pointed out that normalization of data it is very useful since you have to be able to compare between departments and divisions but also with other industry peers. Normalization places all the measures on a similar footing by equalizing them across a common organizational base
Besides, metrics are rarely raw data but some derivative number (ratio, index, percentage or weighted average)
Critical to successful implementation of metrics is the understanding and acceptance that they take an important commitment and use in time and resources
Regarding IC, each organization needs to decide how important each attribute is for their business and this profile expresses the enterprise’s position and appetite for risk
CSFs were introduced by John F. Rockart in 1979 and are defined as elements that are vital for a strategy to be successful. In another level they could also be seen as important things for the process in this way: “what you need from others” and “what you can do yourself and deliver to others”
KGIs are a target to achieve, a measure of outcome
We are going to focus today in KPIs since they are the day-to-day metrics, the one being monitored constantly
In this context we need to remember that IT is a major enabler of the business and, therefore, KPIs are a measure of performance
As you can see in the graphic on the left, KGIs are just above generic IT goals and KPIs are next to IT processes showing their area or influence. Consequently, we could define KGIs as “lag” indicators while KPIs could be “lead” indicators. By the way, both measures could also be expressed negatively showing not having reached the goal or not performing well
KPIs have a cause-effect relationship with KGIs of the process
In summary, KGIs are business-driven while KPIs are process-oriented
I think that KGIs and KPIs do reflect organizational goals. Once a company has analyzed its mission, identified all its stakeholders and defined its goals, it needs a way to measure progress. KPIs are those measurements. Take into account that some analysts and consultants call KPIs also KSI (Key Success Indicators) but it is extremely more common the former acronym (with a P from performance) giving it a sense of direction and continuous monitoring.
Top-down approach
KPIs are quantifiable measurements, agreed to beforehand. However, I would like to deviate from the idea that there is a kind of negotiation with KGIs and KPIs. There should be an agreement but what really matters is the strategy and how a company is going to measure the achievement of the target. In the same way, scaling down to the IT or security department, there should be an agreement (again, not a biased negotiation) of what is needed and how security brings and adds value to the business (by preventing threats exploiting a vulnerability better than last month or year or some other measures that we are going to see in a moment).
This takes us to a whole new level of data visualization and integration: dashboards and balanced scorecards. Introduced by Robert Kaplan and David Norton in the early 90s, (1992 to be precise), balanced scorecards convert strategy into action by showing in a centralized single place all the metrics that executive management needs to take decisions. In fact, not only management but also operational teams and divisional managers are empowered by balanced scorecards since different views and information is provided depending on the role and profile of the viewer.
The definition of BSCs given by Mr. Kaplan and Mr. Norton is very interesting. Listen for the words: comprehensive view, performance, management tool. A BSC is a method and a management tool for ensuring enterprise’s activities in terms of its vision and strategies by giving managers a fast, comprehensive view of the performance of a business. It is here where we should introduce the 4 different perspectives of a balanced scorecard: financial, customer, internal process and learning/innovation.
Scorecards - Most strategic level of the business decision while dashboard work more in the operational side giving key users metrics of their area of influence
Level 0 - Non existent
Level 1 - Initial
Level 2 - Repeatable
Level 3 - Defined
Level 4 - Managed
Level 5 - Optimised
“Knowledge resides in the person, not in the data…it is the response and action to information that counts”
We built upon other disciplines like network management, asset management (CMDB) and storage management (backup & contigency plan) so as to provide a unique repository of information and began escalating in what we called “The road to management”
“You need to know what you have to be able to protect it”
3-layer architecture
We focused so much in showing a KPI regarding critical operations, which nodes out of 1453 where at risk and, consequently, which operations were being threatened. Remember, at this point the definition of what is risk: the potential that a given threat will exploit a vulnerability with an impact in an asset or group of assets
(meaning alignment with the business)
(since KPIs are “lead” indicators)
FOCUS