Measurement, Qualitative vs Quantitative Methods, and other Cool Stuff

Risk Centric Security, Inc.
www.riskcentricsecurity.com
Authorized reseller of ModelRisk from Vose Software

Risk Centric Security, Inc. Confidential and Proprietary . Risk Analysis for the 21st Century®
Copyright © 2011 Risk Centric Security, Inc . All rights reserved.

Patrick Florer has worked in information technology for
30 years. In addition, he worked a parallel track in
medical outcomes research, analysis, and the creation of
evidence-based guidelines for medical treatment. His
roles have included IT operations, programming, and
systems analysis. From 1986 until now, he has worked as
an independent consultant, helping customers with
strategic development, analytics, risk analysis, and
decision analysis. He is a cofounder of Risk Centric
Security and currently serves as Chief Technology Officer.

Risk Centric Security, Inc. Confidential and Proprietary .

This webinar is about language and ideas.

No one owns the definitions of words, but it is often
useful to:

Understand what we mean when we use certain
words and express certain ideas

Understand what others mean when they do the
same

Risk Centric Security, Inc. Confidential and Proprietary.

When speaking with our customers, we recognized:

Information Security Professionals are comfortable
speaking the technical language of firewalls, logs,
threats, vulnerabilities, and exploits.

Business managers are comfortable speaking the
language of return on investment, discounted cash
flows, and risk as financial impact.

Mutual misunderstanding can occur, and it is often a
source of frustration for everyone.


By learning to speak about risk in business terms,
Information Security Professionals can reach out and
bridge the language gap.

The technical details of sql injection attacks may be
important to you, but your business counterparts
may not understand, and they usually don’t care.


Instead of talking about threats, vulnerabilities, and
controls, talk about risk in terms of financial impact.
Tell the business people what a sql injection attack
could cost.

They will understand that!
(They may not believe you, but they will understand
what you are saying!)


Risk – What it is and what it isn’t
Risk and Opportunity
Possibility vs. probability
Measurement
Variability and Uncertainty
Precision vs. accuracy
Qualitative vs. quantitative methods
The “not enough data” syndrome
Monte Carlo simulation
Modeling expert opinion and the PERT distribution


From The American Heritage dictionary*:
The possibility of suffering harm or loss; danger.
A factor, thing, element, or course involving uncertain
danger; a hazard.
The danger or probability of loss to an insurer.
The amount that an insurance company stands to lose.
The variability of returns from an investment.
The chance of nonpayment of a debt.
*The American Heritage® Dictionary of the English Language, Fourth Edition. Copyright © 2006, 2000. Houghton
Mifflin Company.


From ISO 31000:
1.1 risk - effect of uncertainty on objectives
NOTE 1 An effect is a deviation from the expected —positive and/or
negative.
NOTE 2 Objectives can have different aspects (such as financial, health and
safety, and environmental goals) and can apply at different levels (such as
strategic, organization-wide, project, product and process).
NOTE 3 Risk is often characterized by reference to potential events
(3.5.1.3) and consequences (3.6.1.3), or a combination of these.
NOTE 4 Risk is often expressed in terms of a combination of the
consequences of an event (including changes in circumstances) and the
associated likelihood (3.6.1.1) of occurrence.

NOTE 5 Uncertainty is the state, even partial, of deficiency of information
related to, understanding or knowledge of, an event, its consequence, or
likelihood


In the USA, NIST, Special Publication 800-30 describes
risk in the following way:

Risk is:
“the net mission impact considering the
probability that a particular threat-source will
exercise (accidentally trigger or intentionally
exploit) a particular information system
vulnerability, and the resulting impact if this
should occur.”


NIST (The National Institute of Standards and
Technology), provides an additional definition of risk
in Special Publication 800-39:
Risk
A measure of the extent to which an entity is threatened by a
potential circumstance or event, and typically a function of: (i) the
adverse impacts that would arise if the circumstance or event
occurs; and (ii) the likelihood of occurrence.

Information system-related security risks are those risks that arise
from the loss of confidentiality, integrity, or availability of
information or information systems and reflect the potential
adverse impacts to organizational operations (including mission,
functions, image, or reputation), organizational assets, individuals,
other organizations, and the Nation.
NIST, The National Institute of Standards and Technology, Special Publication 800-39, Appendix B, Page B-7.


A probability that something will happen

A probable impact if something does happen


The probability that something will happen to cause a
negative impact in financial terms:

For example, a 50% chance that it will cost 50 million
dollars if our data are stolen.

Another way to express this is to multiply the two
numbers together and say that:

Risk = 25 million dollars on an annualized basis


Threats and exploits are not risks.

Threat agents are not risks.

Vulnerabilities are not risks.

Ineffective controls are not risks.

Each of these elements factors into understanding
risk, but none of them constitute risk.

For our discussion today, Risk will be used to indicate
loss or harm.
Opportunity can be viewed as the positive aspect of
Risk.
The techniques that apply to Risk analysis can also be
applied to Opportunity analysis.


Let’s look at tossing a coin:
What are the possibilities?
What are the probabilities?
Does knowing either help us predict what will happen
when we toss the coin next time?


A possibility is something that is “capable of
happening, existing, or being true without
contradicting proven facts, laws, or circumstances
known to be true. *”

A probability is "the likelihood that a given event will
occur.”*

*All
quotes from The American Heritage® Dictionary of the English Language, Fourth Edition. Copyright © 2006,
2000. Houghton Mifflin Company.


In statistics, a probability is “a number expressing the
likelihood that a specific event will occur, expressed as
the ratio of the number of actual occurrences to the
number of possible occurrences.“
Probability is calculated after tossing the coin many
times.
Probability is always a number between 0 and 1,
sometimes expressed as:

*All quotes from The American Heritage® Dictionary of the English Language, Fourth Edition. Copyright © 2006, 2000. Houghton Mifflin Company.

How can we use this in information security risk
analysis?

The fact that something can happen (possibility)
doesn't tell us how likely it is to happen (probability),
or how much impact it might have if it does happen
(probability).

Estimating these values helps us prioritize our
activities in a rational way.


What is a measurement?

An observation that “ascertains the dimensions,
quantity, or capacity of” an object or process”*

A set of observations that reduce uncertainty where
the result is expressed as a quantity**

*TheAmerican Heritage® Dictionary of the English Language, Fourth Edition. Copyright © 2006, 2000. Houghton Mifflin
Company
** Hubbard, Douglas W., “How to Measure Anything 2nd Edition”, John Wiley & Sons, New Jersey, 2010, p. 23


What are the properties of a measurement?

Validity – does the measurement actually do what
you think it does?

Reproducibility – when repeated, does the
measurement give a consistent answer?

Detail – does the measurement provide a useful
level of detail?


What are some sources of error in measurement?

Random error – a function of the instrument

Bias – a function of the measurement taker


Why do we make measurements?
Measurements are a way to collect data.

Making measurements should be about reducing
uncertainty.

A measurement only has to be good enough for the
decision at hand.

Sometimes, you cannot get the data you think you
need, so you have to use a proxy.

Variability and uncertainty are not the same thing,
and it can be useful to be aware of the difference.
“Variability is the effect of chance and is a function of
the system. It is not reproducible through either
study or further measurement, but may be reduced
by changing the physical system” *

In order to reduce variability in a system, you have to
change the system.

*David Vose, Risk Analysis, A Quantitative Guide, 3rd edition, 2008, pp. 47-48


“Uncertainty is the assessor’s lack of knowledge (level
of ignorance) about the parameters that characterize
the physical system being modeled. It is sometimes
reducible through further measurement or study, or
by consulting more experts”

Uncertainty is a normal characteristic of the
assessor/measurer of an IT system. Lack of perfect
certainty, and having to operate with less than
perfect information is the norm here.

In order to reduce uncertainty, you have to collect
and analyze data.
*David Vose, Risk Analysis, A Quantitative Guide, 3rd edition, 2008, pp. 47-48


Using a coin for an example:
Imagine that you don’t know anything about the
possible outcomes -
Variability: heads or tails is a random event.
Uncertainty: what is the probability of heads or
tails?
After a few trials, we begin to get an idea.
After many trials, we have a better idea.

Precision is “the ability of a measurement to be
consistently reproduced.”

Accuracy is “the ability of a measurement to match
the actual value of the quantity being measured.”

*All quotes from The American Heritage® Dictionary of the English Language, Fourth Edition. Copyright © 2006, 2000. Houghton
Mifflin Company.


Precision: a machine can produce the same part to
within 1/1000th mm all day long. This is no
guarantee that the part is the correct length,
however.

Accuracy: a machine can produce the same part to
within +/- 2/1000th mm of the correct length.
Although some parts are a bit shorter and some are a
bit longer, every part is within spec.


Precision: 100.001, or 10.233%

Accuracy: 100 or 10%, or 10.2%


Prefer Accuracy to Precision.

Precise Accuracy? – it would be nice!


Qualitative methods:
Categorical: green/yellow/red; male/female
Nominal: male = 1/female = 2
Ordinal: rank ordering – first/second/third; 0 – 5
Interval: intervals are equal, but no zero point –
temperature in Faherenheit
Quantitative methods: real numbers (cardinal
numbers, ratio scale).

Most of the time, quantitative methods are easier.


Benefits of qualitative methods?

They are useful in certain scenarios, and can be
quick and good enough.

Good for quick ordering/prioritization of options

Be careful with aggregation – if done incorrectly, an
erroneous picture will be presented


Benefits of Qualitative Methods
From ISO 17999


Problems with qualitative methods?

No ability to compare between different qualitative
scales
Variability between assessors
Inconsistency of a single assessor
Arithmetic and statistical operations not possible
Problems near the boundaries of categories
Loss of information


No ability to compare between different qualitative
scales

Red – Yellow – Green don’t mean the same thing in my
method as they do in your method.

0 – 5 on my scale doesn’t mean the same thing as 0 – 5
on your scale.


Variability between assessors

Faced with the same set of facts, different assessors
apply a scale differently.

Two QSA’s apply the PCI standards differently.

Two risk analysts classify risks differently – one says
low, one say medium


Inconsistency of a single assessor

Given the same set of facts, an assessor might make
different assessments when the only difference is
the passage of time.


Difficulty with arithmetic and statistical operations
From ISO 17999


Difficulty with arithmetic and statistical operations

Imagine if money worked this way:

The value of a dollar would be relative to the
purchase price of an item.

The value of a dollar might vary from store to store.


Problems with aggregation and estimates near the
boundaries of categories

Assume that:

Low = < 1M
Medium = 1M – 5M
High = >5M


And assume that the following risks have been
identified and put into categories:

$100K, 500K, 800K: all in Low category

$1M, 3M, 3M, 4M: all in Medium category


What happens when you aggregate based upon
qualitative scales?
What is the real difference between a very “high
Low” and a very “low Medium”?
How can we justify and defend category boundaries
that are essentially arbitrary?


Loss of information
Most of the time, we get a number in mind.
Then, we assign it to a category.
Why not just keep the number?
Or better yet, create a distribution around a range
of estimates to better express our beliefs and
confidence?


Benefits of quantitative methods?
The numbers mean what they are (cardinality).
Arithmetic and statistical methods are possible.
Problems with quantitative methods?
Data are required.
Estimates are estimates – the future hasn’t
happened yet.
Formal training in calibration techniques is very
helpful.

They say: there isn’t enough “good” data, so you are
just processing “garbage in and garbage out.”

The reason we need data is to reduce uncertainty in
decision-making.

The decision we need to make will define the data we
need – some decisions require very little data, others
require quite a bit.


A sample can be smaller than you think.

Parametric vs. non-parametric methods

Contact us for more information on these topics.


We often hear that the data are poor –

What does this mean?

Data are just data – some data may be more
interesting than other data – it depends on what you
are doing.


Dan Geer et al.:
The Index of Cybersecurity
(http://www.cybersecurityindex.org/)

Prediction Market Project

The Beewise Project
(http://beewise.org/markets/metricon.ctrl)


Please refer to the slides at the end of this
presentation.


Monte Carlo simulation is a game changer for
information security risk analysis.
Less sophisticated methods use single-point
estimates or even simple ranges of estimates:
35%, or from 20% - 51%
Monte Carlo methods sample thousands or tens of
thousands of values, and provide a much clearer
picture of the possible outcomes.


Minimum:
What is the least or lowest (best or worst) numerical
estimate that you believe to be reasonable? This will be the
smallest number you come up with.

Most Likely:
What is the most likely or most probable numerical estimate
in your opinion? This number must fall between the
minimum and maximum. It may equal either the minimum
or the maximum, but should not equal both


Maximum:
What is the greatest or highest (best or worst) numerical
estimate that you believe to be reasonable?

Note that “best” or “worst” case estimates could be either
minimum or maximum values, depending upon the scenario.

In a risk / loss exposure scenario, lower is better, so the
minimum represents the lowest loss, or best outcome. The
maximum represents the highest loss, or worst outcome.

In a sales or opportunity scenario, it’s the reverse: lower is
not better, so the minimum represents the worst case.
Higher is better, so the maximum represents the best case.


Confidence:
On a scale that includes “Very Low”, “Low”, “Average”, “High”,
and “Very High”, how confident are you in the accuracy of
your estimates?

This parameter controls the sampling around the most likely
value, and thereby also controls the height of the histogram
or slope of the cumulative plot.

For most analyses, using “Average” for the confidence
parameter works well. In this instance, “Average” really
means having no strong feeling about the matter – being
evenly divided between under-confidence and over-
confidence.


Percentile Tables


Percentile Tables

1% of values are <= 10,044 and 99% are > 10,044

The 50th percentile has another name - it’s called
the Median.

The Median is the mid-point in a list of values - half
of the values in the list are less and half are greater
than the Median.

Histogram


Cumulative Plot


Thank you !
Risk Analysis for the 21st Century ®
Patrick Florer
CTO and Co-founder
Risk Centric Security, Inc
patrick@riskcentricsecurity.com

214.828.1172

Authorized reseller of ModelRisk from Vose Software


Open Security Foundation: datalossdb and osvdb
http://www.opensecurityfoundation.org/

Computer Security Institute (CSI): http://gocsi.com/

Office of Inadequate Security: http://www.databreaches.net/

Identity Theft Resource Center: http://www.idtheftcenter.org/

ISACA: www.isaca.org

ISSA: www.issa.org


Mitre Corporation: www.mitre.org

OWASP: http://owasp.com/index.php/Main_Page

Privacy Rights Clearing House: http://www.privacyrights.org/

SANS: www.sans.org

The Ponemon Institute: www.ponemon.org


Conference procedings: Black Hat, RSA, Source Conferences, BSides

Internet tools:

Search engines: Google, Bing, Yahoo, Ask.com

Trend Analyzers:

Google trends: http://www.google.com/trends

Twitter Trends: www.trendistic.com

Amazon: http://www.metricjunkie.com/


Securitymetrics.org – mailing list

Society of Information Risk Analysts (SIRA)

Books:
How to Measure Anything – Hubbard
The Failure of Risk Management – Hubbard
Risk Analysis: A Quantitative Guide – Vose
Clinical Epidemiology and Biostatistics - Kramer


Measurement, Qualitative vs Quantitative Methods, and other Cool Stuff

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (19)

Destaque

Destaque (20)

Semelhante a Measurement, Qualitative vs Quantitative Methods, and other Cool Stuff

Semelhante a Measurement, Qualitative vs Quantitative Methods, and other Cool Stuff (20)

Último

Último (20)

Measurement, Qualitative vs Quantitative Methods, and other Cool Stuff