1.
Challenging
Conventional Wisdom:
A New Approach to
Risk Management
Alex Hutton
Jay Jacobs

2.
What’s this We think you’re getting bad
information!
about? We think our industry can
do better!
We think this will make us
“more secure!”

3.
Security is now so
essential a concern
that we can no longer
use adjectives and
adverbs but must
instead use numbers.
– Dan Geer

4.
How are you making
decisions now?

5.
What’s the quality of
those decisions?

6.
Effective Decisions
need quality data,
models, execution

7.
Our vendors and
standards aren’t
helping us
(-:

8.
hey, why are
you getting
lousy
information
from
standards
and vendors?

9.
The science
hey, why are of information
you getting security & risk
management
lousy is hard
information 1. Pseudo Science &
Proto Science
from 2. Models & Data
standards 3. Complexity
and vendors?

10.
The science
hey, why are of information
you getting security & risk
management
lousy is hard
information 1. Pseudo Science &
Proto Science
from 2. Models & Data
standards 3. Complexity
and vendors?

11.
State of the Industry (a)
(Thomas Kuhn is way smarter than we are)
proto-science
somewhat random fact
gathering (mainly of readily
accessible data)
a“morass”of interesting,
trivial, irrelevant
observations
a variety of theories (that are
spawned from what he calls
philosophical speculation) that
provide little guidance to
data gathering

12.
State of the Industry (b)
At our present skill in measurement of
security, we generally have an ordinal
scale at best, not an interval scale and
certainly not a ratio scale. In plain
terms, this means we can say whether
X is better than Y but how much better
and compared to what is not so easy.
– More from Dan Geer

13.
If Science is based on
inductive observations to
derive meaning and
understanding and
measurement on quality
(ratio) scales, how about
InfoSec?
Where do we sit in the
family of sciences?

14.
We’re the Crazy Uncle
with tinfoil hat antennae
used to talk to the space
aliens of Regulus V, has
47 cats, and who too
frequently (but
benignly) forgets to
wear pants.

15.
Take, for example, CVSS

16.
“the Base Equation multiplies
Impact by 0.6 and
Exploitability by 0.4”

17.
Jet Engine X Peanut Butter = Shiny

18.
decimals aren’t magic.
adding one
willy-nilly doesn’t
suddenly
transform
ordinal rankings
into ratio values.

19.
The science
hey, why are of information
you getting security & risk
management
lousy is hard
information 1. Pseudo Science &
Proto Science
from 2. Models & Data
standards 3. Complexity
and vendors?

20.
Data must exist in order to feed our
models...
... but creating the right models are
dependent on understanding what
data is useful!
20

21.
Data, Models, Execution:
Garbage in-Garbage Out

22.
Data, Models, Execution:
Treat Data Poorly

23.
Data, Models, Execution:
Adapting to Situations

24.
The science
hey, why are of information
you getting security & risk
management
lousy is hard
information 1. Pseudo Science &
Proto Science
from 2. Models & Data
standards 3. Complexity
and vendors?

25.
These “risk”
statements you’re
making...
I don’t think
you’re doing it
right.
- (Chillin’
Friederich Hayek)

26.
A Comforting Thought...
“Given Newton's laws
and the current position
and velocity of every
particle in the universe,
it was possible, in
principle, to predict
everything for all time.”
-- Simon-Pierre LaPlace, 1814

27.
8
4 4
2 2 2 2
Reductionism

28.
8
?
4 4
?
2 2 2 2
Functionalism

29.
Asset Reductionism
Functionalism
Comp. Comp.
Sub. Sub.
Attribute
Attribute
Attribute
Attribute

30.
Awww man...
...even if it were the case that the
natural laws had no longer any
secret for us, we could still only
know the initial situation
approximately. ... small
differences in the initial conditions
produce very great ones in the
final phenomenon. A small error in
the former will produce an
enormous error in the latter.
Prediction becomes impossible...
-- Henri Poincare,
1887

31.
ty non
lexi -l i
p nea
C om r
13
5 6
2 2 2 2
Systems Approach
Holism

32.
Complex systems contain changing
mixtures of failures latent within them.
The complexity of these systems makes it impossible for
them to run without multiple flaws being present.
... individually insufficient to cause failure
...failures change constantly because of
changing technology, work organization,
and efforts to eradicate failures.
Complex systems run in degraded mode.
“How Complex Systems Fail”
- Richard Cook

33.
Security is a characteristic of systems
and not of their components
Security is an emergent property of systems; it does not
reside in a person, device or department of an organization
or system.
... it is not a feature that is separate from
the other components of the system.
...the state of Security in any system is
always dynamic
“How Complex Systems Fail”
- Richard Cook

34.
We may want to
rethink our
approach.

35.
Overcoming the problem
• Medicine uses an “Evidence-
Based” approach to solving
problems in the complex
system that is the body.
• Dr. Peter Tippett (MD, PhD)
applies Evidence-Based
principles to Information
Security.
36

36.
What to study: Sources of Knowledge
Suggested
context:
Capability
to
manage
(skills,
resources,
asset decision
quality…)
landscape
impact
landscape
risk
threat
landscape
controls
landscape

37.
How: Data Quality in Evidence-Based Practice
Evidence
level
D Evidence
level
C Evidence
level
B Evidence
level
A
Evidence
level
A Case-­‐series
Consistent
Consistent
“Expert
opinion
study
or
Retrospec8ve
Randomized
without
explicit
extrapola8ons
Cohort,
Exploratory
Controlled
Clinical
cri8cal
appraisal,
from
level
B
Cohort,
Ecological
Trial,
cohort
study,
or
based
on
studies. Study,
Outcomes
all
or
none,
clinical
physiology,
bench
Research,
case-­‐ decision
rule
research
or
first
control
study;
or
validated
in
principles.” extrapola8ons
from
different
level
A
studies. popula8ons.
beNer

38.
Evidence-Based Risk Management
State of Nature State of Knowledge State of Wisdom
Evidence level D Lists Feeling like we’ve done
something
Evidence level C Simple derived values Outcomes with ad-hoc
with ad-hoc modeling deductive selections
Evidence level B Formal Modeling Decision making
constructs
Evidence level A

39.
Evidence-Based Risk Management
State of Nature State of Knowledge State of Wisdom
Evidence level D Lists Feeling like we’ve done
something
Evidence level C Simple derived values Outcomes with ad-hoc
with ad-hoc modeling deductive selections
Evidence level B Formal Modeling Decision making
constructs
Evidence level A

40.
Evidence-Based Risk Management
State of Nature State of Knowledge State of Wisdom
Evidence level D Lists Feeling like we’ve done
something
Evidence level C Simple derived values Outcomes with ad-hoc
with ad-hoc modeling deductive selections
You
are
here
Evidence level B Formal Modeling Decision making
constructs
Evidence level A

41.
So
How
Do
We
Change?
Data
Models…
Standards
START
WITH
THE
OUTCOMES!

42.
Two True Security
Outcomes:
Success and
Failure

43.
Knowing Success in
InfoSec is hard
- Known Success (anti-Threat ops)
- Unknown success (controls work
without us knowing)
- Dumb luck (We’re not targeted, but our
neighbor is)

44.
Getting the
outcomes:
Success

45.
Getting the
outcomes:
Success
stronger
processes
result in fewer
availability
incidents

46.
Getting the outcomes
- Successes:
- Existences of processes
- Operational (performance) metrics
- Maturity ratings
WHAT WE WANT ARE PATTERNS!

47.
Knowing Failure is
(somewhat) easier

48.
Getting The Outcomes:
Failures
VERIS | Verizon
Enterprise Risk and
Information Sharing
VERIS takes the
incident narrative
and creates metrics
(risk determinants)

49.
VERIS | Verizon
Enterprise Risk and
Information Sharing
A
free
(as
in
beer*)
framework
created
for
metrics,
modeling,
and
compara8ve
analy8cs.
A
security
incident
(or
threat
scenario)
is
modeled
as
a
series
of
events.
Every
event
is
comprised
of
the
following
4
A’s:
Agent:
Whose
acLons
affected
the
asset
AcLon:
What
acLons
affected
the
asset
Asset:
Which
assets
were
affected
AOribute:
How
the
asset
was
affected

50.
VERIS takes this :
INCIDENT REPORT
“An attacker from a Russian IP address
initiated multiple SQL injection attacks
against a public-facing web application.
They were able to introduce keyloggers
and network sniffers onto internal
systems. The keyloggers captured
several domain credentials which the
attackers used to further infiltrate the
corporate network. The packet sniffers
captured data for several months which
the attacker periodically returned to
collect…”
and…

51.
…and translates it to this…
Event 1
Agent: External (Org crime)
Action: Hacking (SQLi)
Asset: Server (Web server, Database)
Attribute: Integrity
Event 2
Agent: External (Org crime)
Action: Malware (Keylogger)
1 > 2 > 3 > 4 >
Asset: Server (Web server)
Attribute: Confidentiality
Event 3
Agent: External (Org crime)
Action: Hacking (Use of stolen creds)
Asset: Server, Network (multiple)
Attribute: Confidentiality, Integrity
Event 4…

52.
patterns!

53.
Framework
=
∑
∩ ∫√
Models Data

54.
Framework Framework
Data Process
= Process
∑
∩ ∫√
Models =
∑
∩ ∫√
Data Models
Process
Process
Data

55.
Using your metrics
program
- Identify & Measure your processes
- Identify & Measure your failures
- Get into loss factors (ABC)
- Share data
- Support data sharing efforts

56.
Bring it Home:
your metrics program

57.
Bring it Home:
your metrics program
or

58.
Bring it Home:
your metrics program
or
The Amazing
Technicolor Scorecard

59.
Priority #1:
no more surrogate data

60.
Priority #1: (meaning)
no more risk analysts*

61.
Priority #1: (really)
create data analysts

62.
Data analysts need to
focus on quality data,
models, execution

63.
Evidence-Based Risk Management
State of Nature State of Knowledge State of Wisdom
Evidence level D Lists Feeling like we’ve done
something
Evidence level C Simple derived values Outcomes with ad-hoc
with ad-hoc modeling deductive selections
Evidence level B Formal Modeling Decision making
constructs
Evidence level A

64.
asset
landscape
A balanced
scorecard of
sorts
threat impact
landscape landscape
risk
controls
landscape

65.
Where to look? The
Two True Security
Outcomes:
Success and
Failure

66.
Failures:
threat
landscape incidents, red/blue team
asset vulnerabilities, misconfigurations,
landscape
unknowns...
gaps in coverage, known lack of
controls
landscape effectiveness, known underskilled/
utilized...
impact Cost-Based Accounting around
landscape
incidents, cost of operations, etc...

67.
Successes:
threat
landscape intel, red/blue teams, SIEM
asset vulnerabilities, misconfigurations,
landscape
unknowns, skills, training
controls positive threat outcomes (tOps), skills,
landscape
training
impact
landscape ROI? ROSI? (ducks to avoid tomatoes)

68.
What to look? Two
types of data to find:
Focus initially
on Visibility,
then look to find
Variability.

69.
How to look? The
GQM Approach:
For each
“where” for each
“what” use the
following “how”

70.
How to look? The
GQM Approach:
For each
“where” for each
“what”, start by
using GQM as
“how.”

71.
Goal, Question,
Metric
Conceptual level (goal)
goals defined for an object for a variety of
reasons, with respect to various models, from
various points of view.
Operational level (question)
questions are used to define models of
the object of study and then focuses on
that object to characterize the assessment
or achievement of a specific goal.
Quantitative level (metric)
Victor Basili
metrics, based on the models, is
associated with every question in order to
answer it in a measurable way.

72.
The Book You
Should Buy
(Jay & Alex aren’t getting a
kickback, in case you’re
wondering)

73.
GQM for Fun & Profit
Goals establish
what we want to Goal 1 Goal 2
accomplish.
Questions help us
understand how to
meet the goal. They Q1 Q2 Q3 Q4 Q5
address context.
Metrics identify the
measurements that
are needed to answer M1 M2 M3 M4 M5 M6 M7
the questions.

74.
GQM for Fun & Profit
Execution Goal 1 Goal 2
Models Q1 Q2 Q3 Q4 Q5
Data M1 M2 M3 M4 M5 M6 M7

75.
data about defined success
and failures
models of assets, controls,
threats contributing to impact
execution by data analysts
...Feeding standards, audits and governance

76.
Using your metrics
program
- Identify & Measure your processes
- Identify & Measure your failures
- Get into loss factors (ABC)
- Share data
- Support data sharing efforts

77.
Using your metrics
program
- Identify & Measure your processes
- Identify & Measure your failures
- Get into loss factors (ABC)
- Share data
- Support data sharing efforts

78.
Security is now so
essential a concern
that we can no longer
use adjectives and
adverbs but must
instead use numbers.
– Dan Geer

79.
Questions?
Jay Jacobs Alex Hutton
@jayjacobs @alexhutton
jay@beechplane.com alex@alexhutton.com

80.
Approaching the system
as a system
asset
landscape
impact
Prioritize
landscape
risk
threat
landscape
controls
landscape De-prioritize

81.
Suggested context:
Capability to manage
(skills, resources,
decision quality…)
asset
landscape
impact
landscape
risk
threat
landscape
controls
landscape

82.
Data Sharing:
- Sources:
- Qualify this Intel according to
framework
- Treat with appropriate data quality
listings (let models shape the certainty)

83.
Get Into Accounting
- Use existing models that take
advantage of accounting concepts
(ABC) to Talk to the LOBs

84.
Using your metrics
program
- Identify & Measure your processes
- Identify & Measure your failures
- Share data
- Support data sharing efforts
- Get into loss factors (ABC)

85.
Challenging
Conventional Wisdom
Conventional Wisdom may not be wrong
- Question current practices
- Seek Evidence and Feedback