Slides from Tony Martin-Vegue's presentation at SIRAcon (Cincinatti, OH) on April 30, 2019 Abstract: We make estimates every day in the process of performing risk assessments. We regularly estimate the probability of a data breach, effectiveness of awareness training or projected staffing levels for the next 5 years. Here’s the problem: humans are horrible at making estimates! All sorts of bias cloud our judgement, making it difficult to make good security decisions. Here’s the good news: it is possible to overcome some of these inherent biases with many of the same techniques that professional bookmakers use to set odds when placing and taking bets. Attend this hands-on session to learn how to overcome your bias and become a better estimator. All attendees will take a test to determine estimation skills and will receive personalized feedback on what kind of bias is present. Bring a $20 bill to place bets – don’t worry, you’ll get it back!

1. #SIRAcon
1
Expert
Estimation and
Calibration
Training

2. #SIRAcon
About Me
Tony Martin-Vegue
www.tonym-v.com
tony.martinvegue@gmail.com
@tdmv
• 20 years in Technology; last 10 in Cyber Risk
• FAIR practitioner for about 8 years now
• Reside in the Bay Area

3. #SIRAcon
• What is this?
• Quiz 1
• Calibration Training
• Quiz 2
• Q&A / Wrap up
3
Agenda

4. #SIRAcon4

5. #SIRAcon
5
Quiz 1

6. #SIRAcon
• Instructions:
• Go to this link
• This is not pub trivia : I don’t care if you get the answer
right!
• I only care if you think you got the answer right
• Resist the urge to Google the answers or ask your
neighbor!
6
http://bit.ly/2UVsP68
Quiz 1

7. #SIRAcon
• Who are the superforecasters?
• Why?
• Are you surprised by your results?
Results

8. #SIRAcon
Calibration Training
8

9. #SIRAcon
Expert Estimation
• Interdisciplinary
• Acquired knowledge
• Predictive
judgements

10. #SIRAcon

11. #SIRAcon
•Data is not available, or not available to you
•Data is too expensive (see “value of information”
principles)
•You have data, but it needs adjustment
• You are forecasting some change in the landscape
that just using historical data won’t reflect, or
• Need an interpretation of incomplete, missing data or there’s
a degree of uncertainty
When to Use Expert Judgement

12. #SIRAcon
Some Cognitive Biases

13. #SIRAcon
Being
Calibrated

14. #SIRAcon
Calibration Test

15. #SIRAcon
Name
Question
score
Calibration
Score
Calibration
Respondent 1 8/10 8.2 Perfectly calibrated
Respondent 2 7/10 8.2 Slightly overconfident
Respondent 3 8/10 8.6 Perfectly calibrated
Respondent 4 7/10 7.4 Perfectly calibrated
Respondent 5 8/10 7.7 Perfectly calibrated
Respondent 6 6/10 7.2 Slightly overconfident
Respondent 7 9/10 7.2 Underconfident
Respondent 8 6/10 7.8 Overconfident
Respondent 9 7/10 6.9 Perfectly calibrated
Respondent 10 5/10 6.7 Overconfident
Respondent 11 8/10 7.7 Perfectly calibrated
Respondent 12 5/10 7.4 Overconfident
Respondent 13 7/10 7.1 Perfectly calibrated
Respondent 14 8/10 7.1 Perfectly calibrated
Respondent 15 8/10 8.7 Perfectly calibrated
An Example

16. #SIRAcon
• Substitute for constant
feedback
• Questions do not need to
be cyber related (pub trivia)
• Simply training people to
recognize bias
Make
forecast
Observe
event
Receive
feedback
on prior
forecast
Why Does Trivia Work?

17. #SIRAcon
Respondent Calibrated Min Mode Max
Respondent 1 Yes 10 25 35
Respondent 2 No 27 32 34
Respondent 3 Yes 15 35 65
Respondent 4 Yes 1 5 36
Respondent 5 Yes 1 2 65
Respondent 6 No 20 25 40
Respondent 7 Yes 10 20 60
Respondent 8 Yes 1 50 100
Respondent 9 No 27 30 34
Respondent 10 No 25 31 35
Respondent 11 Yes 0 5 40
Respondent 12 No 5 10 20
Respondent 13 No 1 5 20
Respondent 14 No 5 35 80
Respondent 15 Yes 20 30 40
In Practice…

18. #SIRAcon
Controlling for Bias

19. #SIRAcon
Overconfident professionals
sincerely believe they have
expertise, act as experts and
look like experts. You will have
to struggle to remind yourself
that they may be in the grip of
an illusion.
- Daniel Kahneman
”
“

20. #SIRAcon
Lesson 1: Accuracy versus Precision
William Tell is forced by the tyrant Gessler to shoot an apple from his son's head

21. #SIRAcon
21
Accurate Precise
Accuracy vs Prediction

22. #SIRAcon
22
How Much Does a Ford Pinto
Weigh?
1,376.762 pounds is precise
but not accurate
2,015–2,270 pounds accurate,
but not precise
Accuracy vs Prediction: An Example

23. #SIRAcon
“Perfect is the
enemy of good.”
- Italian Proverb
23

24. #SIRAcon
Lesson #2: Decomposition

25. #SIRAcon
But I Don’t Know
Anything About That
• Start with the absurd
• Apply anything that you
know
• Decompose it!
• Can you bring the ranges
in?
• Remember the goal:
accuracy, not precision
25

26. #SIRAcon
What is the Wingspan of a 747?
26
• Start with the absurd
• Apply anything that you
know
• Decompose it!
• Can we bring the ranges
in?
• Goal: accuracy, not
precision

27. #SIRAcon
27

28. #SIRAcon
Lesson #3: Equivalent Bet Test

29. #SIRAcon
Game Rules:
• Place $20 down to play the game;
house also places down $20
• I’m going to ask you to provide an
estimate within a range in which
you are 90% confident the answer
is correct
• You now have choice as to which
game to play: see if your answer
was right or spin the wheel
How This Works

30. #SIRAcon
How many
credits does
Steven Spielberg
have as director?
Game #1: Movie Trivia

31. #SIRAcon
$20$20
WinLose
SPIN
Equivalent Bet Test
Make your choice!
Game 1:
Spin the wheel
Game 2:
See if your answer was
right

32. #SIRAcon
Answer:
55
(source: imdb.com)

33. #SIRAcon
The idea behind the equivalent best test is to test
whether or not you are truly 90% confident about
your estimation
• If you choose option a, spin the wheel – you are less than
90% confident about your estimation (ranges are too tight)
• If you choose option b, try your luck with your answer – you
are more than 90% confident (ranges are too wide)
• The perfect balance would be that you don’t care.
Why This Works

34. #SIRAcon
How tall is Steph
Curry?
Game #2: Sports Trivia

35. #SIRAcon
$20$20
WinLose
SPIN
Equivalent Bet Test
Make your choice!
Game 1:
Spin the wheel
Game 2:
See if your answer was
right

36. #SIRAcon
Source: User JeopardyTempest @ Sports Stack Exchange
How Tall is Steph Curry?
Answer:
6’3”
(source: nba.com)

37. #SIRAcon
Lesson #4: How might you be right or
wrong?

38. #SIRAcon
Why might you be
wrong?
• Anchoring
• Blind spot bias
• Confirmation bias
• Dunning–Kruger effect
Why might you be
right?
• You are an expert
• Did you apply everything
you already know?
• Evaluated the options
• Thought about the
probability of being right
Considering Pros and Cons

39. #SIRAcon
Lesson 5: Probability
“Kick-up at the Hazard Table” -- Painting by Thomas Rowlandson

40. #SIRAcon
Lesson 5: Probability
Source:
https://www.mathsisfun.com/data/probability.html

41. #SIRAcon
Question:
•”Ramoray” is Chandler’s last name in the sitcom
Friends. True or false?
•Confidence that you are correct? (50%, 60%,
70%...)
41
Applying to Binary Questions

42. #SIRAcon
42
• Apply anything that you know
• Decompose it!
• Make a bet with yourself
• Can you improve your confidence rating?
(Is your guess better than a coin flip?)
http://bit.ly/2VvLept
Quiz 2

43. #SIRAcon
More Calibration Quizzes
• The Credence Calibration Game - http://acritch.com/credence-game/
• Calibrated Probability Assessments - http://calibratedprobabilityassessment.org/
Measurement
• How to Measure Anything by Douglas Hubbard
• How to Measure Anything in Cyber Risk by Douglas Hubbard and Richard Seiersen
Cognitive Biases
• Thinking Fast and Slow by Daniel Kahneman
Expert judgement
• The Wisdom of Crowds – by James Surowiecki
• Superforecasters – by Philip Tetlock
• The Good Judgement Project - https://goodjudgment.com/
Further Reading

44. #SIRAcon