Global CISO Forum 2017: How To Measure Anything In Cybersecurity Risk
•Download as PPTX, PDF•
1 like•721 views
Richard is a security executive with ~20 years experience ranging from start-ups to global organizations. He is currently the CISO/VP of Trust for Twilio and most recently the VP/GM Cybersecurity and Privacy for GE Healthcare. His background is in Information Security, Digital Risk Management and Product Development with an analytics bent. His current focus is developing quantitatively informed strategies, building agile teams that scale and making digital risk measurable. Likewise, he recently co-authored a decision analysis book called “How To Measure Anything In Cybersecurity Risk” (Wiley 2016) This book targets those looking to improve risk management strategies using predictive analytics.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Global CISO Forum 2017: How To Measure Anything In Cybersecurity Risk
Similar to Global CISO Forum 2017: How To Measure Anything In Cybersecurity Risk (20)
More from EC-Council
More from EC-Council (20)
Recently uploaded
Recently uploaded (20)
Global CISO Forum 2017: How To Measure Anything In Cybersecurity Risk
- 1. Richard Seiersen: VP Trust & CISO Twilio Forecasting Breach Perspectives and Code from HTMA Cyber
- 2. ?
- 3. Agenda: Forecasting breach, what’s that!? What are the challenges? Why would you want to? The Security Analytics Framework
- 4. The Security Analytics Framework Our Focus
- 5. Forecasting breach, what’s that!? What are the challenges? Why would you want to? The Security Analytics Framework
- 6. Internal Use Only Forecasting Breach, What’s That!? 2017-2019 Breach Forecast 16% chance of losing $20M+ 1% chance of losing $70M+ Who else uses these methods? Actuaries, big-pharma, military logisticians, nuclear engineers, epidemiologists, meteorologist, project managers, movie producers etc…anyone making forecasts with seemingly irriducible uncertainty. It’s Tolerance Based: This one has an imagined insurance threshold It’s Capability Based: We model security capability improvement over time It’s Risk As A Curve!: We build a model that relates impact (money) and likelihood. It Uses Probabilities: We only use probabilities and dollars – no Red, Yellow, Green or High, Medium, Low. It’s Time Based: We make multi-year forecasts to help drive strategy. …. but if correct, it has a similar structure to the territory, which accounts for its usefulness. — Alfred Korzybski in Science & Sanity A map is not the territory Data For Demo Purposes Only
- 7. Internal Use Only Key Question: Is a ~16% chance of losing $20M or more, at least once in 3 years, OK? Can it be benchmarked in some way? Fortune 500 Healthcare • Yearly Avg Rate: 3.85% • 3 Year Avg Rate: 11% Fortune 500 Finance • Yearly Avg Rate: 2.46% • 3 Year Avg Rate: 7.2% Fortune 500 Retail • Yearly Avg Rate: 2.02% • 3 Year Avg Rate: 5.9% *Research conducted by Hubbard Decision Research Inc. • Public disclosed data breaches from 2014-2015 • Outcomes are uncertain, but update our beliefs • We will show you how to predict like this with marbles shortly! Observations Forecasting Breach, What’s That!?
- 8. Forecasting breach, what’s that!? What are the challenges? Why would you want to? The Security Analytics Framework
- 9. Challenge #1: The Concept or Misconception Of Measurement
- 11. Challenge #2: Probabilities & Breach What proportion is Red? Assume you don’t know…….total • F500 2014-2015 healthcare breach forecast • We can update as we get more info, be it red or blue marbles.
- 12. Challenge #3: Forecasting Financial Impact Win $1000 if you guess the average weight in tons of an adult male African elephant? = 100 lbs. 1000 Tons
- 13. It’s Demo Time! Forecasting the value of “security capabilities” in reducing the likelihood and impact of breach
- 14. Forecasting breach, what’s that!? What are the challenges? Why would you want to? The Security Analytics Framework
- 15. Why would you want to do this?
Editor's Notes
- Time: 1:15 Seconds What do the following / four things have in common? - The probability of a mine flooding, predicting the value of drought resistance, forecasting fuel needs for a ground war, and / the likelihood of breach? They’re all catastrophic risks Most people think they are impossible to measure, and Most people / would be wrong! Are you like most people? If so, I have some good news, they have all been measured before! And you don’t need to be a rocket scientist, a movie producer or a professional sports GM / to do this! Although / they measure intangibles / with uncertain outcomes - often daily. Think money ball and how that approach is rampant in sports now... So / what are the uncertain things you want to predict? I’ve got one, and its a hairy ball of uncertainty / wrapped in a thick veneer / of intangibility. How about predicting / the value of Trust That’s my aim these days! Your aim / I believe, is a critical subset of Trust I would frame it as: “Predicting the value of security operations in reducing both the likelihood and impact of breach.” So, that will end up being a key focus of this discussion, and particularly my demo.
- Time: 30 Seconds I plan on covering the following: A security analytics framework ”Forecasting Breach What’s That? What are the challenges / in implementing it? And / why would you want to? And for those of you / who just can’t wait…for anything, who read the last page / of murder mysteries first, who start dinner / with dessert - let me quickly tell you / the moral of the story. Forecasting Breach helps us choose what, and how much, to improve security capabilities…based on value. The “what” / are security capabilities like vulnerability management, security architecture, incident response etc. The “how much” / are maturity levels for those capabilities like "Nothing, Adhoc, Scheduled and Continous" Value is R-O-I, on the capabilities…. Now lets look at our security analytics framework!
- I am unsure how many of you have read out book? This is a model from the 3rd section of the book, largely on enterprise implementation It's starts with the questions you might have. Questions mature as your organization and strategy matures Questions lead to data sources. Data sources complexity grows with your questions We have four levels of capability maturity that represents the correlation of your questions and their expected data sources. Our focus is the first area. Its foundational, its additive, it persists and its the most strategic
- I explained our security analytics framework Its only a tool for thinking about security analytics maturity And I have passed through all of them Small data analytics is not a phase - its the foundation - i have come to the conclusion that it is the most important - my book reflects that belief. Its both the easiest and most difficult phase.....which I will explain shortly Most importantly, it's where we start considering our #1 threat - Breach It's where we "Forecast Breach" Let's explain what "Forecasting Breach" means now.....
- Time: 1 Minute What’s the ROI on zero-trust networks vs application white listing vs user behavior analytics. Howabout DISA Stigs vs CIS Benchmark vs a 24 hour SLA on patches. OR PCI vs HIPAA vs SOC ll Type 2? Most security approaches and frameworks speak to the what and the how of operational security - but they don't always do a good job of answering “why”....or at least not in business terms. Fortunately, we have some / “quant magic” that may help. That “Magic” answers WHY we want to make one improvement / over another / based on ROI. And normally / I would demonstrate that “magic” / now But, I found that people / get down right / snappy / if I jump to code / and don’t first address / a few intellectual / and emotional / “challenges” And this is particularly true for security pros! The first challenge / is Measurement - it ain’t what we think Second / Probabilities / also ain’t / what we think Third / Impacts / easier to predict / than we think. Lets look at our first challenge // Measurement
- Time: 2:00 “Have you ever thought that's Impossible To Measure!” The concept or misconception obstacle Breach that’s impossible to measure” or You can’t determine the value of drought resistance then this is your slide. What is measurement? To quantify, compute, reduce Measurement pros - scientists, actuaries and statisticians say differntly What do the pros say?:(CLICK) George Box “All models are wrong, but some / are useful.” Claude Shannon defines information as“the amount of uncertainty reduction in a signal”. Nobel prize winning, history making, world changing scientists / see measurement as a reduction, not an elimination, / of uncertainty. As we look to forecast breach, / we need to keep the “uncertainty reduction” theme in mind. Now on to our next challenge - Probabilities - a Yuge stumbling block for some. Particularly for those who remember just enough stats from college / to get it all wrong.
- Time: 2 Minutes “Where on earth did you get those probabilities!” this is what my mother use to say My answer, (click) Bookies! When a bookie says something has a 90% chance of occurring, how often are they right? 80%? What makes them so good / at predicting the future? Rapid feedback on predictions, playing with money, and fear of pain. Pain they will inflict on others, and that will be inflicted on them And making predictions / is a trainable skill… that’s what we do…. We train SMEs to be bookies. I’ve participated in one of the largest / ongoing studies on this - over twenty years - we’ve seen over a thousand SMEs trained / to be virtually indistinguishable from bookies. Again, don’t take our word for it…..(click) Have you read / or heard of “Thinking Fast, Thinking Slow?” Obama put it on his list of “8 must read books” It’s about making judgements under uncertainty With chapters on “taming intuitive predictions” and “rare events” and “expert intuition”. Methods like these have been applied to: Medical Diagnosis, Legal Judgements, Intelligence Analysis, Finance, Military Strategy….and now Cyber Security And, the author’s research culminated / in a Nobel Prize / in Economics. So, bookie skills / can help us generate probabilities, Particularly when we think we lack data But, more good news, we often have more data than we think, and need less than we think / when making bets. To that end, lets see how playing with marbles / can help us refine our predictions - in this case /about something highly uncertain, potentially catastrophic, and Security Breach.
- Total: 2 Mins This: 1:30 You have a jar / that has red and blue marbles in it. (Click) You have no idea / how many, could be in the billions Red is for breach, blue - no breach (Click) You randomly draw 5. If your asked to bet / on how much red there is, the math says your best bet / is somewhere between 1-45% Its not saying that is the truth, its just putting the odds / in your favor. (Click)You put those back, draw 15 more, you best bet is as stated. (Click) Now, you randomly draw 75. You have more data now, and your uncertainty / is significantly reduced, but not / eliminated. It seems to be / between 1-8% (Click) This just so happens to be the breach forecast / for fortune 500 healthcare, based on public breach data. This: 30 This is an example of how actuaries use probability / when faced with seemingly / irreducible uncertainty - like breach / and or like the probability of a mine flooding. I should add, two of my co-authors books / are required reading / for the Society of Actuaries exam prep, our new book / extends this to cyber…and 10% of the exam questions / are right from these books…. So, perhaps you can take our word on this….a bit. So we covered the challenges with measurement and probabilities /and now comes impacts. After that, we will do a quick / probabilistic programming demo // that bring this all together…
- Time: 2:10 (Click) Anyone / wanna make this bet…but no cheating. Unless you grew up in the circus, or you’re zoologist, or frequently on safari - you’re likely uncertain about this. Let’s make this more / interesting for you. I’ll let you put a range on your bet, and if the true value / is within that range - you win! (Click) Because your likely / greedy. I bet your going to put an impossibly large range on this! What we really want, is what the pros call / your 90% confidence interval. (Click) This is how we do it. You adjust your range / until you think your chances of winning $1000 / is just as good with the spinner / as it is with the Elephant. (Click) Measurement pros call this / “making an equivalent bet” Believe it or not, this is how bookies forecast impacts, and its how we do it too / when doing Forecasting Breach.
- It’s Demo Time! Our goal (CLICK) Determining the value of security capabilities / in reducing the likelihood and impact of breach, Capabilities, think "Capability Maturity Model" Limited, Adhoc, Scheduled, Continous A key part of this demo / is something called a Monte Carlo Simulation. Anyone / heard of that before? Monte Carlo simulations were originally invented / by scientist in the 1940’s/ in the development of the first atomic bomb. Despite its nefarious start, they found that randomly running / thousands of trials / was a way to work out probabilities / with highly uncertain inputs. Our demo is essentially / a monte carl based gambling machine, or gambling robot. It consumes bookie generated / collective intelligence, and helps us make / better bets / to get better ROI / given our uncertainty. Again, its not the truth / its just better / card counting / when forecasting breach. <<go to R>>
- Time: 30 seconds Our demo was a mashup of / Security Program Maturity States, Bookie Collective Intelligence and Gambling Machines. We also asked Albert, Bertrand, George and Claude their thoughts on measurement, and they all agreed - it’s uncertainty reduction - not an impossible standard of exactness We also hope we addressed / intellectual and emotional concerns Now, we are going to conclude our talk / by answering the question “Why would you want to do this?”
- (Jason) Conclusion: 2 Mins Im assuming most everyone here has either seen money ball or read the book? The image on the right is the asst general manager and statistician for the Oakland As - John DePodesta played by Jonah Hill. What were the A’s after? They wanted to know how to compete with the likes of the big market teams like the Yankees, but with a fraction of the budget. They knew they couldn’t spend their way to a championship - so they used data and modeling to determine what would give the best ROI for winning. One of the biggest epiphanies for the A’s was that “RBIs” or “Runs Batted In” did not highly correlate with wins, which seems counterintuitive…you bat a run in, you score, you do that a lot, you win, right?!? Apparently, just getting on base correlates much higher with wins. Even more importantly, the cost of hiring people who were good at just getting on base was significantly less than those who had high RBI’s. So they used the predictions from running their models and made very targeted investments in players who simply tended to get on base… The As ran with this, and started winning Yuge at a tremendous ROI! Admittedly, baseball has less moving parts than the infinite complexities of security applications, products and companies. Baseball has rules everyone plays by: four bases, 9 players, 9 innings etc. Conversely, as I stated early, we really only have one rule for Reducing the likelihood and impact of breach And how do we do this? Using bookies, collective intelligence and gambling machines - identify capabilities and maturity levels that give you the best ROI for winning. (Rich) It’s our way of focusing on the plays - that most likely make wins Its nothing more than choosing what, and how much, to improve…based on value - the why! And we understand value quantitatively, predictively dare we say scientifically as a “Return On Investment” And our hope? That this type of modeling approach will emerge as an industry best practice In my experience being able to forecast outcomes like breach is a key skill for developing a strategy. ..particularly for those of you who are responsible for defending and recovering from breach... Thank you If you want to play with some of tools in an easy to use form, please go to our site....they are all there for free...in excel.