Strata San Jose 2016 - Reduce False Positives in Security
•Transferir como PPTX, PDF•
2 gostaram•11,501 visualizações
How to reduce false positives in security systems through feedback and rules. You will learn about: 1) Implicit Feedback 2) Applying Rules above ML systems 3) Applying Rules as Features 4) Combining them using MLN
Recomendados
Recomendados
Mais conteúdo relacionado
Mais procurados
Mais procurados (20)
Destaque
Destaque (6)
Semelhante a Strata San Jose 2016 - Reduce False Positives in Security
Semelhante a Strata San Jose 2016 - Reduce False Positives in Security (20)
Último
Último (20)
Strata San Jose 2016 - Reduce False Positives in Security
- 1. Powerball Predictor Photo Credit: Sean McGrath Crystal ball tells me with 99% accuracy if a powerball prediction is a winner.
- 2. Powerball Predictor Photo Credit: Sean McGrath ● ~300 million samples. ● ~ 3 million false positives. ● 1 true positive.
- 3. Powerball Predictor Photo Credit: Sean McGrath The overwhelming majority of tickets are not winners. Failing to recognize this is falling victim to the base rate fallacy.
- 4. Security Crystal Ball Photo Credit: Sean McGrath The overwhelming majority of log entries and data points do not represent fraud and intrusions. Failing to recognize this is falling victim to the base rate fallacy.
- 9. Case Study: Outlier Detection Using an outlier detection system to identify fraudsters within the environment.
- 10. For a set of generating mechanisms find the unusual ones.
- 12. Photo credit SuperCar-RoadTrip.fr under Creative Commons Attribution 2.0 Change in the data over time in unforeseen ways. Concept Drift
- 14. Explicit Feedback Loop Photo credit Alan Levine under Creative Commons Attribution 2.0
- 15. Explicit Feedback Loop Photo credit Alan Levine under Creative Commons Attribution 2.0 Implicit Feedback Loop
- 16. Fraud: Takeaways - Concept Drift is a shift in behavior. - Feedback combats concept drift. - Implicit Feedback > Explicit Feedback
- 17. IDS: Anatomy of Successful Detection
- 19. Red team Kill Chain
- 20. Blue team Kill Chain
- 21. False positives: Lose Ability to Triage
- 22. Fact: You cannot salvage a false positive with Contextual Info or Visualization
- 23. What is a Successful detection? Properties + Frameworks
- 24. Successful detection captures Adversary TTP from Sensor data ignoring Expected activity Source: @MSwannMSFT
- 26. Basic Advanced LessUsefulMoreUseful Sophistication of Algorithms UsefulnessofAlerts SecurityDomainKnowledge Framework for a Successful detection
- 27. Basic Advanced LessUsefulMoreUseful Sophistication of Algorithms UsefulnessofAlerts SecurityDomainKnowledge Outlier
- 28. Basic Advanced LessUsefulMoreUseful Sophistication of Algorithms UsefulnessofAlerts SecurityDomainKnowledge Outlier Anomaly
- 29. Basic Advanced LessUsefulMoreUseful Sophistication of Algorithms UsefulnessofAlerts SecurityDomainKnowledge Outlier Anomaly Security Interesting Alerts Successful Detections incorporate Domain Knowledge Alerts
- 30. How to encode Domain Knowledge: Embrace Rules • Business Heuristics to filter out the “Security interesting anomalies” • Rules can take many forms: •TI feeds •IOCs, IOAs •TTPs • Rules are awesome • Credible, Interpretable, Adaptable (to some extent), Actionable! • Highest Precision • Highest Recall
- 31. Three ways to combine ML and Rules
- 32. Three Ways to combine Rules and ML 1.Above Machine Learning Systems a.Business Heuristics to filter alerts i. “For account _foo_, only raise sev 2 alerts until March 28th, 2016”,
- 33. Work by Dan Mace et. al, Microsoft
- 34. 2. Below Machine Learning Systems a. Featurizations - “If IP address present in List of malicious IP dataset, flag 1” b. Utilizes Threat Intel feeds (Cymru, Virus total, FireEye)
- 35. 3: Combining Rules and Machine Learning together using Markov Logic Networks Initial Ideas given by Vinod Nair, MSR
- 36. Intuition •Rules alone place a set of hard constraints on the set of possible worlds •Let’s make them soft constraints: When a world violates a formula, It becomes less probable, not impossible •Give each formula a weight (Higher weight ⇒ Stronger constraint) Source: Lectures by Pedro Domingos
- 37. Interactive logons from service accounts causes attack Similar service accounts tend to have similar logon behavior Example: Service Accounts Domain Knowledge
- 38. Example: Service Accounts Encode as First Order Logic
- 39. Example: Service Accounts 1.5 1.1 Example: Service Accounts Associate Each Rule With the Learned Weight
- 40. Example: Service Accounts 1.5 1.1 Attack( A) InteractiveLo gon(A) InteractiveLo gon(B) Attack( B) Example: Service Accounts Consider two service accounts: A,B
- 44. •How to learn the structure? •Begin with hand-coded rules •Use Inductive Logic Programming, but need to infer arbitrary clause •How to learn the weights? •For generative learning, depend on pseudolikelihood •Checkout Alchemy -- http://alchemy.cs.washington.edu/
- 45. Call for Action - After the conference • One Week • Review •@CodyRioux - IPython Notebook •@Ram_ssk - Follow Up material • Think comprehensively about Rules • One Month •Ask your data scientists to literature review section •Implement the rules on TOP of ML systems • One quarter •Implement a feedback system to capture training data •Implement all TI feeds within an ML System •Play with Alchemy
- 46. Literature ● The Base-Rate Fallacy and its Implications for the Difficulty of Intrusion Detection (Alexsson, 1999) ● Enhancing Performance Prediction Robustness by Combining Analytical Modeling and Machine Learning (Didona et al., 2015) ● Richardson, Matthew, and Pedro Domingos. "Markov logic networks."Machine learning 62.1-2 (2006): 107-136.