Thin Slicing a Black Swan: When Less Is More
•
0 likes•500 views
As infosec professionals we are swimming in prodigious amounts of data, but it isn’t making us better at our jobs, it seems to make us worse. In Verizon’s 2012 Data Breach Investigations Report, it was found that across organizations, an external party discovers 92% of breaches. We continue to desperately grasp at that straw of, “more data,” but what if this is simply information gluttony? Incident response's bloated model drives it closer to a form of security archeology rather than its promise of real-time relevance.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Thin Slicing a Black Swan: When Less Is More
Similar to Thin Slicing a Black Swan: When Less Is More (20)
Recently uploaded
Recently uploaded (20)
Thin Slicing a Black Swan: When Less Is More
- 2. Michele Chubirka, aka Mrs. Y, is a senior security architect and blogger. Hosts Healthy Paranoia, a security podcast. Researches and speaks on topics such as affective neuroscience and the psychology of decision making. Ronald P. Reck is formally trained in theoretical syntax, an author of numerous papers on linguistics and a book on RDF. He has worked extensively with the intelligence community and law enforcement, implementing standards for data and knowledge representation. Disclaimer: No swans were harmed in the making of this presentation
- 3. "The entire security industry is wired so that the oldest and least effective methods will profit most….” Josh Corman, Director of Security Intelligence at Akamai, the content delivery network.
- 4. In Verizon’s 2012 Data Breach Investigations Report, it was found that across organizations, an external party discovers 92% of breaches. Something’s Broken
- 5. “WHEN YOU CONSIDER THE METHODS USED BY ATTACKERS TO GAIN A FOOTHOLD IN ORGANIZATIONS—BRUTE FORCE, STOLEN CREDS, PHISHING, TAMPERING—IT’S REALLY NOT ALL THAT SURPRISING THAT NONE RECEIVE THE HIGHLY DIFFICULT RATING. WOULD YOU FIRE A GUIDED MISSILE AT AN UNLOCKED SCREEN DOOR?” “…three-quarters of breaches are of low or very low difficulty for initial compromise, and the rest land in the moderate category.”
- 8. ► We believe we can solve the issue of the unknowns, intrusions, with more data. ► The more information we have, the less we know. ► This makes us no better than security archeologists. From Compromise To Discovery: Verizon Data Breach Report 2012
- 9. Imperva conducted a study and released a report in 2012 on the effectiveness of antivirus software. ►Out of approximately 80 pieces of malware, the initial detection rate for new malware was less than 5%. ►For some AV vendors, it may take up to four weeks to detect a new virus from the time of the initial scan. ►Software cost wasn’t a factor. Some free programs performed better. ►In 2011, Gartner reported that consumers spent $4.5 billion on antivirus and enterprises spent $2.9 billion. The total of $7.4 billion is more than a third of the total of $17.7 billion spent on security software. Anti-Virus Ineffective
- 10. ► An unknown unknown. ► Can’t be predicted by probability theories. ► Rationalized after the fact. ► How often do we try to predict the Black Swan Event in security and fail? The Black Swan Event
- 11. “Military drone operators amass untold amounts of data that never is fully analyzed because it is simply too much.” Michael W. Isherwood, defense analyst and former Air Force fighter pilot. Information Gluttony?
- 12. • From beginning of recorded time to 2003 - five exabytes of information. • 2011 - that much created every two days. • 2012 - prediction is every 10 minutes. Digital Kudzu
- 13. SANS surveyed how much time is spent on log-data analysis: 35% - none to a few hours per week 18% - unknown 11% - one day per week 2% - outsourced to a managed security service provider 24% - integrated into normal workflow 50% of the smaller organizations spent zero to just a few hours analyzing logs. Big Data or Big Garbage?
- 14. “I don’t call it big data, I call it garbage Data.” Jerry Sto. Thomas, Director of Global Information Security, Allergan
- 15. ► SIEMs: never gets fully implemented. ► Predictions using Logistic Regression/Bayesian Probability. ► Huge amounts of data, not enough time. ► “Open world” problem using “closed world” assumptions. ► More staff, more money. Current Solutions
- 16. “…the ability of our unconscious to find patterns in situations and behavior based on very narrow slices of experience.” Malcolm Gladwell, Blink Alternative Model: Thin Slicing
- 17. ► Cook County Hospital struggled with identifying patients in danger of an imminent heart attack. ► Coronary care unit was overwhelmed. ► Public hospital, limited resources. ► ICU is dangerous. Case Study: A Hospital in Trouble
- 18. ► Lee Goldman, a cardiologist, created a protocol based upon an algorithm developed in partnership with mathematicians. ► After two years of using a decision tree, hospital staff were 70% more effective at recognizing patients at risk. ► Less information led to greater success. ► Technique used by first-responders every day. Applied Thin-Slicing
- 19. “Violations of logical reasoning [are] interpreted as cognitive fallacies, yet what appears to be a fallacy can often also be seen as adaptive behavior, if one is willing to rethink the norm.” Gerd Gigerenzer, Rationality for Mortals Bounded Rationality
- 20. Fast and Frugal Trees
- 21. ► Remember Spock’s tricorder? ► It could identify anything in the galaxy. ► Fiction, right? ► Not according to Carlos Garcia-Robledo, postdoctoral fellow in the department of botany at the Smithsonian's National Museum of Natural History. Star Trek and Thin Slicing
- 22. ► Goal - quickly identify species using short DNA sequences. ► Master list of sequences, then match samples. ► Changes in mitochondria is the marker between species. ► Technique is already being used commercially. ► Oceana released a controversial report revealing 33% of 1200 fish samples sold were mislabeled. ► Data resulted from DNA barcoding. Reality: DNA Barcoding
- 23. ► Semantic Web technology. ► Queries based on relationships or mental associations. ► Graphs treat each packet from capture file as a discrete event with properties. ► TCP header info in a metadata model. ► Model replicates human cognitive economy. Method: Resource Description Framework (RDF)
- 24. ► SPARQL query language uses a concise approach for quickly traversing large data sets while capturing similarities between packets as generalizations. ► RDF statement contains a subject, predicate and an object. ► Subject defines the event. ► Predicate defines a characteristic or property. ► Object contains the value for the predicate. Thin-Slicing with SPARQL
- 25. sparql select * { ?s ?p ?o.}; sparql select *{ ?e1 <http://www.rrecktek.com/demo/src> ?ip1.}; Example: Building A Query
- 26. • All source IPs and their destination IPs. • For each source, count how many times it went to a destination. • Report source destination and count. sparql SELECT ?src ?dst (count (?dst) as ?count) { ?e1 <http://www.rrecktek.com/demo/src> ?src. ?e1 <http://www.rrecktek.com/demo/dst> ?dst. } ORDER BY DESC (?count); Example
- 27. Which machines were the destination of the most traffic? sparql select * { ?event <http://www.rrecktek.com/demo/dst> ?dst. } limit 10; sparql select distinct (?dst) (COUNT (?src) as ?count) { ?event <http://www.rrecktek.com/demo/dst> ?dst. ?event <http://www.rrecktek.com/demo/src> ?src. } ORDER BY DESC(?count) limit 10; Example 2
- 28. What times did the machines talk to each other ? sparql select * { ?e <http://www.rrecktek.com/demo/src> "135.8.60.182". ?e <http://www.rrecktek.com/demo/dst> "172.16.113.50". ?e <http://www.rrecktek.com/demo/date> ?date. FILTER regex(?date, "1998-06-04"). ?e <http://www.rrecktek.com/demo/time> ?time }; Example 3
- 30. ► What we can do ► Build strong infrastructures and secure applications minimizing technical debt. ► Create data classification schemes based upon the business and technical service catalogs to better create better segmentation. ► Add the equivalent of air bags to the architecture for when intrusions occur. ► Recognize signature limitations. ► Investigate the creation of real-time fast and frugal trees. Our patient is dying on the table. It’s up to us to change the outcome. We Can’t Fight All Unknowns
- 31. ► Michele Chubirka www.healthyparanoia.com Twitter @MrsYisWhy networksecurityprincess@gmail.com ► Ronald P. Reck rreck@rrecktek.com Thanks!
- 32. Works Cited: "Eclectic Tech." Semantic Web Introduction. N.p., n.d. Web. 20 Dec. 2012. Erwin, Sandra I. "Too Much Information, Not Enough Intelligence." National Defense Magazine. N.p., May 2012. Web. <http://www.nationaldefense.org>. Gigerenzer, Gerd. Gut Feelings: The Intelligence of the Unconscious. New York: Viking, 2007. Print. Gigerenzer, Gerd. Rationality for Mortals: How People Cope with Uncertainty. Oxford: Oxford UP, 2008. Print. Gladwell, Malcolm. Blink: The Power of Thinking without Thinking. New York: Little, Brown and, 2005. Print. Hacker Intelligence Initiative, Monthly Trend Report #14. Rep. Imperva, Dec. 2012. Web. Dec. 2012. Koerth-Baker, Maggie. "The Technology That Links Taxonomy and Star Trek." BoingBoing.net. BoingBoing, 15 May 2013. Web. 15 May 2013. Luan, Shenghua, Lael J. Schooler, and Gerd Gigerenzer. "A Signal-detection Analysis of Fast-and-frugal Trees." Psychological Review 118.2 (2011): 316-38. Print. Marewski, Julian N., PhD, and Gerd Gigerenzer, PhD. "Heuristic Decision Making in Medicine." Dialogues in Clinical Neuroscience 14.1 (2012): 77-89. Print. Messmer, Ellen. "SANS Warns IT Groups Fail to Focus on Logs for Security Clues." TechWorld.com. TechWorld, 3 May 2012. Web. 15 Aug. 2012. "RDF." -Semantic Web Standards. W3C, n.d. Web. 02 Jan. 2013. "Resource Description Framework (RDF)Model and Syntax." RDF Model and Syntax. W3C, n.d. Web. 02 Jan. 2013. Rieland, Randy. "Big Data or Too Much Information?" Innovations. Smithsonian, 7 May 2012. Web. Sandoval, Greg. "Foreign Hackers Steal More Than a Terabyte of Data per Day in Ongoing Cyberwar." The Verge. N.p., 27 Feb. 2013. Web. 27 Feb. 2013. "Semantic Web Standards." W3C. W3C, n.d. Web. 02 Jan. 2013. Taleb, Nassim. The Black Swan: The Impact of the Highly Improbable. New York: Random House, 2007. Print. Trustwave 2013 Global Security Report. Rep. Trustwave, 2013. Web. Turek, Dave. "The Case Against Digital Sprawl." The Management Blog. Bloomberg Businessweek, 2 May 2012. Web. Verizon 2012 Data Breach Investigation Report. Rep. Verizon, 2012. Web. Verizon 2013 Data Breach Investigation Report. Rep. Verizon, 2013. Web. References
Editor's Notes
- http://www.theverge.com/2013/2/27/4035378/new-report-finds-hackers-stealing-terabyte-daily
- Chart and statistics from Verizon 2012 Data Breach Investigation Report.
- “ WHILE IT MIGHT BE DIFFICULT TO DETECT, POSITIVELY IDENTIFY, AND RESPOND TO AN INTRUSION WITHIN SECONDS OR MINUTES, OUR ABILITY TO DO SO SHOULD OSTENSIBLY INCREASE THE LONGER THEY POKE AROUND OUR INTERNAL NETWORKS. BUT UNFORTUNATELY, WE’RE NOT REALLY SEEING THAT IMPROVEMENT.”
- Chart from Verizon 2012 Data Breach Investigation Report. Trustwave’s 2012 Global Security Report was just as sobering, indicating that only 16% of compromises were self-detected and attackers had an average of 173.5 days within a victim’s organization before detection. From Trustwave’s 2013 Global Security Report: Often compromises are detected at greatly varying intervals, and the time from initial breach date to containment may be six to 12 months or more. The Timeline: Intrusion to Containment graph represents investigations that took place in 2012, demonstrating that initial entry may have been up to four years before the investigation. Unfortunately, it’s not always possible to determine the specific date on which a compromise occurred, because victim organizations often do not maintain relevant forensic artifacts, like log files, and attackers sometimes cover their tracks.
- The black swan theory or theory of black swan events is a metaphor that describes an event that is a surprise (to the observer), has a major effect, and after the fact is often inappropriately rationalized with the benefit of hindsight . The theory was developed by Nassim Nicholas Taleb to explain: The disproportionate role of high-profile, hard-to-predict, and rare events that are beyond the realm of normal expectations in history, science, finance, and technology The non-computability of the probability of the consequential rare events using scientific methods (owing to the very nature of small probabilities) The psychological biases that make people individually and collectively blind to uncertainty and unaware of the massive role of the rare event in historical affairs Unlike the earlier philosophical " black swan problem ," the "black swan theory" refers only to unexpected events of large magnitude and consequence and their dominant role in history. Such events, considered extreme outliers , collectively play vastly larger roles than regular occurrences. [1] http://en.wikipedia.org/wiki/Black_swan_theory “ First, it is an outlier, as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.” Nassim Taleb ‘ … almost everything studied about social life focuses on the "normal," particularly with "bell curve" methods of inference that tell you close to nothing. Why? Because the bell curve ignores large deviations, cannot handle them, yet makes us confident that we have tamed uncertainty. Its nickname in this book is GIF, Great Intellectual Fraud.’ Nassim Taleb
- 1.quoted in the article “Too Much Information, Not Enough Intelligence” in National Defense Magazine http://www.nationaldefensemagazine.org/archive/2012/May/Pages/TooMuchInformation,NotEnoughIntelligence.aspx 2. When the IT professionals were asked how much time they normally spend on log-data analysis, the largest group (35%) replied, "none to a few hours per week." As for the rest, 18% didn't know, 11% said one day per week, 2% outsourced this task to a managed security service provider, and 24% defined it as "integrated into normal workflow." The SANS survey report, which notes analysis time overall actually seems down from last year, noted that about 50% of the smaller organisations spent zero to just a few hours analising logs.’ http://news.techworld.com/security/3355455/sans-warns-it-groups-fail-focus-on-logs-for-security-clues/ 3. At the recent RSA Security Conference in San Francisco, Jerry Sto. Thomas, Director of Global Information Security at Allergan said, “I don’t call it big data, I call it garbage Data.”
- In a recent article from Smithsonian Magazine, Big Data or Too Much Information , the author made a comparison between the copious amounts of digital information we’re collecting and the invasive Asian perennial, kudzu, brought to the U.S. in a misguided attempt to prevent soil erosion during the Great Depression. Problem is that it worked *really* well and was declared a weed in 1972 by USDA. Information from Dave Turek, in charge of supercomputer development at IBM “From the year 2003 and working backwards to the beginning of human history, we generated, according to IBM’s calculations, five exabytes–that’s five billion gigabytes–of information. By last year, we were cranking out that much data every two days. By next year, predicts Turek, we’ll be doing it every 10 minutes.”
- “ logistic regression is a type of regression analysis used for predicting the outcome of a categorical (a variable that can take on a limited number of categories) dependent variable based on one or more predictor variables. The probabilities describing the possible outcome of a single trial are modeled, as a function of explanatory variables, using a logistic function.” Definition : Logistic regression is a technique for making predictions when the dependent variable is a dichotomy, and the independent variables are continuous and/or discrete. In statistics, regression analysis is a statistical technique for estimating the relationships among variables. It includes many techniques for modeling and analyzing several variables, when the focus is on the relationship between a dependent variable and one or more independent variables. More specifically, regression analysis helps one understand how the typical value of the dependent variable changes when any one of the independent variables is varied, while the other independent variables are held fixed. the white elephant that’s a fork-lift to install = SEIM http://semanticweb.com/introduction-to-open-world-assumption-vs-closed-world-assumption_b33688 "The Closed World Assumption (CWA) is the assumption that what is not known to be true must be false. The Open World Assumption (OWA) is the opposite. In other words, it is the assumption that what is not known to be true is simply unknown.” "On the other hand, OWA applies when a system has incomplete information. This is the case when we want to represent knowledge (a.k.a Ontologies) and want to discover new information. For example, consider a patient’s clinical history system. If the patient’s clinical history does not include a particular allergy, it would be incorrect to state that the patient does not suffer from that allergy. It is unknown if the patient suffers from that allergy, unless more information is given to disprove the assumption.” http://en.wikipedia.org/wiki/Closed_world_assumption The closed world assumption (CWA) is the presumption that what is not currently known to be true is false. The same name also refers to a logical formalization of this assumption by Raymond Reiter. The opposite of the closed world assumption is the open world assumption (OWA), stating that lack of knowledge does not imply falsity. Decisions on CWA vs. OWA determine the understanding of the actual semantics of a conceptual expression with the same notations of concepts. A successful formalization of natural language semantics usually can not avoid an explicit revelation of the implicit logical backgrounds based on whether CWA or OWA. Negation as failure is related to the closed world assumption, as it amounts to believing false every predicate that cannot be proved to be true. In the knowledge management arena, the closed world assumption is used in at least two situations: 1) when the knowledge base is known to be complete (e.g., a corporate database containing records for every employee), and 2) when the knowledge base is known to be incomplete but a "best" definite answer must be derived from incomplete information. http://en.wikipedia.org/wiki/Open_world_assumption In formal logic, the open world assumption is the assumption that the truth-value of a statement is independent of whether or not it is known by any single observer or agent to be true. It is the opposite of the closed world assumption, which holds that any statement that is not known to be true is false. The open world assumption (OWA) is used in knowledge representation to codify the informal notion that in general no single agent or observer has complete knowledge, and therefore cannot make the closed world assumption. The OWA limits the kinds of inference and deductions an agent can make to those that follow from statements that are known to the agent to be true. In contrast, the closed world assumption allows an agent to infer, from its lack of knowledge of a statement being true, anything that follows from that statement being false. Heuristically, the open world assumption applies when we represent knowledge within a system as we discover it, and where we cannot guarantee that we have discovered or will discover complete information. In the OWA, statements about knowledge that are not included in or inferred from the knowledge explicitly recorded in the system may be considered unknown, rather than wrong or false.
- Also Smart Heuristics, Bounded Rationality or rapid cognition. Example: catching a fly ball, which uses a gaze heuristic. The explanation is that the angle of the gaze remains constant. The evolved capacities: tracking objects with eyes and maintaining balance while moving. Evolved capacity for tracking prey, avoiding predators. The building blocks of the gaze heuristic are 1. fix gaze on ball, 2. start running 3. adjust running speed so angle of gaze remains constant. Financial studies demonstrate that equal allocation investment strategies based upon recognition heuristics beating optimal allocation strategies. Dr. Gerd Gigerenzer is one of the leading researchers in the field of human decision making. His research tries to find the mechanisms by which humans make decisions in real-world situations. Unlike many other models, Dr. Gigerenzer's approach assumes that much of the relevant information is not known to an individual, that it is unclear what the probabilities of certain events are, what the consequences of a decision are, etc. This sets his research program apart from those approaches which try to model optimal human decision making behavior in so-called 'small worlds' - in which all of the important information is known to the decision maker. In his work, Dr. Gigerenzer has focused on the role of fast and frugal heuristics that can drive good decision making without taking up too many resources of the decision maker. In addition, his group at the Max Planck Institute for Human Development looks at how efficient decision strategies have emerged through evolution and under what conditions a particular decision making strategy or heuristic is 'ecologically rational'. http://gocognitive.net/interviews/gerd-gigerenzer-decision-making.
- To understand why we're failing, we have to understand the three basic types of decision-making in medicine: Defensive (too many tests to prevent being sued). This leads to overtreatment and overmedication. logistic regression (ask lots of questions, gather data and make predictions. Slightly better than overkill of the defensive method. Smart heuristics aka rapid cognition, thin-slicing. Using a Fast and Frugal Tree which always has a stopping rule. What does that mean? rapid cognition, coarseness, a frugality or economy of thought. The human mind naturally does this to conserve energy. unbounded rationality vs. optimization under constraints. In the real world, a human being doesn't have all the information, all the time to solve a problem. Sentient beings actually utilize variances of bounded rationality, "an adaptive toolbox, fast and frugal rules for decision making under uncertainty. " Gigerenzer
- This is based upon the idea of a Fast and Frugal Tree developed by Gerd Gigerenzer and others in the Decision Making Analysis field called Heuristics or Bounded Rationality. See “Heuristic decision making in medicine” Julian N. Marewski, PhD; Gerd Gigerenzer, PhD “ Intuitions based on only one good reason tend to be accurate when one has to predict the future (or some unknown present state of affairs), when the future is difficult to foresee, and when one has only limited information. They are also more efficient in using time and information. Complex analysis, by contrast, pays when one has to explain the past, when the future is highly predictable, or when there are large amounts of information. From “Gut Feelings” by Gerd Gigerenzer
- From Bounded Rationality: The Adaptive Toolbox “ Bounded rationality…dispenses with optimization, and for the most part, with calculations of probabilities and utilities as well.” “ These models dispense with the fictions of optimization, which in many real-world situations demands unrealistic assumptions about the knowledge, time, attention and other resources available to humans. Note that dispensing with optimization (as a model of cognitive processes) does not imply that the outcome of a non-optimizing strategy is bad. …it is possible that simple and robust heuristics can match or even outperform a specific optimizing strategy.”
- From “A Signal-Detection Analysis of Fast-and-Frugal Trees” Shenghua Luan Lael J. Schooler and Gerd Gigerenzer From “Gut Feelings” by Gigerenzer Fast and frugal tree contains 3 elements: Search rule: look up factors of importance Stopping rule: Stop the search if a factor allows it. Decision rule: Classify the object according to this factor. An FFT is a way to codify intuition. It is a habituated heuristic informed by empirical data. “ Fast and frugal tree is different from a full decision tree. Full trees are not rules of thumb; they are information-greedy and complex rahter than simple and transparent.” Full tree = 2n FFT = n+1 N is the number of factors This process is looking for MARKERS in order to shortcut an over-long rational process.
- http://boingboing.net/2013/05/15/the-technology-that-links-taxo.html
- http://www.w3.org/standards/semanticweb/ “ The Semantic Web is a Web of data — of dates and titles and part numbers and chemical properties and any other data one might conceive of. RDF provides the foundation for publishing and linking your data. Various technologies allow you to embed data in documents (RDFa, GRDDL) or expose what you have in SQL databases, or make it available as RDF files.” “ Query languages go hand-in-hand with databases. If the Semantic Web is viewed as a global database, then it is easy to understand why one would need a query language for that data. SPARQL is the query language for the Semantic Web.” Graph from http://eclectic-tech.blogspot.com/2010/05/semantic-web-introduction-part-2-rdf.html “ The Resource Description Framework is one of the fundamental Semantic Web standards in that it specifies how data in the form of statements are structured. A statement is made up of three parts, a subject, a predicate, and an object. For example, a statement stating the origins of Penicillin could be written in the following form. AlexanderFleming discovered Penicillin. In this example AlexanderFleming is the subject, discovered is the predicate, and Penicillin is the object. Statements in this form are referred to as triples and a software component that provides storage and access to them is called a triple store. A set of triples can be represented as a graph. For example, if we further want to state that Penicillin is used to treat Staphylococcus we might represent the statements in the form of triples as the following. AlexanderFleming discovered Penicillin. Penicillin treats Staphylococcus.”
- http://www.w3.org/rdf/ http://www.w3.org/TR/WD-rdf-syntax-971002/ RDF – the Resource Description Framework – is a foundation for processing metadata; it provides interoperability between applications that exchange machine-understandable information on the Web. RDF emphasizes facilities to enable automated processing of Web resources. RDF metadata can be used in a variety of application areas; for example: in resource discovery to provide better search engine capabilities; in cataloging for describing the content and content relationships available at a particular Web site, page, or digital library; by intelligent software agents to facilitate knowledge sharing and exchange; in content rating ; in describing collections of pages that represent a single logical "document"; for describing intellectual property rights of Web pages, and in many others. RDF with digital signatures will be key to building the "Web of Trust" for electronic commerce, collaboration, and other applications. Metadata is "data about data" or specifically in the context of RDF "data describing web resources." The distinction between "data" and "metadata" is not an absolute one; it is a distinction created primarily by a particular application. Many times the same resource will be interpreted in both ways simultaneously. RDF encourages this view by using XML as the encoding syntax for the metadata. The resources being described by RDF are, in general, anything that can be named via a URI. The broad goal of RDF is to define a mechanism for describing resources that makes no assumptions about a particular application domain, nor defines the semantics of any application domain. The definition of the mechanism should be domain neutral, yet the mechanism should be suitable for describing information about any domain. This document introduces a model for representing RDF metadata and one syntax for expressing and transporting this metadata in a manner that maximizes the interoperability of independently developed web servers and clients. The syntax described in this document is best considered as a "serialization syntax" for the underlying RDF representation model. The serialization syntax is XML, XML being the W3C's work-in-progress to define a richer Web syntax for a variety of applications. RDF and XML are complementary; there will be alternate ways to represent the same RDF data model, some more suitable for direct human authoring. Future work may lead to including such alternatives in this document.
- Sample Queries: sparql select * {<f9ce44e7d05a16982e3a221d156beb45> ?p ?o}; sparql select * { ?event2 <http://www.rrecktek.com/demo/src> ?ip1. ?event2 <http://www.rrecktek.com/demo/handshake> "syn". ?event2 <http://www.rrecktek.com/demo/dst> ?ip2. ?event <http://www.rrecktek.com/demo/src> ?ip2. ?event <http://www.rrecktek.com/demo/handshake> "ack". ?event <http://www.rrecktek.com/demo/dst> ?ip1. } LIMIT 5; sparql select * { ?e1 <http://www.rrecktek.com/demo/src> ?ip1. ?e1 <http://www.rrecktek.com/demo/flag> "[S]". ?e1 <http://www.rrecktek.com/demo/dst> ?ip2. ?e2 <http://www.rrecktek.com/demo/src> ?ip2. ?e2 <http://www.rrecktek.com/demo/handshake> "ack". ?e2 <http://www.rrecktek.com/demo/dst> ?ip1. } LIMIT 5; sparql select * { ?e1 <http://www.rrecktek.com/demo/src> ?ip1. ?e1 <http://www.rrecktek.com/demo/flag> "[S]". ?e1 <http://www.rrecktek.com/demo/dst> ?ip2. ?e2 <http://www.rrecktek.com/demo/src> ?ip2. ?e2 <http://www.rrecktek.com/demo/handshake> "[S.]". ?e2 <http://www.rrecktek.com/demo/dst> ?ip1. ?e3 <http://www.rrecktek.com/demo/src> ?ip1. ?e3 <http://www.rrecktek.com/demo/handshake> "[.]". ?e3 <http://www.rrecktek.com/demo/dst> ?ip2. } LIMIT 5; which machines were the destination of the most traffic sparql select * { ?event <http://www.rrecktek.com/demo/dst> ?dst. } limit 10; sparql select distinct (?dst) (COUNT (?src) as ?count) { ?event <http://www.rrecktek.com/demo/dst> ?dst. ?event <http://www.rrecktek.com/demo/src> ?src. } ORDER BY DESC(?count) limit 10; http://www.ll.mit.edu/mission/communications/cyber/CSTcorpora/ideval/docs/attacks.html 1 Thurs dict 09:06:03 135.8.60.182 zeno alie tcp - http://www.rrecktek.com/demo/date http://www.rrecktek.com/demo/length http://www.rrecktek.com/demo/src http://www.rrecktek.com/demo/dst_port http://www.rrecktek.com/demo/ss http://www.rrecktek.com/demo/dst http://www.rrecktek.com/demo/flag http://www.rrecktek.com/demo/time http://www.rrecktek.com/demo/src_port http://www.rrecktek.com/demo/seq http://www.rrecktek.com/demo/win http://www.rrecktek.com/demo/handshake 1998-06-04 what times did the machines talk to each other sparql select * {?e <http://www.rrecktek.com/demo/src> "135.8.60.182". ?e <http://www.rrecktek.com/demo/dst> "172.16.113.50". ?e <http://www.rrecktek.com/demo/date> ?date. FILTER regex(?date, "1998-06-04"). ?e <http://www.rrecktek.com/demo/time> ?time }; what dates are loaded sparql select distinct (?date) {?e <http://www.rrecktek.com/demo/date> ?date. }; how many triples are loaded sparql select count (?e) {?e ?p ?o }; sparql select * { ?e1 <http://www.rrecktek.com/demo/src> ?ip1. ?e1 <http://www.rrecktek.com/demo/flag> "[S]". ?e1 <http://www.rrecktek.com/demo/dst> ?ip2. ?e2 <http://www.rrecktek.com/demo/src> ?ip2. ?e2 <http://www.rrecktek.com/demo/flag> "[S.]". ?e2 <http://www.rrecktek.com/demo/dst> ?ip1. ?e3 <http://www.rrecktek.com/demo/src> ?ip1. ?e3 <http://www.rrecktek.com/demo/flag> "[S]". ?e3 <http://www.rrecktek.com/demo/dst> ?ip2. } LIMIT 6; sparql select * { ?e1 <http://www.rrecktek.com/demo/src> ?ip1. ?e1 <http://www.rrecktek.com/demo/flag> "[S]". ?e1 <http://www.rrecktek.com/demo/dst> ?ip2. ?e1 <http://www.rrecktek.com/demo/date> ?date FILTER NOT EXISTS { ?e2 <http://www.rrecktek.com/demo/src> ?ip2. ?e2 <http://www.rrecktek.com/demo/flag> "[S.]". ?e2 <http://www.rrecktek.com/demo/dst> ?ip1. ?e2 <http://www.rrecktek.com/demo/date> ?date } } LIMIT 6; sparql select * { ?e <http://www.rrecktek.com/demo/dst> "172.16.115.234". ?e <http://www.rrecktek.com/demo/src> "135.13.216.191". ?e <http://www.rrecktek.com/demo/date> ?date. FILTER regex(?date, "1998-07-09"). ?e <http://www.rrecktek.com/demo/time> ?time }; none at the time Thurs pod 10:20:11 209.30.71.165 linux10 - tcp sparql select * { ?e <http://www.rrecktek.com/demo/dst> "172.16.118.100". ?e <http://www.rrecktek.com/demo/src> "209.30.71.165". ?e <http://www.rrecktek.com/demo/date> ?date. FILTER regex(?date, "1998-07-09"). ?e <http://www.rrecktek.com/demo/time> ?time }; sparql select * { ?e <http://www.rrecktek.com/demo/dst> "172.16.112.50". ?e <http://www.rrecktek.com/demo/src> "207.103.80.104". ?e <http://www.rrecktek.com/demo/date> ?date. FILTER regex(?date, "1998-07-09"). ?e <http://www.rrecktek.com/demo/time> ?time }; well what dates then ? sparql select distinct (?date) { ?e <http://www.rrecktek.com/demo/dst> "172.16.112.50". ?e <http://www.rrecktek.com/demo/src> "207.103.80.104". ?e <http://www.rrecktek.com/demo/date> ?date. }; sparql select * { ?e <http://www.rrecktek.com/demo/src> "207.103.80.104". ?e <http://www.rrecktek.com/demo/date> ?date. FILTER regex(?date, "1998-07-07"). ?e <http://www.rrecktek.com/demo/time> ?time }; sparql select * { ?e <http://www.rrecktek.com/demo/dst> "172.16.118.100". ?e <http://www.rrecktek.com/demo/src> "207.75.239.115". ?e <http://www.rrecktek.com/demo/date> ?date. FILTER regex(?date, "1998-07-09"). ?e <http://www.rrecktek.com/demo/time> ?time }; sparql select distinct (?date) { ?e <http://www.rrecktek.com/demo/dst> "172.16.118.100". ?e <http://www.rrecktek.com/demo/src> "207.75.239.115". ?e <http://www.rrecktek.com/demo/date> ?date. }; sparql select distinct (?date) { ?e <http://www.rrecktek.com/demo/src> "207.75.239.115". ?e <http://www.rrecktek.com/demo/date> ?date. };
- Another example: Identify a 3-way handshake: sparql select * { ?e1 <http://www.rrecktek.com/demo/src> ?ip1. ?e1 <http://www.rrecktek.com/demo/flag> "[S]". ?e1 <http://www.rrecktek.com/demo/dst> ?ip2. ?e2 <http://www.rrecktek.com/demo/src> ?ip2. ?e2 <http://www.rrecktek.com/demo/flag> "[S.]". ?e2 <http://www.rrecktek.com/demo/dst> ?ip1. ?e3 <http://www.rrecktek.com/demo/src> ?ip1. ?e3 <http://www.rrecktek.com/demo/flag> "[S]". ?e3 <http://www.rrecktek.com/demo/dst> ?ip2. }; ---------------------- which machines were the destination of the most traffic sparql select * { ?event <http://www.rrecktek.com/demo/dst> ?dst. } limit 10; sparql select distinct (?dst) (COUNT (?src) as ?count) { ?event <http://www.rrecktek.com/demo/dst> ?dst. ?event <http://www.rrecktek.com/demo/src> ?src. } ORDER BY DESC(?count) limit 10; ---------------------- what times did the machines talk to each other sparql select * { ?e <http://www.rrecktek.com/demo/src> "135.8.60.182". ?e <http://www.rrecktek.com/demo/dst> "172.16.113.50". ?e <http://www.rrecktek.com/demo/date> ?date. FILTER regex(?date, "1998-06-04"). ?e <http://www.rrecktek.com/demo/time> ?time };
- take all source IP and their respective destination IPs. for each source count how many times it went to a destination, report source destination and count. took 42 seconds over 70M events sparql SELECT ?src ?dst (count (?dst) as ?count) { ?e1 <http://www.rrecktek.com/demo/src> ?src. ?e1 <http://www.rrecktek.com/demo/dst> ?dst. } ORDER BY DESC (?count) LIMIT 10; src dst count VARCHAR VARCHAR VARCHAR _______________________________________________________________________________ 135.13.216.191 172.16.112.50 524932 172.16.112.50 135.13.216.191 277379 172.16.112.194 194.27.251.21 152972 194.27.251.21 172.16.112.194 146144 172.16.112.194 194.7.248.153 132120 194.7.248.153 172.16.112.194 122537 192.168.1.10 172.16.112.20 58477 206.186.80.111 172.16.112.50 53937 135.8.60.182 172.16.114.50 50606 202.77.162.213 172.16.114.50 43560 10 Rows. -- 42544 msec.
- No actual swans or dancers were harmed in the creation of this presentation. http://whatdoesdemocracylooklike.wordpress.com/2011/07/11/bertolt-brechts-interpretation-of-the-buddhas-parable-of-the-burning-house/ “ Lately I saw a house. It was burning. The flame Licked at its roof. I went up close and observed That there were people still inside. I entered the doorway and called Out to them that the roof was ablaze, so exhorting them To leave at once. But those people Seemed in no hurry. One of them, While the heat was already scorching his eyebrows, Asked me what it was like outside, whether there was Another house for them, and more of this kind. Without answering I went out again. These people here, I thought, Must burn to death before they stop asking questions. And truly friends, Whoever does not yet feel such heat in the floor that he’ll gladly Exchange it for any other, rather than stay, to that man I have nothing to say.” From Bertolt Brecht’s poem, “The Buddha’s Parable of the Burning House”