Scanning the Internet for External Cloud Exposures via SSL Certs
Debs2009 Event Processing Languages Tutorial
1. epts
event processing technical society
Event Processing
Language Tutorial
François Bry, Michael Eckert, Opher Etzion, Adrian Paschke, Jon Riecke
on behalf of the epts languages analysis working group
2. epts
event processing technical society
TUTORIAL OUTLINE
• Introduction Opher Etzion
• stream processing languages Jon Riecke
• rule oriented languages Adrian Paschke Languages
Styles
Break
• Agent oriented languages Opher Etzion
• Temporal semantics Jon Riecke
Advanced
• Integrated development environments Jon Riecke
Topics
• Formal approaches Adrian Paschke
• Conclusion Opher Etzion
2
3. epts
event processing technical society
TUTORIAL OUTLINE
• Introduction Opher Etzion
• stream processing languages Jon Riecke
• rule oriented languages Adrian Paschke Languages
Styles
Break
• Agent oriented languages Opher Etzion
• Temporal semantics Jon Riecke
Advanced
• Integrated development environments Jon Riecke
Topics
• Formal approaches Adrian Paschke
• Conclusion Opher Etzion
3
4. epts
event processing technical society
What is it all about?
Filter, transform, enrich, route,
Detect patterns, derive events
Event Processing
Event Event
Producer Consumer
Generate and emit Consume and react
events perform operations on to either raw or
events complex events
Event processing has
gained a lot of interest in
recent years; according to
analysts it is the fastest
growing segment of
middleware software
In this tutorial we are Pattern
exploring the language Match
issue – how the various
event processing
functions are expressed?
4
5. epts
event processing technical society
An Observation The Babylon Tower symbolizes the tendency
Of humanity to talk in multiple languages.
The Event Processing area is no different: most languages in the industry really follow
the hammer and nails syndrome – and extended existing approaches
• imperative script language
• SQL extensions
• Extension of inference rule language
It does not seem that we’ll succeed to settle
In the near future around a single programming style
The epts language analysis workgroup is aimed to understand the various styles
And extract common functions that can be used to define what is an event
processing language; this tutorial is an interim report
5
6. epts
event processing technical society
Existing Styles for EP languages (samples)
Inference TIBCO
Rules
Prova WBE XChangeEQ
ECA
Rules RuleCore
AMiT
Starview Agent Agent Logic
Oriented
EventZero Spade
SQL
Aleri extension Streambase
Coral8
State Oracle
oriented
Esper
Imperative/
Netcool Impact Script Based Apama
* - if we add simple and mediated event processing the picture is even more diversified
6
7. epts
event processing technical society
Dimensions to describe languages
• Events Data and Meta-data
• State
• Computation/execution model
• Programming model
7
8. epts
event processing technical society
The event Data and meta-data dimension
• Event structures supported (structured, semi-
structured, unstructured)
• Data types supported for event’s payload
• Event identifiers/identities
• Event time-stamps and ordering
• Event relations
• Other meta-data entities in the language
8
9. epts
event processing technical society
The state dimension
• State of events / data
– Events and other data in states
– Life-cycle, duration policies
– Explicit visibility and manipulation of states
• State of execution
– Context-related grouping (e.g. time windows)
– Explicit visibility into state of execution
9
10. epts
event processing technical society
The execution model dimension
• Execution timing – immediate or deferred ?
• Guarantees: real-time, determinism, latency ?
• Time point or time interval semantics
• Concurrency – implicit / explicit
• Event at a time / set at a time processing
• Explicit event flow
10
11. epts
event processing technical society
The programming model dimension
• Programming style
• Abstractions
• Pattern matching
• Enrichment capabilities
• Transformation capabilities
• Filter capabilities
11
12. epts
event processing technical society
TUTORIAL OUTLINE
Introduction Opher Etzion
stream processing languages Jon Riecke
• rule oriented languages Adrian Paschke Languages
Styles
Break
• Agent oriented languages Opher Etzion
• Temporal semantics Jon Riecke
Advanced
• Integrated development environments Jon Riecke
Topics
• Formal approaches Adrian Paschke
• Conclusion Opher Etzion
12
13. epts
event processing technical society
Hallmarks of Stream Processing
• Events are processed in a directed graph
• Nodes = computational elements
• Edges = communication of events between nodes
• Languages inspired by SQL/relational algebra (though not
all use an extended SQL as the language)
• Difference: write the query first
• Results of query are updated as data flows in
• Academic examples: STREAM, Aurora, Cayuga, …
• Commercial examples: Aleri, Coral8, Esper/Nesper,
StreamBase, System S, …
13
14. epts
event processing technical society
Some pictures
14
15. epts
event processing technical society
Some pictures
15
16. epts
event processing technical society
Some pictures
16
17. epts
event processing technical society
Historical Roots
• Relational Databases
• Triggers: change table when another table changes
• Materialized view maintenance: calculate small set of
changes to view given set of changes to base tables
• Signal processing
• Control and Audio: Ptolemy, LabVIEW, …
• Music and video: OpenMusic, jMax, Max, vvvv,
17
18. epts
event processing technical society
Historical Roots: Static Dataflow Architecture
• Developed by Jack Dennis in 1960s
• Each arc carries one token
• Node fires when all data available on all arcs
18
19. epts
event processing technical society
Historical Roots: Dynamic Dataflow Architecture
• Matching section in front of each node
• Tokens are tagged by invocation instance; when all
tokens with same tag are ready, node fires
• Allows more parallelism than static model
19
21. epts
event processing technical society
Stream Processing versus
Signal Processing and Dataflow Architecture
• Events carry more information than simple data:
records instead of, e.g., decimal numbers
• Slightly slower data rates (up to millions of events
per second)
• Publish/subscribe model of input/output
21
22. epts
event processing technical society
Dimension Commonalities in Stream Processing
• Events: Non-mutable records (fields with values)
Values have scalar types (integer, float, string, blob, xml)
• Metadata: describes streams and types of records
• State: Unprocessed and processed events, and internal
state of operations (e.g., aggregations)
• Computational model: records traverse the dataflow graph
• Programming model: extensions of SQL/relational algebra
22
23. epts
event processing technical society
Dimension Differences
• Events: small differences in scalar types
• State
• User-defined data structures (arrays, maps, …)?
• Disk-based persistence? What is persisted?
• Programming model
• Visual programming or text-based?
• Embedded in larger language, e.g., Java?
• Smaller language embedded in it?
23
24. epts
event processing technical society
Dimension Differences
• Computational model
• Cycles in dataflow graph?
• Deterministic?
• Consistent? What form of consistency?
• Parallel or concurrent execution?
• Distributed computation? Distributed data?
• Synchronous or asynchronous communication?
24
25. epts
event processing technical society
Aleri Streaming Platform Dataflow Networks
• One queue per node (instead of per edge)
• Nodes store records in table
• Events (insert/update/delete) change the table
• Nodes run in separate threads
• Optional persistence with roll-forward recovery
25
26. epts
event processing technical society
Aleri Streaming Platform Dataflow Networks
• Computational model
• Events may arrive in different orders, and may cause
different output events
• Correctness: standard operators (join, compute, …)
satisfy “eventual consistency”: the sequence of
insert/update/deletes will produce the same table, no
matter how the events flow through the graph
26
27. epts
event processing technical society
Aleri Streaming Platform Dataflow Networks
B
A C
• Node A sends insert:[Id=3, Symbol=IBM, Price=45.0] Node B sends update:
[Symbol=IBM, AvgPrice=43.40]
• Different arrival orders can cause different events
• (A then B)
• insert: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=42.00]
• update: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=43.40]
• (B then A)
• insert: [Id=3, Symbol=IBM, Price=45.0, AvgPrice=43.40]
27
28. epts
event processing technical society
Coral8 Dataflow Networks
• No queues; synchronization buffer in front
• Two kinds of nodes:
• Stream nodes process records but don’t store them
• Window nodes process and store records
• Optional persistence for unprocessed events, records
28 in windows, aggregate state
29. epts
event processing technical society
Coral8 Dataflow Networks
• Computational model (simplified)
• External data flows into synchronization buffer
• Events carry a timestamp (point in time, microsecond
granularity): set on arrival, or by source
• Events with same timestamp are fed into the network in
one step, and are processed until network quiesces
29
30. epts
event processing technical society
Coral8 Dataflow Networks
• Computational model
• Deterministic: CCL programs produce the same output
events given the same input
• Priorities assigned to nodes in graph to maintain
determinacy
30
31. epts
event processing technical society
Coral8 Dataflow Networks
• Computational model: concurrent/distributed
• Project = group of CCL streams, windows, queries
(code for calculating new events from old)
• Projects can send/receive events to/from other projects
• Each project has a thread (there are other threads too)
31
• Projects can be distributed across machines
32. epts
event processing technical society
Examples in CCL: Simple Market Data
• Schema: type of records
– CREATE SCHEMA Trades_t
– (Symbol STRING, Qty INTEGER, Price FLOAT);
– CREATE SCHEMA Book_t
– (Symbol STRING, Qty INTEGER, BuyPrice
FLOAT);
• Other scalar types: BOOLEAN, TIMESTAMP,
INTERVAL, XML, BLOB
• CCL streams: INPUT, OUTPUT, local
– CREATE INPUT STREAM Trades_s SCHEMA Trades_t;
– CREATE INPUT STREAM Book_s SCHEMA Book_t;
32
33. epts
event processing technical society
CCL Windows
• Windows are like streams, but store records
– CREATE WINDOW Book_w SCHEMA Book_t KEEP ALL;
– INSERT INTO Book_w
– SELECT * FROM Book_s;
• Sample of KEEP policies (there are others):
– KEEP LAST PER Id
– KEEP 3 MINUTES
– KEEP EVERY 3 MINUTES
– KEEP UNTIL (”MON 17:00:00”)
– KEEP 10 ROWS
– KEEP LAST ROW
– KEEP 10 ROWS PER Symbol
33
34. epts
event processing technical society
CCL Queries: SQL-based syntax
• Aggregation
– CREATE STREAM Vwap_s SCHEMA (Symbol STRING, Vwap FLOAT);
– INSERT INTO Vwap_s
– SELECT Symbol, sum(Qty * Price)/sum(Qty)
– FROM Trades_s KEEP 30 MINUTES
– GROUP BY Symbol;
• Join
– CREATE SCHEMA Value_t
– INHERITS FROM Book_t (CurVal FLOAT, AvgVal FLOAT);
– CREATE WINDOW BookValue_w
– SCHEMA Value_t KEEP LAST PER Symbol;
– INSERT INTO BookValue_w
– SELECT B.Symbol, B.SharesHeld, B.BuyPrice,
– B.SharesHeld * L.Price, B.SharesHeld*V.Vwap
– FROM Book_w B, LastTrade_w L, Vwap_s V
– WHERE B.Symbol = L.Symbol and B.Symbol = V.Symbol;
34
35. epts
event processing technical society
CCL Pattern Matching
• MATCHING clause
– INSERT INTO OrdersMatch_s
– SELECT B.ID, S.ID, B.Price
– FROM BuyOrders_s B, Trades_s T, SellOrders_s S,
– MATCHING [30 SECONDS: B, !T, S]
– ON B.Price = S.Price = T.Price;
• Operators in patterns (cf. finite-state automata):
• Three boolean operators: ! (not), & (and), | (or)
• One temporal operator: , (followed by)
• Temporal scope can be nested
– MATCHING [30 SECONDS: [10 SECONDS: B, !T], S]
35
36. epts
event processing technical society
Modularity
• Note: concrete syntax in future version of CCL
• Group of streams can be abstracted into a module
– CREATE MODULE OptionPrice
– CREATE INPUT STREAM Option_s;
– CREATE OUTPUT STREAM OptionPrice_s;
– CREATE PARAMETER FLOAT InterestRate = 0.05;
– // internal stream definitions
– END MODULE;
• Modules can be instantiated
● LOAD MODULE OptionPrice as LondonOptionPrices
● STREAMS
● Input = LondonOptions_s,
● Output = LondonOptionPrices_s
● PARAMETERS
● InterestRate = 0.03;
36
37. epts
event processing technical society
Other Features from Other Languages
• Esper (Java)/Nesper (C#)
• Extensions of SQL
• Embedded within host language
• AleriML
• SPLASH: a C-like embedded language for writing nodes
• Data structures: vectors, dictionaries, event caches
• Mechanisms for discarding stored records (time, # records)
• StreamBase
• Primitive types include nested records and vectors (lists)
• LOCK/UNLOCK for holding and releasing records
37
38. epts
event processing technical society
TUTORIAL OUTLINE
Introduction Opher Etzion
stream processing languages Jon Riecke
Rule oriented languages Adrian Paschke Languages
Styles
• Break
• Agent oriented languages Opher Etzion
• Temporal semantics Jon Riecke
Advanced
• Integrated development environments Jon Riecke
Topics
• Formal approaches Adrian Paschke
• Conclusion Opher Etzion
38
39. epts
event processing technical society
Production Rules
39
40. epts
event processing technical society
Production Rules and CEP
• Production rules react to states changes (not events)
• PR systems with object model and external fact updates might
be extended to CEP
– Event types and classes are defined in the rule declarations
– New event data instances are added to the fact base / working
memory
• might be an external fact base, e.g. a event queue
– The instances of declarations are filtered and joined in the
conditions via typed pattern matching and if they pass the condition
list the action part of the rule is triggered
• Might be further extended with mechanisms like query
languages, state models and temporal constraints
• Examples: TIBCO Business Events, Drools
40
41. epts
event processing technical society
Production Rules
• A production rules is a statement of the form:
“if Condition then Action”
• Members, e.g. OPS5, Clips, Jess, Drools, Fair Isaac Blaze
Advisor, ILog jRules, CA Aion, Haley, ESI Logist, …
• Operational semantics: forward-chaining
– primitive update actions (assert, retract, …);
41
42. epts
event processing technical society
Operational Semantics
• Operational processing
– forward-chaining order-independent execution (e.g. based
on variations of the RETE algorithm)
– Procedural order-dependent sequential execution in a
(compiled) execution environment (rule engine).
• Conflict resolution
– Refraction
– Priority
– Recency
– Specificity
– …
42
43. epts
event processing technical society
The Rete Pattern-Matching Algorithm
• Rete Pattern-Matching Algorithm (and variations of it)
– matches facts against the patterns in rules to determine which rule
conditions are satisfied
– incremental evaluation of production rules' conditions
• Developed by Charles L. Forgy in the late 70s for OPS5 (Official
Production System) shell
– stores information about the antecedents in a network
– in every cycle, it only checks for changes in the networks
this greatly improves efficiency
43
44. epts
event processing technical society
OMG Production Rules Representation (OMG PRR)
• Formal model for vendor-neutral rule-model
representation in UML for production rules
• Consortium of developers and supporters
– Rule vendors (including Fair Isaac, IBM/ILOG, LibRT, Pega,
Corticon, TIBCO)
– Academic community (RuleML Group)
– Related vendor community (Fujitsu)
• PRR is currently “adopted" as a standard and in
“finalization", meaning that it is in Beta, with a final
version 1.0 in 2009.
44
45. epts
event processing technical society
OMG Production Rules Representation
• OMG MDA PIM model
• PRR beta spec applies to a PRR Core using OMG
MOF defined in UML
– Extends UML so production rules are 1st class citizens
alongside objects
– Spec. excludes an explicit expression language
• Can use other standards as expression language
– e.g. W3C Rule Interchange Format (W3C RIF) and the
production rule language of RuleML
45
46. epts
event processing technical society
OMG PRR ProductionRule Classes
46 if [condition] then [action]
47. epts
event processing technical society
W3C Rule Interchange Format
• W3C Rule Interchange Format (W3C RIF)
http://www.w3.org/2005/rules/wiki/RIF_Working_Group
• Goal
Standardization of a web-based Rule Interchange Format (RIF)
• RIF 1.0 three dialects:
– RIF Production Rules Dialect (RIF-PRD)
• specifies the RIF production rules dialect to enable the interchange
of production rules
– RIF Basic Logic Dialect (RIF-BLD)
• specifies a basic interchange format that allows logic rules (definite
Horn rules with equality) to be exchanged
– RIF Core
• RIF-Core, a common subset of RIF-BLD and RIF-PRD
47
48. epts
event processing technical society
W3C RIF Production Rules Dialect
• Production Rules (Condition-Action)
– Based on RIF condition language (from RIF Core)
– Actions: Retract, Assert, Modify, Execute
– Negation (inflationary negation)
• Syntax
– Normative XML syntax
• Might be concrete representation language for OMG PRR
– Non-normative EBNF-based presentation syntax
• for presentation purposes
48
49. epts
event processing technical society
RIF Production Rule
<Implies>
<if> FORMULA </if>
<then rif:ordered="yes">
<Do>
ACTION*
</Do>
</then> Syntax close to PR
RuleML syntax
</Implies>
49
51. epts
event processing technical society
Event Condition Action Rules
• ECA Rule
“on Event if Condition do Action”;
– Explicit complex event part; separated from conditions and
actions
• e.g. expresses customer order (event); check if credit card is
valid (condition)
• Evolved from active databases
– extend databases with reactions, e.g. HiPac, ACCOOD, Chimera,
ADL, COMPOSE, NAOS
– Composite event algebra, e.g. SAMOS, COMPOSE, Snoop
• Sequence | Disjunction | Xor | Conjunction |
Concurrent | Not | Any | Aperiodic | Periodic
51
52. epts
event processing technical society
Time-point vs. Interval-based Semantics
– Time-point semantics
• Event definition: B; (A;C) (Sequence)
event instance sequence EIS = b, a, c => detect event
event instance sequence EIS = a, b, c => detect event
– Interval-based semantics
• Event definition B;(A;C) (Sequence)
event instance sequence EIS = b, a, c => detect event
event instance sequence EIS = a, b, c => do not detect event
T1 T2 T3 T4
B A C
52
53. epts
event processing technical society
Operational Semantics
1. Specification Phase
– Definition of event/action pattern e.g. by event algebra
– Based on declarative formalization or procedural implementation
– Defined over an atomic instant or an interval of time,
events/actions, situation, transition etc.
2. Selection
– Defines selection function to select one event from several
occurred events of a particular type in an event instance
sequence, e.g. “first”, “last”
– Crucial for the outcome of a reaction rule, since the events may
contain different (context) information, e.g. different message
payloads or sensing information
53
54. epts
event processing technical society
Operational Semantics
1. Consumption
– Optionally defines which events are consumed after the detection of a
composite event
– An event may contribute to the detection of several composite events, if
it is not consumed
– Distinction in event messaging between “multiple receive” and “single
receive”
– Events which can no longer contribute, e.g. are outdated, should be
removed
2. Execution
– Actions might have an internal effect i.e. change the knowledge state
leading to state transition from (pre)-condition state to post-condition
state.
– Actions might have an external side effect
54
55. epts
event processing technical society
Example: REWERSE MARS
• MARS (Modular Active Rules for the Semantic Web)
– rule uses specialized heterogeneous event language (E), one
or more query languages (Q), a test language (T), and an
action language (A)
– each of these languages and their constructs are described by
metadata and an ontology of its semantics
– well-defined interface for communication between the E, Q&T,
and A components by variables
– SNOOP (E), OWLQ (Q&T) and CCS - Calculus of
Communicating Systems (A)
55
56. epts
event processing technical society
Example: XChangeEQ
• ECA rules ON Event query IF Web query DO Action
– the condition part is a declarative query to XML data
– the event part is as well a declarative query to XML event
data
• distributed and web-based: events issued by web nodes (with both
emission and reception times)
– - action: simple or complex updates send to web nodes
• Complex events are expressed as (compound) queries to the
event stream
– temporal restrictions and event compositions
• conjunction, disjunction, exclusion, quantifications, repetitions, and
ranks
• absolute temporal restrictions, also relative temporal restrictions
(within)
56
57. epts
event processing technical society
XChangeEQ
• The four dimensions of event querying are kept separated:
– data extraction (eg oder number, book title)
– event composition (eg oder of a dictionary and a grammar both from the same
publisher)
– temporal (and other) relationships (eg two orders within a week)
– event accumulation (eg no default of paiement between the order of the two
abovementioned buys)
• Equivalent Semantics:
– Declarative semantics: Model and fixpoint theories
– Operational semantics: CERA (Complex Event Relational Algebra)
• XChange^EQ operational semantics:
– by language design, the system never wait forever for a complex event to possibly
resume.
– Incremental evaluation (refining RETE).
– Built-in garbage collection of irrelevant complex events (eg events that cannot
resume).
57
58. epts
event processing technical society
Example: XChange Composite Event
andthen [
xchange:event {{
xchange:sender {"http://airline.com"},
cancellation-notification {{
flight {{ number { var Number } }} }}
}},
xchange:event {{
xchange:sender {"http://airline.com"},
important {"Accomodation is not granted!"}
}}
] within 2 h
58
59. epts
event processing technical society
ruleCore Markup Language (rCML)
• XML language for ECA rules
• Complex Event defined in four sub-elements:
– <detect-event>. The <detect-event> element specifies the
event that is generated when the composite event is detected.
– <no-detect-event>. The <no-detect-event> element specifies
the event that is generated when the composite event can not
be detected.
– <event-selector> The <event-selector> element specifies a
logical condition (or filter) for the composite event.
– <detector>. The composite event detector consists of a number
of event algebra operators (the SNOOP algebra operators) that
describe the composite event.
59
60. epts
event processing technical society
IBM AMIT Situation Manager Rule Language
• Definition of situations and their detection conditions
– syntactically equivalent to (complex) event patterns
– lifespans which hold references to events with relevant context for a
particular pattern calculation
• Conceptual event model defines an event type generalization
hierarchy
– Root event type defines a set of standard attributes e.g.,
• the event source, creation type, temporal dimensions like occurrence time,
detection time and /or transaction time of an event, and quantifications such
as status, priority, severity, count
– Application specific event types inherit from root event type and define
further user-defined attributes
– Two types of event relationships
• Based-on - a pure syntactical relationship to ease the development
process. If an event A is 'basedon' event B, event A inherits the entire
structure of event B and can extend it.
• Reference - a relationship that allows deriving events without pattern
detection. Useful for transformation and enrichment purposes.
60
61. epts
event processing technical society
Example AMIT SMRL
AMIT SMRL markup language provides an event algebra with semantic notations such as
context (lifespan intervals), and semantic association (event group) to define situations
<eventTypes>
<eventType name="doctorEnterRoom">
<attributeType name="doctor" xsi:type="string"/>
<attributeType name="room" xsi:type="integer"/> </eventType>
<eventType name="patientEnterRoom">
<attributeType name="patient" xsi:type="string"/>
<attributeType name="r_o_o_m" xsi:type="integer"/>
</eventType>
<eventType name="lockRoom">
<attributeType name="room" xsi:type="integer"/> </eventType>
<eventType name="unlockRoom">
<attributeType name="room" xsi:type="integer"/>
</eventType>
</eventTypes>
<lifespan name="roomOpen">
<initiator> <eventInitiator name="unlockRoom"/> </initiator>
<terminator> <eventTerminator name="lockRoom"/> </terminator>
<keyBy name="sameRoom"/>
</lifespan>
61
62. epts
event processing technical society
(Reaction) RuleML
• General (reaction) rule form that can be specialized as needed
• Platform-independent XML-based rule interchange format
– translation into platform-specific executable rule languages, e.g. Prova
• Three general execution styles:
– Active: 'actively' polls/detects occurred events in global ECA style, e.g. by a ping on
a service/system or a query on an internal or external event database
– Messaging: Waits for incoming complex event message
– Reasoning: KR event/action logic reasoning and transitions (as e.g. in Event
Calculus, Situation Calculus, TAL formalizations)
• Appearance
– Global: ‘globally’ defined reaction rule
– Local: ‘locally’ defined reaction rule in a specific context
62
64. epts
event processing technical society
Selected Reaction RuleML Extended Features
• Action Algebra:
Succession (Ordered Succession of Actions), Choice
(Non-Determenistic Choice), Flow (Parallel Flow), Loop
(Loops)
• Event Algebra:
Sequence (Ordered), Disjunction (Or) , Xor (Mutal
Exclusive), Conjunction (And), Concurrent , Not, Any,
Aperiodic, Periodic
• Support for event / action messaging
• Support for different detection, selection and consumption
policies
• Support for intervals (Time, Event)
• Support for situations (States, Fluents)
• Support for external event query languages
• ...
64
65. epts
event processing technical society
10 minutes break
TUTORIAL OUTLINE
Introduction Opher Etzion
stream processing languages Jon Riecke
rule oriented languages Adrian Paschke Languages
Styles
Break
• Agent oriented languages Opher Etzion
• Temporal semantics Jon Riecke
Advanced
• Integrated development environments Jon Riecke
Topics
• Formal approaches Adrian Paschke
• Conclusion Opher Etzion
65
66. epts
event processing technical society
TUTORIAL OUTLINE
Introduction Opher Etzion
stream processing languages Jon Riecke
rule oriented languages Adrian Paschke Languages
Styles
Break
• agent oriented languages Opher Etzion
• Temporal semantics Jon Riecke
Advanced
• Integrated development environments Jon Riecke
Topics
• Formal approaches Adrian Paschke
• Conclusion Opher Etzion
66
69. epts
event processing technical society
What is the idea ?
• The EPN (event processing network) represents the event
processing application as a directed graph:
– The nodes represent event processing agents (and states in some
models)
– The edges represent either individual events or event streams
(depends on the processing type). A generalized EPN may support
both.
• The coverage of EPN varies
– It may cover some language operator (e.g. SQL like language)
– It may take a broader approach and cover also routing decisions, pub/
sub etc…
– EPN can have a specific programming model, or an hybrid
programming model, where one or more languages represent the
agents.
69
70. epts
event processing technical society
Anatomy of Agents
Performs the processing
Selects the events that are
Derives output events
relevant for the processing detect patterns)
(e.g.
Selector Processor Deriver
70
73. epts
event processing technical society
Making the agents context sensitive
Time interval that starts by: {event, clock, offset from event}
and terminates by: {event, clock, offset from event, count of
certain events} – each can have multiple instances (e.g. several
events, periodic clock, multiple offsets}
When?
Examples: [9:00AM, angry-customer-alert], [angry-
customer-alert, +1 hour], [request – 2 hours,
request]
Spatial dimensions/ abstraction
Where? Inside building A, In Italy, within 1KM from here
(given GPS coordinate)
An entity or a collection of entities that determine the partition
of the context
Who? Class of customer, customer + order, large order…
The event happens when some state is in effect
Which state?
Traffic Jam, Red Alert, Bullish Market
73
74. epts
event processing technical society
Filtering
Many of the raw events are not
relevant for processing, in some
applications only a sample is
selected for processing.
74
75. epts
event processing technical society
Transformation
Translation / projection:
Translation and simple
derivations
Aggregation/ composition:
statistical aggregator or
concatenating events, can be
stand alone agent or a scalar
derivation. May be in network
edge.
Splitting: Splits events to
multiple events
75
76. epts
event processing technical society
Enrichment
Enriches the content of events
from reference data in databases,
spreadsheets, Email messages,
text files etc..
76
77. epts
event processing technical society
Pattern Matching Pattern definition
example
In many cases, the agent does not
consume raw events, but event that
are derived from pattern matching.
DEBS 2008 tutorial on
event processing patterns
provided a deep dive
77
78. epts
event processing technical society
Dimensions - I
• Meta-data
– EPN and agent definitions may be defined as meta-data
– Event schema is implementation dependent
• State
– State can be represented as another node in the EPN
– Some EPAs may be state oriented, e.g. the input event is
change in state
– State is one of the context dimensions
78
79. epts
event processing technical society
Dimensions - II
• Execution model:
– EPN representation is natural one for
concurrency/parallelism since the dependencies are
explicit
– Since it models explicitly the end-to-end event flow, it
serve as a convenient model for QoS related
optimizations (latency, throughput, real-time constraints)
• Programming model:
– Can support multiple programming styles
– Can support all types of event processing functionality
79
81. epts
event processing technical society
Event Zero example
81
82. epts
event processing technical society
System S Example
Congestive Heart Failure
Freq alert
Arrhythmia
Analyzer
QRS RR P and T
Source PE Glucose
alert
BP S/D CHF
Weight trend IHE
detector
Adapter
*Daby Sow
82
83. epts
event processing technical society
Agent Logic example
83
84. epts
event processing technical society
TUTORIAL OUTLINE
Introduction Opher Etzion
stream processing languages Jon Riecke
rule oriented languages Adrian Paschke Languages
Styles
Break
agent oriented languages Opher Etzion
• Temporal semantics Jon Riecke
Advanced
• Integrated development environments Jon Riecke
Topics
• Formal approaches Adrian Paschke
• Conclusion Opher Etzion
84
85. epts
event processing technical society
Varieties of Temporal Semantics
• Do events carry timestamps? Where does the timestamp
come from (CEP system or external world)?
• Granularity? Second, millisecond, microsecond, …?
• Point in time or interval of time?
• Can events arrive out-of-order from a single data source?
• Can events be reordered if they arrive out-of-order?
• How long does the system wait?
• What happens if system moves ahead, and old event arrives?
• In distributed case, is there a difference between emission
and reception time?
85
86. epts
event processing technical society
Simple Temporal Semantics
• Events arrive in order from a single data source
• Events have timestamps, but not reordered
• Advantage
• Process as soon as possible
• Disadvantages
• Non-deterministic
• Hard to manage if time really matters
• Emission times de-synchronized in distributed case
86
87. epts
event processing technical society
Global Clock
• Events have timestamps consistent across data sources
• Wait for events for specified time; discard if wait too long
• Advantages
• Deterministic
• Time might be relevant
• Disadvantages
• Speed of processing
• Buffering required for reordering events
87
88. epts
event processing technical society
TUTORIAL OUTLINE
Introduction Opher Etzion
stream processing languages Jon Riecke
rule oriented languages Adrian Paschke Languages
Styles
Break
agent oriented languages Opher Etzion
Temporal semantics Jon Riecke
Advanced
• Integrated development environments Jon Riecke
Topics
• Formal approaches Adrian Paschke
• Conclusion Opher Etzion
88
89. epts
event processing technical society
Integrated Development Environments (IDEs)
• Two main forms
• Text based
• Visual or graphical based
• Not an either-or; most systems have some
combination of text and visual display of dataflow
89
90. epts
event processing technical society
Visual IDEs
• Streams = nodes, flows = edges
• Drag-and-drop streams from palette onto canvas
and wire together
• Configure streams with property sheets or text
• Advantages: simple metaphor; easy to see flow
• Disadvantages: managing screen real-estate
90
91. epts
event processing technical society
Visual IDE Example
91
92. epts
event processing technical society
Text-based IDEs
• Streams represented in text
• Advantages
• Appeals to programmers
• Can be quicker to write code
• Disadvantages
• Hard to see the data flow
• Hard to find bottlenecks in program
92
93. epts
event processing technical society
Text-based IDE
93
94. epts
event processing technical society
Another Text-based IDE
94
95. epts
event processing technical society
Advanced Features in IDEs
• Ability to start/stop programs
• Debugging, including viewing intermediate events
and setting breakpoints
• Tests, including ability to control data feeds
• Dashboards for displaying results
95
96. epts
event processing technical society
Breakpoints and Debugging
96
97. epts
event processing technical society
Performance Monitoring
97
98. epts
event processing technical society
Dashboard Construction
98
99. epts
event processing technical society
Dashboard Construction
99
100. epts
event processing technical society
Assessment: IDEs still have a ways to go
• Need more features that programmers have come
to expect
• Single-stepping with line numbers
• Examination of events and data
• Forward and backward replay of time
• Need to be integrated into more dashboards and
other visualization tools
100
101. epts
event processing technical society
TUTORIAL OUTLINE
Introduction Opher Etzion
stream processing languages Jon Riecke
rule oriented languages Adrian Paschke Languages
Styles
Break
agent oriented languages Opher Etzion
Temporal semantics Jon Riecke
Advanced
Integrated development environments Jon Riecke
Topics
• Formal approaches Adrian Paschke
• Conclusion Opher Etzion
101
102. epts
event processing technical society
Formal Approaches
Temporal Event/Action Logics for
Reasoning on Changes
102
103. epts
event processing technical society
(Temporal) Event Logics
• (Temporal) event/action logics
– Events with effects on changeable properties / states / fluents
– Focus: reasoning on effects of events/actions on knowledge
states and properties
– Not Complex Event Processing Languages, but can be used
in reasoning on/with complex events
• Members e.g.
– event calculus and variants,
– situation calculus,
– features and fluents calculus,
– various (temporal) action languages (TAL),
– fluent calculi and versatile event logics.
103
104. epts
event processing technical society
The Situation Calculus
• A situation is a snapshot of the
world at some point in time
• Every true or false statement is
made with respect to a
particular situation
• When an agent performs an action A is situation S1, the
result is a new situation S2
• McCarthy 1963, McCarthy & Hayes 1969, Green 1969
104
105. epts
event processing technical society
Event Calculus
• Kowalski and Sergot’s EC
– a formalism for temporal reasoning about events and their effects
• computation of earlier events (long-term "historical" perspective)
– model of change
• events happen at time-points and initiate and/or terminate properties (time-
varying fluents) of the world
– Law of inertia:
Things normally tend to stay the same
• Main differences in EC to SC
– branching time in SC vs. linear time in EC
– explicit notion of previous state of the world / situation in SC
– state transitions are functions in SC
105
106. epts
event processing technical society
Classical Event Calculus Example
• EC Basic Axioms:
– happens(E,T) event E happens at time point T
– initiates(E,F,T) event E initiates fluent F for all time>T
– terminates(E,F,T) event E terminates fluent F for all time>T
– holdsAt(F,T) fluent F holds at time point T
• Many EC Extensions, e.g.:
– valueAt(P,T,X) parameter P has changeable value X at time point T
– planned(E,T) event E is believed to happen at time point T
Example:
initiates(stopService,serviceUnavailable,T) stopService startService
terminates(startService,serviceUnavailable,T)
happens(stopService,t1); happens(startService,t5)
t1 t3 ? t5 t7 ?
holdsAt(serviceUnavailable,t3)? true
holdsAt(serviceUnavailable,t7)? false
106
107. epts
event processing technical society
Example: Interval-based Event Calculus Event Algebra
– Event initiate and terminate Situations (Fluents) which hold at an time interval
– Interval-based Event Calculus semantics (model-theory + proof theory)
based on time intervals modeled as fluents
I: Ti x Fl {true, false}
– Example: B;(A;C) (Sequence)
T1 T2 T3 T4
A B C
(A;B;C) [T1,T3]) <=
holdsInterval([a,b] [T1,T2]) Λ holdsInterval([b,c] [T2,T3]) Λ [T1,T2]<=[T2,T3]
– Rule-based implementation of EC event algebra, e.g. as meta logic program
– Rule-based arithmetic involving times and durations, e.g. Allen’s interval logic
107
108. epts
event processing technical society
Combinations of Rule Types
1. Rules that influence the operational / decision processes:
• Derivation rules (deduction rules): establish / derive new
information that is used e.g. in a decision process.
• Reaction rules that establish when certain activities should place
(e.g. ECA rules)
2. Constraints on system/organisation's structure, behavior or information:
• Structural constraints (e.g. deontic assignments), State constraints,
Process / flow constraints
108
110. epts
event processing technical society
Messaging Reaction Rules in Prova
• Send a message
sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive a message
rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive multiple messages
rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
Syntax:
– XID is the conversation / event instance sequence identifier
– Protocol: ESB transport protocols (>30 e.g. self, jade, jms, soap,…)
– Agent: denotes the target or sender of the message
– Performative: pragmatic context, e.g. FIPA ACL primitives
– [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload
110
111. epts
event processing technical society
TUTORIAL OUTLINE
Introduction Opher Etzion
stream processing languages Jon Riecke
rule oriented languages Adrian Paschke Languages
Styles
Break
agent oriented languages Opher Etzion
Temporal semantics Jon Riecke
Advanced
Integrated development environments Jon Riecke
Topics
Formal approaches Adrian Paschke
Conclusion Opher Etzion
111
112. epts
event processing technical society
Event Processing Languages – insights from this tutorial
epts
event processing technical society
epts
event processing technical society
An Observation
Dimensions toTower symbolizeslanguages
The Babylon describe the tendency
Of humanity to talk in multiple languages.
• Events Data and Meta-data
• State
• no different: most languages in the model
The Event Processing area is Computation/execution industry really follow
the hammer and nails syndrome – and extended existing approaches
• Programming model
• imperative script language
• SQL extensions
• Extension of inference rule language
It does not seem that we’ll succeed to settle
In the near future around a single programming style
The epts language analysis workgroup is aimed to understand the various styles
And extract common functions that can be used to define what is an event
processing language; this tutorial is an interim report
112
113. epts
event processing technical society
Towards event processing languages standard ?
• Eventually – a single language ?
1st generation
SQL standard approved
of relational DBMS products
Mid 1970-ies 1990
1st generation
EP standard approved
of EP products
Mid 200-ies ?
• OR – variety of language style standards ?
• Meta-language as an intermediate standard ?
113
114. epts
event processing technical society
The epts “Event Processing Language Analysis” workgroup
Mission:
The purpose of this working group is to conduct a study of the features that exist in the contemporary and planned languages
in the area of event processing (including event stream processing).
The goal is to understand the functional properties that exist in event processing languages and abstract out semantic functions,
regardless of syntax and implementation considerations.
This is a first step in a way to determine future possible standardization in the event processing area
WG Leaders: Opher Etzion (IBM) and Jon Riecke (Aleri)
Members:
• Alex Kozlenkov (Betfair)
• Adrian Paschke (FU Berlin)
• Arno Jacobsen (U of Toronto)
• Bala Maniymaran (U of Toronto)
• Bob Hagmann (Aleri)
• David Tucker (EventZero)
• Dermot McPeake (FirstDerivatives)
• Francois Bry (LMU)
• Guy Sharon (IBM)
• Louis Lovas (Progress/Apama)
• Michael Eckert (TIBCO)
• Pedro Bizarro (University of Coimbra)
• Richard Tibbetts (Streambase)
• Robert McKeown (IBM)
• Susan Urban (Texas Tech. University)
• Simon Courtenage (Westminister U.)
• Serge Mankovskii (CA)
114
115. epts
event processing technical society
If you want to participate in these discussions, contribute to this
topic and/or others topics, and be part of the community effort to
advance the state of the art and state of the practice:
JOIN EPTS
For details: http://www.ep-ts.com/