Event Processing - The Next Generation; March 2009

IBM Haifa Research Lab – Event Processing

!
quot; #$ % quot;
&

© 2008IBM Corporation


' (

I am wearing three
different hats today

The industrial hat: IBM
(which pays my salary and
in return occupies most of
my time)
Once a week I am
wearing the Professor
hat and teach an event
processing course

The EPTS chair hat – a
catalyst in establishing
this community
IBM Haifa Research Lab – Event Processing © 2008 IBM Corporation


)* + ,-./

0 12
- ) 3
4 5 6
5 4 6
'# # )

! #) # # #
4
quot; 4 # 7
7# ) 4 4 7 4 ! 8
' 2 ! #& 4 ) !4
#
# 4 # #
0 $ $ $# 3



4

What is event processing ? What is it used for?

A quick Introduction second generation

What are the current Research Challenges ?

Survey of recent projects

Summary



# ) !4
Every business person reacts to events, from time
to time

Well… there are all sort of reactions…



We answer
phone calls
everywhere



We have to make arrangements when our
flight is delayed



7 4
4
I closed the deal with the
I closed Australians
the deal with the
Canadians

We closed two huge deals in a single day,
It is a good opportunity to send all the team to
Las-Vegas



' (

9 0 )
3 0 ) 3 4 0
! 4$ 3 )
4 # 0 4 )
5 63 #

) 4 +
:# # !
; < 8 7
7
& # !
9 # !
9 )5!4 4 6 #!)

) 07 4 4 #!) 3$
2 0 3$ # ) # #



9 4
? $ $ $ 4 $
$#

Event Processing

Event Event
Producer Consumer

Generate and Consume and
may publish react to either
perform operations on
events raw or complex
events
events

' 4 #
8
&=4 2& #4 $
# !4 #
# + 4
CEP
# # )# >
8 7



4 4 * 4
Getting the
right
information
in the right
granularity BAM
to the right Information Detect Quick observation into exceptional business
Dissemination
person at Decide behavior and notification to the appropriate
the right Respond people.
time

Diagnose Predictive Mitigate or
Active Diagnostics Processing
problems eliminate
based on predicted events
RTE
symptoms and
Reactions to events are done as part of business
resolve them
transactions – achieving low latency decisions,
and quick reaction to threats and opportunities



? )
&
4 ! !4 # 4 ?
# ## #! 8
# ) 4 ! Solution:
$ 2 8 # 8 0# 3
Configurable aggregation rules
# 4 # ) 4 4 #
@ •Thresholds by currency
•Thresholds by time
•Thresholds by
# 4#! # 4# 7 counterparty/client
# 4#! # 4# 4

Transactional Context Complex Event Processing Transactional Context

INPUT Filter and Persist CEP Routing, Output
Transactional Aggregation Database Transactional
JMS Input Queues Transform Orders to Aggregation,
Output Queues For
Temporal Logic Logic Update
Pub/Sub Logic Database execution

Aggregated Order Notification to CEP

Direct passthrough of selected messages


; & 4 8 2 9



% + #%
A
+
! #
4
4 )
7 , #)

Detect .-B
Decide
)
Respond

+
8 #4
4 0 .- ?
1$
32 ) #! #
4 4 ! # 4
4 4 # 4
> )7 # ) #)
# 7 +
4



& 9 +

# #4 9 +
,C
7 4 # #4
, 4 4 # ! 8
1-
#4 !
# .C
4
9 #4 # !
! 4
4 quot; 9 +
9; #4 7
! 8 )
4
9 ! 8
$ 4 ##
!
84 7 ,
4



9D 8
4 #4 , #



. #, #

. +
$ #$
#2 7
) 4 , # +
) )!4 7 ) ##
$ ! #
4 $ !# #
# $
) # ! #05 63$ * 4 D
&=4 $



+
! 4 #
7 8

# #
#4

&4 ;

9 !

!

4

. C
. C
? + !
#
F
.,
# +
9 ) ., H

# 4 # ..
E .-
, +
, # ! ( F
/
.,
9 +
H
? ; .G ..
+ .1
1 ;,? ./
? ? 1
? ? .C
+
G 9 4
E
.F
,-
.-
+ .E
; 44 ?

.H /



' 4
(



4

:

?4



#;

7 5 4 6 7
# # $ @

4 0 # 3 +

#
# <
<
4 22
4 <
<
9
9

! ###
#
!# #


IBM Research

Engineering Oriented Challenges

&4 #
# # 9
4 +
!
# 2 4

9
9

EPN
EPN
Event
EPA Event
Producer
EPA EPA Consumer
Pattern
Event Event
Producer Consumer
EPA EPA

Event Pattern Pattern Event
Producer Consumer

#

© 2008 IBM Corporation


?4 #; +

#
2 % #
? # #
?4 # 4

; ! #



: #;

4#2
+
& 2
4
# $
+
4 $
$
#!4
4#

4
quot; 4


Business User
View Point

# ! 4 ##

Why a decision is needed?
• Event Occurred Islands of
• Event Pattern has detected tools
• Fact has changed
• An explicit request
• Related to other decision

Implementation
Which information is needed
driven tools
In order to make this decision?
• Current state
• Past events
• Past states
• Future states and events. Gaps
In coverage
How the decision should be
Done?
• Inference from facts and rules
• Match patterns on event history Ability to manage
• Apply optimization tools Large complex
Decision network
Dependent
Decisions


#
)# # 4 +
+
) 0 3 EP
) 0 3$ Gaps
) # 0 3
+;4 0 & 3
+
I 7 44 0 3
8 7 0 & 3 BRMS
# # 09 ) 3 Analytics

Most Important
Connection
) 4 4 +
J # Business User Orientation:
J The user should be able to author,
J modify, understand, and audit individual
Computerized decisions, as well as the
J # Relationships among them



?4 ;

&

% #
94 #
:



9 4 ) & K



% quot;
&
&
?4 : ! )
'L 4
) 9 )

4 2

: ) #
quot;

&

#4
4#



9) M? 74 7 8
N &!
4
Stratum_1 Stratum_2 Stratum_S
Producer_1 Consumer_1

Agent_11 Agent_21 Agent_S1
Producer_2 Consumer_2

Agent_12 Agent_22 Agent_S2
. … .
. . .
. .
. . .
. .
. . .

Agent_1N1 Agent_2N2 Agent_SN2

Producer_P Consumer_C


IBM Research

RFID Store Scenario - 3 Stratum Levels
Stratum1 Stratum2 Stratum3
TagRead Filter
ProductAtCashier AtLeast5ProductsOf
Loc= “Cashier”
SameKindReturnedToShelf
ProductExitStore ProductLeftAtCashier
TagRead Filter
Tag Sequence Atleast
Loc = “Exit”
Tag
Reader
Tag
Reader TagRead
Tag Filter AtLeast10ProductsOf
Reader ProductInCart
Loc = “Cart” ProductReturnedToShelf
Reader SameKindSold
Sequence Atleast
Filter
TagRead Loc = “Shelf” ProductOnShelf

Levels Average Average Implementation Input events Derived
input events derived total events total
throughput events rate throughput rate
(event/s) (event/s)
Centralized 1,399 38
First stratum – 4 agents on 21,419 21,418
single node 3 Stratum Levels – second 21,419*4+69 4,485
stratum has the slowest 3*107+
Second stratum – 2 agents on 693 277 performance. Maximal flow 29,904 =
single node performance can be achieved 189,731
by combination of 1 node in the
Third stratum – 2 agents on 29,904 4,485 (upper limit)
third level, 107 nodes in the
single node
second layer, 4 nodes in the
Centralized – all the patterns 1,399 38 first level
are detected by multiple
112 nodes improvement 13562% 11802%
agents on same node (8
agents) Additional node improvement 121% 105.37%

© 2008 IBM Corporation


2 9 2quot; 4 +
9 # +
+
quot; 4 ; 4 + 4 991
4 ;
# 8

quot; 4 2; 4 +
991
4 # # 9
@

quot; 4 2 2
; 4 + 9
9
# 2
4
+
@



& 2O $
% quot;
&
4 4 # ) 7
! # 4 )# #
! & # )
2 !4 9! 4 4
# 4 ) # )
7 # 8 #7 # #
4 )



& 9

% quot;0
& # ,- /
-3
&!
9
) +9 ) 7 8
# ! 0# $ $
4 4
! $ # $ #@3
) ) +9 4 ) $
$ # $ 4 # #



+ ' 84
0K 4 7 9 3
9 4 # 4 # 4
) +

& 4 ) #
4 4

9 0 3 4 ! ! )=4
; 4 !! # #! 4
)
# 4 # #
4



9 I

quot; # # 4 4
quot; 4 # ! 7 8 7
7 8 # #
! quot; ! #) #
4



? ?4 # 7 ?O
I 4
# : K
4 2



! quot; # $ %
%
& #

& '
4 +P
4 ,- E
-
#!) # !4
0 7 3
4# 4# # 7 $ 4$
9 # M )
) ! # ) # # $
# M # ) # #
9 ) # 0!4 7 ! # #
4 3$ 8 4 # #
)

% !
# #



; 4



' # 7 # #
quot; & !
' ) 4 # #(
05 4 &4 9 #6$ ! * % 3
4# )! #
7 # 4# # )07 @
3$ ## # #4
7 #! 8 =4 ) #
)$ ! $ # )$ 4 )
# # # 4
0 ) ## 3+ Dquot; 2 # # ! #
! )
0 ) ## 3+ 9 &Q #
4 0 $4 )
= $
4 ) 3@ # ! #
4
! #



# # 4

quot; 8 # # # # !
7
) =4 # # #+
# # 7 )
#
! ) # # 0 ! !) ' 2
<' 3
2 4 # #0 4 # #3
# # #
' ! # !) 4 # #
0 )3 ##
# #



' ) +
) 4 !
0 7 7 4# #
R
3
0 ) $
3 5 6 4 )
8 ! 4 7 )@ #7 7 #
! 0 !4
4 #
# 7
# 44 3
' # S $ ! $ $
T #4 7 # !
4 $ #7 # # #


Event Processing - The Next Generation; March 2009

Recommended

Recommended

More Related Content

Similar to Event Processing - The Next Generation; March 2009

Similar to Event Processing - The Next Generation; March 2009 (7)

More from Opher Etzion

More from Opher Etzion (20)

Recently uploaded

Recently uploaded (20)

Event Processing - The Next Generation; March 2009