WSO2 Complex Event Processor (CEP) 4.0 provides real-time analytics capabilities. Some key features of CEP 4.0 include a re-architecture of the CEP server, improvements to the Siddhi query engine (version 3.0), scalable distributed processing using Apache Storm, and better high availability support. CEP 4.0 also offers a domain specific execution manager, real-time dashboards, improved usability, and pluggable event receivers and publishers. It can execute queries in a distributed manner across multiple nodes for scalability.

1. WSO2 Complex Event
Processor 4.0
Sriskandarajah Suhothayan
Technical Lead
Product Release Webinar
Lasantha Fernando
Senior Software Engineer

2. WSO2 Analytics Platform

3. WSO2 Analytics Platform

4. Data Processing Pipeline

5. Realtime Analytics

6. What is WSO2 CEP ?

7. New in CEP 4.0
o Re Architecture of the CEP Server
o Rewrite of Siddhi 3.0
o Scalable Distributed Processing with Apache Storm
o Better High Availability support
o Domain Specific Execution Manager
o Realtime Dashboard
o Improved usability

8. Event Flow of WSO2 CEP 4.0
CEP 3.x Input Adaptor + Event Builder = CEP 4.0 Event Receiver
CEP 3.x Output Adaptor + Event Formatter = CEP 4.0 Event Publisher

9. Receiving & Notifying Events

10. *Supports custom event receivers via its pluggable architecture!
Highly Pluggable Event Receiver Architecture

11. Highly Pluggable Event Publisher Architecture
*Supports custom event publishers via its pluggable architecture!

12. Execution of Queries

13. What’s new in Siddhi 3.0
o New architecture
o Optimized for better performance
o Improved Query API and Compiler (Antlr 4)
o Integration of Disruptor
o Better Extendability
o Annotation support
o Native support for distributed processing via Partition
Grouping
o Event Trigger Support

14. What’s new in Siddhi 3.0 ...
o Inbuilt Scripting support
o JavaScript, R & Scala
o Event Table with
o RDBMS & Data Analytics Server Tables
o New Window support
o Cron & Sorting
o Realtime Machine learning
o R, WSO2ML & PMML
o Rich set of Siddhi Extensions

15. define stream StockStream (symbol string, price double, volume int);
partition by (symbol of StockStream)
begin
from t1=StockStream,
t2=StockStream [(t2[last] is null and t1.price < price) or
(t2[last].price < price)]+
within 5 min
select t1.price as initialPrice, t2[last].price as finalPrice,t1.symbol
insert into IncreasingMyStockPriceStream
end;
Siddhi Query (Partition)

16. define table CardUserTable (name string, cardNum long) ;
@from(eventtable = 'rdbms' , datasource.name = ‘CardDataSource’ , table.name =
‘UserTable’, caching.algorithm’=‘LRU’)
define table CardUserTable (name string, cardNum long)
Cache types supported
o Basic: A size-based algorithm based on FIFO.
o LRU (Least Recently Used): The least recently used event is dropped
when cache is full.
o LFU (Least Frequently Used): The least frequently used event is dropped
when cache is full.
Siddhi Query (Event Table)
Supported for RDBMS, In-
Memory, Analytics Table

17. define function concatJ [JavaScript] return string {
var str1 = data[0];
var str2 = data[1];
var str3 = data[2];
var res = str1.concat(str2,str3);
return res;
};
from cseEventStream select price , concatJ(symbol,' ',price) as concatStr
group by volume insert into mailOutput;
Siddhi Query (Scripting)
Supported for Javascript,
Scala and R. Can be
extended for other languages

18. o Function extension
o Aggregator extension
o Window extension
o Stream Processor extension
define stream SalesStream (brand string, price double, currency string);
from SalesStream
select brand, custom:toUSD(price, currency) as priceInUSD
insert into OutputStream ;
Referred with namespaces
Siddhi Query (Extension) ...

19. Provides support for common functionalities across multiple domains out of the
box
o geo: Geographical processing
o nlp: Natural language Processing (with Stanford NLP)
o ml: Running machine learning models of WSO2 Machine Lerner
o pmml: Running PMML models learnt by R
o timeseries: Regression and time series
o math: Mathematical operations
o str: String operations
o regex: Regular expression
o ...
New Siddhi Extensions

20. Siddhi Try-it Tool
Allows users to
o Try out different queries
o Validate queries by sending events

21. Event Simulator
Allows users to test execution environment and
simulate complex execution scenarios
o Ability to send events to a stream without configuring
event receivers
o Send events
o Directly via UI
o Upload a file and send/replay events
o Point to a database table and play those events
o Configurable delay between events when sending via
file/database

22. Event Simulator

23. Ideal for business users, Form based UI to create/edit/delete execution logic
o Allows to create a predefined execution plan template to a domain
o Which can be managed via UI
Domain Specific Execution Manager

24. Event Flow
o Visualizes the event flow to provide an overall context

25. Scaling & HA

26. High Availability with WSO2 CEP 4.0

27. Scalability: Why not use existing CEPs ...
Spark Streaming
o Supports distributed processing
o Runs micro batches
o Does not support pattern & sequence detection
Apache Storm
o Supports distributed processing
o Stream processing engine

28. Scalability: Why not use Apache Storm ?
Advantages
o Supports distributed processing
o Supports Partitioning
o Extendable
o Opensource
Disadvantages
o Need to write Java code
o Need to start from basic principles ( & data structures )
o Adoption for change is slow
o No support to govern artifacts

29. WSO2 CEP 4.0 Scalability
Distributed Realtime = Siddhi +
Advantages over Apache Storm
o No need to write Java code (Supports SQL like query language)
o No need to start from basic principles (Supports high level
language)
o Adoption for change is fast
o Govern artifacts using Toolboxes
o etc ...

30. How we scale ?

31. Scaling with Storm
Handling Stateless
& Stateful Queries

32. Siddhi QL
define stream StockStream (symbol string, volume int, price double);
@name(‘Filter Query’)
from StockStream[price > 75]
select *
insert into HighPriceStockStream ;
@name(‘Window Query’)
from HighPriceStockStream#window.time(10 min)
select symbol, sum(volume) as sumVolume
insert into ResultStockStream ;

33. Siddhi QL - with partition
define stream StockStream (symbol string, volume int, price double);
@name(‘Filter Query’)
from StockStream[price > 75]
select *
insert into HighPriceStockStream ;
@name(‘Window Query’)
partition with (symbol of HighPriceStockStream)
begin
from HighPriceStockStream#window.time(10 min)
select symbol, sum(volume) as sumVolume
insert into ResultStockStream ;
end;

34. Siddhi QL - distributed
define stream StockStream (symbol string, volume int, price double);
@name(Filter Query’)
@dist(parallel= ‘3')
from StockStream[price > 75]
select *
insert into HightPriceStockStream ;
@name(‘Window Query’)
@dist(parallel= ‘2')
partition with (symbol of HighPriceStockStream)
begin
from HighPriceStockStream#window.time(10 min)
select symbol, sum(volume) as sumVolume
insert into ResultStockStream ;
end;

35. Distributed Execution on Storm UI

36. Realtime Dashboard

37. Realtime Dashboard
Provides an overall idea on
what's going on
o Dashboard using
o Google Gadget
o HTML5 + javascripts
o Support custom gadgets and
dashboards
o Gadget generation with
o D3 and Vega

38. Gadget Generation Tool
Customize with few clicks …
Support realtime and
stored data visualization
How ?
○ Start with data in tabular format
○ Select chart type (Bar, Line, Map, ...)
○ Map columns to dimensions (X Y, color, point size, … )
○ Also do drill downs

39. Geo Dashboard
With configurable alerting &
Monitoring capabilities from UI.
http://wso2.com/library/articles/2015/01/article-geo-spatial-data-analysis-using-wso2-
complex-event-processor-0/

40. Demo

41. Questions?

42. More Information
o WSO2 CEP 4.0.0 download -
http://wso2.com/products/complex-event-processor/
o Product documentation -
https://docs.wso2.
com/display/CEP400/WSO2+Complex+Event+Processor+Documentation
o WSO2 Analytics Platform -
http://wso2.com/analytics
o Quick Start Guide doc - https://docs.wso2.
com/display/CEP400/Quick+Start+Guide
o Quick Start Guide video - https://www.youtube.com/playlist?
list=PLp0TUr0bmhX5v5yBa0TINUktuF-_eDPC_
o Pre Release Webinar on CEP 4.0 -
http://wso2.com/library/webinars/2015/07/real-time-analytics-from-data-to-actions-
in-milliseconds/

43. Contact us !