Enterprise systems such as enterprise resources planning (EPR), system monitoring, and stock transaction and financial transaction systems generate an enormous amount of events that can contain useful data; processing these data efficiently can help to gain a competitive advantage. Implementing custom solutions to capture, analyze, and act on these Big Data imposes challenges and operational complexities. WSO2 CEP and WSO2 BAM together provide a solution to deliver a low-latency, high-volume, and scalable environment enabling data collection, real-time as well as batch analysis and firing notifications of multiple types across numerous endpoints.

2. Conceptual Architecture - BigData

3. ●
●
CEP Is & Is NOT!

4. Realizing Real-time Analytics
●
●
●

5. WSO2 CEP Big Picture

6. WSO2 CEP Big Picture (contd..)

7. ●
●
●
●
●
●
●
What’s WSO2 CEP

8. WSO2 CEP Architecture

9. WSO2 CEP Artifacts

10. Data Collection through Adaptors
● Can receive events via
SOAP, HTTP, JMS, ..
● WSO2 Events is highly
optimized version
(400K events TPS)
● Default Agents and
you can write custom
agents.
Agent agent = new Agent(agentConfiguration);
publisher = new AsyncDataPublisher(
"tcp://localhost:7612", .. );
StreamDefinition definition =
new StreamDefinition(STREAM_NAME,
VERSION);
definition.addPayloadData("sid", STRING);
...
publisher.addStreamDefinition(definition);
...
Event event = new Event();
event.setPayloadData(eventData);
publisher.publish(STREAM_NAME, VERSION, event);

11. Monitoring & Debugging (Event Tracer & Event St
●

12. Monitoring (Event Tracer & Event St

13. Realtime Dashboard

14. Getting insights of Event Streams
- writing queries in Siddhi

15. ●
●
●
●
●
●
●
Siddhi Supports,

16. Filter
…

17. Window
from <input-stream-name> [<conditions>]?
#window.<window-name>(<parameters>)
select <attributes>
insert into <output-stream-name>

18. Join
from <stream>#<window> [unidirectional] join
<stream>#<window>
on <condition> within <time>
select <attributes>
insert into <stream>

19. Pattern
a1 x1 k5 a2 n7y1
from [every] <condition> -> [every] <condition> … <condition>
within <time>
select <attributes>
insert into StockQuote

20. Sequence
from <event-regular-expression> within <time>
select <attributes>
insert into <stream>
a1 b1 b1 b2 n7y1

21. Partition
define <partition-id> by <partition-type> (,<partition-type>)*

22. Event Table
define table <table-name> ( {<attribute-name> <type>}*)
( from <db-configs>)?

23. High availability and Scalability

24. High Availability
High Availability (Recommended approach)

25. Scaling

26. Performance Stats - Siddhi

27. DEBS Grand Challenge 2013

28. Integration with WSO2 BAM

29. Demo

30. Scenario
MyPizzaShop – On time delivery or free Pizza Offer !!!

31. Scenario
• Pizza Order : XML Event over http

32. Scenario
Inputs

33. Scenario
Outputs

34. Scenario
Outputs

35. Why WSO2 CEP ?
●
●
●
●

36. WSO2 Business Activity Monitor

37. Outline
● Introduction to Business Activity Monitoring
● WSO2 Business Activity Monitor
● WSO2 BAM Components
● WSO2 BAM Toolboxes
● WSO2 BAM High Availability Distributed Deployment
● Demo

38. Business Activity Monitoring
“The aggregation, analysis, and presentation of real-time
information about activities inside organizations and
involving customers and partners” - Gartner

39. Aggregation
● Capturing data
● Data storage
● What data to capture?

40. Analysis
● Data operations
● Building KPIs
● Operate on large amounts of historic data or new data
● Building BI

41. Presentation
● Visualizing KPIs/BI
● Custom Dashboards
● Visualization tools
● Not just dashboards!

42. WSO2 Business Activity Monitor
● Fully-open source solution for aggregating and analyzing data
and presenting information about business activities.
● Cloud-enabled, lightweight, developer-friendly and
easy-to-deploy
● High performance data capture framework
● Elastically-scalable data analysis powered by Apache Hadoop
● Pre-built Data Agents for WSO2 products

43. Architecture

44. Data Agents

45. Data Agents
● Compatible with CEP/BAM
● Get data across to BAM
− Service monitoring feature – WSO2 AS, DSS, ESB, API Manager
− Mediation monitoring feature – BAM Mediator for WSO2 ESB
− Custom data-agents
● Asynchronous & non-blocking
● Thrift for high performance message throughput

46. Custom Agents
● Data agents are meant to be installed/plugged into the server
to be monitored
● Through Thrift, different languages can be supported

47. Data Receiver

48. Data Receiver
● Receives data and stores it in Cassandra
○ Scalable, big data repository
● Asynchronous & non-blocking
○ Combination of Cassandra, Thrift and the non-blocking nature results in
extremely fast writes
● Shared with WSO2 CEP for real time analysis
● Supports Thrift & REST API

49. Data Model
● Data is sent over using strongly typed Data Streams
{
'name':'phone.retail.shop',
'version':'1.0.0',
'nickName': 'Phone_Retail_Shop',
'description': 'Phone Sales',
'metaData':[
{'name':'clientType','type':'STRING'}
],
'payloadData':[
{'name':'brand','type':'STRING'},
{'name':'quantity','type':'INT'},
{'name':'total','type':'INT'},
{'name':'user','type':'STRING'}
]
}
● Data Streams are versioned
○ Allows for easier analysis

50. The Analyzer Engine

51. The Analyzer Engine
● Powered by Apache Hadoop with querying/managing through
Apache Hive
● Data transfer to and from data sources is handled through
custom storage handlers
● Parallel, distributed processing through the MapReduce
programming model

52. The Analyzer Engine
● Runs on local Hadoop node or delegates to Hadoop cluster
○ Scalable analytics
○ Cluster can range from a couple of nodes to 1000s
● Analysis is carried out based on analytics scripts
● Scripts are based on an easy-to-learn, SQL-like query language
INSERT OVERWRITE TABLE UserTable SELECT userName, COUNT(DISTINCT
orderID),SUM(quantity) FROM PhoneSalesTable WHERE version= "1.0.0" GROUP
BY userName;

53. The Analyzer Engine
● Scripts can be scheduled
○ Ex: once a minute, every Wednesday at 4:15 p.m., every 30th
at 12 midnight
● Polyglot data architectures are supported
○ Write summarised information to any kind of datastore
○ Custom written Hive JDBC handler used by default for output to relational DBs

54. The Presentation Layer

55. The Presentation Layer
● Gadget Portal for the dashboard
● Gadget Generation Wizard – Tool for generating custom
gadgets for the dashboard
● Activity Dashboard for correlating activities
● Message Console
● Plug in your own report server/ dashboard server

56. The Presentation Layer
BAM Dashboards

57. The Presentation Layer
Gadget Portal

58. BAM Toolboxes
● BAM Toolboxes are installable and hot deployable artefacts
used for deploying functionalities to a BAM server
○ Stream definitions
○ Analytics scripts
○ Dashboards
● Supports plugging in of one or more of the above features
● Toolboxes for monitoring and auditing most WSO2 products are
available OOTB
● Toolboxes for custom scenarios can be created easily

59. HA Distributed BAM deployment
● WSO2 BAM can be clustered and deployed in a distributed
manner to enable high-availability, fail-over scenarios
○ Partially distributed deployment
■ Cassandra (storage) and Hadoop (analyzer) nodes are clustered
○ Fully distributed deployment
■ All components of BAM are clustered (data receiver, storage, analyzer and
presentation)
● Hazelcast in-memory data grids are used for clustering
implementation

60. Partially Distributed BAM deployment

61. Fully Distributed BAM deployment

62. Demo

63. …