Complex Event Processors are capable of handling large volumes of events - by filtering, aggregating or detecting patterns. Java Applications use a CEP to pre-process incoming signals. These applications can also generate the events - for example the user click and navigation behavior in the web application - and report them to the CEP. The web application can subsequently utilize the outcomes from the CEP to for example intelligently guide the user or present relevant details. This session will show various ways in which a CQL based CEP can be integrated into a Java application to enhance the web application's behavior. The intended audience for this presentation consists of experience Web Java and Enterprise Java developers. - introduction to Complex Event Processing - demonstration of CQL event processing on events arriving on JMS - discussion of how the web application can absorb the CEP results - demonstration of a simple Web Shop application that published events to the CEP and utilizes the CEP results - discussion on when and how CEP can add value to Java applications

1. Why and How to Engage a Complex Event Processor from a Java Web Application Lucas Jellema CTO, AMIS (The Netherlands) Oracle ACE Director

2. An Event A fact When it becomes a fact Timestamp! Of potential interest to anyone With some information associated with it When the fact is changed, that too may constitute an event Note that even when no change occurs, that could be considered an event (absence of expected event)

3. The Data Avalanche The league of real time events Continuous stream of a multitude of tiny events with hardly any payload, most of which are not interesting at all Sent from physical sensors (temperature, pressure, RFID, security gates), process sensors, Twitter, manufacturing equipment, database triggers, web servers, ESBs, stock trade tickers, sport statistics, RSS, network switches, …

4. Most of these events….

5. Making Sense of Non-senseComplex Event Processing Useful information could be hidden in those streams of events – and CEP extracts it By detecting patterns, deviations, aggregations in these event streams CEP publishes ‘meaningful’ (business) events to report its findings – by executing some Event Processing Langugae events Complex Event Processor

6. Continuous Query Language (CQL) Standard extension/complement to SQL For handling large volumes of continuously arriving events Executing the queries continuously Operates on streams of continually arriving events instead of static relational data sets Events can be enriched with context Results keep being produced as event stream Is used for comparing events of same type, to find Trends and Patterns Exceptions Aggregates Complex Event Processor CQL

7. Example CQL Query Report the average temperature Aggregated over last 60 seconds Reported every 30 seconds Determined per Cluster Temperaturereadings Complex Event Processor Average temperature values (over 60s, per cluster, every 30s)

8. Event Processing Network Adapters – consuming or producing events Channels – carrying specific event types Processors – running CQL

9. Complex Event Processing Alert JMS Dashboard SOA Application DB Data Warehouse WebService File events Complex Event Processor JMS HTTP JMX File DB Event Consumer

10. events events events Java Applications interacting with a complex event processor Complex Event Processor Java application JMS 1 3 2

11. Various CEP offerings available Progress Software (Apama) Aleri (Aleri and Coral8) IBM (WebSphere Business Events, System S/InfoSphere Streams, ILog rule engine) StreamBase Systems, TIBCO Software (Tibco Business Events) Oracle (CEP - fka BEA WLEvs - and Glassfish OpenESB IEP) UC4 (Senactive) EsperTech JBoss Drools Fusion

12. Oracle CEP 11g Architecturefka BEA WLEVS(erver) Design Time Run Time CEP Visualizer(web console) Other WebApp Eclipse http CEP plugin Web (Bayeux)Server CEP Server DB JMS

13. Demo: HelloWorld

14. The EPN source side of things

15. Java Application as Event Consumer events Complex Event Processor Java application JMS

16. Temperature Vigil CEP

17. Case of Temperature Hospital Temperature sensors Cooling units Maternity Ward and Intensive Care units Fire detection Frequent readings (subsecond) Monitoring for: trends of threshold crossing, fire!, fault detection

18. Demo: Temperature Sensors Application should display a dashboard for temperature sensor readings The user is interested in cluster aggregates , deviations from pre-set values and broken sensors How would a Java Application process thousands of events per second extract and report aggregates detect and report missing values Note: user wants real time updates – but not more frequently than once per minute

19. Technical Layout Temperature Sensors are all over the place Clustered per room or location Temperature Readings arrive over JMS Results are sent through JMS to dashboard JMS queue for temperature aggregates Per cluster, once every 30 seconds JMS queue for failed sensor detection

20. Demo of Temperature Aggregations

21. Event Processor for aggregation Group by Cluster Report once every 30 seconds Over all events in the last 60 seconds

23. Sensor Failure When a sensor has not reported for 30 seconds, we consider it failed And potentially broken down CEP needs to trap the missing events and report those

24. Demo of Sensor Failure detection

26. Feeding Temperature Aggregate Findings into real time BAM CEP publishes result events on JMS CEP is not aware of Business Activity Monitoring BAM creates an enterprise message source Based on JMS and MapMessages Feeding into a Data Object in the Active Data Cache BAM listens on the JMS Queue and maps the messages to the ADC Data Object

27. Business Activity Monitoring

28. Java Application as Event Producer events Complex Event Processor Java application JMS 3

29. Java Application producing the events to process The user interaction with an application produces many events Navigation, Selection, Submission (scroll, expand/collapse, drill down) Most events are not recorded persistently The events can reveal interesting information about the application, the data and the users For example in a web shop The information could be used to improve the effectiveness of the application

31. Super Mario

32. Super Mario Galaxy

33. Mario Super Sluggers

34. Super Mario Bros.Search Event ‘mario’ View Event‘Mario Kart’ Buy Event‘Mario Kart’ User interaction events in a WebShop

35. The WebShop Application

36. Derive intelligence from WebShop events Actual Metrics: #searches in last 5 minutes, #buyers in the last 24 hours, #search-to-views drilldowns Top 10 search terms (in last [24] hour) Top 10 useless search terms After search on term there is no view event Most Effective search terms Search is followed by (one or more) view events and subsequently a buy event Items frequently viewed but not often bought #buy/#views is ranks low

37. WebShop & CEP WebShop sends JMS events to single queue Whenever a user in a session searches, drills down or buys Type of event, search term or item identifier, session id CEP reads from JMS channel Routes based on event type Processes events using CQL processor Emits derived events to JMS queue WebShop listens to JMS queue Updates application scope ‘top10’ beans Provides dashboard based on ‘top10’ beans Potentially using Server Push (or periodic poll)

38. Webshop feeding into CEP and receiving refined feedback WebShop Top 10 Popular Search Terms Top 10 Useless Search Terms Most Effective Search Terms search view buy

39. Popular useless search terms(missing viewevent following the search)

40. Real Time Event Driven Dashboard in WebShop application

41. Summary Complex Event Processing For analysis of real time, high volume, continuous streams of fine grained, non-persistent events Used for detecting patterns, spotting exceptions, calculating aggregates and tracking trends CEP uses CQL queries to derive information from event streams – continuously, as events continue to arrive Results are published as (higher level) events – often to JMS queues

42. Summary [Java] Applications can have CEP pre-process events to do complex analysis and filtering producing business events that applications can understand Java Applications produce events that CEP can process And possibly feed back into the applications JMS and http are primary communication channels

43. Resources Presentation and demos are on our blog http://technology.amis.nl/blog Contact me at:lucas.jellema@amis.nl Note: Chapter 19 in Oracle SOA Suite 11gHandbook describesOracle CEP and BAM