Complex Event Processing with Esper
•
19 likes•4,993 views
Nathan Reese's presentation to DOSUG on Esper's complex event processing engine.
![Complex Event Processing with Esper Turn your database upside down DB Nathan Reese [email_address]](https://image.slidesharecdn.com/cepwithesper-100612193323-phpapp01/85/Complex-Event-Processing-with-Esper-1-320.jpg)
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Complex Event Processing with Esper
Similar to Complex Event Processing with Esper (20)
More from Matthew McCullough
More from Matthew McCullough (20)
Recently uploaded
Recently uploaded (20)
Complex Event Processing with Esper
- 1. Complex Event Processing with Esper Turn your database upside down DB Nathan Reese [email_address]
- 2. “ Information processing using computer systems on a global scale has become the foundation of the twenty-first-century” David Luckham – The Power of Events
- 5. horizontal/vertical causality detect patterns filter aggregate correlate
- 6. Event Driven Architecture (EDA) Complex Event Processing (CEP) Event Stream Processing (ESP)
- 7. Store Now Query Later
- 11. Event Streams App1 AppX … CEP
- 12. Event Streams App1 AppX … CEP CEP CEP Derived Event Streams
- 14. Esper Engine Open Source SOA – Jeff Davis EPServiceProvider Event Objects EPL Statements Listeners Subscribers Configuration JDBC Adapter
- 16. select * from Withdrawal.win:length(5) Esper Reference Documentation Ch. 3
- 17. select * from Withdrawal(amount>=200).win:length(5) Esper Reference Documentation Ch. 3
- 18. select * from Withdrawal.win:length(5) where amount >= 200 Esper Reference Documentation Ch. 3
Editor's Notes
- Distributed systems span across the boundaries between enterprises Examples Financial trading systems Same structure: dispersed set of applications communicating with one another by messages transmitted over an IT layer
- Distributed Information Systems aka global communications spaghetti pot Activities are driven by events flowing through IT layers No technology to enable humans to understand what is going on Tools to see events No tools to make sense of events
- Need tools to unravel the communication spaghetti, to be able to understand it at a human level and derive instant insights. Need to be able to determine the following from the event cloud
- New suite of technologies have been created to answer these questions
- A good way to visualize ESP and CEP is to think of an upside down database. A traditional databases or distributed caches are passive data structures that create frozen and slow assets which require explicit querying to make sense of.
- DB model turned upside down. Store your queries and continuously flow data over queries, constantly detecting patterns among events (event correlation), filtering events, aggregate time or length windows of events, join event streams, trigger based on absence of events
- Real time data mining: Query results returned in real time when conditions occur that match queries Continuous execution model: In CEP, the event stream is constantly being examined for query matches and patterns. Database only searches while a query is running. If your application needs data 10 times per second it must query 10 times per second. Disparate data streams: CEP can have any/all enterprise data streams flowing over queries. Break free from data silos Tiny footprint: Do not need large hardware investment. Benchmark tests running 1000 to 100000 messages per second on a laptop.
- Not meant to replace traditional databases, which will always be needed for historical data and adhoc queries. This is meant as another cog in the wheel that will greatly reduce the burden placed on databases while provided much needed real time capabilities.
- CEP nodes can placed into a hierarchy to model target system’s logical layers
- Esper engine is an Open Source java CEP library that makes building CEP applications possible Esper is a lightweight kernel written in Java which is fully embeddable into any Java process, JEE application server or Java-based Enterprise Service Bus. It enables rapid development of applications that process large volumes of incoming messages or events. Esper is actively developed with quarterly releases
- The EPServiceProvider interface represents an engine instance. Each instance of an Esper engine is completely independent of other engine instances and has its own administrative and runtime interface. Create events Set up query statements Specify Engine result consumers (Listener or subscriber) Esper configuration options
- The Event Processing Language (EPL) is a SQL-like language. Streams (and views) replace tables as the source of data with events replacing rows as the basic unit of data. Views represent the different operations needed to structure data in an event stream and to derive data from an event stream Views define the data available for querying and filtering. Some views represent windows over a stream of events. Other views derive statistics from event properties, group events or handle unique event property values. Lots of built in views
- Diagram taken from “Esper Reference Documentation” version 3.4 produced by EsperTech
- Diagram taken from “Esper Reference Documentation” version 3.4 produced by EsperTech
- Diagram taken from “Esper Reference Documentation” version 3.4 produced by EsperTech
- Resources and further reading