Event Stream Intelligence: Esper & NEsper
Esper is a component for complex event processing (CEP), available for Java as Esper, and for .NET as NEsper.
Esper and NEsper enable rapid development of applications that process large volumes of incoming messages or events, regardless of whether incoming messages are historical or real-time in nature. Esper and NEsper filter and analyze events in various ways, and respond to conditions of interest with minimal latency.
Esper and Event Processing Language (EPL) provide a highly scalable, memory-efficient, in-memory computing, SQL-standard, minimal latency, real-time streaming Big Data processing engine for medium to high-velocity and high-variety data.
New - Cloud Application for EPL Learning: Esper EPL Online
Complex event processing (CEP) delivers high-speed processing of many events across all the layers of an organization, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time (source:Wikipedia).
Esper offers a Domain Specific Language (DSL) for processing events. The Event Processing Language (EPL) is a declarative language for dealing with high frequency time-based event data.
Some typical examples of applications are:
- Business process management and automation (process monitoring, BAM, reporting exceptions, operational intelligence)
- Finance (algorithmic trading, fraud detection, risk management)
- Network and application monitoring (intrusion detection, SLA monitoring)
- Sensor network applications (RFID reading, scheduling and control of fabrication lines, air traffic)
- Highly scalable, elastic, distributable and fault tolerant event processing
- Rich web-based multi-window GUI based on Apache Flex for design and management of EPL statements and CEP engine in general
- Real-time continuously-updating displays (charts, timeline, gauge, grids) that are interactive, composable, configurable and easy to integrate
- Data distribution service for efficient 3-tier data fan-out from CEP engine(s) to browsers
- Hot deployment of EPL modules and event-driven applications
- Integration with distributed caches: Oracle Coherence, VMware GemFire, GigaSpaces XAP, Hazelcast, Infinispan, Terracotta, IBM WebSphere eXtreme Scale
- Info and Download and Screenshots
- Resiliency of CEP engine state and EPL statement state, as needed; High write performance and fast recovery
- Info and Download
- Sliding windows: time, length, sorted, ranked, accumulating, time-ordering, externally-timed (value-based windowing), expiry-expression-based with aggregations
- Tumbling windows: time, length and multi-policy; first-event; expiry-expression-based with aggregations
- Combine windows with intersection and union semantics. Partitioned windows. Dynamically shrinking or expanding windows. Expiry-expression-driven windows.
- Grouping, aggregation, sorting, filtering, merging, splitting or duplicating of event streams
- Familiar SQL-standard-based continuous query language using insert into, select, from, where, group-by, having, order-by, limit and distinct clauses
- Inner-joins and outer joins (left, right, full) of an unlimited number of streams or windows
- Subqueries including exists and in
- Output rate limiting and stabilizing, snapshot output, expression support
- Named windows
- Explicit sharing of data windows between statements
- Multiple and custom entry and exit criteria for events
- Support for predefined query execution optimized by indexed access, via on-select and on-demand (fire-and-forget) query and explicit indexes (hash and btree)
- Support for update-insert-delete (aka. merge or upsert) in a single atomic operation
- Enumeration methods, date-time methods, Allan's interval algebra with support for point-in-time events and events with duration, lambda-expressions, expression definition and reuse, 'new' operator for returning multiple results from an expression
- Declarative context dimensions based on consistent-hashes, partitions, categories, overlapping or non-overlapping initiated by event arrival and patterns, with correlated termination; nested context dimensions; Context partition targeting for runtime queries and iteration. Achieves data parallelism and component parallelism.
- High degree of parallelization processing the same statement and processing multiple statements. Stateless statements process lock-free.
- Logical and temporal event correlation
- Crontab-like timer 'at' operator
- Lifecycle of pattern can be controlled by timer and via operators, repeat-number and repeat-until, every-distinct, while
- Pattern-matched events provided to listeners
- Support for match-recognize which is a proposed SQL standard for row pattern matching based on regular expressions
- Supports event-type inheritance and polymorphism as provided by the Java language, for Java object events as well as for Map-type and object-array type events
- Events can be plain Java objects, XML (DOM and streaming through Apache Axiom), object-array (Object) and java.util.Map including nested objects and hierarchical maps
- Event properties can be simple, indexed, mapped or nested - allows querying of deep Java object graphs and XML structures
- Dynamic properties allow dynamic typing of properties, supported by cast, instanceof and exists functions
- Applications can plug-in their own event representation and dynamic type resolution
- Insert-into can directly populate a plain or legacy Java object or Map from a select-clause
- Support for a create-schema syntax, useful to declare event types from a column-and-type list, from existing classes or from other types
- Module parsing and deployment API
- A module files is a plain text file with EPL statements
- Mature API with few significant changes since version 1.0
- Support for both the listener (push/subscription) API and the consumer (pull/receive) API for querying results
- Control event visibility and the concept of timer on a statement level via isolated service provider
- Suspend statements: take statements offline and online without losing state
- Load statements from historical data and merge into real-time streams
- We provide a test framework for unit or regression testing EPL-based applications
- Precompile a statement with substitution parameters and efficiently start the parameterized statement multiple times
- A set of classes providing an object-oriented representation of a statement
- Full and complete specification of a statement via object model
- Round-trip from object model to statement text and back to object model
- Build, change or interrogate statements beyond the textual representation
- CSV input adapter reads comma-separated value formats
- simulate multiple event streams with timed, coordinated playback via timestamp column
- load generator
- preloading of reference data
- JMS input and output adapter based on Spring JMS templates
- DB output adapter for running DML and for keyed update-insert (aka. upsert)
- HTTP input+output adapter
- Socket input adapter
- Executes a large number of simultaneous queries - Esper's query processor can run thousands of queries continuously and simultaneously on a single instance
- Relational database access via SQL-query joins with event streams
- LRU (least-recently used) and expiry-time query result caches
- Keyed cache entries for fast cache lookup
- Engine indexes cached rows for fast filtering within a large number of SQL-query result rows
- Multiple SQL-queries in one statement transparently integrates multiple autonomous database systems
- Expression declarations - expressions that are common to multiple places in an EPL statement can be moved to a named expression declaration.
- Joins to method invocation results allows easy integration with distributed caches, web services and object-oriented databases
- On-demand queries are fire-and-forget EPL queries against named windows for non-continuous query execution.
- Support for "create index" for on-demand query execution against large named windows
- Data Flow Programming
- Variables and Constants can occur in any expression and can dynamically control output rate
- Guarantees of consistency and atomicity of variable updates
- Scalar, un-typed and event-typed variables with support for nested property read/write
- Constants ensure query optimizations for constant values are possible for variables that never change
- Variant event streams allows treating disparate types of events as the same type, such as when the event type can only be known at runtime, when the event type is expected to vary, or when optional properties are desired.
- Concurrency-safe iterator provides complete query capability for all statements, with read-write locking
- Supports externally-provided time as well as Java system time, allowing applications full control over the concept of time within an engine and full control over which thread(s) evaluate timer schedule for statements
- Multithread-safe as of release 1.5
- Multithreaded sends of events into an engine
- Create, start and stop statements during operation without adversely impacting performance
- Applications can retain full control over threading; Inbound, outbound and execution threading configurable
- Efficiently sharing resources between statements and low thread blocking
- Supports multiple independent Esper engines per JavaVM
- Pluggable architecture for event pattern and event stream analysis via user-defined functions, plug-in views, plug-in aggregation functions, plug-in pattern guards and plug-in pattern event observers and event instance methods. Virtual data windows for transparently backing named windows with an external store
- Performance-minded design: query strategy analysis and index building; array-based collections; delta networks and many other techniques
- Benchmark kit available for download; Performance testing results and tips see page in menu
- Unmatched event listener receives a callback if an event does not match any started statement
- Support for all Java data types including BigInteger and BigDecimal
- Contained-Event select syntax for easy handling of coarse-grained, business-level events that themselves contain events or that need to be broken down into rows.
- JSON and XML output event rendering without syntax to learn
- Annotations allow adding information to EPL statements specific to your application, allow debugging and allow tuning memory use and performance
- Assign a priority to your statement; Identify preemptive statements that cause an event to drop in face of multiple statements matching filters.
- Declarative update of event properties via on update for named windows and update istream of events entering a stream.
- Support for versioned events that update, provide a new version or that revise an existing event held by an engine.
- Support for MVEL and JSR 223 scripts
- Both non-consuming and consuming operation (consuming deletes/drops an event after it is considered)
- Actions can be Java methods and can also be declarative through the use of data flows such as for event I/O
- J2EE and non-J2EE, from many different domains
- Java Messaging Service (JMS) server shell demonstrates a multi-threaded JMS server with dynamic statement management using Java Management Extensions (JMX)
What these applications have in common is the requirement to process events (or messages) in real-time or near real-time. This is sometimes referred to as complex event processing (CEP) and event stream analysis.
Key considerations for these types of applications are the complexity of the logic required, throughput and latency.
- Complex computations - applications that detect patterns among events (event correlation), filter events, aggregate time or length windows of events, join event streams, trigger based on absence of events etc.
- High throughput - applications that process large volumes of messages (between 1,000 to 100k messages per second)
- Low latency - applications that react in real-time to conditions that occur (from a few milliseconds to a few seconds)
- Esper requires a Java Virtual Machine version 5.0 runtime, or above.
- Esper will not work with JavaVM versions 1.4.2 or below.
- NEsper is the .NET version of the Java Esper, check the release section for feature information
Esper and NEsper were designed to make it easier to build CEP and ESP applications. Esper and NEsper are open-source software available under the GNU General Public License (GPL) license.
YourKit is kindly supporting open source projects with its full-featured Java Profiler. YourKit, LLC is creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products: YourKit Java Profiler and YourKit .NET Profiler.
About Esper and NEsperNews
About Esper for Java
About NEsper for .NET
License and Trademark Use
Articles and Presentations
Short Case Study
Longer Case Study
DEBS 2011 Challenge
How To Contribute
Last Published: Apr 09, 2013