Home Contact

Event Stream Intelligence: Esper & NEsper

Where Complex Event Processing meets Open Source: Esper & NEsper
Make no compromise on enterprise readiness: discover EsperHA, EsperJDBC and our support services. Embed mainstream CEP in your products and deployments: Learn about our licensing options.

Event Stream Intelligence with Esper and NEsper

Esper is a component for CEP and ESP applications, available for Java as Esper, and for .NET as NEsper.

Esper and NEsper enable rapid development of applications that process large volumes of incoming messages or events. Esper and NEsper filter and analyze events in various ways, and respond to conditions of interest in real-time.

Technology Introduction

Complex Event Processing, or CEP, is technology to process events and discover complex patterns among multiple streams of event data. ESP stands for Event Stream Processing and deals with the task of processing multiple streams of event data with the goal of identifying the meaningful events within those streams, and deriving meaningful information from them.

The Esper engine has been developed to address the requirements of applications that analyze and react to events. Some typical examples of applications are:

  • Business process management and automation (process monitoring, BAM, reporting exceptions, operational intelligence)
  • Finance (algorithmic trading, fraud detection, risk management)
  • Network and application monitoring (intrusion detection, SLA monitoring)
  • Sensor network applications (RFID reading, scheduling and control of fabrication lines, air traffic)

Features

Event Stream Processing

  • Sliding windows: time, length, sorted, accumulating, time-ordering, externally-timed (value-based windowing)
  • Tumbling windows: time, length and multi-policy
  • Grouping, aggregation, sorting, filtering and merging of event streams
  • Tailored SQL-like query language using insert into, select, from, where, group-by, having and order-by clauses
  • Inner-joins and outer joins (left, right, full) of an unlimited number of streams or windows
  • Subqueries including exists and in
  • Output rate limiting and stabilizing, snapshot output
  • Named windows allow sharing data windows between statements; Define multiple explicit entry and exit criteria for events

Event Pattern Matching

  • Logical and temporal event correlation
  • Crontab-like timer 'at' operator
  • Lifecycle of pattern can be controlled by timer and via operators
  • Pattern-matched events provided to listeners

Event Representations

  • Supports event-type inheritance and polymorphism as provided by the Java language
  • Events can be plain Java objects, XML (DOM and streaming through Apache Axiom) and java.util.Map including nested objects and hierarchical maps
  • Event properties can be simple, indexed, mapped or nested - allows querying of deep Java object graphs and XML structures
  • Dynamic properties allow dynamic typing of properties, supported by cast, instanceof and exists functions
  • Applications can plug-in their own event representation and dynamic type resolution

Prepared statements and substitution parameters

  • Precompile a statement with substitution parameters and efficiently start the parameterized statement multiple times

Statement Object Model

  • A set of classes providing an object-oriented representation of a statement
  • Full and complete specification of a statement via object model
  • Round-trip from object model to statement text and back to object model
  • Build, change or interrogate statements beyond the textual representation

Input Adapters

  • CSV input adapter reads comma-separated value formats
    • simulate multiple event streams with timed, coordinated playback via timestamp column
    • load generator
    • preloading of reference data
  • JMS input and output adapter based on Spring JMS templates

Other

  • Executes a large number of simultaneous queries - Esper's query processor can run thousands of queries continuously and simultaneously on a single instance
  • Relational database access via SQL-query joins with event streams
    • LRU (least-recently used) and expiry-time query result caches
    • Keyed cache entries for fast cache lookup
    • Engine indexes cached rows for fast filtering within a large number of SQL-query result rows
  • Joins to method invocation results allows easy integration with distributed caches, web services and object-oriented databases
  • Variables can occur in any expression and can dynamically control output rate
    • Guarantees of consistency and atomicity of variable updates
  • Variant event streams allows treating disparate types of events as the same type, such as when the event type can only be known at runtime, when the event type is expected to vary, or when optional properties are desired.
  • Support for update events that update, provide a new version or that revise an existing event held by an engine.
  • Support for both the listener (push/subscription) API and the consumer (pull/receive) API for querying results
  • Concurrency-safe iterator provides complete query capability for all statements
  • Supports externally-provided time as well as Java system time, allowing applications full control over the concept of time within an engine
  • Multithread-safe as of release 1.5
    • Multithreaded sends of events into an engine
    • Create, start and stop statements during operation without adversely impacting performance
    • Applications can retain full control over threading
    • Efficiently sharing resources between statements and low thread blocking
  • Supports multiple independent Esper engines per JavaVM
  • Pluggable architecture for event pattern and event stream analysis via user-defined functions, plug-in views, plug-in aggregation functions, plug-in pattern guards and plug-in pattern event observers, event instance methods
  • Performance-minded design: query strategy analysis and index building; array-based collections; delta networks and many other techniques
    • Benchmark kit available for download; Performance testing results and tips see page in menu
  • Unmatched event listener receives a callback if an event does not match any started statement

Excellent documentation

Many examples

  • J2EE and non-J2EE, from many different domains
  • Java Messaging Service (JMS) server shell demonstrates a multi-threaded JMS server with dynamic statement management using Java Management Extensions (JMX)

Supportive user and developer community

Performance tested

Typical Uses

What these applications have in common is the requirement to process events (or messages) in real-time or near real-time. This is sometimes referred to as complex event processing (CEP) and event stream analysis.

Key considerations for these types of applications are the complexity of the logic required, throughput and latency.

  • Complex computations - applications that detect patterns among events (event correlation), filter events, aggregate time or length windows of events, join event streams, trigger based on absence of events etc.
  • High throughput - applications that process large volumes of messages (between 1,000 to 100k messages per second)
  • Low latency - applications that react in real-time to conditions that occur (from a few milliseconds to a few seconds)

Java - Known Limitations

  • Esper requires a Java Virtual Machine version 5.0 runtime, or above.
  • Esper will not work with JavaVM versions 1.4.2 or below.

.NET - Known Limitations

  • NEsper is the .NET version of the Java Esper version 1.12

Summary

Esper and NEsper were designed to make it easier to build CEP and ESP applications. Esper and NEsper are open-source software available under the GNU General Public License (GPL) license.



We are happy to use licenses of IntelliJ IDEA, ReSharper, and dotTrace through the JetBrains Open Source Program

YourKit is kindly supporting open source projects with its full-featured Java Profiler. YourKit, LLC is creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products: YourKit Java Profiler and YourKit .NET Profiler. "