Home Contact

Event Stream Intelligence: Esper & NEsper

Where Complex Event Processing meets Open Source: Esper & NEsper
High-availability and Enterprise Readiness: EsperHA, Enterprise Edition and support services. Embed mainstream CEP in your products and deployments: Learn about our licensing options.
Product info | Customers | Contact us

Event Stream Intelligence with Esper and NEsper

Esper is a component for CEP and ESP applications, available for Java as Esper, and for .NET as NEsper.

Esper and NEsper enable rapid development of applications that process large volumes of incoming messages or events. Esper and NEsper filter and analyze events in various ways, and respond to conditions of interest in real-time.

Technology Introduction

Complex Event Processing, or CEP, is technology to process events and discover complex patterns among multiple streams of event data. ESP stands for Event Stream Processing and deals with the task of processing multiple streams of event data with the goal of identifying the meaningful events within those streams, and deriving meaningful information from them. Real-time OLAP (online analytical processing) and continuous query are also terms used frequently for this technology.

The Esper engine has been developed to address the requirements of applications that analyze and react to events. Some typical examples of applications are:

  • Business process management and automation (process monitoring, BAM, reporting exceptions, operational intelligence)
  • Finance (algorithmic trading, fraud detection, risk management)
  • Network and application monitoring (intrusion detection, SLA monitoring)
  • Sensor network applications (RFID reading, scheduling and control of fabrication lines, air traffic)

Features

Event Stream Processing

  • Sliding windows: time, length, sorted, accumulating, time-ordering, externally-timed (value-based windowing)
  • Tumbling windows: time, length and multi-policy; first-event
  • Combine windows with intersection and union semantics.
  • Grouping, aggregation, sorting, filtering, merging, splitting or duplicating of event streams
  • Tailored SQL-like query language using insert into, select, from, where, group-by, having and order-by clauses
  • Inner-joins and outer joins (left, right, full) of an unlimited number of streams or windows
  • Subqueries including exists and in
  • Output rate limiting and stabilizing, snapshot output
  • Named windows
    • Explicit sharing of data windows between statements
    • Multiple and custom entry and exit criteria for events
    • Support for predefined query execution optimized by indexed access, via on-select

Event Pattern Matching

  • Logical and temporal event correlation
  • Crontab-like timer 'at' operator
  • Lifecycle of pattern can be controlled by timer and via operators, repeat-number and repeat-until, every-distinct
  • Pattern-matched events provided to listeners

Event Representations

  • Supports event-type inheritance and polymorphism as provided by the Java language, for Java object events as well as for Map-type events
  • Events can be plain Java objects, XML (DOM and streaming through Apache Axiom) and java.util.Map including nested objects and hierarchical maps
  • Event properties can be simple, indexed, mapped or nested - allows querying of deep Java object graphs and XML structures
  • Dynamic properties allow dynamic typing of properties, supported by cast, instanceof and exists functions
  • Applications can plug-in their own event representation and dynamic type resolution
  • Insert-into can directly populate a plain or legacy Java object or Map from a select-clause

Prepared statements and substitution parameters

  • Precompile a statement with substitution parameters and efficiently start the parameterized statement multiple times

Statement Object Model

  • A set of classes providing an object-oriented representation of a statement
  • Full and complete specification of a statement via object model
  • Round-trip from object model to statement text and back to object model
  • Build, change or interrogate statements beyond the textual representation

Input Adapters

  • CSV input adapter reads comma-separated value formats
    • simulate multiple event streams with timed, coordinated playback via timestamp column
    • load generator
    • preloading of reference data
  • JMS input and output adapter based on Spring JMS templates

Other

  • Executes a large number of simultaneous queries - Esper's query processor can run thousands of queries continuously and simultaneously on a single instance
  • Relational database access via SQL-query joins with event streams
    • LRU (least-recently used) and expiry-time query result caches
    • Keyed cache entries for fast cache lookup
    • Engine indexes cached rows for fast filtering within a large number of SQL-query result rows
    • Multiple SQL-queries in one statement transparently integrates multiple autonomous database systems
  • Joins to method invocation results allows easy integration with distributed caches, web services and object-oriented databases
  • On-demand queries are fire-and-forget EPL queries against named windows for non-continuous query execution
  • Variables can occur in any expression and can dynamically control output rate
    • Guarantees of consistency and atomicity of variable updates
  • Variant event streams allows treating disparate types of events as the same type, such as when the event type can only be known at runtime, when the event type is expected to vary, or when optional properties are desired.
  • Support for update events that update, provide a new version or that revise an existing event held by an engine.
  • Support for both the listener (push/subscription) API and the consumer (pull/receive) API for querying results
  • Concurrency-safe iterator provides complete query capability for all statements
  • Supports externally-provided time as well as Java system time, allowing applications full control over the concept of time within an engine
  • Multithread-safe as of release 1.5
    • Multithreaded sends of events into an engine
    • Create, start and stop statements during operation without adversely impacting performance
    • Applications can retain full control over threading; Inbound, outbound and execution threading configurable
    • Efficiently sharing resources between statements and low thread blocking
  • Supports multiple independent Esper engines per JavaVM
  • Pluggable architecture for event pattern and event stream analysis via user-defined functions, plug-in views, plug-in aggregation functions, plug-in pattern guards and plug-in pattern event observers, event instance methods
  • Performance-minded design: query strategy analysis and index building; array-based collections; delta networks and many other techniques
    • Benchmark kit available for download; Performance testing results and tips see page in menu
  • Unmatched event listener receives a callback if an event does not match any started statement
  • Support for all Java data types including BigInteger and BigDecimal
  • Contained-Event select syntax for easy handling of coarse-grained, business-level events that themselves contain events
  • JSON and XML output event rendering without syntax to learn
  • Annotations allow adding informaton to EPL statements specific to your application
  • Assign a priority to your statement; Identify preemptive statements that cause an event to drop in face of multiple statements matching filters.

Excellent documentation

Many examples

  • J2EE and non-J2EE, from many different domains
  • Java Messaging Service (JMS) server shell demonstrates a multi-threaded JMS server with dynamic statement management using Java Management Extensions (JMX)

Supportive user and developer community

Performance tested

Typical Uses

What these applications have in common is the requirement to process events (or messages) in real-time or near real-time. This is sometimes referred to as complex event processing (CEP) and event stream analysis.

Key considerations for these types of applications are the complexity of the logic required, throughput and latency.

  • Complex computations - applications that detect patterns among events (event correlation), filter events, aggregate time or length windows of events, join event streams, trigger based on absence of events etc.
  • High throughput - applications that process large volumes of messages (between 1,000 to 100k messages per second)
  • Low latency - applications that react in real-time to conditions that occur (from a few milliseconds to a few seconds)

Java - Known Limitations

  • Esper requires a Java Virtual Machine version 5.0 runtime, or above.
  • Esper will not work with JavaVM versions 1.4.2 or below.

.NET - Known Limitations

  • NEsper is the .NET version of the Java Esper, check the release section for feature information

Summary

Esper and NEsper were designed to make it easier to build CEP and ESP applications. Esper and NEsper are open-source software available under the GNU General Public License (GPL) license.



We are happy to use licenses of IntelliJ IDEA, ReSharper, and dotTrace through the JetBrains Open Source Program

YourKit is kindly supporting open source projects with its full-featured Java Profiler. YourKit, LLC is creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products: YourKit Java Profiler and YourKit .NET Profiler.