Esper - Event Stream and Complex Event Processing for Java
		Next

Esper - Event Stream and Complex Event Processing for Java

Reference Documentation

3.3.0

Table of Contents

Preface

1. Technology Overview

1.1. Introduction to CEP and event stream analysis
1.2. CEP and relational databases
1.3. The Esper engine for CEP
1.4. Required 3rd Party Libraries

2. Event Representations

2.1. Event Underlying Java Objects

2.2. Event Properties

2.2.1. Escape Characters

2.3. Dynamic Event Properties

2.4. Fragment and Fragment Type

2.5. Plain-Old Java Object Events

2.5.1. Java Object Event Properties
2.5.2. Property Names
2.5.3. Constants and Enumeration
2.5.4. Parameterized Types
2.5.5. Known Limitations

2.6. java.util.Map Events

2.6.1. Overview

2.6.2. Map Properties

2.6.3. Map Supertypes

2.6.4. Advanced Map Property Types

2.6.4.1. Nested Properties
2.6.4.2. Map Event Type Properties
2.6.4.3. One-to-Many Relationships

2.7. org.w3c.dom.Node XML Events

2.7.1. Schema-Provided XML Events

2.7.1.1. Getting Started
2.7.1.2. Property Expressions and Namespaces
2.7.1.3. Property Expression to XPath Rewrite
2.7.1.4. Array Properties
2.7.1.5. Dynamic Properties
2.7.1.6. Transposing Properties
2.7.1.7. Event Sender

2.7.2. No-Schema-Provided XML Events

2.7.3. Explicitly-Configured Properties

2.7.3.1. Simple Explicit Property
2.7.3.2. Explicit Property Casting and Parsing
2.7.3.3. Node and Nodeset Explicit Property

2.8. Additional Event Representations

2.9. Updating, Merging and Versioning Events

2.10. Coarse-Grained Events

2.11. Event Objects Populated by Insert Into

3. Processing Model

3.1. Introduction

3.2. Insert Stream

3.3. Insert and Remove Stream

3.4. Filters and Where-clauses

3.5. Time Windows

3.5.1. Time Window
3.5.2. Time Batch

3.6. Batch Windows

3.7. Aggregation and Grouping

3.7.1. Insert and Remove Stream

3.7.2. Output for Aggregation and Group-By

3.7.2.1. Un-aggregated and Un-grouped
3.7.2.2. Fully Aggregated and Un-grouped
3.7.2.3. Aggregated and Un-Grouped
3.7.2.4. Fully Aggregated and Grouped
3.7.2.5. Aggregated and Grouped

3.8. Event Visibility and Current Time

4. EPL Reference: Clauses

4.1. EPL Introduction

4.2. EPL Syntax

4.2.1. Specifying Time Periods

4.2.2. Using Comments

4.2.3. Reserved Keywords

4.2.4. Escaping Strings

4.2.5. Data Types

4.2.5.1. Data Type of Constants
4.2.5.2. BigInteger and BigDecimal

4.2.6. Annotation

4.2.6.1. Application-Provided Annotations
4.2.6.2. Built-In Annotations
4.2.6.3. @Name
4.2.6.4. @Description
4.2.6.5. @Tag
4.2.6.6. @Priority
4.2.6.7. @Drop
4.2.6.8. @Hint

4.3. Choosing Event Properties And Events: the Select Clause

4.3.1. Choosing all event properties: select *
4.3.2. Choosing specific event properties
4.3.3. Expressions
4.3.4. Renaming event properties
4.3.5. Choosing event properties and events in a join
4.3.6. Choosing event properties and events from a pattern
4.3.7. Selecting insert and remove stream events
4.3.8. Qualifying property names and stream names
4.3.9. Select Distinct

4.4. Specifying Event Streams: the From Clause

4.4.1. Filter-based Event Streams

4.4.1.1. Specifying an Event Type
4.4.1.2. Specifying Filter Criteria
4.4.1.3. Filtering Ranges
4.4.1.4. Filtering Sets of Values
4.4.1.5. Filter Limitations

4.4.2. Pattern-based Event Streams

4.4.3. Specifying Views

4.4.4. Multiple Data Window Views

4.4.5. Using the Stream Name

4.5. Specifying Search Conditions: the Where Clause

4.6. Aggregates and grouping: the Group-by Clause and the Having Clause

4.6.1. Using aggregate functions
4.6.2. Organizing statement results into groups: the Group-by clause
4.6.3. Selecting groups of events: the Having clause
4.6.4. How the stream filter, Where, Group By and Having clauses interact
4.6.5. Comparing the Group By clause and the std:groupby view

4.7. Stabilizing and Controlling Output: the Output Clause

4.7.1. Output Clause Options

4.7.1.1. Controlling Output Using an Expression
4.7.1.2. Suppressing Output With After

4.7.2. Aggregation, Group By, Having and Output clause interaction

4.7.3. Runtime Considerations

4.8. Sorting Output: the Order By Clause

4.9. Limiting Row Count: the Limit Clause

4.10. Merging Streams and Continuous Insertion: the Insert Into Clause

4.10.1. Transposing a Property To a Stream
4.10.2. Merging Streams By Event Type
4.10.3. Merging Disparate Types of Events: Variant Streams
4.10.4. Decorated Events
4.10.5. Event as a Property
4.10.6. Populating an Underlying Event Object

4.11. Joining Event Streams

4.12. Outer and Inner Joins

4.13. Unidirectional Joins

4.14. Subqueries

4.14.1. The 'exists' Keyword
4.14.2. The 'in' and 'not in' Keywords
4.14.3. The 'any' and 'some' Keywords
4.14.4. The 'all' Keyword

4.15. Accessing Relational Data via SQL

4.15.1. Joining SQL Query Results
4.15.2. SQL Query and the EPL Where Clause
4.15.3. Outer Joins With SQL Queries
4.15.4. Using Patterns to Request (Poll) Data
4.15.5. Polling SQL Queries via Iterator
4.15.6. JDBC Implementation Overview
4.15.7. Oracle Drivers and No-Metadata Workaround

4.16. Accessing Non-Relational Data via Method Invocation

4.16.1. Joining Method Invocation Results
4.16.2. Polling Method Invocation Results via Iterator
4.16.3. Providing the Method
4.16.4. Using a Map Return Type

4.17. Creating and Using Named Windows

4.17.1. Creating Named Windows: the Create Window clause

4.17.1.1. Creation by Modelling after an Existing Type
4.17.1.2. Creation By Defining Columns Names and Types
4.17.1.3. Dropping or Removing Named Windows

4.17.2. Inserting Into Named Windows

4.17.2.1. Named Windows Holding Decorated Events
4.17.2.2. Named Windows Holding Events As Property

4.17.3. Selecting From Named Windows

4.17.4. Triggered Select on Named Windows: the On Select clause

4.17.5. Triggered Playback from Named Windows: the On Insert clause

4.17.6. Populating a Named Window from an Existing Named Window

4.17.7. Updating Named Windows: the On Update clause

4.17.8. Deleting From Named Windows: the On Delete clause

4.17.8.1. Using Patterns in the On Delete Clause

4.17.9. Versioning and Merging Events in Named Windows

4.18. Splitting and Duplicating Streams

4.19. Variables

4.19.1. Creating Variables: the Create Variable clause
4.19.2. Setting Variable Values: the On Set clause
4.19.3. Using Variables

4.20. Contained-Event Selection

4.20.1. Select Clause in a Contained-Event Selection
4.20.2. Where Clause in a Contained-Event Selection
4.20.3. Contained-Event Selection and Joins

4.21. Updating an Insert Stream: the Update IStream Clause

4.21.1. Immutability and Updates

5. EPL Reference: Patterns

5.1. Event Pattern Overview

5.2. How to use Patterns

5.2.1. Pattern Syntax
5.2.2. Patterns in EPL
5.2.3. Subscribing to Pattern Events
5.2.4. Pulling Data from Patterns

5.3. Operator Precedence

5.4. Filter Expressions In Patterns

5.5. Pattern Operators

5.5.1. Every

5.5.1.1. Limiting Subexpression Lifetime
5.5.1.2. Every Operator Example
5.5.1.3. Sensor Example

5.5.2. Every-Distinct

5.5.3. Repeat

5.5.4. Repeat-Until

5.5.4.1. Unbound Repeat
5.5.4.2. Bound Repeat Overview
5.5.4.3. Bound Repeat - Open Ended Range
5.5.4.4. Bound Repeat - High Endpoint Range
5.5.4.5. Bound Repeat - Bounded Range
5.5.4.6. Tags and the Repeat Operator

5.5.5. And

5.5.6. Or

5.5.7. Not

5.5.8. Followed-by

5.5.9. Pattern Guards

5.5.9.1. timer:within

5.6. Pattern Atoms

5.6.1. Filter Atoms

5.6.2. Time-based Observer Atoms

5.6.2.1. timer:interval
5.6.2.2. timer:at

6. EPL Reference: Match Recognize

6.1. Overview

6.2. Comparison of Match Recognize and EPL Patterns

6.3. Syntax

6.3.1. Syntax Example

6.4. Pattern and Pattern Operators

6.4.1. Operator Precedence

6.4.2. Concatenation

6.4.3. Alternation

6.4.4. Quantifiers Overview

6.4.5. Variables Can be Singleton or Group

6.4.5.1. Additional Aggregation Functions

6.4.6. Eliminating Duplicate Matches

6.4.7. Greedy Or Reluctant

6.4.8. Quantifier - One Or More (+ and +?)

6.4.9. Quantifier - Zero Or More (+ and +?)

6.4.10. Quantifier - Zero Or One (? and ??)

6.5. Define Clause

6.5.1. The Prev Operator

6.6. Measure Clause

6.7. Datawindow-Bound

6.8. Interval

6.9. Limitations

7. EPL Reference: Operators

7.1. Arithmetic Operators
7.2. Logical And Comparison Operators
7.3. Concatenation Operators
7.4. Binary Operators
7.5. Array Definition Operator
7.6. The 'in' Keyword
7.7. The 'between' Keyword
7.8. The 'like' Keyword
7.9. The 'regexp' Keyword
7.10. The 'any' and 'some' Keywords
7.11. The 'all' Keyword

8. EPL Reference: Functions

8.1. Single-row Function Reference

8.1.1. The Case Control Flow Function

8.1.2. The Cast Function

8.1.3. The Coalesce Function

8.1.4. The Current_Timestamp Function

8.1.5. The Exists Function

8.1.6. The Instance-Of Function

8.1.7. The Min and Max Functions

8.1.8. The Previous Function

8.1.8.1. Previous Event per Group
8.1.8.2. Restrictions
8.1.8.3. Comparison to the prior Function

8.1.9. The Prior Function

8.2. Aggregate Functions

8.3. User-Defined Functions

9. EPL Reference: Views

9.1. Window views

9.1.1. Length window (win:length)
9.1.2. Length batch window (win:length_batch)
9.1.3. Time window (win:time)
9.1.4. Externally-timed window (win:ext_timed)
9.1.5. Time batch window (win:time_batch)
9.1.6. Time-Length combination batch window (win:time_length_batch)
9.1.7. Time-Accumulating window (win:time_accum)
9.1.8. Keep-All window (win:keepall)
9.1.9. First Length (win:firstlength)
9.1.10. First Time (win:firsttime)

9.2. Standard view set

9.2.1. Unique (std:unique)
9.2.2. Group-By (std:groupby)
9.2.3. Size (std:size)
9.2.4. Last Event (std:lastevent)
9.2.5. First Event (std:firstevent)
9.2.6. First Unique (std:firstunique)

9.3. Statistics views

9.3.1. Univariate statistics (stat:uni)
9.3.2. Regression (stat:linest)
9.3.3. Correlation (stat:correl)
9.3.4. Weighted average (stat:weighted_avg)

9.4. Extension View Set

9.4.1. Sorted Window View (ext:sort)
9.4.2. Time-Order View (ext:time_order)

10. API Reference

10.1. API Overview

10.2. The Service Provider Interface

10.3. The Administrative Interface

10.3.1. Creating Statements

10.3.2. Receiving Statement Results

10.3.3. Setting a Subscriber Object

10.3.3.1. Row-By-Row Delivery
10.3.3.2. Multi-Row Delivery

10.3.4. Adding Listeners

10.3.4.1. Subscription Snapshot and Atomic Delivery

10.3.5. Using Iterators

10.3.6. Managing Statements

10.3.7. Runtime Configuration

10.4. The Runtime Interface

10.4.1. Event Sender

10.4.2. Receiving Unmatched Events

10.4.3. On-Demand Snapshot Query Execution

10.4.3.1. On-Demand Query API

10.5. Event and Event Type

10.5.1. Event Type Metadata
10.5.2. Event Object
10.5.3. Query Example
10.5.4. Pattern Example

10.6. Engine Threading and Concurrency

10.6.1. Advanced Threading

10.6.1.1. Inbound Threading
10.6.1.2. Outbound Threading
10.6.1.3. Timer Execution Threading
10.6.1.4. Route Execution Threading
10.6.1.5. Threading Service Provider Interface

10.7. Controlling Time-Keeping

10.8. Time Resolution

10.9. Service Isolation

10.9.1. Overview
10.9.2. Example: Suspending a Statement
10.9.3. Example: Catching up a Statement from Historical Data
10.9.4. Isolation for Insert-Into
10.9.5. Isolation for Named Windows
10.9.6. Runtime Considerations

10.10. Statement Object Model

10.10.1. Building an Object Model
10.10.2. Building Expressions
10.10.3. Building a Pattern Statement
10.10.4. Building a Select Statement
10.10.5. Building a Create-Variable and On-Set Statement
10.10.6. Building Create-Window, On-Delete and On-Select Statements

10.11. Prepared Statement and Substitution Parameters

10.12. Engine and Statement Metrics Reporting

10.12.1. Engine Metrics
10.12.2. Statement Metrics

10.13. Event Rendering to XML and JSON

10.13.1. JSON Event Rendering Conventions and Options
10.13.2. XML Event Rendering Conventions and Options

10.14. Plug-in Loader

11. Configuration

11.1. Programmatic Configuration

11.2. Configuration via XML File

11.3. XML Configuration File

11.4. Configuration Items

11.4.1. Events represented by Java Classes

11.4.1.1. Package of Java Event Classes
11.4.1.2. Event type name to Java class mapping
11.4.1.3. Non-JavaBean and Legacy Java Event Classes
11.4.1.4. Specifying Event Properties for Java Classes
11.4.1.5. Turning off Code Generation
11.4.1.6. Case Sensitivity and Property Names
11.4.1.7. Factory and Copy Method

11.4.2. Events represented by java.util.Map

11.4.3. Events represented by org.w3c.dom.Node

11.4.3.1. Schema Resource
11.4.3.2. Explicit XPath Property
11.4.3.3. Absolute or Deep Property Resolution
11.4.3.4. XPath Variable and Function Resolver
11.4.3.5. Auto Fragment
11.4.3.6. XPath Property Expression
11.4.3.7. Event Sender Setting

11.4.4. Events represented by Plug-in Event Representations

11.4.4.1. Enabling an Custom Event Representation
11.4.4.2. Adding Plug-in Event Types
11.4.4.3. Setting Resolution URIs

11.4.5. Class and package imports

11.4.6. Cache Settings for From-Clause Method Invocations

11.4.7. Variables

11.4.8. Relational Database Access

11.4.8.1. Connections obtained via DataSource
11.4.8.2. Connections obtained via DataSource Factory
11.4.8.3. Connections obtained via DriverManager
11.4.8.4. Connections-level settings
11.4.8.5. Connections lifecycle settings
11.4.8.6. Cache settings
11.4.8.7. Column Change Case
11.4.8.8. SQL Types Mapping
11.4.8.9. Metadata Origin

11.4.9. Engine Settings related to Concurrency and Threading

11.4.9.1. Preserving the order of events delivered to listeners
11.4.9.2. Preserving the order of events for insert-into streams
11.4.9.3. Internal Timer Settings
11.4.9.4. Advanced Threading Options

11.4.10. Engine Settings related to Event Metadata

11.4.10.1. Java Class Property Names and Case Sensitivity

11.4.11. Engine Settings related to View Resources

11.4.11.1. Sharing View Resources between Statements
11.4.11.2. Configuring Multi-Expiry Policy Defaults

11.4.12. Engine Settings related to Logging

11.4.12.1. Execution Path Debug Logging

11.4.13. Engine Settings related to Variables

11.4.13.1. Variable Version Release Interval

11.4.14. Engine Settings related to Stream Selection

11.4.14.1. Default Statement Stream Selection

11.4.15. Engine Settings related to Time Source

11.4.15.1. Default Time Source

11.4.16. Engine Settings related to Metrics Reporting

11.4.17. Engine Settings related to Language and Locale

11.4.18. Engine Settings related to Expression Evaluation

11.4.18.1. Integer Division and Division by Zero
11.4.18.2. Subselect Evaluation Order
11.4.18.3. User-Defined Function or Static Method Cache
11.4.18.4. Extended Built-in Aggregation Functions

11.4.19. Engine Settings related to Execution of Statements

11.4.19.1. Prioritized Execution

11.4.20. Revision Event Type

11.4.21. Variant Stream

11.5. Type Names

11.6. Runtime Configuration

12. Extension and Plug-in

12.1. Custom View Implementation

12.1.1. Implementing a View Factory
12.1.2. Implementing a View
12.1.3. View Contract
12.1.4. Configuring View Namespace and Name
12.1.5. Requirement for Data Window Views
12.1.6. Requirement for Grouped Views

12.2. Custom Aggregation Functions

12.2.1. Implementing an Aggregation Function
12.2.2. Configuring Aggregation Function Name
12.2.3. Accepting Multiple Parameters

12.3. Custom Pattern Guard

12.3.1. Implementing a Guard Factory
12.3.2. Implementing a Guard Class
12.3.3. Configuring Guard Namespace and Name

12.4. Custom Pattern Observer

12.4.1. Implementing an Observer Factory
12.4.2. Implementing an Observer Class
12.4.3. Configuring Observer Namespace and Name

12.5. Custom Event Representation

12.5.1. How It Works

12.5.2. Steps

12.5.3. URI-based Resolution

12.5.4. Example

12.5.4.1. Sample Event Type
12.5.4.2. Sample Event Bean
12.5.4.3. Sample Event Representation
12.5.4.4. Sample Event Bean Factory

13. Examples, Tutorials, Case Studies

13.1. Examples Overview

13.2. Running the Examples

13.3. AutoID RFID Reader

13.4. JMS Server Shell and Client

13.4.1. Overview
13.4.2. JMS Messages as Events
13.4.3. JMX for Remote Dynamic Statement Management

13.5. Market Data Feed Monitor

13.5.1. Input Events
13.5.2. Computing Rates Per Feed
13.5.3. Detecting a Fall-off
13.5.4. Event generator

13.6. OHLC Plug-in View

13.7. Transaction 3-Event Challenge

13.7.1. The Events
13.7.2. Combined event
13.7.3. Real time summary data
13.7.4. Find problems
13.7.5. Event generator

13.8. Self-Service Terminal

13.8.1. Events

13.8.2. Detecting Customer Check-in Issues

13.8.3. Absence of Status Events

13.8.4. Activity Summary Data

13.8.5. Sample Application for J2EE Application Server

13.8.5.1. Running the Example
13.8.5.2. Building the Example
13.8.5.3. Running the Event Simulator and Receiver

13.9. Assets Moving Across Zones - An RFID Example

13.10. StockTicker

13.11. MatchMaker

13.12. Named Window Query

13.13. Quality of Service

14. Performance

14.1. Performance Results

14.2. Performance Tips

14.2.1. Understand how to tune your Java virtual machine
14.2.2. Compare Esper to other solutions
14.2.3. Input and Output Bottlenecks
14.2.4. Advanced Theading
14.2.5. Select the underlying event rather than individual fields
14.2.6. Prefer stream-level filtering over post-data-window filtering
14.2.7. Reduce the use of arithmetic in expressions
14.2.8. Remove Unneccessary Constructs
14.2.9. End Pattern Sub-Expressions
14.2.10. Consider using EventPropertyGetter for fast access to event properties
14.2.11. Consider casting the underlying event
14.2.12. Turn off logging
14.2.13. Disable view sharing
14.2.14. Tune or disable delivery order guarantees
14.2.15. Use a Subscriber Object to Receive Events
14.2.16. High-Arrival-Rate Streams and Single Statements
14.2.17. Joins And Where-clause And Data Windows
14.2.18. Patterns and Pattern Sub-Expression Instances
14.2.19. The Keep-All Data Window
14.2.20. Performance, JVM, OS and hardware
14.2.21. Consider using Hints

14.3. Using the performance kit

14.3.1. How to use the performance kit
14.3.2. How we use the performance kit

15. References

15.1. Reference List

A. Output Reference and Samples

A.1. Introduction and Sample Data

A.2. Output for Un-aggregated and Un-grouped Queries

A.2.1. No Output Rate Limiting
A.2.2. Output Rate Limiting - Default
A.2.3. Output Rate Limiting - Last
A.2.4. Output Rate Limiting - First
A.2.5. Output Rate Limiting - Snapshot

A.3. Output for Fully-aggregated and Un-grouped Queries

A.3.1. No Output Rate Limiting
A.3.2. Output Rate Limiting - Default
A.3.3. Output Rate Limiting - Last
A.3.4. Output Rate Limiting - First
A.3.5. Output Rate Limiting - Snapshot

A.4. Output for Aggregated and Un-grouped Queries

A.4.1. No Output Rate Limiting
A.4.2. Output Rate Limiting - Default
A.4.3. Output Rate Limiting - Last
A.4.4. Output Rate Limiting - First
A.4.5. Output Rate Limiting - Snapshot

A.5. Output for Fully-aggregated and Grouped Queries

A.5.1. No Output Rate Limiting
A.5.2. Output Rate Limiting - Default
A.5.3. Output Rate Limiting - All
A.5.4. Output Rate Limiting - Last
A.5.5. Output Rate Limiting - First
A.5.6. Output Rate Limiting - Snapshot

A.6. Output for Aggregated and Grouped Queries

A.6.1. No Output Rate Limiting
A.6.2. Output Rate Limiting - Default
A.6.3. Output Rate Limiting - All
A.6.4. Output Rate Limiting - Last
A.6.5. Output Rate Limiting - First
A.6.6. Output Rate Limiting - Snapshot

B. Reserved Keywords

Index

		Next
		Preface