Chapter 2. Event Representations

This section outlines the different means to model and represent events.

Esper uses the term event type to describe the type information available for an event representation.

Your application may configure predefined event types at startup time or dynamically add event types at runtime via API or EPL syntax. See Section 14.4, “Configuration Items” for startup-time configuration and Section 13.3.7, “Runtime Configuration” for the runtime configuration API.

The EPL create schema syntax allows declaring an event type at runtime using EPL, see Section 5.16, “Declaring an Event Type: Create Schema”.

In Section 13.5, “Event and Event Type” we explain how an event type becomes visible in EPL statements and output events delivered by the engine.

2.1. Event Underlying Java Objects

An event is an immutable record of a past occurrence of an action or state change. Event properties capture the state information for an event.

In Esper, an event can be represented by any of the following underlying Java objects:

Table 2.1. Event Underlying Java Objects

Java ClassDescription
java.lang.ObjectAny Java POJO (plain-old java object) with getter methods following JavaBean conventions; Legacy Java classes not following JavaBean conventions can also serve as events .
java.util.MapMap events are key-values pairs and can also contain objects, further Map, and arrays thereof.
org.w3c.dom.NodeXML document object model (DOM).
org.apache.axiom.om.OMDocument or OMElementXML - Streaming API for XML (StAX) - Apache Axiom (provided by EsperIO package).
Application classesPlug-in event representation via the extension API.

Esper provides multiple choices for representing an event. There is no absolute need for you to create new Java classes to represent an event.

Event representations have the following in common:

  • All event representations support nested, indexed and mapped properties (aka. property expression), as explained in more detail below. There is no limitation to the nesting level.

  • All event representations provide event type metadata. This includes type metadata for nested properties.

  • All event representations allow transposing the event itself and parts of all of its property graph into new events. The term transposing refers to selecting the event itself or event properties that are themselves nestable property graphs, and then querying the event's properties or nested property graphs in further statements. The Apache Axiom event representation is an exception and does not currently allow transposing event properties but does allow transposing the event itself.

  • The Java object and Map representations allow supertypes.

The API behavior for all event representations is the same, with minor exceptions noted in this chapter.

The benefits of multiple event representations are:

  • For applications that already have events in one of the supported representations, there is no need to transform events into a Java object before processing.

  • Event representations are exchangeable, reducing or eliminating the need to change statements when the event representation changes.

  • Event representations are interoperable, allowing all event representations to interoperate in same or different statements.

  • The choice makes its possible to consciously trade-off performance, ease-of-use, the ability to evolve and effort needed to import or externalize events and use existing event type metadata.

2.2. Event Properties

Event properties capture the state information for an event. Event properties be simple as well as indexed, mapped and nested event properties. The table below outlines the different types of properties and their syntax in an event expression. This syntax allows statements to query deep JavaBean objects graphs, XML structures and Map events.

Table 2.2. Types of Event Properties

TypeDescriptionSyntaxExample
SimpleA property that has a single value that may be retrieved.
name
sensorId
IndexedAn indexed property stores an ordered collection of objects (all of the same type) that can be individually accessed by an integer-valued, non-negative index (or subscript).
name[index]
sensor[0]
MappedA mapped property stores a keyed collection of objects (all of the same type).
name('key')
sensor('light')
NestedA nested property is a property that lives within another property of an event.
name.nestedname
sensor.value

Combinations are also possible. For example, a valid combination could be person.address('home').street[0].

2.2.1. Escape Characters

If your application uses java.util.Map or XML to represent events, then event property names may themselves contain the dot ('.') character. The backslash ('\') character can be used to escape dot characters in property names, allowing a property name to contain dot characters.

For example, the EPL as shown below expects a property by name part1.part2 to exist on event type MyEvent:

select part1\.part2 from MyEvent

Sometimes your event properties may overlap with EPL language keywords. In this case you may use the backwards apostrophe ` character to escape the property name.

The next example assumes a Quote event that has a property by name order, while order is also a reserved keyword:

select `order` from Quote

2.3. Dynamic Event Properties

Dynamic (unchecked) properties are event properties that need not be known at statement compilation time. Such properties are resolved during runtime: they provide duck typing functionality.

The idea behind dynamic properties is that for a given underlying event representation we don't always know all properties in advance. An underlying event may have additional properties that are not known at statement compilation time, that we want to query on. The concept is especially useful for events that represent rich, object-oriented domain models.

The syntax of dynamic properties consists of the property name and a question mark. Indexed, mapped and nested properties can also be dynamic properties:

Table 2.3. Types of Event Properties

TypeSyntax
Dynamic Simple
name?
Dynamic Indexed
name[index]?
Dynamic Mapped
name('key')?
Dynamic Nested
name?.nestedPropertyName

Dynamic properties always return the java.lang.Object type. Also, dynamic properties return a null value if the dynamic property does not exist on events processed at runtime.

As an example, consider an OrderEvent event that provides an "item" property. The "item" property is of type Object and holds a reference to an instance of either a Service or Product.

Assume that both Service and Product classes provide a property named "price". Via a dynamic property we can specify a query that obtains the price property from either object (Service or Product):

select item.price? from OrderEvent

As a second example, assume that the Service class contains a "serviceName" property that the Product class does not possess. The following query returns the value of the "serviceName" property for Service objects. It returns a null-value for Product objects that do not have the "serviceName" property:

select item.serviceName? from OrderEvent

Consider the case where OrderEvent has multiple implementation classes, some of which have a "timestamp" property. The next query returns the timestamp property of those implementations of the OrderEvent interface that feature the property:

select timestamp? from OrderEvent

The query as above returns a single column named "timestamp?" of type Object.

When dynamic properties are nested, then all properties under the dynamic property are also considered dynamic properties. In the below example the query asks for the "direction" property of the object returned by the "detail" dynamic property:

select detail?.direction from OrderEvent

Above is equivalent to:

select detail?.direction? from OrderEvent

The functions that are often useful in conjunction with dynamic properties are:

  • The cast function casts the value of a dynamic property (or the value of an expression) to a given type.

  • The exists function checks whether a dynamic property exists. It returns true if the event has a property of that name, or false if the property does not exist on that event.

  • The instanceof function checks whether the value of a dynamic property (or the value of an expression) is of any of the given types.

  • The typeof function returns the string type name of a dynamic property.

Dynamic event properties work with all event representations outlined next: Java objects, Map-based and XML DOM-based events.

2.4. Fragment and Fragment Type

Sometimes an event can have properties that are itself events. Esper uses the term fragment and fragment type for such event pieces. The best example is a pattern that matches two or more events and the output event contains the matching events as fragments. In other words, output events can be a composite event that consists of further events, the fragments.

Fragments have the same metadata available as their enclosing composite events. The metadata for enclosing composite events contains information about which properties are fragments, or have a property value that can be represented as a fragment and therefore as an event itself.

Fragments and type metadata can allow your application to navigate composite events without the need for using the Java reflection API and reducing the coupling to the underlying event representation. The API is further described in Section 13.5, “Event and Event Type”.

2.5. Plain-Old Java Object Events

Plain-old Java object events are object instances that expose event properties through JavaBeans-style getter methods. Events classes or interfaces do not have to be fully compliant to the JavaBean specification; however for the Esper engine to obtain event properties, the required JavaBean getter methods must be present or an accessor-style and accessor-methods may be defined via configuration.

Esper supports JavaBeans-style event classes that extend a superclass or implement one or more interfaces. Also, Esper event pattern and EPL statements can refer to Java interface classes and abstract classes.

Classes that represent events should be made immutable. As events are recordings of a state change or action that occurred in the past, the relevant event properties should not be changeable. However this is not a hard requirement and the Esper engine accepts events that are mutable as well.

The hashCode and equals methods do not need to be implemented. The implementation of these methods by a Java event class does not affect the behavior of the engine in any way.

Please see Chapter 14, Configuration on options for naming event types represented by Java object event classes. Java classes that do not follow JavaBean conventions, such as legacy Java classes that expose public fields, or methods not following naming conventions, require additional configuration. Via configuration it is also possible to control case sensitivity in property name resolution. The relevant section in the chapter on configuration is Section 14.4.1.3, “Non-JavaBean and Legacy Java Event Classes”.

2.5.1. Java Object Event Properties

As outlined earlier, the different property types are supported by the standard JavaBeans specification, and some of which are uniquely supported by Esper:

  • Simple properties have a single value that may be retrieved. The underlying property type might be a Java language primitive (such as int, a simple object (such as a java.lang.String), or a more complex object whose class is defined either by the Java language, by the application, or by a class library included with the application.

  • Indexed - An indexed property stores an ordered collection of objects (all of the same type) that can be individually accessed by an integer-valued, non-negative index (or subscript).

  • Mapped - As an extension to standard JavaBeans APIs, Esper considers any property that accepts a String-valued key a mapped property.

  • Nested - A nested property is a property that lives within another Java object which itself is a property of an event.

Assume there is an NewEmployeeEvent event class as shown below. The mapped and indexed properties in this example return Java objects but could also return Java language primitive types (such as int or String). The Address object and Employee can themselves have properties that are nested within them, such as a street name in the Address object or a name of the employee in the Employee object.

public class NewEmployeeEvent {
	public String getFirstName();
	public Address getAddress(String type);
	public Employee getSubordinate(int index);
	public Employee[] getAllSubordinates();
}

Simple event properties require a getter-method that returns the property value. In this example, the getFirstName getter method returns the firstName event property of type String.

Indexed event properties require either one of the following getter-methods. A method that takes an integer-type key value and returns the property value, such as the getSubordinate method, or a method that returns an array-type, or a class that implements Iterable. An example is the getSubordinates getter method, which returns an array of Employee but could also return an Iterable. In an EPL or event pattern statement, indexed properties are accessed via the property[index] syntax.

Mapped event properties require a getter-method that takes a String-typed key value and returns the property value, such as the getAddress method. In an EPL or event pattern statement, mapped properties are accessed via the property('key') syntax.

Nested event properties require a getter-method that returns the nesting object. The getAddress and getSubordinate methods are mapped and indexed properties that return a nesting object. In an EPL or event pattern statement, nested properties are accessed via the property.nestedProperty syntax.

All event pattern and EPL statements allow the use of indexed, mapped and nested properties (or a combination of these) anywhere where one or more event property names are expected. The below example shows different combinations of indexed, mapped and nested properties in filters of event pattern expressions (each line is a separate EPL statement):

every NewEmployeeEvent(firstName='myName')
every NewEmployeeEvent(address('home').streetName='Park Avenue')
every NewEmployeeEvent(subordinate[0].name='anotherName')
every NewEmployeeEvent(allSubordinates[1].name='thatName')
every NewEmployeeEvent(subordinate[0].address('home').streetName='Water Street')

Similarly, the syntax can be used in EPL statements in all places where an event property name is expected, such as in select lists, where-clauses or join criteria.

select firstName, address('work'), subordinate[0].name, subordinate[1].name
from NewEmployeeEvent
where address('work').streetName = 'Park Ave'

2.5.2. Property Names

Property names follows Java standards: the class java.beans.Introspector and method getBeanInfo returns the property names as derived from the name of getter methods. In addition, Esper configuration provides a flag to turn off case-sensitive property names. A sample list of getter methods and property names is:

Table 2.4. JavaBeans-style Getter Methods and Property Names

MethodProperty NameExample
getPrice()price
select price from MyEvent
getNAME()NAME
select NAME from MyEvent
getItemDesc()itemDesc
select itemDesc from MyEvent
getQ()q
select q from MyEvent
getQN()QN
select QN from MyEvent
getqn()qn
select qn from MyEvent
gets()s
select s from MyEvent

2.5.3. Constants and Enumeration

Constants are public static final fields in Java that may also participate in expressions of all kinds, as this example shows:

select * from MyEvent where property=MyConstantClass.FIELD_VALUE

Event properties that are enumeration values can be compared by their enumeration value:

select * from MyEvent where enumProp=EnumClass.ENUM_VALUE_1

Alternatively, a static method may be employed on a class, such as the enumeration class 'EnumClass' as below:

select * from MyEvent where enumProp=EnumClass.valueOf('ENUM_VALUE_1')

If your application does not import, through configuration, the package that contains the enumeration class, then it must also specify the package name of the class. Enumeration classes that are inner classes must be qualified with $ following Java conventions.

For example, the Color enumeration as an inner class to MyEvent in package org.myorg can be referenced as shown:

select * from MyEvent(enumProp=org.myorg.MyEvent$Color.GREEN).std:firstevent()

Instance methods may also be invoked on event instances by specifying a stream name, as shown below:

select myevent.computeSomething() as result from MyEvent as myevent

Chaining instance methods is supported as this example shows:

select myevent.getComputerFor('books', 'movies').calculate() as result 
from MyEvent as myevent

2.5.4. Parameterized Types

When your getter methods or accessor fields return a parameterized type, for example Iterable<MyEventData> for an indexed property or Map<String, MyEventData> for a mapped property, then property expressions may refer to the properties available through the class that is the type parameter.

An example event that has properties that are parameterized types is:

public class NewEmployeeEvent {
  public String getName();
  public Iterable<EducationHistory> getEducation();
  public Map<String, Address> getAddresses();
}

A sample of valid property expressions for this event is shown next:

select name, education, education[0].date, addresses('home').street
from NewEmployeeEvent

2.5.5. Known Limitations

Esper employs byte code generation for fast access to event properties. When byte code generation is unsuccessful, the engine logs a warning and uses Java reflection to obtain property values instead.

A known limitation is that when an interface has an attribute of a particular type and the actual event bean class returns a subclass of that attribute, the engine logs a warning and uses reflection for that property.

2.6. java.util.Map Events

2.6.1. Overview

Events can also be represented by objects that implement the java.util.Map interface. Event properties of Map events are the values in the map accessible through the get method exposed by the java.util.Map interface.

The Map event type is a comprehensive type system that can eliminate the need to use Java classes as event types, thereby making it easier to change types at runtime or generate type information from another source.

A given Map event type can have one or more supertypes that must also be Map event types. All properties available on any of the Map supertypes are available on the type itself. In addition, anywhere within EPL that an event type name of a Map supertype is used, any of its Map subtypes and their subtypes match that expression.

Your application can add properties to an existing Map event type during runtime using the configuration operation updateMapEventType. Properties may not be updated or deleted - properties can only be added, and nested properties can be added as well. The runtime configuration also allows removing Map event types and adding them back with new type information.

After your application configures a Map event type by providing a type name, the type name can be used when defining further Map event types by specifying the type name as a property type or an array property type.

One-to-Many relationships in Map event types are represented via arrays. A property in a Map event type may be an array of primitive, an array of Java object or an array of Map.

The engine can process java.util.Map events via the sendEvent(Map map, String eventTypeName) method on the EPRuntime interface. Entries in the Map represent event properties. Keys must be of type java.util.String for the engine to be able to look up event property names specified by pattern or EPL statements.

The engine does not validate Map event property names or values. Your application should ensure that objects passed in as event properties match the create schema property names and types, or the configured event type information when using runtime or static configuration.

2.6.2. Map Properties

Map event properties can be of any type. Map event properties that are Java application objects or that are of type java.util.Map (or arrays thereof) offer additional power:

  • Properties that are Java application objects can be queried via the nested, indexed, mapped and dynamic property syntax as outlined earlier.

  • Properties that are of type Map allow Maps to be nested arbitrarily deep and thus can be used to represent complex domain information. The nested, indexed, mapped and dynamic property syntax can be used to query Maps within Maps and arrays of Maps within Maps.

In order to use Map events, the event type name and property names and types must be made known to the engine via Configuration. Please see the examples in Section 14.4.2, “Events represented by java.util.Map”.

The code snippet below creates and processes a Map event. It defines a CarLocationUpdateEvent event type first:

Map event = new HashMap();
event.put("carId", carId);
event.put("direction", direction);
epRuntime.sendEvent(event, "CarLocUpdateEvent");

The CarLocUpdateEvent can now be used in a statement:

select carId from CarLocUpdateEvent.win:time(1 min) where direction = 1

The engine can also query Java objects as values in a Map event via the nested property syntax. Thus Map events can be used to aggregate multiple data structures into a single event and query the composite information in a convenient way. The example below demonstrates a Map event with a transaction and an account object.

Map event = new HashMap();
event.put("txn", txn);
event.put("account", account);
epRuntime.sendEvent(event, "TxnEvent");

An example statement could look as follows.

select account.id, account.rate * txn.amount 
from TxnEvent.win:time(60 sec) 
group by account.id

2.6.3. Map Supertypes

Your Map event type may declare one or more supertypes when configuring the type at engine initialization time or at runtime through the administrative interface.

Supertypes of a Map event type must also be Map event types. All property names and types of a supertype are also available on a subtype and override such same-name properties of the subtype. In addition, anywhere within EPL that an event type name of a Map supertype is used, any of its Map subtypes also matches that expression (similar to the concept of interface in Java).

This example assumes that the BaseUpdate event type has been declared and acts as a supertype to the AccountUpdate event type (both Map event types):

epService.getEPAdministrator().getConfiguration().
    addEventType("AccountUpdate", accountUpdateDef, 
    new String[] {"BaseUpdate"});

Your application EPL statements may select BaseUpdate events and receive both BaseUpdate and AccountUpdate events, as well as any other subtypes of BaseUpdate and their subtypes.

// Receive BaseUpdate and any subtypes including subtypes of subtypes
select * from BaseUpdate

Your application Map event type may have multiple supertypes. The multiple inheritance hierarchy between Maps can be arbitrarily deep, however cyclic dependencies are not allowed. If using runtime configuration, supertypes must exist before a subtype to a supertype can be added.

See Section 14.4.2, “Events represented by java.util.Map” for more information on configuring Map event types.

2.6.4. Advanced Map Property Types

2.6.4.1. Nested Properties

Strongly-typed nested Map-within-Map events can be used to build rich, type-safe event types on the fly. Use the addEventType method on Configuration or ConfigurationOperations for initialization-time and runtime-time type definition.

Noteworthy points are:

  • JavaBean (POJO) objects can appear as properties in Map-within-Map.

  • One may represent Map-within-Map and Map-Array within Map using the name of a previously registered Map event type.

  • There is no limit to the number of nesting levels.

  • Dynamic properties can be used to query Map-within-Map keys that may not be known in advance.

  • The engine returns a null value for properties for which the access path into the nested structure cannot be followed where map entries do not exist.

For demonstration, in this example our top-level event type is an AccountUpdate event, which has an UpdatedField structure as a property. Inside the UpdatedField structure the example defines various fields, as well as a property by name 'history' that holds a JavaBean class UpdateHistory to represent the update history for the account. The code snippet to define the event type is thus:

Map<String, Object> updatedFieldDef = new HashMap<String, Object>();
updatedFieldDef.put("name", String.class);
updatedFieldDef.put("addressLine1", String.class);
updatedFieldDef.put("history", UpdateHistory.class);

Map<String, Object> accountUpdateDef = new HashMap<String, Object>();
accountUpdateDef.put("accountId", long.class);
accountUpdateDef.put("fields", updatedFieldDef);

epService.getEPAdministrator().getConfiguration().
    addEventType("AccountUpdate", accountUpdateDef);

The next code snippet populates a sample event and sends the event into the engine:

Map<String, Object> updatedField = new HashMap<String, Object>();
updatedField.put("name", "Joe Doe");
updatedField.put("addressLine1", "40 Popular Street");
updatedField.put("history", new UpdateHistory());

Map<String, Object> accountUpdate = new HashMap<String, Object>();
accountUpdate.put("accountId", 10009901);
accountUpdate.put("fields", updatedField);

epService.getEPRuntime().sendEvent(accountUpdate, "AccountUpdate");

Last, a sample query to interrogate AccountUpdate events is as follows:

select accountId, fields.name, fields.addressLine1, fields.history.lastUpdate
from AccountUpdate

Note that type information for nested maps is only available to the immediately selecting stream. For example, the second select-query does not work:

insert into MyStream select fields from NestedMapEvent
// this does not work ... instead select the individual fields in the insert-into statement
select fields.name from MyStream 

2.6.4.2. Map Event Type Properties

Your application may declare a Map event type for reuse within other Map event types or for one-to-many properties represented by an array of Maps.

This example declares a Map event type by name AmountCurrency with amount and currency properties:

Map<String, Object> amountAndCurr = new HashMap<String, Object>();
amountAndCurr.put("amount", double.class);
amountAndCurr.put("currency", String.class);

epService.getEPAdministrator().getConfiguration().
    addEventType("AmountCurrency", amountAndCurr);

The AmountCurrency type is now available for use as a property type itself. Below code snippet declares OrderItem to hold an item number and AmountCurrency:

Map<String, Object> orderItem = new HashMap<String, Object>();
orderItem.put("itemNum", int.class);
orderItem.put("price", "AmountCurrency");    // The property type is the name itself

epService.getEPAdministrator().getConfiguration().
    addEventType("OrderItem", orderItem);

2.6.4.3. One-to-Many Relationships

To model repeated properties within a Map, you may use arrays as properties in a Map. You may use an array of primitive types or an array of JavaBean objects or an array of a previously declared Map event type.

When using a previously declared Map event type as an array property, the literal [] must be appended after the event type name.

This following example defines a Map event type by name Sale to hold array properties of the various types. It assumes a SalesPerson Java class exists and a Map event type by name OrderItem was declared:

Map<String, Object> sale = new HashMap<String, Object>();
sale.put("userids", int[].class);
sale.put("salesPersons", SalesPerson[].class);
sale.put("items", "OrderItem[]");	 // The property type is the name itself appended by []

epService.getEPAdministrator().getConfiguration().
    addEventType("SaleEvent", sale);

The three properties that the above example declares are:

  • An integer array of user ids.

  • An array of SalesPerson Java objects.

  • An array of Maps for order items.

The next EPL statement is a sample query asking for property values held by arrays:

select userids[0], salesPersons[1].name, 
    items[1], items[1].price.amount from SaleEvent

2.7. org.w3c.dom.Node XML Events

Events can be represented as org.w3c.dom.Node instances and send into the engine via the sendEvent method on EPRuntime or via EventSender. Please note that configuration is required so the event type name and root element name is known. See Chapter 14, Configuration.

If a XML schema document (XSD file) can be made available as part of the configuration, then Esper can read the schema and appropriately present event type metadata and validate statements that use the event type and its properties. See Section 2.7.1, “Schema-Provided XML Events”.

When no XML schema document is provided, XML events can still be queried, however the return type and return values of property expressions are string-only and no event type metadata is available other then for explicitly configured properties. See Section 2.7.2, “No-Schema-Provided XML Events”.

In all cases Esper allows you to configure explicit XPath expressions as event properties. You can specify arbitrary XPath functions or expressions and provide a property name and type by which result values will be available for use in EPL statements. See Section 2.7.3, “Explicitly-Configured Properties”.

Nested, mapped and indexed event properties are also supported in expressions against org.w3c.dom.Node events. Thus XML trees can conveniently be interrogated via the property expression syntax.

Only one event type per root element name may be configured. The engine recognizes each event by its root element name or you may use EventSender to send events.

This section uses the following XML document as an example:

<?xml version="1.0" encoding="UTF-8"?>
<Sensor xmlns="SensorSchema">
  <ID>urn:epc:1:4.16.36</ID>
  <Observation Command="READ_PALLET_TAGS_ONLY">
    <ID>00000001</ID>
    <Tag>
      <ID>urn:epc:1:2.24.400</ID>
    </Tag>
    <Tag>
      <ID>urn:epc:1:2.24.401</ID>
    </Tag>
  </Observation>
</Sensor>

The schema for the example is:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="Sensor">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="ID" type="xs:string"/>
        <xs:element ref="Observation" />
      </xs:sequence>
    </xs:complexType>
  </xs:element>

  <xs:element name="Observation">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="ID" type="xs:string"/>
        <xs:element ref="Tag" maxOccurs="unbounded" />
      </xs:sequence>
      <xs:attribute name="Command" type="xs:string" use="required" />
    </xs:complexType>
  </xs:element>

  <xs:element name="Tag">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="ID" type="xs:string"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

2.7.1. Schema-Provided XML Events

If you have a XSD schema document available for your XML events, Esper can interrogate the schema. The benefits are:

  • New EPL statements that refer to event properties are validated against the types provided in the schema.

  • Event type metadata becomes available for retrieval as part of the EventType interface.

2.7.1.1. Getting Started

The engine reads a XSD schema file from an URL you provide. Make sure files imported by the XSD schema file can also be resolved.

The configuration accepts a schema URL. This is a sample code snippet to determine a schema URL from a file in classpath:

URL schemaURL = this.getClass().getClassLoader().getResource("sensor.xsd");

Here is a sample use of the runtime configuration API, please see Chapter 14, Configuration for further examples.

epService = EPServiceProviderManager.getDefaultProvider();
ConfigurationEventTypeXMLDOM sensorcfg = new ConfigurationEventTypeXMLDOM();
sensorcfg.setRootElementName("Sensor");
sensorcfg.setSchemaResource(schemaURL.toString());
epService.getEPAdministrator().getConfiguration()
    .addEventType("SensorEvent", sensorcfg);

You must provide a root element name. This name is used to look up the event type for the sendEvent(org.w3c.Node node) method. An EventSender is a useful alternative method for sending events if the type lookup based on the root or document element name is not desired.

After adding the event type, you may create statements and send events. Next is a sample statement:

select ID, Observation.Command, Observation.ID, 
  Observation.Tag[0].ID, Observation.Tag[1].ID
from SensorEvent

As you can see from the example above, property expressions can query property values held in the XML document's elements and attributes.

There are multiple ways to obtain a XML DOM document instance from a XML string. The next code snippet shows how to obtain a XML DOM org.w3c.Document instance:

InputSource source = new InputSource(new StringReader(xml));
DocumentBuilderFactory builderFactory = DocumentBuilderFactory.newInstance();
builderFactory.setNamespaceAware(true);
Document doc = builderFactory.newDocumentBuilder().parse(source);

Send the org.w3c.Node or Document object into the engine for processing:

epService.getEPRuntime().sendEvent(doc);

2.7.1.2. Property Expressions and Namespaces

By default, property expressions such as Observation.Tag[0].ID are evaluated by a fast DOM-walker implementation provided by Esper. This DOM-walker implementation is not namespace-aware.

Should you require namespace-aware traversal of the DOM document, you must set the xpath-property-expr configuration option to true (default is false). This flag causes Esper to generate namespace-aware XPath expressions from each property expression instead of the DOM-walker, as described next. Setting the xpath-property-expr option to true requires that you also configure namespace prefixes as described below.

When matching up the property names with the XSD schema information, the engine determines whether the attribute or element provides values. The algorithm checks attribute names first followed by element names. It takes the first match to the specified property name.

2.7.1.3. Property Expression to XPath Rewrite

By setting the xpath-property-expr option the engine rewrites each property expression as an XPath expression, effectively handing the evaluation over to the underlying XPath implementation available from classpath. Most JVM have a built-in XPath implementation and there are also optimized, fast implementations such as Jaxen that can be used as well.

Set the xpath-property-expr option if you need namespace-aware document traversal, such as when your schema mixes several namespaces and element names are overlapping.

The below table samples several property expressions and the XPath expression generated for each, without namespace prefixes to keep the example simple:

Table 2.5. Property Expression to XPath Expression

Property ExpressionEquivalent XPath
Observeration.ID/Sensor/Observation/ID
Observeration.Command/Sensor/Observation/@Command
Observeration.Tag[0].ID/Sensor/Observation/Tag[position() = 1]/ID

For mapped properties that are specified via the syntax name('key'), the algorithm looks for an attribute by name id and generates a XPath expression as mapped[@id='key'].

Finally, here is an example that includes all different types of properties and their XPath expression equivalent in one property expression:

select nested.mapped('key').indexed[1].attribute from MyEvent

The equivalent XPath expression follows, this time including n0 as a sample namespace prefix:

/n0:rootelement/n0:nested/n0:mapped[@id='key']/n0:indexed[position() = 2]/@attribute

2.7.1.4. Array Properties

All elements that are unbound or have max occurs greater then 1 in the XSD schema are represented as indexed properties and require an index for resolution.

For example, the following is not a valid property expression in the sample Sensor document: Observeration.Tag.ID. As no index is provided for Tag, the property expression is not valid.

Repeated elements within a parent element in which the repeated element is a simple type also are represented as an array.

Consider the next XML document:

<item>
  <book sku="8800090">
    <author>Isaac Asimov</author>
    <author>Robert A Heinlein</author>
  </book>
</item>

Here, the result of the expression book.author is an array of type String and the result of book.author[0] is a String value.

2.7.1.5. Dynamic Properties

Dynamic properties are not validated against the XSD schema information and their result value is always org.w3c.Node. You may use a user-defined function to process dynamic properties returning Node. As an alternative consider using an explicit property.

An example dynamic property is Origin?.ID which will look for an element by name Origin that contains an element or attribute node by name LocationCode:

select Origin?.LocationCode from SensorEvent

2.7.1.6. Transposing Properties

When providing a XSD document, the default configuration allows to transpose property values that are themselves complex elements, as defined in the XSD schema, into a new stream. This behavior can be controlled via the flag auto-fragment.

For example, consider the next query:

insert into ObservationStream
select ID, Observation from SensorEvent

The Observation as a property of the SensorEvent gets itself inserted into a new stream by name ObservationStream. The ObservationStream thus consists of a string-typed ID property and a complex-typed property named Observation, as described in the schema.

A further statement can use this stream to query:

select Observation.Command, Observation.Tag[0].ID from ObservationStream

Before continuing the discussion, here is an alternative syntax using the wildcard-select, that is also useful:

insert into TagListStream
select ID as sensorId, Observation.* from SensorEvent

The new TagListStream has a string-typed ID and Command property as well as an array of Tag properties that are complex types themselves as defined in the schema.

Next is a sample statement to query the new stream:

select sensorId, Command, Tag[0].ID from TagListStream

Please note the following limitations:

  • The XPath standard prescribes that XPath expressions against org.w3c.Node are evaluated against the owner document of the Node. Therefore XPath is not relative to the current node but absolute against each node's owner document. Since Esper does not create new document instances for transposed nodes, transposing properties is not possible when the xpath-property-expr flag is set.

  • Complex elements that have both simple element values and complex child elements are not transposed. This is to ensure their property value is not hidden. Use an explicit XPath expression to transpose such properties.

Esper automatically registers a new event type for transposed properties. It generates the type name of the new XML event type from the XML event type name and the property names used in the expression. The synposis is type_name.property_name[.property_name...]. The type name can be looked up, for example for use with EventSender or can be created in advance.

2.7.1.7. Event Sender

An EventSender sends events into the engine for a given type, saving a type lookup based on element name.

This brief example sends an event via EventSender:

EventSender sender = epRuntime.getEventSender("SensorEvent");
sender.sendEvent(node);

The XML DOM event sender checks the root element name before processing the event. Use the event-sender-validates-root setting to disable validation. This forces the engine to process XML documents according to any predefined type without validation of the root element name.

2.7.1.8. Limitations

The engine schema interrogation is based on the Xerces distribution packaged into Sun Java runtimes. Your application may not replace the JRE's Xerces version and use XML schemas, unless your application sets the DOM implementation registry as shown below before loading the engine configuration:

System.setProperty(DOMImplementationRegistry.PROPERTY, 
  "com.sun.org.apache.xerces.internal.dom.DOMXSImplementationSourceImpl");

2.7.2. No-Schema-Provided XML Events

Without a schema document a XML event may still be queried. However there are important differences in the metadata available without a schema document and therefore the property expression results. These differences are outlined below.

All property expressions against a XML type without schema are assumed valid. There is no validation of the property expression other then syntax validation. At runtime, property expressions return string-type values or null if the expression did not yield a matching element or attribute result.

When asked for property names or property metadata, a no-schema type returns empty array.

In all other aspects the type behaves the same as the schema-provided type described earlier.

2.7.3. Explicitly-Configured Properties

Regardless of whether or not you provide a XSD schema for the XML event type, you can always fall back to configuring explicit properties that are backed by XPath expressions.

For further documentation on XPath, please consult the XPath standard or other online material. Consider using Jaxen or Apache Axiom, for example, to provide faster XPath evaluation then your Java VM built-in XPath provider may offer.

2.7.3.1. Simple Explicit Property

Shown below is an example configuration that adds an explicit property backed by a XPath expression and that defines namespace prefixes:

epService = EPServiceProviderManager.getDefaultProvider();
ConfigurationEventTypeXMLDOM sensorcfg = new ConfigurationEventTypeXMLDOM();
sensorcfg.addXPathProperty("countTags", "count(/ss:Sensor/ss:Observation/ss:Tag)", 
    XPathConstants.NUMBER);
sensorcfg.addNamespacePrefix("ss", "SensorSchema");
sensorcfg.setRootElementName("Sensor");
epService.getEPAdministrator().getConfiguration()
    .addEventType("SensorEvent", sensorcfg);

The countTags property is now available for querying:

select countTags from SensorEvent

The XPath expression count(...) is a XPath built-in function that counts the number of nodes, for the example document the result is 2.

2.7.3.2. Explicit Property Casting and Parsing

Esper can parse or cast the result of your XPath expression to the desired type. Your property configuration provides the type to cast to, like this:

sensorcfg.addXPathProperty("countTags", "count(/ss:Sensor/ss:Observation/ss:Tag)", 
    XPathConstants.NUMBER, "int");

The type supplied to the property configuration must be one of the built-in types. Arrays of built-in type are also possible, requiring the XPathConstants.NODESET type returned by your XPath expression, as follows:

sensorcfg.addXPathProperty("idarray", "//ss:Tag/ss:ID", 
    XPathConstants.NODESET, "String[]");

The XPath expression //ss:Tag/ss:ID returns all ID nodes under a Tag node, regardless of where in the node tree the element is located. For the example document the result is 2 array elements urn:epc:1:2.24.400 and urn:epc:1:2.24.40.

2.7.3.3. Node and Nodeset Explicit Property

An explicit property may return XPathConstants.NODE or XPathConstants.NODESET and can provide the event type name of a pre-configured event type for the property. The method name to add such properties is addXPathPropertyFragment.

This code snippet adds two explicit properties and assigns an event type name for each property:

sensorcfg.addXPathPropertyFragment("tagOne", "//ss:Tag[position() = 1]", 
    XPathConstants.NODE, "TagEvent");
sensorcfg.addXPathPropertyFragment("tagArray", "//ss:Tag", 
    XPathConstants.NODESET, "TagEvent");

The configuration above references the TagEvent event type. This type must also be configured. Prefix the root element name with "//" to cause the lookup to search the nested schema elements for the definition of the type:

ConfigurationEventTypeXMLDOM tagcfg = new ConfigurationEventTypeXMLDOM();
tagcfg.setRootElementName("//Tag");
tagcfg.setSchemaResource(schemaURL);
epAdministrator.getConfiguration()
    .addEventType("TagEvent", tagcfg);

The tagOne and tagArray properties are now ready for selection and transposing to further streams:

insert into TagOneStream select tagOne.* from SensorEvent

Select from the new stream:

select ID from TagOneStream

An example with indexed properties is shown next:

insert into TagArrayStream select tagArray as mytags from SensorEvent

Select from the new stream:

select mytags[0].ID from TagArrayStream

2.8. Additional Event Representations

Part of the extension and plug-in features of Esper is an event representation API. This set of classes allow an application to create new event types and event instances based on information available elsewhere, statically or dynamically at runtime when EPL statements are created. Please see Section 16.8, “Event Type And Event Object” for details.

Creating a plug-in event representation can be useful when your application has existing Java classes that carry event metadata and event property values and your application does not want to (or cannot) extract or transform such event metadata and event data into one of the built-in event representations (POJO Java objects, Map or XML DOM).

Further use of a plug-in event representation is to provide a faster or short-cut access path to event data. For example, access to event data stored in a XML format through the Streaming API for XML (StAX) is known to be very efficient. A plug-in event representation can also provide network lookup and dynamic resolution of event type and dynamic sourcing of event instances.

Currently, EsperIO provides the following additional event representations:

  • Apache Axiom: Streaming API for XML (StAX) implementation

Please see the EsperIO documentation for details on the above.

The chapter on Section 16.8, “Event Type And Event Object” explains how to create your own custom event representation.

2.9. Updating, Merging and Versioning Events

To summarize, an event is an immutable record of a past occurrence of an action or state change, and event properties contain useful information about an event.

The length of time an event is of interest to the event processing engine (retention time) depends on your EPL statements, and especially the data window, pattern and output rate limiting clauses of your statements.

During the retention time of an event more information about the event may become available, such as additional properties or changes to existing properties. Esper provides three concepts for handling updates to events.

The first means to handle updating events is the update istream clause as further described in Section 5.20, “Updating an Insert Stream: the Update IStream Clause”. It is useful when you need to update events as they enter a stream, before events are evaluated by any particular consuming statement to that stream.

The second means to update events is the on-merge and on-update clauses, for use with named windows only, as further described in Section 5.15.9, “Triggered Upsert using the On-Merge Clause” and Section 5.15.7, “Updating Named Windows: the On Update clause”. On-merge is similar to the SQL merge clause and provides what is known as an "Upsert" operation: Update existing events or if no existing event(s) are found then insert a new event, all in one atomic operation provided by a single EPL statement. On-update can be used to update individual properties of events held in a named window.

The third means to handle updating events is the revision event types, for use with named windows only, as further described in Section 5.15.11, “Versioning and Revision Event Type Use with Named Windows”. With revision event types one can declare, via configuration only, multiple different event types and then have the engine present a merged event type that contains a superset of properties of all merged types, and have the engine merge events as they arrive without additional EPL statements.

Note that patterns do not reflect changes to past events. For the temporal nature of patterns, any changes to events that were observed in the past do not reflect upon current pattern state.

2.10. Coarse-Grained Events

Your application events may consist of fairly comprehensive, coarse-grained structures or documents. For example in business-to-business integration scenarios, XML documents or other event objects can be rich deeply-nested graphs of event properties.

To extract information from a coarse-grained event or to perform bulk operations on the rows of the property graph in an event, Esper provides a convenient syntax: When specifying a filter expression in a pattern or in a select clause, it may contain a contained-event selection syntax, as further described in Section 5.19, “Contained-Event Selection”.

2.11. Event Objects Instantiated and Populated by Insert Into

The insert into clause can populate instantiate new instances of Java object events and java.util.Map events directly from the results of select clause expressions and populate such instances. Simply use the event type name as the stream name in the insert into clause as described in Section 5.10, “Merging Streams and Continuous Insertion: the Insert Into Clause”.

If instead you have an existing instance of a Java object returned by an expression, such as a single-row function or static method invocation for example, you can transpose that expression result object to a stream. This is described further in Section 5.10.7, “Transposing an Expression Result” and Section 9.4, “Select-Clause transpose Function”.

The column names specified in the select and insert into clause must match available writable properties in the event object to be populated (the target event type). The expression result types of any expressions in the select clause must also be compatible with the property types of the target event type.

If populating a POJO-based event type and the class provides a matching constructor, the expression result types of expressions in the select clause must be compatible with the constructor parameters in the order listed by the constructor. The insert into clause column names are not relevant in this case.

Consider the following example statement:

insert into com.mycompany.NewEmployeeEvent 
select fname as firstName, lname as lastName from HRSystemEvent

The above example specifies the fully-qualified class name of NewEmployeeEvent. The engine instantianes NewEmployeeEvent for each result row and populates the firstName and lastName properties of each instance from the result of select clause expressions. The HRSystemEvent in the example is assumed to have lname and fname properties, and either setter-methods and a default constructor, or a matching constructor.

Note how the example uses the as-keyword to assign column names that match the property names of the NewEmployeeEvent target event. If the property names of the source and target events are the same, the as-keyword is not required.

The next example is an alternate form and specifies property names within the insert into clause instead. The example also assumes that NewEmployeeEvent has been defined or imported via configuration since it does not specify the event class package name:

insert into NewEmployeeEvent(firstName, lastName) 
select fname, lname from HRSystemEvent

Finally, this example populates HRSystemEvent events. The example populates the value of a type property where the event has the value 'NEW' and populates a new event object with the value 'HIRED', copying the fname and lname property values to the new event object:

insert into HRSystemEvent 
select fname, lname, 'HIRED' as type from HRSystemEvent(type='NEW')

The matching of the select or insert into-clause column names to target event type's property names is case-sensitive. It is allowed to only populate a subset of all available columns in the target event type. Wildcard (*) is also allowed and copies all fields of the events or multiple events in a join.

For Java object events, your event class must provide setter-methods according to JavaBean conventions or, alternatively, a matching constructor. If the event class provides setter methods the class should also provide a default constructor taking no parameters. If the event class provides a matching constructor there is no need for setter-methods. If your event class does not have a default constructor and setter methods, or a matching constructor, your application may configure a factory method via ConfigurationEventTypeLegacy.

The engine follows Java standards in terms of widening, performing widening automatically in cases where widening type conversion is allowed without loss of precision, for both boxed and primitive types and including BigInteger and BigDecimal.

When inserting array-typed properties into a Java or Map-type underlying event the event definition should declare the target property as an array.

Please note the following limitations:

  • Event types that utilize XML org.w3c.dom.Node underlying event objects cannot be target of an insert into clause.

2.12. Comparing Event Types

Each of the event representations of Java object, Map and XML document has advantages and disadvantages that are summarized in the table below:

Table 2.6. Comparing Event Representations

 Java Object (POJO/Bean or other)MapXML Document
PerformanceGoodGoodNot comparable and depending on use of XPath
Memory UseGoodGoodDepends on DOM and XPath implementation used
Call Method on EventYesYes, if contains Object(s)No
Nested, Indexed, Mapped and Dynamic PropertiesYesYesYes
Course-grained event syntaxYesYesYes
Insert-into that RepresentationYesYesNo
Runtime Type ChangeReload class, yesYesYes
Create-schema SyntaxYesYesNo, runtime and static configuration

© 2011 EsperTech Inc. All Rights Reserved