Correlation and Filter Operators

Correlation enables you to obtain data based on the relationship of two pieces of existing data. Filtering enables you to provide information to obtain a subset of data you want.

The following table lists the input and output for each operator.

Relation Map

The Relation Map operator performs a select on two or more incoming relations, equivalent to a SQL join view of a minimum of one relation and additional tables and relations. The Relation Map operator can take multiple inputs.

You can use a Relation Map to join input from other operator sources, for example, from two or more tuple based windows, or from two or more partitioned windows.

For example, with the latest two-hour window of stock transactions and the latest two-hour window of trader information as input, you can compute the latest two-hour window of possible trades by joining the trader's name with the name provided in the trader information.

To Create a Relation Map Operator

1. Drag a Relation Map operator from the Palette to the Design view.

2. Connect at least two inputs to the Relation Map operator.

3. Double-click the Relation Map operator.

   The property editor opens.

4. In the Expression box, enter the expression for the SQL SELECT statement. Add attribute names, data types and sizes, as applicable. Add, delete, or move attributes.

5. In the From field, define the input on which to perform the selection.

6. In the Where clause field, provide additional search criteria.

7. Click OK.

Stream Projection and Filter

The Stream Projection and Filter operator enables you to join a stream with multiple relations and tables, in order to create new events or to filter existing events based on specified conditions.

The input to the Stream Projection and Filter operator must include one (and only one) stream. The input can also include any number of relations and tables.

The output from the Stream Projection and Filter operator is a stream.

Configuring the Stream Projection and Filter operator resembles the process of writing a SQL SELECT statement. The property editor includes a SELECT area, a FROM area, and a WHERE area. The property editor also includes a tree structure that contains the input operators and their attributes.

In the SELECT area, you populate a table with one or more attributes. You can quickly add an existing attribute by selecting the attribute in the tree structure and dragging it to the Expression column. You can create a new attribute by using a mathematical computation in the Expression column. In the following example, two existing attributes are multiplied to create the new attribute.

total=Products.price*Products.tax

You can remove an attribute from the input stream by not including it in the SELECT area.

In the WHERE area, you can specify a search condition in the form of a boolean value expression. The boolean value expression can include the following types of predicates: comparison, between, in, like, null, quantified comparison, and exists.

One of the IEP tutorials uses the Stream Projection and Filter operator to check whether the price of each stock transaction is significantly different from the average price of all stock transactions during the last two minutes. The operator contains the following WHERE clause:

StockTransactions.symbol=RollingAvgPrice.symbol AND
(StockTransactions.price > 1.1 * RollingAvgPrice.price OR
StockTransactions.price < 0.9 * RollingAvgPrice.price)

To Create a Stream Projection and Filter Operator

1. Drag a Stream Projection and Filter operator from the Palette to the Design view.

2. Connect the output of an operator that has stream output to the input of the Stream Projection and Filter operator.

3. (Optional) Connect the output of one or more operators that have relation output to the input of the Stream Projection and Filter operator.

4. (Optional) Connect the output of one or more operators that have table output to the input of the Stream Projection and Filter operator.

5. Double-click the Stream Projection and Filter operator.

   The property editor opens.

6. If you want to change the default name, then change the value in the Name field.

7. In the SELECT area, populate the table with one or more attributes.

   By default, the SELECT area contains only one attribute row. To create additional rows, click Add Attribute.

   You can drag and drop attributes from the tree structure into the Expression column. If you use this approach, then the operator name is automatically added to the FROM area.

8. In the FROM area, specify the names of one or more input operators.

   You can drag and drop operators from the tree structure into the FROM area.

   You can specify an alias for a table (for example, TableInput0 t) and then use the alias in the WHERE clause. You cannot specify an alias for a stream or a relation.

   If you specify more than one operator name, then separate the names with a comma (for example, StockTransactions,RollingAvgPrice).

9. (Optional) In the WHERE area, specify a search condition in the form of a boolean value expression.

   You can drag and drop operators and attributes from the tree structure into the WHERE area.

10. Click OK.

Tuple Serial Correlation

The Tuple Serial Correlation operator provides correlation of sequential events; that is, the operator uses a specification of a number of events from an existing stream to create a stream that contains larger events.

Assume that you have a stream of transactions of a stock, and you want to compute a new stream in which each event is composed of three consecutive events from the original stream. Using the latest two stock prices as output, you can build a trading model that predicts the next stock prices.

To Create a Tuple Serial Correlation Operator

1. Drag a Tuple Serial Correlation operator from the Palette to the Design view.

2. Double-click the Tuple Serial Correlation operator.

   The property editor opens.

3. In the Increment field, enter how often you want to obtain events via the Tuple Serial Correlation operator, based on the number of events.

4. In the Size field, specify the number of small consecutive events that you want to add to the larger cumulative event.

5. Click OK.