Adding Stages to a Pipeline

Adding a Query Stage

You can include simple or complex queries on the data stream without any coding to obtain refined results in the output.

1. Open a pipeline in the Pipeline Editor.
2. Right-click the stage after which you want to add a query stage, click Add a Stage, and then select Query.
3. Enter a Name and Description for the Query Stage.
4. Click Save.

Adding a Filter to a Query Stage

You can add filters in a pipeline to obtain more accurate streaming data.

To add a filter:

1. Open a pipeline in the Pipeline Editor.
2. Select the required query stage.
3. Navigate to the Filters tab.
4. Click Add a Filter.
5. Select the required column and a suitable operator and value.

   You can also calculate fields within filters.

   Note:

   IN operator is available as an operator in the drop-down list. This operator is not supported for Interval, Interval YM, Timestamp, and SDO Geometry datatypes.

   You can use the IN filter to refer to a column in a database table. When you change the database column values at runtime, the pipeline picks up the latest values from the DB column, without republishing the pipeline.

6. Click Add a Condition to add and apply a condition to the filter.
7. Click Add a Group to add nested conditions.
8. Repeat these steps for as many filters, conditions, or groups as you want to add.

   You can create blocks without adding condition expression, which you can add at any later stage.

   Link the blocks using AND/ OR

   Define complex conditions.

   Example:

   

Adding a Summary to a Query Stage

To add a summary:

1. Open a pipeline in the Pipeline Editor.
2. Select the required query stage and click the Summaries tab.
3. Click Add a Summary.
4. Select the suitable function and the required column.
5. Repeat the above steps to add as many summaries you want.

Adding a Summary with Group By

To add a group by:

1. Open a pipeline in the Pipeline Editor.
2. Select the required query stage and click the Summaries tab.
3. Click Add a Group By.
4. Click Add a Field and select the column on which you want to group by.

When you create a group by, the live output table shows the group by column alone by default. Turn ON Retain All Columns to display all columns in the output table.

You can add multiple group by's.

Adding a Query Group Stage

A query group is a combination of summaries (aggregation functions), group-bys, filters and a range window. Different query groups process your input in parallel and the results are combined in the query group stage output. You can also define input filters that process the incoming stream before the query group logic is applied, and result filters that are applied on the combined output of all query groups together.

A query group stage of the stream type applies processing logic to a stream. It is in essence similar to several parallel query stages grouped together for the sake of simplicity.

A query group stage of the table type can be added to a stream containing transactional semantic. For example, change data capture stream produced by the Oracle GoldenGate BigData plugin. The stage of this type will recreate the original database table in memory using the transactional semantics contained in the stream. You can then apply query groups to this table in memory, to run real-time analytics on your transactional data, without affecting the performance of your database.

Adding Query Group: Stream

You can apply aggregate functions with different groupbys and window ranges to your streaming data.

To add a query group stage of type stream:

1. Open a pipeline in the Pipeline Editor.
2. Right-click the stage after which you want to add a query group stage, click Add a Stage, select Query Group, and then Stream.

   You can add a query stage group only at the end of the pipeline.

3. Enter a name and a description for the query group stage of the type stream and click Save.

   The query group stage of the type stream appears in the pipeline.

4. On the Input Filters tab, click Add a Filter. See Adding a Filter to a Query Stage.

   These filters process data before it enters the query group stage. Hence, you can only see fields of the original incoming shape.

5. On the Groups tab, click Add a Group. A group can consist one or many of summaries, filters, and group bys.
   See Adding a Summary to a Query Stage and Adding a Summary with GroupBy.
6. Repeat the previous step to add as many groups as you want.
7. On the Result Filters tab, click Add a Filter to filter the results.

   These filters process data before it exits the query group stage. Hence, you can see a combined set of fields in the outgoing shape.

8. On the Visualizations tab, click Add a Visualization and add the required type of visualization. See Adding Chart Visualizations.

Adding Query Group: Table

You can apply aggregate functions with different groupbys and window ranges to a database table data recreated in memory.

To add a query group stage of the type table:

1. Open a pipeline in the Pipeline Editor.
2. Right-click the stage after which you want to add a query group stage, click Add a Stage, select Query Group, and then Table.
3. Enter a name and a description for the Query Group Table and click Next.
4. On the Transactions Settings screen, select a column in the Transaction Field drop-down list.

   The transaction column is a column from the output of the previous stage that carries the transaction semantics (insert/update/delete). Make sure that you use the values that correspond to your change data capture dataset. The default values work for Oracle GoldenGate change data capture dataset.

5. On the Field Mappings screen, select the columns that carry the before and after transaction values from the original database table. For example, in case of Oracle GoldenGate, the before and after values have before_ and after_ as prefixes, respectively. Specify a column as primary key in the table.
6. Click Save to create a query group stage of the type table.
   You can see the table configuration that you have specified while creating the table stage in the Table Configuration tab.
7. On the Input Filters tab, click Add a Filter. See Adding a Filter to a Query Stage.
8. On the Groups tab, click Add a Group. A group can consist one or many of summaries, filters, and groupbys.
   See Adding a Summary to a Query Stage and Adding a Summary with GroupBy.
9. Repeat the previous step to add as many groups as you want.
10. On the Result Filters tab, click Add a Filter to filter the results.
11. On the Visualizations tab, click Add a Visualization and add the required type of visualization. See Adding Chart Visualizations.

Adding a Rule Stage

Using a rule stage, you can add the IF-THEN logic to your pipeline. A rule is a set of conditions and actions applied to a stream. There is no specific sequence to add rules.

To add a rule stage:

1. Open a pipeline in the Pipeline Editor.
2. Right-click the stage after which you want to add a rule stage, click Add a Stage, and then select Rule.
3. Enter a Name and Description for the rule stage.
4. Click Add a Rule.
5. Enter Rule Name and Description for the rule and click Done to save the rule.
6. Select a suitable condition in the IF statement, THEN statement, and click Add Action to add actions within the business rules.
   Actions can also be expressions. For example, SET Revenue TO =-Revenue, will convert the current value of Revenue to a negative number.
   Expressions must always start with a '=' sign. For a constant text value, just type in the text. For example, SET CustomerType TO GOLD.

The rules are applied to the incoming events one by one and actions are triggered if the conditions are met.

Adding a Pattern Stage

A pattern is a template of an Oracle GoldenGate Stream Analytics application, with a business logic built into it. You can create pattern stages within the pipeline. Patterns are not stand-alone artifacts, they need to be embedded within a pipeline.

For detailed information about the various type of patterns, see Transforming and Analyzing Data using Patterns.

To add a pattern stage:

1. Open a pipeline in the Pipeline Editor.
2. Right-click the stage after which you want to add a pattern stage, click Add a Stage, and then select Pattern.
3. Choose the required pattern from the list of available patterns.
4. Enter a Name and Description for the pattern stage.
   The selected pattern stage is added to the pipeline.
5. Click Parameters and provide the required values for the parameters.
6. Click Visualizations and add the required visualizations to the pattern stage.

Adding a Scoring Stage

To add a scoring stage:

1. Open the required pipeline in Pipeline Editor.
2. Right-click the stage after which you want to add a scoring stage, click Add a Stage, and then select Scoring.
3. Enter a meaningful name and suitable description for the scoring stage and click Save.
4. In the stage editor, enter the following details:
   1. Model name: Select the predictive model that you want to use in the scoring stage
   2. Model Version: Select the version of the predictive model
   3. Mapping: Select the corresponding model fields that appropriately map to the stage fields

You can add multiple scoring stages based on your use case.

Adding a Target Stage

To add a target stage:

1. Open the required pipeline in Pipeline Editor.
2. Right-click the stage after which you want to add a target stage, click Add a Stage, and then select Target.
3. Enter a name and suitable description for the target.
4. Click Save.

For more information on creating different target types, see unresolvable-reference.html#GUID-F8E38E61-6F76-46E9-A405-E4F612171E54.

Adding a Custom CQL Stage

To add a custom stage:

1. Open the required pipeline in Pipeline Editor.
2. Right-click the stage after which you want to add a custom stage. Click Add a Stage, and Custom, and then select Custom CQL.
3. Enter a name and suitable description for the custom stage and click Save.
4. Type your custom CQL query in the right pane of the pipeline editor.