Stage Details

Click a Name in the Name column of the Pipeline Stages table to navigate to stage details page.

This page provides details about all the transformations in specific stage.

• Pipeline ID: Unique pipeline id in Spark Cluster

• Pipeline Name: Name of Oracle Stream Analytics Pipeline given by user in Oracle Stream Analytics UI.

• Stage ID: Unique stage id in DAG of stages for Oracle Stream Analytics Pipeline.

• Stage DAG: This is a visual representation of all transformations in form of a DAG where it displays the parent-child relation various pipeline transformations.

  Note Oracle Stream Analytics allows different transformation types including Business Rules but drilldowns are allowed only on CQLDStream transformations. Inside CQLDStream transformation, the input data is transformed using a continuously running query (CQL) in a CQL Engine. Note that there will be one CQL engine associated with one Executor.

• Stage Transformations Table:This table displays details about all transformations being performed in a specific stage. Each entry in the table is corresponding to a transformation operation used in computation of the stage. You can observe that final transformation in every stage is MonitorDStream. The reason is that MonitorDStream pipes output of stage to Oracle Stream Analytics UI Live Table.

  • Transformation Name: Name of output DStream for the transformation. Every transformation in spark results into an output dstream.

  • Transformation Type: This is category information of each transformation being used in the stage execution. If transformation type is "Oracle", it is based on Oracle's proprietary transformation algorithm. If the transformation type is "Native", then transformation is provided by Apache Spark implementation.

CQLDStream transformation allows further drill down as described in the Query Details section.