2.4.5.2 Use and Execute the Source Data Quality Check Process

Use this Run Pipeline (Process) to generate and view the Data Quality report for any of the Data Quality run execution. The Data Quality Reporting Engine Pipeline uses the 'As of Date' and the 'Run Identifier' parameters to generate the Data Quality reports for run executions which are in 'Completed' status either for a passed or a failed run execution.

Note:

Once the DQ Report has been generated for the failed DQ run pipeline, post the execution of this PMF, users must provide the Process Instance ID of the failed DQ run to the public API to view the DQ Groups that are breaching the threshold limit. For more information, see Data Operations Guide.

1. To access the Source Data Quality Check Process Pipeline, select the Process Orchestration from the home page.
   The Process Modeller page is displayed.
2. On the Process Modeller page, search and select the Source Data Quality Check Process Pipeline.
   The Process Flow page is displayed. This Process Flow is designed on the Drawing Canvas using the Transition, Activity, and Widgets Components available in the floating toolbar. RUN DQ RULE widgets representing Data Quality Groups are set up in parallel to each other. A Data Service widget called as Data Quality Reporting Engine is added at the end meant for reporting Data Quality Checks.
3. To view the details of any widget, double-click on the widget and the details related to its Activity, Transition, and Notification are displayed. On the drawing canvas, you can select and see the Definition, Data Fields, and Application Rule details.
4. To execute the Run, you can select the Run Parameter Values using the Execution button on the Process Flow page or on the Process Modeller page.
5. Go to the Process Modeller page to execute the Run.
6. Click the menu button corresponding to the Source Data Quality Check Process that needs to be executed.
7. Click Execute Run.
   The Execution page is displayed.
8. On the Execution page, to execute the Run with parameters, select With Parameters in the Execution Type list.
9. Select the required As of Date for which the Data Quality Checks need to be processed.
10. Click Apply to initiate the Run Pipeline execution.

    Note:

    The execution of the Run Pipeline is triggered using the selected Extraction Date. See the Process Orchestration section for more details about the Process Orchestration framework.
    To verify the Run Execution of the Source Data Quality Check Process, do the following:
    1. To open the Process Monitor page, on the Process Modeller page, click the Process Monitor button or select Process Flow Monitor on the Process Modeller menu.
       The Process Monitor page is displayed, which lists all the Run Instances corresponding to the Source Data Quality Check Process.
    2. On the Process Monitor page, search by the Process ID, or by the Process Name Source Data Quality Check Process, and select the Process Instance for the required Run Pipeline (Process) that was executed.
11. The Process Flow page is displayed with the Run Execution Status on each Node of the Source Data Quality Check Process.
12. To verify the Run Execution Logs, do the following:
    1. On the Process Monitor page, click the required Process Instance for which you need to verify the Execution Logs. The Process Flow page is displayed with the Run Execution Status on each Node.
    2. To see the Execution Status details of a Node, double-click on that Node. The Execution Status details page is displayed.
    3. Click Execution Logs.
       The Log Viewer page is displayed, which lists all the Logs related to the Process Instance. To see the details of a log entry, click the Show More Button. Click outside the Log Viewer page to close it.