5.6.2 Use and Execute the Source Data Quality Check Process

Use this Run Pipeline (Process) to perform the Data Quality Checks on source records for various data elements.

To use and execute the Source Data Quality Check Process in the Process Orchestration, do the following:

1. To access the Source Data Quality Check Process Pipeline, on the Home Page, select the Process Orchestration. The Process Modeller Page is displayed.
2. On the Process Modeller Page, search and select the Source Data Quality Check Process. The Process Flow Page is displayed. This Process Flow is designed on the Drawing Canvas using the Transition, Activity, and Widgets Components available in the floating toolbar. RUN DQ RULE Widgets representing Data Quality Groups are set up in parallel to each other. A Data Service Widget called as Data Quality Reporting Engine is added at the end meant for reporting Data Quality Checks.
3. To view the details of any Widget, double-click on the Widget and the details related to its Activity, Transition, and Notification are displayed. On the drawing canvas, you can select and see the Definition, Data Fields, and Application Rule details.
4. To execute the Run, you can select the Run Parameter Values using the Execution Button on the Process Flow Page or on the Process Modeller Page.
   Go to the Process Modeller Page to execute the Run. Click the Menu Button corresponding to the Source Data Quality Check Process that needs to be executed. Click Execute Run. The Execution Page is displayed.
5. On the Execution Page, to execute the Run with parameters, select With Parameters in the Execution Type List. Select the required As of Date and Incremental Execution from the drop down list for which the Data Quality Checks need to be processed. The Incremental execution can be Yes if the newly ingested records must be validated else select as No. Click the Execute Button to initiate the Run Pipeline execution.
   Beginning with 26C Release, AFCS supports Incremental Data Quality (DQ) Execution, which enables data quality validation to be performed only on newly ingested records instead of the entire dataset. This feature reduces execution time and improves overall processing efficiency.

   • To use Incremental DQ execution in existing custom DQ pipelines in the 26C release, create the Incremental DQ data field and then enable the Incremental DQ parameters using the Incremental AOM API. After these configurations are complete, the Incremental DQ execution options become available in the user interface. For more information, see the Incremental DQ Parameter Configuration and Incremental DQ Data Field Configuration tables.
   • To use Incremental DQ execution in the new custom DQ pipelines in the 26C release, enable the Incremental DQ parameters using the Incremental AOM API to make Incremental DQ execution available in the user interface. For more information, see the Incremental DQ Parameter Configuration table.

   Create the Incremental DQ data field with the following information in the PMF. For more information, see the Data Fields section in the AFCS User Guide.

   Table 5-7 Incremental DQ Data Field Configuration

   Fields	Description
   Data Field Code	IS_INCREMENTAL_EXE
   Data Field Name	Incremental Execution
   Data Field Description	Incremental Execution
   Data Field Type	AOM
   Is Mandatory	Yes
   Scope	Process

   With the 26C release, the following parameters have been introduced for the Incremental DQ feature. To make these parameters available in the user interface, users must configure them using the Incremental AOM API.

   Table 5-8 Incremental DQ Parameter Configuration

   Parameter Identifier	Parameter Name
   120010	Incremental Execution

   For more information on the Incremental AOM Parameter Process, see the AFCS API User Guide.

   Note:

   The execution of the Run Pipeline is triggered using the selected Extraction Date.

   See the Process Orchestration Section for more details about the Process Orchestration Framework.

6. To verify the Run Execution of the Source Data Quality Check Process, do the following:
   1. To open the Process Monitor Page, on the Process Modeller Page, click the Process Monitor Button or select Process Flow Monitor on the Process Modeller Menu.
   2. The Process Monitor Page is displayed, which lists all the Run Instances corresponding to the Source Data Quality Check Process. On the Process Monitor Page, search by the Process ID, or by the Process Name Source Data Quality Check Process, and select the Process Instance for the required Run Pipeline (Process) that was executed
7. The Process Flow Page is displayed with the Run Execution Status on each Node of the Source Data Quality Check Process.
8. To verify the Run Execution Logs, do the following:
   1. On the Process Monitor Page, click the required Process Instance for which you need to verify the Execution Logs. The Process Flow Page is displayed with the Run Execution Status on each Node.
   2. To see the Execution Status details of a Node, double-click on that Node. The Execution Status details Page is displayed. Click Execution Logs. The Log Viewer Page is displayed, which lists all the Logs related to the Process Instance. To see the details of a log entry, click the Show More Button. Click outside the Log Viewer Page to close it.