Configurable Data Correction

Implementers can use plug-in driven batches to perform some types of one-off data corrections but the process lacks the built in approval step supported by the request based data correction flows. Moreover, each custom plug-in driven batch involves creating a few custom algorithms, scripts and a batch control, all generated for a one-off update.

The product supports a simplified way to request, approve and perform plug-in driven based data corrections.

The following points describe the functionality at a high level:

  • A data correction request is created to capture the following details:
    • An SQL query that selects the records to correct.
    • Either a business object or a service script to perform the data correction for each selected record.
    • The number of threads to submit the data correction batch process with.

  • When the request is approved and a user submits it for processing, a batch job is created for the specific request. The batch job uses the query on the request to select the records to correct and calls the corresponding business object or script to perform the update for each record. The request remains in the 'Submitted' status while the batch process runs.

  • When the batch completes, it updates the corresponding request to the 'Processed' status, which then creates a monitoring To Do Entry for a user to review the results of the batch. The user marks the request as complete and if further corrections are needed the user can create another request with different correction details as needed.

The following sections provide more information about this tool.

Request Type Setup

A "Data Correction via Batch" (F1-DataCorrectTypeByBatch) Request Type business object is provided to define a request type record for creating such requests. To enable this functionality, a request type should be created using this business object.

The request type should reference the "Configurable Data Correction" (F1-DataCorrectPlugIn) as the request business object. This business object is a subclass of the data correction root business object and as such leverages the same approval based lifecycle support.

Data Correction Implementation

The following is a suggested description for a one-off data correction procedure:

  • If the change is not a simple update of fields using a business object, implement a service script that gets the prime key of a selected record and uses standard scripting logic to perform the update. The script can be tested using the Test action on the Script page.

  • Use any tool to compose and test an SQL that selects the records from a primary table of a maintenance object to correct.

  • If the change is a simple update of non-list fields on a business object, note the xpath of the primary key fields and the xpath of each element that needs to be updated on that business object. Adjust the SQL to set the xpath element name as the alias of each field.

    This is an example of a data correction SQL that selects some batch control records and assigns each with a new "next batch run number". The "batchControl" and "nextBatchNumber" aliases are xpath references to the corresponding fields in the F1-BatchControlPhysicalBO business object. 
    select batch_cd "batchControl", 
case
when batch_cd = '...' then <new next batch run number>
...
else ...
end "nextBatchNumber"
from ci_batch_ctrl 
where batch_cd in ('.... list of batch control records to update ... ')

    Note:

    The SQL is provided for illustration purposes only.
  • If the change is not as simple, implement a service script that gets the fields from a selected record and uses standard scripting logic to perform the update.

    • The script's schema must include the F1-DataCorrectionProcessRecord data area. It is via this data area that your script gets the selected field from each row to update. The structure of this data area is similar to the plug-in driven Process Record plug-in spot.

      All selected fields are provided to your script in the 'selectFields' list of the data area. In addition, the primary key of the selected record may also be provided via the 'mo', and 'pk1-5' elements, if the 'MAINT_OBJ_CD', and 'PK_VALUE1-5' designated aliases are used in your query.

      For example:
      select 'F1-FACT' MAINT_OBJ_CD, fact_id PK_VALUE1, ... 
from f1_fact ...

    • The script can be tested using the 'Test' action on the Script page.

  • Create a 'Configurable Data Correction' Request using the corresponding Request Type. Specify on the request the SQL and the business object or script to use for data correction.
  • Progress the request through approval and submit for processing when ready.
  • Review the data once the submitted batch is complete.

Approving and Submitting the Request

Once a data correction request is captured, there are several key steps to perform.

  • If the deletion details are correct, the request can be transitioned to Ready for Approval. A to do entry is created to notify potential approvers. The request must be approved by someone other than the user who created the record. The notification to do entry type and role are configured on the Request Type.

  • The request must be approved before it can be submitted for processing.
  • Once the request is in the Approved state, it can be submitted. The Submitted state is configured to be processed by a plug-in driven batch process that uses details from the request to perform the deletion in background. The batch job is submitted automatically. When batch job completes, the request transitions to the Processed state so that the results can be reviewed.
  • A user marks the request as complete and if further corrections are needed the user can create another request with different correction details as needed.

Parent topic: Configuring Requests