Run BDF Ingestion in Parallel for Multiple Batches

Start Mantas Batch. This can be a default batch (DLY).

Set Mantas Date

Create a copy of KDD_PRCSNG_BATCH_CONTROL for each batch you plan to run in parallel. The KDD_PRCSNG_BATCH_CONTROL table will need to be suffixed with

<_first two characters of the batch
                    name>

.

This has to be done only once, unless a new batch is being added.

Example: To create copy of KDD_PRCSNG_BATCH_CONTROL for the France (FR) batch, run

CREATE TABLE KDD_PRCSNG_BATCH_CONTROL_FR AS SELECT * FROM KDD_PRCSNG_BATCH_CONTROL WHERE 1=2;

Insert data for running multiple batches into

“KDD_PRCSNG_BATCH”,
                        “KDD_PRCSNG_BATCH_CONTROL_<first two characters of the batch name>” and
                        “KDD_PRCSNG_BATCH_SRC

” tables.

Example: Insert for France (FR) batch.

 INSERT INTO KDD_PRCSNG_BATCH (PRCSNG_BATCH_NM,PRCSNG_BATCH_DSPLY_NM, PRCSNG_ORDER, EOD_BATCH_NM, PRCSNG_BATCH_DESC)
VALUES ('FR','France Batch',11,'FR', NULL);  
INSERT INTO KDD_PRCSNG_BATCH_CONTROL_FR (PRCSNG_BATCH_ID, DATA_DUMP_DT, PRCSNG_BATCH_NM, EOD_PRCSNG_BATCH_FL)
VALUES (100, '10-DEC-15', 'FR', 'Y');  
INSERT INTO KDD_PRCSNG_BATCH_SRC (PRCSNG_BATCH_NM, SRC_ORIGIN, SRC_DESC)
VALUES ('FR', 'FR', 'France');
  COMMIT;

Note:

The KDD_PRCSNG_BATCH_CONTROL_<first two characters of the batch name> table must be truncated as per Step 12. For each subsequent batch run, insert the data for the next day’s run into


                            KDD_PRCSNG_BATCH_CONTROL_<first two characters of the batch
                            name>

table.

The KDD_PRCSNG_BATCH_JRSDCN_MAP table is specifically used by the Watchlist Management UI. It includes mapping between PRCSNG_BATCH_NM and JRSDCN_CD. Insert relevant data into the KDD_PRCSNG_BATCH_JRSDCN_MAP table

 Example:

 INSERT INTO KDD_PRCSNG_BATCH_JRSDCN_MAP (PRCSNG_BATCH_NM, JRSDCN_CD)
 VALUES ('FR', 'EMEA');  
INSERT INTO KDD_PRCSNG_BATCH_JRSDCN_MAP (PRCSNG_BATCH_NM, JRSDCN_CD)
VALUES ('SG', 'APAC');
  COMMIT;

Create the following directories under $FIC_HOME/bdf/config:

datamaps_org
queries_org

Grant 755 permission to these directories.

Copy the original BDF datamaps from $FIC_HOME/bdf/config/datamaps to

 $FIC_HOME/
                        bdf/config/datamaps_org

directory and CSA queries from


                        $FIC_HOME/bdf/config/ queries

to $FIC_HOME/bdf/config/queries_org directory (this is a onetime activity).

Note:

If a patch is released for BDF datamap or queries then the patch will place the updated files in the datamaps or queries directory. Once the patch is applied, you must copy the datamap xml file from the datamaps folder to datamaps_org folder and SQL query from queries folder to queries_org folder manually.

There are six sub-directories under the $FIC_HOME/bdf/config/derivations directory. For each of these sub-directories, create a copy (along with contents of the sub-directory) of and suffix <_first two characters of the batch name> to it.

Example: For France (FR) batch the sub-directory account will be copied (along with its contents) as Account_FR. If your batch is for Singapore (SG) the sub-directory will be copied as Account_SG.

If your implementation uses a custom BDF datamap parameter file under $FIC_HOME/bdf/ config/custom directory, copy the custom file and suffix

<_first two characters of the batch
                        name>.xml

.

Example: The custom parameter file for an account should be copied as Account_FR for France (FR) batch.

In the BDF.xml file, make full refresh FALSE by changing the following parameters:

<Parameter name="Load.FullRefresh" type ="BOOLEAN” value=”true”/> to 
<Parameter name="Load.FullRefresh" type ="BOOLEAN" value="false"/>

Change the value of the following parameter from "FILE" to "FSDW". This will only work for CSA ingestion.

<Parameter name="DIS.Source" type ="STRING" value="FILE"/>
change to
<Parameter name="DIS.Source" type ="STRING" value="FSDW"/>

Before the daily batch run, truncate the following tables:

FO_TRXN_STAGE
FO_TRXN_PARTY_STAGE
FO_TRXN_PARTY_STAGE_RISK
CLIENT_BANK_SMRY_MNTH
ACCT_PEER_TRXN_SMRY_MNTH
HH_SMRY_MNTH
HH_BAL_POSN_SMRY

If the tables are not truncated, an ORA-00001: unique constraint violation error will generate for these tables for the next day run. This is only applicable when running multiple batches in parallel using parallel_bdf.sh script.

The parallel_bdf.sh" file is located at $FIC_HOME/bdf/scripts. Grant 755 permission to this folder.

The parallel_bdf.sh takes two input parameters; the BDF datamap name and the <first two characters of the batch name>.

Command:

 parallel_bdf.sh <BDF datamap name> <first two
                        characters of the batch name>

To run this command, navigate to $FIC_HOME/bdf/scripts folder and run "parallel_bdf.sh" by giving it the required two parameters

Example command for the France (FR) batch:

parallel_bdf.sh Account FR
parallel_bdf.sh AccountAddress FR
parallel_bdf.sh AccountPhone FR
parallel_bdf.sh Account_AccountCustRiskUpd FR
parallel_bdf.sh PreviousWatchList_WatchList FR
parallel_bdf.sh NameMatchStaging FR
parallel_bdf.sh WireTransaction_FrontOfficeTransaction FR
parallel_bdf.sh WireTransaction_FrontOfficeTransactionRevAdj FR

Example command for the Singapore (SG) batch::

parallel_bdf.sh Account SG
parallel_bdf.sh AccountAddress SG
parallel_bdf.sh AccountPhone SG
parallel_bdf.sh Account_AccountCustRiskUpd SG
parallel_bdf.sh PreviousWatchList_WatchList SG
parallel_bdf.sh NameMatchStaging SG
parallel_bdf.sh WireTransaction_FrontOfficeTransaction SG
parallel_bdf.sh WireTransaction_FrontOfficeTransactionRevAdj SG

When the BDF ingestion ends for a batch, truncate the respective

KDD_PRCSNG_BATCH_CONTROL_<first two characters of batch
                        name>

tables. This is required for WLM UI functionality for the particular batch to be unlocked.

Once BDF ingestion completes for all the batches, end the default batch started at Step 1 by executing End Mantas Batch.

Detection (which includes Scenarios) and Post Processing jobs will continue to run in sequence.

In order to proceed with detection and post processing jobs you must run the start mantas batch and set the mantas date for a particular batch. Once all jobs for the batch are completed, run the end mantas batch. Repeat this process for all the batches in your system in sequence.

Example: To run detection and post processing jobs for the France (FR) batch, start the mantas batch for FR (France), set the mantas date and run the jobs for FR batch. Once all jobs are over for FR batch, run the end mantas batch for FR. Do the same for other batches such as US (United States), SG (Singapore), and so on.

4.1.3.1 Run BDF Ingestion in Parallel for Multiple Batches