4.3.3.3 Hive CSA Data Load

Hive-to-Table (H2T) is used in the AAI Framework for data loading. The source for H2T data is the hive database.

About AAI H2T Data Loading

AAI (Analytical Applications Infrastructure) is a complete end-to-end Business Intelligence solution. It is a single interface that lets you access your company's operational data and use it to track and respond to business trends. It also facilitates the analysis of processed data.

The AAI framework is the process of retrieving structured data from data sources for further data processing, storage, or migration. The intermediate extraction process is followed by data transformation and metadata addition before exporting it to the Business Data Model. For more information, see Chapter 2, Section - Data Mapping, Oracle Financial Services Analytical Applications Infrastructure User Guide.

This section covers the following topics:

Process Flow for AAI H2T

The following figure shows the process flow for AAI H2T:

Figure 4-4 Process Flow for AAI H2T


Description of Figure 4-4 follows
Description of "Figure 4-4 Process Flow for AAI H2T"

Starting a Batch

Note:

Ensure that the staging data has the same batch date records.

To start the batch, run the start_mantas_batch.sh and set_mantas_date.sh scripts. For more information, see Managing Batch Control Utility.

Verifying the Data Quality Using DQ

Note:

Execute the Update_DQ_Tables.sql file available in the FIC_HOME path on the ATOMIC schema database.

Data Quality (DQ) is a check that is done at every level based on the FSDM table. When data is moved from CSA to FSDM, a check is done on CSA. This check is done in order to move only useful data into the FSDM table. For example, a column in FSDM should not be blank if it is mandatory. These checks are also called rules.

The following data quality checks are done:
  • Length Validation Check: If the length of data in source column is more than the length of target column, then an error message is generated. For example, if the ACCT_INTRL_ID column, which has a column length of 50 characters, needs to be populated from the source table column V_ACCOUNT_NUMBER, which has a few data with length more than 50 characters. An error message is raised
  • Domain Check: If any data does not qualify for the domain values, then an error message is generated. For example, if the valid value that column ADDR_USAGE_CD accepts is one of M|B|L|A|O|P|D|H|X|V, but the source column V_ADDRESS_PURPOSE_TYPE_IND has additional values such as E or C. An error message is raised.
  • Mandatory Check: If a column which must have a value for the record to be valid has a null value, then an error message is generated. For example, if the column ADDR_STRT_LINE1_TX needs to have a value for the record to be valid and is mapped to the source table column V_ADDRESS_LINE1. If the column ADDR_STRT_LINE1_TX has a null value, an error message is raised.
  • Threshold Test: If a target table column must have a value that is greater than 0 but has a value of 0, then an error message is generated. For example, if the target table column LDGR_AM must have a value that is greater than 0 but the source table column N_LEDGER_BAL has a value as 0 or null, an error message is raised.

    Note:

    In addition to these data checks, another data check is done for duplicate data during data loading through AAI H2T.

Moving Data through T2H

Data is exported or moved from the Oracle RDBMS to the hive database using AAI T2H. For moving data through T2H, the T2H batch must be executed from RefData_Source with the following parameters:

[EXEC_ENV_TARGET]=<hive_infodom_name>,which is the ICC batch parameter

“EXEC_ENV_TARGET”,”<hive_infodom_name>”,which is the batch parameter if the batch is running through RRF.

For more information, see Chapter 2, Section - Data Mapping, Oracle Financial Services Analytical Applications Infrastructure User Guide.

Executing Data Transformation Using DT

The Data Transformation (DT) functionality allows you to delete the existing data in the AAI. For more information, see Adding Tasks to a TBAML Batch.

Moving Data through H2T

Data is exported or moved from the hive database to Oracle RDBMS using AAI H2T. For more information on moving data through H2T, see Chapter 2, Section - Data Mapping, Oracle Financial Services Analytical Applications Infrastructure User Guide

For the table to be loaded, the list of H2Ts are in Managing Data.

Executing Behavior Detection Jobs

After the data quality check, data transformation, and data movement through T2H and H2T is completed, execute the following jobs:

Ending the Batch

To end the batch, run the end_mantas_batch.sh script and the DT. For more information, see Managing Batch Processing Utilities.