4.3.3.4 Ingestion Flat File Data Load

The loading process receives, transforms, and loads Business and Reference data that event detection and assessment investigation processing requires. After loading the base tables, the Oracle client’s job scheduling system invokes datamaps to derive and aggregate data.

This section covers the following topics:

Overview

The following figure provides an overview of the data management flow using Flat File Interface:

Figure 4-5 Data Loading Flow Using Flat File Interface


Description of Figure 4-5 follows
Description of "Figure 4-5 Data Loading Flow Using Flat File Interface"

Note:

All DIS datamaps in the Flat File Interface for which staging representation is marked as Yes are applicable for Flat File loading. For more information, see TBAML Flat File Interface.

Using TBAML Datamaps

The datamap takes the data from the flat files, enhances it, and then loads it into a target database table (FCDM).

To load data in the FCDM using Flat Files, follow these steps:
  1. Place the ASCII.dat flat files in the <OFSAAI Installed Directory>/bdf/inbox directory.
  2. Configure the DIS.source parameter to FILE.

    Configure the DIS.Source parameter to FILE-EXT for loading flat files through the external table. In order to load the flat files using the external table, the ext_tab_dir_path variable must also be set to the inbox directory and the database UNIX account must have read and write privileges to it.

  3. Execute the Account datamap which loads into the Account (ACCT) table:
    <OFSAAI Installed Directory>/bdf/scripts/execute.sh Account

    Note:

    If there are any errors in loading, refer to the <OFSAAI Installed Directory>/bdf/logs path.

Ways of Data Loading

Full Refresh Data Loading

Incremental (Delta) Data Loading

This section covers the following topics:

Note:

The following ways of data loading is applicable only for DIS files defined with load operation as Overwrite.

For full refresh data loading, first data is truncated and then new data is inserted. For example, suppose five records are loaded on Day 1. If new data is required on Day 2 based on the business keys defined on the DIS files, a full refresh data load can be done.

To do a full refresh data load, set load.fullrefresh to true in the <OFSAAI Installed Directory>/bdf/config/BDF.xml path. For more information, see BDF.xml Configuration Parameters.

The time taken to do a full refresh data load is less than for an incremental load, although complete data must be provided every time.

For incremental data loading, the following can be done:
  • Data can be merged
  • Existing data can be updated
  • New data can be inserted

For example, five records are loaded on Day 1. If four new records need to be inserted and one existing record needs to be updated based on the business keys defined on the DIS files, an incremental data load can be done.

To do an incremental data load, set load.fullrefresh to false in the <OFSAAI Installed Directory>/bdf/config/BDF.xml path. For more information, see BDF.xml Configuration Parameters.

Note:

It takes more time to do an incremental data load than a full refresh data load, although there is no need to give complete data every time. Only updated or new data is required.