3.2.3.2 Instrument and Ledger Data Loaders

After the Data Files are uploaded to the Object Store, the Data Loaders are used to move the data from the files to the staging tables and then to processing tables.

File to Stage

To load the Data to Staging Tables:

  1. Define a new Batch and save it.
  2. Add the following Tasks to the above Batch:

    Table 3-7 Data Loader – File to Stage Data

    Task Code Task Name Component Parameters
    1 * Custom Task Name * Stage Data Loader

    Table Name: select the stage table name from the available list.

    Data File Specification: select the data file specification definition from the available list.

    Data File Name: free text where file name uploaded to the object store to be provided. This is optional.

    • When the Data File Name is not provided, Stage Loader fetches the file from object store with the selected Data File Specification name.
    • If name of file is different from Data File Specification, then put exact file name including extension but excluding the prefix.

      Note:

      You can also zip the file and then upload. Ensure the file name in the zip file is inline with the Data File Name in the Task Parameters UI.
    • Folder Name: Enter the physical folder name. This is the location where file saved in the Object Store. For more information, see File Upload and Download Utility. You can use the MIS Date in the input file name for easier identification.

    Note:

    Beginning with the 26B release, the Logical Unique Key used by the Stage Data Loader to identify duplicate Management Ledger records includes DATA_SOURCE_CODE, in addition to AS_OF_DATE, CONSOLIDATION_CODE, ISO_CURRENCY_CD, and all seeded and placeholder KPDs. This enhancement allows you to load the same logical data set more than once for the same As-of Date by providing a unique DATA_SOURCE_CODE value for each load.

    Note:

    Beginning with 26C release, DATA_SOURCE_CODE is mandatory (NOT NULL) for all Stage Management Ledger loads.

    * Task Code and Task Name in the above table are for illustration purpose only. You can name them as per your requirements.

    Table 3-8 Sample Data File Specification and File Name

    Data File Specification Data File Name in Task Parameters UI File in Object Store
    Asset_Loan.dat Asset_Loan_1.dat input_yyyymmdd_Asset_Loan_1.dat
    Asset_Loan.dat Not entered input_yyyymmdd_Asset_Loan.dat

    To upload multiple files using same Data File Specification, add separate tasks for each file within the batch.

  3. Execute the Batch for the As-of-Date used in the Data File.

    For detailed instructions on Creating a Batch, Defining a Task, Execute the Task, and Schedule a Batch, see Scheduler Services.

Stage to Processing

To load the Data from Staging Tables to Processing:

  1. Define a new Batch and save it.
  2. Add the following Tasks to the above Batch:

    Table 3-9 Data Loader – Stage to Processing Data

    Task Code Task Name Component Parameters
    1 * Custom Task Name *

    Select the relevant Component depending on the Data that you want to process. The options are:

    • Instrument Data Loader
    • Ledger Data Loader
    • Transaction Summary Loader

    Note:

    The system validates the DATA_SOURCE_CODE column from the staging table against the FSI_INSTRUMENT_DATA_SOURCE_CD table. If the data source is enabled, its INSTRUMENT_DATA_SOURCE_CD is pushed to the FSI tables; otherwise, a default value of -1 is assigned.

    Note:

    For the Ledger Data Loader, the DATA_SOURCE_CODE value from the staging Management Ledger table is copied as-is into the DATA_SOURCE column of the corresponding processing (FSI) Management Ledger table. Unlike the Instrument Loader, no validation against a Data Source dimension table is performed.

    Stage Table: Select the stage table name from the available list.

    Data File Specification: Select the Data File Specification name from the available list.

    Folder Name: Enter the folder or prefix for the data file. This field is optional. A trailing / is automatically appended if missing.

    Data File Name: Enter the name of the data file to be processed. Supports / within the file name to represent subfolders.