3.2.1.4 Loading Multiple As-of-Date Data in a Single File

Loading data for multiple As-of-Dates within a single data file requires an accompanying Manifest File. The Stage Management Ledger loader program uses the Manifest File, loaded together with the input data file, to identify the As-of-Dates for which data must be loaded into the Stage Management Ledger tables. Loading the Manifest File is mandatory whenever a data file contains data for more than one As-of-Date. The Manifest File option in the Data File Specification definition screen must be selected.

Multiple As-of-Date load does not support parallel loading.

Note:

• If a file has multiple As-of-Dates, each As-of-Date is loaded sequentially, one after the other.
• When loading multiple files into the same stage table, and each file has multiple As-of-Dates, load those files sequentially — one after the other. Do not attempt to load them in parallel, as this can result in partially loaded files.

By default, the Stage Management Ledger program assumes that the incoming data file contains data for only one As-of-Date. To override this assumption, include the following two additional properties in the Manifest JSON file:

• load_type: A value of MULTI_AS_OF_DATE for this property signifies to the loader program that the file has multiple As-of-Dates.
• as_of_dates: A comma-separated list of distinct As-of-Dates included within the data file. The list of dates need not appear in sorted order.

A sample structure of a file with multiple As-of-Dates is shown below:

{
      "file_name": "input_20001201_CASCN2AODSTGMGMTLDGR.csv",
      "as_of_date": "2000-12-01",
      "checksum": "9050e936d47bbb110bdebb799587b136",
      "record_count": "120",
      "rejection_threshold": "0",
      "load_type": "MULTI_AS_OF_DATE",
      "as_of_dates": [
            "2000-12-01",
            "2000-11-01",
            "2000-10-01"
      ]
}

To load multiple As-of-Date data:

1. Prepare a single data file containing records for one or more As-of-Dates.
2. If the file contains records for more than one As-of-Date, create an accompanying Manifest File listing each As-of-Date under as_of_dates, with load_type set to MULTI_AS_OF_DATE.
3. Upload the data file and its Manifest File to the object store.
4. The loader reads the Manifest File to identify the As-of-Dates present in the data file, then stages records into the Stage Management Ledger tables against the appropriate As-of-Date.

Note:

Existing single-As-of-Date batches continue to function as before, whether or not a Manifest File is used.