Manifest File

Batch processes that create files, for example data export related batch processes, may create one or more files in each processing thread.  External applications that consume these files need to process them only when the entire batch process is successfully completed. To automate the consumption of such files, the external application may leverage a function to be triggered upon the creation of a manifest file to process all the files referenced by the manifest. The manifest file includes details such as content hash value, file size information etc that assist in confirming that file content is completed and has not been tampered with prior to processing the files.

The following sections provide additional information about manifest files.

The Manifest Option Batch Parameter

Manifest file creation is supported on certain base product extract related batch controls only. These batch controls include the Manifest Option batch parameter that controls whether a manifest file should be created or not upon successful completion of the batch process. You may use the Batch Control Query portal to find all batch controls supporting this option.

Manifest File Information

Manifest file content is provided in JSON format and includes the following information:

  • The version of the manifest details structure.

  • The batch control code, run number and rerun number identifying the batch run producing the files

  • Total number of files created by the batch run.

  • Total number of records processed by the batch run.

  • The manifest creation timestamp in ISO format.

  • The previously completed batch run number, if any.

  • The manifest file produced by the previously completed batch run, if any. This may be used to ensure files are processed in a proper order and assist in detecting potential situations of missing files.

  • A list of files created by the batch run that includes the following details for each file: the file’s name, its size in bytes and the SHA-256 hash value of the file’s content.

The manifest file name is constructed as follows:
[batch control]_[batch run number]_[batch rerun number]_manifest.json

The following is an example of the manifest file structure.

{
  "version": 1,
  "batchCode": "HD-PDE",
  "batchRunNumber": 106,
  "batchRerunNumber": 0,
  "totalFiles": 3,
  "recordsProcessed": 10,
  "timeStamp": "2020-05-02T16:55:43.000353Z",
  "previousBatchRunNumber":"105"
  "previousManifestFileName":"HD-PDE_105_0_manifest.json",
  "fileDetails": [
    {
      "fileName": "SOME_FILE_NAME_001.json.gz",
      "fileSize": 1543,
      "hashAlgorithm": "SHA-256",
      "hashValue": "ff7d4e466f93cadc4a18ab009080916baff631e03b728e2f2684c4a4f92ccbe1"
    },
    {
      "fileName": "SOME_FILE_NAME_002.json.gz",
      "fileSize": 1182,
      "hashAlgorithm": "SHA-256",
      "hashValue": "78869e7ce971f28a70f5b35693a05726ba66109fecf492edb60291a62ae1ba9f"
    },
    {
      "fileName": "SOME_FILE_NAME_003.json.gz",
      "fileSize": 1182,
      "hashAlgorithm": "SHA-256",
      "hashValue": "78869e7ce971f28a70f5b35693a05726ba66109fecf492edb60291a62ae1ba9f"
    }
  ]
}{

Verifying The Manifest File

The Verify Manifest File web service may be used by an external system to validate that the manifest file it has received for a specific batch run is not corrupted or tampered with.  The service returns the manifest file name and the hash value of its content for the requested batch run allowing the caller to compare their file details with the returned values.