Fileset Transfer Manifest Specification

A transfer manifest is a small metadata file that describes the set of files that follow it. This file is optional in most cases. Depending on the terms of your contract, your Oracle Utilities Delivery Team will inform you whether this file is required. However, it is highly recommended for several reasons.

  • Transfer manifests allow Oracle Utilities to validate that we have received all the files that we are supposed to receive, and that all of them have arrived intact. In the event of a transfer failure, such as a file that gets truncated due to a connection dropping, a transfer manifest will allow Oracle Utilities to detect the error condition and notify the utility. This typically leads to faster resolution.
  • Transfer manifests also allow the utility to specify the order in which files should be considered. File order coercion is currently supported in some Oracle Utilities Opower importers, but not in others. If you expect to depend on the order in your manifest, contact your Oracle Utilities Delivery Team about whether the files will be processed by an importer that currently supports file order coercion.

Oracle Utilities recommends that utilities use a transfer manifest every time they send a set of files. Files sent without a manifest will also be processed, but in some cases Oracle Utilities will have no other way to detect issues with the file.

If your utility chooses to use the transfer manifests, note that the transfer manifest file must always precede the first data file. A data file cannot be sent to Oracle Utilities before the manifest file is sent.

Transfer Manifest File Contents

Fileset Transfer Manifest files must be delivered in YAML format. The file must contain the following:

  • meta.version: Version identifier.
  • transfer.validity_window (int): Expiration time in seconds for this manifest, measured from when the file was received. If any of the specified files have not appeared when the time window ends, then the transfer is failed and notifications are sent. Files that have started but not yet finished transferring count as having appeared.
  • fileset: A list of fileset entries. This must be a list to preserve ordering.
  • fileset.[FILENAME].size (int): Size in octets of the object being transferred.
  • fileset.[FILENAME].cksum (str): Checksum of the object being transferred.
  • fileset.[FILENAME].ckalg (str-enum): Checksum algorithm used to generate the checksum string. Valid values are md5, sha1, sha256, ripemd-160. If your system cannot support any of these checksum algorithms, you can discuss alternatives with your Delivery Team.

Back to Top

File Indentation Structure

Fileset Transfer Manifest files adhere to the following indentation rules:

  • Section headers, like meta, transfer, and fileset, are preceded by zero spaces.
  • Section content, like meta.version and transfer.validity_window, is preceded by two spaces.
  • Names of the files in the file list, like opwr_glbx_customer_ABCDEFG.gz.pgp, are preceded by two spaces, a '-' (ASCII code 0x2d), and then another space.
  • Contents of the file list, like size: 10945113, are preceded by six spaces.

Example Fileset Transfer Manifest File

The following shows the contents of an example fileset transfer manifest.

meta:
  version: 0
transfer:
  validity_window: 600
fileset:
  - opwr_glbx_interval15_electric_ABCDEFG.DAT.gz.pgp:
      size: 10945113
      cksum: abcdef0123456789abcd
      ckalg: md5
  - opwr_glbx_interval15_gas_ABCDEFG.DAT.gz.pgp:
      size: 10112608
      cksum: 0123456789abcdabcdef
      ckalg: md5
  - opwr_glbx_customer_ABCDEFG.gz.pgp:
      size: 2149602
      cksum: abc123abc123abc123ab
      ckalg: md5

Back to Top

File Naming Conventions

Manifest files must use the following naming convention:

transfer_manifest_<UNIQUE_ID>.yaml

Back to Top