File Layout Modes

You can organize the Oracle Object Storage, SFTP, Microsoft Azure source files in standard folders or flat files.

File Layout Types

Layout Enable Flat File Layout Source Layout
Standard folder false Remote Host Extract Files Directory/datastore-name/data-files
Flat file true Remote Host Extract Files Directory/data-files

Standard Folder Layout

Use the standard folder layout when each data store has its own folder under the configured base folder.

The base folder of the standard folder layout must match the folder path provided in the connection configuration.

Inside the base folder standard folder layout:

  • Create one folder for each data store.
  • The data store folder name must match the data store name.
  • Data file names must start with the data store name. Avoid special characters, including spaces, in data store names, folder names, and file names.

Example standard folder layout:

<Remote Host Extract Files Directory>/
  META_DATASTORES.csv
  ACTIVITY_TYPES/
    ACTIVITY_TYPES.csv
    META_DATASTORES_ACTIVITY_TYPES_COL.csv

In the base folder, create the data store metadata file, META_DATASTORES.csv to list the supported data stores.

Column Name Required Description
DATA_STORE_NAME Yes Data store name. The Data store name match the data store folder name.
DATA_STORE_LABEL No Description or display label for the data store.

Example META_DATASTORES.csv file:


DATA_STORE_NAME,DATA_STORE_LABEL 

ACTIVITY_TYPES,Activity Types

Each data store folder must contain:

  • A data file that contains the data to load into Fusion Data Intelligence.
  • A column metadata file that describes the columns in the data file.

The data file must start with the data store name. For example, ACTIVITY_TYPES.csv and ACTIVITY_TYPES.xlsx

The column metadata file must use the following naming pattern:

META_DATASTORES_<DATA_STORE_NAME>_COL.csv

For example, META_DATASTORES_ACTIVITY_TYPES_COL.csv

Flat File Layout

Use flat file layout when the source system publishes all the files directly in one folder instead of creating one folder per data store.

Place all data files directly in the configured Remote Host Extract Files Directory.

Example flat file layout:

<Remote Host Extract Files Directory>/
  OFFICE_BLDG.csv.zip
  OFFICE_BLDG_1.csv.zip
  CTLDA.csv.zip
  INVDA.csv.zip
  META_DATASTORES.csv

In the flat file layout, the data store names are derived from file names.

For example:

File Name Datastore Identified
OFFICE_BLDG.csv.zip OFFICE_BLDG
CTLDA.csv.zip CTLDA
INVDA.csv.zip INVDA

Note:

If you have multiple flat files for the same data store, those files can be treated as data files for the same data store by using a configured flat file delimiter. If you need this behavior, create a service request (SR) on My Oracle Support.

For example, If the configured delimiter is _, then the following files are treated as files for data store

OFFICE_BLDG.csv.zip
OFFICE_BLDG_1.csv.zip
OFFICE_BLDG_2.csv.zip

Flat file layout supports two metadata extraction patterns:

  • Metadata extraction with META_DATASTORES.csv
  • Metadata extraction without META_DATASTORES.csv

For Metadata extraction with META_DATASTORES.csv in the root directory, META_DATASTORES.csv is used as the data store list.

  • The extractor reads data store names from META_DATASTORES.csv.
  • Metadata extraction is performed only for data stores listed in META_DATASTORES.csv.
  • Files that don't belong to listed data stores are ignored during metadata extraction.
  • If a column metadata file exists, it's used first.
  • If a column metadata file doesn't exist, column metadata can be inferred from a matching data file.

For example, you have these files, if META_DATASTORES.csv contains only INCLUDED, the extractor processes INCLUDED_1.csv and ignores EXCLUDED_1.csv.

  • META_DATASTORES.csv
  • INCLUDED.csv
  • EXCLUDED.csv

For Metadata extraction without META_DATASTORES.csv, the extractor discovers data stores from eligible data files in the root directory.

For example, you have these files:

  • OFFICE_BLDG.csv.zip
  • CTLDA.csv.zip
  • INVDA.csv.zip

Discovered data stores are:

  • OFFICE_BLDG
  • CTLDA
  • INVDA

If column metadata files are provided in flat files, place them in the root directory using this naming pattern:

META_DATASTORES_<DATA_STORE_NAME>_COLS.csv

When the META_DATASTORES_<DATA_STORE_NAME>_COLS.csv file is available, it's preferred for column metadata. When it isn't available, the extractor can infer column metadata from a matching data file.