Preparing the Source Data File

You first generate a source data file and register it as an Incremental File Adapter in Data Management. You then set up a data rule that designates the source data file. Data loads are run from the file designated in the data rule. The initial source data file is compared with a subsequent file; the last run file becomes the basis against which the subsequent load is compared. The Incremental File Adapter loads only the differences, which results in faster data loads.

Tip:

You'll experience better performance if you provide a presorted source file.

About a source data file:

  • The source data file must be a delimited data file. The data field is the last column in the file.
  • Supported delimiters: comma, vertical bar, exclamation, semicolon, colon, tab, and tilde.
  • Data files used must contain a one-line header, which lists the dimensions you're loading. For example: Entity,Employee,Job,Project,Version,Account,Property,Data.

  • Both numeric and non-numeric data can be loaded.

  • Any deleted records between the two files is ignored. You must manually handle deleted records.

  • If the file is missing (or you change the last ID to a nonexistent run), the load completes with an error.

  • Presorting the file decreases the processing time.

  • Only single Period data loads are supported for an incremental load. Multi-period loads are not supported.

  • Drill down is not supported for incremental loads because incremental files are loaded in Replace mode and only the last version of the file comparison is present in the staging table. As a workaround, you can load the same data file to another location using the full data load method. In this case, you should import data only and not export it to the target application.

  • Copies of the source data file are archived for future comparison. Only the last 5 versions are retained. Files are retained for a maximum of 60 days. If no incremental load is performed for more than 60 days, then set the Last Process ID to 0 and perform the load.

You can load data using the Incremental File Adapter multiple times.