The record adapters load the source data.

To start, here is a quick review of how sample data included with the deployment template is processed. The sample application includes a sample dataset in <app dir>/test_data/baseline directory. When processing the sample data, the load_baseline_test_data script copies the contents of this directory into the <app dir>/data/incoming/ directory and sets a flag in the EAC.

This flag, named baseline_data_ready, indicates to the deployment template scripts that the data extraction process is complete and data is ready for processing. Once that has occurred, the baseline update process copies these files into the <app dir>/data/processing directory before running the Forge process.

When using a default deployment template application, it is therefore necessary for all input record adapters to look in the <app dir>/data/processing directory for incoming data extracts. The deployment template handles this automatically by specifying the --inputDir flag when running the primary forge process. This flag overrides any absolute path specified for specific input adapters with the proper deployment template path: <app dir>/data/processing. However, the --inputDir flag respects relative paths, resolving them relative to the path specified as the input directory.

The URL property of any record adapter component therefore only needs to specify the relative path to a specific file or subdirectory within the <app dir>/data/incoming directory. (Remember that files and subdirectories in the incoming directory are copied to the processing directory by the deployment template before Forge is run.)

For example, if a single extract file called data.txt is copied into the <app dir>/data/incoming directory before running a baseline, the URL property of that data's input record adapter should specify a URL of data.txt.

For a more complex deployment where, for instance, multiple text extract files are copied into the <app dir>/data/incoming/extracted_data directory before running a baseline update, the URL property of a single input record adapter configured to read these files should be set to extracted_data/*.txt.