Processing data loads from files
You can load text or SAS files (data sets, XPORT, or CPORT files) into an input data model. You set up a File Watcher for each input data model. The File Watcher then detects when a data file appears in a specified location and proceeds to load the data.
The system supports three processing modes for loading data from files:
-
Full Reload
-
Incremental Reload
-
Note:
- Do not use Full mode if the table-level file being loaded contains only new data or a subset of data because any data not reloaded will be deleted.
- Do not use UOW Load mode if the table-level file being loaded contains only new data or a subset of data for subjects or subject visits because any data not reloaded in processed units of work will be deleted. Instead, use Incremental (Reload) processing.
-
All deletions are "soft" deletions: records have an end timestamp equal to the load's date and time and are no longer available in the system. However, they still exist in the database and have an audit trail.
Table 8-9 Deletion Behavior in Data Loading Processing Modes
Processing Mode | Uses UOW Logic? | Deletion Behavior |
---|---|---|
UOW Load |
Yes |
Deletes nonreloaded records within units of work —subjects or subject visits—with new, changed, or refreshed records. |
Incremental Reload |
No |
Does not delete any data. |
Full Reload |
No |
Deletes all records that are not reloaded in tables that are reloaded. |
Parent topic: Data processing during data loading