Introduction to Persist Widget with Incremental and Full Load Modes
The Persist widget is used to write processed data into database tables for reuse in other pipelines. It must be connected to a Data Forge widget to function.
Basic Configuration includes:
- Name (user-defined)
- Target Table
- Source Data Forge (selected based on the established connection)
Advanced Configuration supports Joins and Filters (no hints available). At least two sources must be connected for Joins to be enabled.
In the Map tab, users can select one of the following modes:
- Incremental Mode
Maps data incrementally from the source dataset to the target table. Only new records are inserted during each run.
For example, if 100 records are loaded on Day 1 and 110 records exist on Day 2, only the additional 10 new records are inserted into the target table. - Full Load Mode
Inserts all filtered data into the target table during each run, along with Processing Date and Data Origin.
Users must apply filters using the following formats:- to_date($MISDATE, 'YYYY-MM-DD') for Processing Date
- $DATAORIGIN for Data Origin
If a batch fails or gets stuck, users must run the PurgeAMIngestion batch and update the BatchRunId with the corresponding DTPIngestion batch run ID.
This enhancement provides greater control over data ingestion strategies by allowing users to choose between incremental updates and complete data reloads, improving efficiency, accuracy, and operational flexibility in data processing workflows.
Steps to enable and configure
You don't need to do anything to enable this feature.