3.1.1 Using the Persist Widget

Use this widget to write data to database tables so that it can be used in other pipelines.

Before adding or configuring Persist, ensure that the Persist widget is properly connected to a Data Forge on the Pipeline Canvas; otherwise, it will not function as expected.
  1. In the Pipeline Designer page, select the pipeline where you want to add or modify the Persist. The Pipeline Canvas displays.
  2. To add a Persist to a Data Forge, click Add To Add To icon, and then select Persist.Persist Widget
    • Alternately, you can right-click on the Canvas to display the list of widgets, select Persist, and associate the Persist with the Data Forge.
    • You can also use the hamburger menu on the left side to display the Node Picker, which lists any externalized data forges already created. Drag and drop the data forge to the pipeline as desired.
  3. Click Options Options iconthen click EditEdit icon. The Data Forge pane displays.
  4. Under Basic Configuration, provide the following details:
    Ensure that the Persist widget is connected to a Data Forge on the Pipeline Canvas. The Persist widget will not function unless it is associated with a Data Forge.
    1. In the Name field, enter a user-defined name for the Persist widget. (This field is not auto-populated.)
    2. Select the Target Table from the drop-down list.
    3. The Source Data Forges field displays the Data Forge selected in the Data Forge widget. This field is system-generated and disabled.

      Note:

      This value is inherited from the associated Data Forge and cannot be modified in the Persist widget.
    4. You can add a Runtime Parameter to the Data Forge by clicking Add Run Time Parameter Add Run Time Parameter icon. For more information about how to create run time parameters, see Creating Runtime Parameters.
  5. The Advanced Configuration tab displays any Joins and Filters associated with this Data Forge.

    Note:

    At least two sources (dataforges) must be selected or connected for the join section to be enabled in the Advance Configuration tab.
    • Expand Joins to view or create Joins associated with this Data Forge. The Output box provides a summary of all current Joins, with the Join configurations listed in detail below. You can add new Joins using the steps in .Creating Joins
    • Expand Filters to view or create Filters associated with this Data Forge. The Output box provides a summary of all current Filters, with the Filter configurations listed in detail below. You can add new Filters using the steps in Creating Filters .
  6. Use the Map tab to view and map columns within the Persist widget. To map columns, follow these steps:
    1. Click the Map tab.
    2. Under Type, select one of the following options:
      • Incremental: Maps data from the source dataset to the target table incrementally. Only new records are inserted during each run.

        For example, if 100 records are loaded on Day 1 and 110 records exist on Day 2, only the additional 10 new records are inserted into the target table.

      • Full Load: Inserts all filtered data into the target table along with Processing Date and Data Origin values during each run.
    3. The Select a Data Forge field displays a drop-down list of the Data Forges connected to the Persist widget. Select the required Data Forge from the list to define the source dataset for column mapping.
    4. The Target Table field displays the target table selected under Basic Configuration.
    5. Do either of the following:
      • If you select Incremental as Type, do the following:
        1. From the Source Column list, select the required column. Use the search bar to locate specific columns.
        2. Select the corresponding Target Column.
        3. In the Selected Type dialog box, select either Unique or Direct.
        4. Click Map Map Column iconto map the column. The column displays in the Mapped Columns list. You can edit the Mapped Column Name, if desired.
        5. The mapped column is displayed in the mapped columns list.
        6. Repeat the steps to map additional columns, if required.
      • If you select Full Load as Type, do the following:
        1. From the Source Column list, select the required column.
        2. Select the corresponding Target Column.
        3. Click Map Map Column iconto map the column. The column displays in the Mapped Columns list. You can edit the Mapped Column Name, if desired.

          Note:

          In Full Load, all filtered data is inserted into the target table during each run along with Processing Date and Data Origin.
    6. Alternately, click Add Expression Add expression iconto open the Expression Builder. The Expression Builder is used to define free flow text filter conditions. To use the Expression Builder, follow these steps:
    1. Select the required Data Forge, Attribute and Runtime Parameters and operators. The resulting condition is displayed in the Condition field.
    2. Click Save Save iconto save the changes.
  7. When you have finished configuring your Persist, click Save.

    Note:

    If a batch fails or gets stuck, run the PurgeAMIngestion batch. While running the batch, edit the BatchRunId and pass the corresponding DTPIngestion batch run ID.