Apply Incremental Processing to a Data Flow

Use incremental processing to determine the last data processed in the data flow and to process only the newly added data.

1. Select a data element column as an incremental identifier for the data set.

   You can select an incremental identifier only for those data sets that are sourced through database connections.

   1. Go to the Data page and select Data Sets.

   2. Select a data set and click the Actions menu or right-click, then select Open.

   3. Click Edit Data Set on the Results toolbar.

   4. Select the data set node in the diagram. From the New Data Indicator list, select a column, then click Save.

2. Apply incremental processing to the data flow using the data sets for which you’ve selected the incremental identifier.

   1. Create or open the data flow in which you want to apply incremental processing.

   2. In the Data Flow editor select the data set.

   3. In the Step editor pane, select Add new data only to mark the data set as incremental.

   4. Click Save.

In a data flow with multiple data sets, you can select only one data set as incremental. If you try to select a second data set as incremental, you see a warning message. Click Yes to enable incremental processing for the second data set for which you’ve selected Add new data only. Incremental processing is deselected for the first data set.