Apply Incremental Processing to a Data Flow
Use incremental processing to determine the last data processed in the data flow and to process only the newly added data.
-
Select a data element column as an incremental identifier for the data set.
You can select an incremental identifier only for those data sets that are sourced through database connections.
-
Go to the Data page and select Data Sets.
-
Select a data set and click the Actions menu or right-click, then select Open.
-
Click Edit Data Set on the Results toolbar.
-
Select the data set node in the diagram. From the New Data Indicator list, select a column, then click Save.
-
-
Apply incremental processing to the data flow using the data sets for which you’ve selected the incremental identifier.
-
Create or open the data flow in which you want to apply incremental processing.
-
In the Data Flow editor select the data set.
-
In the Step editor pane, select Add new data only to mark the data set as incremental.
-
Click Save.
-
In a data flow with multiple data sets, you can select only one data set as incremental. If you try to select a second data set as incremental, you see a warning message. Click Yes to enable incremental processing for the second data set for which you’ve selected Add new data only. Incremental processing is deselected for the first data set.