Apply Incremental Processing in a Data Flow

Apply incremental processing in a data flow to load only new or updated records from a database.

Before you start, create a connection to one of the supported databases, for example Oracle, Oracle Autonomous Data Warehouse, Apache Hive, Hortonworks Hive, or Map R Hive.

1. Create or open the data flow in which you want to apply incremental processing.
2. In the Data Flow editor select the Save Data step to display the Step editor pane.
3. In the Dataset field, specify the name of the input dataset (the dataset specified in the Add Data step).
4. At the Save data to option select Database Connection.
5. Click Select Connection and select a connection to one of the supported target databases.
6. In the Data field, specify the name of the target table that you're writing to.
7. In the When run option, select Add new data to existing data.
8. Click Save.
   You're now ready to schedule your data flow to load new data regularly.