Creating Data Extraction API Pipelines

You can create a data extraction API pipeline using the API Forge widget to define the API which will be used to extract the data.

You must first create a new pipeline. Follow the steps in Creating Pipelines to create a new pipeline, with the Pipeline Type as API Pipeline.

1. Navigate to the Pipeline Designer page.
2. Open your newly created API Pipeline.
3. Drag and drop the Dataset widget from the widgets pane in the upper-right corner of the designer pane. You can add multiple datasets to each pipeline.
4. Hover on the Dataset widget and click Edit.
5. Provide the details as described in the following table.

   Table 17-2 Dataset Widget Details

   Field	Description
   Name	Enter the name for your dataset.
   Tables	Select a table from the Tables drop-down list. This list consists of all the staging tables that are available. The columns of the selected table are displayed in the Attributes pane. The attributes include the Logical Name, Column name, and Column Type.

6. Click Save to save the changes. The dataset is created and is visible on the canvas. It is also available for use in the Dataset pane.
7. Drag and drop the API Forge widget from the widgets pane in the upper-right corner of the designer pane.
8. Hover on the API Forge widget and click Edit.
9. Provide the details as described in the following table. Click on the + icon to add rows as needed.

   Table 17-3 API Forge Details

   Field	Description
   Name	Enter the name for this API.
   URI	Enter the URI to be used in API Execution. This must be unique to this API Forge widget.
   Source Dataset	The datasets that are connected to the ForgeAPI Widget display.

10. Enter the following Input Parameters. Click on the + icon to add rows as needed.
    • Key: Enter the JSON Key to be passed in the request Object. This Input Parameter displays in the Join/Filter conditions under the Expression builder and is also given as a value during the API call, as shown in the sample payload.
    • Datatype: Select the datatype for the Key from the following options:
      • String
      • Date
      • Number
      This Input Parameter displays in the Join/Filter conditions under the Expression builder and sets the value during the API call.
11. Provide the following Join information. This section only displays when more than one dataset is connected to the APIForge widget.
    • Parent Dataset: Select the Parent Dataset as required for the desired output.
    • Relation: Displays the relation type between the parent and child as Has Many [For every parent entity, there can be 0..n child entities].
    • Child Dataset: Select the Child Dataset as required for the desired output.
    • Join Conditions: You can add multiple join conditions on the parent and child datasets. Use the drop-down lists to select:
      • Datasets: Select from the table names.
      • Attributes: Select from the column names
      • Operator: Select an operator to connect the Datasets and Attributes.
      Click on the + icon to add join conditions as needed.
    • Join Condition Modes: Define the mode of the join condition between the tables. You can select:
      • Exp: Use this option when one dataset is Entity and the other is Expression. The Expression Builder displays.
      • Text: Use this option when one dataset is Entity and the other is static text
      • Table: Use this option when both datasets selected are tables.
12. Provide the following Filter information.
    • Filter Conditions: You can apply filters to datasets based on specific criteria. Select the dataset to apply the filter on and use the drop-down lists to select:
      • Datasets: Select from the table names.
      • Attributes: Select from the column names
      • Operator: Select an operator to connect the Datasets and Attributes.
      Click on the + icon to add filter conditions as needed.
    • Filter Condition Modes: Define the mode of the filter condition between the tables. You can select:
      • Exp: Use this option when one dataset is Entity and the other is Expression. The Expression Builder displays.
      • Text: Use this option when one dataset is Entity and the other is static text
      • TableUse this option when both datasets selected are tables.
13. Map and configure the Output Parameters.
    • Select the Dataset Name from the drop-down list to populate the attributes in the dataset
    • Select fields which you want to display in the output from the source datasets and click Map.
    • Assign valid JSON keys for each mapped field to structure the output.
14. Check the sample payload to ensure the setup aligns with your requirements. If changes are required, you can edit the necessary settings using the steps above. After making changes, click Regenerate Payload to refresh the sample payload.
15. Click Save to save the changes.
16. In the Pipeline Designer page, click Save to save all updates.