Creating REST Tasks
A REST task lets you call a REST API endpoint to perform an operation.
After sending the REST request, by default Data Integration waits for the call to return the response before performing other tasks (synchronous API mode). The REST invocation completes when the response is received for the REST request.
If the REST request invokes an operation that is long running, you can specify a polling condition, and an optional termination configuration, to identify the completion of the REST invocation. By using polling, Data Integration does not wait for the REST call to return the response before proceeding to other tasks (asynchronous API mode).
The maximum size supported for a response returned from a REST task is 512K. Larger responses are truncated.
Parts of a REST Task
To invoke a REST API endpoint, provide the information for executing and completing the REST request in a REST task.
Creating a REST task involves the following main steps:
Define execution details: Define the execution details for the REST request, including the HTTP method, URL, and request header.
The URL is the REST server to contact and the relative path to the resource that you want to access in the request. For example:
http://myserver.com:8081/workspaces/finance1234/dataAssets
For a POST or PUT method, you also provide the request body. Currently, only JSON is the supported format for a request body.
Defining execution details in Data Integration is the same for short-running API operations and long-running API operations.
Specify completion criteria: Specify the success condition criteria that must be satisfied to consider the REST task a successful completion.
Write the condition that evaluates to true or false using the response from the REST call in the execution step. If the condition returns true, the outcome of the REST task is successful. If false, the outcome is a failed REST task.
Data Integration provides a default success condition that you can use or edit.
Specifying completion criteria for a long-running API operation requires adding a polling condition and other polling values. Adding a termination request is optional.
See also how to use parameters in a REST task, and what you need to set up before you use the Oracle Cloud Infrastructure Resource Principal to authenticate an API endpoint in a REST task.
Polling in Long-Running Operations
For a long-running REST API operation, in addition to a success condition, you must specify a polling URL and condition, and polling interval and timeout values to identify the completion of the REST invocation.
The polling configuration is used to periodically poll the status of the REST call that is invoked at the execution step.
In polling, the success condition is an expression that is written on the response of the polling request. The polling call evaluates the polling condition to determine whether polling stops or continues. Data Integration issues a polling call repeatedly at the specified polling interval until the value for the specified polling timeout is reached, or until the polling condition returns false, whichever occurs first.
The polling interval is the length of time to wait before sending the next polling request. The polling timeout is the maximum length of time that is allowed for repeated polling to occur at the specified interval rate. The interval value must be less than the timeout value.
Suppose that the polling timeout is 24 hours, and the polling interval is 1 hour. After the REST request at the execution step is invoked, the polling REST request is sent at 1-hour intervals, but only for up to 24 hours, or until the polling condition returns false.
If the criteria specified in the success condition is satisfied, the REST call is completed successfully, and the REST task run is considered a success. If the success condition returns false, the outcome of the REST task run is considered as failed.
For example, suppose the polling configuration polls the progress of a task run by evaluating the status in the response:
cast(json_path(SYS.RESPONSE_PAYLOAD, '$.status') as String) != 'SUCCESS' or cast(json_path(SYS.RESPONSE_PAYLOAD, '$.status') as String) != 'ERROR'
Polling stops when the success condition is met.
cast(json_path(SYS.RESPONSE_PAYLOAD, '$.status') as String) == 'SUCCESS'
Termination in Long-Running Operations
Configuring the HTTP method and REST URL to terminate the REST call is optional for a long-running API operation.
Defining the details for the REST termination request in Data Integration is similar to defining the REST request in the execution step. You can, however, include expressions that use the API response from the REST execution step in the termination URL.
If you terminate a REST task run, and a termination configuration is not provided in that REST task, Data Integration:
- Stops the REST task run
- Does not stop the underlying API operation invoked by the REST task
If a termination configuration is provided, then Data Integration stops the REST task run and the underlying API operation.
Authentication
Provide the authentication for invoking the REST API endpoint in the REST task.
Currently, you can use Oracle Cloud Infrastructure Resource Principal as the method to authenticate Oracle Cloud Infrastructure API endpoints. You can use workspace or application resource principal.
Resource Principal Authentication
The Oracle Cloud Infrastructure Resource Principal can be specified as the authentication method for executing an Oracle Cloud Infrastructure API endpoint in a REST task.
A resource principal enables resources to be authorized principal actors that can perform actions on service resources. The Oracle Cloud Infrastructure Resource Principal authentication method uses a temporary resource provider session token (RPST) for authentication, and authorization is established through Oracle Cloud Infrastructure Identity and Access Management (IAM) dynamic groups and policies. Permission must be granted to the Data Integration workspace or application, and network resources. Set up the following in IAM before you use Oracle Cloud Infrastructure Resource Principal:
-
Create a dynamic group.
-
To define the resources that belong to the dynamic group, create the following matching rule in the dynamic group:
ALL{resource.type = 'disworkspace', resource.compartment.id = <compartment-ocid>}
-
Add the following policy statement that permits the dynamic group access to network resources.
allow dynamic-group <dynamic-group-name> to use virtual-network-family in tenancy
-
To use the Oracle Cloud Infrastructure application resource principal to authenticate an API endpoint in a REST task:
allow any-user to use ai-service-language-family in tenancy where ALL {request.principal.type = 'disapplication', request.principal.id = '<disapplication-ocid>'}
Parameters
You can include parameters in your REST task, so that you can reuse the task in different runtime situations.
You can use parameters for the following parts of the REST request in a REST task:
- One or more sections of the REST execution URL, polling URL, or termination URL. In Data Integration, the parameterized variable sections of the URL are known as URL parameters.
- The value of a header in the request.
- The entire request body in a POST or PUT request.
- The success condition that determines the criteria for a successful completion of the REST call.
- The polling condition (written on the response that is returned from the polling request) that determines whether polling stops or continues.
Creating a REST Task
A REST task lets you call a REST API endpoint using the HTTP protocol.
You create a REST task in a project or folder. Data Integration includes one default project to get you started, but if you want to create your own, see Using Projects and Folders.
To create a REST task:
Configuring the HTTP method, URL, and URL Parameters
Specify the HTTP method and URL for the REST request to execute, and configure any URL parameters for the REST task.
The method is a standard HTTP method such as GET or POST. The URL is the REST API endpoint to the resource to access in the REST request.
For example, the following REST endpoint lists the data assets in the workspace that has the ID finance123:
http://myserver.com:8081/workspaces/finance123/dataAssets
The URL can include parameters by using the syntax ${}
. For example, the following REST endpoint lists the data assets in a workspace by using parameter syntax for the variable sections of the URL that specify the server host and the workspace ID.
http://${myserver}:8081/workspaces/${workspaceId}/dataAssets
When you specify parts of the URL using parameter syntax, those parts are converted into String URL parameters for the REST task. A default value must be assigned to each URL parameter.
When the task is run, you can change the values of those URL parameters. For example, at runtime you can supply different values for the server and the workspace ID.
To configure the URL for a REST task:
After entering the REST URL and pressing Enter, any section in the URL that uses the parameter syntax ${}
are converted into URL parameters for the REST task.
The default data type of all newly added URL parameters is String. On the URL parameters tab, you can add default values for the REST task parameters, and also change the parameter data type, if needed.
To configure URL parameters for the REST task:
- On the Define execution details step of the Configure REST API Details page, select URL parameters.
- In the table row for a newly added URL parameter, select Edit from the Actions menu.
- In the Edit parameter panel, the Identifier read-only field is the REST task parameter name for the URL variable section that is parameterized.
- To show or hide the REST URL using the parameter default values you have configured, click Show preview URL or Hide preview URL.
Adding and Managing a Header
Define one or more headers to pass to the header section of the REST request. A header has a key and a value.
For example, you can add the Content-Type
header for a POST or PUT request.
All standard REST API request headers are supported.
To create a header:
You can edit the key and value of a header.
To edit a header in a REST task:
Deleting a header removes the header key-value pair from the REST task.
To delete a header in a REST task:
After defining the key-value pair for a header, you can assign a task parameter to the header value.
To parameterize the header value of a header in a REST task:
You can modify the default value of the parameter only.
To edit the default value for a parameterized header value:
- On the Define execution details step of the Configure REST API Details page, select Header.
- In the table row for the header you want to modify, next to the parameter name, click Edit parameter.
- In the Edit parameter panel, modify the Value.
- Click Save.
Removing a header value parameter removes only the association of the parameter to the header value of the header.
Removing the header value parameter does not delete the header key-value pair.
To remove a header value parameter:
Providing a Request Body
Provide the request body that is required for a REST request that uses the POST or PUT method.
Currently, only JSON is the supported format for a request body.
To create a request body:
In addition to using parameters within the request body, you can parameterize the entire request body.
After you create the request body in the editor, then you can assign a task parameter to the entire body.
To parameterize the entire request body in a REST task:
You can modify the default value of the task parameter for the request body.
To edit the parameterized request body:
Removing the parameter removes only the association of the parameter to the request body for the REST task.
To unassign the parameter from the entire request body:
Providing Completion Criteria
Specify the condition that determines the criteria for a successful completion of the REST execution.
A success condition is required to complete any REST invocation, whether the API invokes a short-running or long-running operation.
To define the completion of the REST invocation for a long-running operation by using polling, see also Providing Completion Criteria by Using Polling.
Basic String and Operator functions are supported in success and polling conditions.
The expression for a success or polling condition can
include only the functions as shown in the following section. To extract values from
JSON structures, use the json_path
function.
Basic arithmetic |
|
Basic boolean |
|
Basic compare |
|
Basic null |
|
Basic string |
|
Basic date/time |
|
json_path function |
For JSONPath syntax elements, see:
For example, to retrieve the current status value from the following
response, use the syntax:
|
The success condition in a REST task is an expression that determines a successful completion of the REST API call. Polling stops when the success condition is met.
By default, Data Integration provides a success condition for a REST task. The default success condition is an HTTP status code of 200 or greater but less than 300, written as the following expression:
SYS.RESPONSE_STATUS >= 200 AND SYS.RESPONSE_STATUS < 300
The success condition expression references output in the JSON response that is received from the REST request defined in the execution step.
You can use the default success condition, or you can write your own condition expression using system outputs or extracted JSON property values from the response.
To edit the success condition:
You can assign a task parameter to the success condition.
To parameterize the success condition in a REST task:
You can modify the description and default value of the task parameter for the success condition.
To edit the parameterized success condition:
- On the Configure REST API details page, go to the Specify completion criteria step.
- If your REST API invokes a long-running operation, select the Configure a polling and termination condition for a no-wait REST call check box.
- In the Conditions block, next to the parameterized Success condition, click Edit parameter.
- In the Edit parameter panel, edit only the description and the expression for the success condition. You cannot change the data type.
- Click Save changes.
Removing the condition parameter removes only the association of the parameter to the success condition specified for the REST task.
To unassign the parameter from the success condition:
Providing Completion Criteria by Using Polling
For a REST task that invokes a long-running REST API operation, select the Configure a polling and termination condition for a no-wait REST call check box on the Specify completion criteria step to specify the polling configuration for identifying the completion of the REST invocation.
In addition to the success condition, polling configuration includes a polling REST URL and condition, and values for a polling interval and polling timeout.
Optionally, you can specify a termination configuration to cancel the long-running API operation.
To use the API response from the execution step in the polling or termination configuration, create expressions and then include the named expressions in the polling and termination URLs, and the polling and success conditions.
Before you configure polling or termination, create the expressions to fetch the API response values you need.
See also Polling in Long-Running Operations.
Expressions let you assign a value or expression to a variable that you can then use when configuring the completion criteria step.
Create expressions that use the API response from the REST execution step. You
provide a name for the expression when you create it. To include the named
expression in the polling or termination URL, or the polling or success condition,
use the syntax #{expression_name}
.
For example, if the REST task invokes the API operation to create a project using a specific project name, the polling request can check the project key to determine whether polling continues or stops. To use the project key as a variable in the polling URL, create an expression that extracts the key from the API payload response, converting it to a value of datatype String.
To add an expression:
You can edit and delete expressions that you have created in polling.
To edit or delete an expression:
- On the Configure REST API details page, Specify completion criteria step, expand the Expressions section.
- In the Expressions table, locate the expression you want to edit or delete.
-
From the Actions menu of the expression, select Edit or
Delete.
- In the Edit expression panel, make the changes, then click Save.
- In the Delete expression dialog, confirm that the named expression is the one you want to delete, then click Delete.
In a long-running API operation, define the HTTP method and the URL to poll the status of the REST call.
Similar to the REST URL in the execution step, you can include parameters in the
polling URL by using the syntax ${parameter_name}
.
In addition, you can include named expressions using the syntax
#{expression_name}
. Create
the expressions you need before you configure the polling URL.
To configure the polling method and URL:
The polling configuration includes a polling condition, and values for a polling interval and polling timeout.
The polling condition is an expression that determines whether polling stops or continues. When the expression returns false, polling stops.
Data Integration issues a polling call repeatedly at the specified polling interval until the value for the specified polling timeout is reached, or until the polling condition returns false, whichever occurs first.
To configure the polling condition:
- On the Configure REST API details page, Specify completion criteria step, select the Configure a polling and termination condition for a no-wait REST call check box.
-
On the Polling tab, Conditions
section, do the following:
-
In the Conditions section, below Polling
condition, enter a value and choose a unit of measurement for
Polling interval and Polling
timeout.
- Polling timeout: The maximum length of time that is allowed for repeated polling to occur at the specified interval rate. The timeout value must be greater than or equal to 120 seconds but less than or equal to 30 days.
- Polling interval: The length of time to wait before sending the next polling request. The interval value must be greater than or equal to 60 seconds, and less than the specified timeout value.
- To assign a task parameter to the condition, click Assign parameter. See Parameterizing the Polling Condition.
After adding a polling condition, you can assign a task parameter to the condition.
To parameterize the polling condition in a REST task:
You can modify the description and default value of the task parameter for the polling condition.
To edit the parameterized polling condition:
- On the Configure REST API details page, go to the Specify completion criteria step.
- In the Conditions block, next to the parameterized Polling condition, click Edit parameter.
- In the Edit parameter panel, you can edit only the description and the expression for the polling condition. You cannot change the data type.
- Click Save changes.
Removing the condition parameter removes only the association of the parameter to the polling condition specified for the REST task.
To unassign the parameter from the polling condition:
For a REST task that invokes a long-running REST API operation, you can specify the REST API details for terminating the long-running operation.
On the Termination tab, define the HTTP method and the URL to terminate the REST call.
Similar to the polling URL, you can include parameters and expressions in the
termination URL by using the syntax ${parameter_name}
and
#{expression_name}
. Create
the expressions you need before you configure the termination URL.
To configure the termination method and URL:
The success condition in a REST task is an expression that determines a successful completion of the REST API call. Polling stops when the success condition is met.
By default, Data Integration provides a success condition for a REST task. The default success condition is an HTTP status code of 200 or greater but less than 300, written as the following expression:
SYS.RESPONSE_STATUS >= 200 AND SYS.RESPONSE_STATUS < 300
The success condition expression references output in the JSON response that is received from the REST request defined in the execution step.
You can use the default success condition, or you can write your own condition expression using system outputs or extracted JSON property values from the response.
To edit the success condition:
Providing Authentication in the REST Task
By default, no authentication is needed to execute the REST endpoint in a REST task.
You can, however, use Oracle Cloud Infrastructure Resource Principal to authenticate an Oracle Cloud Infrastructure API endpoint in a REST task.
You must set up Oracle Cloud Infrastructure Identity and Access Management (IAM) dynamic groups and policies before you can use OCI Resource Principal for authentication. See Authentication.
To specify the authentication to use for the REST task:
Viewing and Managing Parameters in a REST Task
View and manage the parameters that are added or used in the scope of one REST task.
Open the REST task in the project or folder to see a list of the parameters created and assigned in the task. Unassigned parameters are also available in the list.
You can add scalar, JSON, and expression parameters in the scope of a REST task, for referencing in parameterized elements within the same REST task.
- Open the REST task in the project or folder.
- In the Parameters section of the REST task page, click Configure.
- On the REST task parameters page, click Add parameter and select the type of parameter to add from the menu.
-
In the Add parameter
panel, you can define the following parameter
types.
- Scalar: Add a conventional name-value pair parameter. Specify a unique identifier, the data type, and the default value.
- JSON: Add a JSON data type parameter. Specify a unique identifier, and a valid JSON as the value.
- Expression: Add an expression type parameter. Specify a unique identifier, then use the condition builder to write the expression.
- Click Add.
- Click Done on the REST task parameters page.
You can edit the description, data type, and value of a parameter.
You can delete a parameter only if the parameter is not in use in the REST task.
The Used in column on the REST Task Parameters page indicates where a parameter is being used in the task.
Validating a REST Task
To validate a REST task at any time:
Editing a REST Task
You can edit a REST task to modify any of the configured details, including parameters and parameter values.
You edit a task from the project or folder details page where the task is saved.
Deleting a REST Task
If you want to delete a REST task, you can do so from the tasks list on a project or folder details page. After a task is deleted, it cannot be restored.
To delete a REST task:
Using the API
For information about using the API and signing requests, see REST APIs and Security Credentials. For information about SDKs, see Software Development Kits and Command Line Interface.
Use the following operations in Oracle Cloud Infrastructure Data Integration to manage tasks: