SFTP Task

3.2.8 SFTP Task

The SFTP system task allows workflows to securely transfer files—including uploading, downloading, and listing—directly to and from remote SFTP servers, eliminating the need to develop custom simple tasks or worker tasks.

This capability supports a widespread enterprise integration pattern commonly found in industries such as finance, healthcare, government, retail, and supply chain, where batch file exchanges are a standard process. By providing a straightforward, no-code, declarative interface for SFTP operations, this task streamlines integration, minimizes development effort, and leverages existing SFTP connector profiles in MicroTx Workflows for simplified configuration. This results in faster implementation, enhanced data security, and easier maintenance of file-based data exchanges.

Note:

The maximum size of a file that can be transferred is 50 MB.

The SFTP task maintains high watermark (an internal timestamp) for each workflow version. When the task runs again, it processes only the files that were added or modified after the last recorded timestamp. For example, let's consider that the SFTP task runs at 5 PM and processes all existing files. During the next execution, the SFTP task considers only files added or updated after 5 PM within the same workflow version.

To add an SFTP Task

Navigate to the Task tab in a workflow and view all the tasks that you can add using the Workflow Builder. See Access the Task Tab in Workflow Builder.
In the More Tasks dialog box, click SFTP Task to add it to the workflow.
Click the task that you have added in the left pane. The Task tab in the right pane displays details about the task, such as its name and parameters. Next, let's provide details for the task.
In the Task Details group, enter the following information.
- Task Name: Mandatory. Enter a unique name for the task. The name must be between 1 to 128 alphanumeric characters in length and cannot contain spaces or any special characters. Optionally, you can use underscore (_) and hyphen (-).
- Task Reference: Mandatory. Enter a value to refer to the task within a workflow definition. This value must be unique within a workflow. The task reference value must be between 1 to 128 alphanumeric characters in length and cannot contain spaces or any special characters. Optionally, you can use underscore (_) and hyphen (-).
In the SFTP Task Parameters group, in the Operation drop-down list select one of the following tasks that you want to perform with the SFTP server.
- Upload: Transfers files to the specified remote SFTP directory. Reads files from the local file system storage repository, filters them by pattern, and securely uploads them to a designated remote SFTP directory with MD5 checksum validation.
- Download: Retrieves files from a remote SFTP directory and saves the files to your chosen destination, such as local file system or Oracle Cloud Infrastructure (OCI) Object Storage. Additionally, based on the settings, it filters the retrieved results by file name pattern, tracks when files were last processed, and calculates MD5 checksum for verification.
- List: Displays names of all the files available in the specified remote SFTP directory.
- Exists: Verifies the presence of a file in the specified remote SFTP directory.
If you select the Upload operation, provide the following information and then skip to the last step to save the changes.
- Source Type: Select one of the following as the source, and then provide the connection details.
  - File Resource: Takes files from the file system repository, and then stores the files in the specified destination SFTP server. When you select this option, the default base path of the file storage is the value specified in the conductor.connector.storage-path property in the application.properties file. When you select this source type, optionally you can provide a Java regular expression to filter file names in the specified remote directory in the File Name Pattern field. Case-insensitive matching is supported. If you do not enter a value, all executable files are filtered by default.
  - OCI Bucket: Takes files in the specified OCI Object Storage bucket in the provided namespace, and then stores the files in the specified destination SFTP server. SFTP task uses the details provided in the selected Cloud profile to establish a connection with the destination Object Storage bucket.
    - Cloud Profile: Select the Cloud profile containing the credentials and configuration required to access OCI Object Storage.
    - OCI Namespace: Enter the Object Storage namespace associated with your OCI Object Storage bucket.
    - OCI Bucket Name: Enter the name of the existing bucket in your OCI Object Storage where files should be stored. Buckets are logical containers for storing objects.
    - OCI Object Prefix: Optional. Specify a prefix to be added to all objects uploaded to the bucket.
- Overwrite: Optional. Select this option to always overwrite any existing file with the new file during upload. This option is disabled by default.
- Maximum File Transfer: Optional. Enter a natural number to specify the maximum number of files to transfer in a single task execution. The default value is 10. This restricts the number of files for better control over large batch transfers.
- Destination SFTP Profile: Mandatory. Select an existing SFTP connector profile that contains the connection settings for the destination SFTP server to which you want to upload files.
- Destination Directory: Mandatory. Enter the path of the directory in SFTP server to which you want to upload files.
If you select the Download operation, provide the following information and then skip to the last step to save the changes.
- Source SFTP Profile: Mandatory. Select an existing SFTP connector profile that contains the connection settings for the source SFTP server from which you want to download files.
- Source Directory: Mandatory. Enter the path of the directory in SFTP server from which you want to download files.
- File Name Pattern: Optional. Provide a Java regular expression to filter file names in the specified remote directory. Case-insensitive matching is supported. If you do not enter a value, all executable files are filtered by default.
- Destination Type: Select one of the following as the destination, and then provide the connection details.
  - File Resource: Stores files received from the specified source SFTP server to the MicroTx Workflows file storage.
  - OCI Bucket: Stores files in the specified OCI Object Storage bucket in the provided namespace and uses the details provided in the Cloud profile selected in the Cloud Profile to establish a connection with the destination Object Storage bucket.
    - Cloud Profile: Select the Cloud profile containing the credentials and configuration required to access OCI Object Storage.
    - OCI Namespace: Enter the Object Storage namespace associated with your OCI object storage bucket.
    - OCI Bucket Name: Enter the name of the existing bucket in your OCI Object Storage where files should be stored. Buckets are logical containers for storing objects.
    - OCI Object Prefix: Optional. Specify a prefix to be added to all objects uploaded to the bucket.
  - SFTP: Transfers files to the SFTP directory specified in the Destination Directory field and uses the details provided in the SFTP profile selected in the Destination SFTP Profile to establish a connection with the destination SFTP server.
- Idempotency Strategy: Select one of the following methods to handle existing files on the destination server to ensure safe, repeatable operations.
  - none: Always perform the operation, without checking for existing files.
  - skipIfExists:Skip the operation if the file already exists on the destination.
  - overwrite: Always overwrite any existing file with the new file.
  - checksum: Compare files using MD5 checksum; only transfer if local and remote files differ.
- Maximum File Transfer: Enter a natural number to specify the maximum number of files to transfer in a single task execution. The default value is 10. This restricts the number of files for better control over large batch transfers.
If you select the List operation, provide the following information and then skip to the last step to save the changes.
- Source SFTP Profile: Mandatory. Select an existing SFTP connector profile that contains the connection settings for the source SFTP server from which you want to list file names.
- Source Directory: Mandatory. Enter the path of the directory in SFTP server from which you want to download files.
- File Name Pattern: Optional. Provide a Java regular expression to filter file names in the specified remote directory. Case-insensitive matching is supported. If you do not enter a value, all executable files are filtered by default.
If you select the Exists operation, provide the following information and then skip to the last step to save the changes.
- Source SFTP Profile: Mandatory. Select an existing SFTP connector profile that contains the connection settings for the source SFTP server from which you want to download files.
- Source File: Mandatory. Enter the path of the source file on the SFTP server to verify if the file exists.
Click Save to save the changes to the workflow.

MicroTx Workflows displays the changes in JSON code.
Review all the changes, and then click Confirm Save to save the changes.

If you do not want to save the changes, click Cancel, and then click Reset to undo all the changes that you have made since the workflow was last saved.

Example

When you enter information in the Task tab, the corresponding code of the task is updated in the JSON tab. The following example displays the JSON code for an SFTP task which downloads files from a specified SFTP server and uploads them to the provided OCI Object Storage bucket.

{
    "name": "Download files from SFTP Server to OCI Object Storage Bucket",
    "taskReferenceName": "download_files_SFTP_server",
    "inputParameters": {
        "operation": "DOWNLOAD",  
        "sourceSftpProfileName": "sftp-profile-name", 
        "sourceDirectory": "/home/ubuntu/inbound",
        "fileNamePattern": "^.*\\.xml$",                
        "destinationType": "OCI_BUCKET",               
        "ociNamespace": "oabcs1",                            
        "ociBucketName": "microtx",
        "ociProfile": "phoenix0",
        "idempotencyStrategy": "skipIfExists",
	 "maximumFileTransfer": 5
    },
    "type": "SFTP"
}

Parent topic: Create System Tasks and Operator Tasks