File Transfer Wrapper Service

Overview

Oracle Cloud Infrastructure Object Storage is an internet-scale, high-performance storage platform that offers reliable and cost-efficient data durability.

In addition to the Legacy File Transfer Integration Service, a new File Transfer Services (FTS) Wrapper Service is added to allows customer to manage files in object storage. The FTS Wrapper service provides a directory-based access to files using a storage prefix and filename. The storage prefix represents a directory path.

File Transfer Wrapper Services provides a common way to allow customer to upload to/download from object storage direct via common platform implementation.

For each customer environment, logical containers (buckets) are created in Object Storage for storing objects used by the cloud application. The file transfer bucket name is created and set when the environment is provisioned. The bucket name is required to move files between Oracle Cloud and your local system using file transfer services.

Access to files is through a Pre-Authenticated Request (PAR), which is a URL that requires no further authentication to upload or download to the application's object storage. To retrieve a PAR, you must use the appropriate FTS Wrapper service.

The File Transfer Wrapper service enables external application to import files to and export files from Object Storage used by the solutions.

These APIs provides following operations:

Ping
List storage prefixes
List files in object storage
Move Files
Delete Files
Request Upload PAR
Request Download PAR

Service Use Cases

Use Case 1: External Application lists the files in object storage via Service API.

Use Case 2: External Application Uploads data files via Service API to object storage.

Use Case 3: External Application downloads data files via Service API from object storage.

Use Case 4: External application user imports/uploads files use File Transfer Service UI.

Prerequisites

Before you can use File Transfer Wrapper Service, you need to have the following resources in your Oracle Cloud Infrastructure tenancy:

Object Storage bucket
File Transfer Wrapper Service URL and Scopes
Setup Client ID and Client Secret
Configure FTS Wrapper User

Some of these resources are automatically created for you upon SIOCS deployments. See details about FTS service in the following sections.

Object Storage Bucket

For each customer environment, logical containers (buckets) are created in Object Storage for storing objects used by the cloud application. The file transfer bucket name is created and set when the environment is provisioned.

File Transfer Wrapper Service URL and Scopes

Upon SIOCS deployment, File Transfer Wrapper Service URL and scopes are automatically created, by default the service is disabled. Customer would need to enable the File Transfer Wrapper Service if they intent to use this service.

OAUTH Security Considerations

Setup Client ID and Client Secret

File Transfer Wrapper Service use OAuth2.0 for authorization. The Customer Administration users must create their own client credential IDCS application in IDCS.

Once the client id and client secret are generated, please notes down the credentials, those will be used to get the Authentication Token for using File Transfer Wrapper Service.

When creating the Credentials, the App name and scope for the FTS Wrapper IDCS client application creation should be environment specific using the formats as below:

App Name: RGBU_SIOCS_<ENV>_EICS_FTS_WRAPPER

Where <ENV>: DEV1, STG1, PROD1

For example, RGBU_SIOCS_STG1_ EICS_FTS_WRAPPER

Scope: rgbu:siocs:psraf-<ENV>

For example, rgbu:siocs:psraf-STG1

See the Oracle® Retail Home Administration Guide OAuth Application Configuration chapter for Creating OAuth Client Applications.

How to Switch Using File Transfer Wrapper Service

Configure FTS Wrapper Credentials

The FTS Wrapper user credentials are required for SIOCS batch upload/download files via File Transfer Wrapper Service.

Login SIOCS JET webclient.
From task menu, under Admin, click Technical Maintenance, then click Credential Administration.
Select Alias ftswrapper-user, update the client id and client secret with the one that was created in step “Setup Client ID and Client Secret”.
Click Apply and Save .

Enable File Transfer Wrapper Service

Upon SIOCS deployment, by default the service will be disabled. To enable the service, customer will need to perform following steps:

Login SIOCS JET.
From task menu, under Admin, click Technical Maintenance, then click External Service Administration.
Edit the FtsWrapperExternalService, set Active to Yes.
Click Apply and Save.

Update Your External Applications to Use FTS Wrapper Service APIs

Evaluate if your application is accessing the object storage via SIOCS legacy FTS endpoint (siocs-int-services/api/fts), then you are on earlier version of the service, update your application to point to the FTS Wrapper service endpoints. See section How to use File Transfer Wrapper Service APIs for details.

How to Use File Transfer Wrapper Service APIs

To interact with FTS, you must use the REST APIs provided. The endpoints URLs are relative to service base URL.

Service Base URL

The File Transfer Wrapper Service Base URL is automatically configurate upon SIOCS deployment.

The URL has the following format:

https://rex.retail.<Region Name>.ocs.oraclecloud.com/<Customer Subnamespace>/RetailAppsPlatformServices/services/private/FTSWrapper/

Note:

The <Region Name> and <Customer Sub Namespace> part of the URL should be replaced with the one specific to your environment. This will be the same as your cloud service application URL provided in the welcome email.

To view the URL via SIOCS External Service Administration Screen.

Login SIOCS JET.
From task menu, under Admin, click Technical Maintenance, then click External Service Administration.

Service Endpoints

The table below lists the API end points for different file operations. See File Transfer Wrapper Service API Specifications for details.

Table 8-1 FTS Endpoints

Service	Method	FTS Wrapper Endpoint URLs
Ping	GET	{Service Base URL}/ping
List Prefixes	GET	{Service Base URL}/listprefixes
List Files	GET	{Service Base URL}/listfiles
Move Files	POST	{ Service Base URL}//movefiles
Delete Files	DELETE	{Service Base URL}/delete
Request Upload PAR	POST	{Service Base URL}/upload
Request Download PAR	POST	{ Service Base URL}/download

Retrieving Access Client Token

IDCS OAuth2 Token URL

IDCS token URL to obtain Oauth2 token.

Example IDCS_TOKENT_URL:

https://idcs-XXXXXXXX.identity.oraclecloud.com/

Using the above URL,

IDCS_TOKENT_URL = {IDCL_BASE_URL}/oauth2/v1/token

The following is required in headers for making OAuth2.0 enabled REST Services.

Please contact customer's IDCS administrator for FTS Client ID and Client Secret.
An access token using the Client ID and secret from IDCS.

Example: get access Token Use Curl

Note:

You need to have curl and jq client tool installed on your client machine for using curl for testing.

export ACCESS_TOKEN="$(curl -u <Client ID>:<Secret> -H 'Content-Type: application/x-www-form-urlencoded;charset=UTF-8' --request POST https:// <IDCS_BASE_URL>/oauth2/v1/token -d 'grant_type=client_credentials&scope=<Scope>' | jq -r '.access_token')"

In the example above, substitute the variables with proper values for your environment.

Service Request Headers

Each call to FTS Endpoint should contain the following Request headers:

Content-Type: application/json
Accept: application/json
Accept-Language: en
Authorization: Bearer {ACCESS_TOKEN}

Before calling FTS Wrapper API, you need to get the ACCESS_TOKEN use step Retrieving Access Client Token.

How to Use Service API to find Object Storage Prefixes

First you need to get the ACCESS_TOKEN use step Retrieving Access Client Token, then you may call the endpoint List Prefixes as below:

Sample Request:

curl --request GET <Service Base URL>/listprefixes -H 'content-type: application/json' -H 'Accept: application/json' -H 'Accept-Language: en' -H "Authorization: Bearer ${ACCESS_TOKEN}"

Sample Response:

{"values":["archives","rejects","imports","exports"]}

How to Use Service API to List Files in Object Storage

First get the ACCESS_TOKEN using step Retrieving Access Client Token, then call the endpoint List Files as below:

Sample Request:

curl --request <Service Base URL>/listfiles?contains=RFID -H 'content-type: application/json' -H 'Accept: application/json' -H 'Accept-Language: en' -H "Authorization: Bearer ${ACCESS_TOKEN}"

Sample Response:

{"limit":999,"count":1,"offset":0,"hasMore":false,"resultSet":[{"name":"imports/ RFID_1.dat","createdDate":"2022-02-13T21:35:26Z","modifiedDate":"2022-02-13T21:35 :26Z","scanStatus":"Passed","scanDate":"2022-02-13T21:35:56.187Z","md5":"xxxx==", "version":"xxxxx","etag":"zzzzzzz","size":75}]}

How to Use Service APIs to Upload Import Data Files to Object Storage

To upload data files to object storage, the external solution application needs to perform following steps:

The external application gets the Oauth2 token from IDCS. See the Retrieving Access Client Token section.
The external application makes an FTS request with the Oauth2 token to requesting Pre-Authentication.

Then call the endpoint Request Upload PAR as below:

Sample Request:

curl --request POST <Service Base URL>/upload -H 'content-type: application/json' -H 'Accept: application/json' -H 'Accept-Language: en' -H "Authorization: Bearer ${ACCESS_TOKEN}" -d "{\"listOfFiles\": [{\"storagePrefix\": \"imports\",\"fileName\": \"EXTPC_1.dat\"}, {\"storagePrefix\": \"imports\",\"fileName\": \"RFID_1.dat\"}]}"

Sample Response:

{"parList":[{"id":"zzzzzzz/:imports/EXTPC_1.dat","name":"EXTPC_1.dat","accessUri":"https://objectstorage.us-ZZZ-siocs/o/imports/ EXTPC_1.dat","accessType":"ObjectWrite","timeExpires":"2022-02-13T21:39:40.265Z", "timeCreated":"2022-02-13T21:34:40.329Z","objectName":"imports/EXTPC_1.dat"}, {"id":"ZZZZ:imports/RFID_1.dat","name":"RFID_1.dat","accessUri":"https://zzzz-siocs/o/imports/ RFID_1.dat","accessType":"ObjectWrite","timeExpires":"2022-02-13T21:39:40.411Z"," timeCreated":"2022-02-13T21:34:40.472Z","objectName":"imports/RFID_1.dat"}]}
Once the PAR is received, the external application uploads the file to object storage using the URL included within the response.

Sample Request:

curl https://ZZZZZ-siocs/o/imports/RFID_1.dat --upload-file C:\\temp\\RFID_1.dat
Files uploaded to application object storage will be processed by cloud application batches.

How to Use Service APIs to Download Data Files from Object Storage

For retailer to download the export data files from application object storage, perform the following steps:

The external solution application gets the Oauth2 token from IDCS. See the Retrieving Access Client Token section.
The external solution application calls the FTS service with the Oauth2 token to list the available export files in Object Storage which are generated by cloud app.

Sample Request:

curl --request GET <Service Base URL>/listfiles?contains=RFID -H 'content-type: application/json' -H 'Accept: application/json' -H 'Accept-Language: en' -H "Authorization: Bearer ${ACCESS_TOKEN}"

Sample Response:

{"limit":999,"count":1,"offset":0,"hasMore":false,"resultSet":[{"name":"imports/ RFID_1.dat","createdDate":"2022-02-13T21:35:26Z","modifiedDate":"2022-02-13T21:35 :26Z","scanStatus":"Passed","scanDate":"2022-02-13T21:35:56.187Z","md5":"xxxxx==" ,"version":"xxxxx","etag":"ZZZZZ","size":75}]}
The external solution application calls the FTS service with the Oauth2 token, requesting Pre-Authentication to download files from object storage used by cloud app.

Sample Request:

curl --request POST <Service Base URL>/download -H 'content-type: application/json' -H 'Accept: application/json' -H 'Accept-Language: en' -H "Authorization: Bearer ${ACCESS_TOKEN}" -d "{\"listOfFiles\":[{\"storagePrefix\": \"imports\",\"fileName\": \"RFID_1.dat\"}]}"

Sample Response:

{"parList":[{"id":"i91P0nFIIsgj05qrUH2ibTZ2npmbTdq1TKsGtWOerAYaE6/MYZE7840lR/ QEhaFk:imports/RFID_1.dat","name":"RFID_1.dat","accessUri":"https:// objectstorage.us-phoenix-1.oraclecloud.com/p/ ZG89KsLS_5SY7D2p7nVQt8KfJ6rLJ40FSmI97zASLRK2VrsICbvoRP0bgoQGxk3S/n/ZZZZZ-siocs/o/ imports/RFID_1.dat","accessType":"ObjectRead","timeEx-pires":"2022-02-13T23:07:00.962Z","timeCreated":"2022-02-13T23:02:01.105Z","objec tName":"imports/RFID_1.dat"}]}
Once the PAR is received, the file is downloaded using the URL included within its response. A PAR is valid for 10 minutes. A download can take longer than 10 minutes, but it must be started within 10 minutes of the PAR being received.

curl -o <destinationFileName> -X GET <PAR>

For example:

curl -o RFID_1_download.dat -X GET https://ZZZZZ-siocs/o/imports/RFID_1.dat

File Transfer Service UI

In addition to the FTS Wrapper Service REST Services, SIOCS File Transfer Service UI provides user to upload/download a file or view a list of files in object storage.

Security Consideration to access File Transfer Service UI:

The customer admin user should be assigned the following PSRAF groups to access the FTS JET UI

PSRAF Users and PSRAF Admin Users

The naming convention for the group is <EICS_IDCS_APP_DISPLAY_NAME>.psraf_users where EICS_IDCS_APP_DISPLAY_NAME is the display name of EICS app on IDCS or OCI IAM.

The naming convention for the group is <EICS_IDCS_APP_DISPLAY_NAME>.psraf_admin_users where EICS_IDCS_APP_DISPLAY_NAME is the display name of EICS app on IDCS or OCI IAM
Security Permission Access File Transfer Service must be assigned to user who needs to access the File Transfer Service UI.
The IDCS or OCI IAM Application role admin_users is required for the user to perform the upload/download operations.

Figure 8-1 File Transfer Service UI

The main form lists the recently uploaded files.

Actions:

To filter the files by store prefix, select a file storage prefix.
To filter by file name by choosing the Actions choice selector on the screen.
To upload new files, click Upload New File button:

Figure 8-2 Upload New File

In the Upload New File popup dialog, choose storage prefix Imports and click Add File button.

Figure 8-3 Upload New File Dialog

Next, choose files from your client machine, then click Upload:

Figure 8-4 File Added

Once the uploaded file has passed a virus scan, the file is ready for a cloud application batch to import the file from object storage into the application.

Figure 8-5 Recent Uploads

File Transfer Wrapper Service API Specification

This section describes FTS API specifications.

Ping

Returns the status of the service and provides an external health-check.

Method	GET
Endpoint	{Service Base URL}/ping
HTTP Header	See FTS Wrapper API Call Headers.
Request Body	None
Response	`"200": { "description": "OK - The service operation produced a successful response." }, "400": { "description": "Bad Request - The path params or query params or body was not valid for this operation." }`

List Prefixes

Returns a list of the known storage prefixes. These are analogous to directories and are restricted to predefined choices per service. SIOCS has list of pre-defined storage prefixes: import, exports, rejects and archives.

Method	GET
Endpoint	{Service Base URL}/listprefixes
HTTP Header	See FTS Wrapper API Call Headers.
Request Body	None
Response	A JSON array of strings containing the known prefixes. `{ "200": { "description": "OK - The service operation produced a successful response." }, "400": { "description": "Bad Request - The path params or query params or body was not valid for this operation." } }`

List Files

Returns a list of the files within a given storage prefix.

Method	GET
Endpoint	{Service Base URL}/listfiles
HTTP Header	See FTS Wrapper API Call Headers.
Parameters	{ "name": "prefix", "description": "The object filter in object storage.", "in": "query", "required": false, "schema": { "type": "string" } }, { "name": "contains", "description": "The object filter in object storage.", "in": "query", "required": false, "schema": { "type": "string" } }, { "name": "scanStatus", "description": "The object filter in object storage.", "in": "query", "required": false, "schema": { "type": "string" } }, { "name": "offset", "description": "The object filter in object storage.", "in": "query", "required": false, "schema": { "type": "integer" } }, { "name": "limit", "description": "The object filter in object storage.", "in": "query", "required": false, "schema": { "type": "integer" } }, { "name": "sort", "description": "The object filter in object storage.", "in": "query", "required": false, "schema": { "type": "string" } } ],
Request Body	None
Response	A JSON resultSet containing array of files. For each file, there is metadata including: name, size, created and modified dates, scan status and date, scan output message. `{ "200": { "description": "OK - The service operation produced a successful response." }, "400": { "description": "Bad Request - The path params or query params or body was not valid for this operation." }`

Move Files

Moves one or more files between storage prefixes, while additionally allowing the name to be modified.

Method	POST
Endpoint	{Service Base URL}/movefiles
HTTP Header	See FTS Wrapper API Call Headers.
Request Body	`{"listOfFiles": [ {"currentPath": { "storagePrefix": "string", "fileName": "string"}, "newPath": { "storagePrefix": "string", "fileName": "string" } } }}`

Delete Files

Deletes one or more files.

Method	DELETE
Endpoint	{Service Base URL}/ delete
HTTP Header	See FTS API Call Common Headers
Request Body	A JSON array of files to be deleted. One or more pairs of storagePrefix and filename elements can be specified within the array. Required: `true{ " {"listOfFiles": [ [ { " {"storagePrefix": "string", "fileName": "string" } ]}`
Response	200 The request has succeeded. 500 The server has encountered a situation it does not know how to handle.

Request Upload PAR

Request PAR for uploading one or more files.

Method	POST
Endpoint	{Service Base URL}/upload
HTTP Header	See FTS Wrapper Request Headers.
Request Body	A JSON array of files to be uploaded. One or more pairs of storagePrefix and filename elements can be specified within the array. Required: true `{ "listOfFiles": [ { "storagePrefix": "string", "fileName": "string" } ] }`
Response	A parList containing an array containing elements corresponding to the request including the PAR accessUri and name of file. `{ "parList": [ { "id": "string", "name": "string", "accessUri": "string", "objectName": "string", "accessType": "string", "timeExpires": "2021-09-07T16:35:27.390Z", "timeCreated": "2021-09-07T16:35:27.390Z" } ] } Response Status: { "200": { "description": "OK - The service operation produced a successful response." }, "400": { "description": "Bad Request - The path params or query params or body was not valid for this operation." }`

Request Download PAR

Request PAR for downloading one or more files.

Method	POST
Endpoint	{Service Base URL}/download
HTTP Header	FTS Wrapper Service Request Headers.
Request Body	A JSON array of files to be downloaded. One or more pairs of storagePrefix and filenames can be specified within the array. Required: true `{ "listOfFiles": [ { "storagePrefix": "string", "fileName": "string" } ] }`
Response	A parList containing an array containing elements corresponding to the request including the PAR accessUri and name of file. `"parList": [ { "id": "string", "name": "string", "accessUri": "string", "objectName": "string", "accessType": "string", "timeExpires": "2021-09-07T16:35:27.390Z", "timeCreated": "2021-09-07T16:35:27.390Z" } ] } Response Status: { "200": { "description": "OK - The service operation produced a successful response.`

File Transfer Wrapper Service Troubleshooting

These troubleshooting topics covers common file transfer service issues and possible solutions.

Troubleshooting File Transfer Service Internal Server Error

Try to connect to File Transfer Ping endpoint. If you can connect ping endpoints, continue to step2.
Try to invoke List Files endpoint, if get response status 200.
If the above steps do not resolve the internal server error, you may raise a Service Request on My Oracle Support.