7 Batch Administration

This chapter addresses the following batch administration tasks:

Run Batch Task Group
Run Batch Measure Load Group
Run Batch Measure Export Group
Run Batch Calc Group
Uploading and Downloading Data Files

Application-Specific Batch Tasks

The application-configured batch tasks are a set of tasks that help the administrator perform routine batch processing operations, either in a recurring schedule (such as nightly and/or weekly) or loading and exporting a predefined set of data and hierarchies. These tasks are found in the Online Administration Tools as Configured Batch Tasks.

Each task in turn calls one or more batch services to do the required tasks. The catalog of available batch services to configure is listed and described in the Oracle Retail Predictive Application Server Cloud Edition Implementation Guide.

. Some tasks require data files (such as measure/fact load files or hierarchy/dimension load files) as input. These files must be uploaded via Object Storage before the tasks are scheduled. Some tasks export files from the application's data store. Those files are copied to the Object Storage location, and Administrators can download these exported files. For more information, see "Uploading and Downloading Data Files."

The following table shows the list of Online Admin task entries available for running or scheduling the application-configured batch tasks. Note that these are only a selection of the configured task types available. Any that are not directly available here may be run by executing the Batch Task Group of which they are a part.

Configured Batch Tasks
Run Batch Task Group
Run Batch Measure Load Group
Run Batch Measure Export Group
Run Batch Calc Group

Run Batch Task Group

This task is used to schedule a predefined set of batch tasks in a sequential order. The batch control file batch_exec_list.txt defines the steps in the specific order that will be executed during batch task execution. This task is typically used to schedule a daily or weekly batch. If any task aborts, a log is created and the process terminates. This task has the option to restart from the last failed service rather than running all the completed services again.

Depending on the type of application implementation, the Run Batch Task Group task can either be extended (see the application-specific Implementation Guides) or completely customized (see RPASCE Implementation Guide), via the batch_exec_list.txt control file. For more details about the list of internal services this task can call and how to change the order of services to this task, see the Oracle Retail Predictive Application Server Cloud Edition Implementation Guide.

Run Batch Measure Load Group

This task is used to load a predefined set of measures grouped as a measure set in the batch control file batch_loadmeas_list.txt. All required data files must be uploaded to the Object Storage location before scheduling this task. For more information, see "Uploading and Downloading Data Files." If no file is present to load a particular measure in that measure set, it will be logged as not found and skipped. This process will only load the data into those measures and will not run any subsequent calculations. (If follow-on operations such as calculations or workbook refreshes are desired, then the Batch Task Group must be used, rather than the Measure Load Group.)

The Run Batch Measure Load Group task can be customized to load different sets of measures by making changes to the batch control file. For information about changing the batch control files and how to upload them, see the Oracle Retail Predictive Application Server Cloud Edition Implementation Guide.

Run Batch Measure Export Group

This task exports a predefined set of measure data into flat files. The set of measures to be exported are defined in the batch control file batch_exportmeas_list.txt. This set of exported data can be used for integration with other applications. Exported files from this task are put into the Object Storage location; the administrator can download the files from this location. For more information, see "Uploading and Downloading Data Files."

The Run Batch Measure Export Group task can be customized by changes to the batch control file. For information about changing the batch control files and how to upload them, see the Oracle Retail Predictive Application Server Cloud Edition Implementation Guide.

Run Batch Calc Group

This task is used to run a predefined set of batch rules to do the necessary data processing, as defined in the batch control file batch_calc_list.txt. The Run Batch Calc group task can be customized to change the batch calculations and the order in which they run against the application by changing the batch control file. For information about changing the batch control files and how to upload them, see the Oracle Retail Predictive Application Server Cloud Edition Implementation Guide.

Uploading and Downloading Data Files

Oracle Cloud Infrastructure Object Storage service is used to upload and download data files for batch processing.

A public API has been provided to perform various actions on Object Storage.

Terminology:

PAR: PAR is a Pre-Authenticated Request that can be used to upload or download files. This is valid only for a limited duration and is set to five minutes.

Storage Prefix: The Object Storage is a flat storage area and does not include the concept of directories. Storage Prefix is a directory structure such as prefix (for example, planning/incoming) that is the prefixed to a filename to give it a more readable and distinguishable name.

Example 7-1

Action: Get PAR (Pre-Authenticated Requests) for upload of files
Endpoint: /uploadFiles
Payload: [  {"storagePrefix": "string", "fileName": "string"}, ...]
Response: PAR URI

This endpoint is used to generate PARs which are then used to upload files. 

Data:

{
"listOfFiles": [
{
"storagePrefix": "planning/<subnamespace>/incoming/input",
"fileName": "prod.csv.dat"
},
{
"storagePrefix": "planning/<subnamespace>/incoming/config",
"fileName": "RetailHomeConfig.json"
}
]
}

curl -X POST "<url>/<BucketName>/uploadFiles" -H "accept: application/json" -H "Accept-Language: en" -H "Authorization: Bearer <AccessToken>" -H "Content-Type: application/json" -d "{\"listOfFiles\":[{\"storagePrefix\":\"planning/<subnamespace>/incoming/input\",\"fileName\":\"prod.csv.dat\"},{\"storagePrefix\":\"planning/<subnamespace>/incoming/config/\",\"fileName\":\"RetailHomeConfig.json\"}]}"


Response 200:

{
"parList": [
{
"id": "WhmJ1Jn6v1GufaHDVISg2sUyiOLEoDAG3rWGUp1J/yj7NgvpJtf77q6Qehsz9h9R:planning/<subnamespace>/incoming/input/prod.csv.dat",
"name": "prod.csv.dat",
"accessUri": "https://objectstorage.us-phoenix-1.oraclecloud.com/p/oKDbsdO0T3LsfEWgys5WMD85DOyB7W4AvNK_lYXGtVMkGobebU6NIdvuz5AqrtO4/n/oraclegbudevcorp/b/<BucketName>/o/planning/<subnamespace>/incoming/input/prod.csv.dat",
"objectName": "planning/<subnamespace>/incoming/input/prod.csv.dat",
"accessType": "ObjectWrite",
"timeExpires": 1629914865691,
"timeCreated": 1629914565988
},
{
"id": "I0zDNVNqpQG8KEgGoJokViRIZ6yAQNX7haWIvBfDME+UabAShOhVu7zNvmERkZLm:planning/<subnamespace>/incoming/config/RetailHomeConfig.json",
"name": "RetailHomeConfig.json",
"accessUri":
 "https://objectstorage.us-phoenix-1.oraclecloud.com/p/Oja49u66xQqkzgOC6i5GA1PooED1XQV2bsnDwurqKbwetAnqX1RzVb4-e1mPiWWs/n/oraclegbudevcorp/b/<BucketName>/o/planning/<subnamespace>/incoming/config/RetailHomeConfig.json",
"objectName": "planning/<subnamespace>/incoming/config/RetailHomeConfig.json",
"accessType": "ObjectWrite",
"timeExpires": 1629914866213,
"timeCreated": 1629914566301
}
]
}

Files can then be uploaded using the generated PARs (accessURI in the response).

curl --request PUT --data-binary <File> <PAR>

Example 7-2

Action: Get PAR for download of files
Endpoint: /downloadFiles
Payload: [  {"storagePrefix": "string", "fileName": "string"}, ...]
Response: PAR URI

This endpoint is used to generate PARs which are then used to download files. 

Data:

{
"listOfFiles": [
{
"storagePrefix": "planning/<subnamespace>/incoming/input",
"fileName": "prod.csv.dat"
},
{
"storagePrefix": "planning/<subnamespace>/incoming/config/",
"fileName": "RetailHomeConfig.json"
}
]
}

curl -X POST "<url>/<BucketName>/download" -H "accept: application/json" -H "Accept-Language: en" -H "Authorization: Bearer <AccessToken>" -H "Content-Type: application/json" -d "{\"listOfFiles\":[{\"storagePrefix\":\"planning/<subnamespace>/incoming/input\",\"fileName\":\"prod.csv.dat\"},{\"storagePrefix\":\"planning/<subnamespace>/incoming/config/\",\"fileName\":\"RetailHomeConfig.json\"}]}"


Response 200:

{
"parList": [
{
"id": "SMINYf9O09IQmfkIxb8gHb0oRN2VrgAD88knbvo3pcNvvrNEhG8btWlPOdnfcLRG:planning/<subnamespace>/incoming/input/prod.csv.dat",
"name": "prod.csv.dat",
"accessUri": "https://objectstorage.us-phoenix-1.oraclecloud.com/p/M7JSr6Jp4up_XnVgib6xXqCuc17YHoWZDVzwGmt1x76_sqIRTQGHS3X_5aMar2HV/n/oraclegbudevcorp/b/<BucketName>/o/planning/<subnamespace>/incoming/input/prod.csv.dat",
"objectName": "planning/<subnamespace>/incoming/input/prod.csv.dat",
"accessType": "ObjectRead",
"timeExpires": 1629916720360,
"timeCreated": 1629916420483
},
{
"id": "8Jtqp6T2g3k+1KY9RhO9D7Kftmm5OBn6PIhCzxIQAH9jbvyQmi54SNoXuEzx0eqM:planning/<subnamespace>/incoming/config/RetailHomeConfig.json",
"name": "RetailHomeConfig.json",
"accessUri": "https://objectstorage.us-phoenix-1.oraclecloud.com/p/5wvRsj2D00nRkjX7-IMZoo9xOSRkMJUM37yyy4wKL1PKwTpdgQ8BWQKlg_jHXnrh/n/oraclegbudevcorp/b/<BucketName>/o/planning/<subnamespace>/incoming/config/RetailHomeConfig.json",
"objectName": "planning/<subnamespace>/incoming/config/RetailHomeConfig.json",
"accessType": "ObjectRead",
"timeExpires": 1629916720632,
"timeCreated": 1629916420742
}
]
}

Files can then be downloaded using the generated PARs (accessURI in the response).

curl --request GET <PAR> --output <file>

Example 7-3

Action: Delete Files
Endpoint: /deleteFiles
Payload: [  {"storagePrefix": "string", "fileName": "string"}, ...]
Response: Status: ok/error per file

This endpoint is used to delete files from Object Storage.

Data:

{
"listOfFiles": [
{
"storagePrefix": "planning/<subnamespace>/incoming/input",
"fileName": "prod.csv.dat"
},
{
"storagePrefix": "planning/<subnamespace>/incoming/config/",
"fileName": "RetailHomeConfig.json"
}
]
}

curl -X DELETE "<url>/<BucketName>/delete" -H "accept: */*" -H "Accept-Language: en" -H "Authorization: Bearer <AccessToken>" -H "Content-Type: application/json" -d "{\"listOfFiles\":[{\"storagePrefix\":\"planning/<subnamespace>/incoming/input\",\"fileName\":\"prod.csv.dat\"},{\"storagePrefix\":\"planning/<subnamespace>/incoming/config/\",\"fileName\":\"RetailHomeConfig.json\"}]}"


Response 200:
{
"filesDeleted": [
{
"filePath": {
"storagePrefix": "planning/<subnamespace>/incoming/input",
"fileName": "prod.csv.dat"
},
"responseMessage": "File successfully deleted planning/<subnamespace>/incoming/input/prod.csv.dat"
},
{
"filePath": {
"storagePrefix": "planning/<subnamespace>/incoming/config/",
"fileName": "RetailHomeConfig.json"
},
"responseMessage": "File successfully deleted planning/<subnamespace>/incoming/config/RetailHomeConfig.json"
}
],
"filesFailedDeletion": []
}

Example 7-4

Action: Move/Rename Files
Endpoint: /moveFiles
Payload: [

  {    currentPath:{"storagePrefix": "string", "fileName": "string"},    newPath: {"storagePrefix": "string", "fileName": "string"}  }, ...

]
Response: Status: ok/error per file

This endpoint is used to move files within Object Storage.

Data:

{
"listOfFiles":[
{
"currentPath":{
"storagePrefix":"planning/<subnamespace>/incoming/input2",
"fileName":"prod.csv.dat"
},
"newPath":{
"storagePrefix":"planning/<subnamespace>/incoming/config1/",
"fileName":"prod.csv.dat"
}
}
]
}

curl -X POST "<url>/<BucketName>/movefiles" -H "accept: */*" -H "Accept-Language: en" -H "Authorization: Bearer <AccessToken>" -H "Content-Type: application/json" -d "{\"listOfFiles\":[{\"currentPath\":{\"storagePrefix\":\"planning/<subnamespace>/incoming/input2\",\"fileName\":\"prod.csv.dat\"},\"newPath\":{\"storagePrefix\":\"planning/<subnamespace>/incoming/config1/\",\"fileName\":\"prod.csv.dat\"}}]}"


Response:

{
"failedMove": [],
"successfulMove": [
{
"moveFile": {
"currentPath": {
"storagePrefix": "planning/<subnamespace>/incoming/input2",
"fileName": "prod.csv.dat"
},
"newPath": {
"storagePrefix": "planning/<subnamespace>/incoming/config1/",
"fileName": "prod.csv.dat"
}
},
"responseMessage": "Successfully moved."
}
]
}

Example 7-5

Action: List storage prefixes
Endpoint: /listStoragePrefixes
Payload: {"storagePrefix": "string", "fileNameStartsWith": "optional_string"}
Response: Storage prefix name, Size, Number of files

This endpoint is used to list Storage Prefixes in Object Storage.

Allows customer to discover consumption and storage prefix names

curl -X GET "<url>/<BucketName>/listprefixes" -H "accept: application/json" -H "Accept-Language: en" -H "Authorization: Bearer <AccessToken>"


Response: 200
Response body
[
"planning/<subnamespace>/incoming/config/",
"planning/<subnamespace>/incoming/input"
]

Example 7-6

Action: Register storage prefix
Endpoint: {bucketname}/register/{prefix}
Payload:
Response: 

This endpoint is used to register Storage Prefixes.

curl -X PUT "<url>/<BucketName>/register/planning/<subnamespace>/incoming/input" -H "accept: */*" -H "Accept-Language: en" -H "Authorization: Bearer <AccessToken>"


Response: 200

Example 7-7

Action: Unregister storage prefix
Endpoint: {bucketname}/unregister/{prefix}
Payload:
Response:

This endpoint is used to unregister Storage Prefixes.

curl -X DELETE "<url>/<BucketName>/unregister/planning/<subnamespace>/incoming/config1/" -H "accept: */*" -H "Accept-Language: en" -H "Authorization: Bearer <AccessToken>"


Response: 200

Example 7-8

Action: List files within a storage prefix
Endpoint: /listFiles
Payload: {"storagePrefix": "optional_string"}
Response: storage prefix name, file name, size, md5 checksum, created date, modified date, scan date, scan status

This endpoint is used to find and list files from Object Storage,

curl -X GET "<url>/<BucketName>/listfiles?prefix=planning/<subnamespace>/incoming&contains=prod&sort=size:asc" -H "accept: application/json" -H "Accept-Language: en" -H "Authorization: Bearer <AccessToken>"


Response:

{
"resultSet": [
{
"name": "planning/<subnamespace>/incoming/config1/prod.csv.dat",
"size": 177023,
"md5": "vlN/r8gbNyQ1EdsC++Hv1w==",
"version": "ca60582d-bacb-4309-bbbf-89d4a1aa2843",
"etag": "7d6dde3b-5b5b-430d-8494-78e090418946",
"createdDate": "2021-08-25T20:36:40Z",
"modifiedDate": "2021-08-25T20:36:40Z",
"scanStatus": "Passed"
}
],
"totalResults": 1,
"limit": 0,
"count": 1,
"offset": 0,
"hasMore": false
}

Example 7-9

Action: Find file
Endpoint: /findFileByName
Payload: {"stringContains": "string"}
Response: 
storage prefix name, file name, size, md5 checksum, created date, modified date, scan date, scan status

This endpoint is used to find and list files from Object Storage,

curl -X GET "<url>/<BucketName>/findFileByName?prefix=planning/<subnamespace>/incoming&contains=prod&sort=size:asc" -H "accept: application/json" -H "Accept-Language: en" -H "Authorization: Bearer <AccessToken>"

Response:


{
"resultSet": [
{
"name": "planning/<subnamespace>/incoming/config1/prod.csv.dat",
"size": 177023,
"md5": "vlN/r8gbNyQ1EdsC++Hv1w==",
"version": "ca60582d-bacb-4309-bbbf-89d4a1aa2843",
"etag": "7d6dde3b-5b5b-430d-8494-78e090418946",
"createdDate": "2021-08-25T20:36:40Z",
"modifiedDate": "2021-08-25T20:36:40Z",
"scanStatus": "Passed"
}
],
"totalResults": 1,
"limit": 0,
"count": 1,
"offset": 0,
"hasMore": false
}

Example 7-10

Action: Liveness test
Endpoint: /ping
Payload:
Response:

This endpoint allows customer to check for service liveness

curl -X GET "<url>/ping" -H "accept: */*" -H "Accept-Language: en" -H "Authorization: Bearer <AccessToken>"


Response:

{
"appStatus": 200
}