| Oracle® Retail Bulk Data Integration Cloud Service Implementation Guide Release 19.1.000 F31810-01 |
|
 Previous |
 Next |
| Oracle® Retail Bulk Data Integration Cloud Service Implementation Guide Release 19.1.000 F31810-01 |
|
Previous |
Next |
The BDI suite provides two CLI (Command-Line Interface) tools as part of this release.
BDI CLI Job Executor BDI
CLI Batch Transmitter
The following sections describe in detail the above CLI components, their setup and usage.
The BDI CLI Job Executor is a standalone command line utility that helps in basic operation of BDI batch jobs through commands. It uses the REST services that the BDI Batch Job Admin provides to list jobs and executions, get status of a job, and start, stop and restart a batch job.
To prepare the tool for use, follow these steps.
The bdi-cli-job-executor home directory (where the tool software package is extracted) contains a 'conf' directory where the tool related configuration file will be present, and 'bin' directory where the executable to run the tool will be present.
Configure BDI Batch Job Admin URL and alias name for the credentials to access Job Admin URL.
Edit conf/bdi-job-admin-info.json file to add the BDI Batch Job Admin URL value for the jobAdminUrl property.
Example: "jobAdminUrl":"http://<hostname>:<port>/bdi-rms-batch-job-admin/"
Add alias name in the property jobAdminUserAlias.
Example:
"jobAdminUserAlias":"rmsJobAdminBaseUrlUserAlias"
Run: bdi-cli-job-executor.sh -setup-credentials
|
Note: bdi-cli-job-executor.sh will be in the 'bin' directory. |
This prompts for the credentials for the given alias. Enter the corresponding username and password to be used to access the Job Admin URL. The credentials will be stored in the wallet and used to invoke the BDI Job Admin REST services.
The BDI CLI Job Executor tool is run using the shell script: bdi-cli-job-executor.sh from the 'bin' directory.
Usage: bdi-cli-job-executor.sh -[option]
| Option | Description |
|---|---|
| list | Lists all available job names and details.
bdi-cli-job-executor.sh -list |
| list runningJobs | Lists all currently running jobs and job execution IDs.
bdi-cli-job-executor.sh -list runningJobs |
| start <jobname> | Starts a job of given name.
Example: bdi-cli-job-executor.sh -start MyBatchJob |
| restart <jobname> <executionId> | Restarts a failed job execution with the corresponding execution Id.
Example: bdi-cli-job-executor.sh -restart MyBatchJob 12345 |
| stop | Stops all the running job executions.
bdi-cli-job-executor.sh -stop |
| stop <executionId> | Stops the currently running job execution of given execution Id.
Example: bdi-cli-job-executor.sh -stop 12345 |
| status <jobname> | Gets the status of the job of given job name.
Example: bdi-cli-job-executor.sh -status MyBatchJob |
| status <jobname> <instanceId> | Gets the status of the job of given job name and job instance Id.
Example: bdi-cli-job-executor.sh -status MyBatchJob 54321 |
The BDI CLI Transmitter is a standalone command line tool to transmit batch interface data files to a destination BDI receiver system. It is particularly used where the source system is non-BDI (that is, the source system does not have or use BDI Batch Job Admin application) but needs to send interface data files to a receiver system running the BDI Job Admin application.
The tool uses the BDI Job Admin Receiver REST service URL to transmit the data to the destination system. So it is necessary that the destination system runs the BDI Job Admin application to use the tool.
To prepare the tool for use, follow these steps.
The bdi-cli-transmitter home directory (where the tool software package is extracted) contains 'conf' directory where the tool related configuration files will be present, and 'bin' directory where the executable to run the tool will be present.
Configure conf/bdi-file-transmitter.properties. The following describes the properties to be configured. The properties file provides some sample values for reference to start with. The values specified in the properties file can be overridden using the command-line input options if required, when running the tool for file transmission.
| Property | Description |
|---|---|
| source.system.name | The name of the source system or application that provides the source data to be transmitted.
For example, source.system.name=RMS |
| <receiverAppName>.receiver.url | The Receiver REST service URL of the BDI Receiver application indicated by <receiverAppName> (should be in lowercase).
For example, if the BDI receiver application is RPAS, then specify the property and value as: rpas.receiver.url=http://<bdi-rpas-app-hostname>:<port>/bdi-rpas-batch-job-admin/resources/receiver |
| <receiverAppName>.receiver.url.useralias | Alias name for the credentials to be used to connect to the corresponding receiver service. The alias name with the credentials are stored in a wallet. <receiverAppName> should be in lowercase.
Example: rpas.receiver.url.useralias=rpasReceiverUrlUserAlias |
| <InterfaceModuleName>.receiver.appname | Name of the BDI receiver application for the interface module <InterfaceModuleName>.
Specify the name in lowercase. Example: Diff_Fnd.receiver.appname=rpas Store_Fnd.receiver.appname=sim |
| <InterfaceModuleName>.dataset.type | The data set type of the data to be transmitted for the interface module identified by <InterfaceModuleName>.
Valid value is FULL or PARTIAL. Example: Diff_Fnd.dataset.type=FULL |
| <InterfaceModuleName>.interfaceShortNames | The interface name(s) for the corresponding interface module <InterfaceModuleName>. Multiple interface names can be specified (each separated by a comma) as multiple interfaces can be part of an interface module. The interface module name and interface names should be the same as expected by the BDI receiver application where the files are transmitted.
Example: Diff_Fnd.interfaceShortNames=Diff DiffGrp_Fnd.interfaceShortNames=Diff_Grp,Diff_Grp_Dtl |
| <InterfaceModuleName>.<InterfaceShortName>.input.filepath | Specify the file location where the corresponding interface data files to be transmitted are present. Each interface in a interface module should have separate file locations.
Example: Diff_Fnd.Diff.input.filepath=/home/bdi/diff_fnd/diff/files DiffGrp_Fnd.Diff_Grp.input.filepath=/home/bdi/diffgrp_fnd/diff_grp/files DiffGrp_Fnd.Diff_Grp_Dtl.input.filepath=/home/bdi/diffgrp_fnd/diff_grp_dtl/files |
Run: bdi-file-transmitter.sh -setup-credentials.
|
Note: bdi-file-transmitter.sh will be in the 'bin' directory. |
Run -setup-credentials to configure the BDI Receiver service user credentials. Running this command will prompt for the username and password for each of the <receiverAppName>.receiver.url.useralias specified in bdi-file-transmitter.properties file.
The credentials entered for each alias will be stored in a secure wallet and used to connect to the corresponding BDI Receiver service for transmission of files.
This is a prerequisite step to use the tool but usually a one-time setup before running bdi-file-transmitter.sh for transmission of files.
Run: bdi-file-transmitter.sh -get-interface-metadata <receiver-app-name>
|
Note: bdi-file-transmitter.sh will be in the 'bin' directory. |
For example: bdi-file-transmitter.sh -get-interface-metadata sim
Run bdi-file-transmitter.sh -get-interface-metadata to configure metadata for a receiver app. On running -get-interface-metadata option the metadata for the receiver app will be saved in the bdi-cli-transmitter/conf/meta-data/<receiver-app-name>.
This is a prerequisite step to use the tool usually one-time setup before running bdi-file-transmitter.sh for transmission of files to a receiver app.
Optionally, configure conf/bdi-file-transmitter-runtime.properties that contains parameters (described below) for performance tuning of the tool.
Start with default values as present in the properties file, analyze the performance and choose optimal values for the parameters for better performance if required. The tool will use default values for the parameters (mentioned below) when no values are specified in the properties file.
| Property | Description |
|---|---|
| multiple_files_process_limit | The maximum number of files to process in parallel at any given time. Default value is 5. |
| file_transmission_thread_limit | The number of parallel threads to run to process a single file. Default value is 3. |
| transmission_record_size | The maximum number of records per block or chunk to transmit to the receiver service per service call. Default value is 20000. |
| transmission_timeout | The timeout in minutes for file transmission. The process will timeout and end when the file transmission is still not complete after the specified time. Default value is 300 minutes. |
The BDI CLI Transmitter tool is run using the shell script: bdi-file-transmitter.sh from the 'bin' directory.
The tool can be run in interactive and noninteractive modes.
Interactive Mode: Run bdi-file-transmitter.sh
For user interactive mode where the program prompts for input, just run bdi-file-transmitter.sh with no options.
This will prompt for each input with descriptions which will be self-explanatory. The user can enter value as required or skip optional parameters. When no value is specified for optional parameters, the tool will try to use the default values as specified in the bdi-file-transmitter.properties file or stop executing when no default value is present.
Non-Interactive Mode: Run bdi-file-transmitter.sh [input]
The tool can be run with the following inputs as described below.
|
Note: The only required input is interface module name, when the other input values are specified in bdi-file-transmitter.properties file. |
| Input | Description |
|---|---|
| -m or --interfacemodule <interfaceModuleName> | (Required) The interface module name. Should be the same as the interface module name expected by the BDI receiver application. |
| -i or --interfaceshortnames <interfaceShortNames> | (Optional) Multiple interface names should be separated by comma. If not specified, the program will use the interface names corresponding to the interface module as specified in bdi-file-transmitter.properties file.
The interface names should be the same as expected by the BDI receiver application |
| -s or --sourcesystem <sourceSystemName> | (Optional) The source system name. If not specified, the program will use the source.system.name given in the properties file. |
| -f or --filelocation <inputFilePaths> | (Optional) The location of interface data file(s) that are to be transmitted. This can be single file or a directory path with multiple data files of the interface. Multiple file paths should be separated by comma, for each interface in the corresponding order. If not specified, the program will use the input file paths given for the interfaces as given in the properties file. |
| -a or --receiverapp <receiverAppName> | (Optional) The BDI receiver app name. This is used to get the receiver url and/or useralias from properties file if any of those values are not provided. If not specified, the program will use the receiver app name specified for the interface in the properties file. |
| -r or --receiverurl <fileReceiverUrl> | (Optional) The receiver url. If not specified, the program will use the receiver url of the receiver app specified for the interface in the properties file, for transmission of files. |
| -u or --useralias <receiverUrlUserAlias> | (Optional) The alias name for the credentials to be used to connect to the receiver service url. The credentials corresponding to the alias should exist in the wallet. If not specified, the program will use the receiver url useralias of the receiver app specified for the interface in the properties file. |
| -d or --datasettype <dataSetType> | (Optional) The data set type that specifies the data transmitted is full or delta load. Valid value: 'FULL' or 'PARTIAL'. If not specified, the program will use the interface specific data set type as given in the properties file. |
Some examples of running the transmitter tool command-line:
bdi-file-transmitter.sh -m Diff_Fnd
bdi-file-transmitter.sh -m Diff_Fnd -i Diff
bdi-file-transmitter.sh -m DiffGrp_Fnd -i Diff_Grp,Diff_Grp_Dtl
bdi-file-transmitter.sh -m Diff_Fnd -a sim
bdi-file-transmitter.sh -m DiffGrp_Fnd -i Diff_Grp,Diff_Grp_Dtl -f /home/bdi/diffgrp_fnd/diff_grp/files,/home/bdi/diffgrp_fnd/diff_grp_dtl/files
bdi-file-transmitter.sh -m "Diff_Fnd" -i "Diff" -s "RMS" -d "FULL"
bdi-file-transmitter.sh -m Diff_Fnd -i Diff -s "RMS" -f "/home/bdi/diffgrp_fnd/diff/files" -a "sim" -r "https://bdisimapphost:9001/bdi-sim-batch-job-admin/resources/receiver" -d "FULL"
The BDI Transmitter tool supports transmission of flat files, for example, .csv files, in UTF-8 format. The BDI Receiver application supports only csv files. Hence the interface data files to be transmitted need to contain records with comma-separated field values.
The order of the fields in the file should be as expected by the BDI Receiver application, so that each value is inserted in the right columns of the destination interface tables. No header line should be present in the file (each line is treated as data record). Each record should be in a newline.
The interface module name and interface names for the files to be transmitted should be same as expected by the BDI Receiver application.
The transmitter tool can process a single file or a directory containing multiple files. But the tool does not process files recursively in subdirectories.
Files are processed and transmitted per interface module. Each run of processing of files of the interface module will be considered a transaction and a Transaction Id will be generated and associated to the transmission of files (at the interface module level). Files of multiple interfaces in an interface module will be part of the same transaction.
Each file transmission within a transaction will have a Transmission Id associated to it. The same transaction Id and transmission Id are sent to the BDI Receiver application, so the corresponding transmission details can be seen in the Job Admin console of the BDI Receiver application.
After successful transmission, the file will be moved to the archive directory:
<inputFileDirectory>/archive/<interfaceModuleName>/<transactionId>
For example,
if the input file location is '/home/bdi/interface/files' and the interface module of the files is 'Diff_Fnd', and the transaction Id of the file transmission is 'Tx#5263_1568696470665_RMS', then after successful transmission the file will be moved to the directory:
/home/bdi/interface/files/archive/Diff_Fnd/Tx#5263_1568696470665_RMS.
The transmitter tool outputs messages and logs to the terminal console where the command is run.
The tool also creates a log file that contains detailed logs about the processing of files. The log will show the Transaction Id and Transmission Id of each file transmission among other details.
The log file is created in the logs directory under the tool home directory (bdi-cli-transmitter/logs).
The name of the log file will be in the format: bdi-file-transmitter_yyyy-mm-dd_hh:mm:ss, for example bdi-file-transmitter_2016-07-04_10:38:59.
In case of any error in file processing, error in transmission of file to the receiver service, timeout of file transmission, or any other failure, the file will be moved to the 'failed' directory: <inputFileDirectory>/failed/<interfaceModuleName>/<transactionId>
For example, if the input file location is '/home/bdi/interface/files' and the interface module of the files is 'Diff_Fnd', and the transaction Id of the file transmission is 'Tx#5263_1568696470665_RMS', then if the transmission of file fails, the file will be moved to the directory: /home/bdi/interface/files/failed/Diff_Fnd/Tx#5263_1568696470665_RMS.
A properties file containing the input details corresponding to the failed file will be created. For example, if the file named 'Item_1.csv' has failed, then a file named 'Item_1.csv.properties' will be created in the 'failed' directory. This acts as the input context that will be used when the file is reprocessed. The user should not delete or modify this properties file, if the data file has to be re-processed with the original input context.
Due to parallel processing of files by the transmitter, there may be a scenario where some records in the file may have been transmitted successfully, but part of the file transmission may have failed. Even in this case, the entire file will be treated as failed and moved to the 'failed' directory.
Reprocessing will be at the file level and not at the block level where the transmission may have failed. In the case of partial transmission of file, the BDI Receiver application also marks the whole transmission as failed and hence the entire file can be retransmitted to be processed again by the receiver application.
To retry failed files (that did not get transmitted successfully in previous transmission) use the below command:
bdi-file-transmitter.sh -retry-failed <inputFileDir or inputFilePath>
For example, bdi-file-transmitter.sh -retry-failed /home/bdi/interface/files/failed/Diff_Fnd/Tx#5263_1568696470665_RMS
bdi-file-transmitter.sh -retry-failed /home/bdi/interface/files/failed/Diff_Fnd/Tx#5263_1568696470665_RMS/Diff_1.csv
Once a file is successfully reprocessed, it will be renamed as <filename>-retransmitted. For example, Diff_1.csv-retransmitted. And, the corresponding properties file will be deleted.
