Go to primary content
Oracle® Retail Bulk Data Integration Implementation Guide
Release 16.0
E81413-01
  Go To Table Of Contents
Contents

Previous
Previous
 
Next
Next
 

3 Job Admin Services

This chapter discusses the Job Admin Services.

Job Admin RESTful Services

Job Admin provides below RESTful services. These services are secured with SSL and basic authentication.

  • Batch Service - Ability to start/stop/restart, check status, and so on of jobs. This service is typically used by the BDI Process Flow engine.

  • Receiver Service - Ability to stream data from one system to another system. This service is used by the Downloader-Transporter job.

  • System Setting Service - Ability to view, change system settings, and credentials. Refer to Appendix D for details on System Setting REST resources.

  • Data Service - Ability to get data set information using job information such as job name, execution id or instance id.

Receiver Service

The Receiver Service is a RESTful service that provides various endpoints to send data transactionally. Receiver Service is part of Job Admin. It stores data as files and keeps track of metadata in the database. The Receiver Service also supports various merge strategies for merging files at the end. The Receiver Service is used by the Downloader-Transporter job to transmit data from source to destination.Seed data for Receiver Service is generated during design time and loaded during deployment of the Job Admin application.

BDI_RECEIVER_OPTIONS

The Receiver Service options can be configured at interface level.

Table 3-1 Receiver Options

Column Type Comments
ID NUMBER Primary key
INTERFACE_MODULE VARCHAR2(255) Name of the interface module
INTERFACE_SHORT_NAME VARCHAR2(255) Name of the interface
BASE_FOLDER VARCHAR2(255) This is the base folder for storing files created by Receiver Service. Receiver Service creates a subfolder ”bdi-data” under base folder. Base folder can be changed from ”Manage Configurations” tab of Job Admin GUI.
FOLDER_TEMPLATE VARCHAR2(255) Folder template provides the folder structure for storing files created by Receiver Service.Default value is ”${basefolder}/${TxId}/${TId}/${BId}/”. TxId - Transaction Id TId - Transmission Id BId - Block Id This value can't be changed.
MERGE_STRATEGY VARCHAR2(255) The strategy for merging files. The default value is ”NO_MERGE”. The valid values are NO_MERGE, MERGE_TO_PARTITION_LEVEL, and MERGE_TO_INTERFACE_LEVEL. MERGE_TO_PARTITION_LEVEL Merges all files for that partition and creates the merged file in ”${TId}” folder. MERGE_TO_INTERFACE_LEVEL Merges all files for interface and creates the merged file in ”${TxId}” folder. MERGE strategies are only supported in cases where the Uploader is not used.

Endpoints

Ping

This endpoint can be used to check whether the Receiver Service is up or not.

HTTP Method: GET

Path: receiver/ping

Response: alive

Begin Receiver Transaction

This is the first endpoint to be called by the client (for example Downloader-Transporter job) before sending any data. It stores the following metadata in the BDI_RECEIVER_TRANSACTION table and returns the response in JSON format.

Parameter Description
Transaction Id Transaction Id (Tx#<Job Instance Id> of the Downloader-Transporter job
Interface Module Name of the interface module (for example Diff_Fnd)
Source System Name Name of the source system (for example RMS)
sourceSystemId ID of the source system (for example URL)
sourceDataSetId ID of the data set
Data Set Type Type of data set (FULL or PARTIAL)
Source System Data Set Ready Time Time when source data set was ready in the outbound tables

HTTP Method: POST

Path: receiver/beginTransaction/{transactionId}/{interfaceModule}/{sourceSystemName}/{sourceSystemId}/{sourceDataSetId}/{dataSetType}/{sourceSystemDataSetReadyTime}

Sample Response
 
{
        ”receiverTransactionId”: ”1”,
        ”transactionId”: ”Tx#1”,
        ”sourceSystemName”: ”RMS”,
        ”interfaceModule”: ”Diff_Fnd”,
        ”sourceSystemId”: ””,
        ”sourceDataSetId”: ””,
        ”dataSetType”: ”FULL”,
        ”sourceSystemDataSetReadyTime”: ””,
        ”dir”: ””,
        ”fileName”: ””,
        ”receiverTransactionStatus”: ””,
        ”receiverTransactionBeginTime”: ””,
        ”receiverTransactionEndTime”: ””    
}

Begin Receiver Transmission

This end point needs to be called by client (for example Downloader-Transporter job) before sending any data for a partition. It stores the following metadata in BDI_RECEIVER_TRANSMISSION table and returns response in JSON format.

Parameter Description
TransmissionId Generated for each partition
InterfaceModule) Name of the interface module (for example Diff_Fnd
InterfaceShortName) Name of the interface (for example Diff
partitionName Partition number
partitionBeginSeqNum Begin sequence number in the partition
partitionEndSeqNum End sequence number in the partition
beginBlockNumber Begin block number

HTTP Method: POST

Path: receiver/beginTransmission/{transactionId}/{transmissionId}/{sourceSystemName}/{interfaceMod-ule}/{interfaceShortName}/{partitionName}/{partitionBeginSeqNum}/{partitionEndSeqNum}/{beginBlockNumber}

Sample Response:
 
{
        ”transmissionId”: ”1”,
        ”interfaceModule”: ”Diff_Fnd”,
        ”interfaceShortName”: ”Diff”,
        ”sourceSystemPartitionName”: ”1”,
        ”sourceSystemPartitionBeginSequenceNumber”: ”1”,
        ”sourceSystemPartitionEndSequenceNumber”: ”100”,
        ”beginBlockNumber”: ”1”,
        ”endBlockNumber”: ””,
        ”dir”: ””,
        ”fileName”: ””,
        ”receiverTransmissionStatus”: ””
}

Update Data Block

Clients use this endpoint to send data. This endpoint is typically called by the client multiple times until there is no more data. It creates a csv file with the data it received at the below location.

${BASE_FOLDER}/bdi-data/${TxId}/${TId}/${BId}

BASE_FOLDER - Obtained from the BDI_RECEIVER_OPTIONS table

TxId - Transaction Id of the remote Downloader-Transporter job

TId - Transmission Id associated with the transaction id

BId - Block Id associated with transmission id

It also stores the following metadata in the RECEIVER_TRANSMISSION_BLOCK table.

Parameter Description
BlockNumber Number of the block
ItemCountInBlock Number of items in the block
Dir Directory where file is created
FileName Name of the file
ReceiverBlockStatus Status of the block
CreateTime Time when the block is created

HTTP Method: POST

Path: receiver/uploadDataBlock/{transactionId}/{transmissionId}/{sourceSystemName}/{interfaceModule}/{interfaceShortName}/{blockNumber}/{itemCountInBlock}

Sample Response
 
{
        ”blockId”: ”1”,
        ”transmissionId”: ”1”,
        ”blockNumber”: ”1”,
        ”blockItemCount”: ”100”,
        ”dir”: ””,
        ”fileName”: ””,
        ”receiverBlockStatus”: ””,
        ”createTime”: ””
}

End Transmission

This end point ends transmission for a partition. It updates ”endBlockNumber” and ”receiverTransmisionStatus” in the RECEIVER_TRANSMISSION table.

HTTP Method: POST

Path: receiver/endTransmission/{transmissionId}/{sourceSystemName}/{interfaceModule}/{interfaceShortName}/{numBlocks}

Sample Response
 
{
        ”transmissionId”: ”1”,
        ”interfaceModule”: ”Diff_Fnd”,
        ”interfaceShortName”: ”Diff”,
        ”sourceSystemPartitionName”: ”1”,
        ”sourceSystemPartitionBeginSequenceNumber”: ”1”,
        ”sourceSystemPartitionEndSequenceNumber”: ”100”,
        ”beginBlockNumber”: ”1”,
        ”endBlockNumber”: ””,
        ”dir”: ””,
        ”fileName”: ””,
        ”receiverTransmissionStatus”: ””
}

End Transaction

This end point ends the transaction and called once by the client. It updates ”receiverTransactionStatus” and ”receiverTranasctionEndTime” in the RECEIVER_TRANSACTION table. If ”mergeStrategy” is set to ”MERGE_TO_PARTITION_LEVEL” or ”MERGE_TO_INTERFACE_LEVEL”, then it merges the files and creates the merged file(s) at the appropriate directory. It creates an entry in the BDI_UPLDER_IFACE_MOD_DATA_CTL table so that Uploader job can pick up the data.

HTTP Method: POST

Path: receiver/endTransaction/{transactionId}/{sourceSystemName}/{interfaceModule}

Sample Response
 
{
        ”receiverTransactionId”: ”1”,
        ”transactionId”: ”Tx#1”,
        ”sourceSystemName”: ”RMS”,
        ”interfaceModule”: ”Diff_Fnd”,
        ”sourceSystemId”: ””,
        ”dataSetType”: ”FULL”,
        ”sourceSystemDataSetReadyTime”: ””,
        ”dir”: ””,
        ”fileName”: ””,
        ”receiverTransactionStatus”: ””,
        ”receiverTransactionBeginTime”: ””,
        ”receiverTransactionEndTime”: ””    
}

Uploader Interface Module Data Control Table

The BDI_UPLDER_IFACE_MOD_DATA_CTL table acts as a handshake between the downloader and uploader jobs. When the downloader-transporter job calls endTransaction on Receiver Service, the Receiver Service creates an entry in this table if it successfully received data and created files.

An entry in this table indicates to the uploader job that a data set is ready to be uploaded.

BDI_UPLDER_IFACE_MOD_DATA_CTL

Table 3-2 Module Data Control

Column Type Comments
UPLOADER_IFACE_MOD_DATA_CTLID NUMBER Primary key
INTERFACE_MODULE VARCHAR2(255) Name of the interface module
REMOTE_TRANSACTION_ID VARCHAR2(255) Transaction Id of Downloader-Transporter job
SOURCE_DATA_SET_ID NUMBER NUMBER ID of the source data set
SRC_SYS_DATA_SET_READY_TIME TIMESTAMP Source Data Set Ready Time
DATA_SET_TYPE VARCHAR2(255) Type of data set (FULL or PARTIAL)
DATA_SET_READY_TIME TIMESTAMP Time when data set was available in the outbound tables
DATA_FILE_MERGE_LEVEL VARCHAR2(255) Merge level for the files (NO_MERGE, MERGE_TO_PARTITION_LEVEL, MERGE_TO_INTERFACE_LEVEL)
SOURCE_SYSTEM_NAME VARCHAR2(255) Name of the source system (for example RMS)

Receiver Side Split for Multiple Destinations

If there are multiple destinations that receive data from a source, one of the options is to use the Receiver Service at one destination to receive data from the sender and then multiple destinations use the data from one Receiver Service to upload to inbound tables. The requirements for the Receiver Side Split are such that:

  • The Receiver Service database schema is shared by all the destinations

  • The File system is shared by all destinations

The performance of BDI can be improved by using the receiver side split if there are multiple destinations.

Batch Service

Batch service is a RESTful service that provides various endpoints to manage batch jobs in the bulk data integration system. Batch Service is part of Job Admin.

Table 3-3 Batch Service

REST Resource HTTP Method Description
/batch/jobs GET Gets all available batch jobs
/batch/jobs/{jobName} GET Gets all instances for a job
/batch/jobs/{jobName}/executions GET Gets all executions for a job
/batch/jobs/executions GET Gets all executions
/batch/jobs/currently-running-jobs GET Gets currently running jobs
/batch/jobs/{jobName}/{jobInstanceId}/executions GET Gets job executions for a job instance
/batch/jobs/{jobName}/{jobExecutionId} GET Gets job instance and execution for a job execution id
/batch/jobs/{jobName} POST Starts a job asynchronously
/batch/jobs/executions/{jobExecutionId} POST Restarts a stopped or failed job
/batch/jobs/executions DELETE Stops all running job executions
/batch/jobs/executions/{jobExecutionId} DELETE Stops a job execution
/batch/jobs/executions/{jobExecutionId} GET Gets execution steps with details
/batch/jobs/executions/{jobExecutionId}/steps GET Gets execution steps
/batch/jobs/executions/{jobExecutionId}/steps/{stepExecutionId} GET Gets step details
/batch/jobs/job-def-xml-files GET Gets all job xml files

Key End Points

Start Job

This end point starts a job asynchronously based on a job name and returns the execution id of the job in the response.

Path: /batch/jobs/{jobName}

HTTP Method: POST

Inputs

Job Name as path parameter

Job Parameters as a query parameter. Job Parameters is a comma separated list of name value pairs. This parameter is optional.

Sample Request

http://localhost:7001/bdi-batch-job-admin/resources/batch/jobs/DiffGrp_Fnd_ImporterJob?jobParameters=dataSetId=1

Successful Response

XML
 
<executionIdVo targetNamespace=””>
<executionId>1</executionId>
        <jobName>DiffGrp_Fnd_ImporterJob</jobName>
</executionIdVo>
 
JSON
 
{
”executionId”: 1,
        ”jobName”: ”DiffGrp_Fnd_ImporterJob”
}

Error Response

XML
<exceptionVo targetNamespace=””>
        <statusCode>404</statusCode>
        <status>NOT_FOUND</status>
        <message>HTTP 404 Not Found</message>
        <stackTrace></stackTrace>  <!-- optional -->
</exceptionVo>
 
JSON
{
        ”statusCode”: ”404”,
        ”status”: ”NOT_FOUND”,
        ”message”: ”HTTP 404 Not Found”,
        ”stackTrace”: ””
}

Restart Job

This end point restarts a job asynchronously using the job execution id and returns the new job execution id.

Path: /batch/jobs/executions/{executionId}

HTTP Method: POST

Inputs

executionId as path parameter

Sample Request

http://localhost:7001/bdi-batch-job-admin/resources/batch/jobs/executions/2

Successful Response

XML
 
<executionIdVo targetNamespace=””>
<executionId>2</executionId>
<jobName>DiffGrp_Fnd_ImporterJob</jobName>
</executionIdVo>
 
JSON
 
{
        ”executionId”: 2,
        ”jobName”: ”DiffGrp_Fnd_ImporterJob”
}

Error Response

XML
XML
 
<exceptionVo targetNamespace=””>
<statusCode>500</statusCode>
<status>INTERNAL_SERVER_ERROR</status>
        <message>Internal Server Error</message>
<stackTrace></stackTrace>  <!-- optional -->
</exceptionVo>
 
JSON
 
{
        ”statusCode”: ”500”,
        ”Status”: ”INTERNAL_SERVER_ERROR”,
        ”Message”: ”Internal Server Error”,
        ”stackTrace”: ””
}

Check Status of a Job

This endpoint returns the status of a job using the job name and execution id.

Path: /batch/jobs/jobName/{jobExecutionId}

HTTP Method: GET

Inputs

jobName as path parameter

jobExecutionId as path parameter

Sample Request

http://localhost:7001/bdi-batch-job-admin/resources/batch/jobs/DiffGrp_Fnd_ImporterJob/1

Successful Response

XML
<jobInstanceExecutionsVo targetNamespace=””>
    <jobName>DiffGrp_Fnd_ImporterJob</jobName>
    <jobInstanceId>1</jobInstanceId>
<resource>http://localhost:7001/bdi-batch-job-admin/resources/batch/jobs/DiffGrp_Fnd_ImporterJob/1</resource>
    <jobInstanceExecutionVo>
        <executionId>1<>executionId>
        <executionStatus>COMPLETED</executionStatus>
        <executionStartTime>2016-07-11 15:45:27.356</executionStartTime>
        <executionDuration>10</executionDuration>
    </jobInstanceExecutionVo>
</jobInstanceExecutionsVo>

JSON
{
    ”jobName”: ”DiffGrp_Fnd_ImporterJob”,
    ”jobInstanceId”: 1,
    ”resource”: ”http://localhost:7001/bdi-batch-job-admin/resources/batch/jobs/DiffGrp_Fnd_ImporterJob/1”,
    [”jobInstanceExecutionVo”: {
            ”executionId”: 1,
            ”executionStatus”: ”COMPLETED”,
            ”executionStartTime”:”2016-07-11 15:45:27.356”,
            ”executionDuration”: ”10”
    }]
  }    
}

Error Response

XML

<exceptionVo targetNamespace=””>
<statusCode>500</statusCode>
<status>INTERNAL_SERVER_ERROR</status>
        <message>Internal Server Error</message>
<stackTrace></stackTrace>  <!-- optional -->
</exceptionVo>
 
JSON
 
{
        ”statusCode”: ”500”,
        ”Status”: ”INTERNAL_SERVER_ERROR”,
        ”Message”: ”Internal Server Error”,
        ”stackTrace”: ””
}

Data Service

Data Service is a RESTful service that provides end points to get data set information based on job level information.

Table 3-4 Data Service

REST Resource HTTP Method Description
/data/dataset/{jobName}/executions/{jobExecutionId} GET Gets a data set based on job name and job execution id
/data/dataset/{jobName}/instances/{jobInstanceId} GET Gets a data set based on job name and job instance id
/data/dataset/{jobName}nextPending GET Gets next pending data set based on job name

Get Data Set for job name and execution id

Job name - Extractor or downloader-transmitter or uploader job name

Execution id - Job execution id

This endpoint is used by a process flow to get the data set id after the extractor job is run successfully.

Sample Request

http://localhost:7001/bdi-batch-job-admin/resources/data/dataset/Diff_Fnd_ExtractorJob/executions/1

Sample Response

Returns response in either XML or JSON format.

<jobDataSetVo>
  <interfaceModule>Diff_Fnd</interfaceModule>
  <interfaceModuleDataControlId>2</interfaceModuleDataControlId>
  <jobName>Diff_Fnd_ExtractorJob</jobName>
  <jobDataSetInstance>
     <jobInstanceId>1</jobInstanceId>
     <jobDataSetExecutions>
         <jobExecutionId>1</jobExecutionId>
     </jobDataSetExecutions>
  </jobDataSetInstance>
</jobDataSetVo>

Get Data Set for job name and instance id

Job name - Extractor or downloader-transmitter or uploader job

Instance id - Job instance id

Sample Request

http://localhost:7001/bdi-batch-job-admin/resources/data/dataset/Diff_Fnd_ExtractorJob/instances/1

Sample Response

<jobDataSetVo>
  <interfaceModule>Diff_Fnd</interfaceModule>
  <interfaceModuleDataControlId>2</interfaceModuleDataControlId>
  <jobName>Diff_Fnd_ExtractorJob</jobName>
  <jobDataSetInstance>
     <jobInstanceId>1</jobInstanceId>
     <jobDataSetExecutions>
         <jobExecutionId>1</jobExecutionId>
     </jobDataSetExecutions>
  </jobDataSetInstance>
</jobDataSetVo>

Get next pending data set for job name

This endpoint is applicable only to the downloader-transporter or uploader jobs.

Sample Request

http://localhost:7001/bdi-batch-job-admin/resources/data/dataset/Diff_Fnd_DownloaderAndTransporterToRxmJob/nextPending

Sample Response

<jobDataSetVo>
    <interfaceModule>Diff_Fnd</interfaceModule>
    <interfaceModuleDataControlId>9</interfaceModuleDataControlId>
</jobDataSetVo>

Configuration of Job Admin

During the deployment of Job Admin, seed data gets loaded to various tables. Seed data files are located in the bdi-<app>-home/setup-data/dml folder. If seed data is changed, Job Admin need to be reinstalled and redeployed. For loading seed data again during the redeployment, LOADSEEDDATA flag in the BDI_SYSTEM_OPTIONS table need to be set to TRUE.

Jobs

The following job properties can be changed to improve the performance of the jobs.

Item-count - Number of items read by a job before it writes. Default size is 1000.

fetchSize - Number of items cached by JBDC driver. Default size is 1000.

Job xml files are located in bdi-<app>-home/setup-data/job/META-INF/batch-jobs folder. Job xml files can be changed with new values in this folder. If Job xml files are changes, Job Admin need to be reinstalled and redeployed.

Receiver Service

The Receiver Service allows maximum number of blocks for transaction based on the following system option in BDI_SYSTEM_OPTIONS table.

receiverMaxNumBlocks - Default value is set to 10000

Seed data need to be changed to update the maximum number of blocks for the Receiver Service. To update the seed data, set the LOADSEEDDATA flag to TRUE, reinstall and redeploy Job Admin. The Value of the LOADSEEDDATA flag can be changed from the Job Admin Manage Configurations Tab.