Go to primary content
Oracle® Retail Bulk Data Integration Cloud Service Implementation Guide
Release 19.1.000
F31810-01
  Go To Table Of Contents
Contents

Previous
Previous
 
Next
Next
 

2 Job Administrator

BDI Job Admin is a web application that provides the runtime and GUI for managing batch jobs. It provides the following high level functionality.

Job Admin Core Components

The BDI Job Admin contains the batch jobs for moving bulk data from source (senders) systems (for example RMS) to destination (receiver) systems (for example SIM, RPAS and so on). A bulk integration flow moves data for one family from source to destination application(s).

An Integration Flow is made up of the multiple activities: Extractor, Downloader, Transporter, FileCreator, Uploader, and Importer. These activities are implemented as batch jobs.

An Extractor Job extracts data for a Family from a source system and moves data to the outbound Interface Tables.

Outbound Interface Tables typically exist in the integration database schema and the schema resides in the source system database.

Extractor Job

The Extractor Job uses a Batchlet and PL/SQL stored procedures to move data from transactional tables of source system (for example RMS) to outbound tables. A PL/SQL stored procedure calls BDI PL/SQL stored procedure to insert data set information in the outbound data control tables. Extractor jobs are currently implemented to provide full data (not delta) for an interface.

extractor job

BDI Extractor (PL/SQL Application)

  1. The Extractor job is run from App A (for example RMS) Extractor Job Admin application through REST or UI.

  2. The Extractor job invokes PL/SQL stored procedure in App A database.

  3. A PL/SQL stored procedure is run in the App A database.

  4. The PL/SQL stored procedure moves data from transactional tables to the outbound tables in the BDI schema.

  5. The PL/SQL stored procedure inserts entries in downloader data control tables to indicate the data set is ready for download.


Note:

Review Appendix E.

Sample Extractor – PL/SQL application code that calls procedures in PL/SQL package.


The Downloader Data Control Tables act as a handshake between the Extractor and the Downloader. There are two Outbound Data Control Tables:

  • BDI_DWNLDR_IFACE_MOD_DATA_CTL

  • BDI_DWNLDR_IFACE_DATA_CTL

The Extractor job inserts entries in the downloader data control tables to indicate that data is ready to be downloaded after it completes moving data to outbound interface tables.

Downloader-Transporter job

A Downloader-Transporter job downloads the data set from outbound interface tables for an Interface Module (family) and streams data to a BDI destination application using the Receiver Service.

If there are multiple Interfaces for an Interface Module, data for all interfaces for that interface module are downloaded and streamed concurrently to the Receiver Service of BDI destination application.

download transporter

BDI Downloader Transporter

  1. The Downloader Transporter job is run from the BDI App A Job Admin application through REST or UI.

  2. The Downloader Transporter job checks for new data sets in Downloader Data Control Tables.

  3. If a Data Set is available for download, the Downloader Transporter job downloads a block of data from the outbound table.

  4. The Downloader Transporter job streams downloaded blocks to Receiver Service.

  5. The Receiver Service stores meta data in Receiver Service database.

  6. By default, the Receiver Service inserts the data directly into receiver and inbound tables.

  7. The Receiver Service inserts an entry in the importer data control table indicating that the data set is ready for upload.

Rules for processing a data set by Downloader Job

  1. A full data set is available for download, if it is not processed by a downloader job yet and if a newer full data set is not processed successfully.

  2. If data set id is passed through job parameters (for example jobParameters=dataSetId=1) to downloader job, it will use the data set if it is available as per the above rule. Otherwise job will fail.

  3. If the data set id is not passed through job parameters to downloader job, it will identify the next available data set if there is one. Otherwise job completes without processing any data set.

  4. If the downloader-transporter job fails for whatever reason, the data set that it tried to download can only be downloaded by restarting the job after fixing the issues.

  5. If the downloader-transporter job is started instead of a restart, it will either pick up a new data set or none.

Downloader Data Sets

A Data Set consists of rows between a begin and end sequence number (bdi_seq_id column) in the Outbound Interface Table. The BDI_SEQ_ID column is automatically incremented when data is inserted into the outbound table.

The Downloader-Transporter job downloads a single data set that is not downloaded yet from the outbound interface tables.

If a data set id is passed as job parameter (for example jobParameters=dataSetId=1) to Downloader-Transporter job, it will use that data set if it is available for download. Job Parameters as a query parameter. Job Parameters is a comma separated list of name value pairs. This parameter is optional.

If there are multiple data sets in the outbound tables that are available for download, then the Downloader-Transporter job picks up the oldest data set.

If there is no data set available in the outbound tables, the Downloader-Transporter job completes without downloading any data.

If a newer data set is processed by the Downloader-Transporter job, then older data set cannot be processed.

A data set is divided into Logical Partitions and data in each partition is downloaded by a separate thread. The Downloader-Transporter job tries to allocate data equally between partitions. Data in each partition is divided into blocks based on the ”item-count” value in the job and each block is retrieved from an outbound table and streams it to the destination application using the Receiver Service.

A data set is divided into logical partitions based on the number of partitions specified in the BDI_DWNLDR_TRNSMITTR_OPTIONS table and the number of rows in the data set.

The number of rows is calculated by subtracting the begin sequence number from the end sequence number provided in the BDI_DWNLDR_IFACE_DATA_CTL table. The number of rows may be approximate as there can be gaps in sequence numbers.

The number of rows allocated to each logical partition is calculated by dividing the approximate row count with the number of partitions.

Example 1:
Begin Sequence number = 1
End Sequence number = 100
Number of partitions = 2
 
Approximate row count = 100 - 1 + 1 
Items for partition = 100/2 = 50
Data assigned to partition 1
                        Begin Sequence number = 1
         End Sequence number = 1 + 50 - 1 = 50
Data assigned to partition 2
         Begin Sequence number = 51
         End Sequence number = 51 + 50 - 1 = 100
 
Example 2:
Begin Sequence number = 1
End Sequence number = 75
Number of partitions = 2
 
Approximate row count = 75 - 1 + 1 
Items for partition = 75/2 = 37
Extra items = 75 % 2 = 1
Data assigned to partition 1
        Begin Sequence number = 1
         End Sequence number = 1 + 37 - 1 = 37
Data assigned to partition 2
         Begin Sequence number = 38
         End Sequence number = 38 + 37 + 1 - 1 = 75

The Downloader-Transporter job deletes data from outbound tables after the successful completion of the job if AUTO_PURGE_DATA flag in BDI_DWNLDR_TRNSMITTR_OPTIONS table is set to TRUE. The default value for this flag is FALSE. If sender side split topology is used, this flag needs to be changed to FALSE. Otherwise all destination applications may not get the data.

When a Downloader-Transporter job fails, the error information such as stack trace gets stored in BDI_JOB_ERROR and BDI_DOWNLOADER_JOB_ERROR tables. Errors are displayed in the ”Diagnostics” tab of the Job Admin GUI. The error information can be used to fix the issues before restarting the failed job. Note that if there are exceptions in Batch runtime, then those exceptions won't show up in the Job Error tables and so in the Diagnostics tab of the Job Admin GUI.

Downloader-Transporter Job Configuration

Seed data for the Downloader-Transporter jobs is loaded to the database during the deployment of Job Admin. Some of the seed data can be changed from the Job Admin GUI.

BDI_SYSTEM_OPTIONS

During the installation of Job Admin, the following information is provided by the user and that information is loaded into the BDI_SYSTEM_OPTIONS table.

Table 2-1 System Options

Column Type Comments

VARIABLE_NAME

VARCHAR2(255)

Name of the system variable

APP_TAG

VARCHAR2(255)

The application name

VARIABLE_VALUE

VARCHAR2(255)

Value of the variable

CREATE_TIME

TIMESTAMP

Time it was created

UPDATE_TIME

TIMESTMP

Time it was updated


<app>JobAdminBaseUrl - Base URL for Job Admin of destination applications such as sim/rpas

<app>JobAdminBaseUrlUserAlias - User alias for Job Admin of destination applications such as sim/rpas

<app> - Destination application name (for example sim or rpas)

Example:
MERGE INTO BDI_SYSTEM_OPTIONS USING DUAL ON (VARIABLE_NAME='rpasJobAdminBaseUrl' and APP_TAG='rms-batch-job-admin.war') WHEN MATCHED THEN UPDATE SET VARIABLE_VALUE='http://rxmhost:7001/rpas-batch-job-admin', UPDATE_TIME=SYSDATE WHEN NOT MATCHED THEN INSERT (VARIABLE_NAME, APP_TAG, VARIABLE_VALUE, CREATE_TIME) VALUES('rpasJobAdminBaseUrl', 'rms-batch-job-admin.war', 'http://rpashost:7001/rpas-batch-job-admin', SYSDATE)

BDI_INTERFACE_CONTROL

During the design time, seed data for the BDI_INTERFACE_CONTROL table is generated for all interface modules (aka families) for a job type (DOWNLOADER, UPLOADER) so that interface modules are active.

Table 2-2 Interface Control

Column Type Comments

ID

NUMBER

Primary Key

INTERFACE_CONTROL_COMMAND

VARCHAR2(255)

ACTIVE or IN_ACTIVE

INTERFACE_MODULE

VARCHAR2(255)

Name of interface module

SYSTEM_COMPONENT_TYPE

VARCHAR2(255)

DOWNLOADER or UPLOADER


Example:
insert into BDI_INTERFACE_CONTROL (ID, INTERFACE_CONTROL_COMMAND, INTERFACE_MODULE, SYSTEM_COMPONENT_TYPE) values (1, 'ACTIVE', 'Diff_Fnd', 'DOWNLOADER')

BDI_DWNLDR_TRNSMITTR_OPTIONS

Seed data for BDI_DWNLDR_TRNSMITTR_OPTIONS specifies various configuration options for the Downloader-Transmitter job. Seed data is generated during design time and executed during deployment.

Table 2-3 Transmitter Options

Column Type Comments

ID

NUMBER

Primary Key

INTERFACE_MODULE

VARCHAR2(255)

Name of interface module

INTERFACE_SHORT_NAME

VARCHAR2(255)

Name of the interface

RECVR_END_POINT_URL

VARCHAR2(255)

Name of the URL variable in BDI_SYSTEM_OPTIONS table

RECVR_END_POINT_URL_ALIAS

VARCHAR2(255)

Name of the URL alias variable in BDI_SYSTEM_OPTIONS table

PARTITION

NUMBER

Number of partitions used by Downloader-Transporter job. Default value is 10. This value can be changed through Job Admin GUI

THREAD

NUMBER

Number of threads used by Downloader-Transporter job. Default value is 10. This value can be changed through Job Admin GUI.

QUERY_TEMPLATE

VARCHAR2(255)

Query to be run by downloader job

AUTO_PURGE_DATA

VARCHAR2(255)

This flag indicates Downloader-Transporter job whether to clean data set in the outbound table after the job successfully downloads the data set. Default value is set to True. This value need to be changed based on the deployment topology used for bulk data integration.

COLUMN_FILTER

VARCHAR2(4000)


ROW_FILTER

VARCHAR2(4000)



Example:
 
MERGE INTO BDI_DWNLDR_TRNSMITTR_OPTIONS USING DUAL ON (ID=1) WHEN MATCHED THEN UPDATE SET INTERFACE_MODULE='Diff_Fnd', INTERFACE_SHORT_NAME='Diff', RECVR_END_POINT_URL='rpasJobAdminBaseUrl', RECVR_END_POINT_URL_ALIAS='rpasJobAdminBaseUrlUserAlias', PARTITION=10, THREAD=10, QUERY_TEMPLATE='select * from InterfaceShortName where (bdi_seq_id between ? and ?) QueryFilter order by bdi_seq_id', AUTO_PURGE_DATA='TRUE'  WHEN NOT MATCHED THEN INSERT (ID, INTERFACE_MODULE, INTERFACE_SHORT_NAME, RECVR_END_POINT_URL, RECVR_END_POINT_URL_ALIAS, PARTITION, THREAD, QUERY_TEMPLATE, AUTO_PURGE_DATA) values (1, 'Diff_Fnd', 'Diff', 'rpasJobAdminBaseUrl', 'rpasJobAdminBaseUrlUserAlias', 10, 10, 'select * from InterfaceShortName where (bdi_seq_id between ? and ?) QueryFilter order by bdi_seq_id', 'TRUE')
 
MERGE INTO BDI_DWNLDR_TRNSMITTR_OPTIONS USING DUAL ON (ID=1) WHEN MATCHED THEN UPDATE SET INTERFACE_MODULE='Diff_Fnd', INTERFACE_SHORT_NAME='Diff', RECVR_END_POINT_URL='rpasJobAdminBaseUrl', RECVR_END_POINT_URL_ALIAS='rpasJobAdminBaseUrlUserAlias', PARTITION=10, THREAD=10, QUERY_TEMPLATE='select * from InterfaceShortName where (bdi_seq_id between ? and ?) QueryFilter order by bdi_seq_id', AUTO_PURGE_DATA='TRUE'  WHEN NOT MATCHED THEN INSERT (ID, INTERFACE_MODULE, INTERFACE_SHORT_NAME, RECVR_END_POINT_URL, RECVR_END_POINT_URL_ALIAS, PARTITION, THREAD, QUERY_TEMPLATE, AUTO_PURGE_DATA) values (2, 'Diff_Fnd', 'Diff', 'rpasJobAdminBaseUrl', 'rpasJobAdminBaseUrlUserAlias', 10, 10, 'select * from InterfaceShortName where (bdi_seq_id between ? and ?) QueryFilter order by bdi_seq_id', 'TRUE')

Downloader-Transporter Job Properties

The following job properties can be changed in the Downloader-Transporter jobs to tune the performance.

item-count

Item Count is an attribute of the ”chunk” element in the Downloader-Transporter job. The default value for ”item-count” is set to 1000. The Downloader job retrieves 1000 rows of data from the database before it sends data to the Receiver Service.

<chunk checkpoint-policy="item" item-count="1000">

This value can be changed to fine tune the performance of the Downloader-Transporter job. You need to manually change the value in the job xml files in bdi-<app>-home/setup-data/job/META-INF/batch-jobs folder and reinstall the app. Increasing the item count will increase memory utilization.

fetchSize

The Fetch Size is a property in the Downloader-Transporter job.

FetchSize property is used by JDBC to fetch n number of rows and cache them. The default value is set to 1000. Typically ”item-count” and ”fetchSize” values are identical to get better performance.

<property name="fetchSize" value="1000"/>

This value can be changed to fine tune the performance of the Downloader-Transporter job. You need to manually change the value in the job xml files.

Cleanup

The Downloader-Transporter job deletes data from outbound tables after the successful completion of the job if the AUTO_PURGE_DATA flag in BDI_DWNLDR_TRNSMITTR_OPTIONS table is set to TRUE. The default value for this flag is FALSE. If sender side split topology is used, this flag needs to be changed to FALSE. Otherwise all destination applications may not get the data.

Auto Purge Delay

New system options have been added to introduce delay in purging data.

Auto Purge System Options for auto purge outbound data

autoPurgeOutboundData.global - Valid values are TRUE or FALSE

autoPurgeOutboundData.<Interface Module> - Interface module level system option that overrides global system option

autoPurgeOutboundDataDelay.global - The value of can be specified in days or hours.Examples are 24h = 24 hours or 30d = 30 days

autoPurgeOutboundDataDelay.<Interface Module> - Interface module level system option that overrides global system option. The value of can be specified in days or hours.Examples are 24h = 24 hours or 30d = 30 days

Auto Purge System Options for auto purge of inbound data

autoPurgeInboundData.global - Valid values are TRUE or FALSE

autoPurgeInboundData.<Interface Module> - Interface module level system option that overrides global system option

autoPurgeInboundDataDelay.global - The value of can be specified in days or hours.Examples are 24h = 24 hours or 30d = 30 days

autoPurgeInboundDataDelay.<Interface Module> - Interface module level system option that over-rides global system option. The value of can be specified in days or hours.Examples are 24h = 24 hours or 30d = 30 days

The value of autoPurgeDelay system option can be specified in days or hours.

Examples

30d - 30 days

24h - 24 hours

If "d" or "h" is not included in the value, then it is considered days.

The default autoPurgeDelay is 30 days. This value applies when autoPurgeDelay system option is not specified.

Extractor Cleanup Job

Extractor cleanup job purges data from outbound tables for data sets that have been successfully processed. It uses the above mentioned system options to decide whether to purge and when to purge data.

Importer Job

Importer job purges data from inbound tables for data sets that have been successfully processed. It uses the above mentioned system options to decide whether to purge and when to purge data.

Error Handling

When a Downloader-Transporter job fails, error information like the stack trace gets stored in the BDI_JOB_ERROR and BDI_DOWNLOADER_JOB_ERROR tables. Errors are displayed in the ”Diagnostics” tab of the Job Admin GUI. The error information can be used to fix the issues before restarting the failed job.


Note:

If there are exceptions in Batch runtime, then those exceptions won't show up in Job Error tables and so in Diagnostics tab of Job Admin GUI.

BDI_DOWNLOADER_JOB_ERROR

Table 2-4 Downloader Job Error

Column Type Comments

DOWNLOADER_JOB_ERROR_ID

NUMBER

Primary key

PARTITION_INDEX

VARCHAR2(255)

Partition number of data set

BLOCK_NUMBER

NUMBER

Block number in the partition

BEGIN_SEQ_NUM_IN_BLOCK

NUMBER

Begin sequence number in the block

END_SEQ_NUM_IN_BLOCK

NUMBER

End sequence number in the block

JOB_ERROR_ID

NUMBER

Foreign key to JOB_ERROR table


BDI_JOB_ERROR

Table 2-5 Job Error

Column Type Comments

JOB_ERROR_ID

NUMBER

Primary key

CREATE_TIME

TIMESTAMP

Time when error occurred

TRANSACTION_ID

VARCHAR2(255)

Transaction Id of the job

INTERFACE_MODULE

VARCHAR2(255)

Name of the interface module

INTERFACE_SHORT_NAME

VARCHAR2(255)

Name of the interface

DESCRIPTION

VARCHAR2(1000)

Error description

STACK_TRACE

VARCHAR2(4000)

Stack trace


Downloader-FileCreator Job

A DownloaderAndFileCreator job downloads data from outbound interface table for an interface module (family) and creates a file. It also creates a zero byte trigger file.

The names of the data and trigger files are specified as properties in the DownloaderAndFileCreator job. It copies the data and trigger files to outbound locations for destinations specified in "destination" property of the job. The outbound locations are specified in system options and the job derives the system option key using destination name. DownloaderAndFileCreator jobs are used by RMS Job Admin to create files for RPAS.

Example properties for Calendar_Fnd_DownloaderAdnFileCreatorToRpasJob

<property name="fileName" value="rms_clnd.csv.dat"/>
<property name="fileDataFormat" value="CSV"/>
<property name="triggerFileName" value="rms_clnd.complete"/>
<property name="destination" value="MFP,RDF,AP,IP"/>

DownloaderAndFileCreator job allows the order of the columns in the file and columns to be filtered. The columns specified in the columnFilter property are excluded in the file. Data in the file is in the order of the columns specified in columnOrder property.

Example properties for columnFilter and columnOrder. The delimiter for columnFilter is comma and for columnOrder is pipe character.

<property name="columnFilter" value="BDI_SEQ_ID,BDI_APP_NAME,BDI_DATASET_TYPE,BDI_DATASET_ACTION"/>
<property name="columnOrder" value="TO_CHAR(Day,'YYYYMMDD')|TO_CHAR(Week,'YYYYMMDD')|Month|Quarter|Half|Year|Week_of_Year|Day_of_Week"/>

DownloaderAndFileCreator job uses multiple job partitions to download data and create files. It then merges files from all partitions to create a single data file.The following system options are used by the job for providing flexibility.

mergeFilesFlag

Files created by various partitions are merged by the job by default. Job won't merge files if this flag is set to False.

triggerFileFlag

Trigger file is created by the job by default. Job won't create trigger file if this flag is set to False.

copyFilesFlag

Files are copied to outbound locations by default. Job won't copy files to outbound locations if this flag is False.

overwriteOutboundFilesFlag

Files are not overwritten at outbound location by default. Job will overwrite files if this flag is set to True.

<Destination>_outboundLocation

Job uses this system option to find out the outbound location for a destination. If there are multiple destinations, then multiple system options need to be set.

Receiver Service

The Receiver Service is a RESTful service that provides various endpoints to send data transactionally.

The Receiver Service is part of Job Admin. It stores data as files and keeps track of metadata in the database. The Receiver Service also supports various merge strategies for merging files at the end.

The Receiver Service is used by the Downloader-Transporter job to transmit data from source to destination. By default, the Receiver Service inserts the data directly into receiver and inbound tables. There is an option to configure whether to transfer data to database or file system using system option property, receiverOutputType. If system option is not provided then receiver service defaults to database output type.

Importer Job

The tables BDI_IMPRTR_IFACE_MOD_DATA_CTL and BDI_IMPORTER_IFACE_DATA_CTL act as a handshake between the receiver service and importer jobs. When the Receiver Service completes processing a data set successfully, it creates an entry in these tables.

An entry in the table BDI_IMPRTR_IFACE_MOD_DATA_CTL indicates to the Importer Job that a data set is ready to be imported.

The Importer job imports a data set for an Interface Module from inbound tables into application specific transactional tables. Importer jobs are application (for example SIM/RPAS) specific jobs. It uses the Importer Data Control Tables to identify whether a data set is ready for import or not.

importer

RPAS Importer

  1. Importer job is run from App B RPAS Job Admin application through REST or UI.

  2. Importer job checks for data sets in importer data control tables.

  3. If data set is available for import, importer job downloads data from inbound table.

  4. Importer job loads data to App B RPAS staging tables.

BDI_IMPRTR_IFACE_MOD_DATA_CTL

Importer File Creator Job

Importer File Creator job imports data from inbound table and creates a file at the outbound location. This job provides the functionality mentioned below.

  • Column Filter - Filters data for columns specified in columnFilter property of the job.

  • Row Filter - Filters data based on the predicate provided in the rowFilter property of the job

  • Column Order - Column order can be specified in the columnOrder property of the job. If it is not specified, job uses order of the columns in the inbound table.

  • Merge files - Merges files created for each partition. The system option "importerMergeFilesFlag" can be set to FALSE not to merge the files. By default, job merges the files.

  • Copy files - Copies files to outbound location. The system option "importerCopyFilesFlag" can be set to FALSE not to copy the files. By default, job copies the files. Job expects the system option <Destination>_importerOutboundLocation if importerCopyFilesFlag is set to TRUE.

  • Purge Data - Purges data from inbound tables. AUTO_PURGE_DATA flag in BDI_IMPORTER_OPTIONS can be set to FALSE not to purge data after successful completion of the job. Seed data sets the flag to TRUE.

  • Import File Location - By default, files are created in "bdi-data" folder. A system option <Job Name>.importFileLocation can be specified so that files are created in configured directory. A global system option "importFileLocation" can be specified for all importer file creator jobs.

  • Job won't create any file if dataSetId is not specified

  • If dataset has no data, then job will create an empty file.

  • Job won't allow processing of a dataset if it is already processed.

Importer File Creator job is currently added only for OMS InvAvailWh_Tx.

Table 2-6 Importer Data

Column Type Comments

IMPORTER_IFACE_MOD_DATACTL_ID

NUMBER

Primary key

INTERFACE_MODULE

VARCHAR2(255)

Name of the interface module

SOURCE_SYSTEM_NAME

VARCHAR2(255)

Name of the source system

SOURCE_DATA_SET_ID

NUMBER

Source data set id

SRC_SYS_DATA_SET_READY_TIME

TIMESTAMP

Time when data set was ready in outbound tables

DATA_SET_TYPE

VARCHAR2(255)

Type of data set (FULL or PARTIAL)

DATA_SET_READY_TIME

TIMESTAMP

Time when data set was available in inbound tables

UPLOADER_TRANSACTION_ID

NUMBER

Transaction id of the uploader job


BDI_IMPORTER_IFACE_DATA_CTL

Table 2-7 Importer Data

Column Type Comments

IMPORTER_IFACE_DATA_CTL_ID

NUMBER

Primary key

INTERFACE_SHORT_NAME

VARCHAR2(255)

Name of the interface

INTERFACE_DATA_BEGIN_SEQ_NUM

NUMBER

Beginning sequence number of the data set in the inbound table

INTERFACE_DATA_END_SEQ_NUM

NUMBER

Ending sequence number of the data set in the inbound table

IMPORTER_IFACE_MOD_DATACTL_ID

NUMBER

Foreign key to BDI_IMPRTR_IFACE_MOD_DATA_CTL table

SRC_SYS_INTERFACE_DATA_COUNT

NUMBER


INTERFACE_DATA_COUNT NUMBER

NUMBER