| Oracle® Retail Job Orchestration and Scheduler Cloud Service Implementation Guide Oracle® Retail Job Orchestration and Scheduler Cloud Service Implementation Guide Release 19.1.000 F31814-01 |
|
 Previous |
 Next |
| Oracle® Retail Job Orchestration and Scheduler Cloud Service Implementation Guide Oracle® Retail Job Orchestration and Scheduler Cloud Service Implementation Guide Release 19.1.000 F31814-01 |
|
Previous |
Next |
This chapter describes the Job Admin functionality.
Here is a list of the Job Admin features.
Provides management and monitoring of batch jobs.
Services available to start/restart/monitor jobs programmatically.
Monitors executions and logs.
Built on JavaEE JSR352 JavaBatch standard and available on WebLogic.
GUI to manage/start/restart batch jobs.
JavaBatch Runtime Engine to execute job.xml files.
The following section includes information about the Job Admin components.
This section describes the RESTFul services.
The Batch service is a RESTful service that provides various endpoints to manage batch jobs. Here are the key endpoints in the Batch Service.
Start Job
This endpoint starts a job asynchronously based on a job name and returns the execution ID of the job in the response. If the given job is disabled it throws the exception "Cannot start the disabled Job {jobName}. Enable the Job and start it."
Path: /batch/jobs/<jobName>
HTTP Method: POST
Inputs:
Job Name as path parameter
Job Parameters is a query parameter. Job Parameters is a comma-separated list of name value pairs. This is optional.
Sample Request
http://localhost:7001/jos-batch-job-admin/resources/batch/jobs/ShellCommandRunnerJob?jobParameters=key=value
Successful Response
Error Response in case of disable jobs
JSON
{
"statusCode": 500,
"status": "Internal Server Error",
"message": "Cannot start the disabled Job {jobName}. Enable the Job
and start it.",
"stackTrace": "java.lang.RuntimeException:....."
}
Restart Job
This endpoint restarts a job asynchronously using the job execution ID and returns the new job execution ID. If the given job is disabled it throws the exception "Cannot restart the disabled Job {jobName}. Enable the Job and restart."
Path: /batch/jobs/executions/{executionId}
HTTP Method: POST
Inputs: executionId as path parameter
Sample Request
http://localhost:7001/jos-batch-job-admin/resources/batch/jobs/executions/2
Error Response in case of disable jobs
JSON
{
"statusCode": 500,
"status": "Internal Server Error",
"message": "Cannot restart the disabled Job {jobName}. Enable the
Job and restart.",
"stackTrace": "java.lang.RuntimeException:....."
}
Inputs
jobName as path parameter
jobExecutionId as path parameter
Sample Request
http://localhost:7001/jos-batch-job-admin/resources/batch/jobs/ShellCommandRunnerJob/1
Enable or Disable the jobs
This endpoint enables or disables the jobs using jobId, jobEnableStatus and returns the jobId, status and message.
Path: /batch/jobs/enable-disable
HTTP Method: POST
Inputs
JSON
[
{
"jobId": "Diff_Fnd_DownloaderAndTransporterToRxmJob",
"jobEnableStatus": "false"
},
{
"jobId": "CodeHead_Fnd_DownloaderAndTransporterToSimJob",
"jobEnableStatus": "true"
}
]
Sample Request
http://localhost:7001/bdi-batch-job-admin/resources/batch/jobs/enable-disable
Successful Response
JSON
[
{
"jobId": "DiffGrp_Fnd_DownloaderAndTransporterToSimJob",
"jobEnableStatus": "DISABLED",
"message": "Job Disabled Successfully"
},
{
"jobId": "CodeHead_Fnd_DownloaderAndTransporterToSimJob",
"jobEnableStatus": "ENABLED",
"message": "Job Enabled Successfully"
}
]
System Setting Service
See Appendix A for details about System Setting Service.
Job Metrics service provides an end point that produces metrics for jobs based on time range.
Path: /telemetry/jobs
HTTP Method: GET
Inputs: fromTime as query parametertoTime as a query parameter
Sample Response
{
"data-requested-at": "2017-10-09T10:49:57.208-06:00",
"data-requested-from-time": "2017-10-08T10:49:57.197-06:00",
"data-requested-to-time": "2017-10-09T10:49:57.197-06:00",
"jobs-server-runtime-info": {
"id": "bdi-rms-batch-job-admin.war",
"app-type": "BDI",
"app-status": "RUNNING",
"up-since": 1642,
"total-jobs-count": 1,
"total-executions-count": 4,
"successful-executions-count": 0,
"failed-executions-count": 4,
"job": [
{
"name": "ShellCommandRunnerBatchlet",
"slowest-run-duration": 0,
"fastest-run-duration": 0,
"avg-run-duration": 0,
"executions": {
"execution_count": 4,
"success_count": 0,
"failure_count": 4,
"execution": [
{
"execution-id": 121,
"instance_id": 121,
"status": "FAILED",
"startTime": "2017-10-09T10:06:43-06:00",
"endTime": "2017-10-09T10:06:43-06:00",
"step": [
{
"step-execution-id": 121,
"name": "shellCmd",
"duration": 0,
"status": "FAILED"
}
]
},
{
"execution-id": 122,
"instance_id": 122,
"status": "FAILED",
"startTime": "2017-10-09T10:07:36-06:00",
"endTime": "2017-10-09T10:07:36-06:00",
"step": [
{
"step-execution-id": 122,
"name": "shellCmd",
"duration": 0,
"status": "FAILED"
}
]
},
{
"execution-id": 123,
"instance_id": 123,
"status": "FAILED",
"startTime": "2017-10-09T10:08:20-06:00",
"endTime": "2017-10-09T10:08:20-06:00",
"step": [
{
"step-execution-id": 123,
"name": "shellCmd",
"duration": 0,
"status": "FAILED"
}
]
},
{
"execution-id": 124,
"instance_id": 124,
"status": "FAILED",
"startTime": "2017-10-09T10:13:20-06:00",
"endTime": "2017-10-09T10:13:20-06:00",
"step": [
{
"step-execution-id": 124,
"name": "shellCmd",
"duration": 0,
"status": "FAILED"
}
]
}
]
}
}
]
}
}
End points have been added to Batch Service to allow CRUD operations on Job XML.
CREATE Job XML
This end point creates an entry in BDI_JOB_DEFINITION table. It will throw an exception if job already exists.
HTTP Method: PUT
Path: /resources/batch/jobs/job-def-xml/{jobName}
Inputs:
Job Name (e.g. Diff_Fnd_ExtractorJob)
Job XML - It has to be valid XML that conforms to Job XML schema. The value of "id" in the XML should match "jobName" path parameter.
Update Job XML
This end point updates an entry in BDI_JOB_DEFINITION table. It will update if job is not in running state. This end point throws an exception if job doesn't exist in the table.
HTTP Method: POST
Path: /resources/batch/jobs/job-def-xml/{jobName}
Inputs:
Job Name (e.g. Diff_Fnd_ExtractorJob)
Job XML - It has to be valid XML that conforms to Job XML schema. The value of "id" in the XML should match "jobName" path parameter.
Delete Job XML
This end point deletes an entry in BDI_JOB_DEFINITION table. It will delete if job is not in running state and if there is no history in batch database.
HTTP Method: DELETE
Path: /resources/batch/jobs/job-def-xml/{jobName}
Inputs:
Job Name (e.g. Diff_Fnd_ExtractorJob)
Delete Job History
This end point deletes history for a job from batch database. It will delete history if job is not in running state.
HTTP Method: DELETE
Path: /resources/batch/jobs/{jobName}
Inputs:
Job Name (e.g. Diff_Fnd_ExtractorJob)
ReST Endpoints have been introduced for Bulk create/update and Delete of Jobs in BDI/JOS Job admin application. There were existing end points to create/update/delete the individual jobs but it would be a great effort to use them sequentially for high number of jobs. A single end point service which can be used to take care of bulk create OR update operation for new job set has been added. Another end point for deleting multiple jobs has been added. Both the end points provide Job Level success and failure response for Create/Update/Delete.
End point for create/update job definitions
Http method:
Post
Path:
/batch/jobs/bulk/job-definitions
Consumes:
MediaType.APPLICATION_JSON/MediaType.APPLICATION_XML
Sample request:
http://localhost:7001/bdi-batch-job-admin/resources/batch/jobs/bulk/job-definitions
Input:
JSON
{"jobDefinitionsVo":[
{ "jobId": "Job11",
"jobDefXml": "PGpvYi-BpZD0iSm9iMTEiIHZlcnNpb249IjEuMCIgeG1sbnM9Imh0dHA6Ly94bWxucy5qY3Aub3JnL3htbC9ucy9qYXZhZWUiPgogIAgHyb3BlcnRpZXM==" },
{"jobId": "Job12",
"jobDefXml": "PGpvYi-BpZD0iSm9iMTIiIHZlcnNpb249IjEuMCIgeG1sbnM9Imh0dHA6Ly94bWxucy5qY3Aub3JnL3htbC9ucy9qYXZhZWUiPgoJPHByb3BlcnRpZXM+ " } ]
}
Successful Response:
If the result is complete success from endpoint then provide the list of jobDefinitionVo(List<JobDefinitionVo>) for customer reference.
JSON
{
"jobDefinitionsVo": [
{"jobId": "Job1",
"createTime": "create_timestamp",
"updateTime": "update_timestamp",
"status": "Success"
},
{"jobId": "Job2",
"createTime": "create_timestamp",
"updateTime": "update_timestamp",
"status": "Success"
}
]
}
Error Response:
There would be individual calls for each job xml, if any job fails we can still process the next job and can show the list at the end for failed and success jobs.
Possible issues could be:
Cannot update job XML if that job is running.
Cannot create/update the Job if job id was not found in input job xml.
Cannot create/update the Job if input job xml is not readable/parsable.
JSON
{
"jobDefinitionsVo" [
{"jobId": "Job1",
"createTime": "create_timestamp",
"updateTime": "update_timestamp"
"status": "Success"
},
{"jobId": "Job2",
"status": "Failure"
"message": "Currently job is running"
},
{"jobId": "Job3",
"status": "Failure"
"message": "Job id not found in job xml"
}
]
}
Exception Response
End point returns exception response if it is unable to process the request.
{
"statusCode": 500
"status": "Internal Server Error"
"message": "Unable to process"
"stackTrace": ""
}
End point for delete job definitions
Http method:
Delete
Path:
/batch/jobs/bulk/job-definitions
Consumes:
MediaType.APPLICATION_JSON/MediaType.APPLICATION_XML
Sample request:
http://localhost:7001/bdi-batch-job-admin/resources/batch/jobs/bulk/job-definitions
Input:
JSON
{
"jobDefinitionsVo": [
{"jobId":"Job1"},
{"jobId":"Job2"}
]
}
Successful Response:
If the result is complete success from endpoint then provide the list of jobDefinition-Vo(List<JobDefinitionVo>) for customer reference.
JSON
{
"jobDefinitionsVo": [
{"jobId": "Job1",
"deleteTime": "delete_timestamp",
"status": "Success"
},
{"jobID": "Job2",
"deleteTime": "delete_timestamp",
"status": "Success"
},
]
}
Error Response:
There would be individual calls for each job xml, if any job fails to delete we can still process the next job and can show the list at the end for failed and success jobs.
Possible issues could be:
Can't delete job if that job is running.
Can't delete the Job if job id was not found in existing job list.
Can't delete the Job if job is in enabled status.
Can't delete the Job if input job list is not readable/parsable.
JSON
{
"jobDefinitionsVo": [
{"jobID": "Job1",
"deleteTime": "delete_timestamp",
"status": "Success"
},
{"JobId": "Job2",
"status": "Failure",
"message": "Currently job is running"
},
{"jobID": "Job3",
"status": "Failure",
"message": "Cant delete job XML as job job3 is not valid. "
}
]
}
Exception Response
End point returns exception response if it's unable to process the main request.
JSON
{
"statusCode": 500
"status": "Internal Server Error"
"message": "Unable to process"
"stackTrace": ""
}
The Group and Group member service is a RESTful service available in BDI Job Admin that provides endpoints to manage group and members.
Table 3-1 Group and Group Member REST Endpoints
| REST Resource | HTTP Method | Description |
|---|---|---|
|
/manage-group/groups |
GET |
Returns list of all Groups |
|
/manage-group/group/ {groupId} |
GET |
Returns details of the group with input group ID |
|
/manage-group/group |
PUT |
Updates the group details |
|
/manage-group/group |
POST |
Creates new group |
|
/manage-group/group/{groupName} |
DELETE |
Deletes the group with input group name |
|
/manage-group/group/{groupName}/group-members |
GET |
Returns all groups members for the input group name |
|
/manage-group/group/group-member/{groupMemberId} |
GET |
Returns details of the group member with input group member ID |
|
/manage-group/group/group-member |
PUT |
Updates the group member details |
|
/manage-group/group/{groupName}/group-member |
POST |
Create a new member and adds it to the group with input group name |
|
/manage-group/group/{groupName}/group-member/{groupMemberName} |
DELETE |
Deletes the group member with input group name and group member name |
|
/manage-group/groups |
POST |
Adds multiple groups provided in input at a time. |
|
/manage-group/group/{groupName}/group-members |
POST |
Adds multiple members to a given groups at a time. |
|
/manage-group/group/{groupName}/group-members |
DELETE |
Delete all members from given group. |
|
/manage-group/groups |
DELETE |
Deletes all groups provided in input. |
The Job Admin UI provides functionality to manage and monitor jobs. The Job Admin UI is described in detail in Appendix A.
Best practices include:
Use Batchlet if a job must run a script or program that resides locally.
Use ItemReader, ItemWriter, and so on, if a job requires more control over the processing of data. Chunk processing allows data to be processed in chunks, and, if a job fails, it can only process the remaining chunks during a restart.
Use job parameters to dynamically pass parameters to a job.
Use PartitionMapper to specify the number of partitions and the threads for chunk processing so that all data is processed concurrently.
Both Job Admin UI and REST Services are secured with SSL and basic authentication. The following roles are defined to restrict access to operations in Job Admin.
JobAdminRole
JobOperatorRole
JobMonitorRole
Batch jobs can be run from the Job Admin UI or through the Batch REST service. Here are the operations that can be performed by the users based on their roles.
During the deployment of Job Admin, seed data is loaded to various tables. Seed data files are located in the jos-<app>-home/setup-data/dml folder. If seed data is changed, Job Admin must be reinstalled and redeployed. In order to load seed data again during the redeployment, the LOADSEEDDATA flag in the BDI_SYSTEM_OPTIONS table must be set to TRUE.
During the deployment, Job XMLs get loaded to BDI_JOB_DEFINITION table. Job XML files are located in the "jos-job-home/setup-data/META-INF/batch-jobs" folder. If job xmls are changed, Job Admin must be reinstalled and redeployed. In order to load job xmls during redeployment, the LOADJOBDEF flag in the BDI_SYSTEM_OPTIONS table must be set to TRUE.
|
Note: Restart of job does not load job definition from the BDI_JOB_DEFINITION table. Java Batch loads job xml from JOBSTATUS table during the restart of a job.If there is an issue with Job XML, job needs to be started after fixing the job XML. |
Throttling is the capability of regulating the rate of input for a system where output rate is slower than input.
Java Batch runtime will not allow to run multiple instances of a job at same time, it will say job is currently running and fail the job. There can be only one instance of a job at any time (unless the job parameters are different).
Throttling is introduced to address the issue caused when there are many job start requests at the same time. In order to honor the throttle limits "throttleSystemLimit" is introduced to make sure the system never runs more than the throttle limit for the group and the system.
Three new tables are added to job schema to handle throttling, these are BDI_GROUP, BDI_GROUP_LOCK, BDI_GROUP_MEMBER.
Table 3-3 BDI Group
| Column | Type | Comments |
|---|---|---|
|
GROUP_ID |
NUMBER |
Primary Key |
|
APP_TAG |
VARCHAR2(255) |
Name of the application |
|
COMMENTS |
VARCHAR2(255) |
Comments |
|
GROUP_ATTRIB_NAME_1 |
VARCHAR2(255) |
Name of the group attribute ex - THROTTLE_JOBS_IN_SAME_GROUP |
|
GROUP_ATTRIB_NAME_2 |
VARCHAR2(255) |
Name of the group attribute |
|
GROUP_ATTRIB_NAME_3 |
VARCHAR2(255) |
Name of the group attribute |
|
GROUP_ATTRIB_NAME_4 |
VARCHAR2(255) |
Name of the group attribute |
|
GROUP_ATTRIB_NAME_5 |
VARCHAR2(255) |
Name of the group attribute |
|
GROUP_ATTRIB_VAL_1 |
VARCHAR2(255) |
Value of the group attribute |
|
GROUP_ATTRIB_VAL_2 |
VARCHAR2(255) |
Value of the group attribute |
|
GROUP_ATTRIB_VAL_3 |
VARCHAR2(255) |
Value of the group attribute |
|
GROUP_ATTRIB_VAL_4 |
VARCHAR2(255) |
Value of the group attribute |
|
GROUP_ATTRIB_VAL_5 |
VARCHAR2(255) |
Value of the group attribute |
|
GROUP_NAME |
VARCHAR2(255) |
Name of the group |
Table 3-4 BDI Group Member
| Column | Type | Comments |
|---|---|---|
|
GROUP_MEMBER_ID |
NUMBER |
Primary Key |
|
APP_TAG |
VARCHAR2(255) |
Name of the application |
|
GROUP_ID |
NUMBER |
Group id |
|
MEMBER_NAME |
VARCHAR2(255) |
Name of the job |
|
MEMBER_TYPE |
VARCHAR2(255) |
Type of the member ex - job |
Table 3-5 BDI Group Lock
| Column | Type | Comments |
|---|---|---|
|
LOCK_ID |
NUMBER |
Primary Key |
|
APP_TAG |
VARCHAR2(255) |
Name of the application |
|
GROUP_NAME |
VARCHAR2(255) |
Name of the group |
Prerequisite: In the WebLogic console, make sure the job admin data source has "Supports Global Transactions" and "Logging Last Resource" checked in the Transaction tab.
Example on how throttling is handled at runtime:
Group1 <--(job1, job2, job3)-Throttle value 3
Group2 <-- (job1, job2) - Throttle value2
Step1:
Start job1, job2, job3 from process1, process2, process3 respectively. All 3 start running.
Step2:
Then start again process1 and process2. Both job1 and job2 get throttled.
There can be only one instance of a job at any time (unless the job parameters are different).
This section describes the Job Admin errors and troubleshooting.
Issue: Job Admin deployment can encounter this error if the database credentials are invalid:
Caught: javax.management.RuntimeMBeanException: java.lang.RuntimeException: weblogic.management.provider.EditFailedException: java.lang.NullPointerException
javax.management.RuntimeMBeanException: java.lang.RuntimeException: weblogic.management.provider.EditFailedException: java.lang.NullPointerException
at weblogic.utils.StackTraceDisabled.unknownMethod()
Caused by: java.lang.RuntimeException: weblogic.management.provider.EditFailedException: java.lang.NullPointerException
1 more
Caused by: weblogic.management.provider.EditFailedException:
java.lang.NullPointerException
1 more
Caused by: java.lang.NullPointerException
1 more
Solution:
Undo all changes in the WebLogic domain session. Redeploy the application, setting up new credentials and verifying that the deployment has been successful.
Issue: Log files contain this exception:
oracle.wsm.common.sdk.WSMException: WSM-07620 : Agent cannot enforce policies due to either failure in retrieving policies or error in validations, detail= "WSM-02557 The documents required to configure the Oracle Web Services Manager runtime have not been retrieved from the Policy Manager application (wsm-pm), possibly because the application is not running or has not been deployed in the environment. The query "&(@appliesTo~="REST-CLIENT()")(policysets:global:%)" is queued for later retrieval.
Solution: Follow the BDI Installation guide and verify that the WSM- policy manager is configured for the admin server URL.
Open the WebLogic domain console and target the wsm-pm application to Admin Server. Bounce Admin server and verify that the wsm-pm application is in an active state.
Job Fails and Job Admin Log Files Contain No Details of the Failure
Issue: A job fails and the Job Admin log files contain no evidence of or details about the failure.
Solution: Examine the WebLogic server log files to identify the root cause of the job failure. A possible root cause is the improper data source configuration.
Job Admin UI Throwing Error: Job XML Not Found
Issue: Log files contain this exception:
Caused By: javax.ejb.EJBException: EJB Exception: : java.lang.RuntimeException: Could not find jobName(ShellCommandRunnerBatchlet) xml file. You may have renamed the job file or your job repository has more jobs than your application. To resolve the issue either delete the job repository or add the correct job xml file to the app.
Solution: The job has been deleted from the jos-job-home before redeployment. Either add the missing job XML or delete the execution records of the job from the batch database.
Job Admin UI Throwing Error: Table or View Doesn't Exists
Issue: Log files contain this exception:
<Error> <javax.enterprise.resource.webcontainer.jsf.application> <BEA-000000> <Error Rendering View[/index.xhtml] javax.el.ELException: /index.xhtml @15,84 value="#{batchSummaryRequestBean.refreshPage}": javax.ejb.EJBException: EJB Exception: : com.ibm.jbatch.container.exception.PersistenceException: java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist
Solution: BatchInfrastructure database is not pointing to a valid schema. Check if the schema has the following tables: CHECKPOINTDATA, STEPEXECUTIONINSTANCEDATA, STEPSTATUS, EXECUTIONINSTANCEDATA, JOBINSTANCEDATA, JOBSTATUS. If not, then run the DDL:
$Oracle_Home/oracle_common/common/sql/wlsservices/batch/oracle/jbatch.sql
IO Exception or Permissions Issue On Running A Shell Runner Job
Issue: java.io.IOException: Cannot run program "./TestShell1.sh" (in directory "/u00/webadmin/19.1.000/Scripts"): error=13, Permission denied
Solution: Check if the script the job is running is present in the specified location or not.
Check if the required permissions are provided for running the script.
Missing Credentials Access Permission
Issue: Caused by: java.lang.RuntimeException: Cannot get the Credential Map with the specified appLevelKeyPartitionName(DEFAULT_KEY_PARTITION_NAME).at com.oracle.retail.integration.common.security.credential.CredentialStoreManager.getAllUserNameAliases(CredentialStoreManager.java:1171) ~[retail-public-security-api-19.1.000.jar:?]Caused by: java.security.AccessControlException: access denied ("oracle.security.jps.service.credstore.CredentialAccessPermission" "context=SYSTEM,mapName=DEFAULT_KEY_PARTITION_NAME,keyName=*" "read")
Solution: Verify weblogic.policy file. The credential access permissions should be added for the domain where the applications are deployed. Add the permissions and restart the Admin and managed server.
Issue: Caused by: java.lang.IllegalArgumentException: alias(processCallBackServiceUrlUserAlias) not found in the credential store.
Solution: Add the system credentials using the UI or REST service.
Issue: 2017-03-31T02:49:42,628 [Thread-132] DEBUG Logger$debug - Starting job: null/resources/batch/jobs/DiffGrp_Fnd_ExtractorJob 2017-03-31T02:49:42,658 [Thread-132] ERROR Logger$error$0 - Error calling activity. java.lang.NullPointerException: Cannot invoke method getBytes() on null object
Solution: Add the system options using the UI or ReST service.
