Resolving the Schedule Job Errors

22.6.2 Resolving the Schedule Job Errors

Diagnosing scheduled job errors include activities such as, enabling scheduler logging, and verifying that scheduler is running, the job is enabled, and clocks are in sync on all nodes.

To diagnose this issue:

Verify whether scheduler service is running or not. Scheduler service is deployed on each node of the cluster until this service is not explicitly disabled. This can be disabled by setting the scheduler.disabled server level property to false for that node. The following URL can be used to verify the status of the scheduler service:

http://OIM_HOST:OIM_PORT/SchedulerService-web/status

In this URL, OIM_HOST is the name of the computer hosting the Oracle Identity Manager server and OIM_PORT is the port on which Oracle Identity Manager server is listening.
Verify whether the specific job is enabled or not. This can be verified from the Scheduler section of Identity System Administration. The job must be enabled to run per the schedule.
Verify whether clocks are in sync for all nodes. Clocks must be within a second of each other
Delete the existing trigger from Scheduler UI, and schedule a new trigger from the UI. Verify whether the issue persists or not.
Enable scheduler logs by changing log level to DEBUG. This can be done by changing log level for the oracle.iam.scheduler.impl package from Oracle Enterprise Manager. Verify whether the following messages are traced in logs or not:
```
Job Listener, Job was executed '$JOB_NAME'
Job Listener, Job to be executed '$JOB_NAME'
```
Here, $JOB_NAME is the name of the job that is supposed to be executed at that time.

If the messages are not logged, then contact Oracle Support.
In Oracle Enterprise Manager, check the threadPoolSize parameter for the schedulerConfig segment in the oim-config.xml file. This is the number of threads that are available for concurrent execution of jobs. Therefore, the number of jobs that can be executed on a particular time cannot be more than the configured threadPoolSize count. Running of such jobs is skipped and executed as per the next scheduled time, which gives an impression that the job is not executed per the scheduled time. The default value of this parameter is 10, but is can be tuned as required.
Restart the server and verify whether the job has been run or not.

Verify whether the following exception is logged:

Caused By: java.lang.NullPointerException at org.quartz.SimpleTrigger.computeNumTimesFiredBetween(SimpleTrigger.java:800)

Run following query to fix this issue:

UPDATE QRTZ92_TRIGGERS SET NEXT_FIRE_TIME=1 WHERE  NEXT_FIRE_TIME<1;

Sometimes the trigger status is not updated in the QRTZ92_TRIGGER table from BLOCKED to PAUSED state. This situation happens if the environment is not tuned properly, and database connections from the pool are exhausted by other parallel operations running on the server. As a result, QUARTZ framework is not able to get connection from the pool to update the running job. This situation can be identified if exceptions related to database connection pool is observed in the server logs. Usually, such triggers get fixed after server restart, but if trigger status still remains the same, then running the following query can help:
```
UPDATE QRTZ92_TRIGGERS SET TRIGGER_STATUS='WAITING' WHERE JOB_NAME ='$JOB_NAME'
```
Replace $JOB_NAME with the job name.
Sometimes manual trigger for a job is not updated in the QRTZ92_TRIGGER table. Manual trigger is created in the system when you execute the job by clicking Run Now from the Scheduler UI or use the Scheduler runNow() API. Such trigger is supposed to be deleted after the job is executed successfully. To fix this issue:
1. Shutdown the server.
2. Run the following queries on Oracle Identity Manager database:
```
DELETE FROM QRTZ92_FIRED_TRIGGERS where TRIGGER_NAME like ('MT_%');
DELETE FROM QRTZ92_SIMPLE_TRIGGERS where TRIGGER_NAME like ('MT_%');
DELETE FROM QRTZ92_TRIGGERS where TRIGGER_NAME like ('MT_%');
```
Automatic deletion of such manual triggers is maintained by the Quartz framework.