5 Troubleshooting Oracle Enterprise Scheduler

This chapter describes common problems that you might encounter when using Oracle Enterprise Scheduler and explains how to solve them.

This chapter contains the following topics:

• Introduction to Troubleshooting Oracle Enterprise Scheduler

• Getting Started with Troubleshooting Oracle Enterprise Scheduler Jobs

• Problems and Solutions

To gain insight into the log details generated by Oracle Fusion applications, see the "Troubleshooting Oracle Fusion Applications Using Incidents, Logs, QuickTrace, and Diagnostic Tests" chapter in the Oracle Fusion Applications Administrator's Guide. Also, review the Oracle Fusion Middleware Error Messages Reference for information about the error messages you may encounter.

5.1 Introduction to Troubleshooting Oracle Enterprise Scheduler

This section provides guidelines and a process for using the information in this chapter. Using the following guidelines and process will focus and minimize the time you spend resolving problems.

Guidelines

When using the information in this chapter, Oracle recommends:

• After performing any of the solution procedures in this chapter, immediately retrying the failed task that led you to this troubleshooting information. If the task still fails when you retry it, perform a different solution procedure in this chapter and then try the failed task again. Repeat this process until you resolve the problem.

• Making notes about the solution procedures you perform, symptoms you see, and data you collect while troubleshooting. If you cannot resolve the problem using the information in this chapter and you must log a service request, the notes you make will expedite the process of solving the problem.

Process

Follow the process outlined in Table 5-1 when using the information in this chapter. If the information in a particular section does not resolve your problem, proceed to the next step in this process.

Table 5-1 Process for Using the Information in this Chapter

Step	Section to Use	Purpose
1	Section 5.2	Get started troubleshooting Oracle Enterprise Scheduler jobs.
2	Section 5.3	Perform problem-specific troubleshooting procedures. This section describes: Possible causes of the problems Solution procedures corresponding to each of the possible causes
3	Section 13.1	Use My Oracle Support to get additional troubleshooting information about Oracle Fusion Applications or Oracle Enterprise Scheduler. My Oracle Support provides access to several useful troubleshooting resources, including Knowledge Base articles and Community Forums and Discussions.
4	Section 13.1	Log a service request if the information in this chapter and My Oracle Support does not resolve your problem. You can log a service request using My Oracle Support at https://support.oracle.com.

5.2 Getting Started with Troubleshooting Oracle Enterprise Scheduler Jobs

You may want to troubleshoot the following typical issues that can arise when running Oracle Enterprise Scheduler jobs.

• Asynchronous jobs remain in running state indefinitely.

• An asynchronous job hangs or crashes.

• Oracle Enterprise Scheduler is down when the remote scheduled job completes or there are network problems such that Oracle Enterprise Scheduler does not receive the completion status from the remote job.

• A scheduled job is ready to execute, but does not execute.

• A scheduled job is placed in manual error recovery state where troubleshooting is needed.

• Oracle Enterprise Scheduler is throwing errors.

• A scheduled job ends in error.

For troubleshooting Oracle Enterprise Scheduler, use the standard Oracle WebLogic Server system log. For information about viewing job request logs, see "Managing Logging for Oracle Enterprise Scheduler" in the Oracle Fusion Applications Administrator's Guide. For more information about troubleshooting Oracle Enterprise Scheduler, see the "Troubleshooting Oracle Enterprise Scheduler" chapter in the Oracle Fusion Middleware Administrator's Guide for Oracle Enterprise Scheduler.

Troubleshooting Asynchronous Oracle Business Intelligence Publisher Jobs

When viewed in Fusion Applications Control, a given Oracle Business Intelligence Publisher job includes a direct URL pointing to the Oracle Business Intelligence Publisher server on the Job Details page. Once the Oracle Business Intelligence Publisher job starts running on the Oracle Business Intelligence Publisher server, the scheduled job request attains the property value bip.status_url. This property value holds the URL of the Oracle Business Intelligence Publisher server which is used to diagnose Oracle Business Intelligence Publisher report execution. For more information about viewing job request details, see "Viewing Job Request Details" section in the Oracle Fusion Middleware Administrator's Guide for Oracle Enterprise Scheduler.

Oracle Business Intelligence Publisher jobs are supported only in Oracle Fusion Applications.

5.3 Problems and Solutions

This section describes common problems and solutions for Oracle Enterprise Scheduler. It contains the following topics:

• BI Publisher Job Remains In RUNNING State

• BI Publisher Job States Mismatch with Oracle Enterprise Scheduler States

• Job Never Executes and Goes into ERROR State Immediately

• Job Completes, but Goes into a WARNING state

• Metadata Access Denied Error When Accessing Job Metadata

• Insufficient Privilege Error on Request Error

• Empty Process List in Schedule New Process Dialog

• Slow Display of Job Definition List of Values in Schedule New Process Dialog

In addition to the recommended solutions, consider reviewing the section "Tuning Oracle Enterprise Scheduling Service System Performance" in Oracle Fusion Middleware Administrator's Guide for Oracle Enterprise Scheduler for tuning tips.

5.3.1 BI Publisher Job Remains In RUNNING State

Problem

Oracle Business Intelligence Publisher jobs run on separate computers. In these cases, Oracle Enterprise Scheduler depends on the remote job sending a completion status at end of processing that defines the job outcome. However this message may never get generated due to various reasons. It is required to troubleshoot the native job implementation to resolve.

Solution

To resolve this issue, you must troubleshoot the native job. See Section 5.3.2 for more information.

5.3.2 BI Publisher Job States Mismatch with Oracle Enterprise Scheduler States

Problem

Under normal circumstances, the BI Publisher server state should match the Oracle Enterprise Scheduler job state. However, in case of any BI Publisher job errors, server issues or other network issues, the BI Publisher server state may not match the Oracle Enterprise Scheduler job state.

Solution

It is important to understand that the Oracle Enterprise Scheduler runtime invokes the Oracle BI Publisher web service asynchronously and then needs to wait for it to successfully complete to update the Oracle Enterprise Scheduler job status. Therefore, between the initial web service invocation, there could be various things that could go wrong. The table below lists the typical job life cycle, the potential failure points, and the Oracle Enterprise Scheduler and BI Publisher state values seen at each of those states. Use the table below to debug any state mismatch between the Oracle Enterprise Scheduler job and the BI Publisher server:

Processing Step/Error	Server	Expected ESS Request State	Expected BI Publisher State	Description
1. Oracle Enterprise Scheduler request is in the RUNNING state, calls BI Publisher job	Oracle Enterprise Scheduler	RUNNING	None
2. BI Publisher Oracle Enterprise Scheduler job begins to execute.	Oracle Enterprise Scheduler	RUNNING	None
3. BI Publisher Oracle Enterprise Scheduler job invokes BI Publisher web service.		RUNNING	None
3a. Error calling the web service, for example, the BI Publisherserver is down or crashes while processing the web service.	Oracle Enterprise Scheduler	ERROR_MANUAL_RECOVERY	Could be None or RUNNING, depending on where the crash occurs	An error message is seen in the Oracle Enterprise Scheduler server log and with Oracle Enterprise Scheduler request in Fusion Applications Control. BI Publisher job state depends on why the web service failed. This requires manual recovery.
4. BI Publisher web service creates "BIP Job ID" with RUNNING status, enqueues to JMS, and commits.	BI Publisher	RUNNING	RUNNING
5. BI Publisher Oracle Enterprise Scheduler job finishes after web service processing.	Oracle Enterprise Scheduler	RUNNING	RUNNING
5a. Error in the BI Publisher Oracle Enterprise Scheduler job before returning.		ERROR_MANUAL_RECOVERY	RUNNING	Error message seen in the Oracle Enterprise Scheduler server log and with Oracle Enterprise Scheduler request in Fusion Applications Control. BI Publisher job is running, but Oracle Enterprise Scheduler job that invoked it has failed. This needs manual recovery because the Oracle Enterprise Scheduler job cannot be in a terminal SUCCESS or ERROR state while the BI Publisher job is RUNNING.
5b. Oracle Enterprise Scheduler server crash anywhere during BI Publisher Oracle Enterprise Scheduler job execution (crash during BI Publisher Oracle Enterprise Scheduler job executable call between Steps 2-5).		ERROR_MANUAL_RECOVERY	Could be None or RUNNING, depending on where the crash occurs	Error message seen in the Oracle Enterprise Scheduler server log and with Oracle Enterprise Scheduler request in Fusion Applications Control. It is not known if the web service has been invoked or not, so manual intervention is required.
6. BI Publisher server begins processing the BI Publisher job	BI Publisher	RUNNING	RUNNING
6a. BI Publisher server crashes during processing and restarts		RUNNING to ERROR (See steps 11 through 13)	RUNNING (before restart) to ERROR (after restart)
6b. Job is taking a long time to run		RUNNING	RUNNING	BI Publisher "request job history" UI should show which stage of processing the job is in. When BI Publisher server restarts, it will mark the BI Publisher state as ERROR and then notify Oracle Enterprise Scheduler that the job had an error.
BI Publisher report is successful
7. BI Publisher job completes successfully, BI Publisher sets the BI Publisher job to SUCCESS and commits	BI Publisher	RUNNING	SUCCESS
8. BI Publisher invokes Oracle Enterprise Scheduler web service to notify Oracle Enterprise Schedulerof job completion.	BI Publisher	RUNNING	SUCCESS
8a. Oracle Enterprise Scheduler server is down while BI Publisher makes web service call to notify Oracle Enterprise Scheduler of job completion, other Oracle Enterprise Scheduler servers in the cluster are running.	BI Publisher	See Steps 9 through 10.	SUCCESS	Fusion Applications Control should show that the Oracle Enterprise Scheduler server is down. Service invocation should be routed successfully to another Oracle Enterprise Scheduler server in the cluster.
8b. All Oracle Enterprise Scheduler servers are down while BI Publisher makes web service call to notify Oracle Enterprise Scheduler of job completion.	BI Publisher	RUNNING	SUCCESS	Can detect this case after the Oracle Enterprise Scheduler servers are restarted and the BI Publisher request is SUCCESS while the Oracle Enterprise Scheduler request is RUNNING (or timed out). Job will remain RUNNING and administrator has to cancel and recover request in Fusion Applications Control.
9. Oracle Enterprise Scheduler calls job post-processor.	Oracle Enterprise Scheduler	COMPLETED	SUCCESS
9a. Oracle Enterprise Scheduler job post-processor has an error.	Oracle Enterprise Scheduler	WARNING	SUCCESS	Error message seen in the Oracle Enterprise Scheduler server log and with ESS request in Fusion Applications Control.
10. Oracle Enterprise Scheduler completes the request after post-processing finishes successfully	Oracle Enterprise Scheduler	SUCCESS	SUCCESS
BI Publisher report has error.				Error message seen in the Oracle Enterprise Scheduler server log and with Oracle Enterprise Scheduler request in Fusion Applications Control as well as in BI Publisher console for the job.
11. BI Publisher job completes with error, BI Publisher sets the BI Publisher job to ERROR and commits	BI Publisher	RUNNING	ERROR
12. BI Publisher invokes Oracle Enterprise Scheduler web service to notify Oracle Enterprise Scheduler of job completion	BI Publisher	RUNNING	ERROR
12a. Oracle Enterprise Scheduler server is down while BI Publisher makes web service call to notify Oracle Enterprise Scheduler of job completion, other Oracle Enterprise Scheduler servers in the cluster are running.	BI Publisher	RUNNING	ERROR	See Steps 8a and 8b. BI Publisher retries delivery of the status update in all cases if Oracle Enterprise Scheduler web service invocation fails
13. Oracle Enterprise Scheduler updates request status in the Oracle Database to ERROR	Oracle Enterprise Scheduler	ERROR	ERROR
Request is cancelled.				Note: The timing refers to when the AsyncCancellable method is invoked, not when cancel is initiated. The states in this section assume that if the BI Publisher runs, it runs successfully unless canceled. For more details on Oracle Enterprise Scheduler cancellation, see the "Cancelling Oracle Enterprise Scheduler Job Requests" section in the Oracle Fusion Middleware Administrator's Guide for Oracle Enterprise Scheduler.
14. Request in RUNNING state canceled prior to BI Publisher Oracle Enterprise Scheduler job execution.		CANCELLING to CANCELLED	None	Process phase prior to ExecuteInitiate. Should automatically transition from CANCELLING to CANCELLED.
15. Request canceled during BI Publisher Oracle Enterprise Scheduler job execution before BI Publisher job begins.		CANCELLING to CANCELLED	None	BI Publisher Oracle Enterprise Scheduler job should prevent BI Publisher job from being initiated by having checkpoints in the execute method. On return, request should transition to CANCELLED.
16. Request canceled during BI Publisher Oracle Enterprise Scheduler job execution after BI Publisher job initiated.		CANCELLING to CANCELLED	RUNNING to CANCELLED	BI Publisher Oracle Enterprise Scheduler job execute method can ignore cancel. Cancel() method should attempt to cancel BI Publisher job. BI Publisher job returns status (CANCELLED or other). Request ends up in CANCELLED state.
17. Request canceled while BI Publisher job is running.		CANCELLING to CANCELLED	RUNNING to CANCELLED	Cancel() method should attempt to cancel BI Publisher job. BI Publisher job returns status (CANCELLED or other). Request ends up in CANCELLED state.
18. Request canceled after BI Publisher job, before callback received (when is BI Publisher state set to SUCCESS.)		CANCELLING to CANCELLED	SUCCESS	Cancel() method should attempt to cancel BI Publisher job. BI Publisher job returns status (CANCELLED or other). Request ends up in CANCELLED state.
19. Request canceled after callback received, before post-processor		CANCELLING to CANCELLED	SUCCESS
20. Request canceled while post-processor is running		SUCCESS	SUCCESS	Once post-processing is begun, the cancel operation has no effect.
21. Request canceled after post-processing		SUCCESS	SUCCESS

5.3.3 Job Never Executes and Goes into ERROR State Immediately

Problem

When the user submits a spawn job, it goes into ERROR state immediately.

Solution

To resolve this problem, verify that the RequestFileDirectory directory is created and set up on the server. This directory can be located anywhere. The Oracle WebLogic Server writes and reads from it, so the Oracle Fusion Middleware administration account server can access this directory. If this directory does not exist, all jobs will move to ERROR state. The directory is a shared file system shared across the Oracle Enterprise Scheduler cluster.

To find the RequestFileDirectory value in the Fusion Applications Control:

1. Search the logs to diagnose any specific issues found. See the section "Viewing and Searching Log Files" in the Oracle Fusion Middleware Administrator's Guide.

2. Verify that RequestFileDirectory directory is created:

   1. From the navigation pane, expand the farm, WebLogic Domain, and select the Oracle Enterprise Scheduler server.

   2. From the WebLogic Server menu, choose System MBean Browser.

   3. In the System MBean Browser page, expand Application Defined MBeans.

   4. Expand oracle.adf.share.connections, Server: ess_server_name, Application: ESSAPP, ADFConnections, ADFConnections, EssConnection.

      
      Description of the illustration essmbean.gif
      
      

   5. Click EssConnection1.

   6. In the Application Defined MBeans: EssConnection:EssConnection1 page, view the attribute value for RequestFileDirectory.

5.3.4 Job Completes, but Goes into a WARNING state

Problem

Oracle Enterprise Scheduler tries to upload log/output files to Oracle WebCenter Content. If upload fails, the request will be marked to WARNING state.

Solution

To resolve this problem, use Fusion Applications Control:

1. Check that the Oracle WebCenter Content Content Server is up and running.

   Oracle WebCenter Content is located in the CommonDomain domain in the Oracle Fusion Setup product family.

   1. From the navigation pane, expand the farm, Content Management, and then Content Server.

   2. Select the Oracle Universal Content Management - Content Server application for the appropriate Managed Server.

   3. In the UCM home page, in the Scheduler Components section, ensure the Request Processor has a status of Started.

      If it is not running, start it. See "Starting and Stopping a Request Processor or Dispatcher" in the Oracle Fusion Middleware Administrator's Guide for Oracle Enterprise Scheduler.

   4. In the Scheduling Service home page, in the General section, ensure the Content Server has a state of Active.

   5. If the state is not Active, from the UCM menu, choose Control > Start.

2. Search and view log records for issues related to Oracle WebCenter Content or attachments. See the "Viewing and Searching Log Files" section in the Oracle Fusion Middleware Administrator's Guide.

3. Check the attachments configuration for the job hosting application is correct:

   1. From the navigation pane, expand the farm, Application Deployments.

   2. Expand domain_nameEssApp, and then select domain_nameEssApp.

      The Application Deployment page displays.

   3. From the Application Deployment menu, choose WebCenter > Service Configuration.

      The WebCenter Service Configuration page displays.

   4. Click Content Repository.

   5. In the Manage Content Repository Connection sections, click Edit to view and modify the entry for the FusionAppsContentRepository connection.

5.3.5 Metadata Access Denied Error When Accessing Job Metadata

Problem

When accessing job metadata, users receive a metadata access denied error in the Standard Report Submission, when there is an attempt to submit a job.

Solution

To resolve this problem,

1. Determine the application role that is supposed to have metadata permissions for the job. See the "Mapping External Roles to an Application Role" section in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).

2. From Fusion Applications Control, perform the following steps to verify the permissions:

   1. In the navigation pane, expand the farm and then WebLogic Server Domain.

   2. Select the domain.

   3. From the WebLogic Domain menu, choose Security > Application Policies.

      The Application Policies page displays.

   4. In the Search section, choose the application or application stripe to search, enter the data to match (a principal name or a permission name or both), and click the blue Search application security grants icon. In the results table at the bottom of the page, search the grants for the application role and see if the permissions are granted. If

   5. Add the permissions, as described in the "Managing Application Policies" section of the Oracle Fusion Middleware Application Security Guide.

   6. Determine the enterprise role that is supposed to map to the application role. See the "Mapping External Roles to an Application Role" section in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).

3. From the Oracle WebLogic Server Administration Console, perform the following to steps to determine the enterprise role of the user.

   1. From the left pane, from Domain Structure, select Security Realms.

   2. On the Summary of Security Realms page select the name of the realm.

   3. On the Settings for Real Name page, click the Users and Groups tab to check the user's group.

      If you do not find the user, add the user to the group, as described in Step 4.

4. Add the user to the group:

   1. In the Users table, select the user you want to add to a group.

   2. On the Settings for User Name page select Groups.

   3. Select a group or groups from the Available list box:

      - To locate a group in a large list, type the first few characters of the name.

      - To select multiple groups, Ctrl-click each group.

      - To add a user to a group, click the right arrow to move the selection to the Chosen list box.

      - To remove a user from a group, select the group in the Chosen list box and click the left arrow.

   4. Click Save.

5.3.6 Insufficient Privilege Error on Request Error

Problem

The following error is reported:

User name does not have sufficient privilege to do name operation on request number

Solution

To resolve this problem,

1. Check who submitted the request. A user should be able to operate on the requests submitted. If the login user is not the request submitter, go to Step 2.

2. Use Oracle Authorization Policy Manager to check if the submitter is assigned privileges on the request. See the "Locating Policies Associated with a Database Resource" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition). When using Oracle Authorization Policy Manager, in the Search area of the Manage Database Resources and Policies tab, search for the ESS_REQUEST_HISTORY database resource.

3. Search the policies for the application role that is expected to receive the grants. See "Searching Database Resources" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).

4. If the policy or condition is missing, use Oracle Authorization Policy Manager to add them. See the "Managing Database Resource Conditions" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).

5. Use Oracle Authorization Policy Manager to check if the submitter is assigned privileges on the request. See the "Searching Database Resources" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).

5.3.7 Empty Process List in Schedule New Process Dialog

Problem

When an application user logs into an Oracle Fusion application to schedule a new Oracle Enterprise Scheduler job by using the following procedure and finding no value for the process:

1. Click the Navigator link.

2. Choose Tools > Scheduled Processes.

3. In the Scheduled Processes page, in the Search Results section, click Schedule New Process.

4. In the Schedule New Process dialog, click the arrow button next to the Process Name list and find no values.

This issue is usually the result of a permissions problem.

Solution

To resolve this problem,

1. Check who submitted the request. A user should be able to operate on the requests submitted. If the login user is not the request submitter, go to Step 2.

2. Use Oracle Authorization Policy Manager to check if the submitter is assigned privileges on the request. See the "Locating Policies Associated with a Database Resource" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition). When using Oracle Authorization Policy Manager, in the Search area of the Manage Database Resources and Policies tab, search for the ESS_REQUEST_HISTORY database resource.

3. Search the policies for the application role that is expected to receive the grants. See the "Searching Database Resources" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).

4. If the policy or condition is missing, use Oracle Authorization Policy Manager to add them. See the "Managing Database Resource Conditions" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).

5. Use Oracle Authorization Policy Manager to check if the submitter is assigned privileges on the request. See "Searching Database Resources" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).

5.3.8 Slow Display of Job Definition List of Values in Schedule New Process Dialog

Problem

When an application user logs into an Oracle Fusion application to schedule a new Oracle Enterprise Scheduler job by using the following procedure and finding slow performance in the list of values.

1. Click the Navigator link.

2. Choose Tools > Scheduled Processes.

3. In the Scheduled Processes page, in the Search Results section, click Schedule New Process.

4. In the Schedule New Process dialog, click the arrow button next to the Process Name list and discover the performance is unacceptably slow.

Solution

To resolve this issue, update the MDS table statistics for the query optimizer to use optimal query plans:

1. Generate Oracle Enterprise Scheduler MDS schema statistics:

   BEGIN
   dbms_stats.gather_schema_stats(
   ownname =>'FUSION_MDS',
   METHOD_OPT =>'FOR ALL COLUMNS SIZE AUTO',
   CASCADE =>TRUE,
   ESTIMATE_PERCENT =>NULL);
   END;
   

2. Flush the shared pool:

   SQL> ALTER SYSTEM FLUSH SHARED_POOL;