Oracle® Fusion Applications Administrator's Troubleshooting Guide 11g Release 6 (11.1.6) Part Number E25450-05 |
|
|
PDF · Mobi · ePub |
This chapter describes common problems that you might encounter when using Oracle Enterprise Scheduler and explains how to solve them.
This chapter contains the following topics:
Section 6.1, "Introduction to Troubleshooting Oracle Enterprise Scheduler"
Section 6.2, "Getting Started with Troubleshooting Oracle Enterprise Scheduler Jobs"
To gain insight into the log details generated by Oracle Fusion applications, see the "Troubleshooting Oracle Fusion Applications Using Incidents, Logs, QuickTrace, and Diagnostic Tests" chapter in the Oracle Fusion Applications Administrator's Guide. Also, review the Oracle Fusion Middleware Error Messages Reference for information about the error messages you may encounter.
This section provides guidelines and a process for using the information in this chapter. Using the following guidelines and process will focus and minimize the time you spend resolving problems.
Guidelines
When using the information in this chapter, Oracle recommends:
After performing any of the solution procedures in this chapter, immediately retrying the failed task that led you to this troubleshooting information. If the task still fails when you retry it, perform a different solution procedure in this chapter and then try the failed task again. Repeat this process until you resolve the problem.
Making notes about the solution procedures you perform, symptoms you see, and data you collect while troubleshooting. If you cannot resolve the problem using the information in this chapter and you must log a service request, the notes you make will expedite the process of solving the problem.
Process
Follow the process outlined in Table 6-1 when using the information in this chapter. If the information in a particular section does not resolve your problem, proceed to the next step in this process.
Table 6-1 Process for Using the Information in this Chapter
Step | Section to Use | Purpose |
---|---|---|
1 |
Get started troubleshooting Oracle Enterprise Scheduler jobs. |
|
2 |
Perform problem-specific troubleshooting procedures. This section describes:
|
|
3 |
Use My Oracle Support to get additional troubleshooting information about Oracle Fusion Applications or Oracle Enterprise Scheduler. My Oracle Support provides access to several useful troubleshooting resources, including Knowledge Base articles and Community Forums and Discussions. |
|
4 |
Log a service request if the information in this chapter and My Oracle Support does not resolve your problem. You can log a service request using My Oracle Support at |
You may want to troubleshoot the following typical issues that can arise when running Oracle Enterprise Scheduler jobs.
Asynchronous jobs remain in running state indefinitely.
An asynchronous job hangs or crashes.
Oracle Enterprise Scheduler is down when the remote scheduled job completes or there are network problems such that Oracle Enterprise Scheduler does not receive the completion status from the remote job.
A scheduled job is ready to execute, but does not execute.
A scheduled job is placed in manual error recovery state where troubleshooting is needed.
Oracle Enterprise Scheduler is throwing errors.
A scheduled job ends in error.
For troubleshooting Oracle Enterprise Scheduler, use the standard Oracle WebLogic Server system log. For information about viewing job request logs, see "Managing Logging for Oracle Enterprise Scheduler" in the Oracle Fusion Applications Administrator's Guide. For more information about troubleshooting Oracle Enterprise Scheduler, see the "Troubleshooting Oracle Enterprise Scheduler" chapter in the Oracle Fusion Middleware Administrator's Guide for Oracle Enterprise Scheduler.
Troubleshooting Asynchronous Oracle Business Intelligence Publisher Jobs
When viewed in Fusion Applications Control, a given Oracle Business Intelligence Publisher job includes a direct URL pointing to the Oracle Business Intelligence Publisher server on the Job Details page. Once the Oracle Business Intelligence Publisher job starts running on the Oracle Business Intelligence Publisher server, the scheduled job request attains the property value bip.status_url
. This property value holds the URL of the Oracle Business Intelligence Publisher server which is used to diagnose Oracle Business Intelligence Publisher report execution. For more information about viewing job request details, see "Viewing Job Request Details" section in the Oracle Fusion Middleware Administrator's Guide for Oracle Enterprise Scheduler.
Oracle Business Intelligence Publisher jobs are supported only in Oracle Fusion Applications.
This section describes common problems and solutions for Oracle Enterprise Scheduler. It contains the following topics:
Section 6.3.1, "Oracle BI Publisher Job Remains In RUNNING State"
Section 6.3.2, "Oracle BI Publisher Job States Mismatch with Oracle Enterprise Scheduler States"
Section 6.3.3, "Job Never Executes and Goes into ERROR State Immediately"
Section 6.3.4, "Job Completes, but Goes into a WARNING state"
Section 6.3.5, "Metadata Access Denied Error When Accessing Job Metadata"
Section 6.3.6, "Insufficient Privilege Error on Request Error"
Section 6.3.7, "Empty Process List in Schedule New Process Dialog"
Section 6.3.8, "Slow Display of Job Definition List of Values in Schedule New Process Dialog"
In addition to the recommended solutions, consider reviewing the section "Tuning Oracle Enterprise Scheduling Service System Performance" in Oracle Fusion Middleware Administrator's Guide for Oracle Enterprise Scheduler for tuning tips.
Problem
Oracle Business Intelligence Publisher jobs run on separate computers. In these cases, Oracle Enterprise Scheduler depends on the remote job sending a completion status at end of processing that defines the job outcome. However this message may never get generated due to various reasons. It is required to troubleshoot the native job implementation to resolve.
Solution
To resolve this issue, you must troubleshoot the native job. See Section 6.3.2 for more information.
Problem
Under normal circumstances, the Oracle BI Publisher server state should match the Oracle Enterprise Scheduler job state. However, in case of any BI Publisher job errors, server issues or other network issues, the Oracle BI Publisher server state may not match the Oracle Enterprise Scheduler job state.
Solution
It is important to understand that the Oracle Enterprise Scheduler runtime invokes the Oracle BI Publisher web service asynchronously and then needs to wait for it to successfully complete to update the Oracle Enterprise Scheduler job status. Therefore, between the initial web service invocation, there could be various things that could go wrong. The table below lists the typical job life cycle, the potential failure points, and the Oracle Enterprise Scheduler and Oracle BI Publisher state values seen at each of those states. Use the table below to debug any state mismatch between the Oracle Enterprise Scheduler job and the Oracle BI Publisher server:
Processing Step/Error | Server | Expected ESS Request State | Expected Oracle BI Publisher State | Description |
---|---|---|---|---|
1. Oracle Enterprise Scheduler request is in the |
Oracle Enterprise Scheduler |
|
None |
|
2. Oracle BI Publisher Oracle Enterprise Scheduler job begins to execute. |
Oracle Enterprise Scheduler |
|
None |
|
3. Oracle BI Publisher Oracle Enterprise Scheduler job invokes Oracle BI Publisher web service. |
|
None |
||
3a. Error calling the web service, for example, the Oracle BI Publisherserver is down or crashes while processing the web service. |
Oracle Enterprise Scheduler |
|
Could be |
An error message is seen in the Oracle Enterprise Scheduler server log and with Oracle Enterprise Scheduler request in Fusion Applications Control. Oracle BI Publisher job state depends on why the web service failed. This requires manual recovery. |
4. Oracle BI Publisher web service creates |
Oracle BI Publisher |
|
|
|
5. Oracle BI Publisher Oracle Enterprise Scheduler job finishes after web service processing. |
Oracle Enterprise Scheduler |
|
|
|
5a. Error in the Oracle BI Publisher Oracle Enterprise Scheduler job before returning. |
|
|
Error message seen in the Oracle Enterprise Scheduler server log and with Oracle Enterprise Scheduler request in Fusion Applications Control. Oracle BI Publisher job is running, but Oracle Enterprise Scheduler job that invoked it has failed. This needs manual recovery because the Oracle Enterprise Scheduler job cannot be in a terminal |
|
5b. Oracle Enterprise Scheduler server crash anywhere during Oracle BI Publisher Oracle Enterprise Scheduler job execution (crash during Oracle BI Publisher Oracle Enterprise Scheduler job executable call between Steps 2-5). |
|
Could be |
Error message seen in the Oracle Enterprise Scheduler server log and with Oracle Enterprise Scheduler request in Fusion Applications Control. It is not known if the web service has been invoked or not, so manual intervention is required. |
|
6. Oracle BI Publisher server begins processing the Oracle BI Publisher job |
Oracle BI Publisher |
|
|
|
6a. Oracle BI Publisher server crashes during processing and restarts |
|
|
||
6b. Job is taking a long time to run |
|
|
Oracle BI Publisher "request job history" UI should show which stage of processing the job is in. When Oracle BI Publisher server restarts, it will mark the Oracle BI Publisher state as |
|
Oracle BI Publisher report is successful |
||||
7. Oracle BI Publisher job completes successfully, Oracle BI Publisher sets the Oracle BI Publisher job to |
Oracle BI Publisher |
|
|
|
8. Oracle BI Publisher invokes Oracle Enterprise Scheduler web service to notify Oracle Enterprise Schedulerof job completion. |
Oracle BI Publisher |
|
|
|
8a. Oracle Enterprise Scheduler server is down while Oracle BI Publisher makes web service call to notify Oracle Enterprise Scheduler of job completion, other Oracle Enterprise Scheduler servers in the cluster are running. |
Oracle BI Publisher |
See Steps 9 through 10. |
|
Fusion Applications Control should show that the Oracle Enterprise Scheduler server is down. Service invocation should be routed successfully to another Oracle Enterprise Scheduler server in the cluster. |
8b. All Oracle Enterprise Scheduler servers are down while Oracle BI Publisher makes web service call to notify Oracle Enterprise Scheduler of job completion. |
Oracle BI Publisher |
|
|
Can detect this case after the Oracle Enterprise Scheduler servers are restarted and the Oracle BI Publisher request is |
9. Oracle Enterprise Scheduler calls job post-processor. |
Oracle Enterprise Scheduler |
|
|
|
9a. Oracle Enterprise Scheduler job post-processor has an error. |
Oracle Enterprise Scheduler |
|
|
Error message seen in the Oracle Enterprise Scheduler server log and with ESS request in Fusion Applications Control. |
10. Oracle Enterprise Scheduler completes the request after post-processing finishes successfully |
Oracle Enterprise Scheduler |
|
|
|
Oracle BI Publisher report has error. |
Error message seen in the Oracle Enterprise Scheduler server log and with Oracle Enterprise Scheduler request in Fusion Applications Control as well as in Oracle BI Publisher console for the job. |
|||
11. Oracle BI Publisher job completes with error, Oracle BI Publisher sets the Oracle BI Publisher job to |
Oracle BI Publisher |
|
|
|
12. Oracle BI Publisher invokes Oracle Enterprise Scheduler web service to notify Oracle Enterprise Scheduler of job completion |
Oracle BI Publisher |
|
|
|
12a. Oracle Enterprise Scheduler server is down while Oracle BI Publisher makes web service call to notify Oracle Enterprise Scheduler of job completion, other Oracle Enterprise Scheduler servers in the cluster are running. |
Oracle BI Publisher |
|
|
See Steps 8a and 8b. Oracle BI Publisher retries delivery of the status update in all cases if Oracle Enterprise Scheduler web service invocation fails |
13. Oracle Enterprise Scheduler updates request status in the Oracle Database to |
Oracle Enterprise Scheduler |
|
|
|
Request is cancelled. |
Note: The timing refers to when the |
|||
14. Request in |
|
None |
Process phase prior to |
|
15. Request canceled during Oracle BI Publisher Oracle Enterprise Scheduler job execution before Oracle BI Publisher job begins. |
|
None |
Oracle BI Publisher Oracle Enterprise Scheduler job should prevent Oracle BI Publisher job from being initiated by having checkpoints in the execute method. On return, request should transition to |
|
16. Request canceled during Oracle BI Publisher Oracle Enterprise Scheduler job execution after Oracle BI Publisher job initiated. |
|
|
Oracle BI Publisher Oracle Enterprise Scheduler job execute method can ignore cancel. |
|
17. Request canceled while Oracle BI Publisher job is running. |
|
|
|
|
18. Request canceled after Oracle BI Publisher job, before callback received (when is Oracle BI Publisher state set to |
|
|
|
|
19. Request canceled after callback received, before post-processor |
|
|
||
20. Request canceled while post-processor is running |
|
|
Once post-processing is begun, the cancel operation has no effect. |
|
21. Request canceled after post-processing |
|
|
Problem
When the user submits a spawn job, it goes into ERROR
state immediately.
Solution
To resolve this problem, verify that the RequestFileDirectory
directory is created and set up on the server. This directory can be located anywhere. The Oracle WebLogic Server writes and reads from it, so the Oracle Fusion Middleware administration account server can access this directory. If this directory does not exist, all jobs will move to ERROR
state. The directory is a shared file system shared across the Oracle Enterprise Scheduler cluster.
To find the RequestFileDirectory
value in the Fusion Applications Control:
Search the logs to diagnose any specific issues found. See the section "Viewing and Searching Log Files" in the Oracle Fusion Middleware Administrator's Guide.
Verify that RequestFileDirectory
directory is created:
From the navigation pane, expand the farm, WebLogic Domain, and select the Oracle Enterprise Scheduler server.
From the WebLogic Server menu, choose System MBean Browser.
In the System MBean Browser page, expand Application Defined MBeans.
Expand oracle.adf.share.connections, Server: ess_server_name, Application: ESSAPP, ADFConnections, ADFConnections, EssConnection.
Click EssConnection1.
In the Application Defined MBeans: EssConnection:EssConnection1 page, view the attribute value for RequestFileDirectory.
Problem
Oracle Enterprise Scheduler tries to upload log/output files to Oracle WebCenter Content. If upload fails, the request will be marked to WARNING
state.
Solution
To resolve this problem, use Fusion Applications Control:
Check that the Oracle WebCenter Content Content Server is up and running.
Oracle WebCenter Content is located in the CommonDomain
domain in the Oracle Fusion Setup product family.
From the navigation pane, expand the farm, Content Management, and then Content Server.
Select the Oracle Universal Content Management - Content Server application for the appropriate Managed Server.
In the home page, in the Scheduler Components section, ensure the Request Processor has a status of Started.
If it is not running, start it. See "Starting and Stopping a Request Processor or Dispatcher" in the Oracle Fusion Middleware Administrator's Guide for Oracle Enterprise Scheduler.
In the Scheduling Service home page, in the General section, ensure the Content Server has a state of Active.
If the state is not Active, from the UCM menu, choose Control > Start.
Search and view log records for issues related to Oracle WebCenter Content or attachments. See the "Viewing and Searching Log Files" section in the Oracle Fusion Middleware Administrator's Guide.
Check the attachments configuration for the job hosting application is correct:
From the navigation pane, expand the farm, Application Deployments.
Expand domain_nameEssApp, and then select domain_nameEssApp.
The Application Deployment page displays.
From the Application Deployment menu, choose WebCenter > Service Configuration.
The WebCenter Service Configuration page displays.
Click Content Repository.
In the Manage Content Repository Connection sections, click Edit to view and modify the entry for the FusionAppsContentRepository connection.
Problem
When accessing job metadata, users receive a metadata access denied error in the Standard Report Submission, when there is an attempt to submit a job.
Solution
To resolve this problem,
Determine the application role that is supposed to have metadata permissions for the job. See the "Mapping External Roles to an Application Role" section in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).
From Fusion Applications Control, perform the following steps to verify the permissions:
In the navigation pane, expand the farm and then WebLogic Server Domain.
Select the domain.
From the WebLogic Domain menu, choose Security > Application Policies.
The Application Policies page displays.
In the Search section, choose the application or application stripe to search, enter the data to match (a principal name or a permission name or both), and click the blue Search application security grants icon. In the results table at the bottom of the page, search the grants for the application role and see if the permissions are granted. If
Add the permissions, as described in the "Managing Application Policies" section of the Oracle Fusion Middleware Application Security Guide.
Determine the enterprise role that is supposed to map to the application role. See the "Mapping External Roles to an Application Role" section in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).
From the Oracle WebLogic Server Administration Console, perform the following to steps to determine the enterprise role of the user.
From the left pane, from Domain Structure, select Security Realms.
On the Summary of Security Realms page select the name of the realm.
On the Settings for Real Name page, click the Users and Groups tab to check the user's group.
If you do not find the user, add the user to the group, as described in Step 4.
Add the user to the group:
In the Users table, select the user you want to add to a group.
On the Settings for User Name page select Groups.
Select a group or groups from the Available list box:
- To locate a group in a large list, type the first few characters of the name.
- To select multiple groups, Ctrl-click each group.
- To add a user to a group, click the right arrow to move the selection to the Chosen list box.
- To remove a user from a group, select the group in the Chosen list box and click the left arrow.
Click Save.
Problem
The following error is reported:
User name does not have sufficient privilege to do name operation on request number
Solution
To resolve this problem,
Check who submitted the request. A user should be able to operate on the requests submitted. If the login user is not the request submitter, go to Step 2.
Use Oracle Authorization Policy Manager to check if the submitter is assigned privileges on the request. See the "Locating Policies Associated with a Database Resource" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition). When using Oracle Authorization Policy Manager, in the Search area of the Manage Database Resources and Policies tab, search for the ESS_REQUEST_HISTORY
database resource.
Search the policies for the application role that is expected to receive the grants. See "Searching Database Resources" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).
If the policy or condition is missing, use Oracle Authorization Policy Manager to add them. See the "Managing Database Resource Conditions" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).
Use Oracle Authorization Policy Manager to check if the submitter is assigned privileges on the request. See the "Searching Database Resources" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).
Problem
When an application user logs into an Oracle Fusion application to schedule a new Oracle Enterprise Scheduler job by using the following procedure and finding no value for the process:
Click the Navigator link.
Choose Tools > Scheduled Processes.
In the Scheduled Processes page, in the Search Results section, click Schedule New Process.
In the Schedule New Process dialog, click the arrow button next to the Process Name list and find no values.
This issue is usually the result of a permissions problem.
Solution
To resolve this problem,
Check who submitted the request. A user should be able to operate on the requests submitted. If the login user is not the request submitter, go to Step 2.
Use Oracle Authorization Policy Manager to check if the submitter is assigned privileges on the request. See the "Locating Policies Associated with a Database Resource" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition). When using Oracle Authorization Policy Manager, in the Search area of the Manage Database Resources and Policies tab, search for the ESS_REQUEST_HISTORY
database resource.
Search the policies for the application role that is expected to receive the grants. See the "Searching Database Resources" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).
If the policy or condition is missing, use Oracle Authorization Policy Manager to add them. See the "Managing Database Resource Conditions" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).
Use Oracle Authorization Policy Manager to check if the submitter is assigned privileges on the request. See "Searching Database Resources" chapter in the Oracle Fusion Middleware Oracle Authorization Policy Manager Administrator's Guide (Oracle Fusion Applications Edition).
Problem
When an application user logs into an Oracle Fusion application to schedule a new Oracle Enterprise Scheduler job by using the following procedure and finding slow performance in the list of values.
Click the Navigator link.
Choose Tools > Scheduled Processes.
In the Scheduled Processes page, in the Search Results section, click Schedule New Process.
In the Schedule New Process dialog, click the arrow button next to the Process Name list and discover the performance is unacceptably slow.
Solution
To resolve this issue, update the MDS table statistics for the query optimizer to use optimal query plans:
Generate Oracle Enterprise Scheduler MDS schema statistics:
BEGIN dbms_stats.gather_schema_stats( ownname =>'FUSION_MDS', METHOD_OPT =>'FOR ALL COLUMNS SIZE AUTO', CASCADE =>TRUE, ESTIMATE_PERCENT =>NULL); END;
Flush the shared pool:
SQL> ALTER SYSTEM FLUSH SHARED_POOL;