17
OPMN Troubleshooting

This chapter describes some troubleshooting tips for OPMN. It features the following topics:

17.1 Diagnosing OPMN Problems

There are several methods for troubleshooting any problems you may have using OPMN:

17.1.1 OPMN log Files

The OPMN log files enable you to troubleshoot difficulties you might have in execution and use of OPMN and OPMN-managed processes. OPMN and OPMN-managed processes generate log files during processing. You can review the following generated log files to verify successful or unsuccessful execution of an OPMN command:

ORACLE_HOME/opmn/logs: The standard output and standard error of OPMN managed processes. OPMN creates a log file for each component and assigns a unique concatenation of the Oracle Application Server component with a number. For example, the standard output log for OracleAS Web Cache may be WebCache~WebCacheAdmin~1.The process specific console logs are the first and best resource for investigating problems related to starting and stopping components.
ORACLE_HOME/opmn/logs/ipm.log: Review the error codes and messages that are shown in the ipm.log file. OPMN generates and outputs the error messages in this file. The ipm.log file tracks command execution and operation progress. The level of detail that gets logged in the ipm.log can be modified by configuration in the opmn.xml file. Refer to Chapter 4, "opmn.xml Common Configuration" for examples of debug levels.
ORACLE_HOME/opmn/logs/ons.log: Use the ons.log file to debug the ONS portion of OPMN or for early OPMN errors. The ONS portion of OPMN is initialized before PM, and so errors that occur early in OPMN initialization will show up in the ons.log file.

17.1.2 opmnctl debug

Use the opmnctl debug command to verify the status of an Oracle Application Server process and whether any actions are pending. This command generates output that can be used in conjunction with contact to your local Oracle support to diagnose your OPMN problem.

The syntax for the opmnctl debug command is:

opmnctl [<scope>] debug [comp=pm|ons] [interval=<secs> count=<num>]

where @scope is the optional scope for the request.

Output is generated following execution of the opmnctl debug command. Oracle recommends that you contact Oracle support to use the generated output to assist in diagnosis of your problem.

The attributes (<attr>) name for this command are either comp, interval, or count. The value for comp can be either ons or pm, representing ONS and PM, respectively. If comp is not specified, then both ons and pm debug information is reported. For example, the following command outputs debug information for ONS.

prompt > opmnctl debug comp=ons

You can specify the interval in seconds and number of requests sent to OPMN to assist in the debugging process. The values of <interval> and <count> must always be specified together. Values for them should be integers greater than 0. For example, the following command, outputs debug information at an interval of 5 seconds 3 times.

prompt > opmnctl debug comp=pm interval=5 count=3

Contact your local Oracle support to assist you in using the opmnctl debug command to diagnose your OPMN problem.

17.1.3 iHAT

Use the iHAT tool to provide a real time, graphical interface view of your enterprise. iHAT displays all Oracle Application Server processes managed by one or more OPMN servers including useful performance metrics about each process as obtained from DMS. The snapshot of the system is updated continuously at a configurable interval.

To download iHAT, visit the Oracle Technology Network (OTN):

http://otn.oracle.com/membership

17.1.4 Oracle Enterprise Manager Application Server Control

Application Server Control provides a graphical interface that enables diagnosis of Oracle Application Server components in your network and enterprise. Application Server Control features a log page. The log page enables you to view all of the Oracle Application Server log files in one place and trace problems across multiple log files. Application Server Control uses an API that contact OPMN.

You can use Application Server Control to enable or disable Oracle Application Server components: You can disable components so they do not start when you start an Oracle Application Server instance.

See Also:

Oracle Application Server 10g Administrator's Guide

17.1.5 Troubleshooting with Event Scripts

You can create your own event scripts that record Oracle Application Server process event activities. You can create a script that records events prior to the start or stop of Oracle Application Server processes, as well as an unscheduled system crash.

The following Example 17-1 shows a pre-start event script:

Example 17-1 Pre-start Event Script

#!/bin/sh
echo
echo =---===----======---=-----=-----=------======----===---=
echo =---===----===== PRE-START EVENT SCRIPT =====----===---=
echo =---===----======---=-----=-----=------======----===---=

timeStamp="N/A"
instanceName="N/A"
componentId="N/A"
processType="N/A"
processSet="N/A"
processIndex="N/A"
stderrPath="N/A"  # not available w/pre-start unless part of restart
stdoutPath="N/A"  # not available w/pre-start unless part of restart
reason="N/A"
pid="N/A"         # only available with pre-stop, post-crash
startTime="N/A"   # only available with pre-stop, post-crash

while [ $# -gt 0 ]; do
     case $1 in
        -timeStamp)    timeStamp=$2; shift;;
        -instanceName) instanceName=$2; shift;;
        -componentId)  componentId=$2; shift;;
        -processType)  processType=$2; shift;;
        -processSet)   processSet=$2; shift;;
        -processIndex) processIndex=$2; shift;;
        -stderr)       stderrPath=$2; shift;;
        -stdout)       stdoutPath=$2; shift;;
        -reason)       reason=$2; shift;;
        -pid)          pid=$2; shift;;
        -startTime)    startTime=$2; shift;;
        *) echo "Option Not Recognized: [$1]"; shift;;
        esac
        shift
done

echo timeStamp=$timeStamp
echo instanceName=$instanceName
echo componentId=$componentId
echo processType=$processType
echo processSet=$processSet
echo processIndex=$processIndex
echo stderr=$stderrPath
echo stdout=$stdoutPath
echo reason=$reason
echo pid=$pid
echo startTime=$startTime

17.2 Common Problems

This section describes some of the common problems encountered when using OPMN. It features the following topics:

17.2.1 Oracle Application Server Process Does Not Start

If you are unable to start an Oracle Application Server process using OPMN:

Verify and if necessary, correct, the command input. Confirm the spelling and choice of option for the command you are entering.

Warning:
Do not use command line scripts or utilities from previous versions of Oracle9iAS for starting OPMN or Oracle Application Server components.
Review the standard out output log for the Oracle Application Server process. Output from the process console is located in the ORACLE_HOME/opmn/logs directory. For example, the standard output log for OracleAS Web Cache may be WebCache~WebCacheAdmin~1.
Verify the dependency requirements for the Oracle Application Server process you are attempting to start. For example, the Oracle Application Server process you want to start may require that the Oracle Application Server Metadata Repository be up and running.
Verify the element values for the Oracle Application Server component in the opmn.xml file. Use the opmnctl validate command to verify configuration of opmn.xml file. You may have misconfigured the opmn.xml for the Oracle Application Server component you are attempting to start.

17.2.2 Determining if Oracle Application Server Processes are Dying or Unresponsive

If your Oracle Application Server processes are dying or unreachable:

Look at the ORACLE_HOME/opmn/logs/ipm.log for Oracle Application Server processes at level of 4 or higher. Look for process crashed or process unreachable messages. OPMN automatically restarts Oracle Application Server processes that die or become unresponsive.
Create event scripts for any pre-stop or post-crash events. The event scripts could be used to create a specific log file or send you an email about a failure.

See Also:
Section 17.1.5, "Troubleshooting with Event Scripts"
Review the Oracle Application Server component specific output in the ORACLE_HOME/opmn/logs.
Review the Oracle Application Server component specific log file located in Oracle Application Server component directory.

See Also:
Section 17.1.1, "OPMN log Files"
Use iHAT to view the actions of Oracle Application Server

See Also:
Section 17.1.3, "iHAT"

17.2.3 opmnctl Command Execution Times Out

The time it takes to execute an opmnctl command is dependent on the type of Oracle Application Server process and available computer hardware. Because of this the time it takes to execute an opmnctl command may not be readily apparent.

The default start time out for OC4J is approximately five minutes. If an OC4J process is does not start-up after an opmnctl command, OPMN will wait approximately an hour before timing out and aborting the request.

To verify successful execution of the opmnctl command, try the following:

Increase the start element timeout attribute for the component that is not starting. Set the timeout in the opmn.xml file at a level that will allow OPMN to wait for process to come up. This functionality is available with the startproc command which will start all the relevant processes configured in opmn.xml.
Check the start element in the opmn.xml file and change the retry attribute to a higher increment of time.
Look at the ORACLE_HOME/opmn/logs/ for the Oracle Application Server process that is not starting.
Review the component-specific log file for the Oracle Application Server component that is not starting. For example, ORACLE_HOME/discoverer/logs.
Examine the ORACLE_HOME/opmn/logs/ipm.log for any indication of problems. Increase the log level in the file to obtain additional information.

See Also:
Chapter 4, "opmn.xml Common Configuration"
Contact Oracle Technical Support.

If you are having difficulty with an Oracle Application Server instance that is part of a farm, review the ons.log file for the Oracle Application Server instance. The "attempting active connection init" message in the ons.log file indicates that there is another OPMN configured in the farm that is currently shut-down. OPMN tries to consistently connect to the shut-down OPMN. Use the listed steps to determine why the down OPMN is not running.

17.2.4 Oracle Application Server Component Automatically Restarted by OPMN

If an Oracle Application Server component is automatically restarted by OPMN, try the following:

Review the message for the Oracle Application Server component in the ipm.log file.
Verify that the ping timeout for the Oracle Application Server component is sufficient. An Oracle Application Server component that receives a lot of activity may require an increase in the length of time for the timeout. Increase the ping timeout element in the Oracle Application Server component opmn.xml file.

17.2.5 opmn.xml Environment Variables

The environment variable used to launch OPMN server is not inherited by the Oracle Application Server process started by OPMN server. OPMN sets the environment variables at the ias-instance level, with the values extracted either from the ias-instance configuration or from the OPMN run time environment.

17.2.6 Unexpected opmnctl start Behavior

Occasionally, there is unexpected behavior when you use the opmnctl start command to start OPMN; either only OPMN is started or OPMN makes a best effort to start Oracle Application Server OPMN-managed processes. Typically, this unexpected behavior is due to turning-off or rebooting your computer without first shutting down OPMN. When you restart your computer, all OPMN-managed processes are started.

Oracle recommends that you shutdown OPMN before shutting down your computer. Use the opmnctl stopall command to stop OPMN and OPMN-managed processes.

17.2.7 Disabled Element in the opmn.xml File

If you are unable to start an Oracle Application Server process, check if an element in the Oracle Application Server opmn.xml file is disabled. If an element in the opmn.xml file is disabled OPMN will generate an output message of "Missing" or "Disabled".

17.2.8 Management of Oracle9iAS Release 2 (9.0.2 and 9.0.3) Instances

By default, the SSL element in the 10g (9.0.4) opmn.xml file is enabled; however, the SSL element is not enabled in the opmn.xml file for Oracle9iAS Release 2 (9.0.2 and 9.0.3).

If you have an Oracle Application Server farm containing Oracle9iAS Release 2 (9.0.2 and 9.0.3) instances you must enable the SSL element in the Release 2 opmn.xml file.

17 OPMN Troubleshooting