|
Distributed Configuration Management Reference Guide
10g (9.0.4) Part No. B12052-01 |
|
|
Distributed Configuration Management Reference Guide
10g (9.0.4) Part No. B12052-01 |
|
This chapter contains information on troubleshooting problems you may encounter when using DCM. Usually, the problems are reported by the dcmctl utility.
This chapter contains the following major sections:
This section presents a step-by-step approach for troubleshooting problems with DCM. It contains the following steps:
When an error occurs, dcmctl usually provides a detailed error message and contextual information on a possible cause and solution. If an exception occurs, the -d option provides a stack trace. Sometimes this information may allow you to resolve the problem. The stack trace must be supplied if you contact Oracle support. By default, the dcmctl utility sets the -v (verbose) option and the -d (debug) option on. Change these defaults with the set command.
To view the error message and stack trace that most recently occurred, use this command:
ORACLE_HOME/dcm/bin/dcmctl getError
If the information provided by the dcmctl verbose and debug output is not sufficient, try examining the log files. Often, they will contain error messages that will help you determine the problem. Table 3-1 describes the log files related to DCM.
Table 3-1 Distributed Configuration Management Log Files
| File | Description |
|---|---|
ORACLE_HOME/dcm/logs/dcmctl_logs/log.xml
|
DCM client log for dcmctl shell |
ORACLE_HOME/dcm/logs/emd_logs/log.xml
|
DCM client log for Application Server Control |
ORACLE_HOME/dcm/logs/daemon_logs/log.xml
|
DCM daemon log |
ORACLE_HOME/dcm/logs/busrule_logs/log.xml
|
DCM business rule execution log |
ORACLE_HOME/opmn/logs/*
|
Oracle Process Manager and Notification Server logs |
If you have determined the error but are not sure how to proceed, try searching for the problem, or one similar, in "DCM Problems and Solutions".
Many problems reported by DCM have root causes in the operation of a related component, such as OPMN, Oracle HTTP Server, Oracle Application Server Containers for J2EE, Oracle Internet Directory, and the metadata repository. Table 3-2 provides tips on recognizing these types of problems and pointers to more information.
Table 3-2 Problems with Related Technologies
| Type of Problem | May be due to: |
|---|---|
| Starting and stopping | Oracle Process Manager and Notification Server, Oracle HTTP Server, or Oracle Application Server Containers for J2EE problems
See Also: Oracle Process Manager and Notification Server Administrator's Guide See Also: Oracle Application Server Containers for J2EE User's Guide |
| Application deployment or clustering | Oracle Application Server Containers for J2EE problems
See Also: Oracle Application Server Containers for J2EE User's Guide |
| Directory or password problems | Oracle Internet Directory problems
See Also: Troubleshooting Oracle Internet Directory See Also: Oracle Internet Directory Administrator's Guide, Appendix I, Troubleshooting |
| Repository problems (metadata repository) | Metadata Repository problems
See Also: Troubleshooting the Metadata Repository |
| Repository problems (File based repository) | File system problems |
If none of the preceding steps led to a solution, report the problem to Oracle Support. Have the following information ready:
Steps to duplicate the problem, or a summary of the conditions and settings in effect when the problem occurred
The verbose and debug output from the dcmctl command
Log file entries
This section describes problems reported by Distributed Configuration Management and suggested techniques for resolving these problems.
It contains the following topics:
OPMN is Unable to Start Due to Corrupted or Missing opmn.xml File
Handling Unexpected Messages or Failures With a File Based Repository
DCM Repository Unavailable With Start Stop and Restart Commands
The getState Command Cannot Be Used With Older Instances (9.0.2 or 9.0.3)
Registered Plugin Named Discoverer Could Not Be Loaded Found in log.xml
During a configuration operation, the opmn.xml file was corrupted or removed. This error may be accompanied by one of the following error messages:
ADMN-202001 An exception has occurred while accessing DCM repository. Please refer to the accompanying base exception for more details. Base Exception: oracle.ons.ONSException:Unable to open file /private/10g/opmn/conf/opmn.xml
or
ADMN-705002 The most likely causes are: 1. Configuration error; TaskMaster cannot read configuration file; 2. Cannot connect to the repository. Base Exception: oracle.ons.SubscriptionException:Subscription request timed out after 120000 millseconds. Possible causes: OPMN is not running, you may have OPMN running in an alternate ORACLE_HOME using duplicate port values, OPMN may be misconfigured.
Restart OPMN using the last saved version of the opmn.xml configuration file and then perform a dcmctl resyncinstance command to update the configuration files with the version last saved to the repository.
To do this, perform the following steps:
Perform a opmnctl validate operation on the saved opmn.xml.bak file.
% cd $ORACLE_HOME/opmn/conf % $ORACLE_HOME/opmn/bin/opmnctl validate opmn.xml.bak
If step 1 returns the following, then continue to the next step:
opmnctl: opmn validation succeeded!
If step one returns with the message:
opmnctl: opmn validation failed!
Then, if you know how to correct the error in opmn.xml.bak then correct it and try step 1 again. Otherwise, try Solution 2.
Replace the opmn.xml file with the file that was saved in the last DCM operation. The saved version of opmn.xml should be in the file $ORACLE_HOME/opmn/conf/opmn.xml.bak.
For example:
% cd $ORACLE_HOME/opmn/conf % cp opmn.xml.bak opmn.xml
Start the DCM shell and run the resyncinstance command.
% cd $ORACLE_HOME/dcm/bin % dcmctl shell dcmctl> resyncinstance -force All OC4J components are stopped during a resyncinstance -force. They will need to be restarted after the command completes. Current State for Instance:10gM21.tvanraal-sun.us.oracle.com
Use opmnctl to check the status of all components, and if any components that you expect to be running are not running, then start them:
% cd $ORACLE_HOME/opmn/bin % opmnctl status
Modify the last saved version of the opmn.xml file to allow DCM to run, and then perform a dcmctl resyncinstance command to update the configuration files with the version last saved to the repository.
To do this, perform the following steps:
Perform a opmnctl validate operation on the saved opmn.xml.bak file.
% cd $ORACLE_HOME/opmn/conf % $ORACLE_HOME/opmn/bin/opmnctl validate opmn.xml.bak
If step one returns the following:
opmnctl: opmn validation failed!
Then, you need to correct the error in opmn.xml.bak to allow you to start the dcm daemon. First, backup your current version of opmn.xml.bak, in case you have problems, so you can revert to your backup copy. For example, use the following command:
cp opmn.xml.bak opmn.xml.bak.orig
Edit the $ORACLE_HOME/opmn/conf/opmn.xml.bak file and add the attribute status="disabled" for each <ias-component> element in opmn.xml.bak. For only the component dcm-daemon, set the status= "enabled".
For example:
<ias-component id="OC4J" status="disabled"> <ias-component id="dcm-daemon" status="enabled" id-matching="true">
Replace the opmn.xml file with the updated version of opmn.xml.bak.
For example:
% cd $ORACLE_HOME/opmn/conf % cp opmn.xml.bak opmn.xml
Start the DCM shell and run the resyncinstance command.
% cd $ORACLE_HOME/dcm/bin % dcmctl shell dcmctl> resyncinstance -force All OC4J components are stopped during a resyncinstance -force. They will need to be restarted after the command completes. Current State for Instance:10gM21.tvanraal-sun.us.oracle.com
Use opmnctl to check the status of all components; if any components that you expect to be running are not running, start them:
% cd $ORACLE_HOME/opmn/bin % opmnctl status
If Solution 1 and Solution 2 do not solve this problem, then both the opmn.xml and the opmn.xml.bak files have been corrupted. In this case, you need to manually create an opmn.xml file that allows you to start the dcm-daemon and then perform a dcmctl resyncinstance command to update the configuration files with the version last saved to the repository.
To create a version of opmn.xml that is sufficient for starting the dcm-daemon, perform the following steps:
Copy and paste the text shown in Example 3-1 to $ORACLE_HOME/opmn/conf/opmn.xml.
Edit the port element such that the attributes local, remote, and request have the correct values for your environment. These values are shown as XXXX, YYYY, and ZZZZ in Example 3-1.
Using this file, the OPMN Server should start up, which should allow the DCM Daemon to start so that you can restore the opmn.xml file from the DCM repository using the dcmctl resyncInstance command.
Example 3-1 Basic opmn.xml File For Emergency Startup - Replace Port Values Before Using
<opmn xmlns="http://www.oracle.com/ias-instance">
<notification-server>
<port local="XXXX" remote="YYYY" request="ZZZZ"/>
<log-file path="$ORACLE_HOME/opmn/logs/ons.log" level="4"
rotation-size="1500000"/>
<ssl enabled="true" wallet-file="$ORACLE_HOME/opmn/conf/ssl.wlt/default"/>
</notification-server>
<process-manager>
<log-file path="$ORACLE_HOME/opmn/logs/ipm.log" level="4"
rotation-size="1500000"/>
<process-modules>
<module path="$ORACLE_HOME/opmn/lib/libopmndcmdaemon">
<module-id id="DCMDaemon"/>
</module>
</process-modules>
<ias-instance id="foo">
<environment>
<variable id="TMP" value="/tmp"/>
</environment>
<ias-component id="dcm-daemon" status="enabled" id-matching="true">
<process-type id="dcm-daemon" module-id="DCMDaemon">
<process-set id="dcm" numprocs="1">
<module-data>
<category id="start-parameters">
<data id="java-parameters"
value="-Xmx256m -Doracle.ias.sysmgmt.logging.loglevel=ERROR -Djava.net.preferIPv4Stack=true -Djava.io.tmpdir=$TMP" />
<data id="jar-file" value="$ORACLE_HOME/dcm/lib/dcm.jar"/>
<data id="application-parameters" value="daemon -logdir $ORACLE_HOME/dcm/logs/daemon_logs
-o $ORACLE_HOME" /> </category>
</module-data>
</process-set>
</process-type>
</ias-component>
</ias-instance>
</process-manager>
</opmn>
The DCM log file contains the entry "BOM Not Updated". Usually, this error is accompanied by a detailed message resembling the following:
Error writing file path to file for plugin plugin name.
This message is logged when configuration files for the Oracle Application Server instance are not synchronized with the configuration information stored in the DCM repository. DCM was not able to update the configuration files due to a file system full condition, file permissions problem, or an operating system problem. Examine the following log files to find the reasons for which DCM could not write to the file system:
ORACLE_HOME/dcm/logs/dcmctl_logs/log.xml
ORACLE_HOME/dcm/logs/emd_logs/log.xml
ORACLE_HOME/j2ee/home/log/log.xml
Another way to pinpoint a problem writing to the file system is to log in to the operating system as the user that installed Oracle Application Server, and attempt to write to a configuration file. The operating system will return an error message that describes the problem precisely.
After you resolve the file system problem, you can resynchronize the instance with this command:
ORACLE_HOME/dcm/bin/dcmctl resyncinstance
dcmctl getState displays an In Sync status of "False: Configuration" for one or more components.
This indicates that the configuration files for the Oracle Application Server instance are out of sync with the configuration information stored in the DCM repository. This can happen if a previous operation was not able to complete properly. To resolve this problem, you need to restore the configuration information from the DCM repository to the instance as follows:
Stop Oracle Enterprise Manager Application Server Control to ensure that no attempts are made to change the configuration while you are synchronizing it.
Resynchronize the instance with this command:
ORACLE_HOME/dcm/bin/dcmctl resyncinstance
This will usually fix the problem. If not, the resyncinstance operation will return error information that will help you determine what is wrong and how to proceed.
dcmctl updateconfig returns the message "The configuration files for this Oracle Application Server instance are inconsistent with the configuration stored in the repository."
This indicates that the configuration files for the Oracle Application Server instance are out of sync with the configuration information stored in the DCM repository. Stop and resynchronize the instance. After the configuration files for the Oracle Application Server instance are in sync, synchronized, you can make your original modifications. Be sure to run dcmctl updateconfig again after making the changes.
|
See Also: Problem 1 and Solution 1 in this section |
A DCM configuration file has zero bytes after an operation.
A file size of zero bytes indicates there was a problem writing to the configuration file during a DCM operation. To resolve the problem:
Log on to the operating system as the user that installed the Oracle Application Server instance, and:
Check the file permissions.
Attempt to write to the configuration file, noting the operating system error message after any failure. Resolve the condition that caused the failure.
Verify file system quotas.
Ensure that there are no problems with the operating system that are preventing writing to the configuration file.
Resolve any problems found.
Restore the file from the DCM repository with the resyncInstance command.
dcmctl returns ADMN-100999 and the base exception is "Unable to connect to Directory."
This error occurs when DCM cannot access Oracle Internet Directory. DCM needs to connect to the directory in order to retrieve the passwords it uses to connect to the database repository.
See if restarting Oracle Internet Directory solves the problem. If it does not, refer to "Troubleshooting Oracle Internet Directory".
dcmctl returns ADMN-202026 and the base exception is "Unable to connect to Directory."
This error occurs when DCM cannot access Oracle Internet Directory. DCM needs to connect to the directory in order to retrieve the passwords it uses to connect to the database repository.
See if restarting Oracle Internet Directory solves the problem. If it does not, refer to "Troubleshooting Oracle Internet Directory".
The DCM daemon does not start.
This can occur due to insufficient memory problem or an OPMN problem. You can investigate this problem as follows:
Review the error messages in ORACLE_HOME/opmn/logs/dcm-daemon~dcm-daemon~dcm~1.
Review the error messages in the log files in ORACLE_HOME/opmn/logs/*.
Ensure that the ports OPMN uses are free of conflicts. These ports are listed in ORACLE_HOME/opmn/conf/opmn.xml under the <notification-server> element.
A configuration change was not reverted as expected in the following sequence of steps:
Update of DCM-managed configuration file
Execute the updateConfig command
Manually remove change to configuration file
Execute the resyncinstance command
Make DCM aware of the removal of the configuration change in Step 3 by issuing the updateConfig command after Step 3. You can use the -force option to force DCM to resynchronize the instance whether it thinks it necessary or not:
ORACLE_HOME/dcm/bin/dcmctl resyncinstance -force
dcmctl returns "invalid username/password; logon denied"
This could mean that DCM is having problems accessing Oracle Internet Directory or the DCM schema in the metadata repository. Refer to "Troubleshooting Oracle Internet Directory" for the steps for troubleshooting this problem.
dcmctl start times out while trying to start a component.
First, verify whether the component does eventually start. To do this, issue the following command repeatedly until the component starts successfully, or fails to start and returns an error:
ORACLE_HOME/dcm/bin/dcmctl getreturnstatus
If the component eventually starts, then it may be that the component needs more time than allowed by the current timeout setting to start. Some reasons for this are:
The first time an Oracle Application Server Containers for J2EE (OC4J) instance is started, it takes more time than in subsequent starts.
When all OC4J instances are started with one command, DCM starts them sequentially. If there are many OC4J instances, the time needed to start them will probably exceed the timeout.
You can increase the timeout value for all DistributedDCM operations with the following command:
ORACLE_HOME/dcm/bin/dcmctl set -t timeout_in_seconds
If the component fails to start, and returns an error, refer to the documentation for the component to resolve the problem, then try dcmctl start again.
The dcmctl start command returns:
ADMN-705002. Examining
ORACLE_HOME/dcm/logs/dcmctl_logs/log.xml
shows SubscriptionException: Subscription request timed out after 30000 milliseconds.
This indicates that DCM is trying to communicate with Oracle Process Manager and Notification Server. This is usually due to a port conflict with one of the Oracle Notification Server (ONS) ports. Check the ORACLE_HOME/opmn/conf/opmn.xml file, examine the ports used for the <notification-server> element (local, remote, request), and verify that they are free.
When the dcmctl getState -d command is executed, it returns the following and then stops:
Current state for Instance:myinstance.example.com
Component Type Up Status In Sync Status
=============================
Determine which process has actually stopped. It could be that a process associated with one of the related components (that is, OHS, OPMN, OC4J) has stopped.
Determine the PID of the dcm-daemon process, using the following command:
ORACLE_HOME/opmn/bin/opmnctl status
Send a kill -3 signal to the dcm-daemon process with the following command:
kill -3 PID
This tells the JVM to dump an execution stack trace.
Open the ORACLE_HOME/opmn/logs/dcm-daemon~dcm-daemon~dcm~1 file and examine the execution stack trace to identify the process that caused DCM to stop.
When the dcmctl joinfarm command is executed, it returns the following:
ADMN-202026 A problem has occurred accessing the OracleAS 10g infrastructure database. Base Exception: oracle.ias.repository.schema.SchemaException:DCM Repository is not Configured Please refer to the base exception for resolution or call Oracle support. oracle.ias.repository.schema.SchemaException: DCM Repository is not Configured . . .
If you are attempting to join a File based repository, issue the command again, using the -r option to specify the repository ID for the File based repository.
The -r option is required to specify the ID of a File based repository. If you erroneously omit the -r option, dcmctl attempts to join an Infrastructure database farm with the repository ID you provided.
You executed repositoryRelocated and now the repository is unreachable.
The repositoryRelocated command is intended to be used in conjunction with the importRepository command. A farm can only be associated with one repository at a time. When the repository is imported to a new instance of the farm, the old instance must be notified, through the repositoryRelocated command, that it is no longer the repository host. If the repository was not imported to a new instance, then there is currently no instance managing the repository.
The solution depends on the circumstances under which the command was issued.
If the intent was to import an exported repository image to a new instance, simply issue the importRepository command. After the repository is imported to a new instance, all instances will be able to locate the new repository host. If the command was run in error, and you wish to continue to use the current repository:
Stop the instance.
Navigate to the ORACLE_HOME/dcm/config directory.
Compare the files with the command:
diff dcm.conf dcm.conf.bak
The result should be:
< repository_type=cache
> repository_type=cache rep
Restore the dcm.conf file to the original version with the command:
cp dcm.conf.bak dcm.conf
Start the instance.
The repository is now available to the farm.
Occasionally, during configuration management operations, configuration files in are written out as zero length files. This is usually due to disk write problems, such as lack of space on the disk.
Copy the corresponding .bak file, dcm.conf.bak or dcmCache.xml.bak, to dcm.conf or dcmCache.xml respectively, and start the instance.
Occasionally, while using dcmctl commands with file based clustering, unexpected messages or failures may be seen.
Restarting all dcmctl processes, dcm daemons, and Application Server Control on all the instances in the farm may allow you to recover from these problems. Restarting these processes will not result in a loss of configuration information.
The following Application Server Control or dcmctl errors may occur if the configuration file ORACLE_HOME/dcm/config/dcmCache.xml includes a <communication> element with a <coordinator> subelement that specifies an ip attribute for an IP address that is not reachable.
The exception 202504, has occurred in the cache layer of the persistence manager. "Network port is in use.". Resolution: Please, refer to the base exception for the details.
And the following dcmctl error is also associated with this problem:
Local Stack: oracle.ias.cache.group.GroupException: Network port is in use at oracle.ias.cache.group.Transport.ssinit(Unknown Source) at oracle.ias.cache.group.Transport.init(Unknown Source)
The IP address could become unreachable if the IP address for the machine changes, for example, in a DHCP environment.
Use the following procedure to workaround this problem:
Change the ip attribute specified in the <coordinator> element of the ORACLE_HOME/dcm/config/dcmCache.xml configuration file. There are several ip attributes and <coordinator> elements specified in this file; change the <coordinator> element with attribute "original=true". In this element change the ip attribute to the correct value for the machine. Use the hostname and nlslookup commands or similar commands to identify the correct IP address.
Change the VirtualHostName value in ORACLE_ HOME/config/ias.properties to the corresponding correct hostname.
Run the command dcmctl resetHostInformation.
This should correct the problem.
If the DCM repository is not available, the dcmctl start, stop, and restart commands fail.
For example if dcmctl start returns the exception,
ADMN-202037 The DCM repository is unavailable and there is currently insufficient information cached locally for DCM to initialize or the operation to be completed.
Work around: use opmnctl to start, stop, or restart processes.
Invoking the command dcmctl getstate -i instance_name from an Oracle Application Server 10g (9.0.4) Oracle Home results in an ADMN-604104 exeption. This problem occurs when the instance_name is a 9.0.2 or 9.0.3 instance.
When trying to retrieve the state of a 9.0.2 or 9.0.3 instance, invoke the dcmctl utility from a 9.0.2 or 9.0.3 Oracle Home, and not from the Oracle Application Server 10g (9.0.4) Oracle Home.
3266895After installing BI and Forms, the $ORACLE_HOME/dcm/logs/daemon_logs/log.xml file includes messages similar to the following:
<MSG_TEXT>The registered plugin named discoverer could not be loaded and will be not be functional within the iAS Home: /private1/iasinst/OraHome1 until this problem is resolved, due to the following exception: </MSG_TEXT> <SUPPL_DETAIL>< Copyright © 2003 Oracle Corporation All rights reserved |
|