Troubleshooting Common Problems

 

The following section describes how to troubleshoot common problems including:

• Collecting Diagnostic Information

• Addressing Common Problems

 

---

Collecting Diagnostic Information

Before contacting BEA Technical Support for help with cluster-related problems, follow the steps in this section to collect the required diagnostic information for your system. The primary diagnostic information for cluster-related problems is a log file that contains multiple thread dumps (if applicable) from the clustered server. This log file can be helpful in diagnosing a variety of cluster-related problems, but it is especially important for addressing problems related to cluster "freezes" and deadlocks.

Note: If you experience a cluster problem that involves a deadlock between server instances or otherwise causes your cluster to "hang," a log file that contains multiple thread dumps is a prerequisite for diagnosing your problem.

To create the required log file, follow these steps:

1. Remove or back up any log files you may currently have. In practice you should create a new log file each time you boot a WebLogic Server instance, rather than append new sessions to a historical log file.

2. Turn on verbose Garbage Collection (GC) output for your Java VM when you start WebLogic Server. See the next step for an example command line.

3. Redirect both the standard error and standard output to a log file. Doing so places thread dump information in the proper context with WebLogic Server informational and error messages, and provides a more useful log for diagnostic purposes. For example:

   % java -ms64m -mx64m -verbose:gc -classpath $CLASSPATH
   -Dweblogic.domain=mydomain -Dweblogic.Name=clusterServer1
   -Djava.security.policy==$WL_HOME/lib/weblogic.policy
   -Dweblogic.admin.host=192.168.0.101:7001 
   -Dweblogic.management.username=system
   -Dweblogic.management.password=systemPassword weblogic.Server >> 
   logfile.txt 2>&1
   

4. Continue running the WebLogic Server cluster until you have reproduced the problem.

5. For server hangs, use kill -3 or <Ctrl>-<Break> to create the necessary threads dumps for diagnosing your problem. Make sure the log file contains multiple thread dumps on each server, with distinct intervals between thread dumps.

Note: If you are running the JRockit JVM under Linux, see Getting a JRockit Thread Dump Under Linux.

Getting a JRockit Thread Dump Under Linux

If you use the JRockit JVM under Linux, use one of the following methods to generate a thread dump.

• Use the weblogic.admin THREAD_DUMP command. For instructions and limitations, see "THREAD_DUMP" in WebLogic Server Administration Guide.

• If the JVM's management server is enabled (by starting the JVM with the -Xmanagement option), you can generate a thread dump using the JRockit Management Console.

• Use Kill -3 PID, wherePID is the root of the process tree.

  To obtain the root PID, perform a:

  ps -efHl | grep 'java' **. **

  using a grep argument that is a string that will be found in the process stack that matches the server startup command. The first PID reported will be the root process, assuming that the ps command has not been piped to another routine.

Note: Under Linux, each execute thread appears as a separate process under the Linux process stack. To use Kill -3 on Linux you supply must match PID of the main WebLogic execute thread, otherwise no thread dump will be produced.

Providing Diagnostics to BEA Technical Support

After you have created a diagnostic log file (with thread dumps, if applicable), use the following guidelines when providing information to your BEA Technical Support representative:

1. Compress the log file using an operating system compression utility:

   % tar czf logfile.tar logfile.txt
   

2. Append the compressed log file to an e-mail message to your Technical Support representative.

   Note: Always include the compressed log file as an attachment to the message. Do not cut and paste the log file into the body of the e-mail.

3. If the compressed log file is too large to attach to an e-mail message, you can use the BEA Customer Support FTP site.

 

---

Addressing Common Problems

The following sections provide solutions to common cluster-related problems. They also provide information for how to diagnose non-specific problems, such as poor cluster performance.

Server Fails to Join a Cluster

There are several reasons why a WebLogic Server does not join a cluster on startup, including general network availability and WebLogic-specific configuration problems. Use this checklist to check your configuration and startup process.

1. Check your command-line parameters for typos, misspellings, etc.

2. Verify that there are no physical problems with your network connection. Network connections can be verified using the dbping utilities discussed in Testing Connections.

3. Verify that no other application is using the cluster multicast address.

4. Run the utils.MulticastTest utility to verify that multicast is working.

Other items which require troubleshooting include general configuration errors and communications errors, such as:

• Incompatible version numbers. All WebLogic Servers in the cluster must be the same version. If a server attempts to join a cluster with a WebLogic Server whose version does not match the other servers in the cluster, an error message will be generated.

• Unable to find a license for clustering. Your WebLogic license does not include the clustering feature. Contact your sales representative.

• Unable to send service announcement. This could indicate a general network problem, or a misconfigured DNS. Clustered servers communicate among themselves over multicast and must share the same (exclusive) multicast address.

• Cannot set default clusterAddress properties value. This could mean that another server with the same IP address has already joined the cluster. Check to make sure you do not have duplicate IP addresses assigned to multiple machines.

• Unable to create a multicast socket for clustering, Multicast socket send error, or Multicast socket receive error. These communications errors are most likely caused by an incorrect or bad multicast address.

  Note that each operating system has specific configuration requirements for configuring multicast; you should check your operating system documentation for help in correcting this error.