Troubleshooting the Message Queue Component

This section describes how to troubleshoot problems with the Message Queue component and its broker. It contains the following topics:

This chapter contains the following sections:

• Using telnet to Verify That the Message Queue Broker is Running

• Collecting Additional Information About the Message Queue Broker

• Troubleshooting Communication Problems With Directory Server

• Troubleshooting Memory Problems

• To Recover From a Message Queue Broker Low Memory Condition

Using telnet to Verify That the Message Queue Broker is Running

Verify that the Message Queue broker is running. Using the telnet command to connect to the machine and port where the Message Queue broker is running returns a list of the active Message Queue services:

# telnet localhost 7676
Trying 127.0.0.1...
Connected to localhost.
Escape character is \q^]\q.
101 psw-broker 3.0.1
cluster tcp CLUSTER 32914
admin tcp ADMIN 32912
portmapper tcp PORTMAPPER 7676
ssljms tls NORMAL 32913
jms tcp NORMAL 32911
Connection closed by foreign host.

If the ssljms tcp NORMAL service is not listed in the output, then examine the Message Queue logs for potential problems. The location of the log depends on the platform you are using as follows:

• On Solaris, the log is in the following location: /var/imq/instances/psw-broker/log/log.txt

• On Linux, the log is in the following location: /var/opt/sun/mq/instances/isw-broker/log/log.txt

• On Windows, the log is in the following location: installation_root\isw-machine_name\imq\var\instances\isw-broker\log\log.txt

  If the telnet command fails, then either the broker is not running or the wrong port was specified. Check the port number in the broker’s log. For example, the log contains a line for the broker's port as follows:

[13/Mar/2003:18:17:09 CST] [B1004]:
'Starting the portmapper service using tcp
[ 7676, 50 ]
with min threads 1 and max threads of 1'

If the broker is not running, start it on Solaris and Linux by running the /etc/init.d/imq start command. On Windows, start the broker by starting the iMQ Broker Windows service.

If you install Message Queue on Solaris 8, and you run the mquinstall command to install all of the packages, be sure to set IMQ_JAVAHOME propertybefore running the mqinstall command. This ensures that the software picks the correct version of Java.

If you have not yet installed the core component, you do not have to set the IMQ_JAVAHOME property because the Identity Synchronization for Windows installation program tells the Message Queue broker which version of Java to use.

Collecting Additional Information About the Message Queue Broker

You can run the broker with the debug log turned on to help collect additional information about your problem. To turn on the debug log level, use the following command:

# imqbrokerd -loglevel DEBUG

You can get a debug dump of the broker using the following command:

imqcmd dump bkr -edebug -u admin -o file=filename

Troubleshooting Communication Problems With Directory Server

The Message Queue broker authenticates clients with the Directory Server that stores the Identity Synchronization for Windows configuration. If the broker cannot connect to this Directory Server, no clients can connect to the Message Queue. The broker log will contain a javax.naming exception, such as “javax.naming.CommunicationException or javax.naming.NameNotFoundException.

If a javax.naming exception occurs, take the following steps:

• Verify that all imq.user_repository.ldap properties in the /var/imq/instances/isw-broker/props/config.properties file have the correct values. If any of the values are incorrect, stop the Message Queue broker. Correct the errors, save the file, and restart the broker. Note that machine running the Message Queue broker must be able to resolve the Directory Server host name.

• Verify that the imq.user_repository.ldap.password property in the /etc/imq/passfile file is correct.

• Sometimes, the broker can not search for entries if the root suffix contains spaces. Verify that the root suffix name does not contain spaces.

Troubleshooting Memory Problems

During normal operation, the Message Queue broker consumes a modest amount of memory. However, during idsync resync operations, the broker’s memory requirements increase. If the broker reaches its memory limit, undelivered messages will accumulate, the idsync resync operations will slow down dramatically or even, and Identity Synchronization for Windows may be unresponsive.

When the broker enters a low-memory state, the following messages will appear in its log:

[03/Nov/2003:14:07:51 CST] [B1089]:
In low memory condition,
Broker is attempting to f
ree up resources [03/Nov/2003:14:07:51 CST] [B1088]:
Entering Memory State [B0024]:
RED  from previous state [B0023]:
ORANGE - current memory is 1829876K,
90% of total memory

To avoid a low memory state, take the following steps:

• Increase the broker’s memory limit to 1 or 2 GB, as explained in Oracle Directory Server Enterprise Edition Release Notes.

• During an idsync resync operation, keep the log level set to INFO. Changing the log level to FINE or higher increases the load of the broker as more log messages are sent to the central logger.

• Run the idsync resync command for one synchronization user list at a time.

To Recover From a Message Queue Broker Low Memory Condition

1. Verify that the broker has a backlog of undelivered messages.

   Examine the broker's persistent message store in the appropriate directory for your operating system:

   • On Solaris: /var/imq/instances/psw-broker/filestore/message/

   • On Linux: /var/opt/sun/mq/instances/isw-broker/filestore/message/

   • On Windows: installation_root\isw-machine_name\imq\var\instances\isw-broker\filestore\message\

   Each file in this directory contains a single undelivered message. If there are more than 10,000 files in this directory, then the broker has a backlog of messages. Otherwise, an undelivered message backlog is not causing the problem with the broker.

   The message backlog usually contains log files related to an idsync resync operation and you can safely remove them.

2. Stop the Message Queue broker.

   For more information, see Starting and Stopping Services in Oracle Identity Synchronization for Windows 6.0 Installation and Configuration Guide.

3. Remove all files in the persistent message store.

   The easiest way to remove these files is by recursively removing the message/ directory and then recreating it.

4. Restart the Message Queue broker.

   To avoid running out of memory in the future, take the steps described earlier in this section.