Sun Java System Message Queue 3 2005Q4 Administration Guide |
Chapter 12
Troubleshooting ProblemsThis chapter explains how to understand and resolve the following problems:
When problems occur, it is useful to check the version number of the installed Message Queue software. Use the version number to ensure that you are using documentation whose version matches the software version. You also need the version number to report a problem to Sun. To check the version number, issue the following command:
imqcmd -v
A Client Cannot Establish a ConnectionThe symptoms of this problem are as follows:
This section explores the following possible causes:
Client applications are not closing connections, causing the number of connections to exceed resource limitations
To confirm this cause of the problem
List all connections to a broker:
imqcmd list cxn
The output will list all connections and the host from which each connection has been made, revealing an unusual number of open connections for specific clients.
To resolve the problem
Rewrite the offending clients to close unused connections.
Broker is not running or there is a network connectivity problem
To confirm this cause of the problem
To resolve the problem
Connection service is inactive or paused
To confirm this cause of the problem
Check the status of all connection services:
imqcmd list svc
If the status of a connection service is shown as unknown or paused, clients will not be able to establish a connection using that service.
To resolve the problem
- If the status of a connection service is shown as unknown, it is missing from the active service list (imq.service.active). In the case of SSL-based services, the service might also be improperly configured, causing the broker to make the following entry in the broker log: ERROR [B3009]: Unable to start service ssljms: [B4001]: Unable to open protocol tls for ssljms service... followed by an explanation of the underlying cause of the exception.
To properly configure SSL services, see Working With an SSL-Based Service.
- If the status of a connection service is shown as paused, resume the service (see Pausing and Resuming a Connection Service).
Too few threads available for the number of connections required
To confirm this cause of the problem
Check for the following entry in the broker log:
WARNING [B3004]: No threads are available to process a new connection on service ... Closing the new connection.
Also check the number of connections on the connection service and the number of threads currently in use, using one of the following formats:
imqcmd query svc -n serviceName
imqcmd metrics svc -n serviceName -m cxn
Each connection requires two threads: one for incoming messages and one for outgoing messages (see Thread Pool Management).
To resolve the problem
- If you are using a dedicated thread pool model (imq.serviceName. threadpool_model=dedicated), the maximum number of connections is half the maximum number of threads in the thread pool. Therefore, to increase the number of connections, increase the size of the thread pool (imq.serviceName.max_threads) or switch to the shared thread pool model.
- If you are using a shared thread pool model (imq.serviceName. threadpool_model=shared), the maximum number of connections is half the product of the following two properties: the connection Monitor limit (imq.serviceName.connectionMonitor_limit) and the maximum number of threads (imq.serviceName.max_threads). Therefore, to increase the number of connections, increase the size of the thread pool or increase the connection monitor limit.
- Ultimately, the number of supportable connections (or the throughput on connections) will reach input/output limits. In such cases, use a multi-broker cluster to distribute connections among the broker instances within the cluster.
Too few file descriptors for the number of connections required on the Solaris or Linux operating system
For more information about this issue, see Setting the File Descriptor Limit.
To confirm this cause of the problem
Check for an entry in the broker log similar to the following: Too many open files.
To resolve the problem
Increase the file descriptor limit, as described in the ulimit man page.
TCP backlog limits the number of simultaneous new connection requests that can be established
The TCP backlog places a limit on the number of simultaneous connection requests that can be stored in the system backlog (imq.portmapper.backlog) before the Port Mapper rejects additional requests. (On Windows operating systems there is a hard-coded backlog limit: 5 for Windows desktops and 200 for Windows servers.)
The rejection of requests because of backlog limits is usually a transient phenomenon, due to an unusually high number of simultaneous connection requests.
To confirm this cause of the problem
Examine the broker log. First, check to see whether the broker is accepting some connections during the same time period that it is rejecting other connections. Next, check for messages that explain rejected connections. If you find such messages, the TCP backlog is probably not the problem, because the broker does not log connection rejections due to the TCP backlog.
If some successful connections are logged, and no connection rejections are logged, the TCP backlog is probably the problem.
To resolve the problem
The following approaches can be used to resolve TCP backlog limitations:
Operating system limits the number of concurrent connections
The Windows operating system license places limits on the number of concurrent remote connections that are supported.
To confirm this cause of the problem
Check that there are plenty of threads available for connections (using imqcmd query svc) and check the terms of your Windows license agreement. If you can make connections from a local client, but not from a remote client, operating system limitations might be the cause of the problem.
To resolve the problem
Authentication or authorization of the user is failing
The authentication can be failing due to an incorrect password, because there is no entry for the user in the user repository, or because the user does not have access permissions for the connection service.
To confirm this cause of the problem
Check entries in the broker log for the Forbidden error message. This will indicate an authentication error, but will not indicate the reason for it.
- If you are using a file-based user repository, enter the following command:
imqusermgr list -i instanceName -u userName
- If the output shows a user, the wrong password was probably submitted. If the output shows the following error, there is no entry in the user repository:
Error [B3048]: User does not exist in the password file,
- If you are using an LDAP server user repository, use the appropriate tools to check if there is an entry for the user.
- Check the access control properties file to see if there are restrictions on access to the connection service.
To resolve the problem
- If there is no entry for the user in the user repository, add the user to the user repository (see Populating and Managing a User Repository).
- If the wrong password was used, provide the correct password.
- If the access control properties are improperly set, edit the access control properties file to grant connection service permissions (see Access Control for Connection Services).
Connection Throughput Is Too SlowThe symptoms of this problem are as follows:
- Message throughput does not meet expectations.
- The number of supported connections to a broker is not limited as described in A Client Cannot Establish a Connection, but rather by message input/output rates.
This section explores the following possible causes:
Network connection or WAN is too slow
To confirm this cause of the problem
Ping the network to see how long it takes for the ping to return, and then consult a network administrator. Also you can send and receive messages using local clients and compare the delivery time with that of remote clients (which use a network link).
To resolve the problem
If the connection is too slow, upgrade the network link.
Connection service protocol is inherently slow compared to TCP
As an example, SSL-based or HTTP-based protocols are slower than TCP (see Figure 11-5).
To confirm this cause of the problem
If you are using SSL-based or HTTP-based protocols, try using TCP and compare the delivery times.
To resolve the problem
Application requirements usually dictate the protocols being used, so there is little that you can do, other than to attempt to tune the protocol as described in (Tuning Transport Protocols).
Connection service protocol is not optimally tuned
To confirm this cause of the problem
Try tuning the protocol and see if it makes a difference.
To resolve the problem
Try tuning the protocol as described in (Tuning Transport Protocols).
Messages are so large they consume too much bandwidth
To confirm this cause of the problem
Try running your benchmark with smaller-sized messages.
To resolve the problem
What appears to be slow connection throughput is actually a bottleneck in some other step of the message delivery process
To confirm this cause of the problem
If none of the items above appear to be the cause of what appears to be slow connection throughput, consult Figure 11-1 for other possible bottlenecks and check for symptoms associated with the following problems:
To resolve the problem
Follow the problem resolution guidelines provided in the problem troubleshooting sections above.
A Client Cannot Create a Message ProducerThe symptoms of this problem are as follows:
This section explores the following possible causes:
A physical destination has been configured to allow only a limited number of producers
One of the ways of avoiding the accumulation of messages on a physical destination is to limit the number of producers (maxNumProducers) that it supports.
To confirm this cause of the problem
Check the physical destination (see Displaying Information about Physical Destinations):
imqcmd query dst
The output will show the current number of producers and the value of maxNumProducers. If the two values are the same, the number of producers has reached its configured limit. When a new producer is rejected by the broker, the broker returns a ResourceAllocationException [C4088]: A JMS destination limit was reached and makes the following entry in the broker log: [B4183]: Producer can not be added to destination.
To resolve the problem
Increase the value of the maxNumProducers attribute (see Updating Physical Destination Properties).
The user is not authorized to create a message producer due to settings in the access control properties file
To confirm this cause of the problem
When a new producer is rejected by the broker, the broker returns the following message:
The broker also makes the following entries in the broker log:
To resolve the problem
Change the access control properties to allow the user to produce messages (see Access Control for Physical Destinations).
Message Production Is Delayed or SlowedThe symptoms of this problem are as follows:
This section explores the following possible causes:
The message server is backlogged and has responded by slowing message producers
A backlogged server accumulates messages in broker memory.
When the number of messages or number of message bytes in physical destination memory reaches configured limits, the broker attempts to conserve memory resources in accordance with the specified limit behavior. The following limit behaviors slow down message producers:
Similarly, when the number of messages or number of message bytes in brokerwide memory (for all physical destinations) reaches configured limits, the broker will attempt to conserve memory resources by rejecting the newest messages.
Also, when system memory limits are reached because physical destination or brokerwide limits have not been set properly, the broker takes increasingly serious action to prevent memory overload. These actions include throttling back message producers.
To confirm this cause of the problem
When a message is rejected by the broker due to configured message limits, the broker returns the following message:
The broker also makes this entry in the broker log:
The message is followed by a message indicating the limit that has been reached. If the message limit is on a physical destination, the broker makes an entry like the following:
If the message limit is broker wide, the broker makes an entry like the following:
More generally, you can check for message limit conditions before the rejections occur as follows:
- By querying physical destinations and the broker and inspecting their configured message limit settings.
- By monitoring the number of messages or number of message bytes currently in a physical destination or in the broker as a whole, using the appropriate imqcmd commands. See Chapter 18, "Metrics Reference" for information about metrics you can monitor, and the commands you use to obtain them.
To resolve the problem
There are a number of approaches to addressing the slowing of producers due to messages becoming backlogged:
In general, you should manage memory on a destination-by-destination level so that brokerwide message limits are never reached. For more information, see Broker Adjustments.
For example, you can specify the REMOVE_OLDEST and REMOVE_LOW_PRIORITY limit behaviors, which delete messages that accumulate in memory (see Table 15-1).
The broker cannot save a persistent message to the data store
If the broker cannot access a data store or write a persistent message to the data store, the producing client is blocked. This condition can also occur if destination or brokerwide message limits are reached, as described above.
To confirm this cause of the problem
If the broker is unable to write to the data store, it makes one of the following entries in the broker log: [B2011]: Storing of JMS message from connectionID failed… or [B4004]: Failed to persist message messageID…
To resolve the problem
- In the case of file-based persistence, try increasing the disk space of the file-based data store.
- In the case of a JDBC-compliant data store, check that JDBC-based persistence is properly configured (see Chapter 4, "Configuring a Broker"). If so, consult your database administrator to troubleshoot other database problems.
Broker acknowledgment timeout is too short
Due to slow connections or a lethargic message server (caused by high CPU utilization or scarce memory resources), a broker might require more time to acknowledge receipt of a persistent message than allowed by the value of the connection factory’s imqAckTimeout attribute.
To confirm this cause of the problem
If the imqAckTimeout value is exceeded, the broker returns the following message:
JMSException [C4000]: Packet acknowledge failed
To resolve the problem
Change the value of the imqAckTimeout connection factory attribute (see Reliability And Flow Control).
A producing client is encountering JVM limitations
To confirm this cause of the problem
To resolve the problem
Adjust the JVM (see Java Virtual Machine Adjustments).
Messages Are BackloggedThe symptoms of this problem are as follows:
To see whether messages are accumulating, check how the number of messages or message bytes in the broker changes over time and compare to configured limits. First check the configured limits:
Then check for message accumulation in each destination:
To see whether messages have exceeded configured destination or brokerwide limits, check the broker log for the following entry: WARNING [B2011]: Storing of JMS message from…failed. This entry will be followed by another entry explaining the limit that has been exceeded.
This section explores the following possible causes:
There are inactive durable subscriptions on a topic destination
If a durable subscription is inactive, messages are stored in a destination until the corresponding consumer becomes active and can consume the messages.
To confirm this cause of the problem
Check the state of durable subscriptions on each topic destination:
imqcmd list dur -d destName
To resolve the problem
You can take any of the following actions:
- Purge all messages for the offending durable subscriptions (see Managing Durable Subscriptions).
- Specify message limit and limit behavior attributes for the topic (see Table 15-1). For example, you can specify the REMOVE_OLDEST and REMOVE_LOW_PRIORITY limit behaviors, which delete messages that accumulate in memory.
- Purge all messages from the corresponding destinations (see Purging Physical Destinations).
- Limit the time messages can remain in memory. You can rewrite the producing client to set a time-to-live value on each message. You can override any such settings for all producers sharing a connection by setting the imqOverrideJMSExpiration and imqJMSExpiration connection factory attributes (see Message Header Overrides).
There are too few consumers available to consume messages in a queue
If there are too few active consumers to which messages can be delivered, a queue destination can become backlogged as messages accumulate. This condition can occur for any of the following reasons:
To confirm this cause of the problem
To help determine the reason for unavailable consumers, check the number of active consumers on a destination:
imqcmd metrics dst -n destName -t q -m con
To resolve the problem
You can take any of the following actions, depending on the reason for unavailable consumers:
- Create more active consumers for the queue, by starting up additional consuming clients.
- Adjust the imq.consumerFlowLimit broker property to optimize queue delivery to multiple consumers (see Multiple Consumer Queue Performance).
- Specify message limit and limit behavior attributes for the queue (see Table 15-1). For example, you can specify the REMOVE_OLDEST and REMIOVE_LOW_PRIOROTY limit behaviors, which delete messages that accumulate in memory.
- Purge all messages from the corresponding destinations (see Purging Physical Destinations).
- Limit the time messages can remain in memory. You can rewrite the producing client to set a time-to-live value on each message, you can override any such setting for all producers sharing a connection by setting the imqOverrideJMSExpiration and imqJMSExpiration connection factory attributes (see Message Header Overrides).
Message consumers are processing too slowly to keep up with message producers
In this case topic subscribers or queue receivers are consuming messages more slowly than the producers are sending messages. One or more destinations is getting backlogged with messages due to this imbalance.
To confirm this cause of the problem
Check for the rate of flow of messages into and out of the broker:
imqcmd metrics bkr -m rts
Then check flow rates for each of the individual destinations:
imqcmd metrics bkr -t destType -n destName -m rts
To resolve the problem
- Optimize consuming client code.
- For queue destinations, increase the number of active consumers (see Multiple Consumer Queue Performance).
Client acknowledgment processing is slowing down message consumption
Two factors affect the processing of client acknowledgments:
- Significant broker resources can be consumed in processing client acknowledgments. As a result, message consumption might be slowed in those acknowledgment modes in which consuming clients block until the broker confirms client acknowledgments.
- JMS payload messages and Message Queue control messages (such as client acknowledgments) share the same connection. As a result, control messages can be held up by JMS payload messages, slowing message consumption.
To confirm this cause of the problem
To resolve the problem
- Modify the acknowledgment mode used by clients, for example, switch to DUPS_OK_ACKNOWLEDGE or CLIENT_ACKNOWLEDGE.
- If using CLIENT_ACKNOWLEDGE or transacted sessions, group a larger number of messages into a single acknowledgment.
- Adjust consumer and connection flow control parameters (see Client Runtime Message Flow Adjustments).
The broker cannot keep up with produced messages
In this case, messages are flowing into the broker faster than the broker can route and dispatch them to consumers. The sluggishness of the broker can be due to limitations in any or all of the following: CPU, network socket read/write operations, disk read/write operations, memory paging, the persistent store, or JVM memory limits.
To confirm this cause of the problem
Check that none of the other causes of this problem are responsible.
To resolve the problem
Client code defects: consumers are not acknowledging messages
Messages are held in a destination until they have been acknowledged by all consumers to which the messages have been sent. If a client is not acknowledging consumed messages, the messages accumulate in the destination without being deleted.
For example, client code might have the following defects:
To confirm this cause of the problem
First check all other possible causes listed in this section. Next, list the destination with the following command:
imqcmd list dst
Notice whether the number of messages listed under the UnAcked header is the same as the number of messages in the destination. The messages under the UnAcked header were sent to consumers but not acknowledged. If this number is the same as the total number of messages, the broker has sent all the messages and is waiting for acknowledgment.
To resolve the problem
Request the help of application developers in debugging this problem.
Message Server Throughput Is SporadicThe symptom of this problem is as follows:
This section explores the following possible causes:
The broker is very low on memory resources
Because destination and broker limits were not properly set, the broker takes increasingly serious action to prevent memory overload, and this can cause the broker to become very sluggish until the message backlog is cleared.
To confirm this cause of the problem
Check the broker log for a low memory condition ([B1089]: In low memory condition, broker is attempting to free up resources), followed by an entry describing the new memory state and the amount of total memory being used.
Also check the free memory available in the JVM heap:
imqcmd metrics bkr -m cxn
Free memory is low when the value of total JVM memory is close to the maximum JVM memory value.
To resolve the problem
- Adjust the JVM (see Java Virtual Machine Adjustments).
- Increase system swap space.
JVM memory reclamation (garbage collection) is taking place
Memory reclamation periodically sweeps through the system to free up memory. When this occurs, all threads are blocked. The larger the amount of memory to be freed up and the larger the JVM heap size, the larger the delay due to memory reclamation.
To confirm this cause of the problem
Monitor CPU usage on your computer. CPU usage drops when memory reclamation is taking place.
Also start your broker using the following command line options:
-vmargs -verbose:gc
Standard output indicates the time that memory reclamation takes place.
To resolve the problem
In multiple CPU computers, set the memory reclamation to take place in parallel:
-XX:+UseParallelGC=true
The JVM is using the Just-In-Time compiler to speed up performance
To confirm this cause of the problem
Check that none of the other causes of this problem are responsible.
To resolve the problem
Let the system run for a while; performance should improve.
Messages Are Not Reaching ConsumersThe symptom of this problem is as follows:
This section explores the following possible causes:
Limit behaviors are causing messages to be deleted on the broker
When the number of messages or number of message bytes in destination memory reach configured limits, the broker attempts to conserve memory resources. Three of the configurable behaviors taken by the broker when these limits are reached will cause messages to be lost:
As the number of messages or number of message bytes in broker memory reach configured limits, the broker attempts to conserve memory resources by rejecting the newest messages.
To confirm this cause of the problem
Check the dead message queue, as described under The Dead Message Queue Contains Messages. Specifically, use the instructions under The number of messages, or their sizes, exceed destination limits. Look for the REMOVE_OLDEST or REMOVE_LOW_PRIORITY reason.
To resolve the problem
Increase the destination limits. For example:
imqcmd update dst -n MyDest -o maxNumMsgs=1000
Message time-out value is expiring
The broker deletes messages whose time-out value has expired. If a destination gets sufficiently backlogged with messages, messages whose time-to-live value is too short might be deleted.
To confirm this cause of the problem
Check the dead message queue to see whether messages are timing out.
Use the QBrowser demo application to look at the DMQ contents. The QBrowser demo is in an operating system-specific location; for the location, see Appendix A, "Platform-Specific Locations of Message Queue Data" and look in the tables for “Example Applications and Locations.”
This is an example of invocation on Windows:
cd \MessageQueue3\demo\applications\qbrowser java QBrowser
When the QBrowser main window appears, select the queue name mq.sys.dmq and then click Browse. A list like the following appears.
Figure 12-1 QBrowser Window
Double click a message to display details about that message.
Figure 12-2 QBrowser Message Details
Note whether the JMS_SUN_DMQ_UNDELIVERED_REASON property for messages has the value EXPIRED.
To resolve the problem
Contact the application developers and have them increase the time-to-live value.
Clocks are not synchronized
If clocks are not synchronized, broker calculations of message lifetimes can be wrong, causing messages to exceed their expiration times and be deleted.
To confirm this cause of the problem
In the broker log file, look for any of the following messages: B2102, B2103, B2104. These messages all report that possible clock skew was detected.
To resolve this problem
Check that you are running a time synchronization program, as described in Preparing System Resources.
Consuming client failed to start message delivery on a connection
Messages cannot be delivered until client code establishes a connection and starts message delivery on the connection.
To confirm this cause of the problem
Check that client code establishes a connection and starts message delivery.
To resolve the problem
Rewrite the client code to establish a connection and start message delivery.
The Dead Message Queue Contains MessagesThe symptom of this problem is as follows:
After you supply a user name and password, output like the following appears:
In this example, the dead message queue, mq.sys.dmq, contains 35 messages.
This section explores the following possible causes:
The number of messages, or their sizes, exceed destination limits
To confirm this cause of the problem
Use the QBrowser demo application to look at the contents of the dead message queue. The QBrowser demo is in an operating system-specific location; for the location, see Appendix A, "Platform-Specific Locations of Message Queue Data" and look in the tables for “Example Applications and Locations.”
This is an example of invocation on Windows:
cd \MessageQueue3\demo\applications\qbrowser java QBrowser
When the QBrowser main window appears, select the queue name mq.sys.dmq and then click Browse. A list like the one shown in Figure 12-1 appears.
Double click any message to display details about that message. The window shown in Figure 12-2 appears.
Note the values for the following message properties:
Under JMS Headers, note the value for JMSDestination to determine the destination whose messages are becoming dead.
To resolve this problem
Increase the destination limits. For example:
imqcmd update dst -n MyDest -o maxNumMsgs=1000
The broker clock and producer clock are not synchronized
To confirm this cause of the problem:
Using the QBrowser application, view the message details for messages in the dead message queue. Check the value for JMS_SUN_DMQ_UNDELIVERED_REASON, looking for messages with the reason EXPIRED.
In the broker log file, look for any of the following messages: B2102, B2103, B2104. These messages all report that possible clock skew was detected.
To resolve this problem
Check that you are running a time synchronization program, as described in Preparing System Resources.
Consumers are not receiving the messages before messages time out
To verify this cause of the problem
Using the QBrowser application, view the message details for messages in the dead message queue. Check the value for JMS_SUN_DMQ_UNDELIVERED_REASON, looking for messages with the reason EXPIRED.
Check to see whether there any consumers on the destination. For example:
imqcmd query dst -t q -n MyDest
Check the value listed for Current Number of Active Consumers. If there are active consumers, one of the following is true:
To resolve the problem
Request that application developers increase message time-to-live values.
There are too many producers for the number of consumers
To confirm this cause of the problem
Using the QBrowser application, view the message details for messages in the dead message queue. Check the value for JMS_SUN_DMQ_UNDELIVERED_REASON.
If the reason is REMOVE_OLDEST or REMOVE_LOW_PRIORITY, use the imqcmd query dst command to check the number of producers and consumers on the destination. If the number of producers exceeds the number of consumers, production rate might be overwhelming consumption rate.
To resolve the problem
Add more consumer clients or set the destination to use the FLOW_CONTROL limit behavior. The FLOW_CONTROL limit behavior uses consumption rate to control production rate.
Start the flow control behavior by using a command such as the following example:
imqcmd update dst -n myDst -t q -o consumerFlowLimit=FLOW_CONTROL
Producers are faster than consumers
To confirm this cause of the problem
To determine whether slow consumers are causing producers to slow down, set the destination limit behavior to FLOW_CONTROL. The FLOW_CONTROL limit behavior uses consumption rate to control production rate.
Start the flow control behavior by using a command such as the following example:
imqcmd update dst -n myDst -t q -o consumerFlowLimit=FLOW_CONTROL
Use metrics to examine the destination input and output, by issuing a command like the following example:
imqcmd metrics dst -n myDst -t q -m rts
In the metrics output, examine the following values:
- Msgs/sec Out
This value shows how many messages per second the broker is removing. The broker removes messages when all consumers acknowledge receiving them, so the metric reflects consumption rate.
- Msgs/sec In
This value shows how many messages per second the broker is receiving from producers. The metric reflects production rate.
Because flow control aligns production to consumption, note whether production slows or stops. If the rate slows or stops, there is a discrepancy between the processing speed of producers and consumers.
You can also check the number of unacknowledged (UnAcked) sent messages, by using the imqcmd list dst command. If the number of unacknowledged messages is less than the size of the destination. the destination has additional capacity and is being held back by client flow control.
To resolve the problem
If production rate is consistently faster than consumption rate, consider using flow control regularly, to keep the system aligned.
In addition, using the subsequent sections, consider and attempt to resolve each of the following possible factors:
A consumer is too slow
To confirm this cause of the problem
Use metrics to determine the rate of production and consumption, as described under Producers are faster than consumers.
To resolve the problem
Try one or more of the following:
- Set the destinations to use the FLOW_CONTROL limit behavior. Use a command like the following:
imqcmd update dst -n myDst -t q -o consumerFlowLimit=FLOW_CONTROL
Use of flow control slows production to the rate of consumption and prevents the accumulation of messages on the broker. Producer applications hold messages until the destination can process them in a timely manner, with less risk of expiration.
- Find out from application developers whether producers send messages at a steady rate, or in periodic bursts.
If an application sends bursts of messages, follow the instructions in the next item to increase destination limits.
- Increase destination limits based on number of messages or number of bytes, or both.
To change the number of messages on a destination, enter a command that has the following format:
imqcmd update dst -n destName -t {q/t} -o maxNumMsgs=number
To change the size of a destination, enter a command that has the following format:
imqcmd update dst -n destName -t {q/t} -o maxTotalMsgBytes=number
Be aware that raising limits increases the amount of memory that the broker uses. If limits are too high, the broker could run out of memory and become unable to process messages.
Clients are not committing messages
To confirm this cause of the problem
Check with application developers to find out whether the application uses transactions. If the application uses transactions, list the active transactions as follows:
imqcmd list txn
This is an example of the command output:
----------------------------------------------------------------------
Transaction ID State User name # Msgs/# Acks Creation time
----------------------------------------------------------------------
6800151593984248832 STARTED guest 3/2 7/19/04 11:03:08 AM
Note the numbers of messages and number of acknowledgments.
If the number of messages is high, producers may be sending individual messages but failing to commit transactions. Until the broker receives a commit, it cannot route and deliver the messages for that transaction.
If the number of acknowledgments is high, consumers may be sending acknowledgments for individual messages but failing to commit transactions. Until the broker receives a commit, it cannot remove the acknowledgments for that transaction.
To resolve this problem
Contact application developers to fix the coding error.
Consumers are failing to acknowledge messages
To confirm this cause of the problem
Contact application developers to determine whether the application uses system-based acknowledgment or client-based acknowledgment. If the application uses system-based acknowledgment, skip this section.
If the application uses client-based acknowledgment (the CLIENT_ACKNOWLEDGE type), first decrease the number of messages stored on the client. Use a command like the following:
imqcmd update dst -n myDst -t q -o consumerFlowLimit=1
Next, you will determine whether the broker is buffering messages because a consumer is slow, or whether the consumer processes messages quickly but does not acknowledge them.
List the destination, using the following command:
imqcmd list dst
After you supply a user name and password, output like the following appears:
Listing all the destinations on the broker specified by:
---------------------------------
Host Primary Port
---------------------------------
localhost 7676
----------------------------------------------------------------------
Name Type State Producers Consumers Msgs
Total Count UnAck Avg Size
-----------------------------------------------------------------------
MyDest Queue RUNNING 0 0 5 200 1177.0
mq.sys.dmq Queue RUNNING 0 0 35 0 1422.0
Successfully listed destinations.
The UnAck number represents messages that the broker has sent and for which it is waiting for acknowledgment. If the UnAck number is high or increasing, you know that the broker is sending messages, so it is not waiting for a slow consumer. You also know that the consumer is not acknowledging the messages.
To resolve the problem
Contact application developers to fix the coding error.
Durable consumers are inactive
To confirm this cause of the problem
Look at the topic’s durable subscribers, using the following command format:
imqcmd list dur -d topicName
To resolve the problem
An unexpected broker error occurred
To confirm this cause of the problem
Use QBrowser to examine a message, as described under Producers are faster than consumers.
If the value for JMS_SUN_DMQ_UNDELIVERED_REASON is ERROR, a broker error occurred.
To resolve the problem