BEA Logo BEA MessageQ Release 5.0

  Corporate Info  |  News  |  Solutions  |  Products  |  Partners  |  Services  |  Events  |  Download  |  How To Buy

 

   MessageQ Doc Home   |   Configuration Guide for OpenVMS   |   Previous Topic   |   Next Topic   |   Contents   |   Index

Managing a BEA MessageQ Environment

 

BEA MessageQ problems can affect an entire message queuing bus, one or more message queuing groups, or a single BEA MessageQ application. This chapter describes how to troubleshoot problems that affect the operation of your BEA MessageQ environment.

Processing problems in the BEA MessageQ environment are generally caused by:

The information in this chapter will help you to determine whether a problem lies with the BEA MessageQ environment or is application-specific. It describes:

For more information on problems encountered during application development, refer to the BEA MessageQ Programmer's Guide.

BEA MessageQ Error Logging

Before you begin troubleshooting BEA MessageQ problems, you need to understand how BEA MessageQ alerts you to error conditions. This section describes how BEA MessageQ logs informational and error messages, how BEA MessageQ Servers operate, and how to pinpoint the source of the problems you encounter.

BEA MessageQ Output

BEA MessageQ offers several mechanisms to provide BEA MessageQ system managers, maintainers, application developers, and users with information about the status of BEA MessageQ and BEA MessageQ applications.

BEA MessageQ Stream Output

BEA MessageQ outputs messages to inform users about the current status of processing and to record system events. BEA MessageQ can display informational, status, and error messages on a terminal screen, print them on the operator's console, or write them to a log file.

BEA MessageQ messages are designed to serve a number of purposes. Some messages assist in debugging an application. Some messages alert the user to an event, error, or potentially serious problem affecting the entire BEA MessageQ group. Output messages are grouped according to their output streams.

There are three BEA MessageQ output streams. Each stream can be referred to by a logical name as follows:

Stream

Logical Name

Used for

Trace

DMQ$TRACE_OUTPUT

Debugging

Process

DMQ$PROCESS_OUTPUT

Informing the application or user of events or errors of interest ONLY to that application or user.

Group

DMQ$GROUP_OUTPUT

Informing BEA MessageQ Servers or other applications of events or errors.

Both the process and trace streams are of local interest only because they report information or events pertaining to a single application. The group stream output is of interest to the entire message queuing group because it reports information or events which can affect operation of the entire group. Each BEA MessageQ process, whether an internal server or user application process, uses these three output streams.

Stream Destinations

A stream can send output to up to five destinations. A destination can be a terminal screen, operator console, or a log file. The assignment of a stream's destination(s) can be made at either group or application startup.

Stream output can also be changed and redirected using the BEA MessageQ Manager utility (DMQ$MGR_UTILITY) Redirect Output (RO) menu option. Redirecting output includes adding or removing a destination to a stream's output or creating a new output file.

The destinations for stream output are:

Stream output is directed to a destination based on the value set for the DMQ$TRACE_OUTPUT, DMQ$PROCESS_OUTPUT, or DMQ$GROUP_OUTPUT logical names. If you want to direct output to multiple destinations, assign a string containing the destinations (separated by commas) to the logical name.

For example, you would assign SYSOUT,USER_LOG,CONSOLE to the DMQ$GROUP_OUTPUT logical name if you want the group stream to output to your terminal, the user log file, and the system console. The system will take default values, if the stream logical names are not defined.

Stream Switches

Some types of stream data require switches to be set to enable the output. A switch is set if a particular string has been assigned to the corresponding logical name. Output which falls into this category (and their corresponding logical names) are:

Output Type

Switch Logical

Values

Value Interpretation

Debugging Trace

DMQ$DEBUG

ERROR

Display Error messages to DMQ$TRACE_OUTPUT

TRACE

Display Trace messages to DMQ$TRACE_OUTPUT

ALL

Display both Error and Trace messages to DMQ$TRACE_OUTPUT

Display of Message Header Data

DMQ$HEADER

YES

Display BEA MessageQ message headers.

Server Tracing (for DMQ Server processes only)

DMQ$SERVER_TRACE

YES

Display server trace messages.

The Event Logger

Each BEA MessageQ group has an Event Logger Server which logs events, errors, and informational messages to a central file called DMQ$LOG:DMQ$EVL_bbbb_ggggg.log. These messages include announcements of server process startup, the failure of any portion of a BEA MessageQ group, or any log message sent to the event logger by a BEA MessageQ application. To uncover the source of a BEA MessageQ problem, always begin by reading the event log.

Console Output

Stream output sent to the console is normally sent to the central operator console. This output can be redirected by setting the value of the DMQ$ERROUT logical name. DMQ$ERROUT specifies the destination for warning messages from the BEA MessageQ Servers. Messages written to DMQ$ERROUT are also logged to the appropriate log file.

In-Memory Logging

The MEM_LOG value indicates that in-memory tracing is enabled. Tracing information is only written to disk when the group or process is stopped.

The size of the in memory trace buffer is a compile-time constant and is set to receive 20,000 lines of 120 characters in length. The buffer is configured as a ring, therefore only the last 20,000 events are captured.

When the trace routine is first called, it initializes the ring buffer and establishes an exit handler, which is called when the program terminates. When the program is terminated, the exit handler writes out the contents of the trace buffer to the DMQ$LOG:MEMORY_TRACE.LOG file.

BEA MessageQ server tracing is enabled either by the command procedure DMQ$USER:DMQ$SET_SERVER_LOGICALS.COM or the BEA MessageQ Manager Utility (DMQ$MGR_UTILITY). To enable in-memory tracing using the Manager Utility, set the trace, group, and process output fields to MEM_LOG and enable one or more tracing options.

It is recommended that the page file quota be increased for the server process that will be using the in-memory tracing.

Enabling Tracing

You can use the DMQ$DEBUG and DMQ$SERVER_TRACE logical names to enable tracing prior to starting a user or sender program. For more information, see Enabling Tracing Prior to Starting a Program.

You can use the RO option of the DMQ$MGR_UTILITY to dynamically enable tracing and redirect tracing information without stopping and restarting the application. For more information, see Enabling Tracing When a Program is Running.

BEA MessageQ Servers

Internal processing of BEA MessageQ messages is performed by several group server processes. Each server runs as an OpenVMS detached process. As separate processes, each server has its own process quotas and limits and its own stream output.

The main overseer of each BEA MessageQ group is the COM Server. Upon group startup, the COM Server will create other server processes. Whether the COM Server creates a other server process is based upon each server's entry in the DMQ$INIT.TXT file.

BEA MessageQ Server Output Files

Each server process has a log file where its output is written. The log files are contained in the directory specified by the DMQ$LOG logical name. The content of these files can be viewed using the OpenVMS TYPE command. Server log files are very useful for monitoring resources used by the servers and traceback information in the event of a server crash.

Table 12-1 lists the servers and their corresponding output files. In the file names, bbbb is the bus ID, ggggg is the group ID, and eeee is the endpoint. In the CLS file name, x is either S for single-client mode, D for DECnet multi-client mode, or T for TCP/IP multi-client mode:

Table 12-1 Servers and Output Files

Server Process

File

Com Server

DMQ$COM_SERVER_bbbb_ggggg.LOG

SBS Server

DMQ$SBS_SERVER_bbbb_ggggg.LOG

MRS Server

DMQ$MRS_SERVER_bbbb_ggggg.LOG

Event Logger

DMQ$EVENT_bbbb_ggggg.LOG

Journal Server

DMQ$JRN_SERVER_bbbb_ggggg.LOG

Naming Agent Server

DMQ$NA_SERVER_bbbb_ggggg.LOG

QTransfer Server

DMQ$QTRANSFER_bbbb_ggggg.LOG

TCPIP Link Driver

DMQ$TCPIP_LD_bbbb_ggggg.LOG

DECnet Link Driver

DMQ$DECNET_LD_bbbb_ggggg.LOG

Client Library Server

DMQ$CLS_x_eeeee_bbbb_ggggg.LOG

BEA MessageQ Server Logging/Debugging

BEA MessageQ offers the same stream output mechanisms for server processes as are available for user processes. The logical names used to direct output streams are contained in the DMQ$SET_SERVER_LOGICALS.COM procedure located in the DMQ$USER directory.

To isolate and diagnose a server problem, you can modify a server process stream output to direct specific server output such as server debugger tracing. Listing 12-1 is a sample of the COM Server portion of the DMQ$SET_SERVER_LOGICALS.COM file.

Listing 12-1 COM Server Logical Name Settings


$ COM: 
$ trace_output = "SYSOUT"
$ group_output = "EVL_LOG,CONSOLE"
$ process_output = "SYSOUT"
$ debug = ""
$ headers = ""
$ server_trace = ""
$ user_log = ""
$ goto SET_LOGICALS


The symbol definitions in this file equate to the DMQ$* stream logical names for that particular server. For example, the trace_output symbol above equates to the DMQ$TRACE_OUTPUT logical name for the COM Server. When the DMQ$SERVER_TRACE logical name is set to YES, server-specific debug tracing is activated. The output is written to wherever trace_output is specified for that server.

In addition to the capability to set server stream output switches at startup, you can also dynamically alter the stream output switch settings for server processes that are already started. The ability to redirect output for a running server provides the capability to turn on a particular output stream (such as debug tracing) on a running server, capture the output to a specified file, and then turn the stream output off after sufficient data has been captured. It also gives you the capability to capture stream output sessions to files using the DMQ$MGR_UTILITY RO (Redirect Output) option.

Tools for Troubleshooting BEA MessageQ Problems

BEA MessageQ is a complex, multiplatform, networked communication product. Its ability to function is affected by the computer network(s) over which it communicates, the hardware and software of each platform running BEA MessageQ, the configuration of each BEA MessageQ group, and the application environment for each application using BEA MessageQ.

Problems can occur in following areas:

Server Log Messages

Each BEA MessageQ Server logs informational messages to describe events or situations affecting BEA MessageQ. Each message provides a description of an event, reports on a condition of a BEA MessageQ component, or provides information regarding the operation of BEA MessageQ. Error Log Messages, contains a description of warning and error messages logged by the COM Server, SBS Server, MRS Server and link drivers.

The source of many BEA MessageQ processing problems can be found by viewing the contents of the server log files. For example, the COM Server log file can contain messages describing problems that may be encountered with group startup. Of particular importance is the group-wide event log file (DMQ$EVL_bbbb_ggggg.LOG) which contains messages on problems that may affect the entire message queuing group.

For example, a failure to send a message to a remote node would appear in the event logger as:

Listing 12-2 Sample Log File Information for Message Failure


. 
.
.
EVENT_LOGGER 30-DEC-1997 14:00:19.12 I Message Undeliverable
EVENT_LOGGER 30-DEC-1997 14:00:19.14 I + Discarded msg - Src=4.4 Tgt=2.155
Class=1 Type=-100
EVENT_LOGGER 30-DEC-1997 14:00:19.16 I + Org-Src=2.2 Org-Tgt=4.4 Size=10
Seq=00020002:00000C32
EVENT_LOGGER 30-DEC-1997 14:00:19.16 I %PAMS-E-NOTACTIVE, Target
process is not currently active - message not sent


API Return Status Values

BEA MessageQ applications will receive a return status value for each call made to BEA MessageQ API routines. Programmers should check the return status after calling API routines to perform the necessary error recovery steps. For a list of return status values for each BEA MessageQ API routine and a description of their meaning, refer to the BEA MessageQ Programmer's Guide.

Utility Programs

BEA MessageQ provides the following utilities to assist in troubleshooting system problems:

The following describes how each utility can be used to troubleshoot problems. See Table 9-1 for information on running these utilities from the BEA MessageQ main menu.

DMQ$TEST

The DMQ$TEST utility allows you to attach to a queue, send and receive messages, and detach from the queue. This utility can be run directly or executed by choosing Option 6 from the BEA MessageQ main menu.

DMQ$LOOP

The DMQ$LOOP utility allows you to test communication on the message queuing bus by performing a loopback test of messages. The utility can be run directly or by choosing Option 4 from the BEA MessageQ main menu.

DMQ$LLS_VERIFY

DMQ$LLS_VERIFY allows you to check BEA MessageQ global sections. It is used to look for queues with large numbers of pending messages. This utility reports information about the buffer pools such as the number of pending messages and the parameter settings. The utility can be run directly or by choosing Option 3 from the BEA MessageQ main menu.

DMQ$MONITOR

The DMQ$MONITOR utility allows you to view queues, queue counters, queue quotas, links to other groups and cross-group connections. Some monitoring can be performed dynamically in continuous mode. In addition, the utility provides the capability to reset queue counters, kill the COM Server, and set and alternate group. The utility can be run directly or by choosing Option 5 from the BEA MessageQ main menu.

DMQ$MGR_UTILITY

The DMQ$MGR_UTILITY utility allows you to perform several BEA MessageQ management functions. For troubleshooting purposes, this utility offers the capability to set or redirect the stream output and trace switches for processes attached to the group including the server processes. This functionality is obtained by selecting the RO menu item from the DMQ$MGR_UTILITY utility. The utility can be run directly or by choosing Option 7 from the BEA MessageQ main menu.

DMQ$SCAN_SYSTEM_FOR_DMQ.COM

This utility scans processes on the system and indicates which processes are MessageQ processes (processes that are using either DMQ$ENTRYRTL or DMQ$EXECRTL). This utility is run directly from the command line.

Troubleshooting Procedures

This section describes the basics tasks in troubleshooting BEA MessageQ and explains how to solve some of the most common problems.

Basic Troubleshooting Tasks

You can monitor BEA MessageQ output and isolate operational problems by directing and redirecting output streams, switching specific types of stream output on and off, and examining log files.

If you are having difficulty starting a BEA MessageQ group, look at the COM Server log file. Because the COM Server process is central to BEA MessageQ communications, it often contains informative messages that will help you to pinpoint reasons why the message queuing group will not start and why other communications failures occur.

Diagnosing Application Errors

To isolate a problem that occurs during application development, it is helpful to see the flow of BEA MessageQ routines. By defining the value of the DMQ$DEBUG logical name to ALL, you can view informational messages that document the execution of BEA MessageQ API routines. Setting the DMQ$DEBUG logical name to ALL will also cause debugging trace information to be written to wherever DMQ$TRACE_OUTPUT is directed.

These attributes can be set dynamically while the application is running using the DMQ$MGR_UTILITY program and specifying the RO (Redirect Output) option. Use this option to turn a switch on or off, and start and stop log file output as you need.

Enabling Server Tracing

Just as you can turn on tracing information for an application process, you can turn on tracing for a server process. In addition, servers allow a special form of tracing called server tracing. This causes the server to report server-specific processing during its operation. See BEA MessageQ Server Logging/Debugging for instructions on how to activate server tracing.

Verifying BEA MessageQ Group Startup

During the startup process for each message queuing group, BEA MessageQ:

The startup procedure writes messages to SYS$OUTPUT to provide startup progress information. The last portion of a successful startup reports that all servers responded. BEA MessageQ Server processes are OpenVMS detached processes.

Note that these processes were created using the OpenVMS SHOW PROCESS command. The BEA MessageQ Server processes appears with a process name of DMQ_c_bbbbggggg, where c represents the server identifier, bbbb represents the bus id, and ggggg represents the group id. The server identifications are:

For example, the output of the SHOW SYSTEM command lists the COM Server, Event Logger, Journal Server, MRS Server, Qtransfer Server, and SBS Server processes:

Listing 12-3 Sample Output of SHOW SYSTEM Command


$ 
$ show system
OpenVMS V6.2 on node NODE1 9-JAN-1998 11:44:05.45 Uptime 108 03:00:49
Pid Process Name State Pri I/O CPU Page flts Pages
20A00081 SWAPPER HIB 16 0 0 00:04:48.03 0 0
20A00086 CONFIGURE HIB 10 32 0 00:00:00.07 235 192
.
.
.
20A108A0 DMQ_C_009900040 LEF 8 16348 0 00:00:32.73 31491 6723
20A10BA5 DMQ_L_009900040 LEF 8 514 0 00:00:01.29 2336 1438
20A129B6 DMQ_T_009900040 LEF 8 144 0 00:00:00.88 2796 1787
20A141B7 DMQ_D_009900040 LEF 8 88 0 00:00:00.67 2391 1504
20A148B9 DMQ_J_009900040 LEF 7 180 0 00:00:01.43 5418 1827
20A12ABA DMQ_M_009900040 LEF 7 140 0 00:00:01.48 7225 2989
20A0E8BB DMQ_S_009900040 LEF 7 84 0 00:00:01.20 4136 1228
20A06F40 DMQ_N_009900040 LEF 7 1303 0 00:00:03.65 2689 1648
20A11B1F DMQ_Q_009900040 LEF 5 84 0 00:00:00.71 2454 1204
20A0F0C4 DMQ_CLS_D_6000 LEF 7 90 0 00:00:00.67 2248 1204
20A106CB DMQ_CLS_T_5001 LEF 7 112 0 00:00:00.68 2445 1355
.
.
.


To verify that the group has successfully started, you can run the DMQ$TEST utility. Attach to a temporary queue, send a message, read the message and then exit to verify that the group has started and is operational.

Troubleshooting BEA MessageQ Startup Problems

If the group failed to start, you need to isolate the problem and apply corrective action. Answer the following questions to help you pinpoint the source of the group startup failure.

Group Startup Failure

Additional Group Startup Problems

The following questions can also help you identify the source of the group startup failure.

Troubleshooting Problems with Running Groups and Queues

A BEA MessageQ message queuing group is a collection of processes and shared resources. Many factors can affect group operation. One of the most common problems is the depletion of one or more resources.

For example, message queues have resources limits restricting the number of messages and the total amount of message buffer pool they can use at any one time. If a running queue fails to send or receive messages, check the queue counters and quotas with the DMQ$MONITOR utility.

Groups also have a limited number of resources. For example, a group can only store a limited number messages as configured in the buffer pool configuration section of the DMQ$INIT.TXT file. If you experience group-wide failure or performance degradation, you should check the buffer pool memory.

The DMQ$LLS_VERIFY utility can assist in determining what amount of global memory sections are used and how much is available. If you are approaching the maximum for one or more BEA MessageQ resources, adjust the settings for the particular resource in the DMQ$INIT.TXT file and restart the group. Refer to Sizing and Tuning the BEA MessageQ Environment, for information on setting OpenVMS and BEA MessageQ resource parameters.

BEA MessageQ Connectivity Troubleshooting

To deliver a message to its target queue, it must flow through the following components of the BEA MessageQ environment:

Failure of any component along this chain will result in failed message delivery. Answer the following questions to isolate and correct a communications failure:

Due to production constraints, not every application can be tested using the actual application, queue or messages. For those situations, BEA MessageQ provides test utilities such as DMQ$TEST. This will allow you to attach to a test queue and send a test message to a target queue.

You can use the same tools (the DMQ$MONITOR utility, server tracing, logging, and network tools) with the DMQ$TEST utility to test end-to-end connectivity. We recommend that you have a test environment designed and ready where you can rapidly troubleshoot problems when they occur. Your test environment should include:

BEA MessageQ Recovery Troubleshooting

When an application utilizes recoverable messaging, a message must flow through the following BEA MessageQ components:

To isolate a problem with a recoverable message, trace error and event logging for each of these components as well as the following:

BEA MessageQ Application Troubleshooting

BEA MessageQ reports the success or failure of BEA MessageQ operations by returning status values from calls to BEA MessageQ API calls and by placing values in the delivery status field of PAMS Status Block (PSB). For detailed information on obtaining and using BEA MessageQ status values refer to the BEA MessageQ Programmer's Guide.

BEA MessageQ applications should test the return status and perform the proper handling based on the these values. To assist in the debugging of BEA MessageQ application, BEA MessageQ allows tracing to be turned on during application execution. Tracing writes messages to the destinations specified in DMQ$TRACE_OUTPUT (SYS$OUTPUT by default) that describe the execution path through BEA MessageQ code.

Application developers can take advantage of this by defining the DMQ$DEBUG logical name to the value of ERROR, TRACE, or ALL before running their application. When set to ERROR, messages are written to the trace stream when BEA MessageQ routines encounter an error.

When set to TRACE, messages are written as BEA MessageQ executes its API functions. When set to ALL, BEA MessageQ reports both trace and error information to the trace stream. ALL is the most commonly used setting for DMQ$TRACE_OUTPUT during application debugging. An example of trace output is as follows:

Listing 12-4 Sample Trace Output


$ define dmq$debug all 
$ run DMQ$EXE:DMQ$TEST

0.0 DMQ$TEST - Bus 0099 9-JAN-1998 12:21:28.08



-- Using PAMS_ATTACH_Q --
Attach mode (Name=1,Number=2,Temp=3) [3]:
Q type (PQ/SQ/MRQ) [PQ]:

DmQ T 23:40.0 Time Stamp - 9-JAN-1998 12:23:40.04
DmQ T 23:40.0 Tracing/logging event flag number = 61
DmQ T 23:40.0 Entering PAMS_ATTACH_Q
DmQ T 23:40.0 Entering PAMS_COMMON_ATTACH
DmQ T 23:40.0 Entering INIT_PARAMS
DmQ T 23:40.0 (req_q_num:0, &q_num:0X7FE0E8E8, q_type:-200, q_name:""
DmQ T 23:40.0 q_name_len:0, adver_scope:-202, mode:-212, tmo:600)
DmQ T 23:40.0 Entering MAP_AND_CHECK_QUEUE
DmQ T 23:40.0 Entering PAMS_ATTACH_TO_DMQ_ENV
DmQ T 23:40.0 (Is_CS:0 Process name: "+_RTA1:" PID: 20A10C07, LOCKTMO:600)
DmQ T 23:40.0 Using event flag 60
DmQ T 23:40.0 Using event flag 59
DmQ T 23:40.0 DMQ$BLOCKING_CONFIRM == NO
DmQ T 23:40.0 Wakeup Mode = EXEC_MODE_AST
DmQ T 23:40.0 Connecting to message bus #0099 group #00040
.
.
.


In addition to monitoring the execution of a BEA MessageQ application, you may wish to observe the behavior of a group while an application is running. The DMQ$MONITOR utility program allows you to observe several features of a running group. One often used feature is the continuous queue counter display. This allows you to monitor the sending, receiving, and pending queue counters for queues in a group.

You can use this display to determine if a queue is sending out the proper number of messages to the correct queue by looking at the send count for the application's queue and the receive count for the target queue. In addition to queue counters, the DMQ$MONITOR utility allows you to observe several other facets of a running BEA MessageQ group. See Using BEA MessageQ System Management Utilities, for more information on the DMQ$MONITOR utility.