In addition to the capability to set server stream output switches at
startup, you can also dynamically alter the stream output switch
settings for server processes that are already started. The ability to
redirect output for a running server provides the capability to turn on
a particular output stream (such as debug tracing) on a running server,
capture the output to a specified file, and then turn the stream output
off after sufficient data has been captured. It also gives you the
capability to capture stream output sessions to files using the
DMQ$MGR_UTILITY RO (Redirect Output) option.
13.3 Tools for Troubleshooting MessageQ Problems
MessageQ is a complex, multiplatform, networked communication product. Its ability to function is affected by the computer network(s) over which it communicates, the hardware and software of each platform running MessageQ , the configuration of each MessageQ group, and the application environment for each application using MessageQ .
Problems can occur in following areas:
MessageQ troubleshooting tools fall into the following categories:
Each MessageQ Server logs informational messages to describe events or situations affecting MessageQ . Each message provides a description of an event, reports on a condition of a MessageQ component, or provides information regarding the operation of MessageQ . Appendix C of this document contains a description of warning and error messages logged by the COM Server, SBS Server, MRS Server and link drivers.
The source of many MessageQ processing problems can be found by viewing the contents of the server log files. For example, the COM Server log file can contain messages describing problems that may be encountered with group startup. Of particular importance is the group-wide event log file (DMQ$EVL_bbbb_ggggg.LOG) which contains messages on problems that may affect the entire message queuing group.
For example, a failure to send a message to a remote node would appear in the event logger as:
.
.
.
EVENT_LOGGER 30-DEC-1997 14:00:19.12 I Message Undeliverable
EVENT_LOGGER 30-DEC-1997 14:00:19.14 I + Discarded msg - Src=4.4 Tgt=2.155
Class=1 Type=-100
EVENT_LOGGER 30-DEC-1997 14:00:19.16 I + Org-Src=2.2 Org-Tgt=4.4 Size=10
Seq=00020002:00000C32
EVENT_LOGGER 30-DEC-1997 14:00:19.16 I %PAMS-E-NOTACTIVE, Target process is
not currently active - message not sent
MessageQ applications will receive a return status value for each
call made to MessageQ API routines. Programmers should check the return
status after calling MessageQ routines to perform the necessary
error recovery steps. For a list of return status values for each
MessageQ API routine and a description of their meaning, refer to
the BEA MessageQ Programmer's Guide .
13.3.3 Utility Programs
MessageQ provides the following utilities to assist in troubleshooting system problems:
The following describes how each utility can be used to troubleshoot
problems.
13.3.3.1 DMQ$TEST
The DMQ$TEST utility allows you to attach to a queue, send and receive
messages, and detach from the queue. This utility can be run directly
or executed by choosing Option 6 from the MessageQ main menu (see
Table 10-1 ).
13.3.3.2 DMQ$LOOP
The DMQ$LOOP utility allows you to test communication on the message
queuing bus by performing a loopback test of messages. The utility can
be run directly or by choosing Option 4 from the MessageQ main
menu (see Table 10-1 ).
13.3.3.3 DMQ$LLS_VERIFY
DMQ$LLS_VERIFY allows you to check MessageQ global sections. It is
used to look for queues with large numbers of pending messages. This
utility reports information about the buffer pools such as the number
of pending messages and the parameter settings. The utility can be run
directly or by choosing Option 3 from the MessageQ main menu (see
Table 10-1 ).
13.3.3.4 DMQ$MONITOR
The DMQ$MONITOR utility allows you to view queues, queue counters,
queue quotas, links to other groups and cross-group connections. Some
monitoring can be performed dynamically in continuous mode. In
addition, the utility provides the capability to reset queue counters,
kill the COM Server, and set and alternate group. The utility can be
run directly or by choosing Option 5 from the MessageQ main menu
(see Table 10-1 ).
13.3.3.5 DMQ$MGR_UTILITY
The DMQ$MONITOR utility allows you to perform several MessageQ
management functions. For troubleshooting purposes, this utility offers
the capability to set or redirect the stream output and trace switches
for processes attached to the group including the server processes.
This functionality is obtained by selecting the RO menu item from the
DMQ$MGR_UTILITY utility. The utility can be run directly or by choosing
Option 7 from the MessageQ main menu (see Table 10-1 ).
13.4 Troubleshooting Procedures
This section describes the basics tasks in troubleshooting
MessageQ and explains how to solve some of the most commonly
encountered problems.
13.4.1 Basic Troubleshooting Tasks
You can monitor MessageQ output and isolate operational problems by directing and redirecting output streams, switching specific types of stream output on and off, and examining log files.
If you are having difficulty starting a MessageQ group, look at
the COM Server log file. Because the COM Server process is central to
MessageQ communications, it often contains informative messages
that will help you to pinpoint reasons why the message queuing group
will not start and why other communications failures occur.
13.4.1.1 Diagnosing Application Errors
To isolate a problem that occurs during application development, it is helpful to see the flow of MessageQ routines. By defining the value of the DMQ$DEBUG logical name to ALL, you can view informational messages that document the execution of MessageQ API routines. Setting the DMQ$DEBUG logical name to ALL will also cause debugging trace information to be written to wherever DMQ$TRACE_OUTPUT is directed.
These attributes can be set dynamically while the application is
running using the DMQ$MGR_UTILITY program and specifying the RO
(Redirect Output) option. Use this option to turn a switch on or off,
and start and stop log file output as you need.
13.4.1.2 Enabling Server Tracing
Just as you can turn on tracing information for an application process,
you can turn on tracing for a server process. In addition, servers
allow a special form of tracing called server tracing. This causes the
server to report server-specific processing during its operation. See
Section 13.2.2.2 on how to activate server tracing.
13.4.2 Verifying MessageQ Group Startup
During the startup process for each message queuing group, MessageQ executes the following tasks:
The startup procedure will write messages to SYS$OUTPUT to provide startup progress information. The last portion of a successful startup will report that all servers responded. MessageQ Server processes are OpenVMS detached processes.
You can see that the processes have been created using the OpenVMS SHOW PROCESS command. The MessageQ Server processes will appear with a process name of DMQ_c_bbbbggggg, where c represents the server identifier, bbbb represents the bus id, and ggggg represents the group id. The server identifications are:
For example, the output of the SHOW SYSTEM command lists the COM Server, Event Logger, Journal Server, MRS Server, Qtransfer Server, and SBS Server processes:
$ $ show system OpenVMS V6.2 on node NODE1 9-JAN-1998 11:44:05.45 Uptime 108 03:00:49 Pid Process Name State Pri I/O CPU Page flts Pages 20A00081 SWAPPER HIB 16 0 0 00:04:48.03 0 0 20A00086 CONFIGURE HIB 10 32 0 00:00:00.07 235 192 . . . 20A108A0 DMQ_C_009900040 LEF 8 16348 0 00:00:32.73 31491 6723 20A10BA5 DMQ_L_009900040 LEF 8 514 0 00:00:01.29 2336 1438 20A129B6 DMQ_T_009900040 LEF 8 144 0 00:00:00.88 2796 1787 20A141B7 DMQ_D_009900040 LEF 8 88 0 00:00:00.67 2391 1504 20A148B9 DMQ_J_009900040 LEF 7 180 0 00:00:01.43 5418 1827 20A12ABA DMQ_M_009900040 LEF 7 140 0 00:00:01.48 7225 2989 20A0E8BB DMQ_S_009900040 LEF 7 84 0 00:00:01.20 4136 1228 20A06F40 DMQ_N_009900040 LEF 7 1303 0 00:00:03.65 2689 1648 20A11B1F DMQ_Q_009900040 LEF 5 84 0 00:00:00.71 2454 1204 20A0F0C4 DMQ_CLS_D_6000 LEF 7 90 0 00:00:00.67 2248 1204 20A106CB DMQ_CLS_T_5001 LEF 7 112 0 00:00:00.68 2445 1355 . . .
To verify that the group has successfully started, you can run the
DMQ$TEST utility. Attach to a temporary queue, send a message, read the
message and then exit to verify that the group has started and is
operational.
13.4.3 Troubleshooting MessageQ Startup Problems
If the group failed to start, you need to isolate the problem and apply corrective action. Answer the following questions to help you pinpoint the source of the group startup failure:
$ @dmq$startup 99 40
DMQ$STARTUP.COM - 9-JAN-1998 11:48:19.95
Bus: 0099
Group: 00040
Disk: $1$DIA5:
Timeout: 120
Start Servers: Yes
DMQ Version: DMQ$V40
User Area: DMQ$DISK:[DMQ$V40.USER.0099_00040]
Log Area: DMQ$DISK:[DMQ$V40.LOG.0099_00040]
%DMQ-S-SETLNM, Set to MessageQ LNM table DMQ$LNM_0099_00040
%DMQ-W-NOGMTLNM, DMQ$GMT_OFFSET not set
Copyright © BEA Systems, Inc. 1997. All rights reserved.
%DMQ-I-FORCEX, Forcing image termination on process "DMQ_C_009900040"
%DMQ-S-IMGTERM, Process "DMQ_C_009900040" image terminated
...Getting list of installed images
...Verifying DMQ$MSGSHR
...Verifying DMQ$EXECRTL
...Verifying DMQ$ENTRYRTL
...Verifying DMQ$SET_LNM
%DMQ-S-SETLNM, Set to MessageQ LNM table DMQ$LNM_0099_00040
...Starting DMQ_C_009900040 (COM Server)
%RUN-S-PROC_ID, identification of created process is 20A138A8
DmQ I 55:55.6 Time Stamp - 9-JAN-1998 11:55:55.65
DmQ I 55:55.6 loader is running
DmQ I 55:55.7 Parsing PROFILE Section
DmQ I 55:55.8 Parsing GROUP Section
DmQ I 55:55.8 Parsing ROUTE Section
DmQ I 55:55.8 Parsing CLS Section
DmQ I 55:55.9 Parsing BUFFER Section
DmQ I 55:55.9 Parsing QCT Section
DmQ I 55:56.4 Parsing SBS Section
DmQ I 55:56.4 Parsing MRS Section
DmQ I 55:56.5 Parsing GNT Section
DmQ I 55:56.7 Parsing NAM Section
DmQ I 55:56.7 loader is exiting
******* DMQ_C_009900040 HAS NOT INDICATED IT'S UP AFTER 60 SECONDS *******
- This maybe related to system load -
Please examine the newly created version of DMQ$COM_SERVER_0099_00040.LOG
in DMQ$LOG. This file will most likely contain the reason for the
COM Server's failure to boot.
However, should this file not contain enough information there is
another course of action. That is to run the COM Server interactively with
trace turned on so that all errors are displayed on your terminal. This
method requires that your process has sufficent quotas and privileges.
Run the COM Server interactively [Y/N] (Y)? N
%NONAME-F-NOMSG, Message number 00000004
%DMQ-F-STARTFAIL, Failed on startup of DMQ$COM_SERVER
%NONAME-F-NOMSG, Message number 00000004
. . . DmQ I 55:56.7 Group 40 (GRP40) Initialized DmQ I 55:56.7 This group is the primary DmQ E 55:56.8 Failed to open checkpoint file for write access DmQ E 55:56.8 %RMS-E-PRV, insufficient privilege or file protection violation DmQ E 55:56.9 Failed to open Checkpoint file DmQ E 55:56.9 %RMS-E-PRV, insufficient privilege or file protection violation %RMS-E-PRV, insufficient privilege or file protection violation %RMS-E-PRV, insufficient privilege or file protection violation POPP job terminated at 9-JAN-1998 11:55:58.02 Accounting information: Buffered I/O count: 146 Peak working set size: 6128 Direct I/O count: 50 Peak page file size: 32345 Page faults: 30769 Mounted volumes: 0 Charged CPU time: 0 00:00:03.95 Elapsed time: 0 00:00:12.33
A MessageQ message queuing group is a collection of processes and shared resources. Many factors can affect group operation. One of the most common problems is the depletion of one or more resources.
For example, message queues have resources limits restricting the number of messages and the total amount of message buffer pool they can use at any one time. If a running queue experiences failures sending or receiving messages, check the queue counters and quotas with the DMQ$MONITOR utility.
Groups also have a limited number of resources. For example, a group can only store a limited number messages as configured in the buffer pool configuration section of the DMQ$INIT.TXT file. If you experience group-wide failure or performance degradation, you should check the buffer pool memory.
The DMQ$LLS_VERIFY utility can assist in determining what amount of
global memory sections are used and how much is available. If you are
approaching the maximum for one or more MessageQ resources, adjust
the settings for the particular resource in the DMQ$INIT.TXT file and
restart the group. Refer to Chapter 12 for information on setting
OpenVMS and MessageQ resource parameters.
13.4.5 MessageQ Connectivity Troubleshooting
To deliver a message to its target queue, it must flow through the following components of the MessageQ environment:
Failure of any component along this chain will result in failed message delivery. Answer the following questions to isolate and correct a communications failure:
Due to production constraints, not every application can be tested using the actual application, queue or messages. For those situations, MessageQ provides test utilities such as DMQ$TEST. This will allow you to attach to a test queue and send a test message to a target queue.
You can use the same tools (the DMQ$MONITOR utility, server tracing, logging, and network tools) with the DMQ$TEST utility to test end-to-end connectivity. We recommend that you have a test environment designed and ready where you can rapidly troubleshoot problems when they occur. Your test environment should include:
When an application utilizes recoverable messaging, a message must flow through the following MessageQ components:
To isolate a problem with a recoverable message, trace error and event logging for each of these components as well as the following:
MessageQ reports the success or failure of MessageQ operations by returning status values from calls to MessageQ API calls and by placing values in the delivery status field of PAMS Status Block (PSB). For detailed information on obtaining and using MessageQ status values refer to the BEA MessageQ Programmer's Guide .
MessageQ applications should test the return status and perform the proper handling based on the these values. To assist in the debugging of MessageQ application, MessageQ allows tracing to be turned on during application execution. Tracing writes messages to the destinations specified in DMQ$TRACE_OUTPUT (SYS$OUTPUT by default) that describe the execution path through MessageQ code.
Application developers can take advantage of this by defining the DMQ$DEBUG logical name to the value of ERROR, TRACE, or ALL before running their application. When set to ERROR, messages are written to the trace stream when MessagQ routines encounter an error.
When set to TRACE, messages are written as MessageQ executes its API functions. When set to ALL, MessageQ reports both trace and error information to the trace stream. ALL is the most commonly used setting for DMQ$TRACE_OUTPUT during application debugging. An example of trace output is as follows:
$ define dmq$debug all $ run DMQ$EXE:DMQ$TEST 0.0 DMQ$TEST - Bus 0099 9-JAN-1998 12:21:28.08 -- Using PAMS_ATTACH_Q -- Attach mode (Name=1,Number=2,Temp=3) [3]: Q type (PQ/SQ/MRQ) [PQ]: DmQ T 23:40.0 Time Stamp - 9-JAN-1998 12:23:40.04 DmQ T 23:40.0 Tracing/logging event flag number = 61 DmQ T 23:40.0 Entering PAMS_ATTACH_Q DmQ T 23:40.0 Entering PAMS_COMMON_ATTACH DmQ T 23:40.0 Entering INIT_PARAMS DmQ T 23:40.0 (req_q_num:0, &q_num:0X7FE0E8E8, q_type:-200, q_name:"" DmQ T 23:40.0 q_name_len:0, adver_scope:-202, mode:-212, tmo:600) DmQ T 23:40.0 Entering MAP_AND_CHECK_QUEUE DmQ T 23:40.0 Entering PAMS_ATTACH_TO_DMQ_ENV DmQ T 23:40.0 (Is_CS:0 Process name: "+_RTA1:" PID: 20A10C07, LOCKTMO:600) DmQ T 23:40.0 Using event flag 60 DmQ T 23:40.0 Using event flag 59 DmQ T 23:40.0 DMQ$BLOCKING_CONFIRM == NO DmQ T 23:40.0 Wakeup Mode = EXEC_MODE_AST DmQ T 23:40.0 Connecting to message bus #0099 group #00040 . . .
In addition to monitoring the execution of a MessageQ application, you may wish to observe the behaviour of a group while an application is running. The DMQ$MONITOR utility program allows you to observe several features of a running group. One often used feature is the continuous queue counter display. This allows you to monitor the sending, receiving, and pending queue counters for queues in a group.
You can use this display to determine if a queue is sending out the proper number of messages to the correct queue by looking at the send count for the application's queue and the receive count for the target queue. In addition to queue counters, the DMQ$MONITOR utility allows you to observe several other facets of a running MessageQ group. See Chapter 11 for more information on the DMQ$MONITOR utility.
This chapter describes the MessageQ for OpenVMS security features that allow system managers to:
There are two rights identifiers used by MessageQ . The identifiers are created during installation by the kit installation procedure if they do not already exist.