Sun Java System Message Queue 4.1 Administration Guide

Chapter 10 Monitoring Broker Operations

This chapter describes the tools you can use to monitor a broker and how you can get metrics data. The chapter has the following sections:

Reference information on specific metrics is available in Chapter 18, Metrics Reference

Introduction to Monitoring Tools

There are four monitoring interfaces for Message Queue information: log files, interactive monitoring commands, the Sun JavaTM Enterprise System Monitoring Framework (JESMF), and a client API that can obtain metrics. Each has its advantages and disadvantages, as follows:

Table 10–1 compares the different tools.

Table 10–1 Benefits and Limitations of Metrics Monitoring Tools

Metrics Monitoring Tool 

Benefits 

Limitations 

Log files 

  • Regular sampling

  • Creates a historical record

  • Local monitoring only

  • Data format difficult to read; no parsing tools

  • Need to configure broker properties; must shut down and restart broker to take effect

  • Broker metrics only; no destination or connection service metrics

  • No flexibility in selection of data

  • Same reporting interval for all metrics data; cannot be changed on the fly

  • Possible performance penalty if interval set too short

Interactive monitoring commands 

  • Remote monitoring

  • Convenient for spot-checking

  • Data presented in easy-to-read tabular format

  • Easy to select specific data of interest

  • Reporting interval set in command option; can be changed on the fly

  • Difficult to analyze data programmatically

  • No single command gets all data

  • No historical record; difficult to see historical trends

JES Monitoring Framework

  • Web-based graphical interface

  • Data presented in easy-to-read format

  • Common interface shared with other JES components

  • No performance penalty; pulls data from broker’s existing data monitoring infrastructure

  • Limited subset of data available

  • Data cannot be analyzed programmatically

  • No historical record; difficult to see historical trends

Client API

  • Remote monitoring

  • Data can be analyzed programmatically and presented in any format

  • Easy to select specific data of interest

  • Need to write your own monitoring client

  • Need to configure broker properties; must shut down and restart broker to take effect

  • Same reporting interval for all metrics data; cannot be changed on the fly

In addition to the differences shown in the table, each tool gathers a somewhat different subset of the metrics information generated by the broker. For information on which metrics data is gathered by each monitoring tool, see Chapter 18, Metrics Reference.

Configuring and Using Broker Logging

The Message Queue Logger takes information generated by broker code, a debugger, and a metrics generator and writes that information to a number of output channels: to standard output (the console), to a log file, and, on Solaris™ operating systems, to the syslog daemon process. You can specify the type of information gathered by the Logger as well as the type written to each of the output channels. In particular, you can specify that you want metrics information written out to a log file.

This section describes the default logging configuration for the broker and explains how to change that configuration to redirect log information to alternative output channels, change log file rollover criteria, or send metrics data to a log file.

Log Message Format

A logged message consists of a time stamp, a message code, and the message itself. The volume of information included varies with the logging level you have set. The broker supports three logging levels: ERROR, WARNING , and INFO (see Table 10–2). Each level includes those above it (for example, WARNING includes ERROR).

Table 10–2 Logging Levels

Logging Level 

Description 

ERROR

Serious problems that could cause system failure 

WARNING

Conditions that should be heeded but will not cause system failure 

INFO

Metrics and other informational messages 

The default logging level is INFO, so messages at all three levels are logged by default. The following is an example of an INFO message:

   [13/Sep/2000:16:13:36 PDT] [B1004]: Starting the broker service
   using tcp [25374,100] with min threads 50 and max threads of 500

You can change the time zone used in the time stamp by setting the broker configuration property imq.log.timezone (see Table 14–9).

Default Logging Configuration

A broker is automatically configured to save log output to a set of rolling log files. The log files are located in a directory identified by the instance name of the associated broker (see Appendix A, Platform-Specific Locations of Message Queue Data):

   …/instances/instanceName/log

Note –

For a broker whose life cycle is controlled by the Application Server, the log files are located in a subdirectory of the domain directory for the domain for which the broker was started:

   …/appServerDomainDir/imq/instances/imqbroker/log

The log files are simple text files. The system maintains nine backup files named as follows, from earliest to latest:

   log.txt
   log_1.txt
   log_2.txt
         log_9.txt

By default, the log files are rolled over once a week. You can change this rollover interval, or the location or names of the log files, by setting appropriate configuration properties:

See Table 14–9 for further information on these properties.

Changing the Logging Configuration

Log-related properties are described in Table 14–9.

ProcedureTo Change the Logger Configuration for a Broker

  1. Set the logging level.

  2. Set the output channel (file, console, or both) for one or more logging categories.

  3. If you log output to a file, configure the rollover criteria for the file.

    You complete these steps by setting Logger properties. You can do this in one of two ways:

    • Change or add Logger properties in the config.properties file for a broker before you start the broker.

    • Specify Logger command line options in the imqbrokerd command that starts the broker. You can also use the broker option -D to change Logger properties (or any broker property).

    Options passed on the command line override properties specified in the broker instance configuration files. The following imqbrokerd options affect logging:

    -metrics interval

    Logging interval for broker metrics, in seconds

    -loglevel level

    Logging level (ERROR, WARNING, INFO, or NONE)

    -silent

    Silent mode (no logging to console)

    -tty

    Log all messages to console

    The following sections describe how you can change the default configuration in order to do the following:

    • Change the output channel (the destination of log messages)

    • Change rollover criteria

Changing the Output Channel

By default, error and warning messages are displayed on the terminal as well as being logged to a log file. (On Solaris, error messages are also written to the system’s syslog daemon.)

You can change the output channel for log messages in the following ways:


Note –

Before changing Logger output channels, you must make sure that logging is set at a level that supports the information you are mapping to the output channel. For example, if you set the logging level to ERROR and then set the imq.log.console.output property to WARNING, no messages will be logged because you have not enabled the logging of WARNING messages.


Changing Log File Rollover Criteria

There are two criteria for rolling over log files: time and size. The default is to use a time criteria and roll over files every seven days.

If you set both the time-related and the size-related rollover properties, the first limit reached will trigger the rollover. As noted before, the broker maintains up to nine rollover files.

You can set or change the log file rollover properties when a broker is running. To set these properties, use the imqcmd update bkr command.

Sending Metrics Data to Log Files

This section describes the procedure for using broker log files to report metrics information. For general information on configuring the Logger, see Configuring and Using Broker Logging.

ProcedureTo Use Log Files to Report Metrics Information

  1. Configure the broker’s metrics generation capability:

    1. Confirm imq.metrics.enabled=true

      Generation of metrics for logging is turned on by default.

    2. Set the metrics generation interval to a convenient number of seconds.

      imq.metrics.interval=interval

      This value can be set in the config.properties file or using the -metrics interval command line option when starting up the broker.

  2. Confirm that the Logger gathers metrics information:


    imq.log.level=INFO

    This is the default value. This value can be set in the config.properties file or using the -loglevel level command line option when starting up the broker.

  3. Confirm that the Logger is set to write metrics information to the log file:


    imq.log.file.output=INFO

    This is the default value. It can be set in the config.properties file.

  4. Start up the broker.

    The following shows sample broker metrics output to the log file:


    [21/Jul/2004:11:21:18 PDT]
    Connections: 0    JVM Heap: 8323072 bytes (7226576 free) Threads: 0 (14-1010)
          In: 0 msgs (0bytes) 0 pkts (0 bytes)
         Out: 0 msgs (0bytes) 0 pkts (0 bytes)
     Rate In: 0 msgs/sec (0 bytes/sec) 0 pkts/sec (0 bytes/sec)
    Rate Out: 0 msgs/sec (0 bytes/sec) 0 pkts/sec (0 bytes/sec)

    For reference information about metrics data, see Chapter 18, Metrics Reference

Logging Dead Messages

You can monitor physical destinations by enabling dead message logging for a broker. You can log dead messages whether or not you are using a dead message queue.

If you enable dead message logging, the broker logs the following types of events:

If a dead message queue is in use, logging also includes the following types of events:

The following is an example of the log format for dead messages:


[29/Mar/2006:15:35:39 PST] [B1147]: Message 8-129.145.180.87(e7:6b:dd:5d:98:aa)-
35251-1143675279400 from destination Q:q0 has been placed on the DMQ because 
[B0053]: Message on destination Q:q0 Expired: expiration time 1143675279402, 
arrival time 1143675279401, JMSTimestamp 1143675279400

Dead message logging is disabled by default. To enable it, set the broker attribute imq.destination.logDeadMsgs.

Displaying Metrics Interactively

A Message Queue broker can report metrics of the following types:

The imqcmd command can obtain metrics information for the broker as a whole, for individual connection services, and for individual physical destinations. To obtain metrics data, you generally use the metrics subcommand of imqcmd. Metrics data is written at an interval you specify, or the number of times you specify, to the console screen.

You can also use the query subcommand to view similar data that also includes configuration information. See imqcmd query for more information.

imqcmd metrics

The syntax and options of imqcmd metrics are shown in Table 10–3 and Table 10–4, respectively.

Table 10–3 imqcmd metrics Subcommand Syntax

Subcommand Syntax 

Metrics Data Provided 

metrics bkr
   [-b hostName:portNumber]
   [-m metricType]
   [-int interval]
   [-msp numSamples]

Displays broker metrics for the default broker or a broker at the specified host and port. 

metrics svc -n serviceName
   [-b hostName:portNumber]
   [-m metricType]
   [-int interval]
   [-msp numSamples]

Displays metrics for the specified service on the default broker or on a broker at the specified host and port. 

metrics dst -t destType
   -n destName
   [-b hostName:portNumber]
   [-m metricType]
   [-int interval]
   [-msp numSamples]

Displays metrics information for the physical destination of the specified type and name.

Table 10–4 imqcmd metrics Subcommand Options

Subcommand Options 

Description 

-b hostName: portNumber

Specifies the hostname and port of the broker for which metrics data is reported. The default is localhost:7676.

-int interval

Specifies the interval (in seconds) at which to display the metrics. The default is 5 seconds. 

-m metricType

Specifies the type of metric to display: 

ttl Displays metrics on messages and packets flowing into and out of the broker, service, or destination (default metric type).

rts Displays metrics on rate of flow of messages and packets into and out of the broker, connection service, or destination (per second).

cxn Displays connections, virtual memory heap, and threads (brokers and connection services only).

con Displays consumer-related metrics (destinations only).

dsk Displays disk usage metrics (destinations only).

-msp numSamples

Specifies the number of samples displayed in the output. The default is an unlimited number (infinite). 

-n destName

Specifies the name of the physical destination (if any) for which metrics data is reported. There is no default. 

-n serviceName

Specifies the connection service (if any) for which metrics data is reported. There is no default. 

-t destType

Specifies the type (queue or topic) of the physical destination (if any) for which metrics data is reported. There is no default. 

Using the metrics Subcommand to Display Metrics Data

This section describes the procedure for using the metrics subcommand to report metrics information.

ProcedureTo Use the metrics Subcommand

  1. Start the broker for which metrics information is desired.

    See Starting Brokers.

  2. Issue the appropriate imqcmd metrics subcommand and options as shown in Table 10–3 and Table 10–4.

Metrics Outputs: imqcmd metrics

This section contains examples of output for the imqcmd metrics subcommand. The examples show brokerwide, connection service, and physical destination metrics.

Brokerwide Metrics

To get the rate of message and packet flow into and out of the broker at 10 second intervals, use the metrics bkr subcommand:

imqcmd metrics bkr -m rts -int 10 -u admin

This command produces output similar to the following (see data descriptions in Table 18–2):


--------------------------------------------------------
 Msgs/sec   Msg Bytes/sec   Pkts/sec    Pkt Bytes/sec   
 In   Out     In      Out     In   Out     In      Out  
--------------------------------------------------------
 0     0      27      56      0     0      38      66   
 10    0     7365     56      10    10    7457    1132  
 0     0      27      56      0     0      38      73   
 0     10     27     7402     10    20    1400    8459  
 0     0      27      56      0     0      38      73   

Connection Service Metrics

To get cumulative totals for messages and packets handled by the jms connection service, use the metrics svc subcommand:

imqcmd metrics svc -n jms -m ttl -u admin

This command produces output similar to the following (see data descriptions in Table 18–3):


-------------------------------------------------
  Msgs      Msg Bytes      Pkts      Pkt Bytes     
In   Out    In     Out   In   Out    In     Out  
-------------------------------------------------
164  100  120704  73600  282  383  135967  102127
657  100  483552  73600  775  876  498815  149948

Physical Destination Metrics

To get metrics information about a physical destination, use the metrics dst subcommand:

imqcmd metrics dst -t q -n XQueue -m ttl -u admin

This command produces output similar to the following (see data descriptions in Table 18–4):


-----------------------------------------------------------------------------
  Msgs      Msg Bytes         Msg Count         Total Msg Bytes (k)     Largest
In   Out    In     Out    Current  Peak  Avg  Current  Peak     Avg    Msg (k)
-----------------------------------------------------------------------------
200  200  147200  147200     0     200    0      0      143      71        0  
300  200  220800  147200    100    200   10     71      143      64        0  
300  300  220800  220800     0     200    0      0      143      59        0  

To get information about a physical destination’s consumers, use the following metrics dst subcommand:

imqcmd metrics dst -t q -n SimpleQueue -m con -u admin

This command produces output similar to the following (see data descriptions in Table 18–4):


------------------------------------------------------------------
  Active Consumers         Backup Consumers         Msg Count
Current  Peak  Avg      Current  Peak    Avg    Current  Peak  Avg
------------------------------------------------------------------
   1       1      0        0       0      0       944    1000  525

imqcmd query

The syntax and options of imqcmd query are shown in Table 10–5 along with a description of the metrics data provided by the command.

Table 10–5 imqcmd query Subcommand Syntax

Subcommand Syntax 

Metrics Data Provided 


query bkr
   [-b hostName: portNumber]

Information on the current number of messages and message bytes stored in broker memory and persistent store (see Viewing Broker Information).

or 

 

query svc -n serviceName
  [-b  hostName:portNumber]

Information on the current number of allocated threads and number of connections for a specified connection service (see Viewing Connection Service Information).

or 

 

query dst -t destType
  -n destName
  [-b hostName:portNumber]

Information on the current number of producers, active and backup consumers, and messages and message bytes stored in memory and persistent store for a specified destination (see Viewing Physical Destination Information).


Note –

Because of the limited metrics data provided by imqcmd query , this tool is not represented in the tables presented in Chapter 18, Metrics Reference.


Using the JES Monitoring Framework

Message Queue supports the Sun Java Enterprise System Monitoring Framework (JESMF), which allows Java Enterprise System components to be monitored using a common graphical interface. This interface is implemented by a Web-based console called the Sun Java System Monitoring Console. Administrators can use the Monitoring Console to view performance statistics, create rules for automatic monitoring, and acknowledge alarms. If you are running Message Queue along with other JES components, you may find it more convenient to use a single interface to manage all of them.

The JES Monitoring Framework defines a common data model, the Common Monitoring Model (CMM), to be used by all JES component products. This model enables a centralized and uniform view of all JES components. Message Queue exposes the following objects through the Common Monitoring Model:

Each of these objects is mapped to a CMM object whose attributes can be monitored using the JES Monitoring Console. The reference tables in Chapter 19, JES Monitoring Framework Reference identify those attributes that are available for JESMF monitoring. For detailed information about the mapping of Message Queue objects to CMM objects, see the Sun Java Enterprise System Monitoring Guide.

To enable JESMF monitoring, you must do the following:

  1. Install and configure all the components in your deployment (Message Queue and other components) according to instructions given in the Sun Java Enterprise System Installation Guide.

  2. Enable and configure the Monitoring Framework for all of your monitored components, as described in the Sun Java Enterprise System Monitoring Guide.

  3. Install the Monitoring Console on a separate host, start the master agent, and then start the Web server, as described in the Sun Java Enterprise System Monitoring Guide.

Using the JES Monitoring Framework will not affect broker performance, because all the work of gathering metrics is done by the Monitoring Framework, which pulls data from the broker’s existing data monitoring infrastructure.

Writing an Application to Monitor Brokers

Message Queue provides a metrics monitoring capability by which the broker can write metrics data into JMS messages, which it then sends to one of a number of metrics topic destinations, depending on the type of metrics information contained in the message.

You can access this metrics information by writing a client application that subscribes to the metrics topic destinations, consumes the messages in these destinations, and processes the metrics information contained in the messages.

There are five metrics topic destinations, whose names are shown in Table 10–6, along with the type of metrics messages delivered to each destination.

Table 10–6 Metrics Topic Destinations

Topic Name 

Description

mq.metrics.broker

Broker metrics

mq.metrics.jvm

Java Virtual Machine metrics

mq.metrics.destination_list

List of destinations and their types

mq.metrics.destination.queue.queueName

Destination metrics for queue queueName

mq.metrics.destination.topic.topicName

Destination metrics for topic topicName

Setting Up Message-Based Monitoring

This section describes the procedure for using the message-based monitoring capability to gather metrics information. The procedure includes both client development and administration tasks.

ProcedureTo Set Up Message-based Monitoring

  1. Write a metrics monitoring client.

    See the Message Queue Developer’s Guide for Java Clients for instructions on programming clients that subscribe to metrics topic destinations, consume metrics messages, and extract the metrics data from these messages.

  2. Configure the broker’s Metrics Message Producer by setting broker property values in the config.properties file:

    1. Enable metrics message production.

      Set imq.metrics.topic.enabled=true

      The default value is true.

    2. Set the interval (in seconds) at which metrics messages are generated.

      Set imq.metrics.topic.interval=interval .

      The default is 60 seconds.

    3. Specify whether you want metrics messages to be persistent (that is, whether they will survive a broker failure).

      Set imq.metrics.topic.persist .

      The default is false.

    4. Specify how long you want metrics messages to remain in their respective destinations before being deleted.

      Set imq.metrics.topic.timetolive .

      The default value is 300 seconds.

  3. Set any access control you desire on metrics topic destinations.

    See the discussion in Security and Access Considerations below.

  4. Start up your metrics monitoring client.

    When consumers subscribe to a metrics topic, the metrics topic destination will automatically be created. Once a metrics topic has been created, the broker’s metrics message producer will begin sending metrics messages to the metrics topic.

Security and Access Considerations

There are two reasons to restrict access to metrics topic destinations:

Because of these considerations, it is advisable to restrict access to metrics topic destinations.

Monitoring clients are subject to the same authentication and authorization control as any other client. Only users maintained in the Message Queue user repository are allowed to connect to the broker.

You can provide additional protections by restricting access to specific metrics topic destinations through an access control file, as described in User Authorization.

For example, the following entries in an accesscontrol.properties file will deny access to the mq.metrics.broker metrics topic to everyone except user1 and user 2.


topic.mq.metrics.broker.consume.deny.user=*
topic.mq.metrics.broker.consume.allow.user=user1,user2

The following entries will only allow users user3 to monitor topic t1.


topic.mq.metrics.destination.topic.t1.consume.deny.user=*
topic.mq.metrics.destination.topic.t1.consume.allow.user=user3

Depending on the sensitivity of metrics data, you can also connect your metrics monitoring client to a broker using an encrypted connection. For information on using encrypted connections, see Message Encryption.

Metrics Outputs: Metrics Messages

The metrics data outputs you get using the message-based monitoring API is a function of the metrics monitoring client you write. You are limited only by the data provided by the metrics generator in the broker. For a complete list of this data, see Chapter 18, Metrics Reference.