Sun Java System Message Queue 4.1 Administration Guide

Broker Services

Broker configuration properties can be divided into several categories, depending on the services or broker components they affect:

Connection services manage the physical connections between a broker and its clients that provide transport for incoming and outgoing messages.
Routing services route and deliver JMS payload messages, as well as control messages used by the message service to support reliable delivery.
Persistence services manage the writing and retrieval of data to and from persistent storage.
Security services authenticate users connecting to the broker and authorize their actions.
Monitoring services generate metric and diagnostic information about the broker’s performance.

The following sections describe each of these services and the properties you use to customize them for your particular needs.

Connection Services

Message brokers can offer various connection services supporting both application and administrative clients, using a variety of transport protocols. Broker configuration properties related to connection services are listed under Connection Properties.

Table 4–1 shows the available connection services, which are distinguished by two characteristics:

The service type specifies whether the service provides JMS message delivery (NORMAL) or Message Queue administration services ( ADMIN).
The protocol type specifies the underlying transport protocol.

Table 4–1 Message Queue Connection Services


Service Name	Service Type	Protocol Type
`jms`	`NORMAL`	TCP
`ssljms`	`NORMAL`	TLS (SSL-based security)
`httpjms`	`NORMAL`	HTTP
`httpsjms`	`NORMAL`	HTTPS (SSL-based security)
`admin`	`ADMIN`	TCP
`ssladmin`	`ADMIN`	TLS (SSL-based security)

By setting a broker’s imq.service.activelist property, you can configure it to run any or all of these connection services. The value of this property is a list of connection services to be activated when the broker is started up; if the property is not specified explicitly, the jms and admin services will be activated by default.

Each connection service also supports specific authentication and authorization features; see Security Services for more information.

Note –

There is also a special cluster connection service, used internally by the brokers within a cluster to exchange information about the cluster’s configuration and state. This service is not intended for use by clients communicating with a broker. See Chapter 8, Broker Clusters for more information about broker clusters.

Port Mapper

Each connection service is available at a particular port, specified by host name (or IP address) and port number. You can explicitly specify a static port number for a service or have the broker’s Port Mapper assign one dynamically. The Port Mapper itself resides at the broker’s primary port, which is normally located at the standard port number 7676. (If necessary, you can use the broker configuration property imq.portmapper.port to override this with a different port number.) By default, each connection service registers itself with the Port Mapper when it starts up. When a client creates a connection to the broker, the Message Queue client runtime first contacts the Port Mapper, requesting a port number for the desired connection service.

Alternatively, you can override the Port Mapper and explicitly assign a static port number to a connection service, using the imq.serviceName.protocolType. port configuration property (where serviceName and protocolType identify the specific connection service, as shown in Table 4–1). (Only the jms, ssljms, admin, and ssladmin connection services can be configured this way; the httpjms and httpsjms services use different configuration properties, described in Appendix C, HTTP/HTTPS Support) Static ports are generally used only in special situations, however, such as in making connections through a firewall (see Connecting Through a Firewall), and are not recommended for general use.

Note –

In cases where two or more hosts are available (such as when more than one network interface card is installed in a computer), you can use broker properties to specify which host the connection services should bind to. The imq.hostname property designates a single default host for all connection services; this can then be overridden, if necessary, with imq.serviceName. protocolType.hostname (for the jms, ssljms, admin, or ssladmin service) or imq.portmapper.hostname (for the Port Mapper itself).

When multiple Port Mapper requests are received concurrently, they are stored in an operating system backlog while awaiting action. The imq.portmapper.backlog property specifies the maximum number of such backlogged requests. When this limit is exceeded, any further requests will be rejected until the backlog is reduced.

Thread Pool Management

Each connection service is multithreaded, supporting multiple connections. The threads needed for these connections are maintained by the broker in a separate thread pool for each service. As threads are needed by a connection, they are added to the thread pool for the service supporting that connection.

The threading model you choose specifies whether threads are dedicated to a single connection or shared by multiple connections:

In the dedicated model, each connection to the broker requires two threads: one for incoming and one for outgoing messages. This limits the number of connections that can be supported, but provides higher performance.
In the shared model, connections are processed by a shared thread when sending or receiving messages. Because each connection does not require dedicated threads, this model increases the number of possible connections, but at the cost of lower performance because of the additional overhead needed for thread management.

The broker’s imq.serviceName. threadpool_model property specifies which of the two models to use for a given connection service. This property takes either of two string values: dedicated or shared. If you don’t set the property explicitly, dedicated is assumed by default.

You can also set the broker properties imq.serviceName. min_threads and imq.serviceName. max_threads to specify a minimum and maximum number of threads in a service’s thread pool. When the number of available threads exceeds the specified minimum threshold, Message Queue will shut down threads as they become free until the minimum is reached again, thereby saving on memory resources. Under heavy loads, the number of threads might increase until the pool’s maximum number is reached; at this point, new connections are rejected until a thread becomes available.

The shared threading model uses distributor threads to assign threads to active connections. The broker property imq.shared.connectionMonitor_limit specifies the maximum number of connections that can be monitored by a single distributor thread. The smaller the value of this property, the faster threads can be assigned to connections. The imq.ping.interval property specifies the time interval, in seconds, at which the broker will periodically test (“ping”) a connection to verify that it is still active, allowing connection failures to be detected preemptively before an attempted message transmission fails.

Routing Services

Once clients are connected to the broker, the routing and delivery of messages can proceed. In this phase, the broker is responsible for creating and managing different types of physical destinations, ensuring a smooth flow of messages, and using resources efficiently. You can use the broker configuration properties described under Routing Properties to manage these tasks in a way that suits your application’s needs.

The performance and stability of a broker depend on the system resources (such as memory) available and how efficiently they are utilized. You can set configuration properties to prevent the broker from becoming overwhelmed by incoming messages or running out of memory. These properties function at three different levels to keep the message service operating as resources become scarce:

Systemwide message limits apply collectively to all physical destinations on the system. These include the maximum number of messages held by a broker (imq.system.max_count) and the maximum total number of bytes occupied by such messages (imq.system.max_size). If either of these limits is reached, the broker will reject any new messages until the pending messages fall below the limit. There is also a limit on the maximum size of an individual message (imq.message.max_size) and a time interval at which expired messages are reclaimed (imq.message.expiration.interval).
Individual destination limits regulate the flow of messages to a specific physical destination. The configuration properties controlling these limits are described in Chapter 15, Physical Destination Property Reference. They include limits on the number and size of messages the destination will hold, the number of message producers and consumers that can be created for it, and the number of messages that can be batched together for delivery to the destination.

The destination can be configured to respond to memory limits by slowing down the delivery of message by message producers, by rejecting new incoming messages, or by throwing out the oldest or lowest-priority existing messages. Messages deleted from the destination in this way may optionally be moved to the dead message queue rather than discarded outright; the broker property imq.destination.DMQ.truncateBody controls whether the entire message body is saved in the dead message queue, or only the header and property data.

As a convenience during application development and testing, you can configure a message broker to create new physical destinations automatically whenever a message producer or consumer attempts to access a nonexistent destination. The broker properties summarized in Table 14–3 parallel the ones just described, but apply to such auto-created destinations instead of administratively created ones.
System memory thresholds define levels of memory usage at which the broker takes increasingly serious action to prevent memory overload. Four such usage levels are defined:
- Green: Plenty of memory is available.
- Yellow: Broker memory is beginning to run low.
- Orange: The broker is low on memory.
- Red: The broker is out of memory.
  
  The memory utilization percentages defining these levels are specified by the broker properties imq.green.threshold, imq.yellow.threshold , imq.orange.threshold, and imq.red.threshold , respectively; the default values are 0% for green, 80% for yellow, 90% for orange, and 98% for red.
  
  As memory usage advances from one level to the next, the broker responds progressively, first by swapping messages out of active memory into persistent storage and then by throttling back producers of nonpersistent messages, eventually stopping the flow of messages into the broker. (Both of these measures degrade broker performance.) The throttling back of message production is done by limiting the size of each batch delivered to the number of messages specified by the properties imq.resourceState .count, where resourceState is green , yellow, orange, or red , respectively.

The triggering of these system memory thresholds is a sign that systemwide and destination message limits are set too high. Because the memory thresholds cannot always catch potential memory overloads in time, you should not rely on them to control memory usage, but rather reconfigure the systemwide and destination limits to optimize memory resources.

Persistence Services

For a broker to recover in case of failure, it needs to re-create the state of its message delivery operations. To do this, the broker must save state information to a persistent data store. When the broker restarts, it uses the saved data to re-create destinations and durable subscriptions, recover persistent messages, roll back open transactions, and rebuild its routing table for undelivered messages. It can then resume message delivery.

An important use for persistent data stores is for providing high availability message service. In this model, two or more brokers are joined together in a broker cluster sharing the same persistent store. If one of the brokers should fail, another broker within the cluster can then take over ownership of its messages and see that they are delivered to their destinations without interruption of service. (See Chapter 8, Broker Clusters and the Message Queue Technical Overviewfor more information on clusters and how they operate.)

Message Queue supports both file-based and JDBC-based persistence modules (see Figure 4–1). File-based persistence uses individual files to store persistent data; JDBC-based persistence uses the Java Database Connectivity (JDBC) interface to connect the broker to a JDBC-compliant data store. While file-based persistence is generally faster than JDBC-based, some users prefer the redundancy and administrative control provided by a JDBC-compliant store. The broker configuration property imq.persist.store (see Table 14–4) specifies which of the two forms of persistence to use.

Figure 4–1 Persistent Data Storage

Diagram showing that persistence services use either
a flat file-based or a JDBC-based data store.

Note –

Changes in the file formats for both file-based and JDBC-based persistent data stores were introduced in Message Queue 3.7, with further JDBC changes in version 4.0. As a result of these changes, the persistent store version numbers were updated to 370 for file-based and 400 for JDBC-based stores. You can use the imqdbmgr query command (see Displaying Information About the Persistent Store) to determine the version number of your existing data store.

On first startup, the Message Queue Broker utility (imqbrokerd) will check for the presence of an older persistent store and automatically migrate it to the latest format:

File-based store versions 200 and 350 will be migrated to the version 370 format.
JDBC-based store versions 350 and 370 will be migrated to the version 400 format. (If you need to upgrade a version 200 store, you will need to step through an intermediate 3.5 or 3.6 release.)

This upgrade leaves the older copy of the persistent store intact, allowing you to roll back the upgrade if necessary. To do so, you can uninstall Message Queue 4.0 or 4.1 and reinstall the earlier version you were previously running. The older version’s message brokers will locate and use the older copy of the persistent store.

File-Based Persistence

By default, Message Queue uses a file-based persistent data store, in which individual files store persistent data such as messages, destinations, durable subscriptions, and transactions. Broker configuration properties related to file-based persistence are listed under File-Based Persistence Properties.

The file-based store is located in a directory identified by the name of the broker instance (instanceName) to which the data store belongs:

   …/instances/instanceName/fs370

(See Appendix A, Platform-Specific Locations of Message Queue Data for the location of the instances directory.) Each destination on the broker has its own subdirectory holding messages delivered to that destination.

Note –

Because the persistent data store can contain messages of a sensitive or proprietary nature, you should secure the …/instances/instanceName/fs370 directory against unauthorized access; see Securing Persistent Data.

All persistent data other than messages is stored in separate files: one file each for destinations, durable subscriptions, and transaction state information. Most messages are stored in a single file consisting of variable-size records. You can compact this file to alleviate fragmentation as messages are added and removed (see Managing Physical Destination Disk Utilization). In addition, messages above a certain threshold size are stored in their own individual files rather than in the variable-sized record file. You can configure this threshold size with the broker property imq.persist.file.message.max_record_size.

The broker maintains a file pool for these individual message files: instead of being deleted when it is no longer needed, a file is returned to the pool of free files in its destination directory so that it can later be reused for another message. The broker property imq.persist.file.destination.message.filepool.limit specifies the maximum number of files in the pool. When the number of individual message files for a destination exceeds this limit, files will be deleted when no longer needed instead of being returned to the pool.

When returning a file to the file pool, the broker can save time at the expense of storage space by simply tagging the file as available for reuse without deleting its previous contents. You can use the imq.persist.file.message.filepool.cleanratio broker property to specify the percentage of files in each destination’s file pool that should be maintained in a “clean” (empty) state rather than simply marked for reuse. The higher you set this value, the less space will be required for the file pool, but the more overhead will be needed to empty the contents of files when they are returned to the pool. If the broker’s imq.persist.file.message.cleanup property is true, all files in the pool will be emptied at broker shutdown, leaving them in a clean state; this conserves storage space but slows down the shutdown process.

In writing data to the persistent store, the operating system has some leeway in whether to write the data synchronously or “lazily” (asynchronously). Lazy storage can lead to data loss in the event of a system crash, if the broker believes the data to have been written to persistent storage when it has not. To ensure absolute reliability (at the expense of performance), you can require that all data be written synchronously by setting the broker property imq.persist.file.sync.enabled to true. In this case, the data is guaranteed to be available when the system comes back up after a crash, and the broker can reliably resume operation. Note, however, that although the data is not lost, it is not available to any other broker in a cluster, since clustered brokers do not currently share data.

JDBC-Based Persistence

Instead of using file-based persistence, you can set up a broker to access any data store accessible through a JDBC-compliant driver. This involves setting the appropriate JDBC-related broker configuration properties and using the Database Manager utility (imqdbmgr) to create the proper database schema. See Configuring a JDBC-Based Store for specifics.

The properties for configuring a broker to use a JDBC database are listed in Table 14–6. You can specify these properties either in the instance configuration file (config.properties) of each broker instance or by using the -D command line option to the Broker utility (imqbrokerd) or the Database Manager utility (imqdbmgr).

The imq.persist.jdbc.dbVendor property identifies the database vendor being used for the persistent data store; all of the remaining properties are qualified by this vendor name. The imq.persist.jdbc.vendorName.driver property gives the Java class name of the JDBC driver to use in connecting to the database. There are also properties specifying the URLs for connecting to an existing database (imq.persist.jdbc.vendorName.opendburl), creating a new database (imq.persist.jdbc.vendorName.createdburl), and closing a database connection (imq.persist.jdbc.vendorName.closedburl).

The imq.persist.jdbc.vendorName.user and imq.persist.jdbc.vendorName.password properties give the user name and password for accessing the database, if required; imq.persist.jdbc.vendorName.needpassword is a boolean flag specifying whether a password is needed. For security reasons, the password should be specified only in a password file designated with the -passfile command line option; if no such password file is specified, the imqbrokerd and imqdbmgr commands will prompt for the password interactively. Similarly, the user name can be supplied from the command line using the -dbuser option to the imqbrokerd command or the -u option to imqdbmgr. If any additional, vendor-specific properties are required, you can set them by using broker properties of the form imq.persist.jdbc.vendorName.property.propName.

In a JDBC database shared by multiple broker instances, the configuration property imq.brokerid specifies a unique instance identifier for each, to be appended to the names of database tables. (This is usually unnecessary for an embedded database, which stores data for only one broker instance.)

Besides setting all of the broker’s needed JDBC configuration properties, you must also install your JDBC driver’s .jar file in the appropriate directory location, depending on your operating-system platform (as listed in Appendix A, Platform-Specific Locations of Message Queue Data) and then execute the Database Manager command

   imqdbmgr create tbl

to create the database schema for the persistent data store.

Security Services

Message Queue provides security services for user access control (authentication and authorization) and for encryption:

Authentication ensures that only verified users can establish a connection to a broker.
Authorization specifies which users or groups have the right to access resources and to perform specific operations.
Encryption protects messages from being tampered with during delivery over a connection.

As a Message Queue administrator, you are responsible for setting up the information the broker needs to authenticate users and authorize their actions. The broker properties pertaining to security services are listed under Security Properties. The boolean property imq.accesscontrol.enabled acts as a master switch that controls whether access control is applied on a brokerwide basis; for finer control, you can override this setting for a particular connection service by setting the imq.serviceName .accesscontrol.enabled property, where serviceName is the name of the connection service, as shown in Table 4–1: for example, imq.httpjms.accesscontrol.enabled.

Figure 4–2 shows the components needed by the broker to provide authentication and authorization services. These services depend on a user repository containing information about the users of the messaging system: their names, passwords, and group memberships. In addition, to authorize specific operations for a user or group, the broker consults an access control file that specifies which operations a user or group can perform. You can designate a single access control file for the broker as a whole, using the configuration property imq.accesscontrol.file.filename, or for a single connection service with imq.serviceName. accesscontrol.file.filename.

Figure 4–2 Security Support

Diagram showing broker's security services using both
a user repository and an access control file.

As Figure 4–2 shows, you can store user data in a flat-file user repository that is provided with the Message Queue service or you can plug in a preexisting Lightweight Directory Access Protocol (LDAP) repository:

If you choose a flat-file repository, you must use the Message Queue User Manager utility (imqusermgr) to manage the repository. This option is built-in and easy to use.
If you want to use an existing LDAP server, you use the tools provided by the LDAP vendor to populate and manage the user repository. You must also set properties in the broker’s instance configuration file to enable the broker to query the LDAP server for information about users and groups.

The broker’s imq.authentication.basic.user_repository property specifies which type of repository to use. In general, an LDAP repository is preferable if scalability is important or if you need the repository to be shared by different brokers (if you are using broker clusters, for instance). See User Authentication for more information on setting up a flat-file or LDAP user repository.

Authentication

A client requesting a connection to a broker must supply a user name and password, which the broker compares with those stored in the user repository. Passwords transmitted from client to broker are encoded using either base-64 encoding (for flat-file repositories) or message digest (MD5) hashing (for LDAP repositories). The choice is controlled by the imq.authentication.type property for the broker as a whole, or by imq.serviceName. authentication.type for a specific connection service. The imq.authentication.client.response.timeout property sets a timeout interval for authentication requests.

As described under Password Files, you can choose to put your passwords in a password file instead of being prompted for them interactively. The boolean broker property imq.passfile.enabled controls this option. If this property is true, the imq.passfile.dirpath and imq.passfile.name properties give the directory path and file name for the password file. The imq.imqcmd.password property (which can be embedded in the password file) specifies the password for authenticating an administrative user to use the Command utility (imqcmd) for managing brokers, connection services, connections, physical destinations, durable subscriptions, and transactions.

If you are using an LDAP-based user repository, there are a whole range of broker properties available for configuring various aspects of the LDAP lookup. The address (host name and port number) of the LDAP server itself is specified by imq.user_repository.ldap.server. The imq.user_repository.ldap.principal property gives the distinguished name for binding to the LDAP repository, while imq.user_repository.ldap.password supplies the associated password. Other properties specify the directory bases and optional JNDI filters for individual user and group searches, the provider-specific attribute identifiers for user and group names, and so forth; see Security Properties for details.

Authorization

Once authenticated, a user can be authorized to perform various Message Queue-related activities. As a Message Queue administrator, you can define user groups and assign individual users membership in them. The default access control file explicitly refers to only one group, admin (see User Groups and Status). A user in this group has connection permission for the admin connection service, which allows the user to perform administrative functions such as creating destinations and monitoring and controlling a broker. A user in any other group that you define cannot, by default, get an admin service connection.

When a user attempts to perform an operation, the broker checks the user’s name and group membership (from the user repository) against those specified for access to that operation (in the access control file). The access control file specifies permissions to users or groups for the following operations:

Connecting to a broker
Accessing destinations: creating a consumer, a producer, or a queue browser for any given destination or for all destinations
Auto-creating destinations

Encryption

To encrypt messages sent between clients and broker, you need to use a connection service based on the Secure Socket Layer (SSL) standard. SSL provides security at the connection level by establishing an encrypted connection between an SSL-enabled broker and client.

To use an SSL-based Message Queue connection service, you generate a public/private key pair using the Message Queue Key Tool utility (imqkeytool). This utility embeds the public key in a self-signed certificate and places it in a Message Queue key store. The key store is itself password-protected; to unlock it, you must provide a key store password at startup time, specified by the imq.keystore.password property. Once the key store is unlocked, a broker can pass the certificate to any client requesting a connection. The client then uses the certificate to set up an encrypted connection to the broker.

Monitoring Services

The broker includes components for monitoring and diagnosing application and broker performance. These include the following:

Components that generate data, a Metrics Generator and broker code that logs events
A Logger component that writes out information to a number of output channels
A Metrics Message Producer that sends JMS messages containing metric information to topic destinations for consumption by JMS monitoring clients

The general scheme is illustrated in Figure 4–3. Broker properties for configuring the monitoring services are listed under Monitoring Properties.

Figure 4–3 Monitoring Support

Diagram showing inputs to Logger, error levels, and output
channels. Figure explained in text.

Metrics Generator

The Metrics Generator provides information about broker activity, such as message flow in and out of the broker, the number of messages in broker memory and the memory they consume, the number of open connections, and the number of threads being used. The boolean broker property imq.metrics.enabled controls whether such information is logged; imq.metrics.interval specifies how often.

Logger

The Logger takes information generated by broker code and the Metrics Generator and writes that information to standard output (the console), to a log file, and, on Solaris platforms, to the syslog daemon process in case of errors. The log file to use is identified by the imq.log.file.dirpath and imq.log.file.filename broker properties; imq.log.console.stream specifies whether console output is directed to stdout or stderr.

The imq.log.level property controls the categories of metric information that the Logger gathers: ERROR, WARNING, or INFO. Each level includes those above it, so if you specify, for example, WARNING as the logging level, error messages will be logged as well. The imq.log.console.output and imq.log.file.output properties control which of the specified categories will be written to the console and the log file, respectively. In this case, however, the categories do not include those above them; so if you want, for instance, both errors and warnings written to the log file and informational messages to the console, you must explicitly set imq.log.file.output to ERROR|WARNING and imq.log.console.output to INFO. On Solaris platforms another property, imq.log.syslog.output, specifies the categories of metric information to be written to the syslog daemon. There is also an imq.destination.logDeadMsgs property that specifies whether to log when dead messages are discarded or moved to the dead message queue.

In the case of a log file, you can specify the point at which the file is closed and output is rolled over to a new file. Once the log file reaches a specified size (imq.log.file.rolloverbytes) or age (imq.log.file.rolloversecs), it is saved and a new log file created.

See Monitoring Properties for additional broker properties related to logging, and Configuring and Using Broker Logging for further details about how to configure the Logger and how to use it to obtain performance information.

Metrics Message Producer

The Metrics Message Producer receives information from the Metrics Generator at regular intervals and writes the information into metrics messages, which it then sends to one of a number of metric topic destinations, depending on the type of metric information contained in the message (see Table 4–2). Message Queue clients subscribed to these metric topic destinations can consume the messages and process the metric data they contain. This allows developers to create custom monitoring tools to support messaging applications. For details of the metric quantities reported in each type of metrics message, see the Message Queue Developer’s Guide for Java Clients.

Table 4–2 Metric Topic Destinations


Topic Name	Type of Metric Information
`mq.metrics.broker`	Broker metrics
`mq.metrics.jvm`	Java Virtual Machine metrics
`mq.metrics.destination_list`	List of destinations and their types
`mq.metrics.destination.queue.queueName`	Destination metrics for specified queue
`mq.metrics.destination.topic.topicName`	Destination metrics for specified topic

The broker properties imq.metrics.topic.enabled and imq.metrics.topic.interval control, respectively, whether messages are sent to metric topic destinations and how often. The imq.metrics.topic.timetolive and imq.metrics.topic.persist properties specify the lifetime of such messages and whether they are persistent.

Besides the information contained in the body of a metrics message, the header of each message includes properties that provide the following additional information:

The message type
The address (host name and port number) of the broker that sent the message
The time the metric sample was taken

These properties are useful to client applications that process metrics messages of different types or from different brokers.