test

Sun Java System Message Queue 4.3 Administration Guide

Chapter 10 Configuring and Managing Broker Clusters

Message Queue supports the use of broker clusters: groups of brokers working together to provide message delivery services to clients. Clusters enable a message service to scale its operations to meet an increasing volume of message traffic by distributing client connections among multiple brokers.

In addition, clusters provide for message service availability. In the case of a conventional cluster, if a broker fails, clients connected to that broker can reconnect to another broker in the cluster and continue producing and consuming messages. In the case of an enhanced cluster, if a broker fails, clients connected to that broker reconnect to a failover broker that takes over the pending work of the failed broker, delivering messages without interruption of service.

See the Chapter 4, Broker Clusters, in Sun Java System Message Queue 4.3 Technical Overview for a description of conventional and enhanced broker clusters and how they operate.

This chapter describes how to configure and manage both conventional and enhanced broker clusters:

Configuring Broker Clusters

You create a broker cluster by specifying cluster configuration properties for each of its member brokers. Except where noted in this chapter, cluster configuration properties must be set to the same value for each broker in a cluster. This section introduces these properties and the use of a cluster configuration file to specify them:

The Cluster Configuration File

Like all broker properties, cluster configuration properties can be set individually for each broker in a cluster, either in its instance configuration file (config.properties) or by using the -D option on the command line when you start the broker. However, except where noted in this chapter, each cluster configuration property must be set to the same value for each broker in a cluster.

For example, to specify the transport protocol for the cluster connection service, you can include the following property in the instance configuration file for each broker in the cluster: imq.cluster.transport=ssl. If you need to change the value of this property, you must change its value in the instance configuration file for every broker in the cluster.

For consistency and ease of maintenance, it is generally more convenient to collect all of the common cluster configuration properties into a central cluster configuration file that all of the individual brokers in a cluster reference. Using a cluster configuration file prevents the settings from getting out of synch and ensures that all brokers in a cluster use the same, consistent cluster configuration information.

When using a cluster configuration file, each broker’s instance configuration file must point to the location of the cluster configuration file by setting the imq.cluster.url property. For example,

imq.cluster.url=file:/home/cluster.properties

Note –

A cluster configuration file can also include broker properties that are not used specifically for cluster configuration. For example, you can place any broker property in the cluster configuration file that has the same value for all brokers in a cluster. For more information, see Connecting Brokers into a Conventional Cluster

Cluster Configuration Properties

This section reviews the most important cluster configuration properties, grouped into the following categories:

A complete list of cluster configuration properties can be found in Table 16–11

Cluster Connection Service Properties

The following properties are used to configure the cluster connection service used for internal communication between brokers in the cluster. These properties are used by both conventional and enhanced clusters.

Conventional Broker Cluster Properties

The following properties, in addition to those listed in Cluster Connection Service Properties, are used to configure conventional clusters:

Enhanced Broker Cluster Properties

Enhanced broker clusters, which share a JDBC-based data store, require more configuration than do conventional broker clusters. In addition to the properties listed in Cluster Connection Service Properties, the following categories of properties are used to configure an enhanced cluster:

Enhanced Clusters: General Configuration Properties

Enhanced Clusters: JDBC Configuration Properties

The persistent data store for an enhanced cluster is maintained on a highly-available JDBC database.

The highly-availabile database may be Sun’s MySQL Cluster Edition or High Availability Session Store (HADB), or it may be an open-source or third-party product such as Oracle Corporation’s Real Application Clusters (RAC). As described in JDBC-Based Persistence Properties, the imq.persist.jdbc.dbVendor broker property specifies the name of the database vendor, and all of the remaining JDBC-related properties are qualified with this vendor name.

The JDBC-related properties are discussed under JDBC-Based Persistence Properties and summarized in Table 16–6. See the example configurations for MySQL and HADB in Example 8–1 and Example 8–2, respectively.

Note –

In setting JDBC-related properties for an enhanced cluster, note the following vendor-specific issues:

Enhanced Clusters: Failure Detection Properties

The following configuration properties (listed in Table 16–11) specify the parameters for the exchange of heartbeat and status information within an enhanced cluster:

Smaller values for these heartbeat and monitoring intervals will result in quicker reaction to broker failure, but at the cost of reduced performance and increased likelihood of false suspicions and erroneous failure detection.

Displaying a Cluster Configuration

To display information about a cluster’s configuration, use the Command utility’s list bkr subcommand: 

   imqcmd list bkr

This lists the current state of all brokers included in the cluster to which a given broker belongs. The broker states are described in the following table:

Table 10–1 Broker States

Broker State 

Meaning 

OPERATING

Broker is operating 

TAKEOVER_STARTED

For enhanced clusters, broker has begun taking over persistent data store from another broker

TAKEOVER_COMPLETE

For enhanced clusters, broker has finished taking over persistent data store from another broker 

TAKEOVER_FAILED

For enhanced clusters, attempted takeover has failed 

QUIESCE_STARTED

Broker has begun quiescing 

QUIESCE_COMPLETE

Broker has finished quiescing 

SHUTDOWN_STARTED

Broker has begun shutting down 

BROKER_DOWN

Broker is down 

UNKNOWN

Broker state unknown 

The results of the imqcmd list bkr command are shown in Example 10–1 (for a conventional cluster) and Example 10–2 (for an enhanced cluster).

Example 10–1 Configuration Listing for a Conventional Cluster


Listing all the brokers in the cluster that the following broker is a member of:

-------------------------
Host         Primary Port
-------------------------
localHost    7676

Cluster Is Highly Available             False

-------------------------
Address         State
---------------------
whippet:7676    OPERATING
greyhound:7676  OPERATING

Example 10–2 Configuration Listing for an Enhanced Cluster


Listing all the brokers in the cluster that the following broker is a member of:

----------------------------------------------
Host         Primary Port    Cluster Broker ID
----------------------------------------------
localHost    7676            brokerA

Cluster ID                              myClusterID
Cluster Is Highly Available             True

--------------------------------------------------------------------------------------------------------------
                                                                           ID of broker       Time since last
Broker ID       Address         State                   Msgs in store   performing takeover   status timestamp
--------------------------------------------------------------------------------------------------------------
brokerA         localhost:7676  OPERATING               121                                   30 sec
brokerB         greyhound:7676  TAKEOVER_STARTED        52              brokerA               3 hrs
brokerC         jpgserv:7676    SHUTDOWN_STARTED        12346                                 10 sec
brokerD         icdev:7676      TAKEOVER_COMPLETE       0               brokerA               2 min
brokerE         mrperf:7676     *unknown                12                                    0 sec
brokerG         iclab1:7676     QUIESCING               4                                     2 sec
brokerH         iclab2:7676     QUIESCE_COMPLETE        8                                     5 sec

Managing Broker Clusters

The following sections describe how to perform various administrative management tasks for conventional and enhanced clusters, respectively.

Managing Conventional Clusters

The procedures in this section show how to perform the following tasks for a conventional cluster:

Connecting Brokers into a Conventional Cluster

There are two general methods of connecting brokers into a conventional cluster: from the command line (using the -cluster option) or by setting the imq.cluster.brokerlist property in the cluster configuration file.

Whichever method you use, each broker that you start attempts to connect to the other brokers in the cluster every five seconds; the connection will succeed once the master broker is started up (if one is configured). If a broker in the cluster starts before the master broker, it will remain in a suspended state, rejecting client connections, until the master broker starts; the suspended broker then will automatically become fully functional. It is therefore a good idea to start the master broker first and then the others, after the master broker has completed its startup.

When connecting brokers into a conventional cluster, you should be aware of the following issues:

ProcedureTo Connect Brokers Using a Cluster Configuration File

The method best suited for production systems is to use a cluster configuration file to specify the configuration of the cluster:

  1. Create a cluster configuration file that uses the imq.cluster.brokerlist property to specify the list of brokers to be connected.

    If you are using a master broker, identify it with the imq.cluster.masterbroker property in the configuration file.

  2. For each broker in the cluster, set the imq.cluster.url property in the broker’s instance configuration file to point to the cluster configuration file.

  3. Use the imqbrokerd command to start each broker.

    If there is a master broker, start it first, then the others after it has completed its startup.

ProcedureTo Connect Brokers from the Command Line

  1. If you are using a master broker, start it with the imqbrokerd command, using the -cluster option to specify the complete list of brokers to be included in the cluster.

    For example, the following command starts the broker as part of a cluster consisting of the brokers running at the default port (7676) on host1, at port 5000 on host2, and at port 9876 on the default host (localhost): 

       imqbrokerd  -cluster host1,host2:5000,:9876

  2. Once the master broker (if any) is running, start each of the other brokers in the cluster with the imqbrokerd command, using the same list of brokers with the -cluster option that you used for the master broker.

    The value specified for the -cluster option must be the same for all brokers in the cluster.

ProcedureTo Establish Secure Connections Between Brokers

If you want secure, encrypted message delivery between brokers in a cluster, configure the cluster connection service to use an SSL-based transport protocol:

  1. For each broker in the cluster, set up SSL-based connection services, as described in Message Encryption.

  2. Set each broker’s imq.cluster.transport property to ssl, either in the cluster configuration file or individually for each broker.

Adding Brokers to a Conventional Cluster

The procedure for adding a new broker to a conventional cluster depends on whether the cluster uses a cluster configuration file.

ProcedureTo Add a New Broker to a Conventional Cluster Using a Cluster Configuration File

  1. Add the new broker to the imq.cluster.brokerlist property in the cluster configuration file.

  2. Issue the following command to any broker in the cluster: 

       imqcmd reload cls

    This forces each broker to reload the imq.cluster.brokerlist property. It is not necessary to issue this command to every broker in the cluster; executing it for any one broker will cause all of them to reload the cluster configuration.

  3. (Optional) Set the value of the imq.cluster.url property in the new broker’s instance configuration file (config.properties) to point to the cluster configuration file.

  4. Start the new broker.

    If you did not perform step 3, use the -D option on the imqbrokerd command line to set the value of imq.cluster.url to the location of the cluster configuration file.

ProcedureTo Add a New Broker to a Conventional Cluster Without a Cluster Configuration File

  1. (Optional) Set the values of the following properties in the new broker’s instance configuration file (config.properties) :

      imq.cluster.brokerlist


      imq.cluster.masterbroker (if necessary)


      imq.cluster.transport (if you are using a secure cluster connection service)


    When the newly added broker starts, it connects and exchanges data with all the other brokers in the imq.cluster.brokerlist value.

  2. Modify the imq.cluster.brokerlist property of other brokers in the cluster to include the new broker.

    This step is not strictly necessary to add a broker to a functioning cluster. However, should any broker need to be restarted, its imq.cluster.brokerlist value must include all other brokers in the cluster, including the newly added broker.

  3. Start the new broker.

    If you did not perform step 1, use the -D option on the imqbrokerd command line to set the property values listed there.

Removing Brokers From a Conventional Cluster

The method you use to remove a broker from a conventional cluster depends on whether you originally created the cluster using a cluster configuration file or by means of command line options.

ProcedureTo Remove a Broker From a Conventional Cluster Using a Cluster Configuration File

If you originally created a cluster by specifying its member brokers with the imq.cluster.brokerlist property in a central cluster configuration file, it isn’t necessary to stop the brokers in order to remove one of them. Instead, you can simply edit the configuration file to exclude the broker you want to remove, force the remaining cluster members to reload the cluster configuration, and reconfigure the excluded broker so that it no longer points to the same cluster configuration file:

  1. Edit the cluster configuration file to remove the excluded broker from the list specified for the imq.cluster.brokerlist property.

  2. Issue the following command to each broker remaining in the cluster: 

       imqcmd reload cls

    This forces the brokers to reload the cluster configuration.

  3. Stop the broker you’re removing from the cluster.

  4. Edit that broker’s instance configuration file (config.properties), removing or specifying a different value for its imq.cluster.url property.

ProcedureTo Remove a Broker From a Conventional Cluster Using the Command Line

If you used the imqbrokerd command from the command line to connect the brokers into a cluster, you must stop each of the brokers and then restart them, specifying the new set of cluster members on the command line:

  1. Stop each broker in the cluster, using the imqcmd command.

  2. Restart the brokers that will remain in the cluster, using the imqbrokerd command’s -cluster option to specify only those remaining brokers.

    For example, suppose you originally created a cluster consisting of brokers A, B, and C by starting each of the three with the command 

       imqbrokerd  -cluster A,B,C

    To remove broker A from the cluster, restart brokers B and C with the command 

       imqbrokerd  -cluster B,C

Managing a Conventional Cluster's Configuration Change Record

As noted earlier, a conventional cluster can optionally have one master broker, which maintains a configuration change record to keep track of any changes in the cluster’s persistent state. The master broker is identified by the imq.cluster.masterbroker configuration property, either in the cluster configuration file or in the instance configuration files of the individual brokers.

Because of the important information that the configuration change record contains, it is important to back it up regularly so that it can be restored in case of failure. Although restoring from a backup will lose any changes in the cluster’s persistent state that have occurred since the backup was made, frequent backups can minimize this potential loss of information. The backup and restore operations also have the positive effect of compressing and optimizing the change history contained in the configuration change record, which can grow significantly over time.

ProcedureTo Back Up the Configuration Change Record

  1. Use the -backup option of the imqbrokerd command, specifying the name of the backup file.

    For example: 

       imqbrokerd  -backup mybackuplog

ProcedureTo Restore the Configuration Change Record

  1. Shut down all brokers in the cluster.

  2. Restore the master broker’s configuration change record from the backup file.

    The command is 

       imqbrokerd  -restore mybackuplog

  3. If you assign a new name or port number to the master broker, update the imq.cluster.brokerlist and imq.cluster.masterbroker properties accordingly in the cluster configuration file.

  4. Restart all brokers in the cluster.

Managing Enhanced Clusters

This section presents step-by-step procedures for performing a variety of administrative tasks for an enhanced cluster:

Connecting Brokers into an Enhanced Cluster

Because enhanced clusters are self-configuring, there is no need to explicitly specify the list of brokers to be included in the cluster. Instead, all that is needed is to set each broker’s configuration properties appropriately and then start the broker; as long as its properties are set properly, it will automatically be incorporated into the cluster. Enhanced Broker Cluster Properties describes the required properties, which include vendor-specific JDBC database properties.

Note –

In addition to creating an enhanced cluster as described in this section, you must also configure clients to successfully reconnect to a failover broker in the event of broker or connection failure. You do this by setting the imqReconnectAttempts connection factory attribute to a value of -1.

The property values needed for brokers in an enhanced cluster can be set separately in each broker’s instance configuration file, or they can be specified in a cluster configuration file that all the brokers reference. The procedures are as follows:

ProcedureTo Connect Brokers Using a Cluster Configuration File

The method best suited for production systems is to use a cluster configuration file to specify the configuration of the cluster.

  1. Create a cluster configuration file specifying the cluster’s high-availability-related configuration properties.

    Enhanced Broker Cluster Properties shows the required property values. However, do not include the imq.brokerid property in the cluster configuration file; this must be specified separately for each individual broker in the cluster.

  2. Specify any additional, vendor-specific JDBC configuration properties that might be needed.

    The vendor-specific properties required for MySQL and HADB are shown in Example 8–1 and Example 8–2, respectively.

  3. For each broker in the cluster:

    1. Start the broker at least once, using the imqbrokerd command.

      The first time a broker instance is run, an instance configuration file (config.properties) is automatically created.

    2. Shut down the broker.

      Use the imqcmd shutdown bkr command.

    3. Edit the instance configuration file to specify the location of the cluster configuration file.

      In the broker’s instance configuration file, set the imq.cluster.url property to point to the location of the cluster configuration file you created in step 1.

    4. Specify the broker identifier.

      Set the imq.brokerid property in the instance configuration file to the broker’s unique broker identifier. This value must be different for each broker.

  4. Place a copy of, or a symbolic link to, your JDBC driver’s .jar file in the Message Queue external resource files directory, depending on your platform (see Appendix A, Platform-Specific Locations of Message Queue Data):

      Solaris: /usr/share/lib/imq/ext


      Linux: /opt/sun/mq/share/lib/ext


      AIX: IMQ_HOME/lib/ext


      Windows: IMQ_HOME\lib\ext


  5. Create the database schema needed for Message Queue persistence.

    Use the imqdbmgr create tbl command; see Database Manager Utility.

  6. Restart each broker with the imqbrokerd command.

    The brokers will automatically register themselves into the cluster on startup.

ProcedureTo Connect Brokers Using Instance Configuration Files

  1. For each broker in the cluster:

    1. Start the broker at least once, using the imqbrokerd command.

      The first time a broker instance is run, an instance configuration file (config.properties) is automatically created.

    2. Shut down the broker.

      Use the imqcmd shutdown bkr command.

    3. Edit the instance configuration file to specify the broker’s high-availability-related configuration properties.

      Enhanced Broker Cluster Properties shows the required property values. Be sure to set the brokerid property uniquely for each broker.

    4. Specify any additional, vendor-specific JDBC configuration properties that might be needed.

      The vendor-specific properties required for MySQL and HADB are shown in Example 8–1 and Example 8–2, respectively.

  2. Place a copy of, or a symbolic link to, your JDBC driver’s .jar file in the Message Queue external resource files directory, depending on your platform (see Appendix A, Platform-Specific Locations of Message Queue Data):

      Solaris: /usr/share/lib/imq/ext


      Linux: /opt/sun/mq/share/lib/ext


      AIX: IMQ_HOME/lib/ext


      Windows: IMQ_HOME\lib\ext


  3. Create the database schema needed for Message Queue persistence.

    Use the imqdbmgr create tbl command; see Database Manager Utility.

  4. Restart each broker with the imqbrokerd command.

    The brokers will automatically register themselves into the cluster on startup.

Adding and Removing Brokers in an Enhanced Cluster

Because enhanced clusters are self-configuring, the procedures for adding and removing brokers are simpler than for a conventional cluster.

ProcedureTo Add a New Broker to an Enhanced Cluster

  1. Set the new broker’s high-availability-related properties, as described in the preceding section.

    You can do this either by specifying the individual properties in the broker’s instance configuration file (config.properties) or, if there is a cluster configuration file, by setting the broker’s imq.cluster.url property to point to it.

  2. Start the new broker with the imqbrokerd command.

    The broker will automatically register itself into the cluster on startup.

ProcedureTo Remove a Broker from an Enhanced Cluster

  1. Make sure the broker is not running.

    If necessary, use the command 

       imqcmd shutdown bkr

    to shut down the broker.

  2. Remove the broker from the cluster with the command 

       imqdbmgr remove bkr

    This command deletes all database tables for the corresponding broker.

Preventing or Forcing Broker Failover

Although the takeover of a failed broker’s persistent data by a failover broker in an enhanced cluster is normally automatic, there may be times when you want to prevent such failover from occurring. To suppress automatic failover when shutting down a broker, use the -nofailover option to the imqcmd shutdown bkr subcommand: 

   imqcmd shutdown bkr  -nofailover  -b hostName:portNumber

where hostName and portNumber are the host name and port number of the broker to be shut down.

Conversely, you may sometimes need to force a broker failover to occur manually. (This might be necessary, for instance, if a failover broker were to itself fail before completing the takeover process.) In such cases, you can initiate a failover manually from the command line: first shut down the broker to be taken over with the -nofailover option, as shown above, then issue the command 

   imqcmd takeover bkr  -n brokerID

where brokerID is the broker identifier of the broker to be taken over. If the specified broker appears to be running, the Command utility will display a confirmation message: 

   The broker associated with brokerID last accessed the database # seconds ago. 
   Do you want to take over for this broker?

You can suppress this message, and force the takeover to occur unconditionally, by using the -f option to the imqcmd takeover bkr command: 

   imqcmd takeover bkr  -f  -n brokerID
Note –

The imqcmd takeover bkr subcommand is intended only for use in failed-takeover situations. You should use it only as a last resort, and not as a general way of forcibly taking over a running broker.

Backing up a Shared Data Store

For durability and reliability, it is a good idea to back up an enhanced cluster’s shared data store periodically to backup files. This creates a snapshot of the data store that you can then use to restore the data in case of catastrophic failure. The command for backing up the data store is 

   imqdbmgr backup  -dir backupDir

where backupDir is the path to the directory in which to place the backup files. To restore the data store from these files, use the command 

   imqdbmgr restore  -restore backupDir

Converting a Conventional Cluster to an Enhanced Cluster

The best approach to converting a conventional broker cluster to an enhanced broker cluster is to drain your messaging system of all persistent data before attempting the conversion. This lets you create a new shared data store without worrying about loss of data. However, if you are using individual JDBC-based data stores for your brokers, a utility is available for converting a standalone datastore to a shared data store.

ProcedureCluster Conversion : File-Based Data Store

If the brokers in your conventional cluster are using file-based data stores, use the following procedure to convert to an enhanced cluster.

  1. Drain down your messaging system of all persistent data.

    Stop all producer clients from producing messages, and wait for all messages in the system to be consumed.

  2. Shut down all client applications.

  3. Shut down all brokers in the conventional cluster.

  4. Reconfigure all brokers for an enhanced cluster.

    See Enhanced Broker Cluster Properties. It is recommended that you use a cluster configuration file to specify cluster configuration property values, such as the imq.cluster.clusterid, imq.persist.store, and additional shared JDBC database properties.

  5. Start all brokers in the enhanced cluster.

    See Connecting Brokers into an Enhanced Cluster.

  6. Configure client applications to re-connect to failover brokers.

    Client re-connection behavior is specified by connection handling attributes of the connection factory administered objects (see the Connection Handling). In the case of enhanced broker clusters, the imqAddressList, imqAddressListBehavior, and imqAddressListIterations attributes are ignored, however the imqReconnectAttempts attribute should be set to a value of -1 (unlimited).

  7. Start all client applications.

  8. Resume messaging operations

ProcedureCluster Conversion: JDBC-Based Data Store

If the brokers in your conventional cluster are using JDBC-based data stores, use the following procedure to convert to an enhanced cluster. The procedure assumes that individual standalone broker data stores reside on the same JDBC database server.

  1. Back up all persistent data in the standalone JDBC-based data store of each broker.

    Use proprietary JDBC database tools.

  2. Shut down all client applications.

  3. Shut down all brokers in the conventional cluster.

  4. Convert each standalone data store to a shared data store.

    Use the Message Queue Database Manager utility (imqdbmgr) subcommand 

       imqdbmgr upgrade hastore

    to convert an existing standalone JDBC database to a shared JDBC database.

  5. Reconfigure all brokers for an enhanced cluster.

    See Enhanced Broker Cluster Properties. It is recommended that you use a cluster configuration file to specify cluster configuration property values, such as the imq.cluster.clusterid, imq.persist.store, and additional shared JDBC database properties.

  6. Start all brokers in the enhanced cluster.

    See Connecting Brokers into an Enhanced Cluster.

  7. Configure client applications to re-connect to failover brokers.

    Client re-connection behavior is specified by connection handling attributes of the connection factory administered objects (see the Connection Handling). In the case of enhanced broker clusters, the imqAddressList, imqAddressListBehavior, and imqAddressListIterations attributes are ignored, however the imqReconnectAttempts attribute should be set to a value of -1 (unlimited).

  8. Start all client applications.

  9. Resume messaging operations.