Message QueueTM Enterprise Edition supports the use of broker clusters: groups of brokers working together to provide message delivery services to clients. Clusters enable a message service to scale its operations with the volume of message traffic by distributing client connections among multiple brokers. See the Message Queue Technical Overview for a general discussion of clusters and how they operate.
This chapter describes how to manage broker clusters, connect brokers to them, and configure them. It contains the following sections:
You define a cluster by specifying cluster configuration properties for each of its member brokers. You can set these properties individually for each broker in the cluster, but it is generally more convenient to collect them into a central cluster configuration file that all of the brokers reference. This prevents the settings from getting out of agreement and ensures that all brokers in a cluster share the same, consistent configuration information.
The cluster configuration properties are described in detail in Table 14–9. They include the following:
imq.cluster.hostname gives the host name or IP address for the cluster connection service, used for internal communication between brokers in the cluster. This setting can be useful if more than one host is available: for example, if there is more than one network interface card in a computer.
The hostname and port properties can be set independently for each individual broker, but brokerlist, masterbroker, url, and transport must have the same values for all brokers in the cluster.
The following sections describe how to set a broker’s cluster configuration properties, either individually for each broker in a cluster or centrally, using a cluster configuration file.
You can set a broker’s cluster configuration properties in its instance configuration file (or on the command line when you start the broker). For example, to create a cluster consisting of brokers at port 9876 on host1, port 5000 on host2, and the default port (7676) on ctrlhost , you would include the following property in the instance configuration files for all three brokers:
Notice that if you need to change the cluster configuration, this method requires you to update the instance configuration file for every broker in the cluster.
For consistency and ease of maintenance, it is recommended that you collect all of the shared cluster configuration properties into a single cluster configuration file instead of setting them separately for each individual broker. In this method, each broker’s instance configuration file must set the imq.cluster.url property to point to the location of the cluster configuration file: for example,
The cluster configuration file then defines the shared configuration properties for all of the brokers in the cluster, such as the list of brokers to be connected (imq.cluster.brokerlist), the transport protocol to use for the cluster connection service (imq.cluster.transport ), and optionally, the address of the master broker (imq.cluster.masterbroker). The following code defines the same cluster as in the previous example, with the broker running on ctrlhost serving as the master broker:
This section describes how to connect a set of brokers to form a cluster, add new brokers to an existing cluster, and remove brokers from a cluster.
There are two general methods of connecting brokers into a cluster: from the command line (using the -cluster option) or by setting the imq.cluster.brokerlist property in the cluster configuration file. Whichever method you use, each broker that you start attempts to connect to the other brokers every five seconds; the connection will succeed once the master broker is started up (if one is configured). If a broker in the cluster starts before the master broker, it will remain in a suspended state, rejecting client connections, until the master broker starts; the suspended broker then will automatically become fully functional.
To configure a broker cluster from the command line, use the -cluster option to the imqbrokerd command to specify the complete list of brokers in the cluster when you start each one. For example, the following command starts a new broker and connects it to the brokers running at the default port (7676) on host1, at port 5000 on host2, and at port 9876 on the default host (localhost):
imqbrokerd -cluster host1,host2:5000,:9876
An alternative method, better suited for production systems, is to create a cluster configuration file that uses the imq.cluster.brokerlist property to specify the list of brokers to be connected. Each broker in the cluster must then set its own imq.cluster.url property to point to this cluster configuration file.
Whichever method you use, you must make sure that no broker in the cluster is given an address that resolves to the network loopback IP address (127.0.0.1). Any broker configured with this address will be unable to connect to other brokers in the cluster.
Some Linux installers automatically set the localhost entry to the network loopback address. On such systems, you must modify the system IP address so that all brokers in the cluster can be addressed properly.
For all Linux systems that participate in a cluster, check the /etc/hosts file as part of cluster setup. If the system uses a static IP address, edit the /etc/hosts file to specify the correct address for localhost. If the address is registered with Domain Name Service (DNS), edit the file /etc/nsswitch.conf to change the order of the entries so that DNS lookup is performed before consulting the local hosts file. The line in /etc/nsswitch.conf should read as follows:
hosts: dns files
If you want secure, encrypted message delivery between brokers in a cluster, configure the cluster connection service to use an SSL-based transport protocol. For each broker in the cluster, set up SSL-based connection services, as described in Message Encryption. Then set each broker’s imq.cluster.transport property to ssl, either in the cluster configuration file or individually for each broker.
Issue the following command to any broker in the cluster:
imqcmd reload cls
This forces each broker to reload the cluster configuration, ensuring that all persistent information for brokers in the cluster is up to date. Note that it is not necessary to issue this command to every broker in the cluster; executing it for any one broker will cause all of them to reload the cluster configuration.
Start the new broker.
If you did not perform Adding Brokers to a Cluster, use the -D option on the imqbrokerd command line to set the value of imq.cluster.url.
The method you use to remove a broker from a cluster depends on whether you originally created the cluster via the command line or by means of a central cluster configuration file.
If you used the imqbrokerd command from the command line to connect the brokers into a cluster, you must stop each of the brokers and then restart them, specifying the new set of cluster members on the command line. The procedure is as follows:
Stop each broker in the cluster, using the imqcmd command.
For example, suppose you originally created a cluster consisting of brokers A, B, and C by starting each of the three with the command
imqbrokerd -cluster A,B, C
To remove broker A from the cluster, restart brokers B and C with the command
imqbrokerd -cluster B,C
If you originally created a cluster by specifying its member brokers with the imq.cluster.brokerlist property in a central cluster configuration file, it isn’t necessary to stop the brokers in order to remove one of them. Instead, you can simply edit the configuration file to exclude the broker you want to remove, force the remaining cluster members to reload the cluster configuration, and reconfigure the excluded broker so that it no longer points to the same cluster configuration file. Here is the procedure:
Issue the following command to each broker remaining in the cluster:
imqcmd reload cls
This forces the broker to reload the cluster configuration.
Stop the broker you’re removing from the cluster.
A cluster can optionally have one master broker, which maintains a configuration change record to keep track of any changes in the cluster’s persistent state. The master broker is identified by the imq.cluster.masterbroker configuration property, either in the cluster configuration file or in the instance configuration files of the individual brokers.
The configuration change record contains information about changes in the persistent entities associated with the cluster, such as durable subscriptions and administrator-created physical destinations. All brokers in the cluster consult the master broker during startup in order to update their information about these persistent entities. Failure of the master broker makes such synchronization impossible; see When a Master Broker Is Unavailable for more information.
Because of the important information that the configuration change record contains, it is important to back it up regularly so that it can be restored in case of failure. Although restoring from a backup will lose any changes in the cluster’s persistent state that have occurred since the backup was made, frequent backups can minimize this potential loss of information. The backup and restore operations also have the positive effect of compressing and optimizing the change history contained in the configuration change record, which can grow significantly over time.
imqbrokerd -backup mybackuplog
Shut down all brokers in the cluster.
Restore the master broker’s configuration change record from the backup file with the command
If you assign a new name or port number to the master broker, update the imq.cluster.brokerlist and imq.cluster.masterbroker properties accordingly in the cluster configuration file.
Restart all brokers in the cluster.
Because all brokers in a cluster need the master broker in order to perform persistent operations, the following imqcmd subcommands for any broker in the cluster will return an error when no master broker is available:
Auto-created physical destinations and temporary destinations are unaffected.
In the absence of a master broker, any client application attempting to create a durable subscriber or unsubscribe from a durable subscription will get an error. However, a client can successfully specify and interact with an existing durable subscription.