A conventional broker cluster has the following characteristics:
Each broker has its own respective persistent data store in which destinations, persistent messages, and other state information is stored. Some of this information (for example, destinations and durable subscriptions) has been propagated to the broker from other brokers in the cluster. If a broker fails, it is possible for this information to become out of sync with the information stored by other brokers in the cluster. To guard against this possibility in a conventional broker cluster, one broker within the cluster can be designated as a master broker. The master broker maintains a configuration change record to track changes to the cluster’s propagated persistent entities.
When an offline broker comes back online (or when a new broker is added to the cluster), it consults the configuration change record for information about destinations and durable subscribers, then exchanges information with other brokers about its currently active message consumers.
Because brokers cannot complete their initialization without accessing the configuration change record, the master broker should always be the first broker started within the cluster. Furthermore, if the master broker goes offline, destination and durable subscriber information cannot be propagated across the cluster. Under these conditions, you will get an exception if you try to create, reconfigure, or destroy a destination or a durable subscription (auto-created destinations and temporary destinations are not affected), or attempt a related operation. Similarly, in the absence of a master broker, any client application attempting to create a durable subscriber or unsubscribe from a durable subscription will get an error. Nevertheless, client applications can successfully interact with an existing durable subscriber.
Message production, delivery, and consumption can continue uninterrupted without a master broker.
Failure Detection and Recovery
A conventional broker cluster detects failures when one broker tries to send data to another broker and an exception is thrown. When the cluster encounters a failed connection between brokers, it cannot do anything to recover, other than stop sendng data. It is the responsibility of an administrator to monitor brokers in the cluster by using Message Queue administration tools (see Administration Tools) and perform the appropriate recovery operations.
If a broker or its connection to a client fails, the client automatically attempts to reconnect to the same or another broker in the cluster. The reconnect is governed by connection properties that specify the order and frequency by which the client attempts to reconnect to brokers in the cluster. The broker to which the client successfully reconnects becomes the client's new home broker.
In this scenario, the new home broker (if different from the failed broker) does not have all the client-related state information that was previously held by the failed broker. For example, messages to have been consumed by the client or the state of transactions involving the client might have been lost. As a result, the failure of a broker in a conventional cluster can cause a delay in message delivery (until the failed broker restarts and the client reconnects).