4.6 Recovering From Cluster Partitions

Multiple failures (including network partitions) might result in subsets of cluster members attempting to remain in the cluster. Usually, these subsets have lost partial or total communication with each other. In such cases, the software attempts to ensure that there is only one resultant valid cluster. To achieve this, the software might cause some or all nodes to abort. The following discussion explains the criterion used to make these decisions.

The quorum criterion is defined as a subset with at least half the members of the original set of cluster nodes (not only the configured nodes). If a subset does not meet the quorum criterion, the nodes in the subset abort themselves and a reconfig.4014 error message is displayed. Failure to meet the quorum criterion could be due to a network partition or to a simultaneous failure of more than half of the nodes.

Valid clusters only contain nodes that can communicate with each other over private networks.

Consider a four-node cluster that partitions itself into two subsets: one subset consists of one node, while the other subset consists of three nodes. Each subset attempts to meet the quorum criterion. The first subset has only one node (out of the original four) and does not meet the quorum criterion. Hence, the node in the first subset shuts down. The second subset has three nodes (out of the original four), meets the quorum criterion, and therefore stays up.

Alternatively, consider a two-node cluster with a quorum device. If there is a partition in such a configuration, then one node and the quorum device meet the quorum criterion and the cluster stays up.

4.6.1 Split-Brain Partitions (SSVM or CVM Only)

A split-brain partition occurs if a subset has exactly half the cluster members. (The split-brain partition does not include the scenario of a two-node cluster with a quorum device.) During initial installation of Sun Cluster, you were prompted to choose your preferred type of recovery from a split-brain scenario. Your choices were ask and select. If you chose ask, then if a split-brain partition occurs, the system asks you for a decision about which nodes should stay up. If you chose select, the system automatically selects for you which cluster members should stay up.

If you chose an automatic selection policy to deal with split-brain situations, your options were Lowest Nodeid or Highest Nodeid. If you chose Lowest Nodeid, then the subset containing the node with the lowest ID value becomes the new cluster. If you chose Highest Nodeid, then the subset containing the node with the highest ID value becomes the new cluster. For more details, see the section on installation procedures in the Sun Cluster 2.2 Software Installation Guide.

In either case, you must manually abort the nodes in all other subsets.

If you did not choose an automatic selection policy or if the system prompts you for input at the time of the partition, then the system displays the following error message.

SUNWcluster.clustd.reconf.3010
 "*** ISSUE ABORTPARTITION OR CONTINUEPARTITION *** 
Proposed cluster: xxx  
Unreachable nodes: yyy"

Additionally, a message similar to the following is displayed on the console every ten seconds:

*** ISSUE ABORTPARTITION OR CONTINUEPARTITION ***
 If the unreachable nodes have formed a cluster, issue ABORTPARTITION.
 (scadmin abortpartition <localnode> <clustername>)
 You may allow the proposed cluster to form by issuing CONTINUEPARTITION.
 (scadmin continuepartition <localnode> <clustername>)
 Proposed cluster partition:  0  Unreachable nodes: 1

If you did not choose an automatic select process, use the procedure "4.6.2 How to Choose a New Cluster" to choose the new cluster.

To restart the cluster after a split-brain failure, you must wait for the stopped node to come up entirely (it might undergo automatic reconfiguration or reboot) before you bring it back into the cluster using the scadmin startnode command.

4.6.2 How to Choose a New Cluster

1. Determine which subset should form the new cluster. Run the following command on one node in the subset that should abort.

   # scadmin abortpartition

   When the abortpartition command is issued on one node, the Cluster Membership Monitor (CMM) propagates that command to all the nodes in that partition. Therefore, if all nodes in that partition receive the command, they all abort. However, if some of the nodes in the partition cannot be contacted by the CMM, then they have to be manually aborted. Run the scadmin abortpartition command on any remaining nodes that do not abort.

2. Run the following command on one node in the subset that should stay up.

   # scadmin continuepartition

   ---

   Note -

   A further reconfiguration occurs if there has been another failure within the new cluster. At all times, only one cluster is active.

   ---