Quorum Devices (VxVM)

In certain cases--for example, in a two-node cluster when both cluster interconnects fail and both cluster nodes are still members--Sun Cluster needs assistance from a hardware device to solve the problem of cluster quorum. This device is called the quorum device.

Quorum devices must be used in clusters running VERITAS Volume Manager (VxVM), regardless of the number of cluster nodes. Solstice DiskSuite assures cluster quorum through the use of its own metadevice state database replicas, and as such, does not need a quorum device. Quorum devices are neither required nor supported in Solstice DiskSuite configurations. When you install a cluster using Solstice DiskSuite, the scinstall(1M) program will not ask for, or accept a quorum device.

The quorum device is merely a disk or a controller which is specified during the cluster installation procedure by using the scinstall(1M) command. The quorum device is a logical concept; there is nothing special about the specific piece of hardware chosen as the quorum device. VxVM does not allow a portion of a disk to be in a separate disk group, so an entire disk and its plex (mirror) are required for the quorum device.

A quorum device ensures that at any point in time only one node can update the multihost disks that are shared between nodes. The quorum device comes into use if the cluster interconnect is lost between nodes. Each node (or set of nodes in a greater than two-node cluster) should not attempt to update shared data unless it can establish that it is part of the majority quorum. The nodes take a vote, or quorum, to decide which nodes remain in the cluster. Each node determines how many other nodes it can communicate with. If it can communicate with more than half of the cluster, then it is in the majority quorum and is allowed to remain a cluster member. If it is not in the majority quorum, the node aborts from the cluster.

The quorum device acts as the "third vote" to prevent a tie. For example, in a two-node cluster, if the cluster interconnect is lost, each node will "race" to reserve the quorum device. Figure 1-9 shows a two-node cluster with a quorum device located in one of the multihost disk enclosures.

Figure 1-9 Two-Node Cluster With Quorum Device

The node that reserves the quorum device then has two votes toward quorum versus the remaining node that has only one vote. The node with the quorum will then start its own cluster (mastering the multihost disks) and the other node will abort.

Before each cluster reconfiguration, the set of nodes and the quorum device vote to approve the new system configuration. Reconfiguration proceeds only if a majority quorum is reached. After a reconfiguration, a node remains in the cluster only if it is part of the majority partition.

In greater than two-node clusters, each set of nodes that share access to multihost disks must be configured to use a quorum device.

The concept of quorum device changes somewhat in greater than two-node clusters. If there is an even split for nodes that do not share a quorum device--referred to as a "split-brain" partition--you must be able to decide which set of nodes will become a new cluster and which set will abort. This situation is not handled by the quorum device. Instead, as part of the installation process, when you configure the quorum device(s), you are asked questions that determine what will happen when such a partition occurs. One of two events occurs in this partition situation depending on whether you requested to have the cluster software automatically select the new cluster membership or whether you specified manual intervention.

• Automatic selection - If, during installation, you choose select, then the software automatically selects which subset is aborted based on either the Lowest Nodeid or the Highest Nodeid. If you chose the Lowest Nodeid, then the subset containing the node with the lowest node ID value automatically becomes the new cluster. If you chose the Highest Nodeid, then the subset containing the node with the highest node ID value automatically becomes the new cluster. You must manually abort all other subsets.

• Manual intervention - At the time of the partition, the system prompts you to choose the new cluster.

For example, consider a four-node cluster (that might or might not share a storage device common to all nodes) where a network failure results in node 0 and 1 communicating with each other and nodes 2 and 3 communicating with each other. In this situation, the automatic or manual decision of quorum would be used. The cluster monitor software is quite intelligent. It tries to determine on its own which nodes should be cluster members and which should not. It resorts to the quorum device to break a tie or the manual and automatic selection of cluster domains only in extreme situations.

The failure of a quorum device is similar to the failure of a node in a two-node cluster. Although the failure of a quorum device does not cause a failover of services, it does reduce the high availability of a two-node cluster in that no further node failures can be tolerated. A failed quorum device can be reconfigured or replaced while the cluster is running. The cluster can remain running as long as no other component failure occurs while the quorum repair or replacement is in progress.

The quorum device on its own cannot account for all scenarios where a decision must be made on cluster membership. For example, consider a fully operational three-node cluster, where all of the nodes share access to the multihost disks, such as the Sun StorEdge A5000. If one node aborts or loses both cluster interconnects, and the other two nodes are still able to communicate to each other, the two remaining nodes do not have to reserve the quorum device to break a tie. Instead, the majority voting that comes into play (two votes out of three) determines that the two nodes that can communicate with each other can form the cluster. However, the two nodes that form the cluster must still prevent the crashed or hung node from coming back online and corrupting the shared data. They do this by using a technique called failure fencing, as described in "Failure Fencing ".