Failures Addressed by Mediators

With mediators, it is possible to recover from single failures, as well as some double failures. Since Sun Cluster only guarantees automatic recovery from single failures, only the single-failure recovery situation is covered here in detail. The double failure scenarios are included, but only general recovery processes are described.

Figure 9-1 shows a dual-string configuration in the stable state. Note that mediators are established on both Sun Cluster nodes, so both nodes must be up for a mediator quorum to exist and for mediators to be used. If one Sun Cluster node fails, a replica quorum will exist. If a takeover of the diskset is necessary, the takeover will occur without the use of mediators.

The following sections show various failure scenarios and describe how mediators help recover from these failures.

Single Server Failure

Figure 9-2 shows the situation where one Sun Cluster node fails. In this case, the mediator software is not used since there is a replica quorum available. Sun Cluster node phys-hahost2 will take over the diskset previously mastered by phys-hahost1.

The process for recovery in this scenario is identical to the process followed when one Sun Cluster node fails and there are more than two disk strings. No administrator action is required except perhaps to switch over the diskset after phys-hahost1 rejoins the cluster. See the haswitch(1M) man page for more information about the switchover procedure.

Figure 9-2 Single Sun Cluster Server Failure With Mediators

Single String Failure

Figure 9-3 illustrates the case where, starting from the steady state shown in Figure 9-1, a single string fails. When String 1 fails, the mediator hosts on both phys-hahost1 and phys-hahost2 will be updated to reflect the event, and the system will continue to run as follows:

• No takeover occurs.

• Sun Cluster node phys-hahost1 continues to own the diskset.

• Because String 1 failed, it must be resynchronized with String 2. For more information about the resynchronization process, refer to the Solstice DiskSuite User's Guide and the metareplace(1M) man page.

The commit count is incremented and the mediators remain golden.

Figure 9-3 Single String Failure With Mediators

The administration required in this scenario is the same that is required when a single string fails in the three or more string configuration. Refer to the relevant chapter on administration of your disk expansion unit for details on these procedures.

Host and String Failure

Figure 9-4 shows a double failure where both String 1 and phys-hahost2 fail. If the failure sequence is such that the string fails first, and later the host fails, the mediator on phys-hahost1 could be golden. In this case, we have the following conditions:

• The mediator on phys-hahost1 is golden.

• Half of the mediators are available.

• Half of the replicas are accessible.

• The mediator commit count on phys-hahost1 matches the commit count found in the replicas on String 2

Figure 9-4 Multiple Failure - One Server and One String

This type of failure is recovered automatically by Sun Cluster. If phys-hahost2 mastered the diskset, phys-hahost1 will take over mastery of the diskset. Otherwise, mastery of the diskset will be retained by phys-hahost1. After String 1 is fixed, the data on String 1 must be resynchronized with the data on String 2. For more information about the resynchronization process, refer to the Solstice DiskSuite User's Guide and the metareplace(1M) man page.

Although you can recover from this scenario, you must be sure to restore the failed components immediately since a third failure will cause the cluster to be unavailable.

If the mediator on phys-hahost1 is not golden, this case is not automatically recovered by Sun Cluster and requires administrative intervention. In this case, Sun Cluster generates an error message and the logical host is put into maintenance mode (read-only). If this or any other multiple failure occurs, contact your service provider to assist you.