Understanding the Structure of Notifications

Applications that you write can register a callback function to handle notification messages. The cmm_notify_t callback receives the cluster membership change (cmc) callback function. The cmm_cmc_register function takes the service or application data and this callback function. You must provide relevant data when registering. The code in EXAMPLE 6-1 details the related structure, called the cmm_cmc_notification_t structure.

typedef struct {
     cmm_cmchanges_t  cmchange;
     cmm_nodeid_t  nodeid; 
} cmm_cmc_notification_t;

The fields in this structure detail the cluster change and specify the node concerned. These fields are described in TABLE 6-1.

This structure is used by the cmm_notify_t callback function. The cmm_notify_t callback function contains the parameters described in TABLE 6-2.

If change notification data is required for longer than the duration of the callback, it must be handled by the client application or service.

Notification Values

Change notification messages contain the nodeid of the affected node and a cmm_cmchanges_t data type. The cmm_cmchanges_t data type describes the change notification. TABLE 6-3 lists the notifications of the cmm_cmchanges_t structure:

Cluster Initialization Notifications

When neither node A nor B is currently running the Foundation Services, nodes A and B are out. When node A becomes the master node, a MASTER_ELECTED notification is sent to the registered applications and services. At cluster startup, this is the first step in the creation of a cluster. The notification sent for this scenario is shown in TABLE 6-5.

The following scenario describes the election of a qualified node to the vice-master role at cluster initialization. This takes place in one step, when the CMM_VICEMASTER_ELECTED notification is sent. The notification sent for this scenario is shown in TABLE 6-6.

The following scenario describes when a new node joins the cluster. This node does not take the master or vice-master role and could be a diskless node or a dataless node. The notification sent for this scenario is shown in TABLE 6-7. This scenario can occur at cluster initialization or when a new node is added to a running cluster.

The following scenario describes the situation where a node that is in becomes vice-master. This scenario can occur if a node is in the cluster but does not immediately declare itself as master-eligible. When its eligibility to be a master node or a vice-master node is known, the node is elected vice-master. This scenario can also occur if a master-eligible node is disqualified. When the node is requalified, the node becomes that vice-master. The notification sent for this scenario is shown in TABLE 6-8.

Vice-Master Removal Notifications

Provided that there is a running vice-master node, if the master node stops being master because its role has been removed, there is a failover, as explained in Failover Notifications.

If the vice-master node stops being vice-master due to a failure, or because its role has been removed, there is no backup for the master node and the cluster loses its 2N redundancy.

If the vice-master role is removed because of a failure or by using the cmm_membership_remove function, the notification is shown in TABLE 6-9.

Vice-Master Excluded Notification

The vice-master node can be disqualified if you use the cmm_member_setqualif function. The notification sent for this scenario is shown in TABLE 6-10.

For more information about disqualifying a node by using the cmm_member_setqualif function, see Setting the Qualification of a Node. Care must be taken with the use of the cmm_member_setqualif function. Do not trigger a failover. For more information, see Triggering a Failover by Using the cmm_member_setqualif Function. See also the cmm_member_setqualif(3CMM) man page.

Peer Node Removal Notification

If a peer node other than the master or vice-master loses its role in the cluster, it becomes temporarily out of the cluster. This occurs if you use the cmm_membership_remove function on the peer node. The notification sent in this scenario is shown in TABLE 6-11.

Master Node Excluded Notifications

If the master node fails, the node can be excluded from the cluster as described in Removing or Excluding a Node. The notification sent in this scenario is shown in TABLE 6-12

Node Other Than Master Excluded Notification

If a node other than the master fails it can be excluded from the rest of the cluster as described in Removing or Excluding a Node. The notification sent is shown in TABLE 6-13.

The notification sent for this scenario can also be sent for a diskless node.

Switchover Notifications

A switchover is the scheduled transfer of the CMM_MASTER role from the master node to the vice-master node. A switchover is not a failure and does not change the qualification level of the master node. A switchover is not a persistent change. A switchover is usually triggered by the cluster administrator for the maintenance of a node. For more information about the maintenance of nodes, see the Netra High Availability Suite 3.0 1/08 Foundation Services Cluster Administration Guide.

The notifications sent in the case of a switchover from the master to the vice-master node, triggered by calling the cmm_mastership_release function, are shown in TABLE 6-14.

For further information and an example that uses the cmm_mastership_release function to trigger a switchover, see Triggering a Switchover.

Failover Notifications

A failover is the unscheduled transfer of the CMM_MASTER role from the master node to the vice-master node. A failover is a response to the removal or failure of the master node or disqualification of the master node. This section describes two failover scenarios:

• Failover Due to the Removal or Failure of the Master Node.

• Failover Due to Master Disqualification.

Stale Cluster Notification

When information received by a peer node from the master node is more than 10 seconds old, the information is considered to be stale. A stale cluster does not guarantee that there is no change in the cluster. A stale cluster means that information held by the master node is not reaching a peer node. This can happen if the master node is not functioning correctly and does not send information to the peer node. A stale cluster can also occur if the master node does send information but it does not reach the peer node, due for example to a problem in the network. The notification sent for this scenario is shown in TABLE 6-17.

Amnesia

Amnesia is an error condition in which a cluster restarts with stale cluster configuration data. This can happen when a cluster is restarted from a node that was not previously part of the most recent cluster membership list.

Split Brain

Split brain is an error condition in which there are two master nodes. This can be caused by interconnect failure between peer nodes.

During split brain, each master node assumes that it is the only master node in the cluster. A split brain can begin with any combination of roles for nodes A and B. The notification sent for this scenario is shown in TABLE 6-18.

For information about how to recover from a split brain error condition, see the Netra High Availability Suite 3.0 1/08 Foundation Services Troubleshooting Guide.