Configuring Heartbeat-Loss Notification

Language:

This section contains the following information:

Introduction to Configuring Heartbeat-Loss Notification

You can configure the Geographic Edition framework to send email notification and to run an action script when a heartbeat is lost. You configure heartbeat-loss notification by using the optional Notification_emailaddrs and Notification_actioncmd properties.

Heartbeat-loss notification occurs if the heartbeat still fails after the interval you configure with the Query_interval property of the heartbeat. The heartbeat monitor sends out a heartbeat request to the responder on the logical host every Query_interval period. If no response is received within the Query_interval period, an internal count is incremented. If the recount reaches the number that is specified in the heartbeat.retries property, the heartbeat is deemed to have failed.

For example, you can use the default Query_interval value of 120 seconds and the default heartbeat.retries of 3. The heartbeat-lost event will be sent a maximum of 10 minutes after the last heartbeat response from the partner cluster.

120sec (delay since last query) + 3*120sec (wait for normal response)
+ 120 sec (wait for retry response)

Delays can occur between the generation of the heartbeat-loss event and the triggering of the heartbeat-loss notification.

Note - A heartbeat-loss event does not necessarily indicate that the remote cluster has crashed.

The following sections describe how to configure the heartbeat-loss notification properties and how to create a custom action script that the Geographic Edition framework runs after a heartbeat-loss event.

Configuring the Heartbeat-Loss Notification Properties

You can configure heartbeat-loss notification by using two partnership properties, Notification_emailaddrs and Notification_actioncmd. You specify these properties by using the geops command.

You can specify these properties on the default heartbeat during partnership creation. For more information, see How to Create a Partnership in Oracle Solaris Cluster 4.3 Geographic Edition Installation and Configuration Guide. You can also modify these properties by using the procedure that is described in How to Modify the Heartbeat Properties.

If you want to be notified of heartbeat loss by email, set the Notification_emailaddrs property. You can specify a list of email addresses, separated by commas. If you want to use email notification, the cluster nodes must be configured as email clients. For more information about configuring mail services, see Chapter 5, Administering Mail Services in Managing sendmail Services in Oracle Solaris 11.3.

If you want to run a command in response to heartbeat loss, set the Notification_actioncmd property.

Example 16 Configuring Heartbeat-Loss Notification for an Existing Partnership

This example specifies a notification email address and a custom notification script for the partnership, paris-newyork-ps.

phys-paris-1# geops set-prop \
-p Notification_emailaddrs=ops@paris.example.com,ops@newyork.example.com \
-p Notification_actioncmd=/opt/hb_action.sh paris-newyork-ps

Creating an Action Shell Script for Heartbeat-Loss

You can create an action shell script that runs when the local cluster detects a heartbeat-loss in the partner cluster. The script runs with root permissions. The file must have root ownership and execution permissions, but the script should not have write permissions.

If you have configured the Notification_actioncmd property, the action command runs with arguments that provide information about the event in the following command:

# custom-action-command-path -c local-cluster-name -r remote-partner-cluster-name -e 1 \
-n nodename -t time

custom-action-command-path: Specifies a path to the action command you have created.
–c local-cluster-name: Specifies the name of the local cluster.
–p remote-partner-cluster-name: Specifies the name of the remote partner cluster.
–e1: Specifies that HBLOST=1, which indicates that a heartbeat-loss event has occurred. The Geographic Edition framework only supports heartbeat-loss notification, so –e 1 is the only value that can be passed to the action shell script.
–nnodename: Specifies the name of the cluster node that sent the heartbeat-loss event notification.
–t timestamp: Specifies the time of the heartbeat-loss event as the number of milliseconds since January 1, 1970, 00:00:00 GMT.

Caution - You can use this script to perform an automatic takeover on the secondary cluster. However, such an automated action is risky. If the heartbeat-loss notification is caused by a total loss of all heartbeat connectivity on both the primary and secondary clusters, such an automated action could lead to a situation where two primary clusters exist.

Example 17 How a Notification Action Script Parses the Command-Line Information Provided by the Geographic Edition Framework

This example displays the event information that is provided in the command-line being parsed in a notification action shell script.

#!/bin/sh

set -- `getopt abo: $*`
if [ $? != 0]
then
echo $USAGE
exit 2

fi
for i in $*
do

case $i in
-p)      PARTNER_CLUSTER=$1; shift;;
-e)      HB_EVENT=$2; shift;;
-c)      LOCAL_CLUSTER=$3; shift;;
-n)      EVENT_NODE=$4; shift;;
esac
done