Chapter 6 Administering Heartbeats

Sun Cluster Geographic Edition software uses heartbeats over the public network as a way for the individual clusters participating in partnerships to detect cluster failures at partner sites. The heartbeat monitor uses plug-in modules to query the heartbeat status of its partners.

This chapter contains the following sections:

• Introduction to Heartbeats

• Creating a Heartbeat

• Creating a Heartbeat Plug-in

• Modifying a Heartbeat Plug-in Property

• Deleting Heartbeats and Heartbeat Plug-ins

• Displaying Heartbeat Configuration Information

• Tuning the Heartbeat Properties

• Creating a Heartbeat That Uses a Custom Heartbeat Plug-in

• Configuring Heartbeat-Loss Notification

Introduction to Heartbeats

A heartbeat in Sun Cluster Geographic Edition is a container for a collection of heartbeat plug-ins. A heartbeat has a name and one property that you can tune, Query_interval. The Query_interval property specifies the delay between heartbeat status requests.

The heartbeat plug-in facilitates the actual physical monitoring activity. The plug-in is defined by a required query command or query library, an optional requester and responder agent, a type, and a Plugin_properties string.

The Sun Cluster Geographic Edition product provides the following default plug-ins:

• tcp_udp_plugin — Performs a simple heartbeat check on the cluster logical host IP address. If tcp_udp_plugin cannot use UDP port 2084, the plug-in tries to use TCP port 2084.

  ---

  Note –

  The Internet Assigned Numbers Authority (IANA) has officially assigned port number 2084 for use by the Sun Cluster Geographic Edition heartbeats.

  ---

• ping_plugin — Pings the cluster logical hostname on the remote cluster.

A default heartbeat that uses the default heartbeat plug-ins is created every time you run geops create or geops join without specifying a custom heartbeat. The name of the default heartbeat is hb_localclustername~remoteclustername. For more information about the geops command, refer to the geops(1M) man page.

You can create custom heartbeat plug-ins and associate them with existing default heartbeats or with new custom heartbeats.

Custom heartbeats are provided for special circumstances and require careful configuration. Consult your Sun specialist for assistance if your system requires the use of custom heartbeats.

If you create a custom heartbeat, you must add at least one plug-in to prevent the partnership from remaining in degraded mode.

Creating a Heartbeat

This section describes procedures for creating heartbeats.

How to Create a Heartbeat

Use this procedure to create a new heartbeat. To use the heartbeat with a partnership, you must create the heartbeat before you create a partnership. If you create a partnership before you create the custom heartbeat, the default heartbeat that is used by the partnership will prevent the custom heartbeat from being created.

If you create a custom heartbeat, you must add at least one plug-in to prevent the partnership from remaining in degraded mode.

A custom heartbeat prevents the default heartbeat from being used during partnership creation. If you want to use the default heartbeat for your partnership, you must delete the custom heartbeat before running the geops create command.

1. Log in to a cluster node.

   You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.

2. Create the heartbeat.

   # geohb create -r remoteclustername \ [-p propertysetting [-p…]] heartbeatname

   -r remoteclustername

   Specifies the name of the remote, secondary partner cluster.

   -p propertysetting

   Specifies a heartbeat property that is assigned a value by using a name=statement pair. Multiple properties might be set at one time by using multiple statements.

   For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.

   heartbeatname

   Specifies an identifier for the heartbeat.

   If you create a custom heartbeat, you must add at least one plug-in to prevent the partnership from remaining in degraded mode.

   ---

   Caution –

   The name of the custom heartbeat on each cluster in the same partnership must be different. Choose a name that identifies the heartbeat uniquely, such as paris-to-newyork on the cluster cluster-paris and newyork-to-paris on cluster cluster-newyork.

   ---

   For more information about the geohb command, refer to the geohb(1M) man page.

Example 6–1 Creating a Heartbeat

This example creates a heartbeat that is named paris-to-newyork.

# geohb create -r cluster-newyork paris-to-newyork

Creating a Heartbeat Plug-in

This section describes procedures for creating a heartbeat plug-in.

How to Create Heartbeat Plug-in

1. Log in to a cluster node.

   You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.

2. Add the heartbeat plug-in to an existing heartbeat.

   # geohb add-plugin heartbeatname pluginname \ [-p propertysetting [-p…]]

   heartbeatname

   Specifies the identifier for heartbeat on the local cluster.

   pluginname

   Specifies the name of the heartbeat plug-in.

   -ppropertysetting

   Specifies a heartbeat plug-in property that is assigned a value by using a name=statement pair. Multiple properties might be set at one time by using multiple statements.

   For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.

   For more information about the geohb command, refer to the geohb(1M) man page.

Example 6–2 Creating a Heartbeat Plug-in

This example creates a heartbeat plug-in that is named command1.

# geohb add-plugin paris-to-newyork command1 -p Query_cmd=/usr/bin/hb/

Modifying a Heartbeat Plug-in Property

This section describes procedures for modifying heartbeat plug-in properties. When you modify a plug-in property, your changes take effect immediately.

How to Modify the Properties of a Heartbeat Plug-in

1. Log in to a cluster node.

   You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.

2. Modify the heartbeat plug-in properties.

   # geohb modify-plugin -p propertysetting \ [-p…] pluginname heartbeatname

   heartbeatname

   Specifies an identifier for the heartbeat.

   pluginname

   Specifies the name of the heartbeat plug-in.

   -p propertysetting

   Specifies a heartbeat plug-in property that is assigned a value by using a name=statement pair. Multiple properties might be set at one time by using multiple statements.

   For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.

   ---

   Note –

   You cannot edit some properties of the default plug-ins.

   ---

   For information about the names and values that are supported by Sun Cluster Geographic Edition software, see Appendix B, Legal Names and Values of Sun Cluster Geographic Edition Entities.

   For more information about the geohb command, refer to the geohb(1M) man page.

Example 6–3 Modifying the Properties of the Heartbeat Plug-in

This example modifies the settings of the default TCP/UDP plug-in, tcp_udp_plugin, to use only TCP.

# geohb modify-plugin -p Plugin_properties=paris-cluster/TCP/2084 \
tcp_udp_plugin hb_cluster-paris~cluster-newyork

Deleting Heartbeats and Heartbeat Plug-ins

This section describes procedures for deleting heartbeats and heartbeat plug-ins.

How to Delete a Heartbeat

1. Log in to a cluster node.

   You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.

2. Delete the heartbeat.

   # geohb delete heartbeatname

   heartbeatname

   Specifies an identifier for the heartbeat settings.

   For more information about the geohb command, refer to the geohb(1M) man page.

Example 6–4 Deleting a Heartbeat

This example deletes a heartbeat that is named paris-to-newyork.

# geohb delete paris-to-newyork

How to Delete a Plug-in From a Heartbeat

1. Log in to a cluster node.

   You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.

2. Remove the plug-in from the heartbeat.

   # geohb remove-plugin pluginname heartbeatname

   ---

   Caution –

   Do not delete the default heartbeat plug-ins tcp_upd_plugin and ping_plugin.

   ---

   pluginname

   Specifies the name of the custom heartbeat plug-in

   heartbeatname

   Specifies an identifier for the heartbeat that contains this plug-in

   For information about the names and values that are supported by Sun Cluster Geographic Edition software, see Appendix B, Legal Names and Values of Sun Cluster Geographic Edition Entities.

   For more information about the geohb command, refer to the geohb(1M) man page.

Example 6–5 Deleting a Plug-in From a Heartbeat

This example removes the plug-in that is named command1 from the heartbeat that is named paris-to-newyork.

# geohb remove-plugin command1 paris-to-newyork

Displaying Heartbeat Configuration Information

This section describes procedures for displaying heartbeat configuration information.

How to Display Heartbeat Configuration Information

1. Log in a cluster node.

   You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.

2. Display the current configuration information for a specific heartbeat or the whole heartbeat subsystem.

   # geohb list [heartbeatnamelist]

   heartbeatnamelist

   Specifies the names of the heartbeats on the local cluster for which configuration information should be displayed.

   If you do not specify a list of heartbeat names, this command displays information about all the configured heartbeats.

   For more information about the geohb command, refer to the geohb(1M) man page.

Example 6–6 Displaying Heartbeat Configuration Information

This example displays information about the paris-to-newyork heartbeat.

# geohb list paris-to-newyork

Tuning the Heartbeat Properties

Default heartbeats are created as part of partnership creation. If you use a custom heartbeat, the custom heartbeat should be created before you create a partnership. You can modify the properties of the default and custom heartbeats by using the geohb set-prop command. For more information about this command, refer to the geohb(1M) man page.

Custom heartbeats are provided for special circumstances and require careful configuration. Consult your Sun specialist for assistance if your system requires the use of custom heartbeats.

If you modify the default value of the Query_interval property, ensure that the interval is sufficiently long. An interval that is too short causes a timeout and heartbeat-loss event before the logical hostname resource is available. This failover should result in no more than two unanswered heartbeat requests. Setting a default query_interval value of 120 seconds with the default heartbeat.retries parameter of 3 enables the peer cluster to be unresponsive for 6 minutes (120 * 3 ) without having a false failure declared.

The heartbeat.retries parameter is specified in the com.sun.cluster.agent.geocontol.xml file.

If you adjust the delay setting of the Query_interval property, ensure that the following condition is met:

Query_interval > worst-case logical-host failover time / 2

You must empirically determine the logical-host failover time for the cluster in question.

The following must be true to avoid false failures:

Query_interval > worst-case logical-host failover time / 3

You should not change the heartbeat.retries value. If you want to change the default value of the heartbeat.retries property, contact a Sun service representative.

How to Modify the Heartbeat Properties

1. Log in to a cluster node.

   You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.

2. Modify the heartbeat properties.

   # geohb set-prop -p propertysetting \ [-p…] heartbeatname

   -p propertysetting

   Specifies the default properties of the heartbeat.

   A heartbeat property is assigned a value by a name=statement pair. Multiple properties can be set at one time by using multiple statements.

   For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.

   heartbeatname

   Specifies an identifier for the heartbeat settings.

   For information about the names and values that are supported by Sun Cluster Geographic Edition software, see Appendix B, Legal Names and Values of Sun Cluster Geographic Edition Entities.

   For more information about the geohb command, refer to the geohb(1M) man page.

Example 6–7 Modifying the Properties of the Default Heartbeat

This example modifies the settings for the default heartbeat between cluster-paris and cluster-newyork.

# geohb set-prop -p Query_interval=60 hb_cluster-paris~cluster-newyork

Creating a Heartbeat That Uses a Custom Heartbeat Plug-in

You can create a custom heartbeat plug-in and configure an existing default heartbeat or a new custom heartbeat to use this custom heartbeat plug-in.

Custom heartbeats are provided for special circumstances and require careful configuration. Consult your Sun specialist for assistance if your system requires the use of custom heartbeats.

If you configure a custom heartbeat, ensure that the name of your custom heartbeat is different from the name of the custom heartbeat on the partner cluster.

The presence of a custom heartbeat prevents the default heartbeat from being used during partnership creation. If you want to use the default heartbeat for your partnership, you must delete the custom heartbeat before running the geops create command.

Creating a Custom Heartbeat Plug-in

When a heartbeat is created, your custom heartbeat plug-in is passed the following arguments by the Sun Cluster Geographic Edition software:

queryinterval

The value of the Query-interval property, which defines the delay in seconds after which a heartbeat status request is declared a failure.

mode

The mode for the plug-in startup, either Normal or Emergency.

pluginpropertyvalues

The value of the Plugin-properties property that is configured for the heartbeat plug-in, if any.

For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.

Your custom heartbeat plug-in is expected to check the heartbeat on the secondary cluster and return one of the following exit values:

• Zero, if successful — Indicates that the secondary cluster is alive

• Nonzero, on failure — Indicates that the secondary cluster did not respond to the heartbeat check

How to Add a Custom Heartbeat Plug-in to an Existing Default Heartbeat

1. Log in to a node in the primary cluster.

   You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.

2. Add the custom heartbeat plug-in to the default heartbeat.

   # geohb add-plugin -p propertysetting [-p...] \ pluginname hb_localclustername-remoteclustername

   -p propertysetting

   Specifies the properties of the heartbeat plug-in by using a name=statement pair.

   Specify the path to your custom heartbeat plug-in by using the Query_cmd property.

   For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.

   pluginname

   Specifies the name of the custom heartbeat plug-in.

   hb_localclustername-remoteclustername

   Specifies the name of the default heartbeat to which you want to add the custom heartbeat plug-in.

3. Verify that your changes were made correctly.

   # geoadm status

4. Repeat the previous steps on a node of the secondary cluster.

Example 6–8 Adding a Custom Heartbeat Plug-in to the Default Heartbeat

This example adds the custom heartbeat plug-in, command1, to the default heartbeat, hb_cluster-paris~cluster-newyork.

# geohb add-plugin -p query_cmd=/usr/bin/hb command1 \
hb_cluster-paris~cluster-newyork
# geoadm status

How to Create a Custom Heartbeat Plug-in and Add It to a Custom Heartbeat

1. Log in to a cluster node.

   You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.

2. Create the new custom heartbeat.

   # geohb create -r remoteclustername \ [-p propertysetting [-p…]] heartbeatname

   -r remoteclustername

   Specifies the name of the remote, secondary partner cluster.

   -p propertysetting

   Specifies the default properties of the heartbeat.

   A heartbeat property is assigned a value by a name=statement pair.

   For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.

   heartbeatname

   Specifies an identifier for the heartbeat settings.

   ---

   Caution –

   The name of the custom heartbeat on each cluster in the same partnership must be different. Choose a name that uniquely identifies the heartbeat, such as paris-to-newyork on the cluster cluster-paris and newyork-to-paris on cluster cluster-newyork.

   ---

   For more information about the geohb command, refer to the geohb(1M) man page.

3. Add the custom heartbeat plug-in to the heartbeat.

   # geohb add-plugin -p propertysetting [-p…] \ pluginname heartbeatname

   -p propertysetting

   Specifies the properties of the heartbeat plug-in by using a name=statement pair.

   Specify the path to your custom heartbeat plug-in by using the Query_cmd property.

   For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.

   pluginname

   Specifies the name of the custom heartbeat plug-in.

   heartbeatname

   Specifies an identifier for the heartbeat.

4. Create the partnership that will use the heartbeat that you created in the previous step.

   # geops create -c remoteclustername -h heartbeatname \ [-p propertysetting [-p…]] partnershipname

   -c remoteclustername

   Specifies the name of remote cluster that will participate in the partnership.

   This name matches the logical hostname used by the Sun Cluster Geographic Edition infrastructure on the remote cluster.

   -h heartbeatname

   Specifies the custom heartbeat to be used in the partnership to monitor the availability of the partner cluster.

   -p propertysetting

   Sets the value of partnership properties with a string of name=value pair statements.

   For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.

   partnershipname

   Specifies the name of the partnership.

   For more information about using geops create command to create a partnership, see How to Create a Partnership.

5. Verify that your changes were made correctly.

   # geoadm status

Example 6–9 Adding a Custom Heartbeat Plug-in to a New Custom Heartbeat

This example creates the heartbeat paris-to-newyork, which uses a custom heartbeat plug-in, and associates the heartbeat with a new partnership.

# geohb create -r cluster-newyork paris-to-newyork
# geohb add-plugin -p query_cmd=/usr/bin/hb/ command1 paris-to-newyork
# geops create -c cluster-newyork -h paris-to-newyork paris-newyork-ps
# geoadm status

Configuring Heartbeat-Loss Notification

You can configure the Sun Cluster Geographic Edition software to send email notification and to run an action script when a heartbeat is lost. You configure heartbeat-loss notification by using the optional Notification_emailaddrs and Notification_actioncmd properties.

Heartbeat-loss notification occurs if the heartbeat still fails after the interval you configure with the Query_interval property of the heartbeat. The heartbeat monitor sends out a heartbeat request to the responder on the logical host every Query_interval period. If no response is received within the Query_interval, an internal count is incremented. If the recount reaches the number that is specified in the heartbeat.retries property, the heartbeat is deemed to have failed.

For example, you can use the default Query_interval of 120 seconds and the default heartbeat.retries of 3. The heartbeat-lost event will be sent a maximum of 10 minutes after the last heartbeat response from the partner cluster.

120sec (delay since last query) + 3*120sec (wait for normal response) 
+ 120 sec (wait for retry response)

Delays can occur between the generation of the heartbeat-loss event and the triggering of the heartbeat-loss notification.

A heartbeat-loss event does not necessarily indicate that the remote cluster has crashed.

The following sections describe how to configure the heartbeat-loss notification properties and how to create a custom action script that the Sun Cluster Geographic Edition software runs after a heartbeat-loss event.

Configuring the Heartbeat-Loss Notification Properties

You can configure heartbeat-loss notification by using two partnership properties, Notification_emailaddrs and Notification_actioncmd. You specify these properties by using the geops command.

You can specify these properties on the default heartbeat during partnership creation. For more information, see How to Create a Partnership. You can also modify these properties by using the procedure that is described in How to Modify the Heartbeat Properties.

If you want to be notified of heartbeat loss by email, set the Notification_emailaddrs property. You can specify a list of email addresses, separated by commas. If you want to use email notification, the cluster nodes must be configured as email clients. For more information about configuring mail services, see the Solaris System Administration Guide: Network Services.

If you want to run a command in response to heartbeat loss, set the Notification_actioncmd property.

Example 6–10 Configuring Heartbeat-Loss Notification for an Existing Partnership

This example specifies a notification email address and a custom notification script for the partnership, paris-newyork-ps.

phys-paris-1# geops set-prop \
-p Notification_emailaddrs=ops@paris.com,ops@newyork.com \
-p Notification_actioncmd=/opt/hb_action.sh paris-newyork-ps

Creating an Action Shell Script for Heartbeat-Loss

You can create an action shell script that runs when the local cluster detects a heartbeat-loss in the partner cluster. The script runs with root permissions. The file must have root ownership and execution permissions, but the script should not have write permissions.

If you have configured the Notification_actioncmd property, the action command runs with arguments that provide information about the event in the following command line:

# customactioncommandpath -c localclustername -r remoteclustername -e 1 \
-n nodename -t time

customactioncommandpath

Specifies a path to the action command you have created.

-c localclustername

Specifies the name of the local cluster.

-p remoteclustername

Specifies the name of the remote partner cluster.

-e1

Specifies that HBLOST=1, which indicates that a heartbeat-loss event has occurred. The Sun Cluster Geographic Edition software only supports heartbeat-loss notification, so -e 1 is the only value that can be passed to the action shell script.

-nnodename

Specifies the name of the cluster node that sent the heartbeat-loss event notification.

-t timestamp

Specifies the time of the heartbeat-loss event as the number of milliseconds since January 1, 1970, 00:00:00 GMT.

You can use this script to perform an automatic takeover on the secondary cluster. However, such an automated action is risky. If the heartbeat-loss notification is caused by a total loss of all heartbeat connectivity on both the primary and secondary clusters, such an automated action could lead to a situation where two primary clusters exist.

Example 6–11 How a Notification Action Script Parses the Command-Line Information Provided by the Sun Cluster Geographic Edition Software

This example displays the event information that is provided in the command-line being parsed in a notification action shell script.

#!/bin/sh

set -- `getopt abo: $*`
if [ $? != 0]
then
      echo $USAGE
      exit 2

fi
for i in $*
do

      case $i in
      -p)      PARTNER_CLUSTER=$1; shift;;
      -e)      HB_EVENT=$2; shift;;
      -c)      LOCAL_CLUSTER=$3; shift;;
      -n)      EVENT_NODE=$4; shift;;
      esac
done