Sun Cluster Geographic Edition software uses heartbeats over the public network as a way for the individual clusters participating in partnerships to detect cluster failures at partner sites. The heartbeat monitor uses plug-in modules to query the heartbeat status of its partners.
This chapter contains the following sections:
A heartbeat in Sun Cluster Geographic Edition is a container for a collection of heartbeat plug-ins. A heartbeat has a name and one property that you can tune, Query_interval. The Query_interval property specifies the delay between heartbeat status requests.
The heartbeat plug-in facilitates the actual physical monitoring activity. The plug-in is defined by a required query command or query library, an optional requester and responder agent, a type, and a Plugin_properties string.
The Sun Cluster Geographic Edition product provides the following default plug-ins:
tcp_udp_plugin — Performs a simple heartbeat check on the cluster logical host IP address. If tcp_udp_plugin cannot use UDP port 2084, the plug-in tries to use TCP port 2084.
The Internet Assigned Numbers Authority (IANA) has officially assigned port number 2084 for use by the Sun Cluster Geographic Edition heartbeats.
ping_plugin — Pings the cluster logical hostname on the remote cluster.
A default heartbeat that uses the default heartbeat plug-ins is created every time you run geops create or geops join without specifying a custom heartbeat. The name of the default heartbeat is hb_localclustername~remoteclustername. For more information about the geops command, refer to the geops(1M) man page.
You can create custom heartbeat plug-ins and associate them with existing default heartbeats or with new custom heartbeats.
Custom heartbeats are provided for special circumstances and require careful configuration. Consult your Sun specialist for assistance if your system requires the use of custom heartbeats.
If you create a custom heartbeat, you must add at least one plug-in to prevent the partnership from remaining in degraded mode.
This section describes procedures for creating heartbeats.
Use this procedure to create a new heartbeat. To use the heartbeat with a partnership, you must create the heartbeat before you create a partnership. If you create a partnership before you create the custom heartbeat, the default heartbeat that is used by the partnership will prevent the custom heartbeat from being created.
If you create a custom heartbeat, you must add at least one plug-in to prevent the partnership from remaining in degraded mode.
A custom heartbeat prevents the default heartbeat from being used during partnership creation. If you want to use the default heartbeat for your partnership, you must delete the custom heartbeat before running the geops create command.
Log in to a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.
Create the heartbeat.
# geohb create -r remoteclustername \ [-p propertysetting [-p...]] heartbeatname |
Specifies the name of the remote, secondary partner cluster.
Specifies a heartbeat property that is assigned a value by using a name=statement pair. Multiple properties might be set at one time by using multiple statements.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.
Specifies an identifier for the heartbeat.
If you create a custom heartbeat, you must add at least one plug-in to prevent the partnership from remaining in degraded mode.
The name of the custom heartbeat on each cluster in the same partnership must be different. Choose a name that identifies the heartbeat uniquely, such as paris-to-newyork on the cluster cluster-paris and newyork-to-paris on cluster cluster-newyork.
For more information about the geohb command, refer to the geohb(1M) man page.
This example creates a heartbeat that is named paris-to-newyork.
# geohb create -r cluster-newyork paris-to-newyork |
This section describes procedures for creating a heartbeat plug-in.
Log in to a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.
Add the heartbeat plug-in to an existing heartbeat.
# geohb add-plugin heartbeatname pluginname \ [-p propertysetting [-p...]] |
Specifies the identifier for heartbeat on the local cluster.
Specifies the name of the heartbeat plug-in.
Specifies a heartbeat plug-in property that is assigned a value by using a name=statement pair. Multiple properties might be set at one time by using multiple statements.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.
For more information about the geohb command, refer to the geohb(1M) man page.
This example creates a heartbeat plug-in that is named command1.
# geohb add-plugin paris-to-newyork command1 -p Query_cmd=/usr/bin/hb/ |
This section describes procedures for modifying heartbeat plug-in properties. When you modify a plug-in property, your changes take effect immediately.
Log in to a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.
Modify the heartbeat plug-in properties.
# geohb modify-plugin -p propertysetting \ [-p...] pluginname heartbeatname |
Specifies an identifier for the heartbeat.
Specifies the name of the heartbeat plug-in.
Specifies a heartbeat plug-in property that is assigned a value by using a name=statement pair. Multiple properties might be set at one time by using multiple statements.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.
You cannot edit some properties of the default plug-ins.
For information about the names and values that are supported by Sun Cluster Geographic Edition software, see Appendix B, Legal Names and Values of Sun Cluster Geographic Edition Entities.
For more information about the geohb command, refer to the geohb(1M) man page.
This example modifies the settings of the default TCP/UDP plug-in, tcp_udp_plugin, to use only TCP.
# geohb modify-plugin -p Plugin_properties=paris-cluster/TCP/2084 \ tcp_udp_plugin hb_cluster-paris~cluster-newyork |
This section describes procedures for deleting heartbeats and heartbeat plug-ins.
Log in to a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.
Delete the heartbeat.
# geohb delete heartbeatname |
Specifies an identifier for the heartbeat settings.
For more information about the geohb command, refer to the geohb(1M) man page.
This example deletes a heartbeat that is named paris-to-newyork.
# geohb delete paris-to-newyork |
Log in to a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.
Remove the plug-in from the heartbeat.
# geohb remove-plugin pluginname heartbeatname |
Do not delete the default heartbeat plug-ins tcp_upd_plugin and ping_plugin.
Specifies the name of the custom heartbeat plug-in
Specifies an identifier for the heartbeat that contains this plug-in
For information about the names and values that are supported by Sun Cluster Geographic Edition software, see Appendix B, Legal Names and Values of Sun Cluster Geographic Edition Entities.
For more information about the geohb command, refer to the geohb(1M) man page.
This example removes the plug-in that is named command1 from the heartbeat that is named paris-to-newyork.
# geohb remove-plugin command1 paris-to-newyork |
This section describes procedures for displaying heartbeat configuration information.
Log in a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.
Display the current configuration information for a specific heartbeat or the whole heartbeat subsystem.
# geohb list [heartbeatnamelist] |
Specifies the names of the heartbeats on the local cluster for which configuration information should be displayed.
If you do not specify a list of heartbeat names, this command displays information about all the configured heartbeats.
For more information about the geohb command, refer to the geohb(1M) man page.
This example displays information about the paris-to-newyork heartbeat.
# geohb list paris-to-newyork |
Default heartbeats are created as part of partnership creation. If you use a custom heartbeat, the custom heartbeat should be created before you create a partnership. You can modify the properties of the default and custom heartbeats by using the geohb set-prop command. For more information about this command, refer to the geohb(1M) man page.
Custom heartbeats are provided for special circumstances and require careful configuration. Consult your Sun specialist for assistance if your system requires the use of custom heartbeats.
If you modify the default value of the Query_interval property, ensure that the interval is sufficiently long. An interval that is too short causes a timeout and heartbeat-loss event before the logical hostname resource is available. This failover should result in no more than two unanswered heartbeat requests. Setting a default query_interval value of 120 seconds with the default heartbeat.retries parameter of 3 enables the peer cluster to be unresponsive for 6 minutes (120 * 3 ) without having a false failure declared.
The heartbeat.retries parameter is specified in the com.sun.cluster.agent.geocontol.xml file.
If you adjust the delay setting of the Query_interval property, ensure that the following condition is met:
Query_interval > worst-case logical-host failover time / 2 |
You must empirically determine the logical-host failover time for the cluster in question.
The following must be true to avoid false failures:
Query_interval > worst-case logical-host failover time / 3 |
You should not change the heartbeat.retries value. If you want to change the default value of the heartbeat.retries property, contact a Sun service representative.
Log in to a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.
Modify the heartbeat properties.
# geohb set-prop -p propertysetting \ [-p...] heartbeatname |
Specifies the default properties of the heartbeat.
A heartbeat property is assigned a value by a name=statement pair. Multiple properties can be set at one time by using multiple statements.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.
Specifies an identifier for the heartbeat settings.
For information about the names and values that are supported by Sun Cluster Geographic Edition software, see Appendix B, Legal Names and Values of Sun Cluster Geographic Edition Entities.
For more information about the geohb command, refer to the geohb(1M) man page.
This example modifies the settings for the default heartbeat between cluster-paris and cluster-newyork.
# geohb set-prop -p Query_interval=60 hb_cluster-paris~cluster-newyork |
You can create a custom heartbeat plug-in and configure an existing default heartbeat or a new custom heartbeat to use this custom heartbeat plug-in.
Custom heartbeats are provided for special circumstances and require careful configuration. Consult your Sun specialist for assistance if your system requires the use of custom heartbeats.
If you configure a custom heartbeat, ensure that the name of your custom heartbeat is different from the name of the custom heartbeat on the partner cluster.
The presence of a custom heartbeat prevents the default heartbeat from being used during partnership creation. If you want to use the default heartbeat for your partnership, you must delete the custom heartbeat before running the geops create command.
When a heartbeat is created, your custom heartbeat plug-in is passed the following arguments by the Sun Cluster Geographic Edition software:
The value of the Query-interval property, which defines the delay in seconds after which a heartbeat status request is declared a failure.
The mode for the plug-in startup, either Normal or Emergency.
The value of the Plugin-properties property that is configured for the heartbeat plug-in, if any.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.
Your custom heartbeat plug-in is expected to check the heartbeat on the secondary cluster and return one of the following exit values:
Zero, if successful — Indicates that the secondary cluster is alive
Nonzero, on failure — Indicates that the secondary cluster did not respond to the heartbeat check
Log in to a node in the primary cluster.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.
Add the custom heartbeat plug-in to the default heartbeat.
# geohb add-plugin -p propertysetting [-p...] \ pluginname hb_localclustername-remoteclustername |
Specifies the properties of the heartbeat plug-in by using a name=statement pair.
Specify the path to your custom heartbeat plug-in by using the Query_cmd property.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.
Specifies the name of the custom heartbeat plug-in.
Specifies the name of the default heartbeat to which you want to add the custom heartbeat plug-in.
Verify that your changes were made correctly.
# geoadm status |
Repeat the previous steps on a node of the secondary cluster.
This example adds the custom heartbeat plug-in, command1, to the default heartbeat, hb_cluster-paris~cluster-newyork.
# geohb add-plugin -p query_cmd=/usr/bin/hb command1 \ hb_cluster-paris~cluster-newyork # geoadm status |
Log in to a cluster node.
You must be assigned the Geo Management RBAC rights profile to complete this procedure. For more information about RBAC, see Sun Cluster Geographic Edition Software and RBAC.
Create the new custom heartbeat.
# geohb create -r remoteclustername \ [-p propertysetting [-p...]] heartbeatname |
Specifies the name of the remote, secondary partner cluster.
Specifies the default properties of the heartbeat.
A heartbeat property is assigned a value by a name=statement pair.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.
Specifies an identifier for the heartbeat settings.
The name of the custom heartbeat on each cluster in the same partnership must be different. Choose a name that uniquely identifies the heartbeat, such as paris-to-newyork on the cluster cluster-paris and newyork-to-paris on cluster cluster-newyork.
For more information about the geohb command, refer to the geohb(1M) man page.
Add the custom heartbeat plug-in to the heartbeat.
# geohb add-plugin -p propertysetting [-p...] \ pluginname heartbeatname |
Specifies the properties of the heartbeat plug-in by using a name=statement pair.
Specify the path to your custom heartbeat plug-in by using the Query_cmd property.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.
Specifies the name of the custom heartbeat plug-in.
Specifies an identifier for the heartbeat.
Create the partnership that will use the heartbeat that you created in the previous step.
# geops create -c remoteclustername -h heartbeatname \ [-p propertysetting [-p...]] partnershipname |
Specifies the name of remote cluster that will participate in the partnership.
This name matches the logical hostname used by the Sun Cluster Geographic Edition infrastructure on the remote cluster.
Specifies the custom heartbeat to be used in the partnership to monitor the availability of the partner cluster.
Sets the value of partnership properties with a string of name=value pair statements.
For more information about the properties you can set, see Appendix A, Standard Sun Cluster Geographic Edition Properties.
Specifies the name of the partnership.
For more information about using geops create command to create a partnership, see How to Create a Partnership.
Verify that your changes were made correctly.
# geoadm status |
This example creates the heartbeat paris-to-newyork, which uses a custom heartbeat plug-in, and associates the heartbeat with a new partnership.
# geohb create -r cluster-newyork paris-to-newyork # geohb add-plugin -p query_cmd=/usr/bin/hb/ command1 paris-to-newyork # geops create -c cluster-newyork -h paris-to-newyork paris-newyork-ps # geoadm status |
You can configure the Sun Cluster Geographic Edition software to send email notification and to run an action script when a heartbeat is lost. You configure heartbeat-loss notification by using the optional Notification_emailaddrs and Notification_actioncmd properties.
Heartbeat-loss notification occurs if the heartbeat still fails after the interval you configure with the Query_interval property of the heartbeat. The heartbeat monitor sends out a heartbeat request to the responder on the logical host every Query_interval period. If no response is received within the Query_interval, an internal count is incremented. If the recount reaches the number that is specified in the heartbeat.retries property, the heartbeat is deemed to have failed.
For example, you can use the default Query_interval of 120 seconds and the default heartbeat.retries of 3. The heartbeat-lost event will be sent a maximum of 10 minutes after the last heartbeat response from the partner cluster.
120sec (delay since last query) + 3*120sec (wait for normal response) + 120 sec (wait for retry response) |
Delays can occur between the generation of the heartbeat-loss event and the triggering of the heartbeat-loss notification.
A heartbeat-loss event does not necessarily indicate that the remote cluster has crashed.
The following sections describe how to configure the heartbeat-loss notification properties and how to create a custom action script that the Sun Cluster Geographic Edition software runs after a heartbeat-loss event.
You can configure heartbeat-loss notification by using two partnership properties, Notification_emailaddrs and Notification_actioncmd. You specify these properties by using the geops command.
You can specify these properties on the default heartbeat during partnership creation. For more information, see How to Create a Partnership. You can also modify these properties by using the procedure that is described in How to Modify the Heartbeat Properties.
If you want to be notified of heartbeat loss by email, set the Notification_emailaddrs property. You can specify a list of email addresses, separated by commas. If you want to use email notification, the cluster nodes must be configured as email clients. For more information about configuring mail services, see the Solaris System Administration Guide: Network Services.
If you want to run a command in response to heartbeat loss, set the Notification_actioncmd property.
This example specifies a notification email address and a custom notification script for the partnership, paris-newyork-ps.
phys-paris-1# geops set-prop \ -p Notification_emailaddrs=ops@paris.com,ops@newyork.com \ -p Notification_actioncmd=/opt/hb_action.sh paris-newyork-ps |
You can create an action shell script that runs when the local cluster detects a heartbeat-loss in the partner cluster. The script runs with root permissions. The file must have root ownership and execution permissions, but the script should not have write permissions.
If you have configured the Notification_actioncmd property, the action command runs with arguments that provide information about the event in the following command line:
# customactioncommandpath -c localclustername -r remoteclustername -e 1 \ -n nodename -t time |
Specifies a path to the action command you have created.
Specifies the name of the local cluster.
Specifies the name of the remote partner cluster.
Specifies that HBLOST=1, which indicates that a heartbeat-loss event has occurred. The Sun Cluster Geographic Edition software only supports heartbeat-loss notification, so -e 1 is the only value that can be passed to the action shell script.
Specifies the name of the cluster node that sent the heartbeat-loss event notification.
Specifies the time of the heartbeat-loss event as the number of milliseconds since January 1, 1970, 00:00:00 GMT.
You can use this script to perform an automatic takeover on the secondary cluster. However, such an automated action is risky. If the heartbeat-loss notification is caused by a total loss of all heartbeat connectivity on both the primary and secondary clusters, such an automated action could lead to a situation where two primary clusters exist.
This example displays the event information that is provided in the command-line being parsed in a notification action shell script.
#!/bin/sh set -- `getopt abo: $*` if [ $? != 0] then echo $USAGE exit 2 fi for i in $* do case $i in -p) PARTNER_CLUSTER=$1; shift;; -e) HB_EVENT=$2; shift;; -c) LOCAL_CLUSTER=$3; shift;; -n) EVENT_NODE=$4; shift;; esac done |