Sun Cluster Data Services Developer's Guide for Solaris OS

Chapter 5 Sample Data Service

This chapter describes a sample Sun Cluster data service, HA-DNS, for the in.named application. The in.named daemon is the Solaris implementation of the Domain Name Service (DNS). The sample data service demonstrates how to make an application highly available, using the Resource Management API.

The Resource Management API supports a shell script interface and a C program interface. The sample application in this chapter is written using the shell script interface.

This chapter covers the following topics:

Overview of the Sample Data Service

The sample data service starts, stops, restarts, and switches the DNS application among the nodes of the cluster in response to cluster events, such as administrative action, application failure, or node failure.

Application restart is managed by the Process Monitor Facility (PMF). If the number of applications that die exceeds the failure count within the failure time window, the fault monitor fails over the resource group that contains the application resource to another node.

The sample data service provides fault monitoring in the form of a PROBE method that uses the nslookup command to ensure that the application is healthy. If the probe detects a hung DNS service, the probe tries to correct the situation by restarting the DNS application locally. If restarting the DNS application locally does not improve the situation and the probe repeatedly detects problems with the service, the probe attempts to fail over the service to another node in the cluster.

Specifically, the sample data service includes the following elements:

Defining the Resource Type Registration File

The resource type registration (RTR) file in this example defines the static configuration of the DNS resource type. Resources of this type inherit the properties that are defined in the RTR file.

The information in the RTR file is read by the Resource Group Manager (RGM) when the cluster administrator registers the HA-DNS data service. By convention, you place the RTR file in the /opt/cluster/lib/rgm/rtreg/ directory. Note that the package installer places the RTR file that Agent Builder creates in this directory as well.

Overview of the RTR File

The RTR file follows a well-defined format. Resource type properties are defined first in the file, system-defined resource properties are defined next, and extension properties are defined last. See the rt_reg(4) man page and Setting Resource and Resource Type Properties for more information.

The following sections describe the specific properties in the sample RTR file. These sections provide listings of different parts of the file. For a complete listing of the contents of the sample RTR file, see Resource Type Registration File Listing.

Resource Type Properties in the Sample RTR File

The sample RTR file begins with comments followed by resource type properties that define the HA-DNS configuration, as shown in the following listing.


Note –

Property names for resource groups, resources, and resource types are not case sensitive. You can use any combination of uppercase and lowercase letters when you specify property names.


#
# Copyright (c) 1998-2006 by Sun Microsystems, Inc.
# All rights reserved.
#
# Registration information for Domain Name Service (DNS)
#

#pragma ident   “@(#)SUNW.sample   1.1   00/05/24 SMI”

Resource_type = “sample”;
Vendor_id = SUNW;
RT_description = “Domain Name Service on Sun Cluster”;

RT_version =”1.0”; 
API_version = 2;    
Failover = TRUE;

RT_basedir=/opt/SUNWsample/bin;
Pkglist = SUNWsample;

Start         =   dns_svc_start;
Stop          =   dns_svc_stop;

Validate      =   dns_validate;
Update        =   dns_update;

Monitor_start =   dns_monitor_start;
Monitor_stop  =   dns_monitor_stop;
Monitor_check =   dns_monitor_check;

Tip –

You must declare the Resource_type property as the first entry in the RTR file. Otherwise, registration of the resource type fails.


The following information describes these properties:

Resource type properties that are not specified in this RTR file, such as Single_instance, Init_nodes, and Installed_nodes, are set to their default values. Resource Type Properties contains a complete list of the resource type properties, including their default values.

The cluster administrator cannot change the values for resource type properties in the RTR file.

Resource Properties in the Sample RTR File

By convention, you declare resource properties after the resource type properties in the RTR file. Resource properties include system-defined properties that are provided by the Sun Cluster software and extension properties that you define. For either type, you can specify a number of property attributes that are supplied by the Sun Cluster software, such as minimum, maximum, and default values.

System-Defined Properties in the RTR File

The following listing shows the system-defined properties in a sample RTR file.

# A list of bracketed resource property declarations follows the
# resource type declarations. The property-name declaration must be
# the first attribute after the open curly bracket of each entry.

# The <method>_timeout properties set the value in seconds after which
# the RGM concludes invocation of the method has failed.

# The MIN value for all method timeouts is set to 60 seconds. This
# prevents administrators from setting shorter timeouts, which do not
# improve switchover/failover performance, and can lead to undesired
# RGM actions (false failovers, node reboot, or moving the resource group
# to ERROR_STOP_FAILED state, requiring operator intervention). Setting
# too-short method timeouts leads to a *decrease* in overall availability
# of the data service.
{
   PROPERTY = Start_timeout; 
   MIN=60; 
   DEFAULT=300;
}

{
   PROPERTY = Stop_timeout; 
   MIN=60; 
   DEFAULT=300;
}
{
   PROPERTY = Validate_timeout;
   MIN=60;
   DEFAULT=300;
}
{
   PROPERTY = Update_timeout;
   MIN=60;
   DEFAULT=300;
}
{
   PROPERTY = Monitor_Start_timeout;
   MIN=60;
   DEFAULT=300;
}
{
   PROPERTY = Monitor_Stop_timeout;
   MIN=60;
   DEFAULT=300;
}
{
   PROPERTY = Thorough_Probe_Interval;
   MIN=1;
   MAX=3600;
   DEFAULT=60;
   TUNABLE = ANYTIME;
}

# The number of retries to be done within a certain period before concluding 
# that the application cannot be successfully started on this node.
{
   PROPERTY = Retry_count;
   MIN=0;
   MAX=10;
   DEFAULT=2;
   TUNABLE = ANYTIME; 
}

# Set Retry_interval as a multiple of 60 since it is converted from seconds
# to minutes, rounding up. For example, a value of 50 (seconds)
# is converted to 1 minute. Use this property to time the number of 
# retries (Retry_count).
{
   PROPERTY = Retry_interval;
   MIN=60;
   MAX=3600;
   DEFAULT=300;
   TUNABLE = ANYTIME;
}

{
   PROPERTY = Network_resources_used;
   TUNABLE = AT_CREATION;
   DEFAULT = ““;
}

Although the Sun Cluster software provides the system-defined properties, you can set different default values by using resource property attributes. See Resource Property Attributes for a complete list of attributes that are available to you to apply to resource properties.

Note the following points about the system-defined resource properties in the sample RTR file:

Extension Properties in the RTR File

At the end of the sample RTR file are extension properties, as shown in this listing.

# Extension Properties

# The cluster administrator must set the value of this property to point to the
# directory that contains the configuration files used by the application.
# For this application, DNS, specify the path of the DNS configuration file on
# PXFS (typically named.conf).
{
   PROPERTY = Confdir;
   EXTENSION;
   STRING;
   TUNABLE = AT_CREATION;
   DESCRIPTION = “The Configuration Directory Path”;
}

# Time out value in seconds before declaring the probe as failed.
{
   PROPERTY = Probe_timeout;
   EXTENSION;
   INT;
   DEFAULT = 120;
   TUNABLE = ANYTIME;
   DESCRIPTION = “Time out value for the probe (seconds)”;
}

The sample RTR file defines two extension properties, Confdir and Probe_timeout. The Confdir property specifies the path to the DNS configuration directory. This directory contains the in.named file, which DNS requires to operate successfully. The sample data service's Start and Validate methods use this property to verify that the configuration directory and the in.named file are accessible before starting DNS.

When the data service is configured, the Validate method verifies that the new directory is accessible.

The sample data service's PROBE method is not a Sun Cluster callback method but a user-defined method. Therefore, Sun Cluster does not provide a Probe_timeout property for it. You need to define an extension property in the RTR file to enable a cluster administrator to configure a Probe_timeout value.

Providing Common Functionality to All Methods

This section describes the following functionality that is used in all callback methods of the sample data service:

Identifying the Command Interpreter and Exporting the Path

The first line of a shell script must identify the command interpreter. Each method script in the sample data service identifies the command interpreter, as follows:

#!/bin/ksh

All method scripts in the sample application export the path to the Sun Cluster binaries and libraries rather than relying on the user's PATH settings.

#######################################################################
# MAIN
#######################################################################

export PATH=/bin:/usr/bin:/usr/cluster/bin:/usr/sbin:/usr/proc/bin:$PATH

Declaring the PMF_TAG and SYSLOG_TAG Variables

All the method scripts, except Validate, use the pmfadm command to start or to stop either the data service or the monitor, and to pass the name of the resource. Each script defines a variable, PMF_TAG, that can be passed to the pmfadm command to identify either the data service or the monitor.

Likewise, each method script uses the logger command to log messages in the system log. Each script defines a variable, SYSLOG_TAG, that can be passed to logger with the -t option to identify the resource type, resource name, and resource group of the resource for which the message is being logged.

All methods define SYSLOG_TAG in the same way, as shown in the following sample code. The dns_probe, dns_svc_start, dns_svc_stop, and dns_monitor_check methods define PMF_TAG as follows (the use of pmfadm and logger is from the dns_svc_stop method).

#########################################################################
# MAIN
#########################################################################

PMF_TAG=$RESOURCE_NAME.named

SYSLOG_TAG=$RESOURCETYPE_NAME,$RESOURCEGROUP_NAME,$RESOURCE_NAME

   # Send a SIGTERM signal to the data service and wait for 80% of the
   # total timeout value.
   pmfadm -s $PMF_TAG.named -w $SMOOTH_TIMEOUT TERM
   if [ $? -ne 0 ]; then
      logger -p ${SYSLOG_FACILITY}.info \
          -t [$SYSLOG_TAG] \
          “${ARGV0} Failed to stop HA-DNS with SIGTERM; Retry with \
           SIGKILL”

The dns_monitor_start, dns_monitor_stop, and dns_update methods define PMF_TAG as follows (the use of pmfadm is from the dns_monitor_stop method):

#####################################################################
# MAIN
#####################################################################

PMF_TAG=$RESOURCE_NAME.monitor
SYSLOG_TAG=$RESOURCETYPE_NAME,$RESOURCEGROUP_NAME,$RESOURCE_NAME
...
# See if the monitor is running, and if so, kill it. 
if pmfadm -q $PMF_TAG.monitor; then
   pmfadm -s $PMF_TAG.monitor KILL

Parsing the Function Arguments

The RGM runs all of the callback methods, except Validate, as follows:

method-name -R resource-name -T resource-type-name -G resource-group-name

The method name is the path name of the program that implements the callback method. A data service specifies the path name for each method in the RTR file. These path names are relative to the directory that is specified by the RT_basedir property, also in the RTR file. For example, in the sample data service's RTR file, the base directory and method names are specified as follows:

RT_basedir=/opt/SUNWsample/bin;
Start = dns_svc_start;
Stop =  dns_svc_stop;
...

All callback method arguments are passed as flagged values. The -R argument indicates the name of the resource instance. The -T argument indicates the type of the resource. The -G argument indicates the group into which the resource is configured. See the rt_callbacks(1HA) man page for more information about callback methods.


Note –

The Validate method is called with additional arguments, that is, the property values of the resource and resource group on which it is called. See Handling Property Updates for more information.


Each callback method needs a function to parse the arguments that the function is passed. Because the callbacks are all passed the same arguments, the data service provides a single parse function that is used in all the callbacks in the application.

The following sample shows the parse_args() function that is used for the callback methods in the sample application.

#########################################################################
# Parse program arguments.
#
function parse_args # [args ...]
{
      typeset opt

      while getopts 'R:G:T:' opt
      do
             case "$opt" in
             R)
                  # Name of the DNS resource.
                  RESOURCE_NAME=$OPTARG
                  ;;
             G)
                  # Name of the resource group in which the resource is
                  # configured.
                  RESOURCEGROUP_NAME=$OPTARG
                  ;;
             T)
                  # Name of the resource type.
                  RESOURCETYPE_NAME=$OPTARG
                  ;;
             *)
                  logger -p ${SYSLOG_FACILITY}.err \
                  -t [$RESOURCETYPE_NAME,$RESOURCEGROUP_NAME,$RESOURCE_NAME] \
                  "ERROR: Option $OPTARG unknown"
                  exit 1
                      ;;
             esac
    done
}

Note –

Although the PROBE method in the sample application is user defined (not a Sun Cluster callback method), it is called with the same arguments as the callback methods. Therefore, this method contains a parse function that is identical to the one that is used by the other callback methods.


The parse function is called in MAIN as:

parse_args “$@”

Generating Error Messages

Callback methods should use the syslog() function to output error messages to end users. All callback methods in the sample data service use the scha_cluster_get command to retrieve the number of the syslog() function that is used for the cluster log, as follows:

SYSLOG_FACILITY=`scha_cluster_get -O SYSLOG_FACILITY`

The value is stored in a shell variable, SYSLOG_FACILITY, and can be used as the facility of the logger command to log messages in the cluster log. For example, the Start method in the sample data service retrieves the syslog() function and logs a message that the data service has been started, as follows:

SYSLOG_FACILITY=`scha_cluster_get -O SYSLOG_FACILITY`
...
if [ $? -eq 0 ]; then
   logger -p ${SYSLOG_FACILITY}.err \
         -t [$SYSLOG_TAG] \
         "${ARGV0} HA-DNS successfully started"
fi

See the scha_cluster_get(1HA) man page for more information.

Obtaining Property Information

Most callback methods need to obtain information about resource and resource type properties of the data service. The API provides the scha_resource_get() function for this purpose.

Both system-defined properties and extension properties are available. System-defined properties are predefined. You define extension properties in the RTR file.

When you use scha_resource_get() to obtain the value of a system-defined property, you specify the name of the property with the -O option. The command returns only the value of the property. For example, in the sample data service, the Monitor_start method needs to locate the probe program so it can start it. The probe program is located in the base directory for the data service, which is pointed to by the RT_basedir property. The Monitor_start method retrieves the value of RT_basedir and places it in the RT_BASEDIR variable, as follows:

RT_BASEDIR=`scha_resource_get -O RT_basedir -R $RESOURCE_NAME -G \
$RESOURCEGROUP_NAME`

For extension properties, you must use the -O option to specify that the property is an extension property. You must also supply the name of the property as the last argument. For extension properties, the command returns both the type and value of the property. For example, in the sample data service, the probe program retrieves the type and value of the Probe_timeout extension property, and uses the awk command to put the value only in the PROBE_TIMEOUT shell variable, as follows:

probe_timeout_info=`scha_resource_get -O Extension \
-R $RESOURCE_NAME -G $RESOURCEGROUP_NAME Probe_timeout`
PROBE_TIMEOUT=`echo $probe_timeout_info | awk '{print $2}'`

Controlling the Data Service

A data service must provide a Start or Prenet_start method to activate the application daemon in the cluster, and a Stop or Postnet_stop method to stop the application daemon in the cluster. The sample data service implements a Start and a Stop method. See Deciding Which Start and Stop Methods to Use for information about when to use Prenet_start and Postnet_stop instead.

How the Start Method Works

The RGM runs the Start method on a cluster node when the resource group that contains the data service resource is brought online on that node or when the resource group is already online and the resource is enabled. In the sample application, the Start method activates the in.named DNS daemon on the global-cluster voting node on that host.

This section describes the major pieces of the Start method for the sample application. This section does not describe functionality that is common to all callback methods, such as the parse_args() function. This section also does not describe using the syslog() function. Common functionality is described in Providing Common Functionality to All Methods.

For the complete listing of the Start method, see Start Method Code Listing.

What the Start Method Does

Before attempting to start DNS, the Start method in the sample data service verifies that the configuration directory and configuration file (named.conf) are accessible and available. Information in named.conf is essential to the successful operation of DNS.

This callback method uses the PMF (pmfadm) to start the DNS daemon (in.named). If DNS crashes or fails to start, the PMF attempts to start the DNS daemon a prescribed number of times during a specified interval. The number of retries and the interval are specified by properties in the data service's RTR file.

Verifying the Configuration

In order to operate, DNS requires information from the named.conf file in the configuration directory. Therefore, the Start method performs some sanity checks to verify that the directory and file are accessible before attempting to start DNS.

The Confdir extension property provides the path to the configuration directory. The property itself is defined in the RTR file. However, the cluster administrator specifies the actual location when the cluster administrator configures the data service.

In the sample data service, the Start method retrieves the location of the configuration directory by using the scha_resource_get() function.


Note –

Because Confdir is an extension property, scha_resource_get() returns both the type and value. The awk command retrieves just the value and places that value in a shell variable, CONFIG_DIR.


# find the value of Confdir set by the cluster administrator at the time of
# adding the resource.
config_info=`scha_resource_get -O Extension -R $RESOURCE_NAME \
-G $RESOURCEGROUP_NAME Confdir`

# scha_resource_get returns the "type" as well as the "value" for the
# extension properties. Get only the value of the extension property 
CONFIG_DIR=`echo $config_info | awk '{print $2}'`

The Start method uses the value of CONFIG_DIR to verify that the directory is accessible. If it is not accessible, Start logs an error message and exits with an error status. See Start Exit Status.

# Check if $CONFIG_DIR is accessible.
if [ ! -d $CONFIG_DIR ]; then
   logger -p ${SYSLOG_FACILITY}.err \
         -t [$SYSLOG_TAG] \
         "${ARGV0} Directory $CONFIG_DIR is missing or not mounted"
   exit 1
fi

Before starting the application daemon, this method performs a final check to verify that the named.conf file is present. If the file is not present, Start logs an error message and exits with an error status.

# Change to the $CONFIG_DIR directory in case there are relative
# pathnames in the data files.
cd $CONFIG_DIR

# Check that the named.conf file is present in the $CONFIG_DIR directory
if [ ! -s named.conf ]; then
   logger -p ${SYSLOG_FACILITY}.err \
         -t [$SYSLOG_TAG] \
         "${ARGV0} File $CONFIG_DIR/named.conf is missing or empty"
   exit 1
fi

Starting the Application

This method uses the process manager facility (pmfadm) to start the application. The pmfadm command enables you to set the number of times to try to restart the application during a specified time frame. The RTR file contains two properties: Retry_count specifies the number of times to attempt restarting an application, and Retry_interval specifies the time period over which to do so.

The Start method retrieves the values of Retry_count and Retry_interval by using the scha_resource_get() function and stores their values in shell variables. The Start method passes these values to pmfadm by using the -n and -t options.

# Get the value for retry count from the RTR file.
RETRY_CNT=`scha_resource_get -O Retry_count -R $RESOURCE_NAME \
-G $RESOURCEGROUP_NAME`
# Get the value for retry interval from the RTR file. This value is in seconds
# and must be converted to minutes for passing to pmfadm. Note that the 
# conversion rounds up; for example, 50 seconds rounds up to 1 minute.
((RETRY_INTRVAL=`scha_resource_get -O Retry_interval -R $RESOURCE_NAME \
-G $RESOURCEGROUP_NAME` / 60))

# Start the in.named daemon under the control of PMF. Let it crash and restart
# up to $RETRY_COUNT times in a period of $RETRY_INTERVAL; if it crashes
# more often than that, PMF will cease trying to restart it.
# If there is a process already registered under the tag
# <$PMF_TAG>, then PMF sends out an alert message that the
# process is already running.
pmfadm -c $PMF_TAAG -n $RETRY_CNT -t $RETRY_INTRVAL \
    /usr/sbin/in.named -c named.conf

# Log a message indicating that HA-DNS has been started.
if [ $? -eq 0 ]; then
   logger -p ${SYSLOG_FACILITY}.err \
         -t [$SYSLOG_TAG] \
         "${ARGV0} HA-DNS successfully started"
fi
exit 0

Start Exit Status

A Start method should not exit with success until the underlying application is actually running and is available, particularly if other data services depend on it. One way to verify success is to probe the application to make sure that it is running before exiting the Start method. For a complex application, such as a database, be certain to set the value for the Start_timeout property in the RTR file sufficiently high to allow time for the application to initialize and recover from a crash.


Note –

Because the application resource (DNS) in the sample data service starts quickly, the sample data service does not poll to verify that it is running before exiting with success.


If this method fails to start DNS and exits with failure status, the RGM checks the Failover_mode property, which determines how to react. The sample data service does not explicitly set the Failover_mode property, so this property has the default value NONE (unless the cluster administrator overrides the default value and specifies a different value). In this case, the RGM takes no action other than to set the state of the data service. The cluster administrator needs to initiate a restart on the same node or a fail over to a different node.

How the Stop Method Works

The RGM runs the Stop method on a cluster node when the resource group that contains the HA-DNS resource is brought offline on that node or if the resource group is online and the resource is disabled. This method stops the in.named (DNS) daemon on that node.

This section describes the major pieces of the Stop method for the sample application. This section does not describe functionality that is common to all callback methods, such as the parse_args() function. This section also does not describe using the syslog() function. Common functionality is described in Providing Common Functionality to All Methods.

For the complete listing of the Stop method, see Stop Method Code Listing.

What the Stop Method Does

There are two primary considerations when attempting to stop the data service. The first is to provide an orderly shutdown. Sending a SIGTERM signal through pmfadm is the best way to accomplish an orderly shutdown.

The second consideration is to ensure that the data service is actually stopped to avoid putting it in Stop_failed state. The best way to accomplish putting the data service in this state is to send a SIGKILL signal through pmfadm.

The Stop method in the sample data service takes both of these considerations into account. It first sends a SIGTERM signal. If this signal fails to stop the data service, the method sends a SIGKILL signal.

Before attempting to stop DNS, this Stop method verifies that the process is actually running. If the process is running, Stop uses the PMF (pmfadm) to stop the process.

This Stop method is guaranteed to be idempotent. Although the RGM should not call a Stop method twice without first starting the data service with a call to its Start method, the RGM could call a Stop method on a resource even though the resource was never started or the resource died of its own accord. Therefore, this Stop method exits with success even if DNS is not running.

Stopping the Application

The Stop method provides a two-tiered approach to stopping the data service: an orderly or smooth approach using a SIGTERM signal through pmfadm and an abrupt or hard approach using a SIGKILL signal. The Stop method obtains the Stop_timeout value (the amount of time in which the Stop method must return). Stop allocates 80 percent of this time to stopping smoothly and 15 percent to stopping abruptly (5 percent is reserved), as shown in the following sample code.

STOP_TIMEOUT='scha_resource_get -O STOP_TIMEOUT -R $RESOURCE_NAME \
-G $RESOURCEGROUP_NAME'
((SMOOTH_TIMEOUT=$STOP_TIMEOUT * 80/100))
((HARD_TIMEOUT=$STOP_TIMEOUT * 15/100))

The Stop method uses pmfadm -q to verify that the DNS daemon is running. If the DNS daemon is running, Stop first uses pmfadm -s to send a TERM signal to terminate the DNS process. If this signal fails to terminate the process after 80 percent of the timeout value has expired, Stop sends a SIGKILL signal. If this signal also fails to terminate the process within 15 percent of the timeout value, the method logs an error message and exits with an error status.

If pmfadm terminates the process, the method logs a message that the process has stopped and exits with success.

If the DNS process is not running, the method logs a message that it is not running and exits with success anyway. The following code sample shows how Stop uses pmfadm to stop the DNS process.

# See if in.named is running, and if so, kill it. 
if pmfadm -q $PMF_TAG; then
   # Send a SIGTERM signal to the data service and wait for 80% of the
   # total timeout value.
   pmfadm -s $RESOURCE_NAME.named -w $SMOOTH_TIMEOUT TERM
   if [ $? -ne 0 ]; then
      logger -p ${SYSLOG_FACILITY}.err \
          -t [$RESOURCETYPE_NAME,$RESOURCEGROUP_NAME,$RESOURCE_NAME] \
          “${ARGV0} Failed to stop HA-DNS with SIGTERM; Retry with \
           SIGKILL”
      
      # Since the data service did not stop with a SIGTERM signal, use 
      # SIGKILL now and wait for another 15% of the total timeout value.
      pmfadm -s $PMF_TAG -w $HARD_TIMEOUT KILL
      if [ $? -ne 0 ]; then
          logger -p ${SYSLOG_FACILITY}.err \
          -t [$SYSLOG_TAG] \
          “${ARGV0} Failed to stop HA-DNS; Exiting UNSUCCESSFUL”
         exit 1
      fi
fi
else 
   # The data service is not running as of now. Log a message and 
   # exit success.
   logger -p ${SYSLOG_FACILITY}.err \
           -t [$SYSLOG_TAG] \
           “HA-DNS is not started”

   # Even if HA-DNS is not running, exit success to avoid putting
   # the data service resource in STOP_FAILED State.
   exit 0
fi

# Could successfully stop DNS. Log a message and exit success.
logger -p ${SYSLOG_FACILITY}.err \
    -t [$RESOURCETYPE_NAME,$RESOURCEGROUP_NAME,$RESOURCE_NAME] \
    “HA-DNS successfully stopped”
exit 0

Stop Exit Status

A Stop method should not exit with success until the underlying application is actually stopped, particularly if other data services depend on it. Failure to do so can result in data corruption.

For a complex application, such as a database, be certain to set the value for the Stop_timeout property in the RTR file sufficiently high to allow time for the application to clean up while stopping.

If this method fails to stop DNS and exits with failure status, the RGM checks the Failover_mode property, which determines how to react. The sample data service does not explicitly set the Failover_mode property, so this property has the default value NONE (unless the cluster administrator overrides the default value and specifies a different value). In this case, the RGM takes no action other than to set the state of the data service to Stop_failed. The cluster administrator needs to stop the application forcibly and clear the Stop_failed state.

Defining a Fault Monitor

The sample application implements a basic fault monitor to monitor the reliability of the DNS resource (in.named).

The fault monitor consists of the following elements:

How the Probe Program Works

The dns_probe program implements a continuously running process that verifies that the DNS resource that is controlled by the sample data service is running. The dns_probe is started by the dns_monitor_start method, which is automatically run by the RGM after the sample data service is brought online. The data service is stopped by the dns_monitor_stop method, which the RGM runs before the RGM brings the sample data service offline.

This section describes the major pieces of the PROBE method for the sample application. It does not describe functionality that is common to all callback methods, such as the parse_args() function. This section also does not describe using the syslog() function. Common functionality is described in Providing Common Functionality to All Methods.

For the complete listing of the PROBE method, see PROBE Program Code Listing.

What the Probe Program Does

The probe runs in an infinite loop. It uses nslookup to verify that the correct DNS resource is running. If DNS is running, the probe sleeps for a prescribed interval (set by the Thorough_probe_interval system-defined property) and checks again. If DNS is not running, this program attempts to restart it locally, or depending on the number of restart attempts, requests that the RGM relocate the data service to a different node.

Obtaining Property Values

This program requires the values of the following properties:

The scha_resource_get() function obtains the values of these properties and stores them in shell variables, as follows:

PROBE_INTERVAL=`scha_resource_get -O Thorough_probe_interval \
-R $RESOURCE_NAME -G $RESOURCEGROUP_NAME`

PROBE_TIMEOUT_INFO=`scha_resource_get -O Extension -R $RESOURCE_NAME \
-G $RESOURCEGROUP_NAME Probe_timeout` 
Probe_timeout=`echo $probe_timeout_info | awk '{print $2}'`

DNS_HOST=`scha_resource_get -O Network_resources_used -R $RESOURCE_NAME \
-G $RESOURCEGROUP_NAME`

RETRY_COUNT=`scha_resource_get -O Retry_count -R $RESOURCE_NAME -G \
$RESOURCEGROUP_NAME`

RETRY_INTERVAL=`scha_resource_get -O Retry_interval -R $RESOURCE_NAME -G \
$RESOURCEGROUP_NAME`

RT_BASEDIR=`scha_resource_get -O RT_basedir -R $RESOURCE_NAME -G \
 $RESOURCEGROUP_NAME`

Note –

For system-defined properties, such as Thorough_probe_interval, the scha_resource_get() function returns the value only. For extension properties, such as Probe_timeout, the scha_resource_get() function returns the type and value. Use the awk command to obtain the value only.


Checking the Reliability of the Service

The probe itself is an infinite while loop of nslookup commands. Before the while loop, a temporary file is set up to hold the nslookup replies. The probefail and retries variables are initialized to 0.

# Set up a temporary file for the nslookup replies.
DNSPROBEFILE=/tmp/.$RESOURCE_NAME.probe
probefail=0
retries=0

The while loop carries out the following tasks:

Here is the while loop code.

while :
do
   # The interval at which the probe needs to run is specified in the
   # property THOROUGH_PROBE_INTERVAL. Therefore, set the probe to sleep
   # for a duration of THOROUGH_PROBE_INTERVAL.
   sleep $PROBE_INTERVAL

   # Run an nslookup command of the IP address on which DNS is serving.
   hatimerun -t $PROBE_TIMEOUT /usr/sbin/nslookup $DNS_HOST $DNS_HOST \
   > $DNSPROBEFILE 2>&1

      retcode=$?
      if [ $retcode -ne 0 ]; then
            probefail=1
      fi

   # Make sure that the reply to nslookup comes from the HA-DNS
   # server and not from another nameserver mentioned in the 
   # /etc/resolv.conf file.
   if [ $probefail -eq 0 ]; then
# Get the name of the server that replied to the nslookup query.
   SERVER=` awk ' $1=="Server:" { print $2 }' \
   $DNSPROBEFILE | awk -F. ' { print $1 } ' `
   if [ -z "$SERVER" ]; then
      probefail=1
      else
         if [ $SERVER != $DNS_HOST ]; then
            probefail=1
         fi
   fi
fi

Comparing Restart With Failover

If the probefail variable is something other than 0 (success), the nslookup command timed out or the reply came from a server other than the sample service's DNS. In either case, the DNS server is not functioning as expected and the fault monitor calls the decide_restart_or_failover() function to determine whether to restart the data service locally or request that the RGM relocate the data service to a different node. If the probefail variable is 0, a message is generated that the probe was successful.

   if [ $probefail -ne 0 ]; then
         decide_restart_or_failover
   else
         logger -p ${SYSLOG_FACILITY}.err\
         -t [$SYSLOG_TAG]\
         "${ARGV0} Probe for resource HA-DNS successful"
   fi

The decide_restart_or_failover() function uses a time window (Retry_interval) and a failure count (Retry_count) to determine whether to restart DNS locally or request that the RGM relocate the data service to a different node. This function implements the following conditional logic. The code listing for decide_restart_or_failover() in PROBE Program Code Listing contains the code.

If the number of restarts reaches the limit during the time interval, the function requests that the RGM relocate the data service to a different node. If the number of restarts is under the limit, or the interval has been exceeded so the count begins again, the function attempts to restart DNS on the same node.

Note the following points about this function:

Restarting the Data Service

The restart_service() function is called by decide_restart_or_failover() to attempt to restart the data service on the same node.

This function executes the following logic:

function restart_service
{

        # To restart the data service, first verify that the 
        # data service itself is still registered under PMF.
        pmfadm -q $PMF_TAG
        if [[ $? -eq 0 ]]; then
                # Since the TAG for the data service is still registered under
                # PMF, first stop the data service and start it back up again.

                # Obtain the Stop method name and the STOP_TIMEOUT value for
                # this resource.
                STOP_TIMEOUT=`scha_resource_get -O STOP_TIMEOUT \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                STOP_METHOD=`scha_resource_get -O STOP \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                hatimerun -t $STOP_TIMEOUT $RT_BASEDIR/$STOP_METHOD \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME \
                        -T $RESOURCETYPE_NAME

                if [[ $? -ne 0 ]]; then
                        logger-p ${SYSLOG_FACILITY}.err -t [$SYSLOG_TAG] \
                                “${ARGV0} Stop method failed.”
                        return 1
                fi

                # Obtain the START method name and the START_TIMEOUT value for
                # this resource.
                START_TIMEOUT=`scha_resource_get -O START_TIMEOUT \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                START_METHOD=`scha_resource_get -O START \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ
                hatimerun -t $START_TIMEOUT $RT_BASEDIR/$START_METHOD \
                        -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME \
                        -T $RESOURCETYPE_NAME

                if [[ $? -ne 0 ]]; then
                        logger-p ${SYSLOG_FACILITY}.err -t [$SYSLOG_TAG] \
                                “${ARGV0} Start method failed.”
                        return 1
                fi


        else
                # The absence of the TAG for the dataservice 
                # implies that the data service has already
                # exceeded the maximum retries allowed under PMF.
                # Therefore, do not attempt to restart the
                # data service again, but try to failover
                # to another node in the cluster.
                scha_control -O GIVEOVER -G $RESOURCEGROUP_NAME \
                        -R $RESOURCE_NAME
        fi

        return 0
}

Probe Exit Status

The sample data service's PROBE program exits with failure if attempts to restart locally fail and the attempt to fail over to a different node fails as well. This program logs the message Failover attempt failed.

How the Monitor_start Method Works

The RGM calls the Monitor_start method to start the dns_probe method after the sample data service is brought online.

This section describes the major pieces of the Monitor_start method for the sample application. This section does not describe functionality that is common to all callback methods, such as the parse_args() function. This section also does not describe using the syslog() function. Common functionality is described in Providing Common Functionality to All Methods.

For the complete listing of the Monitor_start method, see Monitor_start Method Code Listing.

What the Monitor_start Method Does

This method uses the PMF (pmfadm) to start the probe.

Starting the Probe

The Monitor_start method obtains the value of the RT_basedir property to construct the full path name for the PROBE program. This method starts the probe by using the infinite retries option of pmfadm (-n -1, -t -1), which means that if the probe fails to start, the PMF tries to start it an infinite number of times over an infinite period of time.

# Find where the probe program resides by obtaining the value of the
# RT_basedir property of the resource.
RT_BASEDIR=`scha_resource_get -O RT_basedir -R $RESOURCE_NAME -G \
$RESOURCEGROUP_NAME`

# Start the probe for the data service under PMF. Use the infinite retries
# option to start the probe. Pass the resource name, type, and group to the
# probe program. 
pmfadm -c $RESOURCE_NAME.monitor -n -1 -t -1 \
   $RT_BASEDIR/dns_probe -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME \
   -T $RESOURCETYPE_NAME

How the Monitor_stop Method Works

The RGM calls the Monitor_stop method to stop execution of dns_probe when the sample data service is brought offline.

This section describes the major pieces of the Monitor_stop method for the sample application. This section does not describe functionality that is common to all callback methods, such as the parse_args() function. This section also does not describe using the syslog() function. Common functionality is described in Providing Common Functionality to All Methods.

For the complete listing of the Monitor_stop method, see Monitor_stop Method Code Listing.

What the Monitor_stop Method Does

This method uses the PMF (pmfadm) to check whether the probe is running, and if so, to stop it.

Stopping the Monitor

The Monitor_stop method uses pmfadm -q to see if the probe is running, and if so, uses pmfadm -s to stop it. If the probe is already stopped, the method exits successfully anyway, which guarantees the idempotence of the method.


Caution – Caution –

Be certain to use the KILL signal with pmfadm to stop the probe and not a signal that can be masked, such as TERM. Otherwise, the Monitor_stop method can hang indefinitely and eventually time out. The reason is that the PROBE method calls scha_control() when it is necessary to restart or fail over the data service. When scha_control() calls Monitor_stop as part of the process of bringing the data service offline, if Monitor_stop uses a signal that can be masked, Monitor_stop hangs waiting for scha_control() to complete, and scha_control() hangs waiting for Monitor_stop to complete.


# See if the monitor is running, and if so, kill it.
if pmfadm -q $PMF_TAG; then
   pmfadm -s $PMF_TAG KILL
   if [ $? -ne 0 ]; then
         logger -p ${SYSLOG_FACILITY}.err \
            -t [$SYSLOG_TAG] \
            "${ARGV0} Could not stop monitor for resource " \
            $RESOURCE_NAME
           exit 1
   else
         # could successfully stop the monitor. Log a message.
         logger -p ${SYSLOG_FACILITY}.err \
            -t [$SYSLOG_TAG] \
            "${ARGV0} Monitor for resource " $RESOURCE_NAME \
            " successfully stopped"
   fi
fi
exit 0

Monitor_stop Exit Status

The Monitor_stop method logs an error message if it cannot stop the PROBE method. The RGM puts the sample data service into MONITOR_FAILED state on the primary node, which can panic the node.

Monitor_stop should not exit before the probe has been stopped.

How the Monitor_check Method Works

The RGM calls the Monitor_check method whenever the PROBE method attempts to fail over the resource group that contains the data service to a new node.

This section describes the major pieces of the Monitor_check method for the sample application. This section does not describe functionality that is common to all callback methods, such as the parse_args() function. This section also does not describe using the syslog() function. Common functionality is described in Providing Common Functionality to All Methods.

For the complete listing of the Monitor_check method, see Monitor_check Method Code Listing.

The Monitor_check method must be implemented so that it does not conflict with other methods that are running concurrently.

The Monitor_check method calls the Validate method to verify that the DNS configuration directory is available on the new node. The Confdir extension property points to the DNS configuration directory. Therefore, Monitor_check obtains the path and name for the Validate method and the value of Confdir. It passes this value to Validate, as shown in the following listing.

# Obtain the full path for the Validate method from
# the RT_basedir property of the resource type.
RT_BASEDIR=`scha_resource_get -O RT_basedir -R $RESOURCE_NAME \
   -G $RESOURCEGROUP_NAMÈ

# Obtain the name of the Validate method for this resource.
VALIDATE_METHOD=`scha_resource_get -O Validate \
   -R $RESOURCE_NAME -G $RESOURCEGROUP_NAMÈ

# Obtain the value of the Confdir property in order to start the
# data service. Use the resource name and the resource group entered to
# obtain the Confdir value set at the time of adding the resource.
config_info=`scha_resource_get -O Extension -R $RESOURCE_NAME \
 -G $RESOURCEGROUP_NAME Confdir`

# scha_resource_get returns the type as well as the value for extension
# properties. Use awk to get only the value of the extension property.
CONFIG_DIR=`echo $config_info | awk `{print $2}'`

# Call the validate method so that the dataservice can be failed over
# successfully to the new node.
$RT_BASEDIR/$VALIDATE_METHOD -R $RESOURCE_NAME -G $RESOURCEGROUP_NAME \
   -T $RESOURCETYPE_NAME -x Confdir=$CONFIG_DIR

See How the Validate Method Works to see how the sample application verifies the suitability of a node for hosting the data service.

Handling Property Updates

The sample data service implements Validate and Update methods to handle the updating of properties by a cluster administrator.

How the Validate Method Works

The RGM calls the Validate method when a resource is created and when administrative action updates the properties of the resource or its containing group. The RGM calls Validate before the creation or update is applied, and a failure exit code from the method on any node causes the creation or update to be canceled.

The RGM calls Validate only when resource or resource group properties are changed by the cluster administrator, not when the RGM sets properties or when a monitor sets the resource properties Status and Status_msg.


Note –

The Monitor_check method also explicitly calls the Validate method whenever the PROBE method attempts to fail over the data service to a new node.


What the Validate Method Does

The RGM calls Validate with additional arguments to those that are passed to other methods, including the properties and values that are being updated. Therefore, this method in the sample data service must implement a different parse_args() function to handle the additional arguments.

The Validate method in the sample data service verifies a single property, the Confdir extension property. This property points to the DNS configuration directory, which is critical to the successful operation of DNS.


Note –

Because the configuration directory cannot be changed while DNS is running, the Confdir property is declared in the RTR file as TUNABLE = AT_CREATION. Therefore, the Validate method is never called to verify the Confdir property as the result of an update, but only when the data service resource is being created.


If Confdir is one of the properties that the RGM passes to Validate, the parse_args() function retrieves and saves its value. Validate verifies that the directory pointed to by the new value of Confdir is accessible and that the named.conf file exists in that directory and contains data.

If the parse_args() function cannot retrieve the value of Confdir from the command-line arguments that are passed by the RGM, Validate still attempts to validate the Confdir property. Validate uses scha_resource_get() to obtain the value of Confdir from the static configuration. Validate performs the same checks to verify that the configuration directory is accessible and contains a named.conf file that is not empty.

If Validate exits with failure, the update or creation of all properties, not just Confdir, fails.

Validate Method Parsing Function

Because the RGM passes the Validate method a different set of arguments than the other callback methods, Validate requires a different function for parsing arguments than the other methods. See the rt_callbacks(1HA) man page for more information about the arguments that are passed to Validate and the other callback methods. The following code sample shows the Validate parse_args() function.

#########################################################################
# Parse Validate arguments.
#
function parse_args # [args...]
{

   typeset opt
   while getopts 'cur:x:g:R:T:G:' opt
   do
         case "$opt" in
         R)
                  # Name of the DNS resource.
                  RESOURCE_NAME=$OPTARG
                  ;;
         G)
                  # Name of the resource group in which the resource is
                  # configured.
                  RESOURCEGROUP_NAME=$OPTARG
                  ;;
         T)
                  # Name of the resource type.
                  RESOURCETYPE_NAME=$OPTARG
                  ;;
         r)
                  # The method is not accessing any system defined
                  # properties so this is a no-op
                  ;;
         g)
                  # The method is not accessing any resource group
                  # properties, so this is a no-op
                  ;;
         c)
                  # Indicates the Validate method is being called while
                  # creating the resource, so this flag is a no-op.
                  ;;
         u)
                  # Indicates the updating of a property when the
                  # resource already exists. If the update is to the
                  # Confdir property then Confdir should appear in the
                  # command-line arguments. If it does not, the method must
                  # look for it specifically using scha_resource_get.
                  UPDATE_PROPERTY=1
                  ;;
         x)
                  # Extension property list. Separate the property and
                  # value pairs using "=" as the separator.
                  PROPERTY=`echo $OPTARG | awk -F= '{print $1}'`
                  VAL=`echo $OPTARG | awk -F= '{print $2}'`
                  # If the Confdir extension property is found on the
                  # command line, note its value.
                  if [ $PROPERTY == "Confdir" ]; then
                           CONFDIR=$VAL
                           CONFDIR_FOUND=1
                  fi
                  ;;
         *)
                  logger -p ${SYSLOG_FACILITY}.err \
                  -t [$SYSLOG_TAG] \
                  "ERROR: Option $OPTARG unknown"
                  exit 1
                  ;;
         esac
   done
}

As with the parse_args() function for other methods, this function provides a flag (R) to capture the resource name, (G) to capture the resource group name, and (T) to capture the resource type that is passed by the RGM.

The r flag (which indicates a system-defined property), g flag (which indicates a resource group property), and the c flag (which indicates that the validation is occurring during creation of the resource) are ignored. They are ignored because this method is being called to validate an extension property when the resource is being updated.

The u flag sets the value of the UPDATE_PROPERTY shell variable to 1 (TRUE). The x flag captures the names and values of the properties that are being updated. If Confdir is one of the properties being updated, its value is placed in the CONFDIR shell variable, and the variable CONFDIR_FOUND is set to 1 (TRUE).

Validating Confdir

In its MAIN function, Validate first sets the CONFDIR variable to the empty string and UPDATE_PROPERTY and CONFDIR_FOUND to 0.

CONFDIR=""
UPDATE_PROPERTY=0
CONFDIR_FOUND=0

Validate calls parse_args() to parse the arguments that are passed by the RGM.

parse_args “$@”

Validate checks if Validate is being called as the result of an update of properties. Validate also checks if the Confdir extension property was on the command line. Validate verifies that the Confdir property has a value, and if not, exits with failure status and an error message.

if ( (( $UPDATE_PROPERTY == 1 )) &&  (( CONFDIR_FOUND == 0 )) ); then
         config_info=`scha_resource_get -O Extension -R $RESOURCE_NAME \
            -G $RESOURCEGROUP_NAME Confdir`
         CONFDIR=`echo $config_info | awk '{print $2}'`
fi

# Verify that the Confdir property has a value. If not there is a failure
# and exit with status 1
if [[ -z $CONFDIR ]]; then
         logger -p ${SYSLOG_FACILITY}.err \
            "${ARGV0} Validate method for resource "$RESOURCE_NAME " failed"
         exit 1
fi

Note –

Specifically, the preceding code checks if Validate is being called as the result of an update ($UPDATE_PROPERTY == 1) and if the property was not found on the command line (CONFDIR_FOUND == 0). In this case, the code retrieves the existing value of Confdir by using scha_resource_get(). If Confdir was found on the command line (CONFDIR_FOUND == 1), the value of CONFDIR comes from the parse_args() function, not from scha_resource_get().


The Validate method uses the value of CONFDIR to verify that the directory is accessible. If the directory is not accessible, Validate logs an error message and exits with error status.

# Check if $CONFDIR is accessible.
if [ ! -d $CONFDIR ]; then
   logger -p ${SYSLOG_FACILITY}.err \
         -t [$SYSLOG_TAG] \
         "${ARGV0} Directory $CONFDIR missing or not mounted"
   exit 1
fi

Before validating the update of the Confdir property, Validate performs a final check to verify that the named.conf file is present. If the file is not present, the method logs an error message and exits with error status.

# Check that the named.conf file is present in the Confdir directory
if [ ! -s $CONFDIR/named.conf ]; then
         logger -p ${SYSLOG_FACILITY}.err \
            -t [$SYSLOG_TAG] \
            "${ARGV0} File $CONFDIR/named.conf is missing or empty"
         exit 1
fi

If the final check is passed, Validate logs a message that indicates success and exits with success status.

# Log a message indicating that the Validate method was successful.
logger -p ${SYSLOG_FACILITY}.err \
   -t [$SYSLOG_TAG] \
   "${ARGV0} Validate method for resource "$RESOURCE_NAME \
   " completed successfully"

exit 0

Validate Exit Status

If Validate exits with success (0), Confdir is created with the new value. If Validate exits with failure (1), Confdir and any other properties are not created and a message that indicates the reason is generated.

How the Update Method Works

The RGM runs the Update method to notify a running resource that its properties have been changed. The RGM runs Update after the cluster administrator succeeds in setting properties of a resource or its group. This method is called on nodes where the resource is online.

What the Update Method Does

The Update method does not update properties. The RGM updates properties. The Update method notifies running processes that an update has occurred. The only process in the sample data service that is affected by a property update is the fault monitor. Consequently, the fault monitor process is the process that the Update method stops and restarts.

The Update method must verify that the fault monitor is running and then kill it by using the pmfadm command. The method obtains the location of the probe program that implements the fault monitor, and restarts it by using the pmfadm command.

Stopping the Monitor With Update

The Update method uses pmfadm -q to verify that the monitor is running, and if so, kills it with pmfadm -s TERM. If the monitor is successfully terminated, a message to that effect is sent to the cluster administrator. If the monitor cannot be stopped, Update exits with failure status and sends an error message to the cluster administrator.

if pmfadm -q $RESOURCE_NAME.monitor; then

# Kill the monitor that is running already
pmfadm -s $PMF_TAG TERM
    if [ $? -ne 0 ]; then
       logger -p ${SYSLOG_FACILITY}.err \
              -t [$SYSLOG_TAG] \
                 "${ARGV0} Could not stop the monitor"
       exit 1
    else
    # could successfully stop DNS. Log a message.
       logger -p ${SYSLOG_FACILITY}.err \
              -t [$RESOURCETYPE_NAME,$RESOURCEGROUP_NAME,$RESOURCE_NAME] \
                 "Monitor for HA-DNS successfully stopped"
    fi

Restarting the Monitor

To restart the monitor, the Update method must locate the script that implements the probe program. The probe program is located in the base directory for the data service, which is pointed to by the RT_basedir property. Update retrieves the value of RT_basedir and stores it in the RT_BASEDIR variable, as follows.

RT_BASEDIR=`scha_resource_get -O RT_basedir -R $RESOURCE_NAME -G \
$RESOURCEGROUP_NAME`

Update uses the value of RT_BASEDIR with pmfadm to restart the dns_probe program. If successful, Update exits with success and sends a message to that effect to the cluster administrator. If pmfadm cannot start the probe program, Update exits with failure status and logs an error message.

Update Exit Status

Update method failure causes the resource to be put into an “update failed” state. This state has no effect on RGM management of the resource, but indicates the failure of the update action to administration tools through the syslog() function.