Tuning the Fault Monitor for Oracle Solaris Cluster HA for Sun Java System Application Server

Versions before 8.1 of Oracle Solaris Cluster HA for Sun Java System Application Server have a fault monitor that is contained in a resource whose resource type is SUNW.s1as.

System properties and extension properties of the resource types control the behavior of the fault monitors. The default values of these properties determine the preset behavior of the fault monitors. The preset behavior should be suitable for most Oracle Solaris Cluster installations. Therefore, you should tune the fault monitors only if you need to modify this preset behavior.

Tuning these fault monitors involves the following tasks:

• Setting the interval between fault monitor probes

• Setting the timeout for fault monitor probes

• Defining the criteria for persistent faults

• Specifying the failover behavior of a resource

Perform these tasks when you register and configure Solaris Cluster HA for Sun Java System Application Server, as described in Registering and Configuring Oracle Solaris Cluster HA for Sun Java System Application Server Supported Versions Before 8.1.

For detailed information about these tasks, see Tuning Fault Monitors for Oracle Solaris Cluster Data Services in Oracle Solaris Cluster Data Services Planning and Administration Guide.

This section provides the following information.

• Description of the fault monitor's probing functionality.

• Conditions, messages, and recovery actions associated with unsuccessful probing.

• Conditions and messages associated with successful probing.

Operations by the Oracle Solaris Cluster HA for Sun Java System Application Server Fault Monitor During a Probe

The Oracle Solaris Cluster HA for Sun Java System Application Server fault monitor probe sends a request to the server to query the health of the Sun Java System Application Server server. The probe executes the following steps:

1. The fault monitor probes the Sun Java System Application Server instance according to the time-out value set with the Probe_timeout resource property.

2. The probe connects to the IP address and port combinations defined by the network resource configuration and the Port_list setting for the resource group. If the resource is configured without an empty Port_list this step is skipped. If connection succeeds, the probe disconnects. If the connection fails, the failure is recorded.

   Heavy network traffic, heavy system load, and misconfiguration can cause the query to fail. Misconfiguration can occur if you did not configure the Sun Java System Application Server server to listen on all of the IP address/port combinations that are probed. The Sun Java System Application Server server should service every port for every IP address that is specified for the resource.

3. The probe connects to the Sun Java System Application Server server and performs an HTTP 1.1 GET check by sending a HTTP request and receiving a response to each of the URIs in Monitor_Uri_List.

   The result of each HTTP request is either failure or success. If all of the requests successfully receive a reply from the Sun Java System Application Server server, the probe returns and continues the next cycle of probing and sleeping.

   Heavy network traffic, heavy system load, and misconfiguration can cause the HTTP GET probe to fail. Misconfiguration of the Monitor_Uri_List property can cause a failure if a URI in the Monitor_Uri_List includes an incorrect port or hostname. For example, if the application server instance is listening on logical host schost-1 and the URI was specified as http://schost-2/servlet/monitor, the probe will try to contact schost-2 to request /servlet/monitor.

4. The probe records a failure in the history log if the reply to the probe is not received within the Probe_timeout limit. The probe considers this scenario a failure on the part of the Sun Java System Application Server data service. A Sun Java System Application Server probe failure can be a complete failure or a partial failure.

   If the reply to the probe is received within the Probe_timeout limit, the HTTP response code is checked. If the response code is 500 Internal Server Error, the probe is considered a complete failure. All other response codes are ignored.

   The following are complete probe failures.

   • The following error message is received upon failure to connect to the server. The %s indicates the hostname and %d indicates the port number.

     Failed to connect to the host <%s> and port <%d>. Receiving a
     response code of 500 Internal Server Error HTTP GET
     Response Code for probe of %s is 500. Failover will be in
     progress

   • The following error message is received upon failure to successfully send the probe string to the server. The first %s indicates the hostname, the %d indicates the port number, and the second %s indicates further details about the error.

     Write to server failed: server %s port %d: %s.

5. The monitor accumulates partial failures that occur within the Retry_interval resource property setting until they equal a complete failure.

   The following are partial probe failures:

   • The following error message is received when there is a failure to disconnect before the Probe_timeout setting lapses. The %d indicates the port number and the %s indicates the resource name.

     Failed to disconnect from port %d of resource %s.

   • Failure to complete all probe steps within Probe_timeout time is a partial failure.

   • The following error message is received upon failure to read data from the server for other reasons. The first %s indicates the hostname, the %d indicates the port number, and the second %s indicates further details about the error.

     Failed to communicate with server %s port %d: %s

6. Based on the history of failures, a failure can cause either a local restart or a failover of the data service.