Operation of the HA for Oracle WebLogic Server Fault Monitor

This section describes the HA for Oracle WebLogic Server fault monitor.

Probing Algorithm and Functionality

The fault monitor detects failures and takes action. If the monitor detects a failure in an Oracle WebLogic Server, it first restarts the Oracle WebLogic Server. If the Oracle WebLogic Server fails a certain number of times (configurable by the administrator) within a certain time window (configurable by the administrator), the resource group containing the Oracle WebLogic Server is failed over to another surviving cluster node and restarted.

The fault monitor method, by default, probes the server URL set in the extension property Server_url. The probe connects to the hostname and the port and then sends an HTTP GET request on the URL. If the connection fails, it is considered a complete failure and the resource group containing the Oracle WebLogic Server is restarted or failed over to another surviving cluster node and restarted. If the connection succeeds, but the HTTP response code is 500 (internal server error), it is also considered a complete failure and the resource group is restarted or failed over. All other HTTP response codes are considered a success.

If the Monitor_uri_list extension property is set, the probe method connects to the URIs mentioned in the list and takes action if a failure occurs. The probe sends an HTTP GET on the specified URI or URIs.

In a multi-master configuration, the managed server resources have per-node resource dependencies set on the logical host resources. If the Monitor_uri_list extension property is set, the probing algorithm iterates through the list of URIs and probes only those URIs that match the logical hostname that is online on that particular node. Probing on the rest of the URIs is ignored on that particular node.

For example, if Monitor_uri_list=http://log-host-1:7003/abc.asp,http://log-host-2:7003/abc.asp, and the property resource_dependencies=log-host-1@node1,log-host-2@node2, the probe method on Node1 connects to the URI http://log-host-1:7003/abc.asp and ignores the URI http://log-host-2:7003/abc.asp. The probe method on Node2 connects to the URI http://log-host-2:7003/abc.asp and ignores the other URI. The probe method running on each node takes appropriate action if a failure occurs.

If a complete failure (URL or URI probe) of the Oracle WebLogic Server instance is detected by the probe, and if a database probe script is specified in the extension property DB_probe_script, the probe method will probe the database before taking any action on the Oracle WebLogic Server resource. If the database probe script returns success (database is up), action is taken on the Oracle WebLogic Server resource. If the database probe script returns a failure (database is down), the Oracle WebLogic Server probe will not take any action (restart or failover) until the database is up.

START Method

Before starting the Oracle WebLogic Server that is configured in the resource, the Oracle WebLogic Server configuration and the resource extension properties are validated. If the DB_probe_script extension property is set, the database is probed by running the script that is set in the extension property. If the database is up, the Oracle WebLogic Server is started by running the START script that is configured in the extension property Start_script under pmf. If the database is not up, the START method returns success and lets the probe method handle the starting of the Oracle WebLogic Server. The probe method waits until the database is up to start the Oracle WebLogic Server, as explained in Probing Algorithm and Functionality.

After starting the START script under pmf, the START method waits until the Oracle WebLogic Server is in RUNNING mode before declaring the START method successful. While waiting for the Oracle WebLogic Server to start, the probe method tries to connect to the server to check if it is up. Some messages are displayed on the console during startup. The message Failed to connect to host logical-host-1 and port 7001: Connection refused will continue to be displayed until the Oracle WebLogic Server starts completely. After the Oracle WebLogic Server is in the RUNNING mode, the START method sets the status to Started Successfully.

Oracle WebLogic Server console messages are stored at /var/cluster/logs/hawls/wls-resource-name on the cluster node where the resource is started. You can check this log file for startup messages of Oracle WebLogic Server instances.

STOP Method

The STOP method stops the Oracle WebLogic Server that is configured in the resource. By default, the STOP method kills the Oracle WebLogic Server by sending a SIGKILL to the Oracle WebLogic Server process. If the Smooth_shutdown extension property is set to TRUE, the STOP method tries to bring down the Oracle WebLogic Server instance by running the following command:

java weblogic.Admin -url hostname:port -username $WLS_USER 
-password $WLS_PW SHUTDOWN

If this command fails, the Oracle WebLogic Server is shut down by using SIGKILL. Even if the command succeeds, the STOP method sends SIGKILL to ensure that the Oracle WebLogic Server process is stopped.

If the Smooth_shutdown extension property is set to TRUE, ensure that the boot.properties file with valid user name and password is present in the DOMAIN directory. The Oracle WebLogic Server variables, WLS_USER and WLS_PW do not need to be set in the START script for smooth shutdown. This waiver applies only to Oracle WebLogic Server 8.1 or later versions.