The Fault Monitor detects failures and takes action. If the monitor detects a failure in a WebLogic Server, it first restarts the WebLogic Server. If the WebLogic Server fails a certain number of times (configurable by the administrator) within a certain time window (configurable by the administrator), the resource group containing the WebLogic Server is failed over to another surviving cluster node and restarted.
The Fault Monitor method, by default, probes the server URL set in the extension property Server_url. The probe connects to the hostname and the port and then sends an HTTP GET request on the URL. If the connection fails, it is considered a complete failure and the resource group containing the WebLogic Server is restarted or failed over to another surviving cluster node and restarted. If the connection succeeds, but the HTTP response code is 500 (internal server error), it is also considered a complete failure and the resource group is restarted or failed over. All other HTTP response codes are considered a success.
If the monitor_uri_list extension property is set, the probe method connects to the URIs mentioned in the list and takes action if a failure occurs. The probe sends an HTTP GET on the specified URI or URIs.
In a multi-master configuration, the Managed Server resources have per-node resource dependencies set on the logical host resources. If the monitor_uri_list extension property is set, the probing algorithm iterates through the list of URIs and probes only those URIs that match the logical hostname that is online on that particular node. Probing on the rest of the URIs is ignored on that particular node.
For example, if Monitor_Uri_list=http://log-host-1:7003/abc.asp,http://log-host-2:7003/abc.asp, and the property resource_dependencies=log-host-1@node1,log-host-2@node2, the probe method on Node1 connects to the URI http://log-host-1:7003/abc.asp and ignores the URI http://log-host-2:7003/abc.asp. The probe method on Node2 connects to the URI http://log-host-2:7003/abc.asp and ignores the other URI. The probe method running on each node takes appropriate action if a failure occurs.
If a complete failure (URL or URI probe) of the WebLogic Server instance is detected by the probe, and if a database probe script is specified in the extension property db_probe_script, the probe method will probe the database before taking any action on the WebLogic Server resource. If the database probe script returns success (database is up), action is taken on the WebLogic Server resource. If the database probe script returns a failure (database is down), the WebLogic Server probe will not take any action (restart or failover) until the database is up.
Before starting the WebLogic Server that is configured in the resource, the WebLogic Server configuration and the resource extension properties are validated. If the db_probe_script extension property is set, the database is probed by running the script that is set in the extension property. If the database is up, the WebLogic Server is started by running the START script that is configured in the extension property Start_script under pmf. If the database is not up, the START method returns success and lets the probe method handle the starting of the WebLogic Server. The probe method waits until the database is up to start the WebLogic Server, as explained in Probing Algorithm and Functionality.
After starting the START script under pmf, the START method waits until the WebLogic Server is in RUNNING mode before declaring the START method successful. While waiting for the WebLogic Server to start, the probe method tries to connect to the server to check if it is up. Some messages are displayed on the console during startup. The message “Failed to connect to host logical-host-1 and port 7001: Connection refused” will continue to be displayed until the WebLogic Server starts completely. After the WebLogic Server is in the RUNNING mode, the START method sets the status to “Started Successfully.”
WebLogic Server console messages are stored at /var/cluster/logs/hawls/wls-resource-name on the cluster node where the resource is started. You can check this log file for startup messages of WebLogic Server instances.
The STOP method stops the WebLogic Server that is configured in the resource. By default, the STOP method kills the WebLogic Server by sending a SIGKILL to the WebLogic Server process. If the smooth_shutdown extension property is set to TRUE, the STOP method tries to bring down the WebLogic Server instance by running the following command:
java weblogic.Admin -url hostname:port -username $WLS_USER -password $WLS_PW SHUTDOWN
If this command fails, the WebLogic Server is shut down by using SIGKILL. Even if the command succeeds, the STOP method sends SIGKILL to ensure that the WebLogic Server process is stopped.
If the smooth_shutdown extension property is set to TRUE, ensure that the boot.properties file with valid user name and password is present in the DOMAIN directory. The WebLogic Server variables, WLS_USER and WLS_PW do not need to be set in the START script for smooth shutdown. This waiver applies only to WebLogic Server 8.1 or later versions.