Configuration Guide

     Previous  Next    Open TOC in new window    View as PDF - New Window  Get Adobe Reader - New Window
Content starts here

Improving Failover Performance for Physical Network Failures

The following sections describe how to configure use the WebLogic SIP Server "echo server" process to improve data tier failover performance when a server becomes physically disconnected from the network:

 


Overview of Failover Detection

In a production system, engine tier servers continually access data tier replicas in order to retrieve and write call state data. The WebLogic SIP Server architecture depends on engine tier nodes to detect when a data tier server has failed or become disconnected. When an engine cannot access or write call state data because a replica is unavailable, the engine connects to another replica in the same partition and reports the offline server. The replica updates the current view of the data tier to account for the offline server, and other engines are then notified of the updated view as they access and retrieve call state data.

By default, an engine tier server uses its RMI connection to the replica to determine if the replica has failed or become disconnected. The algorithms used to determine a failure of an RMI connection are reliable, but ultimately they depend on the TCP protocol's retransmission timers to diagnose a disconnection (for example, if the network cable to the replica is removed). Because the TCP retransmission timer generally lasts a full minute or longer, WebLogic SIP Server provides an alternate method of detecting failures that can diagnose a disconnected replica in a matter of a few seconds.

WlssEchoServer Failure Detection

WlssEchoServer is a separate process that you can run on the same server hardware as a data tier replica. The purpose of WlssEchoServer is to provide a simple UDP echo service to engine tier nodes to be used for determining when a data tier server goes offline, for example in the event that the network cable is disconnected. The algorithm for detecting failures with WlssEchoServer is as follows:

  1. For all normal traffic, engine tier servers communicate with data tier replicas using TCP. TCP is used as the basic transport between the engine tier and data tier regardless of whether or not WlssEchoServer is used.
  2. Engine tier servers send a periodic heartbeat message to each configured WlssEchoServer over UDP. During normal operation, WlssEchoServer responds to the heartbeats so that the connection between the engine node and replica is verified.
  3. Should there be a complete failure of the data tier stack, or the network cable is disconnected, the heartbeat messages are not returned to the engine node. In this case, the engine node can mark the replica as being offline without having to wait for the normal TCP connection timeout.
  4. After identifying the offline server, the engine node reports the failure to an available data tier replica, and the data tier view is updated as described in the previous section.

Also, should a data tier server notice that its local WlssEchoServer process has died, it automatically shuts down. This behavior ensures even quicker failover because avoids the time it takes engine nodes to notice and report the failure as described in Overview of Failover Detection.

You can configure the heartbeat mechanism on engine tier servers to increase the performance of failover detection as necessary. You can also configure the listen port and log file that WlssEchoServer uses on data tier servers.

Forced Shutdown for Failed Replicas

If any engine tier server cannot communicate with a particular replica, the engine access another, available replica in the data tier to report the offline server. The replica updates its view of the affected partition to remove the offline server. The updated view is then distributed to all engine tier servers that later access the partition. Propagating the view in this manner helps to ensure that engine servers do not attempt to access the offline replica.

The replica that updates the view also issues a one-time request to the offline replica to ask it to shut down. This is done to try to shut-down running replica servers that cannot be accessed by one or more engine servers due to a network outage. If an active replica can reach the replica marked as "offline," the offline replica shuts down.

 


WlssEchoServer Requirements and Restrictions

Note: Using WlssEchoServer is not required in all WebLogic SIP Server installations. Enable the echo server only when your system requires detection of a network or replica failure faster than the configured TCP timeout interval.

Observe the following requirements and restrictions when using WlssEchoServer to detect replica failures:

 


Starting WlssEchoServer on Data Tier Server Machines

WlssEchoServer is a Java program that you can start directly from a shell or command prompt. The basic syntax for starting WlssEchoServer is:

java -classpath WLSS_HOME/telco/lib/wlss.jar options com.bea.wcp.util.WlssEchoServer

Where WLSS_HOME is the path to the WebLogic SIP Server installation and options may include one of the options described in Table B-1.

Table B-1 WlssEchoServer Options
Option
Description
-Dwlss.ha.echoserver.port
Specifies the port number used to listen for heartbeat messages. Ensure that the port number you specify is not used by any other process on the server machine. By default WlssEchoServer uses port 6734.
-Dwlss.ha.echoserver.logfile
Specifies the log file location and name. By default, log messages are written to ./echo_servertime.log where time is the time expressed in milliseconds.

BEA recommends that you include the command to start WlssEchoServer in the same script you use to start each WebLogic SIP Server data tier instance. If you use the startManagedWebLogic.sh script to start an engine or data tier server instance, add a command to start WlssEchoServer before the final command used to start the server. For example, change the lines:

"$JAVA_HOME/bin/java" ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS}     \
  -Dweblogic.Name=${SERVER_NAME}                                 \
  -Dweblogic.management.username=${WLS_USER}                     \
  -Dweblogic.management.password=${WLS_PW}                       \
  -Dweblogic.management.server=${ADMIN_URL}                      \
  -Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy" \
   weblogic.Server 

to read:

"$JAVA_HOME/bin/java" -classpath WLSS_HOME/telco/lib/wlss.jar    \
  -Dwlss.ha.echoserver.port=6734 com.bea.wcp.util.WlssEchoServer &
"$JAVA_HOME/bin/java" ${JAVA_VM} ${MEM_ARGS} ${JAVA_OPTIONS}     \
  -Dweblogic.Name=${SERVER_NAME}                                 \
  -Dweblogic.management.username=${WLS_USER}                     \
  -Dweblogic.management.password=${WLS_PW}                       \
  -Dweblogic.management.server=${ADMIN_URL}                      \
  -Djava.security.policy="${WL_HOME}/server/lib/weblogic.policy" \
   weblogic.Server 

 


Enabling and Configuring the Heartbeat Mechanism on Servers

To enable the WlssEchoServer heartbeat mechanism, you must include the -Dreplica.host.monitor.enabled JVM argument in the command you use to start all engine and data tier servers. BEA recommends adding this option directly to the script used to start Managed Servers in your system. For example, in the startManagedWebLogic.sh script, change the line:

# JAVA_OPTIONS="-Dweblogic.attribute=value -Djava.attribute=value"

to read:

JAVA_OPTIONS="-Dreplica.host.monitor.enabled"

Several additional JVM options configure the functioning of the heartbeat mechanism. Table B-2 describes the options used to configure failure detection.

Table B-2 WlssEchoServer Options
Option
Description
-Dreplica.host.monitor.enabled
This system property is required on both engine and data tier servers to enable the heartbeat mechanism.
-Dwlss.ha.heartbeat.interval
Specifies the number of milliseconds between heartbeat messages. By default heartbeats are sent every 1,000 milliseconds.
-Dwlss.ha.heartbeat.count
Specifies the number of consecutive, missed heartbeats that are permitted before a replica is determined to be offline. By default, a replica is marked offline if the WlssEchoServer process on the server fails to respond to 3 heartbeat messages.
-Dwlss.ha.heartbeat.SoTimeout
Specifies the UDP socket timeout value.


  Back to Top       Previous  Next