5009 - No Available Servers in Target Set

Alarm Group:

IPFE

Description:

Through monitoring of the application servers, the server learns no server in a target set is available. The associated measurement, TxReject, also shows counts (refer to the DSR Measurements Reference for details about this measurement). This alarm can be triggered during configuration of the IPFE when the target set address has been configured, but application servers have not yet been added to the target set. Setting the Monitoring Connection Timeout to a value less than 2 seconds is the primary cause of this alarm. It is recommended to leave this setting on the default of 3 seconds. Do not set to 1 second. Later releases prohibit this from being set to 1 second.

Each target set is configured with at least one backend application server (DAMP). The IPFE raises the 5009 alarm when the IPFE detects no DAMP is live. The IPFE detects the DAMP liveliness by receiving the DAMP heartbeat on time.

Severity:

Critical

Instance:

tsa N has no available servers where N is 1-16

Note:

When at least one application server in a target set becomes available, this alarm clears.

HA Score:

Normal

Auto Clear Seconds:

N/A

OID:

ipfeIpfeNoAvailableAppServersNotify

Cause:

Setting the Monitoring Connection Timeout to a value less than 2 seconds is the primary cause of this alarm. It is recommended to leave this setting on the default of 3 seconds. Do not set to 1 second. Later releases prohibit this from being set to 1 second.

Each target set is configured with at least one backend application server (DAMP). The IPFE raises the 5009 alarm when the IPFE detects no DAMP is live. The IPFE detects the DAMP liveliness by receiving the DAMP heartbeat on time. The following screen shows the IPFE monitoring the DAMP XSI port 9675 and the heartbeat is received every 3 seconds through this port.

When the IPFE does not receive the heartbeat from a single backend DAMP the IPFE raises alarm 5001. When the IPFE does not receive the heartbeat for all backend DAMPs in its TSA List, the IPFE raises the alarm 5009.

When 5009 alarm raises, the IPFE is not able to route the connection to a backend DAMP server. This alarm is critical.

For example:

TSA1 10.240.10.162 has three backend DAMPs (DAMP1-XSI2-10.240.10.1, DAMP2-XSI2-10.240.10.2, and DAMP3-XSI2-10.240.10.3), when IPFE is not able to receive the heartbeat in time from DAMP1, alarm 5001 raises from its active IPFE server. When IPFE misses all three DAMP heartbeats, alarm 5009 raises from its active IPFE server.

Diagnostic Information:

The Wireshark is the normal tool to monitor if the DAMP is sending a heat beat to IPFE. Follow these steps to diagnose the issues:

1. From the SO GUI, navigate to IPFE, and then Configuration, and then Target Sets, and then TSA#, and then +; at least one DAMP server XSI IP should be present. If yes, go to step 2.
2. Log into the IPFE server. - ping <the DAMP server XSI IP> - telnet <the DAMP server XSI IP> <monitoring port, default 9675>. If fail, go to step 3.
3. ssh admusr@<DAMP server XMI>. Run the sudo netstat -anop | grep <monitoring port, default 9675> command to see if there is a TCP listen socket on that DAMP XSI IP. If no, check the configuration of the DAMP. If yes, check the DAMP XSI network (hardware and software).

Recovery:

1. Make sure the Monitoring Connection Timeout setting is not less than 2 seconds. Change to a higher value, if required
2. From the SO GUID, navigate to IPFE, and then Configuration, and then Target Sets. At least on DAMP server XSI IP address should display.
3. Log into the IPFE server.
   • ssh to admusr@ @<IPFE XMI IP>
   • ping <the DAMP server XSI IP>
   • telnet <the DAMP server XSI IP> <monitoring port, default 9675>

     The telnet terminal prints gibberish at even intervals. These are the raw heartbeat messages. If you see nothing, then the DSR is not sending hearbeats.

   • ssh to admusr@ @<DAMP server XMI>
   • sudo netstat -anop | grep <monitoring port, default 9675> to see if there is a TCP listen socket on the DAMP XSI IP

     If no, check the configuration of the DAMP

     If yes, check the DAMP XSI network (switch/firewall...)

4. If application servers have been configured correctly for the target set and the application server status is healthy, it is recommended to contact My Oracle Support for assistance. Collect this data first:
   • Screenshot of IPFE, and then Configuration, and then Target Sets edit screen.
   • ifconfig>ifconfig_$(hostname)
   • cat /etc/sysconfig/network > network_$(hostname)
   • cat /etc/modeprobe.d/bnx2x.conf > bnx2x.conf_$(hostname)
   • cat /etc/sysconfig/network-scripts/ifcfg-eth01