5009 - No available servers in target set

Alarm Group:

IPFE

Description:

Through monitoring of the application servers, the IPFE learns no server in a target set is available. The associated measurement, TxReject, also shows counts (refer to the DSR Measurements Reference for details about this measurement). This alarm can be triggered during configuration of the IPFE when the target set address has been configured, but application servers have not yet been added to the target set. Setting the Monitoring Connection Timeout to a value less than 2 seconds is the primary cause of this alarm. It is recommended to leave this setting on the default of 3 seconds. Do not set to 1 second. Later releases prohibit this from being set to 1 second.

Each target set is configured with at least one backend application server (DAMP). The IPFE raises the 5009 alarm when the IPFE detects no DAMP is live. The IPFE detects the DAMP liveliness by receiving the DAMP heartbeat on time.

Severity:

Critical

Instance:

tsa N has no available servers where N is 1-16

Note:

When at least one application server in a target set becomes available, this alarm clears.

HA Score:

Normal

Auto Clear Seconds:

N/A

OID:

ipfeIpfeNoAvailableAppServersNotify

Recovery:

1. Make sure the Monitoring Connection Timeout setting is not less than 2 seconds. Change to a higher value, if required
2. From the SO GUID, navigate to IPFE > Configuration > Target Sets. At least on DAMP server XSI IP address should display.
3. Log into the IPFE server.
   • ssh to admusr@ @<IPFE XMI IP>
   • ping <the DAMP server XSI IP>
   • telnet <the DAMP server XSI IP> <monitoring port, default 9675>

     The telnet terminal prints gibberish at even intervals. These are the raw heartbeat messages. If you see nothing, then the DSR is not sending hearbeats.

   • ssh to admusr@ @<DAMP server XMI>
   • sudo netstat -anop | grep <monitoring port, default 9675> to see if there is a TCP listen socket on the DAMP XSI IP

     If no, check the configuration of the DAMP

     If yes, check the DAMP XSI network (switch/firewall...)

4. If application servers have been configured correctly for the target set and the application server status is healthy, it is recommended to contact My Oracle Support (MOS) for assistance. Collect this data first:
   • Screenshot of IPFE > Configuration > Target Sets edit screen.
   • ifconfig>ifconfig_$(hostname)
   • cat /etc/sysconfig/network > network_$(hostname)
   • cat /etc/modeprobe.d/bnx2x.conf > bnx2x.conf_$(hostname)
   • cat /etc/sysconfig/network-scripts/ifcfg-eth01