22350 - Fixed Connection Alarm Aggregation Threshold

Alarm Group:

DIAM

Description:

This alarm occurs when there are a critical number of fixed connection alarms for the DA-MP.

Note:

The alarm thresholds are configurable using the Alarm Threshold Options tab on Diameter > Configuration > System Options.

Severity:

Major, Critical

Note:

The Critical threshold may be disabled by setting the Critical Threshold to zero using the Alarm Threshold Options tab on Diameter > Configuration > System Options.

Instance:

<DA-MP-Hostname>

HA Score:

Normal

Auto Clear Seconds:

0 (zero)

OID:

eagleXgDiameterConnUnavailableThresholdReachedNotify

Cause:

The alarm #22350 raises when there are a critical number of fixed connection alarms for the DA-MP.

Diagnostic Information:

To get further information regarding this issue:

1. Navigate to Diameter > Maintenance > Connection to find all the connections with a problem for the specific MP.
2. For each connection with a problem, verify:
   1. The remote host is reachable from the local MP by using ssh to the MP and pinging the remote server IP (if using IP address) or server FQDN (if using FQDN)
   2. DNS availability should be tested by pinging the DNS server IP
   3. FQDN resolving should be tested by using nslookup to check the FQDN resolving on the MP
3. If the above tests reveal the remote host is not reachable, then verify that there is no network problem on the remote server.
4. If the remote server is reachable, then verify the processes are running correctly.
   1. Verify the local DSR process is running by checking the ps -ef output
   2. Verify the local node is listening on the correct port by using netstat -na and checking the correct transport type, tcp/sctp port is listening
   3. Use wireshark or tcpdump to capture traffic messages, and verify the connection is established (confirm the handshake process is occurring for SCTP or TCP)
5. If the port is not listening, or the handshake procedure is not occurring, then the process or server may be in trouble.
6. If the connection/association is established, then ensure that the Diameter handshake is happening and correct, by checking the Diameter CEX message exchange, for information like server FQDN, IP address, or applications supported; mismatching information causes the connection to abort.
7. If Diameter handshake is good, then observe the health of the Diameter connection by verifying the DWR messages are answered correctly.

Recovery:

1. Navigate to Diameter > Maintenance > Connection to monitor Fixed Connection status.
2. Confirm the peer connection configuration (protocol, remote/local IP address, remote/local port) matches the local connection configuration.
3. Confirm the connection’s transport protocol and/or port are not being blocked by a network firewall or other ACL in the network path.
4. Verify the peers in the Route List are not under maintenance.
5. Navigate to Diameter > Configuration > System Options to check the Alarm Threshold Options setting. Modify the value if it is set too low.
6. It is recommended to contact My Oracle Support for assistance.