Realm Faults
This section explains how to access realm fault statistics.
Signaling
Use the following command to display SIP realm statistics:
- show sipd realms
Viewing SIP Realm Statistics
Display SIP realm statistics by using the show sipd realms command. For example:
ORACLE# show sipd realms
19:38:17-18
---- Inbound ---- --- Outbound ---- -Latency- --- Max ---
Realm Active Rate ConEx Active Rate ConEx Avg Max Burst In Out
external 0 0.0 0 0 0.0 0 0.0 0.0 0 0 0
external-child 0 0.0 0 0 0.0 0 0.0 0.0 0 0 0
internal 0 0.0 0 0 0.0 0 0.0 0.0 0 0 0
Media Statistics
You can use the following commands to display information about mbcd realms:
- show mbcd realms
- show mbcd realms <realm name>
- show flows
There are also alarms that occur when the following events happen:
- out of memory
- internal
- unknown realm
- realm change
- out of bandwidth
- out of ports
Viewing MBCD Steering Port and Bandwidth Usage for Realms
Display steering ports and bandwidth usage for home, public, and private realms by using the show mbcd realms command.
For example:
acmepacket# show mbcd realms
18:46:29-2819
--- Steering Ports --- ----------- Bandwidth Usage ----------
Realm Used Free No Ports Flows Ingrss Egress Total Insuf BW
acme 0 0 0 0 0K 0K 0K 0
h323172 0 30001 0 0 0K 0K 0K 0
sip172 2 29999 0 0 0K 0K 0K 0
sip192 2 29999 0 0 0K 0K 0K 0
The information displayed includes the following:
- Used—Number of steering ports used
- Free—Number of free steering ports
- No Ports—Number of times that a steering port could not be allocated
- Flows—Number of established media flows
- Ingress—Amount of bandwidth being used for inbound flows
- Egress—Amount of bandwidth being used for outbound flows
- Total—Maximum bandwidth set for this realm
- Insuf BW—Number of times that a session was rejected due to insufficient bandwidth.
Viewing MBCD Statistics for a Specific Realm
Display media statistics for a specific realm by using the show mbcd realms <realm-name> command. This information is given for period and lifetime durations.
- Ports Used—Number of ports used
- Free Ports—Number of free ports
- No Ports Avail—Number of times no steering ports were available
- Ingress Band—Amount of bandwidth used for inbound flows
- Egress Band—Amount of bandwidth used for outbound flows
- BW Allocations—Number of times that bandwidth was allocated
- Band Not Avail—Number of times a session was rejected due to insufficient bandwidth
For example:
acmepacket# show mbcd realms sip172
18:47:31-2881 Realm=sip172
-- Period -- -------- Lifetime --------
Active High Total Total PerMax High
Ports Used 2 2 18 18 18 2
Free Ports 29999 30001 30017 30017 30017 30001
No Ports Avail - - 0 0 0 -
Ingress Band 0K 0K 0 0 0 0K
Egress Band 0K 0K 0 0 0 0K
BW Allocations 0 0 0 0 0 0
Band Not Avail - - 0 0 0 -
Total Bandwidth=0K
Steering Ports: 100% Success
Viewing MBCD Task Errors
The show mbcd errors command displays MBCD task error statistics, starting with a time stamp that shows when the current period began.
For example:
ORACLE# show mbcd errors
11:42:37-198
MBC Errors/Events ---- Lifetime ----
Recent Total PerMax
Client Errors 0 0 0
Client IPC Errors 0 0 0
Open Streams Failed 0 0 0
Drop Streams Failed 0 0 0
Monitor Streams Failed 0 0 0
Exp Flow Events 0 0 0
Exp Flow Not Found 0 0 0
Transaction Timeouts 0 0 0
Server Errors 0 0 0
Server IPC Errors 0 0 0
Flow Dup Replace Faile 0 0 0
Flow Duplicated 0 0 0
NatFlow install Errors 0 0 0
NatFlow apply Errors 0 0 0
NatFlow destry Errors 0 0 0
Flow Add Failed 0 0 0
Flow Delete Failed 0 0 0
Flow Update Failed 0 0 0
Flow Latch Failed 0 0 0
Pending Flow Expired 0 0 0
ARP Wait Errors 0 0 0
Exp CAM Not Found 0 0 0
Drop Unknown Exp Flow 0 0 0
Drop/Exp Flow Missing 0 0 0
Exp Notify Failed 0 0 0
Unacknowledged Notify 0 0 0
No Flows In Chain 0 0 0
Main Flow Notify Skips 0 0 0
Notify Not Sent (Skip) 0 0 0
Drop Chain Failures 0 0 0
ACL Deletes 0 0 0
Saved Flows 0 0 0
NoTimeout on otherFlow 0 0 0
Ignore TimerOn Relatch 0 0 0
Relatching Timeouts 0 0 0
Invalid Realm 0 0 0
No Ports Available 0 0 0
Insufficient Bandwidth 0 0 0
Stale Ports Reclaimed 0 0 0
Stale Flows Replaced 0 0 0
Telephone Events Gen 0 0 0
Media Playback Fails 0 0 0
Playback Exh Resources 0 0 0
Playback Flow Inactive 0 0 0
Playback Mismatch 0 0 0
Pipe Alloc Errors 0 0 0
Pipe Write Errors 0 0 0
Not Found In Flows 0 0 0
MPO Realm Mismatch 0 0 0
XCode Internal Errors 0 0 0
XCode Alloc Errors 0 0 0
XCode Update Errors 0 0 0
XCode Delete Errors 0 0 0
XCode Over Cap Errors 0 0 0
XCode Over License Cap 0 0 0
SRTP Flow Add Failed 0 0 0
SRTP Flow Delete Faile 0 0 0
SRTP Flow Update Faile 0 0 0
SRTP Capacity Exceeded 0 0 0
Adds 0 0 0
Deletes 0 0 0
Updates 0 0 0
Client statistics count errors and events encountered by applications that use the MBCD to set up and tear down media sessions:
- Client Errors—Number of errors in the client application related to MBC transactions that are otherwise uncategorized
- Open Streams Failed—Number of MBC transactions creating or updating a media session that could not be sent to the MBCD because the media session state information could not be located
- Drop Streams Failed—Number of MBC transactions deleting a media session that could not be sent to MBCD because the media session state information could not be located
- Client IPC Errors—Number of errors in the client application related to the Inter-Process Communication
- No Session (Open)—Number of MBC transactions creating or updating a media session that could not be sent to MBCD because the media session state information could not be located
- No Session (Drop)—Number of MBC transactions deleting a media session that could not be sent to MBCD because the media session state information could not be located
- Monitor Streams Failed
- Exp Flow Events—Number of flow timer expiration notifications received from the MBCD by all applications
- Exp Flow Not Found—Number of flow timer expiration notifications received from the MBCD by all applications for which no media session or flow information was present in the application
- Transaction Timeouts—Number of MBC transaction timeouts
Server statistics count errors and events encountered by MBCD:
- Server Errors—Number of uncategorized errors in the MBC server
- Server IPC Errors—Number of errors on the server related to the IPC
- Flow Dup Replace Failed—Number of times failed to replace duplicate flow additions Doc Update
- Flow Duplicated—Number of times flow collisions observed when adding flows
- NatFlow install Errors—Number of flow errors occurred when installing NAT flows
- NatFlow apply Errors—Number of flow errors occurred when encountered when trying to apply ppm
- NatFlow destry Errors—Number of flow errors occurred when encountered when trying to delete ppm
- Flow Add Failed—Number of errors encountered when attempting to add an entry to the NAT table
- Flow Delete Failed—Number of errors encountered when attempting to remove an entry from the NAT table
- Flow Update Failed—Number of errors encountered when attempting to update an entry in the NAT table upon receipt of the first packet for a media flow
- Flow Latch Failed—Number of errors when attempting to locate an entry in the NAT table upon receipt of the first packet for a media flow
- Pending Flow Expired—Number of flow timer expirations for pending flows that have not been added to the NAT table
- ARP Wait Errors—Number of errors and timeouts related to obtaining the Layer 2 addressing information necessary for sending media
- Exp CAM Not Found—Number that the NAT table entry for an expired flow could not find in the NAT table. This usually occurs due to a race condition between the removal of the NAT entry and the flow timer expiration notification being sent to MBCD.
- Drop Unknown Exp Flow—Number of flows deleted by the MBCD because of a negative response from the application to a flow timer expiration notification
- Drop/Exp Flow Missing—Number of flows not found when trying to delete them
- Unk Exp Flow Missing—Number of negative responses from the application to a flow timer expiration notification for which the designated flow could not be found in MBCD's tables
- Exp Notify Failed—Number of errors encountered when the MBCD attempted to send a flow timer expiration notification to the application
- Unacknowledged Notify—Number of flow expiration notification messages sent from MBCD to the application for which MBCD did not receive a response in a timely manner
- No Flows In Chain—Number of flows which are not part of other flows
- Main Flow Notify Skips—Number of flows for which NOTIFY is sent to other flows but not to the original flow
- Notify Not Sent (Skip)—Number of flows for which NOTIFY was not sent
- Drop Chain Failures—Number of failures observed for flow chain drops
- ACL Deletes—Number of ACL flow deletes
- Saved Flows—Number of saved flows not teared down in case of timeouts
- NoTimeout on otherFlow—Number of flows for which timeout not occurred for other flow
- Ignore TimerOn Relatch—Number of flows for which relatching is ignored as timeout occurred
- Relatching Timeouts—Number of flows for which relatching timeout has occurred
- Invalid Realm—Number of flow setup failures due to an unknown realm in the request from the application
- No Ports Available—Number of times ports were not available for allocating to flows
- Insufficient Bandwidth—Number of flow setup failures due to insufficient bandwidth in the ingress or egress realm
- Stale Ports Reclaimed—For an HA node, this is the number of ports that were reclaimed when the standby had a stale flow that the active system replaced; when the flow is replaced, the steering ports are also reallocated properly (i.e., according to the active system)
- Stale Flows Replaced—For an HA node, this is the number of times that the standby system had entries in its flow tables that did not match those on the active system; the active system replaced the standby’s stale flows with valid ones
- Telephone Events Gen—Number of times telephony events were generated
- Media Playback Fails—Number of times media playback failures occurred
- Playback Exh Resources—Number of times media playback failures occurred due to out of resources
- Playback Flow Inactive—Number of times playback was stopped as flow was inactive
- Playback Mismatch—Number of times media playback failures was observed due to playback codec mismatch
- Pipe Alloc Errors—Errors Number of times media pipe allocation failures were observed
- Pipe Write Errors—Number of times failed to write data to pipe
- Not Found In Flows—Number of times flow was not found in NAT table
- MPO Realm Mismatch—Number of times realm mismatch errors were observed in mediarealm list
- XCode Internal Errors—Number of uncategorized errors due to Transcoding session error
- XCode Alloc Errors—Number of times that buffer allocation failed for transcoding tasks
- XCode Update Errors—Number of errors encountered when attempting to update an entry in the transcoding table
- XCode Delete Errors—Number of errors encountered when attempting to delete an entry in the transcoding table
- XCode Over Cap Errors—Number of Transcoding sessions denied once session capacity is reached
- XCode Over License Cap—Number of Transcoding sessions denied once capacity is reached
- SRTP Flow Add Failed—Number of times failed to delete SRTP flows to NAT table
- SRTP Flow Delete Failed—Number of times failed to delete SRTP flows from NAT table
- SRTP Flow Update Failed—Number of times failed to update SRTP flows from NAT table
- SRTP Capacity Exceeded—Number of times SRTP maximum capacity reached
- Adds—Number of SRTP flows added to NAT table
- Deletes—Number of SRTP flows deleted from NAT table
- Updates—Number of SRTP flows updated in NAT table
Viewing Realm Configurations
You can use the show realm command to display all realm-specific configurations. For example:
ORACLE# show realm
14:27:38-56SIP Realm Statistics
-- Period -- ------- Lifetime -------
Realm Active Rate High Total Total PerMax High
realm1
Inbound 0 0.0 0 0 0 0 0
Outbound 0 0.0 0 0 0 0 0
Viewing Realm Configurations for a Specific Realm
ORACLE# show realm realm1
realm stats for : Realm: realm1
14:29:22-40
Realm realm1 NO ACTIVITY
Viewing Monthly Minutes for a Specific Realm
You can use the show monthly minutes <realm-id> command to display the monthly minutes for a specified realm. For example:
ORACLE# show monthly-minutes realm1
14:31:33-51
Realm MinutesAllowed MinutesLeft Minutes Exceed Rejects
------------ ------------- ----------- --------------------
Recent Total PerMax
realm1 10 10 0 0 0
Media Alarms
The following table lists information about the different media alarms.
Alarm Name | Alarm ID | Alarm Severity | Cause(s) | Example Log Message | Actions |
---|---|---|---|---|---|
MBCD ALARM OUT OF MEMORY | 262145 | CRITICAL: for flow
MAJOR: for media (if server cannot allocate a new context) |
No further memory can be allocated for MBCD. | Flow: Cannot create free port list for realm.
Media Server: Failed to allocate new context. |
apSyslogMessageGenerated
apSysMgmtMediaOutofMemory trap generated |
MBCD ALARM INTERNAL | 262146 | MINOR | An internal software error. | Internal Error. No agent for socket <IPPort>. | None |
MBCD ALARM UNKNOWN REALM | 262147 | MAJOR: if media server is adding a new flow | Media server is unable to find realm interface. | Realm type (ingress, egress, hairpin) X, not found | apSyslogMessageGenerated
apSysMgmtUnknownRealm |
MBCD ALARM OUT OF BANDWIDTH | 262149 | CRITICAL: failure rate = 100%
MAJOR: failure rate > or = 50% |
The realm is out of bandwidth. | Out of bandwidth | apSyslogMessageGenerated
apSysMgmtMediaBandwidthTrap |
MBCD ALARM OUT OF PORTS | 262150 | CRITICAL: failure rate = 100%
MAJOR: failure rate > or = 50% |
The realm is out of steering ports. | Out of steering ports | apSyslogMessageGenerated
apSysMgmtMediaPortsTrap |
The Steering Pool Threshold Alarm
You can configure the SBC with the thresholds that trigger an alarm when the steering-pool elements in a realm begin to run low on ports. This alarm is always categorized as a MAJOR alarm and is disabled by default. The system issues a separate alarm when it begins to drop calls because all steering ports are exhausted. This alarm gives you the opportunity to take corrective action before the system starts dropping calls.
Each steering-pool has a defined number of ports. Each realm can have multiple steering-pool elements. To address port utilization issues, the system can keep a running total of ports in use as a percentage of total available ports on a per-realm basis. Available ports include those available from both parent and child realms. When configured, the system compares this utilization against your configuration to determine whether it should trigger this alarm.
To have the SBC generate an alarm when a realm begins to run low on steering-pool ports, you configure the following parameters within the realm-config:
- steering-pool-threshold—Specifies, in percent utilization, the value above which the system triggers the alarm. The default value is 0%, the range is 0 - 100% and the recommended setting is 80%.
- steering-pool-lower-threshold—Specifies, in
percent utilization, the value below which the system considers the alarm condition
as no longer in effect. The default value is 0%, the range is 1 - 95% and the
recommended setting is 70%, or your steering-pool-threshold
minus 10.
When port utilization falls below this threshold, the system considers the alarm cleared condition and can re-issue the alarm if and when it is triggered.
- steering-pool-alarm-monitoring-time—Specifies
the duration for which the system monitors port utilization as being between the
steering-pool-threshold and the
steering-pool-lower-threshold. The default is 15 minutes
and the range is 5 - 600:
- If the steering pool utilization remains between the steering-pool-threshold and the steering-pool-lower-threshold for the duration of the steering-pool-alarm-monitoring-time window, the system considers the alarm condition cleared.
- If the percent utilization goes above the steering-pool-threshold at any point during the steering-pool-alarm-monitoring-time window, the system considers the alarm condition as still in effect and does not clear or re-issue the alarm.
For HA deployments, the steering-pool-alarm-monitoring-time is not maintained across systems. if the primary fails over, the system restarts the steering-pool-alarm-monitoring-time timer on the new primary.
The SBC issues a trap corresponding to this alarm simultaneously using the apSipSteeringPoolThresholdCrossedNotify (OID 1.3.6.1.4.1.9148.3.15.6.1.0.1) trap. This trap includes alarm/trap details, including configured threshold and the current percentage of used steering pools. The system does not issue a corresponding clear trap.
When triggered, the alarm appears as follows:
ID Task Severity First Occurred Last Occurred
262154 117 4 2022-02-22 05:36:17 2022-02-22 05:39:17
Count Description
2 Steering ports utilization for one or more realm is over configured
threshold | Realm_1 is at 81 %, configured threshold: 80%.
Note:
Port utilization includes ports configured for parent and child realms, but the alarm text presents the parent realm name only.You can find this alarm in the log.brokerd and acmelog files. You can manually clear this alarm with the clear-alarm command. The system does not clear this alarm itself.
Notice that the alarm includes a First and Last occurrence timestamp, as well as a count. When the system generates multiple steering-pool alarms, which have the same alarm ID, it manages the alarm as follows:
- If you cleared the previous alarm manually, the system completely removes the alarm, including all fields, and generates brand new alarms if the issue occurs again.
- If you do not clear the previous alarm manually, the system labels any ensuing alarms by specifying the realm that is experiencing the current alarm, incrementing the count and updating the “Last Occurred” timestamp.
Viewing Deny ACL List
Display a list of deny ACLI entries by using the acl-show command at the topmost ACLI prompt. The following information is displayed:
- Incoming port, slot, and VLAN tag
- Source IP, bit mask, port, and port mask
- Destination IP address and port
- Protocol
- ACL entry as static and dynamic
- ACL entry index
For example:
ORACLE# acl-show
deny entries:
intf:vlan source-ip/mask:port/mask dest-ip/mask:port/mask prot type index
Total number of deny entries = 0
Denied Entries not allocated due to ACL constraints: 0
task done