8 Alerts

This section provides information on Policy alerts and their configuration.

Note:

The performance and capacity of the system can vary based on the call model, configuration, including but not limited to the deployed policies and corresponding data, for example, policy tables.

You can configure alerts in Prometheus and Alertrules.yaml file.

The following table describes the various severity types of alerts generated by Policy:

Table 8-1 Alerts Levels or Severity Types

Alerts Levels / Severity Types	Definition
Critical	Indicates a severe issue that poses a significant risk to safety, security, or operational integrity. It requires immediate response to address the situation and prevent serious consequences. Raised for conditions can affect the service of Policy.
Major	Indicates a more significant issue that has an impact on operations or poses a moderate risk. It requires prompt attention and action to mitigate potential escalation. Raised for conditions can affect the service of Policy.
Minor	Indicates a situation that is low in severity and does not pose an immediate risk to safety, security, or operations. It requires attention but does not demand urgent action. Raised for conditions can affect the service of Policy.
Info or Warn (Informational)	Provides general information or updates that are not related to immediate risks or actions. These alerts are for awareness and do not typically require any specific response. WARN and INFO alerts may not impact the service of Policy.

For details on how to configure Policy alerts, see Configuring Alerts section in Oracle Communications Cloud Native Core, Converged Policy Installation, Upgrade, and Fault Recovery Guide.

For details on how to configure SNMP Notifier, see Configuring SNMP Notifier section in Oracle Communications Cloud Native Core, Converged Policy Installation, Upgrade, and Fault Recovery Guide.

8.1 List of Alerts

This section provides detailed information about the alert rules defined for Policy. It consists of the following three types of alerts:

1. Common Alerts - This category of alerts is common and required for all three modes of deployment.
2. PCF Alerts - This category of alerts is specific to PCF microservices and required for Converged and PCF only modes of deployment.
3. PCRF Alerts - This category of alerts is specific to PCRF microservices and required for Converged and PCRF only modes of deployment.

8.1.1 Common Alerts

This section provides information about alerts that are common for PCF and PCRF.

8.1.1.1 POD_CONGESTION_L1

Table 8-2 POD_CONGESTION_L1

Field	Details
Name in Alert Yaml File	PodCongestionL1
Description	Alert when CPU of pod is in CONGESTION_L1 state.
Summary	Pod Congestion status of {{$labels.microservice}} service is Congestion_L1
Severity	Critical
Expression	occnp_pod_congestion_state == 2
OID	1.3.6.1.4.1.323.5.3.52.1.2.71
Metric Used	occnp_pod_resource_congestion_state
Recommended Actions	Cause: The pod has entered a platform/app-defined congestion state (L1). Common reasons: rising queue depth, high CPU, memory pressure, threadpool saturation, or downstream slowness. Diagnostic Information: Identify which pods are in L1 (by pod/namespace labels on the metric) - max by (namespace,pod) (occnp_pod_congestion_state) Correlate with resource congestion sub-metrics (queue/cpu) - max by (namespace,pod,type) (occnp_pod_resource_congestion_state) Correlate with Kubernetes/node signals - CPU throttling, restarts, OOMKills, latency, pending requests. Recovery: Relieve pressure: scale out replicas, reduce incoming load (rate limiting/admission controls), and/or restore slow downstream dependencies. If CPU throttling: raise requests/limits appropriately and remove noisy-neighbor issues. If caused by a specific dependency outage, restore it first; congestion should clear as backlog drains.

8.1.1.2 POD_CONGESTION_L2

Table 8-3 POD_CONGESTION_L2

Field	Details
Description	Alert when CPU of pod is in CONGESTION_L2 state.
Summary	Pod Congestion status of {{$labels.microservice}} service is Congestion_L2
Severity	Critical
Expression	occnp_pod_congestion_state == 3
OID	1.3.6.1.4.1.323.5.3.52.1.2.72
Metric Used	occnp_pod_resource_congestion_state
Recommended Actions	Cause: Severe congestion (L2), typically meaning the pod is protecting itself by shedding/limiting work; likely to cause timeouts/errors. Diagnostic Information: Same as L1, but treat as incident: Look for rapidly growing queue, high CPU, increased timeouts/errors. Recovery: Immediate load shedding / reroute / scale out. Restore downstream dependencies. Consider temporarily draining/isolating the worst pods if only some are impacted; validate service stabilizes and state returns < L1.

8.1.1.3 POD_PENDING_REQUEST_CONGESTION_L1

Table 8-4 POD_PENDING_REQUEST_CONGESTION_L1

Field	Details
Description	Pod resource is in Congestion_L1 for Pending Request type
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L1 for resource type queue
Severity	Critical
Expression	occnp_pod_resource_congestion_state{type="queue"} == 2
OID	1.3.6.1.4.1.323.5.3.52.1.2.73
Metric Used	occnp_pod_resource_congestion_state
Recommended Actions	Cause: Pending-request queue/backlog is high enough to trigger L1 congestion; usually due to traffic spike, slow processing, or downstream slowness. Diagnostic Information: Confirm queue congestion pods: - max by (namespace,pod) (occnp_pod_resource_congestion_state{type="queue"}) Correlate with pending/in-flight metrics (if available): - occnp_diam_pending_requests (or equivalent) Recovery: Scale out or add capacity, reduce traffic, or fix slow downstream calls. Identify and remove hotspots (single peer, single route, or single tenant) if applicable.

8.1.1.4 POD_PENDING_REQUEST_CONGESTION_L2

Table 8-5 POD_PENDING_REQUEST_CONGESTION_L2

Field	Details
Description	Alert when queue of pod is in CONGESTION_L2 state.
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L2 for resource type queue
Severity	Critical
Expression	occnp_pod_resource_congestion_state{type="queue"} == 3
OID	1.3.6.1.4.1.323.5.3.52.1.2.74
Metric Used	occnp_pod_resource_congestion_state
Recommended Actions	Cause: Severe queue backlog; likely to manifest as Diameter timeouts and retries. Diagnostic Information: Same as L1 plus check for request retry storms and timeouts. Recovery: Immediately scale out + load shedding + fix downstream slowness; validate queue drains and congestion state returns to normal.

8.1.1.5 POD_CPU_CONGESTION_L1

Table 8-6 POD_CPU_CONGESTION_L1

Field	Details
Description	Alert when cpu of pod is in CONGESTION_L1 state.
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L1 for CPU type
Severity	Critical
Expression	occnp_pod_resource_congestion_state{type="cpu"} == 2
OID	1.3.6.1.4.1.323.5.3.52.1.2.73
Metric Used	occnp_pod_resource_congestion_state
Recommended Actions	Cause: Pod has hit CPU-based congestion threshold: sustained high CPU, throttling, inefficient workload, or hot shard. Diagnostic Information: Confirm affected pods: - max by (namespace,pod) (occnp_pod_resource_congestion_state{type="cpu"}) Check CPU throttling/usage (Kubernetes / node exporter metrics) and GC/log indicators. Recovery: Add CPU capacity (HPA/replicas), adjust CPU requests/limits to reduce throttling, re-balance traffic, and investigate hot loops/regressions. If due to downstream retries/timeouts, fix root cause to stop wasted CPU.

8.1.1.6 POD_CPU_CONGESTION_L2

Table 8-7 POD_CPU_CONGESTION_L2

Field	Details
Description	Alert when CPU of pod is in CONGESTION_L2 state.
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L2 for CPU type
Severity	Critical
Expression	occnp_pod_resource_congestion_state{type="cpu"} == 3
OID	1.3.6.1.4.1.323.5.3.52.1.2.74
Metric Used	occnp_pod_resource_congestion_state
Recommended Actions	Cause: Severe CPU starvation; high likelihood of cascading failures/timeouts. Diagnostic Information: Same as L1 but urgent; correlate with error/timeout spikes. Recovery: Immediate scale out, reduce load, mitigate CPU throttling, rollback recent changes if regression suspected; validate CPU drops and service recovers.

8.1.1.7 RAA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-8 RAA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	RAA Rx fail count exceeds the critical threshold limit.
Summary	Alert RAA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	CRITICAL
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.35
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed RAA (Re-Auth Answer) Diameter responses for application ID 16777236 (Rx interface) has exceeded the critical threshold (greater than 90%). The condition indicates a severe failure in handling re-authorization responses over the Rx interface, which can disrupt policy updates, QoS adjustments, and session control between PCRF/PCF and the Application Function (AF). Likely causes include communication failures with AF, policy inconsistencies, or system instability. Diagnostic Information: Evaluate the following metric: Failed RAA (Rx) Responses / Total RAA (Rx) Responses * 100 over a 5-minute window. Scope: msgType = "RAA" (Re-Auth Answer) appId = "16777236" (Rx interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 90% Perform the following checks: Analyze Diameter response codes (e.g., authorization failures, unable to comply, system errors) Verify connectivity and health of AF (Application Function) peers Check policy synchronization between PCRF/PCF and AF Review logs for re-authorization failures, timeouts, or invalid requests Monitor signaling performance (latency, retransmissions, error spikes) Inspect system resource utilization (CPU, memory, thread pools) Correlate with spikes in re-auth requests or policy update events Recovery: Restore connectivity and stability with AF/Rx peer systems Resolve policy inconsistencies or invalid re-authorization requests Address specific Diameter error codes causing failures Restart or scale affected components handling Rx traffic Tune timeout and retry mechanisms if required Roll back recent configuration or deployment changes if correlated Monitor until failure rate drops below the critical threshold For any additional guidance, contact My Oracle Support.

8.1.1.8 RAA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-9 RAA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	RAA Rx fail count exceeds the major threshold limit.
Summary	Alert RAA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	MAJOR
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.35
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed RAA (Re-Auth Answer) Diameter responses for application ID 16777236 (Rx interface) has exceeded the major threshold (greater than 80% and up to 90%). The condition indicates a high failure rate in re-authorization handling, leading to partial degradation in policy updates, QoS enforcement, and session control between PCRF/PCF and the Application Function (AF). The issue may be due to backend instability, communication issues, or increasing signaling load. Diagnostic Information: Evaluate the following metric: Failed RAA (Rx) Responses / Total RAA (Rx) Responses * 100 over a 5-minute window Scope: msgType = "RAA" appId = "16777236" (Rx interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 80% and ≤ 90% Perform the following checks: Analyze response code distribution (e.g., authorization failures, temporary/system errors) Verify connectivity and health of AF (Application Function) peers Check policy synchronization between PCRF/PCF and AF Review logs for intermittent re-auth failures, timeouts, or malformed requests Monitor signaling performance (latency, retransmissions, error trends) Inspect system resource usage (CPU, memory, thread pools) Correlate with spikes in re-authorization requests or policy update events Recovery: Investigate and stabilize connectivity with AF/Rx peer systems Resolve policy inconsistencies or invalid re-authorization scenarios Address specific Diameter error codes contributing to failures Optimize or scale system resources to handle load Restart affected components if necessary Roll back recent configuration or deployment changes if correlated Closely monitor to prevent escalation to the critical threshold For any additional guidance, contact My Oracle Support.

8.1.1.9 RAA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-10 RAA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	RAA Rx fail count exceeds the minor threshold limit.
Summary	Alert RAA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	MINOR
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.35
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed RAA (Re-Auth Answer) Diameter responses for application ID 16777236 (Rx interface) has exceeded the minor threshold (greater than 60% and up to 80%). The condition indicates an early stage of degradation in re-authorization handling, potentially affecting policy updates, QoS enforcement, and session control. While the system is still functional, a noticeable portion of re-auth responses are failing, signaling underlying issues such as intermittent connectivity problems or rising load. Diagnostic Information: Evaluate the following metric: Failed RAA (Rx) Responses / Total RAA (Rx) Responses * 100 over a 5-minute window Scope: msgType = "RAA" appId = "16777236" (Rx interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 60% and ≤ 80% Perform the following checks: Observe trends to determine if failure rate is increasing toward major/critical levels Analyze response codes for early indicators (e.g., temporary failures, authorization issues) Verify connectivity and responsiveness of AF (Application Function) peers Check policy synchronization between PCRF/PCF and AF Review logs for intermittent re-auth failures, retries, or timeout issues Monitor system performance (CPU, memory, thread pools, latency) Identify if failures correlate with traffic spikes or policy update events Recovery: Investigate and address early signs of issues in Rx signaling or AF connectivity Validate and correct policy configurations or inconsistencies Optimize or scale system resources to handle increasing load Resolve issues based on observed Diameter error codes Apply preventive measures (retry tuning, connection stability improvements) Review and correct recent configuration or deployment changes if applicable Continue close monitoring to prevent escalation to higher severity thresholds For any additional guidance, contact My Oracle Support.

8.1.1.10 ASA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-11 ASA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	ASA Rx fail count exceeds the critical threshold limit.
Summary	Alert ASA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	CRITICAL
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.66
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed ASA (Abort Session Answer) Diameter responses for application ID 16777236 (Rx interface) has exceeded the critical threshold (greater than 90%). The condition indicates a severe failure in handling session abort responses, which can prevent proper session termination initiated by the Application Function (AF). As a result, sessions may remain active incorrectly, leading to policy inconsistencies and potential resource leakage. The issue is likely due to communication failures with AF, session state mismatches, or backend instability. Diagnostic Information: Evaluate the following metric: Failed ASA (Rx) Responses / Total ASA (Rx) Responses * 100 over a 5-minute window Scope: msgType = "ASA" (Abort Session Answer) appId = "16777236" (Rx interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 90% Perform the following checks: Analyze Diameter response codes (e.g., unknown session, unable to comply, system errors) Verify connectivity and health of AF (Application Function) peers Check session state synchronization between PCRF/PCF and AF Review logs for abort session failures, timeouts, or invalid requests Monitor signaling performance (latency, retransmissions, error spikes) Inspect system resources (CPU, memory, session table utilization) Correlate with spikes in abort session requests or abnormal session behavior Recovery: Restore connectivity and stability with AF/Rx peer systems Resolve session state inconsistencies or stale session entries Address specific Diameter error codes causing failures Restart or scale affected components handling Rx interface traffic Clean up orphaned or stuck sessions if required Roll back recent configuration or deployment changes if correlated Monitor until failure rate drops below the critical threshold For any additional guidance, contact My Oracle Support.

8.1.1.11 ASA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-12 ASA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	ASA Rx fail count exceeds the major threshold limit.
Summary	Alert ASA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	MAJOR
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.66
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed ASA (Abort Session Answer) Diameter responses for application ID 16777236 (Rx interface) has exceeded the major threshold (greater than 80% and up to 90%). The condition indicates a high failure rate in handling abort session responses, leading to partial degradation in session termination initiated by the Application Function (AF). This may result in incomplete session cleanup, inconsistent policy states, and increased resource utilization. The issue is likely due to AF connectivity issues, backend instability, or increased signaling load. Diagnostic Information: Evaluate the following metric: Failed ASA (Rx) Responses / Total ASA (Rx) Responses * 100 over a 5-minute window Scope: msgType = "ASA" appId = "16777236" (Rx interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 80% and ≤ 90% Perform the following checks: Analyze response code distribution. For example, unknown session, temporary failures, system errors Verify connectivity and health of AF (Application Function) peers Check session state synchronization between PCRF/PCF and AF Review logs for intermittent abort session failures, timeouts, or malformed requests Monitor signaling performance (latency, retransmissions, error trends) Inspect system resource usage (CPU, memory, session table utilization) Correlate with spikes in abort session requests or abnormal session behavior Recovery: Investigate and stabilize connectivity with AF/Rx peer systems Resolve session state inconsistencies or stale session entries Address specific Diameter error codes contributing to failures Optimize or scale system resources to handle load Restart affected components if necessary Roll back recent configuration or deployment changes if correlated Closely monitor to prevent escalation to the critical threshold For any additional guidance, contact My Oracle Support.

8.1.1.12 ASA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-13 ASA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	ASA Rx fail count exceeds the minor threshold limit.
Summary	Alert ASA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	MINOR
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.66
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed ASA (Abort Session Answer) Diameter responses for application ID 16777236 (Rx interface) has exceeded the minor threshold (greater than 60% and up to 80%). The condition indicates an early stage of degradation in handling abort session responses. While the system remains operational, a significant portion of session abort operations are failing, which may lead to delayed or incomplete session termination, inconsistent policy states, and potential buildup of stale sessions. Diagnostic Information: Evaluate the following metric: Failed ASA (Rx) Responses / Total ASA (Rx) Responses * 100 over a 5-minute window Scope: msgType = "ASA" appId = "16777236" (Rx interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 60% and ≤ 80% Perform the following checks: Observe trends to determine if failure rate is increasing toward major/critical levels Analyze response codes for early indicators (e.g., unknown session, temporary/system errors) Verify connectivity and responsiveness of AF (Application Function) peers Check session state synchronization between PCRF/PCF and AF Review logs for intermittent abort session failures, retries, or timeout issues Monitor system performance (CPU, memory, session table utilization) Identify if failures correlate with traffic spikes or abnormal session behavior Recovery: Investigate and address early signs of issues in Rx signaling or AF connectivity Validate session state consistency across systems Resolve identified issues based on Diameter error codes Optimize or scale system resources to handle load Apply preventive measures (retry tuning, improved session cleanup mechanisms) Review and correct recent configuration or deployment changes if applicable Continue close monitoring to prevent escalation to higher severity thresholds For any additional guidance, contact My Oracle Support.

8.1.1.13 ASA_RX_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-14 ASA_RX_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	ASA Rx timeout count exceeds the minor threshold limit
Summary	Alert ASA_RX_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	MINOR
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.67
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of ASA (Abort Session Answer) Diameter requests timing out for application ID 16777236 (Rx interface) has exceeded the minor threshold (greater than 60% and up to 80%). The condition indicates an early stage of degradation where a noticeable portion of abort session requests are not receiving timely responses. While the system may still be operational, this condition suggests emerging issues such as increasing latency, intermittent AF (Application Function) unresponsiveness, or rising system load. Diagnostic Information: Evaluate the following metric: Timed-out ASA (Rx) Responses / Total ASA (Rx) Requests * 100 over a 5-minute window Scope: msgType = "ASA" appId = "16777236" (Rx interface) responseCode = "timeout" Trigger condition: Timeout rate > 60% and ≤ 80% Perform the following checks: Observe trends to determine if timeout rate is increasing toward major/critical levels Verify connectivity and latency to AF (Application Function) peers Check for intermittent delays or partial unresponsiveness from AF Review logs for timeout events, retries, and delayed responses Monitor Diameter stack metrics (pending requests, queue buildup, retransmissions) Inspect system performance (CPU, memory, thread pools, connection limits) Identify network issues such as packet loss, congestion, or firewall delays Recovery: Investigate and address early signs of latency or connectivity issues with AF/Rx peers Optimize network performance and resolve packet loss or congestion Tune timeout and retry parameters if needed Scale system resources to handle increasing load Apply preventive measures (connection pooling, traffic smoothing) Review and correct recent configuration or deployment changes if applicable Continue close monitoring to prevent escalation to higher severity thresholds For any additional guidance, contact My Oracle Support.

8.1.1.14 ASA_RX_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-15 ASA_RX_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	ASA Rx timeout count exceeds the major threshold limit
Summary	Alert ASA_RX_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	MAJOR
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.67
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of ASA (Abort Session Answer) Diameter requests timing out for application ID 16777236 (Rx interface) has exceeded the major threshold (greater than 80% and up to 90%). The condition indicates a high rate of delayed or missing responses for abort session requests, leading to partial degradation in session termination handling. It may result in incomplete session cleanup and increased resource utilization. The issue is likely due to network latency, partial unresponsiveness of the Application Function (AF), or system overload. Diagnostic Information: Evaluate the following metric: Timed-out ASA (Rx) Responses / Total ASA (Rx) Requests * 100 over a 5-minute window Scope: msgType = "ASA" appId = "16777236" (Rx interface) responseCode = "timeout" Trigger condition: Timeout rate > 80% and ≤ 90% Perform the following checks: Analyze timeout trends to see if they are increasing toward critical levels Verify connectivity and latency to AF (Application Function) peers Check if AF is intermittently slow or partially unresponsive Review logs for timeout events, retransmissions, and delayed responses Monitor Diameter stack health (pending requests, queue buildup, retransmission count) Inspect system resources (CPU, memory, thread pools, connection limits) Check for network congestion, firewall filtering, or load balancer delays Recovery: Investigate and stabilize connectivity with AF/Rx peer systems Reduce network latency and resolve packet loss issues Tune timeout and retry configurations if needed Optimize or scale system resources to handle load Restart affected components if necessary Address infrastructure bottlenecks (firewalls, load balancers, routing issues) Closely monitor to prevent escalation to the critical threshold For any additional guidance, contact My Oracle Support.

8.1.1.15 ASA_RX_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-16 ASA_RX_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	ASA Rx timeout count exceeds the critical threshold limit
Summary	Alert ASA_RX_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	CRITICAL
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA", appId="16777236"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.67
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of ASA (Abort Session Answer) Diameter requests timing out for application ID 16777236 (Rx interface) has exceeded the critical threshold (greater than 90%). The condition indicates a severe degradation where most abort session responses are not received within the expected time, likely due to communication failures with the Application Function (AF), network latency, or overloaded/unresponsive backend systems. Diagnostic Information: Evaluate the following metric: Timed-out ASA (Rx) Responses / Total ASA (Rx) Requests * 100 over a 5-minute window Scope: msgType = "ASA" appId = "16777236" (Rx interface) responseCode = "timeout" Trigger condition: Timeout rate > 90% Perform the following checks: Verify connectivity to AF (Application Function) peers (network reachability, packet loss, latency) Check if AF is responding slowly or not responding at all Review logs for timeout errors, retransmissions, or request drops Monitor Diameter stack metrics (queue size, pending requests, retransmission count) Inspect system resource utilization (CPU, memory, thread pools, connection limits) Check for network congestion, firewall issues, or load balancer delays Correlate with spikes in abort session requests or traffic bursts Recovery: Restore connectivity and responsiveness of AF/Rx peer systems Investigate and resolve network latency or packet loss issues Tune timeout and retransmission parameters if required Scale or restart affected components handling Rx traffic Reduce load or apply traffic throttling if system is overloaded Fix infrastructure issues (firewalls, load balancers, routing) Monitor until timeout rate drops below the critical threshold For any additional guidance, contact My Oracle Support.

8.1.1.16 QOS_MON_SMF_POLICY_FAILURE_CODE_ABOVE_CRITICAL_PERCENT

Table 8-17 QOS_MON_SMF_POLICY_FAILURE_CODE_ABOVE_CRITICAL_PERCENT

Field	Description
Description	More than 70% of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF contains QOS_MON_DECS_ERR
Summary	More than 70% of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF contains QOS_MON_DECS_ERR
Severity	CRITICAL
Expression	sum by (namespace) (rate(occnp_sm_qosmon_decs_error_total[5m]))) / (sum by (namespace) (rate(occnp_pa_qosmon_request_total[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.133
Metric Used	occnp_sm_qosmon_decs_error_total, occnp_pa_qosmon_request_total
Recommended Actions	The number of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF with QOS_MON_DECS_ERR represents 70% or more of the total app sessions. The alert will be cleared once the reject rate reduces below the threshold.

8.1.1.17 SCP_PEER_UNAVAILABLE

Table 8-18 SCP_PEER_UNAVAILABLE

Field	Details
Description	Configured SCP peer is unavailable.
Summary	SCP peer [ {{$labels.peer}} ] is unavailable.
Severity	Major
Expression	occnp_oc_egressgateway_peer_health_status != 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.60
Metric Used	occnp_oc_egressgateway_peer_health_status
Recommended Actions	Cause: One or more SCP peer nodes are unavailable or unhealthy, as indicated by the peer health status not equal to 0. This suggests a failure in connectivity or health monitoring between the system and its configured peers, potentially impacting service routing, request forwarding, and overall signaling reliability. Diagnostic Information: Evaluate the following metric: occnp_oc_egressgateway_peer_health_status != 0 Trigger condition: Peer health status is not equal to 0 (non-healthy state) Interpretation: 0 → Healthy Non-zero → Peer unavailable/unhealthy Perform the following checks: Identify which peer(s) are marked unhealthy Verify network connectivity to the affected peer(s) (ping, TCP checks) Check health check endpoints or heartbeat mechanisms Review logs for connection failures, timeouts, or DNS resolution issues Validate peer configuration (IP, port, certificates, routing rules) Inspect load balancer or gateway behavior if applicable Check for TLS/SSL handshake failures or certificate issues Monitor if multiple peers are affected (indicating broader network/system issue) Recovery: Restore network connectivity to the affected peer(s) Restart or recover the unavailable peer services Fix configuration issues (endpoints, ports, certificates, routing rules) Resolve DNS or service discovery issues if present Ensure health check mechanisms are functioning correctly Failover traffic to healthy peers if supported Scale or redistribute load if peers are overloaded Monitor until peer health status returns to 0 (healthy state)

8.1.1.18 SCP_PEER_SET_UNAVAILABLE

Table 8-19 SCP_PEER_SET_UNAVAILABLE

Field	Details
Description	None of the SCP peer available for configured peerset.
Summary	{{ $value }} SCP peers under peer set {{$labels.peerset}} are currently unavailable.
Severity	Critical
Expression	(occnp_oc_egressgateway_peer_count > 0 and (occnp_oc_egressgateway_peer_available_count) == 0)
OID	1.3.6.1.4.1.323.5.3.52.1.2.61
Metric Used	occnp_oc_egressgateway_peer_count and occnp_oc_egressgateway_peer_available_count
Recommended Actions	Cause: One or more SCP peers within a configured peer set are unavailable. This condition occurs when the number of available peers is less than the total configured peers, indicating partial peer set degradation. While not all peers are down, this reduces redundancy and may impact load balancing, failover capability, and overall service reliability. Diagnostic Information: Evaluate the following metric: (Total Peers - Available Peers) != 0 AND Total Peers > 0 Trigger condition: At least one peer in the configured peer set is unavailable Interpretation: occnp_oc_egressgateway_peer_count → Total configured peers occnp_oc_egressgateway_peer_available_count → Currently healthy/available peers Difference ≠ 0 → Some peers are unavailable Perform the following checks: Identify which peers in the peer set are unavailable Verify network connectivity to affected peers (latency, packet loss, TCP checks) Check health check/heartbeat status for each peer Review logs for failures such as timeouts, connection errors, or TLS issues Validate peer configuration (IP, port, certificates, routing policies) Check if unavailable peers are due to maintenance, scaling events, or crashes Monitor load on remaining peers to ensure they are not overloaded Recovery: Restore connectivity or restart the unavailable peer(s) Fix configuration issues related to endpoints, certificates, or routing Resolve network issues (latency, packet loss, DNS resolution) Re-enable peers if they were intentionally disabled or under maintenance Ensure proper scaling and load distribution across available peers Validate health check mechanisms and thresholds Monitor until all peers in the set are available and stable For any additional guidance, contact My Oracle Support.

8.1.1.19 POLICY_SERVICES_DOWN

Table 8-20 POLICY_SERVICES_DOWN

Field	Details
Description	{{$labels.service}} service is not running.
Summary	{{$labels.service}} service is not running.
Severity	Critical
Expression	sum by(service, namespace, category)(appinfo_service_running{application="occnp",service!~".*altsvc-cache",vendor="Oracle"}) < 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.1
Metric Used	appinfo_service_running{vendor="Oracle", application="occnp", category!=""}!= 1
Recommended Actions	Cause: Indicates that a Policy service becomes unavailable or stops responding. Possible causes include pod failures, service restarts, node issues, network interruptions, deployment problems, or upstream dependency failures. It may also occur when the service is unhealthy due to resource pressure, configuration issues, or backend connectivity problems. Diagnostic Information: Check related Policy service pod status, restarts, and readiness/liveness probe failures. Verify the running status of the service using kubectl get service -n <namespace>. Review service logs from Kibana to identify crashes, startup failures, or connection issues. Check Grafana for CPU, memory, and network trends to see whether the service was under stress before going down. Correlate the alert with recent deployments, config changes, or node-level incidents. Verify whether related dependent services are healthy and reachable. Recovery: Restart the affected service pods or perform a rolling restart if the failure is pod-specific. Roll back recent changes if the issue started after a deployment or configuration update. Fix the underlying cause such as resource exhaustion, probe failure, or dependency outage. If the service does not recover cleanly, escalate to Oracle Support for further investigation.

8.1.1.20 DIAM_TRAFFIC_RATE_ABOVE_THRESHOLD

Table 8-21 DIAM_TRAFFIC_RATE_ABOVE_THRESHOLD

Field	Details
Description	Diameter Connector Ingress traffic Rate is above threshold of Max MPS (current value is: {{ $value }})
Summary	Traffic Rate is above 90 Percent of Max requests per second.
Severity	Major
Expression	sum(rate(ocpm_ingress_request_total{servicename_3gpp=~"rx|sy"}[2m])) >= 900
OID	1.3.6.1.4.1.323.5.3.52.1.2.6
Metric Used	ocpm_ingress_request_total
Recommended Actions	Cause: Indicates that the ingress traffic rate exceeds the configured threshold for the service. This condition indicates a surge in incoming requests due to increased client activity, retries, misbehaving clients, or upstream/downstream issues. Other causes include error loops (4xx/5xx), sudden traffic spikes, or configuration changes affecting routing or throttling. Diagnostic Information: Check ingress traffic graphs to confirm whether the increase is sustained or a short-lived spike. Refer to the Ingress Gateway section in Grafana to identify any rise in 4xx and 5xx response codes. Analyze Ingress Gateway logs in Kibana to determine the source and reason for errors or abnormal traffic patterns. Verify recent deployments or configuration changes, and correlate with traffic sources or client behavior. Review threshold configuration in Common_Alertrules.yaml to ensure it aligns with expected traffic patterns. Recovery: If the traffic spike is legitimate, scale the service or ingress capacity to handle the load. If errors (4xx/5xx) cause the condition, fix the underlying issue (e.g., routing, backend failures, misconfigurations). Mitigate abnormal traffic by rate limiting, blocking faulty clients, or adjusting retry mechanisms. The alert clears automatically once ingress traffic rate falls below the configured threshold.

8.1.1.21 DIAM_INGRESS_ERROR_RATE_ABOVE_10_PERCENT

Table 8-22 DIAM_INGRESS_ERROR_RATE_ABOVE_10_PERCENT

Field	Details
Name in Alert Yaml File	DiamIngressErrorRateAbove10Percent
Description	Transaction Error Rate detected above 10 Percent of Total on Diameter Connector (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 10 Percent of Total Transactions.
Severity	Critical
Expression	sum(rate(ocpm_ingress_response_total{servicename_3gpp=~"rx|sy",response_code!~"2.*"}[24h]))/sum(rate(ocpm_ingress_response_total{servicename_3gpp=~"rx|sy"}[24h])) * 100 >= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.7
Metric Used	ocpm_ingress_response_total
Recommended Actions	Cause: Indicates that failed transactions exceed 10% of total ingress transactions. The condition indicates a high error rate typically caused by backend service failures, misconfigurations, invalid requests, or dependency issues. It may also result from recent deployments, API contract mismatches, or traffic anomalies affecting specific services or methods. Diagnostic Information: Check overall ingress error rate trends to confirm whether the increase is sustained or transient. Analyze service-specific metrics (e.g., ocpm_ingress_response_total{servicename_3gpp="rx",response_code!~"2.*"}) to identify which service is contributing to failures. Further filter errors by HTTP methods (GET, PUT, POST, DELETE, PATCH) to pinpoint problematic operations. Correlate with recent deployments, configuration changes, and dependency health. Review logs in Kibana for detailed error messages and failure patterns. Recovery: Fix underlying service or dependency issues causing failed transactions (e.g., configuration errors, backend failures). Roll back recent changes if the issue started after a deployment. Address API/method-specific issues and validate request/response handling. The alert clears automatically once failed transactions drop below 10% of total transactions.

8.1.1.22 DIAM_EGRESS_ERROR_RATE_ABOVE_1_PERCENT

Table 8-23 DIAM_EGRESS_ERROR_RATE_ABOVE_1_PERCENT

Field	Details
Description	Egress Transaction Error Rate detected above 1 Percent of Total on Diameter Connector (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 1 Percent of Total Transactions
Severity	Minor
Expression	sum(rate(ocpm_egress_response_total{servicename_3gpp=~"rx|sy",response_code!~"2.*"}[24h]))/sum(rate(ocpm_egress_response_total{servicename_3gpp=~"rx|sy"}[24h])) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.8
Metric Used	ocpm_egress_response_total
Recommended Actions	Cause: Indicates that failed egress transactions exceed 1% of total outgoing transactions. This condition indicates issues in downstream communication, such as peer unavailability, response errors, timeout scenarios, or request/response handling problems. Other causes include misconfigurations, increased load, retry storms, or intermittent connectivity issues with the downstream service. Diagnostic Information: Check egress error rate trends to determine whether the increase is sustained or transient. Analyze service-specific metrics (e.g., ocpm_egress_response_total{servicename_3gpp="rx",response_code!~"2.*"}) to identify which service is contributing to the failures. Further filter errors by HTTP methods (GET, PUT, POST, DELETE, PATCH) to isolate problematic operations. Correlate failures with recent deployments, configuration changes, or downstream service behavior. Review logs in Kibana to determine the exact reason for failures such as timeouts, connection issues, or invalid responses. Recovery: Fix the downstream service or connectivity issue causing the failed egress transactions. Roll back recent changes if the issue started after a deployment or configuration update. Adjust timeout, retry, or routing behavior if the failures are caused by transient downstream instability. If the issue persists, escalate to Oracle Support for further investigation.

8.1.1.23 UDR_INGRESS_TRAFFIC_RATE_ABOVE_THRESHOLD

Table 8-24 UDR_INGRESS_TRAFFIC_RATE_ABOVE_THRESHOLD

Field	Details
Description	User service Ingress traffic Rate from UDR is above threshold of Max MPS (current value is: {{ $value }})
Summary	Traffic Rate is above 90 Percent of Max requests per second
Severity	Major
Expression	sum(rate(occnp_userservice_inbound_count_total{service_resource="udr-service"}[2m])) >= 900
OID	1.3.6.1.4.1.323.5.3.52.1.2.9
Metric Used	ocpm_userservice_inbound_count_total{service_resource="udr-service"}
Recommended Actions	Cause: Indicates that the ingress traffic rate to the UDR service exceeds the configured threshold. This condition indicates a surge in incoming requests due to increased client activity, retry storms, or abnormal traffic patterns. Other causes include upstream service issues, misconfigurations, or spikes driven by specific workloads or events. Diagnostic Information: Check ingress traffic graphs for the UDR service to determine whether the increase is sustained or a transient spike. Refer to the Ingress Gateway section in Grafana to identify any corresponding rise in 4xx and 5xx response codes. Analyze Ingress Gateway logs in Kibana to understand the source of traffic and reasons for errors. Correlate traffic patterns with recent deployments, configuration changes, or upstream service behavior. Review threshold configuration in Common_Alertrules.yaml to ensure it aligns with expected traffic levels. Recovery: If traffic increase is expected, scale the UDR service or ingress capacity to handle the load. If errors (4xx/5xx) cause the condition, resolve underlying issues such as routing problems, backend failures, or misconfigurations. Mitigate abnormal traffic using rate limiting or by controlling retry behavior from clients. The alert clears automatically once ingress traffic rate falls below the configured threshold.

8.1.1.24 UDR_EGRESS_ERROR_RATE_ABOVE_10_PERCENT

Table 8-25 UDR_EGRESS_ERROR_RATE_ABOVE_10_PERCENT

Field	Details
Description	Egress Transaction Error Rate detected above 10 Percent of Total on User service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 10 Percent of Total Transactions
Severity	Critical
Expression	sum(rate(ocpm_udr_tracking_response_total{servicename_3gpp="nudr-dr",response_code!~"2.*"}[24h])) / sum(rate(ocpm_udr_tracking_response_total{servicename_3gpp="nudr-dr"}[24h])) * 100 >= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.10
Metric Used	ocpm_udr_tracking_response_total{servicename_3gpp="nudr-dr",response_code!~"2.*"}
Recommended Actions	Cause: Indicates that failed egress transactions to UDR exceed the configured threshold. The condition indicates a high error rate in outbound communication, typically caused by UDR unavailability, latency issues, misconfigurations, or request/response handling failures. It may also result from connectivity issues, retry storms, or increased load impacting communication with UDR. Diagnostic Information: Check egress error rate trends to determine whether the increase is sustained or transient. Refer to the Egress Gateway section in Grafana to identify any increase in 4xx and 5xx error response codes. Analyze Egress Gateway logs in Kibana to determine the reason for failures such as timeouts, connection issues, or invalid responses. Correlate failures with recent deployments, configuration changes, or UDR service behavior. Review threshold configuration in Common_Alertrules.yaml to ensure it aligns with expected traffic patterns. The alert clears automatically once failed transactions fall below the configured threshold. Recovery: Fix underlying issues with UDR such as availability, latency, or configuration problems. Resolve connectivity or network issues between services and UDR. Adjust retry, timeout, or routing configurations if failures are due to transient issues. Roll back recent changes if the issue started after a deployment. If the issue persists, escalate to Oracle Support for further investigation.

8.1.1.25 POLICYDS_INGRESS_TRAFFIC_RATE_ABOVE_THRESHOLD

Table 8-26 POLICYDS_INGRESS_TRAFFIC_RATE_ABOVE_THRESHOLD

Field	Details
Description	Ingress Traffic Rate is above threshold of Max MPS (current value is: {{ $value }})
Summary	Traffic Rate is above 90 Percent of Max requests per second
Severity	Critical
Expression	sum(rate(client_request_total[2m])) >= 900
OID	1.3.6.1.4.1.323.5.3.52.1.2.13
Metric Used	client_request_total Note: This is a Kubernetes metric used for instance availability monitoring. If the metric is not available, use similar metrics exposed by the monitoring system.
Recommended Actions	Cause: Indicates that the ingress traffic rate to the PolicyDS service exceeds the configured threshold. This condition indicates a surge in incoming requests due to increased client activity, retry storms, or abnormal traffic patterns. Other causes include upstream service issues, misconfigurations, or workload-driven spikes impacting PolicyDS. Diagnostic Information: Check ingress traffic graphs for the PolicyDS service to determine whether the increase is sustained or a transient spike. Refer to the Ingress Gateway section in Grafana to identify any rise in 4xx and 5xx response codes. Analyze Ingress Gateway logs in Kibana to determine the source of traffic and the reason for errors or abnormal request patterns. Correlate with recent deployments, configuration changes, or upstream service behavior impacting PolicyDS. Review threshold configuration in Common_Alertrules.yaml to ensure it aligns with expected traffic levels. Recovery: If the traffic increase is expected, scale the PolicyDS service or ingress capacity accordingly. If errors (4xx/5xx) cause the condition, resolve underlying issues such as routing problems, backend failures, or misconfigurations. Mitigate abnormal traffic by applying rate limiting or controlling retry behavior from clients. The alert clears automatically once ingress traffic rate falls below the configured threshold.

8.1.1.26 POLICYDS_INGRESS_ERROR_RATE_ABOVE_10_PERCENT

Table 8-27 POLICYDS_INGRESS_ERROR_RATE_ABOVE_10_PERCENT

Field	Details
Description	Ingress Transaction Error Rate detected above 10 Percent of Total on PolicyDS service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 10 Percent of Total Transactions
Severity	Critical
Expression	sum(rate(client_response_total{response!~"2.*"}[24h])) / sum(rate(client_response_total[24h])) * 100 >= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.14
Metric Used	client_response_total
Recommended Actions	Cause: Indicates that failed transactions for PolicyDS exceed 10% of total ingress transactions. The condition indicates a high error rate typically caused by backend service failures, misconfigurations, invalid requests, or dependency issues. It may also result from recent deployments, API contract mismatches, or traffic anomalies affecting specific operations in PolicyDS. Database-related issues such as increased latency, high concurrency, optimistic lock exceptions, or transaction retries can also contribute to higher error rates. Diagnostic Information: Check overall ingress error rate trends for PolicyDS to determine whether the increase is sustained or transient. Check database exception metrics such as db_exception_total to identify DB-related failures. Check client-side metrics to identify the workflow which is contributing to the failures (e.g., client_response_total{method="DELETE",workflow="SUBSCRIBER-DELETE",response="404 NOT_FOUND"}). Further filter errors by HTTP methods (GET, PUT, POST, DELETE, PATCH) to isolate problematic operations. Refer to Grafana dashboards for 4xx and 5xx trends and correlate with recent deployments or configuration changes. Review PolicyDS and related service logs like UDR connector, CHF connector, and SM Service logs to identify the actual cause of the failure. Recovery: Fix underlying service or dependency issues causing failed transactions (e.g., misconfigurations, backend failures). Investigate and resolve database-related issues such as latency, contention, or optimistic locking conflicts. If concurrency-related issues are observed, evaluate the need to enable Bulwark service for the specific call flow. Rollback recent deployments or changes if the issue started after a release. The alert clears automatically once failed transactions drop below 10% of total transactions.

8.1.1.27 POLICYDS_EGRESS_ERROR_RATE_ABOVE_1_PERCENT

Table 8-28 POLICYDS_EGRESS_ERROR_RATE_ABOVE_1_PERCENT

Field	Details
Description	Egress Transaction Error Rate detected above 1 Percent of Total on PolicyDS service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 1 Percent of Total Transactions
Severity	Minor
Expression	sum(rate(server_response_total{response!~"2.*"}[24h])) / sum(rate(server_response_total[24h])) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.15
Metric Used	server_response_total
Recommended Actions	Cause: Indicates that failed transactions for PolicyDS exceed 1% of total egress transactions. The condition indicates a high error rate typically caused by backend service failures, misconfigurations, invalid requests, or dependency issues. It may also result from recent deployments, API contract mismatches, or traffic anomalies affecting specific operations in PolicyDS. Diagnostic Information: Check overall egress error rate trends for PolicyDS to determine whether the increase is sustained or transient. Check client-side metrics to identify the workflow which is contributing to the failures (e.g., client_response_total{method="DELETE",workflow="SUBSCRIBER-DELETE",response="404 NOT_FOUND"}). Further filter errors by HTTP methods (GET, PUT, POST, DELETE, PATCH) to isolate problematic operations. Refer to Grafana dashboards for 4xx and 5xx trends and correlate with recent deployments or configuration changes. Review PolicyDS and related service logs like UDR connector, CHF connector, SM Service logs to identify the actual cause of the failure. Recovery: Fix underlying service or dependency issues causing failed transactions (e.g., misconfigurations, backend failures). Rollback recent deployments or changes if the issue started after a release. The alert clears automatically once failed transactions drop below 10% of total transactions.

8.1.1.28 UDR_INGRESS_TIMEOUT_ERROR_ABOVE_MAJOR_THRESHOLD

Table 8-29 UDR_INGRESS_TIMEOUT_ERROR_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	Ingress Timeout Error Rate detected above 10 Percent of Total towards UDR service (current value is: {{ $value }})
Summary	Timeout Error Rate detected above 10 Percent of Total Transactions
Severity	Major
Expression	sum(rate(ocpm_udr_tracking_request_timeout_total{servicename_3gpp="nudr-dr"}[24h])) / sum(rate(ocpm_udr_tracking_request_total{servicename_3gpp="nudr-dr"}[24h])) * 100 >=10
OID	1.3.6.1.4.1.323.5.3.52.1.2.16
Metric Used	ocpm_udr_tracking_request_timeout_total{servicename_3gpp="nudr-dr"}
Recommended Actions	Cause: Indicates that the number of ingress requests to UDR resulting in timeout errors exceeds the configured major threshold. This condition indicates delayed or no response from UDR due to high latency, resource saturation, or backend processing delays. Other causes include network issues, misconfigurations, increased load, or downstream dependency slowness impacting UDR response times. Diagnostic Information: Check timeout error trends to determine whether the increase is sustained or intermittent. Refer to ingress-related metrics and Grafana dashboards to identify spikes in latency and timeout responses. Analyze UDR service logs from Kibana to identify slow queries, processing delays, or request handling issues. Check for resource utilization (CPU, memory) on UDR pods to identify saturation. Correlate with recent deployments, configuration changes, or increased traffic patterns. Verify connectivity and response times between dependent services and UDR. Recovery: Investigate and resolve latency issues in UDR such as slow processing, backend delays, or resource constraints. Scale UDR service if the timeouts are due to increased load. Optimize queries or request handling to reduce processing time. Adjust timeout and retry configurations if required. Roll back recent changes if the issue started after a deployment. If the issue persists, escalate to Oracle Support for further investigation. The alert clears automatically once timeout errors fall below the configured major threshold.

8.1.1.29 DB_TIER_DOWN_ALERT

Table 8-30 DB_TIER_DOWN_ALERT

Field	Details
Description	DB cannot be reachable.
Summary	DB cannot be reachable.
Severity	Critical
Expression	appinfo_category_running{category="database"} != 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.18
Metric Used	appinfo_category_running{category="database"}
Recommended Actions	Cause: Database tier is unavailable or unreachable (appinfo_category_running{category="database"} != 1). Diagnostic Information: Verify database pod status, readiness, events, network connectivity, database logs, storage/PVC health, and the appinfo_category_running metric. Recovery: Restore the failed database component/connectivity; resolve storage or configuration issues; confirm the database category reports running again.

8.1.1.30 CPU_USAGE_PER_SERVICE_ABOVE_MINOR_THRESHOLD

Table 8-31 CPU_USAGE_PER_SERVICE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	CPU usage for {{$labels.namespace}}-{{$labels.service}} service is above 60%
Summary	CPU usage for {{$labels.namespace}}-{{$labels.service}} service is above 60%
Severity	Minor
Expression	avg by (namespace, service) (avg_over_time(cgroup_cpu_usage[2m])) >= 60 < 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.19
Metric Used	cgroup_cpu_usage Note: This is a Kubernetes used for instance availability monitoring. If the metric is not available, use similar metrics exposed by the monitoring system.
Recommended Actions	Cause: Indicates that CPU utilization for a service exceeds the configured minor threshold over a defined time window. This condition indicates increased processing load due to higher traffic, inefficient code paths, or resource-intensive operations. Other causes include traffic spikes, background jobs, tight loops, thread contention, or misconfigured CPU limits/requests. Diagnostic Information: Check CPU usage graphs for the service to determine whether utilization is consistently high or showing intermittent spikes. Correlate CPU usage with traffic patterns, batch jobs, or recent deployments. Check pod-level metrics for CPU throttling, restarts, or abnormal usage patterns. Check latency metrics to identify slow responses or performance degradation contributing to higher CPU usage. If there is a sudden surge in traffic leading to higher CPU usage, check logs of upstream services interacting with this service to identify increased egress traffic or retry behavior. Review service logs in Kibana to identify high-latency operations or repeated processing loops. Verify CPU limits and requests configuration and check for node-level CPU contention. Recovery: If CPU usage is consistently high, optimize application logic, reduce unnecessary processing, or tune configurations. If caused by traffic spikes, scale out the service to distribute load. Adjust CPU limits/requests cautiously if required, ensuring no impact on cluster stability. Roll back recent changes if the issue started after a deployment. The alert clears automatically once CPU usage falls below the configured threshold.

8.1.1.31 CPU_USAGE_PER_SERVICE_ABOVE_MAJOR_THRESHOLD

Table 8-32 CPU_USAGE_PER_SERVICE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	CPU usage for {{$labels.namespace}}-{{$labels.service}} service is above 80%
Summary	CPU usage for {{$labels.namespace}}-{{$labels.service}} service is above 80%
Severity	Major
Expression	avg by (namespace, service) (avg_over_time(cgroup_cpu_usage[2m])) >= 80 < 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.20
Metric Used	cgroup_cpu_usage Note: This is a Kubernetes used for instance availability monitoring. If the metric is not available, use similar metrics exposed by the monitoring system.
Recommended Actions	Cause: Indicates that CPU utilization for a service exceeds the configured minor threshold over a defined time window. This condition indicates increased processing load due to higher traffic, inefficient code paths, or resource-intensive operations. Other causes include traffic spikes, background jobs, tight loops, thread contention, or misconfigured CPU limits/requests. Diagnostic Information: Check CPU usage graphs for the service to determine whether utilization is consistently high or showing intermittent spikes. Correlate CPU usage with traffic patterns, batch jobs, or recent deployments. Check pod-level metrics for CPU throttling, restarts, or abnormal usage patterns. Check latency metrics to identify slow responses or performance degradation contributing to higher CPU usage. If there is a sudden surge in traffic leading to higher CPU usage, check logs of upstream services interacting with this service to identify increased egress traffic or retry behavior. Review service logs in Kibana to identify high-latency operations or repeated processing loops. Verify CPU limits and requests configuration and check for node-level CPU contention. Recovery: If CPU usage is consistently high, optimize application logic, reduce unnecessary processing, or tune configurations. If caused by traffic spikes, scale out the service to distribute load. Adjust CPU limits/requests cautiously if required, ensuring no impact on cluster stability. Roll back recent changes if the issue started after a deployment. The alert clears automatically once CPU usage falls below the configured threshold.

8.1.1.32 CPU_USAGE_PER_SERVICE_ABOVE_CRITICAL_THRESHOLD

Table 8-33 CPU_USAGE_PER_SERVICE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name in Alert Yaml File	CPU_USAGE_PER_SERVICE_ABOVE_CRITICAL_THRESHOLD
Description	CPU usage for {{$labels.namespace}}-{{$labels.service}} service is above 90%
Summary	CPU usage for {{$labels.namespace}}-{{$labels.service}} service is above 90%
Severity	Critical
Expression	avg by (namespace, service) (avg_over_time(cgroup_cpu_usage[2m])) >= 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.21
Metric Used	cgroup_cpu_usage Note: This is a Kubernetes used for instance availability monitoring. If the metric is not available, use similar metrics exposed by the monitoring system.
Recommended Actions	Cause: Indicates that CPU utilization for a service exceeds the configured critical threshold over a defined time window. The condition indicates severe resource saturation and risk of service degradation or unresponsiveness. It is typically caused by sustained high traffic, inefficient processing, tight loops, thread contention, or misconfigured CPU limits/requests. It may also result from retry storms, sudden workload spikes, or downstream latency causing increased processing overhead. Diagnostic Information: Check CPU usage graphs for the service to confirm sustained high utilization at critical levels. Correlate CPU usage with traffic patterns, batch jobs, or recent deployments. Check pod-level metrics for CPU throttling, restarts, and saturation indicators. Check latency metrics to identify performance degradation contributing to high CPU usage. If there is a sudden surge in traffic, check logs of upstream services interacting with this service to identify increased egress traffic or retry behavior. Review service logs in Kibana to identify high CPU-consuming operations, loops, or bottlenecks. Verify CPU limits and requests configuration and check for node-level CPU contention. Recovery: Immediately scale out the service to reduce CPU load per pod. If required, perform a rolling restart to stabilize pods under extreme load. Optimize application logic or disable non-critical processing to reduce CPU consumption. Mitigate retry storms or abnormal traffic by controlling client retry behavior. Adjust CPU limits/requests cautiously if needed to stabilize the service. Rollback recent deployments if the issue started after a change. The alert clears automatically once CPU usage falls below the configured critical threshold.

8.1.1.33 MEMORY_USAGE_PER_SERVICE_ABOVE_MINOR_THRESHOLD

Table 8-34 MEMORY_USAGE_PER_SERVICE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	Memory usage for {{$labels.service}} service is above 60%
Summary	Memory usage for {{$labels.service}} service is above 60%
Severity	Minor
Expression	avg by (namespace, service) (avg_over_time(cgroup_memory_usage[2m])) >= 60 < 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.22
Metric Used	container_memory_usage_bytes Note: This is a Kubernetes used for instance availability monitoring. If the metric is not available, use similar metrics exposed by the monitoring system.
Recommended Actions	Cause: Indicates that the service’s memory usage stays at or above the minor threshold for the configured window that is 60% This usually indicates sustained memory growth, repeated spikes from traffic or payload size, cache expansion, or a memory leak in the PRE service. It can also be triggered by recent deployment changes, higher concurrency, or a misconfigured memory limit. Diagnostic Information: Check memory graphs for the PRE service to confirm whether usage is constantly high or only spiking. Review whether the alert aligns with traffic bursts, deployment changes, or a specific job or request pattern. Recovery: Scale out or increase memory limits only after confirming the trend and ruling out a leak or abnormal workload

8.1.1.34 MEMORY_USAGE_PER_SERVICE_ABOVE_MAJOR_THRESHOLD

Table 8-35 MEMORY_USAGE_PER_SERVICE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	Memory usage for {{$labels.service}} service is above 80%
Summary	Memory usage for {{$labels.service}} service is above 80%
Severity	Major
Expression	avg by (namespace, service) (avg_over_time(cgroup_memory_usage[2m])) >= 80 < 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.23
Metric Used	container_memory_usage_bytes Note: This is a Kubernetes used for instance availability monitoring. If the metric is not available, use similar metrics exposed by the monitoring system.
Recommended Actions	Cause: Indicates that the service’s memory usage stays at or above the minor threshold for the configured window that is 80% This usually indicates sustained memory growth, repeated spikes from traffic or payload size, cache expansion, or a memory leak in the PRE service. It can also be triggered by recent deployment changes, higher concurrency, or a misconfigured memory limit. Diagnostic Information: Check memory graphs for the PRE service to confirm whether usage is constantly high or only spiking. Review whether the alert aligns with traffic bursts, deployment changes, or a specific job or request pattern. Recovery: Scale out or increase memory limits only after confirming the trend and ruling out a leak or abnormal workload

8.1.1.35 MEMORY_USAGE_PER_SERVICE_ABOVE_CRITICAL_THRESHOLD

Table 8-36 MEMORY_USAGE_PER_SERVICE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	Memory usage for {{$labels.service}} service is above 90%
Summary	Memory usage for {{$labels.service}} service is above 90%
Severity	Critical
Expression	avg by (namespace, service) (avg_over_time(cgroup_memory_usage[2m])) >= 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.24
Metric Used	container_memory_usage_bytes Note: This is a Kubernetes used for instance availability monitoring. If the metric is not available, use similar metrics exposed by the monitoring system.
Recommended Actions	Cause: Indicates that the service’s memory usage stays at or above the minor threshold for the configured window that is 90% This usually indicates sustained memory growth, repeated spikes from traffic or payload size, cache expansion, or a memory leak in the PRE service. It can also be triggered by recent deployment changes, higher concurrency, or a misconfigured memory limit. Diagnostic Information: Check memory graphs for the PRE service to confirm whether usage is constantly high or only spiking. Review whether the alert aligns with traffic bursts, deployment changes, or a specific job or request pattern. Recovery: If memory usage reaches ~90% of the defined threshold, perform a rolling restart of PRE pods to immediately relieve memory pressure. Investigate and address underlying causes such as leaks, cache growth, or abnormal traffic patterns. If the issue persists or root cause is unclear, reach out to Oracle Support for further analysis and guidance.

8.1.1.36 POD_CONGESTED

Table 8-37 POD_CONGESTED

Field	Details
Description	The pod congestion status is set to congested.
Summary	Pod Congestion status of {{$labels.service}} service is congested
Severity	Critical
Expression	occnp_pod_congestion_state == 4
OID	1.3.6.1.4.1.323.5.3.52.1.2.26
Metric Used	occnp_pod_congestion_state
Recommended Actions	Cause: This alert is caused when the pod has reached the CONGESTED state based on CPU consumption and/or the pending request queue exceeding the active threshold profile. Diagnostic Information: Check occnp_pod_congestion_state (expect 4), pod_resource_congestion_state for cpu/queue, pod_resource_stress, and occnp_http_congestion_message_reject_total (filter by congestionState, requestUri, requestMethod, priority). Recovery: In CNC Console (PCF → Overload and Congestion Control → Congestion Control), ensure the feature is enabled and the intended Thresholds and Load Shedding profiles are active. If rejections are excessive: Raise discard priority for the current state or relax thresholds based on performance baselines. Consider increasing stateChangeSampleCount and/or the calculation interval to reduce flapping due to short spikes.

8.1.1.37 POD_DANGER_OF_CONGESTION

Table 8-38 POD_DANGER_OF_CONGESTION

Field	Details
Description	The pod congestion status is set to Danger of Congestion.
Summary	Pod Congestion status of {{$labels.service}} service is DoC
Severity	Major
Expression	occnp_pod_congestion_state == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.25
Metric Used	occnp_pod_congestion_state
Recommended Actions	Cause: This alert is caused when the pod has reached the DANGER_OF_CONGESTION state based on CPU consumption and/or the pending request queue exceeding the active threshold profile. Diagnostic Information: Check occnp_pod_congestion_state (expect 1), pod_resource_congestion_state for cpu/queue, pod_resource_stress, and occnp_http_congestion_message_reject_total (filter by congestionState, requestUri, requestMethod, priority). Recovery: In CM GUI (PCF → Overload and Congestion Control → Congestion Control), ensure the feature is enabled and the intended Thresholds and Load Shedding profiles are active. If rejections are excessive: Lower the load-shedding aggressiveness for Major (if profiles are staged by state): Raise the discard priority threshold for Danger of Congention state (i.e., be less aggressive in Major). In Major, reject only lower-priority traffic, and allow more mid/high-priority requests through. Keep Critical/Congested configured to reject more aggressively than Major so escalation still protects the system. Also validate the rules are not unintentionally catching important traffic: Check mapping/classification so that critical call flows are assigned higher priority and are not being discarded in Major If frequent transitions between Normal and Major due to brief spikes: Increase stateChangeSampleCount (requires more consecutive samples before state changes) Increase the calculation interval (smooths short spikes) After changes: Monitor Major-state duration, rejection rate, latency, and resource utilization Confirm Major no longer causes excessive rejections, but still escalates to Critical when true overload occurs Apply changes incrementally and follow change control procedures

8.1.1.38 POD_PENDING_REQUEST_CONGESTED

Table 8-39 POD_PENDING_REQUEST_CONGESTED

Field	Details
Description	The pod congestion status is set to congested for PendingRequest.
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is congested for resource type
Severity	Critical
Expression	occnp_pod_resource_congestion_state{type="queue"} == 4
OID	1.3.6.1.4.1.323.5.3.52.1.2.28
Metric Used	occnp_pod_resource_congestion_state{type="queue"}
Recommended Actions	Cause: The pod has reached the CONGESTED state based on the pending request queue exceeding the active threshold profile. Diagnostic Information: Check occnp_pod_congestion_state (expect 4), pod_resource_congestion_state for type queue, pod_resource_stress, and occnp_http_congestion_message_reject_total (filter by congestionState, requestUri, requestMethod, priority). Recovery: In CNC Console (PCF → Overload and Congestion Control → Congestion Control), ensure the feature is enabled and the intended Thresholds and Load Shedding profiles are active. If rejections are excessive: Raise discard priority for the current state or relax pending request queue thresholds based on performance baselines. Consider increasing stateChangeSampleCount and/or the calculation interval to reduce flapping due to short spikes.

8.1.1.39 POD_PENDING_REQUEST_DANGER_OF_CONGESTION

Table 8-40 POD_PENDING_REQUEST_DANGER_OF_CONGESTION

Field	Details
Description	The pod congestion status is set to Danger of Congestion for pending requests.
Summary	Pod Resource Congestion status of {{$labels.service}} service is Danger of Congestion for PendingRequest type.
Severity	Major
Expression	occnp_pod_resource_congestion_state{type="queue"} == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.27
Metric Used	occnp_pod_resource_congestion_state{type="queue"}
Recommended Actions	Cause: This alert is caused when the pod has reached the DANGER_OF_CONGESTION state based on the pending request queue exceeding the active threshold profile. Diagnostic Information: Check occnp_pod_congestion_state (expect 1), pod_resource_congestion_state for queue, pod_resource_stress, and occnp_http_congestion_message_reject_total (filter by congestionState, requestUri, requestMethod, priority). Recovery: In CM GUI (PCF → Overload and Congestion Control → Congestion Control), ensure the feature is enabled and the intended Thresholds and Load Shedding profiles are active. If rejections are excessive: Lower the load-shedding aggressiveness for Major (if profiles are staged by state): Raise the discard priority threshold for Danger of Congention state (i.e., be less aggressive in Major). In Major, reject only lower-priority traffic, and allow more mid/high-priority requests through. Keep Critical/Congested configured to reject more aggressively than Major so escalation still protects the system. Also validate the rules are not unintentionally catching important traffic: Check mapping/classification so that critical call flows are assigned higher priority and are not being discarded in Major If frequent transitions between Normal and Major due to brief spikes: Increase stateChangeSampleCount (requires more consecutive samples before state changes) Increase the calculation interval (smooths short spikes) After changes: Monitor Major-state duration, rejection rate, latency, and resource utilization Confirm Major no longer causes excessive rejections, but still escalates to Critical when true overload occurs Apply changes incrementally and follow change control procedures

8.1.1.40 POD_CPU_CONGESTED

Table 8-41 POD_CPU_CONGESTED

Field	Details
Name in Alert Yaml File	PodCPUCongested
Description	The pod congestion status is set to congested for CPU.
Summary	Pod Resource Congestion status of {{$labels.service}} service is congested for CPU type.
Severity	Critical
Expression	occnp_pod_resource_congestion_state{type="cpu"} == 4
OID	1.3.6.1.4.1.323.5.3.52.1.2.30
Metric Used	occnp_pod_resource_congestion_state{type="cpu"}
Recommended Actions	Cause: The pod has reached the CONGESTED state based on the CPU Consumption exceeding the active threshold profile. Diagnostic Information: Check occnp_pod_congestion_state (expect 4), pod_resource_congestion_state for type cpu, pod_resource_stress, and occnp_http_congestion_message_reject_total (filter by congestionState, requestUri, requestMethod, priority). Recovery: In CM GUI (PCF → Overload and Congestion Control → Congestion Control), ensure the feature is enabled and the intended Thresholds and Load Shedding profiles are active. If rejections are excessive: Raise discard priority for the current state or relax cpu thresholds based on performance baselines. Consider increasing stateChangeSampleCount and/or the calculation interval to reduce flapping due to short spikes.

8.1.1.41 POD_CPU_DANGER_OF_CONGESTION

Table 8-42 POD_CPU_DANGER_OF_CONGESTION

Field	Details
Description	Pod Resource Congestion status of {{$labels.service}} service is DoC for CPU type.
Summary	Pod Resource Congestion status of {{$labels.service}} service is DoC for CPU type.
Severity	Major
Expression	occnp_pod_resource_congestion_state{type="cpu"} == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.29
Metric Used	occnp_pod_resource_congestion_state{type="cpu"}
Recommended Actions	Cause: This alert is caused when the pod has reached the DANGER_OF_CONGESTION state based on the CPU Utilization exceeding the active threshold profile. Diagnostic Information: Check occnp_pod_congestion_state (expect 1), pod_resource_congestion_state for queue, pod_resource_stress, and occnp_http_congestion_message_reject_total (filter by congestionState, requestUri, requestMethod, priority). Recovery: In CM GUI (PCF → Overload and Congestion Control → Congestion Control), ensure the feature is enabled and the intended Thresholds and Load Shedding profiles are active. If rejections are excessive: Lower the load-shedding aggressiveness for Major (if profiles are staged by state): Raise the discard priority threshold for Danger of Congention state (i.e., be less aggressive in Major). In Major, reject only lower-priority traffic, and allow more mid/high-priority requests through. Keep Critical/Congested configured to reject more aggressively than Major so escalation still protects the system. Also validate the rules are not unintentionally catching important traffic: Check mapping/classification so that critical call flows are assigned higher priority and are not being discarded in Major If frequent transitions between Normal and Major due to brief spikes: Increase stateChangeSampleCount (requires more consecutive samples before state changes) Increase the calculation interval (smooths short spikes) After changes: Monitor Major-state duration, rejection rate, latency, and resource utilization Confirm Major no longer causes excessive rejections, but still escalates to Critical when true overload occurs Apply changes incrementally and follow change control procedures

8.1.1.42 SERVICE_OVERLOADED

Table 8-43 SERVICE_OVERLOADED

Field	Details
Description	Overload Level of {{$labels.service}} service is L1
Summary	Overload Level of {{$labels.service}} service is L1
Severity	Minor
Expression	load_level == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.40
Metric Used	load_level
Recommended Actions	Cause: Possible causes include: CPU: brief CPU spike (burst traffic, short expensive operation) Pending requests: short-lived queue buildup due to bursty arrivals or slight latency increase Failures: small error spike (transient dependency wobble, brief network blip) Diagnostic Information: Identify the signal that triggered L1. CPU trend across pods (is it widespread or a few pods?) Gateway pending requests trend (spike vs sustained) Gateway failures (error codes/timeouts; any single dependency implicated?) Check the impact: p95/p99 latency, timeout rate, reject/drop counters Check stability: Determine whether load_level fluctuates between 0 and 1. Recovery: Stabilize the calculation to avoid flapping. Increase stateChangeSampleCount and/or the calculation interval Ensure hysteresis between Normal -> L1 (entry vs exit thresholds) Apply load shedding only when necessary. Priority-based: Configure L1 to discard only lowest-priority traffic. If L1 rejections are too high, make L1 less aggressive by raising the discard cutoff (so fewer requests are discarded). Percentage-based: Keep L1 reject % minimal; prefer smoothing/hysteresis tuning over increasing shedding. If CPU usage remains high at L1: Validate autoscaling (HPA) and check if CPU limits/requests are constraining the service.

Table 8-44 SERVICE_OVERLOADED

Field	Details
Description	Overload Level of {{$labels.service}} service is L2
Summary	Overload Level of {{$labels.service}} service is L2
Severity	Major
Expression	load_level == 2
OID	1.3.6.1.4.1.323.5.3.52.1.2.40
Metric Used	load_level
Recommended Actions	Cause: Possible causes include: CPU high + pending requests rising: service capacity issue (not enough replicas/CPU), or inefficient code path under current traffic mix. Pending requests high but CPU not that high: waiting on something (downstream latency), thread/connection pool bottleneck, or gateway/service concurrency limit. Failures rising (with or without pending): dependency degradation/outage, timeouts, bad config/release, or a retry storm. Diagnostic Information: CPU check Is CPU high across most pods? If yes, likely compute/capacity. Any recent deployment/config changes increasing CPU per request? Pending request check Is backlog steadily increasing? If yes, service throughput < arrival rate. Correlate pending with latency: rising pending usually pairs with rising p95/p99. Failure check What failure types dominate (timeouts vs immediate 5xx/4xx)? Are failures linked to a downstream dependency or a specific endpoint? Amplification check Evidence of retries (repeated attempts, rising incoming rate without real user growth). Recovery: Apply stronger overload protection to prevent L3 Priority-based: In L2, discard low priority (and if required, some medium priority) traffic to preserve core/high-priority flows. Ensure L3 remains stricter than L2. Percentage-based: Increase reject % to a moderate level, ideally targeting non-critical traffic classes/paths first. Address the dominant limiter If CPU-driven: scale out replicas (and/or increase CPU limits) within approved capacity. verify node pool can schedule them. If pending-driven but CPU not high: investigate downstream latency/connection pools; scaling may not help if you’re blocked on a dependency. If failure-driven: prioritize dependency recovery, rollback recent changes if indicated, and reduce retry amplification.

Table 8-45 SERVICE_OVERLOADED

Field	Details
Description	Overload Level of {{$labels.service}} service is L3
Summary	Overload Level of {{$labels.service}} service is L3
Severity	Critical
Expression	load_level == 3
OID	1.3.6.1.4.1.323.5.3.52.1.2.40
Metric Used	load_level
Recommended Actions	Cause: Possible causes include: CPU very high + large backlog: hard capacity ceiling reached (service cannot keep up). Backlog extreme + failures rising: downstream outage/major latency event causing timeouts, then retry storm. Failures very high (even if CPU isn’t maxed): widespread error condition (network/DNS/certs/routing, bad release, dependency down). Diagnostic Information: Confirm which of the three is worst right now: CPU saturation? Pending requests runaway queue? Failures/timeouts spiking? Check cluster/platform health (node readiness, network/DNS/CNI, ingress/LB). Confirm critical traffic classification (so core flows aren’t mistakenly treated as low priority). Recovery: Immediate containment (stop cascading failure) Priority-based: Apply the strictest L3 discard rules—reject low priority and, if needed, some medium priority traffic to keep the highest-priority flows alive. Percentage-based: Increase reject % aggressively (again, prefer non-critical traffic first if supported). Then restore stability If dependency-driven (failures/timeouts): fix/mitigate the dependency first; scaling the service may worsen downstream load. If compute-driven and dependencies are healthy: scale out rapidly (replicas/nodes) and consider temporarily reducing expensive traffic/features if operationally possible. Engage relevant platform/dependency owners—L3 is typically a cross-team incident.

8.1.1.43 SERVICE_RESOURCE_OVERLOADED

Alerts when service is in overload state due to memory usage

Table 8-46 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L1 for {{$labels.type}} type
Summary	{{$labels.service}} service is L1 for {{$labels.type}} type
Severity	Minor
Expression	service_resource_overload_level{type="memory"} == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="memory"}
Recommended Actions	Cause: Service is at overload level L1 specifically for memory. Drivers: memory leak, higher concurrency, cache growth, large payloads, misconfiguration of memory limits/heap, node memory pressure. Diagnostic Information: Memory usage trend per pod; OOMKilled/restarts. GC pressure (if applicable), allocation spikes. Check limits/requests and node memory pressure. Correlate with traffic and recent changes. Recovery: Reduce memory pressure (tune caches/payloads), fix leaks/rollback regressions. Scale out to reduce per-pod footprint; adjust memory limits cautiously. Clears when overload level drops to 0.

Table 8-47 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L2 for {{$labels.type}} type
Summary	{{$labels.service}} service is L2 for {{$labels.type}} type
Severity	Major
Expression	service_resource_overload_level{type="memory"} == 2
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="memory"}
Recommended Actions	Cause: Service is at overload level L2 specifically for memory. Drivers: memory leak, higher concurrency, cache growth, large payloads, misconfiguration of memory limits/heap, node memory pressure. Diagnostic Information: Memory usage trend per pod; OOMKilled/restarts. GC pressure (if applicable), allocation spikes. Check limits/requests and node memory pressure. Correlate with traffic and recent changes. Recovery: Reduce memory pressure (tune caches/payloads), fix leaks/rollback regressions. Scale out to reduce per-pod footprint; adjust memory limits cautiously. Clears when overload level drops below 2.

Table 8-48 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L3 for {{$labels.type}} type.
Summary	{{$labels.service}} service is L3 for {{$labels.type}} type
Severity	Critical
Expression	service_resource_overload_level{type="memory"} == 3
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="memory"}
Recommended Actions	Cause: Service is at overload level L3 specifically for memory. Drivers: memory leak, higher concurrency, cache growth, large payloads, misconfiguration of memory limits/heap, node memory pressure. Diagnostic Information: Memory usage trend per pod; OOMKilled/restarts. GC pressure (if applicable), allocation spikes. Check limits/requests and node memory pressure. Correlate with traffic and recent changes. Recovery: Reduce memory pressure (tune caches/payloads), fix leaks/rollback regressions. Scale out to reduce per-pod footprint; adjust memory limits cautiously. Clears when overload level drops below 3.

Alerts when service is in overload state due to CPU usage

Table 8-49 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L1 for {{$labels.type}} type
Summary	{{$labels.service}} service is L1 for {{$labels.type}} type
Severity	Minor
Expression	service_resource_overload_level{type="cpu"} == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="cpu"}
Recommended Actions	Cause: This alert fires when the metric service_resource_overload_level indicates that a given service is experiencing Level 1 (L1) resource overload for the specified resource type. For this rule, the resource type is CPU, so it indicates CPU pressure at L1 for {{$labels.service}}. Common causes of CPU overload include: Traffic increase causing higher request processing load Background workload spikes (GC pressure, housekeeping tasks, batch jobs) Insufficient CPU limits/requests or reduced node capacity Node contention if running in a shared cluster Recent deployment/config change that increased CPU usage L1 (MINOR) is an early warning. If overload persists or worsens (e.g., L2/L3), treat as potentially service impacting. Diagnostic Information: Identify the impacted service/pod(s). Correlate with pod replica count and pod distribution across nodes recent deploys/config changes for that service Validate CPU usage per pod / container for {{$labels.service}} Check traffic and latency/error correlation determine if CPU overload correlates with: Increased ingress traffic rate p95/p99 latency increases increased increased error rates / timeouts Review logs for overload indicator, such as, overload control activation and request rejections Recovery: Recovery Reduce load (short term): Divert traffic or reduce non-critical workloads impacting the service If a retry storm is suspected, reduce retries/timeouts upstream Scale the service: Increase replicas (horizontal scaling) to distribute CPU load Increase CPU requests/limits if the service is CPU-starved or throttled Rollback recent changes if CPU increase started after a deployment Stabilize dependencies: If CPU is driven by retries or backpressure from downstream slowness, remediate the downstream dependency first (latency/outage can amplify CPU load). Alert clearance: The alert clears automatically once service_resource_overload_level{type="cpu"} returns to 0 (or below L1) for the service.

Table 8-50 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L2 for {{$labels.type}} type
Summary	{{$labels.service}} service is L2 for {{$labels.type}} type
Severity	Major
Expression	The overload level of the service is L2 due to CPU usage.
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="cpu"} == 2
Recommended Actions	Cause: This alert fires when the metric service_resource_overload_level indicates that a given service is experiencing Level 2 (L2) resource overload for the specified resource type. For this rule, the resource type is CPU, so it indicates CPU pressure at L2 for {{$labels.service}}. Common causes of CPU overload include: Traffic increase causing higher request processing load Background workload spikes (GC pressure, housekeeping tasks, batch jobs) Insufficient CPU limits/requests or reduced node capacity Node contention if running in a shared cluster Recent deployment/config change that increased CPU usage L2 (MAJOR) should generally be treated as service-impacting or at high risk of impact; Diagnostic Information: Identify the impacted service/pod(s). Correlate with pod replica count and pod distribution across nodes recent deploys/config changes for that service Validate CPU usage per pod / container for {{$labels.service}} Check traffic and latency/error correlation determine if CPU overload correlates with: Increased ingress traffic rate p95/p99 latency increases increased increased error rates / timeouts Review logs for overload indicator, such as, overload control activation and request rejections Recovery: Recovery Reduce load (short term): Divert traffic or reduce non-critical workloads impacting the service If a retry storm is suspected, reduce retries/timeouts upstream Scale the service: Increase replicas (horizontal scaling) to distribute CPU load Increase CPU requests/limits if the service is CPU-starved or throttled Rollback recent changes if CPU increase started after a deployment Stabilize dependencies: If CPU is driven by retries or backpressure from downstream slowness, remediate the downstream dependency first (latency/outage can amplify CPU load). Alert clearance: The alert clears automatically once service_resource_overload_level{type="cpu"} returns to 0 (or below L2) for the service.

Table 8-51 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L3 for {{$labels.type}} type
Summary	{{$labels.service}} service is L3 for {{$labels.type}} type
Severity	Critical
Expression	service_resource_overload_level{type="cpu"} == 3
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="cpu"}
Recommended Actions	Cause: This alert fires when the metric service_resource_overload_level indicates that a given service is experiencing Level 3 (L3) resource overload for the specified resource type. For this rule, the resource type is CPU, so it indicates CPU pressure at L3 for {{$labels.service}}. Common causes of CPU overload include: Traffic increase causing higher request processing load Background workload spikes (GC pressure, housekeeping tasks, batch jobs) Insufficient CPU limits/requests or reduced node capacity Node contention if running in a shared cluster Recent deployment/config change that increased CPU usage L3 (CRITICAL) often precedes cascading failures; prioritize load shedding + scaling before deep debugging. Diagnostic Information: Identify the impacted service/pod(s). Correlate with pod replica count and pod distribution across nodes recent deploys/config changes for that service Validate CPU usage per pod / container for {{$labels.service}} Check traffic and latency/error correlation determine if CPU overload correlates with: Increased ingress traffic rate p95/p99 latency increases increased increased error rates / timeouts Review logs for overload indicator, such as, overload control activation and request rejections Recovery: Recovery Reduce load (short term): Divert traffic or reduce non-critical workloads impacting the service If a retry storm is suspected, reduce retries/timeouts upstream Scale the service: Increase replicas (horizontal scaling) to distribute CPU load Increase CPU requests/limits if the service is CPU-starved or throttled Rollback recent changes if CPU increase started after a deployment Stabilize dependencies: If CPU is driven by retries or backpressure from downstream slowness, remediate the downstream dependency first (latency/outage can amplify CPU load). Alert clearance: The alert clears automatically once service_resource_overload_level{type="cpu"} returns to 0 (or below L3) for the service.

Alerts when service is in overload state due to number of pending messages

Table 8-52 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L1 for {{$labels.type}} type
Summary	{{$labels.service}} service is L1 for {{$labels.type}} type
Severity	Minor
Expression	service_resource_overload_level{type="svc_pending_count"} == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="svc_pending_count"}
Recommended Actions	Cause: svc_pending_count is the number of pending/in-flight work items queued for processing by {{$labels.service}}. L1 indicates an early backlog build-up. Common contributors: Short traffic spike / bursty arrivals Sustained traffic above capacity Temporary downstream slowdown/backpressure/outage Resource constraints (CPU starvation/throttling, insufficient replicas, node pressure) Reduced processing capacity (pods not ready, fewer replicas, node contention) Increased per-request work after a deploy/config change Database slowness (query latency, lock waits, pool pressure) blocking workers High istio-proxy CPU and/or throttling increasing proxy/transport overhead and slowing completion Diagnostic Information: Confirm backlog behavior: confirm pending count is high and not draining. Check impact signals: p95/p99 latency, timeout rate, error rate, retry rate. Correlate with load: compare ingress/request rate vs completion/throughput. Pod/service health: restarts, readiness failures, uneven load distribution; check recent deploy/config changes. MySQL checks: query latency/timeouts, connection count/pool usage, lock waits/deadlocks, slow query log/top slow queries. Istio-proxy checks: istio-proxy CPU usage and CPU throttling; check whether spikes correlate with queue growth. If available, review Envoy indicators such as upstream timeouts/resets/retries. Logs: queue buildup warnings, slow-processing messages, DB timeout/retry errors, Envoy upstream timeout/reset patterns. Recent changes: deployments/config changes in service, Istio policy/telemetry changes, DB parameter changes. Recovery: If transient, monitor; backlog should drain as dependencies/CPU normalize. If backlog persists: Remediate MySQL performance (address slow queries/indexing, reduce contention, ensure DB capacity, tune pools/timeouts). Reduce istio-proxy overhead (right-size CPU requests/limits, tune telemetry/log volume, verify retry policies aren’t amplifying load). Scale out {{$labels.service}} (add replicas) to increase throughput (ensure DB can handle added concurrency). Apply load shedding if supported to prevent queue growth. Alert clears when overload level drops below 1.

Table 8-53 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L2 for {{$labels.type}} type
Summary	{{$labels.service}} service is L2 for {{$labels.type}} type
Severity	Major
Expression	service_resource_overload_level{type="svc_pending_count"} == 2
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="svc_pending_count"}
Recommended Actions	Cause: svc_pending_count is the number of pending/in-flight work items queued for processing by {{$labels.service}}. L2 indicates an early backlog build-up. Common contributors: Short traffic spike / bursty arrivals Sustained traffic above capacity Temporary downstream slowdown/backpressure/outage Resource constraints (CPU starvation/throttling, insufficient replicas, node pressure) Reduced processing capacity (pods not ready, fewer replicas, node contention) Increased per-request work after a deploy/config change Database slowness (query latency, lock waits, pool pressure) blocking workers High istio-proxy CPU and/or throttling increasing proxy/transport overhead and slowing completion Diagnostic Information: Confirm backlog behavior: confirm pending count is high and not draining. Check impact signals: p95/p99 latency, timeout rate, error rate, retry rate. Correlate with load: compare ingress/request rate vs completion/throughput. Pod/service health: restarts, readiness failures, uneven load distribution; check recent deploy/config changes. MySQL checks: query latency/timeouts, connection count/pool usage, lock waits/deadlocks, slow query log/top slow queries. Istio-proxy checks: istio-proxy CPU usage and CPU throttling; check whether spikes correlate with queue growth. If available, review Envoy indicators such as upstream timeouts/resets/retries. Logs: queue buildup warnings, slow-processing messages, DB timeout/retry errors, Envoy upstream timeout/reset patterns. Recent changes: deployments/config changes in service, Istio policy/telemetry changes, DB parameter changes. Recovery: If transient, monitor; backlog should drain as dependencies/CPU normalize. If backlog persists: Remediate MySQL performance (address slow queries/indexing, reduce contention, ensure DB capacity, tune pools/timeouts). Reduce istio-proxy overhead (right-size CPU requests/limits, tune telemetry/log volume, verify retry policies aren’t amplifying load). Scale out {{$labels.service}} (add replicas) to increase throughput (ensure DB can handle added concurrency). Apply load shedding if supported to prevent queue growth. Alert clears when overload level drops to 0 or below 1.

Table 8-54 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L3 for {{$labels.type}} type
Summary	{{$labels.service}} service is L3 for {{$labels.type}} type
Severity	Critical
Expression	service_resource_overload_level{type="svc_pending_count"} == 3
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="svc_pending_count"}
Recommended Actions	Cause: svc_pending_count is the number of pending/in-flight work items queued for processing by {{$labels.service}}. L3 indicates an early backlog build-up. Common contributors: Short traffic spike / bursty arrivals Sustained traffic above capacity Temporary downstream slowdown/backpressure/outage Resource constraints (CPU starvation/throttling, insufficient replicas, node pressure) Reduced processing capacity (pods not ready, fewer replicas, node contention) Increased per-request work after a deploy/config change Database slowness (query latency, lock waits, pool pressure) blocking workers High istio-proxy CPU and/or throttling increasing proxy/transport overhead and slowing completion Diagnostic Information: Confirm backlog behavior: confirm pending count is high and not draining. Check impact signals: p95/p99 latency, timeout rate, error rate, retry rate. Correlate with load: compare ingress/request rate vs completion/throughput. Pod/service health: restarts, readiness failures, uneven load distribution; check recent deploy/config changes. MySQL checks: query latency/timeouts, connection count/pool usage, lock waits/deadlocks, slow query log/top slow queries. Istio-proxy checks: istio-proxy CPU usage and CPU throttling; check whether spikes correlate with queue growth. If available, review Envoy indicators such as upstream timeouts/resets/retries. Logs: queue buildup warnings, slow-processing messages, DB timeout/retry errors, Envoy upstream timeout/reset patterns. Recent changes: deployments/config changes in service, Istio policy/telemetry changes, DB parameter changes. Recovery: If transient, monitor; backlog should drain as dependencies/CPU normalize. If backlog persists: Remediate MySQL performance (address slow queries/indexing, reduce contention, ensure DB capacity, tune pools/timeouts). Reduce istio-proxy overhead (right-size CPU requests/limits, tune telemetry/log volume, verify retry policies aren’t amplifying load). Scale out {{$labels.service}} (add replicas) to increase throughput (ensure DB can handle added concurrency). Apply load shedding if supported to prevent queue growth. Alert clears when overload level drops to 0 or below 3.

Alerts when service is in overload state due to number of failed requests

Table 8-55 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L1 for {{$labels.type}} type.
Summary	{{$labels.service}} service is L1 for {{$labels.type}} type.
Severity	Minor
Expression	service_resource_overload_level{type="svc_failure_count"} == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="svc_failure_count"}
Recommended Actions	Cause: svc_failure_count indicates elevated service-side failures (internal failures while processing work) reaching overload level L1. Common drivers: Dependency failures (DB down/slow, peer NF failures) Timeouts leading to failures and retries Resource exhaustion (CPU/memory/queue) causing rejections Misconfiguration or regression after deploy Network/service mesh issues (upstream resets/timeouts) Diagnostic Information: Identify which failure types dominate (error codes, exception classes, timeout vs non-timeout). Correlate failures with dependency health (DB latency/errors, connection pool, peer availability). Check resource saturation signals (CPU, memory, pending queue). Review service logs for repeated stack traces, “too busy/queue full”, connection failures. Check recent deployments/config changes. Recovery: Restore dependencies and reduce timeouts/retries amplification. Scale out service and/or adjust resource limits if saturation is contributing. Fix misconfig/regression; rollback if needed. Clears when overload level drops below the level.

Table 8-56 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L2 for {{$labels.type}} type.
Summary	{{$labels.service}} service is L2 for {{$labels.type}} type.
Severity	Major
Expression	service_resource_overload_level{type="svc_failure_count"} == 2
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="svc_failure_count"}
Recommended Actions	Cause: svc_failure_count indicates elevated service-side failures (internal failures while processing work) reaching overload level L2. Common drivers: Dependency failures (DB down/slow, peer NF failures) Timeouts leading to failures and retries Resource exhaustion (CPU/memory/queue) causing rejections Misconfiguration or regression after deploy Network/service mesh issues (upstream resets/timeouts) Diagnostic Information: Identify which failure types dominate (error codes, exception classes, timeout vs non-timeout). Correlate failures with dependency health (DB latency/errors, connection pool, peer availability). Check resource saturation signals (CPU, memory, pending queue). Review service logs for repeated stack traces, “too busy/queue full”, connection failures. Check recent deployments/config changes. Recovery: Restore dependencies and reduce timeouts/retries amplification. Scale out service and/or adjust resource limits if saturation is contributing. Fix misconfig/regression; rollback if needed. Clears when overload level drops below the level.

Table 8-57 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.service}} service is L3 for {{$labels.type}} type.
Summary	{{$labels.service}} service is L3 for {{$labels.type}} type.
Severity	Critical
Expression	service_resource_overload_level{type="svc_failure_count"} == 3
OID	1.3.6.1.4.1.323.5.3.52.1.2.41
Metric Used	service_resource_overload_level{type="svc_failure_count"}
Recommended Actions	Cause: svc_failure_count indicates elevated service-side failures (internal failures while processing work) reaching overload level L3. Common drivers: Dependency failures (DB down/slow, peer NF failures) Timeouts leading to failures and retries Resource exhaustion (CPU/memory/queue) causing rejections Misconfiguration or regression after deploy Network/service mesh issues (upstream resets/timeouts) Diagnostic Information: Identify which failure types dominate (error codes, exception classes, timeout vs non-timeout). Correlate failures with dependency health (DB latency/errors, connection pool, peer availability). Check resource saturation signals (CPU, memory, pending queue). Review service logs for repeated stack traces, “too busy/queue full”, connection failures. Check recent deployments/config changes. Recovery: Restore dependencies and reduce timeouts/retries amplification. Scale out service and/or adjust resource limits if saturation is contributing. Fix misconfig/regression; rollback if needed. L3 should be treated as incident-level if sustained. Clears when overload level drops below the level.

8.1.1.44 SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_CRITICAL_THRESHOLD

Table 8-58 SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	Notification Transaction Error exceeds the critical threshold limit for a given Subscriber Notification server.
Summary	Transaction Error exceeds the critical threshold limit for a given Subscriber Notification server.
Severity	Critical
Expression	sum by(notificationEndpoint)(increase(http_notification_response_total{responseCode!~"2.*"}[2m])) > 1000
OID	1.3.6.1.4.1.323.5.3.44.1.2.42
Metric Used	http_notification_response_total{responseCode!~"2.*"}
Recommended Actions	Cause: SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_CRITICAL_THRESHOLD indicates a critical increase in notification transaction failures for a given notificationEndpoint. The alert fires when non-2xx HTTP responses exceed 1000 within 2 minutes (per endpoint), suggesting the Subscriber Notification server is failing to deliver notifications successfully. Diagnostic Information: Identify which response codes dominate (e.g., 4xx vs 5xx vs 429) for the affected notificationEndpoint. Confirm whether the issue is isolated to one endpoint or widespread across endpoints (compare top failing endpoints). Correlate with latency and timeout signals (HTTP client timeouts, upstream duration histograms if available). Check Subscriber Notification server logs for: repeated errors, payload validation failures, auth errors, TLS handshake failures, connection timeouts/resets. Validate downstream endpoint health: DNS resolution, cert validity, firewall rules, rate limiting (429), maintenance windows. Check internal dependencies used for notification delivery (DB/queue/service mesh): errors, saturation, connection pool exhaustion. Review recent deployments/config changes affecting: endpoint routing, auth credentials, payload format, retry policy, timeout policy. Recovery: If failures are 4xx, fix payload/auth/config issues (credentials, headers, schema) and reprocess if applicable. If failures are 5xx/timeouts, stabilize dependencies and reduce retry amplification (add backoff/jitter, cap concurrency). If downstream endpoint is unhealthy or rate-limiting (429), coordinate with endpoint owner, apply throttling, and/or pause/retry with backoff. If resource saturation is contributing, scale out the service and/or adjust resource limits/queues. Rollback recent changes if a regression is suspected. Treat sustained CRITICAL as incident-level, especially if impacting multiple endpoints or causing backlog growth. Clears when non-2xx responses for each notificationEndpoint drop below 1000 over 2 minutes.

Table 8-59 SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	Notification Transaction Error exceeds the major threshold limit for a given Subscriber Notification server
Summary	Transaction Error exceeds the major threshold limit for a given Subscriber Notification server
Severity	Major
Expression	sum by(notificationEndpoint)(increase(http_notification_response_total{responseCode!~"2.*"}[2m])) > 750 and sum by(notificationEndpoint)(increase(http_notification_response_total{responseCode!~"2.*"}[2m])) <= 1000
OID	1.3.6.1.4.1.323.5.3.44.1.2.42
Metric Used	http_notification_response_total{responseCode!~"2.*"}
Recommended Actions	Cause: SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_MAJOR_THRESHOLD indicates a critical increase in notification transaction failures for a given notificationEndpoint. The alert fires when non-2xx HTTP responses exceed 750 but less than1000 within 2 minutes (per endpoint), suggesting the Subscriber Notification server is failing to deliver notifications successfully. Diagnostic Information: Server failures (5xx due to internal errors, dependency failures, timeouts). Network/service mesh issues (timeouts, resets, MTLS/cert problems, proxy errors). Retry storms/backpressure (retries amplifying failures; queue buildup increasing error volume). Deployment/config regression (route changes, endpoint mapping changes, timeout changes). Identify which response codes dominate (e.g., 4xx vs 5xx vs 429) for the affected notificationEndpoint. Confirm whether the issue is isolated to one endpoint or widespread across endpoints (compare top failing endpoints). Correlate with latency and timeout signals (HTTP client timeouts, upstream duration histograms if available). Check Subscriber Notification server logs for: repeated errors, payload validation failures, auth errors, TLS handshake failures, connection timeouts/resets. Validate downstream endpoint health: DNS resolution, cert validity, firewall rules, rate limiting (429), maintenance windows. Check internal dependencies used for notification delivery (DB/queue/service mesh): errors, saturation, connection pool exhaustion. Review recent deployments/config changes affecting: endpoint routing, auth credentials, payload format, retry policy, timeout policy. Recovery: If failures are 4xx, fix payload/auth/config issues (credentials, headers, schema) and reprocess if applicable. If failures are 5xx/timeouts, stabilize dependencies and reduce retry amplification (add backoff/jitter, cap concurrency). If downstream endpoint is unhealthy or rate-limiting (429), coordinate with endpoint owner, apply throttling, and/or pause/retry with backoff. If resource saturation is contributing, scale out the service and/or adjust resource limits/queues. Rollback recent changes if a regression is suspected. Treat sustained CRITICAL as incident-level, especially if impacting multiple endpoints or causing backlog growth. Clears when non-2xx responses for each notificationEndpoint drop below 1000 over 2 minutes.

Table 8-60 SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	Notification Transaction Error exceeds the minor threshold limit for a given Subscriber Notification server
Summary	Transaction Error exceeds the minor threshold limit for a given Subscriber Notification server
Severity	Minor
Expression	sum by(notificationEndpoint)(increase(http_notification_response_total{responseCode!~"2.*"}[2m])) > 500 and sum by(notificationEndpoint)(increase(http_notification_response_total{responseCode!~"2.*"}[2m])) <= 750
OID	1.3.6.1.4.1.323.5.3.44.1.2.42
Metric Used	http_notification_response_total{responseCode!~"2.*"}
Recommended Actions	Cause: SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_MINOR_THRESHOLD indicates a critical increase in notification transaction failures for a given notificationEndpoint. The alert fires when non-2xx HTTP responses exceed 500 but remains lesser than 750 within 2 minutes (per endpoint), suggesting the Subscriber Notification server is failing to deliver notifications successfully. Diagnostic Information: Server failures (5xx due to internal errors, dependency failures, timeouts). Network/service mesh issues (timeouts, resets, MTLS/cert problems, proxy errors). Retry storms/backpressure (retries amplifying failures; queue buildup increasing error volume). Deployment/config regression (route changes, endpoint mapping changes, timeout changes). Identify which response codes dominate (e.g., 4xx vs 5xx vs 429) for the affected notificationEndpoint. Confirm whether the issue is isolated to one endpoint or widespread across endpoints (compare top failing endpoints). Correlate with latency and timeout signals (HTTP client timeouts, upstream duration histograms if available). Check Subscriber Notification server logs for: repeated errors, payload validation failures, auth errors, TLS handshake failures, connection timeouts/resets. Validate downstream endpoint health: DNS resolution, cert validity, firewall rules, rate limiting (429), maintenance windows. Check internal dependencies used for notification delivery (DB/queue/service mesh): errors, saturation, connection pool exhaustion. Review recent deployments/config changes affecting: endpoint routing, auth credentials, payload format, retry policy, timeout policy. Recovery: If failures are 4xx, fix payload/auth/config issues (credentials, headers, schema) and reprocess if applicable. If failures are 5xx/timeouts, stabilize dependencies and reduce retry amplification (add backoff/jitter, cap concurrency). If downstream endpoint is unhealthy or rate-limiting (429), coordinate with endpoint owner, apply throttling, and/or pause/retry with backoff. If resource saturation is contributing, scale out the service and/or adjust resource limits/queues. Rollback recent changes if a regression is suspected. Treat sustained CRITICAL as incident-level, especially if impacting multiple endpoints or causing backlog growth. Clears when non-2xx responses for each notificationEndpoint drop below 1000 over 2 minutes.

8.1.1.45 SYSTEM_IMPAIRMENT_MAJOR

Table 8-61 SYSTEM_IMPAIRMENT_MAJOR

Field	Details
Description	Major Impairment alert raised for REPLICATION_FAILED or REPLICATION_CHANNEL_DOWN or BINLOG_STORAGE usage > 80%
Summary	Major impairment alert raised for REPLICATION_FAILED or REPLICATION_CHANNEL_DOWN or BINLOG_STORAGE usage must be more than 80% for 10 minutes.
Severity	Major
Expression	(db_tier_replication_status{role="failed"} == 0) or (db_tier_replication_status{role="active"} == 0) or (count by (site_name) (db_tier_replication_status) == count by (site_name) (db_tier_replication_status{role="standby"})) or (count by (site_name) (db_tier_replication_status) == count by (site_name) (db_tier_replication_status{role="failed"})) or (avg_over_time(db_tier_binlog_used_bytes_percentage[5m])>= 80)
OID	1.3.6.1.4.1.323.5.3.52.1.2.43
Metric Used	db_tier_replication_status and db_tier_binlog_used_bytes_percentage
Recommended Actions	Cause: SYSTEM_IMPAIRMENT_MAJOR indicates a major site impairment condition related to the DB tier, typically associated with replication health or binlog storage pressure. This alert is intended to fire when one or more of the following major-risk conditions occur (often mapped to REPLICATION_FAILED, REPLICATION_CHANNEL_DOWN, or BINLOG_STORAGE high usage): Diagnostic Information: Replication failure (replication role reported as failed, or replication threads stopped) Replication channel down / missing standby (expected replication roles are not present for the site, e.g., standby not reporting) Primary/active not reporting (active role missing/unhealthy from metrics perspective) Binlog storage growth due to replication lag, blocked purge, or high write rate, resulting in binlog usage ≥ 80% (as measured by db_tier_binlog_used_bytes_percentage) Network issues between DB nodes/sites (timeouts, resets, packet loss) DB load/resource saturation causing lag (CPU, IO, disk, connection pool) Misconfiguration/regression after deploy/patch (replication settings, credentials, TLS/certs) Determine which condition triggered the alert Check current values and recent history for: db_tier_replication_status{role="active"}, {role="standby"}, {role="failed"} db_tier_binlog_used_bytes_percentage Confirm whether the trigger is replication-related vs binlog-usage-related. Replication health deep-dive Identify impacted site_name and which node(s)/roles are missing or in failed. Check replication lag, SQL/IO thread state, last error, and channel status on the DB nodes. Channel/network validation Check connectivity between active and standby. Look for recent network maintenance, route changes, MTU issues, or service mesh/proxy impacts (if in path). Binlog storage analysis (if binlog usage is high) Confirm binlog filesystem utilization and growth rate; determine if purge is blocked. Correlate with replication lag/backlog (binlogs often accumulate when standby can’t consume). Check for unusually high write volume or long-running transactions. Resource saturation / DB health Check disk IOPS/latency, CPU, memory pressure, and connection pool saturation. Review DB logs for replication errors, disk-full warnings, corruption, or restart events. Change correlation Check recent deploys/patches/config changes (DB parameters, replication config, storage changes). Recovery: If replication failed / channel down: Restore replication connectivity (network path, DNS, firewall, TLS/certs). Restart/repair replication channel as per DB runbook; resolve the root error (permissions, missing binlogs, etc.). If standby is unhealthy, recover/rebuild standby from a known-good snapshot/backup and rejoin replication. Consider temporarily reducing write load or enabling throttling to allow the standby to catch up. If binlog usage ≥ 80%: Treat as urgent to avoid running out of storage (risk of DB outage). Reduce binlog growth by addressing replication lag (fix channel, improve IO/CPU, reduce write amplification). Expand binlog storage if needed (short-term mitigation) following capacity change procedures. Ensure binlog purge/retention settings are correct and purge is not blocked (e.g., by lagging replicas). Operational guidance: Escalate as major if sustained, if multiple sites are impacted, or if lag/space continues to worsen. After remediation, verify: replication roles return to expected states (active/standby healthy; no failed) replication lag stabilizes db_tier_binlog_used_bytes_percentage drops below threshold and continues trending down

8.1.1.46 SYSTEM_IMPAIRMENT_CRITICAL

Table 8-62 SYSTEM_IMPAIRMENT_CRITICAL

Field	Details
Description	Critical Impairment alert raised for REPLICATION_FAILED or REPLICATION_CHANNEL_DOWN or BINLOG_STORAGE usage > 80%
Summary	Critical impairment alert raised for REPLICATION_FAILED or REPLICATION_CHANNEL_DOWN or BINLOG_STORAGE usage must be more than 80% for 30 minutes.
Severity	Critical
Expression	(db_tier_replication_status{role="failed"} == 0) or (db_tier_replication_status{role="active"} == 0) or (count by (site_name) (db_tier_replication_status) == count by (site_name) (db_tier_replication_status{role="standby"})) or (count by (site_name) (db_tier_replication_status) == count by (site_name) (db_tier_replication_status{role="failed"})) or (avg_over_time(db_tier_binlog_used_bytes_percentage[5m])>= 80)
OID	1.3.6.1.4.1.323.5.3.52.1.2.43
Metric Used	db_tier_replication_status and db_tier_binlog_used_bytes_percentage
Recommended Actions	Cause: SYSTEM_IMPAIRMENT_CRITICAL indicates a critical site impairment related to DB-tier replication health and/or binlog storage pressure. It is the higher-severity escalation of the impairment condition, intended to represent that one of the following has become sustained/critical (per description: replication failed, replication channel down, or binlog storage usage ≥ 80% for ~30 minutes). Common drivers: Sustained replication failure (replication role reported as failed, replication threads stopped, unrecoverable SQL error) Replication channel down for an extended period (standby missing/unreachable, broken link, auth/TLS failure) Loss of expected topology (active/standby role metrics missing or inconsistent for a site) Sustained binlog storage pressure (≥ 80%), typically due to replication lag/backlog, blocked purge, or abnormal write volume Network instability between sites/DB nodes causing persistent disconnects and lag Resource exhaustion on DB nodes (disk IO saturation, CPU, memory) preventing replication catch-up Misconfiguration/regression after change (replication config, retention, credentials) Diagnostic Information: Confirm the trigger and impacted site_name Identify whether the critical is driven by replication status vs binlog usage trend. Review current and historical values for: db_tier_replication_status{role="active" | "standby" | "failed"} db_tier_binlog_used_bytes_percentage (trend over last 30–60 minutes) Replication failure/channel-down triage On DB nodes, check replication channel state, IO/SQL thread status, last error, and lag. Determine whether the standby is unreachable vs reachable-but-failing (auth/TLS/permissions vs data/SQL errors). Confirm whether the “active” node is healthy and exporting metrics (avoid false positives from missing scrapes). Topology consistency check Validate expected number of replication endpoints per site (active + standby) and confirm none are missing. If the alert logic is based on counts, verify whether a node is down or metrics are absent. Binlog storage pressure investigation (if ≥ 80%) Check filesystem utilization where binlogs reside and remaining headroom; estimate time-to-full. Confirm whether binlog purge is blocked by lagging replica(s) or retention settings. Correlate with write throughput and long transactions. Dependency and resource saturation Disk IO latency/queue depth, CPU, memory pressure, network throughput/packet loss. DB error logs for disk-full warnings, replication errors, restarts, corruption indicators. Change/event correlation Look for recent deployments, DB patching, parameter changes, certificate rotations, or network maintenance coinciding with onset. Recovery: Immediate posture (CRITICAL): Treat as incident-level due to risk of data protection loss (replication down) and/or imminent DB outage (binlog disk exhaustion). Engage DB/platform/network on-call as required; follow site incident process. If replication failed / channel down: Restore connectivity (network/DNS/firewall) and fix auth/TLS/cert issues. Restart/repair replication channels per runbook; remediate root cause (SQL errors, missing binlogs, GTID divergence). If standby cannot be repaired quickly, rebuild standby from snapshot/backup and re-establish replication. Consider temporarily throttling write load to reduce lag/backlog while recovering. If binlog usage ≥ 80% (sustained): Prevent disk-full: reduce binlog generation (traffic/load throttling where feasible) restore replication so binlogs can be consumed and purged expand binlog storage/capacity as an emergency mitigation (per change controls) Verify purge/retention settings and ensure purge is not blocked by an unreachable or stuck replica. Validation or exit criteria: Replication status returns to expected healthy roles (no failed, standby present and catching up). Replication lag stabilizes within acceptable bounds. db_tier_binlog_used_bytes_percentage trends downward and remains below threshold.

8.1.1.47 SYSTEM_OPERATIONAL_STATE_PARTIAL_SHUTDOWN

Table 8-63 SYSTEM_OPERATIONAL_STATE_PARTIAL_SHUTDOWN

Field	Details
Description	System Operational State is now in partial shutdown state.
Summary	System Operational State is now in partial shutdown state.
Severity	Major
Expression	system_operational_state == 2
OID	1.3.6.1.4.1.323.5.3.52.1.2.44
Metric Used	system_operational_state == 2
Recommended Actions	Cause: The site has entered the PARTIAL SHUTDOWN operational state under the Controlled Shutdown feature. This deliberate isolation mode supports maintenance, such as upgrades or configuration changes, or recovery activities, such as prolonged replication-channel failures or replication delay, while allowing existing sessions to continue. In this state, NRF status is SUSPENDED to prevent consumer discovery; the Ingress Gateway and Diameter Gateway permit only in-session messages and reject new session establishment; and the Egress Gateway has no restriction. This state is typically set through the CM GUI or REST API when enableControlledShutdown is enabled. Diagnostic Information: Confirm the operational state is truly 2 (PARTIAL) and record transition time (use alert timestamp + metric history). Verify whether the change was planned (maintenance/recovery activity) and who/what initiated it (CM GUI/REST). Validate enableControlledShutdown is enabled and configuration is consistent across the site. Verify expected gateway behavior: Ingress rejects new session establishment but permits in-session traffic Diameter permits in-session only Check NRF to confirm NF status marked SUSPENDED and are not being discovered for new sessions. Identify the underlying reason for isolation (maintenance, replication channel failures/delay, dependency instability) and correlate with replication metrics/logs. Assess customer impact: rate of blocked new sessions, admission control counters, error codes returned at ingress/diameter. Recovery: For a planned shutdown, proceed with maintenance or recovery activities, such as replication repair or dependency stabilization, while keeping existing sessions stable. For an unplanned shutdown, confirm that no automation or configuration error triggered the change, correct configuration or access controls, and return to NORMAL when safe. After recovery, verify that NFs return to REGISTERED in NRF and that new session establishment succeeds at Ingress and Diameter. Escalate if the partial shutdown persists unexpectedly or blocks new sessions beyond agreed SLOs.

8.1.1.48 SYSTEM_OPERATIONAL_STATE_COMPLETE_SHUTDOWN

Table 8-64 SYSTEM_OPERATIONAL_COMPLETE_SHUTDOWN

Field	Details
Description	System Operational State is now in complete shutdown state
Summary	System Operational State is now in complete shutdown state
Severity	Critical
Expression	system_operational_state == 3
OID	1.3.6.1.4.1.323.5.3.52.1.2.44
Metric Used	system_operational_state
Recommended Actions	Cause: The site has entered the COMPLETE SHUTDOWN operational state under the Controlled Shutdown feature. This fully stops load and isolates the site during recovery or severe instability. NRF status is SUSPENDED, the Ingress Gateway accepts no requests, the Egress Gateway allows only NRF-bound requests, and Diameter Gateway external-peer connections are closed. Diagnostic Information: Confirm the operational state is 3 (COMPLETE) and capture the exact change time and duration. Verify whether complete isolation was intended (approved maintenance/incident action) and confirm initiator. Validate actual enforcement at gateways: Ingress is refusing all new connections Diameter external peer sessions are closed Egress allows only NRF-bound traffic Confirm NRF state: NFs remain SUSPENDED and are not discoverable by consumers (SMF/AMF/etc.). Identify the triggering condition that required complete shutdown (e.g., severe replication failure, corruption risk, cascading dependency failures) and gather supporting logs/metrics. Confirm traffic has drained/failover behavior is functioning (upstream routing, peer sites capacity, no looping retries). Check for any secondary effects: backlog growth, message queue accumulation, alarms from consumers due to loss of discovery. Recovery: Treat an unscheduled shutdown as an incident. Perform the maintenance or recovery procedure while the site is isolated, such as replication rebuild or data-consistency checks. When ready, transition to PARTIAL if required for validation, or directly to NORMAL; restore gateway connectivity and Egress behavior; update NRF status to REGISTERED; and validate discovery, new-session establishment, and clean Diameter-peer reconnection.

8.1.1.49 TDF_CONNECTION_DOWN

Table 8-65 TDF_CONNECTION_DOWN

Field	Details
Description	TDF connection is down.
Summary	TDF connection is down.
Severity	Critical
Expression	occnp_diam_conn_app_network{applicationName="Sd"} == 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.48
Metric Used	occnp_diam_conn_app_network
Recommended Actions	Cause: TDF_CONNECTION_DOWN indicates the Diameter Gateway has lost connectivity to a configured TDF peer node (for the Sd Diameter application). The alert fires when occnp_diam_conn_app_network{applicationName="Sd"} == 0, meaning the Diameter GW reports the Sd application network connection is down (peer unreachable/disconnected). Common drivers: TDF peer node outage/restart or Diameter service down Network connectivity loss between Diameter GW and TDF. TLS/MTLS or certificate issues (expired/rotated certs, truststore mismatch) Diameter peer configuration mismatch (host/realm, IP/port, capabilities) Transport-level failures (TCP resets, keepalive failures, packet loss, MTU issues) Gateway overload/resource pressure causing session drops (CPU/memory/thread starvation) Diagnostic Information: Identify the affected TDF peer(s) from the Diameter GW reported message/alert details (peer host/IP/realm) and confirm it is a configured peer. Verify whether the connection-down is isolated (single peer) or systemic (multiple peers down simultaneously). Check Diameter GW logs/events around the disconnect time for: disconnect reason codes, watchdog timeouts, capability exchange failures (CER/CEA), DPR/DPA TLS handshake failures / certificate validation errors connection refused / timeout / reset indications Validate network path from Diameter GW to the TDF peer: basic reachability (routing/VLAN/VXLAN), port access, firewall rules recent network changes/maintenance Confirm TDF peer health: process/service status, listener port up, resource saturation peer logs for rejecting connections, overload, or config errors Check recent changes on either side: peer definitions (host/realm/IP), allowed apps (Sd), policy changes certificate rotations, cipher policy updates If available, correlate with other metrics: total active Diameter connections, reconnect attempts, watchdog counters, error rate spikes. Recovery: Restore connectivity to the TDF peer: bring up/restart TDF Diameter service if down correct network/firewall/routing issues and confirm port reachability Fix configuration mismatches (peer host/realm, IP/port, supported application “Sd”). Resolve TLS/certificate issues (renew/replace certs, update truststores, align MTLS settings). If gateway resource pressure contributed, scale/relieve load and investigate saturation triggers. After remediation, verify: occnp_diam_conn_app_network{applicationName="Sd"} returns to 1/up peer sessions re-establish and remain stable (no flapping) Diameter GW reports the peer as reachable in status output/logs

8.1.1.50 DIAM_CONN_PEER_DOWN

Table 8-66 DIAM_CONN_PEER_DOWN

Field	Details
Description	Diameter connection to peer {{ $labels.peerHost }} is down.
Summary	Diameter connection to peer is down.
Severity	Major
Expression	(sum by (namespace,peerHost)(occnp_diam_conn_network) == 0) and (sum by (namespace,peerHost)(max_over_time(occnp_diam_conn_network[24h])) != 0)
OID	1.3.6.1.4.1.323.5.3.52.1.2.50
Metric Used	occnp_diam_conn_network
Recommended Actions	Cause: DIAM_CONN_PEER_DOWN indicates the Diameter Gateway’s network-side connection to a specific external peer (keyed by origHost) is currently down, even though it was successfully connected at least once in the last 24 hours. The alert logic is: Now: sum by(kubernetes_namespace, origHost)(occnp_diam_conn_network) == 0 → no active connections for that peer History: max_over_time(occnp_diam_conn_network[24h]) != 0 → the peer was up recently, so this is not “never configured” This is a MAJOR condition because it typically impacts one peer (or a subset), not necessarily all external Diameter connectivity. Common drivers: Peer node/service down, restarting, or in maintenance Network path issue specific to that peer (routing/ACL/firewall/NAT/DNS) Transport instability (packet loss/latency causing watchdog timeouts, TCP resets) TLS/mTLS handshake failures (cert expiry/rotation mismatch, trust mismatch, hostname/SNI mismatch, cipher/protocol mismatch) Peer overload/throttling causing disconnects or refused connections Configuration mismatch (peer IP/port, host/realm, supported applications) Diameter GW resource pressure leading to selective peer drops/reconnect storms Diagnostic Information: Identify the affected kubernetes_namespace and origHost value(s) and confirm whether this is a single peer or multiple peers. Validate that the peer was up recently: Confirm max_over_time(occnp_diam_conn_network[24h]) is non-zero for the same origHost. Check Diameter GW logs/events for the affected peer around the time it went down: watchdog failures (DWR/DWA), disconnect procedures (DPR/DPA) CER/CEA negotiation failures TCP connection refused/timeout/reset messages TLS handshake/certificate validation failures (if TLS enabled to this peer) Check the peer-side health: service/process status, listener port, recent restarts peer logs for reject/disconnect reasons and overload indicators Validate network reachability from GW to that peer: routing, firewall/NAT rules, DNS resolution confirm whether only this peer path is impacted (vs broader outage). If TLS is enabled for that peer, validate: certificate validity/expiry and correct chain trust bundle alignment on both sides hostname/SNI and protocol/cipher compatibility Correlate with recent changes: gateway/peer config updates, peer list changes, certificate rotations, network maintenance. Recovery: Restore peer availability: restart/repair peer service; coordinate with peer owner if external. Restore connectivity: fix routing/firewall/DNS issues affecting that peer; confirm required ports are open. Resolve TLS issues (if enabled): renew/rotate certificates, update trust bundles, correct hostname/SNI or cipher/protocol settings Address overload/instability: scale/relieve load on GW or peer; ensure reconnect backoff is reasonable to avoid storms. Validate recovery: occnp_diam_conn_network for the affected origHost returns to > 0 connection remains stable (no repeated flapping) Diameter transaction success rates normalize for traffic targeting that peer

8.1.1.51 DIAM_CONN_NETWORK_DOWN

Table 8-67 DIAM_CONN_NETWORK_DOWN

Field	Details
Description	All the diameter network connections are down.
Summary	All the diameter network connections are down.
Severity	Critical
Expression	sum by (kubernetes_namespace)(occnp_diam_conn_network) == 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.51
Metric Used	occnp_diam_conn_network
Recommended Actions	Cause: DIAM_CONN_NETWORK_DOWN indicates the Diameter Gateway has no active Diameter network (external peer) connections in a given kubernetes_namespace. The alert fires when: sum by (kubernetes_namespace)(occnp_diam_conn_network) == 0. This means the total count of established/healthy external/network-side Diameter connections reported by the gateway is zero for that namespace. This results in loss of Diameter connectivity to external peers (for example, TDF or other network peer nodes). Common drivers (TLS may apply for external peers): External peer outage/restart/maintenance (peer service down, listener not available) Network path failure affecting all peers (routing/VRF/VLAN, firewall/ACL, NAT, DNS) TLS/mTLS issues (expired/rotated certificates, CA/trust mismatch, hostname/SNI mismatch, cipher/protocol mismatch) Peer configuration mismatch (wrong IP/port, host/realm, supported application such as Sd) Transport instability (packet loss/latency causing watchdog timeouts, TCP resets) Diameter GW instability/overload causing disconnects and failed reconnect attempts Controlled shutdown/site isolation actions that intentionally close external peer connections (if applicable) Diagnostic Information: Confirm impacted kubernetes_namespace and whether the issue is isolated or widespread. Identify which peers are affected (if metrics/logs provide peer labels/details) and whether all external peers are down simultaneously. Review Diameter GW logs/events around the drop: watchdog failures (DWR/DWA), disconnect procedures (DPR/DPA) CER/CEA failures (capabilities mismatch) TCP timeouts/resets/connection refused Validate external peer health: peer process up, port listening, peer-side logs show accept/reject reasons check whether peers are in maintenance or intentionally isolated. Network reachability validation: routing/firewall/NAT/security list correctness to all peers confirm no recent network maintenance/changes coinciding with onset. If TLS is enabled for external peers, validate TLS posture: certificate expiry and correct certificate chain truststore/CA bundle alignment on both sides hostname/SNI matching (if used), protocol/cipher compatibility Correlate with related alerts/metrics: if backend connections are also down, suspect gateway-wide failure if controlled shutdown is active, confirm external peer connections are expected to be closed. Recovery: Restore external peer availability (restart/repair peer services; coordinate with peer owner teams). Restore network connectivity (fix routing/ACL/firewall/NAT/DNS) and confirm required ports are reachable. If TLS-related: renew/rotate certificates and update truststores/CA bundles correct hostname/SNI/cipher/protocol mismatches re-establish sessions after cert/config updates Correct Diameter peer configuration (IP/port, host/realm, application enablement). If GW resource pressure/instability contributed, scale/repair the gateway and address the underlying saturation. Validate recovery: sum by (kubernetes_namespace)(occnp_diam_conn_network) returns to > 0 external peer sessions remain stable (no frequent reconnects) end-to-end Diameter transactions succeed and error rates normalize

8.1.1.52 DIAM_CONN_BACKEND_DOWN

Table 8-68 DIAM_CONN_BACKEND_DOWN

Field	Details
Description	All the diameter backend connections are down.
Summary	All the diameter backend connections are down.
Severity	Critical
Expression	sum by (kubernetes_namespace)(occnp_diam_conn_backend) == 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.52
Metric Used	occnp_diam_conn_network
Recommended Actions	Cause: DIAM_CONN_BACKEND_DOWN indicates the Diameter Gateway has no active connections to any internal backend services (for example, PCRF Core, Diam-Connector) within a given kubernetes_namespace. The alert fires when: sum by (kubernetes_namespace)(occnp_diam_conn_backend) == 0 meaning the gateway reports the total number of established backend connections is zero for that namespace. With no backend connectivity, the Diameter GW will be unable to route/process Diameter requests that depend on those internal services. Common drivers (no TLS involved for these backend connections): Internal backend services unavailable (PCRF Core / Diam-Connector pods down, crashlooping, not Ready, rolling restart) Service discovery issues (K8s Service selectors wrong, Endpoints/EndpointSlices empty, wrong port/targetPort) Network connectivity/policy issues inside cluster (NetworkPolicy blocking, CNI/kube-proxy/node networking issues) Backend overload/backpressure (connection limits reached, listener not accepting, process unhealthy) Diameter GW overload/resource issues (CPU/memory/file-descriptor exhaustion) causing connection drops/reconnect failures Misconfiguration/regression after deploy (backend host/port/service name changes) Diagnostic Information: Identify impacted kubernetes_namespace and confirm whether the issue is isolated to one namespace or widespread. Break down occnp_diam_conn_backend by available labels (for example, GW pod/instance, and backend identifier if present) to determine whether: all GW replicas have zero backend connections, or only specific GW pods/nodes are affected. Check backend service health (PCRF Core, Diam-Connector): pod status, readiness and liveness, restarts, recent rollouts application logs for “cannot accept connection”, “connection limit”, “listener down”, resource exhaustion Verify Kubernetes Service/Endpoints for each backend: Service exists and selectors match expected pods Endpoints/EndpointSlices are non-empty and point to correct IP:port port/targetPort configuration matches what the GW expects Review Diameter GW logs for connection symptoms: connection refused (backend not listening/not ready) timeout (network path/policy/CNI issue or backend hung) frequent reconnect loops / backend unavailable messages Validate in-cluster networking: NetworkPolicies allowing GW → backend traffic on required ports node/CNI/kube-proxy issues if failures correlate to specific nodes Correlate with recent changes: deploys/config updates of Diameter GW, PCRF Core, Diam-Connector service renames, port changes, scaling events. Recovery: Restore backend availability: fix/restart failing PCRF Core / Diam-Connector pods, ensure they become Ready. Roll back recent backend changes if a regression is suspected. scale backends if overloaded or hitting connection limits. Fix service discovery wiring: correct Service selectors/ports/targetPorts; ensure EndpointSlices repopulate Restore connectivity: adjust/Roll back NetworkPolicies blocking traffic remediate node/CNI/kube-proxy issues if connectivity is node-specific Stabilize gateway/backends if resource pressure is contributing: scale Diameter GW or increase resources/limits. address FD exhaustion or excessive reconnect storms. Validate recovery: sum by (kubernetes_namespace)(occnp_diam_conn_backend) returns to > 0 connections to PCRF Core and Diam-Connector remain stable (no flapping) Diameter transaction error/timeout rates return to baseline

8.1.1.53 PERF_INFO_ACTIVE_OVERLOAD_THRESHOLD_FETCH_FAILED

Table 8-69 PPERF_INFO_ACTIVE_OVERLOAD_THRESHOLD_FETCH_FAILED

Field	Details
Description	The application fails to get the current active overload level threshold data.
Summary	The application fails to get the current active overload level threshold data.
Severity	Major
Expression	active_overload_threshold_fetch_failed == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.53
Metric Used	active_overload_threshold_fetch_failed
Recommended Actions	Cause: The system has failed to fetch or retrieve the active overload threshold data, as indicated by the flag active_overload_threshold_fetch_failed == 1. This suggests that overload protection parameters are unavailable or not being correctly loaded, which may impact the system ability to handle traffic surges safely. Diagnostic Information: Evaluate the following metric: active_overload_threshold_fetch_failed == 1 Trigger condition: Value equals 1 (failure state) Interpretation: The system could not fetch active overload threshold configuration/data Perform the following checks: Verify connectivity to the configuration source (e.g., database, config service, NMS) Check if overload threshold data is properly configured and available Review logs for errors related to config fetch failures or timeouts Validate credentials and access permissions for configuration retrieval Inspect recent configuration changes or deployments affecting overload settings Ensure dependent services providing threshold data are up and responsive Check for initialization issues during service startup Recovery: Restore connectivity to the configuration or data source Correct missing or invalid overload threshold configurations Restart or reinitialize the component responsible for fetching the data Fix authentication/authorization issues if access is denied Roll back recent changes impacting configuration retrieval Ensure high availability of configuration services (failover/replication) Monitor until the metric returns to normal state (value = 0) If the issue still persist, please contact the My Oracle Support.

8.1.1.54 SLA_SY_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-70 SLA_SY_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	SLA Sy fail count exceeds the critical threshold limit
Summary	Alert SLA_SY_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD > 90
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{msgType="SLA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="SLA"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.58
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed SLA (Subscriber Location Answer / Service-Level related Answer) Diameter responses has exceeded the critical threshold (greater than 90%). The condition indicates that the majority of SLA transactions are failing, which can significantly impact subscriber-related operations such as location handling, service validation, or policy enforcement depending on the SLA usage context. Likely causes include backend system failures, data unavailability, or signaling issues. Diagnostic Information: Evaluate the following metric: Failed SLA Responses / Total SLA Responses * 100 over a 5-minute window Scope: msgType = "SLA" Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 90% Perform the following checks: Analyze Diameter response codes (for example, user unknown, system failure, data missing). Verify connectivity to backend systems (for example, subscriber database, HSS/UDM, policy servers). Check logs for request processing failures, timeouts, or invalid responses. Validate subscriber data availability and consistency. Monitor latency and retransmissions for SLA transactions. Inspect system resource utilization (CPU, memory, thread pools). Correlate with recent configuration or schema changes affecting subscriber data. Recovery: Restore connectivity to backend/subscriber data systems. Fix issues related to missing or inconsistent subscriber data. Address specific Diameter error codes causing SLA failures. Restart or scale affected services handling SLA traffic. Roll back recent configuration or deployment changes if correlated. Tune timeout and retry mechanisms if applicable Continue monitoring until failure rate drops below the critical threshold. For any additional guidance, contact My Oracle Support

8.1.1.55 SLA_SY_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-71 SLA_SY_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	SLA Sy fail count exceeds the major threshold limit
Summary	Alert SLA_SY_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD > 80 and <= 90
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{msgType="SLA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="SLA"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="SLA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="SLA"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.58
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed SLA (Subscriber Location Answer / Service-Level related Answer) Diameter responses has exceeded the major threshold (greater than 80% and up to 90%). The condition indicates a high rate of SLA transaction failures, suggesting significant service degradation. While not yet critical, this level of failure can impact subscriber-related operations and may escalate if not addressed promptly. Possible causes include backend instability, partial outages, or increasing system load. Diagnostic Information: Evaluate the following metric: Failed SLA Responses / Total SLA Responses * 100 over a 5-minute window Scope: msgType = "SLA" Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 80% and ≤ 90% Perform the following checks: Analyze response code distribution to identify dominant failure types. Check for partial failures in backend systems (HSS/UDM, subscriber DB). Review logs for intermittent errors, timeouts, or invalid responses. Validate subscriber data availability and consistency. Monitor system performance (CPU, memory, latency, thread utilization). Inspect network connectivity and latency for SLA transactions. Correlate with recent deployments or configuration changes. Recovery: Investigate and stabilize backend systems handling subscriber data. Resolve identified issues based on specific Diameter error codes. Optimize or scale system resources to handle load. Fix data inconsistencies or missing subscriber information. Roll back recent changes if they correlate with the issue. Apply retry or failover mechanisms if applicable. Closely monitor to prevent escalation to the critical threshold For any additional guidance, contact My Oracle Support.

8.1.1.56 SLA_SY_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-72 SLA_SY_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	SLA Sy fail count exceeds the minor threshold limit
Summary	Alert SLA_SY_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD > 60 <= 80
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{msgType="SLA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="SLA"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="SLA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="SLA"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.58
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed SLA (Subscriber Location Answer / Service-Level related Answer) Diameter responses has exceeded the minor threshold (greater than 60% and up to 80%). The condition indicates an early stage of service degradation where a significant portion of SLA transactions are failing. While the system may still be functional, this condition signals potential underlying issues such as intermittent backend failures, rising load, or partial data/service unavailability. Diagnostic Information: Evaluate the following metric: Failed SLA Responses / Total SLA Responses * 100 over a 5-minute window Scope: msgType = "SLA" Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 60% and ≤ 80% Perform the following checks: Observe trends to determine if failure rate is increasing toward major/critical levels Analyze response code distribution for early error patterns. Check connectivity and responsiveness of backend systems (HSS/UDM, subscriber DB). Review logs for intermittent failures, retries, or timeout issues. Validate subscriber data availability and consistency. Monitor system performance (CPU, memory, latency, thread utilization). Identify if failures are isolated to specific request types or scenarios. Recovery: Investigate and address early signs of backend or data-related issues. Validate and correct configurations or recent changes if applicable. Optimize system performance or scale resources to handle increasing load. Resolve identified issues based on Diameter error codes. Apply preventive measures (retry tuning, connection stability improvements). Continue close monitoring to prevent escalation to higher severity thresholds. For any additional guidance, contact My Oracle Support.

8.1.1.57 STA_SY_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-73 STA_SY_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	STA Sy fail count exceeds the critical threshold limit.
Summary	Alert STA_SY_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD >90
Severity	Critical
Expression	The failure rate of Sy STA responses is more than 90% of the total responses.
Expression	sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777302", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777302"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.59
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed STA (Session Termination Answer) Diameter responses for application ID 16777302 has exceeded the critical threshold (greater than 90%). The condition indicates that most session termination requests are failing, which can lead to improper session cleanup, resource leakage, and inconsistencies in subscriber/session state. Likely causes include backend failures, session state mismatches, or signaling issues with peer systems. Diagnostic Information: Evaluate the following metric: Failed STA Responses / Total STA Responses * 100 over a 5-minute window Scope: msgType = "STA" (Session Termination Answer) appId = "16777302" (specific Diameter application, typically Gx) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 90% Perform the following checks: Analyze Diameter response codes (for example, unknown session, unable to comply, system errors). Verify session state consistency between network elements (for example, PCRF/PCF and gateway). Check connectivity and health of peer systems handling session termination. Review logs for session cleanup failures, stale sessions, or timeout issues. Monitor retransmissions, latency, and signaling errors. Inspect system resource utilization (CPU, memory, session table capacity). Correlate with spikes in session terminations (for example, mass disconnect events). Recovery: Restore connectivity and stability with peer systems (for example, PCRF/PCF). Resolve session state inconsistencies or stale session entries. Address specific Diameter error codes causing STA failures. Restart or scale affected components handling session termination. Clean up orphaned or stuck sessions if required Roll back recent changes impacting session handling. Monitor until failure rate drops below the critical threshold. For any additional guidance, contact My Oracle Support.

8.1.1.58 STA_SY_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-74 STA_SY_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	STA Sy fail count exceeds the major threshold limit.
Summary	Alert STA_SY_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD > 80 and <= 90
Severity	Major
Expression	The failure rate of Sy STA responses is more than 80% and less and or equal to 90% of the total responses.
Expression	sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777302", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777302"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777302", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777302"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.59
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed STA (Session Termination Answer) Diameter responses for application ID 16777302 has exceeded the major threshold (greater than 80% and up to 90%). The condition indicates a high rate of session termination failures, which can lead to incomplete session cleanup, inconsistent session states, and potential resource buildup. The issue suggests partial service degradation, possibly due to backend instability, signaling issues, or increased load. Diagnostic Information: Evaluate the following metric: Failed STA Responses / Total STA Responses * 100 over a 5-minute window Scope: msgType = "STA" appId = "16777302" Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 80% and ≤ 90% Perform the following checks: Analyze response code distribution (for example, unknown session, unable to comply, system errors). Verify session state synchronization between network elements (for example, PCRF/PCF and gateway). Check connectivity and health of peer systems handling session termination. Review logs for intermittent session cleanup failures or timeout issues. Monitor signaling performance (latency, retransmissions). Inspect system resource usage (CPU, memory, session table capacity). Correlate with spikes in session termination requests (for example, bulk disconnect events). Recovery: Investigate and stabilize communication with peer systems. Resolve session state inconsistencies or stale session entries. Address specific Diameter error codes contributing to failures. Optimize or scale system resources to handle load. Restart affected components if necessary. Roll back recent configuration or deployment changes if correlated. Closely monitor to prevent escalation to the critical threshold For any additional guidance, contact My Oracle Support.

8.1.1.59 STA_SY_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-75 STA_SY_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	STA Sy fail count exceeds the minor threshold limit.
Summary	Alert STA_SY_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD > 60 and <= 80
Severity	Minor
Expression	The failure rate of Sy STA responses is more than 60% and less and or equal to 80% of the total responses.
Expression	sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777302", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777302"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777302", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777302"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.59
Metric Used	occnp_diam_response_local_total
Recommended Actions	Cause: The percentage of failed STA (Session Termination Answer) Diameter responses for application ID 16777302 has exceeded the minor threshold (greater than 60% and up to 80%). The condition indicates an early stage of degradation in session termination handling. While the system may still be operational, a significant portion of session termination requests are failing, potentially leading to delayed cleanup, partial session inconsistencies, or buildup of stale sessions. Diagnostic Information: Evaluate the following metric: Failed STA Responses / Total STA Responses * 100 over a 5-minute window Scope: msgType = "STA" appId = "16777302" Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 60% and ≤ 80% Perform the following checks: Observe trends to determine if failure rate is increasing toward major/critical levels Analyze response codes for early indicators (for example, unknown session, temporary failures). Verify session state consistency between network elements. Check connectivity and responsiveness of peer systems (for example, PCRF/PCF). Review logs for intermittent session termination or cleanup issues. Monitor system performance (CPU, memory, session table usage). Identify if failures are linked to specific traffic patterns or events. Recovery: Investigate and address early signs of session handling or backend issues. Validate session state synchronization across systems. Resolve identified issues based on Diameter error codes. Optimize or scale system resources to handle load. Apply preventive measures (for example, retry tuning, session cleanup mechanisms). Review and correct recent configuration or deployment changes if applicable. Continue close monitoring to prevent escalation to higher severity thresholds. For any additional guidance, contact My Oracle Support.

8.1.1.60 STA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-76 STA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	The failure rate of Rx STA responses is more than 90% of the total responses.
Summary	Alert STA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD >90
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777236"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.64
Metric Used	occnp_diam_response_local_total{msgType="STA", appId="16777236", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed STA (Session Termination Answer) Diameter responses for application ID 16777236 (Rx interface) has exceeded the critical threshold (greater than 90%). The condition indicates that most session termination operations on the Rx interface are failing, which can lead to improper session cleanup, policy inconsistencies, and potential resource leakage. Likely causes include failures in communication with AF (Application Function), signaling issues, or backend instability. Diagnostic Information: Evaluate the following metric: Failed STA (Rx) Responses / Total STA (Rx) Responses * 100 over a 5-minute window Scope: msgType = "STA" appId = "16777236" (Rx interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 90% Perform the following checks: Analyze Diameter response codes (for example, unknown session, authorization failures, unable to comply). Verify connectivity and health of AF (Application Function) or related Rx peers. Check session state consistency between PCRF/PCF and AF. Review logs for session termination failures, timeouts, or invalid requests. Monitor signaling metrics (latency, retransmissions, error rates). Inspect system resource utilization (CPU, memory, session tables). Correlate with spikes in Rx signaling or session termination events. Recovery: Restore connectivity and stability with AF or Rx peer systems. Resolve session state inconsistencies or stale sessions. Address specific Diameter error codes causing failures. Restart or scale affected components handling Rx traffic. Clean up orphaned or stuck sessions if required Roll back recent configuration or deployment changes if correlated. Monitor until failure rate drops below the critical threshold.

8.1.1.61 STA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-77 STA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	The failure rate of Rx STA responses is more than 80% and less and or equal to 90% of the total responses.
Summary	Alert STA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD > 80 and <= 90
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777236"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777236"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.64
Metric Used	occnp_diam_response_local_total{msgType="STA", appId="16777236", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed STA (Session Termination Answer) Diameter responses for application ID 16777236 (Rx interface) has exceeded the major threshold (greater than 80% and up to 90%). The condition indicates a high failure rate in session termination over the Rx interface, leading to partial service degradation. It may result in delayed or incomplete session cleanup, inconsistent policy enforcement, and increased resource utilization. The issue is likely due to instability in communication with the Application Function (AF), backend issues, or increased signaling load. Diagnostic Information: Evaluate the following metric: Failed STA (Rx) Responses / Total STA (Rx) Responses * 100 over a 5-minute window Scope: msgType = "STA" appId = "16777236" (Rx interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 80% and ≤ 90% Perform the following checks: Analyze response code distribution (for example, unknown session, temporary failures, system errors). Verify connectivity and health of AF (Application Function) peers. Check session state synchronization between PCRF/PCF and AF. Review logs for intermittent failures, timeouts, or malformed requests. Monitor signaling performance (latency, retransmissions, error trends). Inspect system resource usage (CPU, memory, session tables). Correlate with spikes in Rx traffic or session termination events. Recovery: Investigate and stabilize connectivity with AF/Rx peer systems. Resolve session state inconsistencies or stale session entries. Address identified Diameter error codes contributing to failures. Optimize or scale system resources to handle load. Restart affected components if necessary. Roll back recent configuration or deployment changes if correlated. Closely monitor to prevent escalation to the critical threshold

8.1.1.62 STA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-78 STA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	The failure rate of Rx STA responses is more than 60% and less and or equal to 80% of the total responses.
Summary	STA Rx fail count exceeds the minor threshold limit.
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777236"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777236", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="STA", appId="16777236"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.64
Metric Used	occnp_diam_response_local_total{msgType="STA", appId="16777236", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed STA (Session Termination Answer) Diameter responses for application ID 16777236 (Rx interface) has exceeded the minor threshold (greater than 60% and up to 80%). The condition indicates an early stage of degradation in handling session termination over the Rx interface. While the system may still be functioning, a notable portion of session terminations are failing, which can lead to delayed cleanup, partial session inconsistencies, and potential buildup of stale sessions. Diagnostic Information: Evaluate the following metric: Failed STA (Rx) Responses / Total STA (Rx) Responses * 100 over a 5-minute window Scope: msgType = "STA" appId = "16777236" (Rx interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 60% and ≤ 80% Perform the following checks: Observe trends to see if failure rate is increasing toward major/critical levels Analyze response codes for early indicators (for example, unknown session, temporary/system errors). Verify connectivity and responsiveness of AF (Application Function) peers. Check session state synchronization between PCRF/PCF and AF. Review logs for intermittent failures, retries, or timeout issues. Monitor system performance (CPU, memory, session table utilization). Identify if failures correlate with specific traffic patterns or events. Recovery: Investigate and address early signs of issues in Rx signaling or AF connectivity. Validate session state consistency across involved systems. Resolve identified issues based on Diameter error codes. Optimize or scale system resources to handle load. Apply preventive measures (retry tuning, improved session cleanup mechanisms). Review and correct recent configuration or deployment changes if applicable. Continue close monitoring to prevent escalation to higher severity thresholds.

8.1.1.63 SNA_SY_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-79 SNA_SY_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	The failure rate of Sy SNA responses is more than 90% of the total responses.
Summary	Alert SNA_SY_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD > 90
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{msgType="SNA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="SNA"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.65
Metric Used	occnp_diam_response_local_total{msgType="SNA", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed SNA (Subscriber Notification Answer) Diameter responses has exceeded the critical threshold (greater than 90%). The condition indicates a severe failure in handling subscriber notification responses, which may impact event notifications, policy updates, or subscriber state synchronization. The issue is likely due to backend system failures, signaling issues, or data inconsistencies. Diagnostic Information: Evaluate the following metric: Failed SNA Responses / Total SNA Responses * 100 over a 5-minute window Scope: msgType = "SNA" Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 90% Perform the following checks: Analyze Diameter response codes (for example, user unknown, unable to comply, system errors). Verify connectivity and health of backend systems (for example, HSS/UDM, notification handlers). Check logs for notification delivery failures, timeouts, or processing errors. Validate subscriber data availability and correctness. Monitor signaling performance (latency, retransmissions, error spikes). Inspect system resources (CPU, memory, thread pools). Correlate with spikes in notification events or subscriber activity. Recovery: Restore connectivity and stability with backend/subscriber systems. Resolve data inconsistencies or missing subscriber information. Address specific Diameter error codes causing failures. Restart or scale affected components handling notification traffic. Roll back recent configuration or deployment changes if correlated. Tune timeout and retry mechanisms if applicable Monitor until failure rate drops below the critical threshold. For any additional guidance, contact My Oracle Support.

8.1.1.64 SNA_SY_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-80 SNA_SY_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	The failure rate of Sy SNA responses is more than 80% and less and or equal to 90% of the total responses.
Summary	Alert SNA_SY_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD > 80 and <= 90
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{msgType="SNA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="SNA"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="SNA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="SNA"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.65
Metric Used	occnp_diam_response_local_total{msgType="SNA", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed SNA (Subscriber Notification Answer) Diameter responses has exceeded the major threshold (greater than 80% and up to 90%). The condition indicates a high rate of failures in subscriber notification handling, leading to partial service degradation. It may impact timely delivery of notifications, subscriber state synchronization, and policy updates. The issue could stem from backend instability, signaling issues, or increased system load. Diagnostic Information: Evaluate the following metric: Failed SNA Responses / Total SNA Responses * 100 over a 5-minute window Scope: msgType = "SNA" Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 80% and ≤ 90% Perform the following checks: Analyze response code distribution to identify dominant failure types. Verify connectivity and health of backend systems (for example, HSS/UDM, notification handlers). Check logs for intermittent notification failures, timeouts, or processing errors. Validate subscriber data availability and correctness. Monitor signaling performance (latency, retransmissions, error trends). Inspect system resource usage (CPU, memory, thread pools). Correlate with spikes in notification events or subscriber activity. Recovery: Investigate and stabilize backend systems handling notifications. Resolve issues based on specific Diameter error codes. Optimize or scale system resources to handle load. Fix data inconsistencies or missing subscriber information. Restart affected components if necessary. Roll back recent configuration or deployment changes if correlated. Closely monitor to prevent escalation to the critical threshold For any additional guidance, contact My Oracle Support.

8.1.1.65 SNA_SY_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-81 SNA_SY_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	The failure rate of Sy STA responses is more than 60% and less and or equal to 80% of the total responses.
Summary	Alert SNA_SY_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD > 60 <= 80
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{msgType="SNA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="SNA"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="SNA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="SNA"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.65
Metric Used	occnp_diam_response_local_total{msgType="SNA", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed SNA (Subscriber Notification Answer) Diameter responses has exceeded the minor threshold (greater than 60% and up to 80%). The condition indicates an early stage of degradation in subscriber notification handling. While the system may still be operational, a noticeable portion of notification responses are failing, potentially affecting timely updates to subscriber state and policy enforcement. Diagnostic Information: Evaluate the following metric: Failed SNA Responses / Total SNA Responses * 100 over a 5-minute window Scope: msgType = "SNA" Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 60% and ≤ 80% Perform the following checks: Observe trends to determine if failure rate is increasing toward major/critical levels Analyze response code distribution for early failure patterns. Verify connectivity and responsiveness of backend systems (for example, HSS/UDM, notification handlers). Review logs for intermittent errors, retries, or timeout issues. Validate subscriber data availability and correctness. Monitor system performance (CPU, memory, thread pools, latency). Identify if failures are linked to specific traffic spikes or events. Recovery: Investigate and address early signs of backend or signaling issues. Validate and correct configurations or recent changes. Optimize or scale system resources to handle load. Resolve issues based on observed Diameter error codes. Apply preventive measures (retry tuning, connection stability improvements). Continue close monitoring to prevent escalation to higher severity thresholds. For any additional guidance, contact My Oracle Support.

8.1.1.66 DIAM_GATEWAY_CERTIFICATE_EXPIRY_MINOR

Table 8-82 DIAM_GATEWAY_CERTIFICATE_EXPIRY_MINOR

Field	Details
Description	Certificate expiry in less than 6 months.
Summary	Certificate expiry in less than 6 months.
Severity	Minor
Expression	dgw_tls_cert_expiration_seconds - time() <= 15724800
OID	1.3.6.1.4.1.323.5.3.52.1.2.75
Metric Used	dgw_tls_cert_expiration_seconds
Recommended Actions	Cause: Diameter Gateway TLS certificate approaching expiry; risk of future connection failures if not rotated. Diagnostic Information: Identify which cert/instance is expiring (labels on metric, if present). min by (instance,cert_subject,cert_name) (dgw_tls_cert_expiration_seconds - time()) (adjust labels to your environment) Recovery: Start certificate renewal process: create/approve new cert, validate chain/SANs, and schedule rotation before major window. Confirm DGW can load the new cert in staging/non-prod first.

8.1.1.67 DIAM_GATEWAY_CERTIFICATE_EXPIRY_MAJOR

Table 8-83 DIAM_GATEWAY_CERTIFICATE_EXPIRY_MAJOR

Field	Details
Description	Certificate expiry in less than 3 months.
Summary	Certificate expiry in less than 3 months.
Severity	Major
Expression	dgw_tls_cert_expiration_seconds - time() <= 7862400
OID	1.3.6.1.4.1.323.5.3.52.1.2.75
Metric Used	dgw_tls_cert_expiration_seconds
Recommended Actions	Cause: Diameter Gateway TLS certificate approaching expiry; risk of future connection failures if not rotated. Diagnostic Information: Identify which cert/instance is expiring (labels on metric, if present). min by (instance,cert_subject,cert_name) (dgw_tls_cert_expiration_seconds - time()) (adjust labels to your environment) Confirm the dependent peers trust the issuing CA and that intermediate certificate are correct. Recovery: Execute rotation plan: deploy new cert + key via approved secret management, reload/restart DGW as per runbook, and verify successful TLS handshakes with peers.

8.1.1.68 DIAM_GATEWAY_CERTIFICATE_EXPIRY_CRITICAL

Table 8-84 DIAM_GATEWAY_CERTIFICATE_EXPIRY_CRITICAL

Field	Details
Description	Certificate expiry in less than 1 month.
Summary	Certificate expiry in less than 1 month.
Severity	Critical
Expression	dgw_tls_cert_expiration_seconds - time() <= 2592000
OID	1.3.6.1.4.1.323.5.3.52.1.2.75
Metric Used	dgw_tls_cert_expiration_seconds
Recommended Actions	Cause: Imminent expiry; high risk of production outage when cert expires. Diagnostic Information: Identify which cert/instance is expiring (labels on metric, if present). min by (instance,cert_subject,cert_name) (dgw_tls_cert_expiration_seconds - time()) (adjust labels to your environment) Confirm the dependent peers trust the issuing CA and that intermediate certificate are correct. Verify exact expiry timestamp and whether multiple instances are impacted. Recovery: Expedite renewal/rotation immediately. After deployment, validate with: Peer connection stability (no TLS alerts/handshake failures) DGW logs show new cert loaded Metric resets to a larger time-to-expiry.

8.1.1.69 DGW_TLS_CONNECTION_FAILURE

Table 8-85 DGW_TLS_CONNECTION_FAILURE

Field	Details
Description	Alert for TLS connection establishment.
Summary	TLS Connection failure when Diam gateway is an initiator.
Severity	Major
Expression	sum by (namespace,reason)(occnp_diam_failed_conn_network) > 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.81
Metric Used	occnp_diam_failed_conn_network
Recommended Actions	Cause: Diameter Gateway (DGW) is failing to establish TLS connections as a client due to handshake exceptions. Common causes: expired/rotated certs, missing intermediate chain, truststore mismatch/unknown CA, SAN/CN mismatch, TLS policy/cipher mismatch, clock skew, or network middlebox interference. Diagnostic Information: Identify the reason breakdown and where it occurs. sum by(namespace,reason) (occnp_diam_failed_conn_network) If labels exist for peer/endpoint, isolate the target Example: sum by(namespace,reason,destHost,destRealm) (occnp_diam_failed_conn_network) Correlate with cert-expiry alerts and validate time sync (NTP) on DGW nodes. Check DGW logs around handshake failures to confirm: unknown_ca, bad_certificate, certificate_expired, handshake_failure, protocol_version, etc. Recovery: If cert/trust issue: renew/rotate DGW client cert or update truststores/CA bundles (include intermediates), then reload/restart DGW per runbook. If TLS policy mismatch: align TLS versions/ciphers on both sides according to approved security standards. If endpoint-specific: remediate the server cert/config or failover to an alternate server/peer. Validate: handshake failures stop increasing and DGW peer connections remain established.

8.1.1.70 POLICY_CONNECTION_FAILURE

Table 8-86 POLICY_CONNECTION_FAILURE

Field	Details
Description	Connection failure on Egress and Ingress Gateways for incoming and outgoing connections.
Summary	Connection failure on Egress and Ingress Gateways for incoming and outgoing connections.
Severity	Major
Expression	sum(increase(occnp_oc_ingressgateway_connection_failure_total[5m]) >0 or (occnp_oc_ingressgateway_connection_failure_total unless occnp_oc_ingressgateway_connection_failure_total offset 5m )) by (namespace,app, error_reason) > 0 or sum(increase(occnp_oc_egressgateway_connection_failure_total[5m]) >0 or (occnp_oc_egressgateway_connection_failure_total unless occnp_oc_egressgateway_connection_failure_total offset 5m )) by (namespace,app, error_reason) > 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.76
Metric Used	occnp_oc_ingressgateway_connection_failure_total
Recommended Actions	Cause: Gateway is experiencing incoming/outgoing connection failures, typically due to: downstream endpoint down, DNS issues, TLS handshake failures, network ACL/firewall changes, certificate/CA problems, or resource exhaustion causing connect timeouts/refusals. Diagnostic Information: Break down failures by gateway and reason: - Ingress: sum by (namespace,app,error_reason) (increase(occnp_oc_ingressgateway_connection_failure_total[5m])) - Egress: sum by (namespace,app,error_reason) (increase(occnp_oc_egressgateway_connection_failure_total[5m])) Identify affected destination(s) if labeled (e.g., host/cluster/route). Correlate with cert expiry alerts, DNS metrics, and network error logs. Recovery: If TLS/cert-related (error_reason indicates handshake/unknown CA): rotate/fix cert chain and truststores; confirm mTLS policies. If network/DNS: restore name resolution, routes, firewall rules, service endpoints; validate connectivity from gateway pods. If downstream outage/overload: restore downstream service or failover; consider backoff/rate limit to reduce retry storms. Validate recovery by observing failure increases return to ~0 and connections stabilize.

8.1.1.71 AUDIT_NOT_RUNNING

Table 8-87 AUDIT_NOT_RUNNING

Field	Details
Description	Audit has not been running for at least 1 hour.
Summary	Audit has not been running for at least 1 hour.
Severity	CRITICAL
Expression	(increase(data_repository_invocations_seconds_count{method="getQueuedTablesToAudit",state="SUCCESS"}[1h])) == 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.78
Metric Used	data_repository_invocations_seconds_count
Recommended Actions	Cause: No successful executions of getQueuedTablesToAudit have been recorded for at least 1 hour (increase(...[1h]) == 0). This condition indicates the audit job in the application is not running, is stuck, or is failing before it records a SUCCESS. Diagnostic Information: Check in Prometheus/Grafana: Graph data_repository_invocations_seconds_count{method="getQueuedTablesToAudit"} by state to see if attempts are happening but failing (for example, ERROR/FAILURE) vs not happening at all. Confirm the metric series exists for the pod; if missing, verify Prometheus scrape health/target status for that pod. Check pod health and logs around the last hour: Recent restarts/OOMKills, readiness and liveness failures. Application logs for audit scheduler/worker thread issues, DB connectivity, timeouts, deadlocks, or queue starvation. Recovery: If the pod is unhealthy or the audit worker is stuck: restart/roll the pod (following your team’s change/restart procedures). If failures are present (non-success states): remediate the underlying error (for example, restore DB connectivity, fix credentials, address timeouts/locks, clear blocking conditions), then verify SUCCESS invocations resume. If metrics are missing but the service is healthy: fix scraping/metrics emission (Prometheus target, service monitor, endpoint auth), then confirm the counter is increasing. Validate recovery by confirming the metric increases over a short window (for example, increase(...[5m]) > 0) and the audit backlog/health indicators return to normal.

8.1.1.72 DIAMETER_POD_ERROR_RESPONSE_MINOR

Table 8-88 DIAMETER_POD_ERROR_RESPONSE_MINOR

Field	Details
Description	At least 1% of the Diam Response connection requests failed with error DIAMETER_UNABLE_TO_DELIVER.
Summary	At least 1% of the Diam Response connection requests failed with error DIAMETER_UNABLE_TO_DELIVER.
Severity	MINOR
Expression	(sum by (pod) (rate(occnp_diam_response_network_total{responseCode="3002"}[2m])))/ (sum by (pod) (rate(occnp_diam_response_network_total[2m]))) * 100 >=1
OID	1.3.6.1.4.1.323.5.3.52.1.2.79
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: At least one pod is returning/observing an elevated rate of Diameter result code 3002 (DIAMETER_UNABLE_TO_DELIVER). Typically indicates routing/peer selection problems, destination realm/host not reachable, peer connection down, invalid/missing route, or transient network issues. Can also occur during rolling restarts when peer tables/routes are not stable. Diagnostic Information: Identify the worst pod (the alert is already topk(1); list all pods). % by pod: ((sum by(pod)(rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) / sum by(pod)(rate(ocbsf_diam_response_network_total[2m]))) * 100) Confirm whether it’s a volume issue vs ratio artifact. Total by pod: sum by(pod)(rate(ocbsf_diam_response_network_total[2m])) 3002 by pod: sum by(pod)(rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) Break down by any available routing labels (commonly destHost, destRealm, peer, appId, msgType—use what exists in your metric) Example: sum by(pod,destRealm,destHost) (rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) Correlate with peer connection state / disconnects (Diameter stack metrics/logs) and recent deploy/restart events for the affected pod. Recovery: If isolated to one pod: drain/replace the pod (after confirming no broader peer outage) and validate routing tables/peer connections on the new instance. If isolated to a destination realm/host: fix routing configuration, realm definitions, and peer reachability; restore the peer connection. If caused by peer downtime: failover/reroute traffic to alternate peers where supported. Validate: 3002 rate returns to baseline and success (2xxx) responses dominate.

8.1.1.73 DIAMETER_POD_ERROR_RESPONSE_MAJOR

Table 8-89 DIAMETER_POD_ERROR_RESPONSE_MAJOR

Field	Details
Description	At least 5% of the Diam Response connection requests failed with error DIAMETER_UNABLE_TO_DELIVER.
Summary	At least 5% of the Diam Response connection requests failed with error DIAMETER_UNABLE_TO_DELIVER.
Severity	MAJOR
Expression	(sum by (pod) (rate(occnp_diam_response_network_total{responseCode="3002"}[2m])))/ (sum by (pod) (rate(occnp_diam_response_network_total[2m]))) * 100 >=5
OID	1.3.6.1.4.1.323.5.3.52.1.2.79
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: At least one pod is returning/observing an elevated rate of Diameter result code 3002 (DIAMETER_UNABLE_TO_DELIVER). Typically indicates routing/peer selection problems, destination realm/host not reachable, peer connection down, invalid/missing route, or transient network issues. Can also occur during rolling restarts when peer tables/routes are not stable. Indicates significant delivery failures likely impacting sessions/transactions. Diagnostic Information: Identify the worst pod (the alert is already topk(1); list all pods) - % by pod: ((sum by(pod)(rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) / sum by(pod)(rate(ocbsf_diam_response_network_total[2m]))) * 100) Confirm whether multiple pods are impacted: count( ((sum by(pod)(rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) / sum by(pod)(rate(ocbsf_diam_response_network_total[2m]))) * 100) >= 5 ) Confirm whether it’s a volume issue vs ratio artifact. Total by pod: sum by(pod)(rate(ocbsf_diam_response_network_total[2m])) 3002 by pod: sum by(pod)(rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) Break down by any available routing labels (commonly destHost, destRealm, peer, appId, msgType—use what exists in your metric) Example: sum by(pod,destRealm,destHost) (rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) Correlate with peer connection state / disconnects (Diameter stack metrics/logs) and recent deploy/restart events for the affected pod. Same as MINOR, plus confirm whether multiple pods are impacted: count( ((sum by(pod)(rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) / sum by(pod)(rate(ocbsf_diam_response_network_total[2m]))) * 100) >= 5 ) Recovery: Prioritize restoring reachability/routing: failover peers, fix realm/route config, and mitigate pod-level issues (restart/replace) if localized. Treat as incident if sustained.

8.1.1.74 DIAMETER_POD_ERROR_RESPONSE_CRITICAL

Table 8-90 DIAMETER_POD_ERROR_RESPONSE_CRITICAL

Field	Details
Description	At least 10% of the Diam Response connection requests failed with error DIAMETER_UNABLE_TO_DELIVER
Summary	At least 10% of the Diam Response connection requests failed with error DIAMETER_UNABLE_TO_DELIVER
Severity	CRITICAL
Expression	(sum by (pod) (rate(occnp_diam_response_network_total{responseCode="3002"}[2m])))/ (sum by (pod) (rate(occnp_diam_response_network_total[2m]))) * 100 >=10
OID	1.3.6.1.4.1.323.5.3.52.1.2.79
Metric Used	occnp_diam_response_network_total
Recommended Actions	Cause: Severe unable-to-deliver condition. Typically indicates major peer/routing outage, widespread disconnects, or a bad configuration rollout. Diagnostic Information: Identify the worst pod (the alert is already topk(1); list all pods) - % by pod: ((sum by(pod)(rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) / sum by(pod)(rate(ocbsf_diam_response_network_total[2m]))) * 100) Confirm whether multiple pods are impacted: count( ((sum by(pod)(rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) / sum by(pod)(rate(ocbsf_diam_response_network_total[2m]))) * 100) >= 5 ) Confirm whether it’s a volume issue vs ratio artifact. Total by pod: sum by(pod)(rate(ocbsf_diam_response_network_total[2m])) 3002 by pod: sum by(pod)(rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) Break down by any available routing labels (commonly destHost, destRealm, peer, appId, msgType—use what exists in your metric) Example: sum by(pod,destRealm,destHost) (rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])) Correlate with peer connection state / disconnects (Diameter stack metrics/logs) and recent deploy/restart events for the affected pod. identify dominant destination(s) and whether this correlates with connection failures / peer-down alarms. Recovery: Immediately restore routing/peer connectivity or Roll back recent routing/configuration changes. Fail traffic over to healthy peers/realms. Replace unhealthy pods if issue is pod-local. Validate stabilization: 3002 drops well below 10% and end-to-end transaction success recovers.

8.1.1.75 CERTIFICATE_EXPIRY_MINOR

Table 8-91 CERTIFICATE_EXPIRY_MINOR

Field	Details
Description	Certificate expiry in less than 6 months
Summary	Certificate expiry in less than 6 months
Severity	MINOR
Expression	security_cert_x509_expiration_seconds - time() <= 15724800
OID	1.3.6.1.4.1.323.5.3.52.1.2.77
Metric Used	security_cert_x509_expiration_seconds
Recommended Actions	Cause: Some certificate monitored by the security exporter is approaching expiry. Diagnostic Information: Find the minimum time-to-expiry and impacted certs (labels vary) min by (namespace,secret,cert_subject) (security_cert_x509_expiration_seconds - time()) Recovery: Plan renewal/rotation via the approved cert/PKI process; update Kubernetes secrets or keystores; schedule rollout before MAJOR window; verify post-rotation metric increases.

8.1.1.76 CERTIFICATE_EXPIRY_MAJOR

Table 8-92 CERTIFICATE_EXPIRY_MAJOR

Field	Details
Description	Certificate expiry in less than 3 months
Summary	Certificate expiry in less than 3 months
Severity	MAJOR
Expression	security_cert_x509_expiration_seconds - time() <= 7862400
OID	1.3.6.1.4.1.323.5.3.52.1.2.77
Metric Used	security_cert_x509_expiration_seconds
Recommended Actions	Cause: Same as above with higher urgency. Diagnostic Information: Find the minimum time-to-expiry and impacted certs (labels vary) min by (namespace,secret,cert_subject) (security_cert_x509_expiration_seconds - time()) Ensure which workload(s) consume the cert (ingress, egress, internal service, database, etc.). Recovery: Rotate certificate and confirm dependent clients trust the new chain. Consider staged rollout to avoid widespread handshake failures.

8.1.1.77 CERTIFICATE_EXPIRY_CRITICAL

Table 8-93 CERTIFICATE_EXPIRY_CRITICAL

Field	Details
Description	Certificate expiry in less than 1 months
Summary	Certificate expiry in less than 1 months
Severity	CRITICAL
Expression	security_cert_x509_expiration_seconds - time() <= 2592000
OID	1.3.6.1.4.1.323.5.3.52.1.2.77
Metric Used	security_cert_x509_expiration_seconds
Recommended Actions	Cause: Imminent expiry; high likelihood of service disruption at expiry. Diagnostic Information: Find the minimum time-to-expiry and impacted certs (labels vary) min by (namespace,secret,cert_subject) (security_cert_x509_expiration_seconds - time()) Ensure which workload(s) consume the cert (ingress, egress, internal service, database, etc.). Validate exact expiry and blast radius (which services/peers rely on it). Recovery: Emergency rotation: Renew cert, deploy via approved secrets process, reload/restart impacted workloads per runbook, validate: successful TLS handshakes, no connection-failure alerts, and expiration metric resets.

8.1.1.78 UDR_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_MINOR

Table 8-94 UDR_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_MINOR

Field	Details
Description	More than 10% of incoming requests towards UDR-connector is rejected due to request being stale on arrival or during processing by the connector
Summary	More than 10% of incoming requests towards UDR-connector is rejected due to request being stale on arrival or during processing by the connector
Severity	MINOR
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{mode="UDR-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="UDR-C"}[5m])))/(sum by (namespace) (rate(occnp_userservice_inbound_count_total{mode="UDR-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="UDR-C"}[5m]))) * 100 > 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.85
Metric Used	occnp_late_processing_rejection_total
Recommended Actions	Cause: UDR-connector is rejecting a notable share of requests because they are stale on arrival or become stale during processing. Typical causes: request queueing/backlog, downstream UDR slowness/timeouts, connector CPU saturation, threadpool exhaustion, GC pauses, network latency, or burst traffic causing requests to miss SLA/TTL. Diagnostic Information: Split “late arrival” vs “late processing” to locate the bottleneck - Late arrival: sum by(namespace)(rate(occnp_late_arrival_rejection_total{mode="UDR-C"}[5m])) - Late processing: sum by(namespace)(rate(occnp_late_processing_rejection_total{mode="UDR-C"}[5m])) Check inbound traffic rate and burstiness - sum by(namespace)(rate(ocpm_userservice_inbound_count_total{service_resource="udr-service"}[5m])) Correlate with connector/pod congestion signals (CPU/queue) if available - occnp_pod_resource_congestion_state{type="cpu"} / {type="queue"} Check downstream UDR latency/errors (UDR service metrics/logs) and network latency to UDR. Recovery: If “late arrival” dominant: reduce upstream burstiness (rate limiting/backpressure), scale ingress/connector, and investigate network path delays. If “late processing” dominant: scale out connector, increase resources, tune threadpools, and fix hotspots; restore downstream UDR performance. Validate: stale rejection % drops below 10% and end-to-end latency normalizes.

8.1.1.79 UDR_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_MAJOR

Table 8-95 UDR_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_MAJOR

Field	Details
Description	More than 20% of incoming requests towards UDR-connector is rejected due to request being stale on arrival or during processing by the connector
Summary	More than 20% of incoming requests towards UDR-connector is rejected due to request being stale on arrival or during processing by the connector
Severity	MAJOR
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{mode="UDR-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="UDR-C"}[5m])))/(sum by (namespace) (rate(occnp_userservice_inbound_count_total{mode="UDR-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="UDR-C"}[5m]))) * 100 > 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.85
Metric Used	occnp_late_processing_rejection_total
Recommended Actions	Cause: Same as MINOR but indicates material service degradation; often sustained backlog or downstream impairment. Diagnostic Information: Same diagnostics, with urgency: confirm whether this is isolated to a namespace/cluster and whether it aligns with CPU/queue congestion and UDR latency spikes. Recovery: Immediate mitigation: scale out UDR-connector and/or apply temporary rate limits to protect the system. Restore UDR/downstream dependency health. Validate rejection % < 20% quickly, then drive to baseline.

8.1.1.80 UDR_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_CRITICAL

Table 8-96 UDR_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_CRITICAL

Field	Details
Description	More than 30% of incoming requests towards UDR-connector is rejected due to request being stale on arrival or during processing by the connector
Summary	More than 30% of incoming requests towards UDR-connector is rejected due to request being stale on arrival or during processing by the connector
Severity	CRITICAL
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{mode="UDR-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="UDR-C"}[5m])))/(sum by (namespace) (rate(occnp_userservice_inbound_count_total{mode="UDR-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="UDR-C"}[5m]))) * 100 > 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.85
Metric Used	occnp_late_processing_rejection_total
Recommended Actions	Cause: Severe backlog/latency condition; many requests miss TTL/SLA. Likely causes: UDR outage/slowness, connector saturation, or systemic platform/network issue. Diagnostic Information: Same as above, plus check for cascading failures: retries, connection failures, pod restarts/OOM, and saturation across multiple services. Recovery: Treat as incident: immediate traffic shaping/load shedding and rapid scale-out. If downstream UDR is unhealthy, failover if available or restore service urgently. Consider rollback of recent changes if correlated with onset. Validate: stale rejection % < 30% then continue until stable baseline; confirm queues drain and latency normalizes.

8.1.1.81 CHF_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_MINOR

Table 8-97 CHF_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_MINOR

Field	Details
Description	More than 10% of incoming requests towards CHF-connector is rejected due to request being stale on arrival or during processing by the connector
Summary	More than 10% of incoming requests towards CHF-connector is rejected due to request being stale on arrival or during processing by the connector
Severity	MINOR
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{mode="CHF-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="CHF-C"}[5m])))/(sum by (namespace) (rate(occnp_userservice_inbound_count_total{mode="CHF-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="CHF-C"}[5m]))) * 100 > 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.86
Metric Used	occnp_late_processing_rejection_total
Recommended Actions	Cause: - CHF-C stale-request rejection ratio >10% over 5m. - Caused by requests arriving late or becoming late during processing. - Common triggers: tight caller timeout budget, high CHF/UDR/backend latency, or pod backlog/overload. Diagnostic Information: Metrics involved: `occnp_late_processing_rejection_total{mode="CHF-C"}`, `occnp_late_arrival_rejection_total{mode="CHF-C"}`, `ocpm_userservice_inbound_count_total{service_resource="chf-service"}`. - Error observed: `504 Gateway Timeout`, cause `TIMED_OUT_REQUEST`, internal `FAILURE_STALE_REQUEST_REJECT`. Cause value: `((late_processing + late_arrival) / (chf_inbound + late_arrival)) * 100 > 10` over 5m by namespace. Condition: CHF-C only; late-processing can be localized by `stage` (`PRE_PROCESS|PRE_UDR|POST_UDR|PRE_CHF|POST_CHF`) and `operation`. Verification steps: confirm rejection counters rising; check 504/TIMED_OUT_REQUEST logs; break down late-processing by `stage`/`operation`; compare inbound vs rejects; inspect pod saturation and CHF/UDR latency; verify timeout headers/config. Monitoring recommendations: monitor the ratio and raw counters, split by `stage`/`operation`, and correlate with pod health, backlog, downstream latency, and 504s. Recovery: Restore CHF-C latency/backlog: scale pods, relieve load, and recover CHF/UDR dependencies. Increase caller timeout/max-response-time if requests are expiring too early. Verify recovery when rejection rates drop, 504/TIMED_OUT_REQUEST logs stop, and the ratio falls below 10%.

8.1.1.82 CHF_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_MAJOR

Table 8-98 CHF_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_MAJOR

Field	Details
Description	More than 20% of incoming requests towards CHF-connector is rejected due to request being stale on arrival or during processing by the connector
Summary	More than 20% of incoming requests towards CHF-connector is rejected due to request being stale on arrival or during processing by the connector
Severity	MAJOR
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{mode="CHF-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="CHF-C"}[5m])))/(sum by (namespace) (rate(occnp_userservice_inbound_count_total{mode="CHF-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="CHF-C"}[5m]))) * 100 > 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.86
Metric Used	occnp_late_processing_rejection_total
Recommended Actions	Cause: - CHF-C stale-request rejection ratio >20% over 5m. - Caused by requests arriving late or becoming late during processing. - Common triggers: tight caller timeout budget, high CHF/UDR/backend latency, or pod backlog/overload. Diagnostic Information: Metrics involved: `occnp_late_processing_rejection_total{mode="CHF-C"}`, `occnp_late_arrival_rejection_total{mode="CHF-C"}`, `ocpm_userservice_inbound_count_total{service_resource="chf-service"}`. - Error observed: `504 Gateway Timeout`, cause `TIMED_OUT_REQUEST`, internal `FAILURE_STALE_REQUEST_REJECT`. Cause value: `((late_processing + late_arrival) / (chf_inbound + late_arrival)) * 100 > 10` over 5m by namespace. Condition: CHF-C only; late-processing can be localized by `stage` (`PRE_PROCESS|PRE_UDR|POST_UDR|PRE_CHF|POST_CHF`) and `operation`. Verification steps: confirm rejection counters rising; check 504/TIMED_OUT_REQUEST logs; break down late-processing by `stage`/`operation`; compare inbound vs rejects; inspect pod saturation and CHF/UDR latency; verify timeout headers/config. Monitoring recommendations: monitor the ratio and raw counters, split by `stage`/`operation`, and correlate with pod health, backlog, downstream latency, and 504s. Recovery: Restore CHF-C latency/backlog: scale pods, relieve load, and recover CHF/UDR dependencies. Increase caller timeout/max-response-time if requests are expiring too early. Verify recovery when rejection rates drop, 504/TIMED_OUT_REQUEST logs stop, and the ratio falls below 10%.

8.1.1.83 CHF_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_CRITICAL

Table 8-99 CHF_CONNECTOR_STALE_HTTP_REQUEST_CLEANUP_CRITICAL

Field	Details
Description	More than 30% of incoming requests towards CHF-connector is rejected due to request being stale on arrival or during processing by the connector
Summary	More than 30% of incoming requests towards CHF-connector is rejected due to request being stale on arrival or during processing by the connector
Severity	CRITICAL
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{mode="CHF-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="CHF-C"}[5m])))/(sum by (namespace) (rate(occnp_userservice_inbound_count_total{mode="CHF-C"}[5m])) + sum by (namespace) (rate(occnp_late_arrival_rejection_total{mode="CHF-C"}[5m]))) * 100 > 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.86
Metric Used	occnp_late_processing_rejection_total
Recommended Actions	Cause: - CHF-C stale-request rejection ratio >30% over 5m. - Caused by requests arriving late or becoming late during processing. - Common triggers: tight caller timeout budget, high CHF/UDR/backend latency, or pod backlog/overload. Diagnostic Information: Metrics involved: `occnp_late_processing_rejection_total{mode="CHF-C"}`, `occnp_late_arrival_rejection_total{mode="CHF-C"}`, `ocpm_userservice_inbound_count_total{service_resource="chf-service"}`. - Error observed: `504 Gateway Timeout`, cause `TIMED_OUT_REQUEST`, internal `FAILURE_STALE_REQUEST_REJECT`. Cause value: `((late_processing + late_arrival) / (chf_inbound + late_arrival)) * 100 > 10` over 5m by namespace. Condition: CHF-C only; late-processing can be localized by `stage` (`PRE_PROCESS|PRE_UDR|POST_UDR|PRE_CHF|POST_CHF`) and `operation`. Verification steps: confirm rejection counters rising; check 504/TIMED_OUT_REQUEST logs; break down late-processing by `stage`/`operation`; compare inbound vs rejects; inspect pod saturation and CHF/UDR latency; verify timeout headers/config. Monitoring recommendations: monitor the ratio and raw counters, split by `stage`/`operation`, and correlate with pod health, backlog, downstream latency, and 504s. Recovery: Restore CHF-C latency/backlog: scale pods, relieve load, and recover CHF/UDR dependencies. Increase caller timeout/max-response-time if requests are expiring too early. Verify recovery when rejection rates drop, 504/TIMED_OUT_REQUEST logs stop, and the ratio falls below 10%.

8.1.1.84 STALE_BINDING_REQUEST_REJECTION_CRITICAL

Table 8-100 STALE_BINDING_REQUEST_REJECTION_CRITICAL

Field	Details
Description	This alert is triggered when more than 30 % of the received HTTP requests are cancelled due to them being stale (received too late, or took too much time to process them).
Summary	More than 30% of the Binding requests failed with error TIMED_OUT_REQUEST
Severity	Critical
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{microservice=~".*binding"}[5m]))+ sum by (namespace) (rate(occnp_late_arrival_rejection_total{microservice=~".*binding"}[5m])))/ (sum by (namespace) (rate(ocpm_binding_inbound_request_total{microservice=~".*binding"}[5m])))*100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.87
Metric Used	occnp_late_arrival_rejection_total occnp_late_processing_rejection_total ocpm_binding_inbound_request_total
Recommended Actions	Cause: Binding service is rejecting timed-out/stale requests. The alert fires when the 5-minute rate of `occnp_late_processing_rejection_total + occnp_late_arrival_rejection_total` reaches __>= 30%__ of `ocpm_binding_inbound_request_total` in a namespace. In code this occurs when late-arrival or late-processing protection is enabled and `TimeoutHelper.isRequestOnTime(httpHeaders)` returns false, typically meaning requests arrive already expired or become expired during processing (for example while waiting on BSF/DB/internal work). Diagnostic Information: The following metrics are used: `occnp_late_processing_rejection_total` `occnp_late_arrival_rejection_total` `ocpm_binding_inbound_request_total` Error observed: Binding rejects timed-out requests; code throws `ContextBindingTimedOutRequest` Typical response path is `504 Gateway Timeout` Cause value: `cause="TIMED_OUT_REQUEST"` Condition: `((rate(occnp_late_processing_rejection_total)+rate(occnp_late_arrival_rejection_total))/rate(ocpm_binding_inbound_request_total))*100 >= 10` over 5m by `namespace` Verification steps: Check rejection ratio and split by metric: late-arrival vs late-processing. Break down by labels on rejection metrics: `cause`, `operationType`; for late-processing also `stage` Confirm inbound traffic volume with `ocpm_binding_inbound_request_total`. Check binding logs for `Timed out request` / `ContextBindingTimedOutRequest`. Verify whether `lateArrival` / `lateProcessing` handling is enabled in config. If late-processing dominates, inspect slow dependencies/processing around BSF and DB paths; `POST_BSF` stage is explicitly pegged in code Monitoring recommendations: Add dashboards for rejection ratio, absolute rejection rate, and split by `operationType` and `stage` Correlate with `ocpm_binding_inbound_response_total` 5xx/504, HTTP latency metrics, DB latency, and BSF dependency errors/latency. Alert separately on sustained late-arrival and late-processing so upstream timeout issues and internal slowness are distinguishable Recovery: If late-arrival is high: investigate upstream callers/network delay and timeout/header propagation; ensure requests are not reaching binding already expired. If late-processing is high: reduce processing latency in binding and dependencies (especially BSF/DB), check pod saturation, latency spikes, thread starvation, and downstream timeout/error conditions. Review recent config or traffic changes that enabled/shifted late request handling or increased request latency. After mitigation, verify the rejection metrics fall and the ratio drops below 10% for the namespace.

8.1.1.85 STALE_BINDING_REQUEST_REJECTION_MAJOR

Table 8-101 STALE_BINDING_REQUEST_REJECTION_MAJOR

Field	Details
Description	This alert is triggered when more than 20 % of the received HTTP requests are cancelled due to them being stale (received too late, or took too much time to process them).
Summary	More than 20% of the Binding requests failed with error TIMED_OUT_REQUEST
Severity	Major
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{microservice=~".*binding"}[5m]))+ sum by (namespace) (rate(occnp_late_arrival_rejection_total{microservice=~".*binding"}[5m])))/ (sum by (namespace) (rate(ocpm_binding_inbound_request_total{microservice=~".*binding"}[5m])))*100 >= 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.87
Metric Used	occnp_late_arrival_rejection_total occnp_late_processing_rejection_total ocpm_binding_inbound_request_total
Recommended Actions	Cause: Binding service is rejecting timed-out/stale requests. The alert fires when the 5-minute rate of `occnp_late_processing_rejection_total + occnp_late_arrival_rejection_total` reaches __>= 20%__ of `ocpm_binding_inbound_request_total` in a namespace. In code this occurs when late-arrival or late-processing protection is enabled and `TimeoutHelper.isRequestOnTime(httpHeaders)` returns false, typically meaning requests arrive already expired or become expired during processing (for example while waiting on BSF/DB/internal work). Diagnostic Information: The following metrics are used: `occnp_late_processing_rejection_total` `occnp_late_arrival_rejection_total` `ocpm_binding_inbound_request_total` Error observed: Binding rejects timed-out requests; code throws `ContextBindingTimedOutRequest` Typical response path is `504 Gateway Timeout` Cause value: `cause="TIMED_OUT_REQUEST"` Condition: `((rate(occnp_late_processing_rejection_total)+rate(occnp_late_arrival_rejection_total))/rate(ocpm_binding_inbound_request_total))*100 >= 10` over 5m by `namespace` Verification steps: Check rejection ratio and split by metric: late-arrival vs late-processing. Break down by labels on rejection metrics: `cause`, `operationType`; for late-processing also `stage` Confirm inbound traffic volume with `ocpm_binding_inbound_request_total`. Check binding logs for `Timed out request` / `ContextBindingTimedOutRequest`. Verify whether `lateArrival` / `lateProcessing` handling is enabled in config. If late-processing dominates, inspect slow dependencies/processing around BSF and DB paths; `POST_BSF` stage is explicitly pegged in code Monitoring recommendations: Add dashboards for rejection ratio, absolute rejection rate, and split by `operationType` and `stage` Correlate with `ocpm_binding_inbound_response_total` 5xx/504, HTTP latency metrics, DB latency, and BSF dependency errors/latency. Alert separately on sustained late-arrival and late-processing so upstream timeout issues and internal slowness are distinguishable Recovery: If late-arrival is high: investigate upstream callers/network delay and timeout/header propagation; ensure requests are not reaching binding already expired. If late-processing is high: reduce processing latency in binding and dependencies (especially BSF/DB), check pod saturation, latency spikes, thread starvation, and downstream timeout/error conditions. Review recent config or traffic changes that enabled/shifted late request handling or increased request latency. After mitigation, verify the rejection metrics fall and the ratio drops below 10% for the namespace.

8.1.1.86 STALE_BINDING_REQUEST_REJECTION_MINOR

Table 8-102 STALE_BINDING_REQUEST_REJECTION_MINOR

Field	Details
Description	This alert is triggered when more than 10 % of the received HTTP requests are cancelled due to them being stale (received too late, or took too much time to process them).
Summary	More than 10% of the Binding requests failed with error TIMED_OUT_REQUEST
Severity	Minor
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{microservice=~".*binding"}[5m]))+ sum by (namespace) (rate(occnp_late_arrival_rejection_total{microservice=~".*binding"}[5m])))/ (sum by (namespace) (rate(ocpm_binding_inbound_request_total{microservice=~".*binding"}[5m])))*100 >= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.87
Metric Used	occnp_late_arrival_rejection_total occnp_late_processing_rejection_total ocpm_binding_inbound_request_total
Recommended Actions	Cause: Binding service is rejecting timed-out/stale requests. The alert fires when the 5-minute rate of `occnp_late_processing_rejection_total + occnp_late_arrival_rejection_total` reaches __>= 10%__ of `ocpm_binding_inbound_request_total` in a namespace. In code this occurs when late-arrival or late-processing protection is enabled and `TimeoutHelper.isRequestOnTime(httpHeaders)` returns false, typically meaning requests arrive already expired or become expired during processing (for example while waiting on BSF/DB/internal work). Diagnostic Information: The following metrics are used: `occnp_late_processing_rejection_total` `occnp_late_arrival_rejection_total` `ocpm_binding_inbound_request_total` Error observed: Binding rejects timed-out requests; code throws `ContextBindingTimedOutRequest` Typical response path is `504 Gateway Timeout` Cause value: `cause="TIMED_OUT_REQUEST"` Condition: `((rate(occnp_late_processing_rejection_total)+rate(occnp_late_arrival_rejection_total))/rate(ocpm_binding_inbound_request_total))*100 >= 10` over 5m by `namespace` Verification steps: Check rejection ratio and split by metric: late-arrival vs late-processing. Break down by labels on rejection metrics: `cause`, `operationType`; for late-processing also `stage` Confirm inbound traffic volume with `ocpm_binding_inbound_request_total`. Check binding logs for `Timed out request` / `ContextBindingTimedOutRequest`. Verify whether `lateArrival` / `lateProcessing` handling is enabled in config. If late-processing dominates, inspect slow dependencies/processing around BSF and DB paths; `POST_BSF` stage is explicitly pegged in code Monitoring recommendations: Add dashboards for rejection ratio, absolute rejection rate, and split by `operationType` and `stage` Correlate with `ocpm_binding_inbound_response_total` 5xx/504, HTTP latency metrics, DB latency, and BSF dependency errors/latency. Alert separately on sustained late-arrival and late-processing so upstream timeout issues and internal slowness are distinguishable Recovery: If late-arrival is high: investigate upstream callers/network delay and timeout/header propagation; ensure requests are not reaching binding already expired. If late-processing is high: reduce processing latency in binding and dependencies (especially BSF/DB), check pod saturation, latency spikes, thread starvation, and downstream timeout/error conditions. Review recent config or traffic changes that enabled/shifted late request handling or increased request latency. After mitigation, verify the rejection metrics fall and the ratio drops below 10% for the namespace.

8.1.1.87 POLICYDS_PREEXPIRY_RESUBSCRIBE_FAILURE_MINOR

Table 8-103 POLICYDS_PREEXPIRY_RESUBSCRIBE_FAILURE_MINOR

Field	Details
Description	If 30% to 50% of subscriptions which are in PRE_EXPIRY period fail to resubscribe, this alert will be raised.
Summary	If 30% to 50% of subscriptions which are in PRE_EXPIRY period fail to resubscribe, this alert will be raised.
Severity	Minor
Expression	(sum by (microservice, namespace) (rate(occnp_policy_data_resubscription_response_total{expiryStatus="PRE_EXPIRY",response!~"2.*"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_policy_data_resubscription_response_total{expiryStatus="PRE_EXPIRY"}[5m]))) * 100 > 30 <= 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.129
Metric Used	occnp_policy_data_resubscription_response_total
Recommended Actions	The alert will be raised if policy data subscription is eligible for re-subscription but during resubscription errors observed. Could be due to timeout/ any other error from UDR or UDR-Connector. If 4xx (check occnp_policy_data_resubscription_response_total metrics) is received it can be due to subscription deleted from UDR. Check the communication between PDS and UDR. The alert will be cleared after resusbcription succeess rate > 70%

8.1.1.88 POLICYDS_PREEXPIRY_RESUBSCRIBE_FAILURE_MAJOR

Table 8-104 POLICYDS_PREEXPIRY_RESUBSCRIBE_FAILURE_MAJOR

Field	Details
Description	If 50% to 70% of subscriptions which are in PRE_EXPIRY period fail to resubscribe, this alert will be raised.
Summary	If 50% to 70% of subscriptions which are in PRE_EXPIRY period fail to resubscribe, this alert will be raised.
Severity	Major
Expression	(sum by (microservice, namespace) (rate(occnp_policy_data_resubscription_response_total{expiryStatus="PRE_EXPIRY",response!~"2.*"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_policy_data_resubscription_response_total{expiryStatus="PRE_EXPIRY"}[5m]))) * 100 > 50 <= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.129
Metric Used	occnp_policy_data_resubscription_response_total
Recommended Actions	The alert will be raised if policy data subscription is eligible for re-subscription but during resubscription errors observed. Could be due to timeout/ any other error from UDR or UDR-Connector. If 4xx (check occnp_policy_data_resubscription_response_total metrics) is received it can be due to subscription deleted from UDR. Check the communication between PDS and UDR. The alert will be cleared after resusbcription succeess rate > 50%

8.1.1.89 POLICYDS_PREEXPIRY_RESUBSCRIBE_FAILURE_CRITICAL

Table 8-105 POLICYDS_PREEXPIRY_RESUBSCRIBE_FAILURE_CRITICAL

Field	Details
Description	If 70% of subscriptions which are in PRE_EXPIRY period fail to resubscribe, this alert will be raised.
Summary	If 70% of subscriptions which are in PRE_EXPIRY period fail to resubscribe, this alert will be raised.
Severity	Critical
Expression	(sum by (microservice, namespace) (rate(occnp_policy_data_resubscription_response_total{expiryStatus="PRE_EXPIRY",response!~"2.*"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_policy_data_resubscription_response_total{expiryStatus="PRE_EXPIRY"}[5m]))) * 100 > 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.129
Metric Used	occnp_policy_data_resubscription_response_total
Recommended Actions	The alert will be raised if policy data subscription is eligible for re-subscription but during resubscription errors observed. Could be due to timeout/ any other error from UDR or UDR-Connector. If 4xx (check occnp_policy_data_resubscription_response_total metrics) is received it can be due to subscription deleted from UDR. Check the communication between PDS and UDR. The alert will be cleared after resusbcription succeess rate > 30%

8.1.1.90 POLICYDS_EXPIRED_SUBSCRIPTION

Table 8-106 POLICYDS_EXPIRED_SUBSCRIPTION

Field	Details
Description	If more than 10% of audited subscriptions are expired, this alert will be raised.
Summary	If more than 10% of audited subscriptions are expired, this alert will be raised.
Severity	Major
Expression	(sum by (microservice, namespace) (rate(occnp_policy_data_resubscription_request_total{expiryStatus="EXPIRED"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_policy_data_resubscription_request_total[5m]))) * 100 > 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.130
Metric Used	occnp_policy_data_resubscription_request_total
Recommended Actions	For any additional guidance, contact My Oracle Support (https://support.oracle.com).

8.1.1.91 LDAP_PEER_CONNECTION_LOST

Table 8-107 LDAP_PEER_CONNECTION_LOST

Field	Details
Name in Alert Yaml File	LDAP_PEER_CONNECTION_LOST
Description	This alert is triggered when the LDAP Gateway loses connection to its LDAP peer(s). It is based on the value of the occnp_ldap_conn_total metric falling to zero. The connection re-attempt and alert clearance behavior is governed by a new configuration parameter, LDAP_CONNECTION_REVERT_DELAY.
Summary	LDAP Gateway loses connection to its LDAP peer(s).
Severity	major
Expression	sum by (namespace,peer)(occnp_ldap_conn_total) == 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.113
Metric Used	occnp_ldap_conn_total
Recommended Actions	Verify that the LDAP server is running and connectivity between the PCF and LDAP peers is available. If LDAP is reachable, check the configured LDAP_CONNECTION_REVERT_DELAY value since reconnection attempts and alert clearance depend on this setting.

8.1.1.92 MEMORY_USAGE_PER_PRE_POD_ABOVE_THRESHOLD_MINOR

Table 8-108 MEMORY_USAGE_PER_PRE_POD_ABOVE_THRESHOLD_MINOR

Field	Details
Description	Memory usage for PRE Service Pod {{$labels.namespace}}-{{$labels.container}}-{{$labels.pod}} is above 75. Current value is: {{ $value }}'
Summary	Memory usage for PRE Service Pod {{$labels.namespace}}-{{$labels.container}}-{{$labels.pod}} service is above 75
Severity	Minor
Expression	(100 * max by (container, pod, namespace) (container_memory_working_set_bytes{container=~".*pre-service.*"})/ sum by (container, pod, namespace) (kube_pod_container_resource_limits{resource="memory",container=~".*pre-service.*"})) >= 75 < 85
OID	1.3.6.1.4.1.323.5.3.52.1.2.131
Recommended Actions	This is a PRE pod level alert which fires when memory usage of PRE pod equals to or is greater than 75 but less than 85%. The alert will be cleared once the memory usage reduces below the threshold.

8.1.1.93 MEMORY_USAGE_PER_PRE_POD_ABOVE_THRESHOLD_MAJOR

Table 8-109 MEMORY_USAGE_PER_PRE_POD_ABOVE_THRESHOLD_MAJOR

Field	Details
Description	Memory usage for PRE Service Pod {{$labels.namespace}}-{{$labels.container}}-{{$labels.pod}} service is above 85. Current value is: {{ $value }}
Summary	Memory usage for PRE Service Pod {{$labels.namespace}}-{{$labels.container}}-{{$labels.pod}} service is above 85
Severity	Major
Expression	(100 * max by (container, pod, namespace) (container_memory_working_set_bytes{container=~".*pre-service.*"})/ sum by (container, pod, namespace) (kube_pod_container_resource_limits{resource="memory",container=~".*pre-service.*"})) >= 85 < 95
OID	1.3.6.1.4.1.323.5.3.52.1.2.131
Recommended Actions	This is a PRE pod level alert which fires when memory usage of PRE pod equals to or is greater than 85 but less than 95%. The alert will be cleared once the memory usage reduces below the threshold.

8.1.1.94 MEMORY_USAGE_PER_PRE_POD_ABOVE_THRESHOLD_CRITICAL

Table 8-110 MEMORY_USAGE_PER_PRE_POD_ABOVE_THRESHOLD_CRITICAL

Field	Details
Description	Memory usage for PRE Service Pod {{$labels.namespace}}-{{$labels.container}}-{{$labels.pod}} service is above 95. Current value is: {{ $value }}
Summary	Memory usage for PRE Service Pod {{$labels.namespace}}-{{$labels.container}}-{{$labels.pod}} service is above 95
Severity	Critical
Expression	(100 * max by (container, pod, namespace) (container_memory_working_set_bytes{container=~".*pre-service.*"})/ sum by (container, pod, namespace) (kube_pod_container_resource_limits{resource="memory",container=~".*pre-service.*"})) >= 95
OID	1.3.6.1.4.1.323.5.3.52.1.2.131
Recommended Actions	This is a PRE pod level alert which fires when memory usage of PRE pod equals to or is greater than 95%. The alert will be cleared once the memory usage reduces below the threshold.

8.1.1.95 SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_MINOR_THRESHOLD

Table 8-111 SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	Notification Transaction Error exceeds the minor threshold limit for a given Subscriber Notification server.
Summary	Transaction Error exceeds the minor threshold limit for a given Subscriber Notification server.
Severity	Minor
Expression	sum by(notificationEndpoint)(increase(http_notification_response_total{responseCode!~"2.*"}[2m])) > 500 and sum by(notificationEndpoint)(increase(http_notification_response_total{responseCode!~"2.*"}[2m])) <= 750
OID	1.3.6.1.4.1.323.5.3.44.1.2.42
Recommended Actions	Cause: SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_MINOR_THRESHOLD indicates a critical increase in notification transaction failures for a given notificationEndpoint. The alert fires when non-2xx HTTP responses exceed 500 but remains lesser than 750 within 2 minutes (per endpoint), suggesting the Subscriber Notification server is failing to deliver notifications successfully. Diagnostic Information: Identify which response codes dominate (for example, 4xx vs 5xx vs 429) for the affected notificationEndpoint. Confirm whether the issue is isolated to one endpoint or widespread across endpoints (compare top failing endpoints). Correlate with latency and timeout signals (HTTP client timeouts, upstream duration histograms if available). Check Subscriber Notification server logs for: repeated errors, payload validation failures, auth errors, TLS handshake failures, connection timeouts/resets. Validate downstream endpoint health: DNS resolution, cert validity, firewall rules, rate limiting (429), maintenance windows. Check internal dependencies used for notification delivery (DB/queue/service mesh): errors, saturation, connection pool exhaustion. Review recent deployments/configuration changes affecting: endpoint routing, auth credentials, payload format, retry policy, timeout policy. Recovery: If failures are 4xx, fix payload/auth/configuration issues (credentials, headers, schema) and reprocess if applicable. If failures are 5xx/timeouts, stabilize dependencies and reduce retry amplification (add backoff/jitter, cap concurrency). If downstream endpoint is unhealthy or rate-limiting (429), coordinate with endpoint owner, apply throttling, or pause/retry with backoff. If resource saturation is contributing, scale out the service or adjust resource limits/queues. Roll back recent changes if a regression is suspected. Clears when non-2xx responses for each notificationEndpoint drop below 500 over 2 minutes.

8.1.1.96 SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_MAJOR_THRESHOLD

Table 8-112 SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	Notification Transaction Error exceeds the major threshold limit for a given Subscriber Notification server
Summary	Transaction Error exceeds the major threshold limit for a given Subscriber Notification server
Severity	Major
Expression	sum by(notificationEndpoint)(increase(http_notification_response_total{responseCode!~"2.*"}[2m])) > 750 and sum by(notificationEndpoint)(increase(http_notification_response_total{responseCode!~"2.*"}[2m])) <= 1000
OID	1.3.6.1.4.1.323.5.3.44.1.2.42
Recommended Actions	Cause: SUBSCRIBER_NOTIFICATION_ERROR_EXCEEDS_MAJOR_THRESHOLD indicates a critical increase in notification transaction failures for a given notificationEndpoint. The alert fires when non-2xx HTTP responses exceed 750 but less than1000 within 2 minutes (per endpoint), suggesting the Subscriber Notification server is failing to deliver notifications successfully. Diagnostic Information: Identify which response codes dominate (for example, 4xx vs 5xx vs 429) for the affected notificationEndpoint. Confirm whether the issue is isolated to one endpoint or widespread across endpoints (compare top failing endpoints). Correlate with latency and timeout signals (HTTP client timeouts, upstream duration histograms if available). Check Subscriber Notification server logs for: repeated errors, payload validation failures, auth errors, TLS handshake failures, connection timeouts/resets. Validate downstream endpoint health: DNS resolution, cert validity, firewall rules, rate limiting (429), maintenance windows. Check internal dependencies used for notification delivery (DB/queue/service mesh): errors, saturation, connection pool exhaustion. Review recent deployments/configuration changes affecting: endpoint routing, auth credentials, payload format, retry policy, timeout policy. Recovery: If failures are 4xx, fix payload/auth/configuration issues (credentials, headers, schema) and reprocess if applicable. If failures are 5xx/timeouts, stabilize dependencies and reduce retry amplification (add backoff/jitter, cap concurrency). If downstream endpoint is unhealthy or rate-limiting (429), coordinate with endpoint owner, apply throttling, or pause/retry with backoff. If resource saturation is contributing, scale out the service or adjust resource limits/queues. Roll back recent changes if a regression is suspected. Treat sustained MAJOR as incident-level, especially if impacting multiple endpoints or causing backlog growth. Clears when non-2xx responses for each notificationEndpoint drop below 750 over 2 minutes.

8.1.1.97 RAA_RX_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-113 RAA_RX_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	RAA Rx timeout count exceeds the critical threshold limit
Summary	Alert RAA_RX_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236"}[5m])) * 100 > 90
Condition	The timeout rate of RAA Rx messages has exceeded the configured threshold limit.
OID	1.3.6.1.4.1.323.5.3.52.1.2.36
Metric Used	occnp_diam_response_local_total{msgType="RAA", appType="Rx", responseCode!~"timeout"}
Recommended Actions	Cause: Rx RAA timeouts are near-total, meaning most RAR→RAA transactions are not completing within timer. Typical causes: Rx peer AF unreachable/slow, network loss/latency, Diameter connection instability, routing/realm misroute, or local saturation/backlog causing timeouts. Diagnostic Information: Validate ratio + ensure meaningful traffic volume. Total RAA(Rx): sum(rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA"}[5m])) Timeout RAA(Rx): sum(rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA",responseCode="timeout"}[5m])) Identify top affected peers (use labels you have: destHost, destRealm, reqDestHost, reqDestRealm, origHost). sum by (destHost,destRealm) (rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA",responseCode="timeout"}[5m])) Confirm whether timeouts are specific to Rx/RAA or broader - sum by (msgType,appId) (rate(occnp_diam_response_local_total{responseCode="timeout"}[5m])). Check backlog/congestion indicators. occnp_diam_pending_requests (or equivalent) sum(rate(occnp_stale_diam_request_cleanup_total[5m])) (or equivalent) Correlate with peer connectivity signals (Diameter stack logs/metrics): watchdog failures, disconnect/reconnect loops, send failures. Recovery: If isolated to one AF/peer: failover/reroute Rx traffic if supported; restore peer/network health. If local saturation/backlog: scale out, reduce load, fix CPU/memory/thread pool/queue bottlenecks; ensure pending drains. If network/transport issue: remediate SCTP/TCP stability, latency/loss, ACL/firewall, MTU/TLS settings; stabilize sessions. Only adjust timers after confirming the peer is healthy and latency SLOs justify it. Validate: Rx RAA timeout % drops < threshold and success responses recover.

8.1.1.98 RAA_RX_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-114 RAA_RX_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	RAA Rx timeout count exceeds the major threshold limit
Summary	Alert RAA_RX_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.36
Metric Used	occnp_diam_response_local_total{msgType="RAA", appType="Rx", responseCode!~"timeout"}
Recommended Actions	Cause: Rx RAA timeouts are near-total, meaning most RAR→RAA transactions are not completing within timer. Typical causes: Rx peer AF unreachable/slow, network loss/latency, Diameter connection instability, routing/realm misroute, or local saturation/backlog causing timeouts. Partial outage/degradation trending toward critical (intermittent peer slowness, emerging congestion, routing instability). Diagnostic Information: Validate ratio + ensure meaningful traffic volume. Total RAA(Rx): sum(rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA"}[5m])) Timeout RAA(Rx): sum(rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA",responseCode="timeout"}[5m])) Identify top affected peers (use labels you have: destHost, destRealm, reqDestHost, reqDestRealm, origHost). sum by (destHost,destRealm) (rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA",responseCode="timeout"}[5m])) Confirm whether timeouts are specific to Rx/RAA or broader - sum by (msgType,appId) (rate(occnp_diam_response_local_total{responseCode="timeout"}[5m])). Check backlog/congestion indicators. occnp_diam_pending_requests (or equivalent) sum(rate(occnp_stale_diam_request_cleanup_total[5m])) (or equivalent) Correlate with peer connectivity signals (Diameter stack logs/metrics): watchdog failures, disconnect/reconnect loops, send failures. Trend by peer to find the first reason: - topk(5, sum by (destHost,destRealm) (rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA",responseCode="timeout"}[5m]))) Recovery: If isolated to one AF/peer: failover/reroute Rx traffic if supported; restore peer/network health. If local saturation/backlog: scale out, reduce load, fix CPU/memory/thread pool/queue bottlenecks; ensure pending drains. If network/transport issue: remediate SCTP/TCP stability, latency/loss, ACL/firewall, MTU/TLS settings; stabilize sessions. Only adjust timers after confirming the peer is healthy and latency SLOs justify it. Validate: Rx RAA timeout % drops < threshold and success responses recover. Emphasis on preventing escalation: restore degraded peer, preemptive reroute, relieve load, confirm timeouts return to baseline.

8.1.1.99 RAA_RX_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-115 RAA_RX_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	RAA Rx timeout count exceeds the minor threshold limit
Summary	Alert RAA_RX_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appId="16777236"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.36
Metric Used	occnp_diam_response_local_total{msgType="RAA", appType="Rx", responseCode!~"timeout"}
Recommended Actions	Cause: Early warning of Rx RAA timeout increase; may be bursty, peer-specific, or correlated with traffic spikes. Diagnostic Information: Validate ratio + ensure meaningful traffic volume. Total RAA(Rx): sum(rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA"}[5m])) Timeout RAA(Rx): sum(rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA",responseCode="timeout"}[5m])) Identify top affected peers (use labels you have: destHost, destRealm, reqDestHost, reqDestRealm, origHost). sum by (destHost,destRealm) (rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA",responseCode="timeout"}[5m])) Confirm whether timeouts are specific to Rx/RAA or broader - sum by (msgType,appId) (rate(occnp_diam_response_local_total{responseCode="timeout"}[5m])). Check backlog/congestion indicators. occnp_diam_pending_requests (or equivalent) sum(rate(occnp_stale_diam_request_cleanup_total[5m])) (or equivalent) Correlate with peer connectivity signals (Diameter stack logs/metrics): watchdog failures, disconnect/reconnect loops, send failures. Trend by peer to find the first reason: - topk(5, sum by (destHost,destRealm) (rate(occnp_diam_response_local_total{appId="16777236",msgType="RAA",responseCode="timeout"}[5m]))) Compare 5m vs 30m to see persistence: - (example) avg_over_time((sum(rate(…timeout…[5m])) / sum(rate(…total…[5m])) * 100)[30m:]) Recovery: If isolated to one AF/peer: failover/reroute Rx traffic if supported; restore peer/network health. If local saturation/backlog: scale out, reduce load, fix CPU/memory/thread pool/queue bottlenecks; ensure pending drains. If network/transport issue: remediate SCTP/TCP stability, latency/loss, ACL/firewall, MTU/TLS settings; stabilize sessions. Only adjust timers after confirming the peer is healthy and latency SLOs justify it. Validate: Rx RAA timeout % drops < threshold and success responses recover. Emphasis on preventing escalation: restore degraded peer, preemptive reroute, relieve load, confirm timeouts return to baseline. Recovery: Proactive checks: peer health/connectivity, reroute if one peer is degrading, address rising resource use/backlog before MAJOR/CRITICAL.

8.1.1.100 PCF_STATE_NON_FUNCTIONAL_CRITICAL

Table 8-116 PCF_STATE_NON_FUNCTIONAL_CRITICAL

Field	Details
Description	Policy is in non functional state due to DB cluster state down.
Summary	Policy is in non functional state due to DB cluster state down.
Severity	Critical
Expression	appinfo_nfDbFunctionalState_current{nfDbFunctionalState="Not_Running"} == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.102
Metric Used	appinfo_nfDbFunctionalState_current
Recommended Actions	For any additional guidance, contact My Oracle Support.

8.1.1.101 POD_PROTECTION_BY_RATELIMIT_REJECTED_REQUEST_EGW

Table 8-117 POD_PROTECTION_BY_RATELIMIT_REJECTED_REQUEST_EGW

Field	Details
Description	Egress Gateway traffic is getting rejected more than 1% because of ratelimiting.
Summary	EGW traffic getting rejected more than 1% because of ratelimiting
Severity	Major
Expression	(sum by (namespace, pod) (rate(oc_egressgateway_http_request_ratelimit_values_total{Allowed="false",microservice="occnp-occnp-egress-gateway"}[5m]))) / sum by (namespace, pod)(rate(oc_egressgateway_http_request_ratelimit_values_total{microservice="occnp-occnp-egress-gateway"}[5m])) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.114
Metric Used	oc_egressgateway_http_request_ratelimit_values_total
Recommended Actions	The alert is cleared when the failure rate goes below 1% of total tps.

8.1.1.102 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_MINOR

Table 8-118 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_MINOR

Field	Details
Description	UDR returning with POST subscribe response but without user data for SM as part of immediate reporting occurring above 10% for service {{$labels.microservice}} in {{$labels.namespace}} ( current value: {{ $value }} % )
Summary	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting
Severity	Minor
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",imm_reports_present="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post"}[5m]))) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.127
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. The missing SM user data check is based on: service_subresource = "sm-data" (indicates the UDR POST was to get SM user data from UDR) operation_type = "POST" (indicates this is a POST call) imm_reports_present = "false" (indicates no SM user data was returned from UDR as part of the Immediate Reporting capability) If these metric dimensions are satisfied, then the alarm will trigger. Alarm Condition: More than or equal to 10% but less than 20% of the traffic: UDR returned a POST Subscribe response without SM user data as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR in the POST REST API request payload has the 30th byte set to 1 when converted to hex (e.g., 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Inform UDR Operator If the above points are validated and SM user data is still not retrieved, inform the UDR operators to verify whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.1.103 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_MAJOR

Table 8-119 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_MAJOR

Field	Details
Description	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting
Summary	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting
Severity	Major
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",imm_reports_present="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post"}[5m]))) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.127
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. The missing SM user data check is based on: service_subresource = "sm-data" (to indicate the UDR POST was to get SM user data from UDR) operation_type = "POST" (to determine this is a POST call) imm_reports_present = "false" (to indicate no SM user data was returned from UDR as part of the Immediate Reporting capability) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 20% but less than 30% of the traffic: UDR returned a POST Subscribe response without user data for SM as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR in the POST REST API request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Inform UDR Operator If the above points are validated and still no SM user data is retrieved, inform the UDR operators whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.1.104 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_CRITICAL

Table 8-120 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_CRITICAL

Field	Details
Description	More than 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Summary	More than 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Severity	Critical
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx",immediate_report_pcc="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}[5m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.128
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: UDR profile/capability mismatch, partial rollout/config drift across UDR instances, or negotiation disabled/missing in returned `supportedFeatures`. Diagnostic Information: Verification steps: Query `occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}` split by `immediate_report_pcc`, `nf_name`, and `namespace`. Check whether one/few UDR `nf_name` values dominate the `immediate_report_pcc="false"` bucket. Review PCF logs around `handleSuccessResponseForImmRep` / UDR POST response for messages showing returned `supportedFeatures` does not set ImmediateReportPcc. Confirm PCF-side enablement: `optimizeSubscribe=true`, global `UDR.IMMREP` enabled, and Helm `enable.immediate.reporting.sm-data=true`. Compare UDR profile capability / returned `supportedFeatures` across healthy vs affected UDRs. Monitoring recommendations: Break down the metric by `nf_name` to isolate bad UDR instances. Also watch `occnp_immrep_request_total` vs `occnp_immrep_response_total` for request/response symmetry. Track `imm_reports_present` and `error_cause` for adjacent degradation patterns. Alert separately if the ratio increases beyond 30% or becomes concentrated on a single UDR. Recovery: Verify and correct UDR capability/config so returned `supportedFeatures` includes ImmediateReportPcc for `sm-data` subscriptions. If only specific UDR instances are affected, drain/remove or fix those instances and confirm the `immediate_report_pcc="false"` ratio drops. If intentional/not supported in the environment, disable `sm-data` immediate reporting consistently to avoid mixed negotiation behavior. After change, confirm the alert ratio returns below threshold and `immediate_report_pcc="true"` dominates successful POST responses.

8.1.1.105 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_MINOR

Table 8-121 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_MINOR

Field	Details
Description	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Summary	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Severity	Minor
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx",immediate_report_pcc="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}[5m]))) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.128
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: UDR profile/capability mismatch, partial rollout/config drift across UDR instances, or negotiation disabled/missing in returned `supportedFeatures`. Diagnostic Information: Verification steps: Query `occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}` split by `immediate_report_pcc`, `nf_name`, and `namespace`. Check whether one/few UDR `nf_name` values dominate the `immediate_report_pcc="false"` bucket. Review PCF logs around `handleSuccessResponseForImmRep` / UDR POST response for messages showing returned `supportedFeatures` does not set ImmediateReportPcc. Confirm PCF-side enablement: `optimizeSubscribe=true`, global `UDR.IMMREP` enabled, and Helm `enable.immediate.reporting.sm-data=true`. Compare UDR profile capability / returned `supportedFeatures` across healthy vs affected UDRs. Monitoring recommendations: Break down the metric by `nf_name` to isolate bad UDR instances. Also watch `occnp_immrep_request_total` vs `occnp_immrep_response_total` for request/response symmetry. Track `imm_reports_present` and `error_cause` for adjacent degradation patterns. Alert separately if the ratio increases beyond 20% or becomes concentrated on a single UDR. Recovery: Verify and correct UDR capability/config so returned `supportedFeatures` includes ImmediateReportPcc for `sm-data` subscriptions. If only specific UDR instances are affected, drain/remove or fix those instances and confirm the `immediate_report_pcc="false"` ratio drops. If intentional/not supported in the environment, disable `sm-data` immediate reporting consistently to avoid mixed negotiation behavior. After change, confirm the alert ratio returns below threshold and `immediate_report_pcc="true"` dominates successful POST responses.

8.1.1.106 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_MAJOR

Table 8-122 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_MAJOR

Field	Details
Description	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Summary	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Severity	Major
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx",immediate_report_pcc="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}[5m]))) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.128
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: UDR profile/capability mismatch, partial rollout/config drift across UDR instances, or negotiation disabled/missing in returned `supportedFeatures`. Diagnostic Information: Verification steps: Query `occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}` split by `immediate_report_pcc`, `nf_name`, and `namespace`. Check whether one/few UDR `nf_name` values dominate the `immediate_report_pcc="false"` bucket. Review PCF logs around `handleSuccessResponseForImmRep` / UDR POST response for messages showing returned `supportedFeatures` does not set ImmediateReportPcc. Confirm PCF-side enablement: `optimizeSubscribe=true`, global `UDR.IMMREP` enabled, and Helm `enable.immediate.reporting.sm-data=true`. Compare UDR profile capability / returned `supportedFeatures` across healthy vs affected UDRs. Monitoring recommendations: Break down the metric by `nf_name` to isolate bad UDR instances. Also watch `occnp_immrep_request_total` vs `occnp_immrep_response_total` for request/response symmetry. Track `imm_reports_present` and `error_cause` for adjacent degradation patterns. Alert separately if the ratio increases beyond 20% to 30% or becomes concentrated on a single UDR. Recovery: Verify and correct UDR capability/config so returned `supportedFeatures` includes ImmediateReportPcc for `sm-data` subscriptions. If only specific UDR instances are affected, drain/remove or fix those instances and confirm the `immediate_report_pcc="false"` ratio drops. If intentional/not supported in the environment, disable `sm-data` immediate reporting consistently to avoid mixed negotiation behavior. After change, confirm the alert ratio returns below threshold and `immediate_report_pcc="true"` dominates successful POST responses.

8.1.1.107 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_CRITICAL

Table 8-123 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_CRITICAL

Field	Details
Description	More than 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Summary	More than 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Severity	Critical
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",immediate_report_pcc="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post"}[5m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.128
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. The failed feature negotiation is based on: service_subresource = "sm-data" (indicates the UDR POST was to get AM user data from UDR) operation_type = "POST" (indicates this is a POST call) immediate_report_pcc = "false" (indicates that no feature negotiation happened with UDR on the ImmReportPcc feature) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 30% of the traffic: UDR returned a POST Subscribe response with failed feature negotiation for SM as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in its request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Inform UDR Operator If the above points are validated and SM user data is still not retrieved: Inform the UDR operators. Ask them to verify whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.1.108 DNN_REPLACEMENT_MANDATORY_IE_MISSING_ABOVE_MINOR_PERCENT

Table 8-124 DNN_REPLACEMENT_MANDATORY_IE_MISSING_ABOVE_MINOR_PERCENT

Field	Details
Description	20% or more but less than 50% of AM Update requests are rejected due to Mandatory IE Missing for DNN Replacement Control
Summary	20% or more but less than 50% of AM Update requests are rejected due to Mandatory IE Missing for DNN Replacement Control
Severity	Minor
Expression	(sum by (namespace) (rate(occnp_dnn_replacement_total{mandatory_ie_missing!=""}[5m]))) / (sum by (namespace) (rate(occnp_dnn_replacement_total[5m]))) * 100 >= 20 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.137
Metric Used	occnp_dnn_replacement_total
Recommended Actions	Cause: This alert is triggered when the percentage of AM Update requests are rejected due to Mandatory IE Missing for DNN Replacement Control. When occnp_dnn_replacement_total{mandatory_ie_missing!="") increases, it indicates: AMF is sending invalid request body with PCRT (SMF_SELECT_CH or ALLOWED_NSSAI_CH) but allowedSnssais and/or snssai/dnn it's missing. Diagnostics Information: Verification steps: Send a valid AM Update request including all mandatory IEs for DnnReplacement. Ensure allowedSnssais/snssai/dnn are present and that DnnReplacement is negotiated. Confirm the request succeeds. Verify that the 400/MANDATORY_IE_MISSING ratio drops below the alert threshold within one evaluation window. Recovery: Check AMF configuration and behavior as why invalid values are sent. Investigate configuration mismatch between PCF and AMF. Escalate if sustained at Major or Critical level.

8.1.1.109 DNN_REPLACEMENT_MANDATORY_IE_MISSING_ABOVE_MAJOR_PERCENT

Table 8-125 DNN_REPLACEMENT_MANDATORY_IE_MISSING_ABOVE_MAJOR_PERCENT

Field	Details
Description	50% or more but less than 75% of AM Update requests are rejected due to Mandatory IE Missing for DNN Replacement Control
Summary	50% or more but less than 75% of AM Update requests are rejected due to Mandatory IE Missing for DNN Replacement Control
Severity	Major
Expression	(sum by (namespace) (rate(occnp_dnn_replacement_total{mandatory_ie_missing!=""}[5m]))) / (sum by (namespace) (rate(occnp_dnn_replacement_total[5m]))) * 100 >= 50 < 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.137
Metric Used	occnp_dnn_replacement_total
Recommended Actions	Cause: This alert is triggered when the percentage of AM Update requests are rejected due to Mandatory IE Missing for DNN Replacement Control. When occnp_dnn_replacement_total{mandatory_ie_missing!="") increases, it indicates: AMF is sending invalid request body with PCRT (SMF_SELECT_CH or ALLOWED_NSSAI_CH) but allowedSnssais and/or snssai/dnn it's missing. Diagnostics Information: Verification steps: Send a valid AM Update request including all mandatory IEs for DnnReplacement. Ensure allowedSnssais/snssai/dnn are present and that DnnReplacement is negotiated. Confirm the request succeeds. Verify that the 400/MANDATORY_IE_MISSING ratio drops below the alert threshold within one evaluation window. Recovery: Check AMF configuration and behavior as why invalid values are sent. Investigate configuration mismatch between PCF and AMF. Escalate if sustained at Major or Critical level.

8.1.1.110 DNN_REPLACEMENT_MANDATORY_IE_MISSING_ABOVE_CRITICAL_PERCENT

Table 8-126 DNN_REPLACEMENT_MANDATORY_IE_MISSING_ABOVE_CRITICAL_PERCENT

Field	Details
Description	More than 75% of AM Update requests are rejected due to Mandatory IE Missing for DNN Replacement Control
Summary	More than 75% of AM Update requests are rejected due to Mandatory IE Missing for DNN Replacement Control
Severity	Critical
Expression	(sum by (namespace) (rate(occnp_dnn_replacement_total{mandatory_ie_missing!=""}[5m]))) / (sum by (namespace) (rate(occnp_dnn_replacement_total[5m]))) * 100 >= 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.137
Metric Used	occnp_dnn_replacement_total
Recommended Actions	Cause: This alert is triggered when the percentage of AM Update requests are rejected due to Mandatory IE Missing for DNN Replacement Control. When occnp_dnn_replacement_total{mandatory_ie_missing!="") increases, it indicates: AMF is sending invalid request body with PCRT (SMF_SELECT_CH or ALLOWED_NSSAI_CH) but allowedSnssais and/or snssai/dnn it's missing. Diagnostics Information: Verification steps: Send a valid AM Update request including all mandatory IEs for DnnReplacement. Ensure allowedSnssais/snssai/dnn are present and that DnnReplacement is negotiated. Confirm the request succeeds. Verify that the 400/MANDATORY_IE_MISSING ratio drops below the alert threshold within one evaluation window. Recovery: Check AMF configuration and behavior as why invalid values are sent. Investigate configuration mismatch between PCF and AMF. Escalate if sustained at Major or Critical level.

8.1.1.111 QOS_MON_MISSING_ATTRIBUTES_ABOVE_CRITICAL_PERCENT

Table 8-127 QOS_MON_MISSING_ATTRIBUTES_ABOVE_CRITICAL_PERCENT

Field	Description
Description	More than 70% of Npcf_PolicyAuthorization Create requests are missing at least 1 attribute for QosMonitoring
Summary	More than 70% of Npcf_PolicyAuthorization Create requests are missing at least 1 attribute for QosMonitoring
Severity	CRITICAL
Expression	(sum by (namespace) (rate(occnp_pa_qosmon_request_total{incompleteQosMonAttr="true"}[5m]))) / (sum by (namespace) (rate(occnp_pa_qosmon_request_total[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.134
Metric Used	occnp_pa_qosmon_request_total
Recommended Actions	More than 70% of Npcf_PolicyAuthorization Create requests are missing at least 1 attribute for QosMonitoring.The alert will be cleared once the reject rate reduces below the threshold.

8.1.1.112 QOS_MON_MISSING_ATTRIBUTES_ABOVE_MAJOR_PERCENT

Table 8-128 QOS_MON_MISSING_ATTRIBUTES_ABOVE_MAJOR_PERCENT

Field	Details
Description	More than 50% but less than 70% of Npcf_PolicyAuthorization Create requests are missing at least 1 attribute for QosMonitoring
Summary	More than 50% but less than 70% of Npcf_PolicyAuthorization Create requests are missing at least 1 attribute for QosMonitoring
Severity	MAJOR
Expression	(sum by (namespace) (rate(occnp_pa_qosmon_request_total{incompleteQosMonAttr="true"}[5m]))) / (sum by (namespace) (rate(occnp_pa_qosmon_request_total[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.134
Metric Used	occnp_pa_qosmon_request_total
Recommended Actions	More than 50% of Npcf_PolicyAuthorization Create requests are missing at least 1 attribute for QosMonitoring. The alert will be cleared once the reject rate reduces below the threshold.

8.1.1.113 QOS_MON_MISSING_ATTRIBUTES_ABOVE_MINOR_PERCENT

Table 8-129 QOS_MON_MISSING_ATTRIBUTES_ABOVE_MINOR_PERCENT

Field	Description
Description	More than 30% but less than 50% of Npcf_PolicyAuthorization Create requests are missing at least 1 attribute for QosMonitoring
Summary	More than 30% but less than 50% of Npcf_PolicyAuthorization Create requests are missing at least 1 attribute for QosMonitoring
Severity	MINOR
Expression	(sum by (namespace) (rate(occnp_pa_qosmon_request_total{incompleteQosMonAttr="true"}[5m]))) / (sum by (namespace) (rate(occnp_pa_qosmon_request_total[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.134
Metric Used	occnp_pa_qosmon_request_total
Recommended Actions	More than 30% of Npcf_PolicyAuthorization Create requests are missing at least 1 attribute for QosMonitoring. The alert will be cleared once the reject rate reduces below the threshold.

8.1.1.114 QOS_MON_SMF_POLICY_FAILURE_CODE_ABOVE_CRITICAL_PERCENT

Table 8-130 QOS_MON_SMF_POLICY_FAILURE_CODE_ABOVE_CRITICAL_PERCENT

Field	Description
Description	More than 70% of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF contains QOS_MON_DECS_ERR
Summary	More than 70% of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF contains QOS_MON_DECS_ERR
Severity	CRITICAL
Expression	sum by (namespace) (rate(occnp_sm_qosmon_decs_error_total[5m]))) / (sum by (namespace) (rate(occnp_pa_qosmon_request_total[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.133
Metric Used	occnp_sm_qosmon_decs_error_total, occnp_pa_qosmon_request_total
Recommended Actions	The number of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF with QOS_MON_DECS_ERR represents 70% or more of the total app sessions. The alert will be cleared once the reject rate reduces below the threshold.

8.1.1.115 QOS_MON_SMF_POLICY_FAILURE_CODE_ABOVE_MAJOR_PERCENT

Table 8-131 QOS_MON_SMF_POLICY_FAILURE_CODE_ABOVE_MAJOR_PERCENT

Field	Description
Description	More than 50% but less than 70% of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF contains QOS_MON_DECS_ERR
Summary	More than 50% but less than 70% of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF contains QOS_MON_DECS_ERR
Severity	MAJOR
Expression	(sum by (namespace) (rate(occnp_sm_qosmon_decs_error_total[5m]))) / (sum by (namespace) (rate(occnp_pa_qosmon_request_total[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.133
Metric Used	occnp_sm_qosmon_decs_error_total, occnp_pa_qosmon_request_total
Recommended Actions	The number of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF with QOS_MON_DECS_ERR represents 50% or more of the total app sessions. The alert will be cleared once the reject rate reduces below the threshold.

8.1.1.116 QOS_MON_SMF_POLICY_FAILURE_CODE_ABOVE_MINOR_PERCENT

Table 8-132 QOS_MON_SMF_POLICY_FAILURE_CODE_ABOVE_MINOR_PERCENT

Field	Description
Description	More than 30% but less than 50% of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF contains QOS_MON_DECS_ERR
Summary	More than 30% but less than 50% of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF contains QOS_MON_DECS_ERR
Severity	MINOR
Expression	(sum by (namespace) (rate(occnp_sm_qosmon_decs_error_total[5m]))) / (sum by (namespace) (rate(occnp_pa_qosmon_request_total[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.133
Metric Used	occnp_sm_qosmon_decs_error_total, occnp_pa_qosmon_request_total
Recommended Actions	The number of SMF updateNotify responses and/or Npcf_SMPolicyControl_Update requests from SMF with QOS_MON_DECS_ERR represents 30% or more of the total app sessions. The alert will be cleared once the reject rate reduces below the threshold.

8.1.1.117 QOS_MON_FAILED_AUTH_ABOVE_CRITICAL_PERCENT

Table 8-133 QOS_MON_FAILED_AUTH_ABOVE_CRITICAL_PERCENT

Field	Description
Description	More than 70% of PA Create got rejected due to QosMonitoring
Summary	More than 70% of PA Create got rejected due to QosMonitoring
Severity	CRITICAL
Expression	(sum by (namespace) (rate(occnp_pa_qosmon_response_total{cause=~"MISSING_ATTR|FEATURE_UNSUPPORTED|REJECTED"}[5m]))) / (sum by (namespace) (rate(occnp_pa_qosmon_request_total[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.132
Metric Used	occnp_pa_qosmon_response_total, occnp_pa_qosmon_request_total
Recommended Actions	The percentage of Npcf_PolicyAuthorization Create requests that get rejected by policy because of QosMonitoring is equal or greater than 70%. The alert will be cleared once the reject rate reduces below the threshold.

8.1.1.118 QOS_MON_FAILED_AUTH_ABOVE_MAJOR_PERCENT

Table 8-134 QOS_MON_FAILED_AUTH_ABOVE_MAJOR_PERCENT

Field	Description
Description	More than 50% but less than 70% of PA Create got rejected due to QosMonitoring
Summary	More than 50% but less than 70% of PA Create got rejected due to QosMonitoring
Severity	Major
Expression	(sum by (namespace) (rate(occnp_pa_qosmon_response_total{cause=~"MISSING_ATTR|FEATURE_UNSUPPORTED|REJECTED"}[5m]))) / (sum by (namespace) (rate(occnp_pa_qosmon_request_total[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.132
Metric Used	occnp_pa_qosmon_response_total, occnp_pa_qosmon_request_total
Recommended Actions	The percentage of Npcf_PolicyAuthorization Create requests that get rejected by policy because of QosMonitoring is equal or greater than 50%. The alert will be cleared once the reject rate reduces below the threshold.

8.1.1.119 QOS_MON_FAILED_AUTH_ABOVE_MINOR_PERCENT

Table 8-135 QOS_MON_FAILED_AUTH_ABOVE_MINOR_PERCENT

Field	Description
Description	More than 30% but less than 50% of PA Create got rejected due to QosMonitoring
Summary	More than 30% but less than 50% of PA Create got rejected due to QosMonitoring
Severity	MINOR
Expression	(sum by (namespace) (rate(occnp_pa_qosmon_response_total{cause=~"MISSING_ATTR|FEATURE_UNSUPPORTED|REJECTED"}[5m]))) / (sum by (namespace) (rate(occnp_pa_qosmon_request_total[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.132
Metric Used	occnp_pa_qosmon_response_total
Recommended Actions	The percentage of Npcf_PolicyAuthorization Create requests that get rejected by policy because of QosMonitoring is equal or greater than 30%. The alert will be cleared once the reject rate reduces below the threshold.

8.1.2 PCF Alerts

This section provides information on PCF alerts.

8.1.2.1 EGRESS_GATEWAY_DNS_RESOLUTION_FAILURE

Table 8-136 EGRESS_GATEWAY_DNS_RESOLUTION_FAILURE

Field	Details
Description	Egress Gateway encountered DNS SRV resolution failure.
Summary	"{{$labels.microservice}} is unable to route to SCP in namespace {{$labels.namespace}} due to DNS-SRV lookup failure for the FQDN {{$labels.vfqdn}}"
Severity	MAJOR
Expression	(increase(occnp_oc_egressgateway_resolve_fqdn_from_ars_failure_total[2m])) > 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.138
Metric Used	occnp_oc_egressgateway_resolve_fqdn_from_ars_failure_total
Recommended Actions	Cause: Due to misconfiguration, the Egress Gateway was not routing to the SCP properly because of a DNS-SRV lookup failure. Diagnostic Information: Check whether sbirouting is enabled in the egress gateway ConfigMap. Check the configured FQDN or vFQDN value in the egress gateway ConfigMap. Verify the FQDN or vFQDN, value in custom values file. Recovery: If you configure the correct FQDN or vFQDN, the alert is cleared.

8.1.2.2 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_MINOR

Table 8-137 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_MINOR

Field	Details
Description	UDR returning with POST subscribe response but without user data for SM as part of immediate reporting occurring above 10% for service {{$labels.microservice}} in {{$labels.namespace}} ( current value: {{ $value }} % )
Summary	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting
Severity	Minor
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",imm_reports_present="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post"}[5m]))) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.127
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. The missing SM user data check is based on: service_subresource = "sm-data" (indicates the UDR POST was to get SM user data from UDR) operation_type = "POST" (indicates this is a POST call) imm_reports_present = "false" (indicates no SM user data was returned from UDR as part of the Immediate Reporting capability) If these metric dimensions are satisfied, then the alarm will trigger. Alarm Condition: More than or equal to 10% but less than 20% of the traffic: UDR returned a POST Subscribe response without SM user data as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR in the POST REST API request payload has the 30th byte set to 1 when converted to hex (e.g., 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Inform UDR Operator If the above points are validated and SM user data is still not retrieved, inform the UDR operators to verify whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.3 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_MAJOR

Table 8-138 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_MAJOR

Field	Details
Description	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting
Summary	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting
Severity	Major
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",imm_reports_present="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post"}[5m]))) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.127
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. The missing SM user data check is based on: service_subresource = "sm-data" (to indicate the UDR POST was to get SM user data from UDR) operation_type = "POST" (to determine this is a POST call) imm_reports_present = "false" (to indicate no SM user data was returned from UDR as part of the Immediate Reporting capability) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 20% but less than 30% of the traffic: UDR returned a POST Subscribe response without user data for SM as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR in the POST REST API request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Inform UDR Operator If the above points are validated and still no SM user data is retrieved, inform the UDR operators whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.4 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_CRITICAL

Table 8-139 UDR_SM_IMMREP_RESPONSE_MISSING_DATA_CRITICAL

Field	Details
Description	More than 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Summary	More than 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Severity	Critical
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx",immediate_report_pcc="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}[5m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.128
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: UDR profile/capability mismatch, partial rollout/config drift across UDR instances, or negotiation disabled/missing in returned `supportedFeatures`. Diagnostic Information: Verification steps: Query `occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}` split by `immediate_report_pcc`, `nf_name`, and `namespace`. Check whether one/few UDR `nf_name` values dominate the `immediate_report_pcc="false"` bucket. Review PCF logs around `handleSuccessResponseForImmRep` / UDR POST response for messages showing returned `supportedFeatures` does not set ImmediateReportPcc. Confirm PCF-side enablement: `optimizeSubscribe=true`, global `UDR.IMMREP` enabled, and Helm `enable.immediate.reporting.sm-data=true`. Compare UDR profile capability / returned `supportedFeatures` across healthy vs affected UDRs. Monitoring recommendations: Break down the metric by `nf_name` to isolate bad UDR instances. Also watch `occnp_immrep_request_total` vs `occnp_immrep_response_total` for request/response symmetry. Track `imm_reports_present` and `error_cause` for adjacent degradation patterns. Alert separately if the ratio increases beyond 30% or becomes concentrated on a single UDR. Recovery: Verify and correct UDR capability/config so returned `supportedFeatures` includes ImmediateReportPcc for `sm-data` subscriptions. If only specific UDR instances are affected, drain/remove or fix those instances and confirm the `immediate_report_pcc="false"` ratio drops. If intentional/not supported in the environment, disable `sm-data` immediate reporting consistently to avoid mixed negotiation behavior. After change, confirm the alert ratio returns below threshold and `immediate_report_pcc="true"` dominates successful POST responses.

8.1.2.5 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_MINOR

Table 8-140 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_MINOR

Field	Details
Description	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Summary	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Severity	Minor
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx",immediate_report_pcc="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}[5m]))) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.128
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: UDR profile/capability mismatch, partial rollout/config drift across UDR instances, or negotiation disabled/missing in returned `supportedFeatures`. Diagnostic Information: Verification steps: Query `occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}` split by `immediate_report_pcc`, `nf_name`, and `namespace`. Check whether one/few UDR `nf_name` values dominate the `immediate_report_pcc="false"` bucket. Review PCF logs around `handleSuccessResponseForImmRep` / UDR POST response for messages showing returned `supportedFeatures` does not set ImmediateReportPcc. Confirm PCF-side enablement: `optimizeSubscribe=true`, global `UDR.IMMREP` enabled, and Helm `enable.immediate.reporting.sm-data=true`. Compare UDR profile capability / returned `supportedFeatures` across healthy vs affected UDRs. Monitoring recommendations: Break down the metric by `nf_name` to isolate bad UDR instances. Also watch `occnp_immrep_request_total` vs `occnp_immrep_response_total` for request/response symmetry. Track `imm_reports_present` and `error_cause` for adjacent degradation patterns. Alert separately if the ratio increases beyond 20% or becomes concentrated on a single UDR. Recovery: Verify and correct UDR capability/config so returned `supportedFeatures` includes ImmediateReportPcc for `sm-data` subscriptions. If only specific UDR instances are affected, drain/remove or fix those instances and confirm the `immediate_report_pcc="false"` ratio drops. If intentional/not supported in the environment, disable `sm-data` immediate reporting consistently to avoid mixed negotiation behavior. After change, confirm the alert ratio returns below threshold and `immediate_report_pcc="true"` dominates successful POST responses.

8.1.2.6 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_MAJOR

Table 8-141 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_MAJOR

Field	Details
Description	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Summary	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Severity	Major
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx",immediate_report_pcc="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}[5m]))) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.128
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: UDR profile/capability mismatch, partial rollout/config drift across UDR instances, or negotiation disabled/missing in returned `supportedFeatures`. Diagnostic Information: Verification steps: Query `occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",response_code="2xx"}` split by `immediate_report_pcc`, `nf_name`, and `namespace`. Check whether one/few UDR `nf_name` values dominate the `immediate_report_pcc="false"` bucket. Review PCF logs around `handleSuccessResponseForImmRep` / UDR POST response for messages showing returned `supportedFeatures` does not set ImmediateReportPcc. Confirm PCF-side enablement: `optimizeSubscribe=true`, global `UDR.IMMREP` enabled, and Helm `enable.immediate.reporting.sm-data=true`. Compare UDR profile capability / returned `supportedFeatures` across healthy vs affected UDRs. Monitoring recommendations: Break down the metric by `nf_name` to isolate bad UDR instances. Also watch `occnp_immrep_request_total` vs `occnp_immrep_response_total` for request/response symmetry. Track `imm_reports_present` and `error_cause` for adjacent degradation patterns. Alert separately if the ratio increases beyond 20% to 30% or becomes concentrated on a single UDR. Recovery: Verify and correct UDR capability/config so returned `supportedFeatures` includes ImmediateReportPcc for `sm-data` subscriptions. If only specific UDR instances are affected, drain/remove or fix those instances and confirm the `immediate_report_pcc="false"` ratio drops. If intentional/not supported in the environment, disable `sm-data` immediate reporting consistently to avoid mixed negotiation behavior. After change, confirm the alert ratio returns below threshold and `immediate_report_pcc="true"` dominates successful POST responses.

8.1.2.7 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_CRITICAL

Table 8-142 UDR_SM_IMMREP_FEATURE_NEGOTIATION_FAILED_CRITICAL

Field	Details
Description	More than 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Summary	More than 30% of the traffic, UDR returned with POST subscribe response but without user data for SM as part of immediate reporting.
Severity	Critical
Expression	(sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post",immediate_report_pcc="false"}[5m]))) / (sum by (microservice, namespace) (rate(occnp_immrep_response_total{service_subresource="sm-data",operation_type="post"}[5m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.128
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. The failed feature negotiation is based on: service_subresource = "sm-data" (indicates the UDR POST was to get AM user data from UDR) operation_type = "POST" (indicates this is a POST call) immediate_report_pcc = "false" (indicates that no feature negotiation happened with UDR on the ImmReportPcc feature) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 30% of the traffic: UDR returned a POST Subscribe response with failed feature negotiation for SM as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in its request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Inform UDR Operator If the above points are validated and SM user data is still not retrieved: Inform the UDR operators. Ask them to verify whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.8 STALE_DIAMETER_CONNECTOR_REQUEST_CLEANUP_MINOR

Table 8-143 STALE_DIAMETER_CONNECTOR_REQUEST_CLEANUP_MINOR

Field	Details
Description	The Diameter requests are being discarded due to timeout processing occurring above 10% inside pod {{$labels.pod}} for service {{$labels.microservice}} in {{$labels.namespace}}
Summary	More than 10% of the Diam Connector requests failed with error DIAMETER_ERROR_TIMED_OUT_REQUEST.
Severity	Minor
Expression	(sum by (namespace, pod) (rate(occnp_stale_diam_request_cleanup_total{microservice=~".*diam-connector"}[5m]))) / ((sum by (namespace, pod) (rate(ocpm_ingress_request_total{msgType!~"DWR|CER", microservice=~".*diam-connector"}[5m])) or sum by (namespace, pod) (rate(occnp_stale_diam_request_cleanup_total{microservice=~".*diam-connector"}[5m]) * 0)) + sum by (namespace, pod) (rate(occnp_stale_diam_request_cleanup_total{microservice=~".*diam-connector"}[5m]))) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.88
Metric Used	occnp_stale_diam_request_cleanup_total ocpm_ingress_request_total
Recommended Actions	Cause: At least 10 % of the processed requests by Diameter Connector have gone stale before being completed. That means that PCF is taking too long to process the given requests. Diagnostic Information: Analyze Latency: Check how much latency there is between micro-services. Maybe a particular micro-service is slower. Review Logs: Check PCF's Diameter logs to further investigate on why requests are taking too long; take special attention to the timestamps. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: If system is slow, please increase timeout values to consider these latencies. Recovery: As soon as Diameter processes incoming requests on time the Alert will turn off by itself.

8.1.2.9 STALE_DIAMETER_CONNECTOR_REQUEST_CLEANUP_MAJOR

Table 8-144 STALE_DIAMETER_CONNECTOR_REQUEST_CLEANUP_MAJOR

Field	Details
Description	The Diameter requests are being discarded due to timeout processing occurring above 20% inside pod {{$labels.pod}} for service {{$labels.microservice}} in {{$labels.namespace}}
Summary	More than 20% of the Diam Connector requests failed with error DIAMETER_ERROR_TIMED_OUT_REQUEST.
Severity	Major
Expression	(sum by (namespace, pod) (rate(occnp_stale_diam_request_cleanup_total{microservice=~".*diam-connector"}[5m]))) / ((sum by (namespace, pod) (rate(ocpm_ingress_request_total{msgType!~"DWR|CER", microservice=~".*diam-connector"}[5m])) or sum by (namespace, pod) (rate(occnp_stale_diam_request_cleanup_total{microservice=~".*diam-connector"}[5m]) * 0)) + sum by (namespace, pod) (rate(occnp_stale_diam_request_cleanup_total{microservice=~".*diam-connector"}[5m]))) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.88
Metric Used	ocpm_ingress_request_total occnp_stale_diam_request_cleanup_total
Recommended Actions	Cause: At least 20 % of the processed requests by Diameter Connector have gone stale before being completed. That means that PCF is taking too long to process the given requests. Diagnostic Information: Analyze Latency: Check how much latency there is between micro-services. Maybe a particular micro-service is slower. Review Logs: Check PCF's Diameter logs to further investigate on why requests are taking too long; take special attention to the timestamps. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: If system is slow, please increase timeout values to consider these latencies. Recovery: As soon as Diameter processes incoming requests on time the Alert will turn off by itself.

8.1.2.10 STALE_DIAMETER_CONNECTOR_REQUEST_CLEANUP_CRITICAL

Table 8-145 STALE_DIAMETER_CONNECTOR_REQUEST_CLEANUP_CRITICAL

Field	Details
Description	The Diameter requests are being discarded due to timeout processing occurring above 30% inside pod {{$labels.pod}} for service {{$labels.microservice}} in {{$labels.namespace}}
Summary	More than 30% of the Diam Connector requests failed with error DIAMETER_ERROR_TIMED_OUT_REQUEST.
Severity	Critical
Expression	(sum by (namespace, pod) (rate(occnp_stale_diam_request_cleanup_total{microservice=~".*diam-connector"}[5m]))) / ((sum by (namespace, pod) (rate(ocpm_ingress_request_total{msgType!~"DWR|CER", microservice=~".*diam-connector"}[5m])) or sum by (namespace, pod) (rate(occnp_stale_diam_request_cleanup_total{microservice=~".*diam-connector"}[5m]) * 0)) + sum by (namespace, pod) (rate(occnp_stale_diam_request_cleanup_total{microservice=~".*diam-connector"}[5m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.88
Metric Used	ocpm_ingress_request_total occnp_stale_diam_request_cleanup_total
Recommended Actions	Cause: At least 30 % of the processed requests by Diameter Connector have gone stale before being completed. That means that PCF is taking too long to process the given requests. Diagnostic Information: Analyze Latency: Check how much latency there is between micro-services. Maybe a particular micro-service is slower. Review Logs: Check PCF's Diameter logs to further investigate on why requests are taking too long; take special attention to the timestamps. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: If system is slow, please increase timeout values to consider these latencies. Recovery: As soon as Diameter processes incoming requests on time the Alert will turn off by itself.

8.1.2.11 SESSION_BINDING_MISSING_FROM_BSF_EXCEEDS_CRITICAL_THRESHOLD

Table 8-146 SESSION_BINDING_MISSING_FROM_BSF_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	Number of revalidation responses indicated the binding was missing but restored from BSF over all valid session sessions being audited is equal or above 70% of total revalidation responses.
Summary	70% or more of binding were missing but restored from BSF over all bindings being audited.
Severity	Critical
Expression	(sum by (namespace)(rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding", response_code="2xx",action="restored"}[5m])) /sum by (namespace)(rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding",response_code="2xx"}[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.89
Metric Used	occnp_session_binding_revalidation_response_total
Recommended Actions	Cause: At least 70 % of the PCF Bindings registered in Binding Service were missing in the registered BSF and had to be recovered. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and BSF. Maybe the Binding registration requests are not being sent, or taking too long to reach. Review Logs: Check BSF's BSF MGMT logs to further investigate on why it isn't registering the sent PCF Bindings. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: Check BSF's Audit configurations to confirm if they Query to PCF is enabled. Recovery: Once PCF Bindings are properly registered (and not deleted in its lifetime), the percentage of missing bindings will decrease, and the Alert will turn off by itself.

8.1.2.12 SESSION_BINDING_MISSING_FROM_BSF_EXCEEDS_MAJOR_THRESHOLD

Table 8-147 SESSION_BINDING_MISSING_FROM_BSF_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	Number of revalidation responses indicated the binding was missing but restored from BSF over all valid session sessions being audited is equal or above 50% but less than 70% of total revalidation responses.
Summary	50% to 70% of binding were missing but restored from BSF over all bindings being audited
Severity	Major
Expression	(sum by (namespace) (rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding",response_code="2xx",action="restored"}[5m])) / sum by (namespace) (rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding",response_code="2xx"}[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.89
Metric Used	occnp_session_binding_revalidation_response_total
Recommended Actions	Cause: At least 50 % of the PCF Bindings registered in Binding Service were missing in the registered BSF and had to be recovered. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and BSF. Maybe the Binding registration requests are not being sent, or taking too long to reach. Review Logs: Check BSF's BSF MGMT logs to further investigate on why it isn't registering the sent PCF Bindings. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: Check BSF's Audit configurations to confirm if they Query to PCF is enabled. Recovery: Once PCF Bindings are properly registered (and not deleted in its lifetime), the percentage of missing bindings will decrease, and the Alert will turn off by itself.

8.1.2.13 SESSION_BINDING_MISSING_FROM_BSF_EXCEEDS_MINOR_THRESHOLD

Table 8-148 SESSION_BINDING_MISSING_FROM_BSF_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	Number of revalidation responses indicated the binding was missing but restored from BSF over all valid session sessions being audited is equal or above 30% but less than 50% of total revalidation responses.
Summary	30% to 50% of binding were missing but restored from BSF over all bindings being audited.
Severity	Minor
Expression	(sum by (namespace)(rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding",response_code="2xx",action="restored"}[5m])) /sum by (namespace)(rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding",response_code="2xx"}[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.89
Metric Used	occnp_session_binding_revalidation_response_total
Recommended Actions	Cause: At least 30 % of the PCF Bindings registered in Binding Service were missing in the registered BSF and had to be recovered. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and BSF. Maybe the Binding registration requests are not being sent, or taking too long to reach. Review Logs: Check BSF's BSF MGMT logs to further investigate on why it isn't registering the sent PCF Bindings. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: Check BSF's Audit configurations to confirm if they Query to PCF is enabled. Recovery: Once PCF Bindings are properly registered (and not deleted in its lifetime), the percentage of missing bindings will decrease, and the Alert will turn off by itself.

8.1.2.14 SESSION_BINDING_REVALIDATION_WITH_BSF_FAILURE_EXCEEDS_CRITICAL_THRESHOLD

Table 8-149 SESSION_BINDING_REVALIDATION_WITH_BSF_FAILURE_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	Number of unsuccessful revalidation responses with error received from BSF when in PCF the binding association is valid session sessions is equal or above 70% of total revalidation responses.
Summary	70% or more of failed Revalidation Responses received from BSF over total Revalidation Responses
Severity	Critical
Expression	(sum by (namespace)(rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding", response_code!~"2.*"}[5m])) /sum by (namespace)(rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding"}[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.90
Metric Used	occnp_session_binding_revalidation_response_total
Recommended Actions	Cause: At least 70 % of the sent bindings revalidation requests failed with a non-2xx response. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and BSF. Maybe the Binding registration requests are not being sent, or taking too long to reach. Review Logs: Check BSF's IGW and BSF MGMT logs to further investigate on why it isn't receiving its Binding Registration requests. Scale System Resources: High CPU or memory usage may lead to some Binding Registration requests to not be processed. Scale pods as required by the traffic. Recovery: Once binding revalidation requests are properly processed, the percentage of non-2xx responses will decrease, and the Alert will turn off by itself.

8.1.2.15 SESSION_BINDING_REVALIDATION_WITH_BSF_FAILURE_EXCEEDS_MAJOR_THRESHOLD

Table 8-150 SESSION_BINDING_REVALIDATION_WITH_BSF_FAILURE_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	Number of unsuccessful Revalidation Responses with error received from BSF when in PCF the binding association is valid session sessions is equal or above 50% but less than 70% of total Revalidation Responses.
Summary	50% to 70% of failed Revalidation Responses received from BSF over total Revalidation Responses
Severity	Major
Expression	(sum by (namespace)(rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding", response_code!~"2.*"}[5m])) /sum by (namespace)(rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding"}[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.90
Metric Used	occnp_session_binding_revalidation_response_total
Recommended Actions	Cause: At least 50 % of the sent bindings revalidation requests failed with a non-2xx response. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and BSF. Maybe the Binding registration requests are not being sent, or taking too long to reach. Review Logs: Check BSF's IGW and BSF MGMT logs to further investigate on why it isn't receiving its Binding Registration requests. Scale System Resources: High CPU or memory usage may lead to some Binding Registration requests to not be processed. Scale pods as required by the traffic. Recovery: Once binding revalidation requests are properly processed, the percentage of non-2xx responses will decrease, and the Alert will turn off by itself.

8.1.2.16 SESSION_BINDING_REVALIDATION_WITH_BSF_FAILURE_EXCEEDS_MINOR_THRESHOLD

Table 8-151 SESSION_BINDING_REVALIDATION_WITH_BSF_FAILURE_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	Number of unsuccessful Revalidation Responses with error received from BSF when in PCF the binding association is valid session sessions is equal or above 30% but less than 50% of total Revalidation Responses
Summary	30% to 50% of failed Revalidation Responses received from BSF over total Revalidation Responses
Severity	Minor
Expression	(sum by (namespace)(rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding", response_code!~"2.*"}[5m])) /sum by (namespace)(rate(occnp_session_binding_revalidation_response_total{microservice=~".*binding"}[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.90
Metric Used	occnp_session_binding_revalidation_response_total
Recommended Actions	Cause: At least 30 % of the sent bindings revalidation requests failed with a non-2xx response. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and BSF. Maybe the Binding registration requests are not being sent, or taking too long to reach. Review Logs: Check BSF's IGW and BSF MGMT logs to further investigate on why it isn't receiving its Binding Registration requests. Scale System Resources: High CPU or memory usage may lead to some Binding Registration requests to not be processed. Scale pods as required by the traffic. Recovery: Once binding revalidation requests are properly processed, the percentage of non-2xx responses will decrease, and the Alert will turn off by itself.

8.1.2.17 N7_OPTIMIZED_LOOKUP_ERROR_RATE_ABOVE_MINOR_THRESHOLD_PERCENT

Table 8-152 N7_OPTIMIZED_LOOKUP_ERROR_RATE_ABOVE_MINOR_THRESHOLD_PERCENT

Field	Details
Description	Primary Key lookup failed is equal or above 10% but less than 50% of total PA create.
Summary	Primary Key lookup failed is equal or above 10% but less than 50% of total PA create.
Severity	Minor
Expression	sum by (namespace)(increase(occnp_optimized_smpolicyassociation_lookup_query_total{status="failed"}[30m])) / sum by (namespace)(increase(occnp_optimized_smpolicyassociation_lookup_query_total[30m])) * 100 >= 10 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.124
Metric Used	occnp_optimized_smpolicyassociation_lookup_query_total
Recommended Actions	Cause: Metric occnp_optimized_smpolicyassociation_lookup_query_total will increment only when ENABLE_N7_SESSION_LOOKUP_COOKIE flag is enabled at SM service and AAR-I comes with custom AVP and primary key lookup happens. Metrics: occnp_optimized_smpolicyassociation_lookup_query_total This will increment only when ENABLE_N7_SESSION_LOOKUP_COOKIE flag is enabled at SM service and AAR-I comes with custom AVP and primary key lookup happens. Alarm Condition: If Primary lookup fails for more than or equal to 10% but less than 50% of the total PA Create received. Diagnostic Information: Check Network Latency Investigate possible delays in network between PCF/BSF/AF Verify sender information Verify if PA Create (from AF) is sending the correct information in the headers (cookies) Verify NFs that are affected by the flow Verify healthy status of BSF and AF Verify E2E flow if BSF is receiving from PCF(SM) the headers cookie, if AF is sending in Rx-Create/PaCreate the headers cookie Review application Verify if Sm/Diam-Connector/Diam-Gw are not congested Signs like constant error logs are showing in services Monitor System/resource utilization (CPU, Memory, queues) Monitor Dashboard Monitor the following metric occnp_optimized_smpolicyassociation_lookup_query_total Monitor DB Verify healthy status of DB Latency in DB operation Recover: Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF (BSF or AF) and SM Service, Diam-Connector and Diam-Gw at PCF. Resolve any high latency or packet loss immediately if detected. Review SM Service and Diam-Connector Application and Resources Restart or scale up resources temporarily if the system is overloaded Disable feature If this flow is the only one affected, disable this feature as a last option.

8.1.2.18 N7_OPTIMIZED_LOOKUP_ERROR_RATE_ABOVE_MAJOR_THRESHOLD_PERCENT

Table 8-153 N7_OPTIMIZED_LOOKUP_ERROR_RATE_ABOVE_MAJOR_THRESHOLD_PERCENT

Field	Details
Description	Primary Key lookup failed is equal or above 50% but less than 75% of total PA create.
Summary	Primary Key lookup failed is equal or above 50% but less than 75% of total PA create.
Severity	Major
Expression	sum by (namespace)(increase(occnp_optimized_smpolicyassociation_lookup_query_total{status="failed"}[30m])) / sum by (namespace)(increase(occnp_optimized_smpolicyassociation_lookup_query_total[30m])) * 100 >= 50 < 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.124
Metric Used	occnp_optimized_smpolicyassociation_lookup_query_total
Recommended Actions	Cause: Metric occnp_optimized_smpolicyassociation_lookup_query_total will increment only when ENABLE_N7_SESSION_LOOKUP_COOKIE flag is enabled at SM service and AAR-I comes with custom AVP and primary key lookup happens. Metrics: occnp_optimized_smpolicyassociation_lookup_query_total This will increment only when ENABLE_N7_SESSION_LOOKUP_COOKIE flag is enabled at SM service and AAR-I comes with custom AVP and primary key lookup happens. Alarm Condition: If Primary lookup fails for more than or equal to 50% but less than 75% of the total PA Create received. Diagnostic Information: Check Network Latency Investigate possible delays in network between PCF/BSF/AF Verify sender information Verify if PA Create (from AF) is sending the correct information in the headers (cookies) Verify NFs that are affected by the flow Verify healthy status of BSF and AF Verify E2E flow if BSF is receiving from PCF(SM) the headers cookie, if AF is sending in Rx-Create/PaCreate the headers cookie Review application Verify if Sm/Diam-Connector/Diam-Gw are not congested Signs like constant error logs are showing in services Monitor System/resource utilization (CPU, Memory, queues) Monitor Dashboard Monitor the following metric occnp_optimized_smpolicyassociation_lookup_query_total Monitor DB Verify healthy status of DB Latency in DB operation Recover: Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF (BSF or AF) and SM Service, Diam-Connector and Diam-Gw at PCF. Resolve any high latency or packet loss immediately if detected. Review SM Service and Diam-Connector Application and Resources Restart or scale up resources temporarily if the system is overloaded Disable feature If this flow is the only one affected, disable this feature as a last option.

8.1.2.19 N7_OPTIMIZED_LOOKUP_ERROR_RATE_ABOVE_CRITICAL_THRESHOLD_PERCENT

Table 8-154 N7_OPTIMIZED_LOOKUP_ERROR_RATE_ABOVE_CRITICAL_THRESHOLD_PERCENT

Field	Details
Description	Primary Key lookup failed is equal or above 75% of total PA create.
Summary	Primary Key lookup failed is equal or above 75% of total PA create
Severity	Critical
Expression	sum by (namespace)(increase(occnp_optimized_smpolicyassociation_lookup_query_total{status="failed"}[30m])) / sum by (namespace)(increase(occnp_optimized_smpolicyassociation_lookup_query_total[30m])) * 100 >= 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.124
Metric Used	occnp_optimized_smpolicyassociation_lookup_query_total
Recommended Actions	Cause: Metric occnp_optimized_smpolicyassociation_lookup_query_total will increment only when ENABLE_N7_SESSION_LOOKUP_COOKIE flag is enabled at SM service and AAR-I comes with custom AVP and primary key lookup happens. Metrics: occnp_optimized_smpolicyassociation_lookup_query_total This will increment only when ENABLE_N7_SESSION_LOOKUP_COOKIE flag is enabled at SM service and AAR-I comes with custom AVP and primary key lookup happens. Alarm Condition: If Primary lookup fails for more than or equal to 75% of the total PA Create received. Diagnostic Information: Check Network Latency Investigate possible delays in network between PCF/BSF/AF Verify sender information Verify if PA Create (from AF) is sending the correct information in the headers (cookies) Verify NFs that are affected by the flow Verify healthy status of BSF and AF Verify E2E flow if BSF is receiving from PCF(SM) the headers cookie, if AF is sending in Rx-Create/PaCreate the headers cookie Review application Verify if Sm/Diam-Connector/Diam-Gw are not congested Signs like constant error logs are showing in services Monitor System/resource utilization (CPU, Memory, queues) Monitor Dashboard Monitor the following metric occnp_optimized_smpolicyassociation_lookup_query_total Monitor DB Verify healthy status of DB Latency in DB operation Recover: Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF (BSF or AF) and SM Service, Diam-Connector and Diam-Gw at PCF. Resolve any high latency or packet loss immediately if detected. Review SM Service and Diam-Connector Application and Resources Restart or scale up resources temporarily if the system is overloaded Disable feature If this flow is the only one affected, disable this feature as a last option.

8.1.2.20 SM_SVC_REQ_ENHANCED_OVERLOAD_REJECTION_MINOR

Table 8-155 SM_SVC_REQ_ENHANCED_OVERLOAD_REJECTION_MINOR

Field	Details
Description	At least 10% of the received requests have been cancelled due to Overload state of PCF-SM service.
Summary	At least 10% of the received Requests have been rejected due to Overload state of pcf-sm service in namespace {{$labels.namespace}}
Severity	Minor
Expression	( sum by (namespace) (rate(occnp_enhanced_overload_reject_total{microservice=~".*pcf_sm"}[2m])) / (sum by (namespace) (rate(ocpm_ingress_request_total{microservice=~".*pcf_sm"}[2m]) or occnp_enhanced_overload_reject_total * 0) + (sum by (namespace) (rate(session_oam_request_total{microservice=~".*pcf_sm"}[2m]) or occnp_enhanced_overload_reject_total * 0) ) ) ) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.125
Metric Used	occnp_enhanced_overload_reject_total ocpm_ingress_request_total session_oam_request_total
Recommended Actions	Cause: occnp_enhanced_overload_reject_total metric gets pegged whenever a rejection is performed by sm-service. This alert accounts all the incoming traffic and tells the extent of rejection. If, for a given namespace, any of the metric is absent in denominator (that is, session_oam_request_total/ocpm_ingress_request_total), it's value will be considered zero(0) by default as per "or on(namespace) {...}" clause used in the expression. Diagnostic information: To monitor the individual rejection percentage by each interface type and operation type, please monitor the grafana dashboard with metric occnp_enhanced_overload_reject_total and dimensions operationType, interfaceType and loadLevel. Recovery: Whenever the alert is raised, please go to grafana to check which interfaceType/operationType is being rejected the most. If that extent of rejection is not desired for the given interfaceType/operationType and current load level, customer may need to update the overload profile configurations accordingly.

8.1.2.21 SM_SVC_REQ_ENHANCED_OVERLOAD_REJECTION_MAJOR

Table 8-156 SM_SVC_REQ_ENHANCED_OVERLOAD_REJECTION_MAJOR

Field	Details
Description	At least 20 % of the received requests have been cancelled due to Overload state of PCF-SM service.
Summary	At least 20% of the received Requests have been rejected due to Overload state of pcf-sm service in namespace {{$labels.namespace}}
Severity	Major
Expression	( sum by (namespace) (rate(occnp_enhanced_overload_reject_total{microservice=~".*pcf_sm"}[2m])) / (sum by (namespace) (rate(ocpm_ingress_request_total{microservice=~".*pcf_sm"}[2m]) or occnp_enhanced_overload_reject_total * 0) + (sum by (namespace) (rate(session_oam_request_total{microservice=~".*pcf_sm"}[2m]) or occnp_enhanced_overload_reject_total * 0) ) ) ) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.125
Metric Used	occnp_enhanced_overload_reject_total ocpm_ingress_request_total session_oam_request_total
Recommended Actions	Cause: occnp_enhanced_overload_reject_total metric gets pegged whenever a rejection is performed by sm-service. This alert accounts all the incoming traffic and tells the extent of rejection. If, for a given namespace, any of the metric is absent in denominator (that is, session_oam_request_total/ocpm_ingress_request_total), it's value will be considered zero(0) by default as per "or on(namespace) {...}" clause used in the expression. Diagnostic information: To monitor the individual rejection percentage by each interface type and operation type, please monitor the grafana dashboard with metric occnp_enhanced_overload_reject_total and dimensions operationType, interfaceType and loadLevel. Recovery: Whenever the alert is raised, please go to grafana to check which interfaceType/operationType is being rejected the most. If that extent of rejection is not desired for the given interfaceType/operationType and current load level, customer may need to update the overload profile configurations accordingly.

8.1.2.22 SM_SVC_REQ_ENHANCED_OVERLOAD_REJECTION_CRITICAL

Table 8-157 SM_SVC_REQ_ENHANCED_OVERLOAD_REJECTION_CRITICAL

Field	Details
Description	At least 30 % of the received requests have been cancelled due to Overload state of PCF-SM service.
Summary	At least 30% of the received Requests have been rejected due to Overload state of pcf-sm service in namespace {{$labels.namespace}}.
Severity	Critical
Expression	( sum by (namespace) (rate(occnp_enhanced_overload_reject_total{microservice=~".*pcf_sm"}[2m])) / (sum by (namespace) (rate(ocpm_ingress_request_total{microservice=~".*pcf_sm"}[2m]) or occnp_enhanced_overload_reject_total * 0) + (sum by (namespace) (rate(session_oam_request_total{microservice=~".*pcf_sm"}[2m]) or occnp_enhanced_overload_reject_total * 0) ) ) ) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.125
Metric Used	occnp_enhanced_overload_reject_total ocpm_ingress_request_total session_oam_request_total
Recommended Actions	Cause: occnp_enhanced_overload_reject_total metric gets pegged whenever a rejection is performed by sm-service. This alert accounts all the incoming traffic and tells the extent of rejection. If, for a given namespace, any of the metric is absent in denominator (that is, session_oam_request_total/ocpm_ingress_request_total), it's value will be considered zero(0) by default as per "or on(namespace) {...}" clause used in the expression. Diagnostic information: To monitor the individual rejection percentage by each interface type and operation type, please monitor the grafana dashboard with metric occnp_enhanced_overload_reject_total and dimensions operationType, interfaceType and loadLevel. Recovery: Whenever the alert is raised, please go to grafana to check which interfaceType/operationType is being rejected the most. If that extent of rejection is not desired for the given interfaceType/operationType and current load level, customer may need to update the overload profile configurations accordingly.

8.1.2.23 SM_SVC_REQ_ENHANCED_OVERLOAD_SERVICE_INTERACTION_SKIPPED_MINOR

Table 8-158 SM_SVC_REQ_ENHANCED_OVERLOAD_SERVICE_INTERACTION_SKIPPED_MINOR

Field	Details
Description	At least 10 % of the received requests have been treated as Assume Positive due to Overload state of PCF-SM service.
Summary	For at least 10% of the received Requests, various service interactions(including UserData, SpendingLimit, PRE interaction) were skipped due to Overload state of pcf-sm service in namespace {{$labels.namespace}}
Severity	Minor
Expression	( (sum by (namespace) (rate(occnp_enhanced_overload_service_interaction_total{microservice=~".*pcf_sm",interfaceType="Npcf_SMPolicyControl"}[5m])) - (sum by (namespace) (rate(occnp_enhanced_overload_service_interaction_total{microservice=~".*pcf_sm",interfaceType="Npcf_SMPolicyControl",userData="true",spendingLimitData="true",policyEvaluation="true"}[5m])) or sum by (namespace) (rate(occnp_enhanced_overload_service_interaction_total{microservice=~".*pcf_sm"}[5m])) * 0 )) ) / sum by (namespace) (rate(ocpm_ingress_request_total{microservice=~".*pcf_sm",servicename_3gpp="npcf-smpolicycontrol"}[5m]) ) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.126
Metric Used	occnp_enhanced_overload_service_interaction_total ocpm_ingress_request_total
Recommended Actions	Cause: This alert gets triggered when the percentage of requests processed with "assume positive" actions, due to system overload, exceeds a certain threshold percentage(say 30%). Specifically, it tracks instances where calls to UserData, SpendingLimit, or PolicyService are skipped by smservice because of overload conditions. When occnp_enhanced_overload_service_skipped_total increases, it indicates a higher frequency of assume-positive actions—meaning service interactions are being bypassed to maintain performance under heavy load. While this prevents complete service failure, excessive use indicates significant service degradation. Diagnostic Steps: Use Grafana Dashboard: Check the occnp_enhanced_overload_service_skipped_total metric in Grafana. Review Dimensions: Filter by operationType, interfaceType, loadLevel, and skipped operations (for example, userData, spendingLimitData, policyEvaluation). Also, review relevant message attributes such as dnn, snssai, and serviceUrn. Identify Patterns: Note which specific operations and interface types are most affected. Use the gauge metric occnp_service_load_level to get the current load level of the service. Check Grafana to identify which interfaceType/operationType has the highest skip rate. Recovery: Assess if the skip rate is acceptable for the current load level. Adjust configuration if needed: Tune overload reject/assume-positive percentages Enable/disable service interactions in the service profile Disable feature entirely from SM service screen (last option).

8.1.2.24 SM_SVC_REQ_ENHANCED_OVERLOAD_SERVICE_INTERACTION_SKIPPED_MAJOR

Table 8-159 SM_SVC_REQ_ENHANCED_OVERLOAD_SERVICE_INTERACTION_SKIPPED_MAJOR

Field	Details
Description	At least 20 % of the received requests have been treated as Assume Positive due to Overload state of PCF-SM service.
Summary	For at least 20% of the received Requests, various service interactions(including UserData, SpendingLimit, PRE interaction) were skipped due to Overload state of pcf-sm service in namespace {{$labels.namespace}}
Severity	Major
Expression	( (sum by (namespace) (rate(occnp_enhanced_overload_service_interaction_total{microservice=~".*pcf_sm",interfaceType="Npcf_SMPolicyControl"}[5m])) - (sum by (namespace) (rate(occnp_enhanced_overload_service_interaction_total{microservice=~".*pcf_sm",interfaceType="Npcf_SMPolicyControl",userData="true",spendingLimitData="true",policyEvaluation="true"}[5m])) or sum by (namespace) (rate(occnp_enhanced_overload_service_interaction_total{microservice=~".*pcf_sm"}[5m])) * 0 )) ) / sum by (namespace) (rate(ocpm_ingress_request_total{microservice=~".*pcf_sm",servicename_3gpp="npcf-smpolicycontrol"}[5m]) ) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.126
Metric Used	occnp_enhanced_overload_service_interaction_total ocpm_ingress_request_total
Recommended Actions	Cause: This alert gets triggered when the percentage of requests processed with "assume positive" actions, due to system overload, exceeds a certain threshold percentage(say 30%). Specifically, it tracks instances where calls to UserData, SpendingLimit, or PolicyService are skipped by smservice because of overload conditions. When occnp_enhanced_overload_service_skipped_total increases, it indicates a higher frequency of assume-positive actions—meaning service interactions are being bypassed to maintain performance under heavy load. While this prevents complete service failure, excessive use indicates significant service degradation. Diagnostic Steps: Use Grafana Dashboard: Check the occnp_enhanced_overload_service_skipped_total metric in Grafana. Review Dimensions: Filter by operationType, interfaceType, loadLevel, and skipped operations (for example, userData, spendingLimitData, policyEvaluation). Also, review relevant message attributes such as dnn, snssai, and serviceUrn. Identify Patterns: Note which specific operations and interface types are most affected. Use the gauge metric occnp_service_load_level to get the current load level of the service. Check Grafana to identify which interfaceType/operationType has the highest skip rate. Recovery: Assess if the skip rate is acceptable for the current load level. Adjust configuration if needed: Tune overload reject/assume-positive percentages Enable/disable service interactions in the service profile Disable feature entirely from SM service screen (last option).

8.1.2.25 SM_SVC_REQ_ENHANCED_OVERLOAD_SERVICE_INTERACTION_SKIPPED_CRITICAL

Table 8-160 SM_SVC_REQ_ENHANCED_OVERLOAD_SERVICE_INTERACTION_SKIPPED_CRITICAL

Field	Details
Description	At least 30 % of the received Requests have been treated as Assume Positive due to Overload state of PCF-SM service
Summary	For at least 30% of the received Requests, various service interactions(including UserData, SpendingLimit, PRE interaction) were skipped due to Overload state of pcf-sm service in namespace {{$labels.namespace}}
Severity	Critical
Expression	( (sum by (namespace) (rate(occnp_enhanced_overload_service_interaction_total{microservice=~".*pcf_sm",interfaceType="Npcf_SMPolicyControl"}[5m])) - (sum by (namespace) (rate(occnp_enhanced_overload_service_interaction_total{microservice=~".*pcf_sm",interfaceType="Npcf_SMPolicyControl",userData="true",spendingLimitData="true",policyEvaluation="true"}[5m])) or sum by (namespace) (rate(occnp_enhanced_overload_service_interaction_total{microservice=~".*pcf_sm"}[5m])) * 0 )) ) / sum by (namespace) (rate(ocpm_ingress_request_total{microservice=~".*pcf_sm",servicename_3gpp="npcf-smpolicycontrol"}[5m]) ) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.126
Metric Used	occnp_enhanced_overload_service_interaction_total ocpm_ingress_request_total
Recommended Actions	Cause: This alert gets triggered when the percentage of requests processed with "assume positive" actions, due to system overload, exceeds a certain threshold percentage(say 30%). Specifically, it tracks instances where calls to UserData, SpendingLimit, or PolicyService are skipped by smservice because of overload conditions. When occnp_enhanced_overload_service_skipped_total increases, it indicates a higher frequency of assume-positive actions—meaning service interactions are being bypassed to maintain performance under heavy load. While this prevents complete service failure, excessive use indicates significant service degradation. Diagnostic Steps: Use Grafana Dashboard: Check the occnp_enhanced_overload_service_skipped_total metric in Grafana. Review Dimensions: Filter by operationType, interfaceType, loadLevel, and skipped operations (for example, userData, spendingLimitData, policyEvaluation). Also, review relevant message attributes such as dnn, snssai, and serviceUrn. Identify Patterns: Note which specific operations and interface types are most affected. Use the gauge metric occnp_service_load_level to get the current load level of the service. Check Grafana to identify which interfaceType/operationType has the highest skip rate. Recovery: Assess if the skip rate is acceptable for the current load level. Adjust configuration if needed: Tune overload reject/assume-positive percentages Enable/disable service interactions in the service profile Disable feature entirely from SM service screen (last option).

8.1.2.26 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_ABOVE_MINOR_THRESHOLD

Table 8-161 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_ABOVE_MINOR_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_ABOVE_MINOR_THRESHOLD
Description	More than 70% of timer capacity has been occupied for n1n2 transfer failure notification
Summary	More than 70% of timer capacity has been occupied for n1n2 transfer failure notification
Severity	Minor
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_N1N2TransferFailure"})/360000) * 100 > 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.107
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to N1N2 transfer failure notification reaches 70% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with the flow triggering N1N2 transfer failure notification and possibly enable re-transmission. Cause: This alert indicates sustained high utilization of the UE N1N2 Transfer Failure Notification timer pool. The occnp_timer_capacity gauge tracks the current number of outstanding timers per timerName, updated every timer scan. These timers are created when the UE cannot deliver URSP rules and the system initiates a reattempt flow using backoff with a timer. High utilization suggests many failures are triggering the N1N2 transfer failure notification flow. The alert notifies when utilization for timerName "UE_N1N2TransferFailure" exceeds 70% of a baseline capacity of 360,000. Dimensions: timerName: UE_N1N2TransferFailure namespace: as per Prometheus label used in aggregation siteId: underlying metric label; rule aggregates with max by (namespace) Diagnostic Information: Validate the alert metric: Inspect occnp_timer_capacity{timerName="UE_N1N2TransferFailure"} in Prometheus/Grafana (and /actuator/prometheus) and review trends around the alert window. Correlate with triggering failures: Check for spikes in N1N2 transfer failure notifications and URSP delivery failures within the same time window. Review logs around the alert window: In PCF-UE and related components/egress, look for errors leading to N1N2 transfer failure notifications; align timestamps with the alert period. Verify retransmission/backoff settings: Ensure retransmission is enabled; confirm backoff parameters are appropriate (not overly conservative). Check downstream/egress health: Validate connectivity and response health for AMF or upstream endpoints; look for elevated error rates/timeouts. Confirm processing throughput: Verify rate_per_second for this timerName, worker thread health, and pod readiness/liveness; ensure backlog is draining. Watch for capacity rejections: Observe occnp_timer_create_failure_total{timerName="UE_N1N2TransferFailure", errorCause="TIMER_CAPACITY_EXCEEDS"} for signs of hard-cap hits. Recovery: Resolve underlying failures: Work with upstream/AMF and correct misconfigurations causing the flow to trigger N1N2 transfer failure notifications at high rates. Enable or optimize retransmission: Turn on retransmission if disabled; tune backoff to improve success while avoiding downstream overload. Increase draining capacity: Temporarily raise rate_per_second and/or scale pods to drain outstanding timers faster. Adjust capacity if needed: Temporarily increase the registered timer_capacity baseline for this timerName while addressing root causes. Reduce new load temporarily: Throttle or defer non-critical timer creates for this timerName until utilization drops. Monitor until recovered: Continue monitoring the alert metric after remedial actions to confirm the failure rate falls below the alert threshold.

8.1.2.27 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_ABOVE_MAJOR_THRESHOLD

Table 8-162 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_ABOVE_MAJOR_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_ABOVE_MAJOR_THRESHOLD
Description	More than 80% of timer capacity has been occupied for n1n2 transfer failure notification
Summary	More than 80% of timer capacity has been occupied for n1n2 transfer failure notification
Severity	Major
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_N1N2TransferFailure"})/360000) * 100 > 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.107
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to N1N2 transfer failure notification reaches 80% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with the flow triggering N1N2 transfer failure notification and possibly enable re-transmission. Cause: This alert indicates sustained high utilization of the UE N1N2 Transfer Failure Notification timer pool. The occnp_timer_capacity gauge tracks the current number of outstanding timers per timerName, updated every timer scan. - These timers are created when the UE cannot deliver URSP rules and the system initiates a reattempt flow using backoff with a timer. High utilization suggests many failures are triggering the N1N2 transfer failure notification flow. - The alert notifies when utilization for timerName "UE_N1N2TransferFailure" exceeds 80% of a baseline capacity of 360000. Dimensions: timerName : UE_N1N2TransferFailure namespace : as per Prometheus label used in aggregation siteId : underlying metric label; rule aggregates with max by (namespace) Diagnostic Information : Validate the alert metric: Inspect occnp_timer_capacity{timerName="UE_N1N2TransferFailure"} in Prometheus/Grafana (and /actuator/prometheus) and review trends around the alert window. Correlate with triggering failures: Check for spikes in N1N2 transfer failure notifications and URSP delivery failures within the same time window. Review logs around the alert window: In PCF-UE and related components/egress, look for errors leading to N1N2 transfer failure notifications; align timestamps with the alert period. Verify retransmission/backoff settings: Ensure retransmission is enabled; confirm backoff parameters are appropriate (not overly conservative). Check downstream/egress health: Validate connectivity and response health for AMF or upstream endpoints; look for elevated error rates/timeouts. Confirm processing throughput: Verify rate_per_second for this timerName, worker thread health, and pod readiness/liveness; ensure backlog is draining. Watch for capacity rejections: Observe occnp_timer_create_failure_total{timerName="UE_N1N2TransferFailure",errorCause="TIMER_CAPACITY_EXCEEDS"} for signs of hard-cap hits. Recovery : Resolve underlying failures: Work with upstream/AMF and correct misconfigurations causing the flow to trigger N1N2 transfer failure notifications at high rates. Enable or optimize retransmission: Turn on retransmission if disabled; tune backoff to improve success while avoiding downstream overload. Increase draining capacity: Temporarily raise rate_per_second and/or scale pods to drain outstanding timers faster. Adjust capacity if needed: Temporarily increase the registered timer_capacity baseline for this timerName while addressing root causes. Reduce new load temporarily: Throttle or defer non-critical timer creates for this timerName until utilization drops. Monitor until recovered: Continue monitoring the alert metric after remedial actions to confirm the failure rate falls below the alert threshold.

8.1.2.28 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_ABOVE_CRITICAL_THRESHOLD

Table 8-163 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_ABOVE_CRITICAL_THRESHOLD
Description	More than 90% of timer capacity has been occupied for n1n2 transfer failure notification
Summary	More than 90% of timer capacity has been occupied for n1n2 transfer failure notification
Severity	Critical
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_N1N2TransferFailure"})/360000) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.107
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to N1N2 transfer failure notification reaches 90% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with the flow triggering N1N2 transfer failure notification and possibly enable re-transmission. Cause: This alert indicates sustained high utilization of the UE N1N2 Transfer Failure Notification timer pool. The occnp_timer_capacity gauge tracks the current number of outstanding timers per timerName, updated every timer scan. - These timers are created when the UE cannot deliver URSP rules and the system initiates a reattempt flow using backoff with a timer. High utilization suggests many failures are triggering the N1N2 transfer failure notification flow. - The alert notifies when utilization for timerName "UE_N1N2TransferFailure" exceeds 90% of a baseline capacity of 360000. Dimensions: timerName : UE_N1N2TransferFailure namespace : as per Prometheus label used in aggregation siteId : underlying metric label; rule aggregates with max by (namespace) Diagnostic Information : Validate the alert metric: Inspect occnp_timer_capacity{timerName="UE_N1N2TransferFailure"} in Prometheus/Grafana (and /actuator/prometheus) and review trends around the alert window. Correlate with triggering failures: Check for spikes in N1N2 transfer failure notifications and URSP delivery failures within the same time window. Review logs around the alert window: In PCF-UE and related components/egress, look for errors leading to N1N2 transfer failure notifications; align timestamps with the alert period. Verify retransmission/backoff settings: Ensure retransmission is enabled; confirm backoff parameters are appropriate (not overly conservative). Check downstream/egress health: Validate connectivity and response health for AMF or upstream endpoints; look for elevated error rates/timeouts. Confirm processing throughput: Verify rate_per_second for this timerName, worker thread health, and pod readiness/liveness; ensure backlog is draining. Watch for capacity rejections: Observe occnp_timer_create_failure_total{timerName="UE_N1N2TransferFailure",errorCause="TIMER_CAPACITY_EXCEEDS"} for signs of hard-cap hits. Recovery : Resolve underlying failures: Work with upstream/AMF and correct misconfigurations causing the flow to trigger N1N2 transfer failure notifications at high rates. Enable or optimize retransmission: Turn on retransmission if disabled; tune backoff to improve success while avoiding downstream overload. Increase draining capacity: Temporarily raise rate_per_second and/or scale pods to drain outstanding timers faster. Adjust capacity if needed: Temporarily increase the registered timer_capacity baseline for this timerName while addressing root causes. Reduce new load temporarily: Throttle or defer non-critical timer creates for this timerName until utilization drops. Monitor until recovered: Continue monitoring the alert metric after remedial actions to confirm the failure rate falls below the alert threshold.

8.1.2.29 AUDIT_TIMER_CAPACITY_FOR_UE_AMF_DISCOVERY_ABOVE_MINOR_THRESHOLD

Table 8-164 AUDIT_TIMER_CAPACITY_FOR_UE_AMF_DISCOVERY_ABOVE_MINOR_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_AMF_DISCOVERY_ABOVE_MINOR_THRESHOLD
Description	More than 70% of timers capacity has been occupied for amf discovery.
Summary	More than 70% of timers capacity has been occupied for amf discovery.
Severity	Minor
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_AMFDiscovery"})/360000) * 100 > 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.95
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to AMF discovery reaches 70% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with NRF discovery and possibly enable direct or indirect alternate routing from NRF client. Cause: More than 70% of timer capacity has been occupied for AMF discovery. The occnp_timer_capacity metric records the current timer count. These timers are created when the User Equipment (UE) cannot deliver URSP rules, retries with a back-off, and creates a timer. This alert is triggered when capacity for timers corresponding to AMF Discovery reaches a certain percent (over 70%) of the total 360K capacity. Diagnostic Information: High Rate of UE Failures: Many User Equipment (UE) devices are unable to deliver URSP (User Route Selection Policy) rules, causing increased retries and timer creation for AMF discovery. Network Function (NRF or AMF) Issues: Problems or instability with the AMF (Access and Mobility Management Function) or related NRF (Network Repository Function) components might prevent successful discovery or rule delivery, resulting in more timer retries. Resource Bottlenecks: Network resource constraints or congestion could delay or prevent successful URSP rule delivery, again resulting in repeated retries and high timer usage. Excessively Short Timer Values: If the back-off or retry timers are set too short, UEs may repeat attempts too rapidly, compounding timer consumption. Recovery: Review the logs and monitor for trends in UE failures with AMF discovery. Consider enabling direct or indirect alternate routing from the NRF-client to mitigate timer capacity issues. Investigate any recent configuration or software changes, check for network health (especially AMF and NRF), and verify timer-related configurations. If the issue persists, please check with Support team.

8.1.2.30 AUDIT_TIMER_CAPACITY_FOR_UE_AMF_DISCOVERY_ABOVE_MAJOR_THRESHOLD

Table 8-165 AUDIT_TIMER_CAPACITY_FOR_UE_AMF_DISCOVERY_ABOVE_MAJOR_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_AMF_DISCOVERY_ABOVE_MAJOR_THRESHOLD
Description	More than 80% of timer capacity has been occupied for amf discovery.
Summary	More than 80% of timer capacity has been occupied for amf discovery.
Severity	Major
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_AMFDiscovery"})/360000) * 100 > 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.95
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to AMF discovery reaches 80% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with NRF discovery and possibly enable direct or indirect alternate routing from NRF client. Cause: More than 80% of timer capacity has been occupied for AMF discovery. The occnp_timer_capacity metric records the current timer count. These timers are created when the User Equipment (UE) cannot deliver URSP rules, retries with a back-off, and creates a timer. This alert is triggered when capacity for timers corresponding to AMF Discovery reaches a certain percent (over 80%) of the total 360K capacity. Diagnostic Information: High Rate of UE Failures: Many User Equipment (UE) devices are unable to deliver URSP (User Route Selection Policy) rules, causing increased retries and timer creation for AMF discovery. Network Function (NRF or AMF) Issues: Problems or instability with the AMF (Access and Mobility Management Function) or related NRF (Network Repository Function) components might prevent successful discovery or rule delivery, resulting in more timer retries. Resource Bottlenecks: Network resource constraints or congestion could delay or prevent successful URSP rule delivery, again resulting in repeated retries and high timer usage. Excessively Short Timer Values: If the back-off or retry timers are set too short, UEs may repeat attempts too rapidly, compounding timer consumption. Recovery: Review the logs and monitor for trends in UE failures with AMF discovery. Consider enabling direct or indirect alternate routing from the NRF-client to mitigate timer capacity issues. Investigate any recent configuration or software changes, check for network health (especially AMF and NRF), and verify timer-related configurations. If the issue persists, please check with Support team.

8.1.2.31 AUDIT_TIMER_CAPACITY_FOR_UE_AMF_DISCOVERY_ABOVE_CRITICAL_THRESHOLD

Table 8-166 AUDIT_TIMER_CAPACITY_FOR_UE_AMF_DISCOVERY_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_AMF_DISCOVERY_ABOVE_CRITICAL_THRESHOLD
Description	More than 90% of timer capacity has been occupied for amf discovery.
Summary	More than 90% of timer capacity has been occupied for amf discovery.
Severity	Critical
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_AMFDiscovery"})/360000) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.95
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to AMF discovery reaches 90% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with NRF discovery and possibly enable direct or indirect alternate routing from NRF client. Cause: More than 90% of timer capacity has been occupied for AMF discovery. The occnp_timer_capacity metric records the current timer count. These timers are created when the User Equipment (UE) cannot deliver URSP rules, retries with a back-off, and creates a timer. This alert is triggered when capacity for timers corresponding to AMF Discovery reaches a certain percent (over 90%) of the total 360K capacity. Diagnostic Information: High Rate of UE Failures: Many User Equipment (UE) devices are unable to deliver URSP (User Route Selection Policy) rules, causing increased retries and timer creation for AMF discovery. Network Function (NRF or AMF) Issues: Problems or instability with the AMF (Access and Mobility Management Function) or related NRF (Network Repository Function) components might prevent successful discovery or rule delivery, resulting in more timer retries. Resource Bottlenecks: Network resource constraints or congestion could delay or prevent successful URSP rule delivery, again resulting in repeated retries and high timer usage. Excessively Short Timer Values: If the back-off or retry timers are set too short, UEs may repeat attempts too rapidly, compounding timer consumption. Recovery: Review the logs and monitor for trends in UE failures with AMF discovery. Consider enabling direct or indirect alternate routing from the NRF-client to mitigate timer capacity issues. Investigate any recent configuration or software changes, check for network health (especially AMF and NRF), and verify timer-related configurations. If the issue persists, please check with Support team.

8.1.2.32 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_SUBSCRIBE_ABOVE_MINOR_THRESHOLD

Table 8-167 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_SUBSCRIBE_ABOVE_MINOR_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_SUBSCRIBE_ABOVE_MINOR_THRESHOLD
Description	More than 70% of timer capacity has been occupied for n1n2 subscribe.
Summary	More than 70% of timer capacity has been occupied for n1n2 subscribe.
Severity	Minor
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_N1N2MessageSubscribe"})/360000) * 100 > 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.96
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to N1N2 subscribe reaches 70% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with the flow triggering N1N2 subscription or on the AMF side and possibly enable the direct/indirect alternate routing. Cause: More than 70% of timer capacity has been occupied for N1N2 subscription. The occnp_timer_capacity metric tracks the current count of active timers. These timers are created when the User Equipment (UE) fails to deliver URSP rules, retries after back-off, and creates a new timer. This alert is triggered when timer capacity for N1N2 subscription exceeds 70% of the total 360K capacity. Diagnostic Information: Frequent UE Failures in N1N2 Subscription Flows: Multiple User Equipment (UE) devices may be repeatedly failing to complete N1N2 subscription actions, resulting in retries and new timer creations. Persistent Delivery or Communication Issues: Failures in delivering URSP rules or problems communicating with the AMF or other network functions might cause UEs to retrigger the N1N2 subscription flow. Underlying AMF or Network Instability: Instability, health issues, or misconfigurations in the AMF (Access and Mobility Management Function) could prevent successful subscription completion, leading to increased timers. High Traffic Volume or Spikes: Unexpectedly high volumes of N1N2 subscription requests can cause a large number of timers to be in use concurrently. Resource Limitations or Performance Bottlenecks: Processing delays or resource bottlenecks (CPU, memory, network) within the UE-service or supporting backend could slow down or block subscription handling, causing timers to accumulate. Improper Timer or Retry Configuration: Short retry intervals or misconfigured back-off could lead to rapid, repeated subscription attempts and excessive timer usage. Recovery: Review logs and N1N2 subscription flow metrics for unusual error patterns. Investigate AMF and related network function health and recent changes. Check configuration for timer parameters and adjust if necessary. Monitor for spikes in traffic or unusual load patterns. If the issue persists, please check with Support team.

8.1.2.33 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_SUBSCRIBE_ABOVE_MAJOR_THRESHOLD

Table 8-168 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_SUBSCRIBE_ABOVE_MAJOR_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_SUBSCRIBE_ABOVE_MAJOR_THRESHOLD
Description	More than 80% of timer capacity has been occupied for n1n2 subscribe.
Summary	More than 80% of timer capacity has been occupied for n1n2 subscribe.
Severity	Major
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_N1N2MessageSubscribe"})/360000) * 100 > 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.96
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to N1N2 subscribe reaches 80% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with the flow triggering N1N2 subscription or on the AMF side and possibly enable the direct/indirect alternate routing. Cause: More than 80% of timer capacity has been occupied for N1N2 subscription. The occnp_timer_capacity metric tracks the current count of active timers. These timers are created when the User Equipment (UE) fails to deliver URSP rules, retries after back-off, and creates a new timer. This alert is triggered when timer capacity for N1N2 subscription exceeds 80% of the total 360K capacity. Diagnostic Information: Frequent UE Failures in N1N2 Subscription Flows: Multiple User Equipment (UE) devices may be repeatedly failing to complete N1N2 subscription actions, resulting in retries and new timer creations. Persistent Delivery or Communication Issues: Failures in delivering URSP rules or problems communicating with the AMF or other network functions might cause UEs to retrigger the N1N2 subscription flow. Underlying AMF or Network Instability: Instability, health issues, or misconfigurations in the AMF (Access and Mobility Management Function) could prevent successful subscription completion, leading to increased timers. High Traffic Volume or Spikes: Unexpectedly high volumes of N1N2 subscription requests can cause a large number of timers to be in use concurrently. Resource Limitations or Performance Bottlenecks: Processing delays or resource bottlenecks (CPU, memory, network) within the UE-service or supporting backend could slow down or block subscription handling, causing timers to accumulate. Improper Timer or Retry Configuration: Short retry intervals or misconfigured back-off could lead to rapid, repeated subscription attempts and excessive timer usage. Recovery: Review logs and N1N2 subscription flow metrics for unusual error patterns. Investigate AMF and related network function health and recent changes. Check configuration for timer parameters and adjust if necessary. Monitor for spikes in traffic or unusual load patterns. If the issue persists, please check with Support team.

8.1.2.34 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_SUBSCRIBE_ABOVE_CRITICAL_THRESHOLD

Table 8-169 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_SUBSCRIBE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_SUBSCRIBE_ABOVE_CRITICAL_THRESHOLD
Description	More than 90% of timer capacity has been occupied for n1n2 subscribe.
Summary	More than 90% of timer capacity has been occupied for n1n2 subscribe.
Severity	Critical
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_N1N2MessageSubscribe"})/360000) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.96
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to N1N2 subscribe reaches 90% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with the flow triggering N1N2 subscription or on the AMF side and possibly enable the direct/indirect alternate routing. Cause: More than 90% of timer capacity has been occupied for N1N2 subscription. The occnp_timer_capacity metric tracks the current count of active timers. These timers are created when the User Equipment (UE) fails to deliver URSP rules, retries after back-off, and creates a new timer. This alert is triggered when timer capacity for N1N2 subscription exceeds 90% of the total 360K capacity. Diagnostic Information: Frequent UE Failures in N1N2 Subscription Flows: Multiple User Equipment (UE) devices may be repeatedly failing to complete N1N2 subscription actions, resulting in retries and new timer creations. Persistent Delivery or Communication Issues: Failures in delivering URSP rules or problems communicating with the AMF or other network functions might cause UEs to retrigger the N1N2 subscription flow. Underlying AMF or Network Instability: Instability, health issues, or misconfigurations in the AMF (Access and Mobility Management Function) could prevent successful subscription completion, leading to increased timers. High Traffic Volume or Spikes: Unexpectedly high volumes of N1N2 subscription requests can cause a large number of timers to be in use concurrently. Resource Limitations or Performance Bottlenecks: Processing delays or resource bottlenecks (CPU, memory, network) within the UE-service or supporting backend could slow down or block subscription handling, causing timers to accumulate. Improper Timer or Retry Configuration: Short retry intervals or misconfigured back-off could lead to rapid, repeated subscription attempts and excessive timer usage. Recovery: Review logs and N1N2 subscription flow metrics for unusual error patterns. Investigate AMF and related network function health and recent changes. Check configuration for timer parameters and adjust if necessary. Monitor for spikes in traffic or unusual load patterns. If the issue persists, please check with Support team.

8.1.2.35 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_ABOVE_MINOR_THRESHOLD

Table 8-170 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_ABOVE_MINOR_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_ABOVE_MINOR_THRESHOLD
Description	More than 70% of timer capacity has been occupied for n1n2 transfer.
Summary	More than 70% of timer capacity has been occupied for n1n2 transfer.
Severity	Minor
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_N1N2MessageTransfer"})/360000) * 100 > 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.97
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to N1N2 transfer subscribe reaches 70% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with the flow triggering N1N2 transfer and possibly enable direct/indirect alternate routing. Cause: More than 70% of timer capacity has been occupied for N1N2 transfer. The occnp_timer_capacity metric tracks the current count of active timers. These timers are created when the User Equipment (UE) fails to deliver URSP rules, retries after a back-off, and creates a new timer. This alert is triggered when the timer capacity for N1N2 transfer exceeds 70% of the total 360K capacity. Diagnostic Information: Frequent UE Failures in N1N2 Transfer Flows: Many User Equipment (UE) devices are failing to complete N1N2 transfer operations successfully. Each failure leads to retries and the creation of new timers. Delivery or Communication Issues: Persistent network issues preventing successful URSP rule delivery or failures in communication between the UE, AMF (Access and Mobility Management Function), or other relevant network functions can result in repeated N1N2 transfer attempts. Resource Constraints or Performance Bottlenecks: Limited processing resources, high latency, or overload conditions (e.g., CPU/memory/network contention) can slow down or block the completion of transfer requests, causing timers to accumulate. High Volume of Requests: An increased volume of N1N2 transfer requests due to network events or abnormal UE behavior can lead to a rapid consumption of available timer capacity. Improper Timer Configuration: Short back-off intervals or aggressive retry settings can cause repeated rapid reattempts, increasing the number of concurrent timers. AMF or Other NF Instability: Outages or instability in the AMF or related network functions may cause requests to go unprocessed, triggering continual retries from UEs. Recovery: Review recent logs and metrics related to N1N2 transfer failures. Investigate the health status of the AMF and other supporting NFs. Check resource utilization and adjust timer back-off/retry configuration if needed. Look for recent network changes or spikes in request volume. If the issue persists, please check with Support team.

8.1.2.36 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_ABOVE_MAJOR_THRESHOLD

Table 8-171 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_ABOVE_MAJOR_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_ABOVE_MAJOR_THRESHOLD
Description	More than 80% of timer capacity has been occupied for n1n2 transfer.
Summary	More than 80% of timer capacity has been occupied for n1n2 transfer.
Severity	Major
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_N1N2MessageTransfer"})/360000) * 100 > 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.97
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to N1N2 transfer subscribe reaches 80% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with the flow triggering N1N2 transfer and possibly enable direct/indirect alternate routing. Cause: More than 80% of timer capacity has been occupied for N1N2 transfer. The occnp_timer_capacity metric tracks the current count of active timers. These timers are created when the User Equipment (UE) fails to deliver URSP rules, retries after a back-off, and creates a new timer. This alert is triggered when the timer capacity for N1N2 transfer exceeds 80% of the total 360K capacity. Diagnostic Information: Frequent UE Failures in N1N2 Transfer Flows: Many User Equipment (UE) devices are failing to complete N1N2 transfer operations successfully. Each failure leads to retries and the creation of new timers. Delivery or Communication Issues: Persistent network issues preventing successful URSP rule delivery or failures in communication between the UE, AMF (Access and Mobility Management Function), or other relevant network functions can result in repeated N1N2 transfer attempts. Resource Constraints or Performance Bottlenecks: Limited processing resources, high latency, or overload conditions (e.g., CPU/memory/network contention) can slow down or block the completion of transfer requests, causing timers to accumulate. High Volume of Requests: An increased volume of N1N2 transfer requests due to network events or abnormal UE behavior can lead to a rapid consumption of available timer capacity. Improper Timer Configuration: Short back-off intervals or aggressive retry settings can cause repeated rapid reattempts, increasing the number of concurrent timers. AMF or Other NF Instability: Outages or instability in the AMF or related network functions may cause requests to go unprocessed, triggering continual retries from UEs. Recovery: Review recent logs and metrics related to N1N2 transfer failures. Investigate the health status of the AMF and other supporting NFs. Check resource utilization and adjust timer back-off/retry configuration if needed. Look for recent network changes or spikes in request volume. If the issue persists, please check with Support team.

8.1.2.37 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_ABOVE_CRITICAL_THRESHOLD

Table 8-172 AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name in Alert Yaml File	AUDIT_TIMER_CAPACITY_FOR_UE_N1N2_TRANSFER_ABOVE_CRITICAL_THRESHOLD
Description	More than 90% of timer capacity has been occupied for n1n2 transfer.
Summary	More than 90% of timer capacity has been occupied for n1n2 transfer.
Severity	Critical
Expression	(max by (namespace) (occnp_timer_capacity{timerName="UE_N1N2MessageTransfer"})/360000) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.97
Metric Used	occnp_timer_capacity
Recommended Actions	The occnp_timer_capacity metric is pegged during each timer scan, providing the current timers count. These timers were created when UE was not able to deliver the URSP rules and reattempt with back off. In this scenario an alert is triggered when the timers capacity corresponding to N1N2 transfer subscribe reaches 90% of the maximum rate limit of 360K. In this case the operator can troubleshoot and identify the reasons for failures with the flow triggering N1N2 transfer and possibly enable direct/indirect alternate routing. Cause: More than 90% of timer capacity has been occupied for N1N2 transfer. The occnp_timer_capacity metric tracks the current count of active timers. These timers are created when the User Equipment (UE) fails to deliver URSP rules, retries after a back-off, and creates a new timer. This alert is triggered when the timer capacity for N1N2 transfer exceeds 90% of the total 360K capacity. Diagnostic Information: Frequent UE Failures in N1N2 Transfer Flows: Many User Equipment (UE) devices are failing to complete N1N2 transfer operations successfully. Each failure leads to retries and the creation of new timers. Delivery or Communication Issues: Persistent network issues preventing successful URSP rule delivery or failures in communication between the UE, AMF (Access and Mobility Management Function), or other relevant network functions can result in repeated N1N2 transfer attempts. Resource Constraints or Performance Bottlenecks: Limited processing resources, high latency, or overload conditions (e.g., CPU/memory/network contention) can slow down or block the completion of transfer requests, causing timers to accumulate. High Volume of Requests: An increased volume of N1N2 transfer requests due to network events or abnormal UE behavior can lead to a rapid consumption of available timer capacity. Improper Timer Configuration: Short back-off intervals or aggressive retry settings can cause repeated rapid reattempts, increasing the number of concurrent timers. AMF or Other NF Instability: Outages or instability in the AMF or related network functions may cause requests to go unprocessed, triggering continual retries from UEs. Recovery: Review recent logs and metrics related to N1N2 transfer failures. Investigate the health status of the AMF and other supporting NFs. Check resource utilization and adjust timer back-off/retry configuration if needed. Look for recent network changes or spikes in request volume. If the issue persists, please check with Support team.

8.1.2.38 UE_N1N2_SUBSCRIBE_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD

Table 8-173 UE_N1N2_SUBSCRIBE_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_N1N2_SUBSCRIBE_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD
Description	More than 25% of n1n2 subscribe reattempt failed.
Summary	More than 25% of n1n2 subscribe reattempt failed.
Severity	Minor
Expression	(sum by (namespace) (increase(http_out_conn_response_total{isReattempt="true",operationType="subscribe",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(http_out_conn_request_total{isReattempt="true",operationType="subscribe"}[5m]))) * 100 > 25
OID	1.3.6.1.4.1.323.5.3.52.1.2.99
Metric Used	http_out_conn_response_total, http_out_conn_request_total
Recommended Actions	The http_out_conn_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. In this case the alert is notifying when there is a certain amount of reattempt failure for ue n1n2 subscribe. If there is an increase of failure, operator can revise the reason why the flow triggering n1n2 subscription is failing or if the AMF that request are going to is unhealthy. Cause: An elevated percentage of reattempt failures has been detected for UE N1N2 subscriptions. The http_out_conn_response_total metric increments whenever the PCF-UE receives a response for outbound messages, specifically tracking reattempts where the operation type is "subscribe" and the response code is not in the 2xx (success) range. This alert triggers when more than 25% of such reattempts fail over a 5-minute period. Diagnostic Information: AMF (Access and Mobility Management Function) Unavailability or Instability: The target AMF may be experiencing outages, heavy load, or is otherwise unhealthy, causing it to reject or fail to respond to subscription requests. Network Issues or Communication Failures: Network congestion, routing problems, or transient communication errors may prevent successful delivery of N1N2 subscription requests or receipt of responses. Configuration Errors: Misconfiguration of endpoints (such as incorrect URLs, authentication, or authorization settings) may cause subscription requests to be rejected or fail. High Load or Resource Exhaustion: If the AMF or intermediate network components are overloaded or have run out of necessary resources (e.g., memory, threads, process slots), reattempted requests may be rejected. Timeouts or Latency Issues: Prolonged delays in response times could cause requests to time out, leading to apparent failures. Recovery: Review logs and error codes for patterns or specific failure reasons. Check the health and recent activity of the AMF(s) and relevant network paths. Examine configuration settings related to N1N2 subscriptions and ensure they are correct. Investigate any spikes in load or indications of resource bottlenecks. Correlate with recent changes or deployments in the environment. If the issue persists, please check with Support team.

8.1.2.39 UE_N1N2_SUBSCRIBE_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD

Table 8-174 UE_N1N2_SUBSCRIBE_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_N1N2_SUBSCRIBE_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD
Description	More than 50% of n1n2 subscribe reattempt failed.
Summary	More than 50% of n1n2 subscribe reattempt failed.
Severity	Major
Expression	(sum by (namespace) (increase(http_out_conn_response_total{isReattempt="true",operationType="subscribe",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(http_out_conn_request_total{isReattempt="true",operationType="subscribe"}[5m]))) * 100 > 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.99
Metric Used	http_out_conn_response_total, http_out_conn_request_total
Recommended Actions	The http_out_conn_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. In this case the alert is notifying when there is a certain amount of reattempt failure for ue n1n2 subscribe.If there is an increase of failure, operator can revise the reason why the flow triggering n1n2 subscription is failing or if the AMF that request are going to is unhealthy. Cause: An elevated percentage of reattempt failures has been detected for UE N1N2 subscriptions. The http_out_conn_response_total metric increments whenever the PCF-UE receives a response for outbound messages, specifically tracking reattempts where the operation type is "subscribe" and the response code is not in the 2xx (success) range. This alert triggers when more than 50% of such reattempts fail over a 5-minute period. Diagnostic Information: AMF (Access and Mobility Management Function) Unavailability or Instability: The target AMF may be experiencing outages, heavy load, or is otherwise unhealthy, causing it to reject or fail to respond to subscription requests. Network Issues or Communication Failures: Network congestion, routing problems, or transient communication errors may prevent successful delivery of N1N2 subscription requests or receipt of responses. Configuration Errors: Misconfiguration of endpoints (such as incorrect URLs, authentication, or authorization settings) may cause subscription requests to be rejected or fail. High Load or Resource Exhaustion: If the AMF or intermediate network components are overloaded or have run out of necessary resources (e.g., memory, threads, process slots), reattempted requests may be rejected. Timeouts or Latency Issues:Prolonged delays in response times could cause requests to time out, leading to apparent failures. Recovery: Review logs and error codes for patterns or specific failure reasons. Check the health and recent activity of the AMF(s) and relevant network paths. Examine configuration settings related to N1N2 subscriptions and ensure they are correct. Investigate any spikes in load or indications of resource bottlenecks. Correlate with recent changes or deployments in the environment. If the issue persists, please check with Support team.

8.1.2.40 UE_N1N2_SUBSCRIBE_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD

Table 8-175 UE_N1N2_SUBSCRIBE_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_N1N2_SUBSCRIBE_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD
Description	More than 75% of n1n2 subscribe reattempt failed.
Summary	More than 75% of n1n2 subscribe reattempt failed.
Severity	Critical
Expression	(sum by (namespace) (increase(http_out_conn_response_total{isReattempt="true",operationType="subscribe",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(http_out_conn_request_total{isReattempt="true",operationType="subscribe"}[5m]))) * 100 > 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.99
Metric Used	http_out_conn_response_total, http_out_conn_request_total
Recommended Actions	The http_out_conn_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. In this case the alert is notifying when there is a certain amount of reattempt failure for ue n1n2 subscribe.If there is an increase of failure, operator can revise the reason why the flow triggering n1n2 subscription is failing or if the AMF that request are going to is unhealthy. Cause: An elevated percentage of reattempt failures has been detected for UE N1N2 subscriptions. The http_out_conn_response_total metric increments whenever the PCF-UE receives a response for outbound messages, specifically tracking reattempts where the operation type is "subscribe" and the response code is not in the 2xx (success) range. This alert triggers when more than 75% of such reattempts fail over a 5-minute period. Diagnostic Information: AMF (Access and Mobility Management Function) Unavailability or Instability: The target AMF may be experiencing outages, heavy load, or is otherwise unhealthy, causing it to reject or fail to respond to subscription requests. Network Issues or Communication Failures: Network congestion, routing problems, or transient communication errors may prevent successful delivery of N1N2 subscription requests or receipt of responses. Configuration Errors: Misconfiguration of endpoints (such as incorrect URLs, authentication, or authorization settings) may cause subscription requests to be rejected or fail. High Load or Resource Exhaustion: If the AMF or intermediate network components are overloaded or have run out of necessary resources (e.g., memory, threads, process slots), reattempted requests may be rejected. Timeouts or Latency Issues:Prolonged delays in response times could cause requests to time out, leading to apparent failures. Recovery: Review logs and error codes for patterns or specific failure reasons. Check the health and recent activity of the AMF(s) and relevant network paths. Examine configuration settings related to N1N2 subscriptions and ensure they are correct. Investigate any spikes in load or indications of resource bottlenecks. Correlate with recent changes or deployments in the environment. If the issue persists, please check with Support team.

8.1.2.41 UE_N1N2_TRANSFER_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD

Table 8-176 UE_N1N2_TRANSFER_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_N1N2_TRANSFER_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD
Description	More than 25% of n1n2 transfer reattempt failed.
Summary	More than 25% of n1n2 transfer reattempt failed.
Severity	Minor
Expression	(sum by (namespace) (increase(http_out_conn_response_total{isReattempt="true",reattemptType="UE_N1N2MessageTransfer", operationType="transfer",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(http_out_conn_request_total{isReattempt="true",reattemptType="UE_N1N2MessageTransfer", operationType="transfer"}[5m]))) * 100 > 25
OID	1.3.6.1.4.1.323.5.3.52.1.2.100
Metric Used	http_out_conn_response_total, http_out_conn_request_total
Recommended Actions	The http_out_conn_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. In this case the alert is notifying when there is a certain amount of reattempt failure for ue n1n2 transfer.If there is an increase of failure, operator can revise the reason why the flow triggering n1n2 message transfer is failing or if the AMF that request are going to is unhealthy. Cause: An increased percentage of reattempt failures has been detected for UE N1N2 message transfers. The http_out_conn_response_total metric increments when the PCF-UE receives a response for messages being sent out of the Network Function (NF), specifically monitoring reattempts of the "transfer" operation where the response code is not 2xx (success). This alert is triggered when over 25% of such reattempts result in failure within a 5-minute period. Diagnostic Information: AMF (Access and Mobility Management Function) Unavailability or Instability: If the target AMF is down, overloaded, or behaving unpredictably, message transfer requests (especially retries) are more likely to fail. Network Path Issues: Transient or persistent network failures, high latency, or packet loss between the PCF-UE and the target network function can disrupt the successful transfer of N1N2 messages. Configuration Errors: Misconfiguration in endpoints, credentials, or other protocol parameters can cause messages to be consistently rejected or fail to deliver. System Resource Constraints: Resource exhaustion (CPU, memory, file descriptors, etc.) on either the PCF-UE or the AMF could prevent successful handling of transfer requests. Timeouts and Slow Processing: Delayed responses or timeouts can be interpreted as failures, particularly if the operation times out consistently during high load or due to backend issues. Recovery: Review and analyze failure logs and returned error codes. Check the operational health and resource status of the AMF and other involved NFs. Validate network connectivity and latency between all relevant components. Inspect configuration and recent changes for potential misalignments. Correlate the timing of increased failures with network incidents, maintenance windows, or new deployments. If the issue persists, please check with Support team.

8.1.2.42 UE_N1N2_TRANSFER_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD

Table 8-177 UE_N1N2_TRANSFER_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_N1N2_TRANSFER_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD
Description	More than 50% of n1n2 transfer reattempt failed.
Summary	More than 50% of n1n2 transfer reattempt failed.
Severity	Major
Expression	(sum by (namespace) (increase(http_out_conn_response_total{isReattempt="true",reattemptType="UE_N1N2MessageTransfer", operationType="transfer",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(http_out_conn_request_total{isReattempt="true",reattemptType="UE_N1N2MessageTransfer", operationType="transfer"}[5m]))) * 100 > 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.100
Metric Used	http_out_conn_response_total, http_out_conn_request_total
Recommended Actions	The http_out_conn_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. In this case the alert is notifying when there is a certain amount of reattempt failure for ue n1n2 transfer.If there is an increase of failure, operator can revise the reason why the flow triggering n1n2 message transfer is failing or if the AMF that request are going to is unhealthy. Cause: An increased percentage of reattempt failures has been detected for UE N1N2 message transfers. The http_out_conn_response_total metric increments when the PCF-UE receives a response for messages being sent out of the Network Function (NF), specifically monitoring reattempts of the "transfer" operation where the response code is not 2xx (success). This alert is triggered when over 50% of such reattempts result in failure within a 5-minute period. Diagnostic Information: AMF (Access and Mobility Management Function) Unavailability or Instability: If the target AMF is down, overloaded, or behaving unpredictably, message transfer requests (especially retries) are more likely to fail. Network Path Issues: Transient or persistent network failures, high latency, or packet loss between the PCF-UE and the target network function can disrupt the successful transfer of N1N2 messages. Configuration Errors: Misconfiguration in endpoints, credentials, or other protocol parameters can cause messages to be consistently rejected or fail to deliver. System Resource Constraints: Resource exhaustion (CPU, memory, file descriptors, etc.) on either the PCF-UE or the AMF could prevent successful handling of transfer requests. Timeouts and Slow Processing: Delayed responses or timeouts can be interpreted as failures, particularly if the operation times out consistently during high load or due to backend issues. Recovery: Review and analyze failure logs and returned error codes. Check the operational health and resource status of the AMF and other involved NFs. Validate network connectivity and latency between all relevant components. Inspect configuration and recent changes for potential misalignments. Correlate the timing of increased failures with network incidents, maintenance windows, or new deployments. If the issue persists, please check with Support team.

8.1.2.43 UE_N1N2_TRANSFER_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD

Table 8-178 UE_N1N2_TRANSFER_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_N1N2_TRANSFER_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD
Description	More than 75% of n1n2 transfer reattempt failed.
Summary	More than 75% of n1n2 transfer reattempt failed.
Severity	Critical
Expression	(sum by (namespace) (increase(http_out_conn_response_total{isReattempt="true",reattemptType="UE_N1N2MessageTransfer", operationType="transfer",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(http_out_conn_request_total{isReattempt="true",reattemptType="UE_N1N2MessageTransfer", operationType="transfer"}[5m]))) * 100 > 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.100
Metric Used	http_out_conn_response_total, http_out_conn_request_total
Recommended Actions	The http_out_conn_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. In this case the alert is notifying when there is a certain amount of reattempt failure for ue n1n2 transfer.If there is an increase of failure, operator can revise the reason why the flow triggering n1n2 message transfer is failing or if the AMF that request are going to is unhealthy. Cause: An increased percentage of reattempt failures has been detected for UE N1N2 message transfers. The http_out_conn_response_total metric increments when the PCF-UE receives a response for messages being sent out of the Network Function (NF), specifically monitoring reattempts of the "transfer" operation where the response code is not 2xx (success). This alert is triggered when over 75% of such reattempts result in failure within a 5-minute period. Diagnostic Information: AMF (Access and Mobility Management Function) Unavailability or Instability: If the target AMF is down, overloaded, or behaving unpredictably, message transfer requests (especially retries) are more likely to fail. Network Path Issues: Transient or persistent network failures, high latency, or packet loss between the PCF-UE and the target network function can disrupt the successful transfer of N1N2 messages. Configuration Errors: Misconfiguration in endpoints, credentials, or other protocol parameters can cause messages to be consistently rejected or fail to deliver. System Resource Constraints: Resource exhaustion (CPU, memory, file descriptors, etc.) on either the PCF-UE or the AMF could prevent successful handling of transfer requests. Timeouts and Slow Processing: Delayed responses or timeouts can be interpreted as failures, particularly if the operation times out consistently during high load or due to backend issues. Recovery: Review and analyze failure logs and returned error codes. Check the operational health and resource status of the AMF and other involved NFs. Validate network connectivity and latency between all relevant components. Inspect configuration and recent changes for potential misalignments. Correlate the timing of increased failures with network incidents, maintenance windows, or new deployments. If the issue persists, please check with Support team.

8.1.2.44 SM_STALE_REQUEST_PROCESSING_REJECT_MINOR

Table 8-179 SM_STALE_REQUEST_PROCESSING_REJECT_MINOR

Field	Details
Name in Alert Yaml File	SM_STALE_REQUEST_PROCESSING_REJECT_MINOR
Description	More than 10% of the Ingress requests failed with error 504 GATEWAY_TIMEOUT due to request being stale
Summary	More than 10% of the Ingress requests failed with error 504 GATEWAY_TIMEOUT due to request being stale
Severity	Minor
Expression	(sum by (namespace,pod) (rate(occnp_late_processing_rejection_total{microservice=~"occnp_pcf_sm"}[5m])))/(sum by (namespace,pod) (rate(ocpm_ingress_request_total{microservice=~"occnp_pcf_sm"}[5m]))) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.101
Metric Used	occnp_late_processing_rejection_total, ocpm_ingress_request_total
Recommended Actions	The metric occnp_late_processing_rejection_total is pegged when Late Processing finds a stale session. Cause: The metric occnp_late_processing_rejection_total is incremented when the SM Service determines that a request has become stale. For example, if a request includes the following header parameters: sbiSenderTimestamp3GPP='2025-11-03T09:48:01.000Z' (sender timestamp) sbiMaxRSPTime3GPP='3000' (maximum response time in milliseconds) In this scenario, if there is a delay in receiving a response from the external Network Function (NF), a stale check is later performed. If the request is deemed stale during this check, it is counted in the metric. When more than 10% of the Ingress requests failed with error 504 GATEWAY_TIMEOUT, then this alarm will be raised. Diagnostic Information: Validate Timestamps: Ensure that system clocks are synchronized (e.g., via NTP). Analyze Latency: Use tracing or metric data to identify bottlenecks in response time—look for patterns in external NF response delays. Review Configurations: Confirm that max response times (sbiMaxRSPTime3GPP) are correctly set as per the service contract. Scale System Resources: Check for resource constraints (CPU, memory, bandwidth) and scale up your system or services as needed to handle the incoming request load within the allowed response time. Recovery: Once the recommended diagnostic actions are implemented and responses from the external NF are received within the expected timeframe, the percentage of rejected messages will begin to decline, ultimately clearing the alert.

8.1.2.45 SM_STALE_REQUEST_PROCESSING_REJECT_MAJOR

Table 8-180 SM_STALE_REQUEST_PROCESSING_REJECT_MAJOR

Field	Details
Name in Alert Yaml File	SM_STALE_REQUEST_PROCESSING_REJECT_MAJOR
Description	More than 20% of the Ingress requests failed with error 504 GATEWAY_TIMEOUT due to request being stale
Summary	More than 20% of the Ingress requests failed with error 504 GATEWAY_TIMEOUT due to request being stale
Severity	Major
Expression	(sum by (namespace,pod) (rate(occnp_late_processing_rejection_total{microservice=~"occnp_pcf_sm"}[5m])))/(sum by (namespace,pod) (rate(ocpm_ingress_request_total{microservice=~"occnp_pcf_sm"}[5m]))) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.101
Metric Used	occnp_late_processing_rejection_total, ocpm_ingress_request_total
Recommended Actions	The metric occnp_late_processing_rejection_total is pegged when Late Processing finds a stale session. Cause: The metric occnp_late_processing_rejection_total is incremented when the SM Service determines that a request has become stale. For example, if a request includes the following header parameters: sbiSenderTimestamp3GPP='2025-11-03T09:48:01.000Z' (sender timestamp) sbiMaxRSPTime3GPP='3000' (maximum response time in milliseconds) In this scenario, if there is a delay in receiving a response from the external Network Function (NF), a stale check is later performed. If the request is deemed stale during this check, it is counted in the metric. When more than 20% and less than 30% of the Ingress requests failed with error 504 GATEWAY_TIMEOUT, then this alarm will be raised. Diagnostic Information: Validate Timestamps: Ensure that system clocks are synchronized (e.g., via NTP). Analyze Latency: Use tracing or metric data to identify bottlenecks in response time—look for patterns in external NF response delays. Review Configurations: Confirm that max response times (sbiMaxRSPTime3GPP) are correctly set as per the service contract. Scale System Resources: Check for resource constraints (CPU, memory, bandwidth) and scale up your system or services as needed to handle the incoming request load within the allowed response time. Recovery: Once the recommended diagnostic actions are implemented and responses from the external NF are received within the expected timeframe, the percentage of rejected messages will begin to decline, ultimately clearing the alert.

8.1.2.46 SM_STALE_REQUEST_PROCESSING_REJECT_CRITICAL

Table 8-181 SM_STALE_REQUEST_PROCESSING_REJECT_CRITICAL

Field	Details
Name in Alert Yaml File	SM_STALE_REQUEST_PROCESSING_REJECT_CRITICAL
Description	More than 30% of the Ingress requests failed with error 504 GATEWAY_TIMEOUT due to request being stale
Summary	More than 30% of the Ingress requests failed with error 504 GATEWAY_TIMEOUT due to request being stale
Severity	Critical
Expression	(sum by (namespace,pod) (rate(occnp_late_processing_rejection_total{microservice=~"occnp_pcf_sm"}[5m])))/(sum by (namespace,pod) (rate(ocpm_ingress_request_total{microservice=~"occnp_pcf_sm"}[5m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.101
Metric Used	occnp_late_processing_rejection_total, ocpm_ingress_request_total
Recommended Actions	The metric occnp_late_processing_rejection_total is pegged when Late Processing finds a stale session. Cause: The metric occnp_late_processing_rejection_total is incremented when the SM Service determines that a request has become stale. For example, if a request includes the following header parameters: sbiSenderTimestamp3GPP='2025-11-03T09:48:01.000Z' (sender timestamp) sbiMaxRSPTime3GPP='3000' (maximum response time in milliseconds) In this scenario, if there is a delay in receiving a response from the external Network Function (NF), a stale check is later performed. If the request is deemed stale during this check, it is counted in the metric. When more than 30% of the Ingress requests failed with error 504 GATEWAY_TIMEOUT, then this alarm will be raised. Diagnostic Information: Validate Timestamps: Ensure that system clocks are synchronized (e.g., via NTP). Analyze Latency: Use tracing or metric data to identify bottlenecks in response time—look for patterns in external NF response delays. Review Configurations: Confirm that max response times (sbiMaxRSPTime3GPP) are correctly set as per the service contract. Scale System Resources: Check for resource constraints (CPU, memory, bandwidth) and scale up your system or services as needed to handle the incoming request load within the allowed response time. Recovery: Once the recommended diagnostic actions are implemented and responses from the external NF are received within the expected timeframe, the percentage of rejected messages will begin to decline, ultimately clearing the alert.

8.1.2.47 UE_STALE_REQUEST_PROCESSING_REJECT_MAJOR

Table 8-182 UE_STALE_REQUEST_PROCESSING_REJECT_MAJOR

Field	Details
Description	This alert is triggered when more than 20% of the incoming requests towards UE Policy service are rejected due to request going stale, while being processed by the service.
Summary	This alert is triggered when more than 20% of the incoming requests towards UE Policy service are rejected due to request going stale, while being processed by the service.
Severity	Major
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{microservice=~".*pcf_ueservice"}[5m])) / sum by (namespace) (rate(ocpm_ingress_request_total{microservice=~".*pcf_ueservice"}[5m]))) * 100 > 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.104
Metric Used	occnp_late_processing_rejection_total
Recommended Actions	Metric occnp_late_processing_rejection_total is pegged when requests being processed become stale. Cause: More than 20% of incoming requests to the ue-service have been rejected because they became stale during processing. The service flags a request as stale when its processing exceeds an acceptable time window. Diagnostic Information: High System Load or Resource Contention: The ue-service or its backend components may be overloaded (e.g., CPU, memory, I/O), delaying request processing. Inefficient Request Handling or Bottlenecks: There may be inefficiencies or slow operations within the service logic, such as database queries, API calls, or complex computations causing extended processing times. Network Latency or Downstream Delays: High network latency or slow responses from dependent services or databases could increase the time required to process requests. Increased Volume of Requests: A spike in incoming requests can overwhelm the service, leading to request queues and increased wait times. Recovery: Monitor system and service resource utilization. Review recent changes to workload, configuration, or deployments. Tune timeouts and thresholds appropriately based on observed service latency. Analyze logs to pinpoint where delays are occurring in the request processing workflow. If the issue persists, please check with Support team.

8.1.2.48 UE_STALE_REQUEST_PROCESSING_REJECT_CRITICAL

Table 8-183 UE_STALE_REQUEST_PROCESSING_REJECT_CRITICAL

Field	Details
Description	This alert is triggered when more than 30% of the incoming requests towards UE Policy service are rejected due to request going stale, while being processed by the service.
Summary	This alert is triggered when more than 20% of the incoming requests towards UE Policy service are rejected due to request going stale, while being processed by the service.
Severity	Critical
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{microservice=~".*pcf_ueservice"}[5m])) / sum by (namespace) (rate(ocpm_ingress_request_total{microservice=~".*pcf_ueservice"}[5m]))) * 100 > 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.104
Metric Used	occnp_late_processing_rejection_total
Recommended Actions	Metric occnp_late_processing_rejection_total is pegged when requests being processed become stale. Cause: More than 30% of incoming requests to the ue-service have been rejected because they became stale during processing. The service flags a request as stale when its processing exceeds an acceptable time window. Diagnostic Information: High System Load or Resource Contention: The ue-service or its backend components may be overloaded (e.g., CPU, memory, I/O), delaying request processing. Inefficient Request Handling or Bottlenecks: There may be inefficiencies or slow operations within the service logic, such as database queries, API calls, or complex computations causing extended processing times. Network Latency or Downstream Delays: High network latency or slow responses from dependent services or databases could increase the time required to process requests. Increased Volume of Requests: A spike in incoming requests can overwhelm the service, leading to request queues and increased wait times. Recovery: Monitor system and service resource utilization. Review recent changes to workload, configuration, or deployments. Tune timeouts and thresholds appropriately based on observed service latency. Analyze logs to pinpoint where delays are occurring in the request processing workflow. If the issue persists, please check with Support team.

8.1.2.49 UE_STALE_REQUEST_PROCESSING_REJECT_MINOR

Table 8-184 UE_STALE_REQUEST_PROCESSING_REJECT_MINOR

Field	Details
Description	This alert is triggered when more than 10% of the incoming requests towards UE Policy service are rejected due to request going stale, while being processed by the service.
Summary	This alert is triggered when more than 10% of the incoming requests towards UE Policy service are rejected due to request going stale, while being processed by the service.
Severity	Minor
Expression	(sum by (namespace) (rate(occnp_late_processing_rejection_total{microservice=~".*pcf_ueservice"}[5m])) / sum by (namespace) (rate(ocpm_ingress_request_total{microservice=~".*pcf_ueservice"}[5m]))) * 100 > 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.104
Metric Used	occnp_late_processing_rejection_total
Recommended Actions	Metric occnp_late_processing_rejection_total is pegged when requests being processed become stale. Cause: More than 10% of incoming requests to the ue-service have been rejected because they became stale during processing. The service flags a request as stale when its processing exceeds an acceptable time window. Diagnostic Information: High System Load or Resource Contention: The ue-service or its backend components may be overloaded (e.g., CPU, memory, I/O), delaying request processing. Inefficient Request Handling or Bottlenecks: There may be inefficiencies or slow operations within the service logic, such as database queries, API calls, or complex computations causing extended processing times. Network Latency or Downstream Delays: High network latency or slow responses from dependent services or databases could increase the time required to process requests. Increased Volume of Requests: A spike in incoming requests can overwhelm the service, leading to request queues and increased wait times. Recovery: Monitor system and service resource utilization. Review recent changes to workload, configuration, or deployments. Tune timeouts and thresholds appropriately based on observed service latency. Analyze logs to pinpoint where delays are occurring in the request processing workflow. If the issue persists, please check with Support team.

8.1.2.50 UE_STALE_REQUEST_ARRIVAL_REJECT_MINOR

Table 8-185 UE_STALE_REQUEST_ARRIVAL_REJECT_MINOR

Field	Details
Description	This alert is triggered when more than 10% of the incoming requests towards UE Policy service are rejected due to requests being stale upon arrival to the service.
Summary	This alert is triggered when more than 10% of the incoming requests towards UE Policy service are rejected due to requests being stale upon arrival to the service.
Severity	Minor
Expression	(sum by (namespace) (rate(ocpm_late_arrival_rejection_total{microservice=~".*pcf_ueservice"}[5m])) / sum by (namespace)(rate(ocpm_ingress_request_total{microservice=~".*pcf_ueservice"}[5m]))) * 100 > 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.109
Metric Used	ocpm_late_arrival_rejection_total
Recommended Actions	Metric ocpm_late_arrival_rejection_total is pegged when a received requests is stale. Cause: Metric ocpm_late_arrival_rejection_total is pegged when a received requests is stale. Metric: ocpm_late_arrival_rejection_total Increments when the UE Service determines incoming requests are stale (arrived too late to process). The staleness check is based on: 3gpp-Sbi-Sender-Timestamp (preferred) 3gpp-Sbi-Origination-Timestamp (fallback if sender timestamp is unavailable) 3gpp-Sbi-Max-Rsp-Time (maximum allowed response time, in ms) Request Example: 3gpp-Sbi-Sender-Timestamp='2025-11-03T09:48:01.000Z' 3gpp-Sbi-Max-Rsp-Time='3000' (i.e., 3 seconds) If request arrives after (Sender-Timestamp + Max-Rsp-Time), it is considered stale and counted in the metric. Alarm Condition: If more than 10% of ingress requests result in 504 GATEWAY_TIMEOUT errors due to staleness, an alarm is raised. Diagnostic Information: Verify Time Synchronization Ensure all Network Functions (NFs) have synchronized system clocks (using NTP). Time drift between sender and UE Service may falsely trigger staleness. Check Network Latency Investigate possible network delays or congestion between external NF and the UE Service. High or unstable latency can lead to late arrival of requests. Analyze Sender Behavior Validate that the sending NF populates 3gpp-Sbi-Sender-Timestamp (or Origination-Timestamp) correctly. Misconfigured or delayed timestamping can corrupt staleness calculation. Assess Max Response Time Values Review if the 3gpp-Sbi-Max-Rsp-Time value is appropriate for your network and application conditions. Very short response times may not be feasible under current latency conditions. Review Application Load Monitor system/resource utilization (CPU, memory, queue lengths) on the UE Service. Resource exhaustion may delay request processing, even if requests arrive on time. Correlation with Other Metrics Examine related metrics such as total request counts, processing times, error types, etc., to identify trends. Check if certain sources or request types are consistently late. Check for Backlogs Review UE Service logs for any signs of backlogs, bottlenecks, or spikes in the request handling pipeline. Recovery: Verify Time Synchronization Ensure all relevant Network Functions (NFs) have correct system time. Resynchronize clocks if any drift is detected. Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF and the UE Service. Resolve any high latency or packet loss immediately if detected. Review UE Service Application & Resources Check the UE Service for high CPU/memory usage or any request processing backlogs. Restart or scale up resources temporarily if the system is overloaded. Contact Upstream NF Owners Notify owners of external NFs if they are sending delayed or incorrectly timestamped requests so they can take corrective action.

8.1.2.51 UE_STALE_REQUEST_ARRIVAL_REJECT_MAJOR

Table 8-186 UE_STALE_REQUEST_ARRIVAL_REJECT_MAJOR

Field	Details
Description	This alert is triggered when more than 20% of the incoming requests towards UE Policy service are rejected due to requests being stale upon arrival to the service.
Summary	This alert is triggered when more than 20% of the incoming requests towards UE Policy service are rejected due to requests being stale upon arrival to the service.
Severity	Major
Expression	(sum by (namespace) (rate(ocpm_late_arrival_rejection_total{microservice=~".*pcf_ueservice"}[5m])) / sum by (namespace)(rate(ocpm_ingress_request_total{microservice=~".*pcf_ueservice"}[5m]))) * 100 > 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.109
Metric Used	ocpm_late_arrival_rejection_total
Recommended Actions	Metric ocpm_late_arrival_rejection_total is pegged when a received requests is stale. Cause: Metric ocpm_late_arrival_rejection_total is pegged when a received requests is stale. Metric: ocpm_late_arrival_rejection_total Increments when the UE Service determines incoming requests are stale (arrived too late to process). The staleness check is based on: 3gpp-Sbi-Sender-Timestamp (preferred) 3gpp-Sbi-Origination-Timestamp (fallback if sender timestamp is unavailable) 3gpp-Sbi-Max-Rsp-Time (maximum allowed response time, in ms) Request Example: 3gpp-Sbi-Sender-Timestamp='2025-11-03T09:48:01.000Z' 3gpp-Sbi-Max-Rsp-Time='3000' (i.e., 3 seconds) If request arrives after (Sender-Timestamp + Max-Rsp-Time), it is considered stale and counted in the metric. Alarm Condition: If more than 20% of ingress requests result in 504 GATEWAY_TIMEOUT errors due to staleness, an alarm is raised. Diagnostic Information: Verify Time Synchronization Ensure all Network Functions (NFs) have synchronized system clocks (using NTP). Time drift between sender and UE Service may falsely trigger staleness. Check Network Latency Investigate possible network delays or congestion between external NF and the UE Service. High or unstable latency can lead to late arrival of requests. Analyze Sender Behavior Validate that the sending NF populates 3gpp-Sbi-Sender-Timestamp (or Origination-Timestamp) correctly. Misconfigured or delayed timestamping can corrupt staleness calculation. Assess Max Response Time Values Review if the 3gpp-Sbi-Max-Rsp-Time value is appropriate for your network and application conditions. Very short response times may not be feasible under current latency conditions. Review Application Load Monitor system/resource utilization (CPU, memory, queue lengths) on the UE Service. Resource exhaustion may delay request processing, even if requests arrive on time. Correlation with Other Metrics Examine related metrics such as total request counts, processing times, error types, etc., to identify trends. Check if certain sources or request types are consistently late. Check for Backlogs Review UE Service logs for any signs of backlogs, bottlenecks, or spikes in the request handling pipeline. Recovery: Verify Time Synchronization Ensure all relevant Network Functions (NFs) have correct system time. Resynchronize clocks if any drift is detected. Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF and the UE Service. Resolve any high latency or packet loss immediately if detected. Review UE Service Application & Resources Check the UE Service for high CPU/memory usage or any request processing backlogs. Restart or scale up resources temporarily if the system is overloaded. Contact Upstream NF Owners Notify owners of external NFs if they are sending delayed or incorrectly timestamped requests so they can take corrective action.

8.1.2.52 UE_STALE_REQUEST_ARRIVAL_REJECT_CRITICAL

Table 8-187 UE_STALE_REQUEST_ARRIVAL_REJECT_CRITICAL

Field	Details
Description	This alert is triggered when more than 30% of the incoming requests towards UE Policy service are rejected due to requests being stale upon arrival to the service.
Summary	This alert is triggered when more than 30% of the incoming requests towards UE Policy service are rejected due to requests being stale upon arrival to the service.
Severity	Critical
Expression	(sum by (namespace) (rate(ocpm_late_arrival_rejection_total{microservice=~".*pcf_ueservice"}[5m])) / sum by (namespace)(rate(ocpm_ingress_request_total{microservice=~".*pcf_ueservice"}[5m]))) * 100 > 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.109
Metric Used	ocpm_late_arrival_rejection_total
Recommended Actions	Metric ocpm_late_arrival_rejection_total is pegged when a received requests is stale. Cause: Metric ocpm_late_arrival_rejection_total is pegged when a received requests is stale. Metric: ocpm_late_arrival_rejection_total Increments when the UE Service determines incoming requests are stale (arrived too late to process). The staleness check is based on: 3gpp-Sbi-Sender-Timestamp (preferred) 3gpp-Sbi-Origination-Timestamp (fallback if sender timestamp is unavailable) 3gpp-Sbi-Max-Rsp-Time (maximum allowed response time, in ms) Request Example: 3gpp-Sbi-Sender-Timestamp='2025-11-03T09:48:01.000Z' 3gpp-Sbi-Max-Rsp-Time='3000' (i.e., 3 seconds) If request arrives after (Sender-Timestamp + Max-Rsp-Time), it is considered stale and counted in the metric. Alarm Condition: If more than 30% of ingress requests result in 504 GATEWAY_TIMEOUT errors due to staleness, an alarm is raised. Diagnostic Information: Verify Time Synchronization Ensure all Network Functions (NFs) have synchronized system clocks (using NTP). Time drift between sender and UE Service may falsely trigger staleness. Check Network Latency Investigate possible network delays or congestion between external NF and the UE Service. High or unstable latency can lead to late arrival of requests. Analyze Sender Behavior Validate that the sending NF populates 3gpp-Sbi-Sender-Timestamp (or Origination-Timestamp) correctly. Misconfigured or delayed timestamping can corrupt staleness calculation. Assess Max Response Time Values Review if the 3gpp-Sbi-Max-Rsp-Time value is appropriate for your network and application conditions. Very short response times may not be feasible under current latency conditions. Review Application Load Monitor system/resource utilization (CPU, memory, queue lengths) on the UE Service. Resource exhaustion may delay request processing, even if requests arrive on time. Correlation with Other Metrics Examine related metrics such as total request counts, processing times, error types, etc., to identify trends. Check if certain sources or request types are consistently late. Check for Backlogs Review UE Service logs for any signs of backlogs, bottlenecks, or spikes in the request handling pipeline. Recovery: Verify Time Synchronization Ensure all relevant Network Functions (NFs) have correct system time. Resynchronize clocks if any drift is detected. Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF and the UE Service. Resolve any high latency or packet loss immediately if detected. Review UE Service Application & Resources Check the UE Service for high CPU/memory usage or any request processing backlogs. Restart or scale up resources temporarily if the system is overloaded. Contact Upstream NF Owners Notify owners of external NFs if they are sending delayed or incorrectly timestamped requests so they can take corrective action.

8.1.2.53 UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD

Table 8-188 UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD
Description	More than 75% of N1N2 transfer failure notification reattempts failed.
Summary	More than 75% of N1N2 transfer failure notification reattempts failed.
Severity	Critical
Expression	(sum by (namespace) (increase(http_out_conn_response_total{isReattempt="true",reattemptType="UE_N1N2TransferFailure",operationType="transfer",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(http_out_conn_request_total{isReattempt="true",reattemptType="UE_N1N2TransferFailure",operationType="transfer"}[5m]))) * 100 > 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.106
Metric Used	http_out_conn_response_total, http_out_conn_request_total
Recommended Actions	The http_out_conn_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. Then in this case the alert notifies when there is a certain amount of reattempt failure for UE N1N2 transfer failure notification. If there is an increase of failure, operator can investigate on: Why the flow triggering N1N2 transfer failure notification is failing, or Check the health of the AMF to which the request are going to Cause: http_out_conn_response_total metric with indicated dimensions is pegged when PCF-UE receives a response for an outgoing reattempt transfer request triggered for N1N2TransferFailure notification. Dimensions: IsReattempt : true reattemptType : UE_N1N2TransferFailure OperationType : transfer ResponseCode : !2xx In this case more than 75% of outgoing transfer reattempts (due to N1N2TransferFailure as notified by AMF) receive a non-2xx (failure) response in the last 5 minutes (or selected sample frame) Diagnostic Information : Check Recent Logs: Analyze logs for both PCF-UE and Egress Gateway in the relevant namespace for error details (timestamps matching the period of alert). Focus on error responses: look for 4xx/5xx HTTP responses and their reasons. Correlate with Traffic Patterns: Determine if failures are for specific to certain AMFs or random. Check if there's a sudden surge in failures (indicating a broader issue). Inspect Network Health and Configuration: Ensure connectivity and correct routing between PCF-UE and its downstream targets. Validate configurations, especially recently changed ones. Cross-check Incident/Event Timeline: Review recent maintenance, deployments, or network events that could correlate with the increase in failures. Evaluate for Service Overload: Examine resource metrics (CPU, memory, rate of requests) of the affected service(PCF-UE, PCF-EGW) to determine if it’s under duress. Check with Peers: See if corresponding namespaces (other tenants/products) are seeing similar issues, could indicate a platform or shared service problem. Recovery : Resolve Underlying Service Issues: If the upstream service (e.g., AMF or other network function) is unhealthy, work with the respective team to restore normal operation. Address any misconfiguration or errors causing repeated non-2xx responses. Revert Recent Changes: If the issue correlates with recent deployments or configuration changes, consider rolling back to the previous stable state after assessing impact. Mitigate Service Overload: If resource constraints are detected (CPU, memory, connections), scale up resources or reduce load by throttling non-critical requests where possible. Network Remediation: Resolve any detected connectivity or routing issues between PCF-UE and the egress gateway or upstream endpoints. Monitor and Confirm Recovery: Continue monitoring the alert metric after remedial actions to confirm the failure rate falls below the alert threshold. Ensure related services in affected namespaces also recover.

8.1.2.54 UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD

Table 8-189 UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD
Description	More than 50% of N1N2 transfer failure notification reattempts failed.
Summary	More than 50% of N1N2 transfer failure notification reattempts failed.
Severity	Major
Expression	(sum by (namespace) (increase(http_out_conn_response_total{isReattempt="true",reattemptType="UE_N1N2TransferFailure",operationType="transfer",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(http_out_conn_request_total{isReattempt="true",reattemptType="UE_N1N2TransferFailure",operationType="transfer"}[5m]))) * 100 > 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.106
Metric Used	http_out_conn_response_total, http_out_conn_request_total
Recommended Actions	The http_out_conn_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. Then in this case the alert notifies when there is a certain amount of reattempt failure for UE N1N2 transfer failure notification. If there is an increase of failure, operator can investigate on: Why the flow triggering N1N2 transfer failure notification is failing, or Check the health of the AMF to which the request are going to Cause: http_out_conn_response_total metric with indicated dimensions is pegged when PCF-UE receives a response for an outgoing reattempt transfer request triggered for N1N2TransferFailure notification. Dimensions: IsReattempt : true reattemptType : UE_N1N2TransferFailure OperationType : transfer ResponseCode : !2xx In this case more than 50% of outgoing transfer reattempts (due to N1N2TransferFailure as notified by AMF) receive a non-2xx (failure) response in the last 5 minutes (or selected sample frame) Diagnostic Information : Check Recent Logs: Analyze logs for both PCF-UE and Egress Gateway in the relevant namespace for error details (timestamps matching the period of alert). Focus on error responses: look for 4xx/5xx HTTP responses and their reasons. Correlate with Traffic Patterns: Determine if failures are for specific to certain AMFs or random. Check if there's a sudden surge in failures (indicating a broader issue). Inspect Network Health and Configuration: Ensure connectivity and correct routing between PCF-UE and its downstream targets. Validate configurations, especially recently changed ones. Cross-check Incident/Event Timeline: Review recent maintenance, deployments, or network events that could correlate with the increase in failures. Evaluate for Service Overload: Examine resource metrics (CPU, memory, rate of requests) of the affected service(PCF-UE, PCF-EGW) to determine if it’s under duress. Check with Peers: See if corresponding namespaces (other tenants/products) are seeing similar issues, could indicate a platform or shared service problem. Recovery : Resolve Underlying Service Issues: If the upstream service (e.g., AMF or other network function) is unhealthy, work with the respective team to restore normal operation. Address any misconfiguration or errors causing repeated non-2xx responses. Revert Recent Changes: If the issue correlates with recent deployments or configuration changes, consider rolling back to the previous stable state after assessing impact. Mitigate Service Overload: If resource constraints are detected (CPU, memory, connections), scale up resources or reduce load by throttling non-critical requests where possible. Network Remediation: Resolve any detected connectivity or routing issues between PCF-UE and the egress gateway or upstream endpoints. Monitor and Confirm Recovery: Continue monitoring the alert metric after remedial actions to confirm the failure rate falls below the alert threshold. Ensure related services in affected namespaces also recover.

8.1.2.55 UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD

Table 8-190 UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_N1N2_TRANSFER_FAILURE_NOTIFICATION_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD
Description	More than 25% of N1N2 transfer failure notification reattempts failed.
Summary	More than 25% of N1N2 transfer failure notification reattempts failed.
Severity	Minor
Expression	(sum by (namespace) (increase(http_out_conn_response_total{isReattempt="true",reattemptType="UE_N1N2TransferFailure",operationType="transfer",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(http_out_conn_request_total{isReattempt="true",reattemptType="UE_N1N2TransferFailure",operationType="transfer"}[5m]))) * 100 > 25
OID	1.3.6.1.4.1.323.5.3.52.1.2.106
Metric Used	http_out_conn_response_total, http_out_conn_request_total
Recommended Actions	The http_out_conn_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. Then in this case the alert notifies when there is a certain amount of reattempt failure for UE N1N2 transfer failure notification. If there is an increase of failure, operator can investigate on: Why the flow triggering N1N2 transfer failure notification is failing, or Check the health of the AMF to which the request are going to Cause: http_out_conn_response_total metric with indicated dimensions is pegged when PCF-UE receives a response for an outgoing reattempt transfer request triggered for N1N2TransferFailure notification. Dimensions: IsReattempt : true reattemptType : UE_N1N2TransferFailure OperationType : transfer ResponseCode : !2xx In this case more than 25% of outgoing transfer reattempts (due to N1N2TransferFailure as notified by AMF) receive a non-2xx (failure) response in the last 5 minutes (or selected sample frame) Diagnostic Information : Check Recent Logs: Analyze logs for both PCF-UE and Egress Gateway in the relevant namespace for error details (timestamps matching the period of alert). Focus on error responses: look for 4xx/5xx HTTP responses and their reasons. Correlate with Traffic Patterns: Determine if failures are for specific to certain AMFs or random. Check if there's a sudden surge in failures (indicating a broader issue). Inspect Network Health and Configuration: Ensure connectivity and correct routing between PCF-UE and its downstream targets. Validate configurations, especially recently changed ones. Cross-check Incident/Event Timeline: Review recent maintenance, deployments, or network events that could correlate with the increase in failures. Evaluate for Service Overload: Examine resource metrics (CPU, memory, rate of requests) of the affected service(PCF-UE, PCF-EGW) to determine if it’s under duress. Check with Peers: See if corresponding namespaces (other tenants/products) are seeing similar issues, could indicate a platform or shared service problem. Recovery : Resolve Underlying Service Issues: If the upstream service (e.g., AMF or other network function) is unhealthy, work with the respective team to restore normal operation. Address any misconfiguration or errors causing repeated non-2xx responses. Revert Recent Changes: If the issue correlates with recent deployments or configuration changes, consider rolling back to the previous stable state after assessing impact. Mitigate Service Overload: If resource constraints are detected (CPU, memory, connections), scale up resources or reduce load by throttling non-critical requests where possible. Network Remediation: Resolve any detected connectivity or routing issues between PCF-UE and the egress gateway or upstream endpoints. Monitor and Confirm Recovery: Continue monitoring the alert metric after remedial actions to confirm the failure rate falls below the alert threshold. Ensure related services in affected namespaces also recover.

8.1.2.56 UE_AMF_DISCOVERY_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD

Table 8-191 UE_AMF_DISCOVERY_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_AMF_DISCOVERY_REATTEMPT_FAILURE_ABOVE_CRITICAL_THRESHOLD
Description	More than 75% of amf discovery reattempts failed.
Summary	More than 75% of amf discovery reattempts failed.
Severity	Critical
Expression	(sum by (namespace) (increase(occnp_ue_nf_discovery_reattempt_response_total{operationType="timer_expiry_notification",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(occnp_ue_nf_discovery_reattempt_request_total{operationType="timer_expiry_notification"}[5m]))) * 100 > 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.105
Metric Used	occnp_ue_nf_discovery_reattempt_response_total
Recommended Actions	The occnp_ue_nf_discovery_reattempt_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. Then in this case, the alert notifies when there is a certain number of reattempt failure while discovering AMF. If there is an increase of failure, operator can investigate on: Why the AMF discovery flow is failing, or Check the health of the AMF to which the request are going to. Cause: The main cause of the occnp_ue_nf_discovery_reattempt_response_total metric being pegged—indicating a notable number of reattempt failures during AMF discovery—is that the PCF-UE (Policy Control Function - User Equipment) is receiving non-success responses (failures) when retrying AMF (Access and Mobility Management Function) discovery requests. Diagnostic Information: AMF Unavailability or Health Issues: The target AMF may be down, unresponsive, overloaded, or otherwise unhealthy, resulting in failed or rejected discovery attempts. Network Issues or Latency: Communication issues such as network congestion, high latency, or dropped packets between the PCF-UE and the AMF (or intermediary NFs) can cause discovery attempts to fail. Incorrect Configuration: Misconfigurations in the PCF-UE or AMF—such as wrong endpoint addresses, security settings, or authentication parameters—may prevent the successful completion of discovery requests. NRF (Network Repository Function) Problems: If AMF discovery relies on the NRF and the NRF is unhealthy or misconfigured, the PCF-UE may be unable to retrieve up-to-date or correct AMF information. Resource Exhaustion: If the system is under heavy load or resources (CPU, memory, threads) are depleted, discovery requests may not be handled on time. Timeouts and Slow Processing: Slow responses from the AMF or network timeouts can contribute to repeated reattempts and failures. Recovery: Review logs and error responses associated with AMF discovery attempts. Check the health status and recent operational history of the target AMF and NRF. Verify network health and connectivity between all relevant components. Validate all associated configurations (PCF-UE, AMF, NRF). If the issue persists, please check with Support team.

8.1.2.57 UE_AMF_DISCOVERY_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD

Table 8-192 UE_AMF_DISCOVERY_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_AMF_DISCOVERY_REATTEMPT_FAILURE_ABOVE_MAJOR_THRESHOLD
Description	More than 50% of amf discovery reattempts failed.
Summary	More than 50% of amf discovery reattempts failed.
Severity	Major
Expression	(sum by (namespace) (increase(occnp_ue_nf_discovery_reattempt_response_total{operationType="timer_expiry_notification",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(occnp_ue_nf_discovery_reattempt_request_total{operationType="timer_expiry_notification"}[5m]))) * 100 > 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.105
Metric Used	occnp_ue_nf_discovery_reattempt_response_total
Recommended Actions	The occnp_ue_nf_discovery_reattempt_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. Then in this case, the alert notifies when there is a certain number of reattempt failure while discovering AMF. If there is an increase of failure, operator can investigate on: Why the AMF discovery flow is failing, or Check the health of the AMF to which the request are going to. Cause: The main cause of the occnp_ue_nf_discovery_reattempt_response_total metric being pegged—indicating a notable number of reattempt failures during AMF discovery—is that the PCF-UE (Policy Control Function - User Equipment) is receiving non-success responses (failures) when retrying AMF (Access and Mobility Management Function) discovery requests. Diagnostic Information: AMF Unavailability or Health Issues: The target AMF may be down, unresponsive, overloaded, or otherwise unhealthy, resulting in failed or rejected discovery attempts. Network Issues or Latency: Communication issues such as network congestion, high latency, or dropped packets between the PCF-UE and the AMF (or intermediary NFs) can cause discovery attempts to fail. Incorrect Configuration: Misconfigurations in the PCF-UE or AMF—such as wrong endpoint addresses, security settings, or authentication parameters—may prevent the successful completion of discovery requests. NRF (Network Repository Function) Problems: If AMF discovery relies on the NRF and the NRF is unhealthy or misconfigured, the PCF-UE may be unable to retrieve up-to-date or correct AMF information. Resource Exhaustion: If the system is under heavy load or resources (CPU, memory, threads) are depleted, discovery requests may not be handled on time. Timeouts and Slow Processing: Slow responses from the AMF or network timeouts can contribute to repeated reattempts and failures. Recovery: Review logs and error responses associated with AMF discovery attempts. Check the health status and recent operational history of the target AMF and NRF. Verify network health and connectivity between all relevant components. Validate all associated configurations (PCF-UE, AMF, NRF). If the issue persists, please check with Support team.

8.1.2.58 UE_AMF_DISCOVERY_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD

Table 8-193 UE_AMF_DISCOVERY_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD

Field	Details
Name in Alert Yaml File	UE_AMF_DISCOVERY_REATTEMPT_FAILURE_ABOVE_MINOR_THRESHOLD
Description	More than 25% of amf discovery reattempts failed.
Summary	More than 25% of amf discovery reattempts failed.
Severity	Minor
Expression	(sum by (namespace) (increase(occnp_ue_nf_discovery_reattempt_response_total{operationType="timer_expiry_notification",responseCode!~"2.*"}[5m])) / sum by (namespace) (increase(occnp_ue_nf_discovery_reattempt_request_total{operationType="timer_expiry_notification"}[5m]))) * 100 > 25
OID	1.3.6.1.4.1.323.5.3.52.1.2.105
Metric Used	occnp_ue_nf_discovery_reattempt_response_total
Recommended Actions	The occnp_ue_nf_discovery_reattempt_response_total metric is pegged when PCF-UE receives a response from a message that is going out of the NF. Then, in this case the alert notifies when there is a certain number of reattempt failure while discovering AMF. If there is an increase of failure, operator can investigate on: Why the AMF discovery flow is failing, or Check the health of the AMF to which the request are going to. Cause: The main cause of the occnp_ue_nf_discovery_reattempt_response_total metric being pegged—indicating a notable number of reattempt failures during AMF discovery—is that the PCF-UE (Policy Control Function - User Equipment) is receiving non-success responses (failures) when retrying AMF (Access and Mobility Management Function) discovery requests. Diagnostic Information: AMF Unavailability or Health Issues: The target AMF may be down, unresponsive, overloaded, or otherwise unhealthy, resulting in failed or rejected discovery attempts. Network Issues or Latency: Communication issues such as network congestion, high latency, or dropped packets between the PCF-UE and the AMF (or intermediary NFs) can cause discovery attempts to fail. Incorrect Configuration: Misconfigurations in the PCF-UE or AMF—such as wrong endpoint addresses, security settings, or authentication parameters—may prevent the successful completion of discovery requests. NRF (Network Repository Function) Problems: If AMF discovery relies on the NRF and the NRF is unhealthy or misconfigured, the PCF-UE may be unable to retrieve up-to-date or correct AMF information. Resource Exhaustion: If the system is under heavy load or resources (CPU, memory, threads) are depleted, discovery requests may not be handled on time. Timeouts and Slow Processing: Slow responses from the AMF or network timeouts can contribute to repeated reattempts and failures. Recovery: Review logs and error responses associated with AMF discovery attempts. Check the health status and recent operational history of the target AMF and NRF. Verify network health and connectivity between all relevant components. Validate all associated configurations (PCF-UE, AMF, NRF). If the issue persists, please check with Support team.

8.1.2.59 INGRESS_ERROR_RATE_ABOVE_10_PERCENT_PER_POD

Table 8-194 INGRESS_ERROR_RATE_ABOVE_10_PERCENT_PER_POD

Field	Details
Name	INGRESS_ERROR_RATE_ABOVE_10_PERCENT_PER_POD
Description	Ingress Error Rate above 10 Percent in {{$labels.pod}} in {{$labels.namespace}}
Summary	Transaction Error Rate in {{$labels.pod}} (current value is: {{ $value }})
Severity	Critical
Expression	(sum by(pod)(rate(ocpm_ingress_response_total{response_code!~"2.*"}[24h]) or (up * 0 ) )/sum by(pod)(rate(ocpm_ingress_response_total[24h]))) * 100>= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.2
Metric Used	ocpm_ingress_response_total
Recommended Actions	Cause This alert fires when 10% or more of ingress (incoming) HTTP requests handled by any individual pod result in non-2xx (unsuccessful) responses, measured over a 1-day window. A high ingress error rate per pod suggests issues that could impact application availability, reliability, or user experience. Common causes include: Application-level errors (returning 4xx or 5xx status codes) due to bugs, configuration issues, invalid client requests, or backend failures Resource exhaustion (CPU, memory, open connections) or saturation within the affected pod Dependency failures (database, cache, or external service outages), causing the pod to respond with errors Recent deployments, rollouts, or configuration changes introducing regressions or incompatibilities Network problems or timeouts impacting request processing Unhandled exceptions or circuit breaker activations Diagnostic Information Identify affected pods from alert labels Review pod logs to categorize errors by type (4xx client errors, 5xx server errors, timeouts, etc.) Correlate errors with spikes in traffic, resource usage, or specific endpoints Examine resource utilization and health metrics (CPU, memory, connection pools, thread pools) Check readiness/liveness probe status and pod restart history Review changes in deployments, configurations, or dependencies preceding the alert Investigate for signs of dependency issues, cascading failures, or external API problems Recovery Isolate and address root cause: Use logs, error breakdowns, and metrics to determine if issues are within the pod, code, dependencies, or external factors Rollback if needed: If problems started following a recent deployment or config change, consider reverting Increase resources or scale out: Add capacity if the pod is resource-constrained Fix code or configuration: Resolve bugs, correct misconfigurations, or address unhandled cases Remediate downstream/third-party issues: Work with owners of failing dependencies if external Alert resolution: The alert will auto-resolve when the pod’s ingress error rate falls below 10% for the measuring window

8.1.2.60 SM_TRAFFIC_RATE_ABOVE_THRESHOLD

Table 8-195 SM_TRAFFIC_RATE_ABOVE_THRESHOLD

Field	Details
Name	SM_TRAFFIC_RATE_ABOVE_THRESHOLD
Description	SM service Ingress traffic Rate is above threshold of Max MPS (current value is: {{ $value }})
Summary	Traffic Rate is above 90 Percent of Max requests per second
Severity	Major
Expression	sum(rate(ocpm_ingress_request_total{servicename_3gpp="npcf-smpolicycontrol"}[2m])) >= 900
OID	1.3.6.1.4.1.323.5.3.52.1.2.3
Metric Used	ocpm_ingress_request_total
Recommended Actions	Cause: The metric ocpm_ingress_request_total is incremented for every inbound HTTP request reaching the SM component of the SM service with the dimension serviceName3gpp="npcf-smpolicycontrol". If the 2-minute average exceeds 900 mps, this indicates that the system may be experiencing an overload or an abnormal spike in traffic. Diagnostic Information: Examine Current Rate: Query ocpm_ingress_request_total for serviceName3gpp="npcf-smpolicycontrol" to assess the current ingress traffic rate. Review Upstream Sources: Identify if request rates from any upstream SMF, AF, or TDF instances have increased. Inspect Application Logs: Check for WARN or ERROR messages in logs related to overload or congestion control rejections, which can help determine if the system is rejecting requests or experiencing resource pressure. Recovery: Throttle or Rate-Limit: Apply or adjust overload/congestion control configurations to throttle or rate-limit requests from SMF as appropriate, to restore rate to expected levels. Scale Resources: Add more replicas to the sm-service deployment if needed to reduce the average rate per instance. Threshold Adjustment: Adjust the alert threshold if normal traffic patterns or business requirements change. Alert Resolution: When the sustained request rate stays below 900 mps, Prometheus will automatically clear the SM_TRAFFIC_RATE_ABOVE_THRESHOLD alert.

8.1.2.61 SM_INGRESS_ERROR_RATE_ABOVE_10_PERCENT

Table 8-196 SM_INGRESS_ERROR_RATE_ABOVE_10_PERCENT

Field	Details
Name	SM_INGRESS_ERROR_RATE_ABOVE_10_PERCENT
Description	Transaction Error Rate detected above 10 Percent of Total on SM service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 10 Percent of Total Transactions
Severity	Critical
Expression	(sum(rate(ocpm_ingress_response_total{servicename_3gpp="npcf-smpolicycontrol",response_code!~"2.*"}[24h]) or (up * 0 ) ) / sum(rate(ocpm_ingress_response_total{servicename_3gpp="npcf-smpolicycontrol"}[24h]))) * 100 >= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.4
Metric Used	ocpm_ingress_response_total
Recommended Actions	Cause This alert fires when more than 10% of all HTTP responses returned by the SM Service (npcf-smpolicycontrol)over the past day are non-2xx (i.e., not successful). This may be due to: Upstream or downstream system failures Application-level errors (5xx codes) Client-side or bad requests (4xx codes) Misconfiguration, rate limiting, or resource exhaustion Diagnostic Information Break down error rates by response code to differentiate client, server, and other errors. Search for error messages, stack traces, and signs of repeated failure or congestion. Validate that dependencies(upstream services, DB) are functioning correctly. Analyze recent deployments or config changes Check for network latency Recovery: Identify and Address Root Cause: Use error breakdown and logs to pinpoint and fix the underlying issue. Rollback Recent Changes: If a recent deployment is responsible, consider rolling back temporarily. Scale or Resource Adjustment: Add resources if you detect resource exhaustion. Rate Limiting or Throttling: Apply throttling to minimize error propagation from upstream. Alert Resolution: Once the error rate remains below 10% for a sustained period (1 day), the alert will auto-resolve.

8.1.2.62 SM_EGRESS_ERROR_RATE_ABOVE_1_PERCENT

Table 8-197 SM_EGRESS_ERROR_RATE_ABOVE_1_PERCENT

Field	Details
Name in Alert Yaml File	SM_EGRESS_ERROR_RATE_ABOVE_1_PERCENT
Description	Egress Transaction Error Rate detected above 1 Percent of Total Transactions (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 1 Percent of Total Transactions
Severity	Minor
Expression	(sum(rate(ocpm_egress_response_total{servicename_3gpp="npcf-smpolicycontrol",response_code!~"2.*"}[24h]) or (up * 0 ) ) / sum(rate(ocpm_egress_response_total{servicename_3gpp="npcf-smpolicycontrol"}[24h]))) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.5
Metric Used	ocpm_egress_response_total
Recommended Actions	Cause This alert fires when more than 1% of all HTTP responses returned by the SM Service (npcf-smpolicycontrol)over the past day are non-2xx (i.e., not successful). This may be due to: Upstream or downstream system failures Application-level errors (5xx codes) Client-side or bad requests (4xx codes) Misconfiguration, rate limiting, or resource exhaustion Diagnostic Information Break down error rates by response code to differentiate client, server, and other errors. Search for error messages, stack traces, and signs of repeated failure or congestion. Validate that dependencies(upstream services, DB) are functioning correctly. Analyze recent deployments or config changes Check for network latency Recovery: Identify and Address Root Cause: Use error breakdown and logs to pinpoint and fix the underlying issue. Rollback Recent Changes: If a recent deployment is responsible, consider rolling back temporarily. Scale or Resource Adjustment: Add resources if you detect resource exhaustion. Rate Limiting or Throttling: Apply throttling to minimize error propagation from upstream. Alert Resolution: Once the error rate remains below 10% for a sustained period (1 day), the alert will auto-resolve. For any additional guidance, contact My Oracle Support.

8.1.2.63 PCF_CHF_INGRESS_TRAFFIC_RATE_ABOVE_THRESHOLD

Table 8-198 PCF_CHF_INGRESS_TRAFFIC_RATE_ABOVE_THRESHOLD

Field	Details
Name	PCF_CHF_INGRESS_TRAFFIC_RATE_ABOVE_THRESHOLD
Description	User service Ingress traffic Rate from CHF is above threshold of Max MPS (current value is: {{ $value }})
Summary	Traffic Rate is above 90 Percent of Max requests per second
Severity	Major
Expression	sum(rate(occnp_userservice_inbound_count_total{service_resource="chf-service"}[2m])) >= 900
OID	1.3.6.1.4.1.323.5.3.52.1.2.11
Metric Used	ocpm_userservice_inbound_count_total
Recommended Actions	Cause: The metric ocpm_userservice_inbound_count_total with dimension service_resource="chf-service" is incremented for every inbound HTTP request reaching the CHF connector service. If the 2-minute average exceeds 900 mps, this indicates that the system may be experiencing an overload or an abnormal spike in traffic. Diagnostic Information: Examine Current Rate: Query ocpm_userservice_inbound_count_total for service_resource="chf-service" to assess the current ingress traffic rate. Review Upstream Sources: Identify if request rates from any upstream CHF, SMF, AMF instances have increased. Inspect Application Logs: Check for WARN or ERROR messages in logs related to overload or congestion control rejections, which can help determine if the system is rejecting requests or experiencing resource pressure. Recovery: Throttle or Rate-Limit: Apply or adjust congestion control configurations to throttle requests from downstream services as appropriate, to restore rate to expected levels. Scale Resources: Add more replicas to the Chf connector deployment if needed to reduce the average rate per instance. Threshold Adjustment: Adjust the alert threshold if normal traffic patterns or business requirements change. Alert Resolution: When the sustained request rate stays below 900 mps, Prometheus will automatically clear the PCF_CHF_INGRESS_TRAFFIC_RATE_ABOVE_THRESHOLD alert. For any additional guidance, contact My Oracle Support.

8.1.2.64 PCF_CHF_EGRESS_ERROR_RATE_ABOVE_10_PERCENT

Table 8-199 PCF_CHF_EGRESS_ERROR_RATE_ABOVE_10_PERCENT

Field	Details
Name	PCF_CHF_EGRESS_ERROR_RATE_ABOVE_10_PERCENT
Description	Egress Transaction Error Rate detected above 10 Percent of Total on User service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 10 Percent of Total Transactions
Severity	Critical
Expression	(sum(rate(ocpm_chf_tracking_response_total {servicename_3gpp="nchf-spendinglimitcontrol",response_code!~"2.*"} [24h]) or (up * 0 ) ) / sum(rate(ocpm_chf_tracking_response_total {servicename_3gpp="nchf-spendinglimitcontrol"} [24h]))) 100 >= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.12
Metric Used	ocpm_chf_tracking_response_total
Recommended Actions	Cause: This alert fires when more than 10% of all HTTP responses for the PCF (CHF connector the PCF component that calls the external CHF via nchf-spendinglimitcontrol) over the past day are non-2xx (i.e., not successful). This may be due to: External CHF partial outage or dependency failures. Application-level errors (5xx) or timeouts on the CHF path. Client/bad requests (4xx) from the CHF connector due to schema/version or auth issues. Misconfiguration, rate limiting/throttling, TLS/mTLS or DNS problems, or resource exhaustion. Diagnostic Information: Break down error rates by response class (4xx vs 5xx vs timeouts/TLS/connect resets). Search CHF connector service logs and traces for recurring errors, stack traces, circuit-breaker events, or congestion. Validate external CHF health and dependencies (service/DB), and check for throttling indicators. Analyze recent deployments or configuration changes in PCF or CHF (endpoints, timeouts, retries, API versions). Check for traffic spikes, connection pool saturation, CPU/memory pressure, or elevated latency. Recovery: Identify and address root cause: Use error breakdown, logs, and traces to pinpoint whether the issue is in the PCF CHF client, network/TLS/auth, or the external CHF. Roll back recent changes: Temporarily revert relevant PCF/CHF deployments or configs if correlated with the onset. Scale or resource adjustment: Increase capacity or tune connection/thread pools; enable autoscaling if appropriate. Rate limiting or throttling: Use bounded retries with backoff and apply throttling to reduce cascading failures. Alert resolution: Once the non-2xx rate remains below 10% for a sustained period (1 day), the alert will auto-resolve. For any additional guidance, contact My Oracle Support.

8.1.2.65 PCF_CHF_INGRESS_TIMEOUT_ERROR_ABOVE_MAJOR_THRESHOLD

Table 8-200 PCF_CHF_INGRESS_TIMEOUT_ERROR_ABOVE_MAJOR_THRESHOLD

Field	Details
Name	PCF_CHF_INGRESS_TIMEOUT_ERROR_ABOVE_MAJOR_THRESHOLD
Description	Ingress Timeout Error Rate detected above 10 Percent of Total towards CHF service (current value is: {{ $value }})
Summary	Timeout Error Rate detected above 10 Percent of Total Transactions
Severity	Major
Expression	(sum(rate(ocpm_chf_tracking_request_timeout_total{servicename_3gpp="nchf-spendinglimitcontrol"}[24h]) or (up * 0 )) / sum(rate(ocpm_chf_tracking_request_total{servicename_3gpp="nchf-spendinglimitcontrol"}[24h]))) * 100 >=10
OID	1.3.6.1.4.1.323.5.3.52.1.2.17
Metric Used	ocpm_chf_tracking_request_timeout_total
Recommended Actions	Cause: This alert is triggered when more than 10% of all inbound requests from PCF (Policy Control Function) to the CHF (nchf-spendinglimitcontrol) time out over a 1-day window. This may impact charging, quota enforcement, or service delivery. Common causes include: Network latency, intermittent packet loss, or connectivity issues between PCF and CHF Overload, resource congestion, or unresponsiveness in the CHF or its dependencies Resource exhaustion or scaling limits in the PCF, CHF, or intermediary components Misconfiguration of timeout thresholds, retries, or circuit breaker settings Downstream service or database issues affecting CHF’s ability to respond in time Recent changes or deployments that introduced performance bottlenecks or regressions Diagnostic Information: Identify which part of the infrastructure is experiencing timeouts: is it consistent across all traffic or localized? Review logs from PCF, CHF, and network/security appliances for repeated timeout, retry, or connection reset events Check health dashboards for CHF (CPU, memory, response latency, DB availability, etc.) Analyze request/response timings, queue lengths, and backlog at ingress points Correlate with recent deployment, scaling, or network changes Examine resource usage and pod health for PCF and CHF components Recovery: Isolate the root cause: Use logs and health metrics to determine if the problem is with CHF availability, network path, or PCF. Scale or optimize: Increase resources, scale instances, or optimize configuration for PCF and CHF services as needed. Rollback if needed: If the alert correlates with new deployments or config changes, consider reverting. Network remediation: Address any identified network latency, packet loss, or DNS resolution issues. Tune configuration: Adjust timeout settings, connection pools, and retry logic based on observed conditions. Coordinate: Engage CHF, PCF, and platform support teams as needed for collaborative troubleshooting. Alert Resolution: This alert will auto-resolve once the ingress timeout error rate drops below 10% of total requests to CHF over the evaluation window.

8.1.2.66 PCF_PENDING_BINDING_SITE_TAKEOVER

Table 8-201 PCF_PENDING_BINDING_SITE_TAKEOVER

Field	Details
Description	The site takeover configuration has been activated
Summary	The site takeover configuration has been activated
Severity	CRITICAL
Expression	sum by (application, container, namespace) (changes(occnp_pending_binding_site_takeover[2m])) > 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.45
Metric Used	occnp_pending_binding_site_takeover
Recommended Actions	Cause: This alert fires when the site takeover functionality is engaged to handle geo-redundancy scenarios. Site takeover is typically activated when a site in a distributed PCF deployment is down or unreachable, empowering another site to process that site’s pending binding operations for service continuity. Diagnostic Information: Check configuration to confirm the alternate site profile is correctly set and the takeover flag is enabled. Examine PendingOperation records to ensure the alternate site is processing entries from the down site’s site ID. Review service logs for site takeover-related events, handoff messages, and any associated errors during takeover or operation processing. Recovery & Actions: Verify that site takeover activation was intentional and aligns with fail-over or DR (Disaster Recovery) procedures. Monitor processing of pending operations for successful handoff and completion under the alternate site. Communicate with relevant operations/support teams about the takeover to prevent conflicting operations. Disable site takeover once the original site is restored to normal operation, so pending operations revert to their standard ownership and workflow. Audit for any missed or failed operations during the site handover, and remediate as needed. Alert Resolution: The alert will auto-resolve once there are no new site takeover events, and the takeover configuration is deactivated or no longer required. For any additional guidance, contact My Oracle Support.

8.1.2.67 PCF_PENDING_BINDING_THRESHOLD_LIMIT_REACHED

Table 8-202 PCF_PENDING_BINDING_THRESHOLD_LIMIT_REACHED

Field	Details
Description	The Pending Operation table threshold has been reached.
Summary	The Pending Operation table threshold has been reached.
Severity	CRITICAL
Expression	sum by (application, container, namespace) (changes(occnp_threshold_limit_reached_total[2m])) > 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.46
Metric Used	occnp_threshold_limit_reached_total
Recommended Actions	Cause This alert fires when the number of records in the Pending Operation table (to reattempt binding registration in BSF at a later time) reaches a predefined threshold. This means the system’s retry or pending queue for binding operations is saturated and may be at risk of delaying, or failing new operations. Exceeding this threshold typically signals that retry or binding registrations are not clearing at an expected rate. Common causes include: Persistent errors or failures from BSF in response to binding attempts, triggering retries Widespread or systemic service degradation in BSF, Binding Service, or network paths Application bugs resulting in stuck or orphaned PendingOperation records Misconfigured thresholds, retry intervals, or logic in SM or Binding Service Resource starvation (CPU, memory, DB connections) preventing timely processing of pending operations Recent deployments, configuration updates, or load spikes overwhelming the binding flow Diagnostic Information Check the volume, age, and growth trend of records in the Pending Operation table Correlate with other alerts or incident tickets related to BSF, Binding Service, network, or DB health Analyze logs from SM Service, Binding Service, and (if applicable) Audit Service for repeated errors, retry loops, or slow processing Review recent deployments or configuration changes to PCF Service components Inspect resource utilization for relevant pods, containers, and backend storage Confirm correct configuration of the threshold limit, retry intervals, and error code handling Recovery Prioritize clearing pending records: Investigate and remediate the root cause(s) of unprocessed binding operations (BSF issues, infra bottlenecks, logic bugs) Scale resources or prioritize processing: Add capacity or redistribute load if resource constraints are found Tune configuration: Adjust thresholds, error code mappings, and retry intervals as necessary Audit retry and cleanup logic: Ensure orphaned or stale records are purged and retry logic is functioning as intended Rollback if needed: If issue began with a recent deployment or config change, consider reverting Coordinate across teams: Engage with BSF, Infrastructure, and DB owners as required Alert resolution: The alert will auto-resolve once the number of records in the Pending Operation table returns below the configured threshold and normal processing resumes. For any additional guidance, contact My Oracle Support.

8.1.2.68 PCF_PENDING_BINDING_RECORDS_COUNT

Table 8-203 PCF_PENDING_BINDING_RECORDS_COUNT

Field	Details
Name	PCF_PENDING_BINDING_RECORDS_COUNT
Description	Pending Operation Table contains records.
Summary	Pending Operation Table contains records.
Severity	MINOR
Expression	sum by (application, container, namespace) (changes(occnp_pending_operation_records_count[10s])) > 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.47
Metric Used	occnp_pending_operation_records_count
Recommended Actions	Cause This alert fires when a new pending binding operation is inserted into the system by the SM Service(to reattempt binding registration in BSF at a later time). This typically happens when the BSF reattempt settings are configured and the response from BSF to a binding registration indicates an error condition that requires a retry (as per pre-configured error codes). Common causes for entries in the PendingOperation table include:Common causes include: BSF returns a transient or retry-eligible error code in response to binding requests. Temporary unavailability or instability of BSF or related network paths. Application bugs leading to improper handling of BSF responses or retry logic. Recent configuration changes impacting retry or error handling logic. Diagnostic Information Review SM Service and binding service logs to trace binding requests, BSF response codes, and the creation/updating of PendingOperations. Verify resource utilization and health across relevant pods or containers. Analyze timing and Volume of pending operation records—spikes may indicate regression or external service instability. Recovery Monitor pending operation clearance: Confirm that retries triggered by Audit Service notifications are processed and successfully clear pending records. Investigate recurring or persistent errors: If retries are frequently required or repeatedly fail, drill down to BSF responses, retry outcomes, and any correlated infrastructure issues. Coordinate with BSF/service owners: If an underlying BSF or network problem persists, work with those teams to restore normal registration flow. Tune configuration: Adjust error code mapping, retry intervals, or thresholds based on observed workload and service behavior. Rollback if needed: Revert recent deployments or config updates if they correlate with spikes in pending operations. Alert resolution: The alert will auto-resolve when new pending binding operation records are no longer being routinely created, retries are succeeding, and the overall pending queue stabilizes or clears. For any additional guidance, contact My Oracle Support.

8.1.2.69 AUTONOMOUS_SUBSCRIPTION_FAILURE

Table 8-204 AUTONOMOUS_SUBSCRIPTION_FAILURE

Field	Details
Description	Autonomous subscription failed for a configured Slice Load Level
Summary	Autonomous subscription failed for a configured Slice Load Level
Severity	Critical
Expression	The number of failed Autonomous Subscription for a configured Slice Load Level in nwdaf-agent is greater than zero.
OID	1.3.6.1.4.1.323.5.3.52.1.2.49
Metric Used	subscription_failure{requestType="autonomous"}
Recommended Actions	The alert gets cleared when the failed Autonomous Subscription is corrected. To clear the alert, perform the following steps: Delete the Slice Load Level configuration. Re-provision the Slice Load Level configuration. Cause: This alert activates when there is at least one autonomous subscription (such as the NWDAF event subscription process) failure detected for a given S-NSSAI, indicating that the system was unable to successfully initiate or maintain a subscription for a specific network slice. Common causes may include: Remote service (e.g., NWDAF) is unavailable, responds with a failure, or returns an error code. Authentication/authorization failures (invalid tokens, credentials, certificates). Incorrect, missing, or unsupported subscription parameters (S-NSSAI, event types, notification targets). API version or schema mismatches between subscribing and serving systems. Rate limiting, resource exhaustion, or capacity constraints in remote service. Network or DNS/connectivity problems between components. Recent deployment or configuration change introducing new issues. Diagnostic Information: Check which S-NSSAI (network slice) is affected using the alert labels. Review NWDAF gent service logs, and collect relevant error codes and messages from the failed subscription attempts. Examine recent changes or deployments to the NWDAF Agent, remote NWDAF, or related interfaces/services. Assess service health and connectivity between the agent and NWDAF (latency, errors, authentication status). Validate the subscription request payload, endpoint URLs, and configuration for the target S-NSSAI. Look for evidence of transient or repeated network/service issues. Recovery: Identify the failed subscription(s): Use the alert labels and logs to pinpoint the slice(s) affected. Resolve remote or local service issues: Work with relevant teams to restore NWDAF or agent functionality, address authentication or network problems, or resolve configuration mismatches. Retry or re-initiate subscriptions as needed after addressing the root cause. Rollback changes if the alert coincides with recent deployments, configuration modifications, or rollouts. Alert Resolution: This alert will automatically resolve once the system detects that there are no new autonomous subscription failures (i.e., no new increments in the failure counter) for the affected S-NSSAI(s) within the evaluation window. Successful re-establishment or correction will clear the alert. For any additional guidance, contact My Oracle Support.

8.1.2.70 AM_NOTIFICATION_ERROR_RATE_ABOVE_1_PERCENT

Table 8-205 AM_NOTIFICATION_ERROR_RATE_ABOVE_1_PERCENT

Field	Details
Name	AM_NOTIFICATION_ERROR_RATE_ABOVE_1_PERCENT
Description	AMF Notification Error Rate detected above 1 Percent of Total on AM Service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 1 Percent of Total Transactions on AM Notification
Severity	MINOR
Expression	(sum(rate(http_out_conn_response_total{pod=~".*amservice.*",responseCode!~"2.*",servicename3gpp="npcf-am-policy-control"}[1d])) / sum(rate(http_out_conn_response_total{pod=~".*amservice.*",servicename3gpp="npcf-am-policy-control"}[1d]))) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.54
Metric Used	http_out_conn_response_total
Recommended Actions	Cause This alert triggers when 1% or more of notification requests sent from the AM service (part of PCF) to the AMF ( npcf-am-policy-control endpoint) result in non-2xx (unsuccessful) responses over a 1-day window. These notifications inform AMF about access or mobility events. A significant portion of errors could be 404 responses, which occur when AMF does not have the corresponding session in its context. This may indicate attempts to notify AMF about sessions that have already ended or were never established. Other possible causes include: Partial outage, degradation, or overload in the AMF Application errors in the AM service or AMF (e.g., other 4xx or 5xx codes) Schema or API mismatches due to recent deployments or configuration changes Authentication, authorization, or TLS certificate issues Network/connectivity problems Resource exhaustion in the AMF Diagnostic Information Break down non-2xx responses by HTTP status code, especially 404 versus other 4xx/5xx Examine AM service and AMF logs for detailed error messages and patterns Review session establishment, update, and termination flows in both AM service and AMF Investigate recent deployments, configuration changes, or spikes in error rates Assess resource usage and health of both AM service and AMF Validate API contracts, payload formats, and endpoint configurations Check for authentication/authorization or certificate issues Recovery Identify and resolve the root cause: Use logs, traces, and error breakdowns to determine if high 404 rates are expected (due to session lifecycle), or if there is a systematic issue such as stale notifications Tune notification logic: Adjust workflows to minimize duplicate or late notifications when sessions may already have ended Rollback or adjust recent changes: If errors correlate with deployments or config updates, consider reverting them Scale or adjust resources: Add capacity or tune connection/timeouts if resource exhaustion is present Remediate network or security problems: Ensure stable communication and correct authentication/certificates between PCF and AMF Alert resolution: The alert will auto-resolve when the error rate drops below 1% over the measuring window For any additional guidance, contact My Oracle Support.

8.1.2.71 AM_AR_ERROR_RATE_ABOVE_1_PERCENT

Table 8-206 AM_AR_ERROR_RATE_ABOVE_1_PERCENT

Field	Details
Name	AM_AR_ERROR_RATE_ABOVE_1_PERCENT
Description	Alternate Routing Error Rate detected above 1 Percent of Total on AM Service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 1 Percent of Total Transactions on AM Alternate Routing
Severity	MINOR
Expression	(sum by (fqdn) (rate(ocpm_ar_response_total{pod=~".*amservice.*",responseCode!~"2.*",servicename3gpp="npcf-am-policy-control"}[1d])) / sum by (fqdn) (rate(ocpm_ar_response_total{pod=~".*amservice.*",servicename3gpp="npcf-am-policy-control"}[1d]))) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.55
Metric Used	ocpm_ar_response_total
Recommended Actions	Cause This alert fires when 1% or more of alternate routing (AR) requests initiated by the AM service (as part of PCF) to AMF ( npcf-am-policy-control ) result in non-2xx (unsuccessful) responses over a 1-day window, grouped by FQDN. Alternate routing is the process of retrying the original request to a different AMF instance when the initial attempt fails. A rising AR error rate suggests persistent issues with connectivity, service health, or configuration for primary or alternate AMF endpoints. Typical causes include: Persistent unavailability, overload, or partial outages affecting some or all AMF instances Application-level errors from AMF (many 4xx/5xx responses, including 404s for missing sessions) Schema or API incompatibility after deployments or configuration changes Authentication, authorization, or certificate-related failures during retries Network or DNS problems affecting communication with one or more AMF instances Resource exhaustion, scaling issues, or retry storm in the AM service Misconfiguration of alternate endpoint lists or retry logic Diagnostic Information Break down failed AR responses by HTTP status code (4xx, 5xx, timeouts) to pinpoint the failure type Review AM service logs to identify why alternate routing was triggered and the response from each retry Inspect AMF logs for errors and session context associated with AR requests Assess health, status, and readiness of all AMF endpoints relevant to the alerting FQDN Check authentication credentials, certificate validity, and endpoint configuration Correlate AR error spikes with recent deployments, updates, scaling actions, or network incidents Analyze retry logic to ensure backoff and failover policies are working as expected Recovery Isolate the root cause: Use logs and metrics to determine if AR failures are due to persistent AMF unavailability, configuration problems, or retry logic bugs Remediate endpoint or network issues: Restore AMF health, increase capacity, or fix network connectivity to all AMF endpoints Fix authentication or certificate problems: Update or refresh security credentials as necessary Adjust or rollback changes as needed: If increased errors align with a recent deployment or config update Tune retry/backoff policies: Update AR configuration to minimize repeated failures or retry storms Alert resolution: The alert auto-resolves once the AR error rate drops below 1% over the measurement window For any additional guidance, contact My Oracle Support.

8.1.2.72 UE_NOTIFICATION_ERROR_RATE_ABOVE_1_PERCENT

Table 8-207 UE_NOTIFICATION_ERROR_RATE_ABOVE_1_PERCENT

Field	Details
Name	UE_NOTIFICATION_ERROR_RATE_ABOVE_1_PERCENT
Description	AMF Notification Error Rate detected above 1 Percent of Total from UE Service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 1 Percent of Total AMF Notification from UE Service
Severity	MINOR
Expression	(sum(rate(http_out_conn_response_total{pod=~".*ueservice.*",responseCode!~"2.*",servicename3gpp="npcf-ue-policy-control"}[1d])) / sum(rate(http_out_conn_response_total{pod=~".*ueservice.*",servicename3gpp="npcf-ue-policy-control"}[1d]))) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.56
Metric Used	http_out_conn_response_total
Recommended Actions	Cause This alert triggers when 1% or more of notification requests sent from the UE service (part of PCF) to the AMF (npcf-ue-policy-control endpoint) result in non-2xx (unsuccessful) responses over a 1-day window. These notifications inform AMF about UE policy events. A significant portion of errors could be 404 responses, which occur when AMF does not have the corresponding session in its context. This may indicate attempts to notify AMF about sessions that have already ended or were never established. Other possible causes include: Partial outage, degradation, or overload in the AMF Application errors in the AM service or AMF (e.g., other 4xx or 5xx codes) Schema or API mismatches due to recent deployments or configuration changes Authentication, authorization, or TLS certificate issues Network/connectivity problems Resource exhaustion in the AMF Diagnostic Information Break down non-2xx responses by HTTP status code, especially 404 versus other 4xx/5xx Examine AM service and AMF logs for detailed error messages and patterns Review session establishment, update, and termination flows in both AM service and AMF Investigate recent deployments, configuration changes, or spikes in error rates Assess resource usage and health of both AM service and AMF Validate API contracts, payload formats, and endpoint configurations Check for authentication/authorization or certificate issues Recovery Identify and resolve the root cause: Use logs, traces, and error breakdowns to determine if high 404 rates are expected (due to session lifecycle), or if there is a systematic issue such as stale notifications Tune notification logic: Adjust workflows to minimize duplicate or late notifications when sessions may already have ended Rollback or adjust recent changes: If errors correlate with deployments or config updates, consider reverting them Scale or adjust resources: Add capacity or tune connection/timeouts if resource exhaustion is present Remediate network or security problems: Ensure stable communication and correct authentication/certificates between PCF and AMF Alert resolution: The alert will auto-resolve when the error rate drops below 1% over the measuring window For any additional guidance, contact My Oracle Support.

8.1.2.73 UE_AR_FAILURE_RATE_ABOVE_1_PERCENT

Table 8-208 UE_AR_FAILURE_RATE_ABOVE_1_PERCENT

Field	Details
Description	Alternate Routing Error Rate detected above 1 Percent of Total on UE Service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 1 Percent of Total Transactions on UE Alternate Routing
Severity	MINOR
Expression	(sum by (fqdn) (rate(ocpm_ar_response_total{pod=~".*ueservice.*",responseCode!~"2.*",servicename3gpp="npcf-ue-policy-control"}[1d])) / sum by (fqdn) (rate(ocpm_ar_response_total{pod=~".*ueservice.*",servicename3gpp="npcf-ue-policy-control"}[1d]))) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.57
Metric Used	ocpm_ar_response_total
Recommended Actions	Cause This alert fires when 1% or more of alternate routing (AR) requests initiated by the AM service (as part of PCF) to AMF (npcf-ue-policy-control) result in non-2xx (unsuccessful) responses over a 1-day window, grouped by FQDN. Alternate routing is the process of retrying the original request to a different AMF instance when the initial attempt fails. A rising AR error rate suggests persistent issues with connectivity, service health, or configuration for primary or alternate AMF endpoints. Typical causes include: Persistent unavailability, overload, or partial outages affecting some or all AMF instances Application-level errors from AMF (many 4xx/5xx responses, including 404s for missing sessions) Schema or API incompatibility after deployments or configuration changes Authentication, authorization, or certificate-related failures during retries Network or DNS problems affecting communication with one or more AMF instances Resource exhaustion, scaling issues, or retry storm in the AM service Misconfiguration of alternate endpoint lists or retry logic Diagnostic Information Break down failed AR responses by HTTP status code (4xx, 5xx, timeouts) to pinpoint the failure type Review AM service logs to identify why alternate routing was triggered and the response from each retry Inspect AMF logs for errors and session context associated with AR requests Assess health, status, and readiness of all AMF endpoints relevant to the alerting FQDN Check authentication credentials, certificate validity, and endpoint configuration Correlate AR error spikes with recent deployments, updates, scaling actions, or network incidents Analyze retry logic to ensure backoff and failover policies are working as expected Recovery Isolate the root cause: Use logs and metrics to determine if AR failures are due to persistent AMF unavailability, configuration problems, or retry logic bugs Remediate endpoint or network issues: Restore AMF health, increase capacity, or fix network connectivity to all AMF endpoints Fix authentication or certificate problems: Update or refresh security credentials as necessary Adjust or rollback changes as needed: If increased errors align with a recent deployment or config update Tune retry/backoff policies: Update AR configuration to minimize repeated failures or retry storms Alert resolution: The alert auto-resolves once the AR error rate drops below 1% over the measurement window For any additional guidance, contact My Oracle Support.

8.1.2.74 SMSC_CONNECTION_DOWN

Table 8-209 SMSC_CONNECTION_DOWN

Field	Details
Description	Connection to SMSC peer {{$labels.smscName}} is down in notifier service pod {{$labels.pod}}
Summary	Connection to SMSC peer {{$labels.smscName}} is down in notifier service pod {{$labels.pod}}
Severity	MAJOR
Expression	sum by(namespace, pod, smscName)(occnp_active_smsc_conn_count) == 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.63
Metric Used	occnp_active_smsc_conn_count
Recommended Actions	Cause This alert fires when the connection count to a specific SMSC (Short Message Service Center) peer (smscName) drops to zero in a notifier service pod. This means that the notifier service in the indicated pod has lost connectivity with the SMSC peer, which may halt or delay SMS delivery for affected sessions. Common causes include: Network connectivity issues between the notifier pod and the SMSC peer (latency, packet loss, firewall changes) SMSC peer instance is offline, unresponsive, or undergoing maintenance Unexpected restart or crash of the notifier service pod TCP session timeout, reset, or socket exhaustion TLS/certificate negotiation failures (if applicable) Misconfiguration of SMSC endpoint, port, or authentication details Recent pod or infrastructure changes affecting networking or endpoints Diagnostic Information Identify which namespace, pod, and smscName are affected from alert labels Check notifier pod logs for errors, timeouts, or repeated reconnection attempts to the SMSC Confirm SMSC peer health and status via monitoring tools or coordination with peer’s operations Validate network connectivity (test with ping/telnet/traceroute), DNS resolution, and firewall or security rules Review recent changes in deployment, SMSC endpoint configuration, or certificate rotation Check for underlying resource issues (CPU, memory, open file/socket limits) on the notifier pod Recovery Restore connectivity: Address any network or firewall problems between the notifier pod and SMSC peer Restart services: If the notifier pod is in a bad state, restart it to reestablish the connection Engage SMSC operations: If the peer is down, coordinate with the SMSC provider/team to restore service Correct configuration: Verify endpoint settings, authentication, and port assignments in both notifier and SMSC Rollback recent changes: If disconnection began after deployment or configuration change, consider reverting Alert resolution: The alert will auto-resolve once the connection count returns above zero for the affected pod and SMSC For any additional guidance, contact My Oracle Support.

8.1.2.75 LOCK_ACQUISITION_EXCEEDS_MINOR_THRESHOLD

Table 8-210 LOCK_ACQUISITION_EXCEEDS_MINOR_THRESHOLD

Field	Details
Name	LOCK_ACQUISITION_EXCEEDS_MINOR_THRESHOLD
Description	Lock acquisition failures exceed the minor threshold limit. The (current value is: {{ $value }})
Summary	Keys used in Bulwark lock request which are already in locked state detected above 20 Percent of Total Transactions.
Severity	Minor
Expression	(sum by (namespace) (increase(lock_response_total{requestType="acquireLock",responseType="failure"}[5m])) /sum by (namespace) (increase(lock_request_total{requestType="acquireLock"}[5m]))) * 100 >=20 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.69
Metric Used	lock_request_total
Recommended Actions	Cause This alert fires when, within a 5-minute window, between 20% and 50% of lock acquisition requests (acquireLock) to the Bulwark service in any namespace fail. Elevated lock acquisition failure rates may indicate: Lock contention, with multiple clients attempting to acquire the same lock/resource concurrently (hot spots) Stale or orphaned locks that are not being properly released Performance degradation or partial outages in the Coherence distributed cache backend used by Bulwark Misconfigured lock TTL (time to live), expiry, or retry/backoff policies Recent deployment, scaling events, or increased load causing higher lock demand or contention Bugs in the client logic resulting in frequent or incorrect lock requests Diagnostic Information Identify affected namespaces and resources prone to high contention or failure Examine Bulwark and application logs for specific lock acquisition errors or contention/wait messages Review the health of the bulwark service(and coherence cluster), including resource utilization (CPU, memory) Check lock TTL and cleanup mechanisms to ensure timely lock release by both typical and failure pathways Analyze trends following deployments, configuration changes, or traffic spikes Assess and validate the configuration for Bulwark (connection pools, timeouts, backoff settings) Investigate for node clock skew, which can impact distributed locking Recovery Reduce Contention: Identify and resolve any traffic pattern that causes lock contention Backend Remediation: Scale or optimize Bulwark and address any backend health issues Configuration Tuning: Adjust TTLs, retry intervals, and backoff strategies for optimal application behavior Rollback if Needed: Revert recent changes to Bulwark deployments or configurations if correlated to failure spikes Alert Resolution: Alert will auto-resolve once lock acquisition failure rates in a namespace drop below 20%. If the rate exceeds 50%, a higher severity alert will trigger.

8.1.2.76 LOCK_ACQUISITION_EXCEEDS_MAJOR_THRESHOLD

Table 8-211 LOCK_ACQUISITION_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Name	LOCK_ACQUISITION_EXCEEDS_MAJOR_THRESHOLD
Description	Lock acquisition failures exceed the major threshold limit. The (current value is: {{ $value }})
Summary	Keys used in Bulwark lock request which are already in locked state detected above 50 Percent of Total Transactions.
Severity	Major
Expression	(sum by (namespace) (increase(lock_response_total{requestType="acquireLock",responseType="failure"}[5m])) /sum by (namespace) (increase(lock_request_total{requestType="acquireLock"}[5m]))) * 100 >= 50 < 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.69
Metric Used	lock_request_total
Recommended Actions	Cause This alert fires when, within a 5-minute window, between 50% and 75% of lock acquisition requests (acquireLock) to the Bulwark service in any namespace fail. Elevated lock acquisition failure rates may indicate: Lock contention, with multiple clients attempting to acquire the same lock/resource concurrently (hot spots) Stale or orphaned locks that are not being properly released Performance degradation or partial outages in the Coherence distributed cache backend used by Bulwark Misconfigured lock TTL (time to live), expiry, or retry/backoff policies Recent deployment, scaling events, or increased load causing higher lock demand or contention Bugs in the client logic resulting in frequent or incorrect lock requests Diagnostic Information Identify affected namespaces and resources prone to high contention or failure Examine Bulwark and application logs for specific lock acquisition errors or contention/wait messages Review the health of the bulwark service(and coherence cluster), including resource utilization (CPU, memory) Check lock TTL and cleanup mechanisms to ensure timely lock release by both typical and failure pathways Analyze trends following deployments, configuration changes, or traffic spikes Assess and validate the configuration for Bulwark (connection pools, timeouts, backoff settings) Investigate for node clock skew, which can impact distributed locking Recovery Reduce Contention: Identify and resolve any traffic pattern that causes lock contention Backend Remediation: Scale or optimize Bulwark and address any backend health issues Configuration Tuning: Adjust TTLs, retry intervals, and backoff strategies for optimal application behavior Rollback if Needed: Revert recent changes to Bulwark deployments or configurations if correlated to failure spikes Alert Resolution: Alert will auto-resolve once lock acquisition failure rates in a namespace drop below 50%. If the rate exceeds 75%, a higher severity alert will trigger.

8.1.2.77 LOCK_ACQUISITION_EXCEEDS_CRITICAL_THRESHOLD

Table 8-212 LOCK_ACQUISITION_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Name	LOCK_ACQUISITION_EXCEEDS_CRITICAL_THRESHOLD
Description	Lock acquisition failures exceed the critical threshold limit. The (current value is: {{ $value }})
Summary	Keys used in Bulwark lock request which are already in locked state detected above 75 Percent of Total Transactions.
Severity	Critical
Expression	(sum by (namespace) (increase(lock_response_total{requestType="acquireLock",responseType="failure"}[5m])) /sum by (namespace) (increase(lock_request_total{requestType="acquireLock"}[5m]))) * 100 >=75
OID	1.3.6.1.4.1.323.5.3.52.1.2.69
Metric Used	lock_request_total
Recommended Actions	Cause This alert fires when, within a 5-minute window, above 75% of lock acquisition requests (acquireLock) to the Bulwark service in any namespace fail. Elevated lock acquisition failure rates may indicate: Lock contention, with multiple clients attempting to acquire the same lock/resource concurrently (hot spots) Stale or orphaned locks that are not being properly released Performance degradation or partial outages in the Coherence distributed cache backend used by Bulwark Misconfigured lock TTL (time to live), expiry, or retry/backoff policies Recent deployment, scaling events, or increased load causing higher lock demand or contention Bugs in the client logic resulting in frequent or incorrect lock requests Diagnostic Information Identify affected namespaces and resources prone to high contention or failure Examine Bulwark and application logs for specific lock acquisition errors or contention/wait messages Review the health of the bulwark service(and coherence cluster), including resource utilization (CPU, memory) Check lock TTL and cleanup mechanisms to ensure timely lock release by both typical and failure pathways Analyze trends following deployments, configuration changes, or traffic spikes Assess and validate the configuration for Bulwark (connection pools, timeouts, backoff settings) Investigate for node clock skew, which can impact distributed locking Recovery Reduce Contention: Identify and resolve any traffic pattern that causes lock contention Backend Remediation: Scale or optimize Bulwark and address any backend health issues Configuration Tuning: Adjust TTLs, retry intervals, and backoff strategies for optimal application behavior Rollback if Needed: Revert recent changes to Bulwark deployments or configurations if correlated to failure spikes Alert Resolution: Alert will auto-resolve once lock acquisition failure rates in a namespace drop below 75%.

8.1.2.78 SM_UPDATE_NOTIFY_FAILED_ABOVE_50_PERCENT

Table 8-213 SM_UPDATE_NOTIFY_FAILED_ABOVE_50_PERCENT

Field	Details
Description	Update Notify Terminate sent to SMF failed >= 50 < 60
Summary	Update Notify Terminate sent to SMF failed >= 50 < 60
Severity	MINOR
Expression	(sum(occnp_http_out_conn_response_total{operationType="terminate_notify",pod=~".*smservice.*",servicename3gpp="npcf-smpolicycontrol",responseCode!~"2.*"})*100)/ sum(occnp_http_out_conn_response_total{operationType="terminate_notify",pod=~".*smservice.*",servicename3gpp="npcf-smpolicycontrol"}) >= 50 < 60
OID	1.3.6.1.4.1.323.5.3.52.1.2.80
Metric Used	occnp_http_out_conn_response_total
Recommended Actions	Cause This alert fires when, over the evaluation period, between 50% and 60% of terminate_notify HTTP outbound requests sent from PCF (SM Service pods) to SMF result in non-2xx (failed) HTTP responses. In this workflow, PCF notifies SMF to terminate a session. Notably, SMF will return a 404 error if the session does not exist in its current context. Elevated rates of 404 errors could indicate attempts to terminate already-removed sessions or stale references. Other common causes include: SMF service partial outage or overload Application-level errors (4xx other than 404, 5xx) Network issues Configuration mistakes Recent deployments or system changes Diagnostic Information Break down non-2xx responses by HTTP code (especially distinguishing 404s from 5xx or other 4xx) Check PCF and SM Service logs for error details related to terminate_notify requests Review SMF logs for the context and reasoning behind 404 responses Analyze the timing and volume of session termination requests compared to active session counts Correlate with recent maintenance, scaling events, or deployment changes Evaluate resource utilization and connectivity between PCF and SMF Recovery Determine Root Cause: Use error codes, logs, and traces to identify whether high 404s are expected (e.g., requests for sessions already removed) or whether there are issues with session tracking, race conditions, or stale data Rollback if Needed: If recent changes coincide with failures, consider rolling back deployments or configurations. Scale/Resources: Address resource exhaustion or performance bottlenecks as needed Alert Resolution: The alert will auto-resolve once failed response rates fall below 50% for the evaluation window. A higher-severity alert may trigger if failures exceed 60%. For any additional guidance, contact My Oracle Support.

8.1.2.79 SM_UPDATE_NOTIFY_FAILED_ABOVE_60_PERCENT

Table 8-214 SM_UPDATE_NOTIFY_FAILED_ABOVE_60_PERCENT

Field	Details
Description	Update Notify Terminate sent to SMF failed >= 60 < 70
Summary	Update Notify Terminate sent to SMF failed >= 60 < 70
Severity	MAJOR
Expression	(sum(occnp_http_out_conn_response_total{operationType="terminate_notify",pod=~".*smservice.*",servicename3gpp="npcf-smpolicycontrol",responseCode!~"2.*"})*100)/ sum(occnp_http_out_conn_response_total{operationType="terminate_notify",pod=~".*smservice.*",servicename3gpp="npcf-smpolicycontrol"}) >= 60 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.80
Metric Used	occnp_http_out_conn_response_total
Recommended Actions	Cause This alert fires when, over the evaluation period, between 60% and 70% of terminate_notify HTTP outbound requests sent from PCF (SM Service pods) to SMF result in non-2xx (failed) HTTP responses. In this workflow, PCF notifies SMF to terminate a session. Notably, SMF will return a 404 error if the session does not exist in its current context. Elevated rates of 404 errors could indicate attempts to terminate already-removed sessions or stale references. Other common causes include: SMF service partial outage or overload Application-level errors (4xx other than 404, 5xx) Network issues Configuration mistakes Recent deployments or system changes Diagnostic Information Break down non-2xx responses by HTTP code (especially distinguishing 404s from 5xx or other 4xx) Check PCF and SM Service logs for error details related to terminate_notify requests Review SMF logs for the context and reasoning behind 404 responses Analyze the timing and volume of session termination requests compared to active session counts Correlate with recent maintenance, scaling events, or deployment changes Evaluate resource utilization and connectivity between PCF and SMF Recovery Determine Root Cause: Use error codes, logs, and traces to identify whether high 404s are expected (e.g., requests for sessions already removed) or whether there are issues with session tracking, race conditions, or stale data Rollback if Needed: If recent changes coincide with failures, consider rolling back deployments or configurations. Scale/Resources: Address resource exhaustion or performance bottlenecks as needed Alert Resolution: The alert will auto-resolve once failed response rates fall below 60% for the evaluation window. A higher-severity alert may trigger if failures exceed 70%. For any additional guidance, contact My Oracle Support.

8.1.2.80 SM_UPDATE_NOTIFY_FAILED_ABOVE_70_PERCENT

Table 8-215 SM_UPDATE_NOTIFY_FAILED_ABOVE_70_PERCENT

Field	Details
Description	Update Notify Terminate sent to SMF failed >= 70
Summary	Update Notify Terminate sent to SMF failed >= 70
Severity	CRITICAL
Expression	(sum(occnp_http_out_conn_response_total{operationType="terminate_notify",pod=~".*smservice.*",servicename3gpp="npcf-smpolicycontrol",responseCode!~"2.*"})*100)/ sum(occnp_http_out_conn_response_total{operationType="terminate_notify",pod=~".*smservice.*",servicename3gpp="npcf-smpolicycontrol"}) >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.80
Metric Used	occnp_http_out_conn_response_total
Recommended Actions	Cause This alert fires when, over the evaluation period, above 70% of terminate_notify HTTP outbound requests sent from PCF (SM Service pods) to SMF result in non-2xx (failed) HTTP responses. In this workflow, PCF notifies SMF to terminate a session. Notably, SMF will return a 404 error if the session does not exist in its current context. Elevated rates of 404 errors could indicate attempts to terminate already-removed sessions or stale references. Other common causes include: SMF service partial outage or overload Application-level errors (4xx other than 404, 5xx) Network issues Configuration mistakes Recent deployments or system changes Diagnostic Information Break down non-2xx responses by HTTP code (especially distinguishing 404s from 5xx or other 4xx) Check PCF and SM Service logs for error details related to terminate_notify requests Review SMF logs for the context and reasoning behind 404 responses Analyze the timing and volume of session termination requests compared to active session counts Correlate with recent maintenance, scaling events, or deployment changes Evaluate resource utilization and connectivity between PCF and SMF Recovery Determine Root Cause: Use error codes, logs, and traces to identify whether high 404s are expected (e.g., requests for sessions already removed) or whether there are issues with session tracking, race conditions, or stale data Rollback if Needed: If recent changes coincide with failures, consider rolling back deployments or configurations. Scale/Resources: Address resource exhaustion or performance bottlenecks as needed Alert Resolution: The alert will auto-resolve once failed response rates fall below 70%. For any additional guidance, contact My Oracle Support.

8.1.2.81 UPDATE_NOTIFY_FAILURE_ABOVE_30_PERCENT

Table 8-216 UPDATE_NOTIFY_FAILURE_ABOVE_30_PERCENT

Field	Details
Name	UPDATE_NOTIFY_FAILURE_ABOVE_30_PERCENT
Description	{{ $value }} % of update notify sent to SMF that failed.
Summary	More than 30% of update notify sent to SMF failed
Severity	minor
Expression	sum by (namespace) (rate(occnp_http_out_conn_response_total{operationType="update_notify",microservice=~".*pcf_sm",responseCode!~"2.*"}[5m])) / sum by (namespace) (rate(occnp_http_out_conn_response_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.94
Metric Used	occnp_http_out_conn_response_total
Recommended Actions	Cause: At least 30 % of the sent Update Notify Requests have failed. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and SMF. Maybe Update Notify requests are not being sent, or taking too long to reach. Review Logs: Check PCF's EGW and SM logs to further investigate on why Update Notify requests are taking too long. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Recovery: Once Update Notify requests are properly processed, the percentage of failed requests will decrease, and the Alert will turn off by itself.

8.1.2.82 UPDATE_NOTIFY_FAILURE_ABOVE_50_PERCENT

Table 8-217 UPDATE_NOTIFY_FAILURE_ABOVE_50_PERCENT

Field	Details
Name	UPDATE_NOTIFY_FAILURE_ABOVE_50_PERCENT
Description	{{ $value }} % of update notify sent to SMF that failed
Summary	More than 50% of update notify sent to SMF failed
Severity	MAJOR
Expression	(sum by (namespace) (rate(occnp_http_out_conn_response_total{operationType="update_notify",microservice=~".*pcf_sm",responseCode!~"2.*"}[5m])) / sum by (namespace) (rate(occnp_http_out_conn_response_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.94
Metric Used	occnp_http_out_conn_response_total
Recommended Actions	Cause: At least 50 % of the sent Update Notify Requests have failed. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and SMF. Maybe Update Notify requests are not being sent, or taking too long to reach. Review Logs: Check PCF's EGW and SM logs to further investigate on why Update Notify requests are taking too long. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Recovery: Once Update Notify requests are properly processed, the percentage of failed requests will decrease, and the Alert will turn off by itself.

8.1.2.83 UPDATE_NOTIFY_FAILURE_ABOVE_70_PERCENT

Table 8-218 UPDATE_NOTIFY_FAILURE_ABOVE_70_PERCENT

Field	Details
Name	UPDATE_NOTIFY_FAILURE_ABOVE_70_PERCENT
Description	{{ $value }} % of update notify sent to SMF that failed
Summary	More than 70% of update notify sent to SMF failed
Severity	Critical
Expression	(sum by (namespace) (rate(occnp_http_out_conn_response_total{operationType="update_notify",microservice=~".*pcf_sm",responseCode!~"2.*"}[5m])) / sum by (namespace) (rate(occnp_http_out_conn_response_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.94
Metric Used	occnp_http_out_conn_response_total
Recommended Actions	Cause: At least 70 % of the sent Update Notify Requests have failed. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and SMF. Maybe Update Notify requests are not being sent, or taking too long to reach. Review Logs: Check PCF's EGW and SM logs to further investigate on why Update Notify requests are taking too long. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Recovery: Once Update Notify requests are properly processed, the percentage of failed requests will decrease, and the Alert will turn off by itself.

8.1.2.84 POD_PROTECTION_BY_RATELIMIT_REJECTED_REQUEST

Table 8-219 POD_PROTECTION_BY_RATELIMIT_REJECTED_REQUEST

Field	Details
Description	Ingress Gateway traffic gets rejected more than 1% because of ratelimiting.
Summary	Ingress Gateway traffic gets rejected more than 1% because of ratelimiting.
Severity	Major
Expression	(sum by (namespace,pod) (rate(oc_ingressgateway_http_request_ratelimit_values_total {Allowed="false",app_kubernetes_io_name="occnp-ingress-gateway"}[2m])))/ (sum by (namespace,pod) (rate(oc_ingressgateway_http_request_ratelimit_values_total {app_kubernetes_io_name="occnp-ingress-gateway"}[2m]))) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.103
Metric Used	oc_ingressgateway_http_request_ratelimit_values_total
Recommended Actions	Cause: Alert is triggered when percentage of denied requests is above 1% of total tps.. Diagnostic Information: Metric involved: oc_ingressgateway_http_request_ratelimit_values_total Error observed: 429 Too Many Requests, NF_CONGESTION_RISK Cause value: Allowed="false" Condition: podProtectionByRateLimiting.enabled = true and podProtectionByRateLimiting.fillRate settings Verification steps: podProtectionByRateLimiting.fillRate to a lower value and podProtectionByRateLimiting.deniedRequestActions.action=REJECT for lower congestion level Run 4500 TPS or above for SM traffic; Confirm some request dropped with Error 429. Verify that the alert get triggered. Monitoring recommendations: Monitor 4xx error; and counter increase for oc_ingressgateway_http_request_ratelimit_values_total{Allowed="false"} Watch for spikes following client deployments. Recovery: Check Network traffic burst and storm Investigate traffic load balancer issues and network issues. Review SM Service Resources Restart or scale up resources temporarily if the system is congested Reconfig setting for podProtectionByRateLimiting.fillRate to a higher value and assign podProtectionByRateLimiting.deniedRequestActions.action=REJECT to higher congestion level Disable feature if this flow is the only one affected we can disable this feature as a last resource For any additional guidance, contact My Oracle Support.

8.1.2.85 UE_N1N2_NOTIFY_REJECTION_RATE_ABOVE_MINOR_THRESHOLD

Table 8-220 UE_N1N2_NOTIFY_REJECTION_RATE_ABOVE_MINOR_THRESHOLD

Field	Details
Name	UE_N1N2_NOTIFY_REJECTION_RATE_ABOVE_MINOR_THRESHOLD
Description	UE N1N2 Notification Rate containing requests of MANAGE_UE_POLICY_COMMAND_REJECT from AMF is detected to be above 20 Percent of Total n1n2 notify Requests ( current value: {{ $value }} % )
Summary	Above 20 percent of total N1N2 Notify Requests from AMF are containing MANAGE_UE_POLICY_COMMAND_REJECT
Severity	Minor
Expression	sum by (namespace) (rate(ue_n1_transfer_ue_notification_total{commandType="MANAGE_UE_POLICY_COMMAND_REJECT"}[5m])) / sum by (namespace) (rate(ue_n1_transfer_response_total[5m])) * 100 >= 20 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.91
Metric Used	ue_n1_transfer_ue_notification_total ue_n1_transfer_response_total
Recommended Actions	Cause: At least 20 % of the sent UE Notifications sent are due to MANAGE_UE_POLICY_COMMAND_REJECT. Which means that a fragment delivered by PCF was rejected by UE. Diagnostic Information: Configurations: Check UE configurations and PRE Policies to understand why these requests could've been rejected. Logs: Check PCF UE and PRE logs to better understand why it could've been rejected. Recovery: Once UE Notifications are properly processed, the percentage of rejected requests will decrease, and the alert will turn off by itself.

8.1.2.86 UE_N1N2_NOTIFY_REJECTION_RATE_ABOVE_MAJOR_THRESHOLD

Table 8-221 UE_N1N2_NOTIFY_REJECTION_RATE_ABOVE_MAJOR_THRESHOLD

Field	Details
Name	UE_N1N2_NOTIFY_REJECTION_RATE_ABOVE_MAJOR_THRESHOLD
Description	UE N1N2 Notification Rate containing requests of MANAGE_UE_POLICY_COMMAND_REJECT from AMF is detected to be above 50 Percent of Total n1n2 notify Requests ( current value: {{ $value }} % )
Summary	Above 50 percent of total N1N2 Notify Requests from AMF are containing MANAGE_UE_POLICY_COMMAND_REJECT
Severity	Major
Expression	sum by (namespace) (rate(ue_n1_transfer_ue_notification_total{commandType="MANAGE_UE_POLICY_COMMAND_REJECT"}[5m])) / sum by (namespace) (rate(ue_n1_transfer_response_total[5m])) * 100 >= 50 < 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.91
Metric Used	ue_n1_transfer_ue_notification_total ue_n1_transfer_response_total
Recommended Actions	Cause: At least 50 % of the sent UE Notifications sent are due to MANAGE_UE_POLICY_COMMAND_REJECT. Which means that a fragment delivered by PCF was rejected by UE. Diagnostic Information: Configurations: Check UE configurations and PRE Policies to understand why these requests could've been rejected. Logs: Check PCF UE and PRE logs to better understand why it could've been rejected. Recovery: Once UE Notifications are properly processed, the percentage of rejected requests will decrease, and the alert will turn off by itself.

8.1.2.87 UE_N1N2_NOTIFY_REJECTION_RATE_ABOVE_CRITICAL_THRESHOLD

Table 8-222 UE_N1N2_NOTIFY_REJECTION_RATE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name	UE_N1N2_NOTIFY_REJECTION_RATE_ABOVE_CRITICAL_THRESHOLD
Description	UE N1N2 Notification Rate containing requests of MANAGE_UE_POLICY_COMMAND_REJECT from AMF is detected to be above 75 Percent of Total n1n2 notify Requests ( current value: {{ $value }} % )
Summary	Above 75 percent of total N1N2 Notify Requests from AMF are containing MANAGE_UE_POLICY_COMMAND_REJECT
Severity	critical
Expression	sum by (namespace) (rate(ue_n1_transfer_ue_notification_total{commandType="MANAGE_UE_POLICY_COMMAND_REJECT"}[5m])) / sum by (namespace) (rate(ue_n1_transfer_response_total[5m])) * 100 > 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.91
Metric Used	ue_n1_transfer_ue_notification_total ue_n1_transfer_response_total
Recommended Actions	Cause: At least 75 % of the sent UE Notifications sent are due to MANAGE_UE_POLICY_COMMAND_REJECT. Which means that a fragment delivered by PCF was rejected by UE. Diagnostic Information: Configurations: Check UE configurations and PRE Policies to understand why these requests could've been rejected. Logs: Check PCF UE and PRE logs to better understand why it could've been rejected. Recovery: Once UE Notifications are properly processed, the percentage of rejected requests will decrease, and the alert will turn off by itself.

8.1.2.88 UE_N1N2_TRANSFER_FAILURE_RATE_ABOVE_MINOR_THRESHOLD

Table 8-223 UE_N1N2_TRANSFER_FAILURE_RATE_ABOVE_MINOR_THRESHOLD

Field	Details
Name	UE_N1N2_TRANSFER_FAILURE_RATE_ABOVE_MINOR_THRESHOLD
Description	Above 20 percent of total N1N2 Transfer Requests are of N1N2 Transfer failure notification Requests from AMF ( current value: {{ $value }} % )
Summary	UE n1n2 transfer failure Notification Rate is above 20 percent of total n1n2 Transfers requests
Severity	minor
Expression	sum by (namespace) (rate(ue_n1_transfer_failure_notification_total[5m])) / sum by (namespace) (rate(ue_n1_transfer_response_total[5m])) * 100 >= 20 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.92
Metric Used	ue_n1_transfer_failure_notification_total ue_n1_transfer_response_total
Recommended Actions	Cause: At least 20 % of the sent N1N2 Transfer Requests have failed. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and AMF. Maybe N1N2 requests are not being sent, or taking too long to reach. Review Logs: Check PCF's EGW and UE logs to further investigate on why N1N2 Transfer requests are failing. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: Check AMF has proper retransmission and re-attempt configurations in place. Recovery: Once N1N2 Transfer requests are properly processed, the percentage of non-2xx responses will decrease, and the Alert will turn off by itself.

8.1.2.89 UE_N1N2_TRANSFER_FAILURE_RATE_ABOVE_MAJOR_THRESHOLD

Table 8-224 UE_N1N2_TRANSFER_FAILURE_RATE_ABOVE_MAJOR_THRESHOLD

Field	Details
Name	UE_N1N2_TRANSFER_FAILURE_RATE_ABOVE_MAJOR_THRESHOLD
Description	Above 50 percent of total N1N2 Transfer Requests are of N1N2 Transfer failure notification Requests from AMF ( current value: {{ $value }} % )
Summary	UE n1n2 transfer failure Notification Rate is above 50 percent of total n1n2 Transfers requests
Severity	major
Expression	sum by (namespace) (rate(ue_n1_transfer_failure_notification_total[5m])) / sum by (namespace) (rate(ue_n1_transfer_response_total[5m])) * 100 >= 50 < 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.92
Metric Used	ue_n1_transfer_failure_notification_total ue_n1_transfer_response_total
Recommended Actions	Cause: At least 50 % of the sent N1N2 Transfer Requests have failed. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and AMF. Maybe N1N2 requests are not being sent, or taking too long to reach. Review Logs: Check PCF's EGW and UE logs to further investigate on why N1N2 Transfer requests are failing. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: Check AMF has proper retransmission and re-attempt configurations in place. Recovery: Once N1N2 Transfer requests are properly processed, the percentage of non-2xx responses will decrease, and the Alert will turn off by itself.

8.1.2.90 UE_N1N2_TRANSFER_FAILURE_RATE_ABOVE_CRITICAL_THRESHOLD

Table 8-225 UE_N1N2_TRANSFER_FAILURE_RATE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name	UE_N1N2_TRANSFER_FAILURE_RATE_ABOVE_CRITICAL_THRESHOLD
Description	Above 75 percent of total N1N2 Transfer Requests are of N1N2 Transfer failure notification Requests from AMF ( current value: {{ $value }} % )
Summary	UE n1n2 transfer failure Notification Rate is above 75 percent of total n1n2 Transfers requests
Severity	critical
Expression	sum by (namespace) (rate(ue_n1_transfer_failure_notification_total[5m])) / sum by (namespace) (rate(ue_n1_transfer_response_total[5m])) * 100 >= 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.92
Metric Used	ue_n1_transfer_failure_notification_total ue_n1_transfer_response_total
Recommended Actions	Cause: At least 75 % of the sent N1N2 Transfer Requests have failed. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and AMF. Maybe N1N2 requests are not being sent, or taking too long to reach. Review Logs: Check PCF's EGW and UE logs to further investigate on why N1N2 Transfer requests are failing. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: Check AMF has proper retransmission and re-attempt configurations in place. Recovery: Once N1N2 Transfer requests are properly processed, the percentage of non-2xx responses will decrease, and the Alert will turn off by itself.

8.1.2.91 UE_N1N2_TRANSFER_T3501_TIMER_EXPIRY_RATE_ABOVE_MINOR_THRESHOLD

Table 8-226 UE_N1N2_TRANSFER_T3501_TIMER_EXPIRY_RATE_ABOVE_MINOR_THRESHOLD

Field	Details
Name	UE_N1N2_TRANSFER_T3501_TIMER_EXPIRY_RATE_ABOVE_MINOR_THRESHOLD
Description	Out of the total UE n1n2 Transfers more than 20 percent of transfer requests have T3501 Timer expiry before n1n2 notify is reached from AMF for that transfer ( current value: {{ $value }} % )
Summary	UE n1n2 Transfer T3501 timer expiry rate is above 20 percent of total n1n2 Transfers
Severity	minor
Expression	sum by (namespace) (rate(ue_n1_transfer_t3501_expiry_total[5m])) / sum by (namespace) (rate(ue_n1_transfer_response_total[5m])) * 100 >= 20 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.93
Metric Used	ue_n1_transfer_t3501_expiry_total ue_n1_transfer_response_total
Recommended Actions	Cause: At least 20 % of the sent N1N2 Transfer Requests have had its T3501 timer expired. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and AMF. Maybe N1N2 requests are not being sent, or taking too long to reach. Review Logs: Check PCF's EGW and UE logs to further investigate on why N1N2 Transfer requests are failing. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: Check AMF has proper retransmission and re-attempt configurations in place. If latency is too high, increase the T3501 Timer Duration field to a greater value. Recovery: Once N1N2 Transfer requests are properly processed, the percentage of expired T3501 timers will decrease, and the Alert will turn off by itself.

8.1.2.92 UE_N1N2_TRANSFER_T3501_TIMER_EXPIRY_RATE_ABOVE_MAJOR_THRESHOLD

Table 8-227 UE_N1N2_TRANSFER_T3501_TIMER_EXPIRY_RATE_ABOVE_MAJOR_THRESHOLD

Field	Details
Name	UE_N1N2_TRANSFER_T3501_TIMER_EXPIRY_RATE_ABOVE_MAJOR_THRESHOLD
Description	Out of the total UE n1n2 Transfers more than 50 percent of transfer requests have T3501 Timer expiry before n1n2 notify is reached from AMF for that transfer ( current value: {{ $value }} % )
Summary	UE n1n2 Transfer T3501 timer expiry rate is above 50 percent of total n1n2 Transfers
Severity	major
Expression	sum by (namespace) (rate(ue_n1_transfer_t3501_expiry_total[5m])) / sum by (namespace) (rate(ue_n1_transfer_response_total[5m])) * 100 >= 50 < 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.93
Metric Used	ue_n1_transfer_t3501_expiry_total ue_n1_transfer_response_total
Recommended Actions	Cause: At least 50 % of the sent N1N2 Transfer Requests have had its T3501 timer expired. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and AMF. Maybe N1N2 requests are not being sent, or taking too long to reach. Review Logs: Check PCF's EGW and UE logs to further investigate on why N1N2 Transfer requests are failing. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: Check AMF has proper retransmission and re-attempt configurations in place. If latency is too high, increase the T3501 Timer Duration field to a greater value. Recovery: Once N1N2 Transfer requests are properly processed, the percentage of expired T3501 timers will decrease, and the Alert will turn off by itself.

8.1.2.93 UE_N1N2_TRANSFER_T3501_TIMER_EXPIRY_RATE_ABOVE_CRITICAL_THRESHOLD

Table 8-228 UE_N1N2_TRANSFER_T3501_TIMER_EXPIRY_RATE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Name	UE_N1N2_TRANSFER_T3501_TIMER_EXPIRY_RATE_ABOVE_CRITICAL_THRESHOLD
Description	Out of the total UE n1n2 Transfers more than 75 percent of transfer requests have T3501 Timer expiry before n1n2 notify is reached from AMF for that transfer ( current value: {{ $value }} % )
Summary	UE n1n2 Transfer T3501 Timer expiry rate is above 75 percent of total n1n2 Transfers
Severity	critical
Expression	sum by (namespace) (rate(ue_n1_transfer_t3501_expiry_total[5m])) / sum by (namespace) (rate(ue_n1_transfer_response_total[5m])) * 100 >= 75
OID	1.3.6.1.4.1.323.5.3.52.1.2.93
Metric Used	ue_n1_transfer_t3501_expiry_total ue_n1_transfer_response_total
Recommended Actions	Cause: At least 75 % of the sent N1N2 Transfer Requests have had its T3501 timer expired. Diagnostic Information: Analyze Latency: Check how much latency there is between PCF and AMF. Maybe N1N2 requests are not being sent, or taking too long to reach. Review Logs: Check PCF's EGW and UE logs to further investigate on why N1N2 Transfer requests are failing. Scale System Resources: High CPU or memory usage may lead to some requests to not be processed. Scale pods as required by the traffic. Configuration: Check AMF has proper retransmission and re-attempt configurations in place. If latency is too high, increase the T3501 Timer Duration field to a greater value. Recovery: Once N1N2 Transfer requests are properly processed, the percentage of expired T3501 timers will decrease, and the Alert will turn off by itself.

8.1.2.94 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_ERROR_RESPONSE_ABOVE_CRITICAL_THRESHOLD

Table 8-229 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_ERROR_RESPONSE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	This alert is triggered when the number of update notify failed because a timeout is equal or above 70% in a given time period.
Summary	This alert is triggered when the number of update notify failed because a timeout is equal or above 70% in a given time period.
Severity	Critical
Expression	(sum by (namespace) (rate(ocpm_handle_update_notify_error_response_as_pending_confirmation_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m])) / sum by (namespace) (rate(ocpm_rx_update_notify_request_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.111
Metric Used	ocpm_handle_update_notify_error_response_as_pending_confirmation_total
Recommended Actions	Cause: Metric ocpm_handle_update_notify_error_response_as_pending_confirmation_total is pegged when the operation Update Notify towards SMF ends up with an Error Metrics: ocpm_handle_update_notify_error_response_as_pending_confirmation_total This will be incremented when configuration flag SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.ENABLED is enabled and specific error error is added in SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.RESPOMNSE_CODE and timeout happens during update notify triggered by AAR-I and AAR-U. Alarm Condition: If more than or equal to 70% of update_notify total requests fails with configured errorCode, an alarm is raised Diagnostic Information: Check Network Latency Investigate possible delays in network which is resulting in timeouts Verify sender information Verify if the notifUri where we are sending the information is correct Verify receiver NF Verify that the SMF that is receiving the traffic is in a healthy state Review application Verify if Sm is not congested If signs like constant error logs are showing Monitor System/resource utilization (CPU, Memory, queues) Recover: Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF and the SM Service. Resolve any high latency or packet loss immediately if detected. Review SM Service Application and Resources Restart or scale up resources temporarily if the system is overloaded Disable feature if this flow is the only one affected we can disable this feature as a last resource For any additional guidance, contact My Oracle Support.

8.1.2.95 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_ERROR_RESPONSE_ABOVE_MAJOR_THRESHOLD

Table 8-230 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_ERROR_RESPONSE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	This alert is triggered when the number of update notify failed because a timeout is equal or above 50% in a given time period.
Summary	This alert is triggered when the number of update notify failed because a timeout is equal or above 50% in a given time period.
Severity	Major
Expression	(sum by (namespace) (rate(ocpm_handle_update_notify_error_response_as_pending_confirmation_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m])) / sum by (namespace) (rate(ocpm_rx_update_notify_request_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.111
Metric Used	ocpm_handle_update_notify_error_response_as_pending_confirmation_total
Recommended Actions	Cause: Metric ocpm_handle_update_notify_error_response_as_pending_confirmation_total is pegged when the operation Update Notify towards SMF ends up with an Error Metrics: ocpm_handle_update_notify_error_response_as_pending_confirmation_total This will be incremented when configuration flag SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.ENABLED is enabled and specific error error is added in SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.RESPOMNSE_CODE and timeout happens during update notify triggered by AAR-I and AAR-U. Alarm Condition: If more than or equal to 50% but less than 70% of update_notify total requests fails with configured errorCode, an alarm is raised Diagnostic Information: Check Network Latency Investigate possible delays in network which is resulting in timeouts Verify sender information Verify if the notifUri where we are sending the information is correct Verify receiver NF Verify that the SMF that is receiving the traffic is in a healthy state Review application Verify if Sm is not congested If signs like constant error logs are showing Monitor System/resource utilization (CPU, Memory, queues) Recover: Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF and the SM Service. Resolve any high latency or packet loss immediately if detected. Review SM Service Application and Resources Restart or scale up resources temporarily if the system is overloaded Disable feature if this flow is the only one affected we can disable this feature as a last resource For any additional guidance, contact My Oracle Support.

8.1.2.96 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_ERROR_RESPONSE_ABOVE_MINOR_THRESHOLD

Table 8-231 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_ERROR_RESPONSE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	This alert is triggered when the number of update notify failed because a timeout is equal or above 30% but less than 50% of total Rx sessions.
Summary	This alert is triggered when the number of update notify failed because a timeout is equal or above 30% but less than 50% of total Rx sessions.
Severity	Minor
Expression	(sum by (namespace) (rate(ocpm_handle_update_notify_error_response_as_pending_confirmation_total{operationType="update_notify",microservice=~".*pcf_sm", responseCode=~"5xx/4xx"}[5m])) / sum by (namespace) (rate(ocpm_rx_update_notify_request_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.111
Metric Used	ocpm_handle_update_notify_error_response_as_pending_confirmation_total
Recommended Actions	Cause: Metric ocpm_handle_update_notify_error_response_as_pending_confirmation_total is pegged when the operation Update Notify towards SMF ends up with an error. Metrics: ocpm_handle_update_notify_error_response_as_pending_confirmation_total This will be incremented when: Configuration flag SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.ENABLED is enabled, and A specific error is added in SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.RESPONSE_CODE, and A timeout happens during Update Notify triggered by AAR-I and AAR-U. Alarm Condition: If ≥ 50% and < 70% of update_notify total requests fail with the configured error code, an alarm is raised. Diagnostic Information: Check Network Latency Investigate possible delays in the network that are resulting in timeouts. Verify Sender Information Verify if the notifUri where we are sending the information is correct. Verify Receiver NF Verify that the SMF receiving the traffic is in a healthy state. Review Application Verify that SM is not congested. Check for constant error logs. Monitor system/resource utilization (CPU, memory, queues). Recover: Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF and the SM service. Resolve any high latency or packet loss immediately if detected. Review SM Service Application and Resources Restart or scale up resources temporarily if the system is overloaded. Disable Feature If this flow is the only one affected, disable this feature as a last resort.

8.1.2.97 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_TIMEOUT_ABOVE_CRITICAL_THRESHOLD

Table 8-232 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_TIMEOUT_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	This alert is triggered when the number of update notify failed because a timeout is equal or above 70% in a given time period.
Summary	This alert is triggered when the number of update notify failed because a timeout is equal or above 70% in a given time period.
Severity	Critical
Expression	(sum by (namespace) (rate(ocpm_handle_update_notify_timeout_as_pending_confirmation_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m])) / sum by (namespace) (rate(ocpm_rx_update_notify_request_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.112
Metric Used	ocpm_handle_update_notify_timeout_as_pending_confirmation_total
Recommended Actions	Cause: Metric ocpm_handle_update_notify_timeout_as_pending_confirmation_total is pegged when the Update Notify operation towards SMF ends up with a timeout. Metrics: ocpm_handle_update_notify_timeout_as_pending_confirmation_total This will be incremented when: Configuration flag SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.ENABLED is enabled, and A specific error is added in SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.EXCEPTIONS , and A timeout happens during Update Notify triggered by AAR-I and AAR-U. Alarm Condition: If ≥ 70% of update_notify total requests fail with a timeout, an alarm is raised. Diagnostic Information: Check Network Latency Investigate possible delays in the network that are resulting in timeouts. Verify Sender Information Verify if the notifUri where we are sending the information is correct. Verify Receiver NF Verify that the SMF receiving the traffic is in a healthy state. Review Application Verify that SM is not congested. Check for constant error logs. Monitor system/resource utilization (CPU, memory, queues). Recover: Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF and the SM service. Resolve any high latency or packet loss immediately if detected. Review SM Service Application and Resources Restart or scale up resources temporarily if the system is overloaded. Disable Feature If this flow is the only one affected, disable this feature as a last resort. For any additional guidance, contact My Oracle Support.

8.1.2.98 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_TIMEOUT_ABOVE_MAJOR_THRESHOLD

Table 8-233 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_TIMEOUT_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	This alert is triggered when the number of update notify that failed because a timeout is equal or above 50% but less than 70% in a given time period.
Summary	This alert is triggered when the number of update notify that failed because a timeout is equal or above 50% but less than 70% in a given time period.
Severity	Major
Expression	(sum by (namespace) (rate(ocpm_handle_update_notify_timeout_as_pending_confirmation_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m])) / sum by (namespace) (rate(ocpm_rx_update_notify_request_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.112
Metric Used	ocpm_handle_update_notify_timeout_as_pending_confirmation_total
Recommended Actions	Cause: Metric ocpm_handle_update_notify_timeout_as_pending_confirmation_total is pegged when the operation Update Notify towards SMF ends up with a timeout. Metrics: ocpm_handle_update_notify_timeout_as_pending_confirmation_total This will be incremented when the configuration flag SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.ENABLED is enabled and a specific error is added in SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.EXCEPTIONS, and a timeout happens during update notify triggered by AAR-I and AAR-U. Alarm Condition: If more than or equal to 50% but less than 70% of update_notify total requests fail with a timeout, an alarm is raised. Diagnostic Information: Check Network Latency Investigate possible delays in the network which are resulting in timeouts. Verify sender information Verify if the notifUri where we are sending the information is correct. Verify receiver NF Verify that the SMF that is receiving the traffic is in a healthy state. Review application Verify if SM is not congested. Look for signs such as constant error logs. Monitor system/resource utilization (CPU, memory, queues). Recover: Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF and the SM Service. Resolve any high latency or packet loss immediately if detected. Review SM Service Application and Resources Restart or scale up resources temporarily if the system is overloaded. Disable feature If this flow is the only one affected, you can disable this feature as a last resort. For any additional guidance, contact My Oracle Support.

8.1.2.99 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_TIMEOUT_ABOVE_MINOR_THRESHOLD

Table 8-234 RX_PENDING_CONFIRMATION_UPDATE_NOTIFY_TIMEOUT_ABOVE_MINOR_THRESHOLD

Field	Details
Description	This alert is triggered when the number of update notify that failed because a timeout is equal or above 30% but less than 50% of total Rx sessions.
Summary	This alert is triggered when the number of update notify that failed because a timeout is equal or above 30% but less than 50% of total Rx sessions.
Severity	Minor
Expression	(sum by (namespace) (rate(ocpm_handle_update_notify_timeout_as_pending_confirmation_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m])) / sum by (namespace) (rate(ocpm_rx_update_notify_request_total{operationType="update_notify",microservice=~".*pcf_sm"}[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.112
Metric Used	ocpm_handle_update_notify_timeout_as_pending_confirmation_total
Recommended Actions	Cause: Metric ocpm_handle_update_notify_timeout_as_pending_confirmation_total is pegged when the Update Notify operation towards SMF ends up with a timeout. Metrics: ocpm_handle_update_notify_timeout_as_pending_confirmation_total This will be incremented when: Configuration flag SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.ENABLED is enabled, and A specific error is added in SYSTEM.RX.UPDATE_NOTIFY.RULES.PENDING_CONFIRMATION.EXCEPTIONS , and A timeout happens during Update Notify triggered by AAR-I and AAR-U. Alarm Condition: If ≥ 30% and < 50% of update_notify total requests fail with a timeout, an alarm is raised. Diagnostic Information: Check Network Latency Investigate possible delays in the network that are resulting in timeouts. Verify Sender Information Verify if the notifUri where we are sending the information is correct. Verify Receiver NF Verify that the SMF receiving the traffic is in a healthy state. Review Application Verify that SM is not congested. Check for constant error logs. Monitor system/resource utilization (CPU, memory, queues). Recover: Check Network Latency and Connectivity Investigate any current network issues or bottlenecks between the external NF and the SM service. Resolve any high latency or packet loss immediately if detected. Review SM Service Application and Resources Restart or scale up resources temporarily if the system is overloaded. Disable Feature If this flow is the only one affected, disable this feature as a last resort. For any additional guidance, contact My Oracle Support.

8.1.2.100 PCF_STATE_NON_FUNCTIONAL_CRITICAL

Table 8-235 PCF_STATE_NON_FUNCTIONAL_CRITICAL

Field	Details
Description	Policy is in non functional state due to DB cluster state down.
Summary	Policy is in non functional state due to DB cluster state down.
Severity	Critical
Expression	appinfo_nfDbFunctionalState_current{nfDbFunctionalState="Not_Running"} == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.102
Metric Used	appinfo_nfDbFunctionalState_current
Recommended Actions	For any additional guidance, contact My Oracle Support.

8.1.2.101 UDR_GET_REVALIDATION_FAILURE_ABOVE_MAJOR_PERCENT

Table 8-236 UDR_GET_REVALIDATION_FAILURE_ABOVE_MAJOR_PERCENT

Field	Details
Description	This alert is triggered when more than or equal to 50% but less that 70% of the UDR revalidation using method GET call failed.
Summary	This alert is triggered when more than or equal to 50% but less that 70% of the UDR revalidation using method GET call failed.
Severity	Major
Expression	(sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",response_code!~"2.*",service_resource="subscription-revalidation"}[5m])) / sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",service_resource="subscription-revalidation"}[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.108
Metric Used	ocpm_udr_tracking_response_total
Recommended Actions	The ocpm_udr_tracking_response_total metric is pegged whenever a response from UDR is received in UDR connector. This alert is notifying when the number of responses received from UDR for operation resubscribe that failed is above the threshold mentioned. Cause: The ocpm_udr_tracking_response_total metric is pegged whenever a response is received from the UDR in the UDR Connector. In this case, alerts are triggered when the number of failed responses received from UDR for the resubscribe operation exceeds the configured threshold. This alert is triggered when more than 50% but less than 70% of GET calls for UDR revalidation (operation_type=resubscribe, service_resource=subscription-revalidation) sent by the PCF-UserService fail (i.e., receive non-2xx HTTP response codes). Diagnostic Information: Check Recent Logs Review logs from the PCF UDR Connector and Egress Gateway for the relevant time intervals. Review errors at SCP routing. Identify the failure responses—look for non-2xx HTTP status codes and any error payloads. Analyze Failure Patterns Determine if failures are tied to specific UDRs, subscriber groups, or are distributed across all revalidations. Assess whether there is a spike in failed revalidations or if failures are intermittent. Inspect UDR Health and Reachability Verify the health and responsiveness of the UDR service. Check network connectivity from the PCF to UDR; look for timeouts, DNS errors, or other connectivity issues in intermediary services (EGW, SCP). Review PCF–UDR Connector Configuration Ensure proper configuration of endpoints, service credentials, and connection settings between PCF and UDR. Review any recent configuration or deployment changes that might correspond to the start of failures. Check for Resource or Rate Limiting Evaluate whether there are signs of resource exhaustion (CPU, memory, network) on either service. Investigate if the UDR is rate-limiting incoming requests or experiencing overload. Correlate with Related Alerts or Incidents Cross-check whether other alerts in the same namespace indicate broader issues (e.g., infrastructure, dependency outages, authentication errors). Recovery: Restore UDR Service Health Address any service outages, restarts, or degraded performance on the UDR side. If resource constraints are detected, consider scaling UDR or optimizing load. Fix Connectivity or Configuration Issues Resolve network issues (latency, DNS, firewall). Correct any erroneous endpoint URLs or authentication parameters in PCF User or UDR Connector configurations. For any additional guidance, contact My Oracle Support.

8.1.2.102 UDR_GET_REVALIDATION_FAILURE_ABOVE_CRITICAL_PERCENT

Table 8-237 UDR_GET_REVALIDATION_FAILURE_ABOVE_CRITICAL_PERCENT

Field	Details
Description	This alert is triggered when more than 70% of the UDR revalidation using method GET call failed.
Summary	This alert is triggered when more than 70% of the UDR revalidation using method GET call failed.
Severity	Critical
Expression	(sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",response_code!~"2.*",service_resource="subscription-revalidation"}[5m])) / sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",service_resource="subscription-revalidation"}[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.108
Metric Used	ocpm_udr_tracking_response_total
Recommended Actions	The ocpm_udr_tracking_response_total metric is pegged whenever a response from UDR is received in UDR connector. This alert is notifying when the number of responses received from UDR for operation resubscribe that failed is above the threshold mentioned. Cause: The ocpm_udr_tracking_response_total metric is pegged whenever a response is received from the UDR in the UDR Connector. In this case, alerts are triggered when the number of failed responses received from UDR for the resubscribe operation exceeds the configured threshold. This alert is triggered when more than 70% of GET calls for UDR revalidation (operation_type=resubscribe, service_resource=subscription-revalidation) sent by the PCF-UserService fail (i.e., receive non-2xx HTTP response codes). Diagnostic Information: Check Recent Logs Review logs from PCF UDR Connector and Egress Gateway for the relevant time intervals. Review errors at SCP routing. Identify the failure responses — look for non-2xx HTTP status codes and any error payloads. Analyze Failure Patterns Determine if failures are tied to specific UDRs, subscriber groups, or are distributed across all revalidations. Assess whether there is a spike in failed revalidations or if failures are intermittent. Inspect UDR Health and Reachability Verify the health and responsiveness of the UDR service. Check network connectivity from the PCF to UDR; look for timeouts, DNS errors, or other connectivity issues in intermediary services (EGW, SCP). Review PCF–UDR Connector Configuration Ensure proper configuration of endpoints, service credentials, and connection settings between PCF and UDR. Review any recent configuration or deployment changes that might correspond to the start of failures. Check for Resource or Rate Limiting Evaluate whether there are signs of resource exhaustion (CPU, memory, network) on either service. Investigate if the UDR is rate-limiting incoming requests or experiencing overload. Correlate with Related Alerts or Incidents Cross-check whether other alerts in the same namespace indicate broader issues (e.g., infrastructure, dependency outages, authentication errors). Recovery: Restore UDR Service Health Address any service outages, restarts, or degraded performance on the UDR side. If resource constraints are detected, consider scaling UDR or optimizing load. Fix Connectivity or Configuration Issues Resolve network issues (latency, DNS, firewall). Correct any erroneous endpoint URLs or authentication parameters in PCF User or UDR Connector configurations. For any additional guidance, contact My Oracle Support.

8.1.2.103 UDR_GET_REVALIDATION_FAILURE_ABOVE_MINOR_PERCENT

Table 8-238 UDR_GET_REVALIDATION_FAILURE_ABOVE_MINOR_PERCENT

Field	Details
Description	This alert is triggered when more than or equal to 30% but less that 50% of the UDR revalidation using method GET call failed.
Summary	This alert is triggered when more than or equal to 30% but less that 50% of the UDR revalidation using method GET call failed.
Severity	Minor
Expression	(sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",response_code!~"2.*",service_resource="subscription-revalidation"}[5m])) / sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",service_resource="subscription-revalidation"}[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.108
Metric Used	ocpm_udr_tracking_response_total
Recommended Actions	The ocpm_udr_tracking_response_total metric is pegged whenever a response from UDR is received in UDR connector. This alert is notifying when the number of responses received from UDR for operation resubscribe that failed is above the threshold mentioned. Cause: The ocpm_udr_tracking_response_total metric is pegged whenever we receive a response from the UDR in the UDR Connector. In this case, alerts are triggered when the number of failed responses received from UDR for the resubscribe operation is above the configured threshold. This alert is triggered when more than 30% but less than 50% of GET calls for UDR revalidation ( operation_type=resubscribe , service_resource=subscription-revalidation ) sent by the PCF-UserService fail (i.e., receive non-2xx HTTP response codes). Diagnostic Information: Check Recent Logs Review logs from the PCF UDR Connector and Egress Gateway for the relevant time intervals. Review errors at SCP routing. Identify the failure responses — look for non-2xx HTTP status codes and any error payloads. Analyze Failure Patterns Determine if failures are tied to specific UDRs, subscriber groups, or are distributed across all revalidations. Assess if there is a spike in failed revalidations or if failures are intermittent. Inspect UDR Health and Reachability Verify the health and responsiveness of the UDR service. Check network connectivity from the PCF to UDR; look for timeouts, DNS errors, or other connectivity issues in intermediary services (EGW, SCP). Review PCF–UDR Connector Configuration Ensure proper configuration of endpoints, service credentials, and connection settings between PCF and UDR. Review any recent configuration or deployment changes that might correspond to the start of failures. Check for Resource or Rate Limiting Evaluate if there are signs of resource exhaustion (CPU, memory, network) on either service. Investigate if the UDR is rate-limiting incoming requests or experiencing overload. Correlate with Related Alerts or Incidents Cross-check if other alerts in the same namespace indicate broader issues (e.g., infrastructure, dependency outages, authentication errors). Recovery: Restore UDR Service Health Address any service outages, restarts, or degraded performance on the UDR side. If resource constraints are detected, consider scaling UDR or optimizing load. Fix Connectivity or Configuration Issues Resolve network issues (latency, DNS, firewall). Correct any erroneous endpoint URLs or authentication parameters in PCF User or UDR Connector configurations. For any additional guidance, contact My Oracle Support.

8.1.2.104 UDR_GET_REVALIDATION_404_FAILURE_ABOVE_CRITICAL_PERCENT

Table 8-239 UDR_GET_REVALIDATION_404_FAILURE_ABOVE_CRITICAL_PERCENT

Field	Details
Description	This alert is triggered when more than 70% of the UDR revalidation using method GET call failed with status code 404 NOT FOUND.
Summary	This alert is triggered when more than 70% of the UDR revalidation using method GET call failed with status code 404 NOT FOUND.
Severity	Critical
Expression	(sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",response_code="404",service_resource="subscription-revalidation"}[5m])) / sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",service_resource="subscription-revalidation"}[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.110
Metric Used	ocpm_udr_tracking_response_total
Recommended Actions	The ocpm_udr_tracking_response_total metric is pegged whenever a response from UDR in UDR connector is received. This alert notifies when the number of responses received from UDR for operation resubscribe that failed with a 404 Not Found is above the threshold mentioned. Cause: This alert is triggered when more than 70% of UDR revalidation GET operations managed by PCF fail with an HTTP 404 (Not Found) response code within the specified window. A 404 response indicates that the requested subscription for revalidation was not found in UDR. Diagnostic Information: Check UDR logs: Look for 404 errors and the accompanying trigger request details (trigger of UDR revalidation request) for the affected time period. Verify subscription states: Ensure that the subscriptions expected to be present actually exist and are not being deleted, expired, or unavailable. Investigate Missing Subscriptions: Determine why the revalidation call is being made for a non-existent subscription ID. If the subscription has gone stale, verify why an audit was not triggered for it. Check if there is a data synchronization issue between the originator and UDR. Look for patterns: Determine whether the 404s are concentrated in a particular user group. Recovery: Audit Subscription Lifecycle: Ensure proper creation, update, and deletion workflows so stale subscription IDs are not reused or referenced. Review Recent Deployments or Changes: Check whether recent code or configuration changes in pcf_user or UDR might have led to increased 404s. For any additional guidance, contact My Oracle Support.

8.1.2.105 UDR_GET_REVALIDATION_404_FAILURE_ABOVE_MAJOR_PERCENT

Table 8-240 UDR_GET_REVALIDATION_404_FAILURE_ABOVE_MAJOR_PERCENT

Field	Details
Description	This alert is triggered when more than or equal to 50% but less that 70% of the UDR revalidation using method GET call failed.
Summary	This alert is triggered when more than or equal to 50% but less that 70% of the UDR revalidation using method GET call failed.
Severity	Major
Expression	(sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",response_code="404",service_resource="subscription-revalidation"}[5m])) / sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",service_resource="subscription-revalidation"}[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.110
Metric Used	ocpm_udr_tracking_response_total
Recommended Actions	The ocpm_udr_tracking_response_total metric is pegged whenever a response from UDR in UDR connector is received. This alert notifies when the number of responses received from UDR for operation resubscribe that failed with a 404 Not Found is above the threshold mentioned. Cause: This alert is triggered when more than 50% (but less than 70%) of UDR revalidation GET operations managed by PCF fail with an HTTP 404 (Not Found) response code within the specified window. A 404 response indicates that the requested subscription for revalidation was not found in UDR. Diagnostic Information: Check UDR logs: Look for 404 errors and the accompanying trigger request details (trigger of UDR revalidation request) for the affected time period. Verify subscription states: Ensure that the subscriptions expected to be present actually exist and are not being deleted, expired, or unavailable. Investigate missing subscriptions: Determine why the revalidation call is being made for a non-existent subscription ID. If the subscription has gone stale, verify why an audit was not triggered for the same. Check if there is a data synchronization issue between the originator and UDR. Look for patterns: Determine whether the 404s are concentrated in a particular user group. Recovery: Audit subscription lifecycle: Ensure proper creation, update, and deletion workflows so stale subscription IDs are not reused or referenced. Review recent deployments or changes: Check whether recent code or configuration changes in pcf_user or UDR might have led to increased 404s. For any additional guidance, contact My Oracle Support.

8.1.2.106 UDR_GET_REVALIDATION_404_FAILURE_ABOVE_MINOR_PERCENT

Table 8-241 UDR_GET_REVALIDATION_404_FAILURE_ABOVE_MINOR_PERCENT

Field	Details
Description	This alert is triggered when more than or equal to 30% but less that 50% of the UDR revalidation using method GET call failed.
Summary	This alert is triggered when more than or equal to 30% but less that 50% of the UDR revalidation using method GET call failed.
Severity	Minor
Expression	(sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",response_code="404",service_resource="subscription-revalidation"}[5m])) / sum by (namespace) (rate(ocpm_udr_tracking_response_total{operation_type="resubscribe",microservice=~".*pcf_user",service_resource="subscription-revalidation"}[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.110
Metric Used	ocpm_udr_tracking_response_total
Recommended Actions	The ocpm_udr_tracking_response_total metric is pegged whenever a response from UDR in UDR connector is received. This alert notifies when the number of responses received from UDR for operation resubscribe that failed with a 404 Not Found is above the threshold mentioned. Cause: This alert is triggered when more than 30% (but less than 50%) of UDR revalidation GET operations managed by PCF fail with an HTTP 404 (Not Found) response code within the specified window. A 404 response indicates that the requested subscription for revalidation was not found in UDR. Diagnostic Information: Check UDR logs: Look for 404 errors and the accompanying trigger request details (trigger of UDR revalidation request) for the affected time period. Verify subscription states: Ensure that the subscriptions expected to be present actually exist and are not being deleted, expired, or unavailable. Investigate missing subscriptions: Determine why the revalidation call is being made for a non-existent subscription ID. If the subscription has gone stale, verify why an audit was not triggered for the same. Check if there is a data synchronization issue between the originator and UDR. Look for patterns: Determine whether the 404s are concentrated in a particular user group. Recovery: Audit subscription lifecycle: Ensure proper creation, update, and deletion workflows so stale subscription IDs are not reused or referenced. Review recent deployments or changes: Check whether recent code or configuration changes in pcf_user or UDR might have led to increased 404s. For any additional guidance, contact My Oracle Support.

8.1.2.107 UDR_AM_IMMREP_RESPONSE_MISSING_DATA_MINOR

Table 8-242 UDR_AM_IMMREP_RESPONSE_MISSING_DATA_MINOR

Field	Details
Description	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but without user data for AM as part of immediate reporting
Summary	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but without user data for AM as part of immediate reporting
Severity	Minor
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post",imm_reports_present="false"}[5m])) / sum(rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post"}[5m]))) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.116
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. The missing AM user data check is based on: service_subresource = "am-data" (indicates the UDR POST was to get AM user data from UDR) operation_type = "POST" (indicates this is a POST call) imm_reports_present = "false" (indicates no AM user data was returned from UDR as part of the Immediate Reporting capability) If these metric dimensions are satisfied, then the alarm will trigger. Alarm Condition: More than or equal to 10% but less than 20% of the traffic: UDR returned a POST Subscribe response without AM user data as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR in the POST REST API request payload has the 30th byte set to 1 when converted to hex (e.g., 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Inform UDR Operator If the above points are validated and AM user data is still not retrieved, inform the UDR operators to verify whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.108 UDR_AM_IMMREP_RESPONSE_MISSING_DATA_MAJOR

Table 8-243 UDR_AM_IMMREP_RESPONSE_MISSING_DATA_MAJOR

Field	Details
Description	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for AM as part of immediate reporting
Summary	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for AM as part of immediate reporting
Severity	Major
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post",imm_reports_present="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post"}[5m]))) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.116
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. The missing AM user data check is based on: service_subresource = "am-data" (to indicate the UDR POST was to get AM user data from UDR) operation_type = "POST" (to determine this is a POST call) imm_reports_present = "false" (to indicate no AM user data was returned from UDR as part of the Immediate Reporting capability) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 20% but less than 30% of the traffic: UDR returned a POST Subscribe response without user data for AM as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR in the POST REST API request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Inform UDR Operator If the above points are validated and still no AM user data is retrieved, inform the UDR operators whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.109 UDR_AM_IMMREP_RESPONSE_MISSING_DATA_CRITICAL

Table 8-244 UDR_AM_IMMREP_RESPONSE_MISSING_DATA_CRITICAL

Field	Details
Description	More than or equal to 30% of the traffic, UDR returned with POST subscribe response but without user data for AM as part of immediate reporting.
Summary	More than or equal to 30% of the traffic, UDR returned with POST subscribe response but without user data for AM as part of immediate reporting.
Severity	Critical
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post",imm_reports_present="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post"}[5m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.116
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. The missing AM user data check is based on: service_subresource = "am-data" (to indicate the UDR POST was to get AM user data from UDR) operation_type = "POST" (to determine this is a POST call) imm_reports_present = "false" (to indicate no AM user data was returned from UDR as part of the Immediate Reporting capability) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 30% of the traffic: UDR returned a POST Subscribe response without user data for AM as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in the request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Inform UDR Operator If the above points are validated and AM user data is still not retrieved, inform the UDR operators whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.110 UDR_AM_IMMREP_FEATURE_NEGOTIATION_FAILED_MINOR

Table 8-245 UDR_AM_IMMREP_FEATURE_NEGOTIATION_FAILED_MINOR

Field	Details
Description	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for AM as part of immediate reporting
Summary	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for AM as part of immediate reporting
Severity	Minor
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post",immediate_report_pcc="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post"}[5m]))) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.117
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. The failed feature negotiation check is based on: service_subresource = "am-data" (to indicate the UDR POST was to get AM user data from UDR) operation_type = "POST" (to determine this is a POST call) immediate_report_pcc = "false" (to indicate that no feature negotiation happened with UDR on the ImmReportPcc feature) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 10% but less than 20% of the traffic: UDR returned a POST Subscribe response with failed feature negotiation for AM as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR in the POST REST API request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Inform UDR Operator If the above points are validated and AM user data is still not retrieved, inform the UDR operators whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.111 UDR_AM_IMMREP_FEATURE_NEGOTIATION_FAILED_MAJOR

Table 8-246 UDR_AM_IMMREP_FEATURE_NEGOTIATION_FAILED_MAJOR

Field	Details
Description	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for AM as part of immediate reporting
Summary	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for AM as part of immediate reporting
Severity	Major
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post",immediate_report_pcc="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post"}[5m]))) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.117
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. The failed feature negotiation check is based on: service_subresource = "am-data" (to indicate the UDR POST was to get AM user data from UDR) operation_type = "POST" (to determine this is a POST call) immediate_report_pcc = "false" (to indicate that no feature negotiation happened with UDR on the ImmReportPcc feature) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 20% but less than 30% of the traffic: UDR returned a POST Subscribe response with failed feature negotiation for AM as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in the request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Inform UDR Operator If the above points are validated and AM user data is still not retrieved, inform the UDR operators whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.112 UDR_AM_IMMREP_FEATURE_NEGOTIATION_FAILED_CRITICAL

Table 8-247 UDR_AM_IMMREP_FEATURE_NEGOTIATION_FAILED_CRITICAL

Field	Details
Description	More than or equal to 30% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for AM as part of immediate reporting
Summary	More than or equal to 30% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for AM as part of immediate reporting
Severity	Critical
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post",immediate_report_pcc="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="am-data",operation_type="post"}[5m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.117
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. The failed feature negotiation is based on: service_subresource = "am-data" (indicates the UDR POST was to get AM user data from UDR) operation_type = "POST" (indicates this is a POST call) immediate_report_pcc = "false" (indicates that no feature negotiation happened with UDR on the ImmReportPcc feature) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 30% of the traffic: UDR returned a POST Subscribe response with failed feature negotiation for AM as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in its request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Inform UDR Operator If the above points are validated and AM user data is still not retrieved: Inform the UDR operators. Ask them to verify whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.113 UDR_UE_IMMREP_RESPONSE_MISSING_DATA_MINOR

Table 8-248 UDR_UE_IMMREP_RESPONSE_MISSING_DATA_MINOR

Field	Details
Description	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but without user data for UE as part of immediate reporting
Summary	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but without user data for UE as part of immediate reporting
Severity	Minor
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post",imm_reports_present="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post"}[5m]))) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.118
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. The missing UE user data check is based on: service_subresource = "ue-policy-set" (indicates the UDR POST was to get UE user data from UDR) operation_type = "POST" (indicates this is a POST call) imm_reports_present = "false" (indicates no UE user data was returned from UDR as part of the Immediate Reporting capability) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 10% but less than 20% of the traffic: UDR returned a POST Subscribe response without UE user data as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in its request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Inform UDR Operator If the above points are validated and UE user data is still not retrieved: Inform the UDR operators. Ask them to verify whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.114 UDR_UE_IMMREP_RESPONSE_MISSING_DATA_MAJOR

Table 8-249 UDR_UE_IMMREP_RESPONSE_MISSING_DATA_MAJOR

Field	Details
Description	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for UE as part of immediate reporting
Summary	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but without user data for UE as part of immediate reporting
Severity	Major
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post",imm_reports_present="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post"}[5m]))) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.118
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: The metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. The missing UE user data check is based on: service_subresource = "ue-policy-set" (to indicate the UDR POST was to get UE user data from UDR) operation_type = "POST" (to determine this is a POST call) imm_reports_present = "false" (to indicate no UE user data was returned from UDR as part of the Immediate Reporting capability) If these metric dimensions are satisfied, the alarm will trigger. Alarm Condition: More than or equal to 20% but less than 30% of the traffic: UDR returned a POST Subscribe response without UE user data as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in its request payload has the 30th byte set to 1 when converted to hex (for example, 40000000). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Inform UDR Operator If the above points are validated and UE user data is still not retrieved: Inform the UDR operators. Ask them to verify whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.115 UDR_UE_IMMREP_RESPONSE_MISSING_DATA_CRITICAL

Table 8-250 UDR_UE_IMMREP_RESPONSE_MISSING_DATA_CRITICAL

Field	Details
Description	More than or equal to 30% of the traffic, UDR returned with POST subscribe response but without user data for UE as part of immediate reporting
Summary	More than or equal to 30% of the traffic, UDR returned with POST subscribe response but without user data for UE as part of immediate reporting
Severity	CRITICAL
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post",imm_reports_present="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post"}[5m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.118
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: Metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for a POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. The missing UE user data check is based on: service_subresource = "ue-policy-set" (to indicate the UDR POST was to get UE user data from UDR) operation_type = "POST" (to determine this is a POST call) imm_reports_present = "false" (to indicate no UE user data was returned from UDR as part of the Immediate Reporting capability) If these metric dimensions are satisfied, then the alarm will trigger. Alarm Condition: More than or equal to 30% of the traffic: UDR returned a POST Subscribe response without UE user data as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in its request payload has the 30th byte allocated as 1 when converted to hex (for example, "40000000"). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Intimate UDR Operator If the above points are validated and still no UE user data is retrieved, then intimate the same to UDR operators to check whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile being chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.116 UDR_UE_IMMREP_FEATURE_NEGOTIATION_FAILED_MINOR

Table 8-251 UDR_UE_IMMREP_FEATURE_NEGOTIATION_FAILED_MINOR

Field	Details
Description	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for UE as part of immediate reporting
Summary	More than or equal to 10% but less that 20% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for UE as part of immediate reporting
Severity	Minor
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post",immediate_report_pcc="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post"}[5m]))) * 100 >= 10 < 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.119
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: Metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. The failed feature negotiation is based on: service_subresource = "ue-policy-set" (to indicate the UDR POST was to get UE user data from UDR) operation_type = "POST" (to determine this is a POST call) immediate_report_pcc = "false" (to indicate that no feature negotiation happened with UDR on the ImmReportPcc feature) If these metric dimensions are satisfied, then the alarm will trigger. Alarm Condition: More than or equal to 10% but less than 20% of the traffic: UDR returned a POST Subscribe response with failed feature negotiation for UE as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in its request payload has the 30th byte allocated as 1 when converted to hex (for example, "40000000"). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Intimate UDR Operator If the above points are validated and still no UE user data is retrieved, then intimate the same to UDR operators to check whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile being chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.117 UDR_UE_IMMREP_FEATURE_NEGOTIATION_FAILED_MAJOR

Table 8-252 UDR_UE_IMMREP_FEATURE_NEGOTIATION_FAILED_MAJOR

Field	Details
Description	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for UE as part of immediate reporting
Summary	More than or equal to 20% but less that 30% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for UE as part of immediate reporting
Severity	Major
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post",immediate_report_pcc="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post"}[5m]))) * 100 >= 20 < 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.119
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: Metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. The failed feature negotiation is based on: service_subresource = "ue-policy-set" (to indicate the UDR POST was to get UE user data from UDR) operation_type = "POST" (to determine this is a POST call) immediate_report_pcc = "false" (to indicate that no feature negotiation happened with UDR on the ImmReportPcc feature) If these metric dimensions are satisfied, then the alarm will trigger. Alarm Condition: More than or equal to 20% but less than 30% of the traffic: UDR returned a POST Subscribe response with failed feature negotiation for UE as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in its request payload has the 30th byte allocated as 1 when converted to hex (for example, "40000000"). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the UDR POST request payload. Verify UDR Profile Ensure User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Intimate UDR Operator If the above points are validated and still no UE user data is retrieved, then intimate the same to UDR operators to check whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile being chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.118 UDR_UE_IMMREP_FEATURE_NEGOTIATION_FAILED_CRITICAL

Table 8-253 UDR_UE_IMMREP_FEATURE_NEGOTIATION_FAILED_CRITICAL

Field	Details
Description	More than or equal to 30% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for UE as part of immediate reporting
Summary	More than or equal to 30% of the traffic, UDR returned with POST subscribe response but with failed feature negotiation for UE as part of immediate reporting
Severity	Critical
Expression	(sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post",immediate_report_pcc="false"}[5m]))) / (sum by (namespace) (rate(occnp_immrep_response_total{service_subresource="ue-policy-set",operation_type="post"}[5m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.119
Metric Used	occnp_immrep_response_total
Recommended Actions	Cause: Metric occnp_immrep_response_total is pegged when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. Metric: occnp_immrep_response_total Increments when UDR-C receives a user data response from UDR for POST Subscription with Immediate Reporting. The failed feature negotiation is based on: service_subresource = "ue-policy-set" (to indicate the UDR POST was to get UE user data from UDR) operation_type = "POST" (to determine this is a POST call) immediate_report_pcc = "false" (to indicate that no feature negotiation happened with UDR on the ImmReportPcc feature) If these metric dimensions are satisfied, then the alarm will trigger. Alarm Condition: More than or equal to 30% of the traffic: UDR returned a POST Subscribe response with failed feature negotiation for UE as part of Immediate Reporting. Diagnostic Information: Verify ImmReportPcc Ensure the suppFeat attribute sent towards UDR for the POST REST API call in its request payload has the 30th byte allocated as 1 when converted to hex (for example, "40000000"). This is crucial for feature negotiation with UDR. Verify immRep Ensure the immRep attribute is set to true in the request payload for the UDR POST. Verify UDR Profile Ensure that User Data is requested only for those UDR profiles that PCF obtained from NRF with the ImmReportPcc feature enabled. Last Resort – Intimate UDR Operator If the above points are validated and still no UE user data is retrieved, then intimate the same to UDR operators to check whether the Immediate Reporting feature is working and negotiated from their end. Recovery: Verify the suppFeat attribute is sent with the 30th byte allotted for ImmReportPcc. Verify immRep is being sent as true. Verify the UDR profile being chosen to perform the UDR POST has ImmReportPcc enabled after on-demand/autonomous UDR discovery. For any additional guidance, contact My Oracle Support.

8.1.2.119 SMF_REQUESTED_SERVICE_NOT_AUTHORIZED_ABOVE_CRITICAL_THRESHOLD_PERCENT

Table 8-254 SMF_REQUESTED_SERVICE_NOT_AUTHORIZED_ABOVE_CRITICAL_THRESHOLD_PERCENT

Field	Details
Description	{{ $value }} % of patch requests failed in {{$labels.namespace}}.
Summary	This alert is triggered when the number of PATCH request that failed is equal to or above 60% in a given time period.
Severity	Critical
Expression	(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total{responseCode="403"}[5m])))/(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total[5m]))) * 100 >= 60
OID	1.3.6.1.4.1.323.5.3.52.1.2.120
Metric Used	occnp_pa_sponsored_sessions_total
Recommended Actions	If this alert gets triggered, Prometheus metrics or other tools can be used to check what error codes are being thrown and identify if the error comes from the NF being reached (in this case SM). Cause: Alerts are triggered when Sponsored Connectivity requests processed by PA-Service fail with a 403 Requested Service Not Authorized response. This occurs when the client sends a Sponsored request with umcDataIncluded="false" but the requested service is not authorized. As a result, PA rejects the request and increments the occnp_pa_sponsored_sessions_total metric. Diagnostic Information: Metric involved:occnp_pa_sponsored_sessions_total Error observed: 403 Requested Service Not Authorized Condition: Unauthorized Sponsored Connectivity requests Verification steps: Send a Sponsored Connectivity request with valid authorization and supported features. Confirm the request succeeds. Verify that the 403 / Requested Service Not Authorized ratio drops below the alert threshold within one evaluation window. Monitoring recommendations: Track the 4xx error ratio by caller/tenant and by sponsor/ASP. Pay special attention to spikes after client deployments or policy/configuration changes. Recovery: Identify the failing caller. Review the request payload, entitlement, and policy configuration. Confirm that the sponsor/ASP is authorized for the requested service. Correct any misconfigurations in policy rules or subscription data. Ensure that Sponsored Connectivity is supported in both the PCF SM Service and the PA Service. Escalate if the issue persists after authorization or policy fixes, or if it impacts multiple tenants or partners.

8.1.2.120 SMF_REQUESTED_SERVICE_NOT_AUTHORIZED_ABOVE_MAJOR_THRESHOLD_PERCENT

Table 8-255 SMF_REQUESTED_SERVICE_NOT_AUTHORIZED_ABOVE_MAJOR_THRESHOLD_PERCENT

Field	Details
Description	{{ $value }} % of patch requests failed in {{$labels.namespace}}.
Summary	This alert is triggered when the number of PATCH request that failed is equal to or above 40% in a given time period.
Severity	Major
Expression	(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total{responseCode="403"}[5m])))/(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total[5m]))) * 100 >= 40 < 60
OID	1.3.6.1.4.1.323.5.3.52.1.2.120
Metric Used	occnp_pa_sponsored_sessions_total
Recommended Actions	If this alert gets triggered, Prometheus metrics or other tools can be used to check what error codes are being thrown and identify if the error comes from the NF being reached (in this case SM). Cause: Alerts are triggered when Sponsored Connectivity requests processed by PA-Service fail with a 403 Requested Service Not Authorized response. This occurs when the client sends a Sponsored request with umcDataIncluded="false" but the requested service is not authorized. As a result, PA rejects the request and increments the occnp_pa_sponsored_sessions_total metric. Diagnostic Information: Metric involved:occnp_pa_sponsored_sessions_total Error observed: 403 Requested Service Not Authorized Condition: Unauthorized Sponsored Connectivity requests Verification steps: Send a Sponsored Connectivity request with valid authorization and supported features. Confirm the request succeeds. Verify that the 403 / Requested Service Not Authorized ratio drops below the alert threshold within one evaluation window. Monitoring recommendations: Track the 4xx error ratio by caller/tenant and by sponsor/ASP. Pay special attention to spikes after client deployments or policy/configuration changes. Recovery: Identify the failing caller. Review the request payload, entitlement, and policy configuration. Confirm that the sponsor/ASP is authorized for the requested service. Correct any misconfigurations in policy rules or subscription data. Ensure that Sponsored Connectivity is supported in both the PCF SM Service and the PA Service. Escalate if the issue persists after authorization or policy fixes, or if it impacts multiple tenants or partners.

8.1.2.121 SMF_REQUESTED_SERVICE_NOT_AUTHORIZED_ABOVE_MINOR_THRESHOLD_PERCENT

Table 8-256 SMF_REQUESTED_SERVICE_NOT_AUTHORIZED_ABOVE_MINOR_THRESHOLD_PERCENT

Field	Details
Description	{{ $value }} % of patch requests failed in {{$labels.namespace}}
Summary	This alert is triggered when the number of PATCH request that failed is equal to or above 20% in a given time period.
Severity	Minor
Expression	(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total{responseCode="403"}[5m])))/(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total[5m]))) * 100 >= 20 < 40
OID	1.3.6.1.4.1.323.5.3.52.1.2.120
Metric Used	occnp_pa_sponsored_sessions_total
Recommended Actions	If this alert gets triggered, Prometheus metrics or other tools can be used to check what error codes are being thrown and identify if the error comes from the NF being reached (in this case SM). Cause: Alerts are triggered when Sponsored Connectivity requests processed by PA-Service fail with a 403 Requested Service Not Authorized response. This occurs when the client sends a Sponsored request with umcDataIncluded="false", but the requested service is not authorized. As a result, PA rejects the request and increments the occnp_pa_sponsored_sessions_total metric. Diagnostic Information: Metric involved: occnp_pa_sponsored_sessions_total Error observed: 403 Requested Service Not Authorized Condition: Unauthorized Sponsored Connectivity requests Verification steps: Send a Sponsored Connectivity request with valid authorization and supported features. Confirm the request succeeds. Verify that the 403 / Requested Service Not Authorized ratio drops below the alert threshold within one evaluation window. Monitoring recommendations: Track the 4xx error ratio by caller/tenant and by sponsor/ASP. Pay special attention to spikes after client deployments or policy/configuration changes. Recovery: Identify the failing caller. Review the request payload, entitlement, and policy configuration. Confirm that the sponsor/ASP is authorized for the requested service. Correct any misconfigurations in policy rules or subscription data. Ensure that Sponsored Connectivity is supported in both the PCF SM Service and the PA Service. Escalate if the issue persists after authorization or policy fixes, or if it impacts multiple tenants or partners.

8.1.2.122 AF_MANDATORY_IE_MISSING_SC_ABOVE_CRITICAL_THRESHOLD_PERCENT

Table 8-257 AF_MANDATORY_IE_MISSING_SC_ABOVE_CRITICAL_THRESHOLD_PERCENT

Field	Details
Description	{{ $value }} % of patch requests failed in {{$labels.namespace}}.
Summary	This alert is triggered when the number of PATCH request that failed is equal to or above 60% in a given time period.
Severity	Critical
Expression	(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total{responseCode="400",cause="MANDATORY_IE_MISSING"}[5m])))/(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total[5m]))) * 100 >= 60
OID	1.3.6.1.4.1.323.5.3.52.1.2.122
Metric Used	occnp_pa_sponsored_sessions_total
Recommended Actions	If this alert gets triggered, Prometheus metrics or other tools can be used to check what error codes are being thrown and identify if the error comes from the NF being reached (in this case SM). Cause: Alerts are triggered when Sponsored Connectivity requests processed by PA-Service fail with a 400 Bad Request due to cause="MANDATORY_IE_MISSING". This happens when the client sends a Sponsored Connectivity request missing one or more mandatory Information Elements (IEs). As a result, PA rejects the request and increments the occnp_pa_sponsored_sessions_total metric. Diagnostic Information: Metric involved: occnp_pa_sponsored_sessions_total Error observed: 400 Bad Request Cause value: MANDATORY_IE_MISSING Condition: Sponsored Connectivity requests missing mandatory IE fields Common missing IEs: sponId, aspId, afAppId Verification steps: Send a valid Sponsored Connectivity request including all mandatory IEs. Ensure sponId and aspId are present and that Sponsored Connectivity is negotiated. Confirm the request succeeds. Verify that the 400 / MANDATORY_IE_MISSING ratio drops below the alert threshold within one evaluation window. Monitoring recommendations: Monitor the 4xx error ratio by caller/tenant and by sponsor/ASP. Watch for spikes following client deployments or gateway transformation changes. Recovery: Identify the failing caller. Compare the request payload against the API contract. Restore all mandatory IE fields (sponId, aspId, afAppId, etc.). Review and fix any gateway or payload transformation issues. Redeploy the corrected configuration or client. Escalate if the issue persists after fixes or impacts multiple tenants.

8.1.2.123 AF_MANDATORY_IE_MISSING_SC_ABOVE_MAJOR_THRESHOLD_PERCENT

Table 8-258 AF_MANDATORY_IE_MISSING_SC_ABOVE_MAJOR_THRESHOLD_PERCENT

Field	Details
Description	{{ $value }} % of patch requests failed in {{$labels.namespace}}.
Summary	This alert is triggered when the number of PATCH request that failed is equal to or above 40% in a given time period.
Severity	Major
Expression	(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total{responseCode="400",cause="MANDATORY_IE_MISSING"}[5m])))/(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total[5m]))) * 100 >= 40 < 60
OID	1.3.6.1.4.1.323.5.3.52.1.2.122
Metric Used	occnp_pa_sponsored_sessions_total
Recommended Actions	If this alert gets triggered, Prometheus metrics or other tools can be used to check what error codes are being thrown and identify if the error comes from the NF being reached (in this case SM). Cause: Alerts are triggered when Sponsored Connectivity requests processed by PA-Service fail with a 400 Bad Request due to cause="MANDATORY_IE_MISSING". This happens when the client sends a Sponsored Connectivity request missing one or more mandatory Information Elements (IEs). As a result, PA rejects the request and increments the occnp_pa_sponsored_sessions_total metric. Diagnostic Information: Metric involved: occnp_pa_sponsored_sessions_total Error observed: 400 Bad Request Cause value: MANDATORY_IE_MISSING Condition: Sponsored Connectivity requests missing mandatory IE fields Common missing IEs: sponId, aspId, afAppId Verification steps: Send a valid Sponsored Connectivity request including all mandatory IEs. Ensure sponId and aspId are present and that Sponsored Connectivity is negotiated. Confirm the request succeeds. Verify that the 400 / MANDATORY_IE_MISSING ratio drops below the alert threshold within one evaluation window. Monitoring recommendations: Monitor the 4xx error ratio by caller/tenant and by sponsor/ASP. Watch for spikes following client deployments or gateway transformation changes. Recovery: Identify the failing caller. Compare the request payload against the API contract. Restore all mandatory IE fields (sponId, aspId, afAppId, etc.). Review and fix any gateway or payload transformation issues. Redeploy the corrected configuration or client. Escalate if the issue persists after fixes or impacts multiple tenants.

8.1.2.124 AF_MANDATORY_IE_MISSING_SC_ABOVE_MINOR_THRESHOLD_PERCENT

Table 8-259 AF_MANDATORY_IE_MISSING_SC_ABOVE_MINOR_THRESHOLD_PERCENT

Field	Details
Description	{{ $value }} % of patch requests failed in {{$labels.namespace}}.
Summary	This alert is triggered when the number of PATCH request that failed is equal to or above 20% in a given time period.
Severity	Minor
Expression	(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total{responseCode="400",cause="MANDATORY_IE_MISSING"}[5m])))/(sum by (namespace) (rate(occnp_pa_sponsored_sessions_total[5m]))) * 100 >= 20 < 40
OID	1.3.6.1.4.1.323.5.3.52.1.2.122
Metric Used	occnp_pa_sponsored_sessions_total
Recommended Actions	If this alert is triggered, Prometheus metrics or other tools can be used to check what error codes are being thrown and identify if the error comes from the NF being reached (in this case SM). Cause: Alerts are triggered when Sponsored Connectivity requests processed by PA-Service fail with a 400 Bad Request due to cause="MANDATORY_IE_MISSING". This happens when the client sends a Sponsored Connectivity request missing one or more mandatory Information Elements (IEs). As a result, PA rejects the request and increments the occnp_pa_sponsored_sessions_total metric. Diagnostic Information: Metric involved: occnp_pa_sponsored_sessions_total Error observed: 400 Bad Request Cause value: MANDATORY_IE_MISSING Condition: Sponsored Connectivity requests missing mandatory IE fields Common missing IEs: sponId, aspId, afAppId Verification steps: Send a valid Sponsored Connectivity request including all mandatory IEs. Ensure sponId and aspId are present and that Sponsored Connectivity is negotiated. Confirm the request succeeds. Verify that the 400 / MANDATORY_IE_MISSING ratio drops below the alert threshold within one evaluation window. Monitoring recommendations: Monitor the 4xx error ratio by caller/tenant and by sponsor/ASP. Watch for spikes following client deployments or gateway transformation changes. Recovery: Identify the failing caller. Compare the request payload against the API contract. Restore all mandatory IE fields (sponId, aspId, afAppId, etc.). Review and fix any gateway or payload transformation issues. Redeploy the corrected configuration or client. Escalate if the issue persists after fixes or impacts multiple tenants.

8.1.2.125 UE_N1N2_TRANSFER_RETRYAFTER_FAILURE_RATE_ABOVE_MINOR_THRESHOLD

Table 8-260 UE_N1N2_TRANSFER_RETRYAFTER_FAILURE_RATE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	At least 30% (but less than 50%) of N1N2Transfer requests have failed with a "retryAfter IE" timer within the last 5 mins
Summary	At least 30% (but less than 50%) of N1N2Transfer requests have failed with a "retryAfter IE" timer within the last 5 mins
Severity	Minor
Expression	(sum by (namespace) (increase(occnp_n1n2transfer_error_with_retryafter_seconds_count{microservice=~".*pcf_ueservice"}[5m]))) / (sum by (namespace) (increase(http_out_conn_response_total{operationType="transfer",microservice=~".*pcf_ueservice",responseCode!="2xx"}[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.135
Metric Used	occnp_n1n2transfer_error_with_retryafter_seconds_count
Recommended Actions	Cause: This alert gets triggered when the percentage of N1N2Transfer failures containing a retryAfter exceed defined thresholds within a 5-minute window. When occnp_n1n2transfer_error_with_retryafter_seconds_count increases, it indicates: UE is not reachable. AMF is instructing PCF to retry after a specified interval. Transfer is delayed, not permanently failed. High percentage indicates increasing UE unreachability or signaling instability between AMF and UE. Diagnostics Information: Check retryAfter failure percentage trend. Filter by response_code, amf_id, mcc, mnc. Review retryAfter histogram buckets (0–600s, 600–3600s, >3600s). Correlate with UE health and signaling load. Recovery: Validate if spike is transient. Investigate UE reach ability issues. Escalate if sustained at Major or Critical level.

Note:

• UE_N1N2_TRANSFER_RETRYAFTER_FAILURE_RATE_* alerts mean AMF is returning retryAfter IE values for a high percentage of N1N2Transfer requests. This usually means AMF cannot currently reach the UE and requests PCF to wait before retrying.
• UE_N1N2_TRANSFER_INVALID_RETRYAFTER_VALUE_RATE_* alerts mean AMF returned a retryAfter IE value that is <=0 or greater than 600 seconds.
• For a value <=0, PCF uses normal session retry behavior.
• For a value >600, PCF caps the value at 600 seconds.
• The following must be verified manually:
  • Enable Retry After IE is enabled in the Retry Profile.
  • RETRY.AFTER.IE.MAX.SECONDS is between 1 and 600; default 600.
  • RETRY.AFTER.IE.TIMER.CAPACITY is appropriate; default 50,000.
  • RETRY.AFTER.IE.TIMER.RATE is appropriate; default 50 per second.
  • UE Policy service is registered with Timer Service.
  • The affected AMF and UE are reachable.
  • Alert dimensions such as AMF ID, MCC, MNC, and response code identify the affected AMF.
• The feature supports retryAfter IE in the N1N2 response payload only. It does not support the HTTP Retry-After header.

8.1.2.126 UE_N1N2_TRANSFER_RETRYAFTER_FAILURE_RATE_ABOVE_MAJOR_THRESHOLD

Table 8-261 UE_N1N2_TRANSFER_RETRYAFTER_FAILURE_RATE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	At least 50% (but less than 70%) of N1N2Transfer requests have failed with a "retryAfter IE" timer within the last 5 mins
Summary	At least 50% (but less than 70%) of N1N2Transfer requests have failed with a "retryAfter IE" timer within the last 5 mins
Severity	Major
Expression	(sum by (namespace) (increase(occnp_n1n2transfer_error_with_retryafter_seconds_count{microservice=~".*pcf_ueservice"}[5m]))) / (sum by (namespace) (increase(http_out_conn_response_total{operationType="transfer",microservice=~".*pcf_ueservice",responseCode!="2xx"}[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.135
Metric Used	occnp_n1n2transfer_error_with_retryafter_seconds_count
Recommended Actions	Cause: This alert gets triggered when the percentage of N1N2Transfer failures containing a retryAfter exceed defined thresholds within a 5-minute window. When occnp_n1n2transfer_error_with_retryafter_seconds_count increases, it indicates: UE is not reachable. AMF is instructing PCF to retry after a specified interval. Transfer is delayed, not permanently failed. High percentage indicates increasing UE unreachability or signaling instability between AMF and UE. Diagnostics Information: Check retryAfter failure percentage trend. Filter by response_code, amf_id, mcc, mnc. Review retryAfter histogram buckets (0–600s, 600–3600s, >3600s). Correlate with UE health and signaling load. Recovery: Validate if spike is transient. Investigate UE reach ability issues. Escalate if sustained at Major or Critical level.

8.1.2.127 UE_N1N2_TRANSFER_RETRYAFTER_FAILURE_RATE_ABOVE_CRITICAL_THRESHOLD

Table 8-262 UE_N1N2_TRANSFER_RETRYAFTER_FAILURE_RATE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	At least 70% of N1N2Transfer requests have failed with a "retryAfter IE" timer within the last 5 mins
Summary	At least 70% of N1N2Transfer requests have failed with a "retryAfter IE" timer within the last 5 mins
Severity	Critical
Expression	(sum by (namespace) (increase(occnp_n1n2transfer_error_with_retryafter_seconds_count{microservice=~".*pcf_ueservice"}[5m]))) / (sum by (namespace) (increase(http_out_conn_response_total{operationType="transfer",microservice=~".*pcf_ueservice",responseCode!="2xx"}[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.135
Metric Used	occnp_n1n2transfer_error_with_retryafter_seconds_count
Recommended Actions	Cause: This alert gets triggered when the percentage of N1N2Transfer failures containing a retryAfter exceed defined thresholds within a 5-minute window. When occnp_n1n2transfer_error_with_retryafter_seconds_count increases, it indicates: UE is not reachable. AMF is instructing PCF to retry after a specified interval. Transfer is delayed, not permanently failed. High percentage indicates increasing UE unreachability or signaling instability between AMF and UE. Diagnostics Information: Check retryAfter failure percentage trend. Filter by response_code, amf_id, mcc, mnc. Review retryAfter histogram buckets (0–600s, 600–3600s, >3600s). Correlate with UE health and signaling load. Recovery: Validate if spike is transient. Investigate UE reach ability issues. Escalate if sustained at Major or Critical level.

8.1.2.128 UE_N1N2_TRANSFER_INVALID_RETRYAFTER_VALUE_RATE_ABOVE_MINOR_THRESHOLD

Table 8-263 UE_N1N2_TRANSFER_INVALID_RETRYAFTER_VALUE_RATE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	At least 30% (but less than 50%) of N1N2Transfer error responses in the last 5m have an invalid retryAfter IE value (greater than the allowed maximum).
Summary	At least 30% (but less than 50%) of N1N2Transfer error responses in the last 5m have an invalid retryAfter IE value (greater than the allowed maximum).
Severity	Minor
Expression	(sum by (namespace) (increase(occnp_n1n2transfer_error_retryafter_invalid_total{retry_after_gt_config_max="true",microservice=~".*pcf_ueservice"}[5m]))) / (sum by (namespace) (increase(occnp_n1n2transfer_error_with_retryafter_seconds_count{microservice=~".*pcf_ueservice"}[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.52.1.2.136
Metric Used	occnp_n1n2transfer_error_retryafter_invalid_total occnp_n1n2transfer_error_with_retryafter_seconds_count
Recommended Actions	Cause: Triggered when the percentage of N1N2Transfer error responses containing an invalid retryAfter value exceeds defined thresholds within a 5-minute window. Invalid means value greater than configured maximum (Advanced Settings). When occnp_n1n2transfer_error_retryafter_invalid_total increases, it indicates: AMF is sending invalid retryAfter values. If retryAfter ≤ 0, PCF falls back to normal session retry logic. If retryAfter > configured max, PCF caps it to the configured maximum value – RETRY.AFTER.IE.MAX.SECONDS (This is the advance setting key used to set retryAfterIE max) Service remains protected by fallback logic, but high percentage indicates signaling or configuration inconsistency. Diagnostics Information: Check retryAfter failure percentage trend. Filter by response_code, amf_id, mcc, mnc. Review retry_after_le_zero and retry_after_gt_config_max dimensions. Verify configured maximum retryAfter value. Identify if issue is isolated to a specific AMF. Recovery: Validate Advanced Settings configuration. Check AMF configuration and behavior as why invalid values are sent. Investigate configuration mismatch between PCF and AMF. Escalate if sustained at Major or Critical level.

8.1.2.129 UE_N1N2_TRANSFER_INVALID_RETRYAFTER_VALUE_RATE_ABOVE_MAJOR_THRESHOLD

Table 8-264 UE_N1N2_TRANSFER_INVALID_RETRYAFTER_VALUE_RATE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	At least 50% (but less than 70%) of N1N2Transfer error responses in the last 5m have an invalid retryAfter IE value (greater than the allowed maximum).
Summary	At least 50% (but less than 70%) of N1N2Transfer error responses in the last 5m have an invalid retryAfter IE value (greater than the allowed maximum).
Severity	Major
Expression	(sum by (namespace) (increase(occnp_n1n2transfer_error_retryafter_invalid_total{retry_after_gt_config_max="true",microservice=~".*pcf_ueservice"}[5m]))) / (sum by (namespace) (increase(occnp_n1n2transfer_error_with_retryafter_seconds_count{microservice=~".*pcf_ueservice"}[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.136
Metric Used	occnp_n1n2transfer_error_retryafter_invalid_total occnp_n1n2transfer_error_with_retryafter_seconds_count
Recommended Actions	Cause: Triggered when the percentage of N1N2Transfer error responses containing an invalid retryAfter value exceeds defined thresholds within a 5-minute window. Invalid means value greater than configured maximum (Advanced Settings). When occnp_n1n2transfer_error_retryafter_invalid_total increases, it indicates: AMF is sending invalid retryAfter values. If retryAfter ≤ 0, PCF falls back to normal session retry logic. If retryAfter > configured max, PCF caps it to the configured maximum value – RETRY.AFTER.IE.MAX.SECONDS (This is the advance setting key used to set retryAfterIE max) Service remains protected by fallback logic, but high percentage indicates signaling or configuration inconsistency. Diagnostics Information: Check retryAfter failure percentage trend. Filter by response_code, amf_id, mcc, mnc. Review retry_after_le_zero and retry_after_gt_config_max dimensions. Verify configured maximum retryAfter value. Identify if issue is isolated to a specific AMF. Recovery: Validate Advanced Settings configuration. Check AMF configuration and behavior as why invalid values are sent. Investigate configuration mismatch between PCF and AMF. Escalate if sustained at Major or Critical level.

8.1.2.130 UE_N1N2_TRANSFER_INVALID_RETRYAFTER_VALUE_RATE_ABOVE_CRITICAL_THRESHOLD

Table 8-265 UE_N1N2_TRANSFER_INVALID_RETRYAFTER_VALUE_RATE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	At least 70% of N1N2Transfer error responses in the last 5m have an invalid retryAfter IE value (greater than the allowed maximum).
Summary	At least 70% of N1N2Transfer error responses in the last 5m have an invalid retryAfter IE value (greater than the allowed maximum).
Severity	Critical
Expression	(sum by (namespace) (increase(occnp_n1n2transfer_error_retryafter_invalid_total{retry_after_gt_config_max="true",microservice=~".*pcf_ueservice"}[5m]))) / (sum by (namespace) (increase(occnp_n1n2transfer_error_with_retryafter_seconds_count{microservice=~".*pcf_ueservice"}[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.52.1.2.136
Metric Used	occnp_n1n2transfer_error_retryafter_invalid_total occnp_n1n2transfer_error_with_retryafter_seconds_count
Recommended Actions	Cause: Triggered when the percentage of N1N2Transfer error responses containing an invalid retryAfter value exceeds defined thresholds within a 5-minute window. Invalid means value greater than configured maximum (Advanced Settings). When occnp_n1n2transfer_error_retryafter_invalid_total increases, it indicates: AMF is sending invalid retryAfter values. If retryAfter ≤ 0, PCF falls back to normal session retry logic. If retryAfter > configured max, PCF caps it to the configured maximum value – RETRY.AFTER.IE.MAX.SECONDS (This is the advance setting key used to set retryAfterIE max) Service remains protected by fallback logic, but high percentage indicates signaling or configuration inconsistency. Diagnostics Information: Check retryAfter failure percentage trend. Filter by response_code, amf_id, mcc, mnc. Review retry_after_le_zero and retry_after_gt_config_max dimensions. Verify configured maximum retryAfter value. Identify if issue is isolated to a specific AMF. Recovery: Validate Advanced Settings configuration. Check AMF configuration and behavior as why invalid values are sent. Investigate configuration mismatch between PCF and AMF. Escalate if sustained at Major or Critical level.

8.1.2.131 IGW_POD_PROTECTION_CONGESTED_STATE

Table 8-266 IGW_POD_PROTECTION_CONGESTED_STATE

Field	Details
Name	IGW_POD_PROTECTION_CONGESTED_STATE
Description	The Ingress Gateway is in Congested Level for the pod {{$labels.pod}} in namespace {{$labels.namespace}} ( current congestion level: {{ $value }} % )
Summary	Ingress Gateway pod congestion state in Congested level.
Severity	critical
Expression	oc_ingressgateway_congestion_system_state{microservice=~".*ingress-gateway"} == 4
OID	1.3.6.1.4.1.323.5.3.52.1.2.123
Metric Used	oc_ingressgateway_congestion_system_state
Recommended Actions	Cause: Alert get triggered when the pod cpu consumption raises above the configured onset value for the CONGESTED level. Diagnostic Information: Metric involved: oc_ingressgateway_congestion_system_state Error observed: 429 Too Many Requests, NF_CONGESTION_RISK Cause value: congestionConfig.levels.resources.onset value for CONGESTED level Condition: podProtectionByRLConfigMode.enabled = true and congestionConfig.levels.value=4 and congestionConfig.levels.name=CONGESTED Verification steps: Set congestionConfig.levels.resources.onset to a lower value for CONGESTED level configuration; Run 4500 TPS or above for SM traffic; Confirm some request dropped with Error 429. Verify that the alert get triggered. Monitoring recommendations: Monitor 4xx error; and CPU consumption for the pod in dashboard Watch for spikes following client deployments. Recovery: Check Network traffic burst and storm Investigate traffic load balancer issues and network issues. Review SM Service Resources Restart or scale up resources temporarily if the system is congested Disable feature if this flow is the only one affected we can disable this feature as a last resource.

8.1.2.132 IGW_POD_PROTECTION_DOC_STATE

Table 8-267 IGW_POD_PROTECTION_DOC_STATE

Field	Details
Name	IGW_POD_PROTECTION_DOC_STATE
Description	The Ingress Gateway is in Danger_of_Congestion Level for the pod {{$labels.pod}} in namespace {{$labels.namespace}} ( current congestion level: {{ $value }} % )
Summary	Ingress Gateway pod congestion state in Danger_of_Congestion Level.
Severity	Minor
Expression	oc_ingressgateway_congestion_system_state{microservice=~".*ingress-gateway"} == 1
OID	1.3.6.1.4.1.323.5.3.52.1.2.123
Metric Used	oc_ingressgateway_congestion_system_state
Recommended Actions	Cause: Alert get triggered when the pod cpu consumption raises above the configured onset value for the DANGER_OF_CONGESTION level. Diagnostic Information: Metric involved: oc_ingressgateway_congestion_system_state Error observed: 429 Too Many Requests, NF_CONGESTION_RISK Cause value: congestionConfig.levels.resources.onset value for DANGER_OF_CONGESTION level Condition: podProtectionByRLConfigMode.enabled = true and congestionConfig.levels.value=1 and congestionConfig.levels.name=DANGER_OF_CONGESTION Verification steps: Set congestionConfig.levels.resources.onset to a lower value for DANGER_OF_CONGESTION level configuration; Run 4500 TPS or above for SM traffic; Confirm some request dropped with Error 429. Verify that the alert get triggered. Monitoring recommendations: Monitor 4xx error; and CPU consumption for the pod in dashboard Watch for spikes following client deployments. Recovery: Check Network traffic burst and storm Investigate traffic load balancer issues and network issues. Review SM Service Resources Restart or scale up resources temporarily if the system is congested Disable feature if this flow is the only one affected we can disable this feature as a last resource.

8.1.2.133 EGRESS_GATEWAY_DD_UNREACHABLE_MAJOR

Table 8-268 EGRESS_GATEWAY_DD_UNREACHABLE_MAJOR

Field	Details
Name	EGRESS_GATEWAY_DD_UNREACHABLE_MAJOR
Description	Policy Egress Gateway Data Director unreachable for {{$labels.namespace}}.
Summary	kubernetes_namespace: {{$labels.kubernetes_namespace}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} BSF Egress Gateway Data Director unreachable
Severity	Major
Expression	sum(oc_egressgateway_dd_unreachable) by(namespace,container) > 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.84
Metric Used	oc_egressgateway_dd_unreachable
Recommended Actions	Cause: Data Director for Egress Gateway is unreachable Diagnostic Information: Namespace health: Verify that all pods are healthy and have enough CPU / Memory for the expected traffic. Logs: Check EGW's Logs for more information on why it is unreachable. IP's: Check that EGW's configured IP's are the correct, and Data Director is exposed for connections Recovery: As soon as EGW connects to is Data Director the Alert will turn off by itself.

8.1.2.134 INGRESS_GATEWAY_DD_UNREACHABLE_MAJOR

Table 8-269 INGRESS_GATEWAY_DD_UNREACHABLE_MAJOR

Field	Details
Name	INGRESS_GATEWAY_DD_UNREACHABLE_MAJOR
Description	Policy Ingress Gateway Data Director unreachable for {{$labels.namespace}}.
Summary	'kubernetes_namespace: {{$labels.kubernetes_namespace}}, timestamp: {{ with query "time()" }}{{ . | first | value | humanizeTimestamp }}{{ end }} BSF Ingress Gateway Data Director unreachable'
Severity	major
Expression	sum(oc_ingressgateway_dd_unreachable) by(namespace,container) > 0
OID	1.3.6.1.4.1.323.5.3.52.1.2.83
Metric Used	oc_ingressgateway_dd_unreachable
Recommended Actions	Cause: Data Director for Ingress Gateway is unreachable Diagnostic Information: Namespace health: Verify that all pods are healthy and have enough CPU / Memory for the expected traffic. Logs: Check IGW's Logs for more information on why it is unreachable. IP's: Check that IGW's configured IP's are the correct, and Data Director is exposed for connections Recovery: As soon as IGW connects to is Data Director the Alert will turn off by itself.

8.1.3 PCRF Alerts

This section provides information about PCRF alerts.

8.1.3.1 PRE_UNREACHABLE_EXCEEDS_CRITICAL_THRESHOLD

Table 8-270 PRE_UNREACHABLE_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	PRE fail count exceeds the critical threshold limit.
Summary	Alert PRE unreachable NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	PRE fail count exceeds the critical threshold limit.
OID	1.3.6.1.4.1.323.5.3.44.1.2.9
Metric Used	http_out_conn_response_total{container="pcrf-core", responseCode!~"2.*", serviceResource="PRE"}
Recommended Actions	Cause: PRE_UNREACHABLE_EXCEEDS_CRITICAL_THRESHOLD is raised when the PRE service becomes unreachable beyond the configured critical threshold. This usually indicates severe service unavailability caused by pod failures, service crashes, network interruptions, node issues, or deployment failures. It may also occur when the service is failing health checks, has backend connectivity problems, or is under extreme resource pressure. Diagnostic Information: Check the PRE pod status, restarts, and readiness/liveness probe failures. Verify the service is running using kubectl get service -n <namespace>. Review PRE service logs from Kibana to identify crashes, startup failures, or connectivity issues.Check whether there is a sudden surge in traffic or retry behavior from upstream services causing overload. Correlate the alert with recent deployments, configuration changes, or node-level incidents. Verify whether related dependent services are healthy and reachable. Recovery: Restart the affected PRE pods or perform a rolling restart if the failure is pod-specific. Rollback recent changes if the issue started after a deployment or configuration update. Fix the underlying cause such as resource exhaustion, probe failure, network issue, or dependency outage. If the service does not recover quickly, escalate to Oracle Support for further investigation.

8.1.3.2 PRE_UNREACHABLE_EXCEEDS_MAJOR_THRESHOLD

Table 8-271 PRE_UNREACHABLE_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	PRE fail count exceeds the major threshold limit.
Summary	Alert PRE unreachable NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	PRE fail count exceeds the major threshold limit.
OID	1.3.6.1.4.1.323.5.3.44.1.2.9
Metric Used	http_out_conn_response_total{container="pcrf-core", responseCode!~"2.*", serviceResource="PRE"}
Recommended Actions	Cause: PRE_UNREACHABLE_EXCEEDS_MAJOR_THRESHOLD is raised when the PRE service becomes unreachable beyond the configured threshold (between 80% to 90%). This usually indicates severe service unavailability caused by pod failures, service crashes, network interruptions, node issues, or deployment failures. It may also occur when the service is failing health checks, has backend connectivity problems, or is under extreme resource pressure. Diagnostic Information: Check the PRE pod status, restarts, and readiness/liveness probe failures. Verify the service is running using kubectl get service -n <namespace>. Review PRE service logs from Kibana to identify crashes, startup failures, or connectivity issues.Check whether there is a sudden surge in traffic or retry behavior from upstream services causing overload. Correlate the alert with recent deployments, configuration changes, or node-level incidents. Verify whether related dependent services are healthy and reachable. Recovery: Restart the affected PRE pods or perform a rolling restart if the failure is pod-specific. Rollback recent changes if the issue started after a deployment or configuration update. Fix the underlying cause such as resource exhaustion, probe failure, network issue, or dependency outage. If the service does not recover quickly, escalate to Oracle Support for further investigation.

8.1.3.3 PRE_UNREACHABLE_EXCEEDS_MINOR_THRESHOLD

Table 8-272 PRE_UNREACHABLE_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	PRE fail count exceeds the minor threshold limit.
Summary	Alert PRE unreachable NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	minor
Expression	sum(rate(http_out_conn_response_total{container="pcrf-core", responseCode!~"2.*", serviceResource="PRE"}[5m])) / sum(rate(http_out_conn_response_total{container="pcrf-core",serviceResource="PRE"}[5m])) * 100 > 60 and sum(rate(http_out_conn_response_total{container="pcrf-core", responseCode!~"2.*", serviceResource="PRE"}[5m])) / sum(rate(http_out_conn_response_total{container="pcrf-core",serviceResource="PRE"}[5m])) * 100 <=80
OID	1.3.6.1.4.1.323.5.3.44.1.2.9
Metric Used	http_out_conn_response_total{container="pcrf-core", responseCode!~"2.*", serviceResource="PRE"}
Recommended Actions	Cause: PRE_UNREACHABLE_EXCEEDS_MINOR_THRESHOLD is raised when the PRE service becomes unreachable beyond the configured threshold (between 60% to 80%). This usually indicates severe service unavailability caused by pod failures, service crashes, network interruptions, node issues, or deployment failures. It may also occur when the service is failing health checks, has backend connectivity problems, or is under extreme resource pressure. Diagnostic Information: Check the PRE pod status, restarts, and readiness/liveness probe failures. Verify the service is running using kubectl get service -n <namespace>. Review PRE service logs from Kibana to identify crashes, startup failures, or connectivity issues.Check whether there is a sudden surge in traffic or retry behavior from upstream services causing overload. Correlate the alert with recent deployments, configuration changes, or node-level incidents. Verify whether related dependent services are healthy and reachable. Recovery: Restart the affected PRE pods or perform a rolling restart if the failure is pod-specific. Rollback recent changes if the issue started after a deployment or configuration update. Fix the underlying cause such as resource exhaustion, probe failure, network issue, or dependency outage. If the service does not recover quickly, escalate to Oracle Support for further investigation.

8.1.3.4 PCRF_DOWN

Table 8-273 PCRF_DOWN

Field	Details
Description	PCRF Service is down
Summary	Alert PCRF_DOWN NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	appinfo_service_running{service=~".*pcrf-core"} == 0
OID	1.3.6.1.4.1.323.5.3.44.1.2.33
Metric Used	appinfo_service_running{service=~".*pcrf-core"}
Recommended Actions	Cause: PCRF_DOWN is raised when the PCRF service becomes unavailable or stops responding. This can be caused by pod failures, service restarts, node issues, network interruptions, deployment problems, or upstream dependency failures. It may also occur when the service is unhealthy due to resource pressure, configuration issues, or backend connectivity problems. Diagnostic Information: Check PCRF pod status, restarts, and readiness/liveness probe failures. Verify the running status of the service using kubectl get service -n <namespace>. Review PCRF service logs from Kibana to identify crashes, startup failures, or connection issues. Check Grafana for CPU, memory, and network trends to see whether the service was under stress before going down. Correlate the alert with recent deployments, config changes, or node-level incidents. Verify whether related dependent services are healthy and reachable. Recovery: Restart the affected PCRF pods or perform a rolling restart if the failure is pod-specific. Rollback recent changes if the issue started after a deployment or configuration update. Fix the underlying cause such as resource exhaustion, probe failure, or dependency outage. If the service does not recover cleanly, escalate to Oracle Support for further investigation.

8.1.3.5 CCA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-274 CCA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	CCA fail count exceeds the critical threshold limit
Summary	Alert CCA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{msgType=~"CCA.*", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType=~"CCA.*"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.44.1.2.13
Metric Used	occnp_diam_response_local_total{msgType=~"CCA.*", responseCode!~"2.*"}
Recommended Actions	Cause: CCA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD is raised when the number of failed Credit Control Answer (CCA) transactions exceeds the major threshold. This indicates a significant failure rate in credit control responses, typically caused by downstream charging system issues, connectivity problems, or invalid request handling. It may also result from timeout scenarios, misconfigurations, or increased load impacting CCR/CCA processing. Diagnostic Information: Check CCA failure count trends to determine whether the increase is sustained or a sudden spike. Analyze Diameter traffic metrics to identify failure response codes and patterns. Check the policy that is getting evaluated upon receiving CCR-I to identify any misconfigurations or evaluation failures. Correlate with upstream/downstream systems such as CHF/OCS to verify their health and responsiveness. Review logs in Kibana for CCR/CCA transactions to identify timeout errors, rejection causes, or connectivity issues. Check recent deployments or configuration changes impacting Diameter flows. Recovery: Fix underlying issues with downstream systems (e.g., CHF/OCS, UDR, LDAP availability, latency, or misconfiguration). Fix the policy in case the issue is caused by incorrect policy evaluation. Validate request formats and ensure proper handling of CCR/CCA messages. If failures are due to timeouts or load, consider scaling or tuning timeout/retry configurations. Rollback recent changes if the issue started after a deployment. The alert will clear once the CCA failure count drops below the major threshold.

8.1.3.6 CCA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-275 CCA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	CCA fail count exceeds the major threshold limit
Summary	Alert CCA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{msgType=~"CCA.*", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType=~"CCA.*"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType=~"CCA.*", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType=~"CCA.*"}[5m])) * 100 <=90
OID	1.3.6.1.4.1.323.5.3.44.1.2.13
Metric Used	occnp_diam_response_local_total{msgType=~"CCA.*", responseCode!~"2.*"}
Recommended Actions	Cause: CCA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD is raised when the number of failed Credit Control Answer (CCA) transactions exceeds the major threshold. This indicates a significant failure rate in credit control responses, typically caused by downstream charging system issues, connectivity problems, or invalid request handling. It may also result from timeout scenarios, misconfigurations, or increased load impacting CCR/CCA processing. Diagnostic Information: Check CCA failure count trends to determine whether the increase is sustained or a sudden spike. Analyze Diameter traffic metrics to identify failure response codes and patterns. Check the policy that is getting evaluated upon receiving CCR-I to identify any misconfigurations or evaluation failures. Correlate with upstream/downstream systems such as CHF/OCS to verify their health and responsiveness. Review logs in Kibana for CCR/CCA transactions to identify timeout errors, rejection causes, or connectivity issues. Check recent deployments or configuration changes impacting Diameter flows. Recovery: Fix underlying issues with downstream systems (e.g., CHF/OCS, UDR, LDAP availability, latency, or misconfiguration). Fix the policy in case the issue is caused by incorrect policy evaluation. Validate request formats and ensure proper handling of CCR/CCA messages. If failures are due to timeouts or load, consider scaling or tuning timeout/retry configurations. Rollback recent changes if the issue started after a deployment. The alert will clear once the CCA failure count drops below the major threshold.

8.1.3.7 CCA_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-276 CCA_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	CCA fail count exceeds the minor threshold limit
Summary	Alert CCA_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{msgType=~"CCA.*", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType=~"CCA.*"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType=~"CCA.*", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType=~"CCA.*"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.44.1.2.13
Metric Used	occnp_diam_response_local_total{msgType=~"CCA.*", responseCode!~"2.*"}
Recommended Actions	Cause: CCA_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD is raised when the number of failed Credit Control Answer (CCA) transactions exceeds the major threshold. This indicates a significant failure rate in credit control responses, typically caused by downstream charging system issues, connectivity problems, or invalid request handling. It may also result from timeout scenarios, misconfigurations, or increased load impacting CCR/CCA processing. Diagnostic Information: Check CCA failure count trends to determine whether the increase is sustained or a sudden spike. Analyze Diameter traffic metrics to identify failure response codes and patterns. Check the policy that is getting evaluated upon receiving CCR-I to identify any misconfigurations or evaluation failures. Correlate with upstream/downstream systems such as CHF/OCS to verify their health and responsiveness. Review logs in Kibana for CCR/CCA transactions to identify timeout errors, rejection causes, or connectivity issues. Check recent deployments or configuration changes impacting Diameter flows. Recovery: Fix underlying issues with downstream systems (e.g., CHF/OCS, UDR, LDAP availability, latency, or misconfiguration). Fix the policy in case the issue is caused by incorrect policy evaluation. Validate request formats and ensure proper handling of CCR/CCA messages. If failures are due to timeouts or load, consider scaling or tuning timeout/retry configurations. Rollback recent changes if the issue started after a deployment. The alert will clear once the CCA failure count drops below the major threshold.

8.1.3.8 AAA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-277 AAA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	AAA fail count exceeds the critical threshold limit
Summary	Alert AAA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{msgType=~"AAA.*", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType=~"AAA.*"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.52.1.2.34
Metric Used	occnp_diam_response_local_total{msgType=~"AAA.*", responseCode!~"2.*"}
Recommended Actions	Cause:The percentage of failed AAA (Authentication, Authorization, and Accounting) Diameter responses has exceeded the critical threshold (greater than 90%). This indicates that the majority of AAA requests are being rejected or encountering errors (non-2xx response codes), which may be due to backend system failures, misconfiguration, connectivity issues, or overload conditions. --- Diagnostic Information: Metric evaluates: Failed AAA Responses / Total AAA Responses * 100 over a 5-minute window Failure is identified by Diameter response codes not starting with "2" (i.e., non-success responses). Trigger condition: Failure rate > 90% Key checks: Inspect response codes (e.g., 4xxx, 5xxx) for error patterns Verify connectivity to downstream systems (e.g., HSS/UDM, AAA servers) Check recent configuration changes or deployments Review system logs for spikes in errors or timeouts Monitor load, latency, and resource utilization (CPU, memory, thread pools) Validate if specific msgTypes (subset of AAA) are disproportionately failing --- Recovery: Restore connectivity to dependent systems (AAA backend, subscriber databases) Roll back recent configuration or deployment changes if correlated Restart or scale affected services to handle load Fix misconfigurations related to routing, policies, or credentials Address specific error codes (e.g., authentication failure vs system error) Implement traffic throttling or circuit breaking if overload is detected Continue monitoring until failure rate drops below threshold If the issue still persist please contact the support.

8.1.3.9 AAA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-278 AAA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	AAA fail count exceeds the major threshold limit
Summary	Alert AAA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{msgType=~"AAA.*", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType=~"AAA.*"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType=~"AAA.*", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType=~"AAA.*"}[5m])) * 100 <=90
OID	1.3.6.1.4.1.323.5.3.52.1.2.34
Metric Used	occnp_diam_response_local_total{msgType=~"AAA.*", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed AAA (Authentication, Authorization, and Accounting) Diameter responses has exceeded the major threshold (greater than 80%). This indicates a high rate of unsuccessful AAA transactions, though not yet at critical levels. It may signal emerging issues such as partial service degradation, backend instability, configuration inconsistencies, or increasing load impacting successful processing. Diagnostic Information: Metric evaluates: Failed AAA Responses / Total AAA Responses * 100 over a 5-minute window Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 80% Key checks: Analyze distribution of response codes (e.g., 3xxx, 4xxx, 5xxx) Identify whether failures are intermittent or steadily increasing Check health and responsiveness of dependent systems (HSS/UDM, AAA servers) Review recent configuration changes, deployments, or scaling events Examine logs for timeout errors, connection failures, or policy rejections Monitor system performance metrics (CPU, memory, queue depth, latency) Determine if specific AAA message types are more affected Recovery: Investigate and stabilize backend dependencies (e.g., subscriber database, AAA services) Roll back or correct recent configuration or deployment changes if applicable Scale system resources or optimize performance to handle load Resolve specific failure causes based on response codes Apply traffic management strategies (rate limiting, retries, failover) Perform targeted restarts of impacted components if necessary Closely monitor trends to prevent escalation to critical threshold If the issue still persist, Please contact the Support.

8.1.3.10 AAA_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-279 AAA_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	AAA fail count exceeds the minor threshold limit
Summary	Alert AAA_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{msgType=~"AAA.*", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType=~"AAA.*"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType=~"AAA.*", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType=~"AAA.*"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.52.1.2.34
Metric Used	occnp_diam_response_local_total{msgType=~"AAA.*", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed AAA (Authentication, Authorization, and Accounting) Diameter responses has exceeded the minor threshold (greater than 60%). This indicates an early sign of degradation where a noticeable portion of AAA requests are failing. While the system may still be operational, this could point to developing issues such as intermittent backend failures, rising load, or minor misconfigurations. Diagnostic Information: Metric evaluates: Failed AAA Responses / Total AAA Responses * 100 over a 5-minute window Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 60% Key checks: Observe trends to see if failure rate is increasing toward major/critical thresholds Review response code distribution to identify early error patterns Check connectivity and latency to backend systems (HSS/UDM, AAA servers) Inspect logs for intermittent errors, retries, or timeouts Review recent configuration or deployment changes Monitor system resource utilization (CPU, memory, threads, queues) Identify if failures are isolated to specific AAA message types Recovery: Investigate and address early signs of backend or connectivity issues Validate configurations and recent changes for potential impact Optimize system performance or scale resources if load is increasing Resolve identified error patterns based on response codes Apply preventive measures (e.g., retries tuning, connection pooling adjustments) Continue close monitoring to prevent escalation to higher severity levels If the issue still persist, Please contact the Support.

8.1.3.11 RAA_GX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-280 RAA_GX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	RAA Gx fail count exceeds the critical threshold limit
Summary	Alert RAA_GX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.44.1.2.18
Metric Used	occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed RAA (Re-Auth Answer) messages on the Gx interface has exceeded the critical threshold (greater than 90%). This indicates that most re-authorization responses are failing, which can severely impact policy enforcement, session updates, and charging control. The issue is likely due to failures in communication with PCRF/PCF, policy inconsistencies, or system/resource issues. Diagnostic Information: Metric evaluates: Failed RAA (Gx) Responses / Total RAA (Gx) Responses * 100 over a 5-minute window Scope: msgType = "RAA" (Re-Auth Answer) appType = "Gx" (Policy and Charging Control interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 90% Key checks: Analyze Diameter response codes (e.g., authorization failures, unable to comply, system errors) Verify connectivity and health of PCRF/PCF systems Check for policy rule mismatches or invalid re-authorization requests Review logs for session state issues or failed updates Monitor latency, timeouts, and retransmissions on the Gx interface Inspect system resource utilization (CPU, memory, thread pools) Correlate with spikes in re-auth requests or network events (e.g., QoS changes, session updates) Recovery: Restore or stabilize connectivity with PCRF/PCF Fix policy configuration issues or rule inconsistencies Address specific Diameter error codes causing failures Restart or scale affected components handling Gx traffic Tune timeout and retry mechanisms if needed Roll back recent changes impacting policy control or signaling Continue monitoring until failure rate drops below the critical threshold If the issue persists, please contact the Support.

8.1.3.12 RAA_GX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-281 RAA_GX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	RAA Gx fail count exceeds the major threshold limit
Summary	Alert RAA_GX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.44.1.2.18
Metric Used	occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed RAA (Re-Auth Answer) messages on the Gx interface has exceeded the critical threshold (greater than 80%). This indicates that most re-authorization responses are failing, which can severely impact policy enforcement, session updates, and charging control. The issue is likely due to failures in communication with PCRF/PCF, policy inconsistencies, or system/resource issues. Diagnostic Information: Metric evaluates: Failed RAA (Gx) Responses / Total RAA (Gx) Responses * 100 over a 5-minute window Scope: msgType = "RAA" (Re-Auth Answer) appType = "Gx" (Policy and Charging Control interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 80% Key checks: Analyze Diameter response codes (e.g., authorization failures, unable to comply, system errors) Verify connectivity and health of PCRF/PCF systems Check for policy rule mismatches or invalid re-authorization requests Review logs for session state issues or failed updates Monitor latency, timeouts, and retransmissions on the Gx interface Inspect system resource utilization (CPU, memory, thread pools) Correlate with spikes in re-auth requests or network events (e.g., QoS changes, session updates) Recovery: Restore or stabilize connectivity with PCRF/PCF Fix policy configuration issues or rule inconsistencies Address specific Diameter error codes causing failures Restart or scale affected components handling Gx traffic Tune timeout and retry mechanisms if needed Roll back recent changes impacting policy control or signaling Continue monitoring until failure rate drops below the critical threshold If the issue persists, please contact the Support.

8.1.3.13 RAA_GX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-282 RAA_GX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	RAA Gx fail count exceeds the minor threshold limit
Summary	Alert RAA_GX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.44.1.2.18
Metric Used	occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"2.*"}
Recommended Actions	Cause: The percentage of failed RAA (Re-Auth Answer) messages on the Gx interface has exceeded the critical threshold (greater than 60%). This indicates that most re-authorization responses are failing, which can severely impact policy enforcement, session updates, and charging control. The issue is likely due to failures in communication with PCRF/PCF, policy inconsistencies, or system/resource issues. Diagnostic Information: Metric evaluates: Failed RAA (Gx) Responses / Total RAA (Gx) Responses * 100 over a 5-minute window Scope: msgType = "RAA" (Re-Auth Answer) appType = "Gx" (Policy and Charging Control interface) Failure is defined as Diameter response codes not starting with "2" (non-success responses). Trigger condition: Failure rate > 60% Key checks: Analyze Diameter response codes (e.g., authorization failures, unable to comply, system errors) Verify connectivity and health of PCRF/PCF systems Check for policy rule mismatches or invalid re-authorization requests Review logs for session state issues or failed updates Monitor latency, timeouts, and retransmissions on the Gx interface Inspect system resource utilization (CPU, memory, thread pools) Correlate with spikes in re-auth requests or network events (e.g., QoS changes, session updates) Recovery: Restore or stabilize connectivity with PCRF/PCF Fix policy configuration issues or rule inconsistencies Address specific Diameter error codes causing failures Restart or scale affected components handling Gx traffic Tune timeout and retry mechanisms if needed Roll back recent changes impacting policy control or signaling Continue monitoring until failure rate drops below the critical threshold If the issue persists, please contact the Support.

8.1.3.14 ASA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

ASA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-283 ASA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	ASA fail count exceeds the critical threshold limit
Summary	Alert ASA_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.44.1.2.17
Metric Used	occnp_diam_response_local_total{msgType=~"ASA.*", responseCode!~"2.*"}
Recommended Actions	Cause: A high failure rate of ASA responses returned by the remote peer indicates that ASR requests are not being successfully handled. Common causes include: Session does not exist (common): responseCode = 5002 means the session doesn’t exist (already terminated/aged out, state not found, or session identifier mismatch). Remote peer issues: overload, internal errors, session/policy/state problems preventing ASR handling. Connectivity/routing problems: intermittent network issues, routing/realm misconfiguration, peer unreachable leading to failed responses. Configuration mismatch: incorrect peer configuration, realm/route, application/capability negotiation issues. Traffic spike: increased ASR volume causing the peer to reject requests or return error responses. Overload condition: peer-side throttling/rejection or cascading failures due to sudden ASR storms. Widespread configuration issue: incorrect routing tables, realm/peer identity mismatch introduced by a change. Diagnostic Information: Confirm current ASA failure rate Use the alert expression as-is to validate the observed error percentage. Identify dominant failure response codes. Check if overall ASA volume changed Determine whether the issue is peer-specific Review application and Diameter logs. Look for: peer disconnects / flapping connections timeouts, retransmits, watchdog failures “unable to route” / realm routing errors overload/congestion indications any recent config/deploy changes correlated with onset Recovery: Validate peer health and connectivity: confirm the remote peer is up, reachable, and accepting traffic; check network/routing and Diameter peer status. Reduce load if needed: if ASR volume is abnormal, identify the trigger and throttle upstream behavior where applicable. Fix routing/realm/config mismatches: correct peer configuration, routing tables, realm mappings, or capability negotiation issues. Scale/restore capacity (peer or local): if failures correlate with overload, scale the constrained component(s) or adjust resource limits. Rollback recent changes (if applicable): revert recent configuration, routing, or deployment changes that align with the start of failures Mitigate overload: reduce upstream triggers generating ASRs (where feasible) coordinate with peer team to increase capacity or adjust overload controls Verify recovery: the alert clears automatically once the ASA non-2xxx rate stays below 90% for the evaluation period.

8.1.3.15 ASA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

ASA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-284 ASA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	ASA fail count exceeds the major threshold limit
Summary	Alert ASA_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="ASA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.44.1.2.17
Metric Used	occnp_diam_response_local_total{msgType=~"ASA.*", responseCode!~"2.*"}
Recommended Actions	Cause: A high failure rate of ASA responses returned by the remote peer indicates that ASR requests are not being successfully handled. Common causes include: Session does not exist (common): responseCode = 5002 means the session doesn’t exist (already terminated/aged out, state not found, or session identifier mismatch). Remote peer issues: overload, internal errors, session/policy/state problems preventing ASR handling. Connectivity/routing problems: intermittent network issues, routing/realm misconfiguration, peer unreachable leading to failed responses. Configuration mismatch: incorrect peer configuration, realm/route, application/capability negotiation issues. Traffic spike: increased ASR volume causing the peer to reject requests or return error responses. Overload condition: peer-side throttling/rejection or cascading failures due to sudden ASR storms. Widespread configuration issue: incorrect routing tables, realm/peer identity mismatch introduced by a change. Diagnostic Information: Confirm current ASA failure rate Use the alert expression as-is to validate the observed error percentage. Identify dominant failure response codes. Check if overall ASA volume changed Determine whether the issue is peer-specific Review application and Diameter logs. Look for: peer disconnects / flapping connections timeouts, retransmits, watchdog failures “unable to route” / realm routing errors overload/congestion indications any recent config/deploy changes correlated with onset Recovery: Validate peer health and connectivity: confirm the remote peer is up, reachable, and accepting traffic; check network/routing and Diameter peer status. Reduce load if needed: if ASR volume is abnormal, identify the trigger and throttle upstream behavior where applicable. Fix routing/realm/config mismatches: correct peer configuration, routing tables, realm mappings, or capability negotiation issues. Scale/restore capacity (peer or local): if failures correlate with overload, scale the constrained component(s) or adjust resource limits. Rollback recent changes (if applicable): revert recent configuration, routing, or deployment changes that align with the start of failures Mitigate overload: reduce upstream triggers generating ASRs (where feasible) coordinate with peer team to increase capacity or adjust overload controls Verify recovery: the alert clears automatically once the ASA non-2xxx rate stays below 80% for the evaluation period.

8.1.3.16 ASA_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-285 ASA_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	ASA fail count exceeds the minor threshold limit
Summary	Alert ASA_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="ASA", responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.44.1.2.17
Metric Used	occnp_diam_response_local_total{msgType=~"ASA.*", responseCode!~"2.*"}
Recommended Actions	Cause: A high failure rate of ASA responses returned by the remote peer indicates that ASR requests are not being successfully handled. Common causes include: Session does not exist (common): responseCode = 5002 means the session doesn’t exist (already terminated/aged out, state not found, or session identifier mismatch). Remote peer issues: overload, internal errors, session/policy/state problems preventing ASR handling. Connectivity/routing problems: intermittent network issues, routing/realm misconfiguration, peer unreachable leading to failed responses. Configuration mismatch: incorrect peer configuration, realm/route, application/capability negotiation issues. Traffic spike: increased ASR volume causing the peer to reject requests or return error responses. Overload condition: peer-side throttling/rejection or cascading failures due to sudden ASR storms. Widespread configuration issue: incorrect routing tables, realm/peer identity mismatch introduced by a change. Diagnostic Information: Confirm current ASA failure rate Use the alert expression as-is to validate the observed error percentage. Identify dominant failure response codes. Check if overall ASA volume changed Determine whether the issue is peer-specific Review application and Diameter logs. Look for: peer disconnects / flapping connections timeouts, retransmits, watchdog failures “unable to route” / realm routing errors overload/congestion indications any recent config/deploy changes correlated with onset Recovery: Validate peer health and connectivity: confirm the remote peer is up, reachable, and accepting traffic; check network/routing and Diameter peer status. Reduce load if needed: if ASR volume is abnormal, identify the trigger and throttle upstream behavior where applicable. Fix routing/realm/config mismatches: correct peer configuration, routing tables, realm mappings, or capability negotiation issues. Scale/restore capacity (peer or local): if failures correlate with overload, scale the constrained component(s) or adjust resource limits. Rollback recent changes (if applicable): revert recent configuration, routing, or deployment changes that align with the start of failures Mitigate overload: reduce upstream triggers generating ASRs (where feasible) coordinate with peer team to increase capacity or adjust overload controls Verify recovery: the alert clears automatically once the ASA non-2xxx rate stays below 60% for the evaluation period.

8.1.3.17 ASA_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-286 ASA_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	ASA timeout count exceeds the critical threshold limit
Summary	Alert ASA_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.44.1.2.31
Metric Used	occnp_diam_response_local_total{msgType="ASA", responseCode="timeout"}
Recommended Actions	Cause: More than 90% of local Diameter ASA responses over 5 minutes are timing out. This usually means PCRF/Core (or the local Diameter stack) is not receiving ASA in time after sending ASR, most commonly due to peer reachability/latency issues, peer overload, routing/connection problems, or internal congestion causing the transaction to exceed timers. Diagnostic Information: 1) Confirm the ASA timeout ratio and volume (ensure denominator isn’t near-zero) - Total ASA rate: sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) - ASA timeout rate: sum(rate(occnp_diam_response_local_total{msgType="ASA",responseCode="timeout"}[5m])) 2) Identify which peers/endpoints are driving timeouts (common useful labels: destHost, destRealm, reqDestHost, reqDestRealm, origHost) - sum by (destHost, destRealm) (rate(occnp_diam_response_local_total{msgType="ASA",responseCode="timeout"}[5m])) - If available: sum by (reqDestHost, reqDestRealm) (rate(occnp_diam_response_local_total{msgType="ASA",responseCode="timeout"}[5m])) 3) Check whether this is isolated to ASA or systemic across msgTypes - sum by (msgType, responseCode) (rate(occnp_diam_response_local_total{responseCode="timeout"}[5m])) 4) Correlate with pending/backlog and stale cleanup signals (names vary by deployment; use your actual metric names) - Pending in-flight: occnp_diam_pending_requests (or equivalent) - Stale cleanup: sum(rate(occnp_stale_diam_request_cleanup_total[5m])) (or equivalent) 5) Check related ASR activity (if you also export request-side metrics) - Look for spikes in abort activity leading to more ASR/ASA, and whether ASA timeouts track that spike. 6) Validate timer/transport symptoms - Peer connection flaps, TCP/SCTP retransmits, TLS issues (from Diameter stack logs / node metrics), and latency changes around the alert window. Recovery: 1) If timeouts are concentrated on a specific peer (destHost/destRealm). Validate peer health (CPU, thread pools, queue depth), connectivity, routing, firewall/ACL, and Diameter connection state. If supported, fail traffic away from the impacted peer / route to alternate peer. 2) If timeouts correlate with backlog/pending growth. Reduce load (throttling/admission controls), scale out the impacted service, or address the resource bottleneck (CPU/memory/GC/thread pools). Ensure the Diameter stack isn’t saturated (worker threads, queues). 3) If driven by network/transport issues. Fix underlying network faults (loss/latency), stabilize SCTP/TCP sessions, confirm MTU and TLS settings if applicable. 4) If timers are misaligned systemically. Review Diameter transaction timers and peer timeouts; adjust only per established engineering standards to avoid masking real performance problems. 5) Operational action (last resort). Restart/recycle the specific stuck peer process/component only after addressing the underlying cause; verify ASA timeout % returns to baseline and pending requests drain.

8.1.3.18 ASA_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-287 ASA_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	ASA timeout count exceeds the major threshold limit
Summary	Alert ASA_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="ASA", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.44.1.2.31
Metric Used	occnp_diam_response_local_total{msgType="ASA", responseCode="timeout"}
Recommended Actions	Cause: More than 80% of local Diameter ASA responses over 5 minutes are timing out. This usually means PCRF/Core (or the local Diameter stack) is not receiving ASA in time after sending ASR, most commonly due to peer reachability/latency issues, peer overload, routing/connection problems, or internal congestion causing the transaction to exceed timers. Diagnostic Information: 1) Confirm the ASA timeout ratio and volume (ensure denominator isn’t near-zero) - Total ASA rate: sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) - ASA timeout rate: sum(rate(occnp_diam_response_local_total{msgType="ASA",responseCode="timeout"}[5m])) 2) Identify which peers/endpoints are driving timeouts (common useful labels: destHost, destRealm, reqDestHost, reqDestRealm, origHost) - sum by (destHost, destRealm) (rate(occnp_diam_response_local_total{msgType="ASA",responseCode="timeout"}[5m])) - If available: sum by (reqDestHost, reqDestRealm) (rate(occnp_diam_response_local_total{msgType="ASA",responseCode="timeout"}[5m])) 3) Check whether this is isolated to ASA or systemic across msgTypes - sum by (msgType, responseCode) (rate(occnp_diam_response_local_total{responseCode="timeout"}[5m])) 4) Correlate with pending/backlog and stale cleanup signals (names vary by deployment; use your actual metric names) - Pending in-flight: occnp_diam_pending_requests (or equivalent) - Stale cleanup: sum(rate(occnp_stale_diam_request_cleanup_total[5m])) (or equivalent) 5) Check related ASR activity (if you also export request-side metrics) - Look for spikes in abort activity leading to more ASR/ASA, and whether ASA timeouts track that spike. 6) Validate timer/transport symptoms - Peer connection flaps, TCP/SCTP retransmits, TLS issues (from Diameter stack logs / node metrics), and latency changes around the alert window. Recovery: 1) If timeouts are concentrated on a specific peer (destHost/destRealm). Validate peer health (CPU, thread pools, queue depth), connectivity, routing, firewall/ACL, and Diameter connection state. If supported, fail traffic away from the impacted peer / route to alternate peer. 2) If timeouts correlate with backlog/pending growth. Reduce load (throttling/admission controls), scale out the impacted service, or address the resource bottleneck (CPU/memory/GC/thread pools). Ensure the Diameter stack isn’t saturated (worker threads, queues). 3) If driven by network/transport issues. Fix underlying network faults (loss/latency), stabilize SCTP/TCP sessions, confirm MTU and TLS settings if applicable. 4) If timers are misaligned systemically. Review Diameter transaction timers and peer timeouts; adjust only per established engineering standards to avoid masking real performance problems. 5) Operational action (last resort). Restart/recycle the specific stuck peer process/component only after addressing the underlying cause; verify ASA timeout % returns to baseline and pending requests drain.

8.1.3.19 ASA_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-288 ASA_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	ASA timeout count exceeds the minor threshold limit
Summary	Alert ASA_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{msgType="ASA", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="ASA", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.44.1.2.31
Metric Used	occnp_diam_response_local_total{msgType="ASA", responseCode="timeout"}
Recommended Actions	Cause: More than 60% of local Diameter ASA responses over 5 minutes are timing out. This usually means PCRF/Core (or the local Diameter stack) is not receiving ASA in time after sending ASR, most commonly due to peer reachability/latency issues, peer overload, routing/connection problems, or internal congestion causing the transaction to exceed timers. Diagnostic Information: 1) Confirm the ASA timeout ratio and volume (ensure denominator is not near-zero) - Total ASA rate: sum(rate(occnp_diam_response_local_total{msgType="ASA"}[5m])) - ASA timeout rate: sum(rate(occnp_diam_response_local_total{msgType="ASA",responseCode="timeout"}[5m])) 2) Identify which peers/endpoints are driving timeouts (common useful labels: destHost, destRealm, reqDestHost, reqDestRealm, origHost) - sum by (destHost, destRealm) (rate(occnp_diam_response_local_total{msgType="ASA",responseCode="timeout"}[5m])) - If available: sum by (reqDestHost, reqDestRealm) (rate(occnp_diam_response_local_total{msgType="ASA",responseCode="timeout"}[5m])) 3) Check whether this is isolated to ASA or systemic across msgTypes - sum by (msgType, responseCode) (rate(occnp_diam_response_local_total{responseCode="timeout"}[5m])) 4) Correlate with pending/backlog and stale cleanup signals (names vary by deployment; use your actual metric names) - Pending in-flight: occnp_diam_pending_requests (or equivalent) - Stale cleanup: sum(rate(occnp_stale_diam_request_cleanup_total[5m])) (or equivalent) 5) Check related ASR activity (if you also export request-side metrics) - Look for spikes in abort activity leading to more ASR/ASA, and whether ASA timeouts track that spike. 6) Validate timer/transport symptoms - Peer connection flaps, TCP/SCTP re-transmits, TLS issues (from Diameter stack logs / node metrics), and latency changes around the alert window. Recovery: 1) If timeouts are concentrated on a specific peer (destHost/destRealm) - Validate peer health (CPU, thread pools, queue depth), connectivity, routing, firewall/ACL, and Diameter connection state. If supported, fail traffic away from the impacted peer / route to alternate peer. 2) If timeouts correlate with backlog/pending growth - Reduce load (throttling/admission controls), scale out the impacted service, or address the resource bottleneck (CPU/memory/GC/thread pools). Ensure the Diameter stack is not saturated (worker threads, queues). 3) If driven by network/transport issues. Fix underlying network faults (loss/latency), stabilize SCTP/TCP sessions, confirm MTU and TLS settings if applicable. 4) If timers are misaligned systemically. Review Diameter transaction timers and peer timeouts; adjust only per established engineering standards to avoid masking real performance problems. 5) Operational action (last resort). Restart/recycle the specific stuck peer process/component only after addressing the underlying cause; verify ASA timeout % returns to baseline and pending requests drain.

8.1.3.20 RAA_GX_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 8-289 RAA_GX_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	RAA Gx timeout count exceeds the critical threshold limit
Summary	Alert RAA_GX_TIMEOUT_COUNT_EXCEEDS_CRITICAL_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expession	sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx"}[5m])) * 100 > 90
OID	1.3.6.1.4.1.323.5.3.44.1.2.32
Metric Used	occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"timeout"}
Recommended Actions	Cause: More than 90% of Gx RAA responses over 5 minutes are timing out. This indicates that Gx re-auth transactions (RARRAA) are overwhelmingly failing to complete, typically due to OCS/PCEF/PGW peer non-responsiveness, severe peer or network degradation, routing issues to a specific Gx peer, or severe internal saturation causing request processing or response handling to miss timeouts. Diagnostic Information: 1) Validate volume and ratio- Total Gx RAA rate: sum(rate(occnp_diam_response_local_total{msgType="RAA",appType="Gx"}[5m])) - Timeout Gx RAA rate: sum(rate(occnp_diam_response_local_total{msgType="RAA",appType="Gx",responseCode="timeout"}[5m])) 2) Pinpoint impacted Gx peers - sum by (destHost, destRealm) (rate(occnp_diam_response_local_total{msgType="RAA",appType="Gx",responseCode="timeout"}[5m])) - If available: sum by (reqDestHost, reqDestRealm) (rate(occnp_diam_response_local_total{msgType="RAA",appType="Gx",responseCode="timeout"}[5m])) 3) Confirm if timeouts are specific to RAA or widespread on Gx - sum by (msgType, responseCode) (rate(occnp_diam_response_local_total{appType="Gx",responseCode="timeout"}[5m])) 4) Check re-auth triggering rate and retry patterns (if you export request/process metrics) - Look for spikes in reauth triggers and retries; correlate with timeout onset. 5) Check backlog / stuck transactions - occnp_diam_pending_requests (or equivalent) grouped by app if possible - stale cleanup metric rate increases during the alert window 6) Review Diameter stack / application logs for Gx - Connection down / watchdog failures, route resolution issues, send failed, peer busy, queue full, timer expirations. Recovery: 1) If isolated to one Gx peer - Validate PCEF/PGW/peer health and connectivity; fix routing, ACL/firewall, DNS/peer resolution, and confirm Diameter connection is established and stable. - Shift traffic to alternate peer/route if supported. 2) If re-auth storm / load spike is driving it - Apply rate-limiting/backoff (per product guidance), reduce triggering conditions upstream, and/or scale out the affected component. Confirm thread pools/queues arent saturated; address resource exhaustion. 3) If network degradation is the driver - Stabilize transport (SCTP/TCP), fix packet loss/latency, validate MTU, and ensure no intermediate device is dropping long-lived Diameter sessions. 4) If timers are too aggressive vs observed latency - Review and align transaction timers with SLOs and real latency; avoid papering over peer slowness. 5) Operational action (last resort) - Restart only the affected Diameter peer component after restoring network/peer health; validate RAA timeout % drops and successful RAA resumes.

8.1.3.21 RAA_GX_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 8-290 RAA_GX_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	RAA Gx timeout count exceeds the major threshold limit
Summary	Alert RAA_GX_TIMEOUT_COUNT_EXCEEDS_MAJOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx"}[5m])) * 100 > 80 and sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx"}[5m])) * 100 <= 90
OID	1.3.6.1.4.1.323.5.3.44.1.2.32
Metric Used	occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"timeout"}
Recommended Actions	Cause: Majority of Gx RAA transactions are timing out. Common drivers: Gx peer (PCEF/PGW) not responding, peer overload, network degradation (loss/latency), Diameter connection instability, routing to wrong/unreachable peer, or local backlog causing transactions to exceed timers. Diagnostic Information: 1) Validate ratio + volume (ensure denominator is not tiny) - Total: sum(rate(occnp_diam_response_local_total{msgType="RAA",appType="Gx"}[5m])) - Timeouts: sum(rate(occnp_diam_response_local_total{msgType="RAA",appType="Gx",responseCode="timeout"}[5m])) 2) Identify impacted peers/endpoints (use available labels like destHost, destRealm, reqDestHost, reqDestRealm) - sum by (destHost,destRealm) (rate(occnp_diam_response_local_total{msgType="RAA",appType="Gx",responseCode="timeout"}[5m])) 3) Check if timeouts are isolated to RAA or widespread on Gx - sum by (msgType) (rate(occnp_diam_response_local_total{appType="Gx",responseCode="timeout"}[5m])) 4) Look for congestion / backlog symptoms (metric names vary) - Pending in-flight: occnp_diam_pending_requests (or equivalent) - Stale cleanups: sum(rate(occnp_stale_diam_request_cleanup_total[5m])) (or equivalent) 5) Correlate with Diameter connectivity events (from logs / stack metrics): watchdog failures, reconnect loops, send failures, peer-down. Recovery: 1) If concentrated on one peer: validate peer health/connectivity; fail traffic over to alternate peer/routing if supported. 2) If backlog/CPU saturation locally: scale out, reduce trigger rate, tune threadpools/queues per product guidance; confirm pending drains. 3) If network driven: fix transport (SCTP/TCP), loss/latency, ACL/firewall, MTU, TLS; stabilize sessions. 4) Review/align timers only if systemic and validated; avoid masking a slow peer. 5) Last resort: restart affected peer component after underlying issue is resolved; confirm timeout % drops below threshold.

8.1.3.22 RAA_GX_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 8-291 RAA_GX_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	RAA Gx timeout count exceeds the minor threshold limit
Summary	Alert RAA_GX_TIMEOUT_COUNT_EXCEEDS_MINOR_THRESHOLD NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx"}[5m])) * 100 > 60 and sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode="timeout"}[5m])) / sum(rate(occnp_diam_response_local_total{msgType="RAA", appType="Gx"}[5m])) * 100 <= 80
OID	1.3.6.1.4.1.323.5.3.44.1.2.32
Metric Used	occnp_diam_response_local_total{msgType="RAA", appType="Gx", responseCode!~"timeout"}
Recommended Actions	Cause: Early indicator of the same failure mode as above, but may be intermittent (bursty peer slowness, brief network impairment, or onset of overload). Diagnostic Information: Use the same diagnostics as the MAJOR alert, but add: 1) Trend it (is it rising?) - avg_over_time((sum(rate(…timeout…[5m])) / sum(rate(…total…[5m])) * 100)[30m:])   2) Compare by peer to see if one peer is degrading first - topk(5, sum by (destHost,destRealm) (rate(occnp_diam_response_local_total{msgType="RAA",appType="Gx",responseCode="timeout"}[5m]))) Recovery: 1) Proactively validate peer and network health before it escalates. 2) If one peer shows degradation, preemptively route/failover if allowed. 3) If local resources trending high, scale/relieve load and verify timeouts return to baseline.

8.1.3.23 RESPONSE_ERROR_RATE_ABOVE_CRITICAL_PERCENT

Table 8-292 RESPONSE_ERROR_RATE_ABOVE_CRITICAL_PERCENT

Field	Details
Description	CCA, AAA, RAA, ASA and STA error rate combined is above 10 percent
Summary	Alert RESPONSE_ERROR_RATE_ABOVE_CRITICAL_PERCENT NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total[5m])) * 100 > 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.37
Metric Used	occnp_diam_response_local_total{ responseCode!~"2.*"}
Recommended Actions	Cause: A broad service-impacting condition: >10% of all local Diameter answers are non-2xx. Causes typically include peer outages, widespread timeouts, routing/realm issues, auth/credit/policy dependency failures, system overload, or misconfiguration causing systematic rejects/errors. Diagnostic Information: 1) Identify top failing response codes and message types - sum by (responseCode) (rate(occnp_diam_response_local_total{responseCode!~"2.*"}[5m])) - sum by (msgType,responseCode) (rate(occnp_diam_response_local_total{responseCode!~"2.*"}[5m])) 2) Find which app is driving it (if appType exists on the metric) - sum by (appType) (rate(occnp_diam_response_local_total{responseCode!~"2.*"}[5m])) 3) Pinpoint impacted peers - sum by (destHost,destRealm,responseCode) (rate(occnp_diam_response_local_total{responseCode!~"2.*"}[5m])) 4) Separate timeouts vs explicit rejects/errors - Timeouts: sum(rate(occnp_diam_response_local_total{responseCode="timeout"}[5m])) - Other errors: `sum(rate(occnp_diam_response_local_total{responseCode!~"2.*timeout"}[5m])) 5) Correlate with saturation and stuck-transaction symptoms - Pending:occnp_diam_pending_requests(or equivalent) - Stale cleanup:sum(rate(occnp_stale_diam_request_cleanup_total[5m]))(or equivalent) Recovery: 1) Stabilize service first (contain impact) - If errors are dominated by timeouts: treat as an availability/latency incident—verify Diameter peer connectivity (TCP/SCTP up, watchdog OK), check packet loss/latency, and remediate network/transport issues. - If concentrated on a specific peer (destHost/destRealm): isolate it (failover/reroute traffic if supported) and engage the peer owner to restore capacity/health. 2) Remove overload / drain backlog - If occnp_diam_pending_requests (or equivalent) is growing or CPU/memory/thread pools are saturated: scale out the service, reduce incoming load (throttle/admission controls), and address bottlenecks so pending requests drain and timeouts fall. 3) Fix systematic rejects / config-induced errors - If non-2xx is driven by specific result codes (not timeouts): verify realm routing, peer configuration, policy/charging dependencies (e.g., OCS/UDR/SPR), and any recent config/deploy changes; roll back or correct configuration as needed. 4) Avoid masking the problem with timer changes - Only adjust Diameter timers if analysis shows they are misaligned with expected latency; do not use timer increases as a substitute for fixing slow/unhealthy peers. 5) Last-resort operational actions - Restart/recycle only the affected component(s) if they are stuck (e.g., wedged Diameter stack) after underlying network/peer issues are corrected; avoid repeated restarts that can amplify load. 6) Recovery validation - Confirm combined non-2xx ratio drops below 10%, timeouts return to baseline, pending/backlog metrics normalize, and per-msgType success rates (2xx) recover across the major peers.

8.1.3.24 RESPONSE_ERROR_RATE_ABOVE_MAJOR_PERCENT

Table 8-293 RESPONSE_ERROR_RATE_ABOVE_MAJOR_PERCENT

Field	Details
Description	CCA, AAA, RAA, ASA and STA error rate combined is above 5 percent
Summary	Alert RESPONSE_ERROR_RATE_ABOVE_MAJOR_PERCENT NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total[5m])) * 100 > 5 and sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total[5m])) * 100 <= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.37
Metric Used	occnp_diam_response_local_total{ responseCode!~"2.*"}
Recommended Actions	Cause: Same category as CRITICAL but less severe; indicates meaningful degradation (often partial peer impairment, rising timeouts, or one message type failing). Diagnostic Information: Same diagnostics as CRITICAL, with focus on identifying the earliest driver: - By msgType: sum by (msgType) (rate(occnp_diam_response_local_total{responseCode!~"2.*"}[5m])) - By peer: sum by (destHost,destRealm) (rate(occnp_diam_response_local_total{responseCode!~"2.*"}[5m])) Recovery:Same recovery patterns as CRITICAL, typically starting with: isolate peer/appType causing the increase; remediate connectivity/perf; apply failover/rate-limit/scale before it escalates.

8.1.3.25 RESPONSE_ERROR_RATE_ABOVE_MINOR_PERCENT

Table 8-294 RESPONSE_ERROR_RATE_ABOVE_MINOR_PERCENT

Field	Details
Description	CCA, AAA, RAA, ASA and STA error rate combined is above 1 percent
Summary	Alert RESPONSE_ERROR_RATE_ABOVE_MINOR_PERCENT NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total[5m])) * 100 > 1 and sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*"}[5m])) / sum(rate(occnp_diam_response_local_total[5m])) * 100 <= 5
OID	1.3.6.1.4.1.323.5.3.52.1.2.37
Metric Used	occnp_diam_response_local_total{ responseCode!~"2.*"}
Recommended Actions	Cause: Early warning of deteriorating conditions: small but non-trivial increase in timeouts, rejects, or application errors. Could be a single peer intermittently failing, a rollout regression, or early overload. Diagnostic Information: 1) Quickly determine whether this is timeouts vs non-timeout errors - sum by (responseCode) (rate(occnp_diam_response_local_total{responseCode!~"2.*"}[5m])) 2) See if it’s localized to a msgType/appType/peer - sum by (appType,msgType,responseCode) (rate(occnp_diam_response_local_total{responseCode!~"2.*"}[5m])) - sum by (destHost,destRealm) (rate(occnp_diam_response_local_total{responseCode!~"2.*"}[5m]))   3) Correlate with recent changes (deployments/config/routing changes) and node health signals. Recovery: 1) If timeouts start trending up: validate network/peer health and capacity early; consider proactive reroute/failover. 2) If rejects/errors rise after a change: roll back or fix config; confirm realm/route/policy alignment.   3) Continue monitoring until stable baseline is restored (<1%).

8.1.3.26 Rx_RESPONSE_ERROR_RATE_ABOVE_CRITICAL_PERCENT

Table 8-295 Rx_RESPONSE_ERROR_RATE_ABOVE_CRITICAL_PERCENT

Field	Details
Description	Rx error rate combined is above 10 percent
Summary	Alert Rx_RESPONSE_ERROR_RATE_ABOVE_CRITICAL_PERCENT NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*", appType="Rx"}[5m])) / sum(rate(occnp_diam_response_local_total{ appType="Rx"}[5m])) * 100 > 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.38
Metric Used	occnp_diam_response_local_total{ responseCode!~"2.*", appType="Rx"}
Recommended Actions	Cause: more than 10% of Rx local Diameter responses over 5 minutes are non-2xx, meaning PCRF-Core is returning/observing too many failed, rejected, or timed-out Rx answers. Diagnostic Information: 1 Check current top failing Rx response codes - Query: `sum by (responseCode, msgType, destHost, destRealm) (rate(occnp_diam_response_local_total{appType="Rx",responseCode!~"2.*"}[5m]))` 2 Compare success vs failure split - Success:`sum(rate(occnp_diam_response_local_total{appType="Rx",responseCode=~"2.*"}[5m]))` - Failure: `sum(rate(occnp_diam_response_local_total{appType="Rx",responseCode!~"2.*"}[5m]))` 3 Identify whether failures are concentrated on a specific Rx message type - Query by `msgType`: - `sum by (msgType, responseCode) (rate(occnp_diam_response_local_total{appType="Rx"}[5m]))` - In this code, Rx message types commonly map to `AAR/AAA`, `RAR/RAA`, `STR/STA`, `ASR/ASA` depending on flow. 4 Identify affected peers/endpoints - Query by `destHost`, `destRealm`, `reqDestHost`, `reqDestRealm`, `origHost` - This helps determine whether one AF/peer is driving the issue. 5 Check for timeouts specifically - Query:`sum by (msgType, reqDestHost, reqDestRealm) (rate(occnp_diam_response_local_total{appType="Rx",responseCode="timeout"}[5m]))` 6 Correlate with internal request/reply metrics - `occnp_app_request_local_process_total` - `occnp_app_response_local_process_total` These carry labels like `reason`, `msgType`, `destHost`, `destRealm`, `retryAttempts`, and help determine whether failures are tied to retries, reauth, or specific internal reasons. 7 Check stale or congestion-related symptoms if suspected - `occnp_stale_diam_request_cleanup_total` (actual full name depends on prefix/suffix rules around `stale_diam_request_cleanup_total`) - `occnp_diam_pending_requests` - congestion reject metrics if request throttling is involved 8 Review logs around rejection paths - Relevant code paths log warnings when: - no PEP / enforcement session found - AN-GW failure - duplicate or stale handling - pending transaction retry / timeout scenarios Recovery: Recommended recovery depends on the dominant failing `responseCode` / `msgType`, but generally: 1 If failures are timeout-heavy - verify Diameter peer connectivity and peer health - check whether AF / peer is reachable and responding in time - inspect recent latency increase and pending request growth - validate timer/profile configuration if timeouts are systemic 2 If failures are concentrated on one peer (`destHost`/`destRealm`) - isolate the peer issue - verify peer routing, reachability, and configuration - fail traffic away from the impacted peer if supported 3 If failures are dominated by rejection/result codes - inspect policy rejection, binding, or session-correlation paths - verify enforcement session existence / PEP availability - check for admission/session-limit conditions - verify AN-GW or enforcement-side dependencies 4 If stale or overload symptoms are present - inspect queueing delays and pending Diameter requests - check congestion / resource exhaustion - scale or relieve traffic if the service is saturated 5 Operational recovery actions - restart or recover the affected Diameter peer/component only if the issue is stuck and peer-specific - clear underlying downstream/network fault before restarting PCRF-Core - after recovery, confirm: - Rx non-2xx ratio falls below 10% - timeout counts return to baseline - successful 2xx Rx responses resume

8.1.3.27 Rx_RESPONSE_ERROR_RATE_ABOVE_MAJOR_PERCENT

Table 8-296 Rx_RESPONSE_ERROR_RATE_ABOVE_MAJOR_PERCENT

Field	Details
Description	Rx error rate combined is above 5 percent
Summary	Alert Rx_RESPONSE_ERROR_RATE_ABOVE_MAJOR_PERCENT NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*", appType="Rx"}[5m])) / sum(rate(occnp_diam_response_local_total{ appType="Rx"}[5m])) * 100 > 5 and sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*", appType="Rx"}[5m])) / sum(rate(occnp_diam_response_local_total{ appType="Rx"}[5m])) * 100 <= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.38
Metric Used	occnp_diam_response_local_total{ responseCode!~"2.*", appType="Rx"}
Recommended Actions	Cause: more than 5% and less than 10% of Rx local Diameter responses over 5 minutes are non-2xx, meaning PCRF-Core is returning/observing too many failed, rejected, or timed-out Rx answers. Diagnostic Information: 1 Check current top failing Rx response codes - Query: `sum by (responseCode, msgType, destHost, destRealm) (rate(occnp_diam_response_local_total{appType="Rx",responseCode!~"2.*"}[5m]))` 2 Compare success vs failure split - Success:`sum(rate(occnp_diam_response_local_total{appType="Rx",responseCode=~"2.*"}[5m]))` - Failure: `sum(rate(occnp_diam_response_local_total{appType="Rx",responseCode!~"2.*"}[5m]))` 3 Identify whether failures are concentrated on a specific Rx message type - Query by `msgType`: - `sum by (msgType, responseCode) (rate(occnp_diam_response_local_total{appType="Rx"}[5m]))` - In this code, Rx message types commonly map to `AAR/AAA`, `RAR/RAA`, `STR/STA`, `ASR/ASA` depending on flow. 4 Identify affected peers/endpoints - Query by `destHost`, `destRealm`, `reqDestHost`, `reqDestRealm`, `origHost` - This helps determine whether one AF/peer is driving the issue. 5 Check for timeouts specifically - Query:`sum by (msgType, reqDestHost, reqDestRealm) (rate(occnp_diam_response_local_total{appType="Rx",responseCode="timeout"}[5m]))` 6 Correlate with internal request/reply metrics - `occnp_app_request_local_process_total` - `occnp_app_response_local_process_total` These carry labels like `reason`, `msgType`, `destHost`, `destRealm`, `retryAttempts`, and help determine whether failures are tied to retries, reauth, or specific internal reasons. 7 Check stale or congestion-related symptoms if suspected - `occnp_stale_diam_request_cleanup_total` (actual full name depends on prefix/suffix rules around `stale_diam_request_cleanup_total`) - `occnp_diam_pending_requests` - congestion reject metrics if request throttling is involved 8 Review logs around rejection paths - Relevant code paths log warnings when: - no PEP / enforcement session found - AN-GW failure - duplicate or stale handling - pending transaction retry / timeout scenarios Recovery: Recommended recovery depends on the dominant failing `responseCode` / `msgType`, but generally: 1 If failures are timeout-heavy - verify Diameter peer connectivity and peer health - check whether AF / peer is reachable and responding in time - inspect recent latency increase and pending request growth - validate timer/profile configuration if timeouts are systemic 2 If failures are concentrated on one peer (`destHost`/`destRealm`) - isolate the peer issue - verify peer routing, reachability, and configuration - fail traffic away from the impacted peer if supported 3 If failures are dominated by rejection/result codes - inspect policy rejection, binding, or session-correlation paths - verify enforcement session existence / PEP availability - check for admission/session-limit conditions - verify AN-GW or enforcement-side dependencies 4 If stale or overload symptoms are present - inspect queueing delays and pending Diameter requests - check congestion / resource exhaustion - scale or relieve traffic if the service is saturated 5 Operational recovery actions - restart or recover the affected Diameter peer/component only if the issue is stuck and peer-specific - clear underlying downstream/network fault before restarting PCRF-Core - after recovery, confirm: - Rx non-2xx ratio falls below 10% - timeout counts return to baseline - successful 2xx Rx responses resume

8.1.3.28 Rx_RESPONSE_ERROR_RATE_ABOVE_MINOR_PERCENT

Table 8-297 Rx_RESPONSE_ERROR_RATE_ABOVE_MINOR_PERCENT

Field	Details
Description	Rx error rate combined is above 1 percent
Summary	Alert Rx_RESPONSE_ERROR_RATE_ABOVE_MINOR_PERCENT NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*", appType="Rx"}[5m])) / sum(rate(occnp_diam_response_local_total{ appType="Rx"}[5m])) * 100 > 1 and sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*", appType="Rx"}[5m])) / sum(rate(occnp_diam_response_local_total{ appType="Rx"}[5m])) * 100 <= 5
OID	1.3.6.1.4.1.323.5.3.52.1.2.38
Metric Used	occnp_diam_response_local_total{ responseCode!~"2.*", appType="Rx"}
Recommended Actions	Cause: more than 1% and less than 5% of Rx local Diameter responses over 5 minutes are non-2xx, meaning PCRF-Core is returning/observing too many failed, rejected, or timed-out Rx answers. Diagnostic Information: 1 Check current top failing Rx response codes - Query: `sum by (responseCode, msgType, destHost, destRealm) (rate(occnp_diam_response_local_total{appType="Rx",responseCode!~"2.*"}[5m]))` 2 Compare success vs failure split - Success:`sum(rate(occnp_diam_response_local_total{appType="Rx",responseCode=~"2.*"}[5m]))` - Failure: `sum(rate(occnp_diam_response_local_total{appType="Rx",responseCode!~"2.*"}[5m]))` 3 Identify whether failures are concentrated on a specific Rx message type - Query by `msgType`: - `sum by (msgType, responseCode) (rate(occnp_diam_response_local_total{appType="Rx"}[5m]))` - In this code, Rx message types commonly map to `AAR/AAA`, `RAR/RAA`, `STR/STA`, `ASR/ASA` depending on flow. 4 Identify affected peers/endpoints - Query by `destHost`, `destRealm`, `reqDestHost`, `reqDestRealm`, `origHost` - This helps determine whether one AF/peer is driving the issue. 5 Check for timeouts specifically - Query:`sum by (msgType, reqDestHost, reqDestRealm) (rate(occnp_diam_response_local_total{appType="Rx",responseCode="timeout"}[5m]))` 6 Correlate with internal request/reply metrics - `occnp_app_request_local_process_total` - `occnp_app_response_local_process_total` These carry labels like `reason`, `msgType`, `destHost`, `destRealm`, `retryAttempts`, and help determine whether failures are tied to retries, reauth, or specific internal reasons. 7 Check stale or congestion-related symptoms if suspected - `occnp_stale_diam_request_cleanup_total` (actual full name depends on prefix/suffix rules around `stale_diam_request_cleanup_total`) - `occnp_diam_pending_requests` - congestion reject metrics if request throttling is involved 8 Review logs around rejection paths - Relevant code paths log warnings when: - no PEP / enforcement session found - AN-GW failure - duplicate or stale handling - pending transaction retry / timeout scenarios Recovery: Recommended recovery depends on the dominant failing `responseCode` / `msgType`, but generally: 1 If failures are timeout-heavy - verify Diameter peer connectivity and peer health - check whether AF / peer is reachable and responding in time - inspect recent latency increase and pending request growth - validate timer/profile configuration if timeouts are systemic 2 If failures are concentrated on one peer (`destHost`/`destRealm`) - isolate the peer issue - verify peer routing, reachability, and configuration - fail traffic away from the impacted peer if supported 3 If failures are dominated by rejection/result codes - inspect policy rejection, binding, or session-correlation paths - verify enforcement session existence / PEP availability - check for admission/session-limit conditions - verify AN-GW or enforcement-side dependencies 4 If stale or overload symptoms are present - inspect queueing delays and pending Diameter requests - check congestion / resource exhaustion - scale or relieve traffic if the service is saturated 5 Operational recovery actions - restart or recover the affected Diameter peer/component only if the issue is stuck and peer-specific - clear underlying downstream/network fault before restarting PCRF-Core - after recovery, confirm: - Rx non-2xx ratio falls below 10% - timeout counts return to baseline - successful 2xx Rx responses resume

8.1.3.29 Gx_RESPONSE_ERROR_RATE_ABOVE_CRITICAL_PERCENT

Table 8-298 Gx_RESPONSE_ERROR_RATE_ABOVE_CRITICAL_PERCENT

Field	Details
Description	Gx error rate combined is above 10 percent
Summary	Alert Gx_RESPONSE_ERROR_RATE_ABOVE_CRITICAL_PERCENT NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Critical
Expression	sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*", appType="Gx"}[5m])) / sum(rate(occnp_diam_response_local_total{ appType="Gx"}[5m])) * 100 > 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.39
Metric Used	occnp_diam_response_local_total{ responseCode!~"2.*", appType="Gx"}
Recommended Actions	Cause: This means >10% of Gx responses seen by PCRF-Core in the last 5m are non-success Diameter responses or timeouts. Diagnostic Information: - Metrics involved: `occnp_diam_response_local_total{appType="Gx",responseCode,...}`; supporting metric: `occnp_diam_response_local_processing_latency{...}`. - Error observed: elevated non-2xx Gx responses / timeouts. - Cause value: `sum(rate(occnp_diam_response_local_total{responseCode!~"2.*",appType="Gx"}[5m])) / sum(rate(occnp_diam_response_local_total{appType="Gx"}[5m])) * 100 > 10`. - Condition: non-success Gx responses exceed 10% of all Gx responses in 5m. - Likely causes: PGW/peer returning Gx failures such as `DIAMETER_PENDING_TRANSACTION`, `DIAMETER_UNABLE_TO_COMPLY`, `UNKNOWN_SESSION_ID`, authorization rejects, peer busy/unavailable, or request timeout. - Verification steps: 1) break down failures by `responseCode,msgType,destHost,destRealm`; 2) check for `responseCode="timeout"`; 3) correlate with logs for CCA/RAA failures and peer issues; 4) review latency spikes in `occnp_diam_response_local_processing_latency`. - Monitoring recommendations: alert separately on top `responseCode`; split by `destHost`/`destRealm`; add dedicated timeout alert; track latency with error ratio. Recovery: - Identify dominant failing `responseCode` and impacted Gx peer/realm. - If timeouts / peer transport errors dominate: check Diameter peer connectivity, session state, overload, and transport health. - If `DIAMETER_PENDING_TRANSACTION` dominates: investigate Gx race/retry pressure and pending-transaction behavior. - If `UNKNOWN_SESSION_ID` dominates: verify stale/missing session state, audit/cleanup effects, and session sync. - If `UNABLE_TO_COMPLY` / auth rejects dominate: inspect recent policy/config/data/backend changes and rollback/fix as needed. - Recovery is complete when the non-2xx Gx response ratio drops below 10% and failing codes return to baseline.

8.1.3.30 Gx_RESPONSE_ERROR_RATE_ABOVE_MAJOR_PERCENT

Table 8-299 Gx_RESPONSE_ERROR_RATE_ABOVE_MAJOR_PERCENT

Field	Details
Description	Gx error rate combined is above 5 percent
Summary	Alert Rx_RESPONSE_ERROR_RATE_ABOVE_MAJOR_PERCENT NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Major
Expression	sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*", appType="Gx"}[5m])) / sum(rate(occnp_diam_response_local_total{ appType="Gx"}[5m])) * 100 > 5 and sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*", appType="Gx"}[5m])) / sum(rate(occnp_diam_response_local_total{ appType="Gx"}[5m])) * 100 <= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.39
Metric Used	occnp_diam_response_local_total{ responseCode!~"2.*", appType="Gx"}
Recommended Actions	Cause: This means >5% and <10% of Gx responses seen by PCRF-Core in the last 5m are non-success Diameter responses or timeouts. Diagnostic Information: - Metrics involved: `occnp_diam_response_local_total{appType="Gx",responseCode,...}`; supporting metric: `occnp_diam_response_local_processing_latency{...}`. - Error observed: elevated non-2xx Gx responses / timeouts. - Cause value: `sum(rate(occnp_diam_response_local_total{responseCode!~"2.*",appType="Gx"}[5m])) / sum(rate(occnp_diam_response_local_total{appType="Gx"}[5m])) * 100 > 10`. - Condition: non-success Gx responses exceed 10% of all Gx responses in 5m. - Likely causes: PGW/peer returning Gx failures such as `DIAMETER_PENDING_TRANSACTION`, `DIAMETER_UNABLE_TO_COMPLY`, `UNKNOWN_SESSION_ID`, authorization rejects, peer busy/unavailable, or request timeout. - Verification steps: 1) break down failures by `responseCode,msgType,destHost,destRealm`; 2) check for `responseCode="timeout"`; 3) correlate with logs for CCA/RAA failures and peer issues; 4) review latency spikes in `occnp_diam_response_local_processing_latency`. - Monitoring recommendations: alert separately on top `responseCode`; split by `destHost`/`destRealm`; add dedicated timeout alert; track latency with error ratio. Recovery: - Identify dominant failing `responseCode` and impacted Gx peer/realm. - If timeouts / peer transport errors dominate: check Diameter peer connectivity, session state, overload, and transport health. - If `DIAMETER_PENDING_TRANSACTION` dominates: investigate Gx race/retry pressure and pending-transaction behavior. - If `UNKNOWN_SESSION_ID` dominates: verify stale/missing session state, audit/cleanup effects, and session sync. - If `UNABLE_TO_COMPLY` / auth rejects dominate: inspect recent policy/config/data/backend changes and rollback/fix as needed. - Recovery is complete when the non-2xx Gx response ratio drops below 10% and failing codes return to baseline.

8.1.3.31 Gx_RESPONSE_ERROR_RATE_ABOVE_MINOR_PERCENT

Table 8-300 Gx_RESPONSE_ERROR_RATE_ABOVE_MINOR_PERCENT

Field	Details
Description	Gx error rate combined is above 1 percent
Summary	Alert Rx_RESPONSE_ERROR_RATE_ABOVE_MINOR_PERCENT NS:{{ $labels.kubernetes_namespace }}, PODNAME:{{ $labels.kubernetes_pod_name }}, INST:{{ $labels.instance }} REL:{{ $labels.release }}
Severity	Minor
Expression	sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*", appType="Gx"}[5m])) / sum(rate(occnp_diam_response_local_total{ appType="Gx"}[5m])) * 100 > 1 and sum(rate(occnp_diam_response_local_total{ responseCode!~"2.*", appType="Gx"}[5m])) / sum(rate(occnp_diam_response_local_total{ appType="Gx"}[5m])) * 100 <= 5
OID	1.3.6.1.4.1.323.5.3.52.1.2.39
Metric Used	occnp_diam_response_local_total{ responseCode!~"2.*", appType="Gx"}
Recommended Actions	Cause: This means 1>% and <5% of Gx responses seen by PCRF-Core in the last 5m are non-success Diameter responses or timeouts. Diagnostic Information: - Metrics involved: `occnp_diam_response_local_total{appType="Gx",responseCode,...}`; supporting metric: `occnp_diam_response_local_processing_latency{...}`. - Error observed: elevated non-2xx Gx responses / timeouts. - Cause value: `sum(rate(occnp_diam_response_local_total{responseCode!~"2.*",appType="Gx"}[5m])) / sum(rate(occnp_diam_response_local_total{appType="Gx"}[5m])) * 100 > 10`. - Condition: non-success Gx responses exceed 10% of all Gx responses in 5m. - Likely causes: PGW/peer returning Gx failures such as `DIAMETER_PENDING_TRANSACTION`, `DIAMETER_UNABLE_TO_COMPLY`, `UNKNOWN_SESSION_ID`, authorization rejects, peer busy/unavailable, or request timeout. - Verification steps: 1) break down failures by `responseCode,msgType,destHost,destRealm`; 2) check for `responseCode="timeout"`; 3) correlate with logs for CCA/RAA failures and peer issues; 4) review latency spikes in `occnp_diam_response_local_processing_latency`. - Monitoring recommendations: alert separately on top `responseCode`; split by `destHost`/`destRealm`; add dedicated timeout alert; track latency with error ratio. Recovery: - Identify dominant failing `responseCode` and impacted Gx peer/realm. - If timeouts / peer transport errors dominate: check Diameter peer connectivity, session state, overload, and transport health. - If `DIAMETER_PENDING_TRANSACTION` dominates: investigate Gx race/retry pressure and pending-transaction behavior. - If `UNKNOWN_SESSION_ID` dominates: verify stale/missing session state, audit/cleanup effects, and session sync. - If `UNABLE_TO_COMPLY` / auth rejects dominate: inspect recent policy/config/data/backend changes and rollback/fix as needed. - Recovery is complete when the non-2xx Gx response ratio drops below 10% and failing codes return to baseline.

8.1.3.32 STALE_DIAMETER_REQUEST_CLEANUP_CRITICAL

Table 8-301 STALE_DIAMETER_REQUEST_CLEANUP_CRITICAL

Field	Details
Description	The Diameter requests are being discarded due to timeout processing occurring above 30%
Summary	The Diameter requests are being discarded due to timeout processing occurring above 30%
Severity	Critical
Expression	(sum by (namespace, microservice, pod) (increase(occnp_stale_diam_request_cleanup_total[24h])) / sum by (namespace, microservice, pod) (increase(occnp_diam_request_local_total{msgType!~"DWR|CER"}[24h]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.52.1.2.82
Metric Used	occnp_stale_diam_request_cleanup_total
Recommended Actions	Cause: More than 30% of Diameter requests sent in the last 24 hours did not receive a response in time and were discarded by the cleanup mechanism. This indicates Diameter peers are slow to respond, unreachable, or the node is overloaded. Diagnostic Information: Check the affected pod and microservice for signs of overload or connectivity issues with Diameter peers. Review Diameter peer status, response times, and whether the cleanup rate is increasing over time. Exclude DWR/CER traffic from analysis as these are control-plane messages. Recovery: Verify connectivity and health of Diameter peers. If the node is overloaded, consider scaling the affected microservice. Investigate and resolve any network latency or timeout misconfigurations between the node and its Diameter peers.

8.1.3.33 STALE_DIAMETER_REQUEST_CLEANUP_MAJOR

Table 8-302 STALE_DIAMETER_REQUEST_CLEANUP_MAJOR

Field	Details
Description	The Diameter requests are being discarded due to timeout processing occurring above 20%
Summary	The Diameter requests are being discarded due to timeout processing occurring above 20%
Severity	Major
Expression	(sum by (namespace, microservice, pod) (increase(occnp_stale_diam_request_cleanup_total[24h])) / sum by (namespace, microservice, pod) (increase(occnp_diam_request_local_total{msgType!~"DWR|CER"}[24h]))) * 100 >= 20
OID	1.3.6.1.4.1.323.5.3.52.1.2.82
Metric Used	occnp_stale_diam_request_cleanup_total
Recommended Actions	Cause: More than 20% of Diameter requests sent in the last 24 hours did not receive a response in time and were discarded by the cleanup mechanism. This indicates Diameter peers are slow to respond, unreachable, or the node is overloaded. Diagnostic Information: Check the affected pod and microservice for signs of overload or connectivity issues with Diameter peers. Review Diameter peer status, response times, and whether the cleanup rate is increasing over time. Exclude DWR/CER traffic from analysis as these are control-plane messages. Recovery: Verify connectivity and health of Diameter peers. If the node is overloaded, consider scaling the affected microservice. Investigate and resolve any network latency or timeout misconfigurations between the node and its Diameter peers.

8.1.3.34 STALE_DIAMETER_REQUEST_CLEANUP_MINOR

Table 8-303 STALE_DIAMETER_REQUEST_CLEANUP_MINOR

Field	Details
Description	The Diameter requests are being discarded due to timeout processing occurring above 10%
Summary	The Diameter requests are being discarded due to timeout processing occurring above 10%
Severity	Minor
Expression	(sum by (namespace, microservice, pod) (increase(occnp_stale_diam_request_cleanup_total[24h])) / sum by (namespace, microservice, pod) (increase(occnp_diam_request_local_total{msgType!~"DWR|CER"}[24h]))) * 100 >= 10
OID	1.3.6.1.4.1.323.5.3.52.1.2.82
Metric Used	occnp_stale_diam_request_cleanup_total
Recommended Actions	Cause: More than 10% of Diameter requests sent in the last 24 hours did not receive a response in time and were discarded by the cleanup mechanism. This indicates Diameter peers are slow to respond, unreachable, or the node is overloaded. Diagnostic Information: Check the affected pod and microservice for signs of overload or connectivity issues with Diameter peers. Review Diameter peer status, response times, and whether the cleanup rate is increasing over time. Exclude DWR/CER traffic from analysis as these are control-plane messages. Recovery: Verify connectivity and health of Diameter peers. If the node is overloaded, consider scaling the affected microservice. Investigate and resolve any network latency or timeout misconfigurations between the node and its Diameter peers.