BSF Alerts

Alert Levels / Severity Types	Definition
Critical	Indicates a severe issue that poses a significant risk to safety, security, or operational integrity. It requires immediate response to address the situation and prevent serious consequences. Raised for conditions can affect the service of BSF.
Major	Indicates a more significant issue that has an impact on operations or poses a moderate risk. It requires prompt attention and action to mitigate potential escalation. Raised for conditions can affect the service of BSF.
Minor	Indicates a situation that is low in severity and does not pose an immediate risk to safety, security, or operations. It requires attention but does not demand urgent action. Raised for conditions can affect the service of BSF.
Info or Warn (Informational)	Provides general information or updates that are not related to immediate risks or actions. These alerts are for awareness and do not typically require any specific response. WARN and INFO alerts may not impact the service of BSF.

5.1.1 AAA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 5-2 AAA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	AAA Rx fail count exceeds the critical threshold limit.
Summary	AAA Rx fail count exceeds the critical threshold limit.
Severity	CRITICAL
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="AAA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="AAA", direction="in"}[5m]))) 100 > 90
OID	1.3.6.1.4.1.323.5.3.37.1.2.40
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	For any additional guidance, contact My Oracle Support.

5.1.2 AAA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 5-3 AAA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	AAA Rx fail count exceeds the major threshold limit
Summary	AAA Rx fail count exceeds the major threshold limit.
Severity	MAJOR
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="AAA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="AAA", direction="in"}[5m]))) 100 > 80 <=90
OID	1.3.6.1.4.1.323.5.3.37.1.2.40
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	For any additional guidance, contact My Oracle Support.

5.1.3 AAA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 5-4 AAA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	AAA Rx fail count exceeds the minor threshold limit.
Summary	AAA Rx fail count exceeds the minor threshold limit.
Severity	MINOR
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="AAA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="AAA", direction="in"}[5m]))) 100 > 60 <=80
OID	1.3.6.1.4.1.323.5.3.37.1.2.40
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	For any additional guidance, contact My Oracle Support.

5.1.4 SCP_PEER_UNAVAILABLE

Table 5-5 SCP_PEER_UNAVAILABLE

Field	Details
Description	Configured SCP peer is unavailable.
Summary	SCP peer [ {{$labels.peer}} ] is unavailable.
Severity	Major
Expression	ocbsf_oc_egressgateway_peer_health_status == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.38
Metric Used	ocbsf_oc_egressgateway_peer_health_status
Recommended Actions	Cause: One or more SCP peer nodes are unavailable or unhealthy. This can affect service routing, request forwarding, and signaling reliability. Diagnostic Information: Identify unhealthy peers and verify network connectivity, health-check endpoints, heartbeat mechanisms, connection failures, timeouts, DNS resolution, IP addresses, ports, certificates, routing rules, load balancer or gateway behavior, TLS handshake failures, and whether multiple peers are affected. Recovery: Restore network connectivity, restart or recover unavailable peer services, correct endpoint, port, certificate, or routing configuration, resolve DNS or service-discovery issues, confirm health checks function correctly, fail over traffic to healthy peers where supported, and scale or redistribute load if peers are overloaded. Monitor until peer health returns to zero.

5.1.5 SCP_PEER_SET_UNAVAILABLE

Table 5-6 SCP_PEER_SET_UNAVAILABLE

Field	Details
Description	None of the SCP peer available for configured peerset.
Summary	{{ $value }} SCP peers under peer set {{$labels.peerset}} are currently unavailable.
Severity	Critical
Expression	(ocbsf_oc_egressgateway_peer_count > 0 and (ocbsf_oc_egressgateway_peer_available_count) == 0)
OID	1.3.6.1.4.1.323.5.3.37.1.2.39
Metric Used	ocbsf_oc_egressgateway_peer_*
Recommended Actions	Cause: One or more SCP peers in a configured peer set are unavailable. This reduces redundancy and can affect load balancing, failover capability, and service reliability. Diagnostic Information: Confirm the total and available peer counts, identify unavailable peers, and verify connectivity, latency, packet loss, health checks, heartbeat status, logs, TLS configuration, IP addresses, ports, routing policies, maintenance activity, scaling events, and crashes. Monitor the load on remaining peers. Recovery: Restore connectivity or restart unavailable peers. Correct endpoint, certificate, or routing configuration; resolve network or DNS issues; re-enable peers that were intentionally disabled; ensure suitable scaling and load distribution; validate health-check mechanisms; and monitor until all peers are healthy and stable.

5.1.6 BSF_SERVICES_DOWN

Table 5-7 BSF_SERVICES_DOWN

Field	Details
Description	{{$labels.service}} service is not running!
Summary	{{$labels.service}} is not running!
Severity	Critical
Expression	appinfo_service_running{application="ocbsf"} != 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.1
Metric Used	appinfo_service_running
Recommended Actions	Perform the following steps: Check for service specific alerts that may be causing the issues with service exposure. Verify if the POD is in a Running state by using the following command: `kubectl -n <namespace> get pod` If the output shows any pod that is not running, copy the pod name and run the following command: `kubectl describe pod <podname> -n <namespace>` Check the application logs on Kibana and look for database related failures such as connectivity, invalid secrets, and so on. The logs can be easily filtered for different services. Check for Helm status to ensure no errors are present by using the following command: `helm status <release-name> -n <namespace>` If it is not in `STATUS: DEPLOYED`, capture the logs and events again. In case the issue persists, capture the outputs for the preceding steps and contact My Oracle Support.

5.1.7 BSF_TRAFFIC_RATE_ABOVE_MINOR_THRESHOLD

Table 5-8 BSF_TRAFFIC_RATE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	BSF service Ingress traffic Rate is above threshold of Max MPS(1000) (current value is: {{ $value }}) The total Binding Management service Ingress traffic rate has crossed the configured threshold of 700 TPS. The default value of this alert trigger point in the `BSF_Alertrules.yaml` file is when the Binding management service Ingress Rate crosses 70% of maximum ingress requests per second.
Summary	Traffic Rate is above 70 Percent of Max requests per second(1000)
Severity	Minor
Expression	sum(rate(ocbsf_ingress_request_total[2m])) >= 700
OID	1.3.6.1.4.1.323.5.3.37.1.2.2
Metric Used	ocbsf_ingress_request_total
Recommended Actions	The alert gets cleared when the Ingress traffic rate falls below the threshold. Note: Threshold levels can be configured using the `BSF_Alertrules.yaml` file. It is recommended to assess the reason for additional traffic. Perform the following steps to analyze the cause of increased traffic: Refer Ingress Gateway section in Grafana to determine an increase in 4xx and 5xx error response codes. Check Ingress Gateway logs on Kibana to determine the reason for the errors. For any assistance, contact My Oracle Support.

5.1.8 BSF_TRAFFIC_RATE_ABOVE_MAJOR_THRESHOLD

Table 5-9 BSF_TRAFFIC_RATE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	BSF service Ingress traffic Rate is above threshold of Max MPS(1000) (current value is: {{ $value }})
Summary	Traffic Rate is above 80 Percent of Max requests per second(1000)
Severity	Major
Expression	sum(rate(ocbsf_ingress_request_total[2m])) >= 800
OID	1.3.6.1.4.1.323.5.3.37.1.2.2
Metric Used	ocbsf_ingress_request_total
Recommended Actions	The alert gets cleared when the Ingress traffic rate falls below the threshold. Note: Threshold levels can be configured using the `BSF_Alertrules.yaml` file. It is recommended to assess the reason for additional traffic. Perform the following steps to analyze the cause of increased traffic: Refer Ingress Gateway section in Grafana to determine an increase in 4xx and 5xx error response codes. Check Ingress Gateway logs on Kibana to determine the reason for the errors. For any assistance, contact My Oracle Support.

5.1.9 BSF_TRAFFIC_RATE_ABOVE_CRITICAL_THRESHOLD

Table 5-10 BSF_TRAFFIC_RATE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	BSF service Ingress traffic Rate is above threshold of Max MPS(1000) (current value is: {{ $value }})
Summary	Traffic Rate is above 90 Percent of Max requests per second(1000)
Severity	Critical
Expression	sum(rate(ocbsf_ingress_request_total[2m])) >= 900
OID	1.3.6.1.4.1.323.5.3.37.1.2.2
Metric Used	ocbsf_ingress_request_total
Recommended Actions	The alert gets cleared when the Ingress traffic rate falls below the threshold. Note: Threshold levels can be configured using the `BSF_Alertrules.yaml` file. It is recommended to assess the reason for additional traffic. Perform the following steps to analyze the cause of increased traffic: Refer Ingress Gateway section in Grafana to determine an increase in 4xx and 5xx error response codes. Check Ingress Gateway logs on Kibana to determine the reason for the errors. For any assistance, contact My Oracle Support.

5.1.10 BINDING_QUERY_RESPONSE_ERROR_MINOR

Table 5-11 BINDING_QUERY_RESPONSE_ERROR_MINOR

Field	Details
Description	At least 30% of the Binding Query connection requests failed.
Summary	At least 30% of the Binding Query requests failed.
Severity	Minor
Expression	(sum(rate(ocbsf_binding_query_response_total{response_code!~"2."}[10m]) or (appinfo_service_running 0 ) ) / sum(rate(ocbsf_binding_query_response_total[10m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.37.1.2.36
Metric Used	ocbsf_binding_query_response_total.
Recommended Actions	For any assistance, contact My Oracle Support.

5.1.11 BINDING_QUERY_RESPONSE_ERROR_MAJOR

Table 5-12 BINDING_QUERY_RESPONSE_ERROR_MAJOR

Field	Details
Description	At least 50% of the Binding Query connection requests failed.
Summary	At least 50% of the Binding Query requests failed.
Severity	Major
Expression	(sum(rate(ocbsf_binding_query_response_total{response_code!~"2."}[10m]) or (appinfo_service_running 0 ) ) / sum(rate(ocbsf_binding_query_response_total[10m]))) * 100 >= 50
OID	1.3.6.1.4.1.323.5.3.37.1.2.36
Metric Used	ocbsf_binding_query_response_total.
Recommended Actions	For any assistance, contact My Oracle Support.

5.1.12 BINDING_QUERY_RESPONSE_ERROR_CRITICAL

Table 5-13 BINDING_QUERY_RESPONSE_ERROR_CRITICAL

Field	Details
Description	At least 70% of the Binding Query connection requests failed.
Summary	At least 70% of the Binding Query requests failed.
Severity	Critical
OID	1.3.6.1.4.1.323.5.3.37.1.2.36
Expression	(sum(rate(ocbsf_binding_query_response_total{response_code!~"2."}[10m]) or (appinfo_service_running 0 ) ) / sum(rate(ocbsf_binding_query_response_total[10m]))) * 100 >= 70
Metric Used	ocbsf_binding_query_response_total.
Recommended Actions	For any assistance, contact My Oracle Support.

5.1.13 DIAM_RESPONSE_NETWORK_ERROR_MINOR

Table 5-14 DIAM_RESPONSE_NETWORK_ERROR_MINOR

Field	Details
Description	At least 20% of the Diam Response connection requests failed with error 'DIAMETER_UNABLE_TO_DELIVER'.
Summary	At least 20% of the Diam Response requests failed with error 'DIAMETER_UNABLE_TO_DELIVER'.
Severity	Minor
Expression	(sum(rate(ocbsf_diam_response_network_total{responseCode="3002"}[10m]) or (appinfo_service_running * 0 ) ) / sum(rate(ocbsf_diam_response_network_total[10m]))) * 100 >= 20
OID	1.3.6.1.4.1.323.5.3.37.1.2.35
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: At least one pod is reporting an elevated rate of Diameter result code 3002, `DIAMETER_UNABLE_TO_DELIVER`. This typically indicates routing or peer-selection issues, an unreachable destination realm or host, a peer connection outage, an invalid or missing route, or transient network issues. It can also occur during rolling restarts when peer tables or routes are unstable. Diagnostic Information: Identify the worst affected pod and list all affected pods. Compare the 3002 response rate with total Diameter response volume to distinguish a true volume issue from a ratio artifact. Break down failures by available routing labels, such as destination host, destination realm, peer, application ID, or message type. Correlate the condition with peer connection state, disconnects, and recent deployment or restart events for the affected pod. Recovery: If the issue is isolated to one pod, drain or replace it after confirming that there is no broader peer outage, then validate routing tables and peer connections. If it is isolated to a destination realm or host, correct routing configuration, realm definitions, and peer reachability. If peer downtime is responsible, fail over or reroute traffic to alternate peers where supported. Confirm that the 3002 response rate returns to baseline and successful 2xx responses dominate.

5.1.14 DIAM_RESPONSE_NETWORK_ERROR_MAJOR

Table 5-15 DIAM_RESPONSE_NETWORK_ERROR_MAJOR

Field	Details
Description	At least 50% of the Diam Response connection requests failed with error 'DIAMETER_UNABLE_TO_DELIVER'.
Summary	At least 50% of the Diam Response requests failed with error 'DIAMETER_UNABLE_TO_DELIVER'.
Severity	Major
Expression	(sum(rate(ocbsf_diam_response_network_total{responseCode="3002"}[10m]) or (appinfo_service_running * 0 ) ) / sum(rate(ocbsf_diam_response_network_total[10m]))) * 100 >= 50
OID	1.3.6.1.4.1.323.5.3.37.1.2.35
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: Significant `DIAMETER_UNABLE_TO_DELIVER` failures are occurring and are likely to affect sessions or transactions. Diagnostic Information: Perform the same checks as for the minor alert and determine whether multiple pods are affected. Confirm the number of pods whose 3002 response ratio is at or above 5%. Recovery: Prioritize restoration of reachability and routing. Fail over peers, correct realm or route configuration, and restart or replace pods if the issue is localized. Treat the condition as an incident if it persists.

5.1.15 DIAM_RESPONSE_NETWORK_ERROR_CRITICAL

Table 5-16 DIAM_RESPONSE_NETWORK_ERROR_CRITICAL

Field	Details
Description	At least 75% of the Diam Response connection requests failed with error 'DIAMETER_UNABLE_TO_DELIVER'.
Summary	At least 75% of the Diam Response requests failed with error 'DIAMETER_UNABLE_TO_DELIVER'.
Severity	Critical
Expression	(sum(rate(ocbsf_diam_response_network_total{responseCode="3002"}[10m]) or (appinfo_service_running * 0 ) ) / sum(rate(ocbsf_diam_response_network_total[10m]))) * 100 >= 75
OID	1.3.6.1.4.1.323.5.3.37.1.2.35
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: A severe `DIAMETER_UNABLE_TO_DELIVER` condition is occurring. This typically indicates a major peer or routing outage, widespread disconnects, or an incorrect configuration rollout. Diagnostic Information: Urgently identify the dominant destination or destinations and determine whether the condition correlates with connection-failure or peer-down alerts. Recovery: Immediately restore routing or peer connectivity, or roll back recent routing or configuration changes. Fail traffic over to healthy peers or realms and replace unhealthy pods if the issue is pod-local. Validate recovery when the 3002 error rate falls well below 10% and end-to-end transaction success recovers.

5.1.16 DUPLICATE_BINDING_REQUEST_ERROR_MINOR

Table 5-17 DUPLICATE_BINDING_REQUEST_ERROR_MINOR

Field	Details
Description	At least 30% of the Binding Registration requests failed were duplicate failures.
Summary	At least 30% of the Binding Registration requests failed were duplicate failures.
Severity	Minor
Expression	(sum(rate({__name__=~"ocbsf_collision_detection."}[10m]) or (appinfo_service_running 0)) / sum(rate(ocbsf_ingress_request_total{operation_type="register"}[10m]))) * 100 >= 30
OID	1.3.6.1.4.1.323.5.3.37.1.2.37
Metric Used	ocbsf_ingress_request_total ocbsf_collision_detection
Recommended Actions	For any assistance, contact My Oracle Support.

5.1.17 DUPLICATE_BINDING_REQUEST_ERROR_MAJOR

Table 5-18 DUPLICATE_BINDING_REQUEST_ERROR_MAJOR

Field	Details
Description	At least 50% of the Binding Registration requests failed were duplicate failures.
Summary	At least 50% of the Binding Registration requests failed were duplicate failures.
Severity	Major
Expression	(sum(rate({__name__=~"ocbsf_collision_detection."}[10m]) or (appinfo_service_running 0)) / sum(rate(ocbsf_ingress_request_total{operation_type="register"}[10m]))) * 100 >= 50
OID	1.3.6.1.4.1.323.5.3.37.1.2.37
Metric Used	ocbsf_ingress_request_total ocbsf_collision_detection
Recommended Actions	For any assistance, contact My Oracle Support.

5.1.18 DUPLICATE_BINDING_REQUEST_ERROR_CRITICAL

Table 5-19 DUPLICATE_BINDING_REQUEST_ERROR_CRITICAL

Field	Details
Description	At least 70% of the Binding Registration requests failed were duplicate failures.
Summary	At least 70% of the Binding Registration requests failed were duplicate failures.
Severity	Critical
Expression	(sum(rate({__name__=~"ocbsf_collision_detection."}[10m]) or (appinfo_service_running 0)) / sum(rate(ocbsf_ingress_request_total{operation_type="register"}[10m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.37.1.2.37
Metric Used	ocbsf_ingress_request_total ocbsf_collision_detection
Recommended Actions	For any assistance, contact My Oracle Support.

5.1.19 INGRESS_TOTAL_ERROR_RATE_ABOVE_MINOR_THRESHOLD

Table 5-20 INGRESS_TOTAL_ERROR_RATE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	Transaction Error Rate detected above 1 Percent of Total on BSF service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 1 Percent of Total Transactions
Severity	Minor
Expression	(sum(rate(ocbsf_ingress_response_total{response_code!~"2."}[24h])) / sum(rate(ocbsf_ingress_response_total[24h]))) 100 >= 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.3
Metric Used	ocbsf_ingress_response_total
Recommended Actions	The alert gets cleared when the number of failed transactions is below 1% of the total transactions. For any assistance, contact My Oracle Support.

5.1.20 INGRESS_TOTAL_ERROR_RATE_ABOVE_MAJOR_THRESHOLD

Table 5-21 INGRESS_TOTAL_ERROR_RATE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	Transaction Error Rate detected above 5 Percent of Total on BSF service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 5 Percent of Total Transactions
Severity	Major
Expression	(sum(rate(ocbsf_ingress_response_total{response_code!~"2."}[24h])) / sum(rate(ocbsf_ingress_response_total[24h]))) 100 >= 5
OID	1.3.6.1.4.1.323.5.3.37.1.2.3
Metric Used	ocbsf_ingress_response_total
Recommended Actions	The alert gets cleared when the number of failed transactions is below 5% of the total transactions. For any assistance, contact My Oracle Support.

5.1.21 INGRESS_TOTAL_ERROR_RATE_ABOVE_CRITICAL_THRESHOLD

Table 5-22 INGRESS_TOTAL_ERROR_RATE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	Transaction Error Rate detected above 10 Percent of Total on BSF service (current value is: {{ $value }})
Summary	Transaction Error Rate detected above 10 Percent of Total Transactions
Severity	Critical
Expression	(sum(rate(ocbsf_ingress_response_total{response_code!~"2."}[24h])) / sum(rate(ocbsf_ingress_response_total[24h]))) 100 >= 10
OID	1.3.6.1.4.1.323.5.3.37.1.2.3
Metric Used	ocbsf_ingress_response_total
Recommended Actions	The alert gets cleared when the number of failed transactions is below 10% of the total transactions. For any assistance, contact My Oracle Support.

5.1.22 PCF_BINDING_ERROR_RATE_ABOVE_MINOR_THRESHOLD

Table 5-23 PCF_BINDING_ERROR_RATE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	PCF Binding Error Rate above 1 Percent in {{$labels.microservice}} in {{$labels.namespace}}
Summary	PCF Binding Error Rate in {{$labels.kubernetes_node}} (current value is: {{ $value }})
Severity	Minor
Expression	(sum by (microservice,namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service", status!~"200\|204",method="GET"}[24h])) / sum by (microservice,namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service",method="GET"}[24h]))) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.5
Metric Used	http_server_requests_seconds_count
Recommended Actions	The alert gets cleared when the number of failed transactions is below 1% of the total transactions. To assess the reason for failed transactions, check the service specific metrics for the GET method. For any assistance, contact My Oracle Support.

5.1.23 PCF_BINDING_ERROR_RATE_ABOVE_MAJOR_THRESHOLD

Table 5-24 PCF_BINDING_ERROR_RATE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	PCF Binding Error Rate above 5 Percent in {{$labels.microservice}} in {{$labels.namespace}}
Summary	PCF Binding Error Rate in {{$labels.kubernetes_node}} (current value is: {{ $value }})
Severity	Major
Expression	(sum by (microservice,namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service", status!~"200\|204",method="GET"}[24h])) / sum by (microservice,namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service",method="GET"}[24h]))) * 100 >= 5
OID	1.3.6.1.4.1.323.5.3.37.1.2.5
Metric Used	http_server_requests_seconds_count
Recommended Actions	The alert gets cleared when the number of failed transactions is below 5% of the total transactions. To assess the reason for failed transactions, check the service specific metrics for the GET method. For any assistance, contact My Oracle Support.

5.1.24 INGRESS_CREATE_ERROR_RATE_ABOVE_MINOR_THRESHOLD

Table 5-25 INGRESS_CREATE_ERROR_RATE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	BSF Ingress Create Error Rate above 1 Percent in {{$labels.microservice}} in {{$labels.namespace}}
Summary	Transaction Create Error Rate in {{$labels.kubernetes_node}} (current value is: {{ $value }})
Severity	Minor
Expression	sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service", status!~"200\|201",method="POST"}[24h])) / sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service",method="POST"}[24h])) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.4
Metric Used	http_server_requests_seconds_count
Recommended Actions	The alert gets cleared when the number of failed transactions is below 1% of the total transactions. To assess the reason for failed transactions, check the service specific metrics for the POST method. For any assistance, contact My Oracle Support.

5.1.25 INGRESS_CREATE_ERROR_RATE_ABOVE_CRITICAL_THRESHOLD

Table 5-26 INGRESS_CREATE_ERROR_RATE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	BSF Ingress Create Error Rate above 10 Percent in {{$labels.microservice}} in {{$labels.namespace}}
Summary	Transaction Create Error Rate in {{$labels.kubernetes_node}} (current value is: {{ $value }})
Severity	Critical
Expression	sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service", status!~"200\|201",method="POST"}[24h])) / sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service",method="POST"}[24h])) * 100 >= 10
OID	1.3.6.1.4.1.323.5.3.37.1.2.4
Metric Used	http_server_requests_seconds_count
Recommended Actions	The alert gets cleared when the number of failed transactions is below 10% of the total transactions. To assess the reason for failed transactions, check the service specific metrics for the POST method. For any assistance, contact My Oracle Support.

5.1.26 INGRESS_CREATE_ERROR_RATE_ABOVE_MAJOR_THRESHOLD

Table 5-27 INGRESS_CREATE_ERROR_RATE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	BSF Ingress Create Error Rate above 5 Percent in {{$labels.microservice}} in {{$labels.namespace}}
Summary	Transaction Create Error Rate in {{$labels.kubernetes_node}} (current value is: {{ $value }})
Severity	Major
Expression	sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service", status!~"200\|201",method="POST"}[24h])) / sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service",method="POST"}[24h])) * 100 >= 5
OID	1.3.6.1.4.1.323.5.3.37.1.2.4
Metric Used	http_server_requests_seconds_count
Recommended Actions	The alert gets cleared when the number of failed transactions is below 5% of the total transactions. To assess the reason for failed transactions, check the service specific metrics for the POST method. For any assistance, contact My Oracle Support.

5.1.27 INGRESS_DELETE_ERROR_RATE_ABOVE_MINOR_THRESHOLD

Table 5-28 INGRESS_DELETE_ERROR_RATE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	Ingress Delete Error Rate above 1 Percent in {{$labels.microservice}} in {{$labels.namespace}}
Summary	Ingress Delete Error Rate in {{$labels.kubernetes_node}} (current value is: {{ $value }})
Severity	Minor
Expression	sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service", status!="204",method="DELETE", uri="/nbsf-management/v1/pcfBindings/{bindingId}"}[24h])) / sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service",method="DELETE", uri="/nbsf-management/v1/pcfBindings/{bindingId}"}[24h])) * 100 >= 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.6
Metric Used	http_server_requests_seconds_count
Recommended Actions	The alert gets cleared when the number of failed transactions is below 1% of the total transactions. To assess the reason for failed transactions, check the service specific metrics for the DELETE method. For any assistance, contact My Oracle Support.

5.1.28 INGRESS_DELETE_ERROR_RATE_ABOVE_MAJOR_THRESHOLD

Table 5-29 INGRESS_DELETE_ERROR_RATE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	Ingress Delete Error Rate above 5 Percent in {{$labels.microservice}} in {{$labels.namespace}}
Summary	Ingress Delete Error Rate in {{$labels.kubernetes_node}} (current value is: {{ $value }})
Severity	Major
Expression	sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service", status!="204",method="DELETE", uri="/nbsf-management/v1/pcfBindings/{bindingId}"}[24h])) / sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service",method="DELETE", uri="/nbsf-management/v1/pcfBindings/{bindingId}"}[24h])) * 100 >= 5
OID	1.3.6.1.4.1.323.5.3.37.1.2.6
Metric Used	http_server_requests_seconds_count
Recommended Actions	The alert gets cleared when the number of failed transactions is below 5% of the total transactions. To assess the reason for failed transactions, check the service specific metrics for the DELETE method. For any assistance, contact My Oracle Support.

5.1.29 INGRESS_DELETE_ERROR_RATE_ABOVE_CRITICAL_THRESHOLD

Table 5-30 INGRESS_DELETE_ERROR_RATE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	Ingress Delete Error Rate above 10 Percent in {{$labels.microservice}} in {{$labels.namespace}}
Summary	Ingress Delete Error Rate in {{$labels.kubernetes_node}} (current value is: {{ $value }})
Severity	Critical
Expression	sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service", status!="204",method="DELETE", uri="/nbsf-management/v1/pcfBindings/{bindingId}"}[24h])) / sum by(namespace)(rate(http_server_requests_seconds_count{microservice="bsf-management-service",method="DELETE", uri="/nbsf-management/v1/pcfBindings/{bindingId}"}[24h])) * 100 >= 10
OID	1.3.6.1.4.1.323.5.3.37.1.2.6
Metric Used	http_server_requests_seconds_count
Recommended Actions	The alert gets cleared when the number of failed transactions is below 10% of the total transactions. To assess the reason for failed transactions, check the service specific metrics for the DELETE method. For any assistance, contact My Oracle Support.

5.1.30 DB_TIER_DOWN_ALERT

Table 5-31 DB_TIER_DOWN_ALERT

Field	Details
Description	DB cannot be reachable!
Summary	DB cannot be reachable!
Severity	Critical
Expression	appinfo_category_running{category="database", application="ocbsf"} != 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.7
Metric Used	appinfo_category_running
Recommended Actions	Cause: The database tier is unavailable or unreachable, as indicated by `appinfo_category_running{category="database"} != 1`. Diagnostic Information: Verify database-pod status, readiness, events, network connectivity, database logs, storage and PVC health, and the `appinfo_category_running` metric. Recovery: Restore the failed database component or connectivity, resolve storage or configuration issues, and confirm that the database category reports as running again.

5.1.31 CPU_USAGE_PER_SERVICE_ABOVE_MINOR_THRESHOLD

Table 5-32 CPU_USAGE_PER_SERVICE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	CPU usage for {{$labels.microservice}} service is above 60
Summary	CPU usage for {{$labels.microservice}} service is above 60
Severity	Minor
Expression	avg by (namespace,service) (avg_over_time(cgroup_cpu_usage{application="ocbsf"}[2m])) >= 60 < 80
OID	1.3.6.1.4.1.323.5.3.37.1.2.8
Metric Used	cgroup_cpu_usage
Recommended Actions	Cause: CPU utilization for a service exceeds the configured minor threshold over the defined time window. This typically indicates increased processing load due to higher traffic, inefficient code paths, resource-intensive operations, traffic spikes, background jobs, tight loops, thread contention, or misconfigured CPU limits or requests. Diagnostic Information: Check CPU-usage graphs to determine whether utilization is consistently high or intermittent. Correlate CPU usage with traffic patterns, batch jobs, and recent deployments. Check pod metrics for CPU throttling, restarts, or abnormal usage. Review latency metrics, upstream-service logs for increased egress traffic or retries, service logs for high-latency operations or repeated loops, CPU limits and requests, and node-level CPU contention. Recovery: If CPU usage remains high, optimize application logic, reduce unnecessary processing, or tune configuration. If traffic spikes are responsible, scale out the service. Cautiously adjust CPU limits or requests if required, ensuring cluster stability, and roll back recent changes if the issue started after a deployment. The alert clears when CPU usage falls below the configured threshold.

5.1.32 CPU_USAGE_PER_SERVICE_ABOVE_MAJOR_THRESHOLD

Table 5-33 CPU_USAGE_PER_SERVICE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	CPU usage for {{$labels.microservice}} service is above 80
Summary	CPU usage for {{$labels.microservice}} service is above 80
Severity	Major
Expression	avg by (namespace,service) (avg_over_time(cgroup_cpu_usage{application="ocbsf"}[2m])) >= 80 < 90
OID	1.3.6.1.4.1.323.5.3.37.1.2.9
Metric Used	cgroup_cpu_usage
Recommended Actions	Cause: CPU utilization for a service exceeds the configured major threshold of 80% over the defined time window. This can indicate sustained high traffic, inefficient processing, tight loops, thread contention, retry storms, workload spikes, downstream latency, or misconfigured CPU limits or requests. Diagnostic Information: Check CPU-usage graphs to confirm sustained utilization above 80%. Correlate CPU usage with traffic patterns, batch jobs, and recent deployments. Check pod metrics for CPU throttling, restarts, and saturation indicators. Review latency metrics, upstream-service logs for increased egress traffic or retries, service logs for CPU-intensive operations or loops, CPU limits and requests, and node-level CPU contention. Recovery: Scale out the service to reduce CPU load per pod. Optimize application logic or disable non-critical processing, mitigate retry storms or abnormal traffic, cautiously adjust CPU limits or requests, and roll back recent deployments if applicable. The alert clears when CPU usage falls below the configured major threshold.

5.1.33 CPU_USAGE_PER_SERVICE_ABOVE_CRITICAL_THRESHOLD

Table 5-34 CPU_USAGE_PER_SERVICE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	CPU usage for {{$labels.microservice}} service is above 90
Summary	CPU usage for {{$labels.microservice}} service is above 90
Severity	Critical
Expression	avg by (namespace,service) (avg_over_time(cgroup_cpu_usage{application="ocbsf"}[2m])) >= 90
OID	1.3.6.1.4.1.323.5.3.37.1.2.10
Metric Used	cgroup_cpu_usage
Recommended Actions	Cause: CPU utilization for a service exceeds the configured critical threshold over the defined time window. This indicates severe resource saturation and a risk of service degradation or unresponsiveness. Typical causes include sustained high traffic, inefficient processing, tight loops, thread contention, misconfigured CPU limits or requests, retry storms, workload spikes, or downstream latency. Diagnostic Information: Check CPU-usage graphs to confirm sustained critical utilization. Correlate CPU usage with traffic patterns, batch jobs, and recent deployments. Check pod metrics for CPU throttling, restarts, and saturation indicators. Review latency metrics, upstream-service logs for increased egress traffic or retries, service logs for CPU-intensive operations or loops, CPU limits and requests, and node-level CPU contention. Recovery: Scale out the service immediately to reduce CPU load per pod. If required, perform a rolling restart to stabilize pods under extreme load. Optimize application logic or disable non-critical processing, mitigate retry storms or abnormal traffic, cautiously adjust CPU limits or requests, and roll back recent deployments if applicable. The alert clears when CPU usage falls below the configured critical threshold.

5.1.34 MEMORY_USAGE_PER_SERVICE_ABOVE_MINOR_THRESHOLD

Table 5-35 MEMORY_USAGE_PER_SERVICE_ABOVE_MINOR_THRESHOLD

Field	Details
Description	Memory usage for {{$labels.microservice}} service is above 60
Summary	Memory usage for {{$labels.microservice}} service is above 60
Severity	Minor
Expression	avg by (namespace,service) (avg_over_time(cgroup_memory_usage{application="ocbsf"}[2m])) >= 60 < 80
OID	1.3.6.1.4.1.323.5.3.37.1.2.11
Metric Used	cgroup_memory_usage
Recommended Actions	Cause: Service memory usage remains at or above 60% for the configured window. This can indicate sustained memory growth, repeated traffic or payload spikes, cache expansion, a memory leak in the PRE service, recent deployment changes, higher concurrency, or a misconfigured memory limit. Diagnostic Information: Check memory graphs for the PRE service to determine whether usage is consistently high or only spiking. Review whether the alert aligns with traffic bursts, deployment changes, or a particular job or request pattern. Recovery: Scale out the service or increase memory limits only after confirming the trend and ruling out a memory leak or abnormal workload.

5.1.35 MEMORY_USAGE_PER_SERVICE_ABOVE_MAJOR_THRESHOLD

Table 5-36 MEMORY_USAGE_PER_SERVICE_ABOVE_MAJOR_THRESHOLD

Field	Details
Description	Memory usage for {{$labels.microservice}} service is above 80
Summary	Memory usage for {{$labels.microservice}} service is above 80
Severity	Major
Expression	avg by (namespace,service) (avg_over_time(cgroup_memory_usage{application="ocbsf"}[2m])) >= 80 < 90
OID	1.3.6.1.4.1.323.5.3.37.1.2.12
Metric Used	cgroup_memory_usage
Recommended Actions	Cause: Service memory usage remains at or above 80% for the configured window. This can indicate sustained memory growth, repeated traffic or payload spikes, cache expansion, a memory leak in the PRE service, recent deployment changes, higher concurrency, or a misconfigured memory limit. Diagnostic Information: Check memory graphs for the PRE service to determine whether usage is consistently high or only spiking. Review whether the alert aligns with traffic bursts, deployment changes, or a particular job or request pattern. Recovery: Scale out the service or increase memory limits only after confirming the trend and ruling out a memory leak or abnormal workload.

5.1.36 MEMORY_USAGE_PER_SERVICE_ABOVE_CRITICAL_THRESHOLD

Table 5-37 MEMORY_USAGE_PER_SERVICE_ABOVE_CRITICAL_THRESHOLD

Field	Details
Description	Memory usage for {{$labels.microservice}} service is above 90
Summary	Memory usage for {{$labels.microservice}} service is above 90
Severity	Critical
Expression	avg by (namespace,service) (avg_over_time(cgroup_memory_usage{application="ocbsf"}[2m])) >= 90
OID	1.3.6.1.4.1.323.5.3.37.1.2.13
Metric Used	cgroup_memory_usage
Recommended Actions	Cause: Service memory usage remains at or above 90% for the configured window. This can indicate sustained memory growth, repeated traffic or payload spikes, cache expansion, a memory leak in the PRE service, recent deployment changes, higher concurrency, or a misconfigured memory limit. Diagnostic Information: Check memory graphs for the PRE service to determine whether usage is consistently high or only spiking. Review whether the alert aligns with traffic bursts, deployment changes, or a particular job or request pattern. Recovery: If memory usage reaches approximately 90% of the defined threshold, perform a rolling restart of PRE pods to relieve memory pressure. Investigate and correct underlying causes, such as leaks, cache growth, or abnormal traffic patterns. If the issue persists or the root cause is unclear, contact Oracle Support for further analysis.

5.1.37 NRF_COMMUNICATION_FAILURE

Table 5-38 NRF_COMMUNICATION_FAILURE

Field	Details
Description	There has been an external failure communication error with NRF.
Summary	There has been an external failure communication error with NRF.
Severity	Critical
Expression	ocbsf_nrfclient_nrf_operative_status == 0
OID	1.3.6.1.4.1.323.5.3.37.1.2.33
Metric Used	ocbsf_nrfclient_nrf_operative_status
Recommended Actions	For any assistance, contact My Oracle Support.

5.1.38 NRF_SERVICE_REQUEST_FAILURE

Table 5-39 NRF_SERVICE_REQUEST_FAILURE

Field	Details
Description	There has been a Service Request Failure with NRF, either a Registration failure, Heartbeat failure, or Profile Update Failure.
Summary	There has been a Service Request Failure with NRF, either a Registration failure, Heartbeat failure, or Profile Update Failure.
Severity	Critical
Expression	ocbsf_nrfclient_nfUpdate_status == 0
OID	1.3.6.1.4.1.323.5.3.37.1.2.34
Metric Used	ocbsf_nrfclient_nfUpdate_status
Recommended Actions	For any assistance, contact My Oracle Support.

5.1.39 PERF_INFO_ACTIVE_OVERLOAD_THRESHOLD_FETCH_FAILED

Table 5-40 PERF_INFO_ACTIVE_OVERLOAD_THRESHOLD_FETCH_FAILED

Field	Details
Description	The application fails to get the current active overload level threshold data.
Summary	The application raises `PERF_INFO_ACTIVE_OVERLOAD_THRESHOLD_FETCH_FAILED` alert when it fails to fetch the current active overload level threshold data and `active_overload_threshold_fetch_failed == 1`.
Severity	Major
Expression	active_overload_threshold_fetch_failed == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.20
Metric Used	active_overload_threshold_fetch_failed
Recommended Actions	Cause: The system cannot fetch active overload-threshold data, as indicated by `active_overload_threshold_fetch_failed == 1`. Overload-protection parameters might be unavailable or incorrectly loaded, affecting the system’s ability to handle traffic surges safely. Diagnostic Information: Verify connectivity to the configuration source, such as the database, configuration service, or NMS. Confirm that overload-threshold data is configured and available. Review logs for fetch failures or timeouts, validate credentials and access permissions, inspect recent configuration or deployment changes, confirm that dependent services are available, and check for initialization issues during service startup. Recovery: Restore connectivity to the configuration or data source. Correct missing or invalid overload-threshold configuration, restart or reinitialize the component that retrieves the data, resolve authentication or authorization failures, and roll back recent changes that affect configuration retrieval. Confirm high availability of configuration services and monitor until the metric returns to zero. If the issue persists, contact Oracle Support.

5.1.40 POD_DOC

Table 5-41 POD_DOC

Field	Details
Description	Pod Congestion status of {{$labels.microservice}} service is DoC
Summary	Pod Congestion status of {{$labels.microservice}} service is DoC
Severity	Major
Expression	ocbsf_pod_congestion_state == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.25
Metric Used	ocbsf_pod_congestion_state
Recommended Actions	Cause: The pod entered DANGER_OF_CONGESTION (DOC) due to rising CPU and/or queue close to configured limits. Diagnostic Information: Check pod_congestion_state == 1; review pod_resource_stress (cpu and queue) and pod_cong_state_report_total to see recent transitions. Recovery: Confirm the DOC thresholds and active Load Shedding rule. If DOC is triggered by brief spikes, increase stateChangeSampleCount or the calculation interval. If sustained, consider slightly increasing discard aggressiveness for low-value calls per policy.

5.1.41 POD_CONGESTED

Table 5-42 POD_CONGESTED

Field	Details
Description	Pod Congestion status of {{$labels.microservice}} service is congested
Summary	Pod Congestion status of {{$labels.microservice}} service is congested
Severity	Critical
Expression	ocbsf_pod_congestion_state==4
OID	1.3.6.1.4.1.323.5.3.37.1.2.26
Metric Used	ocbsf_pod_congestion_state
Recommended Actions	Cause: The pod has entered the `CONGESTED` state because CPU consumption or the pending-request queue exceeds the active threshold profile. Diagnostic Information: Check `pod_congestion_state` and confirm a value of 4. Review `pod_resource_congestion_state` for CPU and queue conditions, `pod_resource_stress`, and `ocbsf_http_congestion_message_reject_total`, filtered by congestion state, request URI, request method, and priority. Recovery: In the CM GUI, navigate to BSF, Overload and Congestion Control, and Congestion Control. Confirm that the feature is enabled and that the intended Thresholds and Load Shedding profiles are active. If rejections are excessive, raise the discard priority for the current state or relax thresholds according to performance baselines. Consider increasing `stateChangeSampleCount` or the calculation interval to reduce flapping caused by short spikes.

5.1.42 POD_CONGESTION_L1

Table 5-43 POD_CONGESTION_L1

Field	Details
Description	Pod Congestion status of {{$labels.microservice}} service is Congestion_L1.
Summary	Pod Congestion status of {{$labels.microservice}} service is Congestion_L1.
Severity	Critical
Expression	ocbsf_pod_congestion_state == 2
OID	1.3.6.1.4.1.323.5.3.37.1.2.52
Metric Used	ocbsf_pod_congestion_state
Recommended Actions	Cause: The pod reached `CONGESTION_L1` based on CPU or queue thresholds in the active profile. Diagnostic Information: Confirm that `pod_congestion_state == 2`. Identify the driver by reviewing `pod_resource_congestion_state` for CPU or queue conditions, and review `ocbsf_http_congestion_message_reject_total` with `congestionState=CONGESTION_L1`. Recovery: Confirm the L1 discard priority, which defaults to 24, and the configured thresholds. If important calls are being dropped, adjust the discard priority or tune thresholds to match the expected load profile.

5.1.43 POD_CPU_CONGESTION_L1

Table 5-44 POD_CPU_CONGESTION_L1

Field	Details
Description	Pod resource is in Congestion_L1 for CPU type.
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L1 for CPU type.
Severity	Critical
Expression	ocbsf_pod_resource_congestion_state{type="cpu"} == 2
OID	1.3.6.1.4.1.323.5.3.37.1.2.54
Metric Used	ocbsf_pod_resource_congestion_state
Recommended Actions	Cause: CPU utilization has reached `CONGESTION_L1`. Diagnostic Information: Confirm that `pod_resource_congestion_state{resourceType="cpu"} == 2`. Check CPU stress and state transitions. Recovery: Validate the L1 CPU thresholds and discard priority. If short spikes cause churn, increase `stateChangeSampleCount`; otherwise, increase shedding at L1.

5.1.44 POD_CONGESTION_L2

Table 5-45 POD_CONGESTION_L2

Field	Details
Description	Pod Congestion status of {{$labels.microservice}} service is Congestion_L2
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L2.
Severity	Critical
Expression	ocbsf_pod_congestion_state == 3
OID	1.3.6.1.4.1.323.5.3.37.1.2.53
Metric Used	ocbsf_pod_congestion_state
Recommended Actions	Cause: The pod reached `CONGESTION_L2`, indicating higher stress than L1. Diagnostic Information: Confirm that `pod_congestion_state == 3`. Validate resource-specific congestion states and stress metrics, and inspect rejection counters at L2. Recovery: Use the L2 discard priority, which defaults to 18, to shed more low-priority traffic. Consider tuning thresholds and sample counts to balance protection and availability.

5.1.45 POD_CPU_CONGESTION_L2

Table 5-46 POD_CPU_CONGESTION_L2

Field	Details
Description	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L2 for CPU type.
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L2 for CPU type.
Severity	Critical
Expression	ocbsf_pod_resource_congestion_state{type="cpu"} == 3
OID	1.3.6.1.4.1.323.5.3.37.1.2.55
Metric Used	ocbsf_pod_resource_congestion_state
Recommended Actions	Cause: CPU utilization has reached `CONGESTION_L2`. Diagnostic Information: Confirm that `pod_resource_congestion_state{resourceType="cpu"} == 3`. Check CPU stress, EMA interval and ratio, and L2 rejection counters. Recovery: Raise the L2 discard priority to protect the pod. Tune CPU thresholds or EMA cadence only after comparison with test baselines.

5.1.46 POD_PENDING_REQUEST_DOC

Table 5-47 POD_PENDING_REQUEST_DOC

Field	Details
Description	Pod Resource Congestion status of {{$labels.microservice}} service is DoC for PendingRequest type
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is DoC for PendingRequest type
Severity	Major
Expression	ocbsf_pod_resource_congestion_state{type="queue"} == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.27
Metric Used	ocbsf_pod_resource_congestion_state{type="queue"}
Recommended Actions	Cause: The pending request queue is in the `DANGER_OF_CONGESTION` state. Diagnostic Information: Confirm that `pod_resource_congestion_state{resourceType="queue"} == 1` and review queue-related `pod_resource_stress`. Recovery: Review queue DoC thresholds. If early protection is required, allow gentle shedding of the lowest-priority traffic at DoC; otherwise, tune thresholds to match observed load.

5.1.47 POD_PENDING_REQUEST_CONGESTED

Table 5-48 POD_PENDING_REQUEST_CONGESTED

Field	Details
Description	Pod Resource Congestion status of {{$labels.microservice}} service is congested for PendingRequest type
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is congested for PendingRequest type
Severity	Critical
Expression	ocbsf_pod_resource_congestion_state{type="queue"} == 4
OID	1.3.6.1.4.1.323.5.3.37.1.2.28
Metric Used	ocbsf_pod_resource_congestion_state{type="queue"}
Recommended Actions	Cause: The pending HTTP request queue is in the `CONGESTED` state. Diagnostic Information: Confirm that `pod_resource_congestion_state{resourceType="queue"} == 4` and review `pod_resource_stress{resourceType="queue"}`. Check `ocbsf_http_congestion_message_reject_total` for low-priority discards at this level. Recovery: Validate the queue thresholds in the active profile and the `CONGESTED` discard priority. If backlog persists, increase shedding by raising the discard priority so that lower-priority requests are rejected earlier.

5.1.48 POD_CPU_DOC

Table 5-49 POD_CPU_DOC

Field	Details
Description	Pod Resource Congestion status of {{$labels.microservice}} service is DoC for CPU type
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is DoC for CPU type
Severity	Major
Expression	ocbsf_pod_resource_congestion_state{type="cpu"} == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.29
Metric Used	ocbsf_pod_resource_congestion_state{type="cpu"}
Recommended Actions	Cause: CPU utilization is in the `DANGER_OF_CONGESTION` state according to the current thresholds and EMA settings. Diagnostic Information: Confirm that `pod_resource_congestion_state{resourceType="cpu"} == 1` and check `pod_resource_stress{resourceType="cpu"}`. Confirm the EMA parameters, including the interval and 70:30 ratios. Recovery: If the condition is transient, increase `stateChangeSampleCount` to avoid oscillation. If it is sustained, adjust the CPU DoC threshold or enable mild shedding for non-critical, low-priority requests.

5.1.49 POD_CPU_CONGESTED

Table 5-50 POD_CPU_CONGESTED

Field	Details
Description	Pod Resource Congestion status of {{$labels.microservice}} service is congested for CPU type
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is congested for CPU type
Severity	Critical
Expression	ocbsf_pod_resource_congestion_state{type="cpu"} == 4
OID	1.3.6.1.4.1.323.5.3.37.1.2.30
Metric Used	ocbsf_pod_resource_congestion_state
Recommended Actions	Cause: CPU utilization has reached the `CONGESTED` state. Diagnostic Information: Validate `pod_resource_congestion_state{resourceType="cpu"}` for values 2 or 4, according to the alert rule, and check `pod_resource_stress{resourceType="cpu"}`. Review message rejections at this congestion level. Recovery: Tighten protection by raising the discard priority for this state so that more low-priority requests are dropped. Reassess CPU thresholds and EMA intervals only after reviewing performance benchmarks.

5.1.50 POD_MEMORY_DOC

Table 5-51 POD_MEMORY_DOC

Field	Details
Description	Pod Resource Congestion status of {{$labels.microservice}} service is DoC for Memory type
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is DoC for Memory type
Severity	Major
Expression	ocbsf_pod_resource_congestion_state{type="memory"} == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.31
Metric Used	ocbsf_pod_resource_congestion_state{type="memory"}
Recommended Actions	Cause: Pod memory utilization has entered the `DANGER_OF_CONGESTION` state according to the configured memory thresholds and EMA. This indicates increasing memory pressure that can lead to congestion and request shedding. Diagnostic Information: Confirm that `pod_resource_congestion_state{type="memory"} == 1`. Review the `pod_resource_stress{resourceType="memory"}` trend to determine whether pressure is sustained or spiky. Correlate the condition with working-set or RSS metrics, garbage-collection activity, traffic spikes, configuration changes, pod restarts, OOM indicators, and application logs. Recovery: For transient spikes, increase `stateChangeSampleCount` or the congestion-calculation interval to reduce flapping. For sustained pressure, reduce load by adjusting shedding for low-priority traffic or increase pod memory limits and requests according to the capacity plan. Investigate memory-growth or leak patterns and large in-flight requests; remediate at the application level if memory does not stabilize.

5.1.51 POD_MEMORY_CONGESTED

Table 5-52 POD_MEMORY_CONGESTED

Field	Details
Description	Pod Resource Congestion status of {{$labels.microservice}} service is congested for Memory type
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is congested for Memory type
Severity	Critical
Expression	ocbsf_pod_resource_congestion_state{type="memory"} == 2
OID	1.3.6.1.4.1.323.5.3.37.1.2.32
Metric Used	ocbsf_pod_resource_congestion_state{type="memory"}
Recommended Actions	Cause: Memory utilization has reached the `CONGESTED` state according to the active thresholds and EMA configuration. The pod is under high memory stress and might reject or drop low-priority requests to protect stability. Diagnostic Information: Confirm that `pod_resource_congestion_state{type="memory"} == 2`. Inspect `pod_resource_stress{resourceType="memory"}` and container memory consumption against limits. Check for OOM kills, restarts, Kubernetes events, request-size increases, traffic surges, cache growth, and increasing request rejections. Recovery: Increase shedding aggressiveness by raising the discard priority so that lower-priority requests are rejected earlier. If congestion persists, scale out the workload or increase memory limits according to performance baselines, then reassess thresholds. If OOM kills or restarts occur, stabilize memory by reducing load or rolling back recent changes before tuning thresholds.

5.1.52 SERVICE_OVERLOADED

Table 5-53 SERVICE_OVERLOADED

Field	Details
Description	Overload Level of {{$labels.microservice}} service is L1
Summary	Overload Level of {{$labels.microservice}} service is L1
Severity	Minor
Expression	load_level == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.14
Metric Used	load_level
Recommended Actions	Cause: The service can be overloaded because of high CPU utilization and a large backlog, an extreme backlog with increasing failures caused by downstream latency or an outage, or widespread failures caused by network, DNS, certificate, routing, release, or dependency issues. Diagnostic Information: Determine whether CPU saturation, a runaway pending-request queue, or failures and timeouts are the primary condition. Check cluster and platform health, including node readiness, network, DNS, CNI, ingress, and load balancer status. Confirm that critical traffic is correctly classified. Recovery: Apply strict L3 discard rules to reject low-priority and, if required, medium-priority traffic to preserve high-priority flows. Increase percentage-based rejection where supported. If dependency failures are responsible, fix the dependency before scaling the service. If compute capacity is the issue and dependencies are healthy, rapidly scale replicas or nodes and temporarily reduce expensive traffic or features. Engage the relevant platform and dependency owners because L3 is typically a cross-team incident.

Table 5-54 SERVICE_OVERLOADED

Field	Details
Description	Overload Level of {{$labels.microservice}} service is L2
Summary	Overload Level of {{$labels.microservice}} service is L2
Severity	Major
Expression	load_level == 2
OID	1.3.6.1.4.1.323.5.3.37.1.2.14
Metric Used	load_level
Recommended Actions	Cause: The service can be overloaded because of high CPU utilization and a large backlog, an extreme backlog with increasing failures caused by downstream latency or an outage, or widespread failures caused by network, DNS, certificate, routing, release, or dependency issues. Diagnostic Information: Determine whether CPU saturation, a runaway pending-request queue, or failures and timeouts are the primary condition. Check cluster and platform health, including node readiness, network, DNS, CNI, ingress, and load balancer status. Confirm that critical traffic is correctly classified. Recovery: Apply strict L3 discard rules to reject low-priority and, if required, medium-priority traffic to preserve high-priority flows. Increase percentage-based rejection where supported. If dependency failures are responsible, fix the dependency before scaling the service. If compute capacity is the issue and dependencies are healthy, rapidly scale replicas or nodes and temporarily reduce expensive traffic or features. Engage the relevant platform and dependency owners because L3 is typically a cross-team incident.

Table 5-55 SERVICE_OVERLOADED

Field	Details
Description	Overload Level of {{$labels.service}} service is L3
Summary	Overload Level of {{$labels.service}} service is L3
Severity	Critical
Expression	load_level == 3
OID	1.3.6.1.4.1.323.5.3.37.1.2.14
Metric Used	load_level
Recommended Actions	Cause: The service can be overloaded because of high CPU utilization and a large backlog, an extreme backlog with increasing failures caused by downstream latency or an outage, or widespread failures caused by network, DNS, certificate, routing, release, or dependency issues. Diagnostic Information: Determine whether CPU saturation, a runaway pending-request queue, or failures and timeouts are the primary condition. Check cluster and platform health, including node readiness, network, DNS, CNI, ingress, and load balancer status. Confirm that critical traffic is correctly classified. Recovery: Apply strict L3 discard rules to reject low-priority and, if required, medium-priority traffic to preserve high-priority flows. Increase percentage-based rejection where supported. If dependency failures are responsible, fix the dependency before scaling the service. If compute capacity is the issue and dependencies are healthy, rapidly scale replicas or nodes and temporarily reduce expensive traffic or features. Engage the relevant platform and dependency owners because L3 is typically a cross-team incident.

5.1.53 SERVICE_RESOURCE_OVERLOADED

Alerts when service is in overload state due to memory usage

Table 5-56 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L1 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L1 for {{$labels.type}} type
Severity	Minor
Expression	service_resource_overload_level{type="memory"} == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="memory"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Table 5-57 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L2 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L2 for {{$labels.type}} type
Severity	Major
Expression	service_resource_overload_level{type="memory"} == 2
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="memory"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Table 5-58 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L3 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L3 for {{$labels.type}} type
Severity	Critical
Expression	service_resource_overload_level{type="memory"} == 3
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="memory"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Alerts when service is in overload state due to CPU usage

Table 5-59 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L1 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L1 for {{$labels.type}} type
Severity	Minor
Expression	service_resource_overload_level{type="cpu"} == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="cpu"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Table 5-60 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L2 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L2 for {{$labels.type}} type
Severity	Major
Expression	service_resource_overload_level{type="cpu"} == 2
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="cpu"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Table 5-61 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L3 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L3 for {{$labels.type}} type
Severity	Critical
Expression	service_resource_overload_level{type="cpu"} == 3
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="cpu"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Alerts when service is in overload state due to number of pending messages

Table 5-62 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L1 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L1 for {{$labels.type}} type
Severity	Minor
Expression	service_resource_overload_level{type="svc_pending_count"} == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="svc_pending_count"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Table 5-63 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L2 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L2 for {{$labels.type}} type
Severity	Major
Expression	service_resource_overload_level{type="svc_pending_count"} == 2
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="svc_pending_count"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Table 5-64 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L3 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L3 for {{$labels.type}} type
Severity	Critical
Expression	service_resource_overload_level{type="svc_pending_count"} == 3
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="svc_pending_count"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Alerts when service is in overload state due to number of failed requests

Table 5-65 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L1 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L1 for {{$labels.type}} type
Severity	Minor
Expression	service_resource_overload_level{type="svc_failure_count"} == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="svc_failure_count"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Table 5-66 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L2 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L2 for {{$labels.type}} type
Severity	Major
Expression	service_resource_overload_level{type="svc_failure_count"} == 2
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="svc_failure_count"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

Table 5-67 SERVICE_RESOURCE_OVERLOADED

Field	Details
Description	{{$labels.microservice}} service is L3 for {{$labels.type}} type
Summary	{{$labels.microservice}} service is L3 for {{$labels.type}} type
Severity	Critical
Expression	service_resource_overload_level{type="svc_failure_count"} == 3
OID	1.3.6.1.4.1.323.5.3.37.1.2.15
Metric Used	service_resource_overload_level{type="svc_failure_count"}
Recommended Actions	Cause: The service is at overload level L1 for memory. Possible drivers include a memory leak, higher concurrency, cache growth, large payloads, misconfigured memory limits or heap settings, and node memory pressure. Diagnostic Information: Review the memory-usage trend for each pod, OOM-kill and restart events, garbage-collection pressure where applicable, allocation spikes, configured limits and requests, node memory pressure, traffic, and recent changes. Recovery: Reduce memory pressure by tuning caches or payloads, fixing leaks, or rolling back regressions. Scale out to reduce per-pod memory usage and cautiously adjust memory limits. The alert clears when the overload level returns to zero.

5.1.54 SYSTEM_IMPAIRMENT_MAJOR

Table 5-68 SYSTEM_IMPAIRMENT_MAJOR

Field	Details
Description	Major Impairment alert raised for REPLICATION_FAILED or REPLICATION_CHANNEL_DOWN or BINLOG_STORAGE usage > 80%
Summary	Major impairment alert raised for REPLICATION_FAILED or REPLICATION_CHANNEL_DOWN or BINLOG_STORAGE usage must be more than 80% for 10 minutes.
Severity	Major
Expression	(db_tier_replication_status{role="failed"} == 0) or (db_tier_replication_status{role="active"} == 0) or (count by (site_name) (db_tier_replication_status) == count by (site_name) (db_tier_replication_status{role="standby"})) or (count by (site_name) (db_tier_replication_status) == count by (site_name) (db_tier_replication_status{role="failed"})) or (avg_over_time(db_tier_binlog_used_bytes_percentage[5m])>= 80)
OID	1.3.6.1.4.1.323.5.3.52.1.2.43
Metric Used	db_tier_replication_status and db_tier_binlog_used_bytes_percentage
Recommended Actions	Cause: SYSTEM_IMPAIRMENT_MAJOR indicates a major site impairment condition related to the DB tier, typically associated with replication health or binlog storage pressure. This alert is intended to fire when one or more of the following major-risk conditions occur (often mapped to REPLICATION_FAILED, REPLICATION_CHANNEL_DOWN, or BINLOG_STORAGE high usage): Diagnostic Information: Replication failure (replication role reported as failed, or replication threads stopped) Replication channel down / missing standby (expected replication roles are not present for the site, e.g., standby not reporting) Primary/active not reporting (active role missing/unhealthy from metrics perspective) Binlog storage growth due to replication lag, blocked purge, or high write rate, resulting in binlog usage ≥ 80% (as measured by db_tier_binlog_used_bytes_percentage) Network issues between DB nodes/sites (timeouts, resets, packet loss) DB load/resource saturation causing lag (CPU, IO, disk, connection pool) Misconfiguration/regression after deploy/patch (replication settings, credentials, TLS/certs) Determine which condition triggered the alert Check current values and recent history for: db_tier_replication_status{role="active"}, {role="standby"}, {role="failed"} db_tier_binlog_used_bytes_percentage Confirm whether the trigger is replication-related vs binlog-usage-related. Replication health deep-dive Identify impacted site_name and which node(s)/roles are missing or in failed. Check replication lag, SQL/IO thread state, last error, and channel status on the DB nodes. Channel/network validation Check connectivity between active and standby. Look for recent network maintenance, route changes, MTU issues, or service mesh/proxy impacts (if in path). Binlog storage analysis (if binlog usage is high) Confirm binlog filesystem utilization and growth rate; determine if purge is blocked. Correlate with replication lag/backlog (binlogs often accumulate when standby can’t consume). Check for unusually high write volume or long-running transactions. Resource saturation / DB health Check disk IOPS/latency, CPU, memory pressure, and connection pool saturation. Review DB logs for replication errors, disk-full warnings, corruption, or restart events. Change correlation Check recent deploys/patches/config changes (DB parameters, replication config, storage changes). Recovery: If replication failed / channel down: Restore replication connectivity (network path, DNS, firewall, TLS/certs). Restart/repair replication channel as per DB runbook; resolve the root error (permissions, missing binlogs, etc.). If standby is unhealthy, recover/rebuild standby from a known-good snapshot/backup and rejoin replication. Consider temporarily reducing write load or enabling throttling to allow the standby to catch up. If binlog usage ≥ 80%: Treat as urgent to avoid running out of storage (risk of DB outage). Reduce binlog growth by addressing replication lag (fix channel, improve IO/CPU, reduce write amplification). Expand binlog storage if needed (short-term mitigation) following capacity change procedures. Ensure binlog purge/retention settings are correct and purge is not blocked (e.g., by lagging replicas). Operational guidance: Escalate as major if sustained, if multiple sites are impacted, or if lag/space continues to worsen. After remediation, verify: replication roles return to expected states (active/standby healthy; no failed) replication lag stabilizes db_tier_binlog_used_bytes_percentage drops below threshold and continues trending down

5.1.55 SYSTEM_IMPAIRMENT_CRITICAL

Table 5-69 SYSTEM_IMPAIRMENT_CRITICAL

Field	Details
Description	Critical Impairment alert raised for REPLICATION_FAILED or REPLICATION_CHANNEL_DOWN or BINLOG_STORAGE usage > 80%
Summary	Critical impairment alert raised for REPLICATION_FAILED or REPLICATION_CHANNEL_DOWN or BINLOG_STORAGE usage must be more than 80% for 30 minutes.
Severity	Critical
Expression	(db_tier_replication_status{role="failed"} == 0) or (db_tier_replication_status{role="active"} == 0) or (count by (site_name) (db_tier_replication_status) == count by (site_name) (db_tier_replication_status{role="standby"})) or (count by (site_name) (db_tier_replication_status) == count by (site_name) (db_tier_replication_status{role="failed"})) or (avg_over_time(db_tier_binlog_used_bytes_percentage[5m])>= 80)
OID	1.3.6.1.4.1.323.5.3.52.1.2.43
Metric Used	db_tier_replication_status and db_tier_binlog_used_bytes_percentage
Recommended Actions	Cause: SYSTEM_IMPAIRMENT_CRITICAL indicates a critical site impairment related to DB-tier replication health and/or binlog storage pressure. It is the higher-severity escalation of the impairment condition, intended to represent that one of the following has become sustained/critical (per description: replication failed, replication channel down, or binlog storage usage ≥ 80% for ~30 minutes). Common drivers: Sustained replication failure (replication role reported as failed, replication threads stopped, unrecoverable SQL error) Replication channel down for an extended period (standby missing/unreachable, broken link, auth/TLS failure) Loss of expected topology (active/standby role metrics missing or inconsistent for a site) Sustained binlog storage pressure (≥ 80%), typically due to replication lag/backlog, blocked purge, or abnormal write volume Network instability between sites/DB nodes causing persistent disconnects and lag Resource exhaustion on DB nodes (disk IO saturation, CPU, memory) preventing replication catch-up Misconfiguration/regression after change (replication config, retention, credentials) Diagnostic Information: Confirm the trigger and impacted site_name Identify whether the critical is driven by replication status vs binlog usage trend. Review current and historical values for: db_tier_replication_status{role="active" \| "standby" \| "failed"} db_tier_binlog_used_bytes_percentage (trend over last 30–60 minutes) Replication failure/channel-down triage On DB nodes, check replication channel state, IO/SQL thread status, last error, and lag. Determine whether the standby is unreachable vs reachable-but-failing (auth/TLS/permissions vs data/SQL errors). Confirm whether the “active” node is healthy and exporting metrics (avoid false positives from missing scrapes). Topology consistency check Validate expected number of replication endpoints per site (active + standby) and confirm none are missing. If the alert logic is based on counts, verify whether a node is down or metrics are absent. Binlog storage pressure investigation (if ≥ 80%) Check filesystem utilization where binlogs reside and remaining headroom; estimate time-to-full. Confirm whether binlog purge is blocked by lagging replica(s) or retention settings. Correlate with write throughput and long transactions. Dependency and resource saturation Disk IO latency/queue depth, CPU, memory pressure, network throughput/packet loss. DB error logs for disk-full warnings, replication errors, restarts, corruption indicators. Change/event correlation Look for recent deployments, DB patching, parameter changes, certificate rotations, or network maintenance coinciding with onset. Recovery: Immediate posture (CRITICAL): Treat as incident-level due to risk of data protection loss (replication down) and/or imminent DB outage (binlog disk exhaustion). Engage DB/platform/network on-call as required; follow site incident process. If replication failed / channel down: Restore connectivity (network/DNS/firewall) and fix auth/TLS/cert issues. Restart/repair replication channels per runbook; remediate root cause (SQL errors, missing binlogs, GTID divergence). If standby cannot be repaired quickly, rebuild standby from snapshot/backup and re-establish replication. Consider temporarily throttling write load to reduce lag/backlog while recovering. If binlog usage ≥ 80% (sustained): Prevent disk-full: reduce binlog generation (traffic/load throttling where feasible) restore replication so binlogs can be consumed and purged expand binlog storage/capacity as an emergency mitigation (per change controls) Verify purge/retention settings and ensure purge is not blocked by an unreachable or stuck replica. Validation or exit criteria: Replication status returns to expected healthy roles (no failed, standby present and catching up). Replication lag stabilizes within acceptable bounds. db_tier_binlog_used_bytes_percentage trends downward and remains below threshold.

5.1.56 SYSTEM_OPERATIONAL_STATE_PARTIAL_SHUTDOWN

Table 5-70 SYSTEM_OPERATIONAL_STATE_PARTIAL_SHUTDOWN

Field	Details
Description	System Operational State is now in partial shutdown state.
Summary	System Operational State is now in partial shutdown state.
Severity	Major
Expression	system_operational_state == 2
OID	1.3.6.1.4.1.323.5.3.52.1.2.44
Metric Used	system_operational_state == 2
Recommended Actions	Cause: The site has entered the `PARTIAL SHUTDOWN` operational state under the Controlled Shutdown feature. This deliberate isolation mode supports maintenance, such as upgrades or configuration changes, or recovery activities, such as prolonged replication-channel failures or replication delay, while allowing existing sessions to continue. In this state, NRF status is `SUSPENDED` to prevent consumer discovery; the Ingress Gateway and Diameter Gateway permit only in-session messages and reject new session establishment; and the Egress Gateway has no restriction. This state is typically set through the CM GUI or REST API when `enableControlledShutdown` is enabled. Diagnostic Information: Confirm the operational state is truly 2 (PARTIAL) and record transition time (use alert timestamp + metric history). Verify whether the change was planned (maintenance/recovery activity) and who/what initiated it (CM GUI/REST). Validate enableControlledShutdown is enabled and configuration is consistent across the site. Verify expected gateway behavior: Ingress rejects new session establishment but permits in-session traffic Diameter permits in-session only Check NRF to confirm NF status marked SUSPENDED and are not being discovered for new sessions. Identify the underlying reason for isolation (maintenance, replication channel failures/delay, dependency instability) and correlate with replication metrics/logs. Assess customer impact: rate of blocked new sessions, admission control counters, error codes returned at ingress/diameter. Recovery: For a planned shutdown, proceed with maintenance or recovery activities, such as replication repair or dependency stabilization, while keeping existing sessions stable. For an unplanned shutdown, confirm that no automation or configuration error triggered the change, correct configuration or access controls, and return to `NORMAL` when safe. After recovery, verify that NFs return to `REGISTERED` in NRF and that new session establishment succeeds at Ingress and Diameter. Escalate if the partial shutdown persists unexpectedly or blocks new sessions beyond agreed SLOs.

5.1.57 SYSTEM_OPERATIONAL_STATE_COMPLETE_SHUTDOWN

Table 5-71 SYSTEM_OPERATIONAL_COMPLETE_SHUTDOWN

Field	Details
Description	System Operational State is now in complete shutdown state
Summary	System Operational State is now in complete shutdown state
Severity	Critical
Expression	system_operational_state == 3
OID	1.3.6.1.4.1.323.5.3.52.1.2.44
Metric Used	system_operational_state
Recommended Actions	Cause: The site has entered the `COMPLETE SHUTDOWN` operational state under the Controlled Shutdown feature. This fully stops load and isolates the site during recovery or severe instability. NRF status is `SUSPENDED`, the Ingress Gateway accepts no requests, the Egress Gateway allows only NRF-bound requests, and Diameter Gateway external-peer connections are closed. Diagnostic Information: Confirm the operational state is 3 (COMPLETE) and capture the exact change time and duration. Verify whether complete isolation was intended (approved maintenance/incident action) and confirm initiator. Validate actual enforcement at gateways: Ingress is refusing all new connections Diameter external peer sessions are closed Egress allows only NRF-bound traffic Confirm NRF state: NFs remain SUSPENDED and are not discoverable by consumers (SMF/AMF/etc.). Identify the triggering condition that required complete shutdown (e.g., severe replication failure, corruption risk, cascading dependency failures) and gather supporting logs/metrics. Confirm traffic has drained/failover behavior is functioning (upstream routing, peer sites capacity, no looping retries). Check for any secondary effects: backlog growth, message queue accumulation, alarms from consumers due to loss of discovery. Recovery: Treat an unscheduled shutdown as an incident. Perform the maintenance or recovery procedure while the site is isolated, such as replication rebuild or data-consistency checks. When ready, transition to `PARTIAL` if required for validation, or directly to `NORMAL`; restore gateway connectivity and Egress behavior; update NRF status to `REGISTERED`; and validate discovery, new-session establishment, and clean Diameter-peer reconnection.

5.1.58 DIAM_CONN_PEER_DOWN

Table 5-72 DIAM_CONN_PEER_DOWN

Field	Details
Description	Diameter connection to peer {{ $labels.peerHost }} is down.
Summary	Diameter connection to peer down.
Severity	Major
Expression	(sum by (namespace,peerHost)(ocbsf_diam_conn_network) == 0) and (sum by (namespace,peerHost)(max_over_time(ocbsf_diam_conn_network[24h])) != 0)
OID	1.3.6.1.4.1.323.5.3.37.1.2.18
Metric Used	ocbsf_diam_conn_network
Recommended Actions	Cause: The Diameter Gateway network-side connection to a specific external peer is down, although it was connected at least once during the preceding 24 hours. Common causes include peer downtime or maintenance, a peer-specific network-path issue, transport instability, TLS or mTLS handshake failures, peer overload, configuration mismatches, or gateway resource pressure. Diagnostic Information: Identify the affected `kubernetes_namespace` and `origHost`, and determine whether one or multiple peers are affected. Confirm that `max_over_time(occnp_diam_conn_network[24h])` is non-zero for the same peer. Review Diameter Gateway logs for watchdog failures, disconnect procedures, CER or CEA failures, TCP errors, and TLS validation failures. Check peer health, listener availability, logs, overload indicators, routing, firewall, NAT, DNS, recent changes, and TLS certificate, trust-bundle, hostname, SNI, protocol, and cipher configuration. Recovery: Restore peer availability and repair peer-specific routing, firewall, or DNS issues. If TLS is enabled, renew or rotate certificates, update trust bundles, and correct hostname, SNI, cipher, or protocol settings. Address overload or instability by scaling or reducing load on the gateway or peer. Verify that `occnp_diam_conn_network` for the affected `origHost` is greater than zero, the connection is stable, and Diameter transaction success rates return to normal.

5.1.59 DIAM_CONN_NETWORK_DOWN

Table 5-73 DIAM_CONN_NETWORK_DOWN

Field	Details
Description	All diameter network connections are down.
Summary	All diameter network connections are down.
Severity	Critical
Expression	sum by (namespace)(ocbsf_diam_conn_network) == 0
OID	1.3.6.1.4.1.323.5.3.37.1.2.19
Metric Used	ocbsf_diam_conn_network
Recommended Actions	Cause: The Diameter Gateway has no active external Diameter-network connections for a `kubernetes_namespace`. Common causes include an external-peer outage or maintenance, network-path failures, TLS or mTLS issues, peer-configuration mismatches, transport instability, gateway overload, or controlled shutdown or site isolation. Diagnostic Information: Confirm the affected namespace and whether the outage is isolated or widespread. Identify affected peers and determine whether all external peers are down. Review Diameter Gateway logs for watchdog failures, disconnect procedures, CER or CEA failures, TCP errors, and TLS errors. Validate peer health, listener availability, maintenance status, routing, firewall, NAT, DNS, and recent network changes. If TLS is enabled, verify certificate validity and chain, truststore or CA alignment, hostname or SNI matching, and protocol or cipher compatibility. Correlate the alert with backend-connection alerts and controlled-shutdown status. Recovery: Restore external-peer availability and network connectivity. If TLS-related, renew or rotate certificates, update truststores or CA bundles, correct hostname, SNI, cipher, or protocol mismatches, and re-establish sessions. Correct Diameter-peer configuration and address gateway resource pressure or instability. Verify that `sum by (kubernetes_namespace)(occnp_diam_conn_network)` is greater than zero, external-peer sessions are stable, and end-to-end Diameter transactions succeed.

5.1.60 DIAM_RESPONSE_REALM_VALIDATION_ERROR_CRITICAL

Table 5-74 DIAM_RESPONSE_REALM_VALIDATION_ERROR_CRITICAL

Field	Details
Description	At least 75% of the Diam Response failed with error 'DIAMETER_REALM_NOT_SERVED', either of BSF realm or PCF Realm doesn't match with received destination realm in diameter message.
Summary	{{ $value }}% of the Diam Response failed with error 'DIAMETER_REALM_NOT_SERVED'.
Severity	CRITICAL
Expression	(sum by (namespace, microservice) (increase(ocbsf_diam_realm_validation_failed_total{responseCode="3003", appId="16777236"}[10m])) / sum by (namespace, microservice) (increase(ocbsf_diam_response_network_total{appId="16777236"}[10m]))) * 100 >= 75
OID	1.3.6.1.4.1.323.5.3.37.1.2.41
Metric Used	ocbsf_diam_realm_validation_failed_total ocbsf_diam_response_network_total
Recommended Actions	Check if the value of the following keys under Advanced settings of diameter settings page are set to true: DIAMETER.Enable.Validate.Realm DIAMETER.BSF.Enable.Validate.Binding.Realm Check the destination-realm in diameter request.

5.1.61 DIAM_RESPONSE_REALM_VALIDATION_ERROR_MAJOR

Table 5-75 DIAM_RESPONSE_REALM_VALIDATION_ERROR_MAJOR

Field	Details
Description	At least 50% of the Diam Response failed with error 'DIAMETER_REALM_NOT_SERVED', either of BSF realm or PCF Realm doesn't match with received destination realm in diameter message.
Summary	{{ $value }}% of the Diam Response failed with error 'DIAMETER_REALM_NOT_SERVED'.
Severity	MAJOR
Expression	(sum by (namespace, microservice) (increase(ocbsf_diam_realm_validation_failed_total{responseCode="3003", appId="16777236"}[10m])) / sum by (namespace, microservice) (increase(ocbsf_diam_response_network_total{appId="16777236"}[10m]))) * 100 >= 50
OID	1.3.6.1.4.1.323.5.3.37.1.2.41
Metric Used	ocbsf_diam_realm_validation_failed_total ocbsf_diam_response_network_total
Recommended Actions	Check if the value of the following keys under Advanced settings of diameter settings page are set to true: DIAMETER.Enable.Validate.Realm DIAMETER.BSF.Enable.Validate.Binding.Realm Check the destination-realm coming in diameter request.

5.1.62 DIAM_RESPONSE_REALM_VALIDATION_ERROR_MINOR

Table 5-76 DIAM_RESPONSE_REALM_VALIDATION_ERROR_MINOR

Field	Details
Description	At least 20% of the Diam Response failed with error 'DIAMETER_REALM_NOT_SERVED', either of BSF realm or PCF Realm doesn't match with received destination realm in diameter message.
Summary	{{ $value }}% of the Diam Response failed with error 'DIAMETER_REALM_NOT_SERVED'.
Severity	MINOR
Expression	(sum by (namespace, microservice) (increase(ocbsf_diam_realm_validation_failed_total{responseCode="3003", appId="16777236"}[10m])) / sum by (namespace, microservice) (increase(ocbsf_diam_response_network_total{appId="16777236"}[10m]))) * 100 >= 20
OID	1.3.6.1.4.1.323.5.3.37.1.2.41
Metric Used	ocbsf_diam_realm_validation_failed_total ocbsf_diam_response_network_total
Recommended Actions	Check if the value of the following keys under Advanced settings of diameter settings page are set to true: DIAMETER.Enable.Validate.Realm DIAMETER.BSF.Enable.Validate.Binding.Realm Check the destination-realm coming in diameter request.

5.1.63 AUDIT_STALE_NOTIFY_ERROR_RESPONSE_MINOR

Table 5-77 AUDIT_STALE_NOTIFY_ERROR_RESPONSE_MINOR

Field	Details
Description	At least 20 % of the BSF Audit Notification Requests sent to PCF to check for Suspected Stale Sessions have responded with a 5xx or 4xx (excluding 404) Status in the last 10 minutes.
Summary	At least 20% of the BSF Notification Request for Audit have responded with a 5xx or 4xx (not 404) Status in the last 10 minutes.
Severity	MINOR
Expression	(sum by (namespace, microservice) (increase(ocbsf_query_response_count_total{response_code=~"5xx\|4xx\|timeout"}[10m])) / sum by (namespace, microservice) (increase(ocbsf_query_response_count_total[10m]))) * 100 >= 20 < 40
OID	1.3.6.1.4.1.323.5.3.37.1.2.42
Metric Used	ocbsf_query_response_count_total
Recommended Actions	Cause: More than 20% and less than 40% of BSF audit notification requests sent to PCF to check suspected stale sessions returned 4xx, 5xx, or timeout responses in the last 10 minutes. Diagnostic Information: Verify whether PCF returns 4xx, 5xx, or timeout responses for the BSF PCF bindings used by audit query or notification requests. Recovery: The alert clears when BSF audit requests to PCF receive a successful 2xx response or another HTTP error response.

5.1.64 AUDIT_STALE_NOTIFY_ERROR_RESPONSE_MAJOR

Table 5-78 AUDIT_STALE_NOTIFY_ERROR_RESPONSE_MAJOR

Field	Details
Description	At least 40 % of the BSF Audit Notification Requests sent to PCF to check for Suspected Stale Sessions have responded with a 5xx or 4xx (excluding 404) Status in the last 10 minutes.
Summary	{{ $value }} % of the BSF Audit Notification Requests sent to PCF to check for Suspected Stale Sessions have responded with a 5xx or 4xx (excluding 404) Status in the last 10 minutes.
Severity	MAJOR
Expression	(sum by (namespace, microservice) (increase(ocbsf_query_response_count_total{response_code=~"5xx\|4xx\|timeout"}[10m])) / sum by (namespace, microservice) (increase(ocbsf_query_response_count_total[10m]))) * 100 >= 40 < 60
OID	1.3.6.1.4.1.323.5.3.37.1.2.42
Metric Used	ocbsf_query_response_count_total
Recommended Actions	Cause: More than 40% and less than 60% of BSF audit notification requests sent to PCF to check suspected stale sessions returned 4xx, 5xx, or timeout responses in the last 10 minutes. Diagnostic Information: Verify whether PCF returns 4xx, 5xx, or timeout responses for the BSF PCF bindings used by audit query or notification requests. Recovery: The alert clears when BSF audit requests to PCF receive a successful 2xx response or another HTTP error response.

5.1.65 AUDIT_STALE_NOTIFY_ERROR_RESPONSE_CRITICAL

Table 5-79 AUDIT_STALE_NOTIFY_ERROR_RESPONSE_CRITICAL

Field	Details
Description	At least 60 % of the BSF Audit Notification Requests sent to PCF to check for Suspected Stale Sessions have responded with a 5xx or 4xx (excluding 404) Status in the last 10 minutes.
Summary	At least 60% of the BSF Notification Request for Audit to PCF (or its respective NF) failed with a 5xx or 4xx (not 404) Status in the last 10 minutes. The threshold default value is defined at `BSF_Alertrules.yaml`.
Severity	CRITICAL
Expression	(sum by (namespace, microservice) (increase(ocbsf_query_response_count_total{response_code=~"5xx\|4xx\|timeout"}[10m])) / sum by (namespace, microservice) (increase(ocbsf_query_response_count_total[10m]))) * 100 >= 60
OID	1.3.6.1.4.1.323.5.3.37.1.2.42
Metric Used	ocbsf_query_response_count_total
Recommended Actions	Cause: More than 60% of BSF audit notification requests sent to PCF to check suspected stale sessions returned 4xx, 5xx, or timeout responses in the last 10 minutes. Diagnostic Information: Verify whether PCF returns 4xx, 5xx, or timeout responses for the BSF PCF bindings used by audit query or notification requests. Recovery: The alert clears when BSF audit requests to PCF receive a successful 2xx response or another HTTP error response. .

5.1.66 BSF_CONNECTION_FAILURE

Table 5-80 BSF_CONNECTION_FAILURE

Field	Details
Description	Connection failure on Egress and Ingress Gateways for incoming and outgoing connections.
Summary	Connection failure on Egress and Ingress Gateways for incoming and outgoing connections.
Severity	Major
Expression	sum(increase(ocbsf_oc_ingressgateway_connection_failure_total[5m]) >0 or (ocbsf_oc_ingressgateway_connection_failure_total unless ocbsf_oc_ingressgateway_connection_failure_total offset 5m )) by (namespace,app, error_reason) > 0 or sum(increase(ocbsf_oc_egressgateway_connection_failure_total[5m]) >0 or (ocbsf_oc_egressgateway_connection_failure_total unless ocbsf_oc_egressgateway_connection_failure_total offset 5m )) by (namespace,app, error_reason) > 0
OID	1.3.6.1.4.1.323.5.3.37.1.2.43
Metric Used	ocbsf_oc_ingressgateway_connection_failure_total ocbsf_oc_egressgateway_connection_failure_total
Recommended Actions	For any additional guidance, contact My Oracle Support (https://support.oracle.com).

5.1.67 INGRESS_GATEWAY_DD_UNREACHABLE_MAJOR

Table 5-81 INGRESS_GATEWAY_DD_UNREACHABLE_MAJOR

Field	Details
Description	'BSF Ingress Gateway Data Director unreachable for {{$labels.namespace}}'
Summary	'kubernetes_namespace: {{$labels.kubernetes_namespace}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }} BSF Ingress Gateway Data Director unreachable'
Severity	Major
Expression	sum(oc_ingressgateway_dd_unreachable) by(namespace,container) > 0
OID	1.3.6.1.4.1.323.5.3.37.1.2.48
Metric Used	oc_ingressgateway_dd_unreachable
Recommended Actions

5.1.68 EGRESS_GATEWAY_DD_UNREACHABLE_MAJOR

Table 5-82 EGRESS_GATEWAY_DD_UNREACHABLE_MAJOR

Field	Details
Description	'BSF Egress Gateway Data Director unreachable for {{$labels.namespace}}'
Summary	'kubernetes_namespace: {{$labels.kubernetes_namespace}}, timestamp: {{ with query "time()" }}{{ . \| first \| value \| humanizeTimestamp }}{{ end }} BSF Egress Gateway Data Director unreachable'
Severity	Major
Expression	sum(oc_egressgateway_dd_unreachable) by(namespace,container) > 0
OID	1.3.6.1.4.1.323.5.3.37.1.2.49
Metric Used	oc_egressgateway_dd_unreachable
Recommended Actions	Cause: No cause information is provided in the Confluence Resolutions/Workarounds column. Diagnostic Information: No diagnostic information is provided in the Confluence Resolutions/Workarounds column. Recovery: No recovery information is provided in the Confluence Resolutions/Workarounds column.

5.1.69 DIAM_GATEWAY_CERTIFICATE_EXPIRY_MAJOR

Table 5-83 DIAM_GATEWAY_CERTIFICATE_EXPIRY_MAJOR

Field	Details
Description	Diam-gw certificate expiry in less than 3 months for {{$labels.namespace}}.
Summary	Diam-gw certificate expiry in less than 3 months.
Severity	Major
Expression	dgw_tls_cert_expiration_seconds - time() <= 7862400
OID	1.3.6.1.4.1.323.5.3.37.1.2.47
Metric Used	dgw_tls_cert_expiration_seconds
Recommended Actions	Cause: A Diameter Gateway TLS certificate is approaching expiry with increased urgency. Diagnostic Information: Identify the expiring certificate and affected instance from available metric labels. Confirm that dependent peers trust the issuing CA and that intermediate certificates are correctly configured. Recovery: Execute the certificate-rotation plan. Deploy the new certificate and key through the approved secret-management process, then reload or restart the Diameter Gateway according to the runbook. Verify successful TLS handshakes with peers.

5.1.70 DIAM_GATEWAY_CERTIFICATE_EXPIRY_CRITICAL

Table 5-84 DIAM_GATEWAY_CERTIFICATE_EXPIRY_CRITICAL

Field	Details
Description	Diam-gw certificate expiry in less than a month for {{$labels.namespace}}.
Summary	Diam-gw certificate expiry in less than a month.
Severity	Critical
Expression	dgw_tls_cert_expiration_seconds - time() <= 2592000
OID	1.3.6.1.4.1.323.5.3.37.1.2.47
Metric Used	dgw_tls_cert_expiration_seconds
Recommended Actions	Cause: A Diameter Gateway certificate is close to expiry, creating a high risk of a production outage when it expires. Diagnostic Information: Verify the exact certificate-expiry timestamp and determine whether multiple instances are affected. Recovery: Expedite certificate renewal or rotation. After deployment, verify peer-connection stability without TLS alerts or handshake failures, confirm that Diameter Gateway logs show the new certificate has loaded, and confirm that the metric resets to a longer time to expiry.

5.1.71 DGW_TLS_CONNECTION_FAILURE

Table 5-85 DGW_TLS_CONNECTION_FAILURE

Field	Details
Description	{{$labels.reason}} failure when Diam gateway is an initiator.
Summary	TLS Connection failure when Diam gateway is an initiator.
Severity	Major
Expression	sum by (namespace,reason)(ocbsf_diam_failed_conn_network) > 0
OID	1.3.6.1.4.1.323.5.3.37.1.2.50.
Metric Used	ocbsf_diam_failed_conn_network
Recommended Actions	Cause: Diameter Gateway is failing to establish TLS connections as a client because of handshake exceptions. Common causes include expired or rotated certificates, a missing intermediate certificate chain, truststore mismatch or an unknown CA, SAN or CN mismatch, TLS policy or cipher mismatch, clock skew, or network middlebox interference. Diagnostic Information: Review failures by namespace and reason using `sum by(namespace,reason) (occnp_diam_failed_conn_network)`. If peer or endpoint labels are available, isolate the affected target. Correlate the issue with certificate-expiry alerts, validate NTP synchronization on Diameter Gateway nodes, and review Diameter Gateway logs for `unknown_ca`, `bad_certificate`, `certificate_expired`, `handshake_failure`, or `protocol_version` errors. Recovery: Renew or rotate the Diameter Gateway client certificate, update trust stores or CA bundles including intermediate certificates, and reload or restart Diameter Gateway according to the runbook. Align TLS versions and ciphers on both sides. If the issue is endpoint-specific, remediate the server certificate or configuration, or fail over to an alternate server or peer. Confirm that handshake failures stop increasing and peer connections remain established.

5.1.72 BINDING_REVALIDATION_PCF_BINDING_MISSING_MINOR

Table 5-86 BINDING_REVALIDATION_PCF_BINDING_MISSING_MINOR

Field	Details
Description	At least 30% but less than 50% of the PCF BINDING missing among all binding revalidation records in the last 5 minutes.
Summary	At least 30% but less than 50% of the PCF BINDING missing among all Binding Revalidation records in the last 5 minutes.
Severity	Minor
Expression	(sum by (namespace) (rate(ocbsf_binding_revalidation_pcfBinding_missing_total[5m])) / sum by (namespace) (rate(ocbsf_binding_revalidation_response_total[5m]))) * 100 >= 30 < 50
OID	1.3.6.1.4.1.323.5.3.37.1.2.51
Metric Used	ocbsf_binding_revalidation_pcfBinding_missing_total
Recommended Actions	Check BSF Management service health history. Increase binding audit frequency. For any additional guidance, contact My Oracle Support (https://support.oracle.com).

5.1.73 BINDING_REVALIDATION_PCF_BINDING_MISSING_MAJOR

Table 5-87 BINDING_REVALIDATION_PCF_BINDING_MISSING_MAJOR

Field	Details
Description	At least 50% but less than 70% of the PCF BINDING missing among all binding revalidation records in the last 5 minutes.
Summary	At least 50% but less than 70% of the PCF BINDING missing among all binding revalidation records in the last 5 minutes.
Severity	Major
Expression	(sum by (namespace) (rate(ocbsf_binding_revalidation_pcfBinding_missing_total[5m])) / sum by (namespace) (rate(ocbsf_binding_revalidation_response_total[5m]))) * 100 >= 50 < 70
OID	1.3.6.1.4.1.323.5.3.37.1.2.51
Metric Used	ocbsf_binding_revalidation_pcfBinding_missing_total
Recommended Actions	Check BSF Management service health history. Increase binding audit frequency. For any additional guidance, contact My Oracle Support (https://support.oracle.com).

5.1.74 BINDING_REVALIDATION_PCF_BINDING_MISSING_CRITICAL

Table 5-88 BINDING_REVALIDATION_PCF_BINDING_MISSING_CRITICAL

Field	Details
Description	At least 70% of the PCF BINDING missing among all binding revalidation records in the last 5 minutes.
Summary	At least 70% of the PCF BINDING missing among all binding revalidation records in the last 5 minutes.
Severity	Critical
Expression	(sum by (namespace) (rate(ocbsf_binding_revalidation_pcfBinding_missing_total[5m])) / sum by (namespace) (rate(ocbsf_binding_revalidation_response_total[5m]))) * 100 >= 70
OID	1.3.6.1.4.1.323.5.3.37.1.2.51
Metric Used	ocbsf_binding_revalidation_pcfBinding_missing_total
Recommended Actions	Check BSF Management service health history. Increase binding audit frequency. For any additional guidance, contact My Oracle Support (https://support.oracle.com).

5.1.75 BSF_STATE_NON_FUNCTIONAL_CRITICAL

Table 5-89 BSF_STATE_NON_FUNCTIONAL_CRITICAL

Field	Details
Description	BSF is in non functional state due to DB Cluster state down
Summary	BSF is in non functional state due to DB Cluster state down
Severity	Critical
Expression	appinfo_nfDbFunctionalState_current{nfDbFunctionalState="Not_Running"} == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.56
Metric Used	appinfo_nfdbfunctionalstate_current
Recommended Actions	Cause: The alert is raised because the BSF network function is non-functional due to the database cluster being down. Diagnostic Information: System monitoring indicates that the database cluster state is "Not Running" and is unreachable, preventing the BSF network function from operating normally. Recovery: Check and restore the database cluster to a running state. After recovery, verify that the BSF network function returns to operational status. Escalate to database administration if the issue persists.

5.1.76 POD_PENDING_REQUEST_CONGESTION_L1

Table 5-90 POD_PENDING_REQUEST_CONGESTION_L1

Field	Details
Description	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L1 for resource type queue.
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L1 for resource type queue.
Severity	Critical
Expression	ocbsf_pod_resource_congestion_state{type="queue"} == 2
OID	1.3.6.1.4.1.323.5.3.37.1.2.54
Metric Used	ocbsf_pod_resource_congestion_state
Recommended Actions	Cause: The pending HTTP request queue has reached `CONGESTION_L1`. Diagnostic Information: Confirm that `pod_resource_congestion_state{resourceType="queue"} == 2`. Verify queue-related `pod_resource_stress` values and review rejections by request URI and request method at L1. Recovery: Ensure that L1 queue thresholds are correct. If queues continue to grow, raise the L1 discard priority so that lower-priority requests are rejected earlier.

5.1.77 POD_PENDING_REQUEST_CONGESTION_L2

Table 5-91 POD_PENDING_REQUEST_CONGESTION_L2

Field	Details
Description	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L2 for resource type queue.
Summary	Pod Resource Congestion status of {{$labels.microservice}} service is Congestion_L2 for resource type queue.
Severity	Critical
Expression	ocbsf_pod_resource_congestion_state{type="queue"} == 3
OID	1.3.6.1.4.1.323.5.3.37.1.2.55
Metric Used	ocbsf_pod_resource_congestion_state
Recommended Actions	Cause: The pending HTTP request queue has reached `CONGESTION_L2`. Diagnostic Information: Confirm that `pod_resource_congestion_state{resourceType="queue"} == 3`. Examine queue-stress trends and rejection counters for low-priority traffic at L2. Recovery: Increase shedding at L2 by raising the discard priority, and review queue thresholds to prevent saturation.

5.1.78 AUDIT_NOT_RUNNING

Table 5-92 AUDIT_NOT_RUNNING

Field	Details
Description	Audit has not been running for at least 1 hour in pod {{$labels.pod}}.
Summary	Audit has been stuck in an unhealthy state for over 1 hour.
Severity	Critical
Expression	(increase(data_repository_invocations_seconds_count{method="getQueuedTablesToAudit",state="SUCCESS"}[1h])) == 0
OID	1.3.6.1.4.1.323.5.3.37.1.2.45
Metric Used	data_repository_invocations_seconds_count
Recommended Actions	Cause: No successful executions of `getQueuedTablesToAudit` have been recorded for at least one hour, as indicated by `increase(...[1h]) == 0`. This typically indicates that the audit job is not running, is stuck, or is failing before it records a successful execution. Diagnostic Information: Check in Prometheus or Grafana: Graph `data_repository_invocations_seconds_count{method="getQueuedTablesToAudit"}` by state to determine whether attempts are failing, for example with `ERROR` or `FAILURE`, or whether no attempts are occurring. Confirm that the metric series exists for the pod. If it is missing, verify Prometheus scrape health and target status for that pod. Check pod health and logs for the previous hour: Check for recent restarts, OOM kills, and readiness or liveness failures. Review application logs for audit scheduler or worker-thread issues, database connectivity problems, timeouts, deadlocks, or queue starvation. Recovery: If the pod is unhealthy or the audit worker is stuck, restart or roll the pod according to the team’s approved change and restart procedures. If non-success states are present, remediate the underlying error, for example by restoring database connectivity, correcting credentials, resolving timeouts or locks, or clearing blocking conditions. Then verify that successful invocations resume. If metrics are missing while the service is healthy, correct metrics emission or scraping configuration, including the Prometheus target, ServiceMonitor, or endpoint authentication. Then confirm that the counter is increasing. Validate recovery by confirming that the metric increases over a short interval, for example `increase(...[5m]) > 0`, and that audit backlog and health indicators return to normal.

5.1.79 BSF_PCF_BINDING_TABLE_MIGRATED_PERCENTAGE

Table 5-93 BSF_PCF_BINDING_TABLE_MIGRATED_PERCENTAGE

Field	Details
Description	Pcf binding table migration configuration should be updated to only use the pcf binding v2 table
Summary	Pcf binding table migration configuration should be updated to only use the pcf binding v2 table
Severity	Minor
Expression	max_over_time(((100 * sum by (namespace) (oc_db_active_session_count{Service="bsf-management-service",Table="pcf_binding_v2"}) / clamp_min(sum by (namespace) (oc_db_active_session_count{Service="bsf-management-service",Table=~"pcf_binding\|pcf_binding_v2"}),1) == 100) and (100 * sum by (namespace) (oc_db_active_session_count{Service="bsf-management-service",Table="pcf_binding_v2"} offset 1m) / clamp_min(sum by (namespace) (oc_db_active_session_count{Service="bsf-management-service",Table=~"pcf_binding\|pcf_binding_v2"} offset 1m),1) < 100))[3h:]) == 100
OID	1.3.6.1.4.1.323.5.3.37.1.2.57
Metric Used	oc_db_active_session_count
Recommended Actions	Cause: The alert is raised because all BSF pcf binding records in legacy v1 on current site have been migrated to pcf binding v2 table. Diagnostic Information: Verify that pcf_binding table is empty and transition Advanced Settings PCF_BINDING_TABLE_LOOKUP to 3. Recovery: No recovery steps are needed as it indicates to move into a migration complete status. Alert is cleared after 24 hours.

5.1.80 BSF_PCF_BINDING_TABLE_MIGRATION_INVALID_CONFIGURATION

Table 5-94 BSF_PCF_BINDING_TABLE_MIGRATION_INVALID_CONFIGURATION

Field	Details
Description	Pcf binding table migration configuration should be reviewed and updated to a valid configuration, invalid configurations: {{$labels.incompatibleFeatures}}. Examples of configurations that could trigger the alert: ENABLE_PCF_BINDING_TABLE_MIGRATION :true, PCF_BINDING_TABLE_LOOKUP : 0 ENABLE_PCF_BINDING_TABLE_MIGRATION :false, PCF_BINDING_TABLE_LOOKUP : 3
Summary	Pcf binding table migration, invalid configuration was set, latest valid values are used.
Severity	Critical
Expression	ocbsf_feature_incompatibility == 1
OID	1.3.6.1.4.1.323.5.3.37.1.2.58
Metric Used	ocbsf_feature_incompatibility
Recommended Actions	Cause: The alert is raised because the current configuration for Remove Index Based Lookup feature is having an incorrect combination for ENABLE_PCF_BINDING_TABLE_MIGRATION and PCF_BINDING_TABLE_LOOKUP_VALUE. Diagnostic Information: Verify configuration is valid according to the following rules: If ENABLE_PCF_BINDING_TABLE_MIGRATION is false, PCF_BINDING_TABLE_LOOKUP_VALUE can only have value 0. If ENABLE_PCF_BINDING_TABLE_MIGRATION is true, PCF_BINDING_TABLE_LOOKUP_VALUE can only have values between 1-3. Recovery: Alert is cleared once the configuration is updated to a valid configuration.

5.1.81 CERTIFICATE_EXPIRY_MINOR

Table 5-95 CERTIFICATE_EXPIRY_MINOR

Field	Details
Description	Certificate expiry in less than 6 months for {{$labels.namespace}}
Summary	Certificate expiry in less than 6 months
Severity	Minor
Expression	security_cert_x509_expiration_seconds - time() <= 15724800
OID	1.3.6.1.4.1.323.5.3.37.1.2.44
Metric Used	security_cert_x509_expiration_seconds
Recommended Actions	Cause: A certificate monitored by the security exporter is approaching expiry. Diagnostic Information: Identify the certificate with the shortest remaining validity and the affected resources. For example, review `min by (namespace,secret,cert_subject) (security_cert_x509_expiration_seconds - time())`. Recovery: Plan renewal or rotation through the approved certificate or PKI process. Update Kubernetes secrets or keystores, schedule the rollout before the major-alert window, and confirm that the expiration metric increases after rotation.

5.1.82 CERTIFICATE_EXPIRY_MAJOR

Table 5-96 CERTIFICATE_EXPIRY_MAJOR

Field	Details
Description	Certificate expiry in less than 3 months for {{$labels.namespace}}
Summary	Certificate expiry in less than 3 months.
Severity	Major
Expression	security_cert_x509_expiration_seconds - time() <= 7862400
OID	1.3.6.1.4.1.323.5.3.37.1.2.44
Metric Used	security_cert_x509_expiration_seconds
Recommended Actions	Cause: A certificate is approaching expiry with increased urgency. Diagnostic Information: Identify the workloads that use the certificate, such as ingress, egress, internal services, and databases. Recovery: Rotate the certificate and confirm that dependent clients trust the new certificate chain. Consider a staged rollout to avoid widespread TLS handshake failures.

5.1.83 CERTIFICATE_EXPIRY_CRITICAL

Table 5-97 CERTIFICATE_EXPIRY_CRITICAL

Field	Details
Description	Certificate expiry in less than a month for {{$labels.namespace}}
Summary	Certificate expiry in less than a month.
Severity	Critical
Expression	security_cert_x509_expiration_seconds - time() <= 2592000
OID	1.3.6.1.4.1.323.5.3.37.1.2.44
Metric Used	security_cert_x509_expiration_seconds
Recommended Actions	Cause: A certificate is close to expiry, creating a high likelihood of service disruption when it expires. Diagnostic Information: Validate the exact expiry time and determine the blast radius by identifying the services and peers that depend on the certificate. Recovery: Perform an emergency certificate rotation. Renew the certificate, deploy it through the approved secrets process, and reload or restart affected workloads according to the runbook. Confirm successful TLS handshakes, the absence of connection-failure alerts, and reset of the expiration metric.

5.1.84 AUDIT_STALE_NOTIFY_404_RESPONSE_CRITICAL

Table 5-98 AUDIT_STALE_NOTIFY_404_RESPONSE_CRITICAL

Field	Details
Description	At least 60 % of the BSF Audit Notification Requests sent to PCF to check for Suspected Stale Sessions have responded with a 5xx or 4xx (excluding 404) Status in the last 10 minutes.
Summary	{{ $value }} % of the BSF Audit Notification Requests sent to PCF to check for Suspected Stale Sessions have responded with a 5xx or 4xx (excluding 404) Status in the last 10 minutes.
Severity	CRITICAL
Expression	(sum by (namespace) (increase(ocbsf_query_response_count_total{response_code="404"}[10m])) / sum by (namespace) (increase(ocbsf_query_response_count_total[10m]))) * 100 >= 60
OID	1.3.6.1.4.1.323.5.3.37.1.2.62
Metric Used	ocbsf_query_response_count_total
Recommended Actions	Cause: More than 60% of BSF audit notification requests sent to PCF to check suspected stale sessions returned HTTP 404 in the last 10 minutes. Diagnostic Information: Verify whether PCF returns `404 NOT_FOUND` for the BSF PCF bindings used by audit query or notification requests. Recovery: The alert clears when BSF audit requests to PCF receive a successful 2xx response or another HTTP error response.

5.1.85 AUDIT_STALE_NOTIFY_404_RESPONSE_MAJOR

Table 5-99 AUDIT_STALE_NOTIFY_404_RESPONSE_MAJOR

Field	Details
Description	At least 40 % of the BSF Audit Notification Requests sent to PCF to check for Suspected Stale Sessions have responded with a 5xx or 4xx (excluding 404) Status in the last 10 minutes.
Summary	{{ $value }} % of the BSF Audit Notification Requests sent to PCF to check for Suspected Stale Sessions have responded with a 5xx or 4xx (excluding 404) Status in the last 10 minutes.
Severity	MAJOR
Expression	(sum by (namespace) (increase(ocbsf_query_response_count_total{response_code="404"}[10m])) / sum by (namespace) (increase(ocbsf_query_response_count_total[10m]))) * 100 >= 40 < 60
OID	1.3.6.1.4.1.323.5.3.37.1.2.62
Metric Used	ocbsf_query_response_count_total
Recommended Actions	Cause: More than 40% and less than 60% of BSF audit notification requests sent to PCF to check suspected stale sessions returned HTTP 404 in the last 10 minutes. Diagnostic Information: Verify whether PCF returns `404 NOT_FOUND` for the BSF PCF bindings used by audit query or notification requests. Recovery: The alert clears when BSF audit requests to PCF receive a successful 2xx response or another HTTP error response.

5.1.86 AUDIT_STALE_NOTIFY_404_RESPONSE_MINOR

Table 5-100 AUDIT_STALE_NOTIFY_404_RESPONSE_MINOR

Field	Details
Description	At least 20 % of the BSF Audit Notification Requests sent to PCF to check for Suspected Stale Sessions have responded with a 5xx or 4xx (excluding 404) Status in the last 10 minutes.
Summary	{{ $value }} % of the BSF Audit Notification Requests sent to PCF to check for Suspected Stale Sessions have responded with a 5xx or 4xx (excluding 404) Status in the last 10 minutes.
Severity	MINOR
Expression	(sum by (namespace) (increase(ocbsf_query_response_count_total{response_code="404"}[10m])) / sum by (namespace) (increase(ocbsf_query_response_count_total[10m]))) * 100 >= 20 < 40
OID	1.3.6.1.4.1.323.5.3.37.1.2.62
Metric Used	ocbsf_query_response_count_total
Recommended Actions	Cause: More than 20% and less than 40% of BSF audit notification requests sent to PCF to check suspected stale sessions returned HTTP 404 in the last 10 minutes. Diagnostic Information: Verify whether PCF returns `404 NOT_FOUND` for the BSF PCF bindings used by audit query or notification requests. Recovery: The alert clears when BSF audit requests to PCF receive a successful 2xx response or another HTTP error response.

5.1.87 RAA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 5-101 RAA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	RAA Rx fail count exceeds the critical threshold limit.
Summary	RAA Rx fail count exceeds the critical threshold limit.
Severity	CRITICAL
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="RAA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="RAA", direction="in"}[5m]))) 100 > 90
OID	1.3.6.1.4.1.323.5.3.37.1.2.59
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: More than 90% of received Diameter RAA responses are non-success responses. Diagnostic Information: Check the response-code distribution, peer reachability, TLS, routing, realm configuration, and logs for protocol or AVP errors. Recovery: Restore connectivity or correct the peer configuration. If load-related, reduce traffic, scale capacity, or apply traffic shedding. Escalate with diagnostics if the issue persists.

5.1.88 RAA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 5-102 RAA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	RAA Rx fail count exceeds the major threshold limit.
Summary	RAA Rx fail count exceeds the major threshold limit.
Severity	MAJOR
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="RAA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="RAA", direction="in"}[5m]))) 100 > 80 <=90
OID	1.3.6.1.4.1.323.5.3.37.1.2.59
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: Between 80% and 90% of received Diameter RAA responses are non-success responses. Diagnostic Information: Check the response-code distribution, peer reachability, TLS, routing, and realm configuration. Recovery: Correct connectivity or routing issues. If systemic, reduce load or scale capacity. Monitor the trend and escalate if it persists or worsens.

5.1.89 RAA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 5-103 RAA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	RAA Rx fail count exceeds the minor threshold limit.
Summary	RAA Rx fail count exceeds the minor threshold limit.
Severity	MINOR
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="RAA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="RAA", direction="in"}[5m]))) 100 > 60 <=80
OID	1.3.6.1.4.1.323.5.3.37.1.2.59
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: Between 60% and 80% of received Diameter RAA responses are non-success responses. Diagnostic Information: Check the response-code distribution and correlate the increase with peer latency, recent configuration changes, and logs. Recovery: Correct any identified peer, routing, TLS, or protocol issue, and monitor the trend.

5.1.90 STA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 5-104 STA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	STA Rx fail count exceeds the critical threshold limit.
Summary	STA Rx fail count exceeds the critical threshold limit.
Severity	CRITICAL
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="STA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="STA", direction="in"}[5m]))) 100 > 90
OID	1.3.6.1.4.1.323.5.3.37.1.2.60
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: More than 90% of received Diameter STA responses for Rx, application ID 16777236, are non-2xx. This indicates severe failures in session termination or related Rx procedures. Diagnostic Information: Validate the response ratio using `ocbsf_diam_response_network_total{msgType="STA", appId="16777236", direction="in"}`, grouped by response code. Identify the dominant failing response code and whether failures originate from a specific peer. Correlate the condition with Diameter peer connectivity, request timeouts, and congestion or overload state. Recovery: Restore Diameter peer connectivity or routing if impaired. If the condition is overload-driven, reduce load, increase resources, or apply stricter shedding for low-priority traffic. Collect the failing response-code distribution and logs, then escalate to peer or dependency owners with supporting evidence if required.

5.1.91 STA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 5-105 STA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	STA Rx fail count exceeds the major threshold limit.
Summary	STA Rx fail count exceeds the major threshold limit.
Severity	MAJOR
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="STA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="STA", direction="in"}[5m]))) 100 > 80 <=90
OID	1.3.6.1.4.1.323.5.3.37.1.2.60
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: Between 80% and 90% of received STA responses are non-2xx, indicating major degradation. Diagnostic Information: Perform the same checks as for the critical alert. Determine whether one response code dominates and whether the issue is isolated to specific pods or peers. Recovery: Apply early mitigation by correcting connectivity or routing, performing controlled restarts where the issue is localized, or reducing load where it is systemic. Escalate if the percentage increases or customer KPIs are affected.

5.1.92 STA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 5-106 STA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	STA Rx fail count exceeds the minor threshold limit.
Summary	STA Rx fail count exceeds the minor threshold limit.
Severity	MINOR
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="STA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="STA", direction="in"}[5m]))) 100 > 60 <=80
OID	1.3.6.1.4.1.323.5.3.37.1.2.60
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: Between 60% and 80% of received STA responses are non-2xx, providing an early warning of degradation. Diagnostic Information: Confirm the response ratio and dominant response code, and correlate the condition with peer health and latency. Recovery: Correct the most common failure cause, monitor the trend closely, and apply proactive capacity or load measures if the condition worsens.

5.1.93 ASA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Table 5-107 ASA_RX_FAIL_COUNT_EXCEEDS_CRITICAL_THRESHOLD

Field	Details
Description	ASA Rx fail count exceeds the critical threshold limit.
Summary	ASA Rx fail count exceeds the critical threshold limit.
Severity	CRITICAL
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="ASA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="ASA", direction="in"}[5m]))) 100 > 90
OID	1.3.6.1.4.1.323.5.3.37.1.2.61
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: More than 90% of received Diameter ASA responses are non-success responses. Diagnostic Information: Check the response-code distribution, peer reachability, TLS, routing, realm configuration, and logs for protocol or AVP errors. Recovery: Restore connectivity or correct the peer configuration. If load-related, reduce traffic, scale capacity, or apply traffic shedding. Escalate with diagnostics if the issue persists.

5.1.94 ASA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Table 5-108 ASA_RX_FAIL_COUNT_EXCEEDS_MAJOR_THRESHOLD

Field	Details
Description	ASA Rx fail count exceeds the major threshold limit.
Summary	ASA Rx fail count exceeds the major threshold limit.
Severity	MAJOR
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="ASA", responseCode!~"2.",direction="in"}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="ASA", direction="in"}[5m]))) 100 > 80 <=90
OID	1.3.6.1.4.1.323.5.3.37.1.2.61
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: Between 80% and 90% of received Diameter ASA responses are non-success responses. Diagnostic Information: Check the response-code distribution, peer reachability, TLS, routing, and realm configuration. Recovery: Correct connectivity or routing issues. If systemic, reduce load or scale capacity. Monitor the trend and escalate if it persists or worsens.

5.1.95 ASA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Table 5-109 ASA_RX_FAIL_COUNT_EXCEEDS_MINOR_THRESHOLD

Field	Details
Description	ASA Rx fail count exceeds the minor threshold limit.
Summary	ASA Rx fail count exceeds the minor threshold limit.
Severity	MINOR
Expression	(sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="ASA", responseCode!~"2."}[5m])) / sum by (namespace)(rate(ocbsf_diam_response_network_total{appId="16777236", msgType="ASA", direction="in"}[5m]))) 100 > 60 <=80
OID	1.3.6.1.4.1.323.5.3.37.1.2.61
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: Between 60% and 80% of received Diameter ASA responses are non-success responses. Diagnostic Information: Check the response-code distribution and correlate the increase with peer latency, recent configuration changes, and logs. Recovery: Correct any identified peer, routing, TLS, or protocol issue, and monitor the trend.

5.1.96 EGRESS_GATEWAY_DNS_RESOLUTION_FAILURE

Table 5-110 EGRESS_GATEWAY_DNS_RESOLUTION_FAILURE

Field	Details
Description	Egress Gateway encountered DNS SRV resolution failure.
Summary	"$labels.microservice is unable to route to SCP in namespace $labels.namespace due to DNS-SRV lookup failure for the FQDN $labels.vfqdn"
Severity	Major
Expression	(increase(ocbsf_oc_egressgateway_resolve_fqdn_from_ars_failure_total[2m])) > 0
OID	1.3.6.1.4.1.323.5.3.37.1.2.63
Metric Used	ocbsf_oc_egressgateway_resolve_fqdn_from_ars_failure_total
Recommended Actions	Cause: A misconfiguration prevents the Egress Gateway from routing requests to the SCP because DNS SRV lookup fails. Diagnostic Information: Confirm that SBI routing is enabled in the Egress Gateway ConfigMap. Check the configured FQDN or virtual FQDN value in the Egress Gateway ConfigMap. Verify that the FQDN or virtual FQDN value in the custom values file is correct. Recovery: Configure the correct FQDN or virtual FQDN. The alert clears after the Egress Gateway can successfully resolve and route to the SCP.

5.1.97 DIAMETER_POD_ERROR_RESPONSE_CRITICAL

Table 5-111 DIAMETER_POD_ERROR_RESPONSE_CRITICAL

Field	Details
Description	At least 10% of the Diam Response connection requests failed with error DIAMETER_UNABLE_TO_DELIVER in pod {{$labels.pod}}
Summary	At least 10% of the Diam Response requests failed with error DIAMETER_UNABLE_TO_DELIVER.
Severity	Critical
Expression	(sum by (pod) (rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])))/ (sum by (pod) (rate(ocbsf_diam_response_network_total[2m]))) * 100 >=10
OID	1.3.6.1.4.1.323.5.3.37.1.2.46
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: The Diameter pod is returning a critical level of error responses. Diagnostic Information: Check the affected pod, peer connectivity, TLS status, routing, and error logs. Recovery: Restore connectivity, correct configuration or protocol errors, and restart only the affected pod if necessary. Reduce traffic or scale capacity if overload is contributing. Escalate if recovery is unsuccessful.

5.1.98 DIAMETER_POD_ERROR_RESPONSE_MAJOR

Table 5-112 DIAMETER_POD_ERROR_RESPONSE_MAJOR

Field	Details
Description	At least 5% of the Diam Response connection requests failed with error DIAMETER_UNABLE_TO_DELIVER in pod {{$labels.pod}}
Summary	At least 5% of the Diam Response requests failed with error DIAMETER_UNABLE_TO_DELIVER.
Severity	Major
Expression	(sum by (pod) (rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])))/ (sum by (pod) (rate(ocbsf_diam_response_network_total[2m]))) * 100 >=5
OID	1.3.6.1.4.1.323.5.3.37.1.2.46
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: The Diameter pod is returning a major level of error responses. Diagnostic Information: Check the affected pod, peer connectivity, TLS status, routing, and recent changes. Recovery: Correct identified issues and monitor recovery. If the condition persists or increases, reduce load, scale capacity, or escalate with diagnostics.

5.1.99 DIAMETER_POD_ERROR_RESPONSE_MINOR

Table 5-113 DIAMETER_POD_ERROR_RESPONSE_MINOR

Field	Details
Description	At least 1% of the Diam Response connection requests failed with error DIAMETER_UNABLE_TO_DELIVER in pod {{$labels.pod}}
Summary	At least 1% of the Diam Response requests failed with error DIAMETER_UNABLE_TO_DELIVER.
Severity	Minor
Expression	(sum by (pod) (rate(ocbsf_diam_response_network_total{responseCode="3002"}[2m])))/ (sum by (pod) (rate(ocbsf_diam_response_network_total[2m]))) * 100 >=1
OID	1.3.6.1.4.1.323.5.3.37.1.2.46
Metric Used	ocbsf_diam_response_network_total
Recommended Actions	Cause: The Diameter pod is returning a minor level of error responses. Diagnostic Information: Review pod logs, response codes, peer latency, and recent configuration changes. Recovery: Correct any identified connectivity, routing, TLS, or protocol issue, and monitor for recurrence.

5.1.100 DIAM_GATEWAY_CERTIFICATE_EXPIRY_MINOR

Table 5-114 DIAM_GATEWAY_CERTIFICATE_EXPIRY_MINOR

Field	Details
Description	Diam-gw certificate expiry in less than 6 months for {{$labels.namespace}}
Summary	Diam-gw certificate expiry in less than 6 months
Severity	Minor
Expression	dgw_tls_cert_expiration_seconds - time() <= 15724800
OID	1.3.6.1.4.1.323.5.3.37.1.2.47
Metric Used	dgw_tls_cert_expiration_seconds
Recommended Actions	Cause: A Diameter Gateway TLS certificate is approaching expiry, creating a risk of future connection failures if it is not rotated. Diagnostic Information: Identify the expiring certificate and affected instance. Where available, review `min by (instance,cert_subject,cert_name) (dgw_tls_cert_expiration_seconds - time())`, adjusting labels for the environment. Recovery: Start the certificate renewal process. Create and approve the new certificate, validate the certificate chain and SANs, and schedule rotation before the major-alert window. Confirm that Diameter Gateway can load the new certificate in a staging or non-production environment first.

5 BSF Alerts

5.1 List of Alerts