OSO Metrics and Alerts

4 OSO Metrics and Alerts

4.1 OSO Metrics

This section includes information about metrics for Oracle Communications Cloud Native Core, Operations Services Overlay

The name of the metrics may contain suffix such as total, seconds, max and so on. It gets added by the micrometer registry if it is not present in the metrics name. The metric name has the following format for suffix: <Basename of the metric>_<Suffix>

Table 4-1 Metrics type and Suffix

Metric Type	Suffix	Description
Counter	_total	Represents the total number of occurrences of an event or traffic, such as measuring the total amount of traffic received and transmitted by OSO, and so on.
Gauge	NA	Represents a single numerical value that changes randomly. This metric type is used to measure various parameters, such as OSO load values, memory usage, and so on.
Histogram	_max, _bucket, _count, or _sum	Represents the sample observations like request durations or response sizes and counts them in configurable buckets.

Latency Metrics Format for OSO

The following metrics *_latency_seconds_[suffix] continue to be supported.

*_latency_seconds_max
*_latency_seconds_bucket
*_latency_seconds_count
*_latency_seconds_sum

Dimensions Legend for the Metrics

The following table includes the details about the metrics dimensions:

Table 4-2 Dimensions Legend

Dimension	Description
`client_id`	Indicates the Id of APM provided in custom values (kafkaGroupId). Sample Values: my-python-producer
`error_code`	Indicates the error code received for the request sent. Sample Values: 408/503/500
`resp_code`	Indicates the response code received with successful message. Sample Values: 200
`severity`	Indicates the severity.

4.1.1 Alert Processing Microservice Metrics

Table 4-3 kafka_producer_tx_messages

Field	Details
Description	Indicates the cumulative count of Kafka messages successfully produced by this client.
Type	Counter
Dimensions	client_id

Table 4-4 kafka_producer_tx_bytes

Field	Details
Description	Indicates the cumulative number of payload bytes successfully produced to Kafka.
Type	Counter
Dimensions	client_id

Table 4-5 kafka_producer_msg_queue_count

Field	Details
Description	Indicates the current number of messages waiting in the producer’s internal queue.
Type	Gauge
Dimensions	client_id

Table 4-6 kafka_producer_msg_queue_bytes

Field	Details
Description	Indicates the current total size (in bytes) of messages waiting in the producer queue.
Type	Gauge
Dimensions	client_id

Table 4-7 kafka_producer_outgoing_buffer_count

Field	Details
Description	Indicates the current number of queued messages ready to be transmitted to Kafka brokers.
Type	Gauge
Dimensions	client_id

Table 4-8 kafka_producer_requests

Field	Details
Description	Indicates the cumulative count of Kafka protocol requests sent by the producer.
Type	Counter
Dimensions	client_id

Table 4-9 kafka_producer_errors

Field	Details
Description	Indicates the cumulative count of producer-side errors, broken down by error_code.
Type	Counter
Dimensions	client_id, error_code

Table 4-10 kafka_producer_latency_ms_[suffix]

Field	Details
Description	Indicates the end-to-end produce latency distribution in milliseconds (tracked by histogram buckets), labeled by response code. Note: `_latency_ms_[suffix]` can represent any of the supported formats: `_latency_ms_max` `_latency_ms_bucket` `_latency_ms_count` `*_latency_ms_sum`
Type	Histogram
Dimensions	resp_code

Table 4-11 kafka_producer_msgs_received

Field	Details
Description	Indicates the cumulative count of messages received by the APM service, labeled by severity.
Type	Counter
Dimensions	severity

Table 4-12 kafka_producer_msgs_acked

Field	Details
Description	Indicates the cumulative count of messages acknowledged by the APM service, labeled by response code.
Type	Counter
Dimensions	resp_code

4.2 OSO Alerts

This section includes information about the OSO alerts.

The following table describes the various alert levels generated by OSO:

Table 4-13 Alerts Levels or Severity Types

Alerts Levels/Severity Types	Definition
Critical	Indicates a severe issue that poses a significant risk to safety, security, or operational integrity. It requires immediate response to address the situation and prevent serious consequences. Raised for conditions may affect the service of OSO.
Major	Indicates a more significant issue that has an impact on operations or poses a moderate risk. It requires prompt attention and action to mitigate potential escalation. Raised for conditions may affect the service of OSO.
Minor	Indicates a situation that is low in severity and does not pose an immediate risk to safety, security, or operations. It requires attention but does not demand urgent action. Raised for conditions may affect the service of OSO.
Info or Warn (Informational)	Provides general information or updates that are not related to immediate risks or actions. These alerts are for awareness and do not typically require any specific response. WARN and INFO alerts may not impact the service of OSO.

4.2.1 KafkaLostAck

Table 4-14 KafkaLostAck

Field	Details
Description	Ack loss ratio exceeded threshold for sustained traffic indicating message acknowledgment failures.
Summary	APM Kafka producer ack failures detected.
Severity	Critical
Condition	Ack loss ratio has stayed above 2% for 10m while traffic is sustained (>200 messages in 5m)
Metric Used	`kafka_producer_msgs_received_total`, `kafka_producer_msgs_acked_total` Note: Used to calculate acknowledgment loss ratio
Recommended Actions	The alert is cleared automatically when acknowledgment rates return to normal. Steps: Check APM-Kafka configuration (topic, Kafka IP, port, TLS/mTLS). Verify Kafka availability (pods, services, deployments). Check APM service pod logs for high traffic or large message saturation. Investigate network issues (latency, packet loss). Correlate with other alerts such as producer errors or queue issues. If the issue persists, contact My Oracle Support.

4.2.2 KafkaProducerMsgsErrors

Table 4-15 KafkaProducerMsgsErrors

Field	Details
Description	Kafka producer is experiencing sustained message send failures.
Summary	APM Kafka producer message failure detected.
Severity	Critical
Condition	Producer send errors exceeded 5 in 2m and persisted for 5m, indicating a sustained failure condition.
Metric Used	`kafka_producer_errors` Note: Tracks producer-side errors categorized by error_code.
Recommended Actions	The alert is cleared automatically when error rate drops below threshold. Steps: Check APM-Kafka configuration (topic, Kafka IP, port, TLS/mTLS). Inspect APM service pod logs for specific error codes. Validate TLS/mTLS certificates used by APM. Verify topic existence and partition availability. Check for broker overload or request rejection issues.

4.2.3 KafkaProducerQueueCritical

Table 4-16 KafkaProducerQueueCritical

Field	Details
Description	Kafka producer queue depth is critically high indicating backlog in message delivery.
Summary	APM Kafka producer queue critically high.
Severity	Critical
Condition	Producer queue depth exceeded 1000 within the last 5m and remained elevated for 5m.
Metric Used	`kafka_producer_msg_queue_count` Note: Represents number of messages pending in producer queue.
Recommended Actions	The alert is cleared automatically when queue depth returns to normal levels. Steps:: Check if related alerts (KafkaLostAck, KafkaProducerMsgsErrors) are also triggered. Compare producer send rate vs acknowledgment rate using Prometheus. Run queries: `kafka_producer_msgs_received_total` and `kafka_producer_msgs_acked_total` Check APM service pod logs for retries and reconnects. If traffic is high, reduce producer rate or increase partitions. If Kafka is slow, investigate broker, network, or storage bottlenecks.