6 cnDBTier Alerts

cnDBTier generates alerts when cnDBTier meets a specified condition. You can access the alerts using the Prometheus dashboard and take necessary actions. Prometheus gets installed as part of common services during the vCNE installation. This section provides details about the available cnDBTier alerts.

6.1 cnDBTier Remote Server Backup Transfer Status Alerts

This section provides details about the cnDBTier remote server backup transfer status alerts.

Table 6-1 REMOTE_SERVER_BACKUP_TRANSFER_FAILED

Field Details
Description This alert is triggered with major severity when the transfer of backup to a remote server fails.
Summary Secure transfer of backup to remote server failed on cnDBTier site {{ $labels.site_name }}
Severity major
Condition db_tier_remote_server_backup_transfer_status == 1
Expression Validity NA
SNMP Trap ID 2031
Affects Service (Y/N) N
Recommended Action

Cause: The transfer of backup to remote server failed.

Diagnostic Information: Check the status of the db_tier_remote_server_backup_transfer_status metric (Table 5-80).

Recovery: This alert is cleared automatically when the backup transfer status is updated from the failed state to other states, that is, the db_tier_remote_server_backup_transfer_status metric value is updated to any value other than 1.

To recover from this issue:
  • check if the remote server is in a healthy state
  • check the network
  • check if enough space is available

For any assistance, collect the logs and contact My Oracle Support.

6.2 cnDBTier Backup Transfer Status Alerts

This section provides details about the cnDBTier backup transfer status alerts.

Table 6-2 BACKUP_TRANSFER_LOCAL_FAILED

Field Details
Description This alert is triggered with major severity when the system fails to transfer the backup from the data node to the replication service pod on the cnDBTier site (db_tier_backup_transfer_status metric value is 2).
Summary Failed to transfer backup from data node to replication service pod on cnDBTier site {{ $labels.site_name }}
Severity major
Condition db_tier_backup_transfer_status == 2
Expression Validity NA
SNMP Trap ID 2026
Affects Service (Y/N) Y
Recommended Action

Cause: The system failed to transfer a backup from the data node to the replication service pod on a cnDBTier site.

Diagnostic Information: The db_tier_backup_transfer_status metric (Table 5-79) provides information about the backup transfer status.

Recovery: This alert is cleared automatically when the db_tier_backup_transfer_status metric is updated to a value other than 2.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-3 BACKUP_TRANSFER_FAILED

Field Details
Description This alert is triggered with major severity when the backup transfer failed as the system failed to transfer the backup to the remote site from the cnDBTier site (db_tier_backup_transfer_status metric value is 3).
Summary Failed to transfer backup to remote site from cnDBTier site {{ $labels.site_name }}
Severity major
Condition db_tier_backup_transfer_status == 3
Expression Validity NA
SNMP Trap ID 2027
Affects Service (Y/N) Y
Recommended Action

Cause: The system failed to transfer a backup from the cnDBTier site to a remote site.

Diagnostic Information: The db_tier_backup_transfer_status metric (Table 5-79) provides information about the backup transfer status.

Recovery: This alert is cleared automatically when the db_tier_backup_transfer_status metric is updated to a value other than 3.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-4 BACKUP_TRANSFER_IN_PROGRESS

Field Details
Description This alert is triggered with info severity when the backup transfer is in progress on the cnDBTier site (db_tier_backup_transfer_status metric value is 1).
Summary Backup Transfer is In Progress on cnDBTier site {{ $labels.site_name }}
Severity info
Condition db_tier_backup_transfer_status == 1
Expression Validity NA
SNMP Trap ID 2028
Affects Service (Y/N) N
Recommended Action

Cause: Backup transfer is in progress on the cnDBTier site.

Diagnostic Information: The db_tier_backup_transfer_status metric (Table 5-79) provides information about the backup transfer status.

Recovery: This alert is cleared automatically when the db_tier_backup_transfer_status metric is updated to a value other than 1.

For any assistance, collect the logs and contact My Oracle Support.

6.3 cnDBTier Heartbeat Alerts

This section provides details about cnDBTier heartbeat alerts.

Table 6-5 HEARTBEAT_FAILED

Field Details
Description This alert is triggered with critical severity when HeartBeat fails on a remote site.
Summary HeartBeat failed on cnDBTier site {{ $labels.site_name }} connected to mate site {{ $labels.mate_site_name }} on replication channel group id {{ $labels.replchannel_group_id }} and kubernetes namespace {{ $labels.namespace }}"
Severity critical
Condition db_tier_heartbeat_failure == 1
Expression Validity NA
SNMP Trap ID 2025
Affects Service (Y/N) Y
Recommended Action

Cause: The system is unable to connect to remote site and Heartbeat failed.

Diagnostic Information: The db_tier_heartbeat_failure metric (Table 5-78) provides information about the heartbeat status and indicates whether the remote site is reachable or not.

Recovery: This alert is cleared automatically when the db_tier_heartbeat_failure metric is 0.

For any assistance, collect the logs and contact My Oracle Support.

6.4 cnDBTier BinLog Injector Thread Alerts

This section provides details about cnDBTier BinLog injector alerts.

Table 6-6 BINLOG_INJECTOR_STOPPED

Field Details
Description This alert is triggered with critical severity when Bin Log Injector stops working.
The value of db_tier_binlog_injector_thread or db_tier_binlog_injector_thread_latest_epoch indicates the status of Bin Log Injector:
  • 0: indicates that the Bin Log Injector thread is not stopped for the specified node ID
  • 1: indicates that the Bin Log Injector thread is stopped for the specified node ID
Summary BinLog Injector Thread is stopped for MySQL node having node id {{ $labels.node_id }} on cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition

db_tier_binlog_injector_thread_latest_epoch == 1

or

db_tier_binlog_injector_thread == 1

Expression Validity NA
SNMP Trap ID 2024
Affects Service (Y/N) Y
Recommended Action

Cause: Bin Log Injector thread stalled for the replication SQL node.

Diagnostic Information: The db_tier_binlog_injector_thread_latest_epoch or db_tier_binlog_injector_thread metrics (Table 5-77 or Table 5-76) provide information whether the Bin Log Injector thread is stalled or not.

Recovery: This alert is cleared automatically when the db_tier_binlog_injector_thread_latest_epoch or db_tier_binlog_injector_thread metric is 0.

For any assistance, collect the logs and contact My Oracle Support.

6.5 cnDBTier Replication Error Skip Alerts

This section provides details about the cnDBTier replication error skip alertss.

Table 6-7 REPLICATION_SWITCHOVER_DUE_CLUSTERDISCONNECT

Field Details
Description This alert is triggered when switch over happens on an API node due to configured cluster disconnect error, if skip replication error is enabled.
Summary Replication channel on SQL node with node ID {{ $labels.node_id }} had switchover due to cluster disconnecterror number {{ $labels.error_number }}
Severity info
Condition db_tier_replication_switchover_due_to_clusterdisconnect == 1
Expression Validity NA
SNMP Trap ID 2019
Affects Service (Y/N) N
Recommended Action

Cause: Skip replication error is enabled on an API node and a switchover occurred on the node as the configured cluster disconnected.

Diagnostic Information: The db_tier_replication_switchover_due_to_clusterdisconnect metric (Table 5-75) provides information whether a switchover occurred on an API node.

Recovery: This alert is cleared automatically one hour after the event.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-8 REPLICATION_TOO_MANY_EPOCHS_LOST

Field Details
Description This alert is triggered when the epochs lost due to skip error is greater than 10000 and less than or equal to 80000.

This alert is cleared one hour after the event.
Summary Too many epochs are lost for skipping replication errors
Severity major
Condition (db_tier_epochs_lost_due_to_skiperror > 10000) and (db_tier_epochs_lost_due_to_skiperror <= 80000)
Expression Validity NA
SNMP Trap ID 2020
Affects Service (Y/N) N
Recommended Action

Cause: Between 10000 and 80000 epochs are lost due to skip errors.

Diagnostic Information: The db_tier_epochs_lost_due_to_skiperror metric (Table 5-74) provides information about the number of epochs lost due to skip errors.

Recovery: This alert is cleared automatically one hour after the event.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-9 REPLICATION_SKIP_ERRORS_LOW

Field Details
Description This alert is triggered when the replication is halted due to skip error count less than or equal to 5.

This alert is cleared one hour after the event.
Summary Cross-site replication errors are skipped
Severity minor
Condition (db_tier_replication_halted_due_to_skiperror > 0) and (db_tier_replication_halted_due_to_skiperror <= 5)
Expression Validity NA
SNMP Trap ID 2021
Affects Service (Y/N) N
Recommended Action

Cause: Replication halted due to less than five skip errors.

Diagnostic Information: The db_tier_replication_halted_due_to_skiperror metric (Table 5-73) provides information about the number of skip errors due to which the replication halted.

Recovery: This alert is cleared automatically one hour after the event.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-10 REPLICATION_SKIP_ERRORS_HIGH

Field Details
Description This alert is triggered when the replication is halted due to skip error counts greater than 5.

This alert is cleared one hour after the event.
Summary Cross-site replication errors skipped are high
Severity major
Condition db_tier_replication_halted_due_to_skiperror > 5
Expression Validity NA
SNMP Trap ID 2022
Affects Service (Y/N) N
Recommended Action

Cause: Replication halted due to more than five skip errors.

Diagnostic Information: The db_tier_replication_halted_due_to_skiperror metric (Table 5-73) provides information about the number of skip errors due to which the replication halted.

Recovery: This alert is cleared automatically one hour after the event.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-11 REPLICATION_EPOCHS_LOST

Field Details
Description This alert is triggered when the epochs lost due to skip error is greater than 0 and less than 2000.

This alert is cleared one hour after the event.
Summary Epochs are lost for skipping replication errors
Severity info
Condition db_tier_epochs_lost_due_to_skiperror > 0 and db_tier_epochs_lost_due_to_skiperror <= <Configured epoch interval lower threshold>
Expression Validity NA
SNMP Trap ID 2023
Affects Service (Y/N) N
Recommended Action

Cause: Less than 2000 epochs are lost due to skip errors.

Diagnostic Information: The db_tier_epochs_lost_due_to_skiperror metric (Table 5-74) provides information about the number of epochs lost due to skip errors.

Recovery: This alert is cleared automatically one hour after the event.

For any assistance, collect the logs and contact My Oracle Support.

6.6 cnDBTier Georeplication Recovery Status Alerts

This section provides details about the cnDBTier georeplication recovery status alerts.

Table 6-12 GEOREPLICATION_RECOVERY_IN_PROGRESS

Field Details
Description This alert is triggered with critical severity when the georeplication recovery is in progress and the alert is cleared when georeplication recovery is complete.
Summary Identified cnDBTier Site {{ $labels.site_name }} georeplication recovery is in progress for kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition db_tier_georeplication_recovery_state == 1
Expression Validity 1m
SNMP Trap ID 2018
Affects Service (Y/N) Y
Recommended Action

Cause: When you perform georeplication recovery to recover failed site from a healthy site, that is when georeplication recovery is in progress.

Diagnostic Information: The db_tier_georeplication_recovery_state metric (Table 5-29) provides information whether georeplication recovery is in progress.

Recovery: This alert is cleared automatically when the georeplication recovery is complete.

For any assistance, collect the logs and contact My Oracle Support.

6.7 cnDBTier Cluster Status Alerts

This section provides details about cnDBTier cluster status alerts.

Table 6-13 CLUSTER_DOWN

Field Details
Description This alert is triggered with critical severity when cnDBTier NDB cluster is not UP.
Summary MySQL Cluster is down for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition db_tier_cluster_status == 0
Expression Validity 1m
SNMP Trap ID 2017
Affects Service (Y/N) Y
Recommended Action
Cause:
  • When pod restarts due to Kubernetes liveliness or readiness probe failures.
  • When cnDBTier application restarts or fails to start.
Diagnostic Information:
  • Run the following command to check the status of cnDBTier namespace:
    kubectl -n <namespace> exec -it ndbmgmd-0 -- ndb_mgm -e show
    The cluster is down if:
    • the ndbappmysqld pods are down, not running, and not connected
    • the remaining pods are not running and not connected
  • Check Kubernetes events for probe failures in the platform logs.
  • Check if any exception is reported in the cnDBTier application logs.

Recovery: This alert is cleared automatically when the inactive pod is active.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-14 MYSQL_NDB_CLUSTER_DISCONNECT

Field Details
Description This alert is triggered with critical severity when cnDBTier NDB cluster is not UP.
Summary MySQL NDB Cluster Disconnected {{ $value }} times for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition db_tier_cluster_disconnect > 0
Expression Validity 1m
SNMP Trap ID 2034
Affects Service (Y/N) Y
Recommended Action
Cause:
  • When all ndbmtd pods or all ndbmtd pods of the same node group restart due to Kubernetes probe failures or infrastructure related issues.
Diagnostic Information:
  • Run the following command to check the status of cnDBTier namespace:
    kubectl -n <namespace> exec -it ndbmgmd-0 -- ndb_mgm -e show

    The cluster is down if: all data nodes or all data nodes of the same node group are not connected.

  • Check Kubernetes events for probe failures in the platform logs.
  • Check if there is any network fluctuation or platform related issue which can cause the ndbmtd pods to restart.
  • Check if any exception is reported in the cnDBTier application logs.

Recovery: This alert can be cleared by calling the /db-tier/reset/parameter/cluster_restart_disconnect REST API. For more information about this API, see cnDBTier Cluster Events APIs.

For any assistance, collect the logs and contact My Oracle Support.

6.8 cnDBTier Automated Backup Alerts

This section provides details about the cnDBTier automated backup alerts.

Table 6-15 BACKUP_FAILED

Field Details
Description This alert is triggered with minor severity when the backup service fails to complete the backup successfully.
Summary Could not backup database for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity minor
Condition db_tier_backup{status='FAILED'}
Expression Validity N/A
SNMP Trap ID 2011
Affects Service (Y/N) N
Recommended Action
Cause:
  • When backup service fails to complete the backup successfully.
  • When PVC size is not enough and as a result the backup fails.

Diagnostic Information: The db_tier_backup metric (Table 5-27) provides information if the backup failed or not.

Recovery: This alert is cleared automatically when the db_tier_backup metric status changes from the FAILED state to other states.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-16 BACKUP_PURGED_EARLY

Field Details
Description This alert is triggered with minor severity when the backup service purges old backups earlier than expected to create space for new backup.
Summary A backup was deleted prematurely for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity minor
Condition db_tier_backup{status='PURGED_EARLY'}
Expression Validity N/A
SNMP Trap ID 2012
Affects Service (Y/N) N
Recommended Action

Cause: When the backup service purges the old backups earlier than the expected time, to create space for a new backup.

Diagnostic Information: The db_tier_backup metric (Table 5-27) provides information if the backup is purged earlier than expected.

Recovery: This alert is cleared automatically when the db_tier_backup metric status changes from the PURGED_EARLY state to other states.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-17 BACKUP_SIZE_GROWTH

Field Details
Description This alert is triggered with minor severity whenever the current backup size exceeds 20% of the average of the previous backups.
Summary Backup size exceeded expected size for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity minor
Condition (db_tier_backup_used_disk_percentage/(avg_over_time(db_tier_backup_used_disk_percentage[5d])))>1.20
Expression Validity N/A
SNMP Trap ID 2013
Affects Service (Y/N) N
Recommended Action

Cause: When the current backup size exceeds 20% of the average of the previous backups.

Diagnostic Information: The db_tier_backup_used_disk_percentage metric (Table 5-25) provides information if the current backup size exceeds 20% of the average.

Recovery: This alert is cleared automatically when the db_tier_backup_used_disk_percentage metric value is reduced to the threshold percentage.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-18 BACKUP_STORAGE_LOW

Field Details
Description This alert is triggered with minor severity when the total backup size of the data node is >= 70% and < 80% of the total data node disk size.
Summary Disk storage on DATA node with node ID {{ $labels.node_id }} at {{ $value }} percent for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity minor
Condition (avg_over_time(db_tier_backup_used_disk_percentage[5m])>=70) and (avg_over_time(db_tier_backup_used_disk_percentage[5m])<80)
Expression Validity N/A
SNMP Trap ID 2014
Affects Service (Y/N) N
Recommended Action

Cause: When the total backup size of the data node is >= 70% and < 80% of the total data node disk size.

Diagnostic Information: The db_tier_backup_used_disk_percentage metric (Table 5-25) provides information if the current backup size is >= 70% and < 80% of the total data node disk size.

Recovery:
  • This alert is cleared automatically when the db_tier_backup_used_disk_percentage metric value is reduced to the threshold percentage.
  • Increase the disk size by performing the scaling procedure.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-19 BACKUP_STORAGE_LOW

Field Details
Description This alert is triggered with major severity when the total backup size of the data node is >= 80% and < 95% of the total data node disk size.
Summary Disk storage on DATA node with node ID {{ $labels.node_id }} at {{ $value }} percent for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity major
Condition (avg_over_time(db_tier_backup_used_disk_percentage[5m])>=80) and (avg_over_time(db_tier_backup_used_disk_percentage[5m])<95)
Expression Validity N/A
SNMP Trap ID 2015
Affects Service (Y/N) N
Recommended Action

Cause: When the total backup size of the data node is >= 80% and < 95% of the total data node disk size.

Diagnostic Information: The db_tier_backup_used_disk_percentage metric (Table 5-25) provides information if the current backup size is >= 80% and < 95% of the total data node disk size.

Recovery:
  • This alert is cleared automatically when the db_tier_backup_used_disk_percentage metric value is reduced to the threshold percentage.
  • Increase the disk size by performing the scaling procedure.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-20 BACKUP_STORAGE_FULL

Field Details
Description This alert is triggered with critical severity when the total backup size of the data node is >= 95% of the total data node disk size.
Summary Disk storage on DATA node with node ID {{ $labels.node_id }} is full for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition (avg_over_time(db_tier_backup_used_disk_percentage[5m])>=95)
Expression Validity N/A
SNMP Trap ID 2016
Affects Service (Y/N) N
Recommended Action

Cause: When the total backup size of the data node is >= 95% of the total data node disk size.

Diagnostic Information: The db_tier_backup_used_disk_percentage metric (Table 5-25) provides information if the current backup size is >= 95% of the total data node disk size.

Recovery:

  • This alert is cleared automatically when the db_tier_backup_used_disk_percentage metric value is reduced to the threshold percentage.
  • Increase the disk size by performing the scaling procedure.

Note: Take immediate action to avoid the cnDBTier cluster going out of service.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-21 DB_TIER_NDB_BACKUP_IN_PROGRESS

Field Details
Description This alert is triggered with minor severity when a data node backup is in progress in the current site.
Summary Indicates that a data node backup process is in progress in the current site.
Severity minor
Condition db_tier_ndb_backup_in_progress == 1
Expression Validity N/A
SNMP Trap ID 2037
Affects Service (Y/N) N
Recommended Action

Cause: When a data node backup is in progress in the current site.

Diagnostic Information: The db_tier_ndb_backup_in_progress metric (Table 5-28) provides information if a data node backup is in progress or not. Ensure that you don't make any schema changes until the backup completes.

Recovery: This alert is cleared automatically when the backup completes in the MySQL NDB cluster.

For any assistance, collect the logs and contact My Oracle Support.

6.9 cnDBTier Bin Log Usage Alerts

This section provides details about the cnDBTier binlog usage alerts.

Table 6-22 BINLOG_STORAGE_LOW

Field Details
Description This alert is triggered with a minor severity when the total BinLog size of the SQL node is >= 70% and < 80% of the total SQL node disk size.
Summary Disk storage on SQL node with node ID {{ $labels.node_id }} at {{ $value }} percent for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity minor
Condition (avg_over_time( db_tier_binlog_used_bytes_percentage[5m]) >= 70) and (avg_over_time( db_tier_binlog_used_bytes_percentage[5m] ) < 80)
Expression Validity 5m
SNMP Trap ID 2007
Affects Service (Y/N) N
Recommended Action

Cause: When the total BinLog size of the SQL node is >= 70% and < 80% of the total SQL node disk size.

Diagnostic Information: The db_tier_binlog_used_bytes_percentage metric (Table 5-22) provides information if the total BinLog size of the SQL node is >=70% and <80% of total SQL node disk size.

Recovery: This alert is cleared automatically when the value of the db_tier_binlog_used_bytes_percentage metric is reduced to the threshold value.

For any assistance, contact My Oracle Support.

Table 6-23 BINLOG_STORAGE_LOW

Field Details
Description This alert is triggered with major severity when the total BinLog size of the SQL node is >=80% and <95% of the total SQL node disk size.
Summary Disk storage on SQL node with node ID {{ $labels.node_id }} at {{ $value }} percent for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity major
Condition (avg_over_time( db_tier_binlog_used_bytes_percentage[5m]) >= 80) and (avg_over_time( db_tier_binlog_used_bytes_percentage[5m]) < 95)
Expression Validity 5m
SNMP Trap ID 2036
Affects Service (Y/N) N
Recommended Action

Cause: When the total BinLog size of the SQL node is >=80% and <95% of the total SQL node disk size.

Diagnostic Information: The db_tier_binlog_used_bytes_percentage metric (Table 5-22) provides information if the total BinLog size of the SQL node is >=80% and <95% of total SQL node disk size.

Recovery: This alert is cleared automatically when the value of the db_tier_binlog_used_bytes_percentage metric is reduced to the threshold value.

For any assistance, contact My Oracle Support.

Table 6-24 BINLOG_STORAGE_FULL

Field Details
Description This alert is triggered with critical severity when the total BinLog size of the SQL node is >= 95% of the total SQL node disk size.
Summary Disk storage on SQL node with node ID {{ $labels.node_id }} is full for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition avg_over_time( db_tier_binlog_used_bytes_percentage[5m]) >= 95
Expression Validity N/A
SNMP Trap ID 2008
Affects Service (Y/N) Y
Recommended Action

Cause: When the total BinLog size of the SQL node is >= 95% of the total SQL node disk size.

Diagnostic Information: The db_tier_binlog_used_bytes_percentage metric (Table 5-22) provides information if the total BinLog size of the SQL node is >= 95% of total SQL node disk size.

Recovery:

This alert is cleared automatically when the value of the db_tier_binlog_used_bytes_percentage metric is reduced to the threshold value.

Note: Take immediate action to avoid the SQL node going into Crashbackloop and becoming inaccessible.

For any assistance, contact My Oracle Support.

6.10 cnDBTier Replication Alerts

This section provides details about cnDBTier replication alerts.

Table 6-25 REPLICATION_CHANNEL_DOWN

Field Details
Description This alert is triggered with major severity when an ACTIVE channel goes to the FAILED state.
Summary Cross-site replication is down on node {{ $labels.node_id }} for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity major
Condition (db_tier_replication_status{role="failed"} == 0) or

(db_tier_replication_status{role="active"} == 0)
Expression Validity N/A
SNMP Trap ID 2005
Affects Service (Y/N) N
Recommended Action

Cause: When any ACTIVE channel goes to the FAILED state when the crosssite replication is down on a node.

Diagnostic Information: The following metrics provide information if the replication channel is down:
  • db_tier_replication_status{role="failed"} == 0 (Table 5-23)
  • db_tier_replication_status{role="active"} == 0 (Table 5-23)

Recovery:

This alert is cleared automatically when the cross-site replication is UP on the node and ACTIVE.

Note: Take immediate action to avoid the cnDBTier cluster going out of service.

For any assistance, contact My Oracle Support.

Table 6-26 REPLICATION_FAILED

Field Details
Description This alert is triggered with critical severity when all the channels are in the STANDBY or FAILED state.
Summary Cross-site replication is down for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition (count by (site_name, namespace, replchannel_group_id) (db_tier_replication_status) == count by (site_name, namespace, replchannel_group_id) (db_tier_replication_status{role="standby"})) or (count by (namespace, replchannel_group_id, site_name) (db_tier_replication_status) == count by (namespace, replchannel_group_id, site_name) (db_tier_replication_status{role="failed"}))
Expression Validity N/A
SNMP Trap ID 2006
Affects Service (Y/N) Y
Recommended Action

Cause: When all the channels are in the STANDBY or FAILED state as the cross-site replication is down for the cnDBTier site.

Diagnostic Information: The db_tier_replication_status metric (Table 5-23) provides information if the replication failed.

Recovery:

This alert is cleared automatically when the cross-site replication of the cnDBTier site is ACTIVE.

For any assistance, contact My Oracle Support.

Table 6-27 REPLICA_REPLICATION_DELAY_HIGH

Field Details
Description This alert is triggered when the last record read by the replica is more than five minutes behind the latest record written by the source.
Summary Replica replication on SQL node at {{ $labels.replica_node_ip }} is {{ $value }} seconds behind the source for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity major
Condition avg(avg_over_time(db_tier_replication_replica_delay[5m])) by (source_node_ip,replica_node_ip) >= 300 and avg(avg_over_time(db_tier_replication_replica_delay[5m])) by (source_node_ip,replica_node_ip) < 48*3600
Expression Validity 1m
SNMP Trap ID 2009
Affects Service (Y/N) N
Recommended Action

Cause: When the last record read by the worker node is more than 5 minutes and less than 48 hours behind the latest record written by the controller.

Diagnostic Information: The db_tier_replication_replica_delay metric (Table 5-24) provides information if there is a delay in the worker node replication.

Recovery:

This alert is cleared automatically when the when the db_tier_replication_replica_delay metric value is reduced below the defined threshold value.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-28 REPLICA_REPLICATION_FAILED

Field Details
Description This alert is triggered when the last record read by the replica is more than 48 hours behind the latest record written by the source.
Summary Replica replication has fallen more than 48 hours behind the source. Manual restore from backup may be required for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition avg(avg_over_time(db_tier_replication_replica_delay[5m])) by (source_node_ip,replica_node_ip) >= 48*3600
Expression Validity 1m
SNMP Trap ID 2010
Affects Service (Y/N) Y
Recommended Action

Cause: When the last record read by the worker node is more than 48 hours behind the latest record written by the controller.

Diagnostic Information: The db_tier_replication_replica_delay metric (Table 5-24) provides information if the worker node replication failed.

Recovery: Perform georeplication recovery for the DB sync. For procedures, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

This alert is cleared automatically when the db_tier_replication_replica_delay metric value is reduced below the defined threshold value.

For any assistance, collect the logs and contact My Oracle Support.

Table 6-29 GEOREPLICATION_RECOVERY_FAILED

Field Details
Description This alert is triggered with critical severity when georeplication recovery fails on a unhealthy site where georeplication recovery was started.
Summary Georeplication recovery has failed on cnDBTier Site {{ $labels.site_name }} from kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition db_tier_georeplication_recovery_state == 2
Expression Validity NA
SNMP Trap ID 2033
Affects Service (Y/N) Y
Recommended Action

Cause: Incorrect disk size, incorrect SSH key configurations, or other similar reasons.

Diagnostic Information: This alert indicates that georeplication recovery failed on a unhealthy site and replication couldn't be reestablished using the georeplication recovery procedure. This alert requires immediate attention.

Recovery: Check the configurations like SSH key or disk size. Contact My Oracle Support for additional support.

6.11 cnDBTier Memory Usage Alerts

This section provides details about the cnDBTier memory usage alerts.

Table 6-30 LOW_MEMORY

Field Details
Description This alert is triggered when the RAM usage of any node is greater than or equal to 80%.
Summary Node ID {{ $labels.node_id }}, memory utilization at {{ $value }} percent for memory type {{ $labels.memory_type }} for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity major
Condition ((avg_over_time(db_tier_memory_used_bytes{memory_type="Data memory"}[1m]) / avg_over_time(db_tier_memory_total_bytes{memory_type="Data memory"}[1m])) * 100) >= 80
Expression Validity 1m
SNMP Trap ID 2003
Affects Service (Y/N) N
Recommended Action

Cause: When the RAM or memory usage of any node reaches the major level of threshold value.

Diagnostic Information: Check if the memory usage of the following metrics are too high:

Recovery:

Reduce the incoming service request rate. This alert is cleared automatically when the memory usage of the cnDBTier worker pod is reduced below the defined threshold value.

For any assistance, contact My Oracle Support.

Table 6-31 OUT_OF_MEMORY

Field Details
Description This alert is triggered with critical severity when the RAM usage of any node is greater than or equal to 90%.
Summary Node ID {{ $labels.node_id }} out of memory for memory type {{ $labels.memory_type }} for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition ((avg_over_time(db_tier_memory_used_bytes{memory_type="Data memory"}[1m]) / avg_over_time(db_tier_memory_total_bytes{memory_type="Data memory"}[1m])) * 100) >= 90
Expression Validity 1m
SNMP Trap ID 2004
Affects Service (Y/N) Y
Recommended Action

Cause: When the RAM or memory usage of any node reaches the critical level of threshold value.

Diagnostic Information: Check if the memory usage of the following metrics are too high:

Recovery:

Reduce the incoming service request rate. This alert is cleared automatically when the memory usage of the cnDBTier worker pod is reduced below the defined threshold value.

Note: Take immediate action to avoid the cnDBTier cluster going out of service.

For any assistance, contact My Oracle Support.

6.12 cnDBTier CPU Usage Alerts

This section provides details about cnDBTier CPU usage alerts.

Table 6-32 HIGH_CPU

Field Details
Description This alert is triggered with major severity when the CPU usage of any data node is greater than or equal to 80%, and less than 90%.
Summary Node ID {{ $labels.node_id }} CPU utilization at {{ $value }} for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity major
Condition ((100 - (avg(avg_over_time(db_tier_cpu_os_idle[10m])) by (node_id))) >= 80) and ((100 - (avg(avg_over_time(db_tier_cpu_os_idle[10m])) by (node_id))) < 90)
Expression Validity 1m
SNMP Trap ID 2002
Affects Service (Y/N) N
Recommended Action

Cause: When the CPU utilization of any data node is greater than or equal to 80%, and less than 90%.

Diagnostic Information: Check the CPU threshold level status from the cnDBTier worker pod logs.

Recovery:

Reduce the incoming service request rate. This alert is cleared automatically when the CPU utilization is reduced below the threshold value.

For any assistance, contact My Oracle Support.

Table 6-33 HIGH_CPU

Field Details
Description This alert is triggered with critical severity when the CPU usage of any data node is greater than or equal to 90%.
Summary Node ID {{ $labels.node_id }} CPU utilization at {{ $value }} for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition (100 - (avg(avg_over_time(db_tier_cpu_os_idle[10m]))BY (node_id)))>= 90
Expression Validity 1m
SNMP Trap ID 2035
Affects Service (Y/N) N
Recommended Action

Cause: When the CPU utilization of any data node is greater than or equal to 90%.

Diagnostic Information: Check the CPU threshold level status from the cnDBTier worker pod logs.

Recovery:

Reduce the incoming service request rate. This alert is cleared automatically when the CPU utilization is reduced below the threshold value.

Note: Take immediate action to avoid the cnDBTier cluster going out of service.

For any assistance, contact My Oracle Support.

6.13 cnDBTier Node Status Alerts

The section provides details about cnDBTier node status alerts.

Table 6-34 NODE_DOWN

Field Details
Description This alert is raised with critical severity when the data node is down. db_tier_node_status value:
  • 0: indicates that a node is DOWN
  • 1: indicates that the node is UP
Summary MySQL {{ $labels.node_type }} node having node id {{ $labels.node_id }} is down for cnDBTier site {{ $labels.site_name }} and kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition db_tier_node_status == 0
Expression Validity N/A
SNMP Trap ID 2001
Affects Service (Y/N) Y
Recommended Action
Cause:
  • When pod restarts due to Kubernetes liveliness or readiness probe failures
  • When cnDBTier application restarts or fails to start
Diagnostic Information:
  • Run the following command to check the status of cnDBTier namespace:
    kubectl -n <namespace> exec -it ndbmgmd-0 -- ndb_mgm -e show
  • Check the Kubernetes events for probe failures in the platform logs.
  • Check if any exception is reported in the cnDBTier application logs.

Recovery:

This alert is cleared automatically when the inactive pod becomes active.

For any assistance, collect the application logs and contact My Oracle Support.

6.14 cnDBTier Node Data Volume Alerts

This section provides details about cnDBTier node data volume alerts.

Table 6-35 DB_TIER_API_SEND_NODE_DATA_VOLUME_LOW

Field Details
Description This alert is triggered when any NDB application node sends less data to NDB when compared to the other NDB application nodes.
Summary Send Node Data Volume Low for API Node ID {{ $labels.remote_node_id }} at kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition (((sum by (remote_node_id,namespace) (avg_over_time(rate(db_tier_node_transporter_bytes_received{node_type=~"ndbapp_node",namespace="<${CNDBTIER_NAMESPACE}>"}[5m])[15m:5m])))/scalar(sum (avg_over_time(rate(db_tier_node_transporter_bytes_received{node_type=~"ndbapp_node",namespace="<${CNDBTIER_NAMESPACE}>"}[5m])[15m:5m]))))*100) < (100/(scalar(count(count by (remote_node_id) (db_tier_node_transporter_bytes_received{node_type=~"ndbapp_node",namespace="<${CNDBTIER_NAMESPACE}>"})))*1.6))
Expression Validity NA
SNMP Trap ID 3001
Affects Service (Y/N) Y
Recommended Action

Cause: When NDB application node sends less data to NDB when compared to other NDB application nodes.

Diagnostic Information: The alert indicates that the NDB application node is slow, therefore check the underlying infrastructure.

Recovery:

This alert is cleared automatically when the NDB application node sends the data at the same rate as the other NDB application node.

For any assistance, contact My Oracle Support.

Table 6-36 DB_TIER_API_RECEIVE_NODE_DATA_VOLUME_LOW

Field Details
Description This alert is triggered when any NDB sends less data to any specific NDB application node when compared to the other NDB application nodes.
Summary Receive Node Data Volume Low for API Node ID {{ $labels.remote_node_id }} at kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition (((sum by (remote_node_id,namespace) (avg_over_time(rate(db_tier_node_transporter_bytes_sent{node_type=~"ndbapp_node",namespace="<${CNDBTIER_NAMESPACE}>"}[5m])[15m:5m])))/scalar(sum (avg_over_time(rate(db_tier_node_transporter_bytes_sent{node_type=~"ndbapp_node",namespace="<${CNDBTIER_NAMESPACE}>"}[5m])[15m:5m]))))*100) < (100/(scalar(count(count by (remote_node_id) (db_tier_node_transporter_bytes_sent{node_type=~"ndbapp_node",namespace="<${CNDBTIER_NAMESPACE}>"})))*1.6))
Expression Validity NA
SNMP Trap ID 3002
Affects Service (Y/N) Y
Recommended Action

Cause: When NDB application node sends less data to any specific NDB application node when compared to other NDB application nodes.

Diagnostic Information: The alert indicates that the NDB application node is slow, therefore check the underlying infrastructure.

Recovery:

This alert is cleared automatically when the NDB application node sends the data at the same rate as the other NDB application node.

For any assistance, contact My Oracle Support.

Table 6-37 DB_TIER_SEND_DATA_NODE_DATA_VOLUME_LOW

Field Details
Description This alert is triggered when any NDB doesn't send the traffic data in the required speed or when the speed is slower when compared to another data node.
Summary Send Data Node Data Volume Low for DATA Node ID {{ $labels.node_id }} at kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition ((sum by (node_id,namespace) (avg_over_time(rate(db_tier_node_transporter_bytes_sent{namespace="<${CNDBTIER_NAMESPACE}>"}[5m])[15m:5m]))/scalar(sum (avg_over_time(rate(db_tier_node_transporter_bytes_sent{namespace="<${CNDBTIER_NAMESPACE}>"}[5m])[15m:5m])))) * 100) < (100/(scalar(count(count by (node_id) (db_tier_node_transporter_bytes_sent{namespace="<${CNDBTIER_NAMESPACE}>"})))*1.6))
Expression Validity NA
SNMP Trap ID 3003
Affects Service (Y/N) Y
Recommended Action

Cause: When any NDB doesn't send the traffic data in the required speed or when the speed is slower when compared to another data node.

Diagnostic Information: The alert indicates that the NDB application node is slow, therefore check the underlying infrastructure.

Recovery:

This alert is cleared automatically when the NDB application node sends the same amount of data to other data node.

For any assistance, contact My Oracle Support.

Table 6-38 DB_TIER_RECEIVE_DATA_NODE_DATA_VOLUME_LOW

Field Details
Description This alert is triggered when any NDB doesn't receive the traffic data in the required speed or when the speed is slower when compared to another data node.
Summary Receive Data Node Data Volume Low for DATA Node ID {{ $labels.node_id }} at kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition ((sum by (node_id,namespace) (avg_over_time(rate(db_tier_node_transporter_bytes_received{namespace="<${CNDBTIER_NAMESPACE}>"}[5m])[15m:5m]))/scalar(sum (avg_over_time(rate(db_tier_node_transporter_bytes_received{namespace="<${CNDBTIER_NAMESPACE}>"}[5m])[15m:5m])))) * 100)< (100/(scalar(count(count by (node_id) (db_tier_node_transporter_bytes_received{namespace="<${CNDBTIER_NAMESPACE}>"})))*1.6))
Expression Validity NA
SNMP Trap ID 3004
Affects Service (Y/N) Y
Recommended Action

Cause: When any NDB doesn't receive the traffic data in the required speed or when the speed is slower when compared to another data node.

Diagnostic Information: The alert indicates that the NDB application node is slow, therefore check the underlying infrastructure.

Recovery:

This alert is cleared automatically when the NDB application node receives the same amount of data to other data node.

For any assistance, contact My Oracle Support.

Table 6-39 DB_TIER_DATA_NODE_SCAN_FRAGMENT_SLOW

Field Details
Description This alert is triggered when any data node scan fragment is slow when compared with other data nodes.
Summary Scan Fragment is Slow for DATA Node ID {{ $labels.node_id }} at kubernetes namespace {{ $labels.namespace }}
Severity critical
Condition (((sum by (node_id,comm_node_id,namespace) (rate(db_tier_tc_time_track_stats_total_scan_fragments_time{path_type="INTERNAL",namespace="<${CNDBTIER_NAMESPACE}>"}[15m]))/sum by (node_id,comm_node_id,namespace) (rate(db_tier_tc_time_track_stats_total_scan_fragments_count{path_type="INTERNAL",namespace="<${CNDBTIER_NAMESPACE}>"}[15m])))/ (scalar(sum(sum by (node_id,comm_node_id,namespace) (rate(db_tier_tc_time_track_stats_total_scan_fragments_time{path_type="INTERNAL",namespace="<${CNDBTIER_NAMESPACE}>"}[15m]))/sum by (node_id,comm_node_id,namespace) (rate(db_tier_tc_time_track_stats_total_scan_fragments_count{path_type="INTERNAL",namespace="<${CNDBTIER_NAMESPACE}>"}[15m]))))))*100) > ((100/scalar(count(sum by (node_id,comm_node_id,namespace)(db_tier_tc_time_track_stats_total_scan_fragments_time{path_type="INTERNAL",namespace="<${CNDBTIER_NAMESPACE}>"}))))*1.6)
Expression Validity NA
SNMP Trap ID 3005
Affects Service (Y/N) Y
Recommended Action

Cause: When the scan fragment for any particular data node is slow.

Diagnostic Information: The alert indicates that the NDB application node is slow, therefore check the underlying infrastructure.

Recovery:

This alert is cleared automatically when the data node fragment scan is fast as compared to other data nodes.

For any assistance, contact My Oracle Support.

6.15 cnDBTier PVC Health Monitoring Alerts

This section provides details about the cnDBTier PVC health monitoring alerts.

Table 6-40 PVC_NOT_ACCESSIBLE

Field Details
Description This alert is triggered with a critical severity when PVC is not accessible (db_tier_pvc_is_accesible metric is 0).
Summary PVC is not accessible on cnDBTier site {{ $labels.site_name }}
Severity critical
Condition db_tier_pvc_is_accesible == 0
Expression Validity 1m
SNMP Trap ID 2029
Affects Service (Y/N) Y
Recommended Action

Cause: The system is unable to access PVC for read or write operation.

Diagnostic Information: The db_tier_pvc_is_accesible metric (Table 5-86) provides information if the PVC is accessible or not.

Recovery: This alert is cleared automatically when the PVC is accessible.

Check the PVC and ensure that the PVC is accessible and doesn't have any errors.

For any assistance, contact My Oracle Support.

Table 6-41 PVC_FAILURE_COUNT

Field Details
Description This alert is triggered with info severity when the value of db_tier_pvc_failure_count was greater than 0 during the last scrape. The value of db_tier_pvc_failure_count metric indicates the number of times the PVC was not accessible.
Summary PVC of {{ $labels.hostname }} was unavailable for {{ $value }} times in the last 10m on cnDBTier site {{ $labels.site_name }}
Severity info
Condition sum(sum_over_time(db_tier_pvc_failure_count[10m])) by (hostname, site_name, namespace, mount_path) > 0
Expression Validity 5m
SNMP Trap ID 2030
Affects Service (Y/N) N
Recommended Action

Cause: The system is unable to access the PVC for read or write operation.

Diagnostic Information: The db_tier_pvc_failure_count metric (Table 5-87) provides information about the number of times the PVC was not accessible during the last scrape.

Recovery: This alert is cleared automatically when the db_tier_pvc_failure_count metric is zero.

Check the PVC and ensure that the PVC is accessible and doesn't have any errors.

For any assistance, contact My Oracle Support.

Table 6-42 PVC_STORAGE_FULL

Field Details
Description This alert is triggered with critical severity when a pod's PVC reaches full capacity.
Summary PVC on host {{ $labels.hostname }} is full in the cnDBTier site {{ $labels.site_name }}, within the {{ $labels.namespace }} Kubernetes namespace.
Severity critical
Condition ((avg_over_time(db_tier_volume_stats_used_bytes[1m])/avg_over_time(db_tier_volume_stats_capacity_bytes[1m])) * 100) > 90
Expression Validity 1m
SNMP Trap ID 2032
Affects Service (Y/N) Y
Recommended Action

Cause: The PVC reaches full capacity, preventing further write operations.

Diagnostic Information: The metric provides information that the PVC has no available space, leading to storage-related failures.

Recovery: Free up space or expand the PVC as needed. Ensure that the application managing the PVC properly handles storage utilization. This alert is cleared automatically when sufficient space becomes available.

For any assistance, contact My Oracle Support.

6.16 cnDBTier Backup Manager Svc Down Alerts

This section provides details about cnDBTier backup manager Svc down alerts.

Table 6-43 DB_BACKUP_MANAGER_SVC_DOWN

Field Details
Description This alert is triggered with critical severity when db_backup_manager_svc pod is down.
Summary PVC is not accessible on cnDBTier site {{ $labels.site_name }
Severity critical
Condition kube_deployment_status_replicas_available{deployment=~".*db-backup-manager-svc.*"} == 0
OID 1.3.6.1.4.1.323.5.3.50.1.2.2039
Metric Used 
kube_deployment_status_replicas_available{deployment=~".*db-backup-manager-svc.*"}
Expression Validity 1m
Affects Service (Y/N) Y
Recommended Action

Cause: When db_backup_manager_svc pod is down.

Diagnostic Information: The system detects that the db_backup_manager_svc pod is not up and unable to connect to database.

Recommended steps:
  1. Check the db-backup-manager service pod status. Look for CrashLoopBackOff, ImagePullBackOff, OOMKilled, Init container failures.

    Run the following command to check the pod status:
    kubectl -n <cnDBTier namespace> get pods | grep "db-backup-manager-svc"
    Run the following command to describe the pod:
    kubectl -n <cnDBTier namespace> describe pod <backup-manager-pod name>
  2. Check the deployment status.

    Look for Available replicas: 0, Events section for scheduling or image pull errors.
    Run the following command to get the db-backup-manager-svc deployment:
    kubectl -n <cnDBTier namespace> get deployment | grep "db-backup-manager-svc"
    Run the following command to describe the deployment:
    kubectl -n <cnDBTier namespace> describe deployment <backup-manger-svc
    deployment>
  3. Check the DB Backup Manager service pod logs to see if there are a lot of database connectivity issues.
    Run the following command to get the backup-manager-svc pod:
    kubectl -n <cnDBTier namespace> get pods | grep "db-backup-manager-svc"

    Run the following command to check the logs of the backup-manger svc pod:

    kubectl -n <cnDBTier namespace> logs < db-backup-manager-svc pod name>
  4. In case if the issue persists, capture all the outputs for the above steps and contact My Oracle Support.

Note:

Use CNC NF Data Collector tool for capturing logs. For more information on how to collect logs using Data Collector tool, see Oracle Communications Cloud Native Core, Network Function Data Collector User Guide.