5 cnDBTier Metrics

cnDBTier generates metrics to record or measure the specified values in cnDBTier. You can access the metrics using the Prometheus dashboard and take necessary actions. Prometheus gets installed as part of common services during the vCNE installation. This section provides details about the available cnDBTier metrics.

Dimensions Legend for Metrics

The following table provides information about metrics dimensions:

Table 5-1 Dimensions Legend

Dimension	Description
namespace	The name of the namespace in which the cnDBTier cluster is deployed.
site_name	The name of the site in which the cnDBTier cluster is deployed.
mate_site_name	The name of the remote site where replication is established.
node_id	The node ID of the database node.
remote_node_id	The node ID of the database node in the remote site.
node_type	The type of the database node. Sample values: Data node, Management node, SQL node
node_version	The version of MySQL NDB cluster software.
block_name	The name of the associated NDB kernel block. A kernel block is responsible for managing distinct operations. For more information about NDB kernal block, see SQL documentation. Sample Values: DBLQH, DBTC
block_instance	The instance number of the NDB kernel block.
counter_name	The name of the counter. Each counter is associated with a particular NDB kernel block. For more information about counters, see SQL documentation.
thr_no	The thread ID that is specific to a node.
thr_nm	The name of the threat. Sample Values: ldm, main, recv, rep
memory_type	Type of memory. Sample Values: Data memory, Index memory, Long message buffer
error_number	The error number for which the replication error is skipped. Sample Values: 13119, 1296, 1007, 1008, 1050, 1051.
service_name	Name of the cnDBTier microservice.
mount_path	The path within a container's file system where a volume is mounted. This path allows the container to access the contents of that volume.
hostname	The Fully Qualified Domain Name (FQDN). This value is provided in the following format: host-name.service-name.namespace.svc.domain_name. Sample value: ndbmtd-0.ndbmtdsvc.occne-cndbtier.svc.cluster.local ndbmysqld-1.ndbmysqldsvc.occne-cndbtier.svc.cluster.local
role	The role of the mysqld replication channel. Sample Values: active, standby
replchannel_group_id	The ID of the replication channel group.
channel_id	The ID of the mysqld replication channel.
source_node_ip	The IP address of the active replication SQL service in the remote site.
replica_node_ip	The IP address or hostname of the active replication SQL service in the local site.
backup_id	The ID of the backup.
status	The status of the backup operation. Sample values: STARTED, FAILED, COMPLETED, PURGED, PURGED_EARLY, BACKUP_TRANSFER_FAILED, BACKUP_TRANSFER_IN_PROGRESS, BACKUP_TRANSFER_COMPLETED
table_id	The unique ID of a table that is generated internally by NDB.
fq_name	The fully qualified name of the fragment. Fragments are the logical partitions of the data within a table. A fragment refers to a portion of a table's data that is distributed across multiple data nodes in a cluster. For more information, see SQL documentation.
parent_fq_name	The fully qualified fragment name for the parent or any fragment of the object.
remote_server_ip	The IP address of the remote server.

5.1 cnDBTier API Node Read and Write Metrics

This section provides details about the cnDBTier API node read and write metrics.

Table 5-2 db_tier_api_node_bytes_writes

Field	Details
Description	This metric provides information about the number of byte writes for ndbappmysqld pods and ndbmysqld pods.
Type	Gauge
Dimensions	namespace node_id hostname site_name

Table 5-3 db_tier_api_node_bytes_reads

Field	Details
Description	This metric provides information about the number of byte reads for ndbappmysqld pods and ndbmysqld pods.
Type	Gauge
Dimensions	namespace node_id hostname site_name

5.2 cnDBTier Remote Server Backup Transfer Status Metrics

This section provides details about the cnDBTier remote server backup transfer status metrics.

Table 5-4 db_tier_remote_server_backup_transfer_status

Field	Details
Description	Provides the status of backup transfer to a remote server. The possible values are: "1": indicates that the transfer of backup to remote server failed. "0": indicates that the transfer of backup to remote server completed. "2": indicates that the transfer of backup to remote server is in progress.
Type	Gauge
Dimensions	namespace site_name backup_id remote_server_ip status

5.3 cnDBTier Backup Transfer Status Metrics

This section provides details about the cnDBTier backup transfer status metrics.

Table 5-5 db_tier_backup_transfer_status

Field	Details
Description	Provides the status of the backup transfer process. The possible status values are: 0: indicates success 1: indicates in progress 2 and 3: indicates failed
Type	Gauge
Dimensions	site_name namespace backup_id status

5.4 cnDBTier Parallel Backup Transfer Progress Metrics

This section provides details about the cnDBTier parallel backup transfer progress metrics.

Table 5-6 db_tier_local_backup_transfer_progress

Field	Details
Description	Provides the percentage of the backup transfer that has completed from data node to replication service on a healthy backup site. This metric can be seen when a georeplication recovery is in progress on a failed site.
Type	Gauge
Dimensions	namespace node_id site_name

Table 5-7 db_tier_remote_backup_transfer_progress

Field	Details
Description	Provides the percentage of the backup transfer that has completed from the replication service of a backup site to the replication service of an unhealthy site. This metric can be seen when a georeplication recovery is in progress on a failed site.
Type	Gauge
Dimensions	namespace node_id site_name

5.5 cnDBTier Heartbeat Metrics

This section provides details about the cnDBTier heartbeat metrics.

Table 5-8 db_tier_heartbeat_failure

Field	Details
Description	Indicates the success or failure of heartbeat when trying to connect to a remote site. 0: indicates that Heartbeat is successful and cnDBTier is able to connect to the remote site. 1: indicates that Heartbeat failed and cnDBTier is unable to connect to the remote site.
Type	Gauge
Dimensions	site_name mate_site_name replchannel_group_id namespace

5.6 cnDBTier Node Status Metrics

The section provides details about the cnDBTier node status metrics.

Table 5-9 db_tier_node_status

Field	Details
Description	The status of the cnDBTier node. The possible values are: "0": indicates that the node is DOWN "1": indicates that the node is UP
Type	Gauge
Dimensions	node_id node_type node_version site_name namespace

Table 5-10 db_tier_cluster_status

Field	Details
Description	The status of the cnDBTier cluster. The possible values are: "0": indicates that the cluster is DOWN "1": indicates that the cluster is UP
Type	Gauge
Dimensions	site_name namespace

Table 5-11 db_tier_cluster_disconnect

Field	Details
Description	The disconnect status of the cnDBTier cluster. the value indicates the number of cluster disconnect incidents happened in the cluster.
Type	Gauge
Dimensions	site_name namespace

5.7 cnDBTier Table Read Write Metrics

The section provides details about the cnDBTier table read write metrics.

Table 5-12 db_tier_local_operations

Field	Details
Description	The total number of local operations in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-13 db_tier_transactions

Field	Details
Description	The total number of transactions in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-14 db_tier_commits

Field	Details
Description	The total number of commits in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-15 db_tier_reads

Field	Details
Description	The total number of reads in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-16 db_tier_local_reads

Field	Details
Description	The total number of local reads in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-17 db_tier_writes

Field	Details
Description	The total number of writes in cnDBTier cluster for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-18 db_tier_local_writes

Field	Details
Description	The total number of local writes in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-19 db_tier_aborts

Field	Details
Description	The total number of aborted transactions in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-20 db_tier_table_scans

Field	Details
Description	The total number of table scans in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-21 db_tier_range_scans

Field	Details
Description	The total number of range scans in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-22 db_tier_transporter_overload

Field	Details
Description	The total number of transporter overload in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

Table 5-23 db_tier_scan_slowdown

Field	Details
Description	The total number of scan slowdowns in cnDBTier for a node.
Type	Gauge
Dimensions	node_id block_name block_instance counter_name site_name namespace

5.8 cnDBTier CPU Usage Metrics

This section provides details about the cnDBTier CPU usage metrics.

Table 5-24 db_tier_cpu_os_user

Field	Details
Description	Provides the CPU user statistics per thread for the specific node.
Type	Gauge
Dimensions	node_id thr_no site_name namespace

Table 5-25 db_tier_cpu_os_system

Field	Details
Description	Provides the CPU system statistics per thread for the specific node.
Type	Gauge
Dimensions	node_id thr_no site_name namespace

Table 5-26 db_tier_cpu_os_idle

Field	Details
Description	Provides the idle CPU statistics per thread for the specific node.
Type	Gauge
Dimensions	node_id thr_no site_name namespace

5.9 cnDBTier Memory Usage Metrics

This section provides details about the cnDBTier memory usage metrics.

Table 5-27 db_tier_memory_used_bytes

Field	Details
Description	Indicates the amount of memory used by the node in bytes.
Type	Gauge
Dimensions	node_id memory_type site_name namespace

Table 5-28 db_tier_memory_total_bytes

Field	Details
Description	Indicates the total memory assigned for the node in bytes.
Type	Gauge
Dimensions	node_id memory_type site_name namespace

5.10 cnDBTier Bin Log Usage Metrics

This section provides details about the cnDBTier binlog usage metrics.

Table 5-29 db_tier_binlog_used_bytes_percentage

Field	Details
Description	Indicates the percentage of total memory used by bin log in the SQL node.
Type	Gauge
Dimensions	node_id site_name namespace

5.11 cnDBTier Replication Metrics

This section provides details about the cnDBTier replication metrics.

Table 5-30 db_tier_replication_status

Field	Details
Description	Indicates the status of replication. The possible values are: "0": indicates that the replication channel status of the local site is OFF. "1": indicates that the replication channel status of the local site is ON. "2": indicates that the replication channel status of the local site is CONNECTING.
Type	Gauge
Dimensions	node_id role replchannel_group_id mate_site_name channel_id site_name namespace

Table 5-31 db_tier_replication_replica_delay

Field	Details
Description	Indicates the time (in seconds) by which the last record read by the replica is behind the latest record written by the source.
Type	Gauge
Dimensions	channel_id source_node_ip replica_node_ip mate_site_name site_name namespace

5.12 cnDBTier Automated Backup Metrics

This section provides details about the cnDBTier automated backup metrics.

Table 5-32 db_tier_backup_used_disk_percentage

Field	Details
Description	This metric provides the percentage of disk space used by the backup. This metric is pegged after the old backups are purged and a new one is created.
Type	Gauge
Dimensions	node_id site_name namespace

Table 5-33 db_tier_backup_size

Field	Details
Description	This metric provides the size of cnDBTier backup created. This metric is pegged only when a backup completes successfully.
Type	Gauge
Dimensions	backup_id node_id site_name namespace

Table 5-34 db_tier_backup

Field	Details
Description	This metric fetches the status of cnDBTier backup. This metric is pegged at each stage of a backup life cycle: on creation, when it fails or completes, and when it is deleted.
Type	Gauge
Dimensions	site_name namespace status

Table 5-35 db_tier_ndb_backup_in_progress

Field	Details
Description	This metric states if a data node backup is in progress in the current site. The possible values are: "0": Indicates that no database backup is in progress. "1": Indicates that cnDBTier backup is in progress.
Type	Gauge
Dimensions	site_name namespace

5.13 cnDBTier Fault Recovery State Metrics

This section provides details about the cnDBTier georeplication recovery state metrics.

Table 5-36 db_tier_georeplication_recovery_state

Field	Details
Description	Indicates if the current site is undergoing georeplication recovery.
Type	Gauge
Dimensions	site_name (Name of the site that is undergoing georeplication recovery) namespace

Table 5-37 db_tier_ndb_restore_meta_progress

Field	Details
Description	Provides the completion percentage of the NDB metadata restore progress.
Type	Gauge
Dimensions	site_name (Name of the site that is undergoing georeplication restore) namespace

Table 5-38 db_tier_ndb_restore_data_progress

Field	Details
Description	Provides the completion percentage of the NDB data restore progress.
Type	Gauge
Dimensions	site_name (Name of the site that is undergoing georeplication restore) namespace node_id

5.14 cnDBTier Replica Status Metrics

This section provides details about the cnDBTier replica status metrics.

Table 5-39 db_tier_api_trans_commit_count

Field	Details
Description	The number of transactions committed by this replica.
Type	Gauge
Dimensions	node_id (of the DB node) site_name namespace

Table 5-40 db_tier_api_wait_exec_complete_count

Field	Details
Description	The number of times a thread has been blocked by this replica while waiting for execution of an operation to complete.
Type	Gauge
Dimensions	node_id (of the DB node) site_name namespace

Table 5-41 db_tier_api_bytes_sent_count

Field	Details
Description	The amount of data (in bytes) sent to the data nodes by this replica.
Type	Gauge
Dimensions	node_id (of the DB node) site_name namespace

Table 5-42 db_tier_api_pk_op_count

Field	Details
Description	The number of operations performed by this replica based on or using primary keys.
Type	Gauge
Dimensions	node_id (of the DB node) site_name namespace

5.15 cnDBTier ndbinfo Transporters Metrics

This section provides details about the cnDBTier ndbinfo transporters metrics.

Table 5-43 db_tier_node_transporter_bytes_sent

Field	Details
Description	The bytes sent from data node to other nodes.
Type	Gauge
Dimensions	node_id node_type remote_node_id site_name namespace

Table 5-44 db_tier_node_transporter_bytes_received

Field	Details
Description	The bytes received from data node to other nodes.
Type	Gauge
Dimensions	node_id node_type remote_node_id site_name namespace

Table 5-45 db_tier_node_transporter_overload_count

Field	Details
Description	Indicates the number of times the current connection has entered overload state since the start of the connection.
Type	Gauge
Dimensions	node_id node_type remote_node_id site_name namespace

Table 5-46 db_tier_node_transporter_slowdown_count

Field	Details
Description	Indicates the number of times the current connection has entered slowdown state since the start of the connecting.
Type	Gauge
Dimensions	node_id node_type remote_node_id site_name namespace

5.16 cnDBTier ndbinfo Threadstat Metrics

This section provides details about the cnDBTier ndbinfo thread stats metrics.

Table 5-47 db_tier_threadstat_os_time

Field	Details
Description	Indicates the OS time of the thread (ms).
Type	Gauge
Dimensions	node_id, thr_no thr_nm site_name namespace

Table 5-48 db_tier_threadstat_os_user_cpu_time

Field	Details
Description	Indicates the OS CPU time taken by the user (µs).
Type	Gauge
Dimensions	node_id, thr_no thr_nm site_name namespace

Table 5-49 db_tier_threadstat_os_system_cpu_time

Field	Details
Description	Indicates the OS CPU time taken by the system (µs).
Type	Gauge
Dimensions	node_id, thr_no thr_nm site_name namespace

Table 5-50 db_tier_threadstat_os_voluntary_context_switches

Field	Details
Description	The number of OS voluntary context switches that happened.
Type	Gauge
Dimensions	node_id, thr_no thr_nm site_name namespace

Table 5-51 db_tier_threadstat_os_involuntary_context_switches

Field	Details
Description	The number of OS involuntary context switches that happened.
Type	Gauge
Dimensions	node_id, thr_no thr_nm site_name namespace

5.17 cnDBTier ndbinfo Operations Per Fragment Metrics

This section provides details about the cnDBTier ndbinfo operations per fragment metrics.

Note:

cnDBTier ndbinfo operations per fragment metrics provide insights on the operations performed on individual fragments and their replicas within a database. For more information, see MySQL documentation.

Table 5-52 db_tier_operations_per_fragment_tot_key_reads

Field	Details
Description	The total number of key reads.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-53 db_tier_operations_per_fragment_tot_key_inserts

Field	Details
Description	The total number of key inserts.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-54 db_tier_operations_per_fragment_tot_key_updates

Field	Details
Description	The total number of key updates.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-55 db_tier_operations_per_fragment_tot_key_writes

Field	Details
Description	The total number of key writes.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-56 db_tier_operations_per_fragment_tot_key_deletes

Field	Details
Description	The total number of key deletes.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-57 db_tier_operations_per_fragment_tot_key_bytes_returned

Field	Details
Description	The total size of data and metadata returned from key read operations.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-58 db_tier_operations_per_fragment_tot_frag_scans

Field	Details
Description	The total number of scans.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-59 db_tier_operations_per_fragment_tot_scan_rows_returned

Field	Details
Description	The total number of rows returned to the client.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-60 db_tier_operations_per_fragment_tot_scan_bytes_returned

Field	Details
Description	The total size of data and metadata returned to the client.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-61 db_tier_operations_per_fragment_tot_qd_frag_scans

Field	Details
Description	The total number of times the scans were queued.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-62 db_tier_operations_per_fragment_tot_commits

Field	Details
Description	The total number of row changes committed.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-63 db_tier_operations_per_fragment_tot_scan_rows_examined

Field	Details
Description	The total number of rows examined.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

5.18 cnDBTier ndbinfo Locks Per Fragment Metrics

This section provides details about the cnDBTier ndbinfo locks per fragment metrics.

Note:

cnDBTier ndbinfo locks per fragment metrics provide details about the number of lock claim requests and their outcomes for each fragment within a database. For more information, see MySQL documentation.

Table 5-64 db_tier_locks_per_fragment_ex_req

Field	Details
Description	The number of exclusive lock requests started.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-65 db_tier_locks_per_fragment_ex_imm_ok

Field	Details
Description	The number of exclusive lock requests granted immediately.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-66 db_tier_locks_per_fragment_ex_wait_ok

Field	Details
Description	The number of exclusive lock requests granted following a wait.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-67 db_tier_locks_per_fragment_ex_wait_fail

Field	Details
Description	The number non-granted exclusive lock requests.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-68 db_tier_locks_per_fragment_sh_req

Field	Details
Description	The number of shared lock requests started.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-69 db_tier_locks_per_fragment_sh_imm_ok

Field	Details
Description	The number of shared lock requests granted immediately.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-70 db_tier_locks_per_fragment_sh_wait_ok

Field	Details
Description	The number of shared lock requests granted following a wait.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-71 db_tier_locks_per_fragment_sh_wait_fail

Field	Details
Description	The number non-granted shared lock requests.
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-72 db_tier_locks_per_fragment_wait_ok_millis

Field	Details
Description	The waiting time of the granted lock requests (in milliseconds).
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

Table 5-73 db_tier_locks_per_fragment_wait_fail_millis

Field	Details
Description	The waiting time of the failed lock requests (in milliseconds).
Type	Gauge
Dimensions	table_id fq_name parent_fq_name site_name namespace

5.19 cnDBTier ndbinfo Disk Write Speed Aggregate Metrics

This section provides details about the cnDBTier ndbinfo disk write speed aggregate metrics.

Table 5-74 db_tier_disk_write_speed_aggregate_backup_lcp_speed_last_60sec

Field	Details
Description	The number of bytes written to the disk by the backup and LCP processes per second (averaged over the last 60 seconds).
Type	Gauge
Dimensions	node_id thr_no site_name namespace

Table 5-75 db_tier_disk_write_speed_aggregate_redo_speed_last_60sec

Field	Details
Description	The number of bytes written to the REDO log per second (averaged over the last 60 seconds).
Type	Gauge
Dimensions	node_id thr_no site_name namespace

Table 5-76 db_tier_disk_write_speed_aggregate_slowdowns_due_to_io_lag

Field	Details
Description	The number of seconds since the last node start the disk writes were slowed due to the REDO log I/O lag.
Type	Gauge
Dimensions	node_id thr_no site_name namespace

Table 5-77 db_tier_disk_write_speed_aggregate_slowdowns_due_to_high_cpu

Field	Details
Description	The number of seconds since the last node start the disk writes were slowed due to high CPU usage.
Type	Gauge
Dimensions	node_id thr_no site_name namespace

Table 5-78 db_tier_disk_write_speed_aggregate_disk_write_speed_set_to_min

Field	Details
Description	The number of seconds since the last node start the disk write speed was set to minimum.
Type	Gauge
Dimensions	node_id thr_no site_name namespace

Table 5-79 db_tier_disk_write_speed_aggregate_current_target_disk_write_speed

Field	Details
Description	The actual speed of the disk writes per LDM thread (aggregated).
Type	Gauge
Dimensions	node_id thr_no site_name namespace

5.20 cnDBTier Replication Error Skip Info Metrics

This section provides details about the cnDBTier Replication Error Skip Info Metrics.

Table 5-80 db_tier_replication_halted_due_to_skiperror

Field	Details
Description	The number of times an error is skipped.
Type	Gauge
Dimensions	site_name mate_site_name replchannel_group_id error_number namespace

Table 5-81 db_tier_epochs_lost_due_to_skiperror

Field	Details
Description	The number of epochs lost due to a skipped replication error.
Type	Gauge
Dimensions	site_name mate_site_name replchannel_group_id error_number namespace

Table 5-82 db_tier_replication_switchover_due_to_clusterdisconnect

Field	Details
Description	The number of switchover that happens due to cluster disconnect error.
Type	Gauge
Dimensions	node_id site_name mate_site_name replchannel_group_id error_number namespace

5.21 cnDBTier BinLog Injector Thread Info Metrics

This section provides details about the cnDBTier replication metrics.

Table 5-83 db_tier_binlog_injector_thread

Field	Details
Description	Indicates if the Bin Log Injector thread is stalled for every replication SQL node.
Type	Gauge
Dimensions	node_id site_name namespace

Table 5-84 db_tier_binlog_injector_thread_latest_epoch

Field	Details
Description	Provides the latest epoch of the Bin Log Injector thread for every SQL node.
Type	Gauge
Dimensions	node_id site_name namespace