8.7 Using dbttriage

This section describes how to use the dbttriage command-line tool for deeper health checks and triage of cnDBTier sites. dbttriage connects to a Kubernetes or OpenShift cluster to discover the relevant cnDBTier components, and runs a suite of targeted tests. dbttriage operates by running kubectl / oc commands from the host where it is used. Many checks are performed by executing commands inside cnDBTier pods.

Prerequisties

Before running dbttriage, ensure the following requirements are met:

  • Your operating system is CentOS Linux 7 (Core) or later.

  • The dbttriage binary file is available as a compiled run file, requiring no additional setup or Python dependencies to run.

  • kubectl (or OpenShift oc) is installed and configured on the host where you run dbttriage, with permissions to read resources and exec into cnDBTier pods.

Tool Overview

The dbttriage utility connects to a Kubernetes (or OpenShift) cluster, discovers cnDBTier components (pods, StatefulSets, and services), and runs grouped diagnostic checks. Results are printed as tests run and a final text summary report is printed at the end.

Key Features:
  • Kubernetes / OpenShift integration: Targets cnDBTier running in Kubernetes via kubectl or OpenShift via oc.
  • Flexible targeting:
    • Prompt for namespace when no flags are provided.
    • Use an explicit namespace (--use-namespace).
    • Use a kubeconfig file (--use-kubeconfig).
  • Deeper triage test suite: Runs grouped diagnostic checks and prints PASSED / FAILED / SKIPPED test results as they run.
  • Actionable troubleshooting hints: Failed tests include a short troubleshooting message when available.
  • Configurable timeouts:--connect-timeout controls timeouts for commands executed from within pods (MySQL and curl).
  • Logging controls:
    • By default, writes a log file to the current directory.
    • --log-dir controls the log directory.
    • --no-log disables file logging.
  • Debug mode:--debug enables verbose logging.

Installation of dbttriage

As dbttriage is a compiled program, no installation is required. Copy the dbttriage binary to your system and ensure it has permissions to run:
chmod +x dbttriage

Ensure that either kubectl or oc is installed and configured on the host where you run dbttriage.

Run dbttriage with optional arguments to triage a cnDBTier site. If no arguments are provided, the tool will prompt for a namespace and will use kubectl -n <namespace>.

Usage and Command Syntax

Run dbttriage with optional arguments to triage a cnDBTier site. If no arguments are provided, the tool will prompt for a namespace and will use kubectl -n <namespace>.

Common Options

Table 8-4 Common Options

Option Description
--connect-timeout SECONDS Connection timeout in seconds for commands executed from within pods (MySQL and curl). Default is 15.
--print-test-info Print DbtTestRunner test descriptions (grouped test reference) and exit.
--use-namespace NAMESPACE Use kubectl with a specific namespace. Mutually exclusive with --use-kubeconfig and --use-oc.
--use-kubeconfig KUBECONFIG_FILE Use kubectl with a specific kubeconfig file. Mutually exclusive with --use-namespace and --use-oc.
--use-oc Use the OpenShift oc CLI instead of kubectl. Mutually exclusive with --use-namespace and --use-kubeconfig.
--log-dir DIR Directory where the log file will be generated. Default is current directory (.).
--no-log Disable default file logging (do not create a log file).
--debug Enable debug logging level.
--version Show the version and exit.

Note:

Namespace, kubeconfig, and oc selection options are mutually exclusive. Only one may be specified. If no site-selection argument is provided, dbttriage prompts for a namespace and defaults to using kubectl with that namespace.
Here are some example commands:
  • Run with defaults (will prompt for namespace): 
    ./dbttriage
  • Run against a specific namespace:
    ./dbttriage --use-namespace my-namespace
  • Run using OpenShift oc:
    ./dbttriage --use-oc
  • Run with debug logging and no file logging:
    ./dbttriage --debug --no-log

Output and Reports:

dbttriage produces two primary types of output:

  • Log file (default):
    • By default, dbttriage writes a log file to the current directory.
    • The filename is generated in UTC with the format: <program>_%Y_%m_%dT%H_%M_%SZ.log (for example: dbttriage_2026_03_09T21_42_26Z.log)
    • Use --log-dir to change where the log file is written, or --no-log to disable it.
  • Console output:
    • A startup banner and environment/namespace information.
    • Streaming test results (PASSED / FAILED / SKIPPED).
    • A final Test Report Summary (text) with detailed lists and per-group totals.

Test Types

dbttriage executes a suite of grouped tests implemented in dbttriage.core.dbt_test_runner.DbtTestRunner. Each test type corresponds to a _test_* method and emits one or more dbttest_reporter.record_test(...) entries.

Note:

  • Many tests are not applicable in SINGLE_SITE mode; in that topology the replication deployments do not exist and the runner intentionally skips replication/georeplication checks.
  • The Group/Subgroup names below match the sections shown in the final test summary report.
pod_local / monitor

MON-VER-### — Monitor service version API (_test_monitor_svc_version_api)

  • What it does: execs curl inside each dbmonitorsvc pod to: http://[::1]:${HTTP_SERVER_PORT}/db-tier/version.
  • Why is it important: confirms the monitor service is reachable from within the pod and is reporting a sane version payload; failures often indicate networking, pod health, or misconfiguration.
  • PASS: curl succeeds, output is valid JSON, and JSON contains non-empty cndbtier_version and ndb_version.
  • FAIL: no monitor pods, curl failure/non-200, invalid JSON, or missing required keys.
pod_local / status

POD-STATUS-### — Pod STATUS must be Running (_test_dbtier_pods_running_and_ready)

  • What it does: calls Site.get_dbtier_pods() once and parses the kubectl get pods tabular output.
  • Why is it important: quickly surfaces crash loops, scheduling issues, and node problems that prevent the site from functioning.
  • PASS: parsed STATUS == "Running".
  • FAIL: any pod row has a non-Running status.

POD-READY-### — Pod READY must be x/x (_test_dbtier_pods_running_and_ready)

  • What it does: for each parsed pod row, validates READY matches ^\d+/\d+$ and that the two integers are equal (for example, 1/1, 2/2).
  • Why is it important: ensures all containers in each pod are actually ready to serve (not just running), avoiding false confidence during partial rollouts or failing sidecars.
  • PASS: READY is x/x.
  • FAIL: READY is missing/malformed or not x/x.

PODS-PARSE-ERROR — kubectl output parse failure (_test_dbtier_pods_running_and_ready)

  • What it does: emitted only when a non-header line in the kubectl get pods output cannot be parsed.
  • Why is it important: indicates the tool cannot reliably interpret cluster state (format changes, localization, or unexpected output), which can invalidate subsequent checks.
  • FAIL: includes raw output in troubleshooting for debugging.

MGM-NODE-### — NDB management node state must be OK (_test_management_mgm_nodes_ok)

  • What it does: for each management (ndbmgmd) pod, calls management_sts.status(mgm_pod=<pod>) and emits one test per parsed node.
  • Why is it important: validates NDB cluster control-plane health; non-OK management state can lead to degraded cluster operations and cascading data/node failures.
  • PASS: node state is exactly "OK".
  • FAIL: no mgm pods, status cannot be computed, parsed nodes are missing, or any node state is not OK.
  • Noise reduction: excludes “empty API nodes” when management_sts.first_api_empty_node_id is configured.
connect_internal / backup (skipped in SINGLE_SITE)

DATA-REPL-SSH-### — DataSts backup-executor SSH to leader repl svc

(_test_data_sts_backup_executor_local_ssh_to_repl_leader)

  • What it does: from each local data (ndbmtd) pod, execs in the backup-executor sidecar and runs an SSH command to the leader dbreplicationsvc’s internal service name/port.
  • Why is it important: ensures internal service discovery and credentials allow required intra-cluster access paths used by backup/replication workflows.
  • PASS: SSH exits with code 0 and stdout is non-empty (remote hostname).
  • FAIL: required objects/fields are missing, there are no local data pods, or SSH fails.
connect_external / replication (skipped in SINGLE_SITE)

REPL-REST-VER-### — Remote dbreplicationsvc /db-tier/version

(_test_db_replication_svc_remote_rest_version_api)

  • What it does: from each local dbreplicationsvc pod, execs curl to each remote repl-svc endpoint: http://<remote_ip>:<remote_http_port>/db-tier/version.
  • Why is it important: confirms cross-site reachability and that remote replication services are alive and reporting expected metadata; helps catch DNS/IP/firewall or service exposure issues.
  • PASS: curl succeeds, output is valid JSON, and JSON contains a non-empty version.
  • FAIL: missing local pods, missing remote endpoints, curl/JSON failures, or missing remote http_port.
  • Special behavior: if the remote list is [], the runner records a single PASS ("nothing to test").

REPL-SSH-### — Remote dbreplicationsvc SSH connectivity

(_test_db_replication_svc_remote_ssh_connectivity)

  • What it does: from each local dbreplicationsvc pod, execs SSH to each remote repl-svc endpoint: ssh ... -p <remote_sftp_port> mysql@<remote_ip> "uname -n".
  • Why is it important: verifies the secure transport path used for replication file transfer/control is working end-to-end (networking, keys, ports).
  • PASS: SSH exits with code 0 and stdout is non-empty.
  • FAIL: missing local pods/remotes, invalid remote endpoint fields, or SSH failure.
connect_external / sql

SQL-CONN-### — SQL pod connectivity to its remote source

(_test_sql_pod_remote_mysql_connectivity)

  • What it does: from each local SQL (ndbmysqld) pod, runs a MySQL query against the remote endpoint described by SqlPod.source: SELECT COUNT(*) FROM replication_info.DBTIER_SITE_INFO;.
  • Why is it important: ensures the data plane can read from its upstream/source; broken connectivity leads to stale data, replication lag, or outright service outage.
  • PASS: mysql command succeeds, stdout parses as int, and count > 1.
  • FAIL: local SQL pods missing, derived SqlPod objects missing, missing remote connection fields, mysql failure, non-integer output, or count <= 1.
  • SKIP: SqlPod.source is missing (partial MULTI_SITE install).
geo_replication / health (skipped in SINGLE_SITE)

REPL-HEALTH-### — dbreplicationsvc self health + mate health

(_test_geo_replication_repl_health)

  • What it does: for each DbReplicationSvcDeployment (leader + standard), records two checks: (1) Self health via svc.get_health(). (2) Mate health via svc.repl_mate.get_health(...) (exec’d from inside a local repl-svc pod).
  • Why is it important: provides an early indicator that replication services are operational on both sides; unhealthy replication services are a common root cause of georeplication incidents.
  • PASS: self health returns a non-empty dict and every pod’s status is "UP"; mate health returns status "UP".
  • FAIL: non-dict/empty health payload, any non-UP value, missing repl_mate wiring, missing pods, or exceptions.
  • SKIP (mate health only): svc.repl_mate is missing (partial MULTI_SITE wiring).
geo_replication / repl_channel_info (skipped in SINGLE_SITE)

GEO-REPL-ROLE-### — Local role pairing ACTIVE/STANDBY

(_test_geo_replication_repl_channel_info_roles)

  • What it does: for each repl service, validates that primary/secondary roles are exactly one ACTIVE and one STANDBY.
  • Why is it important: confirms the control plane has an unambiguous primary/secondary assignment; incorrect role pairing can cause split-brain behavior or replication not advancing.
  • PASS: roles are the valid pairing.
  • FAIL: pods missing, roles missing, or invalid role combination.
  • SKIP: per-service when its configured mate site is not installed (partial MULTI_SITE).
  • No-op: if SqlSts is missing (roles are not available), this test emits nothing to avoid noisy failures.

GEO-REPL-ROLE-EQ-### — Source role equals local role

(_test_geo_replication_source_roles_equal_local)

  • What it does: validates each local pod’s role equals its source.role.
  • Why is it important: detects disagreement between sites about who is ACTIVE/STANDBY, which is dangerous during failover and can lead to writes going to the wrong place.
  • PASS: both primary and secondary have matching role pairs.
  • FAIL: missing pod/source references or any mismatch.
  • SKIP: per-service when its configured mate site is not installed (partial MULTI_SITE).
geo_replication / show_replica (skipped in SINGLE_SITE)

GEO-REPL-STATE-### — SHOW REPLICA STATUS state pairing ACTIVE/STANDBY

(_test_geo_replication_show_replica_states)

  • What it does: derives local replica state from SHOW REPLICA STATUS for primary+secondary and validates they are opposite.
  • Why is it important: validates actual MySQL replication behavior (not just configured intent); mismatched states often indicate broken replication threads or miswired channels.
  • PASS: exactly one ACTIVE and one STANDBY.
  • FAIL: invalid pairing or missing pods; troubleshooting includes Last_Errno/Last_Error and may include gated IO/SQL error details.
  • SKIP: per-service when its configured mate site is not installed (partial MULTI_SITE).

GEO-REPL-STATE-EQ-### — Source state equals local state

(_test_geo_replication_source_states_equal_local)

  • What it does: validates each local pod’s derived state matches its source’s derived state.
  • Why is it important: ensures both sides have a consistent view of replication state, helping catch partial failures where one site is lagging or stuck while the other appears healthy.
  • PASS: local==source for both primary and secondary.
  • FAIL: mismatch or missing references; troubleshooting includes error details for both target=local and target=source.
  • SKIP: per-service when its configured mate site is not installed (partial MULTI_SITE).

GEO-REPL-ROLE-STATE-EQ-### — Configured role equals derived state

(_test_geo_replication_role_equals_state)

  • What it does: validates SqlPod.role matches SHOW REPLICA STATUS-derived state on the local side.
  • Why is it important: catches configuration drift where the declared role does not match the database’s real replication state, which can break automation and operator expectations.
  • PASS: both pods have roles in {ACTIVE, STANDBY} and role==state.
  • FAIL: roles missing or mismatch; troubleshooting includes replica error details.
  • SKIP: per-service when its configured mate site is not installed (partial MULTI_SITE).

GEO-REPL-SID-### — Source_Server_Id symmetry

(_test_geo_replication_source_server_id_symmetry)

  • What it does: per pod, verifies Source_Server_Id is consistent in both directions: local.server_id source-side Source_Server_Id; local-side Source_Server_Id source.server_id.
  • Why is it important: verifies the replication topology is correctly wired and each side is following the intended server; mismatched server IDs can indicate wrong source, cloning errors, or misconfiguration.
  • PASS: symmetry holds.
  • SKIP: one (or both) Source_Server_Id readings is 0 on a STANDBY channel (role-dependent as described).
  • FAIL: otherwise, for mismatches, missing references, or cache problems.

SHOW-REPL-PUBKEY-### — Get_Source_public_key == 1 (local + source)

(_test_geo_replication_show_replica_get_source_public_key)

  • What it does: for each local SQL pod, checks the SHOW REPLICA STATUS Get_Source_public_key field for both targets (target=local and target=source).
  • Why is it important: ensures secure replication authentication is configured as expected; incorrect public key settings can cause replication connection failures or insecure fallback behavior.
  • PASS: value is integer 1.
  • SKIP: SqlPod is missing local MySQL creds (cannot run SHOW REPLICA STATUS), or for target=source when SqlPod.source is missing (partial MULTI_SITE).
  • FAIL: value is not 1 or exceptions occur.

GEO-REPL-CTR-### — Replication counters advance + ACTIVE stability

(_test_geo_replication_replication_counters_progress)

  • What it does: takes two SHOW REPLICA STATUS readings (with a minimum elapsed time between them) and verifies key counters change between the two readings. Only measures pods whose local state is ACTIVE.
  • Why is it important: confirms replication is making forward progress (not stuck) and the ACTIVE side remains stable; stagnant counters are a strong signal of replication breakage.
  • Counters validated: Read_Source_Log_Pos (with Source_Log_File), Relay_Log_Pos (with Relay_Log_File), Exec_Source_Log_Pos (with Source_Log_File), Relay_Log_Space (with Relay_Log_File).
  • PASS: counter values parse as ints and advance; ACTIVE pods remain ACTIVE across both readings.
  • FAIL: missing/non-integer counters, unchanged values, or missing second measurements.

GEO-REPL-DELAY-### — SQL_Delay <= 10s (two-reading)

(_test_geo_replication_sql_delay_two_readings)

  • What it does: uses the same two-reading window as GEO-REPL-CTR and validates replication lag (SQL_Delay) for each (pod, target).
  • Why is it important: verifies georeplication delay is staying under 10 seconds, excluding occasional spikes.
  • PASS: at least one of the two readings is <= 10 seconds.
  • FAIL: both readings are > 10, or the second reading is missing / not an integer.