Using dbtfastrecover

8.8 Using dbtfastrecover

This section describes how to use the dbtfastrecover command-line utility to restore cnDBTier bi-directional database replication when replication between local and remote sites is interrupted and the db_replication_svc is unable to restore the replication channel automatically.

The dbtfastrecover utility skips the problematic epoch and re-establishes replication by using the next available epoch from the remote site. It is intended to reduce manual intervention and help restore replication in a controlled manner. This utility can help reduce the need for a full Georeplication Recovery and Restore (GRR) scenario, where a backup must be taken from the healthy site, restored on the affected site, and replication then re-established.

Caution:

dbtfastrecover restores replication by skipping one or more epochs. Any transactions contained in the skipped epochs are not replayed on the recovered site. As a result, data inconsistency may exist between the sites even after replication is re-established.

After running dbtfastrecover, validate data consistency between the affected sites. If skipped data must be recovered, use the appropriate full recovery path, such as GRR, or contact My Oracle Support for assistance before returning the site to service.

Prerequisites

Before running dbtfastrecover, ensure that the following requirements are met:

Bi-directional replication is already configured between the cnDBTier sites.
Bash version 4.3 or later is available.
cnDBTier version 23.4.5 or later is installed.
Network communication between the internal Kubernetes cluster and the external load balancer IPs is functioning correctly.
All cnDBTier pods in the replication channel group are in the running state.
The source_me file delivered with dbtfastrecover is sourced before running the utility.
The latest applied epoch and the related ndb_bin_log_index entries are available for inspection.
Operators have confirmed that the affected replication direction can be recovered by skipping epochs.

The source_me file sets the required paths and environment variables for dbtfastrecover. It must be sourced from the directory where it is located.

cd Artifacts/Scripts/tools
source ./source_me

Tool Overview

The dbtfastrecover utility is designed for cnDBTier bi-directional replication topologies, such as replication between site1 and site2, where each site acts as both source and replica.

For example, if site1 replicates from site2, and site2 replicates from site1, a failure in the replica on site1 to read the binlog from site2 causes replication from site2 to site1 to stop. If db_replication_svc cannot recover the channel automatically, dbtfastrecover can be used to restore replication.

The utility identifies the latest applied epoch from the local site. It then checks the remote source SQL pods and selects the earliest epoch that is greater than the latest applied epoch and is available for recovery. The local replica is then configured to resume replication from the corresponding binlog file and position.

Before changing the replication channel, record the current replication state, including the source host, source log file, source log position, relay log file, and relay source log file. This information may be required for troubleshooting, rollback analysis, or support escalation if recovery does not complete successfully.

After the local recovery is complete, dbtfastrecover performs remote-side operations to align the remote replication channel with the active source-replica mapping used by the native db_replication_svc switchover model.

The utility does not perform a full remote replication health validation before recovery starts. It performs only the remote operations required for switchover and channel alignment. If replication also needs to be recovered on the remote site, run dbtfastrecover separately from the remote site.

Key Features

Automatic Replication Recovery: Restores interrupted replication channels when db_replication_svc cannot recover them automatically.
Epoch-Based Recovery: Identifies the latest applied epoch on the local site and resumes replication from the next available epoch on the remote site.
Bi-Directional Replication Support: Supports cnDBTier deployments where each site acts as both source and replica.
Channel Mapping Alignment: Aligns the remote replication channel with the native db_replication_svc switchover model, such as ndbmysqld-0 to ndbmysqld-0 and ndbmysqld-1 to ndbmysqld-1.
Graceful and Forceful Recovery Modes: Supports graceful recovery when the latest applied epoch is available on the remote site and forceful recovery when the epoch is not available, such as when binlog rotation has occurred.
Dry-Run and Verbose Modes: Provides dry-run and verbose options to help validate recovery behavior and troubleshoot issues before applying changes.

Recovery Modes

dbtfastrecover supports the following recovery modes.

Graceful Mode
Graceful mode is the default recovery mode. In this mode, dbtfastrecover proceeds only if the latest applied epoch from the local site is available on the remote site. If the epoch is not found, skip-error handling is stopped for that local-to-remote replication channel group.

Use graceful mode when the required epoch is still available in the remote site's binlog index and the replication channel can be recovered without moving beyond the expected recovery point.
Forceful Mode
Forceful mode is enabled by using the --force option. In this mode, dbtfastrecover proceeds even when the latest applied epoch from the local site is not available on the remote site. This mode can be used when the required epoch is unavailable because of binlog rotation.

Before using forceful mode, ensure that:
- The affected replication direction has been identified.
- The latest applied epoch is not available on the remote site.
- The business impact of skipping additional epochs has been reviewed.
- Required logs and replication status details have been collected.
- A post-recovery validation plan is available.
Note:

Use forceful mode only after confirming that graceful recovery cannot be used. Forceful mode can skip additional epochs when the latest applied epoch from the local site is not available on the remote site. This can increase the risk of data inconsistency between sites.

Workflow

If local replication is healthy, dbtfastrecover exits without making any changes.

If local replication is down, dbtfastrecover performs the following actions:

Stops switchover in the local and remote db_replication_svc pods by calling the stop-switchover REST API.
Identifies the latest applied epoch from the local site.
Finds ndb_bin_log_index entries around the latest applied epoch on the remote site.
Selects the earliest available epoch on the remote site that is greater than the latest applied epoch from the local site.
Uses the selected epoch, along with the corresponding binlog file and position, to configure the local replica to read from the selected remote source.
Starts the local replica from the selected binlog file and position.
Performs switchover on the remote site so that the remote replica aligns with the local active replica mapping.

Channel Mapping

The channel mapping identifies the relationship between the local site SQL pod and the remote site SQL pod that is used as the source during recovery.

For example, if site1 is the local replica site and site2 is the remote source site, the mapping is as follows:

Local Site Replica	Remote Site Source	Mapping
`site1 / ndbmysqld-0`	`site2 / ndbmysqld-0`	`0-0`
`site1 / ndbmysqld-1`	`site2 / ndbmysqld-1`	`1-1`

In this example, 0-0 means that ndbmysqld-0 on the local replica site is mapped to ndbmysqld-0 on the remote source site. Similarly, 1-1 means that ndbmysqld-1 on the local replica site is mapped to ndbmysqld-1 on the remote source site.

This mapping helps operators understand which local SQL pod is being recovered from which remote SQL pod during recovery validation and troubleshooting.

Usage and Command Syntax

The general syntax for running dbtfastrecover is:

dbtfastrecover [options]

The following are the common options:

Option	Description
`-h, --help`	Shows the help message and exits.
`-n NAMESPACE, --namespace NAMESPACE`	Specifies the Kubernetes namespace for operations. This option overrides `DBTIER_NAMESPACE`.
`--kubectl-cmd KUBECTL_CMD`	Overrides the `kubectl` or `oc` command.
`-v, --verbose`	Enables verbose mode and sets `LOG_LEVEL=DEBUG`.
`-d, --dry-run`	Enables dry-run mode without applying changes.
`--log-level LOG_LEVEL`	Sets the log level to `INFO`, `DEBUG`, `WARNING`, or `ERROR`.
`--mysql-cmd MYSQL_CMD`	Overrides the MySQL command template used by `dbtfastrecover`
`--curl-cmd CURL_CMD`	Overrides the curl command template.
`--localhost LOCALHOST`	Overrides the localhost IP address or host name used by internal commands.
`--thread-count THREAD_COUNT`	Specifies the worker thread count for parallel operations.
`--max-recovery-retry MAX_RECOVERY_RETRY`	Specifies the maximum retry iterations for the recovery loop. The default value is `10`.
`--ignore-sites IGNORE_SITES`	Specifies comma-separated remote site names to ignore.
`--force`	Enables forceful recovery mode. Use this option only after reviewing the forceful mode warning and completing the required checks.

Example Commands

Run this command to recover replication in the site1 namespace:

dbtfastrecover -n site1

Run this command to recover replication with a retry limit:

dbtfastrecover -n site1 --max-recovery-retry 10

Run this command to ignore selected remote sites:

dbtfastrecover --ignore-sites site2,site3

Run this command to run dbtfastrecover in dry-run mode:

dbtfastrecover --dry-run

Forceful Recovery Example

Warning:

The --force option can skip additional epochs and increase the risk of data inconsistency. Use this option only when graceful mode cannot be used and after collecting the required replication status and log details.

Run this command to run dbtfastrecover in forceful mode:

dbtfastrecover --force

Run this command to use a custom OpenShift command with an explicit namespace:

dbtfastrecover --kubectl-cmd="oc -n <namespace>"

OpenShift Namespace Handling

dbtfastrecover checks whether the oc binary is available. If oc is available, the utility treats the environment as an OpenShift environment and uses oc.

In OpenShift environments where the oc context is already set to the required namespace, do not pass -n <namespace> inside the oc command.

Before running dbtfastrecover, verify whether oc can list the expected cnDBTier pods:

oc get pods

If the expected cnDBTier pods are not displayed, run dbtfastrecover with an explicit namespace in the oc command:

dbtfastrecover --kubectl-cmd="oc -n <namespace>"

Replace <namespace> with the cnDBTier namespace.

Post-Recovery Validation

After dbtfastrecover completes, validate that replication has been restored and determine whether data inconsistency exists between the sites.

Perform the following checks:

Verify that the local and remote replication channels are running.
Verify that the recovered replica is reading from the expected remote source SQL pod.
Confirm that the binlog file and position match the selected recovery point.
Check replication delay and ensure that the recovered channel is progressing.
Compare schema and row counts across the affected sites by using an available validation tool, such as dbtaudit.
Review application or NF-level validation results to confirm whether any data impact is visible.
Review logs from dbtfastrecover, db_replication_svc, and the affected SQL pods.

If validation shows missing or inconsistent data, do not treat the recovery as complete. Use the required full recovery path, such as GRR, or contact My Oracle Support for guidance.

Failure of dbtfastrecover

If dbtfastrecover fails or completes only partially, do not retry forceful recovery without reviewing the failure details.

Perform the following actions:

Collect the dbtfastrecover console output and log files.
Collect the replication status from the affected local and remote SQL pods.
Record the current source host, source log file, source log position, relay log file, and relay source log file.
Check whether any replication channel group was skipped because of unavailable pods, network issues, or missing epoch information.
Verify the status of db_replication_svc pods on the local and remote sites.
Check whether switchover was stopped and whether it needs to be restored.
Run post-recovery validation to determine whether any data inconsistency exists.
If replication cannot be safely restored, use the full GRR procedure or contact My Oracle Support.

Do not manually change the binlog file or position unless directed by the recovery procedure or Oracle Support.

Recovering Missed Data

dbtfastrecover does not replay transactions from skipped epochs. If skipped epochs contain required data, that data is not automatically recovered by dbtfastrecover.

To recover missed data, use one of the following approaches based on Oracle Support or engineering guidance:

Perform a full GRR from a healthy site.
Restore the affected site from a valid backup.
Reconcile the missing data at the application or NF level if a supported reconciliation method is available.
Contact My Oracle Support with the collected logs, replication status, and validation results.

Limitations

The following limitations apply to dbtfastrecover:

It does not add an entry to the event info table.
It does not validate whether remote replication is healthy before recovery starts.
It performs remote-side operations only as required for switchover and channel alignment.
It skips epochs during recovery. Therefore, data inconsistency may exist between the sites after replication is re-established.
It does not automatically recover data from skipped epochs.
If replication must also be recovered on the remote site, run dbtfastrecover separately on the remote site.
If recovery fails or only partially completes, manual validation and escalation may be required.

Exit Status and Recovery Outcomes

dbtfastrecover returns the following exit status values:

Exit Status	Description
`0`	Command completed successfully.
Non-zero	Command failed.

Review the command output to determine the actual recovery outcome:

Outcome	Description	Required action
No-op success	Local replication is already healthy, so no changes are made.	No recovery action is required. Continue monitoring replication.
Complete recovery	Replication is re-established for all affected replication channel groups.	Perform post-recovery validation before returning the site to service.
Partial recovery	Recovery succeeds for some replication channel groups but not all.	Validate the recovered groups and troubleshoot the failed or skipped groups.
Skipped recovery	Recovery is skipped for one or more groups because required pods, network access, or epoch information is unavailable.	Resolve the blocking condition and rerun recovery only after validation.
Failed recovery	Recovery fails before replication is restored.	Collect logs and replication status, then use the full GRR procedure or contact My Oracle Support.