7 Fault Recovery

This chapter provides information about fault recovery for Oracle Communications Cloud Native Core, Binding Support Function (BSF) deployment.

7.1 Overview

You must take database backup and restore it either on the same or a different cluster. It uses the BSF database (MySQL NDB Cluster) is used to run any command or follow instructions.

Database Model of BSF

BSF database consists of the following two data types:
  • Configuration Data: The configuration data is exclusive for a given site. Thus, an exclusive logical database is created and used by a site to store its configuration data. Using CNC Console and Configuration Management service, an operator can configure data in the respective site only.
  • Binding Data: The binding data is shared across sites. Thus, a common logical database is created and used by all sites. The data is replicated across sites to preserve and share binding information with mated sites. In case of cross sites messaging or a site failure, shared binding data helps in continuity of service.

The following image shows the BSF database model in three different sites:

Figure 7-1 Database Model

Diagram to explain database model

7.2 Impacted Areas

The following table shares information about the impacted areas during BSF fault recovery:

Scenario Requires Fault Recovery or re-install of CNE? Requires Fault Recovery or re-install of cnDBTier? Requires Fault Recovery or re-install of CNC BSF? Other
Scenario 1A: Recovering Corrupted Binding Database(s) No Yes

Restoring cnDBTier from older backup is the only way to restore back to restore point.

 No

Only if cnDBTier credentials are changed.

 All sites require Fault Recovery.
Scenario: All or Multiple Site Failure Yes Yes Yes NA

7.3 Prerequisites

Before performing any fault recovery procedure, ensure that the following prerequisites are met:

  • DBTier must be in a healthy state and available on multiple sites along with BSF. To check the cnDBTier status, perform the following steps:
    1. Run the following command to ensure that all the nodes are connected:

      ndb_mgm> show

    2. Run the following command to check the pod status:

      kubectl get pods -n <namespace>

      If the pod status is Running, cnDBTier is in healthy state.

    3. Run the following command to check if the replication is up:

      mysql> show slave status\G

      In case there is any error, see Oracle Communications Cloud Native Core, DBTier Installation and Upgrade Guide.

    4. Run the following command to check which DBTier is having ACTIVE replication to take backup:

      select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;

  • You must enable automatic backup on DBTier. Enabling automatic backup helps in achieving the following:
    • Restore stable version of the network function database
    • Minimize significant loss of data due to upgrade or roll back failures
    • Minimize loss of data due to system failure
    • Minimize loss of data due to data corruption or deletion due to external input
    • Migrate database information for a network function from one site to another
  • The following files must be available for fault recovery:
    • Custom values file
    • Helm charts
    • Secrets and Certificates
    • RBAC resources

For details on enabling automatic backup, see Fault Recovery section in Oracle Communications Cloud Native Core, cnDBTier (cnDBTier) Installation, Upgreade and Fault Recovery Guide.

7.4 Fault Recovery Scenarios

This section describes the fault recovery procedures for various scenarios.

Note:

This chapter describes scenario based procedures to restore BSF databases only. To restore all the databases that are part of DBTier, see Fault Recovery chapter in Oracle Communications Cloud Native Core, DBTier Installation, Upgrade and Fault Recovery Guide available on My Oracle Support (MOS).

7.4.1 Scenario: Binding Database Corruption

This section describes how to recover BSF when its binding database corrupts.

When the binding database corrupts, the database on all other sites can also corrupt due to data replication. It depends on the replication status after the corruption has occurred. If the data replication is broken due to database corruption, DBTier fails in either single or multiple sites (not all sites). And if the data replication is successful, then database corruption replicates to all the DBTier sites and DBTier fails in all sites.

The fault recovery procedure covers following sub-scenarios:

7.4.1.1 When cnDBTier Failed in All Sites

This section describes how to recover binding database when successful data replication corrupts all the cnDBTier sites.

To recover binding database, perform the following steps:

  1. Uninstall BSF Helm charts on all sites. For more information about uninstalling Helm charts, see Oracle Communication Cloud Native Core, Binding Support Function Installation, Upgrade, and Fault Recovery Guide available on MOS.
  2. Perform DBTier fault recovery procedure:
    1. Use auto-data backup file for restore procedure. For more information about cnDBTier restore, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide available on MOS.
  3. Install BSF Helm charts. For more information about installing Helm charts, see Oracle Communication Cloud Native Core, Binding Support Function Installation, Upgrade and Fault Recovery Guide available on MOS.

    Note:

    You can also refer to the ocbsf_custom_values_25.2.100.yaml file used at the time of BSF installation for Helm charts installation.

7.4.2 Scenario: Site Failure

This section describes how to perform fault recovery when either one or many of your sites have software failure. This section consists of the following:
7.4.2.1 Single or Multiple Site Failure
This scenario applies when one or more sites, and not all sites, have failed and there is a requirement to perform fault recovery. It is assumed that the user has cnDBTier and BSF installed on multiple sites with automatic data replication and backup enabled.

Note:

It is assumed that one of the cnDBTier is in healthy state.
To recover the failed sites, perform the following steps:

Note:

Ensure that all the prerequisites mentioned are met.
  1. Uninstall BSF. For more information, see the Uninstalling BSF section in Oracle Communications Cloud Native Core, Binding Support Function Installation, Upgrade, and Fault Recovery Guide.
  2. Install a new cluster by performing the Cloud Native Environment (CNE) installation procedure. For more information, see Oracle Communications Cloud Native Core, Cloud Native Environment Installation, Upgrade, and Fault Recovery Guide available on My Oracle Support.
  3. Install cnDBTier, in case replication is down or cnDBTier pods are not up and running. For information about installing cnDBTier, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade and Fault Recovery Guide.
  4. Perform DBTier fault recovery procedure:
    1. Perform DBTier fault recovery procedure to take backup from older healthy site by following the Create On-demand Database Backup procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade and Fault Recovery Guide.
    2. Restore the database to new site by following the Restore Database with Backup procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
  5. Install BSF Helm charts. For more information on installing Helm charts, see Oracle Communications Cloud Native Core, Binding Support Function Installation, Upgrade, and Fault Recovery Guide.