7 Fault Recovery

This chapter describes the procedures to perform fault recovery for Oracle Communications Cloud Native Core, Security Edge Protection Proxy (SEPP) deployment.

7.1 Overview

You must take backup of the databases and restore them either on the same or a different cluster. The SEPP database (MySQL NDB Cluster) is used for running any command or to follow any instruction.

Note:

This section describes recovery procedures to restore SEPP completely or partially.

7.2 Impacted Areas

The following table provides information about the impacted areas during SEPP fault recovery:

Table 7-1 Impacted Areas

Scenario Requires Fault Recovery or re-install of CNE? Requires Fault Recovery or re-install of cnDBTier? Requires Fault Recovery or re-install of SEPP? Other
Scenario 1: Deployment Failure No No Yes SEPP DB is not restored. Only helm uninstall/install is done.
Scenario 2: cnDBTier Corruption No Yes No (use helm upgrade if DB configuration is changed) cnDBTier must be restored from backup and not re-install. If re-install of cnDBTier is needed, then CNE also need to re-installed.
Scenario 2A: When DBTier failed in Single or Multiple (but not all) Sites No Yes No NA
Scenario 2B: When DBTier failed in all Sites No Yes No NA
Scenario 3: Database Corruption No No No SEPP backup and restore of configuration database is required on impacted site. This needs automatic periodic backup.
Scenario 4: Site Failure Yes Yes Yes NA
Scenario 4A: Single or Multiple Site Failure Yes Yes Yes NA
Scenario 4B:All Site Failure Yes Yes Yes NA

7.3 Prerequisites

Before performing any fault recovery procedure, ensure that the following prerequisites are met:
  1. cnDBTier must be in a healthy state and available on multiple sites along with SEPP. To check the cnDBTier status, perform the following steps:
    1. Run the following command to ensure that all the nodes are connected:
      ndb_mgm> show
    2. Run the following command to check the pod status:
      kubectl get pods -n <namespace>

      If the pod status is Running, then cnDBTier is in healthy state.

    3. Run the following command to check if the replication is up:
      mysql> show slave status\G

      In case there is any error, seeOracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

    4. Run the following command to check which cnDBTier has ACTIVE replication to take backup:
      select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;
  2. Automatic backup must be enabled on cnDBTier. Enabling automatic backup helps in:
    • restoring stable version of the SEPP database.
    • minimizing significant loss of data due to upgrades or roll back failures.
    • minimizing loss of data due to system failure.
    • minimizing loss of data due to data corruption or deletion due to external input.
    • migrating database information from one site to another.
  3. The following files must be available for fault recovery:
    • Custom values file used at the time of network function deployment
    • Helm charts used at the time of network function deployment
    • Secrets and Certificates
    • RBAC resources