Fault Recovery Scenarios

7.4.1 Deployment Failure

Perform this procedure to recover SCP when its deployment corrupts.

Restore SCP as described in Restoring SCP.

7.4.2 cnDBTier Corruption

This section describes how to recover cnDBTier from the corrupted database.

When the database corrupts, the database on all other sites may corrupt due to data replication. It depends on the replication status after the corruption has occurred.

If the data replication is interrupted due to database corruption, then cnDBTier fails in either single or multiple sites, not all the sites. If the data replication is successful, then database corruption replicates to all the cnDBTier sites and cnDBTier fails in all the sites.

To recover cnDBTier when cnDBTier corrupts in single, multiple, or all sites, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

7.4.3 SCP Data Corruption

Perform this procedure to recover SCP configuration and routing database (DB) from the corrupted database.

Take a backup of the SCP database (DB) and restore the database on a different Network Database (NDB) cluster. This procedure is for on-demand backup and restore of SCP DB. The commands used for these procedures are provided by the MySQL NDB cluster.

Ensure that the MySQL NDB cluster is in a healthy state, and each database node of it should be in the running state. Run the following command to check the status of cnDBTier service:

kubectl -n <namespace> exec <management node pod> -- ndb_mgm -e show

Where,

<namespace> is the namespace where cnDBTier is deployed
<management node pod> is the management node pod of cnDBTier

Example:

[cloud-user@vcne2-bastion-1 ~]$ kubectl -n scpsvc exec ndbmgmd-0 -- ndb_mgm -e show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.86.202  (mysql-8.0.22 ndb-8.0.22, Nodegroup: 0, *)
id=2    @10.233.81.144  (mysql-8.0.22 ndb-8.0.22, Nodegroup: 0)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.81.154  (mysql-8.0.22 ndb-8.0.22)
id=50   @10.233.86.2  (mysql-8.0.22 ndb-8.0.22)
 
[mysqld(API)]   2 node(s)
id=56   @10.233.81.164  (mysql-8.0.22 ndb-8.0.22)
id=57   @10.233.96.39  (mysql-8.0.22 ndb-8.0.22)
 
[cloud-user@vcne2-bastion-1 ~]$

If the SCP database backup is required, do the following:
1. Log in to any of the SQL node or API node, and then run the following command to take dump of the database:
```
kubectl exec -it <sql node> -n <namespace> bash
mysqldump --quick -h127.0.0.1 –u <username> -p  <databasename>| gzip > <backup_filename>.sql.gz
```
  Where,
  - <sql node> is the SQL node of cnDBTier.
  - <namespace> is the namespace where cnDBTier is deployed.
  - <username> is the database username.
  - <databasename> is the name of the database that has to be backed up.
  - <backup_filename> is the name of the backup dump file.
2. Enter the SCP database name and password in the command when prompted.
  Example:
```
kubectl exec -it ndbmysqld-0 -n scpsvc bash
mysqldump --quick -h127.0.0.1 -uSCPuser -p SCPdb | gzip > SCPdbBackup.sql.gz
```
  Note:
  
  Ensure that there is enough space on the directory to save the backup file.
If the SCP database restore is required, do the following:
1. Transfer the <backup_ filename>.sql.gz file to the SQL node where you want to restore it.
2. Log in to the SQL node of the MySql NDB cluster on the new DB cluster and create a new database where the database needs to be restored.
3. Create database, database user, and grant permissions as described in Configuring Database for SCP.
  
  Note:
  The database name created in this step should be the same as the database name created in the next substep. Also, the Kubernetes secret should be the same as in the values.yaml file used for installing SCP.
4. To restore the database to the new database created, run the following command:
```
gunzip < <backup_filename>.sql.gz | mysql -h127.0.0.1 –u <username> -p <databaseName >
```
  Example:
```
gunzip < SCPdbBackup.sql.gz | mysql -h127.0.0.1 -uSCPuser -p newSCPdb
```
5. Enter the password when prompted.

7.4.4 Single or Multiple Site Failure

This section describes how to perform fault recovery when either one, many, or all of the sites have software failure.

The following are site failure scenarios:

7.4.4.1 Single or Multiple Site Failure

When both cnDBTier and SCP are installed on multiple sites with automatic data replication and backup enabled. It is observed that one or more sites, not all of them, have failed and there is a requirement to perform fault recovery.

Install a new Kubernetes cluster by performing the Cloud Native Environment (CNE) installation procedure as described in Oracle Communications Cloud Native Core, Cloud Native Environment Installation, Upgrade, and Fault Recovery Guide.
Install cnDBTier as described in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Perform cnDBTier fault recovery procedure to take data backup from an older site and restore it to a new site.
For more information about cnDBTier backup, see "Create On-demand Database Backup" and to restore the database to a new site, see "Restore DB with Backup" in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Restore SCP as described in Restoring SCP.

7.4.4.2 All Sites Failure

When both cnDBTier and SCP are installed on multiple sites with automatic data replication and backup enabled. It is observed that all the sites have failed and there is a requirement to perform fault recovery.

Install a new Kubernetes cluster by performing the Cloud Native Environment (CNE) installation procedure as described in Oracle Communications Cloud Native Core, Cloud Native Environment Installation, Upgrade, and Fault Recovery Guide.
Install cnDBTier as described in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
To perform cnDBTier fault recovery, restore the latest backed up data as described in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Restore SCP as described in Restoring SCP.

7.4 Fault Recovery Scenarios