Cloud Native Core CNC Console Disaster Recovery Guide

Introduction

Perform this procedure to take a backup of the Console database (DB) and restore the database on a different cluster. This procedure is for on-demand backup and restore of Console DB. The commands used for these procedures are provided by the MYSQL Network Database (NDB) cluster.

Prerequisite

Ensure that the MYSQL NDB cluster is in a healthy state, and each database node of it should be in the running state. Run the following command to check the status of cnDBTier service:

kubectl -n <namespace> exec <management node pod> -- ndb_mgm -e show

Where,

<namespace> is the namespace where cnDBTier is deployed
<management node pod> is the management node pod of cnDBTier

In case of cnDBTier to verify the prerequisites, check whether the mysql pod is up and running.

Console DB Backup

If the Console database backup is required, do the following:

Log in to any of the SQL node or API node, and then run the following command to take dump of the database:
```
kubectl exec -it <sql node> -n <namespace> bash
mysqldump --quick -h127.0.0.1 –u <username> -p  <databasename>| gzip > <backup_filename>.sql.gz
```
Where,
- <sql node> is the SQL node of cnDBTier
- <namespace> is the namespace where cnDBTier is deployed
- <username> is the database username.
- <databasename> is the name of the database that has to be backed up
- <backup_filename> is the name of the backup dump file
Enter the Console database name and password in the command when prompted.
Example:
```
kubectl exec -i -n occne-ndb ndbmysqld-0 -- mysqldump --single-transaction --no-tablespaces --no-create-info
      -h 127.0.0.1 -u cnccuser -p cnccdb | gzip > cnccdbBackup.sql.gz
```
Note:

Ensure that there is enough space on the directory to save the backup file.

Console Restore Procedure

If the Console database restore is required, do the following:

Drop existing database and recreate database.
Restore this new database with the DB Schema file provided as part of package.
Unzip the cnccdbBackup.sql.gz file.
Rearrange the back up sql file in correct order using given procedure
Populate the DB with rearranged SQL file.
Log in to the deployment cluster, drop the existing database and create a new database. Now restore this new database with the DB Schema file provided as part of package.(occncc_rollback_iam_schema_<version>.sql).
Create database, database user, and grant permissions as described in Oracle Communication Cloud Native Core Console Installation and Upgrade Guide .
Command:
```
DROP DATABASE <CNCC Database>
CREATE DATABASE IF NOT EXISTS <CNCC Database>;
GRANT SELECT, INSERT, CREATE, ALTER, DROP, LOCK TABLES, REFERENCES, INDEX, CREATE TEMPORARY TABLES, DELETE, UPDATE, EXECUTE ON <M-CNCC IAM Database>.* TO'<CNCC IAM DB User Name>'@'%';
```
Example:
```
To be executed in the mysql pod:
DROP DATABASE cnccdb;
CREATE DATABASE IF NOT EXISTS cnccdb;
GRANT SELECT, INSERT, CREATE, ALTER, DROP, LOCK TABLES, REFERENCES, INDEX, CREATE TEMPORARY TABLES, DELETE, UPDATE, EXECUTE ON cnccdb .* TO'cnccusr'@'%';
```
Note:
The database name created in this step should be the same as the database name created in the next sub step. Also, the Kubernetes secret should be the same as in the values.yaml file used for installing Console.

To restore the database to the new database created, run the following command:

kubectl exec -i -n <namespace> <podname> -- mysql  -h 127.0.0.1 -u <username> -p<password> <CNCC Database name> < <backup_filename>

Example:

kubectl exec -i -n cndbtier1 ndbmysqld-0 -- mysql  -h 127.0.0.1 -u cnccusr -pcnccpasswd cnccdb_site1_cluster1 < create-schema-NDB.sql

Procedure to Re-arrange the DB Dump file

The following procedure is to re-arrange the DB Dump file sequentially so that the user wont get any foreign key constraints issue. To do that ENV variables must be created and run it through a for loop.

Run the following command to convert the mysqldump file which was taken as a backup (sql.gz file) to a sql file:
```
gunzip -d <filename.sql.gz> 
```
Example:
```
gunzip -d cnccdbBackup.sql.gz
```

Run the following command to create an ENV which has the sequential table order required:

export KC_TABLES="ADMIN_EVENT_ENTITY RESOURCE_SERVER RESOURCE_SERVER_POLICY
                                    ASSOCIATED_POLICY REALM CLIENT AUTHENTICATION_FLOW
                                    AUTHENTICATION_EXECUTION AUTHENTICATOR_CONFIG
                                    AUTHENTICATOR_CONFIG_ENTRY BROKER_LINK CLIENT_ATTRIBUTES
                                    CLIENT_AUTH_FLOW_BINDINGS KEYCLOAK_ROLE CLIENT_INITIAL_ACCESS
                                    CLIENT_NODE_REGISTRATIONS CLIENT_SCOPE CLIENT_SCOPE_ATTRIBUTES
                                    CLIENT_SCOPE_CLIENT CLIENT_SCOPE_ROLE_MAPPING USER_SESSION
                                    CLIENT_SESSION CLIENT_SESSION_AUTH_STATUS CLIENT_SESSION_NOTE
                                    CLIENT_SESSION_PROT_MAPPER CLIENT_SESSION_ROLE
                                    CLIENT_USER_SESSION_NOTE COMPONENT COMPONENT_CONFIG
                                    COMPOSITE_ROLE DATABASECHANGELOG USER_ENTITY CREDENTIAL
                                    DATABASECHANGELOGLOCK DEFAULT_CLIENT_SCOPE EVENT_ENTITY
                                    FEDERATED_IDENTITY FEDERATED_USER FED_USER_ATTRIBUTE
                                    FED_USER_CONSENT FED_USER_CONSENT_CL_SCOPE FED_USER_CREDENTIAL
                                    FED_USER_GROUP_MEMBERSHIP FED_USER_REQUIRED_ACTION
                                    FED_USER_ROLE_MAPPING KEYCLOAK_GROUP GROUP_ATTRIBUTE
                                    GROUP_ROLE_MAPPING IDENTITY_PROVIDER IDENTITY_PROVIDER_CONFIG
                                    IDENTITY_PROVIDER_MAPPER IDP_MAPPER_CONFIG MIGRATION_MODEL
                                    OFFLINE_CLIENT_SESSION OFFLINE_USER_SESSION POLICY_CONFIG
                                    PROTOCOL_MAPPER PROTOCOL_MAPPER_CONFIG REALM_ATTRIBUTE
                                    REALM_DEFAULT_GROUPS REALM_LOCALIZATIONS
                                    REALM_ENABLED_EVENT_TYPES REALM_EVENTS_LISTENERS
                                    REALM_REQUIRED_CREDENTIAL REALM_SMTP_CONFIG
                                    REALM_SUPPORTED_LOCALES REDIRECT_URIS REQUIRED_ACTION_CONFIG
                                    REQUIRED_ACTION_PROVIDER RESOURCE_SERVER_RESOURCE
                                    RESOURCE_ATTRIBUTE RESOURCE_POLICY RESOURCE_SERVER_SCOPE
                                    RESOURCE_SCOPE RESOURCE_SERVER_PERM_TICKET RESOURCE_URIS
                                    ROLE_ATTRIBUTE SCOPE_MAPPING SCOPE_POLICY USERNAME_LOGIN_FAILURE
                                    USER_ATTRIBUTE USER_CONSENT USER_CONSENT_CLIENT_SCOPE
                                    USER_FEDERATION_PROVIDER USER_FEDERATION_CONFIG
                                    USER_FEDERATION_MAPPER USER_FEDERATION_MAPPER_CONFIG
                                    USER_GROUP_MEMBERSHIP USER_REQUIRED_ACTION USER_ROLE_MAPPING
                                    USER_SESSION_NOTE WEB_ORIGINS";

Create an ENV pointing to the sql file to be filtered:
```
export KC_BACKUP="./<Backup SQL Dump File>"; 
```
Example:
```
export KC_BACKUP="./cnccdbBackup.sql";
```
Run the following command to re-arrange the dump file which you have taken earlier to ensure hat it is in sequential insertion order by running the following for loop command:
```
for i in $KC_TABLES; do grep "INSERT INTO \`$i\`" $KC_BACKUP; done > <file name along with its location>
```
Example:
```
for i in $KC_TABLES; do grep "INSERT INTO \`$i\`" $KC_BACKUP; done > /tmp/restore.sql
```
Run the following command to populate the DB with data using the file which you have right now after filtering the sqldump file:
```
kubectl exec -i -n <namespace> <podname> -- mysql -h 127.0.0.1 -u <username> -p<password><CNCC Database name> < <backup_filename>   
```

Example:

kubectl exec -i -n cndbtier1 ndbmysqld-0 -- mysql -h 127.0.0.1 -u cnccusr -pcnccpasswd cnccdb <restore.sql

Note:

If multiple sites are corrupted, perform the restore procedure on all sites.