Rolling Back cnDBTier

5 Rolling Back cnDBTier

This chapter describes the procedures to roll back cnDBTier deployments.

Note:

If cnDBTier is configured with a single replication channel, then rollback has to be done using single replication channel group. If cnDBTier is configured with multiple replication channel groups, then rollback must be done using multiple replication channel groups.
db-backup-manager-svc is designed to automatically restart in case of errors. Therefore, when the backup-manager-svc encounters a temporary error during the rollback process, it may undergo several restarts. When cnDBTier reaches a stable state, the db-backup-manager-svc pod operates normally without any further restarts.
You can roll back only one georedundant cnDBTier site at a time. If you want to roll back multiple georedundant sites, then complete the rollback of one georedundant site and then move to the next one.
If you had performed the upgrade incrementally (from version n-2 to n-1, and then from n-1 to n), then rollback is not supported from n to n-2, or from n-1 to n-2.
If you had modified the TLS certificates for replication while upgrading the cnDBTier cluster from a TLS enabled version to a TLS enabled version, then before proceeding with the rollback procedure, perform the "Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites" procedure in Oracle Communications Cloud Native Core, cnDBTier User Guide to add the old certificates and retain the existing certificates.
If you had modified the TLS certificates of application SQL pods for NF communication while upgrading the cnDBTier cluster from a TLS enabled version to a TLS enabled version, then before proceeding with the rollback procedure, perform the "Modifying cnDBTier Certificates to Establish TLS for Communication with NFs" procedure in Oracle Communications Cloud Native Core, cnDBTier User Guide to add the old certificates and retain the existing certificates.
The OCCNE_NAMESPACE variable in the rollback procedures must be set to the cnDBTier namespace. Before running any command that contains the OCCNE_NAMESPACE variable, ensure that you have set this variable to the cnDBTier namespace as stated in the following code block:
```
export OCCNE_NAMESPACE=<namespace>
```
where, <namespace> is the cnDBTier namespace.
The namespace name "occne-cndbtier" given in the rollback procedure is only an example. Ensure that you configure the namespace name according to your environment.
cnDBTier 26.1.202 supports Helm 3.12.3 and 3.13.2. Ensure that you upgrade Helm to a supported version.
If HTTPS is enabled, then:
- cnDBTier support only one set of certificates for each site.
- Rollback to a previous release is supported only when each site has the same set of certificates as that of the previous release.

5.1 Supported Rollback Paths

The following table provides the rollback paths that are supported by cnDBTier release 26.1.202.

Table 5-1 Supported Rollback Paths

Source Release	Target Release
26.1.202	26.1.2xx
26.1.202	25.2.2xx
26.1.202	25.2.1xx

Note:

If cnDBTier is configured with single replication channel then downgrade has to be done using single replication channel group. If cnDBTier is configured with multi replication channel group then downgrade has to be done using multi replication channel group.
The db-backup-manager-svc component has been intentionally designed to automatically restart in the event of encountering any errors. Therefore, in the case where the backup-manager-svc encounters a temporary error during the rollback process of cnDBTier, it may undergo several restarts. Once the cnDBTier reaches a stable state, it is expected that the db-backup-manager-svc pod will operate fine without any further restarts.
During rollback user must rollback only one georedundant cnDBTier site at a time. Upon completion of rollback user must move to next georedundant cnDBTier site for rollback.
When the upgrade is performed in incremental way (n-2 -> n-1 -> n cnDBTier versions), rollback is not supported from n to n-2, or n-1 to n-2.
If the tls certificates are modified during the upgrade of the cnDBTier cluster from TLS to TLS, follow the "Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites" section in Oracle Communication Cloud Network Core, cnDBTier User Guide to add old certificates by retaining existing certificates and then proceed with the rollback.
If the TLS certificates for APP SQL pods are modified during the upgrade of the cnDBTier cluster from TLS to TLS, follow the "Modifying cnDBTier Certificates to Establish TLS for Communication with NFs" section in Oracle Communication Cloud Network Core, cnDBTier User Guide to add old certificates by retaining existing certificates and then proceed with the rollback.

5.2 Prerequisites

The following prerequisite must be met before performing a cnDBTier rollback.

There must be no DDL objects created in MySQL after upgrading cnDBTier. If there are any databases or tables, or any other DDL objects that were created after upgrading the cluster, delete them before starting roll back.

Run the following command to delete or drop a table created after upgrade:
```
DROP TABLE table_created_after_upgrade;
```
Run the following command to delete or drop a database created after upgrade:
```
DROP DATABASE database_created_after_upgrade;
```
Note:
To delete other DDL objects, refer to the MySQL manual.

5.3 Rolling Back cnDBTier from Non-TLS to TLS Enabled Version (Replication)

This section describes the procedure to roll back cnDBTier clusters from a version where TLS is not enabled for replication to a version where TLS is enabled for replication.

Note:

In this procedure, the cnDBTier sites are rolled back twice (in step 1 and step 3). Ensure that you follow this procedure as-is to roll back from a non-TLS version to a TLS enabled version.
The namespace name "occne-cndbtier" given in this procedure is only an example. Ensure that you configure the namespace name according to your environment.

Perform the unresolvable-reference.html#GUID-CD67688F-3B1C-4F8E-8DE9-7B8B8A318E88 procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS mode is set to "NONE".
After rolling back each site, run the following command on the site to ensure that the replication is UP:
```
$ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime
```
where, <namespace> is the namespace name of the of cnDBTier cluster.
The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
- "UP": Indicates that the local site is able to replicate data from that remote site.
- "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.
For example, run the following command to check the georeplication status of cnDBTier cluster2 configured with other remote sites:
```
$ kubectl -n cluster2 exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.cluster2:8080/db-tier/status/replication/realtime
```
Sample output:
```
[
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster1",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster3",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster4",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  }
]
```
In the sample output, the replicationStatus is "UP" for the localSiteName cluster2 for remotesiteName cluster1, cluster3, and cluster4. This indicates that the localSiteName cluster2 is able to replicate data from remotesiteName cluster1, cluster3, and cluster4.
Perform the unresolvable-reference.html#GUID-CD67688F-3B1C-4F8E-8DE9-7B8B8A318E88 procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS mode is set to "VERIFY_CA" or "VERIFY_IDENTITY". This configuration ensures that the clients use an encrypted connection and performs verification against the server CA certificate:
- VERIFY_CA instructs the client to check if the server’s certificate is valid.
- VERIFY_IDENTITY instructs the client to check if the server’s certificate is valid and to check if the host name used by the client, matches the identity in the server’s certificate.
After rolling back each site, run the following command on the site to ensure that the replication is UP:
```
$ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime
```
where, <namespace> is the namespace name of the of cnDBTier cluster.
The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
- "UP": Indicates that the local site is able to replicate data from that remote site.
- "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.
For example, see Step 2.

Run the hooks.sh script with the --add-ssltype flag on all cnDBTier sites to set the type of "occnerepluser" as TLS:

Note:

Update the values of the environment variables in the following code block as per your cluster.

export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_ACCESS_KEY="<password for the user occneuser>"
export OCCNE_REPL_USER_NAME="occnerepluser"
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
   
occndbtier/files/hooks.sh --add-ssltype

5.4 Rolling Back cnDBTier from TLS enabled to Non-TLS Version (Replication)

This section describes the procedure to roll back cnDBTier clusters from a version where TLS is enabled for replication to a version where TLS is not enabled for replication.

Note:

In this procedure, the cnDBTier sites are rolled back twice (in step 2 and step 4). Ensure that you follow this procedure as-is to roll back from a TLS enabled version to a non-TLS version.
The namespace name "occne-cndbtier" given in this procedure is only an example. Ensure that you configure the namespace name according to your environment.

Run the hooks.sh script with the

--
                    remove-ssltype

flag on all cnDBTier sites to set the type of


                    "occnerepluser"

as Non-TLS:

Note:

Update the values of the environment variables in the following code block as per your cluster.

export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_ACCESS_KEY="<password for the user occneuser>"
export OCCNE_REPL_USER_NAME="occnerepluser"
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
   
occndbtier/files/hooks.sh --remove-ssltype

Perform the unresolvable-reference.html#GUID-CD67688F-3B1C-4F8E-8DE9-7B8B8A318E88 procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS mode is set to "NONE".
After rolling back each site, run the following command on the site to ensure that the replication is UP:
```
$ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime
```
where, <namespace> is the namespace name of the of cnDBTier cluster.
The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
- "UP": Indicates that the local site is able to replicate data from that remote site.
- "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.
For example, run the following command to check the georeplication status of cnDBTier cluster2 configured with other remote sites:
```
$ kubectl -n cluster2 exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.cluster2:8080/db-tier/status/replication/realtime
```
Sample output:
```
[
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster1",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster3",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster4",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  }
]
```
In the sample output, the replicationStatus is "UP" for the localSiteName cluster2 for remotesiteName cluster1, cluster3, and cluster4. This indicates that the localSiteName cluster2 is able to replicate data from remotesiteName cluster1, cluster3, and cluster4.
Perform the unresolvable-reference.html#GUID-CD67688F-3B1C-4F8E-8DE9-7B8B8A318E88 procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS is disabled.
After rolling back each site, run the following command on the site to ensure that the replication is UP:
```
$ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime
```
where, <namespace> is the namespace name of the of cnDBTier cluster.
The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
- "UP": Indicates that the local site is able to replicate data from that remote site.
- "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.
For example, see Step 3.

5.5 Rolling Back cnDBTier from TLS enabled to Non-TLS Version (NF Communication)

This section describes the procedure to roll back cnDBTier clusters from a version where TLS is enabled in application SQL pod for NF communication to a version where TLS is not enabled in application SQL pod for NF communication.

Note:

The namespace name "occne-cndbtier" given in this procedure is only an example. Ensure that you configure the namespace name according to your environment.

Perform the following steps to set the NF user to NONE such that the application SQL pods of cnDBTier accepts both TLS and non-TLS communication from the NF that uses the NF user:

$ kubectl -n  <namespace of cnDBTier Cluster> exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -p

Example:

$ kubectl -n cluster1 exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -p

Sample output:

Enter Password:

Run the following command to check for NF-specific user account:

$ mysql> select user, host  from mysql.user;

Example:

+------------------+-----------+
| user             | host      |
+------------------+-----------+
| occnerepluser    | %         |
| occneuser        | %         |
| healthchecker    | localhost |
| mysql.infoschema | localhost |
| mysql.session    | localhost |
| mysql.sys        | localhost |
| root             | localhost |
| nfuser           | %         |
+------------------+-----------+
7 rows in set (0.00 sec)

Before altering the user privilege, run the following command to turn off binlogging on one of the ndbappmysqld pods:
```
$ mysql>SET sql_log_bin = OFF;
```

Run the following the commands to alter the user privilege to NONE :

$ mysql> ALTER USER '<USERNAME>'@'%' REQUIRE NONE;   
$ mysql> FLUSH PRIVILEGES;

Turn on the binlogging on the same ndbappmysqld pod before you exit from the session:
```
mysql>SET sql_log_bin = ON;
```
Exit the session:
```
$ mysql> exit;
```

Make the NF flexible enough to communicate with cnDBTier using TLS. However, when cnDBTier is rolled back to a non-TLS version, the NF must use non-TLS communication.
Ensure that /global/ndbappTLS/enable is set to false in the custom_values.yaml file for all cnDBTier sites. This indicates that TLS is disabled in application SQL pods for NF communication:
```
global:
  ndbappTLS:    
    enable: false
```
Perform the unresolvable-reference.html#GUID-CD67688F-3B1C-4F8E-8DE9-7B8B8A318E88 procedure on the cnDBTier cluster to roll back cnDBTier to a version where TLS is not enabled in the application SQL pod for NF communication.

5.6 Rolling Back cnDBTier from Non-TLS to TLS Enabled Version (NF Communication)

This section describes the procedure to roll back cnDBTier clusters from a version where TLS is not enabled in application SQL pod for NF communication to a version where TLS is enabled in application SQL pod for NF communication.

Note:

The namespace name "occne-cndbtier" given in this procedure is only an example. Ensure that you configure the namespace name according to your environment.

Perform the unresolvable-reference.html#GUID-CD67688F-3B1C-4F8E-8DE9-7B8B8A318E88 procedure on the cnDBTier cluster to roll back cnDBTier to a version where TLS is enabled in the application SQL pod for NF communication.
Connect the NF application to cnDBTier using TLS. For the procedure to connect NF to cnDBTier using TLS, see the respective NF documentation.
At this stage, cnDBTier is rolled back and the application SQL pod is configured with TLS certificates. However, the application SQL pod still allows non-TLS communication. To restrict the communication, you must set the mode of the user to X509 such that any NF using the user follows TLS strictly. When the NF user is created, perform the following steps to set the mode of the NF user to X509.

Note:
Before performing this step ensure that the NF is ready to communicate with cnDBTier using TLS.
1. Log in to the ndbappmysqld pod. Enter the password when prompted.
```
$ kubectl -n  <namespace of cnDBTier Cluster> exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -p
```
  Example:
```
$ kubectl -n cluster1 exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -p
```
  Sample output:
```
Enter Password:
```
2. Run the following command to check for NF-specific user account. If there are no NF-specific user accounts, create new user accounts:
```
$ mysql> select user, host  from mysql.user;
```
  Example:
```
+------------------+-----------+
| user             | host      |
+------------------+-----------+
| occnerepluser    | %         |
| occneuser        | %         |
| healthchecker    | localhost |
| mysql.infoschema | localhost |
| mysql.session    | localhost |
| mysql.sys        | localhost |
| root             | localhost |
| nfuser           | %         |
+------------------+-----------+
7 rows in set (0.00 sec)
```
3. Before altering the user and granting privilege, run the following command to turn off binlogging on one of the ndbappmysqld pods:
```
$ mysql>SET sql_log_bin = OFF;
```
4. Run the following the commands to alter the user and grant the privileges:
```
$ mysql> ALTER USER '<USERNAME>'@'%' REQUIRE X509;   
$ mysql> FLUSH PRIVILEGES;
```
5. Turn on the binlogging on the same ndbappmysqld pod before you exit from the session:
```
mysql>SET sql_log_bin = ON;
```
6. Exit the session:
```
$ mysql> exit;
```

5.7 Rolling Back cnDBTier from HTTPS Disabled to HTTPS Enabled Version

This section describes the procedure to roll back cnDBTier clusters from HTTPS disabled to HTTS enabled version.

Procedure to roll back cnDBTier from HTTPS enabled to HTTPS disabled version

Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter is enabled and HTTPS are disabled.
Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter and HTTPS are enabled.
Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter is disabled and HTTPS is enabled.

Note:

This rollback is primarily for removal of the HTTPS port, so that only HTTP connection is allowed in cnDBTier. This step can be performed at any phase of upgrade.

5.8 Rolling Back cnDBTier from HTTPS Enabled to HTTPS Disabled Version

This section describes the procedure to roll back cnDBTier clusters from HTTPS enabled to HTTS disabled version.

Procedure to roll back cnDBTier from HTTPS enabled to HTTPS disabled version

Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter and HTTPS are both enabled.
Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter is enabled and HTTPS is disabled.
Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter and HTTPS are both disabled.

Note:

This rollback is primarily for removal of the HTTPS port so that only HTTP connection is allowed in cnDBTier. This step can be performed at any phase of upgrade.

5.9 Real Time Local Cluster Status API

This API provides the current operational status of a local cnDBTier cluster in real time.

API Definition

URI: http://base-uri/ocdbtier/status/cluster/local/realtime

Method: GET

API Type: Realtime API

Request Payload Response Code Response Payload

200 OK- The local NDB cluster is UP or DOWN

Request Payload	Response Code	Response Payload
NA	200 OK- The local NDB cluster is UP or DOWN	`{ "clusterName":"<current-cluster-name>", "clusterStatus":"UP/DOWN" }`
500 Internal Server Error	`{ "title": "Internal Server Error", "status": 500, "detail": "<Exception details>", "cause": <cause>, "invalidParams": <params> }`

{
  "clusterName":"<current-cluster-name>",
  "clusterStatus":"UP/DOWN"
}

500 Internal Server Error

{
  "title": "Internal Server Error",
  "status": 500,
 "detail": "<Exception details>",
  "cause": <cause>,
  "invalidParams": <params>
}

API Configuration

Field	Description	Details
clusterName	This attribute displays the name of the current cluster in which the cnDBTier cluster is deployed.	DataType: string Constraints: NA Default Value: NA
clusterStatus	This attribute displays the local cluster status for the current cluster.	DataType: string Constraints: UP, DOWN Default Value: NA

Field

Description

Details

clusterName

This attribute displays the name of the current cluster in which the cnDBTier cluster is deployed.

DataType: string

Constraints: NA

Default Value: NA

clusterStatus

This attribute displays the local cluster status for the current cluster.

DataType: string

Constraints: UP, DOWN

Default Value: NA