5 Rolling Back cnDBTier

This chapter describes the procedures to roll back cnDBTier deployments.

Note:

  • If you are upgrading cnDBTier using CDCS, perform all the steps in this section except the Helm rollback command (Step 4 in Rolling Back cnDBTier) and then start the CDCS rollback pipeline. For more information on rolling back cnDBTier using CDCS, see Oracle Communications CD Control Server User Guide.
  • If cnDBTier is configured with a single replication channel, then rollback has to be done using single replication channel group. If cnDBTier is configured with multiple replication channel groups, then rollback must be done using multiple replication channel groups.
  • db-backup-manager-svc is designed to automatically restart in case of errors. Therefore, when the backup-manager-svc encounters a temporary error during the rollback process, it may undergo several restarts. When cnDBTier reaches a stable state, the db-backup-manager-svc pod operates normally without any further restarts.
  • You can roll back only one georedundant cnDBTier site at a time. If you want to roll back multiple georedundant sites, then complete the rollback of one georedundant site and then move to the next one.
  • When upgrade is performed incrementally (from version n-2 to n-1 and then from n-1 to n), then rollback is not supported from n to n-2, or from n-1 to n-2.
  • Rollback is not supported on cnDBTier setups if TLS is enabled in the custom_values.yaml file.
  • Rollback is not supported from a cnDBTier release where password encryption is enabled to a cnDBTier release where password encryption is disabled and vice-versa.
  • The OCCNE_NAMESPACE variable in the rollback procedures must be set to the cnDBTier namespace. Before running any command that contains the OCCNE_NAMESPACE variable, ensure that you have set this variable to the cnDBTier namespace as stated in the following code block:
    export OCCNE_NAMESPACE=<namespace>

    where, <namespace> is the cnDBTier namespace.

5.1 Supported Rollback Paths

The following table provides the rollback paths that are supported by cnDBTier release 23.4.7.

Table 5-1 Supported Rollback Paths

Source Release Target Release
23.4.7 23.4.x
23.4.7 23.3.x
23.4.7 23.2.x

5.2 Prerequisites

The following prerequisites must be met before perfomring a cnDBTier rollback:
  1. There must be no DDL objects created in MySQL after upgrading cnDBTier. If there are any databases or tables, or any other DDL objects that were created after upgrading the cluster, delete them before starting roll back.
    1. Run the following command to delete or drop a table created after upgrade:
      DROP TABLE table_created_after_upgrade;
    2. Run the following command to delete or drop a database created after upgrade:
      DROP DATABASE database_created_after_upgrade;

      Note:

      To delete other DDL objects, refer to the MySQL manual.

5.3 Rolling Back cnDBTier

This section describes the procedure to roll back cnDBTier.

  1. Log in to the Bastion Host of the site being rolled back.
  2. List the history of the cnDBTier deployment:
    1. Run the following command to list the namespace:
      $ helm -n $OCCNE_NAMESPACE list
      Sample output:
      NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
mysql-cluster   Cluster1        2               2023-11-07 20:11:30.592221615 +0000 UTC deployed        occndbtier-23.4.7       23.4.7
    2. Run the following command to get the list of cnDBTier deployments:
      $ helm -n $OCCNE_NAMESPACE history mysql-cluster
      Sample output:
      REVISION        UPDATED                         STATUS          CHART                   APP VERSION     DESCRIPTION
1               Tue Nov 07 10:22:58 2023        superseded      occndbtier-23.3.0       23.3.0          Install complete
2               Tue Nov 07 10:22:58 2023        deployed        occndbtier-23.4.7       23.4.7          Upgrade complete
  3. Before performing the rollback, run the Helm test on all the sites. There are many scenarios where the Helm test can fail, including the health check failures in various components of cnDBTier. If HELM test fails, diagnose and resolve the actual issue and perform the HELM test again on the sites. Proceed further only when the Helm test succeeds.
    $ helm test mysql-cluster --namespace ${OCCNE_NAMESPACE}
    Sample output:
    NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 10:22:58 2023
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 1
TEST SUITE: mysql-cluster-node-connection-test
Last Started: Tue Nov 07 10:22:58 2023
Last Completed: Tue Nov 07 10:22:58 2023
Phase: Succeeded
  4. Run the following command to roll back to the previous version:
    helm --namespace ${OCCNE_NAMESPACE} rollback $CNDBTIER_RELEASE_NAME $REVISION
    Where,
    • $CNDBTIER_RELEASE_NAME is the name of the cnDBTier deployment obtained from Step 3. For example, CNDBTIER_RELEASE_NAME="mysql-cluster"
    • $REVISION is the revision number of the release that you want to rollback to, obtained from Step 3. For example, REVISION=1.
  5. If there is no Upgrade Service Account configured for cnDBTier, run the postrollback script. This script restarts the MGM pods and patches upgradeStrategy.

    Note:

    Replace the values of the environment variables in the following sample with the values of your cluster.
    1. Define the following variable before running the postrollback script:
      export OCCNE_NAMESPACE="occne-cndbtier"
export API_EMP_TRY_SLOTS_NODE_IDS="id=222\|id=223\|id=224\|id=225"
    2. If values.global.ndbapp.ndb_cluster_connection_pool is greater than 1, then declare the following environment variables:
      #export APP_CON_POOL_INGORE_NODE_IDS="id=100"
export APP_CON_POOL_INGORE_NODE_IDS="id=100\|id=101\|id=102\|id=103"
export NDB_MGMD_PODS="ndbmgmd-0 ndbmgmd-1"
export NDB_STS_NAME="ndbmtd"
export API_STS_NAME="ndbmysqld"
export APP_STS_NAME="ndbappmysqld"
export NDBAPP_START_NODE_ID="70"
export NDBAPP_REPLICA_MAX_COUNT="4"
    3. Run the postrollback script:
      occndbtier/files/hooks.sh --post-rollback
  6. Wait for the appropriate nodes to restart and reconnect to the cluster and then run the following command to verify the cluster status from the management pod. The pods that is restarted depends on the pods that are rolled back.
    $ kubectl -n ${OCCNE_NAMESPACE} exec -it ndbmgmd-0 -- ndb_mgm -e show
    Sample output:
    Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.104.176  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.121.175  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.101.154  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.104.174  (mysql-8.0.35 ndb-8.0.35)
 
[mysqld(API)]   8 node(s)
id=56   @10.233.92.169  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.101.155  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.92.170  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.121.176  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)
  7. After rollback, run the Helm test on all the sites and verify if the cnDBTier services are running successfully:
    $ helm test mysql-cluster --namespace ${OCCNE_NAMESPACE}
    Sample output:
    NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 10:22:58 2023
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 1
TEST SUITE: mysql-cluster-node-connection-test
Last Started: Tue Nov 07 10:22:58 2023
Last Completed: Tue Nov 07 10:22:58 2023
Phase: Succeeded