5 Rolling Back cnDBTier

This chapter describes the procedures to roll back cnDBTier deployments.

Note:

  • If cnDBTier is configured with a single replication channel, then rollback has to be done using single replication channel group. If cnDBTier is configured with multiple replication channel groups, then rollback must be done using multiple replication channel groups.
  • db-backup-manager-svc is designed to automatically restart in case of errors. Therefore, when the backup-manager-svc encounters a temporary error during the rollback process, it may undergo several restarts. When cnDBTier reaches a stable state, the db-backup-manager-svc pod operates normally without any further restarts.
  • You can roll back only one georedundant cnDBTier site at a time. If you want to roll back multiple georedundant sites, then complete the rollback of one georedundant site and then move to the next one.
  • If you had performed the upgrade incrementally (from version n-2 to n-1, and then from n-1 to n), then rollback is not supported from n to n-2, or from n-1 to n-2.
  • If you had modifyed the TLS certificates while upgrading the cnDBTier cluster from a TLS enabled version to another TLS enabled version, then before proceeding with the rollback procedure, perform the "Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites" procedure in Oracle Communications Cloud Native Core, cnDBTier User Guide to add the old certificates and retain the existing certificates.
  • The OCCNE_NAMESPACE variable in the rollback procedures must be set to the cnDBTier namespace. Before running any command that contains the OCCNE_NAMESPACE variable, ensure that you have set this variable to the cnDBTier namespace as stated in the following code block:
    export OCCNE_NAMESPACE=<namespace>

    where, <namespace> is the cnDBTier namespace.

  • The namespace name "occne-cndbtier" given in the rollback procedure is only an example. Ensure that you configure the namespace name according to your environment.
  • cnDBTier 24.3.1 supports Helm 3.12.3 and 3.13.2. Ensure that you upgrade Helm to a supported version.

5.1 Supported Rollback Paths

The following table provides the rollback paths that are supported by cnDBTier release 24.3.1.

Table 5-1 Supported Rollback Paths

Source Release Target Release
24.3.1 24.3.x
24.3.1 24.2.x
24.3.1 24.1.x

5.2 Prerequisites

The following prerequisites must be met before perfomring a cnDBTier rollback:
  1. There must be no DDL objects created in MySQL after upgrading cnDBTier. If there are any databases or tables, or any other DDL objects that were created after upgrading the cluster, delete them before starting roll back.
    1. Run the following command to delete or drop a table created after upgrade:
      DROP TABLE table_created_after_upgrade;
    2. Run the following command to delete or drop a database created after upgrade:
      DROP DATABASE database_created_after_upgrade;

      Note:

      To delete other DDL objects, refer to the MySQL manual.

5.3 Rolling Back cnDBTier

This section describes the procedure to roll back cnDBTier.

  1. Log in to the Bastion Host of the site being rolled back.
  2. List the history of the cnDBTier deployment:
    1. Run the following command to list the namespace:
      $ helm -n $OCCNE_NAMESPACE list
      Sample output:
      NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
mysql-cluster   Cluster1        2               2024-02-19 20:11:30.592221615 +0000 UTC deployed        occndbtier-24.3.1       24.3.1
    2. Run the following command to get the list of cnDBTier deployments:
      $ helm -n $OCCNE_NAMESPACE history mysql-cluster
      Sample output:
      REVISION        UPDATED                         STATUS          CHART                   APP VERSION     DESCRIPTION
1                Mon May 20 13:31:35 2024        superseded      occndbtier-24.2.1       24.2.1          Install complete
2                Mon May 20 20:11:30 2024        deployed        occndbtier-24.3.1       24.3.1          Upgrade complete
  3. Before performing the rollback, run the Helm test on all the sites. There are many scenarios where the Helm test can fail, including the health check failures in various components of cnDBTier. If HELM test fails, diagnose and resolve the actual issue and perform the HELM test again on the sites. Proceed further only when the Helm test succeeds.
    $ helm test mysql-cluster --namespace ${OCCNE_NAMESPACE}
    Sample output:
    NAME: mysql-cluster
LAST DEPLOYED:  Mon May 20 10:22:58 2024
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 1
TEST SUITE: mysql-cluster-node-connection-test
Last Started:  Mon May 20 14:15:18 2024
Last Completed:  Mon May 20 14:17:58 2024
Phase: Succeeded
  4. If any of the following parameters were updated during an upgrade, then restart the ndbmtd pods with the --initial flag after the rollback:

    Note:

    Perform this step only if you are upgrading from a previous version (24.2.x or 24.1.x).
    • DataDir
    • FileSystemPath
    • BackupDataDir
    • FragmentLogFileSize
    • InitFragmentLogFiles
    • NoOfFragmentLogFiles
    • NoOfFragmentLogParts
    • FileSystemPathDD
    • FileSystemPathDataFiles
    • FileSystemPathUndoFiles
    • IndexStatSaveSize
    • IndexStatSaveScale
    • IndexStatTriggerPct
    • IndexStatTriggerScale
    • IndexStatUpdateDelay
    • RedoBuffer
    Perform the following steps to instruct the data nodes to restart the ndbmtd pods with the --inital flag after the rollback:

    Note:

    If a setup is deployed with a prefix, then replace the pod name of ndbmtd accordingly.
    1. Run the following command to restart the ndbmtd pods with --inital flag during rolling restart of data node, after the rollback:
      $ for x in {0..N}; do kubectl -n <cndbtier_namespace>  exec -it <NDBMTD_STS_NAME_WITH_PREFIX>-$x -c db-backup-executor-svc -- bash -c  "touch /var/occnedb/useinitial && chmod 777 /var/occnedb/useinitial";  done;
      For example, if a setup has four ndbmtd pods, use the following command:
      $ for x in {0..3}; do kubectl -n occne-cndbtier  exec -it ndbmtd-$x -c db-backup-executor-svc -- bash -c  "touch /var/occnedb/useinitial && chmod 777 /var/occnedb/useinitial"; done;
    2. Verify if the useiinitial file is created for each data node in the /var/occnedb directory:
      $  for x in {0..N}; do kubectl -n <cndbtier_namespace>  exec -it <NDBMTD_STS_NAME_WITH_PREFIX>-$x -c db-backup-executor-svc -- bash -c  "echo ndbmtd-$x;ls -lrth  /var/occnedb/ | grep -i useinitial "; done;
      For example:
      $ for x in {0..3}; do kubectl -n occne-cndbtier  exec -it ndbmtd-$x -c db-backup-executor-svc -- bash -c  "echo ndbmtd-$x;ls -lrth  /var/occnedb/ | grep -i useinitial "; done;
      Sample output:
      ndbmtd-0
-rwxrwxrwx. 1 mysql mysql    0 Sep 23 11:56 useinitial
ndbmtd-1
-rwxrwxrwx. 1 mysql mysql    0 Sep 23 11:56 useinitial
ndbmtd-2
-rwxrwxrwx. 1 mysql mysql    0 Sep 23 11:56 useinitial
ndbmtd-3
-rwxrwxrwx. 1 mysql mysql    0 Sep 23 11:56 useinitial
  5. If there is no upgrade service account configured for cnDBTier or if you are rolling back to a previous cnDBTier, run the pre-rollback-with-tde-to-non-tde script:
    export OCCNE_NAMESPACE="occne-cndbtier"
export NDB_MTD_PODS=$(kubectl -n ${OCCNE_NAMESPACE} get pods | grep 'ndbmtd' | awk '{print $1}')
export NDB_MGMD_PODS=$(kubectl -n ${OCCNE_NAMESPACE} get pods | grep 'ndbmgmd' | awk '{print $1}')
export NDB_STS_NAME=$(kubectl -n ${OCCNE_NAMESPACE} get sts | grep 'ndbmtd' | awk '{print $1}')
 
occndbtier/files/hooks.sh --pre-rollback-with-tde-to-non-tde
  6. If there is no upgrade service account configured for cnDBTier or if you are rolling back to a previous cnDBTier release, then run the pre-rollback script:
    export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_PASSWORD="<password for the user occneuser>"
export DBTIER_REPLICATION_SVC_DATABASE="replication_info"
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0/ndbappmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
export NDB_MGMD_PODS="ndbmgmd-0 ndbmgmd-1"
  
# To enable or disable password encryption , uncomment the below line and set the variable to true or false to enable or disable it.
# export ENABLE_ENCRYPTION="<true/false>"
 
occndbtier/files/hooks.sh --pre-rollback
  7. Run the following command to roll back to the previous version:
    helm --namespace ${OCCNE_NAMESPACE} rollback $CNDBTIER_RELEASE_NAME $REVISION
    Where,
    • $CNDBTIER_RELEASE_NAME is the name of the cnDBTier deployment obtained from Step 3. For example, CNDBTIER_RELEASE_NAME="mysql-cluster"
    • $REVISION is the revision number of the release that you want to rollback to, obtained from Step 3. For example, REVISION=1.
  8. If there is no upgrade service account configured for cnDBTier, run the postrollback script. This script restarts the MGM pods and patches upgradeStrategy.

    Note:

    • Replace the values of the environment variables in the following sample with the values of your cluster.
    • If prefix is used in pod names, then add the prefix to the following environment variable for both the pod and statefulset names. For example:
      export NDB_MGMD_PODS="<global.k8sResource.pod.prefix>-ndbmgmd-0 <global.k8sResource.pod.prefix>-ndbmgmd-1"
export NDB_STS_NAME="<global.k8sResource.pod.prefix>-ndbmtd"
    1. Define the following variables before running the postrollback script:
      export OCCNE_NAMESPACE="occne-cndbtier"
export API_EMP_TRY_SLOTS_NODE_IDS="id=222\|id=223\|id=224\|id=225"
#If values.global.ndbapp.ndb_cluster_connection_pool is greater than 1, then declare the following environment variable
#export APP_CON_POOL_INGORE_NODE_IDS="id=100"
export APP_CON_POOL_INGORE_NODE_IDS="id=100\|id=101\|id=102\|id=103"
export NDB_MGMD_PODS="ndbmgmd-0 ndbmgmd-1"
export NDB_MTD_PODS="ndbmtd-0 ndbmtd-1"
export NDB_STS_NAME="ndbmtd"
export API_STS_NAME="ndbmysqld"
export APP_STS_NAME="ndbappmysqld"
export NDBAPP_START_NODE_ID="70"
export NDBAPP_REPLICA_MAX_COUNT="4"
    2. Run the postrollback script:
      occndbtier/files/hooks.sh --post-rollback
  9. Wait for the appropriate nodes to restart and reconnect to the cluster and then run the following command to verify the cluster status from the management pod. The pods that is restarted depends on the pods that are rolled back.
    $ kubectl -n ${OCCNE_NAMESPACE} exec -it ndbmgmd-0 -- ndb_mgm -e show
    Sample output:
    Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.104.176  (mysql-8.4.2 ndb-8.4.2, Nodegroup: 0)
id=2    @10.233.121.175  (mysql-8.4.2 ndb-8.4.2, Nodegroup: 0, *)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.101.154  (mysql-8.4.2 ndb-8.4.2)
id=50   @10.233.104.174  (mysql-8.4.2 ndb-8.4.2)
 
[mysqld(API)]   8 node(s)
id=56   @10.233.92.169  (mysql-8.4.2 ndb-8.4.2)
id=57   @10.233.101.155  (mysql-8.4.2 ndb-8.4.2)
id=70   @10.233.92.170  (mysql-8.4.2 ndb-8.4.2)
id=71   @10.233.121.176  (mysql-8.4.2 ndb-8.4.2)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)
  10. After rollback, run the Helm test on all the sites and verify if the cnDBTier services are running successfully:
    $ helm test mysql-cluster --namespace ${OCCNE_NAMESPACE}
    Sample output:
    NAME: mysql-cluster
LAST DEPLOYED:  Mon May 20 10:22:58 2024
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 1
TEST SUITE: mysql-cluster-node-connection-test
Last Started:  Mon May 20 14:15:18 2024
Last Completed:  Mon May 20 14:17:58 2024
Phase: Succeeded

5.4 Rolling Back cnDBTier from Non-TLS enabled to TLS Enabled Version

This section describes the procedure to roll back cnDBTier clusters from non-TLS version to a TLS enabled version.

Note:

In this procedure, the cnDBTier sites are rolled back twice (in step 1 and step 3). Ensure that you follow this procedure as-is to roll back from a non-TLS version to a TLS enabled version.
  1. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS mode is set to "NONE".
  2. After rolling back each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.
    For example, run the following command to check the georeplication status of cnDBTier cluster2 configured with other remote sites:
    $ kubectl -n cluster2 exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.cluster2:8080/db-tier/status/replication/realtime
    Sample output:
    [
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster1",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster3",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster4",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  }
]

    In the sample output, the replicationStatus is "UP" for the localSiteName cluster2 for remotesiteName cluster1, cluster3, and cluster4. This indicates that the localSiteName cluster2 is able to replicate data from remotesiteName cluster1, cluster3, and cluster4.

  3. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS mode is set to "VERIFY_CA" or "VERIFY_IDENTITY". This configuration ensures that the clients use an encrypted connection and performs verification against the server CA certificate:
    • VERIFY_CA instructs the client to check if the server’s certificate is valid.
    • VERIFY_IDENTITY instructs the client to check if the server’s certificate is valid and to check if the host name used by the client, matches the identity in the server’s certificate.
  4. After rolling back each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.

    For example, see Step 2.

  5. Run the hooks.sh script with the --add-ssltype flag on all cnDBTier sites to set the type of "occnerepluser" as TLS:

    Note:

    Update the values of the environment variables in the following code block as per your cluster. 
    export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_PASSWORD="<password for the user occneuser>"
export OCCNE_REPL_USER_NAME="occnerepluser"
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
   
occndbtier/files/hooks.sh --add-ssltype

5.5 Rolling Back cnDBTier from TLS enabled to Non-TLS Enabled Version

This section describes the procedure to roll back cnDBTier clusters from TLS enabled version to a non-TLS version.

Note:

In this procedure, the cnDBTier sites are rolled back twice (in step 2 and step 4). Ensure that you follow this procedure as-is to roll back from a TLS enabled version to a non-TLS version.
  1. Run the hooks.sh script with the -- remove-ssltype flag on all cnDBTier sites to set the type of "occnerepluser" as Non-TLS:

    Note:

    Update the values of the environment variables in the following code block as per your cluster. 
    export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_PASSWORD="<password for the user occneuser>"
export OCCNE_REPL_USER_NAME="occnerepluser"
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
   
occndbtier/files/hooks.sh --remove-ssltype
  2. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS mode is set to "NONE".
  3. After rolling back each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.
    For example, run the following command to check the georeplication status of cnDBTier cluster2 configured with other remote sites:
    $ kubectl -n cluster2 exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.cluster2:8080/db-tier/status/replication/realtime
    Sample output:
    [
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster1",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster3",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster4",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  }
]

    In the sample output, the replicationStatus is "UP" for the localSiteName cluster2 for remotesiteName cluster1, cluster3, and cluster4. This indicates that the localSiteName cluster2 is able to replicate data from remotesiteName cluster1, cluster3, and cluster4.

  4. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS is disabled.
  5. After rolling back each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.

    For example, see Step 3.