5 Rolling Back cnDBTier

This chapter describes the procedures to roll back cnDBTier deployments.

Note:

  • If cnDBTier is configured with a single replication channel, then rollback has to be done using single replication channel group. If cnDBTier is configured with multiple replication channel groups, then rollback must be done using multiple replication channel groups.
  • db-backup-manager-svc is designed to automatically restart in case of errors. Therefore, when the backup-manager-svc encounters a temporary error during the rollback process, it may undergo several restarts. When cnDBTier reaches a stable state, the db-backup-manager-svc pod operates normally without any further restarts.
  • You can roll back only one georedundant cnDBTier site at a time. If you want to roll back multiple georedundant sites, then complete the rollback of one georedundant site and then move to the next one.
  • If you had performed the upgrade incrementally (from version n-2 to n-1, and then from n-1 to n), then rollback is not supported from n to n-2, or from n-1 to n-2.
  • If you had modified the TLS certificates for replication while upgrading the cnDBTier cluster from a TLS enabled version to a TLS enabled version, then before proceeding with the rollback procedure, perform the "Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites" procedure in Oracle Communications Cloud Native Core, cnDBTier User Guide to add the old certificates and retain the existing certificates.
  • If you had modified the TLS certificates of application SQL pods for NF communication while upgrading the cnDBTier cluster from a TLS enabled version to a TLS enabled version, then before proceeding with the rollback procedure, perform the "Modifying cnDBTier Certificates to Establish TLS for Communication with NFs" procedure in Oracle Communications Cloud Native Core, cnDBTier User Guide to add the old certificates and retain the existing certificates.
  • The OCCNE_NAMESPACE variable in the rollback procedures must be set to the cnDBTier namespace. Before running any command that contains the OCCNE_NAMESPACE variable, ensure that you have set this variable to the cnDBTier namespace as stated in the following code block:
    export OCCNE_NAMESPACE=<namespace>

    where, <namespace> is the cnDBTier namespace.

  • The namespace name "occne-cndbtier" given in the rollback procedure is only an example. Ensure that you configure the namespace name according to your environment.
  • cnDBTier 25.1.201 supports Helm 3.12.3 and 3.13.2. Ensure that you upgrade Helm to a supported version.
  • If HTTPS is enabled, then:
    • cnDBTier support only one set of certificates for each site.
    • Rollback to a previous release is supported only when each site has the same set of certificates as that of the previous release.

5.1 Supported Rollback Paths

The following table provides the rollback paths that are supported by cnDBTier release 25.1.201.

Table 5-1 Supported Rollback Paths

Source Release Target Release
25.1.201 25.1.2xx
25.1.201 25.1.1xx
25.1.201 24.3.x
25.1.201 24.2.x

Note:

  • If cnDBTier is configured with single replication channel then downgrade has to be done using single replication channel group. If cnDBTier is configured with multi replication channel group then downgrade has to be done using multi replication channel group.
  • The db-backup-manager-svc component has been intentionally designed to automatically restart in the event of encountering any errors. Therefore, in the case where the backup-manager-svc encounters a temporary error during the rollback process of cnDBTier, it may undergo several restarts. Once the cnDBTier reaches a stable state, it is expected that the db-backup-manager-svc pod will operate fine without any further restarts.
  • It is recommended to rollback only one georedundant cnDBTier site at a time. Upon completion of rollback move to next georedundant cnDBTier site for rollback.
  • When the upgrade is performed in an incremental way (n-2 -> n-1 -> n cnDBTier versions), rollback is not supported from n to n-2, or n-1 to n-2.
  • If the TLS certificates are modified during the upgrade of the cnDBTier cluster from TLS to TLS, follow the "Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites" section in Oracle Communication Cloud Network Core, cnDBTier User Guide to add old certificates by retaining existing certificates and then proceed with the rollback.
  • If the TLS certificates for APP SQL pods are modified during the upgrade of the cnDBTier cluster from TLS to TLS, follow the "Modifying cnDBTier Certificates to Establish TLS for Communication with NFs" section in Oracle Communication Cloud Network Core, cnDBTier User Guide to add old certificates by retaining existing certificates and then proceed with the rollback.

5.2 Prerequisites

The following prerequisite must be met before performing a cnDBTier rollback.

There must be no DDL objects created in MySQL after upgrading cnDBTier. If there are any databases or tables, or any other DDL objects that were created after upgrading the cluster, delete them before starting roll back.
  1. Run the following command to delete or drop a table created after upgrade:
    DROP TABLE table_created_after_upgrade;
  2. Run the following command to delete or drop a database created after upgrade:
    DROP DATABASE database_created_after_upgrade;

    Note:

    To delete other DDL objects, refer to the MySQL manual.

5.3 Rolling Back cnDBTier from Non-TLS to TLS Enabled Version (Replication)

This section describes the procedure to roll back cnDBTier clusters from a version where TLS is not enabled for replication to a version where TLS is enabled for replication.

Note:

  • In this procedure, the cnDBTier sites are rolled back twice (in step 1 and step 3). Ensure that you follow this procedure as-is to roll back from a non-TLS version to a TLS enabled version.
  • Downgrade from TLS to Non-TLS is a disruptive procedure that may temporarily impact Geo-replication in the CNE environment. The LoadBalancer service must be recreated for TLS and non-TLS ports to be published. This requires deleting and recreating the db-replication-svc service.
  • The namespace "occne-cndbtier" is given in this procedure is example only. Configure the name according to your environment.
  1. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS mode is set to "NONE".
  2. After rolling back each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.
    For example, run the following command to check the georeplication status of cnDBTier cluster2 configured with other remote sites:
    $ kubectl -n cluster2 exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.cluster2:8080/db-tier/status/replication/realtime
    Sample output:
    [
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster1",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster3",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster4",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  }
]

    In the sample output, the replicationStatus is "UP" for the localSiteName cluster2 for remotesiteName cluster1, cluster3, and cluster4. This indicates that the localSiteName cluster2 is able to replicate data from remotesiteName cluster1, cluster3, and cluster4.

  3. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS mode is set to "VERIFY_CA" or "VERIFY_IDENTITY". This configuration ensures that the clients use an encrypted connection and performs verification against the server CA certificate:
    • VERIFY_CA instructs the client to check if the server’s certificate is valid.
    • VERIFY_IDENTITY instructs the client to check if the server’s certificate is valid and to check if the host name used by the client, matches the identity in the server’s certificate.
  4. After rolling back each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.

    For example, see Step 2.

  5. Run the hooks.sh script with the --add-ssltype flag on all cnDBTier sites to set the type of "occnerepluser" as TLS:

    Note:

    Update the values of the environment variables in the following code block as per your cluster. 
    export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_PASSWORD="<password for the user occneuser>"
export OCCNE_REPL_USER_NAME="occnerepluser"
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
   
occndbtier/files/hooks.sh --add-ssltype

5.4 Rolling Back cnDBTier from TLS enabled to Non-TLS Version (Replication)

This section describes the procedure to roll back cnDBTier clusters from a version where TLS is enabled for replication to a version where TLS is not enabled for replication.

Note:

  • In this procedure, the cnDBTier sites are rolled back twice (in step 2 and step 4). Ensure that you follow this procedure as-is to roll back from a TLS enabled version to a non-TLS version.
  • Downgrade from TLS to Non-TLS is a disruptive procedure that may temporarily impact Geo-replication in the CNE environment. The LoadBalancer service must be recreated for TLS and non-TLS ports to be published. This requires deleting and recreating the db-replication-svc service.
  • The namespace "occne-cndbtier" is given in this procedure is example only. Configure the name according to your environment.
  1. Run the hooks.sh script with the -- remove-ssltype flag on all cnDBTier sites to set the type of "occnerepluser" as Non-TLS:

    Note:

    Update the values of the environment variables in the following code block as per your cluster. 
    export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_PASSWORD="<password for the user occneuser>"
export OCCNE_REPL_USER_NAME="occnerepluser"
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
   
occndbtier/files/hooks.sh --remove-ssltype
  2. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS mode is set to "NONE".
  3. After rolling back each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.
    For example, run the following command to check the georeplication status of cnDBTier cluster2 configured with other remote sites:
    $ kubectl -n cluster2 exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.cluster2:8080/db-tier/status/replication/realtime
    Sample output:
    [
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster1",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster3",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster4",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  }
]

    In the sample output, the replicationStatus is "UP" for the localSiteName cluster2 for remotesiteName cluster1, cluster3, and cluster4. This indicates that the localSiteName cluster2 is able to replicate data from remotesiteName cluster1, cluster3, and cluster4.

  4. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to a version where TLS is disabled.
  5. After rolling back each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.

    For example, see Step 3.

5.5 Rolling Back cnDBTier from TLS enabled to Non-TLS Version (NF Communication)

This section describes the procedure to roll back cnDBTier clusters from a version where TLS is enabled in application SQL pod for NF communication to a version where TLS is not enabled in application SQL pod for NF communication.

Note:

The namespace name "occne-cndbtier" given in this procedure is only an example. Ensure that you configure the namespace name according to your environment.
  1. Perform the following steps to set the NF user to NONE such that the application SQL pods of cnDBTier accepts both TLS and non-TLS communication from the NF that uses the NF user:
    1. Log in to the ndbappmysqld pod. Enter the password when prompted.
      $ kubectl -n  <namespace of cnDBTier Cluster> exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -p
      Example:
      $ kubectl -n cluster1 exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -p
      Sample output:
      Enter Password:
    2. Run the following command to check for NF-specific user account:
      $ mysql> select user, host  from mysql.user;
      Example:
      +------------------+-----------+
| user             | host      |
+------------------+-----------+
| occnerepluser    | %         |
| occneuser        | %         |
| healthchecker    | localhost |
| mysql.infoschema | localhost |
| mysql.session    | localhost |
| mysql.sys        | localhost |
| root             | localhost |
| nfuser           | %         |
+------------------+-----------+
7 rows in set (0.00 sec)
    3. Before altering the user privilege, run the following command to turn off binlogging on one of the ndbappmysqld pods:
      $ mysql>SET sql_log_bin = OFF;
    4. Run the following the commands to alter the user privilege to NONE :
      $ mysql> ALTER USER '<USERNAME>'@'%' REQUIRE NONE;   
$ mysql> FLUSH PRIVILEGES;
    5. Turn on the binlogging on the same ndbappmysqld pod before you exit from the session:
      mysql>SET sql_log_bin = ON;
    6. Exit the session:
      $ mysql> exit;
  2. Make the NF flexible enough to communicate with cnDBTier using TLS. However, when cnDBTier is rolled back to a non-TLS version, the NF must use non-TLS communication.
  3. Perform the Rolling Back cnDBTier procedure on the cnDBTier cluster to roll back cnDBTier to a version where TLS is not enabled in the application SQL pod for NF communication.

5.6 Rolling Back cnDBTier from Non-TLS to TLS Enabled Version (NF Communication)

This section describes the procedure to roll back cnDBTier clusters from a version where TLS is not enabled in application SQL pod for NF communication to a version where TLS is enabled in application SQL pod for NF communication.

Note:

The namespace "occne-cndbtier" is given in this procedure is example only. Configure the name according to your environment.
  1. Perform the Rolling Back cnDBTier procedure on the cnDBTier cluster to roll back cnDBTier to a version where TLS is enabled in the application SQL pod for NF communication.
  2. Connect the NF application to cnDBTier using TLS. For the procedure to connect NF to cnDBTier using TLS, see the respective NF documentation.
  3. At this stage, cnDBTier is rolled back and the application SQL pod is configured with TLS certificates. However, the application SQL pod still allows non-TLS communication. To restrict the communication, you must set the mode of the user to X509 such that any NF using the user follows TLS strictly. When the NF user is created, perform the following steps to set the mode of the NF user to X509.

    Note:

    Before performing this step ensure that the NF is ready to communicate with cnDBTier using TLS.
    1. Log in to the ndbappmysqld pod. Enter the password when prompted.
      $ kubectl -n  <namespace of cnDBTier Cluster> exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -p
      Example:
      $ kubectl -n cluster1 exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -p
      Sample output:
      Enter Password:
    2. Run the following command to check for NF-specific user account. If there are no NF-specific user accounts, create new user accounts:
      $ mysql> select user, host  from mysql.user;
      Example:
      +------------------+-----------+
| user             | host      |
+------------------+-----------+
| occnerepluser    | %         |
| occneuser        | %         |
| healthchecker    | localhost |
| mysql.infoschema | localhost |
| mysql.session    | localhost |
| mysql.sys        | localhost |
| root             | localhost |
| nfuser           | %         |
+------------------+-----------+
7 rows in set (0.00 sec)
    3. Before altering the user and granting privilege, run the following command to turn off binlogging on one of the ndbappmysqld pods:
      $ mysql>SET sql_log_bin = OFF;
    4. Run the following the commands to alter the user and grant the privileges:
      $ mysql> ALTER USER '<USERNAME>'@'%' REQUIRE X509;   
$ mysql> FLUSH PRIVILEGES;
    5. Turn on the binlogging on the same ndbappmysqld pod before you exit from the session:
      mysql>SET sql_log_bin = ON;
    6. Exit the session:
      $ mysql> exit;

5.7 Rolling Back cnDBTier from HTTPS Disabled to HTTPS Enabled Version

This section describes the procedure to roll back cnDBTier clusters from HTTPS disabled to HTTS enabled version.

Procedure to roll back cnDBTier from HTTPS enabled to HTTPS disabled version
  1. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter is enabled and HTTPS are disabled.
  2. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter and HTTPS are enabled.
  3. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter is disabled and HTTPS is enabled.

Note:

This rollback is primarily for removal of the HTTPS port, so that only HTTP connection is allowed in cnDBTier. This step can be performed at any phase of upgrade.

5.8 Rolling Back cnDBTier from HTTPS Enabled to HTTPS Disabled Version

This section describes the procedure to roll back cnDBTier clusters from HTTPS enabled to HTTS disabled version.

Procedure to roll back cnDBTier from HTTPS enabled to HTTPS disabled version
  1. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter and HTTPS are both enabled.
  2. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter is enabled and HTTPS is disabled.
  3. Perform the Rolling Back cnDBTier procedure on all the cnDBTier sites one after the other to roll back cnDBTier to the version where supportDualProtocol parameter and HTTPS are both disabled.

Note:

This rollback is primarily for removal of the HTTPS port so that only HTTP connection is allowed in cnDBTier. This step can be performed at any phase of upgrade.

5.9 Rolling Back cnDBTier

This section describes the procedure to roll back cnDBTier clusters.

Note:

  • The namespace "occne-cndbtier" is given in the documentation only as an indicative purpose. Configure the name according to your environment.
  • If downgrading to 24.3.x with https enabled option, then first downgrade all the sites where /global/https/clientAuthentication was set to WANT. Once all the sites are downgraded, then downgrade the sites to 24.3.x one by one.
  • Rolling back from non-TLS to TLS version or TLS to non-TLS version is a disruptive procedure which may affect the georeplication for some time. Replication may be down for some time.
  1. Log in to the Bastion Host of the site being rolled back.
  2. List the history of the cnDBTier deployment:
    1. Run the following command to list the namespace:
      $ helm -n $OCCNE_NAMESPACE list
      Sample output:
      NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
mysql-cluster   Cluster1        2               2024-02-19 20:11:30.592221615 +0000 UTC deployed        occndbtier-25.1.201       25.1.201
    2. Run the following command to get the list of cnDBTier deployments:
      $ helm -n $OCCNE_NAMESPACE history mysql-cluster
      Sample output:
      REVISION        UPDATED                         STATUS          CHART                   APP VERSION     DESCRIPTION
1                Mon May 20 13:31:35 2024        superseded      occndbtier-25.1.101       25.1.101          Install complete
2                Mon May 20 20:11:30 2024        deployed        occndbtier-25.1.201       25.1.201          Upgrade complete
  3. Before performing the rollback, run the Helm test on all the sites. There are many scenarios where the Helm test can fail, including the health check failures in various components of cnDBTier. If Helm test fails, diagnose and resolve the actual issue and perform the HELM test again on the sites. Proceed further only when the Helm test succeeds.
    $ helm test mysql-cluster --namespace ${OCCNE_NAMESPACE}
    Sample output:
    NAME: mysql-cluster
LAST DEPLOYED:  Mon May 20 10:22:58 2024
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 1
TEST SUITE: mysql-cluster-node-connection-test
Last Started:  Mon May 20 14:15:18 2024
Last Completed:  Mon May 20 14:17:58 2024
Phase: Succeeded
  4. If any of the following parameters were updated during an upgrade, then restart the ndbmtd pods with the --initial flag after the rollback.

    Note:

    Perform this step only if you had updated these parameters while upgrading from 24.2.x to 25.1.2xx and want to rollback to 24.2.x again. This step is automated in release 24.3.x and not need to be performed if you are rolling back to 25.1.2xx or 24.3.x.
    • DataDir
    • FileSystemPath
    • BackupDataDir
    • FragmentLogFileSize
    • InitFragmentLogFiles
    • NoOfFragmentLogFiles
    • NoOfFragmentLogParts
    • FileSystemPathDD
    • FileSystemPathDataFiles
    • FileSystemPathUndoFiles
    • IndexStatSaveSize
    • IndexStatSaveScale
    • IndexStatTriggerPct
    • IndexStatTriggerScale
    • IndexStatUpdateDelay
    • RedoBuffer
    Perform the following steps to instruct the data nodes to restart the ndbmtd pods with the --inital flag after the rollback:

    Note:

    If a setup is deployed with a prefix, then replace the pod name of ndbmtd accordingly.
    1. The value of N has to be one less than the number of ndbmtd pods deployed:
      $ for x in {0..N}; do kubectl -n <cndbtier_namespace>  exec -it <NDBMTD_STS_NAME_WITH_PREFIX>-$x -c db-backup-executor-svc -- bash -c  "touch /var/occnedb/useinitial && chmod 777 /var/occnedb/useinitial";  done;
      For example, if a setup has four ndbmtd pods, use the following command:
      $ for x in {0..3}; do kubectl -n occne-cndbtier  exec -it ndbmtd-$x -c db-backup-executor-svc -- bash -c  "touch /var/occnedb/useinitial && chmod 777 /var/occnedb/useinitial"; done;
      $ for x in {0..3}; do kubectl -n occne-cndbtier  exec -it ndbmtd-$x -c db-backup-executor-svc -- bash -c  "echo ndbmtd-$x;ls -lrth  /var/occnedb/ | grep -i useinitial "; done;
    2. Verify if the useiinitial file is created for each data node in the /var/occnedb directory:
      $  for x in {0..N}; do kubectl -n <cndbtier_namespace>  exec -it <NDBMTD_STS_NAME_WITH_PREFIX>-$x -c db-backup-executor-svc -- bash -c  "echo ndbmtd-$x;ls -lrth  /var/occnedb/ | grep -i useinitial "; done;
      For example:
      $ for x in {0..3}; do kubectl -n occne-cndbtier  exec -it ndbmtd-$x -c db-backup-executor-svc -- bash -c  "echo ndbmtd-$x;ls -lrth  /var/occnedb/ | grep -i useinitial "; done;
      Sample output:
      ndbmtd-0
-rwxrwxrwx. 1 mysql mysql    0 Sep 23 11:56 useinitial
ndbmtd-1
-rwxrwxrwx. 1 mysql mysql    0 Sep 23 11:56 useinitial
  5. If there is no upgrade service account configured for cnDBTier or if you are rolling back to a previous cnDBTier (25.1.2xx to 24.2.x (leapfrog version)), run the pre-rollback-with-tde-to-non-tde script:
    export OCCNE_NAMESPACE="occne-cndbtier"
export NDB_MTD_PODS=$(kubectl -n ${OCCNE_NAMESPACE} get pods | grep 'ndbmtd' | awk '{print $1}')
export NDB_MGMD_PODS=$(kubectl -n ${OCCNE_NAMESPACE} get pods | grep 'ndbmgmd' | awk '{print $1}')
export NDB_STS_NAME=$(kubectl -n ${OCCNE_NAMESPACE} get sts | grep 'ndbmtd' | awk '{print $1}')
 
occndbtier/files/hooks.sh --pre-rollback-with-tde-to-non-tde
  6. If there is no upgrade service account configured for cnDBTier, then run the pre-rollback script:
    export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_ACCESS_KEY="<password for the user occneuser>"
export DBTIER_REPLICATION_SVC_DATABASE="replication_info"
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0/ndbappmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
export NDB_MGMD_PODS="ndbmgmd-0 ndbmgmd-1"
  
# To enable or disable password encryption , uncomment the below line and set the variable to true or false to enable or disable it.
# export ENABLE_ENCRYPTION="<true/false>"
 
occndbtier/files/hooks.sh --pre-rollback
  7. Run the following command to roll back to the previous version:
    helm --namespace ${OCCNE_NAMESPACE} rollback $CNDBTIER_RELEASE_NAME $REVISION
    Where,
    • $CNDBTIER_RELEASE_NAME is the name of the cnDBTier deployment obtained from Step 3. For example, CNDBTIER_RELEASE_NAME="mysql-cluster"
    • $REVISION is the revision number of the release that you want to rollback to, obtained from Step 3. For example, REVISION=1.
  8. If there is no upgrade service account configured for cnDBTier, run the postrollback script. This script restarts the MGM pods and patches upgradeStrategy.

    Note:

    • Replace the values of the environment variables in the following sample with the values of your cluster.
    • If prefix is used in pod names, then add the prefix to the following environment variable for both the pod and statefulset names. For example:
      export NDB_MGMD_PODS="<global.k8sResource.pod.prefix>-ndbmgmd-0 <global.k8sResource.pod.prefix>-ndbmgmd-1"
export NDB_STS_NAME="<global.k8sResource.pod.prefix>-ndbmtd"
    1. Define the following variables before running the postrollback script:
      export OCCNE_NAMESPACE="occne-cndbtier"
export API_EMP_TRY_SLOTS_NODE_IDS="id=222\|id=223\|id=224\|id=225"
#If values.global.ndbapp.ndb_cluster_connection_pool is greater than 1, then declare the following environment variable
#export APP_CON_POOL_INGORE_NODE_IDS="id=100"
export APP_CON_POOL_INGORE_NODE_IDS="id=100\|id=101\|id=102\|id=103"
export NDB_MGMD_PODS="ndbmgmd-0 ndbmgmd-1"
export NDB_MTD_PODS="ndbmtd-0 ndbmtd-1"
export NDB_STS_NAME="ndbmtd"
export API_STS_NAME="ndbmysqld"
export APP_STS_NAME="ndbappmysqld"
export NDBAPP_START_NODE_ID="70"
export NDBAPP_REPLICA_MAX_COUNT="4"
    2. Run the postrollback script:
      occndbtier/files/hooks.sh --post-rollback
  9. Wait for the appropriate nodes to restart and reconnect to the cluster and then run the following command to verify the cluster status from the management pod. The pods that is restarted depends on the pods that are rolled back.
    $ kubectl -n ${OCCNE_NAMESPACE} exec -it ndbmgmd-0 -- ndb_mgm -e show
    Sample output:
    Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.104.176  (mysql-8.4.3 ndb-8.4.3, Nodegroup: 0)
id=2    @10.233.121.175  (mysql-8.4.3 ndb-8.4.3, Nodegroup: 0, *)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.101.154  (mysql-8.4.3 ndb-8.4.3)
id=50   @10.233.104.174  (mysql-8.4.3 ndb-8.4.3)
 
[mysqld(API)]   8 node(s)
id=56   @10.233.92.169  (mysql-8.4.3 ndb-8.4.3)
id=57   @10.233.101.155  (mysql-8.4.3 ndb-8.4.3)
id=70   @10.233.92.170  (mysql-8.4.3 ndb-8.4.3)
id=71   @10.233.121.176  (mysql-8.4.3 ndb-8.4.3)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)
  10. After rollback, run the Helm test on all the sites and verify if the cnDBTier services are running successfully:
    $ helm test mysql-cluster --namespace ${OCCNE_NAMESPACE}
    Sample output:
    NAME: mysql-cluster
LAST DEPLOYED:  Mon May 20 10:22:58 2024
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 1
TEST SUITE: mysql-cluster-node-connection-test
Last Started:  Mon May 20 14:15:18 2024
Last Completed:  Mon May 20 14:17:58 2024
Phase: Succeeded