4 Upgrading cnDBTier

This chapter describes the procedure to upgrade an existing cnDBTier deployment.

Note:

  • The OCCNE_NAMESPACE variable in the upgrade procedures must be set to the cnDBTier namespace. Before running any command that contains the OCCNE_NAMESPACEvariable, ensure that you have set this variable to the cnDBTier namespace as stated in the following code block:
    export OCCNE_NAMESPACE=<namespace>

    where, <namespace> is the cnDBTier namespace.

  • The namespace name "occne-cndbtier" given in the upgrade procedures is only an example. Ensure that you configure the namespace name according to your environment.
  • cnDBTier 24.3.1 supports Helm 3.12.3 and 3.13.2. Ensure that you upgrade Helm to a supported version.

4.1 Supported Upgrade Paths

The following table provides the upgrade paths that are supported by cnDBTier Release 24.3.1.

Table 4-1 Supported Upgrade Paths

Source Release Target Release
24.3.x 24.3.1
24.2.x 24.3.1
24.1.x 24.3.1

4.2 Upgrading cnDBTier Clusters

This section describes the procedure to upgrade the cnDBTier clusters.

Note:

  • If your cnDBTier is configured with a single replication channel, then perform upgrade using a single replication channel group. If your cnDBTier is configured with multiple replication channel groups, then perform the upgrade using multiple replication channel groups.
  • As of 24.3.1, the upgrade service account requires persistentvolumeclaims in its rules.resources. This rule is necessary for the postrollback hook to delete mysqld PVCs when rolling back to an earlier MySQL release.
  • The recommended value of HeartbeatIntervalDbDb is 1250. Check the value of HeartbeatIntervalDbDb in the running cnDBtier instance. If the value is not set to 1250, then update the value to 1250 step by step.
    For example:
    • If the existing value is set to 5000, then perform the following steps to update the value to 1250:
      1. Modify the value of HeartbeatIntervalDbDb to 2500 in the custom values file located in the /global/additionalndbconfigurations/ndb/ directory.
      2. Perform an upgrade by following the procedure given in this section.
      3. When the upgrade completes successfully, modify the value of HeartbeatIntervalDbDb to 1250 in the custom values file located in the /global/additionalndbconfigurations/ndb/ directory.
      4. Perform a cnDBTier upgrade by following the procedure given in this section.
    • If the existing value is set to 500, then perform the following steps to update the value to 1250:
      1. Modify the value of HeartbeatIntervalDbDb to 900 in the custom values file located in the /global/additionalndbconfigurations/ndb/ directory.
      2. Perform an upgrade by following the procedure given in this section.
      3. When the upgrade completes successfully, modify the value of HeartbeatIntervalDbDb to 1250 in the custom values file located in the /global/additionalndbconfigurations/ndb/ directory.
      4. Perform a cnDBTier upgrade by following the procedure given in this section.
  • db-backup-manager-svc is designed to automatically restart in case of errors. Therefore, when the backup-manager-svc encounters a temporary error during the upgrade process, it may undergo several restarts. When cnDBTier reaches a stable state, the db-backup-manager-svc pod operates normally without any further restarts.
  • If you want to enable secure transfer of backups to remote server, then:
    • configure the remote server or storage with SFTP.
    • provide the path where the files must be stored and necessary permissions for cnDBTier to copy the backups on the remote server.
    • configure Private and Public Key to access remote server where SFTP is installed.
    • provide the details configured in the previous points, except Public Key, to cnDBTier either during a fresh install or upgrade so that cnDBTier can store the backups remotely.
    • note that the password used for encryption of the backups isn't stored in the remote server if backup encryption is enabled.

      For more information about the configuration parameters, see Global Parameters.

  • If you have enabled secure transfer of backups to remote server, then make note of the following:
    • cnDBTier doesn't purge the backups that are pushed to remote server. Therefore, when necessary, make sure you manually purge the old backups in remote server.
    • cnDBTier doesn't transfer any existing backups taken using the old cnDBTier version to a remote server or storage.
    • cnDBTier supports secure transfer of backups to only one remote server.
  • If you are upgrading from a release older than 24.3.x, perform the following steps:
    1. Deactivate the network policy feature from the custom_values.yaml file before performing an upgrade:
      global:
  networkpolicy:
    enabled: false
    2. [Optional]: Once you successfully upgrade to 24.3.x, you can reenable network policy in the custom_values.yaml file by following the upgrade procedure again:
      
global:
  networkpolicy:
    enabled: true
  • If you are modifying the TLS certificates while upgrading a cnDBTier cluster from a TLS enabled version to another TLS enabled version, then before proceeding with the upgrade procedure, follow the "Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites" procedure in Oracle Communications Cloud Native Core, cnDBTier User Guide to add new certificates and retain the existing certificates.
  • Upgrade only one georedundant cnDBTier site at a time. Wait for the upgrade to complete before you upgrade the next georedundant cnDBTier site.
  • Starting 24.3.x, you can encrypt the files in the data nodes using TDE. This feature can be enabled by setting EncryptedFileSystem to 1. Therefore, if you are upgrading from a previous version to 24.3.x or from a TDE-disabled version to a 24.3.x TDE-enabled version, ensure that you create the "occne-tde-encrypted-filesystem-secret" secret first. For more information about creating TDE secret, see Creating Secrets.
  • If password encryption is enabled in the version you are upgrading from, then before upgrading, disable the password encryption by performing the Disabling Password Encryption procedure.
  • The database monitor service exposes metrics using the /prometheus endpoint. If Prometheus Operator is installed in the Kubernetes cluster and there is no existing ServiceMonitor for the /prometheus endpoint, then run the following command to create a ServiceMonitor (Prom Operator CRD) with cnDBTier configuration. This enables Prometheus Operator to fetch metrics from the /prometheus REST endpoint for db-monitor-svc.
    $ kubectl apply -f namespace/occndbtier_service_monitor_${OCCNE_VERSION}.yaml -n ${OCCNE_INFRA_NAMESPACE}
    where, ${OCCNE_INFRA_NAMESPACE} is the namespace where Prometheus Operator is installed.
  • Ensure that you don't change the PVC value during an upgrade. To modify the PVC size according to the dimensioning sheet, follow the vertical scaling procedures provided in Oracle Communications Cloud Native Core, cnDBTier User Guide.

Assumption

  • All NDB pods of the cnDBTier cluster are up and running.
  • Helm limits some parameters, such as PVC size, to be changed during the upgrade. Therefore, these parameters must not be changed.
  • The start node ID must be the same as the existing start node ID. Get the start node ID from the existing cluster using the following command:

    As per the following example, the start node ID must be 49 for management, 56 for georeplication SQL, and 70 for nongeoreplication SQL pods.

    Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.94.13  (mysql-8.4.2 ndb-8.4.2, Nodegroup: 0, *)
id=2    @10.233.124.12  (mysql-8.4.2 ndb-8.4.2, Nodegroup: 0)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.124.11  (mysql-8.4.2 ndb-8.4.2)
id=50   @10.233.93.14  (mysql-8.4.2 ndb-8.4.2)
 
[mysqld(API)]   8 node(s)
id=56   @10.233.123.15  (mysql-8.4.2 ndb-8.4.2)
id=57   @10.233.94.14  (mysql-8.4.2 ndb-8.4.2)
id=70   @10.233.120.20  (mysql-8.4.2 ndb-8.4.2)
id=71   @10.233.95.22  (mysql-8.4.2 ndb-8.4.2)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

    Note:

    Node IDs 222 to 225 in the sample output are shown as "not connected" as they are added as empty slot IDs used for georeplication recovery.

Procedure

To upgrade the cnDBTier clusters, perform the following:

  1. Download the latest cnDBTier package for upgrade.

    For more information about downloading the procedure, see Downloading cnDBTier Package.

  2. Create the SSH keys and the secrets for these keys. For creating the keys and secrets, see Creating SSH Keys and Creating Secrets.
  3. Before performing the upgrade, set the https mode and DB encryption to false in the custom_values.yaml file to disable the https mode and DB encryption, as follows:
    https:
  enable: false
   
encryption:
  enable: false
  4. Before performing the upgrade, run the Helm test on the current cnDBTier at all sites, only if the Helm test is success on all sites proceed with the upgrade. 
    $ helm test  mysql-cluster --namespace ${OCCNE_NAMESPACE}
    Sample output:
    NAME: mysql-cluster
LAST DEPLOYED:  Mon May 20 10:22:58 2024
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 1
TEST SUITE:     mysql-cluster-node-connection-test
Last Started:    Mon May 20 14:15:18 2024
Last Completed:  Mon May 20 14:17:58 2024
Phase:          Succeeded
  5. If you are performing an upgrade with fixed LoadBalancer IP for external services, then find the IP addresses of the current cnDBTier cluster by running the following command:
    $ kubectl get svc -n ${OCCNE_NAMESPACE}
  6. Configure the LoadBalancer IP addresses that you obtained in the previous step in the custom_values.yaml file by following the cnDBTier configurations provided in the Customizing cnDBTier section.
  7. If you want to enable backup encryption, perform the following:
    1. Perform Step 5 of Creating Secrets.
    2. Set the "/global/backupencryption/enable" parameter in the custom_values.yaml file to true.
  8. If you want to enable password encryption, create the following secret which is used to configure the key that is used to encrypt the replication username and password. For more information about enabling password encryption, see ""/global/encryption/enable"" configuration in the values.yaml file described in Global Parameters.

    Note:

    cnDBTier supports using same encryption key across all the clusters, so if you are creating a new cluster, ensure that the new cluster uses the same encryption key. 
    $ kubectl -n ${OCCNE_NAMESPACE} create secret generic cndbtier-mysql-encrypt-key --from-literal="dbencryptkey=<encryption-key>"
    For example:
    $ kubectl -n ${OCCNE_NAMESPACE} create secret generic cndbtier-mysql-encrypt-key --from-literal="dbencryptkey=NextGenCne"
  9. If Kubernetes version is above or equal to 1.25 and Kyverno is supported or installed on Kubernetes, then run the following commands as applicable. Otherwise, skip this step.
    • If ASM or Istio is installed or running on Kubernetes, then run the following command:
      $ kubectl apply -f namespace/occndbtier_kyvernopolicy_asm_${OCCNE_VERSION}.yaml -n ${OCCNE_NAMESPACE}
    • If ASM or Istio is not installed or running on Kubernetes, then run the following command:
      $ kubectl apply -f namespace/occndbtier_kyvernopolicy_nonasm_${OCCNE_VERSION}.yaml -n ${OCCNE_NAMESPACE}
  10. If you want to enable secure transfer of backups to remote server, then perform the following steps:
    1. Configure the following global parameters in the custom_values.yaml to enable secure transfer of backups to remote server:
      • "/global/remotetransfer/enable"
      • "/global/remotetransfer/faultrecoverybackuptransfer"
      • "/global/remotetransfer/remoteserverip"
      • "/global/remotetransfer/remoteserverport"
      • "/global/remotetransfer/remoteserverpath"
    2. Create the remote server user name and private key secrets by following step 6 of the Creating Secrets procedure.

    Note:

    Automated cnDBTier upgrade needs a service account for pod rolling restart and patch. If you want to perform an automated cnDBTier upgrade with a service account, then follow the steps given in the Upgrading cnDBTier Clusters with an Upgrade Service Account section. If you want to upgrade cnDBTier manually without using a service account, then follow the steps given in the Upgrading cnDBTier Clusters without an Upgrade Service Account section.

Upgrading cnDBTier Clusters with an Upgrade Service Account

  1. Create an upgrade service account if you don't have one already. Skip this step if you have a service account with the right role to use for an upgrade. You can check the details of your service account and role in the namespace/rbac.yaml file:
    1. Run the following Helm command and note the RELEASE_NAME that is displayed under the NAME column:
      helm -n ${OCCNE_NAMESPACE} list
    2. Set the ${OCCNE_RELEASE_NAME} environment variable with the Helm value of RELEASE_NAME that you obtained from Step a.
      export OCCNE_RELEASE_NAME="mysql-cluster"
    3. Update the service account, role, and rolebinding for upgrade in the namespace/occndbtier_rbac_${OCCNE_VERSION}.yaml file:

      Note:

      You can find the namespace directory at either /Artifacts/Scripts/ or /Scripts/ relative path depending on the CSAR package type. 
      sed -i "s/occne-cndbtier/${OCCNE_NAMESPACE}/" namespace/occndbtier_rbac_${OCCNE_VERSION}.yaml
sed -i "s/mysql-cluster/${OCCNE_RELEASE_NAME}/" namespace/occndbtier_rbac_${OCCNE_VERSION}.yaml
sed -i "s/cndbtier-upgrade/${OCCNE_RELEASE_NAME}-upgrade/" namespace/occndbtier_rbac_${OCCNE_VERSION}.yaml
sed -i "s/rolebinding/role/" namespace/occndbtier_rbac_${OCCNE_VERSION}.yaml
    4. Create upgrade service account, upgrade role, and upgrade rolebinding:
      kubectl -n ${OCCNE_NAMESPACE} apply -f namespace/occndbtier_rbac_${OCCNE_VERSION}.yaml
  2. Configure the upgrade service account in your custom_values.yaml file by following one of the following options:
    1. If you have created the upgrade service account as mentioned in Step 1, run the following commands to configure the account:
      1. Run the following command and note the RELEASE_NAME mentioned in the NAME column:
        helm -n ${OCCNE_NAMESPACE} list
      2. Set the ${OCCNE_RELEASE_NAME} environment variable with the Helm value of RELEASE_NAME that you obtained in Step i:
        export OCCNE_RELEASE_NAME="mysql-cluster"
      3. Run the following command to navigate to the cluster directory:
        cd /var/occne/cluster/${OCCNE_CLUSTER}
      4. Update the service account information in your custom_values.yaml file:
        sed -i "/  serviceAccountForUpgrade:/,/^$/ { /name:/ s/cndbtier-upgrade-serviceaccount/${OCCNE_RELEASE_NAME}-upgrade-serviceaccount/ }" occndbtier/custom_values.yaml
    2. If you have a previously created service account, edit your custom_values.yaml file, and set global.serviceAccountForUpgrade.create to false and global.serviceAccountForUpgrade.name to the name of your service account.
    3. If you are upgrading to cnDBTier 24.3.x, freshly installing 24.2.x, freshly installing 24.1.x, or upgraded to 24.2.x from a fresh install of 24.1.x, then you must already have a service account for upgrade. You can keep the custom_values.yaml file with the same values used for the previous installation or upgrade of cnDBTier.
  3. Upgrade the cnDBTier by running the commands below:
    1. Run the following command and make a note of the release name from the NAME column in the output:
      helm -n ${OCCNE_NAMESPACE} list
export OCCNE_RELEASE_NAME="mysql-cluster"
    2. Navigate to the cluster directory:
      cd /var/occne/cluster/${OCCNE_CLUSTER}
    3. Run the following command to start the upgrade:

      Note:

      Replace the ${OCCNE_RELEASE_NAME} environment variable in the command with the Helm value of RELEASE_NAME that you obtained in Step a. 
      helm -n ${OCCNE_NAMESPACE} upgrade ${OCCNE_RELEASE_NAME} occndbtier -f occndbtier/custom_values.yaml
  4. Wait for all the MGM and NDB pods to restart.
  5. Run the following command to perform a rollout restart of the NDB pods. This restart of the NDB pods is required for the updated HeartbeatDbDb order to take effect. This step is required only if you are upgrading from a release that doesn't have this feature:
    kubectl -n $DBTIER_NAMESPACE rollout restart statefulset ndbmtd
  6. Wait for all the NDB pods to restart.
  7. Verify the cluster status from the management pod by running the following command:
    $ kubectl -n ${OCCNE_NAMESPACE} exec -it ndbmgmd-0 -- ndb_mgm -e show
    Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.104.176  (mysql-8.4.2 ndb-8.4.2, Nodegroup: 0)
id=2    @10.233.121.175  (mysql-8.4.2 ndb-8.4.2, Nodegroup: 0, *)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.101.154  (mysql-8.4.2 ndb-8.4.2)
id=50   @10.233.104.174  (mysql-8.4.2 ndb-8.4.2)
 
[mysqld(API)]   8 node(s)
id=56   @10.233.92.169  (mysql-8.4.2 ndb-8.4.2)
id=57   @10.233.101.155  (mysql-8.4.2 ndb-8.4.2)
id=70   @10.233.92.170  (mysql-8.4.2 ndb-8.4.2)
id=71   @10.233.121.176  (mysql-8.4.2 ndb-8.4.2)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

    Note:

    Node IDs 222 to 225 in the sample output are shown as "not connected" as these are added as empty slot IDs that are used for georeplication recovery.
  8. Run the following Helm test command to verify if the cnDBTier services are upgraded successfully:
    $ helm test  mysql-cluster --namespace ${OCCNE_NAMESPACE}
    Sample output:
    NAME: mysql-cluster
LAST DEPLOYED:  Mon May 20 10:22:58 2024
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 1
TEST SUITE:     mysql-cluster-node-connection-test
Last Started:    Mon May 20 14:15:18 2024
Last Completed:  Mon May 20 14:17:58 2024
Phase:          Succeeded

Upgrading cnDBTier Clusters Without an Upgrade Service Account

  1. Configure the custom_values.yaml file to disable upgrade service account:
    cd /var/occne/cluster/${OCCNE_CLUSTER}
 
# Update the service account information in your custom_values.yaml file
sed -i "/  serviceAccountForUpgrade:/,/^$/ { /create:/ s/true/false/; /name:/ s/cndbtier-upgrade-serviceaccount// }" occndbtier/custom_values.yaml

    Alternatively, edit the custom_values.yaml file, and manually set global.serviceAccountForUpgrade.create to false and global.serviceAccountForUpgrade.name to "" (empty).

  2. If you are upgrading from a previous cnDBTier release or if you want to enable or disable password encryption, perform the following steps to apply the schema changes and run the preupgrade script:
    1. Run the following command on the Bastion Host to apply the schema changes.

      Note:

      • Replace the values of the environment variables in the following commands with the values corresponding to your cluster.
      • Enabling or disabling encryption may cause a brief disruption to the replication between sites if a switchover happens between step 2 and step 4. Therefore, perform steps 3 and 4 immediately after performing step 2.
      export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_PASSWORD="<password for the user occneuser>"
export DBTIER_REPLICATION_SVC_DATABASE="replication_info"
export DBTIER_BACKUP_SVC_DATABASE="backup_info"
export DBTIER_HBREPLICAGROUP_DATABASE="hbreplica_info"
export REPLCHANNEL_GROUP_COUNT=<configured number of replication channel groups i.e either 1/2/3>
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0/ndbappmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
export APP_STS_NAME="ndbappmysqld"

#To enable or disable password encryption , uncomment the following line and set the variable to true or false to enable or disable the feature.
# export ENABLE_ENCRYPTION="<true/false>" 
  
occndbtier/files/hooks.sh --schema-upgrade
    2. Run the following commands on Bastion Host to run the preupgrade procedure.

      Note:

      • Replace the values of the environment variables in the following commands with the values corresponding to your cluster.
      • If prefix is used in pod names, then add the prefix to the following environment variable for both the pod and statefulset names. For example:
        export NDB_MGMD_PODS="<global.k8sResource.pod.prefix>-ndbmgmd-0 <global.k8sResource.pod.prefix>-ndbmgmd-1"
export APP_STS_NAME="<global.k8sResource.pod.prefix>-ndbappmysqld"
      export OCCNE_NAMESPACE="occne-cndbtier"
export NDB_MGMD_PODS="ndbmgmd-0 ndbmgmd-1"
export APP_STS_NAME="ndbappmysqld"

occndbtier/files/hooks.sh --pre-upgrade
  3. Run the following commands to upgrade the cnDBTier:
    1. Run the following command to obtain the Helm value of RELEASE_NAME.
      helm -n ${OCCNE_NAMESPACE} list
export OCCNE_RELEASE_NAME="mysql-cluster"

      The RELEASE_NAME can be found under the NAME column in the output.

    2. Run the following command to upgrade cnDBTier:
      cd /var/occne/cluster/${OCCNE_CLUSTER}
 
helm -n ${OCCNE_NAMESPACE} upgrade ${OCCNE_RELEASE_NAME} occndbtier -f occndbtier/custom_values.yaml --no-hooks

      where, ${OCCNE_RELEASE_NAME} is the Helm value of RELEASE_NAME obtained in Step a.

  4. Perform the following steps to run the postupgrade script. This deletes all MGM pods, waits for them to come up, and patches upgradeStrategy.

    Note:

    • Replace the values of the environment variables in the following commands with the values corresponding to your cluster.
    • If prefix is used in pod names, then add the prefix to the following environment variable for both the pod and statefulset names. For example:
      export NDB_MGMD_PODS="<global.k8sResource.pod.prefix>-ndbmgmd-0 <global.k8sResource.pod.prefix>-ndbmgmd-1"
export APP_STS_NAME="<global.k8sResource.pod.prefix>-ndbappmysqld"
    1. Define the following environment variables:
      export OCCNE_NAMESPACE="occne-cndbtier"
#export API_EMP_TRY_SLOTS_NODE_IDS="id=222"
export API_EMP_TRY_SLOTS_NODE_IDS="id=222\|id=223\|id=224\|id=225"
export MGM_NODE_IDS="id=49\|id=50"
    2. Export all the ndbmtd node IDs in the following environment variable:
      export NDB_NODE_IDS="id=1\|id=2"
    3. Export all the ndbmysqld node IDs in the following environment variables:

      Note:

      ndbmysqld node IDs starts at global.api.startNodeId and ends at (global.api.startNodeId + global.apiReplicaCount - 1) 
      export API_NODE_IDS="id=56\|id=57"
export NDB_MGMD_PODS="ndbmgmd-0 ndbmgmd-1"
export NDB_MTD_PODS="ndbmtd-0 ndbmtd-1" 
export NDB_STS_NAME="ndbmtd"
export API_STS_NAME="ndbmysqld"
export APP_STS_NAME="ndbappmysqld"
    4. If auto scaling is enabled for ndbapp sts, then additionally define the following environment variables:
      #export NDBAPP_START_NODE_ID="<as configured in values.yaml: global.ndbapp.startNodeId>"
#export NDBAPP_REPLICA_MAX_COUNT="<as configured in values.yaml: global.ndbappReplicaMaxCount>"
    5. If values.global.ndbapp.ndb_cluster_connection_pool is greater than one, then declare the following environment variable:
      export APP_CON_POOL_INGORE_NODE_IDS="id=100\|id=101\|id=102 ... \|id=(n-1)\|id=n"

      where, n is calculated using the following formula: n = 100 + (((values.global.ndbapp.ndb_cluster_connection_pool - 1) * values.global.ndbappReplicaMaxCount) - 1).

    6. Run the postupgrade script:
      occndbtier/files/hooks.sh --post-upgrade
  5. Wait for all the MGM and NDB pods to restart.
  6. Run the following command to perform a rollout restart of the NDB pods. This restart of the NDB pods is required for the updated HeartbeatDbDb order to take effect. This step is required only if you are upgrading from a release that doesn't have this feature:
    kubectl -n $DBTIER_NAMESPACE rollout restart statefulset ndbmtd
  7. Wait for all the NDB pods to restart.
  8. Run the following command to verify the cluster status from the management pod:
    $ kubectl -n ${OCCNE_NAMESPACE} exec -it ndbmgmd-0 -- ndb_mgm -e show
    Sample output:
    Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.104.176  (mysql-8.4.2 ndb-8.4.2, Nodegroup: 0)
id=2    @10.233.121.175  (mysql-8.4.2 ndb-8.4.2, Nodegroup: 0, *)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.101.154  (mysql-8.4.2 ndb-8.4.2)
id=50   @10.233.104.174  (mysql-8.4.2 ndb-8.4.2)
 
[mysqld(API)]   8 node(s)
id=56   @10.233.92.169  (mysql-8.4.2 ndb-8.4.2)
id=57   @10.233.101.155  (mysql-8.4.2 ndb-8.4.2)
id=70   @10.233.92.170  (mysql-8.4.2 ndb-8.4.2)
id=71   @10.233.121.176  (mysql-8.4.2 ndb-8.4.2)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

    Note:

    Node IDs 222 to 225 in the sample output are shown as "not connected" as these are added as empty slot IDs that are used for georeplication recovery.
  9. Run the following Helm test command to verify if the cnDBTier services are upgraded successfully:
    $ helm test  mysql-cluster --namespace ${OCCNE_NAMESPACE}
    Sample output:
    $ helm test  mysql-cluster --namespace ${OCCNE_NAMESPACE}
NAME: mysql-cluster
LAST DEPLOYED:  Mon May 20 10:22:58 2024
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 1
TEST SUITE:     mysql-cluster-node-connection-test 
Last Started:    Mon May 20 14:15:18 2024
Last Completed:  Mon May 20 14:17:58 2024
Phase:          Succeeded
  10. After successful upgrade, update the alerts and Grafana dashboards using Postinstallation Tasks.

4.3 Upgrading Cluster from Non-TLS Enabled to TLS Enabled Version

This section describes the procedure to upgrade cnDBTier clusters from non-TLS version to a TLS enabled version.

Note:

In this procedure, the cnDBTier sites are upgraded twice (in step 4 and step 7). Ensure that you follow this procedure as-is to upgrade from a non-TLS version to a TLS enabled version.
  1. Create the necessary secrets in all the cnDBTier sites by following step 7 of the Creating Secrets procedure.
  2. Ensure that TLS is enabled in the custom_values.yaml file for all cnDBTier sites:
    global:
  tls:    
    enable: true
  3. Provide all the necessary certificates such as ca certificate, client certificate, and server certificate for the respective ndbmysqld pods in custom_values.yaml file for all cnDBTier sites:

    Note:

    Set the TLS mode to NONE for all the cnDBTier sites as seen in the following custom_values.yaml file. 
    tls:
  enable: true
  caCertificate: "<ca certificate file name>"
  tlsversion: "TLSv1.3"
  tlsMode: "NONE"
  ciphers:
    - TLS_AES_128_GCM_SHA256
    - TLS_AES_256_GCM_SHA384
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_AES_128_CCM_SHA256
  certificates:
    - name: ndbmysqld-0
      serverCertificate: "<server certificate name>"
      serverCertificateKey: "<server key name>"     
      clientCertificate: "<client certificate name>"
      clientCertificateKey: "<client key name>"
    - name: ndbmysqld-1
      serverCertificate: "<server certificate name>"
      serverCertificateKey: "<server key name>"     
      clientCertificate: "<client certificate name>"
      clientCertificateKey: "<client key name>"
    ...
    For example:
    tls:
  enable: true
  caCertificate: "combine-ca.pem"
  tlsversion: "TLSv1.3"
  tlsMode: "NONE"
  ciphers:
    - TLS_AES_128_GCM_SHA256
    - TLS_AES_256_GCM_SHA384
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_AES_128_CCM_SHA256
  certificates:
    - name: ndbmysqld-0
      serverCertificate: "server1-cert.pem"
      serverCertificateKey: "server1-key.pem"     
      clientCertificate: "client1-cert.pem"
      clientCertificateKey: "client1-key.pem"
    - name: ndbmysqld-1
      serverCertificate: "server1-cert.pem"
      serverCertificateKey: "server1-key.pem"     
      clientCertificate: "client1-cert.pem"
      clientCertificateKey: "client1-key.pem"
    ...
  4. Perform the Upgrading cnDBTier Clusters procedure to upgrade all the cnDBTier sites one after the other using the custom_values.yaml file that you updated in the previous steps.
  5. After upgrading each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.
    For example, run the following command to check the georeplication status of cnDBTier cluster2 configured with other remote sites:
    $ kubectl -n cluster2 exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.cluster2:8080/db-tier/status/replication/realtime
    Sample output:
    [
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster1",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster3",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster4",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  }
]

    In the sample output, the replicationStatus is "UP" for the localSiteName cluster2 for remotesiteName cluster1, cluster3, and cluster4. This indicates that the localSiteName cluster2 is able to replicate data from remotesiteName cluster1, cluster3, and cluster4.

  6. In the custom_values.yaml file, set tlsMode to VERIFY_CA or VERIFY_IDENTITY, depending on your requirement, for all the cnDBTier sites. This configuration ensures that the clients use an encrypted connection and performs verification against the server CA certificate:
    • VERIFY_CA instructs the client to check if the server’s certificate is valid.
    • VERIFY_IDENTITY instructs the client to check if the server’s certificate is valid and to check if the host name used by the client, matches the identity in the server’s certificate. 
    tls:
  enable: true
  caCertificate: "<ca certificate file name>"
  tlsversion: "TLSv1.3"
  tlsMode: "VERIFY_CA/VERIFY_IDENTITY"
  ciphers:
    - TLS_AES_128_GCM_SHA256
    - TLS_AES_256_GCM_SHA384
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_AES_128_CCM_SHA256
  certificates:
    - name: ndbmysqld-0
      serverCertificate: "<server certificate name>"
      serverCertificateKey: "<server key name>"     
      clientCertificate: "<client certificate name>"
      clientCertificateKey: "<client key name>"
    - name: ndbmysqld-1
      serverCertificate: "<server certificate name>"
      serverCertificateKey: "<server key name>"     
      clientCertificate: "<client certificate name>"
      clientCertificateKey: "<client key name>"
    ...
    For example:
    tls:
  enable: true
  caCertificate: "combine-ca.pem"
  tlsversion: "TLSv1.3"
  tlsMode: "VERIFY_CA"
  ciphers:
    - TLS_AES_128_GCM_SHA256
    - TLS_AES_256_GCM_SHA384
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_AES_128_CCM_SHA256
  certificates:
    - name: ndbmysqld-0
      serverCertificate: "server1-cert.pem"
      serverCertificateKey: "server1-key.pem"     
      clientCertificate: "client1-cert.pem"
      clientCertificateKey: "client1-key.pem"
    - name: ndbmysqld-1
      serverCertificate: "server1-cert.pem"
      serverCertificateKey: "server1-key.pem"     
      clientCertificate: "client1-cert.pem"
      clientCertificateKey: "client1-key.pem"
    ...
  7. Perform the Upgrading cnDBTier Clusters procedure to upgrade all the cnDBTier sites one after the other using the custom_values.yaml file that you updated in the previous step.
  8. After upgrading each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.

    For example, see Step 7.

  9. Run the hooks.sh script with the --add-ssltype flag on all cnDBTier sites to set the type of "occnerepluser" as TLS:

    Note:

    Update the values of the environment variables in the following code block as per your cluster. 
    export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_PASSWORD="<password for the user occneuser>"
export OCCNE_REPL_USER_NAME="occnerepluser"
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
  
occndbtier/files/hooks.sh --add-ssltype

4.4 Upgrading Cluster from TLS Enabled to Non-TLS Enabled Version

This section describes the procedure to upgrade cnDBTier clusters from TLS enabled version to an non-TLS version.

Note:

In this procedure, the cnDBTier sites are upgraded twice (in step 3 and step 6). Ensure that you follow this procedure as-is to upgrade from a TLS enabled version to a non-TLS version.
  1. Run the hooks.sh script with the -- remove-ssltype flag on all cnDBTier sites to set the type of "occnerepluser" as Non-TLS:

    Note:

    Update the values of the environment variables in the following code block as per your cluster. 
    export OCCNE_NAMESPACE="occne-cndbtier"
export MYSQL_CONNECTIVITY_SERVICE="mysql-connectivity-service"
export MYSQL_USERNAME="occneuser"
export MYSQL_PASSWORD="<password for the user occneuser>"
export OCCNE_REPL_USER_NAME="occnerepluser"
export MYSQL_CMD="kubectl -n <namespace> exec <ndbmysqld-0 pod name> -- mysql --binary-as-hex=0 --show-warnings"
  
occndbtier/files/hooks.sh --remove-ssltype
  2. In the custom_values.yaml file, set tlsMode to NONE for all the cnDBTier sites:
    tls:
  enable: true
  caCertificate: "<ca certificate file name>"
  tlsversion: "TLSv1.3"
  tlsMode: "NONE"
  ciphers:
    - TLS_AES_128_GCM_SHA256
    - TLS_AES_256_GCM_SHA384
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_AES_128_CCM_SHA256
  certificates:
    - name: ndbmysqld-0
      serverCertificate: "<server certificate name>"
      serverCertificateKey: "<server key name>"     
      clientCertificate: "<client certificate name>"
      clientCertificateKey: "<client key name>"
    - name: ndbmysqld-1
      serverCertificate: "<server certificate name>"
      serverCertificateKey: "<server key name>"     
      clientCertificate: "<client certificate name>"
      clientCertificateKey: "<client key name>"
    ...
    For example:
    tls:
  enable: true
  caCertificate: "combine-ca.pem"
  tlsversion: "TLSv1.3"
  tlsMode: "NONE"
  ciphers:
    - TLS_AES_128_GCM_SHA256
    - TLS_AES_256_GCM_SHA384
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_AES_128_CCM_SHA256
  certificates:
    - name: ndbmysqld-0
      serverCertificate: "server1-cert.pem"
      serverCertificateKey: "server1-key.pem"     
      clientCertificate: "client1-cert.pem"
      clientCertificateKey: "client1-key.pem"
    - name: ndbmysqld-1
      serverCertificate: "server1-cert.pem"
      serverCertificateKey: "server1-key.pem"     
      clientCertificate: "client1-cert.pem"
      clientCertificateKey: "client1-key.pem"
    ...
  3. Perform the Upgrading cnDBTier Clusters procedure to upgrade all the cnDBTier sites one after the other using the custom_values.yaml file that you updated in the previous steps.
  4. After upgrading each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.
    For example, run the following command to check the georeplication status of cnDBTier cluster2 configured with other remote sites:
    $ kubectl -n cluster2 exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.cluster2:8080/db-tier/status/replication/realtime
    Sample output:
    [
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster1",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster3",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster4",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  }
]

    In the sample output, the replicationStatus is "UP" for the localSiteName cluster2 for remotesiteName cluster1, cluster3, and cluster4. This indicates that the localSiteName cluster2 is able to replicate data from remotesiteName cluster1, cluster3, and cluster4.

  5. Ensure that TLS is disabled in the custom_values.yaml file for all cnDBTier sites:
    global:
  tls:    
    enable: false
  6. Perform the Upgrading cnDBTier Clusters procedure to upgrade all the cnDBTier sites one after the other using the custom_values.yaml file that you updated in the previous step.
  7. After upgrading each site, run the following command on the site to ensure that the replication is UP:
    $ kubectl -n <namespace> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace>:8080/db-tier/status/replication/realtime

    where, <namespace> is the namespace name of the of cnDBTier cluster.

    The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
    • "UP": Indicates that the local site is able to replicate data from that remote site.
    • "DOWN": Indicates that the local site is not able to replicate data from the respective remote site.

    For example, see Step 4.