Maintenance Procedures

7.1 Starting or Stopping cnDBTier

This section provides the procedures to start or stop cnDBTier.

7.1.1 Starting cnDBTier

To start cnDBTier, perform the following steps:

Run the following command to scale up the management nodes:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale sts ndbmgmd --replicas=<replica count for MGM pods>

Example:

$ kubectl -n occne-cndbtier scale sts ndbmgmd --replicas=2

Run the following command to scale up the data nodes:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale sts ndbmtd --replicas=<replica count for DATA pods>

Example:

$ kubectl -n occne-cndbtier scale sts ndbmtd --replicas=4

Run the following command to scale up the non-georeplication SQL nodes:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale sts ndbappmysqld --replicas=<replica count for non geo SQL pods>

Example:

$ kubectl -n cluster1 scale sts ndbappmysqld --replicas=2

Run the following command to scale up the SQL nodes:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale sts ndbmysqld --replicas=<replica count for SQL pods>

Example:

$ kubectl -n cluster1 scale sts ndbmysqld --replicas=2

Run the following commands to scale up db-monitor-svc, db-replication-svc, and db-backup-manager-svc:

To scale up the db-monitor-svc:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale deploy <cnDBTier monitor svc deployment name> --replicas=1

Example:

$ kubectl -n occne-cndbtier scale deploy mysql-cluster-db-monitor-svc --replicas=1

To scale up the db-replication-svc:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale deploy <cnDBTier replication svc deployment name> --replicas=1

Example:

$ kubectl -n occne-cndbtier scale deploy mysql-cluster-Cluster1-Cluster2-replication-svc --replicas=1

To scale up the db-backup-manager-svc:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale deploy <cnDBTier backup manager svc deployment name> --replicas=1

Example:

$ kubectl -n occne-cndbtier scale deploy mysql-cluster-db-backup-manager-svc --replicas=1

Run the following command on each mate site to check the replication status:
```
$ kubectl -n <namespace of cnDBTier cluster> exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.<namespace of cnDBTier cluster>:8080/db-tier/status/replication/realtime
```
The value of replicationStatus in the output indicates if the local site is able to replicate data from that remote site:
- "UP": Indicates that the local site is able to replicate data from that remote site.
- "DOWN": Indicates that the local site is not able to replicate data from the respective remote site. In this case, perform a georeplication recovery. For georeplication recovery procedure, see the "Restoring Georeplication Failure" section in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
For example, run the following command to check the georeplication status of cnDBTier cluster2 configured with other remote sites:
```
$ kubectl -n cluster2 exec -it ndbmysqld-0 -- curl http://mysql-cluster-db-monitor-svc.cluster2:8080/db-tier/status/replication/realtime
```
Sample output:
```
[
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster1",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster3",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  },
  {
    "localSiteName": "cluster2",
    "remoteSiteName": "cluster4",
    "replicationStatus": "UP",
    "secondsBehindRemote": 0,
    "replicationGroupDelay": [
      {
        "replchannel_group_id": "1",
        "secondsBehindRemote": 0
      }
    ]
  }
]    
```
In the sample output, the replicationStatus is "UP" for the localSiteName cluster2 for remotesiteName cluster1, cluster3, and cluster4. This indicates that the localSiteName cluster2 is able to replicate data from remotesiteName cluster1, cluster3, and cluster4.

Note:

If georeplication is enabled, then run this procedure on all sites.

7.1.2 Stopping cnDBTier

To stop cnDBTier, perform the following steps:

Run the following command to scale down the non-georeplication SQL nodes:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale sts ndbappmysqld --replicas=0

For example:

$ kubectl -n cluster1 scale sts ndbappmysqld --replicas=0

Run the following commands to scale down db-monitor-svc, db-replication-svc, and db-backup-manager-svc:

Run the following commands to scale down db-monitor-svc:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale deploy <cnDBTier monitor svc deployment name> --replicas=0

For example:

$ kubectl -n cluster1 scale deploy mysql-cluster-db-monitor-svc --replicas=0

Run the following commands to scale down db-replication-svc:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale deploy <cnDBTier replication svc deployment name> --replicas=0

For example:

$ kubectl -n cluster1 scale deploy mysql-cluster-Cluster1-Cluster2-replication-svc --replicas=0

Run the following commands to scale down db-backup-manager-svc:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale deploy <cnDBTier backup manager svc deployment name> --replicas=0

For example:

$ kubectl -n cluster1 scale deploy mysql-cluster-db-backup-manager-svc --replicas=0

Run the following command to scale down the SQL nodes:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale sts ndbmysqld --replicas=0

For example:

$ kubectl -n cluster1 scale sts ndbmysqld --replicas=0

Run the following command to scale down the data nodes:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale sts ndbmtd --replicas=0

For example:

$ kubectl -n cluster1 scale sts ndbmtd --replicas=0

Run the following command to scale down the management nodes:

$ kubectl -n <CNDBTIER_NAMESPACE_NAME> scale sts ndbmgmd --replicas=0

For example:

$ kubectl -n cluster1 scale sts ndbmgmd --replicas=0

7.2 Starting or Stopping cnDBTier Georeplication Service

This section provides the procedures to start or stop cnDBTier georeplication service.

7.2.1 Starting cnDBTier Georeplication Between Sites

This section provides the procedure to start cnDBTier georeplication service using the cnDBTier switchover APIs.

To start cnDBTier georeplication service, perform the following:

Run the following command to get the LoadBalancer IP of the replication service on the site:
```
$ IP=$(kubectl get svc -n <namespace> | grep repl | awk '{print $4}' | head -n 1 )
```
where, <namespace> is the namespace of the failed site.
For example, run the following command to get the LoadBalancer IP of the replication service on cluster1:
```
$ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
```
Run the following command to get the LoadBalancer Port of the replication service on the site:
```
$ PORT=$(kubectl get svc -n <namespace> | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1)
```
where, <namespace> is the namespace of the failed site.
For example, run the following command to get the LoadBalancer port of the replication service on cluster1:
```
$ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1)
```
Run the following command to start the replication service in cnDBTier with respect to siteName:

Note:
Replace $IP and $PORT in the command with the LoadBalancer IP and port numbers obtained from steps 1 and 2.
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/start/sitename/{siteName}
```
For example, run the following command to start the replication service in cnDBTier with respect to cluster1:
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/start/sitename/ cluster1
```
Sample output:
```
{
    "replicationSwitchOver":"start"
}
```
Run the following command to start the replication service in cnDBTier with respect to siteName and remoteSiteName:

Note:
Replace $IP and $PORT in the command with the LoadBalancer IP and port numbers obtained from steps 1 and 2.
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/start/sitename/{siteName}/remotesitename/{remoteSiteName}
```
For example, run the following command to start the replication service in cnDBTier with respect to cluster1 and cluster2:
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/start/sitename/cluster1/remotesitename/cluster2
```
Sample output:
```
{
    "replicationSwitchOver":"start"
}
```
Run the following command to start the replication service in cnDBTier with respect to siteName, remoteSiteName, and replChannelGroupId:

Note:
Replace $IP and $PORT in the command with the LoadBalancer IP and port numbers obtained from steps 1 and 2.
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/start/sitename/{siteName}/remotesitename/{remoteSiteName}/replchannelgroupid/{replChannelGroupId}
```
For example, run the following command to start the replication service in cnDBTier with respect to cluster1, cluster2, and replication channel group ID "1":
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/start/sitename/cluster1/remotesitename/cluster2/replchannelgroupid/1
```
Sample output:
```
{
    "replicationSwitchOver":"start"
}
```

Note:

For more information about API response payloads, error codes, and curl commands for starting georeplication service, see cnDBTier Switchover APIs.

7.2.2 Stopping cnDBTier Georeplication Between Sites

This section provides the procedure to stop cnDBTier georeplication service using cnDBTier switchover and stop replica APIs.

To stop cnDBTier georeplication service, perform the following:

Perform the following steps to stop the cnDBTier replication service switchover:
1. Run the following command to get the LoadBalancer IP of the replication service on the site:
```
$ IP=$(kubectl get svc -n <namespace> | grep repl | awk '{print $4}' | head -n 1 )
```
  where, <namespace> is the namespace of the failed site.
  For example, run the following command to get the LoadBalancer IP of the replication service on cluster1:
```
$ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
```
2. Run the following command to get the LoadBalancer Port of the replication service on the site:
```
$ PORT=$(kubectl get svc -n <namespace> | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1)
```
  where, <namespace> is the namespace of the failed site.
  For example, run the following command to get the LoadBalancer port of the replication service on cluster1:
```
$ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1)
```
3. Run the following command to stop the replication service switchover in cnDBTier with respect to siteName:
  
  Note:
  Replace $IP and $PORT in the command with the LoadBalancer IP and port numbers obtained from steps 1 and 2.
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/stop/sitename/{siteName}
```
  For example, run the following command to stop the replication service switchover in cnDBTier with respect to cluster1:
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/stop/sitename/ cluster1
```
  Sample output:
```
{
    "replicationSwitchOver":"stop"
}
```
4. Run the following command to stop the replication service switchover in cnDBTier with respect to siteName and remoteSiteName:
  
  Note:
  Replace $IP and $PORT in the command with the LoadBalancer IP and port numbers obtained from steps 1 and 2.
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/stop/sitename/{siteName}/remotesitename/{remoteSiteName}
```
  For example, run the following command to stop the replication service switchover in cnDBTier with respect to cluster1 and cluster2:
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/stop/sitename/cluster1/remotesitename/cluster2
```
  Sample output:
```
{
    "replicationSwitchOver":"stop"
}
```
5. Run the following command to stop the replication service switchover in cnDBTier with respect to siteName, remoteSiteName, and replChannelGroupId:
  
  Note:
  Replace $IP and $PORT in the command with the LoadBalancer IP and port numbers obtained from steps 1 and 2.
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/stop/sitename/{siteName}/remotesitename/{remoteSiteName}/replchannelgroupid/{replChannelGroupId}
```
  For example, run the following command to stop the replication service switchover in cnDBTier with respect to cluster1, cluster2, and replication channel group ID "1":
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/switchover/stop/sitename/cluster1/remotesitename/cluster2/replchannelgroupid/1
```
  Sample output:
```
{
    "replicationSwitchOver":"stop"
}
```
Note:
For more information about API response payloads, error codes, and curl commands to stop replication service switchover, see cnDBTier Switchover APIs.
Perform the following steps to stop the replication channel:
1. Run the following command to get the LoadBalancer IP of the replication service on the site:
```
$ IP=$(kubectl get svc -n <namespace> | grep repl | awk '{print $4}' | head -n 1 )
```
  where, <namespace> is the namespace of the failed site.
  For example, run the following command to get the LoadBalancer IP of the replication service on cluster1:
```
$ IP=$(kubectl get svc -n cluster1 | grep repl | awk '{print $4}' | head -n 1 )
```
2. Run the following command to get the LoadBalancer Port of the replication service on the site:
```
$ PORT=$(kubectl get svc -n <namespace> | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1)
```
  where, <namespace> is the namespace of the failed site.
  For example, run the following command to get the LoadBalancer port of the replication service on cluster1:
```
$ PORT=$(kubectl get svc -n cluster1 | grep repl | awk '{print $5}' |  cut -d '/' -f 1 |  cut -d ':' -f 1 | head -n 1)
```
3. Run the following command to stop the replication service in cnDBTier with respect to siteName:
  
  Note:
  Replace $IP and $PORT in the command with the LoadBalancer IP and port numbers obtained from steps 1 and 2.
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/stopreplica/sitename/{siteName}
```
  For example, run the following command to stop the replication service in cnDBTier with respect to cluster1:
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/stopreplica/sitename/ cluster1
```
  Sample output:
```
{
    "stopReplica":"stop"
}
```
4. Run the following command to stop the replication service in cnDBTier with respect to siteName and remoteSiteName:
  
  Note:
  Replace $IP and $PORT in the command with the LoadBalancer IP and port numbers obtained from steps 1 and 2.
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/stopreplica/sitename/{siteName}/remotesitename/{remoteSiteName}
```
  For example, run the following command to stop the replication service in cnDBTier with respect to cluster1 and cluster2:
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/stopreplica/sitename/cluster1/remotesitename/cluster2
```
  Sample output:
```
{
    "stopReplica":"stop"
}
```
5. Run the following command to stop the replication service in cnDBTier with respect to siteName, remoteSiteName, and replChannelGroupId:
  
  Note:
  Replace $IP and $PORT in the command with the LoadBalancer IP and port numbers obtained from steps 1 and 2.
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/stopreplica/sitename/{siteName}/remotesitename/{remoteSiteName}/replchannelgroupid/{replChannelGroupId}
```
  For example, run the following command to stop the replication service in cnDBTier with respect to cluster1, cluster2, and replication channel group ID "1":
```
$ curl -X PUT http://$IP:$PORT/ocdbtier/georeplication/stopreplica/sitename/cluster1/remotesitename/cluster2/replchannelgroupid/1
```
  Sample output:
```
{
    "stopReplica":"stop"
}
```

Note:

For more information about API response payloads, error codes, and curl commands to stop replication channel, see cnDBTier Stop Replica APIs.

7.3 cnDBTier Scaling

cnDBTier 23.4.7 supports horizontal and vertical scaling of pods for improved performance. This section describes the horizontal and vertical scaling procedures for SQL and database pods.

7.3.1 Horizontal Scaling

This section describes the procedures to horizontally scale ndbappmysqld and ndbmtd pods.

7.3.1.1 Scaling ndbappmysqld Pods

cnDBTier 23.4.7 supports autoscaling of cnDBTier ndbappmysqld pods when cnDBTier is deployed with a service account. For more information, see Scaling ndbappmysqld Pods With Service Account. When cnDBTier is deployed without a service account, you must manually scale the ndbappmysqld pods. This section describes the procedures to scale the cnDBTier ndbappmysqld pods when cnDBTier is deployed without service account and configurations to consider when cnDBTier is deployed with service account.

Note:

Before scaling, ensure that the worker nodes have adequate resources to support scaling.

Considerations

The examples in these procedures consider a cnDBTier setup deployed with two ndbmgmd, two ndbmtd, two ndbmysqld, and two ndbappmysqld pods. The ndbappmysqld pods of this cnDBtier setup are scaled up from replica count two to four in this procedure.

7.3.1.1.1 Scaling ndbappmysqld Pods Without Service Account

This section describes the procedure to manually scale the cnDBTier ndbappmysqld pods when cnDBTier is deployed without service account.

Note:

ndbappmysqld auto scaling requires service account. If the cnDBTier is deployed without the service account, then auto scaling is disabled.

Increase the replica count of the ndbappmysqld pods in the custom_values.yaml file under global.ndbappReplicaCount:
```
global:
  ndbappReplicaCount: 4
```
Note:
The cnDBTier is initially deployed with two ndbappmysqld pods and increased to a replica count of four in this step.
Upgrade cnDBTier with the new ndbappReplicaCount by following the upgrade procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Note:
cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.

When the helm upgrade completes, the ndbappmysqld replica count is increased to the value updated in the custom_values.yaml file.

7.3.1.1.2 Scaling ndbappmysqld Pods With Service Account

This section describes the procedure to scale the cnDBTier ndbappmysqld pods when cnDBTier is deployed with service account.

Scaling cnDBTier ndbappmysqld When Autoscaling is Enabled

When cnDBTier is deployed with a service account and autoscaling is enabled, the ndbappmysqld pods are scaled automatically.

Ensure that autoscaling is enabled for cnDBTier ndbappmysqld pods in the custom_values.yaml file under autoscaling.ndbapp:

autoscaling:
  ndbapp:
    enabled: true

Note:

Apply the above change to the cnDBTier Helm chart by following the "Upgrading cnDBTier Clusters" procedure in the Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide. cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.

When autoscaling is enabled, ndbappmysqld is scaled between ndbappReplicaCount and ndbappReplicaMaxCount according to the CPU and RAM usage defined in the custom_values.yaml file:

horizontalPodAutoscaler: 
  memory:
    enabled: true
    averageUtilization: 80
  cpu:
    enabled: false
    averageUtilization: 80

Scaling cnDBTier ndbappmysqld When Autoscaling is Disabled

Increase the replica count of the ndbappmysqld pods in the custom_values.yaml file under global.ndbappReplicaCount:
```
global:
  ndbappReplicaCount: 4
```
Note:
cnDBTier is initially deployed with two ndbappmysqld pods and increased to a replica count of four in this step.
Upgrade cnDBTier with the new ndbappReplicaCount by following the upgrade procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Note:
cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.

When the Helm upgrade completes, the ndbappmysqld replica count is increased to the value updated in the custom_values.yaml file.

7.3.1.2 Scaling ndbmtd Pods

cnDBTier supports only scaling up of data pods, and doesn't support scaling down of data pods. This section describes the manual procedure to scale up the cnDBTier ndbmtd pods.

Note:

Before scaling, ensure that the worker nodes have adequate resources to support scaling.
Divert the traffic during the horizontal scaling of the ndbmtd pods as the MySQL NDB cluster may go offline during the scaling process.

Considerations

The examples in this procedure consider a cnDBTier setup deployed with 2 ndbmgm, 2 ndbmtd, 2 ndbmysqld sql, and 2 ndbappmysqld pods. The ndbmtd pods of this cnDBTier setup are scaled up from replica count 2 to 4 in this procedure.

Procedure

Before scaling the data pods, perform the Helm test on the cnDBTier cluster to check the health of the cluster:

$ helm test mysql-cluster -n <namespace>

Example:

$ helm test mysql-cluster -n occne-cndbtier

Sample output:

NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 08:10:11 2023
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 2
TEST SUITE:     mysql-cluster-node-connection-test
Last Started:   Tue Nov 07 10:03:51 2023
Last Completed: Tue Nov 07 10:04:16 2023
Phase:          Succeeded

You can also check if all the NDB pods are connected by checking the status of cnDBTier from the management ndb_mgm console:

$ kubectl -n occne-cndbtier exec ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.74.65  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.84.68  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.73.61  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.84.67  (mysql-8.0.35 ndb-8.0.35)
 
[mysqld(API)]   10 node(s)
id=56   @10.233.84.69  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.73.62  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.78.70  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.72.49  (mysql-8.0.35 ndb-8.0.35)
id=72 (not connected, accepting connect from ndbappmysqld-2.ndbappmysqldsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=73 (not connected, accepting connect from ndbappmysqld-3.ndbappmysqldsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for fault recovery. You can ignore these nodes.

Enable replicationskiperrors in the values.yaml file and apply the changes using the Helm upgrade procedure for all cnDBTier sites.

Note:
Skip this step if replicationskiperrors is already enabled in the cnDBTier cluster.
```
global:
  replicationskiperrors:
    enable: true
```
Increase the replica count of the ndbmtd pod in the custom_values.yaml file:
Note:
- The cnDBTier is deployed with 2 ndbmtd and increased to a replica count of 4 in this step.
- You can only scale up the ndbmtd pod by an even number of increment and can't scale down the pods.
```
global:
  ndbReplicaCount: 4
```
Run the following command to upgrade the cnDBTier to increase the ndbmtd replica count:

Note:
After the upgrade, the new ndbmtd pods can crash and restart. You can this error. Verify the status of the new pod before partitioning of data in step 14.
```
$ helm upgrade --no-hooks --set global.updateStrategy=OnDelete mysql-cluster --namespace <namespace> occndbtier -f <path to custom_values.yaml>
```
Perform the following steps to scale the ndbmtd pods back to their original replica count (the replica count that the ndbmtd pods had before performing Step 3):

Note:
In this case, the replica count was increased from 2 to 4 in Step 3. Therefore, in this step, scale back the replica count to 2.
1. Run the following commands to patch the horizontal pod autoscaling (hpa):
```
$ kubectl -n <namespace> patch hpa ndbmtd -p '{"spec": {"minReplicas": 2}}'
$ kubectl -n <namespace> patch hpa ndbmtd -p '{"spec": {"maxReplicas": 2}}'
```
2. Run the following command to scale down the ndbmtd sts replica count to 2:
```
$ kubectl -n <namespace> scale sts ndbmtd --replicas=2
```
3. After scaling down the ndbmtd sts replica count, terminate the newly added ndbmtd pods. Retain only the existing ndbmtd pods in the cluster.
Once the upgrade is complete, delete all the management pods simultaneously:
```
$ kubectl -n <namespace> delete pods ndbmgmd-0 ndbmgmd-1
```
Wait for the management pods to come back to the running state and connect to the cluster. Run the following command to verify the status of the pods:
```
$ kubectl -n occne-cndbtier exec ndbmgmd-0 -- ndb_mgm -e show
```
Delete the data pods one at a time. Wait for a deleted pod to return to the running state and connect to cluster before deleting the next one:

Note:
Delete the data pods in descending order (ndbmtd-n, ndbmtd-(n-1), ndbmtd-(n-2), and so on).
For example:
1. Run the following command to delete pod ndbmtd-2:
```
$ kubectl -n <namespace> delete pod ndbmtd-2
```
2. Wait for ndbmtd-2 to return to the running state and connect back to the NDB cluster.
3. Run the following command to delete pod ndbmtd-1:
```
$ kubectl -n <namespace> delete pod ndbmtd-1
```
4. Wait for ndbmtd-1 to return to the running state and connect back to the NDB cluster.
5. Run the following command to delete pod ndbmtd-0:
```
$ kubectl -n <namespace> delete pod ndbmtd-0
```

Wait until all the data pods are up and running, and get connected to the NDB cluster. To check if the data pods are connected to the NDB cluster, log in to one of the management pods and check the status of the NDB cluster from the ndb_mgm console:

$ kubectl -n occne-cndbtier exec ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=1    @10.233.74.65  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.84.68  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
id=3 (not connected, accepting connect from ndbmtd-2.ndbmtdsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=4 (not connected, accepting connect from ndbmtd-3.ndbmtdsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.73.61  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.84.67  (mysql-8.0.35 ndb-8.0.35)
 
[mysqld(API)]   10 node(s)
id=56   @10.233.84.69  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.73.62  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.78.70  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.72.49  (mysql-8.0.35 ndb-8.0.35)
id=72 (not connected, accepting connect from ndbappmysqld-2.ndbappmysqldsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=73 (not connected, accepting connect from ndbappmysqld-3.ndbappmysqldsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The newly added data pods, node IDs 3 and 4 in the sample output, are shown as "starting, no nodegroup". You can ignore these nodes.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Delete the ndbmysqld pods one at a time. Wait for a deleted pod to return to the running state and connect to the cluster before deleting the next one:

Note:
Delete the pods in descending order (ndbmysqld-n, ndbmysqld-(n-1), ndbmysqld-(n-2), and so on).
For example:
1. Run the following command to delete pod ndbmysqld-2:
```
$ kubectl -n <namespace> delete pod ndbmysqld-2
```
2. Wait for ndbmysqld-2 to return to the running state and connect back to the NDB cluster.
3. Run the following command to delete pod ndbmysqld-1:
```
$ kubectl -n <namespace> delete pod ndbmysqld-1
```
4. Wait for ndbmysqld-1 to return to the running state and connect back to the NDB cluster.
5. Run the following command to delete pod ndbmysqld-0:
```
$ kubectl -n <namespace> delete pod ndbmysqld-0
```

Wait for the ndbmysqld pods to return to running state and connect to the NDB cluster. To check if the data pods are connected to the NDB cluster, log in to one of the management pods and check the status of the NDB cluster from the ndb_mgm console:

$ kubectl -n occne-cndbtier exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=1    @10.233.74.65  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.84.68  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
id=3 (not connected, accepting connect from ndbmtd-2.ndbmtdsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=4 (not connected, accepting connect from ndbmtd-3.ndbmtdsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.73.61  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.84.67  (mysql-8.0.35 ndb-8.0.35)
 
[mysqld(API)]   10 node(s)
id=56   @10.233.84.69  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.73.62  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.78.70  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.72.49  (mysql-8.0.35 ndb-8.0.35)
id=72 (not connected, accepting connect from ndbappmysqld-2.ndbappmysqldsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=73 (not connected, accepting connect from ndbappmysqld-3.ndbappmysqldsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Delete the ndbappmysqld pods one at a time. Wait for a deleted pod to return back to the running state before deleting the next pod.

Note:
Delete the pods in descending order (ndbappmysqld-n, ndbappmysqld-(n-1), ndbappmysqld-(n-2), and so on).
For example:
1. Run the following command to delete pod ndbappmysqld-2:
```
$ kubectl -n <namespace> delete pod ndbappmysqld-2
```
2. Wait for ndbappmysqld-2 to return to the running state and connect back to the NDB cluster.
3. Run the following command to delete pod ndbappmysqld-1:
```
$ kubectl -n <namespace> delete pod ndbappmysqld-1
```
4. Wait for ndbappmysqld-1 to return to the running state and connect back to the NDB cluster.
5. Run the following command to delete pod ndbappmysqld-0:
```
$ kubectl -n <namespace> delete pod ndbappmysqld-0
```

Wait for the ndbappmysqld pods to return to the running state and connect to the NDB cluster. To check if the data pods are connected to the NDB cluster, log in to one of the management pods and check the status of the NDB cluster from the ndb_mgm console:

$ kubectl -n occne-cndbtier exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=1    @10.233.74.65  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.84.68  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
id=3 (not connected, accepting connect from ndbmtd-2.ndbmtdsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=4 (not connected, accepting connect from ndbmtd-3.ndbmtdsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.73.61  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.84.67  (mysql-8.0.35 ndb-8.0.35)
 
[mysqld(API)]   10 node(s)
id=56   @10.233.84.69  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.78.70  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.72.49  (mysql-8.0.35 ndb-8.0.35)
id=72 (not connected, accepting connect from ndbappmysqld-2.ndbappmysqldsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=73 (not connected, accepting connect from ndbappmysqld-3.ndbappmysqldsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Perform the following steps to scale up the ndbmtd pods to their increased replica count. That is, the replica count configured at Step 3:

Note:

In this case, the replica count was increased to 4 in Step 3. Therefore, in this step, scale back the replica count to 4.

Run the following commands to patch horizontal pod autoscaler (hpa):

$ kubectl -n <namespace> patch hpa ndbmtd -p '{"spec": {"maxReplicas": 4}}'
$ kubectl -n <namespace> patch hpa ndbmtd -p '{"spec": {"minReplicas": 4}}'

Scale up the ndbmtd sts replica count to 4:
```
$ kubectl -n <namespace> scale sts ndbmtd --replicas=4
```

Wait for the ndbmtd pods to return to the running state and connect to the NDB cluster. To check if the data pods are connected to the NDB cluster, log in to one of the management pods and check the status of the NDB cluster from the ndb_mgm console:

$ kubectl -n occne-cndbtier exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=1    @10.233.74.65  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.84.68  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
id=3    @10.233.117.60  (mysql-8.0.35 ndb-8.0.35, starting, no nodegroup)
id=4    @10.233.111.64  (mysql-8.0.35 ndb-8.0.35, starting, no nodegroup)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.73.61  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.84.67  (mysql-8.0.35 ndb-8.0.35)
 
[mysqld(API)]   10 node(s)
id=56   @10.233.84.69  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.73.62  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.78.70  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.72.49  (mysql-8.0.35 ndb-8.0.35)
id=72 (not connected, accepting connect from ndbappmysqld-2.ndbappmysqldsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=73 (not connected, accepting connect from ndbappmysqld-3.ndbappmysqldsvc.occne-cndbtier.svc.occne1-cgbu-cne-dbtier)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

When the newly added ndbmtd pods are connected to the NDB cluster, they are shown in the "no nodegroup" state.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Run the following command in the NDB Cluster management client to create the new node group. Node ID for the newly added ndbmtd pod is <sequence no of the ndbmtd pod + 1>.
```
ndb_mgm>  CREATE NODEGROUP <node_id_for_pod_ndbmtd-2>,<node_id_for_pod_ndbmtd-3>
```
For example:
```
ndb_mgm>  CREATE NODEGROUP <(sequence no of pod ndbmtd-2) + 1>,<(sequence no of pod ndbmtd-3) + 1>
 
[mysql@ndbmgmd-0 ~]$ ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm>  CREATE NODEGROUP 3,4
```
Sample output:
```
Connected to Management Server at: localhost:1186
Nodegroup 1 created
```
Note:
- This example considers that cnDBTier is initially created with two data pods (node IDs 1 and 2). In this step, the data pod count is scaled up from 2 to 4. Therefore, the node IDs 3 and 4 are assigned to the newly created data nodes.
- If you are adding more than two ndbmtd nodes to the cluster, then create the node groups for the first two nodes, then for the next pair, and so on. For example, if you are adding four ndbmtd nodes with node IDs x1, x2, x3, x4, then first create node groups for x1 and x2 and then for x3 and x4 as shown in the following code block:
```
ndb_mgm> CREATE NODEGROUP x1, x2
ndb_mgm> CREATE NODEGROUP x3, x4
```

Depending on the following conditions, use georeplication recovery procedure or the dbt_reorg_table_partition script to redistribute the cluster data:

If the mate site is available, then perform the georeplication recovery procedure to redistribute the data across all the data nodes (including the new data nodes). For georeplication procedure, see the "Restoring Georeplication Failure" section in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
If the mate site is not available, perform the following steps to run the dbt_reorg_table_partition script to redistribute the data across all the data nodes (including the new data nodes).
If the mate site is available, however you don't want to perform the georeplication recovery, then perform the following steps to run the dbt_reorg_table_partition script to redistribute the data across all the data nodes (including the new data nodes).

Run the following commands to source the source_me file. This adds the bin directory with the program to the user PATH and prompts you to enter the namespace. The namespace is used to set DBTIER_NAMESPACE. Additionally, it sets the DBTIER_LIB environment variable to the path of the directory containing the libraries needed by dbt_reorg_table_partition:

$ cd Artifacts/Scripts/tools
$ source ./source_me

Sample output:

NOTE: source_me must be sourced while your current directory is the directory with the source_me file.

Enter cndbtier namespace: sitea
DBTIER_NAMESPACE = "sitea"

DBTIER_LIB=/home/cloud-user/seshu/deploy_cndbtier_sitea_23.4.1/Artifacts/Scripts/tools/dbtier/lib

Adding /home/cloud-user/seshu/deploy_cndbtier_sitea_23.4.1/Artifacts/Scripts/tools/dbtier/bin to PATH
PATH=/home/cloud-user/.local/bin:/home/cloud-user/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/var/occne/cluster/occne3-arjun-sreenivasalu/artifacts/istio-1.15.3/bin/:/var/occne/cluster/occne3-arjun-sreenivasalu/artifacts:/home/cloud-user/seshu/deploy_cndbtier_sitea_23.4.1/Artifacts/Scripts/tools/dbtier/bin

Run the dbt_reorg_table_partition script:

$ dbt_reorg_table_partition

Sample output:

dbt_reorg_table_partition 23.4.1
Copyright (c) 2023 Oracle and/or its affiliates. All rights reserved.
2024-01-17T05:11:45Z INFO - Getting sts and sts pod info...
2024-01-17T05:11:45Z INFO - Getting MGM sts and sts pod info...
2024-01-17T05:11:45Z INFO - MGM_STS="ndbmgmd"
2024-01-17T05:11:45Z INFO - MGM_REPLICAS="2"
2024-01-17T05:11:45Z INFO - MGM_PODS:
    ndbmgmd-0
    ndbmgmd-1
2024-01-17T05:11:45Z INFO - Getting NDB sts and sts pod info...
2024-01-17T05:11:45Z INFO - NDB_STS="ndbmtd"
2024-01-17T05:11:45Z INFO - NDB_REPLICAS="2"
2024-01-17T05:11:45Z INFO - NDB_PODS:
    ndbmtd-0
    ndbmtd-1
2024-01-17T05:11:45Z INFO - Getting API sts and sts pod info...
2024-01-17T05:11:45Z INFO - API_STS="ndbmysqld"
2024-01-17T05:11:45Z INFO - API_REPLICAS="2"
2024-01-17T05:11:45Z INFO - API_PODS:
    ndbmysqld-0
    ndbmysqld-1
2024-01-17T05:11:45Z INFO - Getting APP sts and sts pod info...
2024-01-17T05:11:45Z INFO - APP_STS="ndbappmysqld"
2024-01-17T05:11:45Z INFO - APP_REPLICAS="2"
2024-01-17T05:11:45Z INFO - APP_PODS:
    ndbappmysqld-0
    ndbappmysqld-1
2024-01-17T05:11:45Z INFO - Getting deployment pod info...
2024-01-17T05:11:45Z INFO - grepping for backup-man (BAK_CHART_NAME)...
2024-01-17T05:11:46Z INFO - BAK_PODS:
    mysql-cluster-db-backup-manager-svc-57fb8ff49c-49p55
2024-01-17T05:11:46Z INFO - BAK_DEPLOY:
    mysql-cluster-db-backup-manager-svc
2024-01-17T05:11:46Z INFO - grepping for db-mon (MON_CHART_NAME)...
2024-01-17T05:11:46Z INFO - MON_PODS:
    mysql-cluster-db-monitor-svc-7b7559cd45-shm8r
2024-01-17T05:11:46Z INFO - MON_DEPLOY:
    mysql-cluster-db-monitor-svc
2024-01-17T05:11:46Z INFO - grepping for repl (REP_CHART_NAME)...
2024-01-17T05:11:46Z INFO - REP_PODS:
    mysql-cluster-siteb-local-replication-svc-9c4c59f87-6zl57
2024-01-17T05:11:46Z INFO - REP_DEPLOY:
    mysql-cluster-siteb-local-replication-svc
2024-01-17T05:11:46Z INFO - Reorganizing table partitions...
mysql.ndb_apply_status  optimize        status  OK
backup_info.DBTIER_BACKUP_INFO  optimize        status  OK
backup_info.DBTIER_BACKUP_COMMAND_QUEUE optimize        status  OK
backup_info.DBTIER_BACKUP_TRANSFER_INFO optimize        status  OK
mysql.ndb_replication   optimize        status  OK
hbreplica_info.DBTIER_HEARTBEAT_INFO    optimize        status  OK
replication_info.DBTIER_SITE_INFO       optimize        status  OK
replication_info.DBTIER_REPL_SITE_INFO  optimize        status  OK
replication_info.DBTIER_REPLICATION_CHANNEL_INFO        optimize        status  OK
replication_info.DBTIER_INITIAL_BINLOG_POSTION  optimize        status  OK
replication_info.DBTIER_REPL_ERROR_SKIP_INFO    optimize        status  OK
replication_info.DBTIER_REPL_EVENT_INFO optimize        status  OK
replication_info.DBTIER_REPL_SVC_INFO   optimize        status  OK
2024-01-17T05:12:10Z INFO - Reorganized Tables Partition Successfully

Run the following command in the management client to check if the data is distributed:

ndb_mgm> ALL REPORT MEMORY

For example:

ndb_mgm> ALL REPORT MEMORY

Sample output:

Connected to Management Server at: localhost:1186
Node 1: Data usage is 0%(58 32K pages of total 12755)
Node 1: Index usage is 0%(45 32K pages of total 12742)
Node 2: Data usage is 0%(58 32K pages of total 12755)
Node 2: Index usage is 0%(45 32K pages of total 12742)
Node 3: Data usage is 0%(15 32K pages of total 12780)
Node 3: Index usage is 0%(20 32K pages of total 12785)
Node 4: Data usage is 0%(15 32K pages of total 12780)
Node 4: Index usage is 0%(20 32K pages of total 12785)

Run the Helm test on the cnDBTier cluster to check the health of the cluster:

$ helm test mysql-cluster -n occne-cndbtier

Sample output:

NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 08:10:11 2023
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 2
TEST SUITE:     mysql-cluster-node-connection-test
Last Started:   Tue Nov 07 10:27:49 2023
Last Completed: Tue Nov 07 10:28:11 2023
Phase:          Succeeded

You can also verify the status of the cnDBTier from the ndb_mgm console by running the following command:

$ kubectl -n occne-cndbtier exec ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     4 node(s)
id=1    @10.233.92.53  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.72.66  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=3    @10.233.117.60  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 1)
id=4    @10.233.111.64  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 1, *)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.111.62  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.117.59  (mysql-8.0.35 ndb-8.0.35)
 
[mysqld(API)]   8 node(s)
id=56   @10.233.72.65  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.92.51  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.72.64  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.92.52  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Disable replicationskiperrors in the values.yaml file and apply the changes using the Helm upgrade procedure.
Note:
Skip this step if you skipped Step 2.
```
global:
  replicationskiperrors:
    enable: false
```

Run the following command to patch the ndbmtd, ndbmysqld, and ndbappmysqld statefulsets and update updateStrategy to RollingUpdate:

$ kubectl -n <namespace> patch sts <ndbmtd sts name> -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}' 
 
$ kubectl -n <namespace> patch sts <ndbmysqld sts name> -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}'  
 
$ kubectl -n <namespace> patch sts <ndbappmysqld sts name> -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}'

7.3.2 Vertical Scaling

Currently, cnDBTier supports only manual vertical scaling of ndbmtd and SQL pods. This section describes the procedure to manually scale ndbmtd, ndbappmysqld, and ndbmysqld pods.

7.3.2.1 Scaling ndbappmysqld Pods

Perform the following procedure to scale up ndbappmysqld pods vertically.

Note:

Before scaling the pods, ensure that the worker nodes have adequate resources to support scaling.
Before you proceed with the vertical scaling procedure for ndbappmysqld, perform a Helm test and ensure that all the cnDBTier services are running smoothly.

Perform the following steps to update the CPU and RAM:
1. Configure the required CPU and memory values in the global section of the custom_values.yaml file.
  For example:
```
ndbapp:
  resources:
    limits:
      cpu: 8
      memory: 10Gi
    requests:
      cpu: 8
      memory: 10Gi
```
2. Upgrade cnDBTier by performing a Helm upgrade with the modified custom_values.yaml file. For more information about the upgrade procedure, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
  
  Note:
  cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.
Perform the following steps to update the PVC value:
1. Update the custom_values.yaml file with the new PVC values for ndbappmysqld pods:
```
global:
  ndbapp:
    ndbdisksize: 10Gi
```
2. Delete the ndbappmysqld statefulset and make the dependencies as orphan:
```
$ kubectl -n occne-cndbtier delete sts --cascade=orphan ndbappmysqld
```
3. Delete the ndbappmysqld-1 pod and patch its PVCs with the new PVC value:
  1. Delete the ndbappmysqld-1 pod:
```
$ kubectl -n occne-cndbtier delete pod ndbappmysqld-1
```
    Sample output:
    
    pod "ndbappmysqld-1" deleted
  2. Patch the PVCs of ndbappmysqld-1 pod with the new PVC values:
```
$ kubectl -n occne-cndbtier patch -p '{ "spec": { "resources": { "requests": { "storage": "10Gi" }}}}' pvc pvc-ndbappmysqld-ndbappmysqld-1
```
  3. Wait for the new PV to attach with the PVC.
4. Upgrade cnDBTier with the modified custom_values.yaml file:
```
$ helm upgrade mysql-cluster occndbtier -f occndbtier/custom_values.yaml -n occne-cndbtier --no-hooks
```
5. Repeat steps b through d for ndbappmysqld-0.
6. As cnDBTier Helm upgrade is performed with the "--no-hooks" option in step d, upgradeStrategy of the cnDBTier StatefulSets is changed to OnDelete. Therefore, perform the cnDBTier upgrade one more time to restore upgradeStrategy to RollingRestart:
```
helm -n ${OCCNE_NAMESPACE} upgrade ${OCCNE_RELEASE_NAME} occndbtier -f occndbtier/custom_values.yaml
```

7.3.2.2 Scaling ndbmysqld Pods

Perform the following procedure to scale up ndbmysqld pods vertically:

Note:

Before scaling the pods, ensure that the worker nodes have adequate resources to support scaling.
Before you proceed with the vertical scaling procedure for ndbmysqld, perform a Helm test and ensure that all the cnDBTier services are running smoothly.

Perform the following steps to update the CPU and RAM:
1. Configure the required CPU and memory values in the global section of the custom_values.yaml file.
  For example:
```
api:
  resources:
    limits:
      cpu: 8
      memory: 10Gi
    requests:
      cpu: 8
      memory: 10Gi
```
2. Upgrade cnDBTier by performing a Helm upgrade with the modified custom_values.yaml file. For more information about the upgrade procedure, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
  
  Note:
  cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.
Perform the following steps to update the PVC value:
1. Update the custom_values.yaml file with the new PVC values for ndbmysqld pods:
```
global:
  api:
    ndbdisksize: 10Gi
```
2. Delete the ndbmysqld statefulset and make the dependencies as orphan:
```
$ kubectl -n occne-cndbtier delete sts --cascade=orphan ndbmysqld
```
3. Delete the ndbmysqld-1 pod and patch its PVCs with the new PVC value:
  1. Delete the ndbmysqld-1 pod:
```
$ kubectl -n occne-cndbtier delete pod ndbmysqld-1
```
    Sample output:
    
    pod "ndbmysqld-1" deleted
  2. Patch the PVCs of ndbmysqld-1 pod with the new PVC values:
```
$ kubectl -n occne-cndbtier patch -p '{ "spec": { "resources": { "requests": { "storage": "10Gi" }}}}' pvc pvc-ndbmysqld-ndbmysqld-1
```
  3. Wait for the new PV to attach with the PVC.
4. Upgrade cnDBTier with the modified custom_values.yaml file:
```
$ helm upgrade mysql-cluster occndbtier -f occndbtier/custom_values.yaml -n occne-cndbtier --no-hooks
```
5. Repeat steps b through d for ndbmysqld-0.
6. As cnDBTier Helm upgrade is performed with the "--no-hooks" option in step d, upgradeStrategy of the cnDBTier StatefulSets is changed to OnDelete. Therefore, perform the cnDBTier upgrade one more time to restore upgradeStrategy to RollingRestart:
```
helm -n ${OCCNE_NAMESPACE} upgrade ${OCCNE_RELEASE_NAME} occndbtier -f occndbtier/custom_values.yaml
```

7.3.2.3 Scaling ndbmtd Pods

Perform the following procedure to vertically scale up ndbmtd pods:

Note:

Before scaling the pods, ensure that the worker nodes have adequate resources to support scaling.

Perform the Helm test on the deployed cnDBTier cluster to check the health of the cluster:

$ helm test mysql-cluster -n occne-cndbtier

Sample output:

NAME: mysql-cluster
LAST DEPLOYED: Wed Aug 09 03:24:03 2023
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 1
TEST SUITE:     mysql-cluster-node-connection-test
Last Started:   Wed Aug 09 04:03:01 2023
Last Completed: Wed Aug 09 04:03:26 2023
Phase:          Succeeded

You can also verify the status of the cnDBTier from the ndb_mgm console by running the following command:

$ kubectl -n occne-cndbtier exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.95.87  (mysql-8.0.34 ndb-8.0.34, Nodegroup: 0, *)
id=2    @10.233.99.254  (mysql-8.0.34 ndb-8.0.34, Nodegroup: 0)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.110.157  (mysql-8.0.34 ndb-8.0.34)
id=50   @10.233.101.213  (mysql-8.0.34 ndb-8.0.34)
 
[mysqld(API)]   10 node(s)
id=56   @10.233.99.243  (mysql-8.0.34 ndb-8.0.34)
id=57   @10.233.101.221  (mysql-8.0.34 ndb-8.0.34)
id=70   @10.233.100.228  (mysql-8.0.34 ndb-8.0.34)
id=71   @10.233.101.217  (mysql-8.0.34 ndb-8.0.34)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Perform the following steps to update the CPU and RAM values in the custom_values.yaml file:
1. Configure the required CPU and memory values in the global section of the custom_values.yaml file.
  For example:
  
  The following code block shows the old CPU and RAM values in the custom_values.yaml file
```
global:
  ndb:
    datamemory: 400M
ndb: 
  resources:
    limits:
      cpu: 1
      memory: 4Gi
    requests:
      cpu: 1
      memory: 4Gi
```
  The following code block shows the updated CPU and RAM values in the custom_values.yaml file:
```
global:
  ndb:
    datamemory: 800M
ndb: 
  resources:
    limits:
      cpu: 2
      memory: 8Gi
    requests:
      cpu: 2
      memory: 8Gi
```
2. Upgrade cnDBTier by performing a Helm upgrade with the modified custom_values.yaml file. For more information about the upgrade procedure, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
  
  Note:
  cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.
Perform the following steps to update the PVC value:
1. Update the custom_values.yaml file with the new PVC values for ndbmtd pods:
  For example:
  The following code block shows the old PVC values in the custom_values.yaml file:
```
global:
  ndb:
    ndbdisksize: 3Gi
    ndbbackupdisksize: 3Gi
```
  The following code block shows the updated PVC values in the custom_values.yaml file:
```
global:
  ndb:
    ndbdisksize: 6Gi
    ndbbackupdisksize: 6Gi
```
2. Run the following command to delete the ndbmtd statefulset and set the dependency to orphan:
```
$ kubectl -n occne-cndbtier delete sts --cascade=orphan ndbmtd
```
3. Run the following command to delete the ndbmtd-1 pod and patch the pod's PVC values with the values updated in Step a:
```
$ kubectl -n occne-cndbtier delete pod ndbmtd-1
 
$ kubectl -n occne-cndbtier patch -p '{ "spec": { "resources": { "requests": { "storage": "6Gi" }}}}' pvc pvc-ndbmtd-ndbmtd-1
$ kubectl -n occne-cndbtier patch -p '{ "spec": { "resources": { "requests": { "storage": "6Gi" }}}}' pvc pvc-backup-ndbmtd-ndbmtd-1
```
4. After patching the PVC values, wait for the new PV to link to the PVC. Run the following command to check if the PV reflects the updated PVC values:
```
$ kubectl get pv | grep -w occne-cndbtier | grep ndbmtd-1
```
  Sample output:
```
pvc-53fe7988-4a81-40d3-a366-fd894de89535  6Gi  RWO  Delete  Bound  occne-cndbtier/pvc-ndbmtd-ndbmtd-1  occne-dbtier-sc  62m
```
5. Upgrade cnDBTier with the modified custom_values.yaml file:
```
$ helm upgrade mysql-cluster occndbtier -f occndbtier/custom_values.yaml -n occne-cndbtier --no-hooks
```
6. Perform Step b through Step e for the ndbmtd-0 pod.
  
  Note:
  If you have more that two data pods, then follow Steps b to e for each ndbmtd pod in the following order: ndbmtd-n, ndbmtd-(n-1), and so on up to ndbmtd-0.
7. As cnDBTier Helm upgrade is performed with the "--no-hooks" option in step e, upgradeStrategy of the cnDBTier StatefulSets is changed to OnDelete. Therefore, perform the cnDBTier upgrade one more time to restore upgradeStrategy to RollingRestart:
```
$ helm upgrade mysql-cluster occndbtier -f occndbtier/custom_values.yaml -n occne-cndbtier
```
8. Run the Helm test on the cnDBTier cluster to check the health of the cluster:
```
$ helm test mysql-cluster -n occne-cndbtier
```
  Sample output:
```
NAME: mysql-cluster
LAST DEPLOYED: Wed Aug 09 04:43:20 2023
NAMESPACE: occne-cndbtier
STATUS: deployed
REVISION: 4
TEST SUITE:     mysql-cluster-node-connection-test
Last Started:   Wed Aug 09 04:45:15 2023
Last Completed: Wed Aug 09 04:45:35 2023
Phase:          Succeeded
```

7.3.2.4 Vertical Scaling of db-replication-svc

This section provides the procedures to scale up the db-replication-svc pods vertically.

Note:

Before scaling the pods, ensure that the worker nodes have adequate resources to support scaling.
Before you proceed with the vertical scaling of db-replication-svc, perform a Helm test and ensure that all the cnDBTier services are running smoothly.

Perform the following steps to update the CPU and RAM:
1. Configure the required CPU and memory values in the custom_values.yaml file.
  For example:
```
db-replication-svc:
  resources:
    limits:
      cpu: 1
      memory: 2048Mi
      ephemeral-storage: 1Gi
    requests:
      cpu: 0.6
      memory: 1024Mi
      ephemeral-storage: 90Mi
```
2. Upgrade cnDBTier by performing a Helm upgrade with the modified custom_values.yaml file. For more information about the upgrade procedure, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
  
  Note:
  cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.

Perform the following steps to update the PVC value:

Perform the following steps to scale down the replication service pod:

Identify the replication service of cnDBTier cluster1 with respect to cnDBTier cluster2:

$ kubectl get deployment --namespace=cluster1

Sample output:

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-cluster1-cluster2-replication-svc   1/1     1            1           18m
mysql-cluster-db-backup-manager-svc               1/1     1            1           18m
mysql-cluster-db-monitor-svc                      1/1     1            1           18m

Scale down the replication service of cnDBTier cluster1 with respect to cnDBTier cluster2:
```
$ kubectl scale deployment mysql-cluster-cluster1-cluster2-replication-svc --namespace=cluster1 --replicas=0
```
Sample output:
```
deployment.apps/mysql-cluster-cluster1-cluster2-replication-svc scaled
```

Update the custom_values.yaml file with the new PVC values for ndbmysqld pods:
The following example shows the PVC value in the custom_values.yaml file updated from 8Gi to 10Gi:
Old PVC value:
```
db-replication-svc:
  dbreplsvcdeployments:
    - name: cluster1-cluster2-replication-svc
      enabled: true
      pvc:
        name: pvc-cluster1-cluster2-replication-svc
        disksize: 8Gi
```
New PVC value:
```
db-replication-svc:
  dbreplsvcdeployments:
    - name: cluster1-cluster2-replication-svc
      enabled: true
      pvc:
        name: pvc-cluster1-cluster2-replication-svc
        disksize: 10Gi
```
Note:
If you are providing a pod prefix for the DB replication service name, then use a name that is unique and small.

Patch PVCs of the DB replication service with the new PVC value:

$ kubectl -n cluster1 patch -p '{ "spec": { "resources": { "requests": { "storage": "10Gi" }}}}' pvc pvc-cluster1-cluster2-replication-svc

Sample output:

persistentvolumeclaim/pvc-cluster1-cluster2-replication-svc patched

Wait until new PV returns and gets attached to the existing PVC. Run the following commands to check the events:

$ kubectl get events -n cluster1
$ kubectl get pv | grep cluster1 | grep repl

Sample output:

pvc-5b9917ff-53d4-44cb-a5ef-3e37b201c376   8Gi        RWO            Delete           Bound    cluster2/pvc-cluster2-cluster1-replication-svc   occne-dbtier-sc            12m
pvc-e6526259-d007-4868-a4e2-d53a8569edc8   10Gi       RWO            Delete           Bound    cluster1/pvc-cluster1-cluster2-replication-svc   occne-dbtier-sc            15m

Upgrade cnDBTier with the modified custom_values.yaml file:
```
$ helm upgrade mysql-cluster occndbtier -f occndbtier/custom_values.yaml -n cluster1 
```
When the helm upgrade is complete, the db-replication-svc pod comes up with the extended PVC size.

7.4 Configuring cnDBTier SQL Nodes to Support TLS for Georeplication

This chapter describes the manual procedure to configure cnDBTier SQL nodes to support Transport Layer Security (TLS) for georeplication channels between two mate sites.

Note:

TLS Support between NF applications and cnDBTier is not in the current scope of this procedure.
You cannot restore georeplication if TLS is enabled in a cnDBTier setup.

7.4.1 Enabling TLS for Georeplication

Perform the following steps to enable TLS for georeplication between cnDBTier sites.

Create all the required secrets using the "Creating Secret for TLS Certificates" procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Enable TLS feature in the custom_values.yaml file:
```
global:
  tls:    
    enable: true
```

Provide all the required certificates (such as, ca certificate, client certificate, and server certificate) for the respective ndbmysqld pods in the custom_values.yaml file:

tls:
  enable: true
  caCertificate: "<ca certificate file name>"
  tlsversion: "TLSv1.3"
  tlsMode: "<TLS mode>"
  ciphers:
    - TLS_AES_128_GCM_SHA256
    - TLS_AES_256_GCM_SHA384
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_AES_128_CCM_SHA256
  certificates:
    - name: ndbmysqld-0
      serverCertificate: "<server certificate name>"
      serverCertificateKey: "<server key name>"      
      clientCertificate: "<client certificate name>"
      clientCertificateKey: "<client key name>"
    - name: ndbmysqld-1
      serverCertificate: "<server certificate name>"
      serverCertificateKey: "<server key name>"      
      clientCertificate: "<client certificate name>"
      clientCertificateKey: "<client key name>"

For example:

tls:
  enable: true
  caCertificate: "combine-ca.pem"
  tlsversion: "TLSv1.3"
  tlsMode: "VERIFY_CA"
  ciphers:
    - TLS_AES_128_GCM_SHA256
    - TLS_AES_256_GCM_SHA384
    - TLS_CHACHA20_POLY1305_SHA256
    - TLS_AES_128_CCM_SHA256
  certificates:
    - name: ndbmysqld-0
      serverCertificate: "server1-cert.pem"
      serverCertificateKey: "server1-key.pem"      
      clientCertificate: "client1-cert.pem"
      clientCertificateKey: "client1-key.pem"
    - name: ndbmysqld-1
      serverCertificate: "server1-cert.pem"
      serverCertificateKey: "server1-key.pem"      
      clientCertificate: "client1-cert.pem"
      clientCertificateKey: "client1-key.pem"

Follow the installation procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide to perform a Helm install with the updated custom_values.yaml file.

7.4.2 Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites

This section provides the procedure to modify cnDBTier certificates to establish an encrypted connection between georeplication sites using TLS.

Note:

If you update cnDBTier certificates in one site, ensure that you update the relevant Certificate Authority (CA) certificates in all the mate sites. This ensures that the new or switchover replication doesn't break due to incorrect certificates.
While recreating the secrets, ensure that the file names of the updated certificates are same as the old certificates.
Steps 1, 4, and 7 in this procedure are optional. You can skip these steps if you know the list of certificates.

[Optional] Get the list of existing CA certificates of the cnDBTier cluster. You can skip this step if you know the list of certificates.

Note:
If the file name has a dot either in the extension or in its name, use an escape character \ before the dot.
```
$ kubectl get secret  cndbtier-trust-store-secret  -n <cndbtier_namespace>  -o jsonpath="{.data.<ca_certificate_file_name>}" | base64 --decode
```
For example:
```
$ kubectl get secret  cndbtier-trust-store-secret  -n occne-cndbtier -o jsonpath="{.data.ca\.pem}" | base64 --decode
```
If you want to modify the secret of the existing CA certificates in the cnDBTier cluster, delete the secret using the following command. Otherwise, skip this step and move to Step 4.
```
$ kubectl -n <namespace> delete secret cndbtier-trust-store-secret
```
where, <namespace> is the name of the cnDBTier namespace.
For example:
```
$ kubectl -n occne-cndbtier delete secret cndbtier-trust-store-secret
```
If you deleted the secret of the existing CA certificates in the previous step, then create the cndbtier-trust-store-secret secret with the new CA certificates. For more information, see the "Create Secret for TLS Certificates" section in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
```
$ kubectl -n ${OCCNE_NAMESPACE} create secret generic cndbtier-trust-store-secret --from-file=<ca_certificate>
```
For example:
```
$ kubectl -n ${OCCNE_NAMESPACE} create secret generic cndbtier-trust-store-secret --from-file=combine-ca.pem
```
[Optional] Perform the following steps to get the existing server certificate and the server key of the cnDBTier cluster. You can skip this step if you know the list of certificates.

Note:
If the file names have dots either in the extension or in its name, use an escape character \ before the dots.
1. Run the following command to get the server certificate:
```
$ kubectl get secret  cndbtier-server-secret  -n <cndbtier_namespace>  -o jsonpath="{.data.<server_certificate_file_name>}" | base64 --decode
```
  For example:
```
$ kubectl get secret  cndbtier-server-secret  -n occne-cndbtier -o jsonpath="{.data.server-cert\.pem}" | base64 --decode
```
2. Run the following command to get the server certificate key:
```
$ kubectl get secret  cndbtier-server-secret  -n <cndbtier_namespace>  -o jsonpath="{.data.<server_certificate_key_file_name>}" | base64 --decode
```
  For example:
```
$ kubectl get secret  cndbtier-server-secret  -n occne-cndbtier -o jsonpath="{.data.server-key\.pem}" | base64 --decode
```
If you want to modify the secret of the existing server certificate and server key secret, delete the secret using the following command. Otherwise, skip this step and move to Step 7.
```
$ kubectl -n <cndbtier_namespace> delete secret cndbtier-server-secret
```
For example:
```
$ kubectl -n occne-cndbtier delete secret cndbtier-server-secret
```
If you deleted the secret of the existing server certificate and server key in the previous step, then create the cndbtier-server-secret secret with the new server certificate and server key. For more information, see the "Creating Secrets for TLS Certificates" section in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
```
$ kubectl -n ${OCCNE_NAMESPACE} create secret generic cndbtier-server-secret --from-file=<server_certificate> --from-file=<server_certificate_key>
```
For example:
```
$ kubectl -n ${OCCNE_NAMESPACE} create secret generic cndbtier-server-secret --from-file=server-cert.pem --from-file=server-key.pem
```
[Optional] Perform the following steps to get the existing client certificate and client key of the cnDBTier cluster. You can skip this step if you know the list of certificates.

Note:
If the file names have dots either in the extension or in its name, use an escape character \ before the dots.
1. Run the following command to get the client certificate:
```
$ kubectl get secret  cndbtier-client-secret  -n <cndbtier_namespace>  -o jsonpath="{.data.<client_certificate_file_name>}" | base64 --decode
```
  For example:
```
$ kubectl get secret  cndbtier-client-secret  -n occne-cndbtier -o jsonpath="{.data.client-cert\.pem}" | base64 --decode
```
2. Run the following command to get the client certificate key:
```
$ kubectl get secret  cndbtier-client-secret  -n <cndbtier_namespace>  -o jsonpath="{.data.<client_certificate_key_file_name>}" | base64 --decode
```
  For example:
```
$ kubectl get secret  cndbtier-client-secret  -n occne-cndbtier -o jsonpath="{.data.client-key\.pem}" | base64 --decode
```
If you want to modify the secret of the existing client certificate and client key secret, delete the secret using the following command. Otherwise, skip this step and move to Step 10.
```
$ kubectl -n <cndbtier_namespace> delete secret cndbtier-client-secret
```
For example:
```
$ kubectl -n occne-cndbtier delete secret cndbtier-client-secret
```
If you deleted the secret of the existing client certificate and client key in the previous step, then create the cndbtier-client-secret secret with the new client certificate and client key. For more information, see the "Creating Secrets for TLS Certificates" section in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
```
$ kubectl -n ${OCCNE_NAMESPACE} create secret generic cndbtier-client-secret --from-file=<client_certificate> --from-file=<client_certificate_key>
```
For example:
```
$ kubectl -n ${OCCNE_NAMESPACE} create secret generic cndbtier-client-secret --from-file=client-cert.pem --from-file=client-key.pem
```
If you modified the CA certificates in the current site, then you must update the certificates in other sites too. Perform Steps 1 through 9 to modify the relevant certificates on all the mate sites.
Wait for the new certificates of the recreated secret to be automatically mounted to the replication SQL pods by the Kubernetes API server.

Note:
This could take around 30 seconds. However, the wait time depends on the Kubernetes configuration and varies from environment to environment.

Perform the following steps to reload the TLS context on each replication SQL pod in the cnDBTier cluster:

Check the number of replication SQL pods present in the cnDBTier cluster:

$ kubectl get pods --namespace=<namespace of cnDBTier cluster> | grep ndbmysql

For example:

$ kubectl get pods --namespace=occne-cndbtier | grep ndbmysql

Sample output:

ndbmysqld-0                                                    3/3     Running   0          103m
ndbmysqld-1                                                    3/3     Running   0          4h39m
ndbmysqld-2                                                    3/3     Running   0          4h37m
ndbmysqld-3                                                    3/3     Running   0          4h36m
ndbmysqld-4                                                    3/3     Running   0          4h34m
ndbmysqld-5                                                    3/3     Running   0          4h32m

Reload the TLS context on each replication SQL pod:

$ kubectl exec -it <replication SQL pod> --namespace=<namespace of cnDBTier cluster> -- mysql -h127.0.0.1 -uroot -p<root password> -e "ALTER INSTANCE RELOAD TLS;"

For example:

Run the following command to reload the TLS context on ndbmysqld-0:

$ kubectl exec -it ndbmysqld-0 --namespace=occne-cndbtier -- mysql -h127.0.0.1 -uroot -pNextGenCne -e "ALTER INSTANCE RELOAD TLS;"

Sample output:

Defaulted container "mysqlndbcluster" out of: mysqlndbcluster, init-sidecar, db-infra-monitor-svc
mysql: [Warning] Using a password on the command line interface can be insecure.

Run the following command to reload the TLS context on ndbmysqld-1:

$ kubectl exec -it ndbmysqld-1 --namespace=occne-cndbtier -- mysql -h127.0.0.1 -uroot -pNextGenCne -e "ALTER INSTANCE RELOAD TLS;"

Sample output:

Defaulted container "mysqlndbcluster" out of: mysqlndbcluster, init-sidecar, db-infra-monitor-svc
mysql: [Warning] Using a password on the command line interface can be insecure.

Run the following command to reload the TLS context on ndbmysqld-2:

$ kubectl exec -it ndbmysqld-2 --namespace=occne-cndbtier -- mysql -h127.0.0.1 -uroot -pNextGenCne -e "ALTER INSTANCE RELOAD TLS;"

Sample output:

Defaulted container "mysqlndbcluster" out of: mysqlndbcluster, init-sidecar, db-infra-monitor-svc
mysql: [Warning] Using a password on the command line interface can be insecure.

Run the following command to reload the TLS context on ndbmysqld-3:

$ kubectl exec -it ndbmysqld-3 --namespace=occne-cndbtier -- mysql -h127.0.0.1 -uroot -pNextGenCne -e "ALTER INSTANCE RELOAD TLS;"

Sample output:

Defaulted container "mysqlndbcluster" out of: mysqlndbcluster, init-sidecar, db-infra-monitor-svc
mysql: [Warning] Using a password on the command line interface can be insecure.

Run the following command to reload the TLS context on ndbmysqld-4:

$ kubectl exec -it ndbmysqld-4 --namespace=occne-cndbtier -- mysql -h127.0.0.1 -uroot -pNextGenCne -e "ALTER INSTANCE RELOAD TLS;"

Sample output:

Defaulted container "mysqlndbcluster" out of: mysqlndbcluster, init-sidecar, db-infra-monitor-svc
mysql: [Warning] Using a password on the command line interface can be insecure.

Run the following command to reload the TLS context on ndbmysqld-5:

$ kubectl exec -it ndbmysqld-5 --namespace=occne-cndbtier -- mysql -h127.0.0.1 -uroot -pNextGenCne -e "ALTER INSTANCE RELOAD TLS;"

Sample output:

Defaulted container "mysqlndbcluster" out of: mysqlndbcluster, init-sidecar, db-infra-monitor-svc
mysql: [Warning] Using a password on the command line interface can be insecure.

Repeat Step 12 on all the mate sites where you have updated the certificates.
Perform the following steps to run a rollout restart of the ndbmysqld pods in the cnDBTier cluster:
1. Run the following command to get the statefulset name of the ndbmysqld pod:
```
$ kubectl get statefulset --namespace=<namespace of cnDBTier cluster>
```
  For example:
```
$ kubectl get statefulset --namespace=occne-cndbtier
```
  Sample output:
```
NAME           READY   AGE
ndbappmysqld   2/2     27h
ndbmgmd        2/2     27h
ndbmtd         2/2     27h
ndbmysqld      2/2     27h
```
2. Perform a rollout restart of the ndbmsqld pod:
```
$ kubectl rollout restart statefulset <statefulset name of ndbmysqld> --namespace=<namespace of cnDBTier cluster>
```
  For example:
```
$ kubectl rollout restart statefulset ndbmysqld  --namespace=occne-cndbtier
```
  Sample output:
```
statefulset.apps/ndbmysqld restarted
```
3. Wait for the rollout restart of the ndbmysqld pod to complete and check the status:
```
$ kubectl rollout status statefulset <statefulset name of ndbmysqld> --namespace=<namespace of cnDBTier cluster>
```
  For example:
```
$ kubectl rollout status statefulset ndbmysqld  --namespace=occne-cndbtier
```
  Sample output:
```
Waiting for 1 pods to be ready...
waiting for statefulset rolling update to complete 1 pods at revision ndbmysqld-7c6b9c9f84...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
statefulset rolling update complete 2 pods at revision ndbmysqld-7c6b9c9f84...
```
Repeat Step 14 on all the mate sites where you have updated the certificates.

7.5 Adding Georedundant cnDBTier Site

This section describes the procedures to add a cnDBTier georedundant site to the existing single site, two site, and three site georedundant cnDBTier clusters.

Note:

If the existing Georedundant cnDBTier cluster is configured with single replication channel, then the new cnDBTier cluster being added by following this procedure must also be configured with a single replication channel.
If the existing Georedundant cnDBTier cluster is configured with multiple replication channels, then the new cnDBTier cluster being added by following this procedure must also be configured with multiple replication channel groups.
Addition of cnDBTier georedundant site is not supported for cnDBTier setups where TLS is enabled.

The procedures in this section use the following terms to identify the clusters:

cnDBTier Cluster1 (a.k.a. cluster1): The first cloud native based DBTier cluster in a two site, three site, or four site georeplication setup.
cnDBTier Cluster2 (a.k.a. cluster2): The second cloud native based DBTier cluster in a two site, three site, or four site georeplication setup.
cnDBTier Cluster3 (a.k.a. cluster3): The third cloud native based DBTier cluster in a two site, three site, or four site georeplication setup.
cnDBTier Cluster4 (a.k.a. cluster4): The fourth cloud native based DBTier cluster in a two site, three site, or four site georeplication setup.

Prerequisites

All the cnDBTier data nodes and SQL nodes that participate in georeplication, and at least one management node in the existing clusters must be up and running.
Georeplication must be established between the existing cnDBTier clusters.
All the cnDBTier clusters must be installed using cnDBTier 22.1.x or above.
All the cnDBTier clusters must have the same number of data nodes and node groups.

7.5.1 Adding cnDBTier Georedundant Cluster to Single Site cnDBTier Cluster

This section describes the procedure to add a cnDBTier georedundant cluster (cnDBTier cluster2) to an existing single site cnDBTier cluster (cnDBTier cluster1).

If TLS was configured on cnDBTier cluster1, then follow the Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites procedure to update the certificates in cnDBTier cluster1.

Note:
Perform this step on cnDBTier cluster1 only if you want to reconfigure the CA certificates for the new cnDBTier cluster.

Check if georeplication is enabled in cnDBTier cluster1 by performing the following steps:

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then check if georeplication is enabled in cnDBTier cluster1 for every replication group.

Ensure that the DB replication service in cnDBTier cluster1 is enabled and running successfully with respect to cnDBTier cluster2.
```
$ kubectl -n cluster1 get pods | grep repl
```
Sample output:
```
mysql-cluster-cluster1-cluster2-replication-svc-5bdc46crzzhb   1/1     Running   0          3m11s
```

Ensure that at least two georeplication SQL nodes are configured in cluster1.

$ kubectl -n cluster1 get pods | grep ndbmysqld

Sample output:

ndbmysqld-0                    3/3     Running   0          4m14s
ndbmysqld-1                    3/3     Running   0          3m53s

Check the status of cnDBTier cluster1.

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
  
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
  
[mysqld(API)]   8 node(s)
id=56   @10.233.114.34   (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.85.91    (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93    (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

If cluster1 satisfies all the conditions in Step 1, then georeplication is enabled in cnDBTier cluster1. Otherwise, georeplication is not enabled in cluster1.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes. This note is applicable to all similar outputs in this procedure.

If georeplication is not enabled in cluster1 as per Step 1, then perform the following steps to enable georeplication in cluster1. Else, skip to Step 3.

Note:

This step is not used to enable Multi Replication Channel Group on the existing cnDBTier cluster1. Multi replication channel groups with the required number of channel groups must be already configured on cnDBTier cluster1.
Do not run a Helm test when performing a conversion procedure.

Configure the remote mate site name (cluster2) in cnDBTier cluster1 by enabling and configuring the replication service using the custom_values.yaml file.

Example:

$ vi occndbtier/custom_values.yaml

Sample output:

dbreplsvcdeployments:
    # if pod prefix is given then use the unique smaller name for this db replication service.
    - name: cluster1-cluster2-replication-svc
      enabled: true
      .....
      .....
      .....
      replication:
        # Local site replication service LoadBalancer ip can be configured.
        localsiteip: ""
        matesitename: "cluster2"
        remotesiteip: ""

Configure the number of SQL pods in cnDBTier cluster1 to at least two in the custom_values.yaml file.
Example with output:
```
$ vi occndbtier/custom_values.yaml
apiReplicaCount: 2
```

Upgrade cnDBTier cluster1 by using the CSAR package.

$ helm upgrade mysql-cluster occndbtier --namespace cluster1 --no-hooks -f occndbtier/custom_values.yaml

Sample output:

Release "mysql-cluster" has been upgraded. Happy Helming!
NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 11:10:32 2023
NAMESPACE: cluster1
STATUS: deployed
REVISION: 2

Scale down the replication service of cluster1 with respect to cluster2 by performing the following steps:

Identify the replication service of cnDBTier cluster1 with respect to cnDBTier cluster2:

$ kubectl get deployment --namespace=cluster1

Sample output:

NAME                                                READY   UP-TO-DATE   AVAILABLE    AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster1-cluster2-replication-svc      0/1     1            0           10s

Scale down the replication service of cnDBTier cluster1 with respect to cnDBTier cluster2:

$ kubectl scale deployment mysql-cluster-cluster1-cluster2-replication-svc --namespace=cluster1 --replicas=0

Sample output:

deployment.apps/mysql-cluster-cluster1-cluster2-replication-svc scaled

Restart all the management pods of cnDBTier cluster1 simultaneously by performing the following steps:

Identify the list of management pods at cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'mgmd'

Sample output:

ndbmgmd-0                    2/2     Running             0          9m34s
ndbmgmd-1                    2/2     Running             0          9m4s

Delete all the management pods of cnDBTier cluster1 such that the pods restart:
```
$ kubectl delete pod ndbmgmd-0 ndbmgmd-1 --namespace=cluster1
```
Sample output:
```
pod "ndbmgmd-0" deleted
pod "ndbmgmd-1" deleted
```
Wait until the management pods restart and check if the management pods are up and running by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'mgmd'
```
Sample output:
```
ndbmgmd-0                    2/2     Running   0          4m29s
ndbmgmd-1                    2/2     Running   0          4m12s
```

Check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
  
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
  
[mysqld(API)]   8 node(s)
id=56   (not connected, accepting connect from ndbmysqld-0.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=57   (not connected, accepting connect from ndbmysqld-1.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)  
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The SQL pods with id=56 and id=57 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Restart the data pods one after the other in descending order by performing the following steps:

Identify the list of data pods in cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'

Sample output:

ndbmtd-0                                          3/3     Running   0          14m
ndbmtd-1                                          3/3     Running   0          13m

Delete the second data pod of cnDBTier cluster1 (in this case, ndbmtd-1) such that the pod restarts:
```
$ kubectl delete pod ndbmtd-1 --namespace=cluster1
```
Sample output:
```
pod "ndbmtd-1" deleted
```

Wait for the second data pod to restart and check if the second pod is up and running by running the following command:

$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'

Sample output:

ndbmtd-0                                               3/3     Running   0          13m
ndbmtd-1                                               3/3     Running   0          65s

Check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
  
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
  
[mysqld(API)]   8 node(s)
id=56   (not connected, accepting connect from ndbmysqld-0.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)  
id=57   (not connected, accepting connect from ndbmysqld-1.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The SQL pods with id=56 and id=57 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Follow Step ii through Step iv to delete and restart the first data pod of cnDBTier cluster1 (ndbmtd-0).

Scale up the replication service of cluster1 with respect to cluster2 by performing the following steps:

Identify the replication service of cnDBTier cluster1 with respect to cnDBTier cluster2:

$ kubectl get deployment --namespace=cluster1

Sample output:

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster1-cluster2-replication-svc      0/0     0            0           4m10s

Scale up the replication service of cnDBTier cluster1 with respect to cnDBTier cluster2:

$ kubectl scale deployment mysql-cluster-cluster1-cluster2-replication-svc --namespace=cluster1 --replicas=1

Sample output:

deployment.apps/mysql-cluster-cluster1-cluster2-replication-svc scaled

Wait until the replication service of cnDBTier cluster1 with respect to cnDBTier cluster2 is up and running. You can check the status of the replication service by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'repl'
```
Sample output:
```
mysql-cluster-cluster1-cluster2-replication-svc-c574d6cjs5mp   1/1     Running        0        119s
```

Perform the following steps to restart the monitor service pod of cnDBTier cluster1:
1. Identify the monitor service pod at cnDBTier cluster1:
```
$ kubectl get pods --namespace=cluster1 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-4cghv           1/1     Running   0                13h
```
2. Delete the monitor service pod of cnDBTier cluster1:
```
$ kubectl delete pod mysql-cluster-db-monitor-svc-8bf9448b8-4cghv --namespace=cluster1
```
  Sample output:
```
pod "mysql-cluster-db-monitor-svc-8bf9448b8-4cghv" deleted
```
3. Wait until the monitor service pod is up and running. You can check the status of the pod by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-w8pvf           1/1     Running   0                109s
```
Upgrade cnDBTier cluster1 by following the procedures in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Note:
cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.

Install cnDBTier cluster2 by configuring cnDBTier cluster1 as mate site with at least two SQL pods. For information on installing a cnDBTier cluster, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Note:
- If Multi Replication Channel Groups is enabled, then install cnDBTier cluster2 by configuring cnDBTier cluster1 as mate site with at least four SQL pods.
- Do not run a Helm test when performing the conversion procedure.

Check if the georeplication channels are established between cnDBTier cluster1 and cnDBTier cluster2 by running the following commands:

Note:

The following commands and examples are applicable for a single replication channel group only. If multi replication channel groups are enabled, then check if the georeplication channels are established between cnDBTier cluster1 and cnDBTier cluster2 for every replication channel group.

$ kubectl -n cluster1 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster1         |             1001 |       7702 | 10.233.46.41        | STANDBY |        NULL | cluster2  |      2001 | 2023-11-07 19:10:45 |
| cluster1         |             1000 |       7601 | 10.233.63.99        | ACTIVE  |        NULL | cluster2  |      2000 | 2023-11-07 19:10:44 |
| cluster2         |             2000 |       7601 | 10.233.22.173       | ACTIVE  |        NULL | cluster1  |      1000 | 2023-11-07 19:10:44 |
| cluster2         |             2001 |       7702 | 10.233.41.221       | STANDBY |        NULL | cluster1  |      1001 | 2023-11-07 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
4 rows in set (0.01 sec)

Restore the cnDBTier cluster1 data in cnDBTier cluster2 and reestablish the georeplication channels by performing the georeplication recovery procedures in the Georeplication Failure Between cnDBTier Clusters in Two Site Replication section of Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Verify the replication status in cnDBTier cluster1 and cnDBTier cluster2 by performing the following steps:

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then verify the replication status in cnDBTier cluster1 and cnDBTier cluster2 for every replication group.

Log in to the cnDBTier cluster1 Bastion Host and check the replication status in the active replication channel:

$ kubectl -n cluster1 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster1         |             1001 |       7702 | 10.233.39.11        | STANDBY |        NULL | cluster2  |      2001 | 2023-11-07 19:10:45 |
| cluster1         |             1000 |       7601 | 10.233.13.41        | ACTIVE  |        NULL | cluster2  |      2000 | 2023-11-07 19:10:44 |
| cluster2         |             2000 |       7601 | 10.233.0.147        | ACTIVE  |        NULL | cluster1  |      1000 | 2023-11-07 19:10:44 |
| cluster2         |             2001 |       7702 | 10.233.9.35         | STANDBY |        NULL | cluster1  |      1001 | 2023-11-07 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
4 rows in set (0.01 sec)

Check if the replication is turned on (that is, Replica_IO_Running and Replica_SQL_Running are set to Yes) in the ACTIVE replication channel.

Example with output:

$ kubectl -n cluster1 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.0.147
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000007
          Read_Source_Log_Pos: 16442
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....
 
 
$ kubectl -n cluster1 exec -it ndbmysqld-1 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.9.35
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 26970
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No

Log in to the cnDBTier cluster2 Bastion Host and check the replication status in the active replication channel:

$ kubectl -n cluster2 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster1         |             1001 |       7702 | 10.233.39.110       | STANDBY |        NULL | cluster2  |      2001 | 2023-11-07 19:10:45 |
| cluster1         |             1000 |       7601 | 10.233.13.41        | ACTIVE  |        NULL | cluster2  |      2000 | 2023-11-07 19:10:44 |
| cluster2         |             2000 |       7601 | 10.233.0.147        | ACTIVE  |        NULL | cluster1  |      1000 | 2023-11-07 19:10:44 |
| cluster2         |             2001 |       7702 | 10.233.9.35         | STANDBY |        NULL | cluster1  |      1001 | 2023-11-07 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
4 rows in set (0.00 sec)

Check if the replication is turned on (that is, Replica_IO_Running and Replica_SQL_Running are set to Yes) in the ACTIVE replication channel.

Example with output:

$ kubectl -n cluster2 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.13.41
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000007
          Read_Source_Log_Pos: 32792
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 17215
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....
 
 
$ kubectl -n cluster2 exec -it ndbmysqld-1 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.39.110
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000007
          Read_Source_Log_Pos: 9785
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: No
          Replica_SQL_Running: No

Note:

As data is replicated to cnDBTier cluster2 and georeplication channels are established between cnDBTier cluster1 and cnDBTier cluster2, the newly added cnDBTier cluster (cluster2) can also be used by the NFs for database operations.

7.5.2 Adding cnDBTier Georedundant Cluster to Two Site cnDBTier Clusters

This section describes the procedure to add a cnDBTier georedundant cluster (cnDBTier cluster3) to the existing two site georedundant cnDBTier clusters (cnDBTier cluster1 and cnDBTier cluster2).

If TLS was configured on cnDBTier cluster1 and cluster 2, then follow the Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites procedure to update the certificates in cnDBTier cluster1 and cluster 2.

Note:
Perform this step on those cnDBTier clusters on which you want to reconfigure the CA certificates for the new cnDBTier cluster.

Check if georeplication is enabled in cnDBTier cluster1 with respect to cnDBTier cluster3 by performing the following steps:

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then check if georeplication is enabled in cnDBTier cluster1 for every replication group with respect to cnDBTier cluster3.

Ensure that the DB replication service in cnDBTier cluster1 is enabled and running successfully with respect to cnDBTier cluster3:

$ kubectl -n cluster1 get pods | grep repl

Sample output:

mysql-cluster-cluster1-cluster2-replication-svc-5bdc46crzzhb   1/1     Running   0          3m11s
mysql-cluster-cluster1-cluster3-replication-svc-7955b6b6fbdc   1/1     Running   2          3m09s

Ensure that at least four georeplication SQL nodes are configured in cluster1:

$ kubectl -n cluster1 get pods | grep ndbmysqld

Sample output:

ndbmysqld-0                                                       3/3     Running   0          4m14s
ndbmysqld-1                                                       3/3     Running   0          3m53s
ndbmysqld-2                                                       3/3     Running   0          3m21s
ndbmysqld-3                                                       3/3     Running   0          3m01s

Check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
   
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
   
[mysqld(API)]   10 node(s)
id=56   @10.233.114.34  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.85.91  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.114.34  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.85.91  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.
If cluster1 satisfies all the conditions in Step 1, then georeplication is enabled in cnDBTier cluster1. Otherwise, georeplication is not enabled.

If georeplication is not enabled in cluster1 as per Step 1, then perform the following steps to enable georeplication in cnDBTier cluster1 with respect to cnDBTier cluster3. Else, skip to Step 3.

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then configure remote mate site name (cluster3) in cnDBTier cluster1 by following the Configuring Multiple Replication channel groups procedure in \Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide and skip steps a and b in the following substeps.

Configure the remote mate site name (cluster3) in cnDBTier cluster1 by enabling and configuring the replication service using the custom_values.yaml file:

$ vi occndbtier/custom_values.yaml

Sample output:

dbreplsvcdeployments:
    ....
    # if pod prefix is given then use the unique smaller name for this db replication service.
    - name: cluster1-cluster3-replication-svc
      enabled: true
      mysql:
        dbtierservice: "mysql-connectivity-service"
        dbtierreplservice: "ndbmysqldsvc"
        # if cndbtier, use CLUSTER-IP from the ndbmysqldsvc-2 LoadBalancer service
        primaryhost: "ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier"
        port: "3306"
        # if cndbtier, use EXTERNAL-IP from the ndbmysqldsvc-2 LoadBalancer service
        primarysignalhost: ""
        # serverid is unique; retrieve it for the site being configured
        primaryhostserverid: "1002"
        # if cndbtier, use CLUSTER-IP from the ndbmysqldsvc-3 LoadBalancer service
        secondaryhost: "ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier"
        # if cndbtier, use EXTERNAL-IP from the ndbmysqldsvc-3 LoadBalancer service
        secondarysignalhost: ""
        # serverid is unique; retrieve it for the site being configured
        secondaryhostserverid: "1003"
      replication:
        # Local site replication service LoadBalancer ip can be configured.        
        localsiteip: ""
        matesitename: "cluster3"
        remotesiteip: ""

Configure the number of SQL pods in cnDBTier cluster1 to at least four in the custom_values.yaml file:

Note:
apiReplicaCount is set to "4" as you are adding the third site. The first two ndbmysqld are used for replication between site one and site two only. The two ndbmysqld pods that are newly added will be responsible for replication between site1 and site3.
```
$ vi occndbtier/custom_values.yaml
```
Sample output:
```
apiReplicaCount: 4
```
Upgrade cnDBTier cluster1 by using the CSAR package:

Note:
The following command and example are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then upgrade cnDBTier cluster1 by providing appropriate multi group values file.
```
$ helm upgrade mysql-cluster occndbtier --namespace cluster1 --no-hooks -f occndbtier/custom_values.yaml
```
Sample output:
```
Release "mysql-cluster" has been upgraded. Happy Helming!
NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 11:10:32 2023
NAMESPACE: cluster1
STATUS: deployed
REVISION: 2
```
Scale down the replication service of cluster1 with respect to cluster3 by performing the following steps:

Note:
The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then scale down the replication service of cluster1 with respect to cluster3 for every replication channel group.
1. Identify the replication service of cnDBTier cluster1 with respect to cnDBTier cluster3:
```
$ kubectl get deployment --namespace=cluster1
```
  Sample output:
```
NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster1-cluster2-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster1-cluster3-replication-svc      0/1     1            0           10s
```
2. Scale down the replication service of cnDBTier cluster1 with respect to cnDBTier cluster3:
```
$ kubectl scale deployment mysql-cluster-cluster1-cluster3-replication-svc --namespace=cluster1 --replicas=0
```
  Sample output:
```
deployment.apps/mysql-cluster-cluster1-cluster3-replication-svc scaled
```

Restart all the management pods of cnDBTier cluster1 by performing the following steps:

Identify the list of management pods in cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'mgmd'

Sample output:

ndbmgmd-0                                            2/2     Running             0          9m34s
ndbmgmd-1                                            2/2     Running             0          9m4s

Delete all the management pods of cnDBTier cluster1 such that the pods restart:
```
$ kubectl delete pod ndbmgmd-0 ndbmgmd-1 --namespace=cluster1
```
Sample output:
```
pod "ndbmgmd-0" deleted
pod "ndbmgmd-1" deleted
```

Wait until the management pods restart and check if the management pods are up and running by running the following command:

$ kubectl get pods --namespace=cluster1 | grep 'mgmd'

Sample output:

ndbmgmd-0                                              1/1     Running   0          4m29s
ndbmgmd-1                                              1/1     Running   0          4m12s

Check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
   
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
   
[mysqld(API)]   10 node(s)
id=56   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135  (mysql-8.0.35 ndb-8.0.35)
id=58   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=59   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)  
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The SQL pods with id=58 and id=59 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Restart the data pods one after the other in descending order by performing the following steps:

Identify the list of data pods in cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'

Sample output:

ndbmtd-0                                               3/3     Running   0          14m
ndbmtd-1                                               3/3     Running   0          13m

Delete the second data pod of cnDBTier cluster1 (in this case, ndbmtd-1) such that the pod restarts:
```
$ kubectl delete pod ndbmtd-1 --namespace=cluster1
```
Sample output:
```
pod "ndbmtd-1" deleted
```

Wait for the second data pod to restart and check if the second pod is up and running by running the following command:

$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'

Sample output:

ndbmtd-0                                               2/2     Running   0          13m
ndbmtd-1                                               2/2     Running   0          65s

Check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
   
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
   
[mysqld(API)]   10 node(s)
id=56   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135  (mysql-8.0.35 ndb-8.0.35)
id=58   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=59   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)  
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The SQL pods with id=58 and id=59 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Follow Step ii through Step iv to delete and restart the first data pod of cnDBTier cluster1 (ndbmtd-0).

Scale up the replication service of cluster1 with respect to cluster3 by performing the following steps:

Note:
The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then scale up the replication service of cluster1 with respect to cluster3 for every replication channel group.
1. Identify the replication service of cnDBTier cluster1 with respect to cnDBTier cluster3:
```
$ kubectl get deployment --namespace=cluster1
```
  Sample output:
```
NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster1-cluster2-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster1-cluster3-replication-svc      0/0     0            0           4m10s
```
2. Scale up the replication service of cnDBTier cluster1 with respect to cnDBTier cluster3:
```
$ kubectl scale deployment mysql-cluster-cluster1-cluster3-replication-svc --namespace=cluster1 --replicas=1
```
  Sample output:
```
deployment.apps/mysql-cluster-cluster1-cluster3-replication-svc scaled
```
3. Wait until the replication service of cnDBTier cluster1 with respect to cnDBTier cluster3 is up and running. You can check the status of the replication service by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'repl'
```
  Sample output:
```
mysql-cluster-cluster1-cluster3-replication-svc-c574d6cjs5mp   1/1     Running           0          119s
```
Perform the following steps to restart the monitor service pod of cnDBTier cluster1:
1. Identify the monitor service pod at cnDBTier cluster1:
```
$ kubectl get pods --namespace=cluster1 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-4cghv           1/1     Running   0                13h
```
2. Delete the monitor service pod of cnDBTier cluster1:
```
$ kubectl delete pod mysql-cluster-db-monitor-svc-8bf9448b8-4cghv --namespace=cluster1
```
  Sample output:
```
pod "mysql-cluster-db-monitor-svc-8bf9448b8-4cghv" deleted
```
3. Wait until the monitor service pod is up and running. You can check the status of the pod by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-w8pvf           1/1     Running   0                109s
```
Upgrade cnDBTier cluster1 by following the procedures in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Note:
cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.

Check if georeplication is enabled in cnDBTier cluster2 with respect to cnDBTier cluster3 by performing the following steps:

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then check if georeplication is enabled in cnDBTier cluster2 for every replication group with respect to cnDBTier cluster3.

Ensure that the DB replication service in cnDBTier cluster2 is enabled and running successfully with respect to cnDBTier cluster3:

$ kubectl -n cluster2 get pods | grep repl

Sample output:

mysql-cluster-cluster2-cluster1-replication-svc-5bdc46crzzhb   1/1     Running   0          3m11s
mysql-cluster-cluster2-cluster3-replication-svc-7955b6b6fbdc   1/1     Running   2          3m39s

Ensure that at least four georeplication SQL nodes are configured in cluster2:

$ kubectl -n cluster2 get pods | grep ndbmysqld

Sample output:

ndbmysqld-0                                                       3/3     Running   0          4m14s
ndbmysqld-1                                                       3/3     Running   0          3m53s
ndbmysqld-2                                                       3/3     Running   0          3m21s
ndbmysqld-3                                                       3/3     Running   0          3m01s

Check the status of cnDBTier cluster2:

$ kubectl -n cluster2 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
   
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
   
[mysqld(API)]   10 node(s)
id=56   @10.233.114.34  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.85.91  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.114.34  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.85.91  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

If cluster2 satisfies all the conditions in Step 3, then georeplication is enabled in cnDBTier cluster2. Otherwise, georeplication is not enabled in cluster2.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

If georeplication is not enabled in cluster2 as per Step 3, then perform the following steps to enable georeplication in cluster2 with respect to cluster3. Else, skip to Step 5.

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then configure remote mate site name (cluster3) in cnDBTier cluster2 by following the Configuring Multiple Replication channel groups procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide and skip steps a and b in the following substeps.

Configure the remote mate site name (cluster3) in cnDBTier cluster2 by enabling and configuring the replication service using the custom_values.yaml file:

$ vi occndbtier/custom_values.yaml

Sample output:

 dbreplsvcdeployments:
    ....
    # if pod prefix is given then use the unique smaller name for this db replication service.
    - name: cluster2-cluster3-replication-svc
      enabled: true
      mysql:
        dbtierservice: "mysql-connectivity-service"
        dbtierreplservice: "ndbmysqldsvc"
        # if cndbtier, use CLUSTER-IP from the ndbmysqldsvc-2 LoadBalancer service
        primaryhost: "ndbmysqld-2.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier"
        port: "3306"
        # if cndbtier, use EXTERNAL-IP from the ndbmysqldsvc-2 LoadBalancer service
        primarysignalhost: ""
        # serverid is unique; retrieve it for the site being configured
        primaryhostserverid: "2002"
        # if cndbtier, use CLUSTER-IP from the ndbmysqldsvc-3 LoadBalancer service
        secondaryhost: "ndbmysqld-3.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier"
        # if cndbtier, use EXTERNAL-IP from the ndbmysqldsvc-3 LoadBalancer service
        secondarysignalhost: ""
        # serverid is unique; retrieve it for the site being configured
        secondaryhostserverid: "2003"
      replication:
        # Local site replication service LoadBalancer ip can be configured.
        localsiteip: ""
        matesitename: "cluster3"
        remotesiteip: ""

Configure the number of SQL pods in cnDBTier cluster2 to at least four in the custom_values.yaml file:

Note:
apiReplicaCount is set to "4" as you are adding the third site. The first two ndbmysqld are used for replication between site one and site two only. The two ndbmysqld pods that are newly added will be responsible for replication between site1 and site3.
```
$ vi occndbtier/custom_values.yaml
```
Sample output:
```
apiReplicaCount: 4
```
Upgrade cnDBTier cluster2 by using the CSAR package:

Note:
The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then upgrade cnDBTier cluster2 by providing appropriate multi group values file.
```
$ helm upgrade mysql-cluster occndbtier --namespace cluster2 --no-hooks -f occndbtier/custom_values.yaml
```
Sample output:
```
Release "mysql-cluster" has been upgraded. Happy Helming!
NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 11:10:32 2023
NAMESPACE: cluster2
STATUS: deployed
REVISION: 2
```
Scale down the replication service of cluster2 with respect to cluster3 by performing the following steps:

Note:
The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then scale down the replication service of cluster2 with respect to cluster3 for every replication channel group.
1. Identify the replication service of cnDBTier cluster2 with respect to cnDBTier cluster3:
```
$ kubectl get deployment --namespace=cluster2
```
  Sample output:
```
NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster2-cluster1-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster2-cluster3-replication-svc      0/1     1            0           10s
```
2. Scale down the replication service of cnDBTier cluster2 with respect to cnDBTier cluster3:
```
$ kubectl scale deployment mysql-cluster-cluster2-cluster3-replication-svc --namespace=cluster2 --replicas=0
```
  Sample output:
```
deployment.apps/mysql-cluster-cluster2-cluster3-replication-svc scaled
```

Restart all the management pods of cnDBTier cluster2 simultaneously by performing the following steps:

Identify the list of management pods at cnDBTier cluster2:

$ kubectl get pods --namespace=cluster2 | grep 'mgmd'

Sample output:

ndbmgmd-0                        2/2     Running             0          9m34s
ndbmgmd-1                        2/2     Running             0          9m4s

Delete all the management pods of cnDBTier cluster2 such that the pods restart:
```
$ kubectl delete pod ndbmgmd-0 ndbmgmd-1 --namespace=cluster2
```
Sample output:
```
pod "ndbmgmd-0" deleted
pod "ndbmgmd-1" deleted
```

Wait until the management pods restart and check if the management pods are up and running by running the following command:

$ kubectl get pods --namespace=cluster2 | grep 'mgmd'

Sample output:

ndbmgmd-0                               2/2     Running   0          4m29s
ndbmgmd-1                               2/2     Running   0          4m12s

Check the status of cnDBTier cluster2:

$ kubectl -n cluster2 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   10 node(s)
id=56   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135  (mysql-8.0.35 ndb-8.0.35)
id=58   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier)
id=59   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier) 
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The SQL pods with id=58 and id=59 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Restart the data pods one after the other in descending order by performing the following steps:

Identify the list of data pods in cnDBTier cluster2:

$ kubectl get pods --namespace=cluster2 | grep 'ndbmtd'

Sample output:

ndbmtd-0                             3/3     Running   0          14m
ndbmtd-1                             3/3     Running   0          13m

Delete the second data pod of cnDBTier cluster2 (in this case, ndbmtd-1) such that the pod restarts:
```
$ kubectl delete pod ndbmtd-1 --namespace=cluster2
```
Sample output:
```
pod "ndbmtd-1" deleted
```

Wait for the second data pod to restart and check if the second pod is up and running by running the following command:

$ kubectl get pods --namespace=cluster2 | grep 'ndbmtd'

Sample output:

ndbmtd-0                                2/2     Running   0          13m
ndbmtd-1                                2/2     Running   0          65s

Check the status of cnDBTier cluster2:

$ kubectl -n cluster2 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   10 node(s)
id=56   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135  (mysql-8.0.35 ndb-8.0.35)
id=58   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier)
id=59   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier) 
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The SQL pods with id=58 and id=59 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Follow Step ii through Step iv to delete and restart the first data pod of cnDBTier cluster2 (ndbmtd-1).

Scale up the replication service of cluster2 with respect to cluster3 by performing the following steps:

Note:
The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then scale up the replication service of cluster2 with respect to cluster3 for every replication channel group.
1. Identify the replication service of cnDBTier cluster2 with respect to cnDBTier cluster3:
```
$ kubectl get deployment --namespace=cluster2
```
  Sample output:
```
NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster2-cluster1-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster2-cluster3-replication-svc      0/0     0            0           4m10s
```
2. Scale up the replication service of cnDBTier cluster2 with respect to cnDBTier cluster3:
```
$ kubectl scale deployment mysql-cluster-cluster2-cluster3-replication-svc --namespace=cluster2 --replicas=1
```
  Sample output:
```
deployment.apps/mysql-cluster-cluster2-cluster3-replication-svc scaled
```
3. Wait until the replication service of cnDBTier cluster2 with respect to cnDBTier cluster3 is up and running. You can check the status of the replication service by running the following command:
```
$ kubectl get pods --namespace=cluster2 | grep 'repl'
```
  Sample output:
```
mysql-cluster-cluster2-cluster3-replication-svc-c574d6cjs5mp   1/1     Running           0          119s
```
Perform the following steps to restart the monitor service pod of cnDBTier cluster2:
1. Identify the monitor service pod at cnDBTier cluster2:
```
$ kubectl get pods --namespace=cluster2 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-4cghv           1/1     Running   0                13h
```
2. Delete the monitor service pod of cnDBTier cluster2:
```
$ kubectl delete pod mysql-cluster-db-monitor-svc-8bf9448b8-4cghv --namespace=cluster2
```
  Sample output:
```
pod "mysql-cluster-db-monitor-svc-8bf9448b8-4cghv" deleted
```
3. Wait until the monitor service pod is up and running. You can check the status of the pod by running the following command:
```
$ kubectl get pods --namespace=cluster2 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-w8pvf           1/1     Running   0                109s
```
Upgrade cnDBTier cluster2 by following the procedures in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Note:
cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.

Install cnDBTier cluster3 by configuring cnDBTier cluster1 as the first mate site and cnDBTier cluster2 as the second mate site with at least four SQL pods. For information on installing the cnDBTier cluster, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Note:
- If multiple replication channel groups are enabled, then install cnDBTier cluster3 by configuring cnDBTier cluster1 as first mate site and cnDBTier cluster2 as second mate site with at least eight SQL pods.
- Do not run a Helm test when performing the conversion procedure.
Check if georeplication channels are established between cnDBTier cluster3 and cnDBTier cluster1, and cnDBTier cluster3 and cnDBTier cluster2 by following the procedures described in the Checking Georeplication Status Between Clusters section.
Restore the data of cnDBTier cluster1 or cnDBTier cluster2 in cnDBTier cluster3 and re-establish the georeplication channels between cnDBTier cluster1 and cnDBTier cluster3, and cnDBTier cluster2 and cnDBTier cluster3. For procedures on restoring the cluster data and re-establishing the georeplication channels, refer to the Georeplication Failure between cnDBTier Clusters in Three Site Replication section of Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Check if georeplication channels are established between cnDBTier cluster3 and cnDBTier cluster1, and cnDBTier cluster3 and cnDBTier cluster2 by following the procedures described in the Checking Georeplication Status Between Clusters section.

Note:
As data is replicated to cnDBTier cluster3 and georeplication channels are established between cnDBTier cluster1 and cnDBTier cluster3, and cnDBTier cluster2 and cnDBTier cluster3, the newly added cnDBTier cluster (cnDBTier cluster3) can also be used by the NFs for database operations.

7.5.3 Adding cnDBTier Georedundant Cluster to Three Site cnDBTier Clusters

This section describes the procedure to add a cnDBTier georedundant cnDBTier cluster4 to the existing three site cnDBTier clusters (cnDBTier cluster1, cnDBTier cluster2, and cnDBTier cluster3).

If TLS was configured on cnDBTier cluster 1, cluster 2, and cluster 3, then follow the Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites procedure to update the certificates in cnDBTier cluster 1, cluster 2, and cluster 3.

Note:
Perform this step on those cnDBTier clusters on which you want to reconfigure the CA certificates for the new cnDBTier cluster.

Check if georeplication is enabled in cnDBTier cluster1 with respect to cnDBTier cluster4 by performing the following steps:

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then check if georeplication is enabled in cnDBTier cluster1 for every replication group with respect to cnDBTier cluster4.

Ensure that the DB replication service in cnDBTier cluster1 is enabled and running successfully with respect to cnDBTier cluster4:

$ kubectl -n cluster1 get pods | grep repl

Sample output:

mysql-cluster-cluster1-cluster2-replication-svc-5bdc46crzzhb   1/1     Running   0          3m11s
mysql-cluster-cluster1-cluster3-replication-svc-7955b6b6fbdc   1/1     Running   2          3m39s
mysql-cluster-cluster1-cluster4-replication-svc-bd977fc5bnd5   1/1     Running   2          3m11s

Ensure that at least six georeplication SQL nodes are configured in cluster1:

$ kubectl -n cluster1 get pods | grep ndbmysqld

Sample output:

ndbmysqld-0                                                       3/3     Running   0          4m14s
ndbmysqld-1                                                       3/3     Running   0          3m53s
ndbmysqld-2                                                       3/3     Running   0          3m21s
ndbmysqld-3                                                       3/3     Running   0          3m01s
ndbmysqld-4                                                       3/3     Running   0          2m48s
ndbmysqld-5                                                       3/3     Running   0          2m31s

Check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   12 node(s)
id=56   @10.233.114.34   (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.85.91    (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.114.34   (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.85.91    (mysql-8.0.35 ndb-8.0.35)
id=60   @10.233.62.226   (mysql-8.0.35 ndb-8.0.35)
id=61   @10.233.14.15    (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93    (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

If cluster1 satisfies all the conditions in Step 1, then georeplication is enabled in cnDBTier cluster1. Otherwise, georeplication is not enabled.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes. This note is applicable to all similar outputs in this procedure.

If georeplication is not enabled in cluster1 as per Step 1, then perform the following steps to enable georeplication in cnDBTier cluster1 with respect to cnDBTier cluster4. Else, skip to Step 3.

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then configure remote mate site name (cluster4) in cnDBTier cluster1 by following the Configuring Multiple Replication channel groups procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide and skip steps a and b in the following substeps.

Configure the remote mate site name (cluster4) in cnDBTier cluster1 by enabling and configuring the replication service using the custom_values.yaml file:

$ vi occndbtier/custom_values.yaml

Sample output:

dbreplsvcdeployments:
    ....
    # if pod prefix is given then use the unique smaller name for this db replication service.
    - name: cluster1-cluster4-replication-svc
      # Local site replication service LoadBalancer ip can be configured.
      enabled: true
      mysql:
        dbtierservice: "mysql-connectivity-service"
        dbtierreplservice: "ndbmysqldsvc"
        # if cndbtier, use CLUSTER-IP from the ndbmysqldsvc-4 LoadBalancer service
        primaryhost: "ndbmysqld-4.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier"
        port: "3306"
        # if cndbtier, use EXTERNAL-IP from the ndbmysqldsvc-4 LoadBalancer service
        primarysignalhost: ""
        # serverid is unique; retrieve it for the site being configured
        primaryhostserverid: "1004"
        # if cndbtier, use CLUSTER-IP from the ndbmysqldsvc-5 LoadBalancer service
        secondaryhost: "ndbmysqld-5.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier"
        # if cndbtier, use EXTERNAL-IP from the ndbmysqldsvc-5 LoadBalancer service
        secondarysignalhost: ""
        # serverid is unique; retrieve it for the site being configured
        secondaryhostserverid: "1005"
      replication:
        localsiteip: ""
        matesitename: "cluster4"
        remotesiteip: ""

Configure the number of SQL pods in cnDBTier cluster1 to at least six in the custom_values.yaml:

Note:
apiReplicaCount is set to "6" as you are adding the third site. The first four ndbmysqld are used for replication between site one, site two, and site three only. The two ndbmysqld pods that are newly added will be responsible for replication between site1 and site4.
```
$ vi occndbtier/custom_values.yaml
```
Sample output:
```
apiReplicaCount: 6
```
Upgrade cnDBTier cluster1 by using the CSAR package:

Note:
The following command and example are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then upgrade cnDBTier cluster1 by providing appropriate multi group values file.
```
$ helm upgrade mysql-cluster occndbtier --namespace cluster1 --no-hooks -f occndbtier/custom_values.yaml
```
Sample output:
```
Release "mysql-cluster" has been upgraded. Happy Helming!
NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 11:10:32 2023
NAMESPACE: cluster1
STATUS: deployed
REVISION: 2
```

Scale down the replication service of cluster1 with respect to cluster4 by performing the following steps:

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then scale down the replication service of cluster1 with respect to cluster4 for every replication channel group.

Identify the replication service of cnDBTier cluster1 with respect to cnDBTier cluster4:

$ kubectl get deployment --namespace=cluster1

Sample output:

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster1-cluster2-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster1-cluster3-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster1-cluster4-replication-svc      0/1     1            0           10s

Scale down the replication service of cnDBTier cluster1 with respect to cnDBTier cluster4:

$ kubectl scale deployment mysql-cluster-cluster1-cluster4-replication-svc --namespace=cluster1 --replicas=0

Sample output:

deployment.apps/mysql-cluster-cluster1-cluster4-replication-svc scaled

Restart all the management pods of cnDBTier cluster1 by performing the following steps:

Identify the list of management pods in cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'mgmd'

Sample output:

ndbmgmd-0                        2/2     Running             0          9m34s
ndbmgmd-1                        2/2     Running             0          9m4s

Delete all the management pods of cnDBTier cluster1 such that the pods restart:
```
$ kubectl delete pod ndbmgmd-0 ndbmgmd-1 --namespace=cluster1
```
Sample output:
```
pod "ndbmgmd-0" deleted
pod "ndbmgmd-1" deleted
```
Wait until the management pods restart and check if the management pods are up and running by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'mgmd'
```
Sample output:
```
ndbmgmd-0                          2/2     Running   0          4m29s
ndbmgmd-1                          2/2     Running   0          4m12s
```

Check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   12 node(s)
id=56   @10.233.124.92    (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135   (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.114.34    (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.85.91     (mysql-8.0.35 ndb-8.0.35)
id=60   (not connected, accepting connect from ndbmysqld-4.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=61   (not connected, accepting connect from ndbmysqld-5.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)   
id=70   @10.233.127.117   (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93     (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The SQL pods with id=60 and id=61 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Restart the data pods one after the other in descending order by performing the following steps:

Identify the list of data pods in cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'

Sample output:

ndbmtd-0                           3/3     Running   0          14m
ndbmtd-1                           3/3      Running   0          13m

Delete the second data pod of cnDBTier cluster1 (in this case, ndbmtd-1) such that the pod restarts:
```
$ kubectl delete pod ndbmtd-1 --namespace=cluster1
```
Sample output:
```
pod "ndbmtd-1" deleted
```
Wait for the second data pod to restart and check if the second pod is up and running by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'
```
Sample output:
```
ndbmtd-0                           3/3     Running   0          13m
ndbmtd-1                           3/3     Running   0          65s
```

Check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   12 node(s)
id=56   @10.233.124.92    (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135   (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.114.34    (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.85.91     (mysql-8.0.35 ndb-8.0.35)
id=60   (not connected, accepting connect from ndbmysqld-4.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=61   (not connected, accepting connect from ndbmysqld-5.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)   
id=70   @10.233.127.117   (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93     (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The SQL pods with id=60 and id=61 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Follow Step ii through Step iv to delete and restart the first data pod of cnDBTier cluster1 (ndbmtd-1).

Scale up the replication service of cluster1 with respect to cluster4 by performing the following steps:

Note:
The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then scale up the replication service of cluster1 with respect to cluster4 for every replication channel group.
1. Identify the replication service of cnDBTier cluster1 with respect to cnDBTier cluster4:
```
$ kubectl get deployment --namespace=cluster1
```
  Sample output:
```
NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster1-cluster2-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster1-cluster3-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster1-cluster4-replication-svc      0/0     0            0           4m10s
```
2. Scale up the replication service of cnDBTier cluster1 with respect to cnDBTier cluster4:
```
$ kubectl scale deployment mysql-cluster-cluster1-cluster4-replication-svc --namespace=cluster1 --replicas=1
```
  Sample output:
```
deployment.apps/mysql-cluster-cluster1-cluster4-replication-svc scaled
```
3. Wait until the replication service of cnDBTier cluster1 with respect to cnDBTier cluster4 is up and running. You can check the status of the replication service by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'repl'
```
  Sample output:
```
mysql-cluster-cluster1-cluster4-replication-svc-c574d6cjs5mp   1/1     Running           0          119s
```
Perform the following steps to restart the monitor service pod of cnDBTier cluster1:
1. Identify the monitor service pod at cnDBTier cluster1:
```
$ kubectl get pods --namespace=cluster1 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-4cghv           1/1     Running   0                13h
```
2. Delete the monitor service pod of cnDBTier cluster1:
```
$ kubectl delete pod mysql-cluster-db-monitor-svc-8bf9448b8-4cghv --namespace=cluster1
```
  Sample output:
```
pod "mysql-cluster-db-monitor-svc-8bf9448b8-4cghv" deleted
```
3. Wait until the monitor service pod is up and running. You can check the status of the pod by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-w8pvf           1/1     Running   0                109s
```
Upgrade cnDBTier cluster1 by following the procedures in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Note:
cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.

Check if georeplication is enabled in cnDBTier cluster2 with respect to cnDBTier cluster4 by performing the following steps:

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then check if georeplication is enabled in cnDBTier cluster2 for every replication group with respect to cnDBTier cluster4.

Ensure that the DB replication service in cnDBTier cluster2 is enabled and running successfully with respect to cnDBTier cluster4:

$ kubectl -n cluster2 get pods | grep repl

Sample output:

mysql-cluster-cluster2-cluster1-replication-svc-5bdc46crzzhb   1/1     Running   0          3m11s
mysql-cluster-cluster2-cluster3-replication-svc-7955b6b6fbdc   1/1     Running   2          3m39s
mysql-cluster-cluster2-cluster4-replication-svc--556c99fcdzh   1/1     Running   2          3m05s

Ensure that at least six georeplication SQL nodes are configured in cluster2:

Note:

apiReplicaCount is set to "6" as you are adding the third site. The first four ndbmysqld are used for replication between site one, site two, and site three only. The two ndbmysqld pods that are newly added will be responsible for replication between site1 and site4.

$ kubectl -n cluster2 get pods | grep ndbmysqld

Sample output:

ndbmysqld-0                                                       3/3     Running   0          4m14s
ndbmysqld-1                                                       3/3     Running   0          3m53s
ndbmysqld-2                                                       3/3     Running   0          3m21s
ndbmysqld-3                                                       3/3     Running   0          3m01s
ndbmysqld-4                                                       3/3     Running   0          2m59s
ndbmysqld-5                                                       3/3     Running   0          2m43s

Check the status of cnDBTier cluster2:

$ kubectl -n cluster2 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   12 node(s)
id=56   @10.233.114.34    (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.85.91     (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.114.34    (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.85.91     (mysql-8.0.35 ndb-8.0.35)
id=60   @10.233.24.121    (mysql-8.0.35 ndb-8.0.35)
id=61   @10.233.85.22     (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.127.117   (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93     (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.
If cluster2 satisfies all the conditions in Step 3, then georeplication is enabled in cnDBTier cluster2. Otherwise, georeplication is not enabled in cluster2.

If georeplication is not enabled in cluster2 as per Step 3, then perform the following steps to enable georeplication in cluster2 with respect to cluster4. Else, skip to Step 5.

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then configure remote mate site name (cluster4) in cnDBTier cluster2 by following the Configuring Multiple Replication channel groups procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide and skip steps a and b in the following substeps.

Configure the remote mate site name (cluster4) in cnDBTier cluster2 by enabling and configuring the replication service using the custom_values.yaml file:

$ vi occndbtier/custom_values.yaml

Sample output:

dbreplsvcdeployments:
    ....
    # if pod prefix is given then use the unique smaller name for this db replication service.
    - name: cluster2-cluster4-replication-svc
      enabled: true
      mysql:
        dbtierservice: "mysql-connectivity-service"
        dbtierreplservice: "ndbmysqldsvc"
        # if cndbtier, use CLUSTER-IP from the ndbmysqldsvc-4 LoadBalancer service
        primaryhost: "ndbmysqld-4.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier"
        port: "3306"
        # if cndbtier, use EXTERNAL-IP from the ndbmysqldsvc-4 LoadBalancer service
        primarysignalhost: ""
        # serverid is unique; retrieve it for the site being configured
        primaryhostserverid: "2004"
        # if cndbtier, use CLUSTER-IP from the ndbmysqldsvc-5 LoadBalancer service
        secondaryhost: "ndbmysqld-5.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier"
        # if cndbtier, use EXTERNAL-IP from the ndbmysqldsvc-5 LoadBalancer service
        secondarysignalhost: ""
        # serverid is unique; retrieve it for the site being configured
        secondaryhostserverid: "2005"
      replication:
        # Local site replication service LoadBalancer ip can be configured.
        localsiteip: ""
        matesitename: "cluster4"
        remotesiteip: ""

Configure the number of SQL pods in cnDBTier cluster2 to at least six in the custom_values.yaml file:

Note:
apiReplicaCount is set to "6" as you are adding the fourth site. The first four ndbmysqld are used for replication between site one, site two, and site three only. The two ndbmysqld pods that are newly added will be responsible for replication between site1 and site4.
```
$ vi occndbtier/custom_values.yaml
```
Sample output:
```
apiReplicaCount: 6
```
Upgrade cnDBTier cluster2 by using the CSAR package:

Note:
The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then upgrade cnDBTier cluster2 by providing appropriate multi group values file.
```
$ helm upgrade mysql-cluster occndbtier --namespace cluster2 --no-hooks -f occndbtier/custom_values.yaml
```
Sample output:
```
Release "mysql-cluster" has been upgraded. Happy Helming!
NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 11:10:32 2023
NAMESPACE: cluster2
STATUS: deployed
REVISION: 2
```

Scale down the replication service of cluster2 with respect to cluster4 by performing the following steps:

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then scale down the replication service of cluster2 with respect to cluster4 for every replication channel group.

Identify the replication service of cnDBTier cluster2 with respect to cnDBTier cluster4:

$ kubectl get deployment --namespace=cluster2

Sample output:

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster2-cluster1-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster2-cluster3-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster2-cluster4-replication-svc      0/1     1            0           10s

Scale down the replication service of cnDBTier cluster2 with respect to cnDBTier cluster4:

$ kubectl scale deployment mysql-cluster-cluster2-cluster4-replication-svc --namespace=cluster2 --replicas=0

Sample output:

deployment.apps/mysql-cluster-cluster2-cluster4-replication-svc scaled

Restart all the management pods of cnDBTier cluster2 simultaneously by performing the following steps:

Identify the list of management pods at cnDBTier cluster2:

$ kubectl get pods --namespace=cluster2 | grep 'mgmd'

Sample output:

ndbmgmd-0                        2/2     Running             0          9m34s
ndbmgmd-1                        2/2     Running             0          9m4s

Delete all the management pods of cnDBTier cluster2 such that the pods restart:
```
$ kubectl delete pod ndbmgmd-0 ndbmgmd-1 --namespace=cluster2
```
Sample output:
```
pod "ndbmgmd-0" deleted
pod "ndbmgmd-1" deleted
```
Wait until the management pods restart and check if the management pods are up and running by running the following command:
```
$ kubectl get pods --namespace=cluster2 | grep 'mgmd'
```
Sample output:
```
ndbmgmd-0                          2/2     Running   0          4m29s
ndbmgmd-1                          2/2     Running   0          4m12s
```

Check the status of cnDBTier cluster2:

$ kubectl -n cluster2 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
     
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
     
[mysqld(API)]   12 node(s)
id=56   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.124.111  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.110.81  (mysql-8.0.35 ndb-8.0.35)
id=60   (not connected, accepting connect from ndbmysqld-4.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier)
id=61   (not connected, accepting connect from ndbmysqld-5.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The SQL pods with id=60 and id=61 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Restart the data pods one after the other in descending order by performing the following steps:

Identify the list of data pods in cnDBTier cluster2:

$ kubectl get pods --namespace=cluster2 | grep 'ndbmtd'

Sample output:

ndbmtd-0                           3/3     Running   0          14m
ndbmtd-1                           3/3     Running   0          13m

Delete the second data pod of cnDBTier cluster2 (in this case, ndbmtd-1) such that the pod restarts:
```
$ kubectl delete pod ndbmtd-1 --namespace=cluster2
```
Sample output:
```
pod "ndbmtd-1" deleted
```
Wait for the second data pod to restart and check if the second pod is up and running by running the following command:
```
$ kubectl get pods --namespace=cluster2 | grep 'ndbmtd'
```
Sample output:
```
ndbmtd-0                           3/3     Running   0          13m
ndbmtd-1                           3/3     Running   0          65s
```

Check the status of cnDBTier cluster2:

$ kubectl -n cluster2 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
     
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
     
[mysqld(API)]   12 node(s)
id=56   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.124.111  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.110.81  (mysql-8.0.35 ndb-8.0.35)
id=60   (not connected, accepting connect from ndbmysqld-4.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier)
id=61   (not connected, accepting connect from ndbmysqld-5.ndbmysqldsvc.cluster2.svc.occne4-cgbu-cne-dbtier)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The SQL pods with id=60 and id=61 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Follow Step ii through Step iv to delete and restart the first data pod of cnDBTier cluster2 (ndbmtd-1).

Scale up the replication service of cluster2 with respect to cluster4 by performing the following steps:

Note:
The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then scale up the replication service of cluster2 with respect to cluster4 for every replication channel group.
1. Identify the replication service of cnDBTier cluster2 with respect to cnDBTier cluster4:
```
$ kubectl get deployment --namespace=cluster2
```
  Sample output:
```
NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster2-cluster1-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster2-cluster3-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster2-cluster4-replication-svc      0/0     0            0           4m10s
```
2. Scale up the replication service of cnDBTier cluster2 with respect to cnDBTier cluster4:
```
$ kubectl scale deployment mysql-cluster-cluster2-cluster4-replication-svc --namespace=cluster2 --replicas=1
```
  Sample output:
```
deployment.apps/mysql-cluster-cluster2-cluster4-replication-svc scaled
```
3. Wait until the replication service of cnDBTier cluster2 with respect to cnDBTier cluster4 is up and running. You can check the status of the replication service by running the following command:
```
$ kubectl get pods --namespace=cluster2 | grep 'repl'
```
  Sample output:
```
mysql-cluster-cluster2-cluster4-replication-svc-c574d6cjs5mp   1/1     Running           0          119s
```
Perform the following steps to restart the monitor service pod of cnDBTier cluster2:
1. Identify the monitor service pod at cnDBTier cluster1:
```
$ kubectl get pods --namespace=cluster2 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-4cghv           1/1     Running   0                13h
```
2. Delete the monitor service pod of cnDBTier cluster2:
```
$ kubectl delete pod mysql-cluster-db-monitor-svc-8bf9448b8-4cghv --namespace=cluster2
```
  Sample output:
```
pod "mysql-cluster-db-monitor-svc-8bf9448b8-4cghv" deleted
```
3. Wait until the monitor service pod is up and running. You can check the status of the pod by running the following command:
```
$ kubectl get pods --namespace=cluster2 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-w8pvf           1/1     Running   0                109s
```
Upgrade cnDBTier cluster2 by following the procedures in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Note:
cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.

Check if georeplication is enabled in cnDBTier cluster3 with respect to cnDBTier cluster4 by performing the following steps:

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then check if georeplication is enabled in cnDBTier cluster3 for every replication group with respect to cnDBTier cluster4.

Ensure that the DB replication service in cnDBTier cluster3 is enabled and running successfully with respect to cnDBTier cluster4:

$ kubectl -n cluster3 get pods | grep repl

Sample output:

mysql-cluster-cluster3-cluster1-replication-svc-5bdc46crzzhb   1/1     Running   0          3m11s
mysql-cluster-cluster3-cluster2-replication-svc-7955b6b6fbdc   1/1     Running   2          3m39s
mysql-cluster-cluster3-cluster4-replication-svc--556c99fcdzh   1/1     Running   2          3m05s

Ensure that at least six georeplication SQL nodes are configured in cluster3:

$ kubectl -n cluster3 get pods | grep ndbmysqld

Sample output:

ndbmysqld-0                                                       3/3     Running   0          4m14s
ndbmysqld-1                                                       3/3     Running   0          3m53s
ndbmysqld-2                                                       3/3     Running   0          3m21s
ndbmysqld-3                                                       3/3     Running   0          3m01s
ndbmysqld-4                                                       3/3     Running   0          2m59s
ndbmysqld-5                                                       3/3     Running   0          2m43s

Check the status of cnDBTier cluster3:

$ kubectl -n cluster3 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   12 node(s)
id=56   @10.233.71.24     (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.119.18    (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.69.23     (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.116.21    (mysql-8.0.35 ndb-8.0.35)
id=60   @10.233.70.16     (mysql-8.0.35 ndb-8.0.35)
id=61   @10.233.78.20     (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.127.117   (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93     (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.
If cluster3 satisfies all the conditions in Step 5, then georeplication is enabled in cnDBTier cluster3. Otherwise, georeplication is not enabled in cluster3.

If georeplication is not enabled in cluster3 as per Step 5, then perform the following steps to enable georeplication in cluster3 with respect to cluster4. Else, skip to Step 7.

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then configure remote mate site name (cluster4) in cnDBTier cluster3 by following the Configuring Multiple Replication channel groups procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide and skip steps a and b in the following substeps.

Configure the remote mate site name (cluster4) in cnDBTier cluster3 by enabling and configuring the replication service using the custom_values.yaml file:

$ vi occndbtier/custom_values.yaml

Sample output:

dbreplsvcdeployments:
    ....
    # if pod prefix is given then use the unique smaller name for this db replication service.
    - name: cndbtiercluster3-cndbtiercluster4-replication-svc
      enabled: true
      mysql:
        dbtierservice: "mysql-connectivity-service"
        dbtierreplservice: "ndbmysqldsvc"
        # if cndbtier, use CLUSTER-IP from the ndbmysqldsvc-4 LoadBalancer service
        primaryhost: "ndbmysqld-4.ndbmysqldsvc.cluster3.svc.occne4-cgbu-cne-dbtier"
        port: "3306"
        # if cndbtier, use EXTERNAL-IP from the ndbmysqldsvc-4 LoadBalancer service
        primarysignalhost: ""
        # serverid is unique; retrieve it for the site being configured
        primaryhostserverid: "3004"
        # if cndbtier, use CLUSTER-IP from the ndbmysqldsvc-5 LoadBalancer service
        secondaryhost: "ndbmysqld-5.ndbmysqldsvc.cluster3.svc.occne4-cgbu-cne-dbtier"
        # if cndbtier, use EXTERNAL-IP from the ndbmysqldsvc-5 LoadBalancer service
        secondarysignalhost: ""
        # serverid is unique; retrieve it for the site being configured
        secondaryhostserverid: "3005"
      replication:
        # Local site replication service LoadBalancer ip can be configured.
        localsiteip: ""
        matesitename: "cluster4"
        remotesiteip: ""

Configure the number of SQL pods in cnDBTier cluster3 to at least six in the custom_values.yaml file:

Note:
apiReplicaCount is set to "6" as you are adding the third site. The first four ndbmysqld are used for replication between site one, site two, and site three only. The two ndbmysqld pods that are newly added will be responsible for replication between site1 and site4.
```
$ vi occndbtier/custom_values.yaml
```
Sample output:
```
apiReplicaCount: 6
```
Upgrade cnDBTier cluster3 by using the CSAR package:

Note:
The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then upgrade cnDBTier cluster3 by providing appropriate multi group values file.
```
$ helm upgrade mysql-cluster occndbtier --namespace cluster3 --no-hooks -f occndbtier/custom_values.yaml
```
Sample output:
```
Release "mysql-cluster" has been upgraded. Happy Helming!
NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 11:10:32 2023
NAMESPACE: cluster3
STATUS: deployed
REVISION: 2
```

Scale down the replication service of cluster3 with respect to cluster4 by performing the following steps:

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then scale down the replication service of cluster3 with respect to cluster4 for every replication channel group.

Identify the replication service of cnDBTier cluster3 with respect to cnDBTier cluster4:

$ kubectl get deployment --namespace=cluster3

Sample output:

NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster3-cluster1-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster3-cluster2-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster3-cluster4-replication-svc      0/1     1            0           10s

Scale down the replication service of cnDBTier cluster3 with respect to cnDBTier cluster4:

$ kubectl scale deployment mysql-cluster-cluster3-cluster4-replication-svc --namespace=cluster3 --replicas=0

Sample output:

deployment.apps/mysql-cluster-cluster3-cluster4-replication-svc scaled

Restart all the management pods of cnDBTier cluster3 simultaneously by performing the following steps:

Identify the list of management pods at cnDBTier cluster3:

$ kubectl get pods --namespace=cluster3 | grep 'mgmd'

Sample output:

ndbmgmd-0                        2/2     Running             0          9m34s
ndbmgmd-1                        2/2     Running             0          9m4s

Delete all the management pods of cnDBTier cluster3 such that the pods restart:
```
$ kubectl delete pod ndbmgmd-0 ndbmgmd-1 --namespace=cluster3
```
Sample output:
```
pod "ndbmgmd-0" deleted
pod "ndbmgmd-1" deleted
```
Wait until the management pods restart and check if the management pods are up and running by running the following command:
```
$ kubectl get pods --namespace=cluster3 | grep 'mgmd'
```
Sample output:
```
ndbmgmd-0                          2/2     Running   0          4m29s
ndbmgmd-1                          2/2     Running   0          4m12s
```

Check the status of cnDBTier cluster3:

$ kubectl -n cluster3 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.89.14  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
id=2    @10.233.78.19  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.69.22  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.116.20  (mysql-8.0.35 ndb-8.0.35)
 
[mysqld(API)]   12 node(s)
id=56   @10.233.71.24  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.119.18  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.69.23  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.116.21  (mysql-8.0.35 ndb-8.0.35)
id=60   (not connected, accepting connect from ndbmysqld-4.ndbmysqldsvc.cluster3.svc.occne4-cgbu-cne-dbtier)
id=61   (not connected, accepting connect from ndbmysqld-5.ndbmysqldsvc.cluster3.svc.occne4-cgbu-cne-dbtier) 
id=70   @10.233.127.117   (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93     (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

The SQL pods with id=60 and id=61 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Restart the data pods one after the other in descending order by performing the following steps:

Identify the list of data pods in cnDBTier cluster3:

$ kubectl get pods --namespace=cluster3 | grep 'ndbmtd'

Sample output:

ndbmtd-0                           3/3     Running   0          14m
ndbmtd-1                           3/3     Running   0          13m

Delete the second data pod of cnDBTier cluster3 (in this case, ndbmtd-1) such that the pod restarts:
```
$ kubectl delete pod ndbmtd-1 --namespace=cluster3
```
Sample output:
```
pod "ndbmtd-1" deleted
```
Wait for the second data pod to restart and check if the second pod is up and running by running the following command:
```
$ kubectl get pods --namespace=cluster3 | grep 'ndbmtd'
```
Sample output:
```
ndbmtd-0                           3/3     Running   0          13m
ndbmtd-1                           3/3     Running   0          65s
```

Check the status of cnDBTier cluster3:

$ kubectl -n cluster3 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.89.14  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
id=2    @10.233.78.19  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
 
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.69.22  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.116.20  (mysql-8.0.35 ndb-8.0.35)
 
[mysqld(API)]   12 node(s)
id=56   @10.233.71.24  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.119.18  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.69.23  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.116.21  (mysql-8.0.35 ndb-8.0.35)
id=60   (not connected, accepting connect from ndbmysqld-4.ndbmysqldsvc.cluster3.svc.occne4-cgbu-cne-dbtier)
id=61   (not connected, accepting connect from ndbmysqld-5.ndbmysqldsvc.cluster3.svc.occne4-cgbu-cne-dbtier) 
id=70   @10.233.127.117   (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93     (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

The SQL pods with id=60 and id=61 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.
The node IDs 222 to 225 in the sample output are shown as "not connected" as these nodes are added as empty slot IDs used for georeplication recovery. You can ignore these nodes.

Follow Step ii through Step iv to delete and restart the first data pod of cnDBTier cluster3 (ndbmtd-1).

Scale up the replication service of cluster3 with respect to cluster4 by performing the following steps:

Note:
The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then scale up the replication service of cluster3 with respect to cluster4 for every replication channel group.
1. Identify the replication service of cnDBTier cluster3 with respect to cnDBTier cluster4:
```
$ kubectl get deployment --namespace=cluster3
```
  Sample output:
```
NAME                                                 READY   UP-TO-DATE   AVAILABLE   AGE
mysql-cluster-db-backup-manager-svc                  1/1     1            1           9m7s
mysql-cluster-db-monitor-svc                         1/1     1            1           9m7s
mysql-cluster-cluster3-cluster1-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster3-cluster2-replication-svc      1/1     1            1           9m7s
mysql-cluster-cluster3-cluster4-replication-svc      0/0     0            0           4m10s
```
2. Scale up the replication service of cnDBTier cluster3 with respect to cnDBTier cluster4:
```
$ kubectl scale deployment mysql-cluster-cluster3-cluster4-replication-svc --namespace=cluster3 --replicas=1
```
  Sample output:
```
deployment.apps/mysql-cluster-cluster3-cluster4-replication-svc scaled
```
3. Wait until the replication service of cnDBTier cluster3 with respect to cnDBTier cluster4 is up and running. You can check the status of the replication service by running the following command:
```
$ kubectl get pods --namespace=cluster3 | grep 'repl'
```
  Sample output:
```
mysql-cluster-cluster3-cluster4-replication-svc-c574d6cjs5mp   1/1     Running           0          119s
```
Perform the following steps to restart the monitor service pod of cnDBTier cluster3:
1. Identify the monitor service pod at cnDBTier cluster3:
```
$ kubectl get pods --namespace=cluster3 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-4cghv           1/1     Running   0                13h
```
2. Delete the monitor service pod of cnDBTier cluster3:
```
$ kubectl delete pod mysql-cluster-db-monitor-svc-8bf9448b8-4cghv --namespace=cluster3
```
  Sample output:
```
pod "mysql-cluster-db-monitor-svc-8bf9448b8-4cghv" deleted
```
3. Wait until the monitor service pod is up and running. You can check the status of the pod by running the following command:
```
$ kubectl get pods --namespace=cluster3 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-w8pvf           1/1     Running   0                109s
```
Upgrade cnDBTier cluster3 by following the procedures in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Note:
cnDBTier supports upgrade in a TLS enabled cluster in this scenario only.

Install cnDBTier cluster4 by configuring cnDBTier cluster1 as the first mate site and cnDBTier cluster2 as the second mate site with at least six SQL pods. For information on installing the cnDBTier cluster, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Note:
- The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then install cnDBTier cluster4 by configuring cnDBTier cluster1 as first mate site, cnDBTier cluster2 as second mate site and cnDBTier cluster3 as third mate site with at least twelve SQL pods.
- Do not run a Helm test when you are performing a conversion procedure.
Check if georeplication channels are established between cnDBTier cluster1 and cnDBTier cluster4, cnDBTier cluster2 and cnDBTier cluster4, and cnDBTier cluster3 and cnDBTier cluster4 by following the procedures described in the Checking Georeplication Status Between Clusters section.
Restore the data of cnDBTier cluster1, cnDBTier cluster2, or cnDBTier cluster3 in cnDBTier cluster4 and re-establish the georeplication channels between cnDBTier cluster1 and cnDBTier cluster4, cnDBTier cluster2 and cnDBTier cluster4, and cnDBTier cluster3 and cnDBTier cluster4. For procedures on restoring the cluster data and re-establishing the georeplication channels, refer to the Georeplication Failure between cnDBTier Clusters in Four Site Replication section of Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Check if georeplication channels are established between cnDBTier cluster1 and cnDBTier cluster4, cnDBTier cluster2 and cnDBTier cluster4, and cnDBTier cluster3 and cnDBTier cluster4 by following the procedures described in the Checking Georeplication Status Between Clusters section.

Note:
As data is replicated to cnDBTier cluster4 and georeplication channels are established between cnDBTier cluster1 and cnDBTier cluster4, cnDBTier cluster2 and cnDBTier cluster4, and cnDBTier cluster3 and cnDBTier cluster4, the newly added cnDBTier cluster (cnDBTier cluster4) can be used by the NFs for database operations.

7.6 Removing a Georedundant cnDBTier Cluster

This section describes the procedures to remove a georedundant cluster from an existing multisite setup (two-site, three-site, or four-site).

The following terminologies are used in the procedures to refer to a cluster in a multisite:

cnDBTier Cluster1 (cluster1): The first cloud native cnDBTier cluster in a two-site, three-site, or four-site georeplication setup.
cnDBTier Cluster2 (cluster2): The second cloud native cnDBTier cluster in a two-site, three-site, or four-site georeplication setup.
cnDBTier Cluster3 (cluster3): The third cloud native cnDBTier cluster in a two-site, three-site, or four-site georeplication setup.
cnDBTier Cluster4 (cluster4): The fourth cloud native cnDBTier cluster in a two-site, three-site, or four-site georeplication setup.

Prerequisites

Before performing the following procedures, ensure that you meet the following prerequisites:

Multiple cnDBTier clusters must be installed.
The cnDBTier cluster that needs to be removed must be a part of the multisite setup.

Note:

To perform these procedures, georeplication may or may not be established correctly between multiple cnDBTier clusters.
Ensure that there is no traffic in the cnDBTier cluster that is being removed from the multisite setup.

7.6.1 Removing cnDBTier Cluster 4

This section provides the procedure to remove cnDBTier cluster 4 from a georeplication multisite setup.

Note:

The cluster and namespace names used in this procedure may vary depending on your namespace and cluster name. Ensure that you replace the values as per your setup.

Perform the following steps to uninstall cnDBTier cluster 4 in a site:
1. Run the following commands to uninstall cnDBTier cluster 4:
```
$ helm uninstall mysql-cluster -n cluster4
```
2. Run the following commands to delete the PVCs of cnDBTier cluster 4:
```
$ kubectl -n cluster4 get pvc
$ kubectl -n cluster4 delete pvc <PVC Names>
```
  where, <PVC Names> is the PVC names of cnDBTier Cluster 4.
Perform the following steps to scale down the DB replication service deployments related to cnDBTier cluster 4 on all the other sites (cluster 1, cluster 2, and cluster 3):
1. Log in to the Bastion Host of cluster 1 and scale down the DB replication services related to cluster 4 on cluster 1:
```
$ kubectl -n cluster1 scale deploy $(kubectl -n cluster1 get deployments | awk '{print $1}' |
      grep -i  cluster1-cluster4 | grep -i repl) --replicas=0
```
2. Repeat step a to scale down the DB replication services related to cluster 4 on cluster 2.
  
  Note:
  Replace cluster1 in the command (in step a) with cluster2. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to scale down the DB replication services related to cluster 4 on cluster 3.
  
  Note:
  Replace cluster1 in the command (in step a) with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
Perform the following steps to delete the records related to cluster 4 from the replication_info, backup_info, and hbreplica_info database tables on all the other sites (cluster 1, cluster 2, and cluster 3):
1. On cluster 1, delete the records related to cluster 4 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session:
  
  Note:
  Ensure that you run all the commands in a single session.
  1. Log in to the Bastion Host of cluster 1:
```
$ kubectl -n cluster1 exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
```
  2. Disable the binary logs for the session:
```
mysql> set SQL_LOG_BIN=OFF;
```
  3. Run the following commands to delete the records related to cluster 4 from the replication_info database tables:
```
mysql> DELETE FROM replication_info.DBTIER_SITE_INFO WHERE site_name='cluster4';
mysql> DELETE FROM replication_info.DBTIER_REPL_SVC_INFO WHERE site_name='cluster4' OR mate_site_name='cluster4';
mysql> DELETE FROM replication_info.DBTIER_REPL_SITE_INFO WHERE site_name='cluster4' OR mate_site_name='cluster4';
mysql> DELETE FROM replication_info.DBTIER_REPL_EVENT_INFO WHERE site_name='cluster4' OR mate_site_name='cluster4';
mysql> DELETE FROM replication_info.DBTIER_REPL_ERROR_SKIP_INFO WHERE site_name='cluster4' OR mate_site_name='cluster4';
mysql> DELETE FROM replication_info.DBTIER_REPLICATION_CHANNEL_INFO WHERE site_name='cluster4' OR remote_site_name='cluster4';
mysql> DELETE FROM replication_info.DBTIER_INITIAL_BINLOG_POSTION WHERE site_name='cluster4';
```
  4. Run the following command to delete the records related to cluster 4 from the DBTIER_HEARTBEAT_INFO table:
```
mysql> DELETE FROM hbreplica_info.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster4' OR mate_site_name='cluster4';
```
  5. If there are more than one replication channel groups, then delete the records related to cluster 4 from DBTIER_HEARTBEAT_INFO table on all the replication channel groups:
```
mysql> DELETE FROM hbreplica_info2.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster4' OR mate_site_name='cluster4';
mysql> DELETE FROM hbreplica_info3.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster4' OR mate_site_name='cluster4';
mysql> DELETE FROM hbreplica_info4.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster4' OR mate_site_name='cluster4';
mysql> DELETE FROM hbreplica_info5.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster4' OR mate_site_name='cluster4';
mysql> DELETE FROM hbreplica_info6.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster4' OR mate_site_name='cluster4';
```
  6. Delete the records related to cluster 4 from the backup_info database tables:
```
mysql> DELETE FROM backup_info.DBTIER_BACKUP_INFO WHERE site_name='cluster4';
mysql> DELETE FROM backup_info.DBTIER_BACKUP_TRANSFER_INFO WHERE site_name='cluster4';
mysql> DELETE FROM backup_info.DBTIER_BACKUP_COMMAND_QUEUE WHERE site_name='cluster4';
```
  7. Delete the records related to cluster 4 from the mysql.ndb_apply_status table:
```
mysql> DELETE FROM mysql.ndb_apply_status WHERE server_id LIKE '40%';
```
  8. Run the following commands to turn on the binary log and exit the session:
```
mysql> set SQL_LOG_BIN=ON;
mysql> exit
```
  9. Run the following commands to update the records in the replication_info.DBTIER_INITIAL_BINLOG_POSTION table in cnDBTier cluster 1:
```
$ export DBTIER_NAMESPACE="cluster1"
$ export DBTIER_CLUSTER_NAME="cluster1"
$ for i in {0..5};do echo $i;kubectl -n ${DBTIER_NAMESPACE} exec -it ndbmysqld-$i -- mysql -h 127.0.0.1 -uroot -pNextGenCne -e "UPDATE replication_info.DBTIER_INITIAL_BINLOG_POSTION AS dest, ( SELECT SUBSTRING_INDEX(next_file, '/', -1) as File,next_position as Position FROM mysql.ndb_binlog_index ORDER BY epoch DESC LIMIT 1 ) AS src SET dest.bin_log_filename = src.File, dest.bin_log_position = src.Position WHERE dest.sql_node_ip like '%ndbmysqld-$i%' AND dest.site_name = '${DBTIER_CLUSTER_NAME}';"; done;
```
2. Repeat step a on cluster 2 to delete the records related to cluster 4 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session.
  
  Note:
  Replace cluster1 in the commands (in step a) with cluster2. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a on cluster 3 to delete the records related to cluster 4 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session.
  
  Note:
  Replace cluster1 in the commands (in step a) with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
Stop and reset the replica related to cluster 4 in the replication SQL (ndbmysqld) pods on all the other sites (cluster 1, cluster 2, and cluster 3):
1. Stop and reset the replica related to cluster 4 in the replication SQL (ndbmysqld) pods on cluster 1:
  1. Log in to the Bastion Host of cluster 1.
  2. Run the following commands to stop and reset the replica related to cluster 4 in the replication SQL (ndbmysqld) pods:
```
$ export DBTIER_NAMESPACE="cluster1"
$ export CLUSTER_NAME="cluster1"
$ export REMOVED_CLUSTER_NAME="cluster4"
$ kubectl -n ${DBTIER_NAMESPACE} get deploy | grep -i "${CLUSTER_NAME}-${REMOVED_CLUSTER_NAME}" | grep -i repl | awk '{print $1}' | xargs -I {} sh -c 'kubectl -n ${DBTIER_NAMESPACE} describe deploy {} | egrep  "MYSQL_PRIMARY_HOST:|MYSQL_SECONDARY_HOST:"' | awk '{print $2}' | awk -F '.'  '{print $1}' | xargs -I {} sh -c 'echo "in ${DBTIER_NAMESPACE} namespace, {} Pod"; kubectl -n ${DBTIER_NAMESPACE} exec -it {} -- mysql -h 127.0.0.1 -uroot -pNextGenCne -e "STOP REPLICA; RESET REPLICA ALL;"'
```
2. Repeat step a to stop and reset the replica related to cluster 4 in the replication SQL (ndbmysqld) pods on cluster 2.
  
  Note:
  Replace cluster1 in the commands (in step a) with cluster2. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to stop and reset the replica related to cluster 4 in the replication SQL (ndbmysqld) pods on cluster 3.
  
  Note:
  Replace cluster 1 in the commands (in step a) with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
Remove the remote site IP address configurations related to cluster 4 and restart all the DB replication service deployments on all the other sites (cluster 1, cluster 2, and cluster 3):
1. Perform the following steps to remove the remote site IP address configurations related to cluster 4 and restart all the DB replication service deployments on cluster 1:
  1. Log in to the Bastion Host of cluster 1.
  2. If fixed IP addresses are not configured to the LoadBalancer services, configure remotesiteip with "" for cluster 4 remote site in the custom_values.yaml file of cluster 1:
```
$ vi cndbtier_cluster1_custom_values.yaml
 
- name: cluster1-cluster4-replication-svc
  enabled: true
  --------------------------------------
  --------------------------------------
  --------------------------------------
    replication:
      localsiteip: ""
      localsiteport: "80"
      channelgroupid: "1"
      matesitename: "cluster4"
      remotesiteip: ""
      remotesiteport: "80"
```
  3. If fixed IP addresses are not configured to the LoadBalancer services, upgrade cnDBTier cluster 1:
```
$ helm upgrade mysql-cluster occndbtier -n cluster1 -f cndbtier_cluster1_custom_values.yaml
```
  4. Scale down and scale up all the DB replication service deployments in cluster 1:
```
$ kubectl -n cluster1 scale deploy $(kubectl -n cluster1 get deployments | awk '{print $1}' | egrep -i 'repl|monitor|backup-manager') --replicas=0
$ kubectl -n cluster1 scale deploy $(kubectl -n cluster1 get deployments | awk '{print $1}' | egrep -i 'repl|monitor|backup-manager') --replicas=1
```
  5. Wait until all the pods are up on cluster 1 and verify:
```
$ kubectl -n cluster1 get pods
```
2. Repeat step a to remove the remote site IP address configurations related to cluster 4 and restart all the DB replication service deployments on cluster 2:
  
  Note:
  Replace cluster1 in the commands (in step a) with cluster2. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to remove the remote site IP address configurations related to cluster 4 and restart all the DB replication service deployments on cluster 3:
  
  Note:
  Replace cluster1 in the commands (in step a) with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.

7.6.2 Removing cnDBTier Cluster 3

This section provides the procedure to remove cnDBTier cluster 3 from a georeplication multisite setup.

Note:

The cluster and namespace names used in this procedure may vary depending on your namespace and cluster name. Ensure that you replace the values as per your setup.

Perform the following steps to uninstall cnDBTier cluster 3 in a site:
1. Run the following commands to uninstall cnDBTier cluster 3:
```
$ helm uninstall mysql-cluster -n cluster3
```
2. Run the following commands to delete the PVCs of cnDBTier cluster 3:
```
$ kubectl -n cluster3 get pvc
$ kubectl -n cluster3 delete pvc <PVC Names>
```
  where, <PVC Names> is the PVC names of cnDBTier Cluster 3.
Perform the following steps to scale down the DB replication service deployments related to cnDBTier cluster 3 on all the other sites (cluster 1, cluster 2, and cluster 4):
1. Log in to the Bastion Host of cluster 1 and scale down the DB replication services related to cluster 3 on cluster 1:
```
$ kubectl -n cluster1 scale deploy $(kubectl -n cluster1 get deployments | awk '{print $1}' | grep -i  cluster1-cluster3 | grep -i repl) --replicas=0
```
2. Repeat step a to scale down the DB replication services related to cluster 3 on cluster 2.
  
  Note:
  Replace cluste 1 in the command with cluster2. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to scale down the DB replication services related to cluster 3 on cluster 4.
  
  Note:
  Replace cluster 1 in the command with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.
Perform the following steps to delete the records related to cluster 3 from the replication_info, backup_info, and hbreplica_info database tables on all the other sites (cluster 1, cluster 2, and cluster 4):

Note:
Ensure that you run all the commands in a single session.
1. On cluster 1, delete the records related to cluster 3 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session:
  1. Log in to the Bastion Host of cluster 1:
```
$ kubectl -n cluster1 exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
```
  2. Disable the binary logs for the session:
```
mysql> set SQL_LOG_BIN=OFF;
```
  3. Run the following commands to delete the records related to cluster 3 from the replication_info database tables:
```
mysql> DELETE FROM replication_info.DBTIER_SITE_INFO WHERE site_name='cluster3';
mysql> DELETE FROM replication_info.DBTIER_REPL_SVC_INFO WHERE site_name='cluster3' OR mate_site_name='cluster3';
mysql> DELETE FROM replication_info.DBTIER_REPL_SITE_INFO WHERE site_name='cluster3' OR mate_site_name='cluster3';
mysql> DELETE FROM replication_info.DBTIER_REPL_EVENT_INFO WHERE site_name='cluster3' OR mate_site_name='cluster3';
mysql> DELETE FROM replication_info.DBTIER_REPL_ERROR_SKIP_INFO WHERE site_name='cluster3' OR mate_site_name='cluster3';
mysql> DELETE FROM replication_info.DBTIER_REPLICATION_CHANNEL_INFO WHERE site_name='cluster3' OR remote_site_name='cluster3';
mysql> DELETE FROM replication_info.DBTIER_INITIAL_BINLOG_POSTION WHERE site_name='cluster3';
```
  4. Run the following command to delete the records related to cluster 3 from the DBTIER_HEARTBEAT_INFO table:
```
mysql> DELETE FROM hbreplica_info.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster3' OR mate_site_name='cluster3';
```
  5. If there are more than one replication channel groups, then delete the records related to cluster 3 from DBTIER_HEARTBEAT_INFO table on all the replication channel groups:
```
mysql> DELETE FROM hbreplica_info2.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster3' OR mate_site_name='cluster3';
mysql> DELETE FROM hbreplica_info3.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster3' OR mate_site_name='cluster3';
mysql> DELETE FROM hbreplica_info4.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster3' OR mate_site_name='cluster3';
mysql> DELETE FROM hbreplica_info5.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster3' OR mate_site_name='cluster3';
mysql> DELETE FROM hbreplica_info6.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster3' OR mate_site_name='cluster3';
```
  6. Delete the records related to cluster 3 from the backup_info database tables:
```
mysql> DELETE FROM backup_info.DBTIER_BACKUP_INFO WHERE site_name='cluster3';
mysql> DELETE FROM backup_info.DBTIER_BACKUP_TRANSFER_INFO WHERE site_name='cluster3';
mysql> DELETE FROM backup_info.DBTIER_BACKUP_COMMAND_QUEUE WHERE site_name='cluster3';
```
  7. Delete the records related to cluster 3 from the mysql.ndb_apply_status table:
```
mysql> DELETE FROM mysql.ndb_apply_status WHERE server_id LIKE '30%';
```
  8. Run the following commands to turn on the binary log and exit the session:
```
mysql> set SQL_LOG_BIN=ON;
mysql> exit
```
  9. Run the following commands to update the records in the replication_info.DBTIER_INITIAL_BINLOG_POSTION table in cnDBTier cluster 1:
```
$ export DBTIER_NAMESPACE="cluster1"
$ export DBTIER_CLUSTER_NAME="cluster1"
$ for i in {0..5};do echo $i;kubectl -n ${DBTIER_NAMESPACE} exec -it ndbmysqld-$i -- mysql -h 127.0.0.1 -uroot -pNextGenCne -e "UPDATE replication_info.DBTIER_INITIAL_BINLOG_POSTION AS dest, ( SELECT SUBSTRING_INDEX(next_file, '/', -1) as File,next_position as Position FROM mysql.ndb_binlog_index ORDER BY epoch DESC LIMIT 1 ) AS src SET dest.bin_log_filename = src.File, dest.bin_log_position = src.Position WHERE dest.sql_node_ip like '%ndbmysqld-$i%' AND dest.site_name = '${DBTIER_CLUSTER_NAME}';"; done;
```
2. Repeat step a on cluster 2 to delete the records related to cluster 3 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session.
  
  Note:
  Replace cluster1 in the commands with cluster2. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a on cluster 4 to delete the records related to cluster 3 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session.
  
  Note:
  Replace cluster1 in the commands with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.
Stop and reset the replica related to cluster 3 in the replication SQL (ndbmysqld) pods on all the other sites (cluster 1, cluster 2, and cluster 4):
1. Stop and reset the replica related to cluster 3 in the replication SQL (ndbmysqld) pods on cluster 1:
  1. Log in to the Bastion Host of cluster 1.
  2. Run the following commands to stop and reset the replica related to cluster 3 in the replication SQL (ndbmysqld) pods:
```
$ export DBTIER_NAMESPACE="cluster1"
$ export CLUSTER_NAME="cluster1"
$ export REMOVED_CLUSTER_NAME="cluster3"
$ kubectl -n ${DBTIER_NAMESPACE} get deploy | grep -i "${CLUSTER_NAME}-${REMOVED_CLUSTER_NAME}" | grep -i repl | awk '{print $1}' | xargs -I {} sh -c 'kubectl -n ${DBTIER_NAMESPACE} describe deploy {} | egrep  "MYSQL_PRIMARY_HOST:|MYSQL_SECONDARY_HOST:"' | awk '{print $2}' | awk -F '.'  '{print $1}' | xargs -I {} sh -c 'echo "in ${DBTIER_NAMESPACE} namespace, {} Pod"; kubectl -n ${DBTIER_NAMESPACE} exec -it {} -- mysql -h 127.0.0.1 -uroot -pNextGenCne -e "STOP REPLICA; RESET REPLICA ALL;"'
```
2. Repeat step a to stop and reset the replica related to cluster 3 in the replication SQL (ndbmysqld) pods on cluster 2.
  
  Note:
  Replace cluster1 in the commands with cluster2. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to stop and reset the replica related to cluster 3 in the replication SQL (ndbmysqld) pods on cluster 4.
  
  Note:
  Replace cluster1 in the commands with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.
Remove the remote site IP address configurations related to cluster 3 and restart all the DB replication service deployments on all the other sites (cluster 1, cluster 2, and cluster 4):
1. Perform the following steps to remove the remote site IP address configurations related to cluster 3 and restart all the DB replication service deployments on cluster 1:
  1. Log in to the Bastion Host of cluster 1.
  2. If fixed IP addresses are not configured to the LoadBalancer services, configure remotesiteip with "" for cluster 3 remote site in the custom_values.yaml file of cluster 1:
```
$ vi cndbtier_cluster1_custom_values.yaml
  
- name: cluster1-cluster3-replication-svc
  enabled: true
  --------------------------------------
  --------------------------------------
  --------------------------------------
    replication:
      localsiteip: ""
      localsiteport: "80"
      channelgroupid: "1"
      matesitename: "cluster3"
      remotesiteip: ""
      remotesiteport: "80"
```
  3. If fixed IP addresses are not configured to the LoadBalancer services, upgrade cnDBTier cluster 1:
```
$ helm upgrade mysql-cluster occndbtier -n cluster1 -f cndbtier_cluster1_custom_values.yaml
```
  4. Scale down and scale up all the DB replication service deployments in cluster 1:
```
$ kubectl -n cluster1 scale deploy $(kubectl -n cluster1 get deployments | awk '{print $1}' | egrep -i 'repl|monitor|backup-manager') --replicas=0
$ kubectl -n cluster1 scale deploy $(kubectl -n cluster1 get deployments | awk '{print $1}' | egrep -i 'repl|monitor|backup-manager') --replicas=1
```
  5. Wait until all the pods are up on cluster 1 and verify:
```
$ kubectl -n cluster1 get pods
```
2. Repeat step a to remove the remote site IP address configurations related to cluster 3 and restart all the DB replication service deployments on cluster 2:
  
  Note:
  Replace cluster1 in the commands with cluster2. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to remove the remote site IP address configurations related to cluster 3 and restart all the DB replication service deployments on cluster 4:
  
  Note:
  Replace cluster1 in the commands with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.

7.6.3 Removing cnDBTier Cluster 2

This section provides the procedure to remove cnDBTier cluster 2 from a georeplication multisite setup.

Note:

The cluster and namespace names used in this procedure may vary depending on your namespace and cluster name. Ensure that you replace the values as per your setup.

Perform the following steps to uninstall cnDBTier cluster 2 in a site:
1. Run the following commands to uninstall cnDBTier cluster 2:
```
$ helm uninstall mysql-cluster -n cluster2
```
2. Run the following commands to delete the PVCs of cnDBTier cluster 2:
```
$ kubectl -n cluster2 get pvc
$ kubectl -n cluster2 delete pvc <PVC Names>
```
  where, <PVC Names> is the PVC names of cnDBTier cluster 2.
Perform the following steps to scale down the DB replication service deployments related to cnDBTier cluster 2 on all the other sites (cluster 1, cluster 3, and cluster 4):
1. Log in to the Bastion Host of cluster 1 and scale down the DB replication services related to cluster 2 on cluster 1:
```
$ kubectl -n cluster1 scale deploy $(kubectl -n cluster1 get deployments | awk '{print $1}' | grep -i  cluster1-cluster2 | grep -i repl) --replicas=0
```
2. Repeat step a to scale down the DB replication services related to cluster 2 on cluster 3.
  
  Note:
  Replace cluste 1 in the command with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to scale down the DB replication services related to cluster 2 on cluster 4.
  
  Note:
  Replace cluster1 in the command with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.
Perform the following steps to delete the records related to cluster 2 from the replication_info, backup_info, and hbreplica_info database tables on all the other sites (cluster 1, cluster 3, and cluster 4):

Note:
Ensure that you run all the commands in a single session.
1. On cluster 1, delete the records related to cluster 2 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session:
  1. Log in to the Bastion Host of cluster 1:
```
$ kubectl -n cluster1 exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
```
  2. Disable the binary logs for the session:
```
mysql> set SQL_LOG_BIN=OFF;
```
  3. Run the following commands to delete the records related to cluster 2 from the replication_info database tables:
```
mysql> DELETE FROM replication_info.DBTIER_SITE_INFO WHERE site_name='cluster2';
mysql> DELETE FROM replication_info.DBTIER_REPL_SVC_INFO WHERE site_name='cluster2' OR mate_site_name='cluster2';
mysql> DELETE FROM replication_info.DBTIER_REPL_SITE_INFO WHERE site_name='cluster2' OR mate_site_name='cluster2';
mysql> DELETE FROM replication_info.DBTIER_REPL_EVENT_INFO WHERE site_name='cluster2' OR mate_site_name='cluster2';
mysql> DELETE FROM replication_info.DBTIER_REPL_ERROR_SKIP_INFO WHERE site_name='cluster2' OR mate_site_name='cluster2';
mysql> DELETE FROM replication_info.DBTIER_REPLICATION_CHANNEL_INFO WHERE site_name='cluster2' OR remote_site_name='cluster2'; 
mysql> DELETE FROM replication_info.DBTIER_INITIAL_BINLOG_POSTION WHERE site_name='cluster2';
```
  4. Run the following command to delete the records related to cluster 2 from the DBTIER_HEARTBEAT_INFO table:
```
mysql> DELETE FROM hbreplica_info.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster2' OR mate_site_name='cluster2';
```
  5. If there are more than one replication channel groups, then delete the records related to cluster 2 from DBTIER_HEARTBEAT_INFO table on all the replication channel groups:
```
mysql> DELETE FROM hbreplica_info2.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster2' OR mate_site_name='cluster2';
mysql> DELETE FROM hbreplica_info3.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster2' OR mate_site_name='cluster2';
mysql> DELETE FROM hbreplica_info4.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster2' OR mate_site_name='cluster2';
mysql> DELETE FROM hbreplica_info5.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster2' OR mate_site_name='cluster2';
mysql> DELETE FROM hbreplica_info6.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster2' OR mate_site_name='cluster2';
```
  6. Delete the records related to cluster 2 from the backup_info database tables:
```
mysql> DELETE FROM backup_info.DBTIER_BACKUP_INFO WHERE site_name='cluster2';
mysql> DELETE FROM backup_info.DBTIER_BACKUP_TRANSFER_INFO WHERE site_name='cluster2';
mysql> DELETE FROM backup_info.DBTIER_BACKUP_COMMAND_QUEUE WHERE site_name='cluster2';
```
  7. Delete the records related to cluster 2 from the mysql.ndb_apply_status table:
```
mysql> DELETE FROM mysql.ndb_apply_status WHERE server_id LIKE '20%';
```
  8. Run the following commands to turn on the binary log and exit the session:
```
mysql> set SQL_LOG_BIN=ON;
mysql> exit
```
  9. Run the following commands to update the records in the replication_info.DBTIER_INITIAL_BINLOG_POSTION table in cnDBTier cluster 1:
```
$ export DBTIER_NAMESPACE="cluster1"
$ export DBTIER_CLUSTER_NAME="cluster1"
$ for i in {0..5};do echo $i;kubectl -n ${DBTIER_NAMESPACE} exec -it ndbmysqld-$i -- mysql -h 127.0.0.1 -uroot -pNextGenCne -e "UPDATE replication_info.DBTIER_INITIAL_BINLOG_POSTION AS dest, ( SELECT SUBSTRING_INDEX(next_file, '/', -1) as File,next_position as Position FROM mysql.ndb_binlog_index ORDER BY epoch DESC LIMIT 1 ) AS src SET dest.bin_log_filename = src.File, dest.bin_log_position = src.Position WHERE dest.sql_node_ip like '%ndbmysqld-$i%' AND dest.site_name = '${DBTIER_CLUSTER_NAME}';"; done;
```
2. Repeat step a on cluster 3 to delete the records related to cluster 2 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session.
  
  Note:
  Replace cluster1 in the commands with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a on cluster 4 to delete the records related to cluster 2 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session.
  
  Note:
  Replace cluster1 in the commands with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.
Stop and reset the replica related to cluster 2 in the replication SQL (ndbmysqld) pods on all the other sites (cluster 1, cluster 3, and cluster 4):
1. Stop and reset the replica related to cluster 2 in the replication SQL (ndbmysqld) pods on cluster 1:
  1. Log in to the Bastion Host of cluster 1.
  2. Run the following commands to stop and reset the replica related to cluster 2 in the replication SQL (ndbmysqld) pods:
```
$ export DBTIER_NAMESPACE="cluster1"
$ export CLUSTER_NAME="cluster1"
$ export REMOVED_CLUSTER_NAME="cluster2"
$ kubectl -n ${DBTIER_NAMESPACE} get deploy | grep -i "${CLUSTER_NAME}-${REMOVED_CLUSTER_NAME}" | grep -i repl | awk '{print $1}' | xargs -I {} sh -c 'kubectl -n ${DBTIER_NAMESPACE} describe deploy {} | egrep  "MYSQL_PRIMARY_HOST:|MYSQL_SECONDARY_HOST:"' | awk '{print $2}' | awk -F '.'  '{print $1}' | xargs -I {} sh -c 'echo "in ${DBTIER_NAMESPACE} namespace, {} Pod"; kubectl -n ${DBTIER_NAMESPACE} exec -it {} -- mysql -h 127.0.0.1 -uroot -pNextGenCne -e "STOP REPLICA; RESET REPLICA ALL;"'
```
2. Repeat step a to stop and reset the replica related to cluster 2 in the replication SQL (ndbmysqld) pods on cluster 3.
  
  Note:
  Replace cluster1 in the commands with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to stop and reset the replica related to cluster 2 in the replication SQL (ndbmysqld) pods on cluster 4.
  
  Note:
  Replace cluster1 in the commands with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.
Remove the remote site IP address configurations related to cluster 2 and restart all the DB replication service deployments on all the other sites (cluster 1, cluster 3, and cluster 4):
1. Perform the following steps to remove the remote site IP address configurations related to cluster 2 and restart all the DB replication service deployments on cluster 1:
  1. Log in to the Bastion Host of cluster 1.
  2. If fixed IP addresses are not configured to the LoadBalancer services, configure remotesiteip with "" for cluster 2 remote site in the custom_values.yaml file of cluster 1:
```
$ vi cndbtier_cluster1_custom_values.yaml
  
- name: cluster1-cluster2-replication-svc
  enabled: true
  --------------------------------------
  --------------------------------------
  --------------------------------------
    replication:
      localsiteip: ""
      localsiteport: "80"
      channelgroupid: "1"
      matesitename: "cluster2"
      remotesiteip: ""
      remotesiteport: "80"
```
  3. If fixed IP addresses are not configured to the LoadBalancer services, upgrade cnDBTier cluster 1:
```
$ helm upgrade mysql-cluster occndbtier -n cluster1 -f cndbtier_cluster1_custom_values.yaml
```
  4. Scale down and scale up all the DB replication service deployments in cluster 1:
```
$ kubectl -n cluster1 scale deploy $(kubectl -n cluster1 get deployments | awk '{print $1}' | egrep -i 'repl|monitor|backup-manager') --replicas=0
$ kubectl -n cluster1 scale deploy $(kubectl -n cluster1 get deployments | awk '{print $1}' | egrep -i 'repl|monitor|backup-manager') --replicas=1
```
  5. Wait until all the pods are up on cluster 1 and verify:
```
$ kubectl -n cluster1 get pods
```
2. Repeat step a to remove the remote site IP address configurations related to cluster 2 and restart all the DB replication service deployments on cluster 3:
  
  Note:
  Replace cluster1 in the commands with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to remove the remote site IP address configurations related to cluster 2 and restart all the DB replication service deployments on cluster 4:
  
  Note:
  Replace cluster1 in the commands with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.

7.6.4 Removing cnDBTier Cluster 1

This section provides the procedure to remove cnDBTier cluster 1 from a georeplication multisite setup.

Note:

The cluster and namespace names used in this procedure may vary depending on your namespace and cluster name. Ensure that you replace the values as per your setup.

Perform the following steps to uninstall cnDBTier cluster 1 in a site:
1. Run the following commands to uninstall cnDBTier cluster 1:
```
$ helm uninstall mysql-cluster -n cluster1
```
2. Run the following commands to delete the PVCs of cnDBTier cluster 1:
```
$ kubectl -n cluster1 get pvc
$ kubectl -n cluster1 delete pvc <PVC Names>
```
  where, <PVC Names> is the PVC names of cnDBTier cluster 1.
Perform the following steps to scale down the DB replication service deployments related to cnDBTier cluster 1 on all the other sites (cluster 2, cluster 3, and cluster 4):
1. Log in to the Bastion Host of cluster 2 and scale down the DB replication services related to cluster 1 on cluster 2:
```
$ kubectl -n cluster2 scale deploy $(kubectl -n cluster2 get deployments | awk '{print $1}' | grep -i  cluster2-cluster1 | grep -i repl) --replicas=0
```
2. Repeat step a to scale down the DB replication services related to cluster 1 on cluster 3.
  
  Note:
  Replace cluster2 in the command with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to scale down the DB replication services related to cluster 1 on cluster 4.
  
  Note:
  Replace cluster2 in the command with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.
Perform the following steps to delete the records related to cluster 1 from the replication_info, backup_info, and hbreplica_info database tables on all the other sites (cluster 2, cluster 3, and cluster 4):

Note:
Ensure that you run all the commands in a single session.
1. On cluster 2, delete the records related to cluster 1 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session:
  1. Log in to the Bastion Host of cluster 2:
```
$ kubectl -n cluster2 exec -it ndbappmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
```
  2. Disable the binary logs for the session:
```
mysql> set SQL_LOG_BIN=OFF;
```
  3. Run the following commands to delete the records related to cluster 1 from the replication_info database tables:
```
mysql> DELETE FROM replication_info.DBTIER_SITE_INFO WHERE site_name='cluster1';
mysql> DELETE FROM replication_info.DBTIER_REPL_SVC_INFO WHERE site_name='cluster1' OR mate_site_name='cluster1';
mysql> DELETE FROM replication_info.DBTIER_REPL_SITE_INFO WHERE site_name='cluster1' OR mate_site_name='cluster1';
mysql> DELETE FROM replication_info.DBTIER_REPL_EVENT_INFO WHERE site_name='cluster1' OR mate_site_name='cluster1';
mysql> DELETE FROM replication_info.DBTIER_REPL_ERROR_SKIP_INFO WHERE site_name='cluster1' OR mate_site_name='cluster1';
mysql> DELETE FROM replication_info.DBTIER_REPLICATION_CHANNEL_INFO WHERE site_name='cluster1' OR remote_site_name='cluster1';
mysql> DELETE FROM replication_info.DBTIER_INITIAL_BINLOG_POSTION WHERE site_name='cluster1';
```
  4. Run the following command to delete the records related to cluster 1 from the DBTIER_HEARTBEAT_INFO table:
```
mysql> DELETE FROM hbreplica_info.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster1' OR mate_site_name='cluster1';
```
  5. If there are more than one replication channel groups, then delete the records related to cluster 1 from DBTIER_HEARTBEAT_INFO table on all the replication channel groups:
```
mysql> DELETE FROM hbreplica_info2.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster1' OR mate_site_name='cluster1';
mysql> DELETE FROM hbreplica_info3.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster1' OR mate_site_name='cluster1';
mysql> DELETE FROM hbreplica_info4.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster1' OR mate_site_name='cluster1';
mysql> DELETE FROM hbreplica_info5.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster1' OR mate_site_name='cluster1';
mysql> DELETE FROM hbreplica_info6.DBTIER_HEARTBEAT_INFO WHERE site_name='cluster1' OR mate_site_name='cluster1';
```
  6. Delete the records related to cluster 1 from the backup_info database tables:
```
mysql> DELETE FROM backup_info.DBTIER_BACKUP_INFO WHERE site_name='cluster1';
mysql> DELETE FROM backup_info.DBTIER_BACKUP_TRANSFER_INFO WHERE site_name='cluster1';
mysql> DELETE FROM backup_info.DBTIER_BACKUP_COMMAND_QUEUE WHERE site_name='cluster1';
```
  7. Delete the records related to cluster 1 from the mysql.ndb_apply_status table:
```
mysql> DELETE FROM mysql.ndb_apply_status WHERE server_id LIKE '10%';
```
  8. Run the following commands to turn on the binary log and exit the session:
```
mysql> set SQL_LOG_BIN=ON;
mysql> exit
```
  9. Run the following commands to update the records in the replication_info.DBTIER_INITIAL_BINLOG_POSTION table in cnDBTier cluster 2:
```
$ export DBTIER_NAMESPACE="cluster2"
$ export DBTIER_CLUSTER_NAME="cluster2"
$ for i in {0..5};do echo $i;kubectl -n ${DBTIER_NAMESPACE} exec -it ndbmysqld-$i -- mysql -h 127.0.0.1 -uroot -pNextGenCne -e "UPDATE replication_info.DBTIER_INITIAL_BINLOG_POSTION AS dest, ( SELECT SUBSTRING_INDEX(next_file, '/', -1) as File,next_position as Position FROM mysql.ndb_binlog_index ORDER BY epoch DESC LIMIT 1 ) AS src SET dest.bin_log_filename = src.File, dest.bin_log_position = src.Position WHERE dest.sql_node_ip like '%ndbmysqld-$i%' AND dest.site_name = '${DBTIER_CLUSTER_NAME}';"; done;
```
2. Repeat step a on cluster 3 to delete the records related to cluster 1 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session.
  
  Note:
  Replace cluster2 in the commands with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a on cluster 4 to delete the records related to cluster 1 from the replication_info, backup_info, and hbreplica_info database tables by disabling the binary logs for a session.
  
  Note:
  Replace cluster2 in the commands with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.
Stop and reset the replica related to cluster 1 in the replication SQL (ndbmysqld) pods on all the other sites (cluster 2, cluster 3, and cluster 4):
1. Stop and reset the replica related to cluster 1 in the replication SQL (ndbmysqld) pods on cluster 2:
  1. Log in to the Bastion Host of cluster 2.
  2. Run the following commands to stop and reset the replica related to cluster 1 in the replication SQL (ndbmysqld) pods:
```
$ export DBTIER_NAMESPACE="cluster2"
$ export CLUSTER_NAME="cluster2"
$ export REMOVED_CLUSTER_NAME="cluster1"
$ kubectl -n ${DBTIER_NAMESPACE} get deploy | grep -i "${CLUSTER_NAME}-${REMOVED_CLUSTER_NAME}" | grep -i repl | awk '{print $1}' | xargs -I {} sh -c 'kubectl -n ${DBTIER_NAMESPACE} describe deploy {} | egrep  "MYSQL_PRIMARY_HOST:|MYSQL_SECONDARY_HOST:"' | awk '{print $2}' | awk -F '.'  '{print $1}' | xargs -I {} sh -c 'echo "in ${DBTIER_NAMESPACE} namespace, {} Pod"; kubectl -n ${DBTIER_NAMESPACE} exec -it {} -- mysql -h 127.0.0.1 -uroot -pNextGenCne -e "STOP REPLICA; RESET REPLICA ALL;"'
```
2. Repeat step a to stop and reset the replica related to cluster 1 in the replication SQL (ndbmysqld) pods on cluster 3.
  
  Note:
  Replace cluster2 in the commands with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to stop and reset the replica related to cluster 1 in the replication SQL (ndbmysqld) pods on cluster 4.
  
  Note:
  Replace cluster2 in the commands with cluster 4. However, the values may vary depending on the cluster and namespace names in your setup.
Remove the remote site IP address configurations related to cluster 1 and restart all the DB replication service deployments on all the other sites (cluster 2, cluster 3, and cluster 4):
1. Perform the following steps to remove the remote site IP address configurations related to cluster 1 and restart all the DB replication service deployments on cluster 2:
  1. Log in to the Bastion Host of cluster 2.
  2. If fixed IP addresses are not configured to the LoadBalancer services, configure remotesiteip with "" for cluster 1 remote site in the custom_values.yaml file of cluster 2:
```
$ vi cndbtier_cluster2_custom_values.yaml
   
- name: cluster2-cluster1-replication-svc
  enabled: true
  --------------------------------------
  --------------------------------------
  --------------------------------------
    replication:
      localsiteip: ""
      localsiteport: "80"
      channelgroupid: "1"
      matesitename: "cluster1"
      remotesiteip: ""
      remotesiteport: "80"
```
  3. If fixed IP addresses are not configured to the LoadBalancer services, upgrade cnDBTier cluster 2:
```
$ helm upgrade mysql-cluster occndbtier -n cluster2 -f cndbtier_cluster2_custom_values.yaml
```
  4. Scale down and scale up all the DB replication service deployments in cluster 2:
```
$ kubectl -n cluster2 scale deploy $(kubectl -n cluster2 get deployments | awk '{print $1}' | egrep -i 'repl|monitor|backup-manager') --replicas=0
$ kubectl -n cluster2 scale deploy $(kubectl -n cluster2 get deployments | awk '{print $1}' | egrep -i 'repl|monitor|backup-manager') --replicas=1
```
  5. Wait until all the pods are up on cluster 2 and verify:
```
$ kubectl -n cluster2 get pods
```
2. Repeat step a to remove the remote site IP address configurations related to cluster 1 and restart all the DB replication service deployments on cluster 3:
  
  Note:
  Replace cluster2 in the commands with cluster3. However, the values may vary depending on the cluster and namespace names in your setup.
3. Repeat step a to remove the remote site IP address configurations related to cluster 1 and restart all the DB replication service deployments on cluster 4:
  
  Note:
  Replace cluster2 in the commands with cluster4. However, the values may vary depending on the cluster and namespace names in your setup.

7.7 Adding and Removing Replication Channel Group

This section describes the procedures to convert a single replication channel group to multiple replication channel groups and vice versa.

The procedures in this section uses the following terms to identify the clusters:

cnDBTier Cluster1 (a.k.a. cluster1): The first cloud native based DBTier cluster in a two site, three site, or four site georeplication setup.
cnDBTier Cluster2 (a.k.a. cluster2): The second cloud native based DBTier cluster in a two site, three site, or four site georeplication setup.
cnDBTier Cluster3 (a.k.a. cluster3): The third cloud native based DBTier cluster in a two site, three site, or four site georeplication setup.
cnDBTier Cluster4 (a.k.a. cluster4): The fourth cloud native based DBTier cluster in a two site, three site, or four site georeplication setup.

Prerequisites

All the cnDBTier data nodes and SQL nodes that participate in georeplication, and at least one management node in the existing clusters must be up and running.
Georeplication must be established between the existing cnDBTier clusters.
All the cnDBTier clusters must be installed using cnDBTier 22.3.x or above.
All the cnDBTier clusters must have the same number of data nodes and node groups.

Note:

cnDBTier supports two replication groups in each cnDBTier mate sites. To convert an existing single replication group to multi replication group, freshly install cnDBTier 22.3.x or above, or upgrade your existing cnDBTier to 22.3.x or above.
Before running this procedure, take a backup of the data and download the backup for safety purposes.
This procedure requires downtime for other cnDBTIer sites.

7.7.1 Converting Single Replication Channel Group to Multiple Replication Channel Groups

This section describes the procedure to convert a single replication channel group to multiple replication channel groups.

Note:

This procedure is not supported for cnDBTier setups where TLS is enabled.

To convert a single replication channel group to multiple replication channel groups:

Consider any one of the cnDBTier clusters as a leader cluster (cluster1) and move all the NFs' traffic to that cluster. All the pods of leader cluster must be up and running.
Wait for replication to be updated in every cnDBTier cluster so that the database is consistent across all the cnDBTier clusters.

Scale down the replication service deployments in each site: Log in to Bastion Host of each of the cnDBTier Clusters and scale down the DB replication service deployments:

$ kubectl -n <namespace of cnDBTier cluster> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <namespace of cnDBTier cluster> scale deployment --replicas=0

Example:

# scaling down the DB replication service deployments of cluster1
$ kubectl -n cluster1 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster1 scale deployment --replicas=0
deployment.apps/mysql-cluster-cluster1-cluster2-repl-svc scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-svc scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-svc scaled
 
# scaling down the DB replication service deployments of cluster2
$ kubectl -n cluster2 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster2 scale deployment --replicas=0
deployment.apps/mysql-cluster-cluster2-cluster1-repl-svc scaled
deployment.apps/mysql-cluster-cluster2-cluster3-repl-svc scaled
deployment.apps/mysql-cluster-cluster2-cluster4-repl-svc scaled 
 
# scaling down the DB replication service deployments of cluster3
$ kubectl -n cluster3 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster3 scale deployment --replicas=0
deployment.apps/mysql-cluster-cluster3-cluster1-repl-svc scaled
deployment.apps/mysql-cluster-cluster3-cluster2-repl-svc scaled
deployment.apps/mysql-cluster-cluster3-cluster3-repl-svc scaled
 
# scaling down the DB replication service deployments of cluster4
$ kubectl -n cluster4 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster4 scale deployment --replicas=0
deployment.apps/mysql-cluster-cluster4-cluster1-repl-svc scaled
deployment.apps/mysql-cluster-cluster4-cluster2-repl-svc scaled
deployment.apps/mysql-cluster-cluster4-cluster3-repl-svc scaled

Log in to the Bastion Host of each cnDBTier cluster, and stop and reset replica of all ndbmysqld pods to gracefully stop the replication channels:

$ kubectl -n <namespace of cnDBTier cluster> exec -ti <ndbmysqld pod name> -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
mysql> reset replica all;

Example to stop and reset replica of all ndbmysqld pods in cluster1:

$ kubectl -n cluster1 exec -ti ndbmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
 
$ kubectl -n cluster1 exec -ti ndbmysqld-1 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
 
$ kubectl -n cluster1 exec -ti ndbmysqld-2 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)  
 
$ kubectl -n cluster1 exec -ti ndbmysqld-3 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
 
$ kubectl -n cluster1 exec -ti ndbmysqld-4 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
 
$ kubectl -n cluster1 exec -ti ndbmysqld-5 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)

Note:

You can use the same example to stop and reset replica of all ndbmysqld pods in cluster2, cluster3, and cluster4 by replacing the cluster names with cluster2, cluster3, and cluster4 respectively.

On the leader site, delete all the entries from all the tables of the replication_info database:

$kubectl -n <namespace of leader cnDBTier cluster> exec -ti ndbmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> DELETE from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;
mysql> DELETE from replication_info.DBTIER_REPL_SITE_INFO;
mysql> DELETE from replication_info.DBTIER_SITE_INFO;
mysql> DELETE from replication_info.DBTIER_INITIAL_BINLOG_POSTION;
mysql> DELETE from replication_info.DBTIER_REPL_ERROR_SKIP_INFO;  
mysql> DELETE from replication_info.DBTIER_REPL_EVENT_INFO;

Example with output:

$kubectl -n cluster1 exec -ti ndbmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> DELETE from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;
Query OK, 24 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_REPL_SITE_INFO;
Query OK, 12 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_SITE_INFO;
Query OK, 4 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_INITIAL_BINLOG_POSTION;
Query OK, 32 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_REPL_ERROR_SKIP_INFO;
Query OK, 0 rows affected (0.00 sec)
mysql> DELETE from replication_info.DBTIER_REPL_EVENT_INFO;
Query OK, 0 rows affected (0.00 sec)

Uninstall cnDBTier Cluster2 if exists.
Uninstall cnDBTier Cluster3 if exists.
Uninstall cnDBTier Cluster4 if exists.

Note:
To uninstall cnDBTier clusters in steps 6, 7 and 8, follow the "Uninstalling DBTier" procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Make changes specific to multiple replication channel groups in the custom_values.yaml file by following the "Configuring Multiple Replication Channel Groups" procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

On updating the custom_values.yaml file, upgrade the cnDBTier leader site (cluster1) by running the following command:

$ helm upgrade  mysql-cluster --namespace <namespace of leader cnDBTier cluster> occndbtier -f occndbtier/custom_values.yaml --no-hooks

Example:

$ helm upgrade  mysql-cluster --namespace cluster1 occndbtier -f occndbtier/custom_values.yaml --no-hooks

Sample output:

Release "mysql-cluster" has been upgraded. Happy Helming!
NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 10:19:42 2023
NAMESPACE: cluster1
STATUS: deployed
REVISION: 2

Restart the management, data, and SQL pods of cnDBTier cluster by performing the following steps:

Log in to the Bastion Host of cnDBTier cluster and scale down the DB replication service deployments of the cnDBTier leader site (cluster1):

$ kubectl -n <namespace of cnDBTier cluster> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <namespace of cnDBTier cluster> scale deployment --replicas=0

Example:

# scale down the db replication service deployments of cluster1
$ kubectl -n cluster1 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster1 scale deployment --replicas=0

Sample output:

deployment.apps/mysql-cluster-cluster1-cluster2-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster2-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-grp2 scaled

Restart all the management pods of cnDBTier leader site (cluster1) by performing the following steps:

Identify the list of management pods at cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'mgmd'

Sample output:

ndbmgmd-0                                            2/2     Running             0          9m34s
ndbmgmd-1                                            2/2     Running             0          9m4s

Delete all the management pods of cnDBTier cluster1 to restart the management pods:
```
$ kubectl delete pod ndbmgmd-0 ndbmgmd-1 --namespace=cluster1
```
Sample output:
```
pod "ndbmgmd-0" deleted
pod "ndbmgmd-1" deleted
```
Wait for the management pods to restart and run the following command to check if the management pods are up and running:
```
$ kubectl get pods --namespace=cluster1 | grep 'mgmd'
```
Sample output:
```
ndbmgmd-0                                              2/2     Running   0          4m29s
ndbmgmd-1                                              2/2     Running   0          4m12s
```

Check the status of cnDBTier cluster1.

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   18 node(s)
id=56   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.113.87  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.114.32  (mysql-8.0.35 ndb-8.0.35)
id=60   @10.233.108.33  (mysql-8.0.35 ndb-8.0.35)
id=61   @10.233.78.230  (mysql-8.0.35 ndb-8.0.35)
id=62   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=63   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier) 
id=64   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=65   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=66   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=67   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The API pods with node IDs 222 to 225 in the sample output are shown as "not connected" as they are added as empty API slots for georeplication recovery. You can ignore these nodes.
The SQL pods with node IDs 62 to 67 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.

Restart the data pods sequentially by performing the following steps:

Identify the list of data pods at cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'

Sample output:

ndbmtd-0                                               3/3     Running   0          14m
ndbmtd-1                                               3/3     Running   0          13m

Delete the first data pod of cnDBTier cluster1 (in this example, ndbmtd-0) such that the pod restarts:
```
$ kubectl delete pod ndbmtd-0 --namespace=cluster1
```
Sample output:
```
pod "ndbmtd-0" deleted
```
Wait for the first data pod to restart and run the following command to check if the first pod is up and running:
```
$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'
```
Sample output:
```
ndbmtd-0                                               3/3     Running   0          65s
ndbmtd-1                                               3/3     Running   0          13m
```

Check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   18 node(s) 
id=56   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.113.87  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.114.32  (mysql-8.0.35 ndb-8.0.35)
id=60   @10.233.108.33  (mysql-8.0.35 ndb-8.0.35)
id=61   @10.233.78.230  (mysql-8.0.35 ndb-8.0.35)
id=62   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=63   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier) 
id=64   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=65   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=66   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=67   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The API pods with node IDs 222 to 225 in the sample output are shown as "not connected" as they are added as empty API slots for georeplication recovery. You can ignore these nodes.
The SQL pods with node IDs 62 to 67 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.

Follow Step ii through Step iv to delete and restart each of the remaining data pods of cnDBTier cluster1.

Log in to Bastion Host of the cnDBTier cluster and scale up the db replication service deployments of cnDBTier leader site (cluster1).

$ kubectl -n <namespace of cnDBTier cluster> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <namespace of cnDBTier cluster> scale deployment --replicas=1

Example:

# scale up the db replication service deployments of cluster1
$ kubectl -n cluster1 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster1 scale deployment --replicas=1

Sample output:

deployment.apps/mysql-cluster-cluster1-cluster2-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster2-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-grp2 scaled

Perform the following steps to restart the monitor service pod of the leader site (cnDBTier cluster1):
1. Identify the monitor service pod at cnDBTier cluster1:
```
$ kubectl get pods --namespace=cluster1 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-4cghv           1/1     Running   0                13h
```
2. Delete the monitor service pod of cnDBTier cluster1:
```
$ kubectl delete pod mysql-cluster-db-monitor-svc-8bf9448b8-4cghv --namespace=cluster1
```
  Sample output:
```
pod "mysql-cluster-db-monitor-svc-8bf9448b8-4cghv" deleted
```
3. Wait until the monitor service pod is up and running. You can check the status of the pod by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-w8pvf           1/1     Running   0                109s
```

Wait until all the pods of the leader site are up and running. Verify if the data node, API nodes, and management nodes are connected to the cnDBTier cluster by running the following command:

# Checking the status of MySQL NDB Cluster in cnDBTier cluster
$ kubectl -n <namespace of leader cnDBTier Cluster> exec -it ndbmgmd-0 -- ndb_mgm -e show

Example:

$ kubectl -n cluster1 exec ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.92.53  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.72.66  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
  
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.111.62  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.117.59  (mysql-8.0.35 ndb-8.0.35)
  
[mysqld(API)]  10 node(s)
id=56   @10.233.72.65  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.92.51  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.81.112  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.64.07  (mysql-8.0.35 ndb-8.0.35)
id=60   @10.233.71.16  (mysql-8.0.35 ndb-8.0.35)
id=61   @10.233.114.196  (mysql-8.0.35 ndb-8.0.35)
id=62   @10.233.84.212  (mysql-8.0.35 ndb-8.0.35)
id=63   @10.233.108.21  (mysql-8.0.35 ndb-8.0.35)
id=64   @10.233.121.20  (mysql-8.0.35 ndb-8.0.35)
id=65   @10.233.109.231  (mysql-8.0.35 ndb-8.0.35)
id=66   @10.233.121.37  (mysql-8.0.35 ndb-8.0.35)
id=67   @10.233.84.38  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.113.109  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

Node IDs 222 to 225 in the sample output are shown as "not connected" as they are added as empty API slots for georeplication recovery. You can ignore these nodes.

Upgrade the leader cnDBTier cluster using the custom_values.yaml file that you updated in Step 9. For more information on upgrading cnDBTier, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Scale down the replication services of cnDBTier leader site (cluster1). Log in to Bastion Host of cnDBTier Cluster and scale down the DB replication service deployments:

$ kubectl -n <namespace of cnDBTier cluster> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <namespace of cnDBTier cluster> scale deployment --replicas=0

For example, run the following command to scale down the DB replication service deployments of cluster1:

$ kubectl -n cluster1 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster1 scale deployment --replicas=0

Sample output:

deployment.apps/mysql-cluster-cluster1-cluster2-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster2-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-grp2 scaled

Delete all entries from all the tables of replication_info database on the leader site:

$kubectl -n <namespace of leader cnDBTier cluster> exec -ti ndbmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> DELETE from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;
mysql> DELETE from replication_info.DBTIER_REPL_SITE_INFO;
mysql> DELETE from replication_info.DBTIER_SITE_INFO;
mysql> DELETE from replication_info.DBTIER_INITIAL_BINLOG_POSTION;
mysql> DELETE from replication_info.DBTIER_REPL_ERROR_SKIP_INFO;  
mysql> DELETE from replication_info.DBTIER_REPL_EVENT_INFO;

Example with output:

$kubectl -n cluster1 exec -ti ndbmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> DELETE from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;
Query OK, 24 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_REPL_SITE_INFO;
Query OK, 12 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_SITE_INFO;
Query OK, 4 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_INITIAL_BINLOG_POSTION;
Query OK, 32 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_REPL_ERROR_SKIP_INFO;
Query OK, 0 rows affected (0.00 sec)
mysql> DELETE from replication_info.DBTIER_REPL_EVENT_INFO;
Query OK, 0 rows affected (0.00 sec)

Scale up the replication services of cnDBTier leader site (cluster1). Log in to Bastion Host of cnDBTier cluster and scale up the DB replication service deployments:

$ kubectl -n <namespace of cnDBTier cluster> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <namespace of cnDBTier cluster> scale deployment --replicas=1

For example, run the following command to scale up the DB replication service deployments of cluster1:

$ kubectl -n cluster1 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster1 scale deployment --replicas=1

Sample output:

deployment.apps/mysql-cluster-cluster1-cluster2-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster2-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-grp2 scaled

Reinstall cnDBTier Cluster2 by following the Adding cnDBTier Georedundant Cluster to Single Site cnDBTier Cluster procedure.
Reinstall cnDBTier Cluster3 by following the Adding cnDBTier Georedundant Cluster to Two Site cnDBTier Clusters procedure.
Reinstall cnDBTier Cluster4 by following the Adding cnDBTier Georedundant Cluster to Four Site cnDBTier Clusters procedure.

7.7.2 Converting Multiple Replication Channel Groups to Single Replication Channel Group

This section describes the procedure to convert multiple replication channel groups to single replication channel group.

Note:

This procedure is not supported for cnDBTier setups where TLS is enabled.

This procedure can be used if multiple replication channel groups are enabled in the cnDBTier cluster and if you want to convert it to a single replication group due to a rollback to the previous versions or any other reasons.

To convert multiple replication channel groups to single replication channel group:

Consider any one of the cnDBTier clusters as a leader cluster (cluster1 ) and move all the NFs' traffic to that cluster. All the pods of leader cluster must be up and running.
Wait for replication to be updated in every cnDBTier cluster so that the database is consistent across all the cnDBTier clusters.

Scale down the replication service deployments in each site: Log in to Bastion Host of each of the cnDBTier Clusters and scale down the DB replication service deployments:

$ kubectl -n <namespace of cnDBTier cluster> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <namespace of cnDBTier cluster> scale deployment --replicas=0

Example:

# scaling down the DB replication service deployments of cluster1
$ kubectl -n cluster1 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster1 scale deployment --replicas=0
deployment.apps/mysql-cluster-cluster1-cluster2-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster2-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-grp2 scaled
 
# scaling down the DB replication service deployments of cluster2 If exists
$ kubectl -n cluster2 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster2 scale deployment --replicas=0
deployment.apps/mysql-cluster-cluster2-cluster1-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster2-cluster1-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster2-cluster3-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster2-cluster3-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster2-cluster4-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster2-cluster4-repl-grp2 scaled
 
# scaling down the DB replication service deployments of cluster3 If exists
$ kubectl -n cluster3 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster3 scale deployment --replicas=0
deployment.apps/mysql-cluster-cluster3-cluster1-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster3-cluster1-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster3-cluster2-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster3-cluster2-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster3-cluster4-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster3-cluster4-repl-grp2 scaled
 
# scaling down the DB replication service deployments of cluster4 If exists
$ kubectl -n cluster4 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster4 scale deployment --replicas=0
deployment.apps/mysql-cluster-cluster4-cluster1-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster4-cluster1-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster4-cluster2-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster4-cluster2-repl-grp2 scaled
deployment.apps/mysql-cluster-cluster4-cluster3-repl-grp1 scaled
deployment.apps/mysql-cluster-cluster4-cluster3-repl-grp2 scaled

Log in to the Bastion Host of each cnDBTier cluster, and stop and reset replica of all ndbmysqld pods to gracefully stop the replication channels:

$ kubectl -n <namespace of cnDBTier cluster> exec -ti <ndbmysqld pod name> -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
mysql> reset replica all;

Example to stop and reset replica of all ndbmysqld pods in cluster1:

$ kubectl -n cluster1 exec -ti ndbmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
  
$ kubectl -n cluster1 exec -ti ndbmysqld-1 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
 
$ kubectl -n cluster1 exec -ti ndbmysqld-2 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
  
$ kubectl -n cluster1 exec -ti ndbmysqld-3 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
 
$ kubectl -n cluster1 exec -ti ndbmysqld-4 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
  
$ kubectl -n cluster1 exec -ti ndbmysqld-5 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
 
$ kubectl -n cluster1 exec -ti ndbmysqld-6 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
  
$ kubectl -n cluster1 exec -ti ndbmysqld-7 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
 
$ kubectl -n cluster1 exec -ti ndbmysqld-8 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
  
$ kubectl -n cluster1 exec -ti ndbmysqld-9 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
 
$ kubectl -n cluster1 exec -ti ndbmysqld-10 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)
  
$ kubectl -n cluster1 exec -ti ndbmysqld-11 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> stop replica;
Query OK, 0 rows affected (0.01 sec)
mysql> reset replica all;
Query OK, 0 rows affected (0.01 sec)

Note:

You can use the same example to stop and reset replica of all ndbmysqld pods in cluster2, cluster3, and cluster4 by replacing the cluster names with cluster2, cluster3, and cluster4 respectively.

On the leader site, delete all the entries from all the tables of the replication_info database:

$kubectl -n <namespace of leader cnDBTier cluster> exec -ti ndbmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> DELETE from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;
mysql> DELETE from replication_info.DBTIER_REPL_SITE_INFO;
mysql> DELETE from replication_info.DBTIER_SITE_INFO;
mysql> DELETE from replication_info.DBTIER_INITIAL_BINLOG_POSTION;
mysql> DELETE from replication_info.DBTIER_REPL_ERROR_SKIP_INFO;  
mysql> DELETE from replication_info.DBTIER_REPL_EVENT_INFO;

Example with output:

$kubectl -n cluster1 exec -ti ndbmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> DELETE from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;
Query OK, 4 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_REPL_SITE_INFO;
Query OK, 2 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_SITE_INFO;
Query OK, 2 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_INITIAL_BINLOG_POSTION;
Query OK, 8 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_REPL_ERROR_SKIP_INFO;
Query OK, 0 rows affected (0.00 sec)
mysql> DELETE from replication_info.DBTIER_REPL_EVENT_INFO;
Query OK, 0 rows affected (0.00 sec)

Uninstall cnDBTier Cluster2 if exists.
Uninstall cnDBTier Cluster3 if exists.
Uninstall cnDBTier Cluster4 if exists.

Note:
To uninstall cnDBTier clusters in steps 6, 7 and 8, follow the "Uninstalling DBTier" procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Remove or disable the configurations specific to multiple replication channel groups from the custom_values.yaml file.

On updating the custom_values.yaml file, upgrade the cnDBTier leader site (cluster1) by running the following command:

$ helm upgrade  mysql-cluster --namespace <namespace of leader cnDBTier cluster> occndbtier -f occndbtier/custom_values.yaml --no-hooks

Example:

$ helm upgrade  mysql-cluster --namespace cluster1 occndbtier -f occndbtier/custom_values.yaml --no-hooks

Sample output:

Release "mysql-cluster" has been upgraded. Happy Helming!
NAME: mysql-cluster
LAST DEPLOYED: Tue Nov 07 10:19:42 2023
NAMESPACE: cluster1
STATUS: deployed
REVISION: 2

Restart the management, data, and SQL pods of cnDBTier cluster by performing the following steps:

Log in to the Bastion Host of cnDBTier cluster and scale down the DB replication service deployments of the cnDBTier leader site (cluster1):

$ kubectl -n <namespace of cnDBTier cluster> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <namespace of cnDBTier cluster> scale deployment --replicas=0

Example:

# scale down the db replication service deployments of cluster1
$ kubectl -n cluster1 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster1 scale deployment --replicas=0

Sample output:

deployment.apps/mysql-cluster-cluster1-cluster2-repl-svc scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-svc scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-svc scaled

Restart all the management pods of cnDBTier leader site (cluster1) by performing the following steps:

Identify the list of management pods at cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'mgmd'

Sample output:

ndbmgmd-0                                            2/2     Running             0          9m34s
ndbmgmd-1                                            2/2     Running             0          9m4s

Delete all the management pods of cnDBTier cluster1 to restart the management pods:
```
$ kubectl delete pod ndbmgmd-0 ndbmgmd-1 --namespace=cluster1
```
Sample output:
```
pod "ndbmgmd-0" deleted
pod "ndbmgmd-1" deleted
```
Wait for the management pods to restart and run the following command to check if the management pods are up and running:
```
$ kubectl get pods --namespace=cluster1 | grep 'mgmd'
```
Sample output:
```
ndbmgmd-0                                              2/2     Running   0          4m29s
ndbmgmd-1                                              2/2     Running   0          4m12s
```

Check the status of cnDBTier cluster1.

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   18 node(s)
id=56   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.113.87  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.114.32  (mysql-8.0.35 ndb-8.0.35)
id=60   @10.233.108.33  (mysql-8.0.35 ndb-8.0.35)
id=61   @10.233.78.230  (mysql-8.0.35 ndb-8.0.35)
id=62   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=63   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier) 
id=64   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=65   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=66   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=67   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The API pods with node IDs 222 to 225 in the sample output are shown as "not connected" as they are added as empty API slots for georeplication recovery. You can ignore these nodes.
The SQL pods with node IDs 62 to 67 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.

Restart the data pods sequentially by performing the following steps:

Identify the list of data pods at cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'

Sample output:

ndbmtd-0                                               3/3     Running   0          14m
ndbmtd-1                                               3/3     Running   0          13m

Delete first data pod of cnDBTier cluster1 (ndbmtd-0) such that the pod restarts:
```
$ kubectl delete pod ndbmtd-0 --namespace=cluster1
```
Sample output:
```
pod "ndbmtd-0" deleted
```
Wait for the first data pod to restart and run the following command to check if the first pod is up and running:
```
$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'
```
Sample output:
```
ndbmtd-0                                               3/3     Running   0          65s
ndbmtd-1                                               3/3     Running   0          13m
```

Check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
    
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
    
[mysqld(API)]   18 node(s)
id=56   @10.233.124.92  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.114.135  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.113.87  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.114.32  (mysql-8.0.35 ndb-8.0.35)
id=60   @10.233.108.33  (mysql-8.0.35 ndb-8.0.35)
id=61   @10.233.78.230  (mysql-8.0.35 ndb-8.0.35)
id=62   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=63   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier) 
id=64   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=65   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=66   (not connected, accepting connect from ndbmysqld-2.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=67   (not connected, accepting connect from ndbmysqld-3.ndbmysqldsvc.cluster1.svc.occne4-cgbu-cne-dbtier)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

The API pods with node IDs 222 to 225 in the sample output are shown as "not connected" as they are added as empty API slots for georeplication recovery. You can ignore these nodes.
The SQL pods with node IDs 62 to 67 in the sample output are the newly added pods. These pods remain in the not connected state until all the data nodes are restarted.

Follow Step ii through Step iv to delete and restart the second data pod of cnDBTier cluster1 (ndbmtd-1).

Log in to Bastion Host of the cnDBTier cluster and scale up the db replication service deployments of cnDBTier leader site (cluster1).

$ kubectl -n <namespace of cnDBTier cluster> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <namespace of cnDBTier cluster> scale deployment --replicas=1

Example:

# scale up the db replication service deployments of cluster1
$ kubectl -n cluster1 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster1 scale deployment --replicas=1

Sample output:

deployment.apps/mysql-cluster-cluster1-cluster2-repl-svc scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-svc scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-svc scaled

Perform the following steps to restart the monitor service pod of the leader site (cnDBTier cluster1):
1. Identify the monitor service pod at cnDBTier cluster1:
```
$ kubectl get pods --namespace=cluster1 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-4cghv           1/1     Running   0                13h
```
2. Delete the monitor service pod of cnDBTier cluster1:
```
$ kubectl delete pod mysql-cluster-db-monitor-svc-8bf9448b8-4cghv --namespace=cluster1
```
  Sample output:
```
pod "mysql-cluster-db-monitor-svc-8bf9448b8-4cghv" deleted
```
3. Wait until the monitor service pod is up and running. You can check the status of the pod by running the following command:
```
$ kubectl get pods --namespace=cluster1 | grep 'monitor'
```
  Sample output:
```
mysql-cluster-db-monitor-svc-8bf9448b8-w8pvf           1/1     Running   0                109s
```

Wait until all the pods of the leader site are up and running. Verify if the data node, API nodes, and management nodes are connected to the cnDBTier cluster by running the following command:

$ kubectl -n <namespace of leader cnDBTier Cluster> exec -it ndbmgmd-0 -- ndb_mgm -e show

Example:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.92.53  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.72.66  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
   
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.111.62  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.117.59  (mysql-8.0.35 ndb-8.0.35)
   
[mysqld(API)]  10 node(s)
id=56   @10.233.72.65  (mysql-8.0.35 ndb-8.0.35)
id=57   @10.233.92.51  (mysql-8.0.35 ndb-8.0.35)
id=58   @10.233.113.87  (mysql-8.0.35 ndb-8.0.35)
id=59   @10.233.114.32  (mysql-8.0.35 ndb-8.0.35)
id=60   @10.233.108.33  (mysql-8.0.35 ndb-8.0.35)
id=61   @10.233.78.230  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.72.64  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.92.52  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

Node IDs 222 to 225 in the sample output are shown as "not connected" as they are added as empty API slots for georeplication recovery. You can ignore these nodes.

Upgrade the leader cnDBTier cluster using the custom_values.yaml file that you updated in Step 9. For more information on upgrading cnDBTier, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Scale down the replication services of cnDBTier leader site (cluster1). Log in to Bastion Host of cnDBTier Cluster and scale down the DB replication service deployments:
```
$ kubectl -n <namespace of cnDBTier cluster> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <namespace of cnDBTier cluster> scale deployment --replicas=0
```
For example, run the following command to scale down the DB replication service deployments of cluster1:
```
$ kubectl -n cluster1 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster1 scale deployment --replicas=0
```
Sample output:
```
deployment.apps/mysql-cluster-cluster1-cluster2-repl-svc scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-svc scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-svc scaled
```

Delete all entries from all the tables of replication_info database on the leader site:

$kubectl -n <namespace of leader cnDBTier cluster> exec -ti ndbmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> DELETE from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;
mysql> DELETE from replication_info.DBTIER_REPL_SITE_INFO;
mysql> DELETE from replication_info.DBTIER_SITE_INFO;
mysql> DELETE from replication_info.DBTIER_INITIAL_BINLOG_POSTION;
mysql> DELETE from replication_info.DBTIER_REPL_ERROR_SKIP_INFO;  
mysql> DELETE from replication_info.DBTIER_REPL_EVENT_INFO;

Example with output:

$kubectl -n cluster1 exec -ti ndbmysqld-0 -- mysql -h 127.0.0.1 -uroot -pNextGenCne
mysql> DELETE from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;
Query OK, 4 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_REPL_SITE_INFO;
Query OK, 2 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_SITE_INFO;
Query OK, 2 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_INITIAL_BINLOG_POSTION;
Query OK, 8 rows affected (0.01 sec)
mysql> DELETE from replication_info.DBTIER_REPL_ERROR_SKIP_INFO;
Query OK, 0 rows affected (0.00 sec)
mysql> DELETE from replication_info.DBTIER_REPL_EVENT_INFO;
Query OK, 0 rows affected (0.00 sec)

Scale up the replication services of cnDBTier leader site (cluster1). Log in to Bastion Host of cnDBTier cluster and scale up the DB replication service deployments:
```
$ kubectl -n <namespace of cnDBTier cluster> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <namespace of cnDBTier cluster> scale deployment --replicas=1
```
For example, run the following command to scale up the DB replication service deployments of cluster1:
```
$ kubectl -n cluster1 get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n cluster1 scale deployment --replicas=1
```
Sample output:
```
deployment.apps/mysql-cluster-cluster1-cluster2-repl-svc scaled
deployment.apps/mysql-cluster-cluster1-cluster3-repl-svc scaled
deployment.apps/mysql-cluster-cluster1-cluster4-repl-svc scaled
```
Reinstall cnDBTier Cluster2 by following the Adding cnDBTier Georedundant Cluster to Single Site cnDBTier Cluster procedure.
Reinstall cnDBTier Cluster3 by following the Adding cnDBTier Georedundant Cluster to Two Site cnDBTier Clusters procedure.
Reinstall cnDBTier Cluster4 by following the Adding cnDBTier Georedundant Cluster to Three Site cnDBTier Clusters procedure.

7.8 Managing Passwords and Secrets

This section provides the procedures to manage cnDBTier passwords and secrets.

7.8.1 Changing cnDBTier Passwords

This section provides the procedures to change cnDBTier passwords such as root user, occneuser, and occnerepluser passwords in a stand-alone and multisite georeplication setup using the dbtpasswd bash script.

Note:

The password changes are applicable only to the current site and are not replicated in the mate sites.
Ensure that your password meets the following password policy requirements:
- Password must be between 20 and 32 characters in length.
- Password must contain at least one lower case letter.
- Password must contain at least one upper case letter.
- Password must include at least one digit.
- Password must include at least one of the following special characters: ,%~+.:_/-

Changing a cnDBTier password involves the following steps:

Adding a new password in MySQL.

Note:
The old password is active until the pods are restarted and the transitional operations are complete.
Replacing the current or old password with the new password in Kubernetes secret.
Restarting the configured cnDBTier pods to use the new Kubernetes secret (This step is not applicable while changing an NF password).
Discarding the old password in MySQL.

The dbtpasswd bash script automates these steps and provides a single script to change all cnDBTier passwords at once. The script also provides options to run selective steps as changing passwords often requires running selective transitional operations, such as restarting pods, while both passwords are still valid.

You can use dbtpasswd to:

change one or more cnDBTier passwords in a single site or cluster. Changing a cnDBTier password on a site includes changing the password in MySQL and Kubernetes secret, restarting all required cnDBTier pods, and updating passwords on cnDBTier database tables if necessary.
change passwords on a live cluster with no service interruption.
change NF passwords. However, when changing an NF password, dbtpasswd can change the password in the Kubernetes secret and database only. You have to manually restart NF pods as a separate user action.

To provide flexibility and support for changing non-cnDBTier passwords (that is, NF passwords), dbtpasswd provides a range of options for controlling the phases of execution. For example, you can configure dbtpasswd to only add the new password to the database, change it in the Kubernetes secret, and then stop. After running the other operations, you can configure the script and run it again to discard the old password only.

MySQL Dual Password

The dbtpasswd script uses the MySQL Dual Password feature to first add the new password and then discard the old MySQL password after database values and secrets are changed and the pods are restarted. Both the passwords are valid until the old password is discarded.

Note:

MySQL Dual Password is not supported for changing the root password.

Support for Non-cnDBTier Secrets on Different Namespace

To support changing NF passwords, that have their secrets on a namespace different from the cnDBTier namespace, dbtpasswd provides the --nf-namespace to configure the namespace in which the secrets are stored. For more information about this option, see Configuration Options.

7.8.1.1 Prerequisites

Before changing the cnDBTier passwords ensure that the following prerequisites are met:

The dbtpasswd scripti must be properly installed. For more information, see Installing and Using dbtpasswd Script.
If you are changing the replication password in a multisite setup, then ensure that the DB Replication is up and running between all of the sites in the system.

7.8.1.2 Installing and Using dbtpasswd Script

This section provides details about installing and using the dbtpasswd script to change cnDBTier passwords.

Installing the dbtpasswd Script

Run the following commands to source and add the bin directory containing the dbtpasswd program to the user's path.

cd Artifacts/Scripts/tools
source ./source_me

The system prompts you to enter the namespace and uses the same to set the DBTIER_NAMESPACE. It also sets the DBTIER_LIB environment variable with the path to the directory containing the libraries needed by dbtpasswd.

Using the dbtpasswd Script

You can use the dbtpasswd script in the following ways:

dbtpasswd [-h | --help]
dbtpasswd [OPTIONS] [SECRET...]

7.8.1.3 Configuration Options

This section describes the list of options that are available to configure the dbtpasswd script.

Table 7-1 Configuration Options

Option	Usage	Example
-h \| --help	This option is used to print the help message and exit.	`dbtpasswd --help`
-v \| --version	This option is used to print script's version.	`dbtpasswd --version`
--debug	This option is used to output DEBUG log message to standard error, stderr.	`dbtpasswd --debug`
--skip-namespace-test	This option is used to skip testing that the namespace in DBTIER_NAMESPACE exists in the current cluster.	`dbtpasswd --skip-namespace-test`
--rollout-watch-timeout=0s	This option is used to define the time to wait before ending the rollout status watch. Set this value to zero if you donot want to wait before ending the rollout status watch. Use a corresponding time unit to set other values. For example: 1s, 2m, 3h.	`dbtpasswd --rollout-watch-timeout=0s`
--no-checks	This option is used to skip verifying if replication to mate sites is UP, when updating a replication password.	`dbtpasswd --no-checks`
--no-discard	This option is used to change a password without discarding the old password.	To change all cnDBTier passwords, but retain the old passwords: `dbtpasswd --no-discard`
--discard-only	This option is used to only discard the old password.	`dbtpasswd --discard-only`
--secrets-only	This option is used to change the password in the secrets only.	`dbtpasswd --secrets-only`
--mysql-only	This option is used to change the passwords in MySQL only. Note: The current secret password must be the current MySQL password. Therefore, if you are using the `--mysql-only` and `--secrets-only` options to change the passwords, you must first change the MySQL password and then the secret password.	`dbtpasswd --mysql-only`
--secrets-and-mysql-only	This option is used to change the passwords in the secrets and MySQL only.	`dbtpasswd --secrets-and-mysql-only`
--restart-only	This option is used to only restart the pods configured to use cnDBTier secrets and doesn't change the passwords.	`dbtpasswd --restart-only`
--nf-namespace=<namespace_where_secrets_are>	Non-cnDBTier secrets may be stored on a namespace different from the cnDBTier namespace. This option is used to provide the details of the namespace where the non-cnDBTier secrets are stored to support changing NF passwords. Note: All non-cnDBTier secrets must be stored in the NF namespace. All cnDBTier secrets must be stored in DBTIER_NAMESPACE.	To change a non-DBTier password in its secret, which is on a different namespace, and in mysql: `dbtpasswd --secrets-and-mysql-only --nf-namespace=other-namespace-name nf-secret-on-diff-namespace` To discard a non-DBTier old password whose secret is on a different namespace: `dbtpasswd --discard-only --nf-namespace=other-namespace-name nf-secret-on-diff-namespace` To change a non-DBTier password in its secret, which is on same namespace as cnDBTier, and in mysql: `dbtpasswd --secrets-and-mysql-only nf-secret-on-same-namespace` To discard a non-DBTier old password whose secret is on the same namespace as cnDBTier: `dbtpasswd --discard-only nf-secret-on-same-namespace`

Note:

Use the dbtpasswd --help command to refer to more examples.

7.8.1.4 Changing All cnDBTier Passwords in a Site

Run the following command to change all the cnDBTier passwords in a cnDBTier site:

$ dbtpasswd

Sample output:

2022-12-23T21:54:24Z INFO - Changing password for user root
Current password:
Enter new password:
Enter new password again:
2022-12-23T21:54:39Z INFO - Changing password for user occneuser
Current password:
Enter new password:
Enter new password again:
2022-12-23T21:54:58Z INFO - Changing password for user occnerepluser
Current password:
Enter new password:
Enter new password again:
2022-12-23T21:55:05Z INFO - Getting sts and sts pod info...
2022-12-23T21:55:12Z INFO - MGM_STS="ndbmgmd"
2022-12-23T21:55:12Z INFO - MGM_REPLICAS="2"
2022-12-23T21:55:12Z INFO -
    ndbmgmd-0
    ndbmgmd-1
2022-12-23T21:55:18Z INFO - NDB_STS="ndbmtd"
2022-12-23T21:55:18Z INFO - NDB_REPLICAS="2"
2022-12-23T21:55:18Z INFO -
    ndbmtd-0
    ndbmtd-1
2022-12-23T21:55:25Z INFO - API_STS="ndbmysqld"
2022-12-23T21:55:25Z INFO - API_REPLICAS="2"
2022-12-23T21:55:25Z INFO -
    ndbmysqld-0
    ndbmysqld-1
2022-12-23T21:55:29Z INFO - APP_STS="ndbappmysqld"
2022-12-23T21:55:29Z INFO - APP_REPLICAS="2"
2022-12-23T21:55:29Z INFO -
    ndbappmysqld-0
    ndbappmysqld-1
2022-12-23T21:55:29Z INFO - Getting deployment pod info...
2022-12-23T21:55:29Z INFO - grepping for backup-man (BAK_CHART_NAME)...
2022-12-23T21:55:36Z INFO -
    mysql-cluster-db-backup-manager-svc-7bb947f5f9-nc45s
2022-12-23T21:55:36Z INFO -
    mysql-cluster-db-backup-manager-svc
2022-12-23T21:55:36Z INFO - grepping for db-mon (MON_CHART_NAME)...
2022-12-23T21:55:42Z INFO -
    mysql-cluster-db-monitor-svc-5cff4bf789-g86rr
2022-12-23T21:55:42Z INFO -
    mysql-cluster-db-monitor-svc
2022-12-23T21:55:42Z INFO - grepping for replicat (REP_CHART_NAME)...
2022-12-23T21:55:47Z INFO -
    mysql-cluster-lfg-site-1-lfg-site-2-replication-svc-b4f8d9g6hbc
2022-12-23T21:55:47Z INFO -
    mysql-cluster-lfg-site-1-lfg-site-2-replication-svc
2022-12-23T21:55:47Z INFO - Labeling pods with dbtier-app...
pod/ndbmgmd-0 not labeled
pod/ndbmgmd-1 not labeled
pod/ndbmtd-0 not labeled
pod/ndbmtd-1 not labeled
pod/ndbappmysqld-0 not labeled
pod/ndbappmysqld-1 not labeled
pod/ndbmysqld-0 not labeled
pod/ndbmysqld-1 not labeled
pod/mysql-cluster-db-backup-manager-svc-7bb947f5f9-nc45s not labeled
pod/mysql-cluster-db-monitor-svc-5cff4bf789-g86rr not labeled
pod/mysql-cluster-lfg-site-1-lfg-site-2-replication-svc-b4f8d9g6hbc not labeled
2022-12-23T21:56:08Z INFO - Pods labeled with dbtier-app
2022-12-23T21:56:08Z INFO - Verifying Geo Replication to mates...
2022-12-23T21:56:18Z INFO - Geo Replication to mates is UP
2022-12-23T21:56:18Z INFO - Changing mysql password for 'root'@'localhost'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-23T21:56:29Z INFO - Mysql password changed
2022-12-23T21:56:33Z INFO - Patching secret, occne-mysqlndb-root-secret, with new password
secret/occne-mysqlndb-root-secret patched
2022-12-23T21:56:36Z INFO - Secret, occne-mysqlndb-root-secret, patched with new password
2022-12-23T21:56:36Z INFO - Adding new mysql password for 'occneuser'@'%'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-23T21:56:48Z INFO - New mysql password added
2022-12-23T21:56:54Z INFO - Patching secret, occne-secret-db-monitor-secret, with new password
secret/occne-secret-db-monitor-secret patched
2022-12-23T21:56:58Z INFO - Secret, occne-secret-db-monitor-secret, patched with new password
2022-12-23T21:56:58Z INFO - Adding new mysql password for 'occnerepluser'@'%'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-23T21:57:10Z INFO - New mysql password added
2022-12-23T21:57:10Z INFO - Changing password in replication_info.DBTIER_REPL_SITE_INFO table...
2022-12-23T21:57:16Z INFO - Using replication pod: mysql-cluster-lfg-site-1-lfg-site-2-replication-svc-b4f8d9g6hbc
2022-12-23T21:57:16Z INFO - Using replication pod container: lfg-site-1-lfg-site-2-replication-svc
2022-12-23T21:57:16Z INFO - MYSQL_REPLICATION_SITE_NAME = lfg-site-1
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-23T21:57:28Z INFO - Password changed in replication_info.DBTIER_REPL_SITE_INFO table
2022-12-23T21:57:35Z INFO - Patching secret, occne-replication-secret-db-replication-secret, with new password
secret/occne-replication-secret-db-replication-secret patched
2022-12-23T21:57:38Z INFO - Secret, occne-replication-secret-db-replication-secret, patched with new password
2022-12-23T21:57:38Z INFO - Starting rollover restarts at 2022-12-23T21:57:38Z
2022-12-23T21:57:38Z INFO - number of db-replication-svc deployments: 1
2022-12-23T21:57:38Z INFO - Patching deployment 0: mysql-cluster-lfg-site-1-lfg-site-2-replication-svc...
deployment.apps/mysql-cluster-lfg-site-1-lfg-site-2-replication-svc patched (no change)
2022-12-23T21:57:41Z INFO - Rollout restarting mysql-cluster-lfg-site-1-lfg-site-2-replication-svc...
deployment.apps/mysql-cluster-lfg-site-1-lfg-site-2-replication-svc restarted
...
2022-12-23T22:01:53Z INFO - ndbmtd pods rollout restarted
...
2022-12-23T22:02:11Z INFO - ndbappmysqld pods rollout restarted
...
2022-12-23T22:02:27Z INFO - ndbmysqld pods rollout restarted
...
2022-12-23T22:02:48Z INFO - db-backup-manager-svc pods rollout restarted
...
2022-12-23T22:03:08Z INFO - db-monitor-svc pods rollout restarted
2022-12-23T22:03:08Z INFO - Discarding old mysql password for 'occneuser'@'%'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-23T22:03:20Z INFO - Old mysql password discarded
2022-12-23T22:03:20Z INFO - Discarding old mysql password for 'occnerepluser'@'%'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-23T22:03:30Z INFO - Old mysql password discarded
2022-12-23T22:03:30Z INFO - Password(s) updated successfully

7.8.1.5 Changing All Passwords in a Stand-Alone or Georeplication Site

This section provides the command to change all cnDBTier passwords in a stand-alone site or a site where Georeplication has been configured but the mate sites are not configured.

$ dbtpasswd

Sample output:

2023-10-12T21:34:05Z INFO - DBTIER_NAMESPACE = occne-cndbtier
2023-10-12T21:34:05Z INFO - Testing namespace, occne-cndbtier, exists...
2023-10-12T21:34:05Z INFO - Should be able to see namespace, occne-cndbtier, with "kubectl get ns -o name occne-cndbtier" - PASSED
2023-10-12T21:34:05Z INFO - Namespace, occne-cndbtier, exists
2023-10-12T21:34:05Z INFO - Changing password for user root 2023-01-03T19:06:31Z INFO - Changing password for user root
Current password:
Enter new password:
Enter new password again:
2023-10-12T21:34:06Z INFO - Changing password for user occneuser
Current password:
Enter new password:
Enter new password again:
2023-10-12T21:34:06Z INFO - Changing password for user occnerepluser
Current password:
Enter new password:
Enter new password again: 2023-10-12T21:34:06Z INFO - Getting sts and sts pod info...
2023-10-12T21:34:06Z INFO - Getting MGM sts and sts pod info...
2023-10-12T21:34:07Z INFO - MGM_STS="ndbmgmd"
2023-10-12T21:34:07Z INFO - MGM_REPLICAS="2"
2023-10-12T21:34:07Z INFO - MGM_PODS:
ndbmgmd-0
ndbmgmd-1
2023-10-12T21:34:07Z INFO - Getting NDB sts and sts pod info...
2023-10-12T21:34:07Z INFO - NDB_STS="ndbmtd"
2023-10-12T21:34:07Z INFO - NDB_REPLICAS="2"
2023-10-12T21:34:07Z INFO - NDB_PODS:
ndbmtd-0
ndbmtd-1
2023-10-12T21:34:07Z INFO - Getting API sts and sts pod info...
2023-10-12T21:34:07Z INFO - API_STS="ndbmysqld"
2023-10-12T21:34:07Z INFO - API_REPLICAS="6"
2023-10-12T21:34:07Z INFO - API_PODS:
ndbmysqld-0
ndbmysqld-1
ndbmysqld-2
ndbmysqld-3
ndbmysqld-4
ndbmysqld-5
2023-10-12T21:34:07Z INFO - Getting APP sts and sts pod info...
2023-10-12T21:34:07Z INFO - APP_STS="ndbappmysqld"
2023-10-12T21:34:07Z INFO - APP_REPLICAS="2"
2023-10-12T21:34:07Z INFO - APP_PODS:
ndbappmysqld-0
ndbappmysqld-1
2023-10-12T21:34:07Z INFO - Getting deployment pod info...
2023-10-12T21:34:07Z INFO - grepping for backup-man (BAK_CHART_NAME)...
2023-10-12T21:34:07Z INFO - BAK_PODS:
mysql-cluster-db-backup-manager-svc-78fdcdfd98-gpml2
2023-10-12T21:34:07Z INFO - BAK_DEPLOY:
mysql-cluster-db-backup-manager-svc
2023-10-12T21:34:07Z INFO - grepping for db-mon (MON_CHART_NAME)...
2023-10-12T21:34:07Z INFO - MON_PODS:
mysql-cluster-db-monitor-svc-ccc9bfbfd-5z45b
2023-10-12T21:34:07Z INFO - MON_DEPLOY:
mysql-cluster-db-monitor-svc
2023-10-12T21:34:07Z INFO - grepping for repl (REP_CHART_NAME)...
2023-10-12T21:34:08Z INFO - REP_PODS:
mysql-cluster-lfg-site-1-lfg-site-2-replication-svc-67d96bg9z5m
mysql-cluster-lfg-site-1-lfg-site-3-replication-svc-8f77b9xrrqt
mysql-cluster-lfg-site-1-lfg-site-4-replication-svc-ddc647dt78q
2023-10-12T21:34:08Z INFO - REP_DEPLOY:
mysql-cluster-lfg-site-1-lfg-site-2-replication-svc
mysql-cluster-lfg-site-1-lfg-site-3-replication-svc
mysql-cluster-lfg-site-1-lfg-site-4-replication-svc
2023-10-12T21:34:08Z INFO - Labeling pods with dbtier-app...
pod/ndbmgmd-0 labeled
pod/ndbmgmd-1 labeled
pod/ndbmtd-0 labeled
pod/ndbmtd-1 labeled
pod/ndbappmysqld-0 labeled
pod/ndbappmysqld-1 labeled
pod/ndbmysqld-0 labeled
pod/ndbmysqld-1 labeled
pod/ndbmysqld-2 labeled
pod/ndbmysqld-3 labeled
pod/ndbmysqld-4 labeled
pod/ndbmysqld-5 labeled
pod/mysql-cluster-db-backup-manager-svc-78fdcdfd98-gpml2 labeled
pod/mysql-cluster-db-monitor-svc-ccc9bfbfd-5z45b labeled
pod/mysql-cluster-lfg-site-1-lfg-site-2-replication-svc-67d96bg9z5m labeled
pod/mysql-cluster-lfg-site-1-lfg-site-3-replication-svc-8f77b9xrrqt labeled
pod/mysql-cluster-lfg-site-1-lfg-site-4-replication-svc-ddc647dt78q labeled
2023-10-12T21:34:41Z INFO - Pods labeled with dbtier-app
2023-10-12T21:34:42Z INFO - DBTIER_REPL_SITE_INFO table is empty (num_of_recs=0); indicating SINGLE SITE SETUP...
2023-10-12T21:34:42Z INFO - Changing mysql password for 'root'@'localhost'...
2023-10-12T21:34:43Z INFO - Mysql password changed
2023-10-12T21:34:43Z INFO - Patching secret, occne-mysqlndb-root-secret, with new password
secret/occne-mysqlndb-root-secret patched
2023-10-12T21:34:43Z INFO - Secret, occne-mysqlndb-root-secret, patched with new password
2023-10-12T21:34:43Z INFO - Adding new mysql password for 'occneuser'@'%'...
2023-10-12T21:34:43Z INFO - New mysql password added
2023-10-12T21:34:43Z INFO - Patching secret, occne-secret-db-monitor-secret, with new password
secret/occne-secret-db-monitor-secret patched
2023-10-12T21:34:44Z INFO - Secret, occne-secret-db-monitor-secret, patched with new password
2023-10-12T21:34:44Z INFO - Adding new mysql password for 'occnerepluser'@'%'...
2023-10-12T21:34:44Z INFO - New mysql password added
2023-10-12T21:34:44Z INFO - Patching secret, occne-replication-secret-db-replication-secret, with new password
secret/occne-replication-secret-db-replication-secret patched
2023-10-12T21:34:44Z INFO - Secret, occne-replication-secret-db-replication-secret, patched with new password
2023-10-12T21:34:44Z INFO - Starting rollover restarts at 2023-10-12T21:34:44Z
2023-10-12T21:34:44Z INFO - number of db-replication-svc deployments: 3
2023-10-12T21:34:44Z INFO - Patching deployment 0: mysql-cluster-lfg-site-1-lfg-site-2-replication-svc...
2023-10-12T21:34:47Z INFO - Patching deployment 1: mysql-cluster-lfg-site-1-lfg-site-3-replication-svc...
2023-10-12T21:34:47Z INFO - Patching deployment 2: mysql-cluster-lfg-site-1-lfg-site-4-replication-svc...
2023-10-12T21:34:48Z INFO - Waiting for deployment mysql-cluster-lfg-site-1-lfg-site-2-replication-svc to rollout restart...
Waiting for deployment "mysql-cluster-lfg-site-1-lfg-site-2-replication-svc" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "mysql-cluster-lfg-site-1-lfg-site-2-replication-svc" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "mysql-cluster-lfg-site-1-lfg-site-2-replication-svc" rollout to finish: 0 out of 1 new replicas have been updated...
Waiting for deployment "mysql-cluster-lfg-site-1-lfg-site-2-replication-svc" rollout to finish: 0 of 1 updated replicas are available...
deployment "mysql-cluster-lfg-site-1-lfg-site-2-replication-svc" successfully rolled out
2023-10-12T21:36:19Z INFO - Waiting for deployment mysql-cluster-lfg-site-1-lfg-site-3-replication-svc to rollout restart...
deployment "mysql-cluster-lfg-site-1-lfg-site-3-replication-svc" successfully rolled out
2023-10-12T21:36:19Z INFO - Waiting for deployment mysql-cluster-lfg-site-1-lfg-site-4-replication-svc to rollout restart...
deployment "mysql-cluster-lfg-site-1-lfg-site-4-replication-svc" successfully rolled out
2023-10-12T21:36:22Z INFO - number of db-backup-manager-svc deployments: 1
2023-10-12T21:36:22Z INFO - Patching deployment 0: mysql-cluster-db-backup-manager-svc...
2023-10-12T21:36:23Z INFO - number of db-monitor-svc deployments: 1
2023-10-12T21:36:23Z INFO - Patching deployment 0: mysql-cluster-db-monitor-svc...
2023-10-12T21:36:23Z INFO - Waiting for statefulset ndbmtd to rollout restart...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
waiting for statefulset rolling update to complete 1 pods at revision ndbmtd-d7bf774f6...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
statefulset rolling update complete 2 pods at revision ndbmtd-d7bf774f6...
2023-10-12T21:37:49Z INFO - Waiting for statefulset ndbappmysqld to rollout restart...
statefulset rolling update complete 2 pods at revision ndbappmysqld-5b8948d47d...
2023-10-12T21:37:49Z INFO - Waiting for statefulset ndbmysqld to rollout restart...
Waiting for 1 pods to be ready...
waiting for statefulset rolling update to complete 4 pods at revision ndbmysqld-94dd7fcbb...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
waiting for statefulset rolling update to complete 5 pods at revision ndbmysqld-94dd7fcbb...
Waiting for 1 pods to be ready...
Waiting for 1 pods to be ready...
statefulset rolling update complete 6 pods at revision ndbmysqld-94dd7fcbb...
2023-10-12T21:38:32Z INFO - Waiting for deployment mysql-cluster-db-backup-manager-svc to rollout restart...
deployment "mysql-cluster-db-backup-manager-svc" successfully rolled out
2023-10-12T21:38:33Z INFO - Waiting for deployment mysql-cluster-db-monitor-svc to rollout restart...
deployment "mysql-cluster-db-monitor-svc" successfully rolled out
2023-10-12T21:38:33Z INFO - Discarding old mysql password for 'occneuser'@'%'...
2023-10-12T21:38:33Z INFO - Old mysql password discarded
2023-10-12T21:38:33Z INFO - Discarding old mysql password for 'occnerepluser'@'%'...
2023-10-12T21:38:34Z INFO - Old mysql password discarded
2023-10-12T21:38:34Z INFO - Password(s) updated successfully

7.8.1.6 Changing All cnDBTier Passwords in Phases

This section provides the procedure to change all cnDBTier Passwords in Phases.

Run the following command to add the new passwords to MySQL (retain the old passwords) and change the passwords in the Kubernetes secret:

$ dbtpasswd --secrets-and-mysql-only

Sample output:

2022-12-24T03:41:08Z INFO - Changing password for user root
Current password:
Enter new password:
Enter new password again:
2022-12-24T03:41:08Z INFO - Changing password for user occneuser
Current password:
Enter new password:
Enter new password again:
2022-12-24T03:41:09Z INFO - Changing password for user occnerepluser
Current password:
Enter new password:
Enter new password again:
2022-12-24T03:41:09Z INFO - Getting sts and sts pod info...
2022-12-24T03:41:09Z INFO - MGM_STS="ndbmgmd"
2022-12-24T03:41:09Z INFO - MGM_REPLICAS="2"
2022-12-24T03:41:09Z INFO -
    ndbmgmd-0
    ndbmgmd-1
2022-12-24T03:41:09Z INFO - NDB_STS="ndbmtd"
2022-12-24T03:41:09Z INFO - NDB_REPLICAS="2"
2022-12-24T03:41:09Z INFO -
    ndbmtd-0
    ndbmtd-1
2022-12-24T03:41:09Z INFO - API_STS="ndbmysqld"
2022-12-24T03:41:09Z INFO - API_REPLICAS="2"
2022-12-24T03:41:09Z INFO -
    ndbmysqld-0
    ndbmysqld-1
2022-12-24T03:41:09Z INFO - APP_STS="ndbappmysqld"
2022-12-24T03:41:09Z INFO - APP_REPLICAS="2"
2022-12-24T03:41:10Z INFO -
    ndbappmysqld-0
    ndbappmysqld-1
2022-12-24T03:41:10Z INFO - Getting deployment pod info...
2022-12-24T03:41:10Z INFO - grepping for backup-man (BAK_CHART_NAME)...
2022-12-24T03:41:10Z INFO -
    mysql-cluster-db-backup-manager-svc-c4648f6bc-jpkt9
2022-12-24T03:41:10Z INFO -
    mysql-cluster-db-backup-manager-svc
2022-12-24T03:41:10Z INFO - grepping for db-mon (MON_CHART_NAME)...
2022-12-24T03:41:10Z INFO -
    mysql-cluster-db-monitor-svc-7d684c7c6f-gvv76
2022-12-24T03:41:10Z INFO -
    mysql-cluster-db-monitor-svc
2022-12-24T03:41:10Z INFO - grepping for replicat (REP_CHART_NAME)...
2022-12-24T03:41:10Z INFO -
    mysql-cluster-lfg-site-1-lfg-site-2-replication-svc-7b689frvfr2
2022-12-24T03:41:10Z INFO -
    mysql-cluster-lfg-site-1-lfg-site-2-replication-svc
2022-12-24T03:41:10Z INFO - Labeling pods with dbtier-app...
pod/ndbmgmd-0 not labeled
pod/ndbmgmd-1 not labeled
pod/ndbmtd-0 not labeled
pod/ndbmtd-1 not labeled
pod/ndbappmysqld-0 not labeled
pod/ndbappmysqld-1 not labeled
pod/ndbmysqld-0 not labeled
pod/ndbmysqld-1 not labeled
pod/mysql-cluster-db-backup-manager-svc-c4648f6bc-jpkt9 not labeled
pod/mysql-cluster-db-monitor-svc-7d684c7c6f-gvv76 not labeled
pod/mysql-cluster-lfg-site-1-lfg-site-2-replication-svc-7b689frvfr2 not labeled
2022-12-24T03:41:11Z INFO - Pods labeled with dbtier-app
2022-12-24T03:41:11Z INFO - Verifying Geo Replication to mates...
2022-12-24T03:41:14Z INFO - Geo Replication to mates is UP
2022-12-24T03:41:14Z INFO - Changing mysql password for 'root'@'localhost'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-24T03:41:15Z INFO - Mysql password changed
2022-12-24T03:41:15Z INFO - Patching secret, occne-mysqlndb-root-secret, with new password
secret/occne-mysqlndb-root-secret patched
2022-12-24T03:41:15Z INFO - Secret, occne-mysqlndb-root-secret, patched with new password
2022-12-24T03:41:15Z INFO - Adding new mysql password for 'occneuser'@'%'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-24T03:41:15Z INFO - New mysql password added
2022-12-24T03:41:16Z INFO - Patching secret, occne-secret-db-monitor-secret, with new password
secret/occne-secret-db-monitor-secret patched
2022-12-24T03:41:16Z INFO - Secret, occne-secret-db-monitor-secret, patched with new password
2022-12-24T03:41:16Z INFO - Adding new mysql password for 'occnerepluser'@'%'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-24T03:41:16Z INFO - New mysql password added
2022-12-24T03:41:16Z INFO - Changing password in replication_info.DBTIER_REPL_SITE_INFO table...
2022-12-24T03:41:16Z INFO - Using replication pod: mysql-cluster-lfg-site-1-lfg-site-2-replication-svc-7b689frvfr2
2022-12-24T03:41:16Z INFO - Using replication pod container: lfg-site-1-lfg-site-2-replication-svc
2022-12-24T03:41:16Z INFO - MYSQL_REPLICATION_SITE_NAME = lfg-site-1
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-24T03:41:17Z INFO - Password changed in replication_info.DBTIER_REPL_SITE_INFO table
2022-12-24T03:41:17Z INFO - Patching secret, occne-replication-secret-db-replication-secret, with new password
secret/occne-replication-secret-db-replication-secret patched
2022-12-24T03:41:17Z INFO - Secret, occne-replication-secret-db-replication-secret, patched with new password
2022-12-24T03:41:17Z INFO - Password(s) updated successfully

Run the following command to restart the appropriate cnDBTier pods:

$ dbtpasswd --restart-only

Sample output:

2022-12-24T03:58:36Z INFO - Changing password for user root
Current password:
2022-12-24T03:58:41Z INFO - Changing password for user occneuser
Current password:
2022-12-24T03:58:46Z INFO - Changing password for user occnerepluser
Current password:
2022-12-24T03:58:49Z INFO - Getting sts and sts pod info...
...
2022-12-24T04:01:39Z INFO - db-monitor-svc pods rollout restarted
2022-12-24T04:01:39Z INFO - Password(s) updated successfully

Once the necessary transition is done, run the following command to discard the old MySQL cnDBTier passwords:

$ dbtpasswd --discard-only

Sample output:

2022-12-24T03:51:40Z INFO - Changing password for user root
Current password:
2022-12-24T03:52:04Z INFO - Changing password for user occneuser
Current password:
2022-12-24T03:52:07Z INFO - Changing password for user occnerepluser
Current password:
2022-12-24T03:52:09Z INFO - Getting sts and sts pod info...
...
2022-12-24T03:52:12Z INFO - Discarding old mysql password for 'occneuser'@'%'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-24T03:52:12Z INFO - Old mysql password discarded
2022-12-24T03:52:12Z INFO - Discarding old mysql password for 'occnerepluser'@'%'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-24T03:52:13Z INFO - Old mysql password discarded
2022-12-24T03:52:13Z INFO - Password(s) updated successfully

7.8.1.7 Changing an NF Password

This section provides a sample procedure to change an NF password when the secret is stored in an NF namespace that is different from the cnDBTier namespace.

Run the following commands to change the password on the secret and add a new password to MySQL.

Note:

When the output prompts for the current password, enter the current password in the NF secret.

$ export DBTIER_NAMESPACE=”dbtier_namespace”
$ dbtpasswd --secrets-and-mysql-only --nf-namespace=name-of-nf-namespace nf-secret-in-nf-namespace

Sample output:

2022-12-15T23:27:19Z INFO - Changing password for user luis
Current password:
Enter new password:
Enter new password again:
...
2022-12-15T23:27:37Z INFO - Adding new mysql password for 'luis'@'%'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-15T23:27:37Z INFO - New mysql password added
2022-12-15T23:27:37Z INFO - Patching secret, nf-secret-in-nf-namespace, with new password
secret/nf-secret-in-nf-namespace patched
2022-12-15T23:27:37Z INFO - Secret, nf-secret-in-nf-namespace, patched with new password
2022-12-15T23:27:37Z INFO - Password(s) updated successfully

Run the following command to discard the old password in MySQL, after restarting NF pods, adding new NF passwords on mate sites, or both.

Note:

When the output prompts for the current password, enter the current password in the NF secret.
The NF secret must be present on nf-namespace and the corresponding MySQL user must be present with the corresponding password.

$ dbtpasswd --discard-only --nf-namespace=name-of-nf-namespace nf-secret-in-nf-namespace

Sample output:

2022-12-15T23:28:48Z INFO - Changing password for user luis
Current password:
...
2022-12-15T23:28:53Z INFO - Discarding old mysql password for 'luis'@'%'...
mysql: [Warning] Using a password on the command line interface can be insecure.
2022-12-15T23:28:54Z INFO - Old mysql password discarded
2022-12-15T23:28:54Z INFO - Password(s) updated successfully

7.8.2 Modifying cnDBTier Backup Encryption Password

This section provides the procedure to modify the cnDBTier backup encryption password.

Get the existing backup encryption password (occne-backup-encryption-secret):
```
$ kubectl get secret  occne-backup-encryption-secret  -n <cndbtier_namespace>  -o jsonpath="{.data.backup_encryption_password}" | base64 --decode
```
For example,
```
$ kubectl get secret  occne-backup-encryption-secret  -n occne-cndbtier -o jsonpath="{.data.backup_encryption_password}" | base64 --decode
```
Note:
Skip this step if you already know the existing occne-backup-encryption-secret password.

Delete the existing backup encryption secret:

$ kubectl -n <cndbtier_namespace> delete secret occne-backup-encryption-secret

For example,

$ kubectl -n occne-cndbtier delete secret occne-backup-encryption-secret

Run the following command to create secret with a new password. For information about the password policies, see the "Creating Secrets" section in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
```
$ kubectl -n <cndbtier_namespace>  create secret generic occne-backup-encryption-secret --from-literal="backup_encryption_password=<new_password>"
```
For example,
```
$ kubectl -n  occne-cndbtier create secret generic occne-backup-encryption-secret --from-literal="backup_encryption_password=NextGenCne"
```

Scale down the replication service and backup manager service deployments. After scaling down, wait until all the replication service and backup manager pods are DOWN:

$ kubectl -n <cndbtier_namespace> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <cndbtier_namespace> scale deployment --replicas=0
$ kubectl -n <cndbtier_namespace> get deployments | egrep 'db-backup-manager-svc' | awk '{print $1}' | xargs -L1 -r kubectl -n <cndbtier_namespace> scale deployment --replicas=0

For example,

$ kubectl -n occne-cndbtier get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n occne-cndbtier scale deployment --replicas=0
$ kubectl -n occne-cndbtier get deployments | egrep 'db-backup-manager-svc' | awk '{print $1}' | xargs -L1 -r kubectl -n occne-cndbtier scale deployment --replicas=0

Scale up the replication service and backup manager service deployments. After scaling up, wait until all the replication service and backup manager pods are UP:

$ kubectl -n <cndbtier_namespace> get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n <cndbtier_namespace> scale deployment --replicas=1
$ kubectl -n <cndbtier_namespace> get deployments | egrep 'db-backup-manager-svc' | awk '{print $1}' | xargs -L1 -r kubectl -n <cndbtier_namespace> scale deployment --replicas=1

For example,

$ kubectl -n occne-cndbtier get deployments | egrep 'repl' | awk '{print $1}' | xargs -L1 -r kubectl -n occne-cndbtier scale deployment --replicas=1
$ kubectl -n occne-cndbtier get deployments | egrep 'db-backup-manager-svc' | awk '{print $1}' | xargs -L1 -r kubectl -n occne-cndbtier scale deployment --replicas=1

7.8.3 Modifying SSH Keys for Transferring Backups

This section provides the procedure to modify Secure Shell (SSH) keys for securely transferring cnDBTier backups.

Perform the following steps to move the existing SSH keys to the backup directory and delete the existing SSH secrets:

Perform the following steps to move the existing SSH keys to the backup directory:

Identify the location of the existing SSH keys:
```
$ ls /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/
```
For example:

$ ls /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/

Sample output:

cndbtier_id_rsa cndbtier_id_rsa.pub

Move the existing SSH keys from the location identified in the previous step to the backup directory:

$ sshBackupDateTime=$(date +"%m-%d-%y-%H-%M-%S")
$ mkdir -p /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/${sshBackupDateTime}
$ mv <SSH private key file name> /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/${sshBackupDateTime}/
$ mv <SSH public key file name> /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/${sshBackupDateTime}/

For example:

$ sshBackupDateTime=$(date +"%m-%d-%y-%H-%M-%S")
$ mkdir -p /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/${sshBackupDateTime}
$ mv /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/cndbtier_id_rsa /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/${sshBackupDateTime}/
$ mv /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/cndbtier_id_rsa.pub /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/${sshBackupDateTime}/

Perform the following steps to delete the existing SSH or Secure File Transfer Protocol (SFTP) secrets from the current cnDBTier cluster:

Identify the SSH or SFTP secrets from the current cluster by running the following command:

$ kubectl get secrets --namespace=<namespace of cnDBTier Cluster> | grep ssh

For example:

$ kubectl get secrets --namespace=cluster1 | grep ssh

Sample output:

cndbtier-ssh-private-key                         Opaque               1      7d
cndbtier-ssh-public-key                          Opaque               1      7d

Delete the SSH or SFTP secrets from the current cluster by running the following commands:

$ kubectl delete secret cndbtier-ssh-private-key --namespace=<namespace of cnDBTier Cluster>
$ kubectl delete secret cndbtier-ssh-public-key --namespace=<namespace of cnDBTier Cluster>

Example to delete a private key:

$ kubectl delete secret cndbtier-ssh-private-key --namespace=cluster1

Sample output:

secret "cndbtier-ssh-private-key" deleted

Example to delete a public key:

$ kubectl delete secret cndbtier-ssh-public-key --namespace=cluster1

Sample output:

secret "cndbtier-ssh-public-key" deleted

Create the SSH keys by running the following commands:
```
$ mkdir -p -m 0700 /var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh
$ ssh-keygen -b 4096 -t rsa -C "cndbtier key" -f "/var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/cndbtier_id_rsa" -q -N ""
```
For more information about creating SSH secrets, see "Creating SSH Keys" in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Create the SFTP secrets by running the following commands:

$ kubectl create secret generic cndbtier-ssh-private-key --from-file=id_rsa=/var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/cndbtier_id_rsa -n ${OCCNE_NAMESPACE}
$ kubectl create secret generic cndbtier-ssh-public-key --from-file=id_rsa.pub=/var/occne/cluster/${OCCNE_CLUSTER}/cndbtierssh/cndbtier_id_rsa.pub -n ${OCCNE_NAMESPACE}

For more information about creating SFTP secrets, see step 2 of "Creating Secrets" in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Perform the following steps to restart the data nodes sequentially in an ascending order (ndbmtd-0, ndbmtd-1, and so on):

Identify the list of data pods in the cnDBTier cluster:

$ kubectl get pods --namespace=<namespace of cnDBTier Cluster> | grep 'ndbmtd'

For example, use the following command to identify the list of data pods in cnDBTier cluster1:

$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'

Sample output:

ndbmtd-0                                               3/3     Running   0          14m
ndbmtd-1                                               3/3     Running   0          13m

Delete the first data pod of the cnDBTier cluster:
```
$ kubectl delete pod ndbmtd-0 --namespace=<namespace of cnDBTier Cluster>
```
For example, use the following command to delete the first data pod of cnDBTier cluster1:
```
$ kubectl delete pod ndbmtd-0 --namespace=cluster1
```
Sample output:
```
pod "ndbmtd-0" deleted
```

Wait for the first data pod to come up and verify if the pod is up by running the following command:

$ kubectl get pods --namespace=<namespace of cnDBTier Cluster> | grep 'ndbmtd'

For example:

$ kubectl get pods --namespace=cluster1 | grep 'ndbmtd'

Sample output:

ndbmtd-0                                               3/3     Running   0          65s
ndbmtd-1                                               3/3     Running   0          13m

Check the status of the cnDBTier cluster:

$ kubectl -n <namespace of cnDBTier Cluster> exec -it ndbmgmd-0 -- ndb_mgm -e show

For example, use the following command to check the status of cnDBTier cluster1:

$ kubectl -n cluster1 exec -it ndbmgmd-0 -- ndb_mgm -e show

Sample output:

Connected to Management Server at: localhost:1186 Cluster Configuration
---------------------
[ndbd(NDB)]     2 node(s)
id=1    @10.233.85.92  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0)
id=2    @10.233.114.33  (mysql-8.0.35 ndb-8.0.35, Nodegroup: 0, *)
   
[ndb_mgmd(MGM)] 2 node(s)
id=49   @10.233.65.167  (mysql-8.0.35 ndb-8.0.35)
id=50   @10.233.127.115  (mysql-8.0.35 ndb-8.0.35)
   
[mysqld(API)]   8 node(s)
id=56   @10.233.120.210  (mysql-8.0.35 ndb-8.0.35)  
id=57   @10.233.124.93  (mysql-8.0.35 ndb-8.0.35)
id=70   @10.233.127.117  (mysql-8.0.35 ndb-8.0.35)
id=71   @10.233.85.93  (mysql-8.0.35 ndb-8.0.35)
id=222 (not connected, accepting connect from any host)
id=223 (not connected, accepting connect from any host)
id=224 (not connected, accepting connect from any host)
id=225 (not connected, accepting connect from any host)

Note:

Node IDs 222 to 225 in the sample output are shown as "not connected" as these are added as empty slot IDs that are used for georeplication recovery.

Repeat steps a through d to delete the remaining data pods (ndbmtd-1, ndbmtd-2, and so on).

Perform the following steps to restart all the replication services of the current cluster:

Identify the replication service running in the current cluster:

$ kubectl get pods --namespace=<namespace of cnDBTier Cluster>

For example:

$ kubectl get pods --namespace=cluster1

Sample output:

NAME                                                              READY   STATUS    RESTARTS        AGE
mysql-cluster-cluster1-cluster2-replication-svc-5d4b8fd685tshzd   1/1     Running   1 (13h ago)     13h

Delete all the replication service pods of the current cluster:

$ kubectl delete pod <replication service pod name> --namespace=<namespace of cnDBTier Cluster>

For example,

$ kubectl delete pod mysql-cluster-cluster1-cluster2-replication-svc-5d4b8fd685tshzd --namespace=cluster1

Sample output:

pod "mysql-cluster-cluster1-cluster2-replication-svc-5d4b8fd685tshzd" delete

Follow steps 1 through 5 to replace the SSH or SFTP keys and secrets on the other georeplication cnDBTier clusters.

Note:
The SSH or SFTP secrets must be the same across all the georeplication cnDBTier clusters.

7.9 Modifying Remote Server Configurations for Secure Transfer of Backups

cnDBTier requires the following remote server configurations to securely transfer backups:

IP address or Fully Qualified Domain Name (FQDN)
Secure File Transfer Protocol (SFTP) port
The path where the backups are to be securely transferred
SSH key and username that are configured in the form of secrets

This section provides the procedure to modify remote server configurations in cnDBTier.

Perform the following steps to update the remote server configurations such as remoteserverip, remoteserverport, and remoteserverpath:

Locate the custom_values.yaml file.

Edit the configurations as per your requirement:

For example,

global:
....
....
  remotetransfer:
    enable: true
    faultrecoverybackuptransfer: true
    remoteserverip: "< IP address of the remote server>"
    remoteserverport: "<SFTP Port of the remote server>"
    remoteserverpath: "<path where cndbtier backups will be stored>"
 
For example:
 
global:
....
....   
  remotetransfer:
     enable: true
     faultrecoverybackuptransfer: true
     remoteserverip: "10.75.216.8"
     remoteserverport: "2022"
     remoteserverpath: "/var/occnedb"

Perform the following steps to change the username of the remote server:
1. Delete the occne-remote-server-username-secret secret by running the following command:
```
$ kubectl -n <namespace> delete secret occne-remote-server-username-secret
```
  Where, <namespace> is the namespace of cnDBTier.
  For example,
```
$  kubectl  -n occne-cndbtier  delete secret occne-remote-server-username-secret
```
2. Recreate the secret with a new username by running the following command:
```
$  kubectl -n <namespace> create secret generic occne-remote-server-username-secret --from-literal="remote_server_user_name=<user_name>"
```
  Where,
  - <namespace> is the namespace of cnDBTier.
  - <user_name> is the new username of the remote server.
  For example,
```
$  kubectl  -n occne-cndbtier  create secret generic occne-remote-server-username-secret --from-literal="remote_server_user_name=cndbtierbackupserver"
```
  As per the example, the username of the remote server is cndbtierbackupserver.
Perform the following steps to update the SSH key of the remote server:
1. Delete the occne-remoteserver-privatekey-secret secret by running the following command:
```
$ kubectl -n <namespace> delete secret occne-remoteserver-privatekey-secret
```
  Where, <namespace> is the namespace of cnDBTier.
  For example,
```
$ kubectl -n occne-cndbtier delete secret occne-remoteserver-privatekey-secret
```
2. Copy the private SSH key of the remote server and save it in a file.
3. Create the secret by using the file created in the previous step:
```
$ kubectl -n <namespace> create secret generic occne-remoteserver-privatekey-secret --from-file=id_rsa=<private key path>
```
  Where,
  - <namespace> is the namespace of cnDBTier
  - <private key path> is the path to the private key file created in step b. The file path must include the file name.
  For example,
```
$ kubectl -n occne-cndbtier create secret generic occne-remoteserver-privatekey-secret --from-file=id_rsa=/var/occne/cluster/dbtier/remoteserver_id_rsa
```
  As per the example, the private key of the remote server is saved in the /var/occne/cluster/dbtier/ directory with the file name remoteserver_id_rsa.
Upgrade the cnDBTier cluster to update all the configurations changed in steps 1, 2, and 3:
```
$  helm  upgrade mysql-cluster occndbtier -f occndbtier/custom_values.yaml -n <namespace>
```
Where, <namespace> is the namespace of cnDBTier.
If the replication service pods are not restarted after helm upgrade in the previous step, run the following command to manually delete the replication service pods:
```
$ kubectl -n <cndbtier_namespace> delete pod <replication_pod_name>
```

7.10 Checking the Georeplication Status Between Clusters

This section describes the following procedures to check the georeplication status of a cnDBTier cluster in a site with a remote site:

Note:

Replace the name of the cnDBTier cluster namespaces(cluster1, cluster2, cluster3, and cluster4) with the actual name of your namespaces in each of the commands in this section.

Checking the Georeplication Status of cnDBTier cluster1 with Remote Sites

Perform the following steps to check the georeplication status of cnDBTier cluster1 with remote sites (cnDBTier cluster2, cnDBTier cluster3, and cnDBTier cluster4):

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then check the georeplication status in cnDBTier cluster1 with remote sites (cnDBTier cluster2, cnDBTier cluster3, and cnDBTier cluster4) for every replication group.

Run the following commands to check the georeplication status of cnDBTier cluster1 with respect to cnDBTier cluster2:

$ kubectl -n cluster1 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (2000, 2001) AND remote_site_name = "cluster2";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster2         |             2000 |       8970 | 10.233.0.147        | ACTIVE  |        NULL | cluster1  |      1000 | 2022-04-04 19:10:44 |
| cluster2         |             2001 |       4597 | 10.233.9.35         | STANDBY |        NULL | cluster1  |      1001 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBTier cluster1 with respect to cnDBTier cluster2.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE replication channel:

$ kubectl -n cluster1 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.0.147
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000007
          Read_Source_Log_Pos: 16442
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBTier cluster1 with respect to cnDBTier cluster2.

Note:

When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.

The following example describes the commands to check the replication status of the STANDBY replication channel:

$ kubectl -n cluster1 exec -it ndbmysqld-1 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.9.35
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 26970
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No

Run the following commands to check the georeplication status of cnDBTier cluster1 with respect to cnDBTier cluster3:

$ kubectl -n cluster1 exec -it ndbmysqld-2 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (3000, 3001) AND remote_site_name = "cluster3";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster3         |             3000 |       5533 | 10.233.16.34        | ACTIVE  |        NULL | cluster1  |      1002 | 2022-04-04 19:10:44 |
| cluster3         |             3001 |       7918 | 10.233.41.15        | STANDBY |        NULL | cluster1  |      1003 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBTier cluster1 with respect to cnDBTier cluster3.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE replication channel:

$ kubectl -n cluster1 exec -it ndbmysqld-2 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.16.34
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000007
          Read_Source_Log_Pos: 18542
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBTier cluster1 with respect to cnDBTier cluster3.

Note:

When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.

The following example describes the commands to check the replication status of the STANDBY replication channel:

$ kubectl -n cluster1 exec -it ndbmysqld-3 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
               Replica_IO_State:
                  Source_Host: 10.233.41.15
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000006
          Read_Source_Log_Pos: 26421
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No

Run the following commands to check the georeplication status of cnDBTier cluster1 with respect to cnDBTier cluster4:

$ kubectl -n cluster1 exec -it ndbmysqld-4 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (4000, 4001) AND remote_site_name = "cluster4";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster4         |             4000 |       5909 | 10.233.57.61        | ACTIVE  |        NULL | cluster1  |      1004 | 2022-04-04 19:10:44 |
| cluster4         |             4001 |       1937 | 10.233.54.145       | STANDBY |        NULL | cluster1  |      1005 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBTier cluster1 with respect to cnDBTier cluster4.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes

The following example describes the commands to check the replication status of the ACTIVE replication channel::

$ kubectl -n cluster1 exec -it ndbmysqld-4 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
               Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.57.61
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000004
          Read_Source_Log_Pos: 185
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBTier cluster1 with respect to cnDBTier cluster4.

Note:
When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.
The following example describes the commands to check the replication status of the STANDBY replication channel:
```
$ kubectl -n cluster1 exec -it ndbmysqld-5 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
```
Sample output:
```
*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.54.145
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 26420
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No
```

Checking the Georeplication Status of cnDBTier cluster2 with Remote Sites

Perform the following steps to check the georeplication status of cnDBTier cluster2 with remote sites (cnDBTier cluster1, cnDBTier cluster3, and cnDBTier cluster4):

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then check the georeplication status in cnDBTier cluster2 with remote sites (cnDBTier cluster1, cnDBTier cluster3, and cnDBTier cluster4) for every replication group.

Run the following commands to check the georeplication status of cnDBtier cluster2 with respect to cnDBTier cluster1:

$ kubectl -n cluster2 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (1000, 1001) AND remote_site_name = "cluster1";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster1         |             1000 |       8970 | 10.233.13.41        | ACTIVE  |        NULL | cluster2  |      2000 | 2022-04-04 19:10:44 |
| cluster1         |             1001 |       4597 | 10.233.39.110       | STANDBY |        NULL | cluster2  |      2001 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBtier cluster2 with respect to cnDBTier cluster1.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE replication channel:

$ kubectl -n cluster2 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.13.41
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000009
          Read_Source_Log_Pos: 16480
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBTier cluster2 with respect to cnDBTier cluster1.

Note:

When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.

The following example describes the commands to check the replication status of the STANDBY replication channel:

$ kubectl -n cluster2 exec -it ndbmysqld-1 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:  
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.39.110
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000007
          Read_Source_Log_Pos: 13270
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No

Run the following commands to check the georeplication status of cnDBTier cluster2 with respect to cnDBTier cluster3:

$ kubectl -n cluster2 exec -it ndbmysqld-2 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (3002, 3003) AND remote_site_name = "cluster3";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster3         |             3002 |       9981 | 10.233.62.226       | ACTIVE  |        NULL | cluster2  |      2002 | 2022-04-04 19:10:44 |
| cluster3         |             3003 |       9319 | 10.233.14.15        | STANDBY |        NULL | cluster2  |      2003 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBTier cluster2 with respect to cnDBTier cluster3.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE Replication channel:

$ kubectl -n cluster2 exec -it ndbmysqld-2 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.62.226
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000008
          Read_Source_Log_Pos: 16485
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBTier cluster2 with respect to cnDBTier cluster3.

Note:
When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.
The following example describes the commands to check the replication status of the STANDBY Replication channel:
```
$ kubectl -n cluster2 exec -it ndbmysqld-3 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
```
Sample output:
```
*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.14.15
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000008
          Read_Source_Log_Pos: 13670
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No
```

Run the following commands to check the georeplication status of cnDBTier cluster2 with respect to cnDBTier cluster4:

$ kubectl -n cluster2 exec -it ndbmysqld-4 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (4002, 4003) AND remote_site_name = "cluster4";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster4         |             4002 |       4500 | 10.233.42.33        | ACTIVE  |        NULL | cluster2  |      2004 | 2022-04-04 19:10:44 |
| cluster4         |             4003 |       9921 | 10.233.48.56        | STANDBY |        NULL | cluster2  |      2005 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBTier cluster2 with respect to cnDBTier cluster4.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE Replication channel:

$ kubectl -n cluster2 exec -it ndbmysqld-4 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.42.33
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 34765
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBTier cluster2 with respect to cnDBTier cluster4.

Note:
When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.
The following example describes the commands to check the replication status of the STANDBY Replication channel:
```
$ kubectl -n cluster2 exec -it ndbmysqld-5 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
```
Sample output:
```
*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.48.56
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 24672
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No
```

Checking the Georeplication Status of cnDBTier cluster3 with Remote Sites

Perform the following steps to check the georeplication status of cnDBTier cluster3 with remote sites (cnDBTier cluster1, cnDBTier cluster2, and cnDBTier cluster4):

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then check the georeplication status in cnDBTier cluster3 with remote sites (cnDBTier cluster1, cnDBTier cluster2, and cnDBTier cluster4) for every replication group.

Run the following commands to check the georeplication status of cnDBtier cluster3 with respect to cnDBTier cluster1:

$ kubectl -n cluster3 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (1002, 1003) AND remote_site_name = "cluster1";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster1         |             1002 |       5533 | 10.233.15.214       | ACTIVE  |        NULL | cluster3  |      3000 | 2022-04-04 19:10:44 |
| cluster1         |             1003 |       7918 | 10.233.24.249       | STANDBY |        NULL | cluster3  |      3001 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBtier cluster3 with respect to cnDBTier cluster1.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE Replication channel:

$ kubectl -n cluster3 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.15.214
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 34765
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBtier cluster3 with respect to cnDBTier cluster1.

Note:
When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.
The following example describes the commands to check the replication status of the STANDBY Replication channel:
```
$ kubectl -n cluster3 exec -it ndbmysqld-1 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
```
Sample output:
```
*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.24.249
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 24672
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No
```

Run the following commands to check the georeplication status of cnDBTier cluster3 with respect to cnDBTier cluster2:

$ kubectl -n cluster3 exec -it ndbmysqld-2 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (2002, 2003) AND remote_site_name = "cluster2";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster2         |             2002 |       9981 | 10.233.34.252       | ACTIVE  |        NULL | cluster3  |      3002 | 2022-04-04 19:10:44 |
| cluster2         |             2003 |       9319 | 10.233.15.228       | STANDBY |        NULL | cluster3  |      3003 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBTier cluster3 with respect to cnDBTier cluster2.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE Replication channel:

$ kubectl -n cluster3 exec -it ndbmysqld-2 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.34.252
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 34765
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBTier cluster3 with respect to cnDBTier cluster2.

Note:
When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.
The following example describes the commands to check the replication status of the STANDBY Replication channel:
```
$ kubectl -n cluster3 exec -it ndbmysqld-3 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
```
Sample output:
```
*************************** 1. row ***************************
               Replica_IO_State:
                  Source_Host: 10.233.15.228
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 24672
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No
```

Run the following commands to check the georeplication status of cnDBTier cluster3 with respect to cnDBTier cluster4:

$ kubectl -n cluster3 exec -it ndbmysqld-4 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (4004, 4005) AND remote_site_name = "cluster4";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster4         |             4004 |       3625 | 10.233.63.121       | ACTIVE  |        NULL | cluster3  |      3004 | 2022-04-04 19:10:44 |
| cluster4         |             4005 |       3837 | 10.233.40.173       | STANDBY |        NULL | cluster3  |      3005 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBTier cluster3 with respect to cnDBTier cluster4.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE Replication channel:

$ kubectl -n cluster3 exec -it ndbmysqld-4 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.63.121
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 34765
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBTier cluster3 with respect to cnDBTier cluster4.

Note:
When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.
The following example describes the commands to check the replication status of the STANDBY Replication channel:
```
$ kubectl -n cluster3 exec -it ndbmysqld-5 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
```
Sample output:
```
*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.40.173
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 24672
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No
```

Checking the Georeplication Status of cnDBTier cluster4 with Remote Sites

Perform the following steps to check the georeplication status of cnDBTier cluster4 with remote sites (cnDBTier cluster1, cnDBTier cluster2, and cnDBTier cluster3):

Note:

The following commands and examples are applicable for a single replication channel group only. If multiple replication channel groups are enabled, then check the georeplication status in cnDBTier cluster4 with remote sites (cnDBTier cluster1, cnDBTier cluster2, and cnDBTier cluster3) for every replication group.

Run the following commands to check the georeplication status of cnDBtier cluster4 with respect to cnDBTier cluster1:

$ kubectl -n cluster4 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (1004, 1005) AND remote_site_name = "cluster1";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster1         |             1004 |       5909 | 10.233.37.108       | ACTIVE  |        NULL | cluster4  |      4000 | 2022-04-04 19:10:44 |
| cluster1         |             1005 |       1937 | 10.233.11.31        | STANDBY |        NULL | cluster4  |      4001 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBtier cluster4 with respect to cnDBTier cluster1.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE Replication channel:

$ kubectl -n cluster4 exec -it ndbmysqld-0 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.37.108
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 34765
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBtier cluster4 with respect to cnDBTier cluster1.

Note:
When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.
The following example describes the commands to check the replication status of the STANDBY Replication channel:
```
$ kubectl -n cluster4 exec -it ndbmysqld-1 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
```
Sample output:
```
*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.11.31
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 24672
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No
```

Run the following commands to check the georeplication status of cnDBTier cluster4 with respect to cnDBTier cluster2:

$ kubectl -n cluster4 exec -it ndbmysqld-2 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (2004, 2005) AND remote_site_name = "cluster2";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster2         |             2004 |       4500 | 10.233.15.245       | ACTIVE  |        NULL | cluster4  |      4002 | 2022-04-04 19:10:44 |
| cluster2         |             2005 |       9921 | 10.233.48.74        | STANDBY |        NULL | cluster4  |      4003 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBTier cluster4 with respect to cnDBTier cluster2.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE Replication channel:

$ kubectl -n cluster4 exec -it ndbmysqld-2 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.15.245
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 34765
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBTier cluster4 with respect to cnDBTier cluster2.

Note:
When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.
The following example describes the commands to check the replication status of the STANDBY Replication channel:
```
$ kubectl -n cluster4 exec -it ndbmysqld-3 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
```
Sample output:
```
*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.48.74
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 24672
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No
```

Run the following commands to check the georeplication status of cnDBTier cluster4 with respect to cnDBTier cluster3:

$ kubectl -n cluster4 exec -it ndbmysqld-4 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO where remote_server_id in (3004, 3005) AND remote_site_name = "cluster3";

Sample output:

+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| remote_site_name | remote_server_id | channel_id | remote_signaling_ip | role    | start_epoch | site_name | server_id | start_ts            |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
| cluster3         |             3004 |       3625 | 10.233.3.32         | ACTIVE  |        NULL | cluster4  |      4004 | 2022-04-04 19:10:44 |
| cluster3         |             3005 |       3837 | 10.233.14.89        | STANDBY |        NULL | cluster4  |      4005 | 2022-04-04 19:10:45 |
+------------------+------------------+------------+---------------------+---------+-------------+-----------+-----------+---------------------+
2 rows in set (0.00 sec)

Run the following commands to check if replication is turned on in the ACTIVE Replication channel of cnDBTier cluster4 with respect to cnDBTier cluster3.

Note:

When replication is turned on, the Replica_IO_Running and Replica_SQL_Running parameters are set to Yes.

The following example describes the commands to check the replication status of the ACTIVE Replication channel:

$ kubectl -n cluster4 exec -it ndbmysqld-4 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;

Sample output:

*************************** 1. row ***************************
             Replica_IO_State: Waiting for master to send event
                  Source_Host: 10.233.3.32
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 34765
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 11203
        Relay_Source_Log_File: mysql-bin.000007
           Replica_IO_Running: Yes
          Replica_SQL_Running: Yes
                    ....
                    ....
                    ....
    Replica_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Source_Retry_Count: 86400
                    ....
                    ....

Run the following commands to check if replication is turned off in the STANDBY Replication channel of cnDBTier cluster4 with respect to cnDBTier cluster3.

Note:
When replication is turned off, the Replica_IO_Running and Replica_SQL_Running parameters are set to No.
The following example describes the commands to check the replication status of the STANDBY Replication channel :
```
$ kubectl -n cluster4 exec -it ndbmysqld-5 -- bash
$ mysql -h 127.0.0.1 -uroot -p
Password:
mysql> SHOW REPLICA STATUS\G;
```
Sample output:
```
*************************** 1. row ***************************
             Replica_IO_State:
                  Source_Host: 10.233.14.89
                  Source_User: occnerepluser
                  Source_Port: 3306
                Connect_Retry: 60
              Source_Log_File: mysql-bin.000005
          Read_Source_Log_Pos: 24672
               Relay_Log_File: mysql-relay-bin.000002
                Relay_Log_Pos: 2228
        Relay_Source_Log_File: mysql-bin.000005
           Replica_IO_Running: No
          Replica_SQL_Running: No
```

7.11 Changing Authentication Plugin on cnDBTier Sites

Users created on cnDBTier setups older than 23.4.x (that is, 23.2.x, and 23.3.0) use the mysql_native_password plugin for authentication. As this plugin is deprecated in mysql version 8.0.34 (cnDBTier 23.3.0), you must use the caching_sha2_password plugin for user authentication. This section provides the steps to change the authentication plugin for users created on cnDBTier setups older than 23.3.1.

Note:

Perform this procedure on the ndbappmysqld pod of one site only.
If you have upgraded to 23.4.0 from 23.3.1, you must perform this procedure to change the authentication plugin after all the sites are upgraded.

Perform the following steps to alter a user to use the caching_sha2_password authentication plugin:

Log in to the ndbappmysqld pod:
```
$ kubectl -n <namespace> exec -it ndbappmysqld-0 -- mysql -h::1 -uroot -p<password>
```
where,
- <namespace> is the namespace name
- <password> is the password to access the ndbappmysqld pod
For example:
```
$ kubectl -n occne-cndbtier exec -it ndbappmysqld-0 -- mysql -h::1 -uroot -p<password>
```

Run the following MySQL command to change the authentication plugin:

mysql> ALTER USER IF EXISTS <USER_NAME> IDENTIFIED WITH 'caching_sha2_password' BY '<password>';

where,

<USER NAME> is the name of the user.
<password> is the password of user.

For example:

mysql> ALTER USER IF EXISTS occneuser IDENTIFIED WITH 'caching_sha2_password' BY 'Password';
mysql> ALTER USER IF EXISTS occnerepluser IDENTIFIED WITH 'caching_sha2_password' BY 'Password';
mysql> ALTER USER IF EXISTS root@localhost IDENTIFIED WITH 'caching_sha2_password' BY 'Password';
mysql> ALTER USER IF EXISTS healthchecker@localhost IDENTIFIED WITH 'caching_sha2_password' BY 'Password';
mysql> ALTER USER IF EXISTS root IDENTIFIED WITH 'caching_sha2_password' BY 'Password';
mysql> ALTER USER IF EXISTS <NF_USER> IDENTIFIED WITH 'caching_sha2_password' BY 'Password';

If you want to roll back to cnDBTier 23.2.x, 23.3.x, or 23.4.x after performing the above procedure, then you must change the authentication plugin to mysql_native_password before performing a rollback:

Log in to the ndbappmysqld pod:

$ kubectl -n <namespace> exec -it ndbappmysqld-0 -- mysql -h::1 -uroot -p<password>

where,

<namespace> is the namespace name.
<password> is the password to access the ndbappmysqld pod.

For example:

$ kubectl -n occne-cndbtier exec -it ndbappmysqld-0 -- mysql -h::1 -uroot -p<password>

Run the following mysql command to change the authentication plugin to mysql_native_password:

mysql> ALTER USER IF EXISTS <USER_NAME> IDENTIFIED WITH 'mysql_native_password' BY '<password>';

where,

<USER NAME> is the name of the user.
<password> is the password of the user.

For example:

mysql> ALTER USER IF EXISTS occneuser IDENTIFIED WITH 'mysql_native_password' BY 'Password';
mysql> ALTER USER IF EXISTS occnerepluser IDENTIFIED WITH 'mysql_native_password' BY 'Password';
mysql> ALTER USER IF EXISTS root@localhost IDENTIFIED WITH 'mysql_native_password' BY 'Password';
mysql> ALTER USER IF EXISTS healthchecker@localhost IDENTIFIED WITH 'mysql_native_password' BY 'Password';
mysql> ALTER USER IF EXISTS <NF_USER> IDENTIFIED WITH 'caching_sha2_password' BY 'Password';

7 Maintenance Procedures

7.1 Starting or Stopping cnDBTier

7.1.1 Starting cnDBTier

7.1.2 Stopping cnDBTier

7.2 Starting or Stopping cnDBTier Georeplication Service

7.2.1 Starting cnDBTier Georeplication Between Sites

7.2.2 Stopping cnDBTier Georeplication Between Sites

7.3 cnDBTier Scaling

7.3.1 Horizontal Scaling

7.3.1.1 Scaling ndbappmysqld Pods

7.3.1.1.1 Scaling ndbappmysqld Pods Without Service Account

7.3.1.1.2 Scaling ndbappmysqld Pods With Service Account

7.3.1.2 Scaling ndbmtd Pods

7.3.2 Vertical Scaling

7.3.2.1 Scaling ndbappmysqld Pods

7.3.2.2 Scaling ndbmysqld Pods

7.3.2.3 Scaling ndbmtd Pods

7.3.2.4 Vertical Scaling of db-replication-svc

7.4 Configuring cnDBTier SQL Nodes to Support TLS for Georeplication

7.4.1 Enabling TLS for Georeplication

7.4.2 Modifying cnDBTier Certificates to Establish TLS Between Georeplication Sites

7.5 Adding Georedundant cnDBTier Site

7.5.1 Adding cnDBTier Georedundant Cluster to Single Site cnDBTier Cluster

7.5.2 Adding cnDBTier Georedundant Cluster to Two Site cnDBTier Clusters

7.5.3 Adding cnDBTier Georedundant Cluster to Three Site cnDBTier Clusters

7.6 Removing a Georedundant cnDBTier Cluster

7.6.1 Removing cnDBTier Cluster 4

7.6.2 Removing cnDBTier Cluster 3

7.6.3 Removing cnDBTier Cluster 2

7.6.4 Removing cnDBTier Cluster 1

7.7 Adding and Removing Replication Channel Group

7.7.1 Converting Single Replication Channel Group to Multiple Replication Channel Groups

7.7.2 Converting Multiple Replication Channel Groups to Single Replication Channel Group

7.8 Managing Passwords and Secrets

7.8.1 Changing cnDBTier Passwords

7.8.1.1 Prerequisites

7.8.1.2 Installing and Using dbtpasswd Script

7.8.1.3 Configuration Options

7.8.1.4 Changing All cnDBTier Passwords in a Site

7.8.1.5 Changing All Passwords in a Stand-Alone or Georeplication Site

7.8.1.6 Changing All cnDBTier Passwords in Phases

7.8.1.7 Changing an NF Password

7.8.2 Modifying cnDBTier Backup Encryption Password

7.8.3 Modifying SSH Keys for Transferring Backups

7.9 Modifying Remote Server Configurations for Secure Transfer of Backups

7.10 Checking the Georeplication Status Between Clusters

7.11 Changing Authentication Plugin on cnDBTier Sites