7.4 Performing OCNADD Backup Procedures

7.4.1 Performing OCNADD Manual Backup

Perform the following steps to take the manual backup:

1. Go to custom-templates folder inside the extracted ocnadd-release package and update the ocnadd_manualBackup.yaml file with the following information:
   1. Values for BACKUP_DATABASES can be set to ALL (i.e. healthdb_schema, configuration_schema, and alarm_schema) or the individual DB names can also be passed. By default, the value is 'ALL'.
   2. Values of BACKUP_ARG can be set to ALL, DB, or KAFKA. By default, the value is as ALL.
   3. Update other values as follows:

      apiVersion: batch/v1
      kind: Job
      metadata:
        name: ocnaddmanualbackup
        namespace: ocnadd-deploy     #---> update the namespace
      spec:
        template:       
          metadata:
            name: ocnaddmanualbackup
          spec:
            volumes:
            - name: backup-vol
              persistentVolumeClaim:
                  claimName: backup-mysql-pvc
            - name: config-vol    
              configMap:
                name: config-backuprestore-scripts
            serviceAccountName: ocnadd-deploy-gitlab-admin      #---> update the service account name. Format:<namespace>-gitlab-admin
            containers:
            - name: ocnaddmanualbackup
              image: <repo-path>/ocnaddbackuprestore:1.0.7      #---> update repository path
              volumeMounts:
                - mountPath: "work-dir"
                  name: backup-vol
                - mountPath: "config-backuprestore-scripts"
                  name: config-vol
              env:
                - name: HOME
                  value: /tmp
                - name: DB_USER
                  valueFrom:
                    secretKeyRef:
                      name: db-secret
                      key: MYSQL_USER
                - name: DB_PASSWORD       
                  valueFrom:
                    secretKeyRef:
                      name: db-secret
                      key: MYSQL_PASSWORD       
                - name: BACKUP_DATABASES
                  value: ALL
                - name: BACKUP_ARG
                  value: ALL 
              command:
              - /bin/sh
              - -c
              - |
                cp /config-backuprestore-scripts/*.sh /tmp
                chmod +x /tmp/*.sh   
                mkdir /work-dir/backup
                echo "Executing manual backup script"
                bash /tmp/backup.sh $BACKUP_DATABASES $BACKUP_ARG
                ls -lh /work-dir/backup
            restartPolicy: Never
      status: {}

2. Execute the below command to run the job:

   kubectl create -f ocnadd_manualBackup.yaml

7.4.2 Verifying OCNADD Backup

Caution:

The connectivity between the external storage via either PV/PVC or network connectivity must be ensured.

To verify the backup, perform the following steps:

1. Go to the custom-templates folder inside the extracted ocnadd-release package and update the ocnadd_verify_backup.yaml file with the following information:
   1. Sleep time is configurable, update it if required. (default sleep is 10m)

   2. Update other values as follows:

      apiVersion: batch/v1
      kind: Job
      metadata:
        name: ocnaddverify
        namespace: ocnadd-deploy          #---> update the namespace
      spec:
        template:       
          metadata:
            name: ocnaddverify
          spec:
            volumes:
            - name: backup-vol
              persistentVolumeClaim:
                  claimName: backup-mysql-pvc
            - name: config-vol    
              configMap:
                name: config-backuprestore-scripts
            serviceAccountName: ocnadd-deploy-gitlab-admin      #---> update the service account name. Format:<namespace>-gitlab-admin
            containers:
            - name: ocnaddverify
              image: <repo-path>/ocnaddbackuprestore:1.0.7      #---> update repository path
              volumeMounts:
                - mountPath: "work-dir"
                  name: backup-vol
                - mountPath: "config-mysql-scripts"
                  name: config-vol
              env:
                - name: HOME
                  value: /tmp
                - name: DB_USER
                  valueFrom:
                    secretKeyRef:
                      name: db-secret
                      key: MYSQL_USER
                - name: DB_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: db-secret
                      key: MYSQL_PASSWORD
              command:
              - /bin/sh
              - -c
              - |
                cp /config-backuprestore-scripts/*.sh /tmp
                chmod +x /tmp/*.sh  
                echo "Checking backup path"
                ls -lh /work-dir/backup
                sleep 10m
            restartPolicy: Never
      status: {}

2. Run the following command to run the verify job for verifying the backup generated at the mounted PV by running inside the running container:

   kubectl create -f ocnadd_verify_backup.yaml

3. If the external storage is used as PV/PVC, get inside the ocnaddverify container using the following commands:

   1. kubectl exec -it <verify_pod> -n <ocnadd namespace> -- bash

   2. Change the directory to /work-dir/backup and inside the latest backup folder OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss.
   3. Verify the DB backup and Kafka metadata backup files

7.4.3 Obtain the OCNADD Backup files

1. Run the Verifying OCNADD Backup procedure to spawn the verify_pod.
2. Get into the running ocnaddverify pod to identify and retrieve the desire backup folder with the following commands:
   1. Access the pod

      kubectl exec -it <ocnaddverify-*> -n <namespace> -- bash

      where namespace = namespace of ocnadd

      ocnaddverify-* = is the verify pod in the namespace

   2. Change the directory to /work-dir/backup and identify the backup folder "OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss"
   3. Exit from the ocnaddverify pod
3. Copy the backup from the pod to the local bastion server by copying the OCNADD Database and kafka_metadata files from the ocnaddverify pod with the following commands:
   1. Onbastion host create a folder with the name similar to backup folder "OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss"
   2. Change to the folder created in step a.
   3. Use below command to copy backup files from the ocnaddverify pod To copy ocnadd Database backup:

      kubectl exec -n <namespace>  <verify_pod> -- cat/work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss/DB_BACKUP_DD-MM-YYYY_hh-mm-ss.sql  > DB_BACKUP_DD-MM-YYYY_hh-mm-ss.sql

      Example:

      kubectl exec -n ocnadd ocnaddverify-ncczz -- cat/work-dir/backup/OCNADD_BACKUP_11-11-2022_16-54-05/DB_BACKUP_11-11-2022_16-54-05.sql  > DB_BACKUP_11-11-2022_16-54-05.sql

      To copy ocnadd Kafka metadata backup:

      
      kubectl exec -n <namespace>  <verify_pod> -- cat/work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss/kafka_metadata_DD-MM-YYYY_hh-mm-ss.txt  > kafka_metadata_DD-MM-YYYY_hh-mm-ss.txt

      Example:

      kubectl exec -n ocnadd ocnaddverify-ncczz -- cat/work-dir/backup/OCNADD_BACKUP_11-11-2022_16-54-05/kafka_metadata_11-11-2022_16-54-05.txt  > kafka_metadata_11-11-2022_16-54-05.txt

      where, namespace = namespace of ocnadd

      verify_pod = is the verify pod in the namespace

7.4.4 Copy and Restore the OCNADD backup

1. Get the ocnadd backup folder.
2. Execute the Verifying OCNADD Backup procedure to spawn the verify_pod.
3. Create a folder inside the verify_pod under directory path:/work-dir/backup/ with the name similar to the backup folder "OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss" .
   1. Access the pod with "kubectl exec -it <ocnaddverify-*> -n <namespace> -- bash".
   2. Create backup folder using command "mkdir -p /work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss".
4. Copy the backup files from the local bastion server to the running ocnaddverify pod.
   1. Go to the ocnadd backup directory
   2. Use the following command to copy backup files inside the ocnaddverify pod:

      cat DB_BACKUP_DD-MM-YYYY_hh-mm-ss.sql | kubectl exec -i -n <namespace> <verify_pod> -- bash -c "cat > /work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss/DB_BACKUP_DD-MM-YYYY_hh-mm-ss.sql"

      Example:

      cat DB_BACKUP_11-11-2022_16-54-05.sql | kubectl exec -i -n ocnadd ocnaddverify-ncczz -- bash
            -c "cat >
            /work-dir/backup/OCNADD_BACKUP_11-11-2022_16-54-05/DB_BACKUP_11-11-2022_16-54-05.sql"

   3. To copy ocnadd Kafka metadata backup:

      cat kafka_metadata_DD-MM-YYYY_hh-mm-ss.txt | kubectl exec -i -n <namespace> <verify_pod> -- bash -c "cat  > /work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss/kafka_metadata_DD-MM-YYYY_hh-mm-ss.txt"

      Example:

      cat kafka_metadata_11-11-2022_16-54-05.txt | kubectl exec -i -n ocnadd ocnaddverify-ncczz -- bash -c "cat > /work-dir/backup/OCNADD_BACKUP_11-11-2022_16-54-05/kafka_metadata_11-11-2022_16-54-05.txt"

5. Verify the backup has been copied by getting into the ocnaddverify pod and path: /work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss
6. Restore OCNADD using the procedure defined in Create OCNADD Restore Job.
7. Restart the ocnaddconfiguration, ocnaddalarm, and ocnaddhealthmonitoring pods.