1. Network Analytics Data Director Disaster Recovery Guide
  2. Performing OCNADD Backup Procedures

4 Performing OCNADD Backup Procedures

Performing OCNADD Manual Backup

Perform the following steps to take the manual backup:

  1. Go to custom-templates folder inside the extracted ocnadd-release package and update the ocnadd_manualBackup.yaml file with the following information:
    1. Values for BACKUP_DATABASES can be set to ALL (i.e. healthdb_schema, configuration_schema, and alarm_schema) or the individual DB names can also be passed. By default, the value is 'ALL'.
    2. Update other values as follows:
      
apiVersion:batch/v1
kind:Job
metadata:
  name:ocnaddmanualbackup
  namespace:ocnadd-deploy     #---> update the namespace
  spec:
     template:
           metadata:
              name:ocnaddmanualbackup    
           spec:  
            volumes:
            -name:backup-vol
            persistentVolumeClaim:   
                claimName:backup-mysql-pvc 
            -name:config-vol  
             configMap:  
               name:config-backuprestore-scripts  
            serviceAccountName:ocnadd-deploy-gitlab-admin      #---> update the service account name. Format:<namespace>-gitlab-admin 
            containers:  
           -name:ocnaddmanualbackup  
            image:<repo-path>/ocnaddbackuprestore:1.0.0        #---> update repository path   
            volumeMounts:       
              -mountPath:"work-dir"      
               name:backup-vol      
              -mountPath:"config-backuprestore-scripts"    
               name:config-vol 
            env:       
              -name:HOME    
               value:/tmp    
              -name:DB_USER    
               valueFrom:         
                 secretKeyRef:          
                   name:db-secret        
                   key:MYSQL_USER  
    
              -name:DB_PASSWORD            
               valueFrom:         
                  secretKeyRef:       
                    name:db-secret    
                    key:MYSQL_PASSWORD     

               -name:BACKUP_DATABASES     
                value:ALL   
         command:       
         - /bin/sh    
         - -c       
         - |        
           cp /config-backuprestore-scripts/*.sh /tmp  
           chmod +x /tmp/*.sh   
           mkdir/work-dir/backup      
           touch/work-dir/backup/DB_BACKUP_$(date +%d-%m-%Y_%H-%M-%S).sql      
           echo "Executing manual backup script"     
           bash /tmp/backup.sh $BACKUP_DATABASES   
          ls -lh /work-dir/backup     
   restartPolicy:Never
status:{}
  2. Execute the below command to run the job:
    kubectl create -f ocnadd_manualBackup.yaml

Verifying OCNADD Backup

Caution:

The connectivity between the external storage via either PV/PVC or network connectivity must be ensured.

To verify the backup, perform the following steps:
  1. Go to the custom-templates folder inside the extracted ocnadd-release package and update the ocnadd_verify_backup.yaml file with the following information:
    1. Sleep time is configurable, update it if required. (default sleep is 10m)

    2. Update other values as follows:

      apiVersion: batch/v1
kind: Job
metadata:
  name: ocnaddverify
  namespace: ocnadd-deploy          #---> update the namespace
spec:
  template:       
    metadata:
      name: ocnaddverify
    spec:
      volumes:
      - name: backup-vol
        persistentVolumeClaim:
            claimName: backup-mysql-pvc
      - name: config-vol    
        configMap:
          name: config-backuprestore-scripts
      serviceAccountName: ocnadd-deploy-gitlab-admin      #---> update the service account name. Format:<namespace>-gitlab-admin
      containers:
      - name: ocnaddverify
        image: <repo-path>/ocnaddbackuprestore:1.0.0      #---> update repository path
        volumeMounts:
          - mountPath: "work-dir"
            name: backup-vol
          - mountPath: "config-mysql-scripts"
            name: config-vol
        env:
          - name: HOME
            value: /tmp
          - name: DB_USER
            valueFrom:
              secretKeyRef:
                name: db-secret
                key: MYSQL_USER
          - name: DB_PASSWORD
            valueFrom:
              secretKeyRef:
                name: db-secret
                key: MYSQL_PASSWORD
        command:
        - /bin/sh
        - -c
        - |
          cp /config-backuprestore-scripts/*.sh /tmp
          chmod +x /tmp/*.sh  
          echo "Checking backup path"
          ls -lh /work-dir/backup
          sleep 10m
      restartPolicy: Never
status: {}
  2. Run the following command to run the verify job for verifying the backup generated at the mounted PV by running inside the running container:
    kubectl create -f ocnadd_verify_backup.yaml
  3. If the external storage is used as PV/PVC, get inside the ocnaddverify container using the following commands: 
    1. kubectl exec -it <verify_pod> -n <ocnadd namespace> -- bash
    2. Change the directory to /work-dir/backup and inside the latest backup folder OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss.
    3. Verify the DB backup and Kafka metadata backup files

Obtain the OCNADD Backup files

  1. Run the Verifying OCNADD Backup procedure to spawn the verify_pod.
  2. Get into the running ocnaddverify pod to identify and retrieve the desire backup folder with the following commands:
    1. Access the pod
      kubectl exec -it <ocnaddverify-*> -n <namespace> -- bash

      where namespace = namespace of ocnadd

      ocnaddverify-* = is the verify pod in the namespace

    2. Change the directory to /work-dir/backup and identify the backup folder "OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss"
    3. Exit from the ocnaddverify pod
  3. Copy the backup from the pod to the local bastion server by copying the OCNADD Database and kafka_metadata files from the ocnaddverify pod with the following commands:
    1. Onbastion host create a folder with the name similar to backup folder "OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss"
    2. Change to the folder created in step a.
    3. Use below command to copy backup files from the ocnaddverify pod To copy ocnadd Database backup:
      kubectl exec -n <namespace>  <verify_pod> -- cat/work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss/DB_BACKUP_DD-MM-YYYY_hh-mm-ss.sql  > DB_BACKUP_DD-MM-YYYY_hh-mm-ss.sql
      Example:
      kubectl exec -n ocnadd ocnaddverify-ncczz -- cat/work-dir/backup/OCNADD_BACKUP_11-11-2022_16-54-05/DB_BACKUP_11-11-2022_16-54-05.sql  > DB_BACKUP_11-11-2022_16-54-05.sql
      To copy ocnadd Kafka metadata backup:
      
kubectl exec -n <namespace>  <verify_pod> -- cat/work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss/kafka_metadata_DD-MM-YYYY_hh-mm-ss.txt  > kafka_metadata_DD-MM-YYYY_hh-mm-ss.txt

      Example:

      kubectl exec -n ocnadd ocnaddverify-ncczz -- cat/work-dir/backup/OCNADD_BACKUP_11-11-2022_16-54-05/kafka_metadata_11-11-2022_16-54-05.txt  > kafka_metadata_11-11-2022_16-54-05.txt

      where, namespace = namespace of ocnadd

      verify_pod = is the verify pod in the namespace

Copy and Restore the OCNADD backup

  1. Get the ocnadd backup folder.
  2. Execute the Verifying OCNADD Backup procedure to spawn the verify_pod.
  3. Create a folder inside the verify_pod under directory path:/work-dir/backup/ with the name similar to the backup folder "OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss" .
    1. Access the pod with "kubectl exec -it <ocnaddverify-*> -n <namespace> -- bash".
    2. Create backup folder using command "mkdir -p /work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss".
  4. Copy the backup files from the local bastion server to the running ocnaddverify pod.
    1. Go to the ocnadd backup directory
    2. Use the following command to copy backup files inside the ocnaddverify pod:
      cat DB_BACKUP_DD-MM-YYYY_hh-mm-ss.sql | kubectl exec -i -n <namespace> <verify_pod> -- bash -c "cat > /work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss/DB_BACKUP_DD-MM-YYYY_hh-mm-ss.sql

      Example:

      cat DB_BACKUP_11-11-2022_16-54-05.sql | kubectl exec -i -n ocnadd ocnaddverify-ncczz -- bash
      -c "cat >
      /work-dir/backup/OCNADD_BACKUP_11-11-2022_16-54-05/DB_BACKUP_11-11-2022_16-54-05.sql
    3. To copy ocnadd Kafka metadata backup: 
      cat kafka_metadata_DD-MM-YYYY_hh-mm-ss.txt | kubectl exec -i -n <namespace> <verify_pod> -- bash -c "cat  > /work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss/kafka_metadata_DD-MM-YYYY_hh-mm-ss.txt

      Example:

      cat kafka_metadata_11-11-2022_16-54-05.txt | kubectl exec -i -n ocnadd ocnaddverify-ncczz -- bash -c "cat > /work-dir/backup/OCNADD_BACKUP_11-11-2022_16-54-05/kafka_metadata_11-11-2022_16-54-05.txt"
  5. Verify the backup has been copied by getting into the ocnaddverify pod and path: /work-dir/backup/OCNADD_BACKUP_DD-MM-YYYY_hh-mm-ss
  6. Restore OCNADD using the procedure defined in Create OCNADD Restore Job.
  7. Restart the ocnaddconfiguration, ocnaddalarm, and ocnaddhealthmonitoring pods.