5 Rolling Back OCNADD

This chapter describes the OCNADD rollback procedure from a target release to a previously supported version. In the current release, centralized deployment is supported, and OCNADD microservices will follow the centralized deployment mode with the management group and the worker group separation concerning microservices functions.

The rollback will be supported for the following cases:

  • Upgrade was done from source releases to the target release in the Centralized deployment mode.
  • Upgrade was done from source releases to the target release in the Non-Centralized deployment mode.

Note:

Changing mode from Centralized deployment mode to Non-Centralized deployment is not supported within the target release (23.4.0.0.1). See Supported Upgrade Paths section of this document.

Table 5-1 Supported Rollback Paths

Source Release Target Release
23.4.0.0.1 23.4.0, 23.3.x
There are three usecases of OCNADD rollback:
  1. Rollback Usecase 1: Upgrade Was Done from Source Releases to Target Release in the Centralized Deployment Mode
  2. Rollback Usecase 2: Upgrade Was Done from Source Releases to Target Release in the Non-Centralized Mode
  3. Rollback Usecase3: Upgrade Was Done to Move to the Centralized Deployment Mode and Got Failed

Rollback Steps

These steps are common for all the rollback cases. To roll back to a previous version, follow the steps as mentioned:

Note:

  • (Optional) A timeout interval of 15 minutes can be set while performing an upgrade as only one POD of the OCNADD services is upgraded at a time.
  • Ensure the status for the target version in the helm history is not in failed or error state.
  • Ensure to delete the Kafka feed if enabled on 23.3.0, before attempting the rollback, follow the section 'Disable Kafka Feed Configuration Support' of the Oracle Communications Network Analytics Data Director User Guide.

Rollback Usecase 1: Upgrade Was Done from Source Releases to Target Release in the Centralized Deployment Mode

To roll back to a previous working version in the target rollback release, follow these steps:

  1. Uninstall the Management Group
    helm uninstall <management-release-name> --namespace <ocndd-namespace>

    Where,

    • <management-release-name> is the name used to identify the management group deployment.
    • <ocndd-namespace>is the namespace of the OCNADD deployment.
    For example:
    helm uninstall ocnadd-mgmt --namespace ocnadd-deploy
  2. Check Revision for Rollback:
    helm history <ocandd-release-name> --namespace <ocndd-namespace>

    Where,

    <ocandd-release-name> is the release name used for default worker group deployment.

    For example: 
    helm history ocnadd --namespace ocnadd-deploy
    Sample Helm history output:
    
REVISION  UPDATED                    STATUS      CHART              APP VERSION  DESCRIPTION
1         Fri Oct 13 04:57:43 2023   superseded  ocnadd-0.0.0-23.3.0 23.3.0.0.0   Install complete
2         Fri Oct 13 05:07:43 2023   superseded  ocnadd-0.0.0-23.3.0 23.3.0.0.0   Upgrade complete (revision required for rollback)
3         Fri Oct 13 05:21:52 2023   superseded  ocnadd-23.3.0      23.3.0.0.0   Upgrade complete
4         Fri Oct 13 05:35:15 2023   deployed    ocnadd-23.4.0      23.4.0.0.0   Upgrade complete
  3. Rollback to Required Revision:

    Note:

    Where the required revision is the revision at which the first successful upgrade was performed during the "Pre-upgrade procedure" step "d in Upgrade Tasks. Perform Helm upgrade on source release to update the backup-pvc changes". 
    helm rollback <ocandd-release-name> <REVISION> --namespace <ocndd-namespace>

    Where,

    <REVISION> is the revision number obtained from the previous step for the desired rollback.

    For example: 
    helm rollback ocnadd 2 --namespace ocnadd-deploy
  4. Restore the Backup-PVC:
    1. Run the command from the source release folder corresponding to 23.4.0.0.1:
      bash custom-templates/copy_backup_pvc.bash <source-release-namespace>
      For example:
      bash custom-templates/copy_backup_pvc.bash ocnadd-deploy

      Note:

      The default storage class is "standard". If the storage class is different, provide its name as a second argument.
      For example:
       bash custom-templates/copy_backup_pvc.bash ocnadd-deploy <storageClass-name>
      Sample output with namespace as the argument:
      
      PVC backup-mysql-pvc-temp exists in namespace ocnadd-deploy.
      PVC backup-mysql-pvc exists in namespace ocnadd-deploy.
      Running job to Copy backup contents from backup-mysql-pvc-temp to backup-mysql-pvc
      job.batch/ocnaddcopybackuppvc-temp-to-orig created
      Copied successfully
      persistentvolumeclaim "backup-mysql-pvc-temp" deleted
    2. Update File References in the Target Release Folder:

      Change the directory to the target release folder.

      For example: For 23.3.0 release the folder is ocnadd-package-23.3.0
      
    sed -i 's/backup-mysql-pvc-temp/backup-mysql-pvc/g' ocnadd/templates/ocnadd-postrollback-version-db.yml
    sed -i 's/backup-mysql-pvc-temp/backup-mysql-pvc/g' ocnadd/templates/ocnadd-preupgrade-db.yaml
    sed -i 's/backup-mysql-pvc-temp/backup-mysql-pvc/g' ocnadd/templates/ocnadd-prerollback-db.yaml
    3. Verify and Update Management Group Services.
      Ensure all Management group services are set to 'true' in the ocnadd-custom-values.yaml of the target release. Update settings if necessary:
      
global:
  ocnaddalarm:
      enabled: false           ##---> to true
  ocnaddconfiguration:
      enabled: false           ##---> to true
  ocnaddhealthmonitoring:
      enabled: false           ##---> to true
  ocnaddbackuprestore:
      enabled: false           ##---> to true
  ocnaddadminsvc:
      enabled: false           ##---> to true
  ocnadduirouter:
      enabled: false           ##---> to true
  ocnaddgui:
      enabled: false           ##---> to true
    4. Rollback Adapter Services to Previous Version.
      Run the command to upgrade all Adapter services to the previous version:
      helm upgrade <ocandd-release-name> -f ocnadd-custom-values-<previous_release>.yaml --namespace <ocnadd-namespace> <helm_chart_previous_release> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-23.3.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
  5. Verify if the rollback is successful.
    • Ensure pods respawned after the upgrade have the latest age (in seconds).
    • Check Adapter pods and data feed status via the GUI.
    • The Kafka broker pods may be in crashloopback state after the rollback due to a service account issue. For any such Kafka broker issues after rollback, refer to the "Kafka brokers in crashloop state after rollback" section in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

    Note:

    If the rollback remains unsuccessful, refer to the troubleshooting steps outlined in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

Rollback Usecase 2: Upgrade Was Done from Source Releases to Target Release in the Non-Centralized Mode

To rollback to a previous working version in the target rollback release:
  1. Run the following command to check the revision to rollback:

    Where the required revision is the revision at which the helm upgrade was performed to disable the 'ocnadduirouter' and 'ocnaddgui' before upgrading to the target release. 

    $ helm history <ocandd-release-name> -n <ocnadd-namespace>

    Where,

    <ocandd-release-name> is the release name of the OCNADD deployment.

    <ocndd-namespace> is the name the namespace of OCNADD deployment.

    For example:

    helm history ocnadd --namespace ocnadd-deploy

    Sample Helm history output with patch 23.3.0 example:

    
REVISION        UPDATED                         STATUS          CHART                           APP VERSION     DESCRIPTION
1               Fri Oct 13 04:57:43 2023        superseded      ocnadd-0.0.0-23.3.0             23.3.0.0.0      Install complete
2               Fri Oct 13 05:07:43 2023        superseded      ocnadd-0.0.0-23.3.0             23.3.0.0.0      Upgrade complete =======> revision required for rollback
3               Fri Oct 13 05:35:15 2023        deployed        ocnadd-23.4.0                   23.4.0.0.0      Upgrade complete
  2. Run the command to rollback to the required revision:
    helm rollback <ocandd-release-name> <REVISION> --namespace <ocnadd-namespace>

    Where, <REVISION> is the revision number obtained in the previous step to which the services needs to be rolled backed.

    For example:

    helm rollback ocnadd 2 --namespace ocnadd-deploy
  3. Run the following command to roll back the Adapter service to the previous version along with the 'ocnadduirouter' and 'ocnaddgui':
    1. Go to the release folder.

      For example: For 23.3.0 release the folder is ocnadd-package-23.3.0.

    2. Update the ocnadd-custom-values.yaml file:
      
global.ocnadduirouter.enabled: false                            ##---> update it to 'true'
global.ocnaddgui.enabled: false                                 ##---> update it to 'true'
    3. Perform helm upgrade using helm charts folder:
      helm upgrade <ocandd-release-name> -f ocnadd-custom-values-<previous_release>.yaml --namespace <ocnadd-namespace> <helm_chart_previous_release> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
      For example: For release 23.3.0
      helm upgrade ocnadd -f ocnadd-custom-values-23.3.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
  4. Verify if the rollback is successful
    1. All the pods that has been respawned after upgrade, have the latest age (in secs)
    2. The Adapter pods also gets respawned for any upgrade. The status can also be verified from GUI for respective data feeds.
    3. The Kafka broker pods may be in crashloopback state after the rollback due to a service account issue. For any such Kafka broker issues after rollback, refer to the "Kafka brokers in crashloop state after rollback" section in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

    Note:

    If the rollback remains unsuccessful, refer to the troubleshooting steps outlined in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

Rollback Usecase3: Upgrade Was Done to Move to the Centralized Deployment Mode and Got Failed

In this case, the upgrade was done in the target release itself to move to the Centralized deployment mode from the Non-Centralized mode. In such a case, there will be only one default worker group running in the same namespace along with the management group services. The procedure is similar to "Rollback Usecase1", except that the target and source release will remain the same. The steps in the "Rollback Usecase1" should be followed, and care must be taken while using the same release charts of the source release (23.4.0) for rollback.

The procedure is applicable for the following scenarios:

The upgrade was successfully done from source releases 23.2.0.0.x/23.3.0.0.x to 23.4.0.0.0 release in Non-Centralized mode. Later on, the upgrade was performed to enable the Centralized deployment mode and it failed.

Note:

The above procedure is not applicable to switch between Non-Centralized and Centralized modes.