5 Rolling Back OCNADD

This chapter describes the OCNADD rollback procedure from a target release to a previously supported version. In the current release, centralized deployment is supported, and OCNADD microservices will follow the centralized deployment mode with the management group and the worker group separation concerning microservices functions.

The rollback will be supported for the following cases:

  • Upgrade was done from source releases to the target release in the Centralized deployment mode.
  • Upgrade was done from source releases to the target release in the Non-Centralized deployment mode.

Note:

Changing mode from Centralized deployment mode to Non-Centralized deployment is not supported within the target release (24.3.0). See Supported Upgrade Paths section of this document.

Table 5-1 Supported Rollback Paths

Source Release Target Release
24.3.0 24.1.0.0.x
24.3.0 24.2.0.0.x
24.3.0.0.1 24.3.0
There are three use cases of OCNADD rollback:
  1. Rollback Usecase 1: Upgrade Was Done from Source Releases (Non-Centralized) to Target Release in the Centralized Deployment Mode
  2. Rollback Usecase 2: Upgrade Was Done from Source Releases (Centralized) to Target Release in the Centralized Deployment Mode
  3. Rollback Usecase 3: Upgrade Was Done from Source Releases (Non-Centralized Mode) to Target Release in the Non-Centralized Mode
  4. Rollback Usecase 4: Upgrade Was Done to Move to the Centralized Deployment Mode and Got Failed

Rollback Steps

These steps are common for all the rollback cases. To roll back to a previous version, follow the steps as mentioned:

Note:

  • (Optional) A timeout interval of 15 minutes can be set while performing an upgrade as only one POD of the OCNADD services is upgraded at a time.
  • Ensure the status for the target version in the helm history is not in failed or error state.

Rollback Usecase 1: Upgrade Was Done from Source Releases (Non-Centralized) to Target Release in the Centralized Deployment Mode

Note:

This scenario is applicable for rollback from 24.3.0 Centralized Deployment Mode to 23.4.x or 24.1.x Non-Centralized Deployment Mode.

To roll back to a previous working version in the target rollback release, follow these steps:

  1. Uninstall the Management Group
    helm uninstall <management-release-name> --namespace <ocndd-namespace>

    Where,

    • <management-release-name> is the name used to identify the management group deployment.
    • <ocndd-namespace>is the namespace of the OCNADD deployment.
    For example:
    helm uninstall ocnadd-mgmt --namespace ocnadd-deploy
  2. Check Revision for Rollback:
    helm history <ocandd-release-name> --namespace <ocndd-namespace>

    Where,

    <ocandd-release-name> is the release name used for default worker group deployment.

    For example: 
    helm history ocnadd --namespace ocnadd-deploy
    Sample Helm history output:
    
REVISION   UPDATED                     STATUS        CHART                  APP VERSION       DESCRIPTION
1          Fri Feb 08 04:57:43 2024    superseded    ocnadd-0.0.0-23.4.0    23.4.0.0.0        Install complete
2          Fri Feb 08 05:07:43 2024    superseded    ocnadd-0.0.0-23.4.0    23.4.0.0.0        Upgrade complete (revision required for rollback)
3          Fri Feb 08 05:21:52 2024    superseded    ocnadd-23.4.0          23.4.0.0.0        Upgrade complete
4          Fri Feb 08 05:35:15 2024    deployed      ocnadd-24.2.0          24.2.0.0.0        Upgrade complete
  3. Before rolling back to 24.1.x/24.2.x, delete kafka-broker STS in orphan mode from 24.3.0 deployment:
    kubectl delete statefulset --cascade=orphan kafka-broker -n <namespace>

    Note:

    If NFs are deployed in same cluster and using the FQDN ".kafka-broker-headless.<ns>.svc.<domain>" as bootstrap of Kafka to connect to OCNADD then this FQDN should be updated as "*.kafka-broker.<ns>.svc.<domain>" in NFs after rollback to OCNADD 24.3.0.
  4. Rollback to Required Revision:
    helm rollback <ocandd-release-name> <REVISION> --namespace <ocndd-namespace>

    Where,

    <REVISION> is the required revision at which the first successful upgrade was performed in the "Preupgrade procedure" step "d". Perform Helm upgrade on source release to update the backup-pvc changes.

    For example: 
    helm rollback ocnadd 2 --namespace ocnadd-deploy
  5. Restore the Backup-PVC:
    1. Run the command from the source release folder corresponding to 24.3.0:
      bash custom-templates/copy_backup_pvc.bash <source-release-namespace> <storageClass-name> <service-account>

      Where

      • source-release-namespace is the OCNADD namespace of the source release
      • storageClass-name is the Storage Class of OCNADD deployment (default used is standard)
      • service-account is the Service Account name set in global.cluster.serviceAccount.name in ocnadd-custom-values-24.3.0.yaml
      For example:
      bash custom-templates/copy_backup_pvc.bash ocnadd-deploy standard ocnadd
      Sample output with namespace as the argument:
      
      PVC backup-mysql-pvc-temp exists in namespace ocnadd-deploy.
      PVC backup-mysql-pvc exists in namespace ocnadd-deploy.
      Running job to Copy backup contents from backup-mysql-pvc-temp to backup-mysql-pvc
      job.batch/ocnaddcopybackuppvc-temp-to-orig created
      Copied successfully
      persistentvolumeclaim "backup-mysql-pvc-temp" deleted
    2. Update File References in the Target Release Folder:

      Change the directory to the target release folder.

      For example: For 23.4.0 release the folder is ocnadd-package-23.4.0
      
    sed -i 's/backup-mysql-pvc-temp/backup-mysql-pvc/g' ocnadd/templates/ocnadd-postrollback-version-db.yml
    sed -i 's/backup-mysql-pvc-temp/backup-mysql-pvc/g' ocnadd/templates/ocnadd-preupgrade-db.yaml
    sed -i 's/backup-mysql-pvc-temp/backup-mysql-pvc/g' ocnadd/templates/ocnadd-prerollback-db.yaml
    3. Verify and Update Management Group Services.
      Ensure all Management group services are set to 'true' in the ocnadd-custom-values.yaml of the target release. Update settings if necessary:
      
global:
  ocnaddalarm:
      enabled: false           ##---> to true
  ocnaddconfiguration:
      enabled: false           ##---> to true
  ocnaddhealthmonitoring:
      enabled: false           ##---> to true
  ocnaddbackuprestore:
      enabled: false           ##---> to true
  ocnaddadminsvc:
      enabled: false           ##---> to true
  ocnadduirouter:
      enabled: false           ##---> to true
  ocnaddgui:
      enabled: false           ##---> to true
    4. Rollback Adapter Services to Previous Version.
      If only adapter configurations are present, run the command to upgrade all Adapter services to the previous version:
      helm upgrade <ocandd-release-name> -f ocnadd-custom-values-<previous_release>.yaml --namespace <ocnadd-namespace> <helm_chart_previous_release> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-24.2.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
    5. If Correlation Configurations are also present:
      helm upgrade <ocandd-release-name> -f ocnadd-custom-values-<previous_release>.yaml --namespace <ocnadd-namespace> <helm_chart_previous_release> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-24.2.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
    6. If ingress adapter configurations are also present:
      helm upgrade <ocandd-release-name> -f ocnadd-custom-values-<previous_release>.yaml --namespace <ocnadd-namespace> <helm_chart_previous_release> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-24.2.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
  6. Verify if the rollback is successful.
    • Ensure pods respawned after the upgrade have the latest age (in seconds).
    • Check Adapter pods and data feed status via the GUI.
    • The Kafka broker pods may be in crashloopback state after the rollback due to a service account issue. For any such Kafka broker issues after rollback, refer to the "Kafka brokers in crashloop state after rollback" section in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.
  7. Replace database name with the actual schema name of storage adapter in the below query and execute in any mysql client by connecting to the database:
    Drop database storageadapter_schema;

    Note:

    If the rollback remains unsuccessful, refer to the troubleshooting steps outlined in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

Rollback Usecase 2: Upgrade Was Done from Source Releases (Centralized) to Target Release in the Centralized Deployment Mode

Note:

This scenario is applicable for rollback from 24.2.0 Centralized Deployment Mode to 23.4.x or 24.1.0 Centralized Deployment Mode.

To roll back to a previous working version in the target rollback release, follow these steps:

Rollback of Management Group:

  1. Check Revision for Rollback:
    helm history <ocandd-release-name> --namespace <ocndd-mgmt-namespace>

    Where,

    <ocandd-release-name> is the release name used for default worker group deployment.

    For example: 
    helm history ocnadd --namespace ocnadd-deploy
    Sample Helm history output:
    
REVISION   UPDATED                     STATUS        CHART                  APP VERSION       DESCRIPTION
1          Fri Feb 08 04:57:43 2024    superseded    ocnadd-0.0.0-23.4.0    23.4.0.0.0        Install complete
2          Fri Feb 08 05:07:43 2024    superseded    ocnadd-0.0.0-23.4.0    23.4.0.0.0        Upgrade complete (revision required for rollback)
3          Fri Feb 08 05:21:52 2024    superseded    ocnadd-23.4.0          23.4.0.0.0        Upgrade complete
4          Fri Feb 08 05:35:15 2024    deployed      ocnadd-24.2.0          24.2.0.0.0        Upgrade complete
  2. Rollback to Required Revision:
    helm rollback <ocandd-release-name> <REVISION> --namespace <ocndd-mgmt-namespace>

    Where,

    <REVISION> is the revision number obtained in the previous step to which the services need to be rolled back.

    For example: 
    helm rollback ocnadd 2 --namespace ocnadd-deploy

Rollback of Worker Groups

  1. Check Revision for Rollback:
    helm history <ocandd-release-name> --namespace <ocnadd-workergroup-namespace>

    Where,

    <ocandd-release-name> is the release name used for default worker group deployment.

    For example: 
    helm history ocnadd --namespace ocnadd-deploy
    Sample Helm history output:
    
REVISION   UPDATED                     STATUS        CHART                  APP VERSION       DESCRIPTION
1          Fri Feb 08 04:57:43 2024    superseded    ocnadd-0.0.0-23.4.0    23.4.0.0.0        Install complete
2          Fri Feb 08 05:07:43 2024    superseded    ocnadd-0.0.0-23.4.0    23.4.0.0.0        Upgrade complete (revision required for rollback)
3          Fri Feb 08 05:21:52 2024    superseded    ocnadd-23.4.0          23.4.0.0.0        Upgrade complete
4          Fri Feb 08 05:35:15 2024    deployed      ocnadd-24.2.0          24.2.0.0.0        Upgrade complete
  2. Rollback to Required Revision:
    helm rollback <ocandd-release-name> <REVISION> --namespace <ocndd-workergroup-namespace>

    Where,

    <REVISION> is the revision number obtained in the previous step to which the services need to be rolled back.

    For example: 
    helm rollback ocnadd 2 --namespace ocnadd-deploy
  3. Rollback Adapter Services to the Previous Version. Run the below commands to rollback all Adapter/Correlation services to the previous version:
    1. Perform helm upgrade using the management group charts folder:
      1. If no correlation configurations are present:
        helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
        For example:
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
      2. If correlation configurations are also present:
        helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
        For example:
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
  4. Check the status of the upgrade, and monitor the pods to come back to the RUNNING state.
  5. Replace database name with the actual schema name of storage adapter in the below query and execute in any mysql client by connecting to the database. The step can be skipped if rolling back to 24.2.0 and storage adapter was configured in the correlation configuration.
    Drop database storageadapter_schema;

Note:

If the rollback is unsuccessful, see troubleshooting steps outlined in the Oracle Communications Network Analytics Data Director Troubleshooting Guide

Rollback Usecase 3: Upgrade Was Done from Source Releases (Non-Centralized Mode) to Target Release in the Non-Centralized Mode

To rollback to a previous working version in the target rollback release:
  1. Run the following command to check the revision to rollback:

    Where the required revision is the revision at which the helm upgrade was performed to disable the 'ocnadduirouter' and 'ocnaddgui' before upgrading to the target release. 

    $ helm history <ocandd-release-name> -n <ocnadd-namespace>

    Where,

    <ocandd-release-name> is the release name of the OCNADD deployment.

    <ocndd-namespace> is the name the namespace of OCNADD deployment.

    For example:

    helm history ocnadd --namespace ocnadd-deploy

    Sample Helm history output with patch 23.4.0 example:

    
REVISION        UPDATED                    STATUS        CHART                  APP VERSION             DESCRIPTION
1               Fri Feb 08 04:57:43 2024   superseded    ocnadd-0.0.0-23.4.0    23.4.0.0.0 Install      complete
2               Fri Feb 08 05:07:43 2024   superseded    ocnadd-0.0.0-23.4.0    23.4.0.0.0 Upgrade      complete =======> revision required for rollback
3               Fri Feb 08 05:35:15 2024   deployed      ocnadd-24.2.0          24.2.0.0.0 Upgrade      complete
  2. Before rolling back to 24.1.x/24.2.x, delete kafka-broker STS in orphan mode from 24.3.0 deployment:
    kubectl delete statefulset --cascade=orphan kafka-broker -n <namespace>

    Note:

    If NFs deployed in same cluster and using the FQDN ".kafka-broker-headless.<ns>.svc.<domain>" as bootstrap of kafka to connect to OCNADD then this FQDN should be updated as "*.kafka-broker.<ns>.svc.<domain>" in NFs after rollback to OCNADD 24.3.0.
  3. Run the command to rollback to the required revision:
    helm rollback <ocandd-release-name> <REVISION> --namespace <ocnadd-namespace>

    Where, <REVISION> is the revision number obtained in the previous step to which the services needs to be rolled backed.

    For example:

    helm rollback ocnadd 2 --namespace ocnadd-deploy
  4. Run the following command to roll back the Adapter service to the previous version along with the 'ocnadduirouter' and 'ocnaddgui':
    1. Go to the release folder.

      For example: For 24.1.0 release the folder is ocnadd-package-24.1.0.

    2. Update the ocnadd-custom-values.yaml file:
      
global.ocnadduirouter.enabled: false                            ##---> update it to 'true'
global.ocnaddgui.enabled: false                                 ##---> update it to 'true'
    3. If only adapter configurations are present, perform helm upgrade using helm charts folder:
      helm upgrade <ocandd-release-name> -f ocnadd-custom-values-<previous_release>.yaml --namespace <ocnadd-namespace> <helm_chart_previous_release> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
      For example: For release 24.1.0
      helm upgrade ocnadd -f ocnadd-custom-values-24.1.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
    4. If correlation configurations are also present, perform helm upgrade using helm charts folder:
      helm upgrade <ocandd-release-name> -f ocnadd-custom-values-<previous_release>.yaml --namespace <ocnadd-namespace> <helm_chart_previous_release> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-24.1.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true

      Note:

      If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag "global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true" in the above command:
      helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true
  5. Verify if the rollback is successful
    1. All the pods that has been respawned after upgrade, have the latest age (in secs)
    2. The Adapter pods also gets respawned for any upgrade. The status can also be verified from GUI for respective data feeds.
    3. The Kafka broker pods may be in crashloopback state after the rollback due to a service account issue. For any such Kafka broker issues after rollback, refer to the "Kafka brokers in crashloop state after rollback" section in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.
  6. Replace database name with the actual schema name of storage adapter in the below query and execute in any mysql client by connecting to the database. The step can be skipped if rolling back to 24.2.0 and storage adapter was configured in the correlation configuration.
    Drop database storageadapter_schema;

    Note:

    If the rollback remains unsuccessful, see troubleshooting steps outlined in the Oracle Communications Network Analytics Data Director Troubleshooting Guide

Rollback Usecase 4: Upgrade Was Done to Move to the Centralized Deployment Mode and Got Failed

In this case, the upgrade was done in the target release itself to move to the Centralized deployment mode from the Non-Centralized mode. In such a case, there will be only one default worker group running in the same namespace along with the management group services. The procedure is similar to "Rollback Usecase1", except that the target and source release will remain the same. The steps in the "Rollback Usecase1" should be followed, and care must be taken while using the same release charts of the source release (24.3.0) for rollback.

The procedure is applicable for the following scenarios:

The upgrade was successfully done from source releases 24.1.0.0.x/24.2.0.0.x to 24.3.0.0.0 release in Non-Centralized mode. Later on, the upgrade was performed to enable the Centralized deployment mode and it failed.

Note:

The above procedure is not applicable to switch between Non-Centralized and Centralized modes.