5 Rolling Back OCNADD

This chapter describes the OCNADD rollback procedure from a target release to a previously supported version. In the current release, centralized deployment is supported, and OCNADD microservices will follow the centralized deployment mode with the management group and the worker group separation concerning microservices functions.

The rollback will be supported for the following cases:

  • Upgrade was done from source releases to the target release in the Centralized deployment mode.

Note:

Changing mode from Centralized deployment mode to Non-Centralized deployment is not supported within the target release (25.1.200). See Supported Upgrade Paths section of this document.

Table 5-1 Supported Rollback Paths

Source Release Target Release
25.1.200 24.3.0.0.x
25.1.200 25.1.1xx
There are three use cases of OCNADD rollback:
  1. Rollback Usecase 1: Upgrade Was Done from Source Releases (Non-Centralized) to Target Release in the Centralized Deployment Mode
  2. Rollback Usecase 2: Upgrade Was Done from Source Releases (Centralized) to Target Release in the Centralized Deployment Mode
  3. Rollback Usecase 3: Upgrade Was Done to Move to the Centralized Deployment Mode and Got Failed

Rollback Steps

These steps are common for all the rollback cases. To roll back to a previous version, follow the steps as mentioned:

Note:

  • (Optional) A timeout interval of 15 minutes can be set while performing an upgrade as only one POD of the OCNADD services is upgraded at a time.
  • Ensure the status for the target version in the helm history is not in failed or error state.

Rollback Usecase 1: Upgrade Was Done from Source Releases (Non-Centralized) to Target Release in the Centralized Deployment Mode

Note:

This scenario is applicable for rollback from 25.1.200 Centralized Deployment Mode to 25.1.1xx or 24.3.x Non-Centralized Deployment Mode.

To roll back to a previous working version in the target rollback release, follow these steps:

  1. Uninstall the Management Group
    helm uninstall <management-release-name> --namespace <ocndd-namespace>

    Where,

    • <management-release-name> is the name used to identify the management group deployment.
    • <ocndd-namespace>is the namespace of the OCNADD deployment.
    For example:
    helm uninstall ocnadd-mgmt --namespace ocnadd-deploy
  2. Check Revision for Rollback:
    helm history <ocandd-release-name> --namespace <ocndd-namespace>

    Where,

    <ocandd-release-name> is the release name used for default worker group deployment.

    For example: 
    helm history ocnadd --namespace ocnadd-deploy
    Sample Helm history output:
    
REVISION   UPDATED                     STATUS        CHART                  APP VERSION       DESCRIPTION
1          Fri Feb 08 04:57:43 2024    superseded    ocnadd-0.0.0-24.3.0    24.3.0.0.0        Install complete
2          Fri Feb 08 05:07:43 2024    superseded    ocnadd-0.0.0-24.3.0    24.3.0.0.0        Upgrade complete (revision required for rollback)
3          Fri Feb 08 05:21:52 2024    superseded    ocnadd-24.3.0          24.3.0.0.0        Upgrade complete
4          Fri Feb 08 05:35:15 2024    deployed      ocnadd-25.1.200        25.1.200.0.0        Upgrade complete
  3. Rollback to Required Revision:
    helm rollback <ocandd-release-name> <REVISION> --namespace <ocndd-namespace>

    Where,

    <REVISION> is the required revision at which the first successful upgrade was performed in the "Preupgrade procedure" step "d". Perform Helm upgrade on source release to update the backup-pvc changes.

    For example: 
    helm rollback ocnadd 2 --namespace ocnadd-deploy
  4. Restore the Backup-PVC:
    1. Run the command from the source release folder corresponding to 25.1.200:
      bash custom-templates/copy_backup_pvc.bash <source-release-namespace> <storageClass-name> <service-account>

      Where

      • source-release-namespace is the OCNADD namespace of the source release
      • storageClass-name is the Storage Class of OCNADD deployment (default used is standard)
      • service-account is the Service Account name set in global.cluster.serviceAccount.name in ocnadd-custom-values-25.1.200.yaml
      For example:
      bash custom-templates/copy_backup_pvc.bash ocnadd-deploy standard ocnadd
      Sample output with namespace as the argument:
      
      PVC backup-mysql-pvc-temp exists in namespace ocnadd-deploy.
      PVC backup-mysql-pvc exists in namespace ocnadd-deploy.
      Running job to Copy backup contents from backup-mysql-pvc-temp to backup-mysql-pvc
      job.batch/ocnaddcopybackuppvc-temp-to-orig created
      Copied successfully
      persistentvolumeclaim "backup-mysql-pvc-temp" deleted
    2. Update File References in the Target Release Folder:

      Change the directory to the target release folder.

      For example: For 24.3.0 release the folder is ocnadd-package-24.3.0.
      
    sed -i 's/backup-mysql-pvc-temp/backup-mysql-pvc/g' ocnadd/templates/ocnadd-postrollback-version-db.yml
    sed -i 's/backup-mysql-pvc-temp/backup-mysql-pvc/g' ocnadd/templates/ocnadd-preupgrade-db.yaml
    sed -i 's/backup-mysql-pvc-temp/backup-mysql-pvc/g' ocnadd/templates/ocnadd-prerollback-db.yaml
    3. Verify and Update Management Group Services.
      Ensure all Management group services are set to 'true' in the ocnadd-custom-values.yaml of the target release. Update settings if necessary:
      
global:
  ocnaddalarm:
      enabled: false              ##---> to true
  ocnaddconfiguration:
      enabled: false              ##---> to true
  ocnaddhealthmonitoring:
      enabled: false              ##---> to true
  ocnaddbackuprestore:
      enabled: false              ##---> to true
  ocnaddadminsvc:
      enabled: false              ##---> to true
  ocnadduirouter:
      enabled: false              ##---> to true
  ocnaddgui:
      enabled: false              ##---> to true
    4. Rollback Adapter Services to Previous Version.
      If only adapter configurations are present, run the command to upgrade all Adapter services to the previous version:
      helm upgrade <ocandd-release-name> -f ocnadd-custom-values-<previous_release>.yaml --namespace <ocnadd-namespace> <helm_chart_previous_release> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-24.3.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
    5. If Correlation Configurations are also present:
      helm upgrade <ocandd-release-name> -f ocnadd-custom-values-<previous_release>.yaml --namespace <ocnadd-namespace> <helm_chart_previous_release> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-24.3.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
    6. If ingress adapter configurations are also present:
      helm upgrade <ocandd-release-name> -f ocnadd-custom-values-<previous_release>.yaml --namespace <ocnadd-namespace> <helm_chart_previous_release> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-24.3.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
    7. Verify if the rollback is successful.
    8. Ensure pods respawned after the upgrade have the latest age (in seconds).
    9. Check Adapter pods and data feed status using the GUI.
    10. The Kafka broker pods may be in crashloopback state after the rollback due to a service account issue. For any such Kafka broker issues after rollback, refer to the "Kafka brokers in crashloop state after rollback" section in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

Note:

  • If the rollback remains unsuccessful, refer to the troubleshooting steps outlined in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.
  • Upon rollback completion, if the export, correlation, aggregation, ingress adapter, and/or consumer adapter services do not receive the configuration notification in a timely manner and fail to start processing data, see the 'Invalid Subscription Entry in the Subscription Table' section in the Oracle Communications Network Analytics Troubleshooting Guide.
  • If the rollback is successful and intraTLS is disabled, then create the certificates of all the OCNADD services. For required certificates, see rollbacked release Oracle Communication Network Analytics Security Guide.

Rollback Usecase 2: Upgrade Was Done from Source Releases (Centralized) to Target Release in the Centralized Deployment Mode

Note:

This scenario is applicable for rollback from 25.1.200 Centralized Deployment Mode to 24.3.x or 25.1.100 Centralized Deployment Mode.

To roll back to a previous working version in the target rollback release, follow these steps:

Rollback of Management Group:

  1. Check Revision of the Management Group Release to Rollback:
    helm history <ocandd-release-name> --namespace <ocndd-mgmt-namespace>

    Where,

    <ocandd-release-name> is the release name used for default worker group deployment.

    For example: 
    helm history ocnadd --namespace ocnadd-deploy
    Sample Helm history output:
    
REVISION   UPDATED                     STATUS        CHART                  APP VERSION       DESCRIPTION
1          Fri Feb 08 04:57:43 2024    superseded    ocnadd-0.0.0-24.3.0    24.3.0.0.0        Install complete
2          Fri Feb 08 05:07:43 2024    superseded    ocnadd-0.0.0-24.3.0    24.3.0.0.0        Upgrade complete (revision required for rollback)
3          Fri Feb 08 05:35:15 2024    deployed      ocnadd-25.1.200        25.1.200.0.0        Upgrade complete
  2. Rollback to Required Revision:
    helm rollback <ocandd-release-name> <REVISION> --namespace <ocndd-mgmt-namespace>

    Where,

    <REVISION> is the revision number obtained in the previous step to which the services need to be rolled back.

    For example: 
    helm rollback ocnadd 2 --namespace ocnadd-deploy

Rollback of Worker Groups

  1. Check Revision for Rollback:
    helm history <ocandd-release-name> --namespace <ocnadd-workergroup-namespace>

    Where,

    <ocandd-release-name> is the release name used for default worker group deployment.

    For example: 
    helm history ocnadd --namespace ocnadd-deploy
    Sample Helm history output:
    
REVISION   UPDATED                     STATUS        CHART                  APP VERSION       DESCRIPTION
1          Fri Feb 08 04:57:43 2024    superseded    ocnadd-0.0.0-24.3.0    24.3.0.0.0        Install complete
2          Fri Feb 08 05:07:43 2024    superseded    ocnadd-0.0.0-24.3.0    24.3.0.0.0        Upgrade complete (revision required for rollback)
3          Fri Feb 08 05:35:15 2024    deployed      ocnadd-25.1.200        25.1.200.0.0        Upgrade complete
  2. Rollback to Required Revision:
    helm rollback <ocandd-release-name> <REVISION> --namespace <ocndd-workergroup-namespace>

    Where,

    <REVISION> is the revision number obtained in the previous step to which the services need to be rolled back.

    For example: 
    helm rollback ocnadd 2 --namespace ocnadd-deploy
  3. Rollback Adapter Services to the Previous Version. Run the below commands to rollback all Adapter/Correlation services to the previous version:
    1. Perform helm upgrade using the management group charts folder:
      1. If no correlation configurations are present:
        helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
        For example:
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
      2. If correlation configurations are also present:
        helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true

        Note:

        If the correlation configuration was enabled for extended storage (from 24.2.0 onwards) then also use the flag "global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true" in the above command.
        For example:
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true

        Note:

        If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command.
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true
      3. If ingress adapter configurations are also present:
        helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
        For example:
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
  4. Check the status of the upgrade, and monitor the pods to come back to the RUNNING state.

Note:

  • If the rollback is unsuccessful, see troubleshooting steps outlined in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.
  • On rollback completion, if the export, correlation, aggregation, ingress-adapter and/or consumer adapter services may not receive the configuration notification in timely manner and start processing data, then see the section "Invalid Subscription entry in the subscription table" in the Oracle Communication Network Analytics Troubleshooting Guide.
  • If the rollback is successful and intraTLS is disabled, then create the certificates of all the OCNADD services. For required certificates, see rollbacked release Oracle Communication Network Analytics Security Guide.

Rollback Usecase 3: Upgrade Was Done to Move to the Centralized Deployment Mode and Got Failed

In this case, the upgrade was done in the target release itself to move to the Centralized deployment mode from the Non-Centralized mode. In such a case, there will be only one default worker group running in the same namespace along with the management group services. The procedure is similar to "Rollback Usecase1", except that the target and source release will remain the same. The steps in the "Rollback Usecase1" should be followed, and care must be taken while using the same release charts of the source release (25.1.200) for rollback.