4 Upgrading OCNADD

This section provides information on how to upgrade an existing OCNADD deployment. The section describes the upgrade order for source NFs, CNC Console, cnDBTier, and the upgrade impact on the source NFs.

Note:

The OCNADD can be upgraded from a source release to a target release using CLI procedures as outlined in the following sections. These steps can also be followed for any hotfix upgrade.

4.1 Supported Upgrade Paths

The following table lists the supported upgrade paths for OCNADD:

Table 4-1 Supported Upgrade Path

Source Release Target Release
23.3.x, 23.4.x 23.4.0.0.1

4.2 Preupgrade Tasks

Note:

Kafka PVC Storage Expansion: In case there is a need to increase the Kafka broker PVC size due to increased throughput support in the target release, then the expansion of the PVC must be done in the source release before initiating the upgrade. To increase the size, follow the section 'Expanding Kafka Storage' from the Oracle Communications Network Analytics Data Director User Guide.
Before starting the procedure to upgrade OCNADD, perform the following tasks:

Note:

  • While performing an upgrade, you must align the ocnadd-custom-values-23.4.0.0.1.yaml file of the target release as per the ocnadd/values.yaml file of the source release or the older release.
  • Do not enable any new feature during the upgrade.
  • The parent or sub-charts values.yaml must not be changed while performing the upgrade, unless it is explicitly specified in this document.

For information about enabling any new feature through Helm parameters, see Oracle Communications Network Analytics Data Director User Guide.

Preupgrade Tasks

  1. Fetch the images and charts of the target release as described in Installing OCNADD.
  2. Keep a backup of the ocnadd-custom-values.yaml file and the extracted chart folder "ocnadd" of the source release as a backup before starting the upgrade procedure.
  3. Create the secrets in the target release with the CA used in the source release. See, Create Secrets For Target Release to create secrets.
  4. Update the following helm chart files of the target release with the parameter values of the source release files:
    • update ocnadd/ocdd-db-resource.sql
    • update ocnadd/templates/ocnadd-secret-hook.yaml
    • update custom-templates/ocnadd-custom-values.x.x.x.yaml

    Note:

    • Ensure the parameters such as serviceAccount, Role, RoleBinding are retained from the source release. These are the deployment parameters and should not be modified as part of the upgrade.
      Ensure to set the upgrade parameter to true as shown below:
      serviceAccount:
    create: true
    name: <must be same as previous release>
    upgrade: true  ## ->> Update this to 'true', default is false 
clusterRole:
     create: true
     name: <must be same as previous release>
clusterRoleBinding:
     create: true
     name: <must be same as previous release>
  5. Update the pvcClaimSize in the target release ocnadd-custom-values-23.4.0.0.1.yaml file.
  6. Ensure to update the "offsetsTopicReplicationFactor" and "transactionStateLogReplicationFactor" in the target release ocnadd-custom-values-23.4.0.0.1.yaml file.
  7. Take the manual backup of the OCNADD before starting the upgrade procedures. See, Performing OCNADD Manual Backup for taking a manual backup of the OCNADD.
  8. Make sure to delete all the existing backup, restore, and ocnaddverify jobs before proceeding with the upgrade. Backup related jobs are "ocnaddbackup", "ocnaddrestore", "ocnaddverify" and "ocnaddmanualbackup".
    Use the following command to check for the currently running or completed backup jobs:
    kubectl get job -n <ocnadd-namespace>
    To delete jobs
    kubectl delete job -n <ocnadd-namespace> <job-name>

4.3 Upgrade Sequence

The upgrade sequence of the procedures to be followed is described in this section.

4.3.1 Upgrade Order for Source NFs

By design, Kafka clients (producers and consumers) and brokers are bidirectionally compatible:
  • Clients with a higher version of Kafka API can communicate with brokers of a lower version
  • Clients with a lower Kafka API version can communicate with brokers of a higher version
Kafka clients and brokers exchange API version information during a handshake.

Upgrade the source NFs and the OCNADD independently of each other, and no specific upgrade order is required. Upgrade to a new release succeeds if compatibility is maintained. See, "Compatibility Matrix" in the Oracle Communications Network Analytics Suite Release Notes.

The OCNADD upgrade requires less time than source NFs (SCP, SEPP, and NRF) upgrade. It is advisable first to upgrade the OCNADD and verify the traffic flow post upgrade for any significant errors or potential roadblocks in the upgrade. If the NFs are upgraded first, rollback of large numbers of source NFs workers and gateway PODs might be required.

Upgrade Order:

  1. Upgrade OCNADD
  2. Upgrade source NFs (NRF, SCP, and SEPP)

4.3.2 Upgrade Order for CNC Console and cnDBTier

The following upgrade order is recommended for CNC Console and cnDBTier:

  1. For CNC Console upgrade, see Oracle Communications Cloud Native Configuration Console Installation, Upgrade, and Fault Recovery Guide.
  2. For OCNADD upgrade, see Upgrade Sequence.
  3. For cnDBTier upgrade, see Oracle Communications Cloud Native Core cnDBTier Installation, Upgrade, and Fault Recovery Guide.

4.4 Upgrade Impact on Source NFs and Third Party Consumers

Listed below are the observed upgrade impacts on the source NFs and the Third Party Consumers:

  • In the case of a Kafka upgrade, the Kafka clients (NF producers and consumers) should not impact the Kafka API as the API compatibility is maintained between clients and brokers by Kafka.
  • The Kafka binary upgrade is a two step procedure in which the Helm upgrade is performed twice. One upgrade for the Kafka binary upgrade and the second (optional) upgrade if the InterBrokerProtocolVersion is changed. During this upgrade, the source NFs (Kafka producers) may face communication disruption with Kafka brokers multiple times as each broker is expected to restart two times. The producer clients should adopt appropriate reconnection and metadata refresh mechanisms. Suppose the producers run with the 'acks=0' and 'retries=0' configurations. There is no guarantee of reliable message delivery between producers and Kafka brokers during the upgrade, as broker instances restart multiple times.
  • The NF producers must maintain the list of servers in the bootstrap-server parameter instead of a single server in the bootstrap-server parameter.
  • Assume that the OCNADD upgrade is performed at 20% (approximately) of the supported traffic rate. The upgrade is performed using the rolling upgrade strategy. The traffic flow between the NFs and the OCNADD Kafka may remain degraded for a few minutes. The traffic rate is expected to become normal after the upgrade when the NFs producers reconnect to the Kafka brokers.
  • Assume that the OCNADD upgrade is performed at 20% (approximately) of the supported traffic rate. The upgrade is performed using the rolling upgrade strategy. The traffic flow between OCNADD consumer adapters and Third party consumers may remain degraded for a few minutes. The traffic rate is expected to be normal after the upgrade when the Kafka broker pods come up, all the consumer adapter pods have been upgraded, and consumer rebalancing is complete.
  • During the OCNADD upgrade, Kafka retention helps prevent data loss for Third party consumers. However, expect some data duplication towards Third party consumers due to multiple Kafka consumer rebalancing.
  • Plan the OCNADD upgrade during a maintenance window.

4.5 Upgrade Tasks

This section includes information about upgrading an existing OCNADD deployment.

When you upgrade an existing OCNADD deployment, the running set of containers and pods are replaced with the new set of containers and pods. However, If there is no change in the pod configuration, the running set of containers and pods are not replaced, unless there is a change in the service configuration of a microservice, the service endpoints will remain unchanged (NodePort and so on).

Note:

  • <Optional> Set a timeout interval of 15 minutes during the upgrade process.
  • Ensure no OCNADD pod is in a failed state.
  • Ensure that the defined in the Preupgrade Tasks are complete
  • During an upgrade that affects all brokers, anticipate approximately a minute of downtime for Kafka brokers. Alternatively, upgrade brokers individually to avoid this downtime if applicable.
  • Kafka upgrade along with PVC storage changes isn't supported.
  • The creation of Consumer Adapter pods or services occurs when Data feeds are created via OCNADD GUI. Ensure the upgrade of these pods is set to "false" (global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE is set to false by default). To perform an upgrade, refer to the "Updating Consumer Adapter Parameters" section in the Oracle Communications Network Analytics Data Director User Guide.
  • The creation of Correlation pods or services takes place when Correlation configurations are created from OCNADD GUI. Ensure the upgrade of these pods is set to "false" (global.env.admin.OCNADD_CORR_UPGRADE_ENABLE is set to false by default). To upgrade, refer to the "Updating Correlation Service Parameters" section in the Oracle Communications Network Analytics Data Director User Guide.
The following upgrades are supported:
  1. Upgrading Source Release to Target Release in Centralized Deployment Mode (Recommended)
  2. Upgrading Source Release to Target Release in Non-Centralized Deployment Mode
  3. Upgrading a 23.4.x Non-Centralized Deployment to Centralized Deployment Mode

Upgrading Source Release to Target Release in Centralized Deployment Mode (Recommended)

  1. Preupgrade procedure:
    1. Change the "repo_host" in target release custom-templates/copy_backup_pvc.bash file
      
cd ocnadd-package-23.4.0.0.1
 
Vi custom-templates/copy_backup_pvc.bash
 
repo_host="occne-repo-host:5000"      ##---> update occne-repo-host:5000 with the <customer image repository path>
    2. Run "copy_backup_pvc.bash" using the namespace of the source release deployment namespace as the argument.
      Run the following command:
      bash custom-templates/copy_backup_pvc.bash <source-release-namespace>
      For example: 
      bash custom-templates/copy_backup_pvc.bash ocnadd-deploy

      Note:

      The default storage class is "standard" if the storage class is different, provide its name as a second argument.
      For example:
      bash custom-templates/copy_backup_pvc.bash ocnadd-deploy <storageClass-name>
      Sample output (with namespace as the argument):
      
PVC backup-mysql-pvc-temp does not exist in namespace ocnadd-deploy.
PVC backup-mysql-pvc exists in namespace ocnadd-deploy.
Creating a new temporary PVC to copy into it.
persistentvolumeclaim/backup-mysql-pvc-temp created
Temporary PVC created successfully
Running job to Copy backup contents from backup-mysql-pvc to backup-mysql-pvc-temp
job.batch/ocnaddcopybackuppvc-orig-to-temp created
Copied successfully
    3. Change directory to the source release folder.

      For example: For 23.3.0 release the folder is ocnadd-package-23.3.0

      Run the following commands in the source release:
      sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-postrollback-version-db.yml
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-preupgrade-db.yaml
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-prerollback-db.yaml
    4. Perform Helm upgrade on source release to update the backup-pvc changes:
      helm upgrade <source-release-name> -f ocnadd-custom-values.yaml --namespace <source-release-namespace> <source-release-helm-chart>

      Where,

      • <source-release-name> is the release name of the source release deployment
      • ocnadd-custom-values.yaml is the custom value file of the source release
      • <source-release-namespace> is the OCNADD namespace of the source release
      • <source-release-helm-chart> is the helm chart folder of the source release
      For example (Example considering 23.3.x release)
      helm upgrade ocnadd -f ocnadd-custom-values.23.3.0.yaml --namespace ocnadd-deploy ocnadd
    5. Now disable the below services in the source release ocnadd-custom-values.yaml file and perform the helm upgrade. These services will be redeployed as the management group services in the target release.
      1. Edit the source release ocnadd-custom-values.yaml file
        For example: (considering 23.3.x release)
        vi ocnadd-custom-values.23.3.0.yaml
      2. Update the following parameters to false
        
global:
        ocnaddalarm:
            enabled: true           ##==> to false
        ocnaddconfiguration:
            enabled: true           ##==> to false
        ocnaddhealthmonitoring:
            enabled: true           ##==> to false
        ocnaddbackuprestore:
            enabled: true           ##==> to false
        ocnaddadminsvc:
            enabled: true           ##==> to false
        ocnadduirouter:
            enabled: true           ##==> to false
        ocnaddgui:
            enabled: true           ##==> to false
      3. Perform helm upgrade:
        helm upgrade <source-release-name> -f ocnadd-custom-values.yaml  --namespace <source-release-namespace> <source-release-helm-chart>
        For example:
        helm upgrade ocnadd -f ocnadd-custom-values.23.3.0.yaml --namespace ocnadd-deploy ocnadd
  2. Install the target release management group services.

    Update the ocnadd-custom-values.yaml of the target release with centralized deployment parameters for the management group.

    Follow the below steps:
    1. Create a copy of the charts and custom values of the target release for the management group and for the default worker group from the ocnadd-package-23.4.0.0.1 folder.
      The user can create copy of helm chart folder and custom-values file in the following suggested way:
      1. For Management Group:
        Create a copy of the following files from extracted folder:
        
     # cd  ocnadd-package-23.4.0.0.1
     # cp -rf ocnadd ocnadd_mgmt
     # cp custom-templates/ocnadd-custom-values-23.4.0.0.1.yaml ocnadd-custom-values-mgmt-group.yaml
      2. For default Worker Group:
        Create a copy of the following files from extracted folder:
        
     # cp -rf ocnadd ocnadd_default_wg
     # cp custom-templates/ocnadd-custom-values-23.4.0.0.1.yaml ocnadd-custom-values-default-wg.yaml
    2. Modify ocnadd-custom-values-mgmt-group.yaml file created above and update it as below:
      
    global.deployment.centralized: true
    global.deployment.management: true
    global.deployment.management_namespace:ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
    global.deployment.nonCenToCen_upgrade: false                    ##---> update it to 'true'
    3. Install using ocnadd_mgmt helm charts folder created for the management group:
      helm install <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart>
      For example:
      helm install ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt
    4. Delete the temp PVC created for the backup during "Preupgrade procedure".
      Run the following command:
      kubectl delete pvc -n ocnadd-deploy backup-mysql-pvc-temp
  3. Now upgrade the default Worker Group and Adapter services using the target release.
    1. Modify the ocnadd-custom-values-default-wg.yaml file created above and update it as below:
      
    global.deployment.centralized: true
    global.deployment.management: true                             ##---> update it to 'false'
    global.deployment.management_namespace:ocnadd-deploy           ##---> update it with source-release-namespace for example ocnadd-deploy
    global.deployment.nonCenToCen_upgrade: false                   ##---> update it to 'true'
    2. Perform helm upgrade using helm charts folder created for the default worker group:
      helm upgrade <source-release-name> -f ocnadd-custom-values-<default-worker-group>.yaml --namespace <source-release-namespace> <target-release-helm-chart>
      Where,
      • <source-release-name> is the release name of the source release deployment
      • ocnadd-custom-values-<default-worker-group>.yaml is the custom values file created for default-worker-group
      • <source-release-namespace> is the OCNADD namespace of the source release
      • <target-release-helm-chart> is the location of the Helm chart of the target release

        For example:

        helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg
    3. Once upgrade is successful perform the helm upgrade to update the adapters, using ocnadd_mgmt helm charts folder created for the management group:
      1. Edit the ocnadd-custom-values-mgmt-group.yaml and update the following parameter:
        global.env.admin.OCNADD_UPGRADE_WG_NS=ocnadd-deploy           ##---> update it with source-release-namespace for example ocnadd-deploy
      2. Perform helm upgrade using the management group charts folder:
        helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
        For example:
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
  4. Check the status of the upgrade, monitor the pods to come back to the RUNNING state, and wait for the traffic rate to be stabilized to the same rate as before the upgrade.
    Run the following command to check the upgrade status:
    helm history <release_name> --namespace <namespace-name>

    For example:

    For ocnadd-mgmt, use:
    helm history ocnadd-mgmt --namespace ocnadd-deplo
    For default worker-group, use:
    helm history ocnadd --namespace ocnadd-deploy

    Sample output:

    The description should be "upgrade complete".

  5. Verify if the upgrade is successful using the following steps:
    1. All the pods that have respawned after the upgrade, have the latest age ( in secs ).
    2. The Adapter pods also get respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds.

      In case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

  6. <Optional> Upgrade InterBrokerProtocolVersion in Kafka brokers.

    This step is required if the Kafka "InterBrokerProtocolVersion" is to be updated in the new release.

    1. Update the InterBrokerProtocolVersion in <ocnadd_default_wg_helm_chart>/charts/ocnaddkafka/values.yaml to the version originally mentioned in the target release charts, and set InterBrokerProtocolVersion to the desired version.
    2. Perform helm upgrade:
      helm upgrade <release-name> -f ocnadd-custom-values-<default-worker-group>.yaml --namespace <ocnadd-namespace> <target-release-helm-chart>
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg
  7. <Optional> To update the SNMP MIBs, follow the section "OCNADD MIB FILES" of the Oracle Communications Network Analytics Data Director User Guide.

Upgrading Source Release to Target Release in Non-Centralized Deployment Mode

  1. Perform the following steps on the source release to disable the 'ocnadduirouter' and 'ocnaddgui'. These services get freshly deployed with target release.

    Note:

    Skip this step for 23.4.0.0.1 patch upgrade.
    1. Go to the source release folder.

      For example: For 23.3.0 release the folder is ocnadd-package-23.3.0

    2. Update the ocnadd-custom-values.yaml file used to deploy the source release:
      
global.ocnadduirouter.enabled: true                            ##---> update it to 'false'
global.ocnaddgui.enabled: true                                 ##---> update it to 'false'
    3. Perform helm upgrade using helm charts folder :
      helm upgrade <source-release-name> -f ocnadd-custom-values_<source-release>.yaml --namespace <source-release-namespace> <source-release-helm-chart>
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values_23.3.0.yaml --namespace ocnadd-deploy ocnadd
  2. Upgrade Steps for OCNADD Microservices:
    1. Update the ocnadd-custom-values.yaml file of the target release to non-centralized deployment parameters as below:

      Note:

      This step expects that "Preupgrade Tasks" have already been performed.
      1. Change directory to the target release folder.

        For example: For 23.4.0 release the folder is ocnadd-package-23.4.0

      2. Update the ocnadd-custom-values-23.4.0.yaml file:
        global.deployment.centralized: true                            ##---> update it to 'false'
    global.deployment.management: true                             ##---> update it to 'false'
    global.deployment.management_namespace:ocnadd-deploy           ##---> update it with source-release-namespace for example ocnadd-deploy                  
    global.env.admin.OCNADD_UPGRADE_WG_NS=ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
      3. Perform helm upgrade to upgrade microservices:
        helm upgrade <source-release-name> -f ocnadd-custom-values_23.4.0.yaml --namespace <source-release-namespace> <target-release-helm-chart>
        Where,
        • <source-release-name> is the release name of the source release deployment
        • ocnadd-custom-values-23.4.0.yaml is the updated custom values file of the target release
        • <source-release-namespace> is the OCNADD namespace of the source release
        • <target-release-helm-chart> is the helm chart folder of the target release
        For example:
        helm upgrade ocnadd -f ocnadd-custom-values-23.4.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
      4. Perform helm upgrade to upgrade the Consumer Adapters:
        helm upgrade <source-release-name> -f ocnadd-custom-values_23.4.0.yaml --namespace <source-release-namespace> <target-release-helm-chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
        For example:
        helm upgrade ocnadd -f ocnadd-custom-values_23.4.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
  3. Check the status of the upgrade, monitor the pods to come back to the RUNNING state, and wait for the traffic rate to be stabilized to the same rate as before the upgrade. Run the command to check the upgrade status:
    helm history <release_name> --namespace <namespace-name>
    For example:
    helm history ocnadd --namespace ocnadd-deploy

    The description should be "upgrade complete".

  4. Verify if the upgrade is successful using the following steps:
    1. All the pods that have respawned after the upgrade, have the latest age (in secs).
    2. The Adapter pods also get respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds.

    In case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

  5. <Optional> Upgrade InterBrokerProtocolVersion in Kafka brokers. This step is required if the Kafka "InterBrokerProtocolVersion" is to be updated in the new release.
    1. Update the InterBrokerProtocolVersion in <helm-charts>/charts/ocnaddkafka/values.yaml to the version originally mentioned in the target release charts, and set InterBrokerProtocolVersion to the desired version.
    2. Run the upgrade command:
      helm upgrade <release_name> -f ocnadd-custom-values-23.4.0.0.1.yaml --namespace <ocnadd-namespace>  <helm_chart>
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-23.4.0.0.1.yaml --namespace ocnadd-deploy ocnadd
  6. <Optional> To update the SNMP MIBs, follow the section "OCNADD MIB FILES" of the Oracle Communications Network Analytics Data Director User Guide

Upgrading a 23.4.x Non-Centralized Deployment to Centralized Deployment Mode

Caution:

Changing mode from Centralized deployment mode to Non-Centralized deployment is not supported.

Follow this section when the upgrade to the 23.4.x release was done in the non-centralized mode and then the user wants to move to the Centralized deployment mode.

  1. Preupgrade procedure
    1. Keep a backup of the ocnadd-custom-values.yaml file and the extracted chart folder "ocnadd" of the current release as a backup before starting the upgrade procedure. Run the following commands:
      
cp custom-templates/ocnadd-custom-values-23.4.0.0.1.yaml ocnadd-custom-values-23.4.0.0.1-non-centralized.yaml
cp -rf ocnadd ocnadd-non-centralized
    2. Change the "repo_host" in 23.4.x release custom-templates/copy_backup_pvc.bash file:
      
cd ocnadd-package-23.4.0.0.1
 
Vi custom-templates/copy_backup_pvc.bash
 
repo_host="occne-repo-host:5000"      ##---> update occne-repo-host:5000 with the <customer image repository path>
    3. Run the "copy_backup_pvc.bash" using the deployment namespace as the argument:
      1. Run the following command: 
        bash custom-templates/copy_backup_pvc.bash <ocnadd-namespace>
        For example:
        bash custom-templates/copy_backup_pvc.bash ocnadd-deploy

        Note:

        The default storage class is "standard" if the storage class is different, provide its name as a second argument.
        For example:
        bash custom-templates/copy_backup_pvc.bash ocnadd-deploy <storageClass-name>
        Sample Output:
        
Output: (with namespace as the argument)
PVC backup-mysql-pvc-temp does not exist in namespace ocnadd-deploy.
PVC backup-mysql-pvc exists in namespace ocnadd-deploy.
Creating a new temporary PVC to copy into it.
persistentvolumeclaim/backup-mysql-pvc-temp created
Temporary PVC created successfully
Running job to Copy backup contents from backup-mysql-pvc to backup-mysql-pvc-temp
job.batch/ocnaddcopybackuppvc-orig-to-temp created
Copied successfully
    4. Run the following commands to change directory to the release folder:

      For example: For 23.4.0.0.1 release the folder is ocnadd-package-23.4.0.0.1 

       
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-postrollback-version-db.yml
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-preupgrade-db.yaml
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-prerollback-db.yaml
    5. Perform Helm upgrade to update the backup-pvc changes:
      helm upgrade <release-name> -f ocnadd-custom-values-23.4.0.0.1.yaml --namespace <ocnadd-namespace> <helm-chart>
      Where,
      • <release-name> is the release name of the deployment
      • ocnadd-custom-values-23.4.0.0.1.yaml is the custom values file of the current
      • ocnadd release <ocnadd-namespace> is the OCNADD namespace of the current release
      • <helm-chart> is the helm chart folder of the current OCNADD deployment
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values.23.4.0.yaml --namespace ocnadd-deploy ocnadd
    6. Now disable the below services in the ocnadd-custom-values.yaml file and perform the helm upgrade. These services will be redeployed as the management group services in the same release.
      1. Edit the ocnadd-custom-values.yaml file:
        vi ocnadd-custom-values.23.4.0.yaml
      2. Update the following parameters to false
        
 global:
        ocnaddalarm:
            enabled: true           ##==> to false
        ocnaddconfiguration:
            enabled: true           ##==> to false
        ocnaddhealthmonitoring:
            enabled: true           ##==> to false
        ocnaddbackuprestore:
            enabled: true           ##==> to false
        ocnaddadminsvc:
            enabled: true           ##==> to false
        ocnadduirouter:
            enabled: true           ##==> to false
        ocnaddgui:
            enabled: true           ##==> to false
      3. Perform helm upgrade:
        helm upgrade <release-name> -f ocnadd-custom-values-23.4.0.0.1.yaml  --namespace <ocnadd-namespace> <helm-chart>
        For example:
        helm upgrade ocnadd -f ocnadd-custom-values-23.4.0.0.1.yaml --namespace ocnadd-deploy ocnadd
  2. Install the management group services

    Update the ocnadd-custom-values.yaml with centralized deployment parameters for the management group.

    Follow the below steps:

    1. Create a copy of the charts and custom values of the current release for the management group and for the default worker group from the backup created during "Preupgrade procedure".

      The user can create copy of helm chart folder and custom-values file in the following suggested way:

      1. For Management Group:
        Create a copy of the following files from extracted folder:
        
     # cd  ocnadd-package-23.4.0.0.1
     # cp -rf ocnadd-non-centralized ocnadd_mgmt
     # cp ocnadd-custom-values-23.4.0.0.1.yaml ocnadd-custom-values-mgmt-group.yaml
      2. For default Worker Group:
        Create a copy of the following files from extracted folder:
        
     # cp -rf ocnadd-non-centralized ocnadd_default_wg
     # cp ocnadd-custom-values-23.4.0.0.1.yaml ocnadd-custom-values-default-wg.yaml
    2. Modify ocnadd-custom-values-mgmt-group.yaml file created above and update it as below:
      
    global.deployment.centralized: false                            ##---> update it to 'true'
    global.deployment.management: false                             ##---> update it to 'true'
    global.deployment.management_namespace:ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
    global.deployment.nonCenToCen_upgrade: false                    ##---> update it to 'true'
    3. Install using ocnadd_mgmt helm charts folder created for the management group:
      helm install <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <mgmt_helm_chart>
      Where,
      • <management-release-name> is the release name of the management release
      • ocnadd-custom-values-<mgmt-group>.yaml is the custom values file created for management group
      • <source-release-namespace> is the OCNADD namespace of the source release
      • <mgmt_helm_chart> is the helm chart folder of the management-group
      For example:
      helm install ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt
    4. Delete the temp PVC created for the backup during "Preupgrade procedure":
      kubectl delete pvc -n ocnadd-deploy backup-mysql-pvc-temp
  3. Now upgrade the default Worker Group and Adapter services of the current release:
    1. Modify the ocnadd-custom-values-default-wg.yaml file created above and update it as below:
      
    global.deployment.centralized: false                           ##---> update it to 'true'
    global.deployment.management: false                           
    global.deployment.management_namespace:ocnadd-deploy           ##---> update it with source-release-namespace for example ocnadd-deploy
    global.deployment.nonCenToCen_upgrade: false                   ##---> update it to 'true'
    2. Perform helm upgrade using helm charts folder created for the default worker group:
      helm upgrade <release-name> -f ocnadd-custom-values-<default-worker-group>.yaml --namespace <ocnadd-namespace> <worker-group-helm-chart>
      Where,
      • <release-name> is the release name of the deployment
      • <ocnadd-custom-values-<default-worker-group>.yaml is the custom values file created fro worker group
      • <ocnadd-namespace> is the OCNADD namespace of the current release
      • <worker-group-helm-chart> is the helm chart folder of the worker-group
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg
    3. Once upgrade is successful perform the helm upgrade to update the adapters, correlation services using ocnadd_mgmt helm charts folder created for the management group:
      1. Edit the ocnadd-custom-values-mgmt-group.yaml and update the following parameter:
        global.env.admin.OCNADD_UPGRADE_WG_NS=ocnadd-deploy           ##---> update it with the ocnadd-namespace or the default worker group namespace for example ocnadd-deploy
      2. Perform helm upgrade using the management group charts folder:
        1. If no Correlation configurations are created:
          helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
          For example:
          helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
        2. If Correlation configurations are created:
           helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
          For example:
          helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
  4. Check the status of the upgrade, monitor the pods to come back to the RUNNING state, and wait for the traffic rate to be stabilized to the same rate as before the upgrade.
    Run the command to check the upgrade status:
    helm history <release_name> --namespace <namespace-name>

    For example:

    For ocnadd-mgmt use:
    helm history ocnadd-mgmt --namespace ocnadd-deploy
    For default worker-group use:
    helm history ocnadd --namespace ocnadd-deploy

    The description should be "upgrade complete".

  5. Verify if the upgrade is successful using the following steps:
    1. All the pods that have respawned after the upgrade, have the latest age ( in secs )
    2. The Adapter pods also get respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds.

      In case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

  6. <Optional> Upgrade InterBrokerProtocolVersion in Kafka brokers. This step is required if the Kafka "InterBrokerProtocolVersion" is to be updated in the new release.
    • Update the InterBrokerProtocolVersion in <ocnadd_default_wg_helm_chart>/charts/ocnaddkafka/values.yaml to the version originally mentioned in the target release charts, and set InterBrokerProtocolVersion to the desired version.
    • Perform helm upgrade:
      helm upgrade <release-name> -f ocnadd-custom-values-<default-worker-group>.yaml --namespace <ocnadd-namespace> <target-release-helm-chart>
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg
  7. <Optional> To update the SNMP MIBs, follow the section "OCNADD MIB FILES" of the Oracle Communications Network Analytics Data Director User Guide.

4.5.1 Hotfix Upgrade

For any patch upgrade follow the Preupgrade Tasks section along with the ReadMe.txt file provided with the patch.

4.5.2 Create Secrets For Target Release

Follow the steps to generate the secrets for the target release.

  1. Navigate directory to the target release "ocnadd-package-23.4.0" folder, and then change directory to the <ssl_certs>/default_values folder.
  2. Edit the 'values' files and update global parameters, CN, and SAN (DNS/IP entries) for each service based on the requirement as follows:
  3. Change the default domain (occne-ocdd) name and default namespace (ocnadd-deploy) in every service_values file with your corresponding cluster domain which will be used to create a certificate as shown below:

    Example:

    
For namespace = <ocnadd-namespace>
                clusterDomain = cluster.local.com
 
    sed -i "s/ocnadd-deploy/<ocnadd-namespace>/g" values
    sed -i "s/occne-ocdd/cluster.local.com/g" values

    Note:

    Edit the corresponding service_values file for global parameters and RootCA common name, and add service blocks of all the services for which the certificate needs to be generated.

    ssl_certs/default_values/values file

    
Global Params:
 
[global]
countryName=<country>
stateOrProvinceName=<state>
localityName=<city>
organizationName=<org_name>
organizationalUnitName=<org_bu_name>
defaultDays=<days to expiry>
  
 
Root CA common name (e.g. rootca common_name=*.svc.domainName)
##root_ca
 
commonName=*.svc.domainName
 
 
Service common name for client and server and SAN(DNS/IP entries). (Make sure to follow exact same format and provide an empty line at the end of each service block)
  
[service-name-1]
client.commonName=client.cn.name.svc1
server.commonName=server.cn.name.svc1
IP.1=127.0.0.1
DNS.1=localhost
  
[service-name-2]
client.commonName=client.cn.name.svc2
server.commonName=server.cn.name.svc2
IP.1= 10.20.30.40
DNS.1 = *.svc2.namespace.svc.domainName
.
.
.
##end
  4. Run the generate_certs.sh script with the following command:
    ./generate_certs.sh -cacert <path to>/cacert.pem -cakey <path to>/cakey.pem

    Where,

    <path to> is the folder path where the CACert and CAKey are present.

  5. Select the mode of deployment:
    
"1" for non-centralized
"2" for upgrade from non-centralized to centralized
"3" for centralized
Select the mode of deployment (1/2/3): 2

    Note:

    If an upgrade is being performed to a Non-Centralized deployment mode, select option 1.
  6. Select the namespace where you would like to generate the certificates.

    Enter Kubernetes namespace: <your_working_namespace>

  7. Enter the passphrase for the CA key when prompted.

    Enter passphrase for CA Key file: <passphrase>

  8. Select “y” when prompted to create CSR for each service.

    Create Certificate Signing Request (CSR) for each service?

  9. Select “y” when prompted to sign CSR for each service with CA Key.

    Would you like to sign CSR for each service with CA key? Y

  10. Run the following command to check if the secrets are created in the specified namespace:
    kubectl get secret -n <namespace>
  11. Run the following command to describe any secret created by the script:
    kubectl describe secret <secret-name> -n <namespace>