4 Upgrading OCNADD

This section provides information on how to upgrade an existing OCNADD deployment. The section describes the upgrade order for source NFs, CNC Console, cnDBTier, and the upgrade impact on the source NFs.

Note:

  • The OCNADD can be upgraded from a source release to a target release using CLI procedures as outlined in the following sections. These steps can also be followed for any hotfix upgrade.

4.1 Supported Upgrade Paths

The following table lists the supported upgrade paths for OCNADD:

Table 4-1 Supported Upgrade Path

Source Release Target Release
24.1.0.0.x 24.3.0
24.2.0.0.x 24.3.0
24.3.0 24.3.0.0.1

4.2 Preupgrade Tasks

Note:

  • (Optional) Configure Helm parameters for OCCM based Certificate Management, refer to the section Helm Parameter Configuration for OCCM for configuration
  • Kafka PVC Storage Expansion: In case there is a need to increase the Kafka broker PVC size due to increased throughput support in the target release, then the expansion of the PVC must be done in the source release before initiating the upgrade. To increase the size, follow the section 'Expanding Kafka Storage' from the Oracle Communications Network Analytics Data Director User Guide.
Before starting the procedure to upgrade OCNADD, perform the following tasks:

Note:

  • While performing an upgrade, you must align the ocnadd-custom-values-24.3.0.yaml file of the target release as per the ocnadd/values.yaml file of the source release or the older release. Do not enable any new feature during the upgrade. The parent or sub-charts values.yaml must not be changed while performing the upgrade, unless it is explicitly specified in this document. For information about enabling any new feature through Helm parameters, see Oracle Communications Network Analytics Data Director User Guide.
  • The current resource profile in helm chart is for 135K MPS, if throughput is increased and it is required to use 270K MPS, update resources as per 270K MPS resource profile mentioned in Oracle Communications Network Analytics Data Director Benchmarking Guide.

Preupgrade Tasks

  1. Fetch the images and charts of the target release as described in Installing OCNADD.
  2. Keep a backup of the ocnadd-custom-values.yaml file and the extracted chart folder "ocnadd" of the source release as a backup before starting the upgrade procedure.
  3. Create the certificates and secrets only for the services that are newly added in the target release with the CA used in the source release. See, Create Secrets For Target Release to create secrets. If not skip this step.
  4. If upgrade to Centralized Mode is targeted, then ensure kubectl-hns is available. Else, see Kubectl HNS Installation for installation.
  5. Update the following helm chart files of the target release with the parameter values of the source release files:
    • update ocnadd/ocdd-db-resource.sql
    • update ocnadd/templates/ocnadd-secret-hook.yaml
    • update custom-templates/ocnadd-custom-values.x.x.x.yaml

    Note:

    • Ensure the parameters such as serviceAccount, Role, RoleBinding are retained from the source release. These are the deployment parameters and should not be modified as part of the upgrade.
      Ensure to set the upgrade parameter to true as shown below:
      serviceAccount:
    create: true
    name: <must be same as previous release>
    upgrade: true  ## ->> Update this to 'true', default is false 
clusterRole:
     create: true
     name: <must be same as previous release>
clusterRoleBinding:
     create: true
     name: <must be same as previous release>
  6. Update the pvcClaimSize in the target release ocnadd-custom-values-24.3.0.yaml file.
  7. Ensure to update the "offsetsTopicReplicationFactor" and "transactionStateLogReplicationFactor" in the target release ocnadd-custom-values-24.3.0.yaml file.
  8. Ensure that the IP Family configuration in the target release matches with the IP configuration in the source release during the upgrade process. Changing the IP Family from IPv4 to IPv6 or vice-versa while performing upgrade is not supported.
  9. Alter the mysql users, <admin_user>, ocddAppUsr and ocddPrivilegedUsr.
    Check user in mysql console:-
 
kubectl -n occne-cndbtier exec -it ndbmysqld-0 -- bash
 
mysql -h 127.0.0.1 -uroot -p $ Enter password:
 
Run the below command to check if auth plugin of mysql user:-
 
SELECT user,plugin FROM mysql.user;
 
Run below commands if and only if auth plugin is 'mysql_native_password' for <admin_user>, ocddPrivilegedUsr and ocddAppUsr
 
For Admin User:-
ALTER USER '<admin_user>'@'%' IDENTIFIED WITH caching_sha2_password BY '<admin_user_password>';
where, <admin_user> is the user created in Section "Creating an Admin User in the Database"
Example:
ALTER USER 'ocdd'@'%' IDENTIFIED WITH caching_sha2_password BY 'ocdd';
 
For ocddPrivilegedUsr:-
ALTER USER 'ocddPrivilegedUsr'@'%' IDENTIFIED WITH caching_sha2_password BY '<ocddPrivilegedUsr_password>';
Example:
ALTER USER 'ocddPrivilegedUsr'@'%' IDENTIFIED WITH caching_sha2_password BY 'ocddPrivilegedPasswd';
 
For ocddAppUsr:-
ALTER USER 'ocddAppUsr'@'%' IDENTIFIED WITH caching_sha2_password BY '<ocddAppUsr_password>';
Example:
ALTER USER 'ocddAppUsr'@'%' IDENTIFIED WITH caching_sha2_password BY 'ocddAppPasswd';
  10. Take the manual backup of the OCNADD before starting the upgrade procedures. See, Performing OCNADD Manual Backup for taking a manual backup of the OCNADD.
  11. Make sure to delete all the existing backup, restore, and ocnaddverify jobs before proceeding with the upgrade. Backup related jobs are "ocnaddbackup", "ocnaddrestore", "ocnaddverify" and "ocnaddmanualbackup".
    Use the following command to check for the currently running or completed backup jobs:
    kubectl get job -n <ocnadd-namespace>
    To delete jobs
    kubectl delete job -n <ocnadd-namespace> <job-name>
  12. Before upgrading to 24.3.0, certificates need to be created or updated, followed by the deletion of the Kafka-broker StatefulSet (STS) in orphan mode from the 24.1.0/24.2.0 deployment.
    1. If OCNADD SSL Certificates Procedure was Used:Follow the procedure “Update Certificate of the Existing Services” in the Oracle Communication Network Analytics Data Director User Guide to edit the default_values/renew_cert_files and comment out all the DD services except the Kafka-broker.
    2. If OCCM Was Used to Manage the Certificate:Follow the step 1 of the “Post Upgrade Task” instructions.
    3. Run the following command in the 24.1.0/24.2.0 OCNADD deployment to delete Kafka-Broker STS:
      kubectl delete statefulset --cascade=orphan kafka-broker -n <namespace>

      Note:

      If NFs are deployed in the same cluster and use the FQDN .kafka-broker.<namespace>.svc.<domain> as the Kafka bootstrap to connect to OCNADD, this FQDN must be updated to *.kafka-broker-headless.<namespace>.svc.<domain> in NFs after upgrading to OCNADD 24.3.0.
  13. If export configurations are present in the source release with exportType=PCAP, follow the steps below. Otherwise, this step can be skipped.
    1. Login to MySQL Database Instance with DB user of Data Director
    2. Run the following commands: 
      1. USE <CONFIGURATION-SCHEMA-NAME>;

        For example:

        USE configuration_schema_dd;
      2. SELECT S.CONFIGURATION_NAME, S.REMAINING_EXPORT_TIME 
FROM DATA_EXPORT_STATUS S 
JOIN EXPORT_CONFIGURATION C 
ON C.CONFIGURATION_NAME = S.CONFIGURATION_NAME 
WHERE EXPORT_TYPE = 'PCAP';

        For example:

        SELECT S.CONFIGURATION_NAME, S.REMAINING_EXPORT_TIME 
FROM DATA_EXPORT_STATUS S 
JOIN EXPORT_CONFIGURATION C 
ON C.CONFIGURATION_NAME = S.CONFIGURATION_NAME 
AND C.EXPORT_TYPE = 'PCAP';

+--------------------+----------------------------+
| CONFIGURATION_NAME | REMAINING_EXPORT_TIME       |
+--------------------+----------------------------+
| pcap_export_test   | +56572-12-29T09:26:48.000Z  |
| after-upgrade-pcap | +56573-01-30T05:01:30.000Z  |
+--------------------+----------------------------+
    3. Retrieve the Corresponding PCAP File: Locate the PCAP file present on the SFTP server for the corresponding configuration. Open the file and go to the last frame of the file. Use the timestamp (for example, highlighted below) to update the REMAINING_EXPORT_TIME column.
      example

      For example:
      
Timestamp format in the file: 2024-07-31 17:10:00.408283
Timestamp format to use for updating the column: 2024-07-31T17:10:00.408Z

      Note:

      If the export configuration exists for exportType=PCAP and the PCAP file is not present on the SFTP server, use the export configuration’s "StartTime+1 second" from the UI to update the REMAINING_EXPORT_TIME column. In this case, the export will start from "StartTime+1 second."
    4. Update REMAINING_EXPORT_TIME: Execute the following command on the MySQL terminal to update the REMAINING_EXPORT_TIME in the DATA_EXPORT_STATUS table and commit the changes:
      
UPDATE DATA_EXPORT_STATUS 
SET REMAINING_EXPORT_TIME = '<UTC timestamp from the above step>' 
WHERE CONFIGURATION_NAME = '<pcap export configuration name>';

COMMIT;
      For example:
      
UPDATE DATA_EXPORT_STATUS 
SET REMAINING_EXPORT_TIME = '2024-07-31T17:10:00.408Z' 
WHERE CONFIGURATION_NAME = 'pcap_export_test';

COMMIT;
    5. Repeat steps 'c' and 'd' to update REMAINING_EXPORT_TIME for all export configurations with exportType=PCAP.

      Note:

      • Post upgrade, if the remaining export time gets updated again(during upgrade) with the incorrect timestamp, then repeat the above steps again after the upgrade.
      • Restart the ocnaddconfiguration and ocnaddexport services after the timestamp update.

4.3 Upgrade Sequence

The upgrade sequence of the procedures to be followed is described in this section.

4.3.1 Upgrade Order for Source NFs

By design, Kafka clients (producers and consumers) and brokers are bidirectionally compatible:
  • Clients with a higher version of Kafka API can communicate with brokers of a lower version
  • Clients with a lower Kafka API version can communicate with brokers of a higher version
Kafka clients and brokers exchange API version information during a handshake.

Upgrade the source NFs and the OCNADD independently of each other, and no specific upgrade order is required. Upgrade to a new release succeeds if compatibility is maintained. See, "Compatibility Matrix" in the Oracle Communications Network Analytics Suite Release Notes.

The OCNADD upgrade requires less time than source NFs (SCP, SEPP, and NRF) upgrade. It is advisable first to upgrade the OCNADD and verify the traffic flow post upgrade for any significant errors or potential roadblocks in the upgrade. If the NFs are upgraded first, rollback of large numbers of source NFs workers and gateway PODs might be required.

Upgrade Order:

  1. Upgrade OCNADD
  2. Upgrade source NFs (NRF, SCP, and SEPP)

4.3.2 Upgrade Order for CNC Console and cnDBTier

The following upgrade order is recommended for CNC Console and cnDBTier:

  1. For CNC Console upgrade, see Oracle Communications Cloud Native Configuration Console Installation, Upgrade, and Fault Recovery Guide.
  2. For OCNADD upgrade, see Upgrade Sequence.
  3. For cnDBTier upgrade, see Oracle Communications Cloud Native Core cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Note:

  • For more information on customizing database parameters required for the OCNADD, see cnDBTier Customization Parameters section.
  • To perform cnDBTier upgrade, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

4.4 Upgrade Impact on Source NFs and Third Party Consumers

Listed below are the observed upgrade impacts on the source NFs and the Third Party Consumers:

  • In the case of a Kafka upgrade, the Kafka clients (NF producers and consumers) should not impact the Kafka API as the API compatibility is maintained between clients and brokers by Kafka.
  • The Kafka binary upgrade is a two step procedure in which the Helm upgrade is performed twice. One upgrade for the Kafka binary upgrade and the second (optional) upgrade if the InterBrokerProtocolVersion is changed. During this upgrade, the source NFs (Kafka producers) may face communication disruption with Kafka brokers multiple times as each broker is expected to restart two times. The producer clients should adopt appropriate reconnection and metadata refresh mechanisms. Suppose the producers run with the 'acks=0' and 'retries=0' configurations. There is no guarantee of reliable message delivery between producers and Kafka brokers during the upgrade, as broker instances restart multiple times.
  • The NF producers must maintain the list of servers in the bootstrap-server parameter instead of a single server in the bootstrap-server parameter.
  • Assume that the OCNADD upgrade is performed at 20% (approximately) of the supported traffic rate. The upgrade is performed using the rolling upgrade strategy. The traffic flow between the NFs and the OCNADD Kafka may remain degraded for a few minutes. The traffic rate is expected to become normal after the upgrade when the NFs producers reconnect to the Kafka brokers.
  • Assume that the OCNADD upgrade is performed at 20% (approximately) of the supported traffic rate. The upgrade is performed using the rolling upgrade strategy. The traffic flow between OCNADD consumer adapters and Third party consumers may remain degraded for a few minutes. The traffic rate is expected to be normal after the upgrade when the Kafka broker pods come up, all the consumer adapter pods have been upgraded, and consumer rebalancing is complete.
  • During the OCNADD upgrade, Kafka retention helps prevent data loss for Third party consumers. However, expect some data duplication towards Third party consumers due to multiple Kafka consumer rebalancing.
  • Plan the OCNADD upgrade during a maintenance window.

4.5 Upgrade Tasks

This section includes information about upgrading an existing OCNADD deployment.

When you upgrade an existing OCNADD deployment, the running set of containers and pods are replaced with the new set of containers and pods. However, If there is no change in the pod configuration, the running set of containers and pods are not replaced, unless there is a change in the service configuration of a microservice, the service endpoints will remain unchanged (NodePort and so on).

Note:

  • <Optional> Set a timeout interval of 15 minutes during the upgrade process.
  • Ensure no OCNADD pod is in a failed state.
  • Ensure that the defined in the Preupgrade Tasks are complete
  • During an upgrade that affects all brokers, anticipate approximately a minute of downtime for Kafka brokers. Alternatively, upgrade brokers individually to avoid this downtime if applicable.
  • Kafka upgrade along with PVC storage changes isn't supported.
  • The creation of Consumer Adapter pods or services occurs when Data feeds are created via OCNADD GUI. Ensure the upgrade of these pods is set to "false" (global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE is set to false by default). To perform an upgrade, refer to the "Updating Consumer Adapter Parameters" section in the Oracle Communications Network Analytics Data Director User Guide.
  • The creation of Correlation pods or services takes place when Correlation configurations are created from OCNADD GUI. Ensure the upgrade of these pods is set to "false" (global.env.admin.OCNADD_CORR_UPGRADE_ENABLE is set to false by default). To upgrade, refer to the "Updating Correlation Service Parameters" section in the Oracle Communications Network Analytics Data Director User Guide.
The following upgrades are supported:
  1. Upgrading Source Release(Non-Centralized) to Target Release in Centralized Deployment Mode (Recommended)
  2. Upgrading Source Release(Centralized) to Target Release in Centralized Deployment Mode
  3. Upgrading Source Release (Non-Centralized) to Target Release in Non-Centralized Deployment Mode
  4. Upgrading a 24.3.x Non-Centralized Deployment to Centralized Deployment Mode

Upgrading Source Release(Non-Centralized) to Target Release in Centralized Deployment Mode (Recommended)

Note:

This scenario is applicable for OCNADD release 24.1.x or 24.2.x Non-Centralized Deployment Mode upgrade to 24.3.x Centralized Deployment Mode.
  1. Preupgrade procedure:
    1. Change the "repo_host" in target release custom-templates/copy_backup_pvc.bash file
      
cd ocnadd-package-24.3.0
 
Vi custom-templates/copy_backup_pvc.bash
 
repo_host="occne-repo-host:5000"      ##---> update occne-repo-host:5000 with the <customer image repository path>
    2. Run "copy_backup_pvc.bash" using the namespace, storageClass and serviceAccount of the source release deployment as the argument.
      Run the following command:
      bash custom-templates/copy_backup_pvc.bash <source-release-namespace> <storageClass-name> <service-account>

      Where,

      • source-release-namespace is the OCNADD namespace of the source release
      • storageClass-name is the Storage Class of OCNADD deployment (default used is standard)
      • service-account is the Service Account name set in global.cluster.serviceAccount.name in ocnadd-custom-values-24.3.0.yaml
      For example: 
      bash custom-templates/copy_backup_pvc.bash ocnadd-deploy standard ocnadd
      Sample output (with namespace, storageClass and serviceAccount as the argument):
      PVC backup-mysql-pvc-temp does not exist in namespace ocnadd-deploy.
PVC backup-mysql-pvc exists in namespace ocnadd-deploy.
Creating a new temporary PVC to copy into it.
persistentvolumeclaim/backup-mysql-pvc-temp created Temporary PVC created successfully
Running job to Copy backup contents from backup-mysql-pvc to backup-mysql-pvc-temp
job.batch/ocnaddcopybackuppvc-orig-to-temp created
Copied successfully
    3. Change directory to the source release folder.

      For example: For 24.2.0 release the folder is ocnadd-package-24.2.0

      Run the following commands in the source release:
      
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-postrollback-version-db.yml
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-preupgrade-db.yaml
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-prerollback-db.yaml
    4. Perform Helm upgrade on source release to update the backup-pvc changes:
      helm upgrade <source-release-name> -f ocnadd-custom-values.yaml --namespace <source-release-namespace> <source-release-helm-chart>

      Where,

      • <source-release-name> is the release name of the source release deployment
      • ocnadd-custom-values.yaml is the custom value file of the source release
      • <source-release-namespace> is the OCNADD namespace of the source release
      • <source-release-helm-chart> is the helm chart folder of the source release
      For example (Example considering 24.2.x release)
      helm upgrade ocnadd -f ocnadd-custom-values.24.2.0.yaml --namespace ocnadd-deploy ocnadd
    5. Now disable the below services in the source release ocnadd-custom-values.yaml file and perform the helm upgrade. These services will be redeployed as the management group services in the target release.
      1. Edit the source release ocnadd-custom-values.yaml file
        For example: (considering 24.2.x release)
        vi ocnadd-custom-values.24.2.0.yaml
      2. Update the following parameters to false
        
global:
     ocnaddalarm:
          enabled: true           ##==> to false
     ocnaddconfiguration:
          enabled: true           ##==> to false
     ocnaddhealthmonitoring:
          enabled: true           ##==> to false
     ocnaddbackuprestore:
          enabled: true           ##==> to false
     ocnaddadminsvc:
          enabled: true           ##==> to false
     ocnadduirouter:
          enabled: true           ##==> to false
     ocnaddgui:
          enabled: true           ##==> to false
      3. Perform helm upgrade:
        helm upgrade <source-release-name> -f ocnadd-custom-values.yaml  --namespace <source-release-namespace> <source-release-helm-chart>
        For example:
        helm upgrade ocnadd -f ocnadd-custom-values.24.2.0.yaml --namespace ocnadd-deploy ocnadd
  2. Install the target release management group services, considering target release to 24.3.0

    Update the ocnadd-custom-values.yaml of the target release with centralized deployment parameters for the management group.

    Follow the below steps:
    1. Create a copy of the charts and custom values of the target release for the management group and for the default worker group from the ocnadd-package-24.3.0 folder.
      The user can create copy of helm chart folder and custom-values file in the following suggested way:
      1. For Management Group:
        Create a copy of the following files from extracted folder:
        
     # cd  ocnadd-package-24.3.0
     # cp -rf ocnadd ocnadd_mgmt
     # cp custom-templates/ocnadd-custom-values-24.3.0.yaml ocnadd-custom-values-mgmt-group.yaml
      2. For default Worker Group:
        Create a copy of the following files from extracted folder:
        
     # cp -rf ocnadd ocnadd_default_wg
     # cp custom-templates/ocnadd-custom-values-24.3.0.yaml ocnadd-custom-values-default-wg.yaml
    2. Modify ocnadd-custom-values-mgmt-group.yaml file created above and update it as below:
      
    global.deployment.centralized: true
    global.deployment.management: true
    global.deployment.management_namespace:ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
    global.deployment.nonCenToCen_upgrade: false                    ##---> update it to 'true'
    3. Install using ocnadd_mgmt helm charts folder created for the management group:
      helm install <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart>
      For example:
      helm install ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt
    4. Delete the temp PVC created for the backup during "Preupgrade procedure".
      Run the following command:
      kubectl delete pvc -n ocnadd-deploy backup-mysql-pvc-temp
  3. Now upgrade the default Worker Group and Adapter services using the target release.
    1. Modify the ocnadd-custom-values-default-wg.yaml file created above and update it as below:
      
    global.deployment.centralized: true
    global.deployment.management: true                             ##---> update it to 'false'
    global.deployment.management_namespace:ocnadd-deploy           ##---> update it with source-release-namespace for example ocnadd-deploy
    global.deployment.nonCenToCen_upgrade: false                   ##---> update it to 'true'
    2. Perform helm upgrade using helm charts folder created for the default worker group:
      helm upgrade <source-release-name> -f ocnadd-custom-values-<default-worker-group>.yaml --namespace <source-release-namespace> <target-release-helm-chart>
      Where,
      • <source-release-name> is the release name of the source release deployment
      • ocnadd-custom-values-<default-worker-group>.yaml is the custom values file created for default-worker-group
      • <source-release-namespace> is the OCNADD namespace of the source release
      • <target-release-helm-chart> is the location of the Helm chart of the target release

        For example:

        helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg

        Note:

        If OCCM was used to manage the certificates, follow the step 1 of the "Post Upgrade Task" before continuing with the adapter upgrades, else continue with the next step.
    3. Once upgrade is successful perform the helm upgrade to update the adapters, using ocnadd_mgmt helm charts folder created for the management group:
      1. Edit the ocnadd-custom-values-mgmt-group.yaml and update the following parameter:
        global.env.admin.OCNADD_UPGRADE_WG_NS=ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
      2. If only adapter configurations are present, perform helm upgrade using the management group charts folder:
        helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
        For example:
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
      3. If Correlation configurations are also present:
        helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true

        Note:

        If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command.
        For example:
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true

        Note:

        If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command:
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true
      4. If ingress adapter configuration are also present:
        helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
        For example:
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
  4. Check the status of the upgrade, monitor the pods to come back to the RUNNING state, and wait for the traffic rate to be stabilized to the same rate as before the upgrade.
    Run the following command to check the upgrade status:
    helm history <release_name> --namespace <namespace-name>

    For example:

    For ocnadd-mgmt, use:
    helm history ocnadd-mgmt --namespace ocnadd-deplo
    For default worker-group, use:
    helm history ocnadd --namespace ocnadd-deploy

    Sample output:

    The description should be "upgrade complete".

  5. Verify if the upgrade is successful using the following steps:
    1. All the pods that have respawned after the upgrade, have the latest age ( in secs ).
    2. The Adapter pods also get respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds.

      In case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

  6. <Optional> Upgrade InterBrokerProtocolVersion in Kafka brokers.

    This step is required if the Kafka "InterBrokerProtocolVersion" is to be updated in the new release.

    1. Update the InterBrokerProtocolVersion in <ocnadd_default_wg_helm_chart>/charts/ocnaddkafka/values.yaml to the version originally mentioned in the target release charts, and set InterBrokerProtocolVersion to the desired version.
    2. Perform helm upgrade:
      helm upgrade <release-name> -f ocnadd-custom-values-<default-worker-group>.yaml --namespace <ocnadd-namespace> <target-release-helm-chart>
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg
  7. The topic for BSF should be created post upgrade. To create OCNADD Kafka topics, see the "Creating Kafka Topic for OCNADD" section of Oracle Communications Network Analytics Data Director User Guide.
  8. <Optional> To update the SNMP MIBs, follow the section "OCNADD MIB FILES" of the Oracle Communications Network Analytics Data Director User Guide.

    For information on the new features like Data Export, see Oracle Communications Network Analytics Data Director User Guide.

Upgrading Source Release(Centralized) to Target Release in Centralized Deployment Mode

Note:

This scenario is applicable for 24.2.x Centralized Deployment Mode upgrade to 24.3.x Centralized Deployment Mode.
  1. Follow the below steps:
    1. Create a copy of the charts and custom values of the target release for the management group and for the default worker group or if the worker group in separate namespace, from the ocnadd-package-24.3.0 folder. The user can create copy of helm chart folder and custom-values file in the following suggested way:
      1. For Management Group: Create a copy of the following files from extracted folder:
        
# cd ocnadd-package-24.3.0
# cp -rf ocnadd ocnadd_mgmt
# cp custom-templates/ocnadd-custom-values-24.3.0.yaml ocnadd-custom-values-mgmt-group.yaml
      2. For Worker Group: Create a copy of the following files from extracted folder:
        
# cp -rf ocnadd ocnadd_wg
# cp custom-templates/ocnadd-custom-values-24.3.0.yaml ocnadd-custom-values-wg.yamlNote: Create multiple copies of the worker group custom values file, depending on the number of worker groups to be upgraded.
    2. Modify ocnadd-custom-values-mgmt-group.yaml file created above and update it as below:
      
global.deployment.centralized: true
global.deployment.management: true
global.deployment.management_namespace:ocnadd-deploy     ##---> update it with source-release-namespace for example ocnadd-deploy
    3. Upgrade using ocnadd_mgmt helm charts folder created for the management group:
      helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart>
      For example:
      helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt
    4. Verify if the Management group upgrade is successful.
  2. Now upgrade the default Worker Group or the Worker Group in separate namespace, Adapter and Correlation services using the target release.
    1. Modify the ocnadd-custom-values-wg.yaml file created above and update it as below:
      
global.deployment.centralized: true
global.deployment.management: true                               ##---> update it to 'false'
global.deployment.management_namespace:ocnadd-deploy             ##---> update it with source-release-namespace for example ocnadd-deploy
    2. Perform helm upgrade using helm charts folder created for the default worker group:
      helm upgrade <source-release-name> -f ocnadd-custom-values-<worker-group>.yaml --namespace <source-release-namespace> <target-release-helm-chart>

      Where,

      • <source-release-name> is the release name of the source release deployment
      • ocnadd-custom-values-<worker-group>.yaml is the custom values file created for default-worker-group or the Worker Group in separate namespace
      • <source-release-namespace> is the OCNADD namespace of the source release
      • <target-release-helm-chart> is the location of the Helm chart of the target release
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-wg.yaml --namespace ocnadd-deploy ocnadd_wg

      Note:

      If OCCM was used to manage the certificates, follow the step 1 of the "Post Upgrade Task" before continuing with the adapter upgrades, else continue with the next step.
    3. Once the upgrade is successful perform the helm upgrade to update the adapters and correlation, using the ocnadd_mgmt helm charts folder created for the management group:
      1. Edit the ocnadd-custom-values-mgmt-group.yaml and update the following parameters:
        global.env.admin.OCNADD_UPGRADE_WG_NS=ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
      2. Perform helm upgrade using the management group charts folder:
        1. If no Correlation configurations are present:
          helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
          For example:
          helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
        2. If correlation configurations are also present:
          helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true

          Note:

          If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command.
          For example:
          helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true

          Note:

          If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command.
          helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true
        3. If ingress adapter configurations are also present:
          helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
          For example:
          helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
  3. Check the status of the upgrade, monitor the pods to come back to the RUNNING state, and wait for the traffic rate to be stabilized to the same rate as before the upgrade. Run the following command to check the upgrade status:
    helm history <release_name> --namespace <namespace-name>

    For example:

    For ocnadd-mgmt, use:
    helm history ocnadd-mgmt --namespace ocnadd-deploy
    For default worker-group, use:
    helm history ocnadd --namespace ocnadd-deploy

    Sample output:

    The description should be "upgrade complete".

  4. Verify if the upgrade is successful using the following steps:
    1. All the pods that have respawned after the upgrade, have the latest age ( in secs ).
    2. The Adapter pods also get respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds. n case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.
  5. Repeat the procedure from Step 2 onwards for upgrading the subsequent worker groups of the same deployment.

Upgrading Source Release (Non-Centralized) to Target Release in Non-Centralized Deployment Mode

Note:

This scenario is applicable for24.1.x or 24.2.x Non-Centralized Deployment Mode upgrade to 24.3.x Non-Centralized Deployment Mode.
  1. Perform the following steps on the source release to disable the 'ocnadduirouter' and 'ocnaddgui'. These services get freshly deployed with target release.
    1. Go to the source release folder.

      For example: For 24.2.0 release the folder is ocnadd-package-24.2.0

    2. Update the ocnadd-custom-values.yaml file used to deploy the source release:
      
global.ocnadduirouter.enabled: true                            ##---> update it to 'false'
global.ocnaddgui.enabled: true                                 ##---> update it to 'false'
    3. Perform helm upgrade using helm charts folder:
      helm upgrade <source-release-name> -f ocnadd-custom-values_<source-release>.yaml --namespace <source-release-namespace> <source-release-helm-chart>
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values_24.2.0.yaml --namespace ocnadd-deploy ocnadd
  2. Upgrade Steps for OCNADD Microservices:
    1. Update the ocnadd-custom-values.yaml file of the target release with centralized deployment parameters:

      Note:

      This step expects that "Preupgrade Tasks" have already been performed.
      1. Change directory to the target release folder.

        For example: For 24.3.0 release the folder is ocnadd-package-24.3.0

      2. Update the ocnadd-custom-values-24.3.0.yaml file:
        
    global.deployment.centralized: true                            ##---> update it to 'false'
    global.deployment.management: true                             ##---> update it to 'false'
    global.deployment.management_namespace:ocnadd-deploy           ##---> update it with source-release-namespace for example ocnadd-deploy                  
    global.env.admin.OCNADD_UPGRADE_WG_NS=ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
      3. If only adapter configurations are available, perform helm upgrade using helm charts folder:
        helm upgrade <source-release-name> -f ocnadd-custom-values-24.3.0.yaml --namespace <source-release-namespace> <target-release-helm-chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
        Where,
        • <source-release-name> is the release name of the source release deployment
        • ocnadd-custom-values-24.3.0.yaml is the updated custom values file of the target release
        • <source-release-namespace> is the OCNADD namespace of the source release
        • <target-release-helm-chart> is the helm chart folder of the target-release
        For example:
        helm upgrade ocnadd -f ocnadd-custom-values-24.3.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
      4. If Correlation configurations are also available:
        helm upgrade <source-release-name> -f ocnadd-custom-values-24.3.0.yaml --namespace <source-release-namespace> <target-release-helm-chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
        For example:
        helm upgrade ocnadd -f ocnadd-custom-values-24.3.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true

        Note:

        If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command.
        helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true
      5. If ingress adapter configurations are also available:
        helm upgrade <source-release-name> -f ocnadd-custom-values-24.3.0.yaml --namespace <source-release-namespace> <target-release-helm-chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
        For example:
        helm upgrade ocnadd -f ocnadd-custom-values-24.3.0.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true

    Note:

    If OCCM was used to manage the certificates, follow the step 1 of the "Post Upgrade Task" before continuing with the adapter upgrades, else continue with the next step.
  3. Check the status of the upgrade, monitor the pods to come back to the RUNNING state, and wait for the traffic rate to be stabilized to the same rate as before the upgrade. Run the command to check the upgrade status:
    helm history <release_name> --namespace <namespace-name>
    For example:
    helm history ocnadd --namespace ocnadd-deploy

    The description should be "upgrade complete".

  4. Verify if the upgrade is successful using the following steps:
    1. All the pods that have respawned after the upgrade, have the latest age (in secs).
    2. The Adapter pods also get respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds.

    In case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

  5. <Optional> Upgrade InterBrokerProtocolVersion in Kafka brokers. This step is required if the Kafka InterBrokerProtocolVersion is to be updated in the new release.
    1. Update the InterBrokerProtocolVersion in <helm-charts>/charts/ocnaddkafka/values.yaml to the version originally mentioned in the target release charts, and set InterBrokerProtocolVersion to the desired version.
    2. Run the upgrade command:
      helm upgrade <release_name> -f ocnadd-custom-values-24.3.0.yaml --namespace <ocnadd-namespace>  <helm_chart>
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-24.3.0.yaml --namespace ocnadd-deploy ocnadd
  6. <Optional> To update the SNMP MIBs, follow the section "OCNADD MIB FILES" of the Oracle Communications Network Analytics Data Director User Guide.

    For information on the new features like Data Export, see Oracle Communications Network Analytics Data Director User Guide.

Upgrading a 24.3.x Non-Centralized Deployment to Centralized Deployment Mode

Caution:

Changing mode from Centralized deployment mode to Non-Centralized deployment is not supported.

Follow this section when the upgrade to the 24.3.x release was done in the non-centralized mode and then the user wants to move to the Centralized deployment mode.

  1. Preupgrade procedure
    1. Keep a backup of the ocnadd-custom-values.yaml file and the extracted chart folder "ocnadd" of the current release as a backup before starting the upgrade procedure. Run the following commands:
      
cp custom-templates/ocnadd-custom-values-24.3.0.yaml ocnadd-custom-values-24.3.0-non-centralized.yaml
cp -rf ocnadd ocnadd-non-centralized
cp custom-templates/ocnadd-custom-values-24.3.0.yaml ocnadd-custom-values-mgmt-group.yaml
    2. Change the "repo_host" in 24.3.x release custom-templates/copy_backup_pvc.bash file:
      
cd ocnadd-package-24.3.0
 
Vi custom-templates/copy_backup_pvc.bash
 
repo_host="occne-repo-host:5000"      ##---> update occne-repo-host:5000 with the <customer image repository path>
    3. Run the "copy_backup_pvc.bash" using the deployment namespace, storageClass and serviceAccount as the argument:
      1. Run the following command: 
        bash custom-templates/copy_backup_pvc.bash <source-release-namespace> <storageClass-name> <service-account>

        Where,

        • source-release-namespace is the OCNADD namespace of the source release
        • storageClass-name is the Storage Class of OCNADD deployment (default used is standard)
        • service-account is the Service Account name set in global.cluster.serviceAccount.name in ocnadd-custom-values-24.3.0.yaml
        For example:
        bash custom-templates/copy_backup_pvc.bash ocnadd-deploy standard ocnadd
        Sample Output:
        
Output: (with namespace, storageClass and serviceAccount as the argument)
PVC backup-mysql-pvc-temp does not exist in namespace ocnadd-deploy.
PVC backup-mysql-pvc exists in namespace ocnadd-deploy.
Creating a new temporary PVC to copy into it.
persistentvolumeclaim/backup-mysql-pvc-temp created Temporary PVC created successfully
Running job to Copy backup contents from backup-mysql-pvc to backup-mysql-pvc-temp
job.batch/ocnaddcopybackuppvc-orig-to-temp created
Copied successfully
    4. Run the following commands to change directory to the release folder:

      For example: For 24.3.0 release the folder is ocnadd-package-24.3.0 

       
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-postrollback-version-db.yml
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-preupgrade-db.yaml
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-prerollback-db.yaml
    5. Perform Helm upgrade to update the backup-pvc changes:
      helm upgrade <release-name> -f ocnadd-custom-values-24.3.0.yaml --namespace <ocnadd-namespace> <helm-chart>
      Where,
      • <release-name> is the release name of the deployment
      • ocnadd-custom-values-24.3.0.yaml is the custom values file of the current
      • ocnadd release <ocnadd-namespace> is the OCNADD namespace of the current release
      • <helm-chart> is the helm chart folder of the current OCNADD deployment
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-24.3.0.yaml --namespace ocnadd-deploy ocnadd
    6. Now disable the below services in the ocnadd-custom-values.yaml file and perform the helm upgrade. These services will be redeployed as the management group services in the same release.
      1. Edit the ocnadd-custom-values.yaml file:
        vi ocnadd-custom-values-24.3.0.yaml
      2. Update the following parameters to false
        global:
         ocnaddalarm:
              enabled: true         ##==> to false
         ocnaddconfiguration:
              enabled: true         ##==> to false
         ocnaddhealthmonitoring:
              enabled: true         ##==> to false
         ocnaddbackuprestore:
              enabled: true         ##==> to false
         ocnaddadminsvc:
              enabled: true         ##==> to false
         ocnadduirouter:
              enabled: true         ##==> to false
         ocnaddgui:
              enabled: true         ##==> to false
         ocnaddredundancyagent:
              enabled: true         ##==> to false
         ocnaddexport:
              enabled: true         ##==> to false
      3. Perform helm upgrade:
        helm upgrade <release-name> -f ocnadd-custom-values-24.3.0.yaml  --namespace <ocnadd-namespace> <helm-chart>
        For example:
        helm upgrade ocnadd -f ocnadd-custom-values-24.3.0.yaml --namespace ocnadd-deploy ocnadd
  2. Install the management group services

    Update the ocnadd-custom-values.yaml with centralized deployment parameters for the management group.

    Follow the below steps:

    1. Create a copy of the charts and custom values of the current release for the management group and for the default worker group from the backup created during "Preupgrade procedure".

      The user can create copy of helm chart folder and custom-values file in the following suggested way:

      1. For Management Group:
        Create a copy of the following files from extracted folder:
        
     # cd  ocnadd-package-24.3.0
     # cp -rf ocnadd-non-centralized ocnadd_mgmt
     # cp ocnadd-custom-values-24.3.0.yaml ocnadd-custom-values-mgmt-group.yaml
      2. For default Worker Group:
        Create a copy of the following files from extracted folder:
        
     # cp -rf ocnadd-non-centralized ocnadd_default_wg
     # cp ocnadd-custom-values-24.3.0.yaml ocnadd-custom-values-default-wg.yaml
    2. Modify ocnadd-custom-values-mgmt-group.yaml file created above and update it as below:
      
    global.deployment.centralized: false                            ##---> update it to 'true'
    global.deployment.management: false                             ##---> update it to 'true'
    global.deployment.management_namespace:ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
    global.deployment.nonCenToCen_upgrade: false                    ##---> update it to 'true'
    3. Install using ocnadd_mgmt helm charts folder created for the management group:
      helm install <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <mgmt_helm_chart>
      Where,
      • <management-release-name> is the release name of the management release
      • ocnadd-custom-values-<mgmt-group>.yaml is the custom values file created for management group
      • <source-release-namespace> is the OCNADD namespace of the source release
      • <mgmt_helm_chart> is the helm chart folder of the management-group
      For example:
      helm install ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt
    4. Delete the temp PVC created for the backup during "Preupgrade procedure":
      kubectl delete pvc -n ocnadd-deploy backup-mysql-pvc-temp
  3. Now upgrade the default Worker Group and Adapter services of the current release:
    1. Modify the ocnadd-custom-values-default-wg.yaml file created above and update it as below:
      
    global.deployment.centralized: false                           ##---> update it to 'true'
    global.deployment.management: false                           
    global.deployment.management_namespace:ocnadd-deploy           ##---> update it with source-release-namespace for example ocnadd-deploy
    global.deployment.nonCenToCen_upgrade: false                   ##---> update it to 'true'
    2. Perform helm upgrade using helm charts folder created for the default worker group:
      helm upgrade <release-name> -f ocnadd-custom-values-<default-worker-group>.yaml --namespace <ocnadd-namespace> <worker-group-helm-chart>
      Where,
      • <release-name> is the release name of the deployment
      • <ocnadd-custom-values-<default-worker-group>.yaml is the custom values file created fro worker group
      • <ocnadd-namespace> is the OCNADD namespace of the current release
      • <worker-group-helm-chart> is the helm chart folder of the worker-group
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg
    3. Once upgrade is successful perform the helm upgrade to update the adapters, correlation services using ocnadd_mgmt helm charts folder created for the management group:
      1. Edit the ocnadd-custom-values-mgmt-group.yaml and update the following parameter:
        global.env.admin.OCNADD_UPGRADE_WG_NS=ocnadd-deploy           ##---> update it with the ocnadd-namespace or the default worker group namespace for example ocnadd-deploy
      2. Perform helm upgrade using the management group charts folder:
        1. If no Correlation configurations are created:
          helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
          For example:
          helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
        2. If Correlation configurations are created:
          helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
          For example:
          helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true

          Note:

          If the correlation configuration was enabled for extended storage, then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command:
          helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true
        3. If ingress adapter configurations are created:
          helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
          For example:
          helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
  4. Check the status of the upgrade, monitor the pods to come back to the RUNNING state, and wait for the traffic rate to be stabilized to the same rate as before the upgrade.
    Run the command to check the upgrade status:
    helm history <release_name> --namespace <namespace-name>

    For example:

    For ocnadd-mgmt use:
    helm history ocnadd-mgmt --namespace ocnadd-deploy
    For default worker-group use:
    helm history ocnadd --namespace ocnadd-deploy

    The description should be "upgrade complete".

  5. Verify if the upgrade is successful using the following steps:
    1. All the pods that have respawned after the upgrade, have the latest age ( in secs )
    2. The Adapter pods also get respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds.

      In case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

4.5.1 Hotfix Upgrade

For any patch upgrade follow the Preupgrade Tasks section along with the ReadMe.txt file provided with the patch.

4.5.2 Create Secrets For Target Release

Follow the steps to generate the secrets for the target release.

  1. Navigate directory to the target release "ocnadd-package-24.3.0" folder, and then change the directory to the <ssl_certs>.
  2. Run the generate_certs.sh script with the following command:
    ./generate_certs.sh -cacert <path to>/CAcert.pem -cakey <path to>/CAkey.pem

    Where, <path to> is the folder path where the CACert and CAKey are present.

    Note:

    In case the certificates are being generated for the worker group separately, ensure that the same CA certificate and private keys are used for generating the certificates as those used for generating the management group certificates. The same command, as mentioned below, can be used for the worker group certificate generation after the management group certificates have been generated:
    ./generate_certs.sh -cacert <path to>/cacert.pem -cakey <path to>/private/cakey.pem
  3. Select the mode of deployment:

    Note:

    • If an upgrade is being performed from Non-centralized to Non-Centralized deployment mode, select option 1.
    • If an upgrade is being performed from Non-centralized to Centralized deployment mode, select option 2.
    
"1" for non-centralized
"2" for upgrade from non-centralized to centralized
"3" for centralised
"4" for simulator
Select the mode of deployement (1/2/3) : 3
  4. Select the namespace where the user would like to generate the certificates.
    Enter Kubernetes namespace: <your_working_namespace>
  5. Select the type of service group the user would like to deploy. The below example is for the Management Group:
    
Choose the group of services:
1. management_group_services
2. worker_group_services
  
Choose a file by entering its corresponding number: (1 or 2) 1
  6. Enter the domain name with which the user wants to change the default domain name(occne-ocdd) in the chosen service_values file which will be used to create the certificate.
    Please enter the domain name: <domain_name>
  7. Enter SAN (DNS/IP entries) for any service if required. 
    Do you want to add any IP for adding SAN entries to existing dd services (y/n): y

    If the user selects 'y,' a list of services will be displayed, and the user can add SAN entries for any of the listed services by selecting the corresponding service number.

    In the following example, the list of management services is presented for the user to add SAN entries. Enter the number corresponding to the service for which the user wants to input IP addresses. After choosing the service, provide IP addresses as input; otherwise, enter 'n' to exit." 

    
For the following services:
1. ocnadduirouter
2. ocnaddadminservice
3. ocnaddalarm
4. ocnaddconfiguration
5. ocnaddhealthmonitoring
6. ocnaddbackuprestore
7. ocnaddredundancyagent
8. ocnaddexport
  
Enter the number corresponding to the service for which you want to add IP: 7
Please enter IP for the service ocnaddredundancyagent or enter "n" to exit : 10.20.30.41
Please enter IP for the service ocnaddredundancyagent or enter "n" to exit : n
Do you want to add IP to any other service (y/n) : n
  8. Select "y" when prompted to create CA.
    Do you want to create Certificate Authority (CA)? (y/n) y
  9. Enter the passphrase for the CA key when prompted.
    Enter passphrase for CA Key file: <passphrase>
  10. Select “y” when prompted to create CSR for each service.
    Create Certificate Signing Request (CSR) for each service?: Y
  11. Select “y” when prompted to sign CSR for each service with CA Key.
    Would you like to sign CSR for each service with CA key? Y
  12. If the centralized mode of deployment is selected while creating management group certificates, once the generation of management group certificates is completed, the user will be prompted to continue the certificate generation process for worker groups.
    Would you like to continue certificate creation for worker group? (y/n) y

    If "y" is selected, the script will execute to recreate the certificates for worker group. The script will repeat its execution from step 4 onwards. During worker group creation flow, select "worker_group_service_values" in step 5 and proceed.If "n" is selected, script will complete its execution

  13. Run the following command to check if the secrets are created in the specified namespace:
    kubectl get secret -n <namespace>
  14. Run the following command to describe any secret created by the script:
    kubectl describe secret <secret-name> -n <namespace>

4.6 Post Upgrade Task

This procedure is required only when OCCM is used to manage certificates in both the source and target releases.

  1. Update OCCM-Managed Certificates:
    • Update the following parameters in the ocnadd-custom-values-24.3.0.yaml file for the required worker group:
      • global.certificates.occm.san.kafka.update_required
      • global.certificates.occm.san.kafka.uuid.client
      • global.certificates.occm.san.kafka.uuid.server

    Kafka SAN Upgrade Example:

    global:
  certificates:
    occm:
      san:
        kafka:
          update_required: true  # Set to true, default is false
          uuid:
            client: 9138b974-2c89-4c9d-bc5c-0ca82752d50b  # Provide the UUID value of the certificate KAFKABROKER-SECRET-CLIENT-<namespace> from OCCM, where <namespace> is the Worker group namespace
            server: 5e765aeb-ae1b-426b-8481-f8f3dcdd645e  # Provide the UUID value of the certificate KAFKABROKER-SECRET-SERVER-<namespace> from OCCM, where <namespace> is the Worker group namespace
  2. Run Helm Upgrade for the Worker Group Namespace:
    helm upgrade <worker-group-release-name> -f <worker-group-custom-values> -n <worker-group-ns> <ocnadd-helm-chart-location>
  3. After running the Helm upgrade, new certificates will be created. Verify them through the OCCM UI. Kafka Brokers will restart after the Helm upgrade is completed and will begin using the newly created certificates.