Upgrading OCNADD

4.1 Supported Upgrade Paths

The following table lists the supported upgrade paths for OCNADD:

Table 4-1 Supported Upgrade Path

Source Release	Target Release
24.2.0.0.x	25.1.100
24.3.0.0.x	25.1.100

4.2 Preupgrade Tasks

Note:

(Optional) Configure Helm parameters for OCCM based Certificate Management, refer to the section Helm Parameter Configuration for OCCM for configuration
Kafka PVC Storage Expansion: In case there is a need to increase the Kafka broker PVC size due to increased throughput support in the target release, then the expansion of the PVC must be done in the source release before initiating the upgrade. To increase the size, follow the section 'Expanding Kafka Storage' from the Oracle Communications Network Analytics Data Director User Guide.

Before starting the procedure to upgrade OCNADD, perform the following tasks:

Note:

While performing an upgrade, you must align the ocnadd-custom-values-25.1.100.yaml file of the target release as per the ocnadd/values.yaml file of the source release or the older release. Do not enable any new feature during the upgrade. The parent or sub-charts values.yaml must not be changed while performing the upgrade, unless it is explicitly specified in this document. For information about enabling any new feature through Helm parameters, see Oracle Communications Network Analytics Data Director User Guide.
The current resource profile in helm chart is for 135K MPS, if throughput is increased and it is required to use 270K MPS, update resources as per 270K MPS resource profile mentioned in Oracle Communications Network Analytics Data Director Benchmarking Guide.
Ensure that if intra-TLS is disabled in the source release, certificates for all the required services are already created. For more details, see the 'Internal TLS Communication' section in the Oracle Communications Network Analytics Suite Security Guide.

Preupgrade Tasks

Fetch the images and charts of the target release as described in Installing OCNADD.
Keep a backup of the ocnadd-custom-values.yaml file and the extracted chart folder "ocnadd" of the source release as a backup before starting the upgrade procedure.
Create the certificates and secrets only for the services that are newly added in the target release with the CA used in the source release. See, Create Secrets For Target Release to create secrets. If not skip this step.
Upgrade is supported only in Centralized mode. The default upgrade involves transitioning from a non-centralized site to a centralized site with the default worker group. This process does not require HNS installation. However, if more than one worker group is needed, HNS installation must be completed before expanding the worker group after the upgrade. For installation details, see Kubectl HNS Installation section.
Update the following helm chart files of the target release with the parameter values of the source release files:
- update ocnadd/ocdd-db-resource.sql
- update ocnadd/templates/ocnadd-secret-hook.yaml
- update custom-templates/ocnadd-custom-values.x.x.x.yaml
Note:
- Ensure the parameters such as serviceAccount, Role, RoleBinding are retained from the source release. These are the deployment parameters and should not be modified as part of the upgrade.
  Ensure to set the upgrade parameter to true as shown below:
```
serviceAccount:
    create: true
    name: <must be same as previous release>
    upgrade: true  ## ->> Update this to 'true', default is false 
clusterRole:
     create: true
     name: <must be same as previous release>
clusterRoleBinding:
     create: true
     name: <must be same as previous release>
```
Update the pvcClaimSize in the target release ocnadd-custom-values-25.1.100.yaml file.
Ensure to update the "offsetsTopicReplicationFactor" and "transactionStateLogReplicationFactor" in the target release ocnadd-custom-values-25.1.100.yaml file.
Ensure that the IP Family configuration in the target release matches with the IP configuration in the source release during the upgrade process. Changing the IP Family from IPv4 to IPv6 or vice-versa while performing upgrade is not supported.

Alter the mysql users, <admin_user>, ocddAppUsr and ocddPrivilegedUsr.

Check user in mysql console:-
 
kubectl -n occne-cndbtier exec -it ndbmysqld-0 -- bash
 
mysql -h 127.0.0.1 -uroot -p $ Enter password:
 
Run the below command to check if auth plugin of mysql user:-
 
SELECT user,plugin FROM mysql.user;
 
Run below commands if and only if auth plugin is 'mysql_native_password' for <admin_user>, ocddPrivilegedUsr and ocddAppUsr
 
For Admin User:-
ALTER USER '<admin_user>'@'%' IDENTIFIED WITH caching_sha2_password BY '<admin_user_password>';
where, <admin_user> is the user created in Section "Creating an Admin User in the Database"
Example:
ALTER USER 'ocdd'@'%' IDENTIFIED WITH caching_sha2_password BY 'ocdd';
 
For ocddPrivilegedUsr:-
ALTER USER 'ocddPrivilegedUsr'@'%' IDENTIFIED WITH caching_sha2_password BY '<ocddPrivilegedUsr_password>';
Example:
ALTER USER 'ocddPrivilegedUsr'@'%' IDENTIFIED WITH caching_sha2_password BY 'ocddPrivilegedPasswd';
 
For ocddAppUsr:-
ALTER USER 'ocddAppUsr'@'%' IDENTIFIED WITH caching_sha2_password BY '<ocddAppUsr_password>';
Example:
ALTER USER 'ocddAppUsr'@'%' IDENTIFIED WITH caching_sha2_password BY 'ocddAppPasswd';

Take the manual backup of the OCNADD before starting the upgrade procedures. See, Performing OCNADD Manual Backup for taking a manual backup of the OCNADD.
Make sure to delete all the existing backup, restore, and ocnaddverify jobs before proceeding with the upgrade. Backup related jobs are "ocnaddbackup", "ocnaddrestore", "ocnaddverify" and "ocnaddmanualbackup".
Use the following command to check for the currently running or completed backup jobs:
```
kubectl get job -n <ocnadd-namespace>
```
To delete jobs
```
kubectl delete job -n <ocnadd-namespace> <job-name>
```
Before upgrading to 25.1.100, certificates need to be created or updated, followed by the deletion of the Kafka-broker StatefulSet (STS) in orphan mode from the 24.2.x deployment. This step can be skipped if upgrading from 24.3.0.
1. If OCNADD SSL Certificates Procedure was Used:Follow the procedure “Update Certificate of the Existing Services” in the Oracle Communication Network Analytics Data Director User Guide to edit the default_values/renew_cert_files and comment out all the OCNADD services except the Kafka-broker.
2. If OCCM Was Used to Manage the Certificate:Follow the step 1 of the Post Upgrade Task instructions.
3. Run the following command in the 24.2.x OCNADD deployment to delete Kafka-Broker STS:
```
kubectl delete statefulset --cascade=orphan kafka-broker -n <namespace>
```
  Note:
  If NFs are deployed in the same cluster and use the FQDN .kafka-broker.<namespace>.svc.<domain> as the Kafka bootstrap to connect to OCNADD, this FQDN must be updated to *.kafka-broker-headless.<namespace>.svc.<domain> in NFs after upgrading to OCNADD 25.1.100.
If export configurations are present in the source release with exportType=PCAP, follow the steps below. Otherwise, this step can be skipped.
1. Login to MySQL Database Instance with DB user of Data Director
2. Run the following commands:
  1. ```
  USE <CONFIGURATION-SCHEMA-NAME>;
```
  For example:
```
  USE configuration_schema_dd;
```
2. ```
SELECT S.CONFIGURATION_NAME, S.REMAINING_EXPORT_TIME 
FROM DATA_EXPORT_STATUS S 
JOIN EXPORT_CONFIGURATION C 
ON C.CONFIGURATION_NAME = S.CONFIGURATION_NAME 
WHERE EXPORT_TYPE = 'PCAP';
```
    For example:
```
SELECT S.CONFIGURATION_NAME, S.REMAINING_EXPORT_TIME 
FROM DATA_EXPORT_STATUS S 
JOIN EXPORT_CONFIGURATION C 
ON C.CONFIGURATION_NAME = S.CONFIGURATION_NAME 
AND C.EXPORT_TYPE = 'PCAP';

+--------------------+----------------------------+
| CONFIGURATION_NAME | REMAINING_EXPORT_TIME       |
+--------------------+----------------------------+
| pcap_export_test   | +56572-12-29T09:26:48.000Z  |
| after-upgrade-pcap | +56573-01-30T05:01:30.000Z  |
+--------------------+----------------------------+
```
3. Retrieve the Corresponding PCAP File: Locate the PCAP file present on the SFTP server for the corresponding configuration. Open the file and go to the last frame of the file. Use the timestamp (for example, highlighted below) to update the REMAINING_EXPORT_TIME column.
  For example:
```
Timestamp format in the file: 2024-07-31 17:10:00.408283
Timestamp format to use for updating the column: 2024-07-31T17:10:00.408Z
```
  Note:
  If the export configuration exists for exportType=PCAP and the PCAP file is not present on the SFTP server, use the export configuration’s "StartTime+1 second" from the UI to update the REMAINING_EXPORT_TIME column. In this case, the export will start from "StartTime+1 second."
4. Update REMAINING_EXPORT_TIME: Execute the following command on the MySQL terminal to update the REMAINING_EXPORT_TIME in the DATA_EXPORT_STATUS table and commit the changes:
```
UPDATE DATA_EXPORT_STATUS 
SET REMAINING_EXPORT_TIME = '<UTC timestamp from the above step>' 
WHERE CONFIGURATION_NAME = '<pcap export configuration name>';

COMMIT;
```
  For example:
```
UPDATE DATA_EXPORT_STATUS 
SET REMAINING_EXPORT_TIME = '2024-07-31T17:10:00.408Z' 
WHERE CONFIGURATION_NAME = 'pcap_export_test';

COMMIT;
```
5. Repeat steps 'c' and 'd' to update REMAINING_EXPORT_TIME for all export configurations with exportType=PCAP.
  Note:
  - Post upgrade, if the remaining export time gets updated again(during upgrade) with the incorrect timestamp, then repeat the above steps again after the upgrade.
  - Restart the ocnaddconfiguration and ocnaddexport services after the timestamp update.

4.3 Upgrade Sequence

The upgrade sequence of the procedures to be followed is described in this section.

4.3.1 Upgrade Order for Source NFs

By design, Kafka clients (producers and consumers) and brokers are bidirectionally compatible:

Clients with a higher version of Kafka API can communicate with brokers of a lower version
Clients with a lower Kafka API version can communicate with brokers of a higher version

Kafka clients and brokers exchange API version information during a handshake.

Upgrade the source NFs and the OCNADD independently of each other, and no specific upgrade order is required. Upgrade to a new release succeeds if compatibility is maintained. See, "Compatibility Matrix" in the Oracle Communications Network Analytics Suite Release Notes.

The OCNADD upgrade requires less time than source NFs (SCP, SEPP, and NRF) upgrade. It is advisable first to upgrade the OCNADD and verify the traffic flow post upgrade for any significant errors or potential roadblocks in the upgrade. If the NFs are upgraded first, rollback of large numbers of source NFs workers and gateway PODs might be required.

Upgrade Order:

Upgrade OCNADD
Upgrade source NFs (NRF, SCP, and SEPP)

4.3.2 Upgrade Order for CNC Console and cnDBTier

The following upgrade order is recommended for CNC Console and cnDBTier:

For CNC Console upgrade, see Oracle Communications Cloud Native Configuration Console Installation, Upgrade, and Fault Recovery Guide.
For OCNADD upgrade, see Upgrade Sequence.
For cnDBTier upgrade, see Oracle Communications Cloud Native Core cnDBTier Installation, Upgrade, and Fault Recovery Guide.

Note:

For more information on customizing database parameters required for the OCNADD, see cnDBTier Customization Parameters section.
To perform cnDBTier upgrade, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

4.4 Upgrade Impact on Source NFs and Third Party Consumers

Listed below are the observed upgrade impacts on the source NFs and the Third Party Consumers:

In the case of a Kafka upgrade, the Kafka clients (NF producers and consumers) should not impact the Kafka API as the API compatibility is maintained between clients and brokers by Kafka.
The Kafka binary upgrade is a two step procedure in which the Helm upgrade is performed twice. One upgrade for the Kafka binary upgrade and the second (optional) upgrade if the InterBrokerProtocolVersion is changed. During this upgrade, the source NFs (Kafka producers) may face communication disruption with Kafka brokers multiple times as each broker is expected to restart two times. The producer clients should adopt appropriate reconnection and metadata refresh mechanisms. Suppose the producers run with the 'acks=0' and 'retries=0' configurations. There is no guarantee of reliable message delivery between producers and Kafka brokers during the upgrade, as broker instances restart multiple times.
The NF producers must maintain the list of servers in the bootstrap-server parameter instead of a single server in the bootstrap-server parameter.
Assume that the OCNADD upgrade is performed at 20% (approximately) of the supported traffic rate. The upgrade is performed using the rolling upgrade strategy. The traffic flow between the NFs and the OCNADD Kafka may remain degraded for a few minutes. The traffic rate is expected to become normal after the upgrade when the NFs producers reconnect to the Kafka brokers.
Assume that the OCNADD upgrade is performed at 20% (approximately) of the supported traffic rate. The upgrade is performed using the rolling upgrade strategy. The traffic flow between OCNADD consumer adapters and Third party consumers may remain degraded for a few minutes. The traffic rate is expected to be normal after the upgrade when the Kafka broker pods come up, all the consumer adapter pods have been upgraded, and consumer rebalancing is complete.
During the OCNADD upgrade, Kafka retention helps prevent data loss for Third party consumers. However, expect some data duplication towards Third party consumers due to multiple Kafka consumer rebalancing.
Plan the OCNADD upgrade during a maintenance window.

4.5 Upgrade Tasks

This section includes information about upgrading an existing OCNADD deployment.

When you upgrade an existing OCNADD deployment, the running set of containers and pods are replaced with the new set of containers and pods. However, If there is no change in the pod configuration, the running set of containers and pods are not replaced, unless there is a change in the service configuration of a microservice, the service endpoints will remain unchanged (NodePort and so on).

Note:

<Optional> Set a timeout interval of 15 minutes during the upgrade process.
Ensure no OCNADD pod is in a failed state.
Ensure that the defined in the Preupgrade Tasks are complete
During an upgrade that affects all brokers, anticipate approximately a minute of downtime for Kafka brokers. Alternatively, upgrade brokers individually to avoid this downtime if applicable.
Kafka upgrade along with PVC storage changes isn't supported.
The creation of Consumer Adapter pods or services occurs when Data feeds are created via OCNADD GUI. Ensure the upgrade of these pods is set to "false" (global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE is set to false by default). To perform an upgrade, refer to the "Updating Consumer Adapter Parameters" section in the Oracle Communications Network Analytics Data Director User Guide.
The creation of Correlation pods or services takes place when Correlation configurations are created from OCNADD GUI. Ensure the upgrade of these pods is set to "false" (global.env.admin.OCNADD_CORR_UPGRADE_ENABLE is set to false by default). To upgrade, refer to the "Updating Correlation Service Parameters" section in the Oracle Communications Network Analytics Data Director User Guide.

The following upgrades are supported:

Upgrading Source Release(Non-Centralized) to Target Release in Centralized Deployment Mode (Recommended)

Note:

This scenario is applicable for OCNADD release 24.3.x or 24.2.x Non-Centralized Deployment Mode upgrade to 25.1.1xx Centralized Deployment Mode.

Preupgrade procedure:

Change the "repo_host" in target release custom-templates/copy_backup_pvc.bash file


cd ocnadd-package-25.1.100
 
Vi custom-templates/copy_backup_pvc.bash
 
repo_host="occne-repo-host:5000"      ##---> update occne-repo-host:5000 with the <customer image repository path>

Run "copy_backup_pvc.bash" using the namespace, storageClass and serviceAccount of the source release deployment as the argument.
Run the following command:
```
bash custom-templates/copy_backup_pvc.bash <source-release-namespace> <storageClass-name> <service-account>
```
Where,
- source-release-namespace is the OCNADD namespace of the source release
- storageClass-name is the Storage Class of OCNADD deployment (default used is standard)
- service-account is the Service Account name set in global.cluster.serviceAccount.name in ocnadd-custom-values-25.1.100.yaml
For example:
```
bash custom-templates/copy_backup_pvc.bash ocnadd-deploy standard ocnadd
```
Sample output (with namespace, storageClass and serviceAccount as the argument):
```
PVC backup-mysql-pvc-temp does not exist in namespace ocnadd-deploy.
PVC backup-mysql-pvc exists in namespace ocnadd-deploy.
Creating a new temporary PVC to copy into it.
persistentvolumeclaim/backup-mysql-pvc-temp created Temporary PVC created successfully
Running job to Copy backup contents from backup-mysql-pvc to backup-mysql-pvc-temp
job.batch/ocnaddcopybackuppvc-orig-to-temp created
Copied successfully
```

Change directory to the source release folder.

For example: For 24.2.0 release the folder is ocnadd-package-24.2.0

Run the following commands in the source release:


sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-postrollback-version-db.yml
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-preupgrade-db.yaml
sed -i 's/backup-mysql-pvc/backup-mysql-pvc-temp/g' ocnadd/templates/ocnadd-prerollback-db.yaml

Perform Helm upgrade on source release to update the backup-pvc changes:
```
helm upgrade <source-release-name> -f ocnadd-custom-values.yaml --namespace <source-release-namespace> <source-release-helm-chart>
```
Where,
- <source-release-name> is the release name of the source release deployment
- ocnadd-custom-values.yaml is the custom value file of the source release
- <source-release-namespace> is the OCNADD namespace of the source release
- <source-release-helm-chart> is the helm chart folder of the source release
For example (Example considering 24.2.x release)
```
helm upgrade ocnadd -f ocnadd-custom-values.24.2.0.yaml --namespace ocnadd-deploy ocnadd
```

Now disable the below services in the source release ocnadd-custom-values.yaml file and perform the helm upgrade. These services will be redeployed as the management group services in the target release.

Edit the source release ocnadd-custom-values.yaml file
For example: (considering 24.2.x release)
```
vi ocnadd-custom-values.24.2.0.yaml
```

Update the following parameters to false

global:
     ocnaddalarm:
          enabled: true           ##==> to false
     ocnaddconfiguration:
          enabled: true           ##==> to false
     ocnaddhealthmonitoring:
          enabled: true           ##==> to false
     ocnaddbackuprestore:
          enabled: true           ##==> to false
     ocnaddadminsvc:
          enabled: true           ##==> to false
     ocnadduirouter:
          enabled: true           ##==> to false
     ocnaddgui:
          enabled: true           ##==> to false
     ocnaddexport:
          enabled: true           ##==> to false

Perform helm upgrade:

helm upgrade <source-release-name> -f ocnadd-custom-values.yaml  --namespace <source-release-namespace> <source-release-helm-chart>

For example:

helm upgrade ocnadd -f ocnadd-custom-values.24.2.0.yaml --namespace ocnadd-deploy ocnadd

Install the target release management group services, considering target release to 25.1.100
Update the ocnadd-custom-values.yaml of the target release with centralized deployment parameters for the management group.
Follow the below steps:
1. Create a copy of the charts and custom values of the target release for the management group and for the default worker group from the ocnadd-package-25.1.100 folder.
  The user can create copy of helm chart folder and custom-values file in the following suggested way:
  1. For Management Group:
    Create a copy of the following files from extracted folder:
    # cd ocnadd-package-25.1.100 # cp -rf ocnadd ocnadd_mgmt # cp custom-templates/ocnadd-custom-values-25.1.100.yaml ocnadd-custom-values-mgmt-group.yaml
  2. For default Worker Group:
    Create a copy of the following files from extracted folder:
    # cp -rf ocnadd ocnadd_default_wg # cp custom-templates/ocnadd-custom-values-25.1.100.yaml ocnadd-custom-values-default-wg.yaml
2. Modify ocnadd-custom-values-mgmt-group.yaml file created above and update it as below:
```
    global.deployment.centralized: true
    global.deployment.management: true
    global.deployment.management_namespace:ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
    global.deployment.nonCenToCen_upgrade: false                    ##---> update it to 'true'
```
3. Install using ocnadd_mgmt helm charts folder created for the management group:
```
helm install <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart>
```
  For example:
```
helm install ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt
```
4. Delete the temp PVC created for the backup during "Preupgrade procedure".
  Run the following command:
```
kubectl delete pvc -n ocnadd-deploy backup-mysql-pvc-temp
```
Now upgrade the default Worker Group and Adapter services using the target release.
1. Modify the ocnadd-custom-values-default-wg.yaml file created above and update it as below:
```
    global.deployment.centralized: true
    global.deployment.management: true                             ##---> update it to 'false'
    global.deployment.management_namespace:ocnadd-deploy           ##---> update it with source-release-namespace for example ocnadd-deploy
    global.deployment.nonCenToCen_upgrade: false                   ##---> update it to 'true'
```
2. Run the following command to delete the Kafka stateful set for the updated Kafka headless service (this step can be skipped if upgrading from 24.3.0):
```
kubectl delete statefulset --cascade=orphan kafka-broker -n <namespace>
```
3. Perform helm upgrade using helm charts folder created for the default worker group:
```
helm upgrade <source-release-name> -f ocnadd-custom-values-<default-worker-group>.yaml --namespace <source-release-namespace> <target-release-helm-chart>
```
  Where,
  - <source-release-name> is the release name of the source release deployment
  - ocnadd-custom-values-<default-worker-group>.yaml is the custom values file created for default-worker-group
  - <source-release-namespace> is the OCNADD namespace of the source release
  - <target-release-helm-chart> is the location of the Helm chart of the target release
    For example:
```
helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg
```
    Note:
    If OCCM was used to manage the certificates, follow the step 1 of the "Post Upgrade Task" before continuing with the adapter upgrades, else continue with the next step.
4. Once upgrade is successful perform the helm upgrade to update the adapters, using ocnadd_mgmt helm charts folder created for the management group:
  1. Edit the ocnadd-custom-values-mgmt-group.yaml and update the following parameter:
```
global.env.admin.OCNADD_UPGRADE_WG_NS=ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
```
  2. If only adapter configurations are present, perform helm upgrade using the management group charts folder:
```
helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
```
    For example:
```
helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
```
  3. If Correlation configurations are also present:
```
helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
```
    Note:
    If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command.
    For example:
```
helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
```
    Note:
    If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command:
    helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true
  4. If ingress adapter configuration are also present:
```
helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
```
    For example:
```
helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
```
Check the status of the upgrade, monitor the pods to come back to the RUNNING state, and wait for the traffic rate to be stabilized to the same rate as before the upgrade.
Run the following command to check the upgrade status:
```
helm history <release_name> --namespace <namespace-name> 
```
For example:
For ocnadd-mgmt, use:
```
helm history ocnadd-mgmt --namespace ocnadd-deplo
```
For default worker-group, use:
```
helm history ocnadd --namespace ocnadd-deploy 
```
Sample output:

The description should be "upgrade complete".
Verify if the upgrade is successful using the following steps:
1. All the pods that have respawned after the upgrade, have the latest age ( in secs ).
2. The Adapter pods also get respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds.
  In case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.
<Optional> Upgrade InterBrokerProtocolVersion in Kafka brokers.
This step is required if the Kafka "InterBrokerProtocolVersion" is to be updated in the new release.
1. Update the InterBrokerProtocolVersion in <ocnadd_default_wg_helm_chart>/charts/ocnaddkafka/values.yaml to the version originally mentioned in the target release charts, and set InterBrokerProtocolVersion to the desired version.
2. Perform helm upgrade:
```
helm upgrade <release-name> -f ocnadd-custom-values-<default-worker-group>.yaml --namespace <ocnadd-namespace> <target-release-helm-chart>
```
  For example:
```
helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg
```
Ensure that the Kafka topics for the required NFs are created post upgrade to current release. To create Kafka topics, see "Creating Kafka Topic for OCNADD" section in Oracle Communications Network Analytics Data User Guide.

Ensure that only required NFs aggregation is enabled on the OCNADD. For example, if the customer is only intendening to use SCP as source NF with OCNADD, then it is recommended to turn off all the other NF specific aggregation instances. The below modifications should be done in <chartpath/ocnadd/charts/ocnaddaggregation/values.yaml>:

#The values file should be modified for the particular NF aggregation instance(s). The replica count should be set to 0
# For example customer is only using SCP, so aggregation instance for NRF, SEPP e.t.c should be updated as suggested below:
 
ocnaddnrfaggregation:
    autoScaling:
        enabled: true
    name: ocnaddnrfaggregation
    defineEnvVariables: true
    replicas: 1                       ==========> this should be updated to 0
    label:
        name: ocnaddnrfaggregation
 
# the similar modification for other non-required NF aggregation instances and file should be saved.

Perform helm upgrade:

helm upgrade <release-name> -f ocnadd-custom-values-<default-worker-group>.yaml --namespace <ocnadd-namespace> <target-release-helm-chart>

For example:

helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg

<Optional> To update the SNMP MIBs, follow the section "OCNADD MIB FILES" of the Oracle Communications Network Analytics Data Director User Guide.

Note:

In case of the intraTLS is disabled, then additional ceritificates that are not required can be removed when the upgrade is verified and it is successful. For more details, see "Internal TLS Communication" section in the Oracle Communications Network Analytics Suite Security Guide.

Upgrading Source Release(Centralized) to Target Release in Centralized Deployment Mode

Note:

This scenario is applicable for 24.2.x/24.3.x Centralized Deployment Mode upgrade to 25.1.1xx Centralized Deployment Mode.

Follow the below steps:
1. Create a copy of the charts and custom values of the target release for the management group and for the default worker group or if the worker group in separate namespace, from the ocnadd-package-25.1.100 folder. The user can create copy of helm chart folder and custom-values file in the following suggested way:
  1. For Management Group: Create a copy of the following files from extracted folder:
```
# cd ocnadd-package-25.1.100
# cp -rf ocnadd ocnadd_mgmt
# cp custom-templates/ocnadd-custom-values-25.1.100.yaml ocnadd-custom-values-mgmt-group.yaml
```
  2. For Worker Group: Create a copy of the following files from extracted folder:
```
# cp -rf ocnadd ocnadd_wg
# cp custom-templates/ocnadd-custom-values-25.1.100.yaml ocnadd-custom-values-wg.yamlNote: Create multiple copies of the worker group custom values file, depending on the number of worker groups to be upgraded.
```
2. Modify ocnadd-custom-values-mgmt-group.yaml file created above and update it as below:
```
global.deployment.centralized: true
global.deployment.management: true
global.deployment.management_namespace:ocnadd-deploy     ##---> update it with source-release-namespace for example ocnadd-deploy
```
3. Upgrade using ocnadd_mgmt helm charts folder created for the management group:
```
helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <source-release-namespace> <helm_chart>
```
  For example:
```
helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt
```
4. Verify if the Management group upgrade is successful.
Now upgrade the default Worker Group or the Worker Group in separate namespace, Adapter and Correlation services using the target release.
1. Modify the ocnadd-custom-values-wg.yaml file created above and update it as below:
```
global.deployment.centralized: true
global.deployment.management: true                               ##---> update it to 'false'
global.deployment.management_namespace:ocnadd-deploy             ##---> update it with source-release-namespace for example ocnadd-deploy 
```
2. Run the following command to delete the Kafka stateful set for the updated Kafka headless service (this step can be skipped if upgrading from 24.3.0):
```
kubectl delete statefulset --cascade=orphan kafka-broker -n <namespace>
```
3. Perform helm upgrade using helm charts folder created for the default worker group:
```
helm upgrade <source-release-name> -f ocnadd-custom-values-<worker-group>.yaml --namespace <source-release-namespace> <target-release-helm-chart>
```
  Where,
  - <source-release-name> is the release name of the source release deployment
  - ocnadd-custom-values-<worker-group>.yaml is the custom values file created for default-worker-group or the Worker Group in separate namespace
  - <source-release-namespace> is the OCNADD namespace of the source release
  - <target-release-helm-chart> is the location of the Helm chart of the target release
  For example:
```
helm upgrade ocnadd -f ocnadd-custom-values-wg.yaml --namespace ocnadd-deploy ocnadd_wg
```
  Note:
  If OCCM was used to manage the certificates, follow the step 1 of the "Post Upgrade Task" before continuing with the adapter upgrades, else continue with the next step.
4. Once the upgrade is successful perform the helm upgrade to update the adapters and correlation, using the ocnadd_mgmt helm charts folder created for the management group:
  1. Edit the ocnadd-custom-values-mgmt-group.yaml and update the following parameters:
```
global.env.admin.OCNADD_UPGRADE_WG_NS=ocnadd-deploy            ##---> update it with source-release-namespace for example ocnadd-deploy
```
  2. Perform helm upgrade using the management group charts folder:
    1. If no Correlation configurations are present:
```
helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
```
      For example:
      helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true
    2. If correlation configurations are also present:
```
helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
```
      Note:
      If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command.
      For example:
      helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true
      Note:
      If the correlation configuration was enabled for extended storage (from 24.2.0 onwards), then also use the flag global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true in the above command.
      helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_STORAGE_ADAPTER_UPGRADE_ENABLE=true
    3. If ingress adapter configurations are also present:
```
helm upgrade <management-release-name> -f ocnadd-custom-values-<mgmt-group>.yaml --namespace <ocnadd-namespace> <mgmt_helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
```
      For example:
      helm upgrade ocnadd-mgmt -f ocnadd-custom-values-mgmt-group.yaml --namespace ocnadd-deploy ocnadd_mgmt --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true,global.env.admin.OCNADD_CORR_UPGRADE_ENABLE=true,global.env.admin.OCNADD_INGRESS_ADAPTER_UPGRADE_ENABLE=true
Check the status of the upgrade, monitor the pods to come back to the RUNNING state, and wait for the traffic rate to be stabilized to the same rate as before the upgrade. Run the following command to check the upgrade status:
```
helm history <release_name> --namespace <namespace-name> 
```
For example:
For ocnadd-mgmt, use:
```
helm history ocnadd-mgmt --namespace ocnadd-deploy
```
For default worker-group, use:
```
helm history ocnadd --namespace ocnadd-deploy
```
Sample output:

The description should be "upgrade complete".
Verify if the upgrade is successful using the following steps:
1. All the pods that have respawned after the upgrade, have the latest age ( in secs ).
2. The Adapter pods also get respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds. n case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.
Ensure that the Kafka topics for the required NFs are created post upgrade to current release. To create Kafka topics, see "Creating Kafka Topic for OCNADD" section in Oracle Communications Network Analytics Data User Guide.

Ensure that only required NFs aggregation is enabled on the OCNADD. For example, if the customer is only intendening to use SCP as source NF with OCNADD, then it is recommended to turn off all the other NF specific aggregation instances. The below modifications should be done in <chartpath/ocnadd/charts/ocnaddaggregation/values.yaml>:

#The values file should be modified for the particular NF aggregation instance(s). The replica count should be set to 0
# For example customer is only using SCP, so aggregation instance for NRF, SEPP e.t.c should be updated as suggested below:
 
ocnaddnrfaggregation:
    autoScaling:
        enabled: true
    name: ocnaddnrfaggregation
    defineEnvVariables: true
    replicas: 1                       ==========> this should be updated to 0
    label:
        name: ocnaddnrfaggregation
 
# the similar modification for other non-required NF aggregation instances and file should be saved.

Perform helm upgrade:
```
helm upgrade <source-release-name> -f ocnadd-custom-values-<worker-group>.yaml --namespace <source-release-namespace> <target-release-helm-chart>
```
Where:
- <source-release-name>: The release name of the source release deployment.
- ocnadd-custom-values-<worker-group>.yaml: The custom values file created for the default worker group or a specific worker group in a separate namespace.
- <source-release-namespace>: The OCNADD namespace of the source release.
- <target-release-helm-chart>: The location of the Helm chart for the target release.
For example:
```
helm upgrade ocnadd -f ocnadd-custom-values-default-wg.yaml --namespace ocnadd-deploy ocnadd_default_wg
```
<Optional>Repeat the procedure from Step 2 onwards for upgrading the subsequent worker groups of the same deployment.
<Optional> To update the SNMP MIBs, follow the instructions in the section "OCNADD MIB FILES" of the Oracle Communications Network Analytics Data Director User Guide.

Note:

In case intraTLS is disabled, additional certificates that are not required can be removed once the upgrade is verified and successful. For more details, see "Internal TLS Communication" section in the Oracle Communications Network Analytics Suite Security Guide.

4.5.1 Hotfix Upgrade

For any patch upgrade follow the Preupgrade Tasks section along with the ReadMe.txt file provided with the patch.

4.5.2 Create Secrets For Target Release

Follow the steps to generate the secrets for the target release.

Navigate directory to the target release "ocnadd-package-25.1.100" folder, and then change the directory to the <ssl_certs>.
Run the generate_certs.sh script with the following command:
```
./generate_certs.sh -cacert <path to>/CAcert.pem -cakey <path to>/CAkey.pem
```
Where, <path to> is the folder path where the CACert and CAKey are present.
Note:
In case the certificates are being generated for the worker group separately, ensure that the same CA certificate and private keys are used for generating the certificates as those used for generating the management group certificates. The same command, as mentioned below, can be used for the worker group certificate generation after the management group certificates have been generated:
```
./generate_certs.sh -cacert <path to>/cacert.pem -cakey <path to>/private/cakey.pem
```
Select the mode of deployment:
Note:
- If an upgrade is being performed from Non-centralized to Non-Centralized deployment mode, select option 1.
- If an upgrade is being performed from Non-centralized to Centralized deployment mode, select option 2.
```
"1" for non-centralized
"2" for upgrade from non-centralized to centralized
"3" for centralised
"4" for simulator
Select the mode of deployement (1/2/3) : 3
```
Select the namespace where the user would like to generate the certificates.
```
Enter Kubernetes namespace: <your_working_namespace>
```

Select the type of service group the user would like to deploy. The below example is for the Management Group:


Choose the group of services:
1. management_group_services
2. worker_group_services
  
Choose a file by entering its corresponding number: (1 or 2) 1

Enter the domain name with which the user wants to change the default domain name(occne-ocdd) in the chosen service_values file which will be used to create the certificate.
```
Please enter the domain name: <domain_name>
```
Enter SAN (DNS/IP entries) for any service if required.
```
Do you want to add any IP for adding SAN entries to existing dd services (y/n): y
```
If the user selects 'y,' a list of services will be displayed, and the user can add SAN entries for any of the listed services by selecting the corresponding service number.

In the following example, the list of management services is presented for the user to add SAN entries. Enter the number corresponding to the service for which the user wants to input IP addresses. After choosing the service, provide IP addresses as input; otherwise, enter 'n' to exit."
```
For the following services:
1. ocnadduirouter
2. ocnaddadminservice
3. ocnaddalarm
4. ocnaddconfiguration
5. ocnaddhealthmonitoring
6. ocnaddbackuprestore
7. ocnaddredundancyagent
8. ocnaddexport
  
Enter the number corresponding to the service for which you want to add IP: 7
Please enter IP for the service ocnaddredundancyagent or enter "n" to exit : 10.20.30.41
Please enter IP for the service ocnaddredundancyagent or enter "n" to exit : n
Do you want to add IP to any other service (y/n) : n
```

Select "y" when prompted to create CA.

Do you want to create Certificate Authority (CA)? (y/n) y

Enter the passphrase for the CA key when prompted.
```
Enter passphrase for CA Key file: <passphrase>
```
Select “y” when prompted to create CSR for each service.
```
Create Certificate Signing Request (CSR) for each service?: Y
```
Select “y” when prompted to sign CSR for each service with CA Key.
```
Would you like to sign CSR for each service with CA key? Y
```
If the centralized mode of deployment is selected while creating management group certificates, once the generation of management group certificates is completed, the user will be prompted to continue the certificate generation process for worker groups.
```
Would you like to continue certificate creation for worker group? (y/n) y
```
If "y" is selected, the script will execute to recreate the certificates for worker group. The script will repeat its execution from step 4 onwards. During worker group creation flow, select "worker_group_service_values" in step 5 and proceed.If "n" is selected, script will complete its execution
Run the following command to check if the secrets are created in the specified namespace:
```
kubectl get secret -n <namespace>
```
Run the following command to describe any secret created by the script:
```
kubectl describe secret <secret-name> -n <namespace>
```

4.6 Post Upgrade Task

This procedure is required only when OCCM is used to manage certificates in both the source and target releases.

Update OCCM-Managed Certificates:

Update the following parameters in the ocnadd-custom-values-25.1.100.yaml file for the required worker group:
- global.certificates.occm.san.kafka.update_required
- global.certificates.occm.san.kafka.uuid.client
- global.certificates.occm.san.kafka.uuid.server

Kafka SAN Upgrade Example:

global:
  certificates:
    occm:
      san:
        kafka:
          update_required: true  # Set to true, default is false
          uuid:
            client: 9138b974-2c89-4c9d-bc5c-0ca82752d50b  # Provide the UUID value of the certificate KAFKABROKER-SECRET-CLIENT-<namespace> from OCCM, where <namespace> is the Worker group namespace
            server: 5e765aeb-ae1b-426b-8481-f8f3dcdd645e  # Provide the UUID value of the certificate KAFKABROKER-SECRET-SERVER-<namespace> from OCCM, where <namespace> is the Worker group namespace

Run Helm Upgrade for the Worker Group Namespace:

helm upgrade <worker-group-release-name> -f <worker-group-custom-values> -n <worker-group-ns> <ocnadd-helm-chart-location>

After running the Helm upgrade, new certificates will be created. Verify them through the OCCM UI. Kafka Brokers will restart after the Helm upgrade is completed and will begin using the newly created certificates.

4 Upgrading OCNADD

4.1 Supported Upgrade Paths

4.2 Preupgrade Tasks

4.3 Upgrade Sequence

4.3.1 Upgrade Order for Source NFs

4.3.2 Upgrade Order for CNC Console and cnDBTier

4.4 Upgrade Impact on Source NFs and Third Party Consumers

4.5 Upgrade Tasks

4.5.1 Hotfix Upgrade

4.5.2 Create Secrets For Target Release

4.6 Post Upgrade Task