4 Upgrading OCNADD

This section provides information on how to upgrade an existing OCNADD deployment. The section describes the upgrade order for source NFs, CNC Console, cnDBTier, and the upgrade impact on the source NFs.

Note:

The OCNADD can be upgraded from a source release to a target release using CLI procedures as outlined in the following sections. These steps can also be followed for any hotfix upgrade.

4.1 Upgrade Not Supported

WARNING:

The Release upgrade for the patch 23.3.0.0.1 is not supported. A fresh installation of this patch should be performed to deploy OCNADD services.

4.2 Preupgrade Tasks

Caution:

Before starting the upgrade, make sure to update the ocnaddcachereplicas on the source release 23.2.0.0.x. Follow the steps below to update the ocnaddcache replicas:
  1. Update the ocnaddcache replicas ocnaddcache.replicas to 3 in the source release 23.2.0.0.x ocnadd-custom-values.yaml file.
  2. Perform helm upgrade to apply the ocnaddcache replica changes:

    For example: 

    helm upgrade ocnadd -f ocnadd-custom-values-23.2.0.0.2.yaml ocnadd --namespace ocnadd-deploy
  3. After the Helm upgrade, verify that the ocnaddcache pod replicas have increased, and the pods are in a running state.
Before starting the procedure to upgrade OCNADD, perform the following tasks:

Note:

  • While performing an upgrade, you must align the ocnadd-custom-values-23.3.0.yaml file of the target release as per the ocnadd/values.yaml file of the source release or the older release.
  • Do not enable any new feature during the upgrade.
  • The parent or sub-charts values.yaml must not be changed while performing the upgrade, unless it is explicitly specified in this document.
  • In case the source release already has Kafka External Access Enabled, ensure that procedure mentioned in the section "Helm Install/Upgrade Failure" in "External Kafka Access Enabled" in Oracle Communications Network Analytics Data Director Troubleshooting Guide is run. These changes are mandatory for the external access to work and else the OCNADD upgrade can result in failure

For information about enabling any new feature through Helm parameters, see Oracle Communications Network Analytics Data Director User Guide.

Preupgrade Tasks

  1. Fetch the images and charts of the target release as described in Installing OCNADD.
  2. Keep a backup of the ocnadd-custom-values-23.2.0.yaml file of the source release as a backup while upgrading to target release.
  3. Create or update the secrets for the newly added services in the target release with the same CA. See, Create Secrets for Newly Added Services to create secrets for ocnaddcache and ocnadduirouter services. Skip this step if no new service is added in the target release. See Oracle Network Analytics Suite Release Notes of target release for more information on newly added feature and services.
  4. Update the following helm chart files of the target release with the parameter values of the source release files:
    • update ocnadd/ocdd-db-resource.sql
    • update ocnadd/templates/ocnadd-secret-hook.yaml
    • update custom-templates/ocnadd-custom-values.x.x.x.yaml

    Note:

    • Ensure the parameters such as serviceAccount, Role, RoleBinding are retained from the source release. These are the deployment parameters and should not be modified as part of the upgrade.
    • In release 23.3.0 when the upgrade is performed from the previous 23.2.0.0.x release, make sure that the below parameters are updated and consistent with the previous release, else pods will not get spawned:
      
serviceAccount:
    create: true
    name: <must be same as previous release>
    upgrade: true  ## ->> Update this to 'true', default is false 
clusterRole:
     create: true
     name: <must be same as previous release>
clusterRoleBinding:
     create: true
     name: <must be same as previous release>
  5. Update the pvcClaimSize in the target release ocnadd-custom-values-23.3.0.yaml file.
  6. Ensure to update the "offsetsTopicReplicationFactor" and "transactionStateLogReplicationFactor" in the target release ocnadd-custom-values-23.3.0.yaml file.
  7. Ensure to update the ocnaddcache replicas 'ocnaddcache.replicas' to 3 in the target release ocnadd-custom-values-23.3.0.yaml file.
  8. Take the manual backup of the OCNADD before starting the upgrade procedures. See, Performing OCNADD Manual Backup for taking a manual backup of the OCNADD.

4.3 Upgrade Sequence

The upgrade sequence of the procedures to be followed is described in this section.

4.3.1 Upgrade Order for Source NFs

By design, Kafka clients (producers and consumers) and brokers are bidirectionally compatible:
  • Clients with a higher version of Kafka API can communicate with brokers of a lower version
  • Clients with a lower Kafka API version can communicate with brokers of a higher version
Kafka clients and brokers exchange API version information during a handshake.

Upgrade the source NFs and the OCNADD independently of each other, and no specific upgrade order is required. Upgrade to a new release succeeds if compatibility is maintained. See, "Compatibility Matrix" in the Oracle Communications Network Analytics Suite Release Notes.

The OCNADD upgrade requires less time than source NFs (SCP, SEPP, and NRF) upgrade. It is advisable first to upgrade the OCNADD and verify the traffic flow post upgrade for any significant errors or potential roadblocks in the upgrade. If the NFs are upgraded first, rollback of large numbers of source NFs workers and gateway PODs might be required.

Upgrade Order:

  1. Upgrade OCNADD
  2. Upgrade source NFs (NRF, SCP, and SEPP)

4.3.2 Upgrade Order for CNC Console and cnDBTier

The following upgrade order is recommended for CNC Console and cnDBTier:

  1. For CNC Console upgrade, see Oracle Communications Cloud Native Configuration Console Installation, Upgrade, and Fault Recovery Guide.
  2. For OCNADD upgrade, see Upgrade Sequence.
  3. For cnDBTier upgrade, see Oracle Communications Cloud Native Core cnDBTier Installation, Upgrade, and Fault Recovery Guide.

4.4 Upgrade Impact on Source NFs and Third Party Consumers

Listed below are the observed upgrade impacts on the source NFs and the Third Party Consumers:

  • In the case of a Kafka upgrade, the Kafka clients (NF producers and consumers) should not impact the Kafka API as the API compatibility is maintained between clients and brokers by Kafka.
  • The Kafka binary upgrade is a two step procedure in which the Helm upgrade is performed twice. One upgrade for the Kafka binary upgrade and the second (optional) upgrade if the InterBrokerProtocolVersion is changed. During this upgrade, the source NFs (Kafka producers) may face communication disruption with Kafka brokers multiple times as each broker is expected to restart two times. The producer clients should adopt appropriate reconnection and metadata refresh mechanisms. Suppose the producers run with the 'acks=0' and 'retries=0' configurations. There is no guarantee of reliable message delivery between producers and Kafka brokers during the upgrade, as broker instances restart multiple times.
  • The NF producers must maintain the list of servers in the bootstrap-server parameter instead of a single server in the bootstrap-server parameter.
  • Assume that the OCNADD upgrade is performed at 20% (approximately) of the supported traffic rate. The upgrade is performed using the rolling upgrade strategy. The traffic flow between the NFs and the OCNADD Kafka may remain degraded for a few minutes. The traffic rate is expected to become normal after the upgrade when the NFs producers reconnect to the Kafka brokers.
  • Assume that the OCNADD upgrade is performed at 20% (approximately) of the supported traffic rate. The upgrade is performed using the rolling upgrade strategy. The traffic flow between OCNADD consumer adapters and Third party consumers may remain degraded for a few minutes. The traffic rate is expected to be normal after the upgrade when the Kafka broker pods come up, all the consumer adapter pods have been upgraded, and consumer rebalancing is complete.
  • During the OCNADD upgrade, Kafka retention helps prevent data loss for Third party consumers. However, expect some data duplication towards Third party consumers due to multiple Kafka consumer rebalancing.
  • Plan the OCNADD upgrade during a maintenance window.

4.5 Upgrade Tasks

This section includes information about upgrading an existing OCNADD deployment.

When you attempt to upgrade an existing OCNADD deployment, the running set of containers and pods are replaced with the new set of containers and pods. However, If there is no change in the pod configuration, the running set of containers and pods are not replaced, unless there is a change in the service configuration of a microservice, the service endpoints will remain unchanged (NodePort and so on).

Note:

  • (Optional) A timeout interval of 15 minutes can be set while performing an upgrade as only one POD of the Data Director services is upgraded at a time.
  • Ensure that no OCNADD pod is in the failed state
  • Ensure that the defined in the Preupgrade Tasks are complete
  • There can be a downtime of Kafka brokers for about a minute while performing an upgrade that affects all of the brokers. You can avoid this downtime by upgrading the brokers one at a time, if applicable. Kafka upgrade along with PVC storage changes are not supported.
  • The Consumer Adapter pods/services are created when Data feed is created from OCNADD GUI. Ensure the upgrade of these pods is set to "false" (global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE is set to false). By default, the upgrade of these pods is set to false. To upgrade them, follow the procedure described in "Upgrade Consumer Adapter" section in the Oracle Communications Network Analytics Data Director User Guide .
Run the following command to upgrade an existing OCNADD deployment:
  1. Upgrade the OCNADD microservices:
    • When using the local Helm chart:
      helm upgrade <release_name> -f ocnadd-custom-values-23.3.0.yaml <helm_chart> --namespace <namespace-name> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true --timeout=15m

      where,

      <release_name> is the release name used by the Helm command

      <helm_chart> is the location of the Helm chart extracted from the target ocnadd-<releaseNumber>.tgz file

      <namespace-name> is the OCNADD namespace in which the release is deployed

      For example:

      helm upgrade ocnadd -f ocnadd-custom-values-23.3.0.yaml ocnadd --namespace ocnadd-deploy --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true --timeout=15m
    • When using the chart from Helm repo:
       helm upgrade <release_name> -f ocnadd-custom-values-23.3.0.yaml <helm_repo/helm_chart> --version <chart_version> --namespace <namespace-name> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true --timeout=15m

      where,

      <helm_repo> is the OCNADD Helm repo.

      <chart_version> is the version of the Helm chart extracted from the ocnadd-<releaseNumber>.tgz file

      <namespace-name> is the OCNADD namespace in which the release is deployed

    Note:

    If the upgrade fails, due to time out, the optional –timeout=15 parameter can be increased.

    For example:

    helm upgrade ocnadd -f ocnadd-custom-values-23.3.0.yaml <helm_repo/helm_chart> --version <chart_version> --namespace ocnadd-deploy --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true --timeout=15m
  2. Check the status of the upgrade, monitor the cluster and wait for the traffic rate to be stabilized to same rate before upgrade. Run the command:
    helm history <release_name> --namespace <namespace-name>
    For example:
    helm history ocnadd --namespace ocnadd-deploy

    The description should be "upgrade complete".

  3. Verify if the upgrade is successful using the following steps:
    1. All the pods that have been respawned after upgrade, have the latest age ( in secs )
    2. The Adapter pods also gets respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds.

      In case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

  4. (Optional) Upgrade InterBrokerProtocolVersion in Kafka brokers. This step is required if the Kafka "InterBrokerProtocolVersion" is to be updated in the new release.
    • Update the InterBrokerProtocolVersion in <helm-charts>/charts/ocnaddkafka/values.yaml to the version originally mentioned in the target release charts, and set InterBrokerProtocolVersion to the desired version.
    • Run the command:
      helm upgrade <release_name> -f ocnadd-custom-values-23.3.0.yaml <helm_chart> --namespace <namespace-name> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=false --timeout=15m
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values-23.3.0.yaml ocnadd --namespace ocnadd-deploy --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=false --timeout=15m

      Or, run the following command if helm repo is used:

      helm upgrade <release_name> -f ocnadd-custom-values-23.3.0.yaml <helm_repo/helm_chart> --version <chart_version> --namespace <namespace-name> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=false --timeout=15m
  5. (Optional) To update the SNMP MIBs, follow the section "OCNADD MIB FILES" of the Oracle Communications Network Analytics Data Director User Guide.

4.5.1 Hotfix Upgrade

For a HotFix patch upgrade, follow the steps mentioned in the Upgrade Tasks section.

4.5.2 Create Secrets for Newly Added Services

Follow the steps to generate the secrets for the newly added services in the target release. This step is only needed when any service is introduced in the target release.

  1. Copy <charts>/ssl_certs/demoCA of the source release to <charts>/ssl_certs/demoCA of the target release.
  2. Change the directory to the target release chart location and comment all services configs in ssl_certs/default_values/values.
  3. Add sections for any newly added services for example ocnaddcache and ocnadduirouter service.

    See the sample values below:

    # Do not modify any keys in global section. Please edit only values present in global section.
# Edit only commonName value for Root CA. Do not modify key
# You can add multiple services in same manner as the sample services are added. The format should be as follows
#service name, common name for service and list of subject alternate name
#e.g.,
#[<service_name>]
#commonName=your.svc.common.name
#IP.1 = 127.0.0.1
#IP.2 = 10.72.31.4
#DNS.1 = localhost
#DNS.2 = svc.cluster.local
# Make sure to provide a single empty line (without space) after end of every section
# Do not add comments anywhere in this script to avoid parsing error
  
[global]
countryName=IN
stateOrProvinceName=KA
localityName=BLR
organizationName=ORACLE
organizationalUnitName=CGBU
defaultDays=365
  
##root_ca
commonName=*.ocnadd-deploy.svc.occne-ocdd
  
#[kafka-broker]
#client.commonName=kafka-broker-zk
#server.commonName=kafka-broker
#DNS.1=*.kafka-broker.ocnadd-deploy.svc.occne-ocdd
#DNS.2=kafka-broker
#DNS.3=*.kafka-broker
  
#[zookeeper]
#client.commonName=zookeeper-zk
#server.commonName=zookeeper
#DNS.1=*.zookeeper.ocnadd-deploy.svc.occne-ocdd
#DNS.2=zookeeper
  
#[egw]
#client.commonName=egw-client
#server.commonName=egw
#DNS.1=*egw.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddegressgateway
  
#[ocnaddthirdpartyconsumer]
#client.commonName=ocnaddthirdpartyconsumer-client
#server.commonName=ocnaddthirdpartyconsumer
#DNS.1=*.ocnaddthirdpartyconsumer.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddthirdpartyconsumer
  
#[oraclenfproducer]
#client.commonName=oraclenfproducer
#server.commonName=oraclenfproducer-server
#DNS.1=*.oraclenfproducer.ocnadd-deploy.svc.occne-ocdd
#DNS.2=oraclenfproducer
  
[ocnadduirouter]
client.commonName=ocnadduirouter-client
server.commonName=ocnadduirouter
DNS.1=*.ocnadduirouter.ocnadd-deploy.svc.occne-ocdd
DNS.2=ocnadduirouter
  
#[ocnaddadminservice]
#client.commonName=ocnaddadminservice-client
#server.commonName=ocnaddadminservice
#DNS.1=*.ocnaddadminservice.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddadminservice
  
#[ocnaddalarm]
#client.commonName=ocnaddalarm-client
#server.commonName=ocnaddalarm
#DNS.1=*.ocnaddalarm.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddalarm
  
#[ocnaddconfiguration]
#client.commonName=ocnaddconfiguration-client
#server.commonName=ocnaddconfiguration
#DNS.1=*.ocnaddconfiguration.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddconfiguration
  
#[ocnaddhealthmonitoring]
#client.commonName=ocnaddhealthmonitoring-client
#server.commonName=ocnaddhealthmonitoring
#DNS.1=*.ocnaddhealthmonitoring.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddhealthmonitoring
  
#[ocnaddscpaggregation]
#client.commonName=ocnaddscpaggregation-client
#server.commonName=ocnaddscpaggregation
#DNS.1=*.ocnaddscpaggregation.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddscpaggregation
  
#[ocnaddnrfaggregation]
#client.commonName=ocnaddnrfaggregation-client
#server.commonName=ocnaddnrfaggregation
#DNS.1=*.ocnaddnrfaggregation.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddnrfaggregation
  
#[ocnaddseppaggregation]
#client.commonName=ocnaddseppaggregation-client
#server.commonName=ocnaddseppaggregation
#DNS.1=*.ocnaddseppaggregation.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddseppaggregation
  
#[adapter]
#client.commonName=adapter
#server.commonName=adapter-server
#DNS.1=*adapter.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddconsumeradapter
  
[ocnaddcache]
client.commonName=ocnaddcache-client
server.commonName=ocnaddcache-server
DNS.1=*ocnaddcache.ocnadd-deploy.svc.occne-ocdd
DNS.2=ocnaddcache
  
  
##end
  4. Perform the steps in section Configuring SSL or TLS Certificates to continue generating secrets for newly added services.

    Note:

    During processing of Step 4, while executing the script generate_certs.sh, provide the answer "n" No for the following prompts:
    • >delete existing demoCA folder?
    • >Create new CA?

    For the remaining queries the answer will be same as given during the installation, existing CAs are used to create secrets for ocnaddcache and ocnadduirouter services.