4 Upgrading OCNADD

This section provides information on how to upgrade an existing OCNADD deployment. The section describes the upgrade order for source NFs, CNC Console, cnDBTier, and the upgrade impact on the source NFs.

Note:

The OCNADD can be upgraded from a source release to a target release using CLI procedures as outlined in the following sections. These steps can also be followed for any hotfix upgrade.

4.1 Upgrade Order for Source NFs

By design, Kafka clients (producers and consumers) and brokers are bidirectionally compatible:
  • Clients with a higher version of Kafka API can communicate with brokers of a lower version
  • Clients with a lower Kafka API version can communicate with brokers of a higher version
Kafka clients and brokers exchange API version information during a handshake.

Upgrade the source NFs and the OCNADD independently of each other, and no specific upgrade order is required. Upgrade to a new release succeeds if compatibility is maintained. See, "Compliance Matrix" in the Oracle Communications Network Analytics Suite Release Notes.

The OCNADD upgrade requires less time than source NFs (SCP, SEPP, and NRF) upgrade. It is advisable first to upgrade the OCNADD and verify the traffic flow post upgrade for any significant errors or potential roadblocks in the upgrade. If the NFs are upgraded first, rollback of large numbers of source NFs workers and gateway PODs might be required.

Upgrade Order:

  1. Upgrade OCNADD
  2. Upgrade source NFs (NRF, SCP, and SEPP)

4.2 Upgrade Order for CNC Console and cnDBTier

The following upgrade order is recommended for CNC Console and cnDBTier:

  1. For CNC Console upgrade, see Oracle Communications Cloud Native Configuration Console Installation, Upgrade, and Fault Recovery Guide.
  2. For OCNADD upgrade, see Upgrade Sequence
  3. For cnDBTier upgrade, see Oracle Communications Cloud Native Core cnDBTier Installation, Upgrade, and Fault Recovery Guide

4.3 Upgrade Impact on Source NFs and Third Party Consumers

Listed below are the observed upgrade impacts on the source NFs and the Third Party Consumers:

  • In the case of a Kafka upgrade, the Kafka clients (NF producers and consumers) should not impact the Kafka API as the API compatibility is maintained between clients and brokers by Kafka.
  • The Kafka binary upgrade is a two step procedure in which the Helm upgrade is performed twice. One upgrade for the Kafka binary upgrade and the second (optional) upgrade if the InterBrokerProtocolVersion is changed. During this upgrade, the source NFs (Kafka producers) may face communication disruption with Kafka brokers multiple times as each broker is expected to restart two times. The producer clients should adopt appropriate reconnection and metadata refresh mechanisms. Suppose the producers run with the 'acks=0' and 'retries=0' configurations. There is no guarantee of reliable message delivery between producers and Kafka brokers during the upgrade, as broker instances restart multiple times.
  • The NF producers must maintain the list of servers in the bootstrap-server parameter instead of a single server in the bootstrap-server parameter.
  • Assume that the OCNADD upgrade is performed at 20% (approximately) of the supported traffic rate. The upgrade is performed using the rolling upgrade strategy. The traffic flow between the NFs and the OCNADD Kafka may remain degraded for a few minutes. The traffic rate is expected to become normal after the upgrade when the NFs producers reconnect to the Kafka brokers.
  • Assume that the OCNADD upgrade is performed at 20% (approximately) of the supported traffic rate. The upgrade is performed using the rolling upgrade strategy. The traffic flow between OCNADD consumer adapters and Third party consumers may remain degraded for a few minutes. The traffic rate is expected to be normal after the upgrade when the Kafka broker pods come up, all the consumer adapter pods have been upgraded, and consumer rebalancing is complete.
  • During the OCNADD upgrade, Kafka retention helps prevent data loss for Third party consumers. However, expect some data duplication towards Third party consumers due to multiple Kafka consumer rebalancing.
  • Plan the OCNADD upgrade during a maintenance window.

4.4 Upgrade Sequence

The upgrade sequence of the procedures to be followed and the supported upgrade path is described in this section.

Table 4-1 Upgrade Sequence

Sequence Upgrade Task
1 Preupgrade Tasks
2 OCNADD Upgrade

Table 4-2 Supported Upgrade Path

Source Release Target Release
23.2.0 23.2.0.0.3
23.2.0.0.2 23.2.0.0.3

4.5 Preupgrade Tasks

Before starting the procedure to upgrade OCNADD, perform the following tasks:

Caution:

Before starting the upgrade make sure to update the ocnaddcache replicas on source release. Follow the below steps to update the ocnaddcache replicas:

  • Update the ocnaddcache replicas 'ocnaddcache.replicas' to 3 in the source release ocnadd-custom-values.yaml file and the "ocnadd/charts/ocnaddcache/values.yaml" file.
  • Perform helm upgrade to apply the ocnaddcache replica changes. For example: For 23.2.0.0.2 release:
    helm upgrade ocnadd -f ocnadd-custom-values-23.2.0.0.2.yaml ocnadd --namespace ocnadd-deploy
  • Verify the ocnaddcache pod replicas are increased, and pods are in running state.

Note:

  • While performing an upgrade, you must align the ocnadd-custom-values.yaml file of the target release and the source release.
  • Do not enable any new feature during the upgrade, unless it is explicitly mentioned.
  • The parent or parameters in sub-charts values.yaml must not be changed while performing the upgrade unless it is explicitly specified.
  • In case the source release already has Kafka External Access Enabled, ensure the following changes:
    • Edit the "ocnadd/templates/ocnadd-rbac.yaml" file.
    • Change automountServiceAccountToken to true.

Traffic through the Kafka 9092 port is disabled. The external access via ClusterIP is no longer supported

For information about enabling any new feature through Helm parameters, see Oracle Communications Network Analytics Data Director User Guide.

Create Secrets for Newly Added Services

Follow the steps to generate the secrets for the newly added services in the target release, skip this step if there no new services introduced in target release:

  1. Copy the <charts>/ssl_certs/demoCA in the source release to the <charts>/ssl_certs/demoCA in the target release.
  2. Change directory to the target release chart location and comment all services configs in ssl_certs/default_values/values.
  3. Add sections for newly added services such as ocnaddcache and ocnadduirouter service. See the sample values below:
    # Do not modify any keys in global section. Please edit only values present in global section.
# Edit only commonName value for Root CA. Do not modify key
# You can add multiple services in same manner as the sample services are added. The format should be as follows
#service name, common name for service and list of subject alternate name
#e.g.,
#[<service_name>]
#commonName=your.svc.common.name
#IP.1 = 127.0.0.1
#IP.2 = 10.72.31.4
#DNS.1 = localhost
#DNS.2 = svc.cluster.local
# Make sure to provide a single empty line (without space) after end of every section
# Do not add comments anywhere in this script to avoid parsing error
 
[global]
countryName=IN
stateOrProvinceName=KA
localityName=BLR
organizationName=ORACLE
organizationalUnitName=CGBU
defaultDays=365
 
##root_ca
commonName=*.ocnadd-deploy.svc.occne-ocdd
 
#[kafka-broker]
#client.commonName=kafka-broker-zk
#server.commonName=kafka-broker
#DNS.1=*.kafka-broker.ocnadd-deploy.svc.occne-ocdd
#DNS.2=kafka-broker
#DNS.3=*.kafka-broker
 
#[zookeeper]
#client.commonName=zookeeper-zk
#server.commonName=zookeeper
#DNS.1=*.zookeeper.ocnadd-deploy.svc.occne-ocdd
#DNS.2=zookeeper
 
#[egw]
#client.commonName=egw-client
#server.commonName=egw
#DNS.1=*egw.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddegressgateway
 
#[ocnaddthirdpartyconsumer]
#client.commonName=ocnaddthirdpartyconsumer-client
#server.commonName=ocnaddthirdpartyconsumer
#DNS.1=*.ocnaddthirdpartyconsumer.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddthirdpartyconsumer
 
#[oraclenfproducer]
#client.commonName=oraclenfproducer
#server.commonName=oraclenfproducer-server
#DNS.1=*.oraclenfproducer.ocnadd-deploy.svc.occne-ocdd
#DNS.2=oraclenfproducer
 
[ocnadduirouter]
client.commonName=ocnadduirouter-client
server.commonName=ocnadduirouter
DNS.1=*.ocnadduirouter.ocnadd-deploy.svc.occne-ocdd
DNS.2=ocnadduirouter
 
#[ocnaddadminservice]
#client.commonName=ocnaddadminservice-client
#server.commonName=ocnaddadminservice
#DNS.1=*.ocnaddadminservice.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddadminservice
 
#[ocnaddalarm]
#client.commonName=ocnaddalarm-client
#server.commonName=ocnaddalarm
#DNS.1=*.ocnaddalarm.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddalarm
 
#[ocnaddconfiguration]
#client.commonName=ocnaddconfiguration-client
#server.commonName=ocnaddconfiguration
#DNS.1=*.ocnaddconfiguration.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddconfiguration
 
#[ocnaddhealthmonitoring]
#client.commonName=ocnaddhealthmonitoring-client
#server.commonName=ocnaddhealthmonitoring
#DNS.1=*.ocnaddhealthmonitoring.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddhealthmonitoring
 
#[ocnaddscpaggregation]
#client.commonName=ocnaddscpaggregation-client
#server.commonName=ocnaddscpaggregation
#DNS.1=*.ocnaddscpaggregation.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddscpaggregation
 
#[ocnaddnrfaggregation]
#client.commonName=ocnaddnrfaggregation-client
#server.commonName=ocnaddnrfaggregation
#DNS.1=*.ocnaddnrfaggregation.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddnrfaggregation
 
#[ocnaddseppaggregation]
#client.commonName=ocnaddseppaggregation-client
#server.commonName=ocnaddseppaggregation
#DNS.1=*.ocnaddseppaggregation.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddseppaggregation
 
#[adapter]
#client.commonName=adapter
#server.commonName=adapter-server
#DNS.1=*adapter.ocnadd-deploy.svc.occne-ocdd
#DNS.2=ocnaddconsumeradapter
 
[ocnaddcache]
client.commonName=ocnaddcache-client
server.commonName=ocnaddcache-server
DNS.1=*ocnaddcache.ocnadd-deploy.svc.occne-ocdd
DNS.2=ocnaddcache
 
 
##end
  4. Perform the steps in section Configuring SSL or TLS Certificates to continue generating secrets for newly added services.

    Note:

    During execution of Step 4, while executing the script generate_certs.sh, provide the answer No for the following prompts:
    • >delete existing demoCA folder?
    • >Create new CA?

    For the remaining queries the answer will be same as given during the installation, existing CAs are used to create secrets for ocnaddcache and ocnadduirouter services.

Preupgrade Tasks

  1. Fetch the images and charts of the target release as described in Installing OCNADD.
  2. Keep a backup of the ocnadd-custom-values.yaml file of the source release as a backup while upgrading to target release.
  3. Create or update the secrets for the newly added services in the target release with the same CA. See, Create Secrets for Newly Added Services, skip this step if there no new services introduced in target release.
  4. Update the following helm chart files of the target release with the parameter values of the source release files:
    • ocnadd-custom-values.yaml

      Note:

      Ensure the parameters such as serviceAccount, clusterRole, clusterRoleBinding are not updated and their values are retained from the source release. These are the deployment parameters and should not be modified as part of the upgrade.

      However, in release 23.2.0, these parameters are controlled by custom value parameters, and in prior releases, the parameters name was taken as the namespace name itself. So the following parameters must be updated with the namespace name in the ocnadd-custom-values.yaml at the time of upgrade to avoid service account permission related errors post upgrade to release 23.2.0. 

      
serviceAccount:
    create: true
    name: <namespace_name_used_in_prior_release>
    upgrade: true                                    ## --->> Update this to 'true', default is false
clusterRole:
     create: true
     name: <namespace_name_used_in_prior_release>
clusterRoleBinding:
     create: true
     name: <namespace_name_used_in_prior_release>

      Post upgrade to 23.2.0, the subsequent upgrade of OCNADD will not require changes in any of the mentioned parameters during the upgrade.

    • ocnadd/ocdd-db-resource.sql
    • ocnadd/templates/ocnadd-secret-hook.yaml
    • Update the pvcClaimSize in ocnadd-custom-values.yaml file
    • Ensure to update the ocnaddcache replicas 'ocnaddcache.replicas' to 3 in the target release ocnadd-custom-values.yaml, and the "ocnadd/charts/ocnaddcache/values.yaml" file replicas 'ocnaddcache.replicas' to 3.
    • Ensure to update the "offsetsTopicReplicationFactor" and "transactionStateLogReplicationFactor" in the target release ocnadd-custom-values.yaml from the source release.
  5. Take the manual back of the OCNADD before starting the upgrade procedures, see, Performing OCNADD Manual Backup for taking a manual backup of the OCNADD.

4.6 OCNADD Upgrade

This section includes information about upgrading an existing OCNADD deployment.

When you attempt to upgrade an existing OCNADD deployment, the running set of containers and pods are replaced with the new set of containers and pods. However, If there is no change in the pod configuration, the running set of containers and pods are not replaced, unless there is a change in the service configuration of a microservice, the service endpoints will remain unchanged (NodePort and so on).

Important:

  • (Optional) A timeout interval of 15 minutes can be set while performing an upgrade as only one POD of the Data Director services is upgraded at a time.
  • Ensure that no OCNADD pod is in the failed state
  • Ensure that the defined in the Preupgrade Tasks are complete
  • There can be a downtime of Kafka brokers for about a minute while performing an upgrade that affects all of the brokers. You can avoid this downtime by upgrading the brokers one at a time, if applicable. Kafka upgrade along with PVC storage changes are not supported.
  • The Consumer Adapter pods/services are created when Data feed is created from OCNADD GUI. Ensure the upgrade of these pods is set to "false" (global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE is set to false). By default, the upgrade of these pods is set to false. To upgrade them, follow the procedure described in "Upgrade Consumer Adapter" section in the Oracle Communications Network Analytics Data Director User Guide .
Run the following procedure to upgrade an existing OCNADD deployment:
  1. Upgrade the OCNADD microservices:
    • When using the local Helm chart:
      helm upgrade <source-release-name> -f ocnadd-custom-values.yaml --namespace <source-release-namespace> <target-release-helm-chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true

      where,

      <source-release-name> is the release name of the source release deployment

      ocnadd-custom-values.yaml is the updated custom values file of the target release

      <source-release-namespace> is the OCNADD namespace of the source release

      <target-release-helm-chart> is the helm chart folder of the target-release

      For example:
      helm upgrade ocnadd -f ocnadd-custom-values_23.2.0.0.3.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true

    Note:

    If the upgrade fails, due to time out, the optional –timeout=15 parameter can be increased.
  2. Check the status of the upgrade, monitor the cluster and wait for the traffic rate to be stabilized to same rate before upgrade. Run the following command:
    helm history <release_name> --namespace <namespace-name>
    For example:
    helm history ocnadd --namespace ocnadd-deploy

    The description status be "upgrade complete" and the current APP VERSION can be seen as "23.2.0.0.3".

  3. Verify if the upgrade is successful using the following steps:
    1. All the pods that have been respawned after upgrade, have the latest age ( in secs )
    2. The Adapter pods also gets respawned for any upgrade. The status can also be verified from GUI for respective Data Feeds.

      In case of any failure, follow the steps mentioned in the Oracle Communications Network Analytics Data Director Troubleshooting Guide.

  4. <Optional> Upgrade InterBrokerProtocolVersion in Kafka brokers. This step is required if the "InterBrokerProtocolVersion" is changed in Kafka.
    • Update the InterBrokerProtocolVersion in <helm-charts>/charts/ocnaddkafka/values.yaml to the version originally mentioned in the target release charts, set InterBrokerProtocolVersion to 3.4.0 in target release 23.2.0.
    • Run the command:
      helm upgrade <release_name> -f ocnadd-custom-values.yaml --namespace <namespace-name> <helm_chart> --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=false --timeout=15m
      For example:
      helm upgrade ocnadd -f ocnadd-custom-values_23.2.0.0.3.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=false --timeout=15m
  5. <Optional> Post upgrade, if you want to enable intraTLS, perform the following steps:
    1. Edit the ocnadd-custom-values-23.2.0.0.3.yaml file and make the following changes:
      1. global.ssl.intraTlsEnabled ## --> Change the value to true
      2. Go to ocnadduirouter.ocnadduirouter.env section and update the below parameters:
        
         DD_CONFIG_API: http://ocnaddconfiguration:12590                     ## --> Change http to https
         DD_ALARM_API: http://ocnaddalarm:9099                               ## --> Change http to https
         DD_HEALTH_API: http://ocnaddhealthmonitoring:12591                  ## --> Change http to https
    2. Run helm upgrade command:

      For example

      helm upgrade ocnadd -f ocnadd-custom-values-23.2.0.0.3.yaml --namespace ocnadd-deploy ocnadd --set global.env.admin.OCNADD_ADAPTER_UPGRADE_ENABLE=true --timeout=15m

    Note:

    For more information on intraTLS, see Oracle Communications Network Analytics Suite Security Guide.
  6. Follow the steps mentioned in the Post Deployment section.

4.6.1 Hotfix Upgrade

For a HotFix patch upgrade, follow the steps mentioned in the OCNADD Upgrade section.