5 Upgrading CNC Console

This section provides details of CNC Console upgrade procedure to upgrade the following CNC Console components:
  • M-CNCC IAM
  • M-CNCC Core
  • A-CNCC Core

Note:

For OCI

Note:

Before proceeding with CNC Console helm upgrade to latest:
  • M-CNCC IAM DB Backup: CNC Console database backup must be taken and stored in a location which can be easily restored. For more information, see CNC Console IAM DB Backup.
  • Support for Dual Stack Networking: Update occncc_custom_values<version>.yaml file and set cnccDeploymentMode flag to the desired value. CNC Console services must be backed up before performing CNC Console Upgrade or Rollback. Services should be deleted and reapplied with preferred IP Family Policy and IP Families. For more information, see Preupgrade Procedure for Dual Stack Networking.

The following steps must be followed while performing the upgrade:

  1. CNCC IAM DB Backup
  2. CNCC Upgrade

This section explains about the CNC Console upgrade procedures for single cluster and multicluster deployments. M-CNCC IAM, M-CNCC Core, and A-CNCC Core can be upgraded from current version to the latest version using helm upgrade feature.

User can upgrade and rollback CNC Console from a source release to a target release using CLI procedures as outlined in the following table:

Upgrade References Non-OCI Environment OCI Environment
Applicable for CLI Applicable for CNC Console deployment using OCI
Supported Upgrade Paths Supported Upgrade Paths Yes Yes
Preupgrade Tasks Preupgrade Tasks Yes Yes
M- CNCC IAM DB Backup CNCC IAM DB Backup Yes No
CNC Console Upgrade CNCC Upgrade Yes Yes
Enabling Automated Certificate Lifecycle Management through OCCM during Console Upgrade Enabling Automated Certificate Lifecycle Management through OCCM during Console Upgrade Yes Yes
CNC Console Upgrade CNC Console Upgrade Yes Yes

Caution:

It is recommended to verify the copy pasted content especially when the hyphens or any special characters are part of copied content.

5.1 Supported Upgrade Paths

This section describes the supported upgrade paths for CNC Console

For more information about supported upgrade paths, see CNC Console Deployment Modes section.

The following table lists the supported upgrade paths for Console:

Table 5-1 CNC Console Upgrade Sequence

Deployment Mode Source Version Target Version Upgrade Sequence
Single Cluster 24.1.x or 24.2.x 24.3.x

Console Upgrade

Upgrade CNC Console

NF Upgrade

Upgrade Instances (NF or CNE/OSO Common Services)

cnDbtier Upgrade
Multi Cluster 24.1.x or 24.2.x 24.3.x

Console Manager Upgrade

Upgrade CNC Console

Console Agent Upgrade

Upgrade A-CNCC Core

NF Upgrade

Upgrade Instances (NF or CNE/OSO Common Services)

cnDbtier Upgrade

Note:

For OCI
  • Starting from version 24.1.0, CNC Console supports deployment on OCI.
  • CNC Console multicluser feature is not supported.

Note:

  • Console should be upgraded first followed by NF for all the supported NFs.
  • cnDBTier only supports APIs from 23.3.x onwards. Users with a combination of cnDBTier 23.3.x or older, and NF version 23.4.x or newer won't have access to the data in cnDBTier configuration items.

Note:

CNC Console supports N-2 NF versions during upgrade window. For example, CNC Console 24.3.0 supports SCP 24.3.0, 24.2.x, and 24.1.x.

Any newly added features in Console which have NF dependency in latest release may not be available in previous release.

5.2 Preupgrade Tasks

While upgrading an existing CNC Console deployment, the running set of containers and pods are replaced with the new set of containers and pods. However, if there is no change in the pod configuration, the running set of containers and pods are not replaced.

Caution:

  • No configuration should be performed during upgrade.
  • Do not exit from helm upgrade command manually. After running the helm upgrade command, it takes some time (depending upon number of Pods to upgrade) to upgrade all the services. In the meantime, you must not press "ctrl+c" to come out from helm upgrade command. It may lead to anomalous behavior.

Preupgrade Steps

  1. Keep current occncc_custom_values_<version>.yaml file as backup, that is 
    occncc_custom_values_<version to be upgraded>.yaml
  2. Update the new custom_values.yaml defined for target CNC Console release. See CNC Console IAM Configuration Parameters section and CNC Console Core Configuration Parameters section for more details about helm configurable parameters.
  3. Install or upgrade the network policies, if applicable. For more information, see Configuring Network Policies.

5.2.1 CNC Console IAM DB Backup

This section describes the procedure to do the DB backup of CNC Console IAM.

Note:

Not applicable for OCI deployment.

Prerequisites

The prerequisites for the backup process are:

  • MySQL NDB cluster should be in a healthy state.
  • Every database node of the MySQL NDB cluster should be in running state.
  • In case of cnDBTier to verify the prerequisites, check mysql pod is up and running.
  • In case of VM based DB Tier to verify the prerequisites, log in to MCM client on one of the SQL node of the cluster. Run the following command to check the node status:
    mcm> show status -r occnendbclustera;
  • Log in to the SQL node and run the following command to take the dump (backup) of the database. The user is required to enter the password.
    kubectl exec -i -n <namespace> <sql-node> -- mysqldump --single-transaction --no-tablespaces --no-create-info -h 127.0.0.1 -u <username> -p <database-name> | gzip > <backup_filename>.sql.gz

    For example:

    kubectl exec -i -n occne-ndb ndbappmysqld-0 -- mysqldump --single-transaction --no-tablespaces --no-create-info
      -h 127.0.0.1 -u cnccusr -p cnccdb | gzip > cnccdbBackup.sql.gz

Note:

The operator must ensure there is enough space on the current directory in order to save the backup file.

5.2.2 Preupgrade Procedure for Dual Stack Networking

Note:

Not applicable for OCI deployment.
The following table outlines the necessary changes to be made before upgrading CNC Console with the cnccDeploymentMode configured in the occncc_custom_values<version>.yaml file. Refer to the following table to upgrade CNC Console to any of the supported modes:

Note:

Refer to the Updating CNC Console Services Before Performing Upgrade section before performing upgrade.

Table 5-2 cnccDeploymentMode Configuration

Cluster DeploymentMode or Dual Stack preferred infrastructure Previously deployed CNC Console IP Family Policy "cnccDeploymentMode" configured in CNC Console custom values.yaml Service configuration with preferred IP Family Policy and IP Families (Pre Upgrade Step)
Dual Stack IPv4 preferred infrastructure

SingleStack IPv4 preferred

Having IP Families:
spec:
  ....
  ipFamilies:
  - IPv4
 ClusterPreferred This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv4 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv6 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv6
IPv4_IPv6 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv6_IPv4 
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv6
  - IPv4

SingleStack IPv6 preferred

Having IP Families:
spec:
  ....
  ipFamilies:
  - IPv6
 ClusterPreferred 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv4
IPv4 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv4
IPv6 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv4_IPv6 
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv4
  - IPv6
IPv6_IPv4 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.

RequireDualStack IPv4 preferred

Having IP Families:
spec:
  ....
  ipFamilies:
  - IPv4_IPv6
 ClusterPreferred 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv4
IPv4 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv6 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv6
IPv4_IPv6 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv6_IPv4 
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv6
  - IPv4

RequireDualStack IPv6 preferred

Having IP Families: 
spec:
  ....
  ipFamilies:
  - IPv6_IPv4
 ClusterPreferred 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv4
IPv4 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv4
IPv6 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv4_IPv6 
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv4
  - IPv6
IPv6_IPv4 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.

Updating CNC Console Services Before Performing Upgrade

This section describes the preupgrade procedure to be performed in case of Dual Stack Networking.

  1. List of console services to be verified and backed up for recovery process.
    1. Run the following commant to get the services that must be verified and backed up before performing CNC Console Upgrade:
      $ kubectl get svc -n <namespace>
      For example: 
      $ kubectl get svc -n cncc
NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)             AGE
cncc-acore-igw-cache         ClusterIP      None            <none>          8000/TCP            3m19s
cncc-acore-ingress-gateway   ClusterIP      10.233.xx.xxx   <none>          80/TCP              3m19s
cncc-iam-igw-cache           ClusterIP      None            <none>          8000/TCP            3m19s
cncc-iam-ingress-gateway     LoadBalancer   10.233.xx.xxx   10.xx.xxx.xxx   80:32007/TCP        3m19s
cncc-iam-kc-headless         ClusterIP      None            <none>          8285/TCP            3m19s
cncc-iam-kc-http             ClusterIP      10.233.xx.xx    <none>          8285/TCP,8443/TCP   3m19s
cncc-mcore-cmservice         ClusterIP      10.233.xx.xxx   <none>          8442/TCP            3m19s
cncc-mcore-igw-cache         ClusterIP      None            <none>          8000/TCP            3m19s
cncc-mcore-ingress-gateway   LoadBalancer   10.233.xx.xxx   10.xx.xxx.xxx   80:30609/TCP        3m19s
 
 
# Note: The following services have 'cncc' as the release name, which needs to be verified and backed up.      
cncc-acore-ingress-gateway 
cncc-iam-ingress-gateway           
cncc-iam-kc-http           
cncc-mcore-cmservice               
cncc-mcore-ingress-gateway
  
# These services do not have any clusterIP assigned and hence can be ignored.
cncc-acore-igw-cache
cncc-mcore-igw-cache
cncc-iam-igw-cache
cncc-iam-kc-headless
    2. Run the following command to verify the service:
      $ kubectl get svc <service-name> -n <namespace>
      For example:
      kubectl get svc cncc-iam-ingress-gateway -n cncc
NAME                     TYPE         CLUSTER-IP    EXTERNAL-IP   PORT(S)       AGE
cncc-iam-ingress-gateway LoadBalancer 10.xx.xx.xx   10.xx.xx.xx   80:30837/TCP  2m35s
    3. Run the following command to take a backup of service and duplicate it for recovery:
      # Command to take backup of service
$ kubectl get svc <service-name> -n namespace -o yaml > <service-name>.yaml;
 
# Command to duplicate service for recovery purpose
$ cp <service-name>.yaml <backup_file_name>.yaml;
      For example:
      $ kubectl get svc cncc-iam-ingress-gateway -n cncc -o yaml > cncc-iam-ingress-gateway.yaml;
$ cp cncc-iam-ingress-gateway.yaml cncc-iam-ingress-gateway_ipv4.yaml;
  2. Run the following command to delete the existing console services:
    $ kubectl delete svc <service-name> -n <namespace>
    For example:
    $ kubectl delete svc cncc-iam-ingress-gateway -n cncc
  3. Update the service file <service-name>.yaml generated as part of step 1.c.
    1. Run the following command to edit <service-name>.yaml:
      $ vim <service-name>.yaml
      For example:
      $ vim cncc-iam-ingress-gateway.yaml
    2. Update the following in <service-name>.yaml:
      1. Delete clusterIP and clusterIPs fields completely, and loadBalancerIP field if present.

        Delete the metallb.universe.tf/loadBalancerIPs annotation which would have been added only in case of IPv4_IPv6 or IPv6_IPv4 modes.

      2. Under spec.ipFamilyPolicy set SingleStack/RequiredDualStack and under spec.ipFamilies set [-IPv4, -IPv6, - IPv4 - IPv6, -IPv6 -IPv4] as per the cnccDeploymentMode configured.
  4. Run the following command to apply the updated <service-name>.yaml file:
    $ kubectl apply -f <service-name>.yaml -n <namespace>
    For example:
    $ kubectl apply -f cncc-iam-ingress-gateway.yaml -n cncc
  5. Run the following command to verify newly created service:
    kubectl get svc <service-name> -n <namespace>
    For example:
    $ kubectl get svc cncc-iam-ingress-gateway -n cncc
NAME                       TYPE           CLUSTER-IP         EXTERNAL-IP                           PORT(S)          AGE
cncc-iam-ingress-gateway   LoadBalancer   fd00:0:0:2::702b   10.xx.xx.xx,xxxx:b400:605:xxxx::a     80:30837/TCP     60s

Sample Configuration Changes to Upgrade CNC Console From IPv4 to IPv6_IPv4 on Dual Stack With IPv4 Preferred Infrastructure

The following example shows the configuration changes to be done before upgrading CNC Console From IPv4 to IPv6_IPv4 on Dual Stack With IPv4 Preferred infrastructure for single service of CNC Console. You must follow a similar procedure for rest of the CNC Console services.

Deployment details:
  • CNC Console deployment namespace: cncc
  • CNC Console release name: cncc
  • Infra Deployment Mode: DualStack with IPv4 Preferred
  • CNC Console Deployment Mode for upgrade ("cnccDeploymentMode") : IPv6_IPv4
  • CNC Console service to be edited: cncc-iam-ingress-gateway
  1. Get the cncc-iam-ingress-gateway service, verify it, and back it up for recovery
    1. Run the following command to get the cncc-iam-ingress-gateway service:
      $ kubectl get svc cncc-iam-ingress-gateway -n cncc
    2. Run the following command to verify cncc-iam-ingress-gateway service is set to IPv4 address:
      $ kubectl get svc cncc-iam-ingress-gateway -n cncc
  
NAME                       TYPE           CLUSTER-IP    EXTERNAL-IP     PORT(S)        AGE
cncc-iam-ingress-gateway   LoadBalancer   10.xx.xx.xx   10.xx.xx.xx     80:30837/TCP   2m35s
    3. Run the following command to take the backup of cncc-iam-ingress-gateway service and duplicate it for recovery:
      $ kubectl get svc cncc-iam-ingress-gateway -n cncc -o yaml > cncc-iam-ingress-gateway.yaml;
  
$ cp cncc-iam-ingress-gateway.yaml cncc-iam-ingress-gateway_ipv4.yaml;
  2. Run the following command to delete the existing cncc-iam-ingress-gateway service:
    $ kubectl delete svc cncc-iam-ingress-gateway -n cncc
  3. Run the command to update following in the cncc-iam-ingress-gateway.yaml: 
    vim cncc-iam-ingress-gateway.yaml
    1. Delete clusterIP and clusterIPs fields completely and loadBalancerIP field if present:
      # Remove the clusterIP and clusterIPs field and loadBalancerIP field if present
...
spec:
  ...
  clusterIP: 1x.xxx.xx.xx
  clusterIPs:
  - 1x.xxx.xx.xx
  ...
  ...
  loadBalancerIP: 10.xx.xx.xx
...
    2. Under spec.ipFamilyPolicy set RequiredDualStack and under spec.ipFamilies set [- IPv6, - IPv4] as follows:
      apiVersion: v1
kind: Service
metadata:
  ...
  name: cncc-iam-ingress-gateway
  namespace: cncc
spec:
  allocateLoadBalancerNodePorts: true
  ...
  ...
  ipFamilies:
  - IPv6
  - IPv4
  ipFamilyPolicy: RequireDualStack
  ...
  ...
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}
  4. Run the following command to apply the cncc-iam-ingress-gateway.yaml file:
    $ kubectl apply -f cncc-iam-ingress-gateway.yaml -n cncc
  5. Run the following command to check the service output. You should find the service with updated the IPv4 and IPv6 addresses:
    $ kubectl get svc cncc-iam-ingress-gateway -n cncc
 
NAME                       TYPE           CLUSTER-IP         EXTERNAL-IP                           PORT(S)          AGE
cncc-iam-ingress-gateway   LoadBalancer   fd00:0:0:2::702b   10.xx.xx.xx,xxxx:b400:605:xxxx::a     80:30837/TCP     60s

5.3 Enabling Automated Certificate Lifecycle Management through OCCM during Console Upgrade

This section provides details on Console upgrade procedure required to enable automated certificate lifecycle management through OCCM for Console.

Note:

This is an optional step which is required to enable OCCM support.

Prerequisites

The following prerequisites must be completed in advance:
  • OCCM Instance must be configured with CNC Console.
  • Certificate creation configuration for OCCM must be done by the operator. For more information, see the "Managing Issuers" and "Managing Certificates" sections in the Oracle Communications Cloud Native Core, Certificate Management User Guide.
  • CNC Console should be deployed with manually created TLS certificates.
  • Operator must reuse the same Kubernetes secret, PrivateKey Name, and Certificate Name as used by CNC Console with manually generated key and certificate. While reusing the existing Kubernetes secret, ensure that the CNC Console custom values configuration are not updated, and kept as it is to match with the existing Kubernetes secret.

Upgrade Procedure to automate certificate lifecycle management through OCCM (Oracle Communications Certificate Manager)

Preupgrade Task
Log in to the deployment cluster and run the following command to take the backup of existing cncc-iam-ingress-secret and cncc-core-ingress-secret:
 $ kubectl get secret <cncc-secret-name> -n <namespace> -o yaml > <secret-backup-file-name>.yaml
For example:
 $ kubectl get secret cncc-iam-ingress-secret -n cncc -o yaml > cncc-iam-secret-backup.yaml

Note:

For OCI Deployment

cncc-iam-ingress-secret is not applicable for oci deployments.

Upgrade CNC Console from Manual Certificate Management to Automated Certificate Management with Same Certificate Authority and Existing Key and Certificate

Note:

ingressGwCertReloadEnabled parameter must be enabled or disabled in all the three sections (cncc-iam.global.ingressGwCertReloadEnabled, mcncc-core.global.ingressGwCertReloadEnabled, and acncc-core.global.ingressGwCertReloadEnabled).

For more information, see M-CNCC IAM, M-CNCC Core, and A-CNCC Core Dynamic reloading of certificates sections under Configuring Secrets for Enabling HTTPS.

This scenario is applicable when the user uses the same secret and manually created certificate and private key. Perform the following procedure to upgrade CNC Console from manual certificate management to automated certificate management:

  1. Set ingressGwCertReloadEnabled to true. To perform CNC Console upgrade, see Upgrading CNC Console.
  2. Create the required CNC Console certificates using OCCM. For more information, see "Introducing OCCM in an Existing NF Deployment" and "Creating NF Certificate Using OCCM - Sample Configuration" sections in the Oracle CommunicationsCloud Native Core, Certificate Management User Guide.
  3. Monitor certificate created by OCCM in Grafana.
Upgrade CNC Console from Manual Certificate Management to Automated Certificate Management with Same Certificate Authority and New Key and Certificate

This Scenario is applicable when user uses the existing secret and new key and certificate created by the OCCM. Perform the following procedure to upgrade CNC Console from manual certificate management to automated certificate management:

  1. Set ingressGwCertReloadEnabled to true. To perform CNC Console upgrade, see Upgrading CNC Console.
  2. Create the required CNC Console certificates using OCCM. For more information, see "Introducing OCCM in an Existing NF Deployment" and "Creating NF Certificate Using OCCM - Sample Configuration" sections in the Oracle CommunicationsCloud Native Core, Certificate Management User Guide.
  3. Monitor certificate created by OCCM in Grafana.
Upgrade CNC Console from Manual Certificate Management to Automated Certificate Management with Different Certificate Authority and New Key and Certificate

This Scenario is applicable when user uses the existing secret and new key and certificate created by the OCCM from an issuer other than the issuer for manually created certificates. Perform the following procedure to upgrade CNC Console from manual certificate management to automated certificate management:

  1. Set ingressGwCertReloadEnabled to false. To perform CNC Console upgrade, see Upgrading CNC Console.
  2. Create the required certificates (cncc-iam-ingress-secret and cncc-core-ingress-secret) using OCCM. For more information, see"Creating NF Certificate Using OCCM - Sample Configuration" section in the Oracle CommunicationsCloud Native Core, Certificate Management User Guide.
  3. Set ingressGwCertReloadEnabled to true. To perform CNC Console upgrade, see Upgrading CNC Console.
  4. Monitor newly created certificate by OCCM in Grafana.

Note:

Restoring old certificates in case of any issue with certificate creation by OCCM:

In case of any failure in certificate creation by OCCM, use secret backup files for restoring the older certificates.

Perform the following steps:

  1. Delete the existing secret if exists.
  2. Run the following command to create the secret from the backup file: 
    $ Kubectl apply -f <secret-backup-file-name>.yaml
    Example:
    $ kubectl apply -f cncc-iam-secret-backup.yaml

5.3.1 Moving Back to Manual Certificate Management

For details about moving back to manual certificate management, see "Moving Back to Manual Certificate Management" section in the Oracle CommunicationsCloud Native Core, Certificate Management User Guide.

5.4 CNC Console Upgrade

This section describes the procedure to upgrade CNC Console.

Note:

  • Existing release name must be used for upgrade.
  • The mCnccIams.port configuration must be added only if port is other than default ones 80 or 443.
  1. Prepare occncc_custom_values_<version>.yaml file for upgrade.
  2. Upgrade CNC Console using existing release release name (cncc-iam or cncc) by running the following command: 
    $ helm upgrade <cncc_iam_release_name> <helm_chart> -f <occncc_custom_values_<version>.yaml> -n <namespace>

    Foe example:
    $ helm upgrade cncc ocspf-helm-repo/cncc -f occncc_custom_values_<version>.yaml -n cncc
  3. Check the status of upgrade by running the following command:
    $ helm status <cncc_iam_release_name> -n <namespace>
    For example:
    $ helm status cncc -n cncc

5.5 Parameters and Definitions During CNC Console Upgrade

Parameters and Definitions during CNC Console Upgrade

Table 5-3 Parameters and Definitions during CNC Console Upgrade

Parameters Definitions
<release_name> CNC Console Helm release deployed. It could be found in the output of 'helm list' command
<namespacename> CNC Console namespace in which release deployed
<helm_chart> CNC Console helm chart
<chart_version> CNC Console helm chart version in case helm charts are referred from helm repo
<helm_repo> CNC Console helm repo
<occncc_custom_values_<version>.yaml>

CNC Console customized values.yaml for target release.
<occncc_rollback_iam_schema_<version>.sql> CNC Console DB schema file used for rollback