4 Upgrading SEPP

This chapter provides information about upgrading Oracle Communications Cloud Native Core, Security Edge Protection Proxy (SEPP) deployment to the latest release. It is recommended to perform SEPP upgrade in a specific order. For more information about the upgrade order, see Oracle Communications Cloud Native Core, Solution Upgrade Guide.

The SEPP upgrade procedures for single cluster and multicluster deployments. SEPP instance can be upgraded from current version to the latest version using Helm upgrade command.

Following steps must be performed while upgrading the SEPP:

  1. SEPP DB Backup
  2. SEPP Upgrade

4.1 Supported Upgrade Paths

The following table lists the supported upgrade paths for SEPP:

Note:

  • SEPP delpoyed in SEPP mode can be upgraded only to SEPP mode.
  • SEPP deployed in Roaming Hub mode can be upgraded only to Roaming Hub mode.
  • Before proceeding with SEPP helm upgrade in this deployment model, SEPP instance database backup must be taken and stored in a location which can be easily restored. Refer SEPP Instance DB Backup section for more information.

Table 4-1 Supported Upgrade Paths

Source Release Target Release
25.1.2xx 25.2.100
25.1.1xx 25.2.100

Table 4-2 Supported Upgrade Paths

Source Roaming Hub Release Target Roaming Hub Release
25.1.2xx 25.2.100
25.1.1xx 25.2.100

Note:

  • SEPP must be upgraded before upgrading cnDBTier.
  • When performing an upgrade from 24.2.x or before source release to 25.2.1xx release, ensure ndb_allow_copying_alter_table in cnDBTier is set to ON. It can be set to OFF after upgrade.

4.2 Upgrade Strategy

SEPP supports in-service upgrade. The supported upgrade strategy is RollingUpdate. The rolling update strategy is a gradual process that allows you to update your Kubernetes system with only a minor effect on performance and no downtime. The advantage of the rolling update strategy is that the update is applied Pod-by-Pod so the greater system can remain active.

Note:

It is recommended to perform SEPP upgrade in a specific order. For more information about the upgrade order, see Oracle Communications Cloud Native Core, Solution Upgrade Guide.

The following engineering configuration parameters are used to define upgrade strategy:

  • upgradeStrategy parameter indicates the update strategy used in SEPP.
  • maxUnavailable parameter determines the maximum number of pods that can be unavailable during upgrade.

Predefined Upgrade Strategy Value

Table 4-3 Predefined Upgrade Strategy Value

Microservice Upgrade Value (maxUnavailable)
nf-mediation 1
n32-ingress-Gateway 25%
n32-egress-Gateway 25%
plmn-ingress-Gateway 25%
plmn-egress-Gateway 25%
pn32f-svc

Not Applicable

Note: maxSurge=25% attribute is used for this microservice.

cn32f-svc

Not Applicable

Note: maxSurge=25% attribute is used for this microservice.

pn32c-svc

Not Applicable

Note: maxSurge=25% attribute is used for this microservice.

cn32c-svc

Not Applicable

Note: maxSurge=25% attribute is used for this microservice.

config-mgr-svc

Not Applicable

Note: maxSurge=25% attribute is used for this microservice.

Appinfo 25%
ocpm-config 50%
performance 50%
sepp-nrf-client-nfmanagement Single Replica
sepp-nrf-client-nfdiscovery 25%
coherence-svc 1
alternate-route 25%

Note:

When SEPP is deployed with OCCM, follow the specific upgrade sequence as mentioned in Oracle Communications, Cloud Native Core Solution Upgrade Guide.

4.3 Preupgrade Tasks

This section provides information about preupgrade tasks to be performed before upgrading SEPP.

While upgrading an existing SEPP deployment, the running set of containers and pods are replaced with the new set of containers and pods. However, if there is no change in the pod configuration, the running set of containers and pods are not replaced.

Note:

  • No configuration should be performed during the upgrade.
  • Do not exit from Helm upgrade command manually. After running the Helm upgrade command, it takes some time (depending upon the number of PODs to upgrade) to upgrade all of the services. In the meantime, you must not press "ctrl+c" to come out from Helm upgrade command. It may lead to anomalous behavior.

Note:

  • The releaseVersion value in the ocsepp_values.yaml file can not be changed.
  • While performing an upgrade from an older release to a newer release, you must align the ocsepp_values.yaml file of the new release as per the ocsepp_values.yaml file of the older release. Ensure that you do not change any Helm parameter values. Do not enable any new feature during the upgrade. Any ocsepp_values.yaml parameter must not be changed while upgrading unless explicitly specified in this guide. For information about enabling any new feature through Helm parameters, see Oracle Communications Cloud Native Core, Security Edge Protection Proxy User Guide.
  • Install or upgrade the network policies, if applicable. For more information, see Configuring Network Policies.
  • While performing an upgrade, the Global Rate Limiting feature must be disabled on both the ingress gateways (n32-ingress-gateway, plmn-ingress-gateway) in the source custom_values.yaml file.
    rateLimiting:
    enabled: false
  rssRateLimiter: 
    enabled: false
  globalIngressRateLimiting:
    enabled: false
  • After the upgrade, the Cat-3 Previous Location Check feature must be restored to its previous state, if the feature was enabled previously.
Preupgrade Steps
  1. Keep current custom_values.yaml file as backup, that is, ocsepp_custom_values_<version>.yaml for upgrading.
  2. Update the new custom_values.yaml defined for target SEPP release. See Customizing SEPP section for more details about Helm configurable parameters.
  3. Perform sanity check using Helm test. See the Performing Helm Test section for the Helm test procedure.
  4. Before starting upgrade, take a manual backup of SEPP REST based configuration. It will help if manual backup is taken in case the data need to be restored before the upgrade. For REST API configuration details, see SEPP Instance DB Backup section to take the backup.
  5. If you are upgrading from version 24.3.x to target SEPP release, follow the steps below:
    • Before starting the procedure to upgrade, perform the following tasks:
      • Perform the following scenarios for enabling or disabling the Open Telemetry tracing as per the requirement, when upgrading from 24.3.x to target SEPP release:
        • To enable tracing in the target SEPP release, perform the following:
          1. Ensure that Jaeger collector service is up and running inside occne-infra, with port specified in ocsepp_custom_values_<version>.yaml file.
          2. The following flag must be set to true to enable the Open Telemetry for n32c and n32f SEPP services:
            enableOpenTelemetry: true
          3. Ensure that following configuration is added or updated in 24.3.x in ocsepp_custom_values_<version>.yaml in n32c and n32f microservices. 
            #Jaeger Tracing
jaegerTracingEnabled: false
bodyInTraceEnabled: false
otel:
  jaeger:
    udpSender:
      host: "jaeger-collector.cne-infra"
      port: 4318
    logSpans: false
    probabilisticSamplingRate: 0.1
          4. Ensure that following configuration is added or updated in 24.3.x in e ocsepp_custom_values_<version>.yaml in n32-ingress-gateway, plmn-ingress-gateway, n32-egress-gateway, and plmn-egress-gateway. 
          # enable jaeger telemetry tracing
jaegerTelemetryTracingEnabled: false

openTelemetry:
  jaeger:
    httpExporter:
      host: "jaeger-collector.cne-infra"
      port: 4318
    probabilisticSampler: 0.5
        • To disable tracing for n32c and n32f microservices, the following flag must be set to false in e ocsepp_custom_values_<version>.yaml file:
          
  enableOpenTelemetry: 'false'

SEPP Instance DB Backup

This section describes the procedure to perform the DB backup of SEPP Instance.

Prerequisites:

  • MySQL NDB cluster should be in a healthy state.
  • Every database node of the MySQL NDB cluster should be in running state.
  • For cnDBTier to verify the prerequisites, check mysql pod is up and running.
  • Log in to the deployment and run the following command to take the dump (backup) of the database. The user is required to enter the password.
    kubectl exec -i -n <namespace> <sql-node> -- mysqldump --single-transaction --no-tablespaces --no-create-info -h 127.0.0.1 -u <username> -p <sepp database-name> | gzip > <backup_filename>.sql.gz

    For example:

    kubectl exec -i -n occne-ndb ndbappmysqld-0 -- mysqldump --single-transaction --no-tablespaces --no-create-info
      -h 127.0.0.1 -u seppusr -p seppdb_sepp1 | gzip > sepp_sepp1_dbBackup.sql.gz

Note:

The operator must ensure there is enough space on the current directory in order to save the backup file.

Verifying cnDBTier

Perform the following steps while upgrading from SEPP 24.2.x or earlier versions to SEPP 24.3.x:
  1. Run the following command to check the current plugin:
    SELECT user, host, plugin FROM mysql.user;
    Example:
    SELECT user, host, plugin FROM mysql.user;
    Output:
    user             | host      | plugin                |
+------------------+-----------+-----------------------+
| cnccusr          | %         | caching_sha2_password |
| occnerepluser    | %         | caching_sha2_password |
| occneuser        | %         | mysql_native_password |
| seppuser1        | %         | caching_sha2_password |
| seppuser3        | %         | mysql_native_password |
| healthchecker    | localhost | caching_sha2_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session    | localhost | caching_sha2_password |
| mysql.sys        | localhost | caching_sha2_password |
| root             | localhost | caching_sha2_password |
+------------------+-----------+-----------------------+
10 rows in set (0.00 sec)
  2. Run the following command to check the plugin that is associated with the seppuser, and used for the SEPP that is planned to be upgraded:

    Example: | seppuser3 | % | mysql_native_password |

  3. If the plugin is ‘mysql_native_password’, run the following command to change the plugin from ‘mysql_native_password’’ to ‘caching_sha2_password’: 
    ALTER USER '<username>'@'%' IDENTIFIED WITH caching_sha2_password BY '<password>';

    Example: 

    ALTER USER 'seppuser3'@'%' IDENTIFIED WITH caching_sha2_password BY 'NextGenCne1';

  4. Run the following command to verify that the plugin is updated: 
    SELECT user, host, plugin FROM mysql.user;
    The plugin should get updated to ‘caching_sha2_password’.

    Output: | seppuser3 | % | caching_sha2_password |

  5. Wait for some time and verify the health of the SEPP pods. Run the following command to check if the pods are up and running: 
    Kubectl get po -n <namespace>

    Note:

    If the plugin selected is ‘caching_sha2_password’, then the above changes are not required.

Database Configuration

Before performing an upgrade to 24.3.x release, run the following command to provide the required permission to MySQL Database:

$ GRANT SELECT,INSERT,CREATE,ALTER,DROP,LOCK TABLES,CREATE TEMPORARY TABLES,DELETE,UPDATE,REFERENCES,EXECUTE ON mysql.* TO '<OCSEPP User Name>'@'%';
Example: 
GRANT SELECT, INSERT, CREATE, ALTER, DROP, LOCK TABLES, CREATE TEMPORARY TABLES,DELETE,UPDATE,REFERENCES,EXECUTE ON mysql.* TO 'seppusr'@'%';

4.4 Upgrade Tasks

This section includes information about upgrading an existing SEPP deployment.

4.4.1 Helm Upgrade

Upgrading an existing deployment replaces the running containers and pods with new containers and pods. If there is no change in the pod configuration, it is not replaced.

Unless there is a change in the service configuration of a microservice, the service endpoints remain unchanged.

4.4.2 Upgrading SEPP

Perform the following procedure to upgrade SEPP.

Caution:

  • Stop the provisioning traffic before you start the upgrade procedure.
  • No configuration should be performed during upgrade.
  • Do not exit from helm upgrade command manually. After running the helm upgrade command, it takes some time (depending upon the number of pods to upgrade) to upgrade all the services. In the meantime, you must not press "ctrl+c" to come out from helm upgrade command. It may lead to anomalous behavior.

Note:

  • If the Helm upgrade results in failure and displays ConfigMap <Egress Rate Limiting ConfigMap Name> in namespace <Namespace> exists and cannot be imported into the current release: invalid ownership metadata error, do the following:

    Delete the Egress Rate Limit ConfigMap using the following command: kubectl delete cm <Egress Rate Limiting ConfigMap Name> -n <namespace> and run the Helm upgrade again.
  1. Untar the latest SEPP package and if required, re-tag and push the images to registry. For more information, see Downloading SEPP package and Pushing the Images to Customer Docker Registry.
  2. Modify the ocsepp_custom_values_25.2.100.yaml file parameters as per site requirement.
  3. Run the following command to upgrade an existing SEPP deployment:
    1. Using local Helm chart:
      helm upgrade <release_name> <helm_chart> -f <ocsepp_customized_values.yaml> --namespace <namespace-name>
      Example:
      helm upgrade ocsepp-release ocsepp-25.2.100.tgz -f ocsepp-custom values-25.2.100.yaml --namespace seppsvc

      Where,

      <release_name> is the SEPP release name.

      <helm_chart> is the Helm chart.

      <sepp_customized_values.yaml> is the latest custom-values.yaml file.

      For example, ocsepp-25.2.100.0.0.custom-values.yaml

      <namespace> is namespace of SEPP deployment.

    2. Using chart from Helm repo:
      helm upgrade <release_name> <helm_repo/helm_chart> --version <chart_version> -f <ocsepp_customized_values.yaml> --namespace <namespace>
      Example:
      helm upgrade ocsepp-release ocsepp-25.2.100.tgz --version <chart_version> -f ocsepp-custom values-25.2.100.yaml --namespace seppsvc

      Where,

      <release_name> is the SEPP release name.

      <helm_chart> is the Helm chart.

      <sepp_customized_values.yaml> is the latest custom-values.yaml file.

      For example, ocsepp-25.2.100.0.0.custom-values.yaml

      <namespace> is namespace of SEPP deployment.

  4. Run the following command to check the status of the upgrade:
    helm status <release_name> --namespace <namespace-name>

    Where,

    <release_name> is the SEPP release name.

    <namespace> is namespace of SEPP deployment.

    For example: 
    $ helm status ocsepp-release --namespace seppsvc
    Sample output of a successful upgrade:
    [cloud-user@occne-22-2-0-cluster-bastion-1 ocsepp-25.2.100]$  helm history ocsepp-release -n seppsvc
REVISION        UPDATED                         STATUS          CHART           APP VERSION     DESCRIPTION
1               Wed May  13 07:19:50 2023        superseded      ocsepp-25.2.100   25.2.100.0.0      Install complete
2               Wed May  13 07:31:21 2023        superseded      ocsepp-25.2.100   25.2.100.0.0      Upgrade complete
3               Wed May  13 07:50:08 2023        superseded      ocsepp-25.2.100   25.2.100.0.0      Rollback to 1
  5. Perform sanity check using Helm test. See the Performing Helm Test section for the Helm test procedure.
  6. If the upgrade fails, see Upgrade or Rollback failure in Oracle Communications Cloud Native Core, Security Edge Protection Proxy Troubleshooting Guide.

4.5 Postupgrade Tasks

Note:

  • To upgrade cnDBTier with resources recommended for SEPP, customize the ocsepp_dbtier_25.2.100_custom_values_25.2.100.yaml file in the ocsepp_csar_25_2_1_0_0_0.zip folder with the required deployment parameters. cnDBTier parameters will vary depending on whether the deployment is on a single site, two site, or three site. For more information about cnDBTier upgrade, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
  • To automate the lifecycle management of the certificates through OCCM, you can migrate the lifecycle of certificates and key management from manual to OCCM. For more information, see "Introducing OCCM in an Existing NF Deployment" section in Oracle Communications Cloud Native Core, Certificate Management User Guide. SEPP application does not manage the LCM of the certificate and key.

4.5.1 Alert Configuration

This section describes how to modify or update SEPP alerts as needed, based on requirements, after performing the upgrade.

For more details, see the following sections:

Alert Configuration