Performing Zero-Downtime Upgrades

18 Performing Zero-Downtime Upgrades

Learn how to upgrade an Oracle Communications Billing and Revenue Management (BRM) cloud native deployment without having to take the environment offline. A zero-downtime upgrade allows your customers to continue using BRM's major services during the upgrade process.

Topics in this document:

In this document, the BRM release running on your production system is called the existing release. The release you are upgrading to is called the new release. For example, if you upgrade from BRM 15.x to BRM 16.26.0, 15.x is the existing release and 16.26.0 is the new one.

Performing a Zero-Downtime Upgrade of BRM

You can perform a zero-downtime upgrade of your BRM cloud native services and the BRM database schema from 15.x to 16.26.0.

To perform a zero-downtime upgrade of BRM cloud native:

Download the BRM 16.26.0 cloud native package from the Oracle Software Delivery website (https://support.oracle.com) or the Oracle Support website (https://support.oracle.com).
Perform the prerequisite steps in the KM KB928854.
Prepare your existing BRM 15.x deployment for the zero-downtime upgrade:
1. In your override-values.yaml file for the BRM 15.x oc-cn-helm-chart, set the following keys:
```
ocbrm:
   refreshInterval: 10
   terminationGracePeriodSeconds: 120
   pcpReconnectDelayOnSocketError: 10
   pcpConnectRetryDelayOnError: 10
```
2. In oc-cn-helm-chart/templates/configmap_pin_conf_cm.yaml, set the following entry:
```
- cm pcm_connect_max_retries 10
```
3. Ensure that at least two replica pods of cm, dm-oracle, dm-ifw-sync, and realtime-pipeline are running.
4. Run the helm upgrade command for the BRM 15.x oc-cn-helm-chart:
```
helm upgrade Brm_15_ReleaseName oc-cn-helm-chart --values OverrideValuesFile --namespace Brm_15_NameSpace
```
  where:
  - Brm_15_ReleaseName is the release name assigned to your existing 15.x oc-cn-helm-chart installation.
  - OverrideValuesFile is the file name and path to the file that overrides the oc-cn-helm-chart/values.yaml file.
  - Brm_15_NameSpace is the namespace for your existing 15.x BRM deployment.
5. Back up your existing BRM 15.x Helm charts.
6. Copy the BRM 16.26.0 oc-init-db-helm-chart and oc-cn-helm-chart Helm charts to your BRM cloud native environment.
Upgrade the BRM cloud native database schema to 16.26.0:
1. Ensure that the ConfigCacheRefreshInterval business parameter in bus_params_system.xml is set to 0.
  
  For information about how to set business parameters, see "Running Load Utilities through Configurator Jobs" in BRM Cloud Native System Administrator's Guide.
2. Upgrade the cloud native database to 16.26.0 using the helm install command:
```
helm install NewInitDbReleaseName oc-cn-init-db-helm-chart --values OverrideValuesFile --namespace NewInitDbNameSpace
```
  where:
  - NewInitDbReleaseName is the release name assigned to your new 16.26.0 oc-cn-init-db-helm-chart installation.
  - NewInitDbNameSpace is the namespace for your new 16.26.0 oc-cn-init-db-helm-chart deployment.
  Verify that the database has upgraded successfully.
Upgrade the BRM cloud native server to 16.26.0 by doing the following:
Note:

To run billing during the BRM cloud native server upgrade for non-production systems, set the next billing date in the oc-cn-helm-chart/templates/configmap_env_common.yaml file to the following:
```
VIRTUAL_TIME_SETTING: "-m 2 billingDate"
```
For example, to set the next billing date to Feb 12, 2025, set billingDate to 021210002025.
1. Ensure the ConfigCacheRefreshInterval business parameter in bus_params_system.xml is set to 0.
  
  For information about how to set business parameters, see "Running Load Utilities through Configurator Jobs" in BRM Cloud Native System Administrator's Guide.
2. In your override-values.yaml file for the 16.26.0 oc-cn-helm-chart, set the following keys:
```
ocbrm:
   refreshInterval: 10
   terminationGracePeriodSeconds: 120
   pcpReconnectDelayOnSocketError: 10
   pcpConnectRetryDelayOnError: 10
   virtual_time:
      enabled: true
      sync_pvt_time: 5
```
  If TLS is enabled in your BRM deployment, also configure the TLS settings and wallet sources required by BRM 16.26.0. Set ocbrm.sslMode to ONE_WAY or TWO_WAY, as required for your deployment.
  
  If you use external Kubernetes secrets for TLS wallets, create any required secrets in Brm_15_NameSpace before you run the Helm upgrade, and set the appropriate component-specific extUserSSLWalletSecret keys in override-values.yaml. You can use ocbrm.extCustomSSLWalletSecret as the shared TLS wallet source when you do not configure a component-specific wallet secret. See "Managing Wallet and KeyStore Certificates".
3. In the 16.26.0 oc-cn-helm-chart/templates/configmap_pin_conf_cm.yaml file, set the following:
```
- cm pcm_connect_max_retries 10
- cm pcm_em_proto_vers 0
```
4. In the 16.26.0 oc-cn-helm-chart/templates/configmap_pin_conf_rtp_pipeline.yaml file, set the following:
```
- cm-em pcm_em_proto_vers 0
```
5. In the 16.26.0 oc-cn-helm-chart/templates/cm.yaml file, set the following:
```
spec.template.spec.containers(- name: cm).livenessProbe.initialDelaySeconds: 60
```
6. In the 16.26.0 oc-cn-helm-chart/templates/configmap_pin_conf_dm_oracle.yaml file, set the following:
```
- dm dm_ignore_fld_mismatch_err 1
```
7. Upgrade the BRM cloud native server to 16.26.0 using the helm upgrade command:
```
helm upgrade Brm_15_ReleaseName oc-cn-helm-chart --values OverrideValuesFile --namespace Brm_15_NameSpace
```
  Verify that all 15.x pods terminate and all 16.26.0 pods come up and run with 16.26.0 images.
Revert the configuration values in your override-values.yaml file for the 16.26.0 oc-cn-helm-chart:
1. In non-production systems, set the VIRTUAL_TIME_SETTING parameter to the default value in your oc-cn-helm-chart/templates/configmap_env_common.yaml file:
```
VIRTUAL_TIME_SETTING: "-m 0"
```
2. In your oc-cn-helm-chart/templates/configmap_pin_conf_rtp_pipeline.yaml file, remove or comment out the following entry:
```
- cm-em pcm_em_proto_vers 0
```
3. In your oc-cn-helm-chart/templates/configmap_pin_conf_cm.yaml file, remove or comment out the following entry:
```
- cm pcm_em_proto_vers 0
```
4. In your override-values.yaml file for oc-cn-helm-chart, set the ocbrm.refreshInterval key to its original value.
5. Set the ConfigCacheRefreshInterval business parameter in bus_params_system.xml to its original value.
  
  For information about how to set business parameters, see "Running Load Utilities through Configurator Jobs" in BRM Cloud Native System Administrator's Guide.
6. Run the helm upgrade command for the 16.26.0 oc-cn-helm-chart:
```
helm upgrade Brm_15_ReleaseName oc-cn-helm-chart --values OverrideValuesFile --namespace Brm_15_NameSpace
```

Performing a Zero Downtime Upgrade of PDC

You can perform a zero downtime upgrade of your PDC cloud native services and the PDC database schema from the 12.0 or 12.0 Patch Set release to the 16.26.0 release.

You do so using a two-namespace approach in which you create an instance of PDC cloud native in a standby namespace, redirect PDC traffic to services in the standby namespace, upgrade PDC cloud native to release 16.26.0 in your original namespace, and then redirect PDC traffic back to your original namespace.

To upgrade PDC in zero downtime upgrade mode:

Create a temporary namespace, such as BrmStandbyNameSpace.
Clone your PDC OverrideValuesFile file to StandbyOverrideValuesFile.
In your StandbyOverrideValuesFile file for oc-cn-op-job-helm-chart, set the following keys:
```
ocpdc:
   configEnv:
      rcuPrefix: NewPrefix
      crossRefSchemaUserName: XrefSchema 
      pdcSchemaUserName: PdcSchema
      siteNo: 2
      upgrade: true
```
where:
- NewPrefix is the new prefix for the PDC domain RCU schema.
- XrefSchema is the same XREF schema name used for deploying PDC in BrmNameSpace.
- PdcSchema is the same PDC schema name used for deploying PDC in BrmNameSpace.
In your StandbyOverrideValuesFile file for oc-cn-helm-chart, set the following keys:
```
ocpdc:
   configEnv:
      upgrade: true
```
These settings will upgrade the PDC and XREF schema.
Copy the following templates from your BrmNameSpace Helm chart template directory to the BrmStandbyNameSpace Helm chart template directory:
- secret_env_brm.yaml
- configmap_pin_conf_brm_apps_2.yaml
- configmap_loadifwconfig_reg.yaml
- configmap_env_common.yaml
- virtual_time_pvc.yaml
- configmap_infranet_properties_brm_apps.yaml
- config_jobs.yaml
- storage_class_green.yaml
- realtime_pipeline_common_pvc.yaml
- configmap_tns_admin.yaml
- secret_wallet_db.yaml
- _helpers_utils.tpl
- configmap_pdc_aux_engines.yaml
- configmap_log_properties_pdc.yaml
- configmap_env_pdc.yaml
- configmap_env_pdc_rre.yaml
- job_ie_pdc.yaml
- domain_pdc.yaml
- _pdchelpers.tpl
- pdc_domain_monitoring_role.yaml
- pdc_domain_monitoring_rbac.yaml
- service_monitor_pdc_domain.yaml
- secret_pdc.yaml
- volume_pdc_brm.yaml
- deployment_pdc_rre.yaml
- deployment_pdc_bre.yaml
- deployment_pdc_syncpdc.yaml

Create a configuration file named cm-service-external-name.yaml and add the following content:

apiVersion: v1
kind: Service
metadata:
  name: cm
  namespace: BrmStandbyNameSpace
spec:
  externalName: cm.BrmNameSpace.svc.cluster.local
  internalTrafficPolicy: Cluster
  ports:
  - port: 11960
    protocol: TCP
    targetPort: 11960
  sessionAffinity: None
  type: ExternalName

Apply the configuration file to a resource:

kubectl apply -f cm-service-external-name.yaml

Deploy the PDC 12.0 Patch Set 7 or 8 Helm charts in your standby namespace:

helm install OpJobStandbyReleaseName oc-cn-op-job-helm-chart --values StandbyOverrideValuesFile --namespace BrmStandbyNameSpace

helm install BrmStandbyReleaseName oc-cn-helm-chart --values StandbyOverrideValuesFile --namespace BrmStandbyNameSpace

Redirect PDC traffic to services in BrmStandbyNameSpace.
Upgrade your PDC cloud native services to release 16.26.0 in BrmNameSpace while requests are temporarily routed to BrmStandbyNameSpace.
1. Set the following keys in your override-values.yaml file for oc-cn-op-job-helm-chart:
```
ocpdc:
   configEnv: 
      siteNo: 1
      upgrade: true
```
2. Follow the instructions in "Upgrading Your PDC Cloud Native Services" to upgrade your original namespace to 16.26.0.
Redirect PDC traffic back to services in your original namespace (BrmNameSpace).

Installing PDC 15.x on More Than Two Sites

You can install PDC on Sites 1 and 2 with the default configuration and start using them immediately.

When you deploy PDC across more than two sites with the same PDC and cross-reference schema, such as during a zero-downtime upgrade of disaster recovery systems, you must follow the steps below on Site 3 and all subsequent sites.

Note:

You can restore PDC using oc-cn-helm-chart if there are any issues with Sites 1 or 2.

To deploy PDC on Site 3 and subsequent sites:

Create a Python script on the machine where you are installing the third site named sample_unique_WLSTORE.py with content similar to the following:

print("\n[INFO] Start - Custom Weblogic WLST - To configure unique WLSTORE  ...")

host = os.getenv("HOSTNAME", "localhost")
connect("WLSAdminUser", "password", "t3://" + host + ":WLSPort")

edit()
startEdit()

cd("/JDBCStores/JDPersistentJDBCStore")
cmo.setDataSource(getMBean("/JDBCSystemResources/JobDispatcherPersistentDS"))
cmo.setPrefixName("PDCJD3_")

save()
activate()
disconnect()

print("\n[INFO] Completed - Custom Weblogic WLST - configured unique WLSTORE ...")

In the "connect" line of the file, replace the following:
- WLSAdminUser is the user name of the administrative user for WebLogic Server.
- password is the password of your WebLogic Server.
- WLSPort is the listener port for the WebLogic administration server.
Save the Sample_unique_WLSTORE.py file to the oc-cn-op-job/pdc/customWLSPython/ directory.
Set the file permissions for Sample_unique_WLSTORE.py to 755.
In your override-values.yaml file for oc-cn-op-job-helm-chart, set the isCustomWLSPython key to true.

Create WebLogic domains by running the 15.x version of oc-cn-op-job-helm-chart from the helmcharts directory:

helm install OpJobReleaseName oc-cn-op-job-helm-chart --namespace BrmNameSpace --values override-values.yaml