Performing Zero-Downtime Upgrades

17 Performing Zero-Downtime Upgrades

Learn how to upgrade an Oracle Communications Billing and Revenue Management (BRM) cloud native deployment without having to take the environment offline. A zero-downtime upgrade allows your customers to continue using BRM's major services during the upgrade process.

Topics in this document:

In this document, the BRM release running on your production system is called the existing release. The release you are upgrading to is called the new release. For example, if you upgrade from BRM 12.0 Patch Set 5 to BRM 15.0, 12.0 Patch Set 5 is the existing release and 15.0 is the new one.

Performing a Zero-Downtime Upgrade of BRM

You can perform a zero-downtime upgrade of your BRM cloud native services and the BRM database schema from 12.0.0.x.0 to 15.0.x.0.0.

To perform a zero-downtime upgrade of BRM cloud native:

Download the BRM 15.0.x.0.0 cloud native package from the Oracle Software Delivery website (https://support.oracle.com) or the Oracle Support website (https://support.oracle.com).
Configure and deploy the BRM 12.0.0.x.0 oc-cn-helm-chart Helm chart on your cloud native environment by doing the following:
1. In your override-values.yaml file for the BRM 12.0.0.x.0 oc-cn-helm-chart, set the following keys:
```
ocbrm:
   refreshInterval: 10
   terminationGracePeriodSeconds: 120
   pcpReconnectDelayOnSocketError: 10
   pcpConnectRetryDelayOnError: 10
```
2. In your oc-cn-helm-chart/templates/configmap_pin_conf_cm.yaml file, set the following key:
```
- cm pcm_connect_max_retries 10
```
3. Ensure that at least two replica pods of cm, dm-oracle, dm-ifw-sync, and realtime-pipeline are up and running.
4. (BRM 12.0.0.7.0 only) Apply BRM 12.0.0.7.0 Interim Patch 34939558 to your cloud native environment.
5. (BRM 12.0.0.7.0 or later) In your BRM 12.0.0.x.0 oc-cn-helm-chart/templates/configmap_pin_conf_dm_oracle.yaml file, set the following key:
```
- dm dm_ignore_fld_mismatch_err 1
```
6. Run the helm upgrade command for the 12.0.0.x.0 oc-cn-helm-chart:
```
helm upgrade Brm_12_ReleaseName oc-cn-helm-chart --values OverrideValuesFile -n Brm_12_NameSpace
```
  where:
  - Brm_12_ReleaseName is the release name assigned to your existing 12.0.0.x.0 oc-cn-helm-chart installation.
  - OverrideValuesFile is the file name and path to the file that overrides the oc-cn-helm-chart/values.yaml file.
  - Brm_12_NameSpace is the namespace for your existing 12.0.0.x.0 BRM deployment.
7. Back up your existing BRM 12.0.0.x.0 Helm charts.
8. Copy the 15.0.x.0.0 oc-init-db-helm-chart and oc-cn-helm-chart Helm charts to your BRM cloud native environment.
9. (12.0.0.6.0 or earlier) Upgrade only the dm-oracle pod from 12.0.0.x.0 to 15.0.x.0.0 by doing the following:
  1. In your override-values.yaml file for the 15.0.x.0.0 oc-cn-helm-chart, set the ocbrm.dm_oracle.deployment.imageTag key to 15.0.x.0.0.
  2. Run the helm upgrade command for the 15.0.x.0.0 oc-cn-helm-chart:
```
helm upgrade Brm_12_ReleaseName oc-cn-helm-chart --values OverrideValuesFile -n Brm_12_NameSpace
```
    Verify that only the dm-oracle pod is running with the 15.0.x.0.0 image. The remaining pods run with 12.0.0.x.0 images.
Upgrade the BRM cloud native database schema to 15.0.x.0.0 by doing the following:
1. Ensure that the ConfigCacheRefreshInterval business parameter in bus_params_system.xml is set to 0.
  
  For information about how to set business parameters, see "Running Load Utilities through Configurator Jobs" in BRM Cloud Native System Administrator's Guide.
2. In your override-values.yaml file for the 15.0.x.0.0 oc-cn-init-db-helm-chart, set the following keys:
```
ocbrm:
   is_upgrade : true
   existing_rootkey_wallet: true
```
3. Copy the wallet files (ewallet.p12 and cwallet.sso) from the /oms/wallet/client/ directory of the 12.0.0.x.0 primary dm-oracle pod to the 15.0.x.0.0 oc-cn-init-db-helm-chart/existing_wallet directory.
4. Upgrade the cloud native database to 15.0.x.0.0 using the helm install command:
```
helm install Brm_New_ReleaseName oc-cn-helm-chart --values OverrideValuesFile -n Brm_New_NameSpace
```
  where:
  - Brm_New_ReleaseName is the release name assigned to your new 15.0.x.0.0 oc-cn-helm-chart installation.
  - Brm_New_NameSpace is the namespace for your new 15.0.x.0.0 BRM deployment.
  Verify that the database has upgraded successfully.
5. (12.0.0.6.0 or earlier) Restart the 15.0.x.0.0 dm-oracle pod.
Upgrade the BRM cloud native server to 15.0.x.0.0 by doing the following:
Note:

To run billing during the BRM cloud native server upgrade for non-production systems, set the next billing date in the oc-cn-helm-chart/templates/configmap_env_common.yaml file to the following:
```
VIRTUAL_TIME_SETTING: "-m 2 billingDate"
```
For example, to set the next billing date to Feb 12, 2030, set billingDate to 021210002030.
1. Ensure the ConfigCacheRefreshInterval business parameter in bus_params_system.xml is set to 0.
  
  For information about how to set business parameters, see "Running Load Utilities through Configurator Jobs" in BRM Cloud Native System Administrator's Guide.
2. In your override-values.yaml file for the 15.0.x.0.0 oc-cn-helm-chart, set the following keys:
```
ocbrm:
   refreshInterval: 10
   terminationGracePeriodSeconds: 120
   pcpReconnectDelayOnSocketError: 10
   pcpConnectRetryDelayOnError: 10
   virtual_time:
      enabled: true
      sync_pvt_time: 5
```
3. In the 15.0.x.0.0 oc-cn-helm-chart/templates/configmap_pin_conf_cm.yaml file, set the following:
```
- cm pcm_connect_max_retries 10
- cm pcm_em_proto_vers 0
```
4. In the 15.0.x.0.0 oc-cn-helm-chart/templates/configmap_pin_conf_rtp_pipeline.yaml file, set the following:
```
- cm-em pcm_em_proto_vers 0
```
5. In the 15.0.x.0.0 oc-cn-helm-chart/templates/cm.yaml file, set the following:
```
spec.template.spec.containers(- name: cm).livenessProbe.initialDelaySeconds: 60
```
6. In the 15.0.x.0.0 oc-cn-helm-chart/templates/configmap_pin_conf_dm_oracle.yaml file, set the following:
```
- dm dm_ignore_fld_mismatch_err 1
```
7. Upgrade the BRM cloud native server to 15.0.x.0.0 using the helm upgrade command:
```
helm upgrade Brm_12_ReleaseName oc-cn-helm-chart --values OverrideValuesFile -n Brm_12_NameSpace
```
  Verify that all 12.0.0.x.0 pods terminate and all 15.0.x.0.0 pods come up and run with 15.0.x.0.0 images.
Revert the configuration values in your override-values.yaml file for the 15.0.x.0.0 oc-cn-helm-chart:
1. In non-production systems, set the VIRTUAL_TIME_SETTING parameter to the default value in your oc-cn-helm-chart/templates/configmap_env_common.yaml file:
```
VIRTUAL_TIME_SETTING: "-m 0"
```
2. In your oc-cn-helm-chart/templates/configmap_pin_conf_rtp_pipeline.yaml file, remove or comment out the following entry:
```
- cm-em pcm_em_proto_vers 0
```
3. In your oc-cn-helm-chart/templates/configmap_pin_conf_cm.yaml file, remove or comment out the following entry:
```
- cm pcm_em_proto_vers 0
```
4. In your override-values.yaml file for oc-cn-helm-chart, set the ocbrm.refreshInterval key to its original value.
5. Set the ConfigCacheRefreshInterval business parameter in bus_params_system.xml to its original value.
  
  For information about how to set business parameters, see "Running Load Utilities through Configurator Jobs" in BRM Cloud Native System Administrator's Guide.
6. Run the helm upgrade command for the 15.0.x.0.0 oc-cn-helm-chart:
```
helm upgrade Brm_12_ReleaseName oc-cn-helm-chart --values OverrideValuesFile -n Brm_12_NameSpace
```

Performing a Zero Downtime Upgrade of PDC

You can perform a zero downtime upgrade of your PDC cloud native services and the PDC database schema from the 12.0 or 12.0 Patch Set release to a 15.0.x release.

You do so using a two-namespace approach in which you create an instance of PDC cloud native in a standby namespace, redirect PDC traffic to services in the standby namespace, upgrade PDC cloud native to release 15.0.x in your original namespace, and then redirect PDC traffic back to your original namespace.

To upgrade PDC in zero downtime upgrade mode:

Create a temporary namespace, such as BrmStandbyNameSpace.
Clone your PDC OverrideValuesFile file to StandbyOverrideValuesFile.
In your StandbyOverrideValuesFile file for oc-cn-op-job-helm-chart, set the following keys:
```
ocpdc:
   configEnv:
      rcuPrefix: NewPrefix
      crossRefSchemaUserName: XrefSchema 
      pdcSchemaUserName: PdcSchema
      deployAndUpgradeSite2: true
      upgrade: true
```
where:
- NewPrefix is the new prefix for the PDC domain RCU schema.
- XrefSchema is the same XREF schema name used for deploying PDC in BrmNameSpace.
- PdcSchema is the same PDC schema name used for deploying PDC in BrmNameSpace.
In your StandbyOverrideValuesFile file for oc-cn-helm-chart, set the following keys:
```
ocpdc:
   configEnv:
      upgrade: true
```
These settings will upgrade the PDC and XREF schema.
Copy the following templates from your BrmNameSpace Helm chart template directory to the BrmStandbyNameSpace Helm chart template directory:
- secret_env_brm.yaml
- configmap_pin_conf_brm_apps_2.yaml
- configmap_loadifwconfig_reg.yaml
- configmap_env_common.yaml
- virtual_time_pvc.yaml
- configmap_infranet_properties_brm_apps.yaml
- config_jobs.yaml
- storage_class_green.yaml
- realtime_pipeline_common_pvc.yaml
- configmap_tns_admin.yaml
- secret_wallet_db.yaml
- _helpers_utils.tpl
- configmap_pdc_aux_engines.yaml
- configmap_log_properties_pdc.yaml
- configmap_env_pdc.yaml
- configmap_env_pdc_rre.yaml
- job_ie_pdc.yaml
- domain_pdc.yaml
- _pdchelpers.tpl
- pdc_domain_monitoring_role.yaml
- pdc_domain_monitoring_rbac.yaml
- service_monitor_pdc_domain.yaml
- secret_pdc.yaml
- volume_pdc_brm.yaml
- deployment_pdc_rre.yaml
- deployment_pdc_bre.yaml
- deployment_pdc_syncpdc.yaml

Create a configuration file named cm-service-external-name.yaml and add the following content:

apiVersion: v1
kind: Service
metadata:
  name: cm
  namespace: BrmStandbyNameSpace
spec:
  externalName: cm.BrmNameSpace.svc.cluster.local
  internalTrafficPolicy: Cluster
  ports:
  - port: 11960
    protocol: TCP
    targetPort: 11960
  sessionAffinity: None
  type: ExternalName

Apply the configuration file to a resource:

kubectl apply -f cm-service-external-name.yaml

Deploy the PDC 12.0 Patch Set 7 or 8 Helm charts in your standby namespace:

helm install OpJobStandbyReleaseName oc-cn-op-job-helm-chart --values StandbyOverrideValuesFile --namespace BrmStandbyNameSpace

helm install BrmStandbyReleaseName oc-cn-helm-chart --values StandbyOverrideValuesFile --namespace BrmStandbyNameSpace

Redirect PDC traffic to services in BrmStandbyNameSpace.
Upgrade your PDC cloud native services to release 15.0.x in BrmNameSpace while requests are temporarily routed to BrmStandbyNameSpace.
1. Set the following keys in your override-values.yaml file for oc-cn-op-job-helm-chart:
```
ocpdc:
   configEnv: 
      deployAndUpgradeSite2: false
      upgrade: true
```
2. Follow the instructions in "Upgrading Your PDC Cloud Native Services" to upgrade your original namespace to 15.0.x.
Redirect PDC traffic back to services in your original namespace (BrmNameSpace).