Upgrading CNC Console

Upgrade /Rollback Task	References	Applicable for CDCS	Applicable for CLI
CNCC IAM DB Backup	CNCC IAM DB Backup	Yes	Yes
CNCC Upgrade	CNCC Upgrade	See Oracle Communications CD Control Server Installation and Upgrade Guide	Yes
CNCC IAM DB Rollback/Restore	CNCC IAM DB Rollback or Restore	Yes	Yes
CNCC Rollback	CNCC Rollback	See Oracle Communications CD Control Server Installation and Upgrade Guide	Yes

Preupgrade Tasks

While upgrading an existing CNC Console deployment, the running set of containers and pods are replaced with the new set of containers and pods. However, if there is no change in the pod configuration, the running set of containers and pods are not replaced.

Caution:

No configuration should be performed during upgrade.
Do not exit from helm upgrade command manually. After running the helm upgrade command, it takes sometime (depending upon number of Pods to upgrade) to upgrade all of the services. In the meantime, you must not press "ctrl+c" to come out from helm upgrade command. It may lead to anomalous behavior.

Preupgrade Steps

Keep current occncc_custom_values_<version>.yaml file as backup, that is
```
occncc_custom_values_<version to be upgraded>.yaml 
```
Update the new custom_values.yaml defined for target CNC Console release. See CNC Console IAM Configuration Parameters section and CNC Console Core Configuration Parameters section for more details about helm configurable parameters.

Supported Upgrade Paths

For information about supported upgrade paths, see CNC Console Deployment Modes section.

CNC Console Upgrade Sequence

The following is the CNC Console Upgrade Sequence:

Table 7-1 CNC Console Upgrade Sequence

Deployment Mode	Source Version	Target Version	Upgrade Sequence	Comments
Single Cluster	22.3.0, 22.3.1	22.3.x	Console Upgrade Upgrade CNCC NF Upgrade Upgrade Instances (NF or CNE/OSO Common Services)	CNC Console can be upgraded first as 22.2.0 onwards NF includes menu api to support Console first upgrade functionality. Helm upgrade CNCC using existing release name (cncc-iam or cncc).
	22.2.x	22.3.x	Console Upgrade Upgrade CNCC IAM Uninstall CNCC Core NF Upgrade Upgrade Instances (NF or CNE/OSO Common Services)	CNC Console can be upgraded first as 22.2.0 onwards NF includes menu api to support Console first upgrade functionality. Helm upgrade using cncc-iam release name. This will use existing IAM database and it will install M-CNCC IAM, M-CNCC Core and A-CNCC Core if enabled. Helm uninstall cncc-core using existing release name (cncc-core or cncc). This will uninstall M-CNCC Core and A-CNCC Core if deployed.
	22.1.x or older	22.3.x	NF Upgrade Upgrade Instances (NF or CNE/OSO Common Services) Console Upgrade Upgrade CNCC IAM Uninstall CNCC Core	NF should be upgraded first as the base NF version does not include menu api to support Console first upgrade functionality. Helm upgrade using cncc-iam release name. This will use existing IAM database and it will install M-CNCC IAM, M-CNCC Core and A-CNCC Core if enabled. Helm uninstall cncc-core using existing release name (cncc-core or cncc). This will uninstall M-CNCC Core and A-CNCC Core if deployed.
Multi Cluster	22.3.0, 22.3.1	22.3.x	Console Manager Upgrade Upgrade CNCC Console Agent Upgrade Upgrade A-CNCC Core NF Upgrade Upgrade Instances (NF or CNE/OSO Common Services)	CNC Console can be upgraded first as 22.2.0 onwards NF includes menu api to support Console first upgrade functionality. Manager Clusters : Helm upgrade CNCC using existing release name(cncc-iam or cncc). Agent Clusters: Helm upgrade A-CNCC using existing release name (cncc-core or cncc). This will upgrade A-CNCC Core.
	22.2.x	22.3.x	Console Manager Upgrade Upgrade CNCC IAM Uninstall CNCC Core Console Agent Upgrade Upgrade A-CNCC Core NF Upgrade Upgrade Instances (NF or CNE/OSO Common Services)	CNC Console can be upgraded first as 22.2.0 onwards NF includes menu api to support Console first upgrade functionality. Manager Clusters : Helm upgrade using cncc-iam release name. This will use existing IAM database and it will install M-CNCC IAM, M-CNCC Core and A-CNCC Core if enabled. Helm uninstall cncc-core using existing release name (cncc-core or cncc). This will uninstall M-CNCC Core and A-CNCC Core if deployed. Agent Clusters: Helm upgrade cncc-core using existing release name (cncc-core or cncc). This will upgrade A-CNCC Core.
	22.1.x or older	22.3.x	NF Upgrade Upgrade Instances (NF or CNE/OSO Common Services) Console Manager Upgrade Upgrade CNCC IAM Uninstall CNCC Core Console Agent Upgrade Fresh install A-CNCC Core	NF should be upgraded first as the base NF version does not include menu api to support Console first upgrade functionality. Exception here is Policy NF, Console can be upgraded first for Policy NF. Manager Clusters : Helm upgrade using cncc-iam release name. This will use existing IAM database and it will install M-CNCC IAM, M-CNCC Core and A-CNCC Core if enabled. Helm uninstall cncc-core using existing release name. Agent Clusters: Fresh install A-CNCC Core or if A-CNCC Core deployment already exists then helm upgrade cncc-core using existing release name (cncc-core or cncc).

CNC Console Rollback Sequence

The following is the CNC Console Rollback Sequence:

CNC Console Rollback Sequence

Table 7-2 CNC Console Rollback Sequence

Deployment Mode	Source Version	Target Version	Rollback Sequence	Comments
Single Cluster	22.3.x	22.3.0, 22.3.1	Console Rollback Rollback CNCC NF Rollback Rollback Instances (NF or CNE/OSO Common Services)	CNC Console can be rolled back first as 22.2.0 onwards NF includes menu api to support Console first upgrade functionality. Helm rollback CNCC using existing release name (cncc-iam or cncc). This will rollback M-CNCC IAM M-CNCC Core and A-CNCC Core if enabled.
	22.3.x	22.3.x	Console Rollback Rollback CNCC Fresh install CNCC Core NF Rollback Rollback Instances (NF or CNE/OSO Common Services)	CNC Console can be rolled back first as 22.2.0 onwards NF includes menu api to support Console first upgrade functionality. Helm rollback CNCC using existing release name (cncc-iam or cncc). This will rollback M-CNCC IAM and uninstall M-CNCC Core and A-CNCC Core if enabled. Fresh install previous version of CNCC Core using previous release name (cncc-core or cncc). It will install M-CNCC Core and A-CNCC Core if enabled. Fresh installation of CNCC Core is needed for rollback as cncc-core was uninstalled during upgrade.
	22.3.x	22.1.x or older	NF Rollback Rollback Instances (NF or CNE/OSO Common Services) Console Rollback Rollback CNCC Fresh install CNCC Core	NF should be rolled back first as the base NF version does not include menu api to support Console first upgrade functionality. Helm rollback CNCC using existing release name (cncc-iam or cncc). This will rollback CNCC IAM and uninstall M-CNCC Core and A-CNCC Core if enabled. Fresh install previous version of CNCC Core using previous release name (cncc-core). It will install only CNCC Core component. Fresh installation of CNCC Core is needed for rollback as cncc-core was uninstalled during upgrade.
Multi Cluster	22.3.x	22.3.0, 22.3.1	Console Manager Rollback Rollback CNCC Console Agent Rollback Rollback A-CNCC Core NF Rollback Rollback Instances (NF or CNE/OSO Common Services)	CNC Console can be rolled back first as 22.2.0 onwards NF includes menu api to support Console first upgrade functionality. Manager Clusters : Helm rollback CNCC using existing release name (cncc-iam or cncc). This will rollback M-CNCC IAM and uninstall M-CNCC Core and A-CNCC Core if enabled. Agent Clusters: Helm rollback cncc-core using existing release name (cncc-core or cncc). This will rollback A-CNCC Core.
	22.3.x	22.2.x	Console Manager Rollback Rollback CNCC Fresh Install CNCC Core Console Agent Rollback Rollback A-CNCC Core NF Rollback Rollback Instances (NF or CNE/OSO Common Services)	CNC Console can be rolled back first as 22.2.0 onwards NF includes menu api to support Console first upgrade functionality. Manager Clusters : Helm rollback CNCC using existing release name (cncc-iam or cncc). This will rollback M-CNCC IAM and uninstall M-CNCC Core and A-CNCC Core if enabled. Fresh install previous version of CNCC Core using previous release name (cncc-core or cncc). It will install M-CNCC Core and A-CNCC Core if enabled. Fresh installation of CNCC Core is needed for rollback as cncc-core was uninstalled during upgrade. Agent Clusters: Helm rollback cncc-core using existing release name (cncc-core or cncc). This will rollback A-CNCC Core.
	22.3.x	22.1.x or older	NF Rollback Rollback Instances (NF or CNE/OSO Common Services) Console Manager Rollback Rollback CNCC Fresh Install CNCC Core Console Agent Rollback Rollback A-CNCC Core	NF should be rolled back first as the base NF version does not include menu api to support Console first upgrade functionality. Exception here is Policy NF, Console can be upgraded first for Policy NF 22.1.x onwards. Manager Clusters : Helm rollback CNCC using existing release name (cncc-iam or cncc). This will rollback CNCC IAM and uninstall M-CNCC Core and A-CNCC Core if enabled. Fresh install previous version of CNCC Core using previous release name (cncc-core or cncc). It will install CNCC Core. Fresh installation of CNCC Core is needed for rollback as cncc-core was uninstalled during upgrade. Agent Clusters: Helm rollback cncc-core using existing release name (cncc-core or cncc). This will rollback A-CNCC Core. Agent cluster rollback is applicable for Policy NF 22.1.x onwards for other NF its not applicable, they can uninstall agent deployment.

Parameters and Definitions during CNC Console Upgrade

Parameters and Definitions during CNCC Upgrade

Table 7-3 Parameters and Definitions during CNCC Upgrade

Parameters	Definitions
<release_name>	CNCC Helm release deployed. It could be found in the output of 'helm list' command
<namespacename>	CNCC namespace in which release deployed
<helm_chart>	CNCC helm chart
<chart_version>	CNCC helm chart version in case helm charts are referred from helm repo
<helm_repo>	CNCC helm repo
<occncc_custom_values_<version>.yaml>	CNCC customized values.yaml for target release.

CNC Console Upgrade and Rollback Procedure

This section provides details of CNCC upgrade procedure to upgrade following CNC Console components.

M-CNCC IAM
M-CNCC Core
A-CNCC Core

The following steps must be followed while performing the upgrade or rollback:

WARNING:

Before proceeding with CNCC IAM helm upgrade, latest CNCC database backup must be taken. See CNCC IAM DB Backup and store backup file in a location which can be easily restored.

Caution:

It is recommended to verify the copy pasted content especially when the hyphens or any special characters are part of copied content.

CNCC IAM DB Backup

This section describes the procedure to do the DB backup of CNCC IAM.

Prerequisites

The prerequisites for the backup process are:

MySQL NDB cluster should be in a healthy state.
Every database node of the MySQL NDB cluster should be in running state.
In case of cnDBTier to verify the prerequisites, check mysql pod is up and running.
In case of VM based DB Tier to verify the prerequisites, log in to MCM client on one of the SQL node of the cluster.
Run the following command to check the node status.

mcm> show status -r occnendbclustera;

cnDBTier

Log in to the SQL node and run the following command to take the dump (backup) of the database. The user is required to enter the password.

kubectl exec -i -n <namespace> <sql-node> -- mysqldump --single-transaction --no-tablespaces -h 127.0.0.1 -u <username> -p <database-name> | gzip > <backup_filename>.sql.gz

Example:

kubectl exec -i -n occne-ndb ndbmysqld-0 -- mysqldump --single-transaction --no-tablespaces --no-create-info -h 127.0.0.1 -u cnccuser -p cnccdb | gzip > cnccdbBackup.sql.gz

CNCC Upgrade

This section describes the procedure to upgrade CNCC.

Note:

CNC Console Upgrade or Rollback procedure depends on base CNC Console version, see CNC Console Upgrade Sequence section for more details.
Existing release name must be used for upgrade or rollback.

Prepare occncc_custom_values_<version>.yaml file for upgrade.

Upgrade CNC Console using existing release release name (cncc-iam or cncc) by running the following command:

$ helm upgrade <release_name> <helm_chart> -f <occncc_custom_values_<version>.yaml> -n <namespace>

Example:

$ helm upgrade cncc ocspf-helm-repo/cncc -f occncc_custom_values_<version>.yaml -n cncc

Check the status of upgrade by running the following command:
```
$ helm status <release_name> -n <namespace>
```
Example:
```
$ helm status cncc -n cncc
```

Uninstall CNCC Core

Note:

Uninstalling CNCC Core procedure is applicable only while upgrading from listed base version to the current version. See CNC Console Upgrade Sequence to check uninstall of CNCC core is needed.

Uninstall cncc core using old release name (cncc-core or cncc). This uninstalls M-CNCC Core and A-CNCC Core if deployed.

Command:

$ helm uninstall <cncc_core_release_name> -n <namespace>

Example:

$ helm uninstall cncc-core -n cncc

Note:

In case of a upgrade failure, for clean up procedure, see CNC Console Cleanup during Failed Upgrade.

CNCC IAM DB Rollback or Restore

This section provides details of CNCC IAM Rollback. In case of CNC Console IAM Upgrade failure, rollback CNC Console IAM DB to previous version by following the below steps.

Note:

The latest backup must be used for rollback.

cnDBTier

Log in to the deployment cluster, drop the existing database and create a new database. Restore the new database with the DB Schema file provided as part of package (rollback-iam-schema-<version>.sql):
Run the following command to drop the Database and create a new Database:
```
DROP DATABASE <CNCCDatabase>
CREATE DATABASE IF NOT EXISTS <CNCC Database>;
```
Example:
```
To be run in the mysql pod:
DROP DATABASE cnccdb;
CREATE DATABASE IF NOT EXISTS cnccdb;
```
Copy the DB Schema file provided as part of package into the MYSQl pod (occncc_rollback_iam_schema_<version>.sql).
Run the following command to copy the DB Schema file:
```
kubectl cp <backup_file name>.sql <namespace>/<pod-name>:<directory where you want your file placed>
```
Example:
```
kubectl cp occncc_rollback_iam_schema_<version>.sql cndbtier1/ndbmysqld-0:/home/mysql
```

Run the following command to connect to the SQL node of the NDB cluster or connect to the cnDbBTier:

$ kubectl -n <cndbtier_namespace> exec -it <cndbtier_sql_pod_name> -c <cndbtier_sql_container_name> -- bash

Example:

$ kubectl -n cndbtier  exec -it ndbmysqld-0 -c mysqlndbcluster -- bash

Restore this new database with the DB Schema file provided as part of package (occncc_rollback_iam_schema_<version>.sql).
Run the following command to Restore DB Schema:
```
mysql -h 127.0.0.1 -u root -p <DB name> < <DB Schema file name>
```
Example:
```
mysql -h 127.0.0.1 -u root -p cnccdb <occncc_rollback_iam_schema_<version>.sql
```

The DB Dump has to be rearranged sequentially not to get any foreign key constraints issue. For that, create the ENV variables and run it in a loop.

Run the following command to convert the mysqldump file which was taken as a backup (sql.gz file) to a sql file to rearrange it:
Unzipping the gz file:
```
gunzip -d <<backup_filename>.sql.gz>
```
Example:
```
gunzip -d cnccdbBackup.sql.gz
```

Rearrange the backup sql file in correct order by using following procedure.

Run the following command to rearrange the Table Data :

export KC_TABLES="ADMIN_EVENT_ENTITY RESOURCE_SERVER RESOURCE_SERVER_POLICY
          ASSOCIATED_POLICY REALM CLIENT AUTHENTICATION_FLOW AUTHENTICATION_EXECUTION
          AUTHENTICATOR_CONFIG AUTHENTICATOR_CONFIG_ENTRY BROKER_LINK CLIENT_ATTRIBUTES
          CLIENT_AUTH_FLOW_BINDINGS KEYCLOAK_ROLE CLIENT_INITIAL_ACCESS CLIENT_NODE_REGISTRATIONS
          CLIENT_SCOPE CLIENT_SCOPE_ATTRIBUTES CLIENT_SCOPE_CLIENT CLIENT_SCOPE_ROLE_MAPPING
          USER_SESSION CLIENT_SESSION CLIENT_SESSION_AUTH_STATUS CLIENT_SESSION_NOTE
          CLIENT_SESSION_PROT_MAPPER CLIENT_SESSION_ROLE CLIENT_USER_SESSION_NOTE COMPONENT
          COMPONENT_CONFIG COMPOSITE_ROLE DATABASECHANGELOG USER_ENTITY CREDENTIAL
          DATABASECHANGELOGLOCK DEFAULT_CLIENT_SCOPE EVENT_ENTITY FEDERATED_IDENTITY FEDERATED_USER
          FED_USER_ATTRIBUTE FED_USER_CONSENT FED_USER_CONSENT_CL_SCOPE FED_USER_CREDENTIAL
          FED_USER_GROUP_MEMBERSHIP FED_USER_REQUIRED_ACTION FED_USER_ROLE_MAPPING KEYCLOAK_GROUP
          GROUP_ATTRIBUTE GROUP_ROLE_MAPPING IDENTITY_PROVIDER IDENTITY_PROVIDER_CONFIG
          IDENTITY_PROVIDER_MAPPER IDP_MAPPER_CONFIG MIGRATION_MODEL OFFLINE_CLIENT_SESSION
          OFFLINE_USER_SESSION POLICY_CONFIG PROTOCOL_MAPPER PROTOCOL_MAPPER_CONFIG REALM_ATTRIBUTE
          REALM_DEFAULT_GROUPS REALM_LOCALIZATIONS REALM_ENABLED_EVENT_TYPES REALM_EVENTS_LISTENERS
          REALM_REQUIRED_CREDENTIAL REALM_SMTP_CONFIG REALM_SUPPORTED_LOCALES REDIRECT_URIS
          REQUIRED_ACTION_CONFIG REQUIRED_ACTION_PROVIDER RESOURCE_SERVER_RESOURCE
          RESOURCE_ATTRIBUTE RESOURCE_POLICY RESOURCE_SERVER_SCOPE RESOURCE_SCOPE
          RESOURCE_SERVER_PERM_TICKET RESOURCE_URIS ROLE_ATTRIBUTE SCOPE_MAPPING SCOPE_POLICY
          USERNAME_LOGIN_FAILURE USER_ATTRIBUTE USER_CONSENT USER_CONSENT_CLIENT_SCOPE
          USER_FEDERATION_PROVIDER USER_FEDERATION_CONFIG USER_FEDERATION_MAPPER
          USER_FEDERATION_MAPPER_CONFIG USER_GROUP_MEMBERSHIP USER_REQUIRED_ACTION USER_ROLE_MAPPING
          USER_SESSION_NOTE WEB_ORIGINS";

Run the following command to create an ENV pointing to the sql file to be filtered:
```
export KC_BACKUP="./<Backup SQL Dump File>";
```
Example:
```
export KC_BACKUP="./cnccdbBackup.sql";
```
Run the following command to rearrange the dump file to make it in sequential insertion order:

for i in $KC_TABLES; do grep "INSERT INTO \`$i\`"$KC_BACKUP; done > <file name along with its location>

Example:

for i in $KC_TABLES; do grep "INSERT INTO \`$i\`"$KC_BACKUP; done > /tmp/restore.sql

Run the following command to Copy file into the pod:

kubectl cp <backup_file name>.sql <namespace>/<pod-name>:<directory where you want your file placed>

Example:

kubectl cp restore.sql cndbtier1/ndbmysqld-0:/home/mysql

Run the following command to connect to the SQL node of the NDB cluster or connect to the cnDBTier:

$ kubectl -n <cndbtier_namespace> exec -it <cndbtier_sql_pod_name> -c <cndbtier_sql_container_name>-- bash

Example:

$ kubectl -n cndbtier  exec -it ndbmysqld-0 -c mysqlndbcluster -- bash

Populate the Database with data using the file that you have, after filtering the sqldump file.

Run the following command to restore Database Data:

mysql -h 127.0.0.1 -u root -p <DB name> < <backup_filename>

Example:

mysql -h 127.0.0.1 -u root -p cnccdb < restore.sql

Login to the MySQL prompt and confirm that the databases are restored.
Run the following command to Delete the sql files copied into the pod after the restore process is complete and successful (by logging into the SQL node):
```
rm -rf <DB Schema file name>
rm -rf <backup_filename>
```
Example:
```
rm -rf occncc_rollback_iam_schema_<version>.sql
rm -rf restore.sql
```

CNCC Rollback

This section describes the procedure to Rollback CNCC.

Note:

CNC Console Upgrade and Rollback procedure depends on base Console version, see CNC Console Upgrade Sequence section for more details.
Existing release name must be used for upgrade or rollback.

Run the following command to check which revision you need to rollback :
```
$ helm history <release_name> -n <namespace>
```
Example:
```
$ helm history cncc -n cncc
```
Run the following command to rollback to the required revision :
```
$ helm history cncc -n cncc
```
Example:
```
$ helm rollback cncc 1 -n cncc
```

Fresh Install CNCC Core

Note:

Fresh installation of CNCC Core is applicable only while performing rollback from listed version to the base version. See CNC Console Rollback Sequence to check if fresh installation CNCC Core is needed.

Install the previous version of CNCC Core using previous release name (cncc-core or cncc). It installs M-CNCC Core and A-CNCC Core, if enabled.

Command:

$ helm install <release name> <helm repo> -f <custom_values_yaml> --namespace <namespace> --version <version no.>

Example:

$ helm install cncc ocscp-helm-repo/cncc -f occncc_custom_values_22.3.2.yaml --namespace cncc --version 22.3.2