6 Rolling Back CNC Console

This section provides information about rolling back Cloud Native Configuration Console deployment to the previous release.

  • M-CNCC IAM
  • M-CNCC Core
  • A-CNCC Core

The following steps must be followed while performing the rollback:

  1. CNCC IAM DB Rollback/Restore
  2. CNCC Rollback

Caution:

It is recommended to verify the copy pasted content especially when the hyphens or any special characters are part of copied content.

User can rollback CNC Console from a source release to a target release using CLI procedures as outlined in the following table:

Rollback Task References Non-OCI Environment OCI Environment
Applicable for CLI Applicable for CNC Console deployment using OCI
Supported Rollback Paths Supported Rollback Paths Yes Yes
Pre-rollback Tasks Pre-rollback Tasks Yes Yes
M- CNCC IAM DB Rollback or Restore CNCC IAM DB Rollback or Restore Yes No
CNC Console Rollback CNCC Rollback Yes Yes
CNC Console Rollback CNC Console Rollback Yes Yes

Note:

For OCI:

The CNCC IAM DB Rollback/Restore is not applicable.

6.1 Supported Rollback Paths

This section describes the supported upgrade paths for CNC Console.

Table 6-1 CNC Console Rollback Sequence

Deployment Mode Source Version Target Version Rollback Sequence
Single Cluster 24.3.x 24.1.x or 24.2.x
  • cnDBTier Rollback
  • NF Rollback

    Rollback Instances (NF or CNE/OSO Common Services)

  • Console Rollback

    Rollback CNC Console
Multi Cluster 24.3.x 24.1.x or 24.2.x
  • cnDBTier Rollback
  • NF Rollback

    Rollback Instances (NF or CNE/OSO Common Services)

  • Console Agent Rollback

    Rollback A-CNCC Core

  • Console Manager Rollback

    Rollback CNC Console

Note:

For OCI:
  • Starting from version 24.1.0, the CNC Console supports deployment on OCI.
  • CNC Console multicluster feature is not supported.

Note:

CNC Console supports N-2 NF versions during upgrade window. For example, CNC Console 24.3.0 supports SCP 24.3.0, 24.2.x, and 24.1.x.

Any newly added features in Console which have NF dependency in latest release may not be available in previous release.

6.2 Pre-rollback Tasks

This section describes the tasks to be performed before a rollback.

6.2.1 M-CNCC IAM DB Rollback or Restore

This section provides details of CNC Console IAM Rollback. In case of CNC Console IAM Upgrade failure, rollback CNC Console IAM DB to previous version by following the below steps.

Note:

Not applicable for OCI deployment.

Note:

The latest backup must be used for rollback.

To perform a CNC Console IAM DB rollback or Restore:
  1. Log in to the deployment cluster, drop the existing database, and create a new database. Restore the new database with the DB Schema file provided as part of package (<occncc_rollback_iam_schema_<version>.sql>):

    Run the following command to drop the Database and create a new database:

    
DROP DATABASE <CNCC Database>
CREATE DATABASE IF NOT EXISTS <CNCC Database>;
    For example:
    
To be run in the mysql pod:
DROP DATABASE cnccdb;
CREATE DATABASE IF NOT EXISTS cnccdb;

    Note:

    You must take the <occncc_rollback_iam_schema_<version>.sql> file from the CNC Console version to which you are performing the rollback. For example, if you are performing rollback from version n to version n-2, you must use the <occncc_rollback_iam_schema_<version>.sql> file given in the CNC Console package of version n-2.
  2. Copy the DB Schema file provided as part of package into the MYSQl pod (occncc_rollback_iam_schema_<version>.sql).

    Run the following command to copy the DB Schema file:

    kubectl cp <occncc_rollback_iam_schema.sql> <namespace>/<pod-name>:<directory where you want your file placed>
    For example:
    kubectl cp occncc_rollback_iam_schema_<version>.sql cndbtier1/ndbappmysqld-0:/home/mysql
  3. Run the following command to connect to the SQL node of the NDB cluster or connect to the cnDbBTier:
    $ kubectl -n <cndbtier_namespace> exec -it <cndbtier_sql_pod_name> -c <cndbtier_sql_container_name> -- bash
    For example:
    $ kubectl -n cndbtier  exec -it ndbappmysqld-0 -c mysqlndbcluster -- bash
  4. Restore this new database with the DB Schema file provided as part of package (occncc_rollback_iam_schema_<version>.sql).
    Run the following command to Restore DB Schema:
    mysql -h 127.0.0.1 -u root -p <DB name> < <DB Schema file name>
    For example:
    mysql -h 127.0.0.1 -u root -p cnccdb < <occncc_rollback_iam_schema_<version>.sql
  5. The DB dump has to be rearranged sequentially not to get any foreign key constraints issue. For that, create the ENV variable,s and run it in a loop.
    1. Run the following command to convert the mysqldump file which was taken as a backup (sql.gz file) to a sql file to rearrange it:

      Unzipping the gz file:

      gunzip -d <backup_filename>.sql.gz
      For example:
      gunzip -d cnccdbBackup.sql.gz
    2. Rearrange the backup sql file in correct order by using following procedure:
      1. Run the following command to rearrange the Table Data : 
        export KC_TABLES="ADMIN_EVENT_ENTITY RESOURCE_SERVER RESOURCE_SERVER_POLICY
          ASSOCIATED_POLICY REALM CLIENT AUTHENTICATION_FLOW AUTHENTICATION_EXECUTION
          AUTHENTICATOR_CONFIG AUTHENTICATOR_CONFIG_ENTRY BROKER_LINK CLIENT_ATTRIBUTES
          CLIENT_AUTH_FLOW_BINDINGS KEYCLOAK_ROLE CLIENT_INITIAL_ACCESS CLIENT_NODE_REGISTRATIONS
          CLIENT_SCOPE CLIENT_SCOPE_ATTRIBUTES CLIENT_SCOPE_CLIENT CLIENT_SCOPE_ROLE_MAPPING
          USER_SESSION CLIENT_SESSION CLIENT_SESSION_AUTH_STATUS CLIENT_SESSION_NOTE
          CLIENT_SESSION_PROT_MAPPER CLIENT_SESSION_ROLE CLIENT_USER_SESSION_NOTE COMPONENT
          COMPONENT_CONFIG COMPOSITE_ROLE DATABASECHANGELOG USER_ENTITY CREDENTIAL
          DATABASECHANGELOGLOCK DEFAULT_CLIENT_SCOPE EVENT_ENTITY FEDERATED_IDENTITY FEDERATED_USER
          FED_USER_ATTRIBUTE FED_USER_CONSENT FED_USER_CONSENT_CL_SCOPE FED_USER_CREDENTIAL
          FED_USER_GROUP_MEMBERSHIP FED_USER_REQUIRED_ACTION FED_USER_ROLE_MAPPING KEYCLOAK_GROUP
          GROUP_ATTRIBUTE GROUP_ROLE_MAPPING IDENTITY_PROVIDER IDENTITY_PROVIDER_CONFIG
          IDENTITY_PROVIDER_MAPPER IDP_MAPPER_CONFIG MIGRATION_MODEL OFFLINE_CLIENT_SESSION
          OFFLINE_USER_SESSION POLICY_CONFIG PROTOCOL_MAPPER PROTOCOL_MAPPER_CONFIG REALM_ATTRIBUTE
          REALM_DEFAULT_GROUPS REALM_LOCALIZATIONS REALM_ENABLED_EVENT_TYPES REALM_EVENTS_LISTENERS
          REALM_REQUIRED_CREDENTIAL REALM_SMTP_CONFIG REALM_SUPPORTED_LOCALES REDIRECT_URIS
          REQUIRED_ACTION_CONFIG REQUIRED_ACTION_PROVIDER RESOURCE_SERVER_RESOURCE
          RESOURCE_ATTRIBUTE RESOURCE_POLICY RESOURCE_SERVER_SCOPE RESOURCE_SCOPE
          RESOURCE_SERVER_PERM_TICKET RESOURCE_URIS ROLE_ATTRIBUTE SCOPE_MAPPING SCOPE_POLICY
          USERNAME_LOGIN_FAILURE USER_ATTRIBUTE USER_CONSENT USER_CONSENT_CLIENT_SCOPE
          USER_FEDERATION_PROVIDER USER_FEDERATION_CONFIG USER_FEDERATION_MAPPER
          USER_FEDERATION_MAPPER_CONFIG USER_GROUP_MEMBERSHIP USER_REQUIRED_ACTION USER_ROLE_MAPPING
          USER_SESSION_NOTE WEB_ORIGINS";
      2. Run the following command to create an ENV pointing to the sql file to be filtered:
        export KC_BACKUP="./<Backup SQL Dump File>";
        For example:
        export KC_BACKUP="./cnccdbBackup.sql";
      3. Run the following command to rearrange the dump file to make it in sequential insertion order:
      for i in $KC_TABLES; do grep "INSERT INTO \`$i\`"$KC_BACKUP; done > <file name>
      Example:
      for i in $KC_TABLES; do grep "INSERT INTO \`$i\`"$KC_BACKUP; done > /tmp/restore.sql
  6. Run the following command to copy file into the pod:
    kubectl cp <backup_file name>.sql <namespace>/<pod-name>:<directory where you want your file placed>
    For example:
    kubectl cp restore.sql cndbtier1/ndbappmysqld-0:/home/mysql
  7. Run the following command to connect to the SQL node of the NDB cluster or connect to the cnDBTier:
    $ kubectl -n <cndbtier_namespace> exec -it <cndbtier_sql_pod_name> -c <cndbtier_sql_container_name>-- bash
    For example:
    $ kubectl -n cndbtier  exec -it ndbappmysqld-0 -c mysqlndbcluster -- bash
  8. Populate the Database with data using the file that you have, after filtering the sqldump file.
  9. Run the following command to restore Database Data: 
    mysql -h 127.0.0.1 -u root -p <DB name> < <backup_filename>
    For example:
    mysql -h 127.0.0.1 -u root -p cnccdb < restore.sql

    Note:

    Verify that the created restore.sql begins with the "INSERT INTO" statement. Anything appended before it must be removed.
  10. Log in to the MySQL prompt and confirm that the databases are restored.
  11. Run the following command to Delete the sql files copied into the pod after the restore process is complete and successful (by logging into the SQL node):
    
rm -rf <DB Schema file name>
rm -rf <backup_filename>
    Example:
    
rm -rf occncc_rollback_iam_schema_<version>.sql
rm -rf restore.sql

6.2.2 Prerollback Procedure for Dual Stack Networking

Note:

Not applicable for OCI deployment.

Refer to the following table to rollback CNC Console to any of the supported modes. The table outlines the necessary changes to be made before rolling back CNC Console.

Note:

Before performing rollback, see Updating CNC Console Services Before Performing Rollback to make necessary changes.

Table 6-2 Prerollback Configurations for Dual Stack Networking

Cluster DeploymentMode or Dual Stack preferred infrastructure Previously deployed CNC Console IP Family Policy "cnccDeploymentMode" configured in CNC Console custom values.yaml Service configuration with preferred IP Family Policy and IP Families (Pre Upgrade Step)
Dual Stack IPv4 preferred infrastructure

SingleStack IPv4 preferred

Having IP Families:
spec:
  ....
  ipFamilies:
  - IPv4
 ClusterPreferred This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv4 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv6 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv4
IPv4_IPv6 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv4
IPv6_IPv4 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv4

SingleStack IPv6 preferred

Having IP Families:
spec:
  ....
  ipFamilies:
  - IPv6
 ClusterPreferred 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv4
IPv4 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv4
IPv6 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv4_IPv6 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv6
IPv6_IPv4 
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv6

RequireDualStack IPv4 preferred

Having IP Families:
spec:
  ....
  ipFamilies:
  - IPv4_IPv6
 ClusterPreferred 
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv4
  - IPv6
IPv4 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv6 
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv4
  - IPv6
IPv4_IPv6 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv6_IPv4 
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv4
  - IPv6

RequireDualStack IPv6 preferred

Having IP Families: 
spec:
  ....
  ipFamilies:
  - IPv6_IPv4
 ClusterPreferred 
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv6
  - IPv4
IPv4 
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv6
  - IPv4
IPv6 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.
IPv4_IPv6 
spec:
  ipFamilyPolicy: RequireDualStack
  ipFamilies:
  - IPv6
  - IPv4
IPv6_IPv4 This will lead to no changes in the service file. Depending on the cluster, IPs will be assigned.

Updating CNC Console Services Before Performing Rollback

This section describes the prerollback procedure to be performed in case of Dual Stack Networking.

  1. List of console services to be verified and backed up for recovery process.
    1. Run the following commant to get the services that must be verified and backed up before performing CNC Console Rollback:
      $ kubectl get svc -n <namespace>
      For example: 
      $ kubectl get svc -n cncc
NAME                         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)             AGE
cncc-acore-igw-cache         ClusterIP      None            <none>          8000/TCP            3m19s
cncc-acore-ingress-gateway   ClusterIP      10.233.xx.xxx   <none>          80/TCP              3m19s
cncc-iam-igw-cache           ClusterIP      None            <none>          8000/TCP            3m19s
cncc-iam-ingress-gateway     LoadBalancer   10.233.xx.xxx   10.xx.xxx.xxx   80:32007/TCP        3m19s
cncc-iam-kc-headless         ClusterIP      None            <none>          8285/TCP            3m19s
cncc-iam-kc-http             ClusterIP      10.233.xx.xx    <none>          8285/TCP,8443/TCP   3m19s
cncc-mcore-cmservice         ClusterIP      10.233.xx.xxx   <none>          8442/TCP            3m19s
cncc-mcore-igw-cache         ClusterIP      None            <none>          8000/TCP            3m19s
cncc-mcore-ingress-gateway   LoadBalancer   10.233.xx.xxx   10.xx.xxx.xxx   80:30609/TCP        3m19s
 
 
# Note: The following have 'cncc' as the release name, which needs to be verified and backed up.      
cncc-acore-ingress-gateway 
cncc-iam-ingress-gateway           
cncc-iam-kc-http           
cncc-mcore-cmservice               
cncc-mcore-ingress-gateway
  
# These services do not have any clusterIP assigned. And hence can be ignored.
cncc-acore-igw-cache
cncc-mcore-igw-cache
cncc-iam-igw-cache
cncc-iam-kc-headless
    2. Run the following command to verify the service:
      $ kubectl get svc <service-name> -n <namespace>
      For example:
      $ kubectl get svc cncc-iam-ingress-gateway -n cncc
NAME                       TYPE           CLUSTER-IP         EXTERNAL-IP                           PORT(S)          AGE
cncc-iam-ingress-gateway   LoadBalancer   fd00:0:0:2::702b   10.xx.xx.xx,xxxx:b400:605:xxxx::a     80:30837/TCP     60s
    3. Run the following command to take a backup of service and duplicate it for recovery:
      # Command to take backup of service
$ kubectl get svc <service-name> -n namespace -o yaml > <service-name>.yaml;
 
# Command to duplicate service for recovery purpose
$ cp <service-name>.yaml <backup_file_name>.yaml;
      For example:
      $ kubectl get svc cncc-iam-ingress-gateway -n cncc -o yaml > cncc-iam-ingress-gateway.yaml;
$ cp cncc-iam-ingress-gateway.yaml cncc-iam-ingress-gateway_ipv6_ipv4.yaml;
  2. Run the following command to delete the existing console services:
    $ kubectl delete svc <service-name> -n <namespace>
    For example:
    $ kubectl delete svc cncc-iam-ingress-gateway -n cncc
  3. Update the service file <service-name>.yaml generated as part of step 1.c.

    Note:

    If service backup was taken before upgrade, then step 3.a can be skipped and you can jump to 3.b to apply the backup service file (<backup_file_name>.yaml) taken as part of Updating CNC Console Services Before Performing Upgrade step 1.c .
      1. Run the following command to edit <service-name>.yaml:
        $ vim <service-name>.yaml
        For example:
        $ vim cncc-iam-ingress-gateway.yaml
      2. Delete clusterIP and clusterIPs fields completely, and loadBalancerIP field if present.

        Delete the metallb.universe.tf/loadBalancerIPs annotation which would have been added only in case of IPv4_IPv6 or IPv6_IPv4 modes.

      3. Under spec.ipFamilyPolicy set SingleStack/RequiredDualStack and under spec.ipFamilies set [-IPv4, -IPv6, - IPv4 - IPv6, -IPv6 -IPv4] as per the cnccDeploymentMode configured.
      4. Run the following command to apply the updated <service-name>.yaml file:
        $ kubectl apply -f <service-name>.yaml -n <namespace>
        For example:
        $ kubectl apply -f cncc-iam-ingress-gateway.yaml -n cncc
    2. Run the following command to apply backup service file (<backup_file_name>.yaml) taken as part of Updating CNC Console Services Before Performing Upgrade step 1.c
      $ kubectl apply -f <backup_file_name>.yaml -n <namespace>
      For example:
      $ kubectl apply -f cncc-iam-ingress-gateway_ipv4.yaml -n cncc
  4. Run the following command to verify newly created service:
    $ kubectl get svc <service-name> -n <namespace>
    For example:
    $ kubectl get svc cncc-iam-ingress-gateway -n cncc 
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
cncc-iam-ingress-gateway   LoadBalancer   10.233.6.181   10.75.205.146   80:31549/TCP   35s

Sample Configuration Changes to Rollback CNC Console From IPv6_IPv4 to IPv4 on Dual Stack With IPv4 Preferred Infrastructure

The following example shows the configuration changes to be done before rolling back CNC Console on Dual Stack With IPv4 Preferred infrastructure for single service of CNC Console.You must follow a similar procedure for rest of the CNC Console services.

Deployment details:
  • CNC Console deployment namespace: cncc
  • CNC Console release name: cncc
  • Infra Deployment Mode: DualStack with IPv4 Preferred
  • CNC Console service to be edited: cncc-iam-ingress-gateway
  1. Get the cncc-iam-ingress-gateway service, verify it, and back it up for recovery
    1. Run the following command to get the cncc-iam-ingress-gateway service:
      $ kubectl get svc cncc-iam-ingress-gateway -n cncc
    2. Run the following command to verify that the cncc-iam-ingress-gateway service is set to IPv4 and IPv6 address:
      $ kubectl get svc cncc-iam-ingress-gateway -n cncc
  
NAME                       TYPE           CLUSTER-IP         EXTERNAL-IP                           PORT(S)          AGE
cncc-iam-ingress-gateway   LoadBalancer   fd00:0:0:2::702b   10.xx.xx.xx,xxxx:b400:605:xxxx::a     80:30837/TCP     60s
    3. Run the following command to take the backup of the cncc-iam-ingress-gateway service and duplicate it for recovery:
      $ kubectl get svc cncc-iam-ingress-gateway -n cncc -o yaml > cncc-iam-ingress-gateway.yaml;
  
$ cp cncc-iam-ingress-gateway.yaml cncc-iam-ingress-gateway_ipv6_ipv4.yaml;
  2. Run the following command to delete the existing cncc-iam-ingress-gateway service:
    $ kubectl delete svc cncc-iam-ingress-gateway -n cncc
  3. Update cncc-iam-ingress-gateway.yaml file either by following step 3.a or step 3.b.
      1. Run the following command to edit <service-name>.yaml>:
        $ vim <service-name>.yaml
        For example:
        $ vim cncc-iam-ingress-gateway.yaml
      2. Delete clusterIP and clusterIPs fields completely, and loadBalancerIP field if present. Delete annotation metallb.universe.tf/loadBalancerIPs which would be added only in case of IPv4_IPv6 or IPv6_IPv4 modes.:
        # Remove the clusterIP and clusterIPs field and loadBalancerIP field if present
...
spec:
  ...
  clusterIP: 1x.xxx.xx.xx
  clusterIPs:
  - 1x.xxx.xx.xx
  ...
  ...
  loadBalancerIP: 10.xx.xx.xx
...
      3. Under spec.ipFamilyPolicy set SingleStack and under spec.ipFamilies set [- IPv4] as follows:
        apiVersion: v1
kind: Service
metadata:
  ...
  name: cncc-iam-ingress-gateway
  namespace: cncc
spec:
  allocateLoadBalancerNodePorts: true
  ...
  ...
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ...
  ...
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer: {}
      4. Run the following command to apply the updated service file:
        $ kubectl apply -f cncc-iam-ingress-gateway.yaml -n cncc
    2. Apply backup service file (<backup_file_name>.yaml) taken as part of Updating CNC Console Services Before Performing Upgrade step 1.c:
      $ kubectl apply -f cncc-iam-ingress-gateway_ipv4.yaml -n cncc
  4. Run the following command to check the service output. You should find the service with updated the IPv4 and IPv6 addresses:
    $ kubectl get svc cncc-iam-ingress-gateway -n cncc
  
NAME                       TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
cncc-iam-ingress-gateway   LoadBalancer   10.233.6.181   10.75.205.146   80:31549/TCP   35s

6.3 CNC Console Rollback

This section describes the procedure to Rollback CNC Console.

Note:

Existing release name must be used for rollback.
  1. Rollback M-CNCC IAM DB.

    Note:

    Not applicable for OCI deployment.
  2. Run the following command to check which revision you need to rollback:
    $ helm history <release_name> -n <namespace>
    For example:
    $ helm history cncc -n cncc
  3. Run the following command to rollback to the required revision:
    $ helm rollback <release_name> <revision_number> -n <namespace>
    For example:
    $ helm rollback cncc 1 -n cncc