Rolling Back SEPP

5 Rolling Back SEPP

This chapter provides information about rolling back Oracle Communications Cloud Native Core, Security Edge Protection Proxy (SEPP) deployment to the previous release. It is recommended to perform SEPP rollback in a specific order. For more information about the rollback order, see Oracle Communications Cloud Native Core, Solution Upgrade Guide.

Following steps must be followed while performing rollback:

SEPP DB Rollback or Restore
SEPP Rollback

Note:

It is recommended to verify the copy pasted content especially when the hyphens or any special characters are part of copied content.

5.1 Supported Rollback Paths

The following table lists the supported rollback paths for SEPP:

Note:

SEPP deployed in SEPP mode can be rolled back only to SEPP mode.
SEPP deployed in Roaming Hub mode can be rolled back only to Roaming Hub mode.

Table 5-1 Supported Rollback Paths

Source Release	Target Release
25.2.100	25.1.2xx
25.2.100	25.1.1xx

Table 5-2 Supported Rollback Paths

Source Roaming Hub Release	Target Roaming Hub Release
25.2.100	25.1.2xx
25.2.100	25.1.1xx

Note:

When performing a rollback from 25.2.1x to 24.2.x release, ensure ndb_allow_copying_alter_table in cnDBTier is set to ON.

5.2 Pre-rollback Tasks

This section describes the tasks to be performed before a rollback.

5.2.1 SEPP Instance DB Rollback or Restore

This section provides details of SEPP instance DB Rollback. In case of SEPP Upgrade failure, rollback SEPP DB to previous version by following the below steps.

Note:

The latest backup must be used for rollback.

To perform a SEPP DB rollback or Restore:

Note:

The following are the summary of steps to be performed:

Log in to the deployment cluster, drop the existing database, and create a new database.
Run the DB Schema file provided as part of package <ocsepp_rollback_schema_<version>.sql> to create the SEPP database schema in the newly created database.
- Use the schema file from the SEPP version to which you are performing the rollback.
Rearrange the DB Dump file (taken during SEPP instance DB backup) sequentially so that there wont be any foreign key constraints issue.
Run the following command to restore Database with rearranged sql file:
```
mysql -h 127.0.0.1 -u occneuser -p seppdb_sepp1 < restore.sql
```

Perform the following steps to do a SEPP DB rollback or Restore:

Run the following command to drop the Database and create a new database:
```
DROP DATABASE <SEPP Database>
CREATE DATABASE IF NOT EXISTS <SEPP Database>;
```
For example:
```
#To be run in the mysql pod:
DROP DATABASE seppdb_sepp1;
CREATE DATABASE IF NOT EXISTS seppdb_sepp1;
```
Note:
- The Database should be dropped from all the sites in case of multi site replication.
- You must take the <ocsepp_rollback_schema_<version>.sql> file from the SEPP version to which you are performing the rollback.
Copy the DB Schema file provided as part of package into the MySQL pod (<ocsepp_rollback_schema_<version>.sql>).
Run the following command to copy the DB Schema file to pod:
```
kubectl cp<ocsepp_rollback_schema_<version>.sql> <namespace>/<pod-name>:<directory where you want your fileplaced>
```
For example:
```
kubectl cp ocsepp_rollback_schema_<version>.sql gr-cndb-site1/ndbappmysqld-0:/home/mysql
```
Run the following command to connect to the SQL node of the NDB cluster or connect to the cnDBTier:
```
$ kubectl -n <cndbtier_namespace> exec -it <cndbtier_sql_pod_name> -c <cndbtier_sql_container_name> -- bash
```
For example:
```
$ kubectl -n cndbtier  exec -it ndbappmysqld-0 -c mysqlndbcluster -- bash
```
Restore this new database with the DB Schema file provided as part of package (ocsepp_rollback_schema_<version>.sql).
Run the following command to Restore DB Schema:
```
mysql -h 127.0.0.1 -u root -p <DB name> < <DB Schema file name>
```
For example:
```
mysql -h 127.0.0.1 -u root -p seppdb_sepp1 < ocsepp_rollback_schema_<version>.sql
```

The DB dump has to be rearranged sequentially not to get any foreign key constraints issue. For that, create the ENV variables and run it in a loop.

Run the following command to convert the mysqldump file which was taken as a backup (sql.gz file) to a sql file to rearrange it:
Unzipping the gz file:
```
gunzip -d <backup_filename>.sql.gz
```
For example:
```
gunzip -d sepp_sepp1_dbBackup.sql.gz
```

Rearrange the backup sql file in correct order by using following procedure:

Run the following command to rearrange the Table Data :

export KC_TABLES="APP_STATE MEDIATION_CONFIG_RELEASE MEDIATION_RULE OCMEDIATION_ROLLBACK_ACTION OCMEDIATION_SYSTEM_OPTIONS ROLLBACK_ACTION ReleaseConfig SiteJsonSchemaVersionInfo UPGRADE_ROLLBACK_EVENTS common_configuration common_configuration_backup snapshot_mapping topic_info topic_info_backup configuration_item configuration_item_backup NrfLeaderPod leader_election discovery_requests_nologging producer_profiles_nologging header topology_body roaming_partner_profile n32f_db_context roaming_partner_set plmn actual_values pseudo_values topology_option default_service_api_list_configuration security_allowed_list_action security_allowed_list_content security_allowed_list_rule security_allowed_list_content_rule sepp_feature_config mediation_trigger_rule_list mediation_trigger_rule mediation_trigger_rule_error_config plmn_validation_header_path_config plmn_validation_body_config security_plmn_list_action security_allowed_plmn_content security_plmn_list_rule security_plmn_list_content_rule sepp_sor_config sepp_sor_allowed_rss_list sepp_sor_url_list sepp_sor_allowed_url_join sepp_feature_error_config ingress_rate_limiting message_validation_list_config message_validation_rule_config message_validation_list_rule_mapping message_validation_api_config message_validation_feature_config previous_location_list_config previous_location_rule_config previous_location_list_rule_config previous_location_serving_header_config previous_location_serving_body_config previous_location_ue_header_config previous_location_ue_body_config cache_refresh_config egress_rate_limit_config egress_rate_limit_list_config sepp_service_log_configuration orig_nw_id_hdr_support verbose_err_rsp_config";

Run the following command to create an ENV pointing to the sql file to be filtered:
```
export KC_BACKUP="./<Backup SQL Dump File>";
```
For example:
```
export KC_BACKUP="./sepp_sepp1_dbBackup.sql";
```
Run the following command to verify that the ENV variables are set:
```
echo$KC_TABLES
echo$KC_BACKUP
```

Run the following command to rearrange the dump file to make it in sequential insertion order:

for i in $KC_TABLES; do if [ "$i" = "topic_info" ]; then echo "DELETE FROM \`$i\` WHERE NAME='nrfclient.cfg';"; fi;  grep "INSERT INTO \`$i\`" "$KC_BACKUP"; done > <file name along with its location>

Example:

for i in $KC_TABLES; do if [ "$i" = "topic_info" ]; then echo "DELETE FROM \`$i\` WHERE NAME='nrfclient.cfg';"; fi;  grep "INSERT INTO \`$i\`" "$KC_BACKUP"; done > /tmp/restore.sql

Run the following command to provide execution permission to the file:
```
chmod +x /tmp/restore.sql
```

Run the following command to copy file into the pod:

kubectl cp <backup_file name>.sql <namespace>/<pod-name>:<directory where you want your file placed>

For example:

kubectl cp restore.sql cndbtier1/ndbappmysqld-0:/home/mysql

Run the following command to connect to the SQL node of the NDB cluster or connect to the cnDBTier:
```
$ kubectl -n <cndbtier_namespace> exec -it <cndbtier_sql_pod_name> -c <cndbtier_sql_container_name>-- bash
```
For example:
```
$ kubectl -n cndbtier  exec -it ndbappmysqld-0 -c mysqlndbcluster -- bash
```
Populate the Database with data using the file that you have, after filtering the sqldump file.
Run the following command to restore Database Data:
```
mysql -h 127.0.0.1 -u root -p <DB name> < <backup_filename>
```
For example:
```
mysql -h 127.0.0.1 -u root -p seppdb_sepp1 < restore.sql
```
Note:
Verify that the created restore.sql begins with the "INSERT INTO" statement. Anything appended before it must be removed.
Log in to the MySQL prompt and confirm that the databases are restored.
Run the following command to delete the sql files copied into the pod after the restore process is complete and successful (by logging into the SQL node):
```
rm -rf <DB Schema file name>
rm -rf <backup_filename>
```
Example:
```
rm -rf ocsepp_rollback_schema_<version>.sql
rm -rf restore.sql
```

5.3 Rollback Steps

Perform this procedure to roll back SEPP deployment from 25.2.100 to the previously supported versions:

Caution:

No configuration should be performed during rollback.
Do not exit from helm rollback command manually. After running the helm rollback command, it takes some time (depending upon number of pods to rollback) to rollback all of the services. In the meantime, you must not press "ctrl+c" to come out from helm rollback command. It may lead to anomalous behavior.
Ensure that no SEPP pod is in the failed state.
If the Helm rollback results in failure and displays Rollback "ocsepp" failed: no ConfigMap with the name "rss-ratelimit-map" found error, it indicates that Helm is not able to merge current and rollback charts. Then run the Helm rollback again with --force.
If the Helm rollback results in failure and displays ConfigMap <Egress Rate Limiting ConfigMap Name> in namespace <Namespace> exists and cannot be imported into the current release: invalid ownership metadata error, do the following:
Delete the Egress Rate Limit ConfigMap using the following command: kubectl delete cm <Egress Rate Limiting ConfigMap Name> -n <namespace> and run the Helm rollback again.

Note:

SEPP might raise alerts while performing rollback. To clear any alert, see "Alerts" section in Oracle Communications Cloud Native Core, Security Edge Protection Proxy User Guide.

Run the following command to check the revision you must roll back to:
```
helm history <release_name> -n <release_namespace>
```
Where,
- <release_name> is the release name used by the Helm command.
- <release_namespace> is the namespace where SEPP is deployed.
For example:
```
helm history ocsepp --namespace seppsvc
```
Run the command to rollback to the required revision:
```
helm rollback <release_name> <revision_number> --namespace <release_namespace>
```
Where, <revision_number> is the release number to which SEPP needs to be rolled back.

For example:
```
helm rollback ocsepp 1 --namespace seppsvc
```
If the rollback fails, see "Upgrade or Rollback Failure" section in Oracle Communications Cloud Native Core, Security Edge Protection Proxy Troubleshooting Guide.

5.4 Post Rollback Tasks

After performing rollback, restore preupgrade data obtained earlier through manual backup.

GET API for different resources can be used to see current values in SEPP, then accordingly if any update required, individual service APIs defined for different resources can be used to reconfigure the data backup taken before upgrade.

Caution:

Run the following command to delete the rateLimitingConfigMap ConfigMap manually, post rollback (applicable only for the roll back to 22.4.x or previous versions):

kubectl delete cm <rateLimitingConfigMap> -n <release-namespace>

See Oracle Communications Cloud Native Core, Security Edge Protection Proxy (SEPP) Rest Specification for API details.