7 Fault Recovery

This chapter provides information about fault recovery for Oracle Communications Cloud Native Core, Service Communication Proxy (SCP) deployment.

7.1 Overview

You must take database backup and restore it either on the same or a different cluster. It uses the SCP database to run any command or follow instructions.

Note:

This document describes recovery procedures to restore SCP completely or partially.

Database Model of SCP

The SCP database consists of the following:

  • Configuration Data: The configuration data is exclusive for a given SCP instance. Therefore, an exclusive logical database is created and used by an SCP instance to store its configuration data. You can configure SCP instance specific configurations using RESTful config API exposed by scpc-configuration service through the Oracle Communications Cloud Native Configuration Console (CNC Console).
  • Routing Data: This is the routing rules data that SCP determines from Network Repository Function (NRF) in 5G Core network topology.
  • Status Data: This data provides the status of upgrade or rollback.
The following image shows SCP database model in three different sites.

Figure 7-1 Database Model

Database Model
This image represents how each SCP instance is using it's dedicated database. The data is getting replicated to all other sites of the cnDBTier cluster so that the data is available on all the cnDBTier cluster sites in case of a cnDBTier instance failure.

Note:

To recover cnDBTier through automated backup file or on-demand backup from mate site, see the restore procedure in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.

7.2 Impacted Areas

The following table shares information about the impacted areas during SCP fault recovery:

Table 7-1 Impacted Areas

Scenario Requires Fault Recovery or Reinstallation of CNE Requires Fault Recovery or Reinstallation of cnDBTier Requires Fault Recovery or Reinstallation of SCP Requires SCP Service Restart
Scenario 1: Recovering SCP (SCP services) when its deployment corrupts No No Yes NA
Scenario 2: Recovering corrupted cnDBTier No Yes No, use Helm upgrade of the same SCP version to update the SCP configuration if required. For example, change in cnDBTier service information, such as cnDB endpoints, DB credentials, and so on. Requires SCPC-Configuration service restart by using the kubectl delete <scpc-configuration pod> -n <namespace> command.
Scenario 3: Recovering corrupted SCP configuration and routing database No No No Requires SCPC-Configuration service restart by using the kubectl delete <scpc-configuration pod> -n <namespace> command.
Scenario 4: Complete site failure due to infrastructure failure, for example, hardware, CNE, and so on. Yes Yes Yes NA

7.3 Prerequisites

Before performing any fault recovery procedure, ensure that the following prerequisites are met:

  • cnDBTier must be in a healthy state and available on multiple sites along with SCP. If cnDBTier is unhealthy, see Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide
  • Automatic backup must be enabled on cnDBTier. Scheduled regular backups help to:
    • Restore stable version of the SCP database
    • Minimize significant loss of data due to upgrade or rollback failure
    • Minimize loss of data due to system failure
    • Minimize loss of data due to data corruption or deletion due to external input
    • Migrate the SCP database information from one site to another
  • Docker images used during the last installation or upgrade must be retained in the external data source.
  • Custom values file used at the time of SCP deployment is retained. If the custom_values.yaml file is not retained, then regenerate it manually. This task increases the overall fault recovery time.

Note:

Do not change DB Secret or cnDBTier MySQL FQDN or IP or PORT configurations.