7 Fault Recovery
This chapter describes the procedures to perform fault recovery for Oracle Communications Cloud Native Core, Security Edge Protection Proxy (SEPP) deployment.
7.1 Overview
Note:
This section describes recovery procedures to restore SEPP completely or partially.7.2 Impacted Areas
The following table provides information about the impacted areas during SEPP fault recovery:
Table 7-1 Impacted Areas
| Scenario | Requires Fault Recovery or re-install of CNE? | Requires Fault Recovery or re-install of cnDBTier? | Requires Fault Recovery or re-install of SEPP? | Other |
|---|---|---|---|---|
| Scenario 1: Deployment Failure | No | No | Yes | SEPP DB is not restored. Only helm uninstall/install is done. |
| Scenario 2: cnDBTier Corruption | No | Yes | No (use helm upgrade if DB configuration is changed) | cnDBTier must be restored from backup and not re-install. If re-install of cnDBTier is needed, then CNE also need to re-installed. |
| Scenario 2A: When DBTier failed in Single or Multiple (but not all) Sites | No | Yes | No | NA |
| Scenario 2B: When DBTier failed in all Sites | No | Yes | No | NA |
| Scenario 3: Database Corruption | No | No | No | SEPP backup and restore of configuration database is required on impacted site. This needs automatic periodic backup. |
| Scenario 4: Site Failure | Yes | Yes | Yes | NA |
| Scenario 4A: Single or Multiple Site Failure | Yes | Yes | Yes | NA |
| Scenario 4B:All Site Failure | Yes | Yes | Yes | NA |
7.3 Prerequisites
- cnDBTier must be in a healthy state and available on multiple sites
along with SEPP. To check the cnDBTier status, perform the following steps:
- Run the following command to ensure that all the nodes are
connected:
ndb_mgm> show - Run the following command to check the pod
status:
kubectl get pods -n <namespace>If the pod status is
Running, then cnDBTier is in healthy state. - Run the following command to check if the replication is
up:
mysql> show slave status\GIn case there is any error, seeOracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
- Run the following command to check which cnDBTier has ACTIVE
replication to take
backup:
select * from replication_info.DBTIER_REPLICATION_CHANNEL_INFO;
- Run the following command to ensure that all the nodes are
connected:
- Automatic backup must be enabled on cnDBTier. Enabling automatic
backup helps in:
- restoring stable version of the SEPP database.
- minimizing significant loss of data due to upgrades or roll back failures.
- minimizing loss of data due to system failure.
- minimizing loss of data due to data corruption or deletion due to external input.
- migrating database information from one site to another.
- The following files must be available for fault recovery:
- Custom values file used at the time of network function deployment
- Helm charts used at the time of network function deployment
- Secrets and Certificates
- RBAC resources
7.4 Fault Recovery Scenarios
This section describes the fault recovery procedures for various scenarios.
7.4.1 Debug Tool
Overview
The Debug Tool provides third-party troubleshooting tools for debugging the runtime issues in a lab environment.
Following are the available tools:
- tcpdump
- ip
- netstat
- curl
- ping
- dig
Preconfiguration Steps
- Configuration in CNEPerform the following configurations in the Bastion Host. You need admin privileges to perform these configurations.
- When NEF is installed on CNE version
23.2.0 or above
Note:
- In CNE version 23.2.0 or above, the default CNE 23.2.0 Kyverno policy, disallow-capabilities, do not allow NET_ADMIN and NET_RAW capabilities that are required for debug tool.
- To run Debug tool on CNE 23.2.0 and above, the user must modify the existing Kyverno policy, disallow-capabilities, as below.
Adding a Namespace to an Empty Resource- Run the following command to verify if the
current disallow-capabilities cluster policy has
namespace in it.
Example:
Sample output:$ kubectl get clusterpolicies disallow-capabilities -oyamlapiVersion: kyverno.io/v1 kind: ClusterPolicy ... ... spec: rules: -exclude: any: -resources:{} - If there are no namespaces, then patch the
policy using the following command to add
<namespace> under
resources:
Example:$ kubectl patch clusterpolicy disallow-capabilities --type=json \ -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources", "value": {"namespaces":["<namespace>"]} }]'
Sample output:$ kubectl patch clusterpolicy disallow-capabilities --type=json \ -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources", "value": {"namespaces":["ocnef"]} }]'apiVersion: kyverno.io/v1 kind: ClusterPolicy ... ... spec: rules: -exclude: resources: namespaces: - sepp1 - If in case it is needed to remove the
namespace added in the above step, use the following
command:
Sample output:$ kubectl patch clusterpolicy disallow-capabilities --type=json \ -p='[{"op": "replace", "path": "/spec/rules/0/exclude/any/0/resources", "value": {} }]'apiVersion: kyverno.io/v1 kind: ClusterPolicy ... ... spec: rules: -exclude: any: -resources:{}
Adding a Namespace to an Existing Namespace List- Run the following command to verify if the
current disallow-capabilities cluster policy has
namespaces in it.
Example:
Sample output:$ kubectl get clusterpolicies disallow-capabilities -oyamlapiVersion: kyverno.io/v1 kind: ClusterPolicy ... ... spec: rules: -exclude: any: -resources: namespaces: -namespace1 -namespace2 -namespace3 - If there are namespaces already added, then
patch the policy using the following command to add
<namespace> to the existing
list:
Example:$ kubectl patch clusterpolicy disallow-capabilities --type=json \ -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/-", "value": "<namespace>" }]'
Sample output:$ kubectl patch clusterpolicy disallow-capabilities --type=json \ -p='[{"op": "add", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/-", "value": "seppsvc" }]'apiVersion: kyverno.io/v1 kind: ClusterPolicy ... ... spec: rules: -exclude: resources: namespaces: -namespace1 -namespace2 -namespace3 - sepp1 - If in case it is needed to remove the
namespace added in the above step, use the following
command:
Example:$ kubectl patch clusterpolicy disallow-capabilities --type=json \ -p='[{"op": "remove", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/<index>"}]'
Sample output:$ kubectl patch clusterpolicy disallow-capabilities --type=json \ -p='[{"op": "remove", "path": "/spec/rules/0/exclude/any/0/resources/namespaces/3"}]'apiVersion: kyverno.io/v1 kind: ClusterPolicy ... ... spec: rules: -exclude: resources: namespaces: -namespace1 -namespace2 -namespace3Note:
While removing the namespace, provide the index value for namespace within the array. The index starts from '0'.
- When NEF is installed on CNE version prior to 23.2.0
- When NEF is installed on CNE version
23.2.0 or above
- Log in to the Bastion Host.
- Create a new PSP by running the following command from the bastion
host. The parameters readOnlyRootFileSystem, allowPrivilegeEscalation,
allowedCapabilities are required by the debug container.
Note:
Other parameters are mandatory for PSP creation and can be customized as per the CNE environment. Default values are recommended.$ kubectl apply -f - <<EOF apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: debug-tool-psp spec: readOnlyRootFilesystem: false allowPrivilegeEscalation: true allowedCapabilities: - NET_ADMIN - NET_RAW fsGroup: ranges: - max: 65535 min: 1 rule: MustRunAs runAsUser: rule: MustRunAsNonRoot seLinux: rule: RunAsAny supplementalGroups: rule: RunAsAny volumes: - configMap - downwardAPI - emptyDir - persistentVolumeClaim - projected - secret EOF
Role Creation
kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: debug-tool-role
namespace: seppsvc
rules:
- apiGroups:
- policy
resources:
- podsecuritypolicies
verbs:
- use
resourceNames:
- debug-tool-psp
EOFRoleBinding Creation
$ kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: debug-tool-rolebinding
namespace: seppsvc
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: debug-tool-role
subjects:
- kind: Group
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccounts
EOFFor parameter details, see Debug Tool Configuration Parameters.
- Configuration in NF specific Helm
Following updates must be performed in custom_values.yaml file.
- Log in to the NF server.
- Open the custom_values
file:
$ vim <custom_values file> - Under global configuration, add the
following:
# Allowed Values: DISABLED, ENABLED podSecurityPolicy: "DISABLED" extraContainers: "DISABLED" debugToolContainerMemoryLimit: 4Gi extraContainersImageDetails: image: ocdebugtool/ocdebug-tools tag: debug_container_tag imagePullPolicy: Always extraContainersVolumesTpl: | - name: debug-tools-dir emptyDir: medium: Memory sizeLimit: {{ .Values.global.debugToolContainerMemoryLimit | quote }} extraContainersTpl: |- - command: - /bin/sleep - infinity name: tools resources: requests: ephemeral-storage: "512Mi" cpu: "0.5" memory: {{ .Values.global.debugToolContainerMemoryLimit | quote }} limits: ephemeral-storage: "512Mi" cpu: "1" memory: {{ .Values.global.debugToolContainerMemoryLimit | quote }} securityContext: allowPrivilegeEscalation: true capabilities: drop: - ALL add: - NET_RAW - NET_ADMIN runAsUser: 1012 volumeMounts: - mountPath: /tmp/tools name: debug-tools-dirNote:
-
Debug Tool Container comes up with the default user ID - 7000. If you want to override this default value, use the `runAsUser` field, or else, you can skip the field.
Default value: uid=7000(debugtool) gid=7000(debugtool) groups=7000(debugtool)
- In case you want to customize the container
name, replace the `name` field in the above values.yaml
with the
following:
This will ensure that the container name is prefixed and suffixed with the necessary values.name: {{ printf "%s-tools-%s" (include "getprefix" .) (include "getsuffix" .) | trunc 63 | trimPrefix "-" | trimSuffix "-" }}
-
- Under service specific configurations for which
debugging is required, add the
following:
# Allowed Values: DISABLED, ENABLED, USE_GLOBAL_VALUE extraContainers: USE_GLOBAL_VALUENote:
- At the global level,
extraContainersflag can be used to enable or disable injecting extra containers globally. This ensures that all the services that use this global value have extra containers enabled or disabled using a single flag. - At the service level,
extraContainersflag determines whether to use the extra container configuration from the global level or enable or disable injecting extra containers for the specific service.
- At the global level,
Run the Debug Tool
Following is the procedure to run Debug Tool.
- Run the following command to retrieve the POD
details:
$ kubectl get pods -n <k8s namespace>Example:
$ kubectl get pods -n seppsvcSample Output:NAME READY STATUS RESTARTS AGE ocsepp-release-appinfo-75894d8d8c-4zzkt 2/2 Running 0 5m54s ocsepp-release-cn32c-svc-5f5cdbfb7f-kspw6 2/2 Running 0 5m55s ocsepp-release-cn32f-svc-5458886cc7-nm7c8 2/2 Running 0 5m55s ocsepp-release-config-mgr-svc-6c94c449f-v8qnv 2/2 Running 0 5m55s ocsepp-release-n32-egress-gateway-55ccbbf46f-bb4tp 3/3 Running 0 5m54s ocsepp-release-n32-ingress-gateway-7bd984c9c6-pcpqd 3/3 Running 0 5m54s ocsepp-release-ocpm-config-65dd85d96d-59t4w 2/2 Running 0 5m54s ocsepp-release-performance-7456bbd8-2j7dx 2/2 Running 0 5m54s ocsepp-release-plmn-egress-gateway-67b7864664-cmcf8 3/3 Running 0 5m54s ocsepp-release-plmn-egress-gateway-67b7864664-lwhxz 3/3 Running 0 4m31s ocsepp-release-plmn-ingress-gateway-596c78f967-sc44c 3/3 Running 0 5m53s ocsepp-release-pn32c-svc-6498f6dc-lrvtt 2/2 Running 0 5m53s ocsepp-release-pn32f-svc-59bcb4c545-c4pqj 2/2 Running 0 5m53s ocsepp-release-sepp-nrf-client-nfdiscovery-9db8957cb-47j6g 1/1 Running 0 5m54s ocsepp-release-sepp-nrf-client-nfmanagement-5ddfd8d754-nbx69 1/1 Running 0 5m54s sepp-mysql-54b7c5699d-5nmzc 1/1 Running 0 3d23h - Run the following command to enter Debug Tool
Container:
$ kubectl exec -it <pod name> -c <debug_container name> -n <namespace> bashExample:$ kubectl exec -it ocsepp-release-cn32c-svc-5f5cdbfb7f-kspw6 -c tools -n seppsvc bash - Run the commands supported by debug
tools:
bash -4.2$ <debug_tools>Example:bash -4.2$ tcpdump - Copy the output files from container to
host:
$ kubectl cp -c <debug_container name> <pod name>:<file location in container> -n <namespace> <destination location>Example:$ kubectl cp -c tools ocsepp-release-cn32c-svc-5f5cdbfb7f-kspw6:/tmp/capture.pcap -n seppsvc /tmp/
Tools Tested in Debug Container
Following is the list of debugging tools that are tested.
Table 7-2 tcpdump
| Options Tested | Description | Output | Capabilities |
|---|---|---|---|
| -D | Print the list of the network interfaces available on the system and on which tcpdump can capture packets. | tcpdump -D
|
NET_ADMIN, NET_RAW |
| -i | Listen on interface | tcpdump -i eth0tcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes12:10:37.381199 IP ocsepp-plmn-ingress-gateway-7ffc49bb7f-2kkhc.46519 > kubernetes.default.svc.cluster.local.https: Flags [P.], seq 1986927241:1986927276, ack 1334332290, win 626, options [nop,nop,TS val 849591834 ecr 849561833], length 3512:10:37.381952 IP ocsepp-plmn--ingress-gateway-7ffc49bb7f-2kkhc.45868 > kube-dns.kube-system.svc.cluster.local.domain: 62870+ PTR? 1.0.96.10.in-addr.arpa. (40) |
NET_ADMIN, NET_RAW |
| -w | Write the raw packets to file rather than parsing and printing them out. | tcpdump -w capture.pcap -i
eth0 |
NET_ADMIN, NET_RAW |
| -r | Read packets from file (which was created with the -w option). | tcpdump -r capture.pcapreading from file /tmp/capture.pcap, link-type EN10MB (Ethernet)12:13:07.381019 IP ocsepp-plmn-ingress-gateway-7ffc49bb7f-2kkhc.46519 > kubernetes.default.svc.cluster.local.https: Flags [P.], seq 1986927416:1986927451, ack 1334332445, win 626, options [nop,nop,TS val 849741834 ecr 849711834], length 3512:13:07.381194 IP kubernetes.default.svc.cluster.local.https > ocsepp-plmn-ingress-gateway-7ffc49bb7f-2kkhc.46519: Flags [P.], seq 1:32, ack 35, win 247, options [nop,nop,TS val 849741834 ecr 849741834], length 3112:13:07.381207 IP ocsepp-plmn-ingress-gateway-7ffc49bb7f-2kkhc.46519 > kubernetes.default.svc.cluster.local.https: Flags [.], ack 32, win 626, options [nop,nop,TS val 849741834 ecr 849741834], length 0 |
NET_ADMIN, NET_RAW |
Table 7-3 ip
| Options Tested | Description | Output | Capabilities |
|---|---|---|---|
| addr show | Look at protocol addresses. |
ip addr show1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group defaultlink/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft forever2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group defaultlink/ipip 0.0.0.0 brd 0.0.0.04: eth0@if190: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1440 qdisc noqueue state UP group defaultlink/ether aa:5a:27:8d:74:6f brd ff:ff:ff:ff:ff:ff link-netnsid 0inet 192.168.219.112/32 scope global eth0valid_lft forever preferred_lft forever |
-- |
| route show | List routes | ip route showdefault via 169.254.1.1 dev eth0 169.254.1.1 dev eth0 scope link |
-- |
| addrlabel list | List address labels |
ip addrlabel listprefix ::1/128 label 0 prefix ::/96 label 3 prefix ::ffff:0.0.0.0/96 label 4 prefix 2001::/32 label 6 prefix 2001:10::/28 label 7 prefix 3ffe::/16 label 12 prefix 2002::/16 label 2 prefix fec0::/10 label 11 prefix fc00::/7 label 5 prefix ::/0 label 1 |
-- |
Table 7-4 netstat
| Options Tested | Description | Output | Capabilities |
|---|---|---|---|
| -a | Show both listening and non-listening (for TCP, this means established connections) sockets. | netstat -aActive Internet connections (servers and established)Proto Recv-Q Send-Q Local Address Foreign Address Statetcp 0 0 0.0.0.0:tproxy 0.0.0.0:* LISTENtcp 0 0 0.0.0.0:websm 0.0.0.0:* LISTENtcp 0 0 cncc-core-ingress:websm 10-178-254-194.ku:47292 TIME_WAITtcp 0 0 cncc-core-ingress:46519 kubernetes.defaul:https ESTABLISHEDtcp 0 0 cncc-core-ingress:websm 10-178-254-194.ku:47240 TIME_WAITtcp 0 0 cncc-core-ingress:websm 10-178-254-194.ku:47347 TIME_WAITudp 0 0 localhost:59351 localhost:ambit-lm ESTABLISHEDActive UNIX domain sockets (servers and established)Proto RefCnt Flags Type State I-Node Pathunix 2 [ ] STREAM CONNECTED 576064861 |
-- |
| -l | Show only listening sockets. | netstat -lActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address Statetcp 0 0 0.0.0.0:tproxy 0.0.0.0:* LISTENtcp 0 0 0.0.0.0:websm 0.0.0.0:* LISTENActive UNIX domain sockets (only servers)Proto RefCnt Flags Type State I-Node Path |
-- |
| -s | Display summary statistics for each protocol. | netstat -sIp:4070 total packets received0 forwarded0 incoming packets discarded4070 incoming packets delivered4315 requests sent outIcmp:0 ICMP messages received0 input ICMP message failed.ICMP input histogram:2 ICMP messages sent0 ICMP messages failedICMP output histogram:destination unreachable: 2 |
-- |
| -i | Display a table of all network interfaces. | netstat -iKernel Interface tableIface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flgeth0 1440 4131 0 0 0 4355 0 0 0 BMRUlo 65536 0 0 0 0 0 0 0 0 LRU |
-- |
Table 7-5 curl
| Options Tested | Description | Output | Capabilities |
|---|---|---|---|
| -o | Write output to <file> instead of stdout. |
curl -o file.txt
http://abc.com/file.txt |
-- |
| -x | Use the specified HTTP proxy. |
curl -x proxy.com:8080 -o
http://abc.com/file.txt |
-- |
Table 7-6 ping
| Options Tested | Description | Output | Capabilities |
|---|---|---|---|
| <ip> | Run a ping test to see whether the target host is reachable or not. |
ping 10.178.254.194 |
NET_ADMIN, NET_RAW |
| -c | Stop after sending 'c' number of ECHO_REQUEST packets. |
ping -c 5 10.178.254.194 |
NET_ADMIN, NET_RAW |
| -f (with non zero interval) | Flood ping. For every ECHO_REQUEST sent, a period ''.'' is printed, while for every ECHO_REPLY received a backspace is printed. |
ping -f -i 2 10.178.254.194 |
NET_ADMIN, NET_RAW |
Table 7-7 dig
| Options Tested | Description | Output | Capabilities |
|---|---|---|---|
| <ip> | It performs DNS lookups and displays the answers that are returned from the name server(s) that were queried. | dig 10.178.254.194Note: The IP should be reachable from inside the container. |
-- |
| -x | Query DNS Reverse lookup. | dig -x 10.178.254.194 |
-- |
7.4.2 Scenario 2: cnDBTier Corruption
This section describes how to recover database when the data replication is broken due to database corruption and cnDBTier has failed in single, multiple sites or all sites.
When the database corrupts, the database on all the other sites may also corrupt due to data replication. It depends on the replication status after the corruption has occurred. If the data replication is broken due to database corruption, then cnDBTier fails in either single or multiple sites (not all sites). And if the data replication is successful, then database corruption replicates to all the cnDBTier sites and cnDBTier fails in all sites.
The following are cnDBTier failure scenarios:
If corrupted database is replicated to mated sites, follow:
If corrupted database cause replication failure and hence local to a site, follow:
- When DBTier failed in all Sites
Note:
This scenario impacts all the NFs using the corrupted cnDBTier. All the NFs sharing cnDBTier needs to do a fault recovery as cnDBTier is corrupted.
7.4.2.1 When cnDBTier failed in Single or Multiple (but not all) Sites
This section describes how to recover database when the data replication is broken due to database corruption and cnDBTier has failed in either single or multiple sites (not all sites).
To recover database:
- Uninstall SEPP Helm chart. For information about uninstalling SEPP, see the Uninstalling SEPP section.
- For cnDBTier fault recovery:
- Create on-demand backup from mated site that has health replication with failed site. For more information about cnDBTier backup, see the "Create On-demand Database Backup" chapter in the Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
- Use the backup data from mate site for restore. For more
information about cnDBTier restore, see the "Restore Georeplication
Failure" chapter in Oracle Communications Cloud Native Core, cnDBTier
Installation, Upgrade, and Fault Recovery Guide.
Note:
The "Restore Georeplication Failure" chapter in Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide has a procedure for two sites where one of the cluster has fatal error. You can perform that procedure for all the sites in a multiple site setup.
- Install SEPP Helm chart. For more information about installing SEPP, see the Installing SEPP section.
7.4.2.2 When cnDBTier failed in all Sites
This section describes how to recover database when successful data replication corrupts all the cnDBTier sites.
To recover database:
- Uninstall SEPP helm charts. For more information about uninstalling SEPP, see the Uninstalling SEPP section.
- For cnDBTier fault recovery:
- Use on-demand backup file to restore database from the previous
data backup. For more information about cnDBTier restore, see the Restore
Georeplication Failure chapter in the Oracle Communications Cloud Native
Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
Note:
The Restore Georeplication Failure chapter has a procedure for two sites where one of the cluster has fatal error. You can perform that procedure for all the sites in a multiple site setup.
- Use on-demand backup file to restore database from the previous
data backup. For more information about cnDBTier restore, see the Restore
Georeplication Failure chapter in the Oracle Communications Cloud Native
Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
- Install SEPP helm charts. For more information about installing SEPP, see the see the Installing SEPP section.
7.4.3 Scenario 3: Configuration Database Corruption
This scenario describes how to recover SEPP when its configuration database corrupts.
The configuration database is stored in a site exclusive database along with its tables. Thus, corruption of configuration database impacts only a particular site.
- Transfer the
<backup_ filename >.sql.gzfile to the SQL node where user wants to restore it. - Log in to MySQL NDB Cluster's SQL node on the new DB cluster and create a new database where the database needs to be restored.
- For details on creating database and user and adding permissions,
see Configuring Database, Creating Users, and Granting Permissions
section.
Note:
The database name created in the above step should be same as the database name created in the following step. The Kubernetes secret must be the same as in the custom_values.yaml used later for Installing SEPP. - Use the following command to restore the database to the new
database:
gunzip < <backup_filename>.sql.gz | mysql -h127.0.0.1 –u <username> -p <backup-database-name>Enter the password when prompted.
Example:
gunzip < SEPPdbBackup.sql.gz | mysql -h127.0.0.1 -u dbuser -p seppdb
7.4.4 Scenario 4: Site Failure
This section describes how to perform fault recovery when either one, multiple, or all sites have a software failure. The following are site failure scenarios:
7.4.4.1 Single or Multiple Site Failure
This scenario applies when one or more sites, and not all sites, have failed and there is a requirement to perform fault recovery. It is assumed that the user has cnDBTier and SEPP installed on multiple sites with automatic data replication and backup enabled.
To recover the failed sites:
- Run the Cloud Native Environment (CNE) installation procedure to install a new cluster. For more information, see Oracle Communications Cloud Native Core, Cloud Native Environment Installation Guide.
- For cnDBTier fault recovery:
- Take on-demand backup from the mate site that has health replication with the failed site or sites. For more information about on-demand backup, see the "Create On-demand Database Backup" chapter in the Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
- Use the backup data from the mate site to restore the database. For more information about database restore, see "Restore Georeplication Failure" chapter in the Oracle Communications Cloud Native Core, cnDBTier Installation, Upgrade, and Fault Recovery Guide.
- Install SEPP Helm chart. For more information about installing SEPP, see the Uninstalling SEPP section.
7.4.4.2 All Sites Failure
This scenario applies when all sites have failed, and there is a requirement to perform fault recovery. It is assumed that the user has cnDBTier and SEPP installed on multiple sites with automatic data replication and backup enabled.
To recover all the failed sites:
- Run the Cloud Native Environment (CNE) installation procedure to install a new cluster. For more information, see Oracle Communications Cloud Native Core, Cloud Native Environment Installation, Upgrade, and Fault Recovery Guide.
- Use on-demand backup file to restore database from previous data backup.
For more information about database restore, see "Restore Georeplication Failure"
chapter in Oracle Communications Cloud Native Core, cnDBTier Installation,
Upgrade, and Fault Recovery Guide.
Note:
- The auto-data backup file is one that is built from scheduled automatic backup.
- The "Restore Georeplication Failure" chapter contains a procedure for two sites where one of the clusters has fatal error. You can perform the same procedure for all the sites in a multiple site setup.
- Install SEPP helm chart.