22 5.1.4 Recovery Scenario 4 (Partial Server Outage with one NOAM server and one SOAM server intact)
For a partial outage with an NOAM server and an SOAM server intact and available, only base recovery of software is needed. The intact NO and SOAM servers are capable of restoring the database via replication to all servers. The major activities are summarized in the list below. Use this list to understand the recovery procedure summary. Do not use this list to execute the procedure. The actual procedures’ detailed steps are in Procedure 4. The major activities are summarized as follows:
Recover Standby NOAM server by recovering software.
- Recover the software.
The database is intact at the active NOAM server and does not require restoration at the standby NOAM server.
- Recover any failed SO and MP servers by recovering software.
- Recover the software.
The database in intact at the active NOAM server and does not require restoration at the SO and MP servers.
- Re-apply signaling networks configuration if the failed VM is an MP.
This procedure performs recovery if at least 1 NOAM server is intact and available and 1 SOAM server is intact and available.
Check off () each step as it is completed. Boxes have been provided for this purpose under each step number.
If this procedure fails, contact My Oracle Support (MOS), and ask for assistance.
Table 22-1 Procedure 8
STEP # | Procedure | Description |
---|---|---|
1 | Workarounds |
Refer to Release to understand/apply any workarounds required during this procedure. |
2 | Gather Required Materials |
Gather the documents and required materials listed in Section 3.1 Required Materials |
3 | NOAM VIP GUI: Login |
Establish a GUI session on the NOAM server by using the VIP IP address of the NOAM server. Open the web browser and enter a URL of: Figure 22-1 NOAM GUI Login ![]() |
4 | Active NOAM: Set Failed Servers to OOS |
Navigate to Main Menu -> Status & Manage -> HA Figure 22-2 HA ![]() Select Edit Set the Max Allowed HA Role drop down box to OOS for the failed servers. Select Ok Figure 22-3 Cancel ![]() |
5 | Recover the Failed Software |
For VMWare based deployments:
For KVM/Openstack based deployments:
For OVM-S/OVM-M based deployments: Execute the following procedures from reference [1]:
Note: While executing Procedure 8, configure the required failed VMs only (NOAMs/SOAMs/MPs) |
6 | Repeat for Remaining Failed Servers |
If necessary, repeat 5 for all remaining failed servers. |
7 | NOAM VIP GUI: Login |
Establish a GUI session on the NOAM server by using the VIP IP address of the NOAM server. Open the web browser and enter a URL of: Figure 22-4 NOAM GUI Login ![]() |
8 | NOAM VIP GUI: Recover Standby NOAM if needed |
Install the second NOAM server by executing procedures from reference [1]: Procedure 15 “Configure the Second NOAM Server” steps 1, 3-7 Procedure 16 “Complete Configuring the NOAM Server Group” Step 4 Note: If Topology or nodeId alarms are persistent after the database restore, refer to Workarounds for Issues not fixed in this Release, or the next step below. |
9 |
(OPTIONAL) NOAM VIP GUI: Recover the Failed SOAM Servers if needed |
If the failed server is an SOAM, recover the remaining SOAM servers (standby, spare) by repeating the following steps for each SOAM server: 1. Install the remaining SOAM servers by executing Procedure 22 “Configure the SOAM Servers”, steps 1, 3- 7 from reference [1]. Note: Wait for server to reboot before continuing. |
10 | (OPTIONAL) NOAM VIP GUI: Set HA on Recovered Servers |
Navigate to Status & Manage -> HA Figure 22-5 HA ![]() Click on Edit at the bottom of the screen. For each server whose Max Allowed HA Role is set to Standby, set it to Active. Press OK |
11 | NOAM VIP GUI: Restart DSR application |
Navigate to Main Menu->Status & Manage->Server, Figure 22-6 Server ![]() Select the recovered server and click on Restart. Figure 22-7 Restart ![]() |
12 | NOAM VIP GUI: Recover the C- Level Server (DA-MP, SBRs, IPFE, vSTP-MP) |
Establish a SSH session to the C Level server being recovered, login as admusr. Execute the following procedures from [1] FOR EACH server that has been recovered: Procedure 25 “Configure the MP Virtual Machines”, Steps 1, 8-14 (& 15 if required). |
13 | NOAM VIP GUI: Set HA on all C- Level Servers |
Navigate to Status & Manage -> HA Figure 22-8 HA ![]() Click on Edit at the bottom of the screen For each server whose Max Allowed HA Role is set to Standby, set it to Active Press OK |
14 | NOAM VIP GUI: Restart DSR Application on recovered C- Level Servers. |
Navigate to Main Menu->Status & Manage->Server Figure 22-9 Manage server ![]() Select the recovered servers and click on Restart. |
15 | ACTIVE NOAM: Login |
Login to the recovered Active NOAM via SSH terminal as admusr user. |
16 | ACTIVE NOAM: Perform key exchange between the active-NOAM and recovered servers. |
Establish an SSH session to the Active NOAM, login as admusr. Execute the following command to perform a keyexchange from the active NOAM to each recovered server: Figure 22-10 Key exchange ![]() |
17 | ACTIVE NOAM: Activate Optional Features |
Establish an SSH session to the active NOAM, login as admusr. Note: For PCA Activation: If you have PCA installed in the system being recovered, execute the procedure “PCA Activation on Stand By NOAM server” on recovered StandBy NOAM Server and procedure “PCA Activation on Stand By SOAM server” on recovered StandBy SOAM Server from [3] to re-activate PCARefer to 1.5 optional features to activate any features that were previously activated. Note:
|
18 | NOAM VIP GUI: Fetch and Store the database Report for the Newly Restored Data and Save it |
Navigate to Main Menu->Status & Manage->Database Figure 22-11 Database ![]() Select the active NOAM server and click on the Report button at the bottom of the page. The following screen is displayed: Figure 22-12 Database report ![]() Click on Save and save the report to your local machine. |
19 | ACTIVE NOAM: Verify Replication Between Servers. |
Login to the Active NOAM via SSH terminal as admusr user.
Execute the following command:
Figure 22-13 Output ![]() |
20 | NOAM VIP GUI: Verify the Database states |
Click on Main Menu->Status and Manager->Database Figure 22-14 Database ![]() Verify that the “OAM Max HA Role” is either “Active” or “Standby” for NOAM and SOAM and “Application Max HA Role” for MPs is “Active”, and that the status is “Normal” as shown below: Figure 22-15 main menu ![]() |
21 | NOAM VIP GUI: Verify the HA Status |
Click on Main Menu->Status and Manage->HA Figure 22-16 HA ![]() Select the row for all of the servers Verify that the “HA Role” is either “Active” or “Standby” Figure 22-17 Manage HA ![]() |
22 | SOAM VIP GUI: Verify the Local Node Info |
Navigate to Main Menu->Diameter->Configuration->Local Node Figure 22-18 Local node ![]() Verify that all the local nodes are shown. |
23 | SOAM VIP GUI: Verify the Peer Node Info |
Navigate to Main Menu->Diameter->Configuration->Peer Node Figure 22-19 Peer node ![]() Verify that all the peer nodes are shown. |
24 | SOAM VIP GUI: Verify the Connections Info |
Navigate to Main Menu->Diameter->Configuration->Connections Figure 22-20 Connections ![]() Verify that all the connections are shown. |
25 | For vSTP Only- SOAM VIP Server Console (Optional): Verify the local nodes info |
To verify the vSTP MP Local nodes info:
Figure 22-21 Output ![]() |
26 | For vSTP Only- SOAM VIP Server Console (Optional): Verify the remote nodes info |
To verify the vSTP MP Remote nodes info:
Figure 22-22 Output ![]() |
27 | For vSTP Only- SOAM VIP Server Console (Optional): Verify the Connections info |
To verify the vSTP MP Connections info:
Figure 22-23 Output ![]() |
28 | MP Servers: Disable SCTP Auth Flag |
For SCTP connections without DTLS enabled, refer to Enable/Disable DTLS Appendix from reference [1]. Execute this procedure on all Failed MP Servers. |
29 | SOAM VIP GUI: Enable Connections if needed |
Navigate to Main Menu->Diameter->Maintenance->Connections Figure 22-24 Connections ![]() Select each connection and click on the Enable button. Alternatively you can enable all the connections by selecting the EnableAll button. Figure 22-25 Enable all ![]() Verify that the Operational State is Available. |
30 | SOAM VIP GUI: Enable Optional Features |
Navigate to Main Menu -> Diameter -> Maintenance -> Applications Figure 22-26 Applications ![]() Select the optional feature application Click the Enable button. Figure 22-27 Enable disable ![]() |
31 | SOAM VIP GUI: Re-enable Transports if Needed |
Navigate to Main Menu->Transport Manager -> Maintenance -> Transport Figure 22-28 Transport ![]() Select each transport and click on the Enable button Figure 22-29 Enable disable ![]() Verify that the Operational Status for each transport is Up. |
32 | SOAM VIP GUI: Re-enable MAPIWF application if needed |
Figure 22-30 Local sccp ![]() Click on the Enable button corresponding to MAPIWF Application Name. Figure 22-31 Enable disable ![]() Verify that the SSN Status is Enabled. |
33 | SOAM VIP GUI: Re-enable links if needed |
Figure 22-32 Maintainence links ![]() Click on Enable button for each link. Figure 22-33 Enable ![]() Verify that the Operational Status for each link is Up. |
34 | SOAM VIP GUI: Examine All Alarms |
Navigate to Main Menu->Alarms & Events->View Active Figure 22-34 View active ![]() Examine all active alarms and refer to the on-line help on how to address them. If needed contact My Oracle Support (MOS). |
35 | NOAM VIP GUI: Examine All Alarms |
Login to the NOAM VIP if not already logged in. Navigate to Main Menu->Alarms & Events->View Active Examine all active alarms and refer to the on-line help on how to address them. If needed contact My Oracle Support (MOS). |
36 | Restart oampAgent if Needed |
Note: If alarm “10012: The responder for a monitored table failed to respond to a table change” is raised, the oampAgent needs to be restarted.Establish an SSH session to each server that has the alarm. Login as admusr. Figure 22-35 oampAgent ![]() Execute the following commands: |
37 | Backup and Archive All the Databases from the Recovered System |
Execute DSR Database Backup to back up the Configuration databases: |