1. Oracle Communications Diameter Signaling Router Cloud Disaster Recovery Guide
  2. 5.1.4 Recovery Scenario 4 (Partial Server Outage with one NOAM server and one SOAM server intact)

22 5.1.4 Recovery Scenario 4 (Partial Server Outage with one NOAM server and one SOAM server intact)

For a partial outage with an NOAM server and an SOAM server intact and available, only base recovery of software is needed. The intact NO and SOAM servers are capable of restoring the database via replication to all servers. The major activities are summarized in the list below. Use this list to understand the recovery procedure summary. Do not use this list to execute the procedure. The actual procedures’ detailed steps are in Procedure 4. The major activities are summarized as follows:

Recover Standby NOAM server by recovering software.

  • Recover the software.

The database is intact at the active NOAM server and does not require restoration at the standby NOAM server.

  • Recover any failed SO and MP servers by recovering software.
  • Recover the software.

The database in intact at the active NOAM server and does not require restoration at the SO and MP servers.

  • Re-apply signaling networks configuration if the failed VM is an MP.

This procedure performs recovery if at least 1 NOAM server is intact and available and 1 SOAM server is intact and available.

Check off () each step as it is completed. Boxes have been provided for this purpose under each step number.

If this procedure fails, contact My Oracle Support (MOS), and ask for assistance.

Table 22-1 Procedure 8

STEP # Procedure Description
1 Workarounds

Refer to Release to understand/apply any workarounds required during this procedure.

2 Gather Required Materials

Gather the documents and required materials listed in Section 3.1 Required Materials

3 NOAM VIP GUI: Login

Establish a GUI session on the NOAM server by using the VIP IP address of the NOAM server. Open the web browser and enter a URL of:

Figure 22-1 NOAM GUI Login


NOAM GUI Login
4 Active NOAM: Set Failed Servers to OOS

Navigate to Main Menu -> Status & Manage -> HA

Figure 22-2 HA


HA

Select Edit

Set the Max Allowed HA Role drop down box to OOS for the failed servers.

Select Ok

Figure 22-3 Cancel


Cancel
5 Recover the Failed Software

For VMWare based deployments:

  • For NOAMs execute the following procedures from reference [1]:
    • Procedure 1 (VMWare). Import DSR OVA [Note: If OVA is already imported and present in the Infrastructure Manager, skip this procedure of importing OVA]
    • Procedure 2 (VMWare Only). Configure NOAM guests based on resource profile
  • For SOAMs execute the following procedures from reference [1]:
    • Procedure 1 (VMWare). Import DSR OVA [Note: If OVA is already imported and present in the Infrastructure Manager, skip this procedure of importing OVA].
    • Procedure 3 (VMWare Only). Configure Remaining DSR guests based on resource profile.
  • For failed MPs execute the following procedures from reference [1]:
    • Procedure 1 (VMWare). Import DSR OVA [Note: If OVA is already imported and present in the Infrastructure Manager, skip this procedure of importing OVA]
    • Procedure 3 (VMWare Only). Configure Remaining DSR guests based on resource profile

For KVM/Openstack based deployments:

  • For NOAMs execute the following procedures from reference [1]:
    • Procedure 4 (KVM/Openstack). "Import DSR OVA" [Note: If OVA is already imported and present in the Infrastructure Manager, skip this procedure of importing OVA]
    • Procedure 5 (KVM/Openstack). "Configure NOAM guests based on resource profile"
  • For SOAMs execute the following procedures from reference [1]:
    • Procedure 4 (KVM/Openstack). "Import DSR OVA" [Note: If OVA is already imported and present in the Infrastructure Manager, skip this procedure of importing OVA]
    • Procedure 6 (KVM/Openstack). "Configure Remaining DSR guests based on resource profile"
  • For failed MPs execute the following procedures from reference [1]:
    • Procedure 4 (KVM/Openstack). "Import DSR OVA" [Note: If OVA is already imported and present in the Infrastructure Manager, skip this procedure of importing OVA]
    • Procedure 6 (KVM/Openstack). "Configure Remaining DSR guests based on resource profile”.

For OVM-S/OVM-M based deployments:

Execute the following procedures from reference [1]:
  • Procedure 7 (OVM-S/OVM-M). Import DSR OVA and prepare for VM creation
  • Procedure 8 (OVM-S/OVM-M). Configure each DSR VM

Note:

While executing Procedure 8, configure the required failed VMs only (NOAMs/SOAMs/MPs)
6 Repeat for Remaining Failed Servers

If necessary, repeat 5 for all remaining failed servers.
7 NOAM VIP GUI: Login

Establish a GUI session on the NOAM server by using the VIP IP address of the NOAM server. Open the web browser and enter a URL of:

Figure 22-4 NOAM GUI Login


NOAM GUI Login
8 NOAM VIP GUI: Recover Standby NOAM if needed

Install the second NOAM server by executing procedures from reference [1]:

Procedure 15 “Configure the Second NOAM Server” steps 1, 3-7

Procedure 16 “Complete Configuring the NOAM Server Group” Step 4

Note:

If Topology or nodeId alarms are persistent after the database restore, refer to Workarounds for Issues not fixed in this Release, or the next step below.
9

(OPTIONAL) NOAM VIP GUI: Recover the Failed SOAM Servers if needed

If the failed server is an SOAM, recover the remaining SOAM servers (standby, spare) by repeating the following steps for each SOAM server:

1. Install the remaining SOAM servers by executing Procedure 22 “Configure the SOAM Servers”, steps 1, 3- 7 from reference [1].

Note:

Wait for server to reboot before continuing.
10 (OPTIONAL) NOAM VIP GUI: Set HA on Recovered Servers

Navigate to Status & Manage -> HA

Figure 22-5 HA


HA

Click on Edit at the bottom of the screen.

For each server whose Max Allowed HA Role is set to Standby, set it to Active.

Press OK
11 NOAM VIP GUI: Restart DSR application

Navigate to Main Menu->Status & Manage->Server,

Figure 22-6 Server


Server

Select the recovered server and click on Restart.

Figure 22-7 Restart


Restart
12 NOAM VIP GUI: Recover the C- Level Server (DA-MP, SBRs, IPFE, vSTP-MP)

Establish a SSH session to the C Level server being recovered, login as admusr.

Execute the following procedures from [1] FOR EACH server that has been recovered:

Procedure 25 “Configure the MP Virtual Machines”, Steps 1, 8-14 (& 15 if required).
13 NOAM VIP GUI: Set HA on all C- Level Servers

Navigate to Status & Manage -> HA

Figure 22-8 HA


HA

Click on Edit at the bottom of the screen

For each server whose Max Allowed HA Role is set to Standby, set it to Active

Press OK
14 NOAM VIP GUI: Restart DSR Application on recovered C- Level Servers.

Navigate to Main Menu->Status & Manage->Server

Figure 22-9 Manage server


Manage server

Select the recovered servers and click on Restart.
15 ACTIVE NOAM: Login

Login to the recovered Active NOAM via SSH terminal as admusr user.

16 ACTIVE NOAM: Perform key exchange between the active-NOAM and recovered servers.

Establish an SSH session to the Active NOAM, login as admusr.

Execute the following command to perform a keyexchange from the active NOAM to each recovered server:

Figure 22-10 Key exchange


Key exchange
17 ACTIVE NOAM: Activate Optional Features

Establish an SSH session to the active NOAM, login as admusr.

Note:

For PCA Activation: If you have PCA installed in the system being recovered, execute the procedure “PCA Activation on Stand By NOAM server” on recovered StandBy NOAM Server and procedure “PCA Activation on Stand By SOAM server” on recovered StandBy SOAM Server from [3] to re-activate PCA

Refer to 1.5 optional features to activate any features that were previously activated.

Note:

  • While running the activation script, the following error message (and corresponding messages) output may be seen, this can safely be ignored: 
    iload#31000{S/W Fault}
  • If any of the MPs are failed and recovered, then these MP servers should be restarted after Activation of the feature.
  • The same procedure applies to all the application.
18 NOAM VIP GUI: Fetch and Store the database Report for the Newly Restored Data and Save it

Navigate to Main Menu->Status & Manage->Database

Figure 22-11 Database


Database

Select the active NOAM server and click on the Report button at the bottom of the page. The following screen is displayed:

Figure 22-12 Database report


Database report

Click on Save and save the report to your local machine.
19 ACTIVE NOAM: Verify Replication Between Servers.
Login to the Active NOAM via SSH terminal as admusr user. Execute the following command:

Figure 22-13 Output


Output
20 NOAM VIP GUI: Verify the Database states

Click on Main Menu->Status and Manager->Database

Figure 22-14 Database


Database

Verify that the “OAM Max HA Role” is either “Active” or “Standby” for NOAM and SOAM and “Application Max HA Role” for MPs is “Active”, and that the status is “Normal” as shown below:

Figure 22-15 main menu


main menu
21 NOAM VIP GUI: Verify the HA Status

Click on Main Menu->Status and Manage->HA

Figure 22-16 HA


HA

Select the row for all of the servers Verify that the “HA Role” is either “Active” or “Standby”

Figure 22-17 Manage HA


Manage HA
22 SOAM VIP GUI: Verify the Local Node Info

Navigate to Main Menu->Diameter->Configuration->Local Node

Figure 22-18 Local node


Local node

Verify that all the local nodes are shown.
23 SOAM VIP GUI: Verify the Peer Node Info

Navigate to Main Menu->Diameter->Configuration->Peer Node

Figure 22-19 Peer node


Peer node

Verify that all the peer nodes are shown.
24 SOAM VIP GUI: Verify the Connections Info

Navigate to Main Menu->Diameter->Configuration->Connections

Figure 22-20 Connections


Connections

Verify that all the connections are shown.
25 For vSTP Only- SOAM VIP Server Console (Optional): Verify the local nodes info

To verify the vSTP MP Local nodes info:

  • Login to the SOAM VIP Server console as admusr.
  • Execute the following command [admusr@SOAM1 ~]$ mmiclient.py /vstp/localhosts.
  • Verify the output similar to the below output.

Figure 22-21 Output


Output
26 For vSTP Only- SOAM VIP Server Console (Optional): Verify the remote nodes info

To verify the vSTP MP Remote nodes info:

  • Login to the SOAM VIP Server console as admusr
  • Execute the following command [admusr@SOAM1 ~]$ mmiclient.py /vstp/remotehosts.
  • Verify the output similar to the below output

Figure 22-22 Output


Output
27 For vSTP Only- SOAM VIP Server Console (Optional): Verify the Connections info

To verify the vSTP MP Connections info:

  • Login to the SOAM VIP Server console as admusr.
  • Execute the following command [admusr@SOAM1 ~]$ mmiclient.py /vstp/connections
  • Verify the output similar to the below output

Figure 22-23 Output


Output
28 MP Servers: Disable SCTP Auth Flag

For SCTP connections without DTLS enabled, refer to Enable/Disable DTLS Appendix from reference [1].

Execute this procedure on all Failed MP Servers.
29 SOAM VIP GUI: Enable Connections if needed

Navigate to Main Menu->Diameter->Maintenance->Connections

Figure 22-24 Connections


Connections

Select each connection and click on the Enable button.

Alternatively you can enable all the connections by selecting the EnableAll button.

Figure 22-25 Enable all


Enable all

Verify that the Operational State is Available.
30 SOAM VIP GUI: Enable Optional Features

Navigate to Main Menu -> Diameter -> Maintenance -> Applications

Figure 22-26 Applications


Applications

Select the optional feature application

Click the Enable button.

Figure 22-27 Enable disable


Enable disable
31 SOAM VIP GUI: Re-enable Transports if Needed

Navigate to Main Menu->Transport Manager -> Maintenance -> Transport

Figure 22-28 Transport


Transport

Select each transport and click on the Enable button

Figure 22-29 Enable disable


Enable disable

Verify that the Operational Status for each transport is Up.
32 SOAM VIP GUI: Re-enable MAPIWF application if needed

Figure 22-30 Local sccp


Local sccp

Click on the Enable button corresponding to MAPIWF Application Name.

Figure 22-31 Enable disable


Enable disable

Verify that the SSN Status is Enabled.
33 SOAM VIP GUI: Re-enable links if needed

Figure 22-32 Maintainence links


Maintainence links

Click on Enable button for each link.

Figure 22-33 Enable


Enable

Verify that the Operational Status for each link is Up.
34 SOAM VIP GUI: Examine All Alarms

Navigate to Main Menu->Alarms & Events->View Active

Figure 22-34 View active


View active

Examine all active alarms and refer to the on-line help on how to address them.

If needed contact My Oracle Support (MOS).
35 NOAM VIP GUI: Examine All Alarms

Login to the NOAM VIP if not already logged in.

Navigate to Main Menu->Alarms & Events->View Active

Examine all active alarms and refer to the on-line help on how to address them. If needed contact My Oracle Support (MOS).
36 Restart oampAgent if Needed

Note:

If alarm “10012: The responder for a monitored table failed to respond to a table change” is raised, the oampAgent needs to be restarted.

Establish an SSH session to each server that has the alarm. Login as admusr.

Figure 22-35 oampAgent


oampAgent

Execute the following commands:
37 Backup and Archive All the Databases from the Recovered System

Execute DSR Database Backup to back up the Configuration databases: