21 5.1.3 Recovery Scenario 3 (Partial Server Outage with all NOAM servers failed and one SOAM server intact)
For a partial server outage with SOAM server intact and available, NOAM servers are recovered using recovery procedures for software and then running a database restore to the active NOAM server using a NOAM database backup file obtained from external backup sources such as customer servers. All other servers are recovered using recovery procedures for software. Database replication from the Active NOAM or Active SOAM server will recover the database on these servers. The major activities are summarized in the list below. Use this list to understand the recovery procedure summary. Do not use this list to run the procedure. The actual procedures detailed steps are in Procedure 7. The major activities are summarized as follows:
Recover Active NOAM server by recovering software and the database.
- Recover the software.
- Recover the database
Recover Standby NOAM servers by recovering software.
- Recover the software.
Recover any failed SOAM and MP servers by recovering software.
- Recover the software.
- Database is already intact at one SOAM server and does not require restoration at the other SOAM and MP servers.
This procedure performs recovery if all NOAM servers are failed but 1 or more SOAM servers are intact. This includes any SOAM server that is in another location (spare SOAM server).
Check each step as it is completed. Boxes have been provided for this purpose under each step number.
If this procedure fails, contact My Oracle Support (MOS), and ask for assistance.
Table 21-1 Procedure 7
| STEP # | Procedure | Description |
|---|---|---|
| 1 | Workarounds |
Refer to workarounds to understand any procedure. |
| 2 | Gather Required Materials |
Gather the documents and required materials listed in section 3.1 Required Materials. |
| 3 | Recover the Failed Software |
For VMware based deployments:
For KVM and Openstack based deployments:
For OVM-S and OVM-M based deployments: Run the following procedures from reference [1]:
Note: While executing Procedure 8, configure the required failed VMs only (NOAMs, SOAMs, and MPs) |
| 4 | Obtain Latest Database Backup and Network Configuration Data. |
Obtain the most recent database backup file from external backup sources (ex. file servers) or tape backup sources. From required materials list in Section 3.1 Required Materials; use site survey documents and Network Element report (if available), to determine network configuration data. |
| 5 | Run DSR Installation Procedure for the First NOAM |
Verify the networking data for Network Elements
Note: Use the backup copy of network configuration data and site surveys (Step 2)Run installation procedures for the first NOAM server from reference [1]: Procedure 13 “Configure the First NOAM NE and Server”. Procedure 14 “Configure the NOAM Server Group”. |
| 6 | NOAM GUI: Login |
Login to the NOAM GUI as the guiadmin user: Figure 21-1 NOAM GUI Login  |
| 7 | NOAM GUI: Upload the Backed up Database File |
Browse to Main Menu->Status & Manage->Files Figure 21-2 Files  Select the Active NOAM server. The following screen will appear: Figure 21-3 Status and manage  Click on Upload as shown below and select the file “NO Provisioning and Configuration:” file backed up after initial installation and provisioning. Figure 21-4 Upload 
Figure 21-5 Backup file box  Click on the Upload button. The file will take a few seconds to upload depending on the size of the backup data. The file will be visible on the list of entries after the upload is complete. |
| 8 | NOAM GUI: Disable Provisioning |
Click on Main Menu->Status & Manage->Database Figure 21-6 Database  Disable Provisioning by clicking on Disable Provisioning button at the bottom of the screen as shown below. Figure 21-7 Disable provisioning  A confirmation window will appear, press OK to disable Provisioning. Figure 21-8 Disable  The message “Warning Code 002” will appear. |
| 9 | NOAM GUI: Verify the Archive Contents and Database Compatibility |
Select the Active NOAM server and click on the Compare. Figure 21-9 Enable  The following screen is displayed; click the button for the restored database file that was uploaded as a part of Step 7 of this procedure. Figure 21-10 Database compare  Verify that the output window matches the screen below. Note: You will get a database mismatch regarding the NodeIDs of the VMs. That is expected. If that is the only mismatch, proceed, otherwise stop and contact My Oracle Support (MOS)Figure 21-11 Database archive  Note: Archive Contents and Database Compatibilities must be the following:Archive Contents: Configuration data Database Compatibility: The databases are compatible. Note: The following is expected Output for Topology Compatibility Check since we are restoring from existing backed up data base to database with just one NOAM:Topology Compatibility THE TOPOLOGY SHOULD BE COMPATIBLE MINUS THE NODEID. Note: We are trying to restore a backed up database onto an empty NOAM database. This is an expected text in Topology Compatibility.If the verification is successful, Click BACK button and continue to next step in this procedure. |
| 10 | ACTIVE NOAM: Restore the Database |
Click on Main Menu->Status & Manage->Database Select the Active NOAM server, and click on Restore as shown below. The following screen will be displayed. Select the proper back up provisioning and configuration file. Figure 21-12 Database compare  Click OK Button. The following confirmation screen will be displayed. Note: You will get a database mismatch regarding the NodeIDs of the servers. That is expected. If that is the only mismatch, proceed, otherwise stop and contact My Oracle Support (MOS)Select the Force checkbox as shown above and Click OK to proceed with the DB restore. Figure 21-13 Database Restore Confirm  Note: After the restore has started, the user will be logged out of XMI NO GUI since the restored Topology is old data. |
| 11 | NOAM VIP GUI: Login |
Establish a GUI session on the NOAM server by using the VIP IP address of the NOAM server. Open the web browser and enter a URL of: Figure 21-14 NOAM GUI Login  |
| 12 | NOAM VIP GUI: Monitor and Confirm database restoral |
Wait for 5-10 minutes for the System to stabilize with the new topology: Monitor the Info tab for “Success”. This will indicate that the backup is complete and the system is stabilized. Following alarms must be ignored for NOAM and MP Servers until all the Servers are configured: Alarms with Type Column as “REPL” , “COLL”, “HA” (with mate NOAM), “DB” (about Provisioning Manually Disabled) Note: Do not pay attention to alarms until all the servers in the system are completely restored. Note: The Configuration and Maintenance information will be in the same state it was backed up during initial backup. |
| 13 | ACTIVE NOAM: Login |
Login to the recovered Active NOAM via SSH terminal as admusr user. |
| 14 | NOAM VIP GUI: Re- enable Provisioning |
Navigate to Main Menu->Status & Manage->Database Figure 21-15 Enable  Click on the Enable Provisioning. A pop-up window will appear to confirm as shown below, press OK. Figure 21-16 Enable provisioning  |
| 15 | NOAM VIP GUI: Recover Standby NOAM |
Install the second NOAM server by executing procedures from reference [1]: Procedure 15 “Configure the Second NOAM Server” steps 1, 3-7 |
| 16 | NOAM VIP GUI: Recover Standby NOAM |
Navigate to Main Menu->Status & Manage->Server and select the second NOAM server. Figure 21-17 Server  Click Restart. Figure 21-18 Restart  Click OK on the confirmation screen. Note: If Topology or nodeId alarms are persistent after the database restore, refer to Workarounds for Issues not fixed in this Release or the next step below. |
| 17 | NOAM VIP GUI: Recover remaining failed SOAM Servers |
Recover the remaining SOAM servers (standby, spare) by repeating the following steps for each SOAM server:
Wait for server to reboot before continuing. |
| 18 | NOAM VIP GUI: Restart DSR application |
Navigate to Main Menu->Status & Manage->Server, Figure 21-19 Server  Select the recovered server and click on Restart. Figure 21-20 Restart |
| 19 | NOAM VIP GUI: Set HA on all C-Level Servers |
Navigate to Status & Manage -> HA Figure 21-21 HA  Click on Edit at the bottom of the screen. For each server whose Max Allowed HA Role is not Active, set it to Active. Press OK |
| 20 | NOAM VIP GUI: Restart DSR application |
Navigate to Main Menu->Status & Manage->Server, Figure 21-22 Server Select each recovered server and click on Restart. Figure 21-23 Restart  |
| 21 | ACTIVE NOAM: Perform key exchange between the active-NOAM and recovered servers. |
Establish an SSH session to the Active NOAM, login as admusr. Run the following command to perform a keyexchange from the active NOAM to each recovered server: Figure 21-24 Key exchange  Note: If an export server is configured, perform this step. |
| 22 | ACTIVE NOAM: Activate Optional Features |
Establish an SSH session to the active NOAM, login as admusr. Note: For PCA Feature Activation. If you have PCA installed in the system being recovered, run the procedure “PCA Activation on Active NOAM server” on recovered Active NOAM Server and procedure “PCA Activation on Stand By SOAM server” on recovered Standby SOAM from [3] to re-activate PCAIf you have PCA installed in the system being recovered, run the procedure “PCA Activation on Active NOAM server” on recovered Active NOAM Server and procedure “PCA Activation on Stand By SOAM server” on recovered Standby SOAM from [3] to re-activate PCA Note:
Refer to section 1.5 Optional Features to activate any features that were previously activated. |
| 23 | NOAM VIP GUI: Fetch and Store the database Report for the Newly Restored Data and Save it |
Navigate to Main Menu->Status & Manage->Database Figure 21-25 Database  Select the active NOAM server and click on the Report button at the bottom of the page. The following screen is displayed: Main menu - Status & manage -Database[Report] Figure 21-26 Main menu  Click on Save and save the report to your local machine. |
| 24 | ACTIVE NOAM: Verify Replication Between Servers |
Login to the Active NOAM through SSH terminal as admusr. Run the following command: Figure 21-27 Output  |
| 25 | NOAM VIP GUI: Verify the Database states |
Click on Main Menu->Status and Manager->Database Figure 21-28 Database  Verify that the “OAM Max HA Role” is either “Active” or “Standby” for NOAM and SOAM and “Application Max HA Role” for MPs is “Active”, and that the status is “Normal” as shown below: Figure 21-29 Main menu  |
| 26 | NOAM VIP GUI: Verify the HA Status |
Click on Main Menu->Status and Manage->HA Figure 21-30 HA Select the row for all of the servers. Verify that the “HA Role” is either “Active” or “Standby”. Figure 21-31 Manage HA  |
| 27 | SOAM VIP GUI: Verify the Local Node Info |
Navigate to Main Menu->Diameter->Configuration->Local Node Figure 21-32 Local node  Verify that all the local nodes are shown. |
| 28 | SOAM VIP GUI: Verify the Peer Node Info |
Navigate to Main Menu->Diameter->Configuration->Peer Node Figure 21-33 Peer node  Verify that all the peer nodes are shown. |
| 29 | SOAM VIP GUI: Verify the Connections Info |
Navigate to Main Menu->Diameter->Configuration->Connections Figure 21-34 Connections  Verify that all the connections are shown. |
| 30 | For vSTP Only- SOAM VIP Server Console (Optional): Verify the local nodes info |
To verify the vSTP MP Local nodes info:
Figure 21-35 Output  |
| 31 | For vSTP Only- SOAM VIP Server Console (Optional): Verify the remote nodes info |
To verify the vSTP MP Remote nodes info:
Figure 21-36 Output  |
| 32 | For vSTP Only- SOAM VIP Server Console (Optional): Verify the Connections info |
To verify the vSTP MP Connections info:
Figure 21-37 Output  |
| 33 | SOAM VIP GUI: Enable Connections if needed |
Navigate to Main Menu->Diameter->Maintenance->Connections Figure 21-38 Connections Select each connection and click on the Enable button. Alternatively you can enable all the connections by selecting the EnableAll button. Figure 21-39 Enable  |
| 34 | SOAM VIP GUI: Enable Optional Features |
Navigate to Main Menu -> Diameter -> Maintenance -> Applications Figure 21-40 Applications  Select the optional feature application configured in step 31 Click the Enable button. Figure 21-41 Enable disable  |
| 35 | SOAM VIP GUI: Re- enable Transports if Needed. |
Navigate to Main Menu->Transport Manager -> Maintenance -> Transport Figure 21-42 Transport  Select each transport and click on the Enable button Figure 21-43 Enable  Verify that the Operational Status for each transport is Up. |
| 36 | SOAM VIP GUI: Re- enable MAPIWF application if needed |
Navigate to Main Menu->Sigtran->Maintenance->Local SCCP Users Figure 21-44 Local SCCP  Click on the Enable button corresponding to MAPIWF Application Name. Figure 21-45 Enable  Verify that the SSN Status is Enabled. |
| 37 | SOAM VIP GUI: Re- enable links if needed |
Navigate to Main Menu->Sigtran->Maintenance->Links Figure 21-46 Links  Click on Enable button for each link. Figure 21-47 Enable disable |
| 38 | SOAM VIP GUI: Examine All Alarms |
Navigate to Main Menu->Alarms & Events->View Active Figure 21-48 View active  Examine all active alarms and refer to the on-line help on how to address them. If needed contact My Oracle Support (MOS) |
| 39 | NOAM VIP GUI: Perform Keyexchange with Export Server |
Navigate to Main Menu -> Administration -> Remote Servers -> Data Export Figure 21-49 Data export  Click on Key Exchange at the bottom of the screen Enter the Password and press OK Figure 21-50 SSH exchange  |
| 40 | NOAM VIP GUI: Examine All Alarms |
Login to the NOAM VIP if not already logged in. Navigate to Main Menu->Alarms & Events->View Active Figure 21-51 View active Examine all active alarms and refer to the on-line help on how to address them. if required, contact My Oracle Support (MOS) . |
| 41 | Restore GUI Usernames and Passwords |
If applicable, run steps in section 6.0 to recover the user and group information restored. |
| 42 | Backup and Archive All the Databases from the Recovered System |
Run DSR Database Backup to back up the Configuration databases |