Replace a SDS NOAM Server Pair

2.2.5 Replace a SDS NOAM Server Pair

Pre-Condition:

Active and standby SDS NOAM servers have stopped functioning.
It has been determined to replace both VM(s) that host SDS NOAM servers.
Recent backup archives of SDS configuration and provisioning databases are available.
DR SDS NOAM servers are not available or are not installed.

Determine SDS backup archive:
1. Make sure you have access to the SDS backup archive containing provisioning data and configuration data. This backup archive should be in uncompressed format.
Replace old SDS VMs with new SDS VMs:
1. For VMWare based deployments:
  1. Open the cloud client of your choice, for example, vSphere client and locate the defective SDS NOAM server VM.
  2. Power down SDS NOAM server VM and remove it from the inventory or disk.
    See, section 2.2 "Creating SDS Guests (VMware)" from Oracle Communications SDS Cloud Installation Guide.
2. For KVM or OpenStack based deployments:
  1. Run the following command to log in to the OpenStack control node:
```
$ ssh admusr@node
```
  2. Run the following command to power down defective DP SOAM server VM and remove it from the inventory or disk:
```
$ nova delete <VM Name>
```
    For more information, see section 2.3 "Creating SDS Guests from OVA (KVM or OpenStack)." from Oracle Communications SDS Cloud Installation Guide.
3. For OVM-S/OVM-M based deployments:
  1. Run the following command to log in to the command line interface of the OVM-M:
```
OVM> ssh -l admin <OVM-M IP> -p 10000
```
    Example:
```
[root@manager01 ~]# ssh -l admin 10.240.16.138 -p
10000
admin@10.240.16.138's password:
```
  2. Run the following command to power down SDS VM(s):
```
OVM> stop Vm name=<VM Name>
```
    Note:
    Wait for the VM to have the state Stopped.
  3. Run the following command to remove the VM from the inventory or disk:
```
OVM> delete Vm name=<VM Name>
```
    Note:
    Repeat steps 2 and 3 for all the failed VMs.
    
    For more information, see section 2.4, "Creating SDS Guests from OVA (OVM-S or OVM-M)" from Oracle Communications SDS Cloud Installation Guide.
Configure the new SDS A server:
See, section 3.1, "Configure SDS NOAM Servers A and B (1st Site Only)" from Oracle Communications SDS Cloud Installation Guide.
Copy SDS backup archive to new SDS A server:
1. Login using ssh to the console of the new SDS NOAM server.
2. Copy the uncompressed backup archive identified in step 1 to /var/TKLC/db/filemgmt area on the newly installed first SDS NOAM server.
3. Run the following command to sudo prod.stop --ignore-cap to stop running applications. Leave database running.
4. Run the following command to restore the configuration DB:
```
sudo idb.restore -n -t /var/TKLC/db/filemgmt -v <full
path to configuration archive file>
```
5. Run the following command to restore the provisioning DB:
```
sudo idb.restore -n -t /var/TKLC/db/filemgmt -v <full
path to provisioning archive file>
```
  Note:
  This step may take up time depending upon the size of provisioning database.
  
  SDS database is now restored.
6. Run the following command to start application:
```
sudo prod.start
```
Re-exchange SSH keys for remote import/export/ data servers:
1. Log in to the primary SDS GUI as admin user using VIP address.
2. Navigate to SDS, Configuration, and Options screen to perform ssh key exchange for Remote Export.
3. Navigate to SDS, Configuration, and Options screen to perform ssh key exchange for Remote Import.
4. Navigate to Administration, Remote, Servers, and Data Export screen to perform ssh key exchange for Data Export.
Install the new second SDS NOAM server: follow recovery steps from this document to restore the second SDS NOAM server.

Post-Condition

Both SDS NOAM servers are back in service.
Provisioning clients are connected to SDS VIP address.
Provisioning continues.