Internal Web Services

11 Internal Web Services

DSR uses many internal web services in support of centralized configuration and management. These web services use the SOAP protocols and implement WS-Security profiles to authenticate internal clients. These services ship with self-signed certificates and default passwords. You must plan to update the default passwords during installation. You can also replace the self-signed certificates with certificates signed by a trusted authority. The following sections provide procedures to perform these actions.

11.1 Changing Internal Web Service Passwords

In general, after the initial configuration is complete and before deploying or turning up services, you must update the internal web service passwords.

11.2 Changing TPD Web Service Password

Perform the following procedures to change the OS-level provisioning web service password.

Updating TPD Web Service Password on Active NO

Log in as admusr on the source server.
```
login: admusr
Password: <current admin user password>
```
Run the following command to reset the TPD web service password:
```
$ /usr/TKLC/appworks/sbin/resetTpdPassword
```
You are prompted to provide a password:
```
password: <enter the new password>
```
Step result: The command copies and installs the new password to each reachable server in the topology, and flushes client password caches.
Run the following command to verify that the web service is still functional:
```
$ AppWorks Network interfaces
```
Step result: You can see a list of network interfaces reported by the Web Service backend:
```
{
“element”:[
“eth0”,
“eth1”
]
}
```

This update command synchronizes the TPD web service (tpdprovd) password on all reachable servers in the topology. Any servers added to the topology after running this command are automatically configured to use the new password. If any servers were not reachable when this command is run, run the command again later when those servers are reachable.

Updating TPD Web Service Password on PMAC

Some DSR deployments include a PMAC system to support installation and growth. Once you update the tpdProvd password on servers in the DSR topology, the PMAC loses the ability to inventory deployed DSR nodes. You can restore the inventory function by running the following procedure on the PMAC:

Update the password on a given server or group of servers (assuming all passwords are the same for the group) either using linux passwd command on the server(s) or by some other means.
From a PMAC shell, use the following command to add the password(s) to the PMAC database and update the PMAC messaging interface. This command prompts the user for the password and echo asterisks as characters are entered.
```
/usr/bin/sudo /usr/TKLC/smac/bin/pmacadm addProvdCredentials --flushBAs=yes
```
Note:
--flushBAs can be set to no if entering multiple passwords and set to yes on the last added password. If --flushBAs is not set to yes on the last password entry, a sentry restart must be performed on the PMAC to flush out all the Broker Agents (server interfaces) in the PMAC messaging system and must be rebuilt using the new passwords.
1. The new password can be verified using the following command:
```
/usr/bin/sudo /usr/TKLC/smac/bin/pmaccli getHostCommStr --ip=<ipv4 address of the server> --accessType=ro
```
  This should return a valid response with a password. If it fails, there may be a tpdProvd password mismatch issue between the PMAC and the server.
2. If a password must be removed and the exact spelling of the password is known, it can be deleted from the PMAC database and messaging system using the following command:
```
/usr/bin/sudo /usr/TKLC/smac/bin/pmacadm deleteProvdCredentials --flushBAs=yes
```
  Note:
  You are prompted for the password.

11.3 Changing the Configuration Web Services Password

The following procedures are used to change the configuration web services password.

Update Configuration Web Service Password on Active NO

Perform the following steps to update configuration web service password on an active NO:

Log in as admusr on the active NOA server.
```
Login: admusr
Password: <current admin user password>
```
Reset the TPD web service password by running:
```
$ /usr/TKLC/appworks/sbin/resetSoapPassword
```
You are not be prompted for a password. The resetSoapPassword command generates a large random string which is used as the new password.

Note:
The command copies and installs the new password to each reachable server in the topology, and flushes client password caches. You might see output related to these activities.
Restart all the servers in the topology from active NOA GUI:
1. Log in to the active NOA GUI.
2. Navigate to Main Menu, and then Status & Manage, and then Server.
3. Restart all the servers in the topology in below mentioned order.
  1. The non-Active OAM Servers, that is Standby or Spare NO, Standby or Spare SO, DR-NO.
  2. All the C-level servers.
  3. The active OAM servers, that is active NO and active SO.
Verify if the web service is functional:
```
$ AppWorks Alarms getData
```
You should see a list of active alarms as reported by the Web Service backend.
```
[
    <alarm list (if any)>
]
```

This update command synchronizes the configuration web services password on all reachable servers in the topology. After running this command, any servers added to the topology is configured to use the new password. If any servers were not reachable when this command is running, then run the command again later when those servers are reachable.

Some DSR deployments include an IDIH system to support message trace and debugging. Once you update the servers in the DSR topology, IDIH loses the ability to interact with the deployed DSR nodes. You can restore the IDIH function by running this procedure on the IDIH:

Log in as admusr on the active NOA server.
```
Login: admusr
Password: <current admin user password>
```
Retrieve the current configuration web services password in plain text. This is needed below in step 4.
```
$ /usr/TKLC/appworks/bin/aw.wallet credential get cmsoapa password
```
The command prints the current plain text configuration web service password
Example:
```
7w57q9U0OvOtKtgtLVTMajDcXfhCj2F4nyXw45qK6EXNHA9jACyQ
```

11.4 Changing Internal Web Service Certificates and Key Material

In general, the TPD and web services are configured to work with self-signed certificates. You can replace the appworks certificates using the procedures described in this section.

Assumptions

The following procedure assumes that you have already obtained a signed certificate or key file from the customer’s certificate authority and that these files are in the .pem format.

Each server in the topology needs its own certificate or key pair. The certificate must have a DN field that matches the hostname of the server. The following procedures assume the customer provides files in this naming convention:

<hostname>_crt.pem – a PEM encoded X.509 certificate for the host <hostname>
<hostname>_priv.pem – a PEM encoded private key for the host <hostname>

Ensure that the private key file is not protected with a passphrase.

Creating and Distributing a separate Certificate and Key PEM File

Perform the following steps to create and distribute a separate certificate and key PEM file:

Log in as admusr on the active NOA server.
```
Login: admusr
Password: <current admin user password>
```
Copy all of the <hostname>_crt.pem and <hostname>_priv.pem files to the home directory for admusr on the active NOA using a utility such as scp or rsync.
Run the following steps to confirm and verify the certificate or key pairs:
1. Confirm each of the certificate or key pairs are compatible (assume <hostname> is noa):
```
$ openssl rsa -noout -modulus -in noa_priv.pem | openssl md5 
(stdin)= eef417fb3f018862034ae8e9f3a0b56e
```
```
$ openssl x509 -noout -modulus -in noa_crt.pem | openssl md5 
(stdin)= eef417fb3f018862034ae8e9f3a0b56e
```
2. Verify the md5 output matches for each <hostname> certificate or private key pair. Additionally, the md5 should be different for different <hostnames>.
Copy the private key and certificate to the server (again, assume <hostname> is noa):
```
$ scp noa_priv.pem admusr@noa:
$ scp noa_crt.pem admusr@noa:
```
Note:
Repeat the above procedure for each <hostname>.

Installing a separate PEM and CERT File on Each Distinct <hostname>

After creating the separate certificate and private key PEM file for each server in the topology, you must log into each server in the topology and install the PEM file as follows:

Run the following commands to copy your new certificate/private key pair PEM file into place (assume <hostname> is noa):

$ sudo cp noa_priv.pem /usr/TKLC/appworks/etc/ssl
$ sudo chown awadmin:awadm /usr/TKLC/appworks/etc/ssl/noa_priv.pem
$ sudo chmod 640 /usr/TKLC/appworks/etc/ssl/noa_priv.pem

$ sudo cp noa_crt.pem /usr/TKLC/appworks/etc/ssl/ 
$ sudo chown awadmin:awadm /usr/TKLC/appworks/etc/ssl/noa_crt.pem
$ sudo chmod 640 /usr/TKLC/appworks/etc/ssl/noa_crt.pem

Run the following commands to replace the existing combined certificate or private key file with the new file:

$ sudo mv /usr/TKLC/appworks/etc/ssl/server.crt /usr/TKLC/appworks/etc/ssl/old_server.crt

$ sudo ln -s /usr/TKLC/appworks/etc/ssl/noa_crt.pem /usr/TKLC/appworks/etc/ssl/server.crt 

$ sudo mv /usr/TKLC/appworks/etc/ssl/server.pem /usr/TKLC/appworks/etc/ssl/old_server.pem

$ sudo ln -s /usr/TKLC/appworks/etc/ssl/noa_priv.pem /usr/TKLC/appworks/etc/ssl/server.pem

Run the following commands to restart the configuration web services and exit:
```
$ sudo pm.kill apwSoapServer
$ sudo pm.kill cmsoapa
$ exit
```
Note:
Repeat the above procedure for each and every distinct <hostname>.

11.5 Restricting Port 80 for HTTPD communication

This section describes the procedure to disable Port 80 using disablePort80.sh script.

Assumptions

The following procedure assumes that you have already updated CA signed server certificates and key files on all the servers of topology by following the Changing Internal Web Service Certificates and Key Material.

Ensure to copy the CA certificate on the Active NOAM Server using a utility such as scp. Copy the file location on the NOAM server, as the disablePort80.sh script prompts for the CA certificate location.

Note:

Ensure that the CA certificate is accessible (644 should be appropriate).

Disabling Port 80

Perform the following procedure to disable Port 80:

Run the following commands as admusr on Active NOAM server to run disablePort80.sh script.
```
$ cd /usr/TKLC/appworks/bin/
$ ./disablePort80.sh
```
Provide CA certificate file location when the script prompts to enter the location.

Note:
The CA certificate file should be present on the Active NOAM Server.

Sample script:


[admusr@SDS-NOAM-SNO00 bin]$  ./disablePort80.sh
===================================================================================
Disabling Port 80 on the setup!
===================================================================================
Checking current status of Port80Disabled in StringParam : No
Please ensure to have the CA (Certificate Authority) certificate file present in the current server i.e. the Active NOAM server of the topology before executing this script
===================================================================================
Checking certificates...
Please confirm that CA-signed certificates and keys are present across all servers of the topology in accordance with DSR Security Guide (yes/no): yes
Enter CA certificate file location : CA-file-complete-path
===================================================================================
===================================================================================
Checking server releases...
Success: All servers are on the same release: SDS-9.1.0.0.0_100.4.0-x86_64.iso
===================================================================================
===================================================================================
Updating httpd.conf to stop listening on Port 80...
Success: Updated httpd.conf on all the servers.
===================================================================================
===================================================================================
Updating iptable rules to remove Port 80...
Success : Updated iptables configuration!
===================================================================================
===================================================================================
Adding CA Certificate to truststore
Adding certificate to CA bundle on SDS-NOAM-SNO00...
Certificate added successfully!
Adding certificate to CA bundle on SDS-NOAM-SNO01...
Certificate added successfully!
Adding certificate to CA bundle on SDS-NOAM-SQS00...
Certificate added successfully!
Adding certificate to CA bundle on SDS-SOAM-SSO00...
Certificate added successfully!
Adding certificate to CA bundle on SDS-SOAM-SSO01...
Certificate added successfully!
Adding certificate to CA bundle on SDS-SOAM-SDP00...
Certificate added successfully!
Adding certificate to CA bundle on SDS-SOAM-SDP01...
Certificate added successfully!
Adding certificate to CA bundle on SDS-DRNOAM-SDRNO00...
Certificate added successfully!
Adding certificate to CA bundle on SDS-DRNOAM-SDRNO01...
Certificate added successfully!
Adding certificate to CA bundle on SDS-DRNOAM-SDRQS00...
Certificate added successfully!
Success : Added CA Certificate to truststore of all servers on the setup.
===================================================================================
===================================================================================
Updating Port80Disabled parameter in StringParam table...
Success: Updated Port80Disabled to Yes in StringParam table!
===================================================================================
Setup completed successfully.
Log file : /var/TKLC/log/disablePort80ScriptExecution.log
===================================================================================

Note:

The script only works when all the servers are on the same release. If backout is run on some servers, these changes are removed and port 80 is re-enabled on that particular server.
Connect to GUI with the following link: https://<Server-IP>.

Re-enabling Port 80

If re-enabling is required for the port 80 at any point, perform the following procedure:

To enable back Port 80, run the following commands as admusr on Active NOAM server.
```
$ cd /usr/TKLC/appworks/bin/
$ ./disablePort80.sh enable
```

Sample script


[admusr@VJ-Noam-DNO00 bin]$ ./disablePort80.sh enable
===================================================================================
Enabling Port 80 on the setup!
===================================================================================
Checking server releases...
Success: All servers are on the same release: DSR-9.1.0.0.0_100.0.1-x86_64.iso
===================================================================================
===================================================================================
Updating httpd.conf to start listening on Port 80...
Success: Updated httpd.conf on all the servers.
===================================================================================
===================================================================================
Updating iptable rules to add Port 80...
Success : Updated iptables configuration!
===================================================================================
===================================================================================
Updating Port80Disabled parameter in StringParam table...
Success: Updated Port80Disabled to No in StringParam table!
===================================================================================
Setup completed successfully.
Log file : /var/TKLC/log/disablePort80ScriptExecution.log
===================================================================================

Note:

Re-enabling Port 80 does not remove the previously added CA certificate from CA bundle that is DSRs truststore. This does not impact any communication.