21 Troubleshooting VNFM

21.1 Debug VNFM

To debug issues during VNFM deployment, check the following log files:

• VNFM logs are located in " /var/vnfm/logs/vnfm.log "
• VNFM boot logs are located in "/var/log/messages".
• Tomcat logs are located in " /usr/share/vnfm/apache-tomcat-9.0.16/logs/catalina.out ".
• SNMP notifier logs are located in "/var/vnfm/logs/snmp_notifier.log"
• Alert Manager logs are located in "/var/vnfm/logs/alertmanager.log"
• Prometheus server logs are located in "/var/vnfm/logs/prometheus.log"
• Node Exporter logs are located in "/var/vnfm/logs/node_exporter.log"
• RSync logs are location in "/var/log/rsync.log"
• Reboot logs are location in "/var/vnfm/logs/reboot.log"
• Cloud-init logs of the DSR/SDS VNFs are located in "/var/TKLC/log/cloud-init/cloud-init.log"

21.2 Adding Route for a New VIM

To add route for a new VIM, execute the following commands in root user mode:

1. Open route-network.sh, and append the new VIM route address to the DataList.

   For example: DataList=10.75.167.0/24,10.75.185.0/24

2. Execute the /route-network.sh, and then execute ifdown eth1 and ifup eth1.

21.3 Reboot Tomcat

To reboot Tomcat, perform the following procedure in the dsrvnfm user mode:

1. Go the / path.

   The ./dsrvnfmStartup.py script is present.

2. Execute this command:

   ./dsrvnfmStartup.py <option for tomcat to start,stop,restart> <ipAddress of vnfm, Don't use [] incase of IPv6>

   Examples:

   ./dsrvnfmStartup.py restart 2606:b400:605:b84a:6e41:6aff:fec7:8100

   ./dsrvnfmStartup.py start 10.75.189.201

21.4 Resolve HA Alarms on VNF through VNFM Deployed Setup

Perform the following to resolve the HA alarms:

1. Check the ping request and response packets from Server-A and Server-B for which alarm has been raised, by executing:

   tcpdump -i eth1 -n "host <server-A>-imi or <server-B>-imi and port 17401 and udp"

   For example:tcpdump -i eth1 -n "host noam00-17badf67-imi or noam01-17badf67-imi and port 17401 and udp"

2. If ping request or response packets are not coming from any server, then add security group rule ingress (response) or egress (request) to that instance.
   The following image shows Ingress response:

   Figure 21-1 Ingress Response

   
3. Check the ping packets again after adding the rule and ensure that imi request and response packets are received from each servers, by executing:

   tcpdump -i eth1 -n "<server-A>-imi or <server-B>-imi and port 17401 and udp"

4. Now restart the cmha process on the node where the alarms are present, by executing:

   pm.set off cmha && sleep 5 && pm.set on cmha

   Note:

   If the Node is HA Active, then restarting cmha will cause switch over.

21.5 Adding a Port in Openstack Security Groups

The Security Group Rules define the traffic that is allowed through instances assigned to the security group.

To allow traffic through ports other than the default ports added by VNFM, execute:

1. Open Security Groups tab on the Openstack Horizon.

   A list of available Security Groups appear.

2. From the list, click Manage Rules for the required Security Group.

3. Select Add Rule, provide all the required details in the dialog box.

   Note:

   In the CIDR field, the values for zero address are:

   • For IPv4 - 0.0.0.0/0
   • For IPv6 - ::/0
4. Click Add Rule.

21.6 Debug SNMP System Alerts

Steps to debug SNMP system alerts:

• Check the log files for any errors. For information about list of log files, see How to debug VNFM.
• If default configurations needs to be changed, perform SNMP System Traps Configurations.

21.7 Configure Flavor and Instantiation Levels in VNFM

Steps to configure Flavor and Instantiation Levels in VNFM:

The number of VMs to be allocated to each VNF Flavor and Instantiation Levels are present in the file: /usr/share/vnfm/openstack/VnfSizing.yaml

A sample of the file is provided below:

dsrSignaling:
  small:
     diameter:
        damp: 2
        ipfe: 2
        stp: 0
        sbr: 0
        udr: 0
  large: 
     diameter:
        damp: 32
        ipfe: 4
        stp: 0
        sbr: 0
        udr: 0

In order to change the default configurations:

• In 'dsrvnfm' user mode, edit this file: /usr/share/vnfm/openstack/VnfSizing.yaml
• Change the number of VMs under the required VNF Type → Instantiation Level Id → Flavor Id and save the file.

For example: In DSR Signaling, under Diameter Flavor Id, small Instantiation Level Id, the user needs 16 DAMPs, 4 IPFEs, the sample of file would be as below:

Edited Sample File

dsrSignaling:
  small:
     diameter:
        damp: 16
        ipfe: 4
        stp: 0
        sbr: 0
        udr: 0
  large: 
     diameter:
        damp: 32
        ipfe: 4
        stp: 0
        sbr: 0
        udr: 0

21.8 Configuring DNS IP in VNFM

Perform the following procedure if DNS IP is not configured.

1. Log in to the VNFM GUI.
2. In the /etc/resolv.conf VNFM parameter file, add the nameserver <DNS Ip> DNS IP if it is unavailable.
   Example:

   • For the IPv6 setup: nameserver 2606:b400:605:b912:200:5eff:fe00:1f7
   • For the IPv4 setup: nameserver 10.250.32.10
3. Add the <IP> cloudname controller IP in the IPv4 section of the /etc/hosts file if the DNS IP is not working.
   Example: 10.190.219.195 controller

21.9 Disable SSH Weak Key Exchange Algorithms

To display the list of key exchange algorithms, use the following command:

sudo sshd -T |grep -i "^kexalgorithms"

Output - Example

kexalgorithms curve25519-sha256,curve25519-sha256@ libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

The following two algorithms are considered as weak algorithm based on the Nessus scan result and they should be disabled for key exchange:

• diffie-hellman-group-exchange-sha1 - weak key algorithm
• diffie-hellman-group1-sha1 - weak key algorithm

To skip the weak key exchange algorithms, add the below line in the /etc/ssh/sshd_config and /etc/ssh/ssh_config files and on all the servers:

KexAlgorithms diffie-hellman-group-exchange-sha256,ecdh-sha2-nistp384

Note:

The diffie-hellman-group-exchange-sha256 and ecdh-sha2-nistp384 are added as they are not reported as Weak key algorithms. This result is overridden if ssh_config and sshd_config contains KexAlgorithms diffie-hellman-group-exchange-sha256,ecdh-sha2-nistp384 line.

Post execution of ssh_config and sshd_config update, run the following command:

sudo sshd -T |grep -i "^kexalgorithms"

Output - Example

diffie-hellman-group-exchange-sha256,ecdh-sha2-nistp384

For GSSAPI Key exchange algorithms, use the following commands to check GSSAPIAuthentication and GSSAPIKeyExchange parameters:

sudo sshd -T | grep -i "gssapiauthentication"
sudo sshd -T | grep -i "gssapikeyexchange"

To disable user authentication and key exchange based on GSSAPI, set the parameters GSSAPIAuthentication and GSSAPIKeyExchange to no in the /etc/ssh/sshd_config and /etc/ssh/ssh_config files, on all the servers.

sudo sed -i 's/.*GSSAPIAuthentication.*/#GSSAPIAuthentication no/i'  /etc/ssh/sshd_config && sudo sed -i '/.*GSSAPIAuthentication.*/{x;/.*GSSAPIAuthentication.*/! {x;h;b;};x;d}' /etc/ssh/sshd_config

sudo sed -i 's/.*GSSAPIAuthentication.*/# GSSAPIAuthentication no/i'  /etc/ssh/ssh_config && sudo sed -i  '/.*GSSAPIAuthentication.*/{x;/.*GSSAPIAuthentication.*/! {x;h;b;};x;d}' /etc/ssh/ssh_config

Note:

If there is a line with the GSSAPIKexAlgorithms or gssapikexalgorithms parameter (case-insensitive), remove that line.

Verify GSSAPIAuthentication and GSSAPIKeyExchange parameters:

sudo sshd -T | grep -i "gssapiauthentication"

sudo sshd -T | grep -i "gssapikeyexchange"

After making the changes in the /etc/ssh/sshd_config and /etc/ssh/ssh_config files, validate the /etc/ssh/sshd_config file for errors.

sudo sshd -T

Then, restart the sshd service.

sudo service sshd restart

21.10 Add or Update Certification

Note:

It is recommended to get a SSL certificate certified from a trusted authority, as Nessus scan results in SSL Certificate Cannot Be Trusted, SSL Self-Signed Certificate issues.

To add or update certification, perform the following steps:

1. Run step 1 and 2 from VNFM Self Signed Certificate Generation procedure to produce dsrVnfm.key and dsrVnfm.csr files.

   Note:

   For information on self signed certificate, refer to VNFM Self Signed Certificate Generation section.
2. Send these 2 files to a trusted CA authority for CA signature.
3. Perform step 4 from VNFM Self Signed Certificate Generation procedure using the dsrVnfm.crt file shared back by the trusted CA authority.

   Note:

   Run the remaining steps in the VNFM Self Signed Certificate Generation section to generate the vnfm_default.jks file.
4. After creating vnfm_default.jks certificate, before copying it into /var/vnfm/certificate/, move the created certificate to backup, if it exists in /var/vnfm/certificate/ path.

21.11 Changing VNF CLI and DSR GUI Log in Credentials

This section provides information to change VNF CLI and DSR GUI log in username and password.

Prerequisites

• The new credentials are supported by the DSR Image.
• The updated username must be a plain text.
• The updated password must be encrypted using the utility function which is /configPword.py'passwordToBeEncrypted'.

  Following is an example showing the usage of the script:

  Figure 21-2 Usage of Script

  
  

  

  
  
• The encrypted password must be updated in VnfmPropeties.yaml file.

Procedure

In the 'dsrvnfm' user mode:

• Open VnfmProperties file - /opt/vnfm/config/VnfmProperties.yaml to edit the properties.
• To change VNF CLI log in username and password, update admUsrUsername and admUsrPword fields.
• To change the DSR GUI username and password, update dsrNoamUsername and dsrNoamPword fields.