Troubleshooting Information

Generic Checklist

The following sections provide generic checklist for troubleshooting tips.

Deployment related tips

Are OCnssf deployment, pods and services created, running and available?
Execute following the command:
```
# kubectl -n <namespace> get deployments,pods,svc
```
Inspect the output, check the following columns:
- AVAILABLE of deployment
- READY, STATUS and RESTARTS of po
- PORT(S) of service
Is the correct image used and the correct environment variables set in the deployment? Execute following the command:
```
# kubectl -n <namespace> get deployment <deployment-name> -o yaml
```

Inspect the output, check the environment and image.

# kubectl -n nssf-svc get deployment ocnssf-nfregistration -o yaml 
apiVersion: extensions/v1beta1 
kind: Deployment metadata:  
              annotations:    
			  deployment.kubernetes.io/revision: "1"
			kubectl.kubernetes.io/last-applied-configuration: |      
			{"apiVersion":"apps/v1","kind":"Deployment","metadata": 
			{"annotations":{},"name":"ocnssf-nfregistration","namespace":"nssfsvc"},
			"spec":{"replicas":1,"selector":{"matchLabels": {"app":"ocnssf-nfregistration"}},
			"template":{"metadata": {"labels":{"app":"ocnssf-nfregistration"}},
			"spec": {"containers":[{"env":[{"name":"MYSQL_HOST","value":"mysql"}, 
			{"name":"MYSQL_PORT","value":"3306"}, 
			{"name":"MYSQL_DATABASE","value":"nssfdb"}, 
			{"name":"nssf_REGISTRATION_ENDPOINT","value":"ocnssfnfregistration"},
			{"name":"nssf_SUBSCRIPTION_ENDPOINT","value":"ocnssfnfsubscription"},
			{"name":"NF_HEARTBEAT","value":"120"}, 
			{"name":"DISC_VALIDITY_PERIOD","value":"3600"}],
			"image":"dsrmaster0:5000/ocnssfnfregistration:latest",
			"imagePullPolicy":"Always","name":"ocnssfnfregistration","ports": 
			[{"containerPort":8080,"name":"server"}]}]}}}}  
			creationTimestamp: 2018-08-27T15:45:59Z  generation: 1  
			name: ocnssf-nfregistration  namespace: nssf-svc  
			resourceVersion: "2336498"  
			selfLink: /apis/extensions/v1beta1/namespaces/ nssf-svc/deployments/ocnssf-nfregistration 
			uid: 4b82fe89-aa10-11e8-95fd-fa163f20f9e2 
			spec:  progressDeadlineSeconds: 600  
			replicas: 1  
			revisionHistoryLimit: 10  
			selector:    
			matchLabels:      
			app: ocnssf-nfregistration  
			strategy:    
			rollingUpdate:      
			maxSurge: 25%      
			maxUnavailable: 25%    
			type: RollingUpdate  
			template:    
			metadata:      
			creationTimestamp: null      
			labels:        
			app: ocnssf-nfregistration    
			spec:      
			containers:      
			- env:       
			- name: MYSQL_HOST          
			value: mysql        
			- name: MYSQL_PORT          
			value: "3306"       
			- name: MYSQL_DATABASE          
			value: nssfdb        
			- name: nssf_REGISTRATION_ENDPOINT          
			value: ocnssf-nfregistration       
			- name: nssf_SUBSCRIPTION_ENDPOINT         
			value: ocnssf-nfsubscription
			
            - name: NF_HEARTBEAT          
			value: "120"        
			- name: DISC_VALIDITY_PERIOD          
			value: "3600"        
			image: dsr-master0:5000/ocnssf-nfregistration:latest        
			imagePullPolicy: Always       
			name: ocnssf-nfregistration        
			ports:        - containerPort: 8080          
			name: server          
			protocol: TCP        
			resources: {}        
			terminationMessagePath: /dev/termination-log        
			terminationMessagePolicy: File      
			dnsPolicy: ClusterFirst      
			restartPolicy: Always      
			schedulerName: default-scheduler      
			securityContext: {}      
			terminationGracePeriodSeconds: 30 
			status:  
			availableReplicas: 1  
			conditions:  - lastTransitionTime: 2018-08-27T15:46:01Z    
			lastUpdateTime: 2018-08-27T15:46:01Z    
			message: Deployment has minimum availability.    
			reason: MinimumReplicasAvailable    
			status: "True"    
			type: Available  
			- lastTransitionTime: 2018-08-27T15:45:59Z    
			lastUpdateTime: 2018-08-27T15:46:01Z    
			message: ReplicaSet 
			"ocnssf-nfregistration-7898d657d9" has successfully progressed.    
			reason: NewReplicaSetAvailable    
			status: "True"    
			type: Progressing  
			observedGeneration: 1  
			readyReplicas: 1  
			replicas: 1  
			updatedReplicas: 1

Check if the micro-services can access each other via REST interface. Execute following command:

# kubectl -n <namespace> exec <pod name> -- curl <uri>

Example:

# kubectl -n nssf-svc exec $(kubectl -n nssf-svc get pods -o name|cut -d'/' -f2|grep nfs) -
        curl http://ocnssf-nfregistration:8080/nnssf-nfm/v1/nfinstances
# kubectl -n nssf-svc exec $(kubectl -n nssf-svc get pods -o name|cut -d'/' -f2|grep nfr) -        
curl http://ocnssf-nfsubscription:8080/nnssf-nfm/v1/nfinstances

Note:

These commands are in their simple form and display the logs only if there is 1 nssf<egistration> and nfs<ubscription> pod deployed.

Application related tips
Check the application logs and look for exceptions, by executing the following command:
```
# kubectl -n <namespace> logs -f <pod name>
```
You can use '-f' to follow the logs or 'grep' for specific pattern in the log output.
Example:
```
# kubectl -n nssf-svc logs -f $(kubectl -n nssf-svc get pods -o name|cut -d'/' -f2|grep nfr)
 # kubectl -n nssf-svc logs -f $(kubectl -n nssf-svc get pods -o name|cut -d'/' -f2|grep nfs) 
```
Note:
These commands are in their simple form and display the logs only if there is 1 nssf<egistration> and nfs<ubscription> pod deployed.

The environment is not working as expected

Problem: The environment is not working as expected.

Solution:

Check if kubectl is installed and working as expected.
Check if kubectl version command works: This must display the versions of client and server.
Check if $ kubectl create namespace test command works.
Check if kubectl delete namespace test command works.
Check if Helm is installed and working as expected.
Check if helm version command works: This must display the versions of client and server.

Debugging of NSSF Installation while Installing using helm

Problem: The user is getting the error: failed to parse ocnssf-custom-values-1.4.0.yaml: error converting YAML to JSON: yaml.

Solution:

Verify the following:

The ocnssf-custom-values-1.4.0.yaml may not be created properly.
The tree structure may not be followed.
There may be tab spaces in the file.
Verify that the ocnssf-custom-values-1.4.0.yaml is proper

Refer Network Slice Selection Function (NSSF) Cloud Native Installation Guide .
If there is no error, helm installation will be deployed.

Helm status can be checked using following command :

helm status <helm release name>

NSSF installation issues

Problem: The NSSF installation is not successful.

Solution:

Verify if NSSF specific pods are working as expected by executing the following command:

kubectl get pods -n <ocnssf_namespace>

Check whether all the pods are up and running.

Sample output:

NAME                            READY   STATUS  RESTARTS  AGE
ocnssf-appinfo-7969c9fbf7-4fmgj                  1/1  Running   0   18m 
ocnssf-config-server-54bf4bc8f9-s82cv            1/1  Running   0   18m 
ocnssf-egress-6b6bff8949-2mf7b                   1/1  Running   0   18m 
ocnssf-ingress-68d76954f5-9fsfq                  1/1  Running   0   18m 
ocnssf-nrf-client-nfdiscovery-cf48cd8d8-l4q2q    1/1  Running   0   18m
ocnssf-nrf-client-nfdiscovery-cf48cd8d8-vmt5v    1/1  Running   0   18m
ocnssf-nrf-client-nfmanagement-7db4598fbb-672hc  1/1  Running   0   18m 
ocnssf-nsavailability-644999bbfb-9gcm5           1/1  Running   0   18m 
ocnssf-nsconfig-577446c487-dzsh6                 1/1  Running   0   18m
ocnssf-nsdb-585f7bd7d-tdth4                      1/1  Running   0   18m 
ocnssf-nsselection-5dfcc94bc7-q9gct              1/1  Running   0   18m 
ocnssf-nssubscription-5c898fbbb9-fqcw6           1/1  Running   0   18m 
ocnssf-performance-6d75c7f966-qm5fq              1/1  Running   0   18m

If status of any pod is shown as ImagePullBackOff or ErrImagePull then it can be due to:
1. Incorrect ImageName provided in ocnssf-custom-values-1.4.0.yaml.
  Then, double check the image name and tags in ocnssf-custom-values-1.4.0.yaml.
2. Docker registry is incorrectly configured.
  Then, check docker registry is properly configured in all master and slave nodes.
If RESTARTS count of the pods is continuously increasing, then it can happen due to the following reasons:
1. MySQL primary and secondary hosts may not be configured properly in ocnssf-custom-values-1.4.0.yaml
2. MySQL servers may not be configured properly according to the pre-installation steps mentioned in Network Slice Selection Function (NSSF) Cloud Native Installation Guide .

Debugging General CNE

Problem: The environment is not working as expected

Solution:

Execute the command kubectl get events -n <ocnssf_namespace>to get all the events related to a particular namespace.

Collect the NSSF Logs to check the error scenarios

Problem: The error scenarios are checked by colllecting the NSSF logs.

Solution:

The following commands must be executed to get the logs from nssf specific pods:

Fetch the list of all pods by executing kubectl get pods -n <ocnssf_namespace>
Collect the logs from the pod and redirect to file by executing kubectl logs <pod_name> -n <ocnssf_namespace> > <Log File>

Example:

kubectl logs ocnssf-nsselection-57cff5665c-skk4l -n ocnssf > ocnssf_logs1.log

Custom Value File Parsing Failure

This section explains troubleshooting procedure in case of failure during parsing custom values file.

Problem

Not able to parse ocnssf-custom-values-x.x.x.yaml, while running helm install.

Error Code/Error Message Error:

failed to parse ocnssf-custom-values-x.x.x.yaml: error converting YAML to JSON: yaml

Symptom

While creating the ocnssf-custom-values-x.x.x.yaml file, if the above mentioned error is received, it means that the file is not created properly. The tree structure may not have been followed and/or there may also be tab spaces in the file.

Solution

Following the procedure as mentioned:

1. Download the latest NSSF templates zip file from OHC. Refer to Installation Tasks for more information.

2. Follow the steps mentioned in the Installation Tasks section.

Curl HTTP2 Not Supported

Problem

curl http2 is not supported on the system.

Error Code/Error Message

Unsupported protocol error is thrown or connection is established with HTTP/1.1 200 OK

Symptom

If unsupported protocol error is thrown or connection is established with http1.1, it is an indication that curl http2 support may not be present on your machine

Solution

Following is the procedure to install curl with HTTP2 support:

1. Make sure git is installed:

$ sudo yum install git -y

2. Install nghttp2:

$ git clone https://github.com/tatsuhiro-t/nghttp2.git
 $ cd nghttp2 $ autoreconf -i 
$ automake 
$ autoconf 
$ ./configure 
$ make 
$ sudo make install  
$ echo '/usr/local/lib' > /etc/ld.so.conf.d/custom-libs.conf 
$ ldconfig

3. Install the latest Curl:


$ wget http://curl.haxx.se/download/curl-7.46.0.tar.bz2 (NOTE: Check for latest version during Installation) 
$ tar -xvjf curl-7.46.0.tar.bz2 
$ cd curl-7.46.0 
$ ./configure --with-nghttp2=/usr/local --with-ssl 
$ make 
$ sudo make install 
$ sudo ldconfig

4. Make sure HTTP2 is added in features by executing the following command:

$ curl --http2-prior-knowledge -v "<http://10.75.204.35:32270/nnrf
disc/v1/nf-instances?requester-nf-type=AMF&target-nf-type=SMF>"

Tiller Pod Failure

Problem

Tiller Pod is not ready to run helm install.

Error Code/Error Message

The error 'could not find a ready tiller pod' message is received.

Symptom

When helm ls is executed, 'could not find a ready tiller pod' message is received.

Solution

Following is the procedure to install helm and tiller using the below commands:

1. Delete the pre-installed helm:

kubectl delete svc tiller-deploy -n kube-system kubectl delete deploy tiller-deploy -n kube-system

2. Install helm and tiller using this commands:

helm init --client-only 
helm plugin install https://github.com/rimusz/
helm-tiller helm tiller install 
helm tiller start kube-system

Error Trigger based debugging

The following table lists the NSSF Alarms, cause for the errors and the solution:

Table 6-1 NSSF Events, causes and solution

Event	Type	Cause for the error	Solution
ocnssfPolicyNotFound	Alert	Rate of messages that did not find a matching policy is above warning threshold (Threshold: <>, Current: <>) Configuration issue: Operator did not configure Rules for the TAI + SNSSAI which is allowed as per AMF	Check TAI and SNSSAI for which there is no matching Rule. Add Rule to allocate slice for TAI+SNSSAI combination.
ocnssfNrfDiscFailed	Alert	NRF discovery message is getting error response	Check connectivity with NRF Validate NRF load
ocnssfNotificationFailure	Alert	Notification message towards AMF is failing	Validate connectivity to notification URL Check NS-Subscription load and capacity

Event

Type

Cause for the error

Solution

ocnssfPolicyNotFound

Alert

Rate of messages that did not find a matching policy is above warning threshold (Threshold: <>, Current: <>)

Configuration issue: Operator did not configure Rules for the TAI + SNSSAI which is allowed as per AMF

Check TAI and SNSSAI for which there is no matching Rule.

Add Rule to allocate slice for TAI+SNSSAI combination.

ocnssfNrfDiscFailed

Alert

NRF discovery message is getting error response

Check connectivity with NRF

Validate NRF load

ocnssfNotificationFailure

Alert

Notification message towards AMF is failing

Validate connectivity to notification URL

Check NS-Subscription load and capacity