Troubleshooting SEPP

3 Troubleshooting SEPP

This section provides information to troubleshoot the common errors which can be encountered during the installation and upgrade of SEPP:

Note:

kubectl commands might vary based on the platform deployment. Replace kubectl with Kubernetes environment-specific command line tool to configure Kubernetes resources through kube-api server. The instructions provided in this document are as per the Oracle Communications Cloud Native Environment (OCCNE) version of kube-api server.

Caution:

User, computer and applications, and character encoding settings may cause an issue when copy-pasting commands or any content from PDF. PDF reader version also affects the copy-pasting functionality. It is recommended to verify the copy-pasted content, especially when hyphens or any special characters are part of the copied content.

3.1 Generic Troubleshooting Scenarios

The following are the generic troubleshooting scenarios:

3.1.1 Generic Checklist

Environment Verification

The following sections provide generic checklist for troubleshooting tips:

a. Deployment related tips

Perform the following checks before the deployment:

Are OCSEPP deployment, pods, and services created, running, and available? .
To check this, run the following command:
```
# kubectl -n get deployments,pods,svc
```

Inspect the output and check the following columns:

AVAILABLE of deployment
READY, STATUS, and RESTARTS of pod
PORT(S) of service

b. Is the correct image used and the correct environment variables set in the deployment?

To check this, run the following the command:

# kubectl -n <namespace> get deployment <deployment-name> -o yaml

c. Inspect the output, check the environment and image.

# kubectl -n seppsvc get deployment sepp-release-1-n32-egress-gateway -o yaml

d. Check if the microservices can access each other through REST interface.

To check this, run following command:

# kubectl -n <namespace> exec <pod name> -- curl <uri>

3.1.2 The environment is not working as expected

Problem:

The environment is not working as expected.

Solution:

Check if kubectl is installed and working as expected.
Check if kubectl version command works: This must display the versions of client and server.
Check if $ kubectl create namespace testcommand works.
Check if kubectl delete namespace test command works.
Check if Helm is installed and working as expected.
Check if helm version command works: This must display the versions of client and server.

3.1.3 Helm Install Failure

This section describes the various scenarios in which helm install command might fail.

Following are some of the scenarios:

Incorrect Image name in ocsepp-custom-values files
Docker registry is incorrectly configured
Continuous Restart of Pods
Custom Value File Parse Failure

Note:

Helm validation is enabled by default for helm v3 and to disable it, customer have to delete values.schema.json file from their charts folder.

Incorrect Image name in ocsepp-custom-values.yaml files

Problem

helm install might fail if an incorrect image name is provided in the ocsepp-custom-values file.

Error Code or Error Message

When you run kubectl get pods -n <ocsepp_namespace>, the status of the pods might be ImagePullBackOff or ErrImagePull.

Solution

Perform the following steps to verify and correct the image name:

Edit ocsepp-custom-values file and provide release specific image name and tags.
Run helm install command.
Run kubectl get pods -n <ocsepp_namespace> to verify if the status of all the pods is running.

Docker registry is incorrectly configured

Problem

helm install might fail if docker registry is not configured in all primary and secondary nodes.

Error Code or Error Message

When you run kubectl get pods -n <ocsepp_namespace>, the status of the pods might be ImagePullBackOff or ErrImagePull.

Solution

Configure docker registry on all primary and secondary nodes.

For more information on configuring docker registry, see Oracle Communications Cloud Native Environment Installation Guide.

Continuous Restart of Pods

Problem

helm install might fail if the MySQL primary and secondary hosts are not be configured properly in ocsepp-custom-values.yaml.

Error Code or Error Message

When you run kubectl get pods -n <ocsepp_namespace>, the pods restart count increases continuously.

Solution

MySQL servers may not be configured properly. For more information about the MySQL server configurations, see the SEPP Predeployment Configuration section in Cloud Native Security Edge Protection Proxy Installation Guide.

Custom Value File Parse Failure

This section explains troubleshooting procedure in case of failure while parsing the custom values file.

Problem

Unable to parse the ocsepp-custom-values-x.x.x.yaml while running helm install.

Error Code or Error Message

Error: failed to parse ocsepp-custom-values-x.x.x.yaml: error converting YAML to JSON: yaml

Symptom

While creating the ocsepp-custom-values-x.x.x.yaml file, if the aforementioned error is received, it means that the file is not created properly. The tree structure may not have been followed and there may also be tab spaces in the file.

Solution

Following the procedure as mentioned:

Download the latest SEPP templates zip file from MOS.
Follow the steps mentioned in the SEPP Predeployment Configuration section in Cloud Native Security Edge Protection Proxy Installation Guide.

3.1.4 Kubernetes Node Failure

Problem

Kubernetes nodes goes down.

Error Code/Error Message

"NotReady" status is displayed against the Kubernetes node.

Symptom

On running the command kubectl get nodes, "NotReady" status is displayed.

Solution

Following is the procedure to identify the kubernetes nodes failure:

Run the following command to describe the node:
```
kubectl describe node <kubernete_node_name>
```
Check the nodes utilization by running the following command:
```
kubectl top nodes
```

3.1.5 SEPP Installation Verification

Problem: The SEPP installation is not successful.

Solution:

Verify if SEPP specific pods are working as expected by running the following command:

kubectl get pods -o wide -n <ocsepp _namespace>

Check whether all the pods are up and running.

Sample output:

NAME     TYPE           CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
ocsepp-release-appinfo-55b8d4f687-wqtgj                      1/1   Running   0     141m
ocsepp-release-cn32c-svc-64cd9c555c-ftd8z                    1/1   Running   0     113m
ocsepp-release-cn32f-svc-dd886fbcc-xr2z8                     1/1   Running   0     4m4s
ocsepp-release-config-mgr-svc-6c8ddf4c4f-lb4zj               1/1   Running   0     141m
ocsepp-release-n32-egress-gateway-5b575bbf5f-z5bbx           2/2   Running   0     131m
ocsepp-release-n32-ingress-gateway-76874c967b-btp46          2/2   Running   0     131m
ocsepp-release-ocpm-config-65978858dc-t4t5k                  1/1   Running   0     141m
ocsepp-release-performance-67d76d9d58-llwmt                  1/1   Running   0     141m
ocsepp-release-plmn-egress-gateway-6dc4759cc7-wn6r8          2/2   Running   0     31m
ocsepp-release-plmn-ingress-gateway-56c9b45658-hfcxx         2/2   Running   0     131m
ocsepp-release-pn32c-svc-57774fdc4-2qpvx                     1/1   Running   0     141m
ocsepp-release-pn32f-svc-586cd87c7b-pxk6m                    1/1   Running   0     3m47s
ocsepp-release-sepp-nrf-client-nfdiscovery-65747884cd-qblqn  1/1   Running   0     141m
ocsepp-release-sepp-nrf-client-nfmanagement-5dd6ff98d6-cr7s7 1/1   Running   0     141m
ocsepp-release-nf-mediation-74bd4dc799-d9ks2                 1/1   Running   0     141m

If status of any pod is shown as ImagePullBackOff or ErrImagePull, then it can be due to:
1. Incorrect ImageName provided in ocsepp_custom_values_<versions>.yaml.
  Then, double check the image name and tags in ocsepp_custom_values_<versions>.yaml.
2. Docker registry is incorrectly configured.
  Then, check docker registry is properly configured in all master and worker nodes.
If RESTARTS count of the pods is continuously increasing, then it can happen due to the following reasons:
1. MySQL primary and secondary hosts may not be configured properly in ocsepp_custom_values_<versions>.yaml
2. MySQL servers may not be configured properly. For more information about the MySQL configuration, see theSEPP Predeployment Configuration section in Cloud Native Core. Security Edge Protection Proxy Installation, Upgrade ,and Fault Recovery Guide.

3.1.6 Debugging General CNE

Problem: The environment is not working as expected

Solution:

Run the command kubectl get events -n <ocsepp_namespace>to get all the events related to a particular namespace.

3.1.7 Collecting the SEPP Logs to Check the Error Scenarios

Problem: The error scenarios are checked by collecting the SEPP logs.

Solution:

Run the following commands to get the logs from SEPP specific pods:

Run the following command to get the pods details:
```
$ kubectl -n get pods
```
Run the following command to collect the logs from the specific pods or containers:
```
kubectl get pods -n <ocsepp_namespace> 
```
Collect the logs from the pod and redirect to file by running kubectl logs <pod_name> -n <ocsepp_namespace> > <Log File>

Example:

kubectl logs - seppsvc-cn32f-svc-57cff5665c-skk41 -n seppsvc > seppsvc_logs1.log

3.1.8 Helm Error During the Rollback

Problem

The Helm rollback causes failure and displays the following error:

"Duplicate value: "cnc-metrics" && cannot patch"

Symptom

The error indicates that the Helm is not able to merge the current and rollback charts.

Solution

Run the helm rollback command again with -–force to resolve the issue.

Note:

If the rollback is to be performed using –force, take the backup of configmap data as the config map data can be cleaned by Helm.

3.1.9 Upgrade or Rollback Failure

When Security Edge Protection Proxy (SEPP) upgrade or rollback fails, perform the following procedure:

Check the pre or post-upgrade or rollback hook logs in Kibana as applicable.
Users can filter upgrade or rollback logs using the pod name filter

example: ocsepp-release-update-db

Check the pod logs in Kibana to analyze the cause of failure.

After detecting the cause of failure, do the following:

For upgrade failure:

If the cause of upgrade failure is database or network connectivity issue, contact your system administrator. When the issue is resolved, rerun the upgrade command.
If the upgrade failure occurs during the preupgrade phase, resolve the issue, then perform a upgrade. Do not perform rollback because SEPP deployment remains in the source or older release.
If the upgrade failure occurs during the postupgrade phase, for example, post upgrade hook failure due to target release pod not moving to ready state, then perform a rollback.

For rollback failure:

If the cause of rollback failure is database or network connectivity issue, contact your system administrator. When the issue is resolved, rerun the rollback command.

If the issue persists, contact My Oracle Support.

3.1.10 Helm Test Failure

Following are the troubleshooting steps if helm test is not getting initiated:

Run the following command to get the cn32f-svc name:
```
kubectl get svc -n namespace
```
Example:
```
kubectl get svc -n seppsvc
```
Add the following property under the global section in the custom-values.yaml file.
```
.global.seppServiceAccountName = cn32f-svc name 
```
Upgrade to the same app version using updated custom-values.yaml file for changes to be updated in the installation.
Run helm test.

3.1.11 Helm Rollback Failure with the Configmap with the Name not Found Error

Problem

Helm rollback results in failure and displays the following error:

Rollback "ocsepp" failed: no ConfigMap with the name "rss-ratelimit-map" found

Symptom

The above-mentioned error indicates that Helm is not able to merge current and rollback charts.

Solution

Run the Helm rollback again with --force to resolve the issue.

Note:

If the rollback is to be performed using –force, take the backup of configmap data as the config map data can be cleaned by Helm.

3.1.12 Continuous Restart of coherence-svc Pods

Problem: Helm install might fail if the coherence-svc pod is restarting repeatedly.

When you run kubectl get pods -n <ocsepp_namespace>, the coherence-svc pods restart count increases continuously.

Solution: Delete the coherence-svc pod using kubectl delete po -n <namespace> <coherence pod> and the pod will be up and running.

3.1.13 IllegalReferenceCount Exception Occurence in Logs of Ingress and Egress Gateways

Problem: In some environments, there can be IllegalReferenceCount exception in the Ingress or Egress logs which results in an unexpected traffic drop. This is visible if the incoming traffic at Gateway is not equal to the outgoing traffic.

Solution:

Check the per second occurrence of IlleagalReferenceCount exception on the Gateway pods.
Update the following configurations in the Config Map of the affected Gateway:
```
nettyInboundExceptions:
  exceptions:
    - io.netty.util.IllegalReferenceCountException
  count: 1000 //Update this to a value less than the per second occurrence
  timePeriod: 1
```
This resets the HTTP2 connection when the count reaches the configured value for this exception within the given time period.

3.1.14 False Message while Doing the Helm Uninstall

Problem: The user gets the following false message while doing the Helm uninstall:


These resources were kept due to the resource policy:
[ConfigMap] egress-ratelimit-map
[ConfigMap] rss-ratelimit-map
release "ocsepp-release" uninstalled.

Solution:

Run the following command to reverify whether the SEPP uninstallation is successful and the config maps are deleted:

kubectl get cm -n <namespace>

Output:


[seppuser@thrust6-bastion-1 ~]$ kubectl get cm -n <namespace>
NAME                 DATA   AGE
istio-ca-root-cert   1      87m
kube-root-ca.crt     1      87m
[seppuser@thrust6-bastion-1 ~]$

Note:

The listed output should not have 'egress-ratelimit-map' and ' rss-ratelimit-map'.

3.2 Feature Specific Troubleshooting Scenarios

The following are the feature specific troubleshooting scenarios:

3.2.1 Cat-2 Network ID Validation Feature

The following are the troubleshooting scenarios of Cat-2 Network ID Validation feature:

The incoming request is rejected at CN32F microservice

Problem:

Incoming request gets rejected with error code configured ( 406 - default error code ) at CN32F microservice.

Solution:

Search for error code SEPP-CN32FSEPP-ERROR-0013 or SEPP-CN32FSEPP-ERROR-0014 in CN32F logs.
Verify if correct regex is configured under Header or Body IE tab under Cat 2 – Network ID Validation Section of Security Countermeasure tab under SEPP.
Verify whether PLMN ID sent in the headers is part of PLMN ID List based on the associated SEPP configured.

The incoming request is rejected at PN32F microservice

Problem:

Incoming request gets rejected with error code configured (406 - default error code) at PN32F microservice.

Solution:

Search for error code SEPP-PN32FSEPP-ERROR-0016 or SEPP-PN32FSEPP-ERROR-0017 in PN32F logs.
Verify if correct regex is configured under Header or Body IE tab under Cat 2 – Network ID Validation Section of Security Countermeasure tab under SEPP.
Verify whether PLMN ID sent in the headers is present in the PLMN ID List based on the associated SEPP configured.

Invalid PLMN ID in Header configurations

Problem:

Invalid PLMN ID in Header configurations is received on SEPP.

Solution:

Verify if correct regex is configured against the header identifier in Header Configuration.
If error is thrown from CN32F microservices, verify if MCC and MNC combination is present in the PLMN ID List based on the associated SEPP configurations.
If error is thrown from PN32F microservices, verify proper configurations PLMN ID list based on the associated SEPP configurations.

Invalid PLMN ID in body configurations

Problem:

Invalid PLMN ID in body configurations is received on SEPP.

Solution:

Verify if correct regex is configured against the body IE in Body IE Configuration.
If the error is from CN32F microservices, verify f MCC and MNC combination is present in the PLMN ID List based on associated SEPP configured.
If the error is from PN32F microservices, verify if MCC and MNC combination is present in the PLMN ID list based on associated SEPP configured.

3.2.2 Troubleshooting Steps for Cat-1 Service API Validation Feature

The incoming request is rejected at CN32F:

Problem: The incoming request is rejected with the configured status code (default status code is 406) at CN32F microservice.

Solution:

Search for error code SEPP-CN32FSEPP-ERROR-0012 in CN32F logs.
Verify whether the proper HTTP method and Resource URI combination is sent in the request in CN32F logs.
Check Allowed list name configured against Remote SEPP set.
Verify whether the proper Resource URI and HTTP method is configured in CNC Console GUI screen of Service API Allowed List for that particular Allowed list name under Security Counter Measure Section for N32 Egress Direction.
If user is configuring a new Resource URI, ensure to configure the proper regex.

3.2.3 Troubleshooting Steps for Overload Control Feature

Problem:

Incoming request does not get rejected with error code configured in onsole (429 - default error code) at N32 Ingress Gateway

Solution:

check if feature is enabled through API: curl -XGET http://<config-server>:<port>/sepp/nf-common-component/v1/igw/n32/ocpolicymapping
Check if correct policy applied through API: curl -XGET http://<config-server>:port/sepp/nf-common-component/v1/igw/n32/ocpolicymapping
Check the svcName parameter to verify whether the release name is correct or not for pn32f-svc.

Problem:

a) Feature is correctly configured and enabled via REST still request is not getting rejected with configured error code

Solution

Fetch the current load-level for N32 Ingress gateway using the following API:

curl 'http://<release-name>-n32-ingress-gateway:80/igw/load-level?svcName=<relase-name>-pn32f-svc'--http2-prior-knowledge

If the above has output the "Normal" ,check CPU and Memory thresholds defined using the API :
```
curl -XGET http://<config-server>:port/sepp/nf-common-component/v1/perf-info/overloadLevelThreshold
```
Check the CPU and Memory statistics from Grafana or Prometheus to check the current CPU and Memory usage.
Use the metrics cgroup_cpu_nanoseconds and cgroup_memory_bytes for the service mapped.
Either of the metrics values should reach the "onsetvalue" for a particular threshold level (defined in step 2) to be applied and feature to run.

b)If the API mentioned in step 1 for case (a) returns following error:

{"type":null,"title":"Service
          Unavailable","status":503,"detail":"Load level
          for service ocsepp-release-pn32f-svc is not Configured at Ingress-Gateway","instance":null,"cause":"Load level for service ocsepp-release-pn32f-svc is not Configured at
          Ingress-Gateway","invalidParams":null}

Solution

Check custom-values.yaml used to deploy OCSEPP
In the Perf-info section , check the tagNamespace value
The above should be either "namespace" or "kubernetes_namespace" depending on the OCCNE version used.
Check the configMap.prometheus value - this should map to prometheus IP and port or service IP path used to access prometheus.
If any of the above have been incorrectly set , kindly change and re-deploy OCSEPP

c)If the API mentioned in step 1 for case (a) returns "Connection refused" error

Solution

Run the following:

kubectl get svc -n <namespace> | grep n32-ingress-gateway

If the above does not have port 80 present in service , do the following:
In the custom-values.yaml , set the following parameter in the N32 Ingress gateway section : enableIncomingHttp to true
Re-deploy OCSEPP or upgrade the N32-Ingress-gateway service.
verify if port 80 if enabled using step 1.

3.2.4 Troubleshooting Steps for Rate Limiting Feature

Problem

Request not getting rejected with configured code.

Solution

Check rateLimiting.enabled must be set to True.
Check globalIngressRateLimiting.enabled must be set to True.
For Egress rate limiting, check egressRateLimiting.enabled must be set to True.

Problem

Request not getting rejected with configured error code.

Solution

In Ingress Gateway check for errorCodeProfiles in OCSEPP custom values.yaml file.
Check profile - name: ERR_1200
Change the error code from 503 to desired value.
Upgrade or re-deploy OCSEPP

3.2.5 Troubleshooting Steps for Message Feed Feature

Problem

Message Copy feature not copying JSON data to Data Director (DD).

Solution

Check whether the feature is enabled or not.
Check if copyPayload is set to false. If yes, set to True.
After re-deploying OCSEPP, verify if the data is copied at DD.

Problem

Message copy not copying data to DD as incorrect IP Port.

Solution

Check whether the feature is enabled or not.
Check whether the security enabled or not.
If security set to false, check whether DD Unreachable<GW> has been raised.
If yes , then Kafka.bootstrapAddress parameter must be set to correct listener IP and port
After re-deploying OCSEPP, verify whether data is copied at DD.

Problem

Message copy not copying data to DD as topic name incorrect.

Solution

Check whether the feature is enabled or not.
Check whether the security enabled or not.
If security set to false, check whether the DDUnreachable<GW> has been raised.
If not, check the topicName parameter. This topic should be created in DD so that data copied can be seen on DD.
After creating topic, verify if data is copied at DD.

Problem

Message copy not copying data to DD (security enabled) - case A.

Solution

Check whether the feature is enabled or not.
Check whether the security enabled or not.
If security set to true ,check whether the DDUnreachable<GW> has been raised
If yes, then Kafka.bootstrapAddress parameter must be set to correct listener IP and security port for DD.
After re-deploying OCSEPP, verify if data is copied at DD.

Problem

Message copy not copying data to DD (security enabled) - case B

Solution

Check whether the feature is enabled or not.
Check whether the security enabled or not.
If security set to true, check whether the DDUnreachable<GW> has been raised
If not , then check the security configurations for DD.
Check the following parameters:
1. userName: must be the same as used to configure DD
2. password: Check the secret name and Namespace details if correct.
After re-deploying OCSEPP, verify if data is copied at DD.

Note:

All the values will be checked in custom-values.yaml as this is a HELM based feature.

3.2.6 Troubleshooting Steps for Hosted SEPP

Problem:

The feature is Enabled and Consumer Remote SEPP Set not found (Default error code = 400)

Error Code or Error Message

Consumer Remote SEPP Set not found

Solution:

Check whether the following error is displayed in logs. The error is displayed if Allowed P-RSS Validation is Enabled and No Consumer RSS is configured.

{"instant":{"epochSecond":1668703429,"nanoOfSecond":698428472},"thread":"reactor-http-epoll-4","level":"ERROR","loggerName":"com.oracle.cgbu.cne.ocsepp.cn32f.handler.Cn32fSeppHandler","message":"HostedSEPPException: Request not allowed as source remote sepp set not found","contextMap":{"ocLogId":"1668703429683_71_ocsepp-release-mohit-plmn-ingress-gateway-7b86f4855c-ph9xj"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":15,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"22-11-17 16:43:49.698+0000","processId":"1","ocLogId":"1668703429683_71_ocsepp-release-mohit-plmn-ingress-gateway-7b86f4855c-ph9xj"}

Verify if Consumer Remote SEPP Set is present on Hosted SEPP.

Problem:

The feature is Enabled and Destination Roaming Partner Set is null (Error code = 400) or Remote SEPP Set Not Found (Error code = 404).

Error Code or Error Message

Destination RPS is null

Solution:

Above error is displayed if Allowed P-RSS Validation is Enabled and No Producer Remote SEPP Set is configured.
Verify if the Producer Remote SEPP Set is present on Hosted SEPP.

Error Code or Error Message

destinationRPS not present

Problem:

The feature is Enabled and destination Roaming Partner Set not present (Default error code = 400)

Solution:

Following logs is displayed:

The following Error is displayed if Destination Sepp set is not configured in allowedProducerRemoteSeppSets of Consumer Sepp Set:

{"instant":{"epochSecond":1668705561,"nanoOfSecond":940785763},"thread":"reactor-http-epoll-3","level":"ERROR","loggerName":"com.oracle.cgbu.cne.ocsepp.cn32f.handler.Cn32fSeppHandler","message":"HostedSEPPException: Request not allowed as remote sepp set psepp not present in allowed list","contextMap":{"ocLogId":"1668705561928_135_ocsepp-release-mohit-plmn-ingress-gateway-7b86f4855c-ph9xj"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":15,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"22-11-17 17:19:21.940+0000","processId":"1","ocLogId":"1668705561928_135_ocsepp-release-mohit-plmn-ingress-gateway-7b86f4855c-ph9xj"}

Verify whether the destinationRPS is present in allowedProducerRemoteSeppSets configured At RSS of Consumer.
If it is present, then wait for cache refresh to take place as configured.

3.2.7 Steering of Roaming (SOR) Feature

The following are the troubleshooting scenarios of Steering of Roaming (SOR) feature:

SOR feature is not enabled.

Problem:

SOR feature is not enabled.

Solution:

Verify the following scenarios:

Check whether the SOR feature is enabled at CNC Console or REST API.
Check the Remote SEPP Set, validate SOR is enabled for the given RSS.
Check Roaming Hub is disabled, and SEPP is deployed in SEPP Mode.

SOR feature is enabled at Global or RSS level but SOR is still disabled.

Problem:

SOR is enabled at Global or RSS level but SOR is still disabled.

Solution:

Verify the following scenarios:

Check Remote SEPP Set Configuration, check the SOR list name associated with RSS.
Verify the method plus URI that is passed in the message request exists in the SOR List.

SOR is configured with Retry as true and server header value is provided but retry is not working.

Problem:

SOR is configured with Retry as true and server header value is provided but retry is not working.

Solution:

Verify that the server header value given at the time of configuration matches the value that reaches in server header in the response.
Example: SOR server header value is configured as SOR-sorfqdn.com, and message request is sent, error response is received with server header value as 'server': 'SOR-sorfqdn.com'.
In this case, if retry is true then retry will be performed.
If server header value is not matched, even when retry is true, retry will not be performed.

3.2.8 Rate Limiting for Ingress Roaming Signaling per Remote SEPP Set Feature

The following are the troubleshooting scenarios of Rate Limiting for Ingress Roaming Signaling per Remote SEPP Set feature:

Problem: Unable to see reject list for traffic sent

Solution:

Check in ocsepp_custom_values_<version>.yaml file if the rssRateLimiter.enabled parameter for N32 Ingress Gateway is set to true.
Check whether the feature is enabled on Options screen under Remote SEPP Set, which is under Ingress Rate Limiting, at CNC Console by checking whether Remote SEPP Set Ingress Rate Limiting Enabled parameter is set to true.
Check whether the feature is enabled on RSS level by checking RSS Ingress Rate Limiting Enabled parameter is set to true on Remote SEPP Set screen of CNC Console for the particular PLMN traffic.
Check whether the the header configured in Originating Network ID Header parameter on Options screen under Remote SEPP Set, which is under Ingress Rate Limiting, at CNC Console, is being sent in traffic.

Problem: Unable to see status code in traffic set on Ingress Rate Limiting at CNC Console

Solution:

Check whether the status code configured is present in the RSS by checking Error configuration under Ingress Rate Limiting option in Remote SEPP Set screen for which the PLMN is being extracted.
Change the error code in RSS by editing Error Configuration under Ingress Rate Limiting option in Remote SEPP Set screen.

Problem: Unable to see error detail in traffic set on Ingress Rate Limiting at CNC Console

Solution:

Check whether the status code configured is present in the RSS by checking Error configuration under Ingress Rate Limiting option in Remote SEPP Set screen for which the PLMN is being extracted.
Change the error code in RSS by editing Error Configuration under Ingress Rate Limiting option in Remote SEPP Set screen.

Problem: Status code set to a different code in Error Configuration but Status code 429 is seen in rejected requests

Solution:

Check if status code set on CNC Console is a valid HTTP Status code or in the series of 3xx. By default, these will be modified to 429.

Problem: Server header observed in logs

Solution:

Server Header is added for the following Status codes - 408, 404, 400, and 429.

3.2.9 Cat-3 Previous Location Check feature

Problem:

Ingress request message gets rejected and displays the error code configured in the CNC Console (406 - default error code) at PN32F microservice.

Solution:

Search for the error codes SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-ERROR-0019 or SEPP-PN32FSEPP-ERROR-0018 or SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-EXCEPTION-0020 in PN32F microservice logs.
Verify if the correct regex is configured in Header or Body IE configuration for UE ID and Serving Network ID under Cat 3 – Previous Location Check Section under Security Countermeasure of SEPP CNC Console.
Verify if the MCC and MNC from serving network configured in either Header or Body is matching with the serving network name. The MCC and MNC values are part of the UDR response. Check whether the UDR response is success.
SUPI must be present in the incoming message, if it is configured for Cat-3 Previous Location Check.
UDR discovery procedure must be successful.
Coherence service must be up and running.
SUPI must be part of the IMSI range coming as part of the UDR profile received in UDR discovery response.
FQDN or IP of UDR must be reachable.
Proper DNS resolutions must be done for UDR discovery call, pn32f-svc for subscription use case.

3.2.10 Cat-0 SBI Message Schema Validation Feature

The following are the troubleshooting scenarios for Cat-0 SBI Message Schema Validation feature:

Problem:

The incoming request gets rejected at CN32F and PN32F microservices.

Solution:

Check the logs or metrics (ocsepp_message_validation_on_body_failure and ocsepp_message_validation_on_header_failure) to find the request has failed for which resource URI and HTTP method, do the following:

If there is a request body failure, the following logs can be find by searching the text "Message validation failed for request body for request" :

{"instant":{"epochSecond":1680084693,"nanoOfSecond":192915132},"thread":"reactor-http-epoll-1","level":"ERROR","loggerName":"com.oracle.cgbu.cne.ocsepp.cn32f.handler.Cn32fSeppHandler","message":"OUT:
          Cn32fSeppHandler::Message validation failed for request body for request:
            /nausf-auth/v1/ue-authentications for method: POST","contextMap":{"ocLogId":"1680084693177_151_ocsepp-release-plmn-ingress-gateway-77c69f7bbc-2fxvg"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":16,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"23-03-29 10:11:33.192+0000","processId":"7","ocLogId":"1680084693177_151_ocsepp-release-shafali-plmn-ingress-gateway-77c69f7bbc-2fxvg"}

If there is a request query parameters failure, the following logs can be find by searching the text "Message validation failed for request query parameter(s) for request" :

{"instant":{"epochSecond":1678638067,"nanoOfSecond":537933800},"thread":"reactor-http-nio-4","level":"ERROR","loggerName":"com.oracle.cgbu.cne.ocsepp.cn32f.handler.Cn32fSeppHandler","message":"OUT:
        Cn32fSeppHandler:: Message  validation failed for
            request query parameter(s) for request:
            //nnssf-nsselection/v2/network-slice-information for method:
        GET","contextMap":{"ocLogId":"1678638061928_34_"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":22,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"23-03-12 21:51:07.537+0530","processId":"37136","ocLogId":"1678638061928_34_ocsepp-release-plmn-ingress-gateway-77c69f7bbc-2fxvg"}

From the left navigation menu, navigate to SEPP and then click Security Countermeasure. Click Cat 0 - SBI Message Schema Validation feature under Security Countermeasure, the Message Validation List appears underneath. Do the following:
1. Search for the problematic resource URI and can get the corresponding schema.
2. Compare the request body or request query parameter value(s) against the corresponding schema and ensure that either the request is complaint with its schema or existing schema needs updation.

If the user wants to know the detailed causes of message validation failures user can generate the debug logs, search for the configured error code and title or text "Error in Request Body" or "Error in Request Parameter(s)" and can get the following logs:

Request body failure case log:

 {"instant":{"epochSecond":1678392753,"nanoOfSecond":435438500},"thread":"reactor-http-nio-4","level":"DEBUG","loggerName":"com.oracle.cgbu.cne.ocsepp.webfluxlog.LoggingResponseDecorator","message":"LoggingResponseDecorator::getBody()  Response {\"title\":\"Message validation
        failed\",\"status\":406,\"detail\":\"Message
        validation for request /nausf-auth/v1/ue-authentications failed for  remote sepp set:
          RS\",\"instance\":\"/nausf-auth/v1/ue-authentications\",\"cause\":\"Error
          in Request Body\",\"invalidParams\":[\"requestBody.traceData.traceDepth:
        should be valid to any of the schemas  string\",\"requestBody.resynchronizationInfo.rand:
        does not match the  regex pattern ^[A-Fa-f0-9]{32}$\",\"requestBody.servingNetworkName:
        does  not match the regex pattern
        ^5G:mnc[0-9]{3}[.]mcc[0-9]{3}[.]3gppnetwork[.]org(:[A-F0-9]{11})?$\",\"requestBody.traceData.eventList:
        is missing but it is required\",\"requestBody.supiOrSuci: is missing  but it is
        required\"]}","contextMap":{"ocLogId":"1678392742101_34_"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":22,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"23-03-10 01:42:33.435+0530","processId":"23320","ocLogId":"1678392742101_34_"}

Request query parameter failure case log:

{"instant":{"epochSecond":1680685967,"nanoOfSecond":535845100},"thread":"reactor-http-nio-4","level":"DEBUG","loggerName":"com.oracle.cgbu.cne.ocsepp.webfluxlog.LoggingResponseDecorator","message":"LoggingResponseDecorator::getBody() Response {\"title\":\"Message validation
              failed\",\"status\":406,\"detail\":\"Message
              validation for request /nudm-sdm/v2/imsi-987654000000001 failed for  remote sepp set:
                RS\",\"instance\":\"/nudm-sdm/v2/imsi-987654000000001\",\"cause\":\"Error
                in Request Parameter(s)\",\"invalidParams\":[\"supported-features:
              does not match the regex pattern  ^[A-Fa-f0-9]*$\",\"parameters.dataset-names: is
              missing but it is  required\"]}","contextMap":{"ocLogId":"1680685963154_34_"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":22,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"23-04-05 14:42:47.535+0530","processId":"13060","ocLogId":"1680685963154_34_"}

On the basis of failure reasons, the user can either correct the request body or request query parameter values or user can update the schema as mentioned in the step 2.

3.2.11 Configuration Failure in Remote SEPP and Remote SEPP Set

Problem:

Configuration operations (Add/ Delete/ Modify) failure in Remote SEPP and Remote SEPP Set, but user receives a 200 OK response code.

Solution:

The user must check the value of metrics ocsepp_configmgr_routefailure_total before and after the configuration operations (Edit/Add/Delete).
An increment in the counter indicates that the operation needs to be triggered again.

3.2.12 Aspen Service Mesh

Problem: SEPP Deployment fails in ASM mode.

Solution:

Check whether istio enabled flag set in namespace. If not, run the following command and deploy again:
```
kubectl label ns seppsvc istio-injection=enabled
```
Check PeerAuthentication is STRICT or PERMISSIVE . If it is set to STRICT, then change to PERMISSIVE and deploy again.
Run the following command to check the IP and host name in Service Entry for Kube-api-server:
```
kubectl get svc
```
Sample Output:
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
```

Check whether the SEPP is able to connect to Database. If CnDBTier is deployed in another namespace, create the DestinationRule(DR) as given below and deploy again:


apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: ocsepp-db-service-dr
  namespace: <ocsepp-namespace>
spec:
  exportTo:
  - "."
  host: <db-service-fqdn>.<db-namespace>.svc.<domain>
  trafficPolicy:
    tls:
      mode: DISABLE

Check whether the Service Account has all the Roles and RoleBindings permissions to access all the resources. If not, give all permissions as given below:
```
...
verbs:
- '*'
```

3.2.13 Rate Limiting for Egress Roaming Signaling per PLMN feature

The following are the troubleshooting scenarios of Rate Limiting for Egress Roaming Signaling per PLMN feature:

Problem: Unable to see the discarded messages for the traffic sent.

Solution:

Check ocsepp_custom_values_<version>.yaml file for the following:
- In SEPP mode, check whether the egressRateLimiter.enabled parameter is set to true in the PLMN Ingress Gateway section of ocsepp_custom_values_<version>.yaml file.
- In Roaming Hub mode, check whether the egressRateLimiter.enabled parameter is set to true in the N32 Ingress Gateway section of ocsepp_custom_values_roaming_hub_<version>.yaml file.

CNC Console

In the CNC Console GUI, from the left navigation menu, navigate to SEPP and click Rate Limiting.
Select Egress Rate Limiting which is defined under Rate Limiting.
The Option and EgressRateLimitingList appears underneath.
Click Option. The option screen appears at the right pane. Check whether Egress Rate Limiting Enabled is true.
Check whether request's PLMN ID is present in any of the EgressRateLimitingList and Egress Rate Limiting Enabled is set totrue.

REST API

Check whether the egressRateLimitingEnabled is set to True using the REST APIs. For more details, see the Egress Rate Limiting Option Configuration and Egress Rate Limiting List Configuration REST APIs sections in the Cloud Native Core, Security Edge Protection Proxy REST Specification Guide.

Problem: Traffic is being forwarded even if tokens for the Egress Rate Limiting List are exhausted.

Solution:

The 3gpp-Sbi-Message-Priority header of the request must be verified before the message is dropped. If the priority in the header is less than (not equal to) Discard Message Priority property of the message in the Egress Rate Limiting List, then the message is not dropped.
If the 3gpp-Sbi-Message-Priority header is not present, then the priority is checked in the route configuration. If a value for 3gpp-Sbi-Message-Priority is present in the route configuration, then the above mentioned condition is considered and the same solution is applied.
If the priority is unknown for the request, 24 is considered as the default value for the request priority, then, the same condition as above is applied.

Problem: Status code is set to a different code in Error Configuration, but status code 429 is seen in rejected requests.

Solution: Check if status code set on CNC Console is a valid HTTP Status code or in the series of 3xx. By default, it should be 429.

Problem: The server header observed in response or logs.

Solution: The server header is observed in the response or logs, if the user cofigured error code is present in the Helm custom values. By default the status codes 400,404,408, and 429 are configured in Helm custom values.

3.2.14 Separate Port Configurations for N32c and N32f on the Egress Routes

The following are the troubleshooting scenarios for separate port configurations for n32c and n32f on the Egress routes feature:

Scenario: The Remote SEPP is changed with new fields of N32F configuration, and the traffic is not proper after changing the profile.

Solution:

Check whether the Remote Partner Set is created. If not, create the Remote Partner Set. For more information about API path, see Oracle Communications Cloud Native Core, Security Edge Protection Proxy REST Specification Guide.
Check if the configuration is stored inside the database correctly. Get the Remote SEPP profile using the following command and verify the configuration:
```
curl -X GET 'http://<ocsepp-release-name>-config-mgr-svc::9091/sepp-configuration/v1/remotesepp/<name>'
```
1. Check if N32fFQDN is correct; whether it is mapped to correct DNS entry and is reachable.
2. Check the N32fAddress, whether it is correctly mapped to the service. To confirm that the IP is mapped to correct service, run the following command:
```
kubectl get endpoints -n <namespace>
```
3. Check whether the IP is mapped correctly to the intended service.
4. Check if N32fPort is configured correctly.
Verify in the database that the routes at n32-egress-gateway are updated according to the new configuration done at Remote SEPP. Run the following command to get the routes created:
```
curl  -X GET 'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/peerconfiguration'#Sample output[{"id":"psepp1","apiPrefix":"","Host":"<n32f-fqdn/IP>", "port": "8888"}]
```
1. Check in the above output if the Host and port parameter are the N32f IP and FQDN and port respectively.
2. Run the following command to check that the ID is mapped correctly in the peer set:
```
curl  -X GET 'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/peersetconfiguration'#Sample output[{"id":"RSS-2","httpConfiguration":[{"priority":1,"peerIdentifier":"psepp1"}]}]
```
3. Run the following command to check in routes configuration that the peerset Id is mapped correctly in the peerSetIndentifier parameter:
```
curl  -X GET  'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/routesconfiguration'
```

3.2.15 Alternate Routing based on the DNS SRV Record for Home Network Functions

The following are the troubleshooting scenarios of alternate routing based on the DNS SRV Record for home network functions feature:

Problem: Virtual FQDNs are configured, but incoming request doesn't match any configured route.

Solution: Verify the routes and the matching criteria (URI and header) associated with each route. If the request is not matching any route, then the request will be routed via the configured default route.

Problem: The incoming requests are not routed according to the configuration defined at plmn-egress-gateway.

Solution:

Get all the routes by using GET API.

Example:

curl -X 'GET' \
  'http://<config-mgr-svc-ip>:<port>/sepp/nf-common-component/v1/egw/plmn/routesconfiguration'

Verify whether the Order id of the each route is configured correctly. Lower the order id, higher will be priority of routes.
User must reconfigure the routes by using REST APIs.

Problem:The incoming requests are not routed to the target FQDNs associated with the virtual FQDNs in the DNS service.

Solution:

You can run below commands for debugging:

Check if alternate-route-svc is up and running.
Use "dig" command to verify if virtual FQDN is resolvable. Example dig -t srv "virtualFqdn". This command should return the list of the target FQDNs associated with the virtual FQDN.

Problem: Configurational issues at plmn-egress-gateway.

Solution: If the user faces difficulty while updating DNS SRV records, the configuration must be cleared in the following order:

Routes Configuration
Peerset Configuration
Peer Configuration

The order for the configurtions must be as follows:

sbiroutingerrorcriteriasets
sbiroutingerroractionsets
Peer Configuration
Peerset Configuration
Routes Configuration

3.2.16 Load Sharing among Multiple Remote SEPP Nodes

The following are the troubleshooting scenarios of load sharing among multiple Remote SEPP nodes feature:

Problem: The Remote SEPP is changed with virtualHost, and the traffic is not working properly after changing the profile

Solution:

Check whether the Remote Partner Set is created. If not, create the Remote Partner Set. For more information about API path, see "Remote Partner Set" section of Oracle Communications Cloud Native Core, Security Edge Protection Proxy REST Specification Guide.
Check if the configuration is stored inside the database correctly. Get the Remote SEPP profile using the following command and verify the configuration:
```
curl -X GET 'http://<ocsepp-release-name>-config-mgr-svc::9091/sepp-configuration/v1/remotesepp/<name>'
```
1. Check if the virtualHost is correct; whether it is mapped to correct DNS entry and is reachable.
2. To verify that the virtual route is created at N32 egress gateway correctly, run the following command:
```
curl  -X GET 'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/peerconfiguration'
#Sample output[{"id":"psepp1","apiPrefix":"","virtualHost":"<virtualHost>"}]
```
3. Check the above output, and whether the virtualHost is mapped to the virtualHost configuration in the Remote SEPP.
4. Check if ID is configured correctly in the peer-set:
```
curl  -X GET 'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/peersetconfiguration'
#Sample output[{"id":"RSS-2","httpConfiguration":[{"priority":1,"peerIdentifier":"psepp1"}]}]
```
Verify that the peerset Id in routes configuration is mapped correctly in peerSetIndentifier parameter:
```
curl  -X GET  'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/routesconfiguration'
```

Problem: Check whether target host is mapped against virtual host correctly.

Solution:

Run the following curl from config mgr pod:

curl --noproxy "*"--http2-prior-knowledge -X GET -H 'Accept: application/json'-H "Content-Type: application/json"'http://<ocsepp-release-name>-alternate-route:80/lookup?fqdn=<virtualhost>g&scheme=http'

Sample output

[{"target":"ocsepp-release-adity-n32-ingress-gateway.sepp3.3gppnetwork.org","port":443,"ttl":60,"type":"SRV","dclass":"IN","priority":10,"weight":10000},{"target":"ocsepp-release-adity-n32-ingress-gateway.sepp2.3gppnetwork.org","port":443,"ttl":60,"type":"SRV","dclass":"IN","priority":10,"weight":10000}]bash-4.4$

3.2.17 5G SBI Message Mediation Support

For troubleshooting the mediation rules using Drools Rule Language (DRL) related scenarios, see "Error Messages for Mediation Rule Configuration" section in Cloud Native Core, Service Communication Proxy Troubleshooting Guide.

3.2.18 Georedundancy Support

The following are the troubleshooting scenarios for the Georedundancy Support feature:

Problem: One of the cnDBTier site data is not reflected on other sites.

Solution:

In the CNC Console GUI, from the left navigation menu, navigate to SEPP and then click Georeplication Status.
If the Replication Status is Down, then user need to perform Recovering a Failed Site procedure.
For more information on how to perform Recovering a Failed Site procedure, see "Recovering a Failed Site" section in Oracle Communications Cloud Native Core, Security Edge Protection Proxy User Guide.

Problem: Traffic is failing at C SEPP.

Solution:

Verify the following on the Grafana dashboard:

If the error is coming from alternate-route service, then check DNS configuration.
If the error is coming on n32-egress-gateway service, then verify routing configuration in Remote SEPP Set.

Problem: Traffic is failed on one of the producer SEPP instances even with equal weights and priorities.

Solution:

Verify n32-ingress-gateway pod of P SEPP is up and running.
Verify n32-egress-gateway logs at C SEPP and n32-ingress-gateway logs at P SEPP to identity the reason for call drop.

Problem: DNS SRV configuration is not reflecting.

Solution:

Verify the DNS settings and run service restart.

Restart the pods on C SEPP:


<release>-alternate-route
<release>-n32-egress-gateway

Problem: cnDBTier health APIs are not working.

Solution:

Check the SEPP and cnDBTier compatibility in SEPP User Guide. If the health APIs are supported from cnDBTier 24.1.x onwards. For the earlier versions of cnDBTier, the health APIs were not supported.

For more details, see 'Support for cnDBTier APIs in CNC Console' section of Oracle Communications Cloud Native Core, Security Edge Protection Proxy User Guide.

Problem: NDB health status Screen is blank.

Solution: Enable the following parameters in cnDBTier yaml file and run the helm upgrade.

pvchealth:

      enable:

        all: true

        mgm: true

        ndb: true

        api: true

3.3 HTTP Response Codes and Error Codes

The following are the HTTP Response Codes:

Table 3-1 HTTP Response Codes and Error Codes

Data Type	Mandat ory(M)/Optional(O)/Conditi onal(C )	Cardinality	Response Code	Description
ProblemDetails	C	1	400 - BAD REQUEST	SEPP config-mgr-svc shall send the response when the Request body received is not as per defined Data Model.
ProblemDetails	C	1	500 - INTERNAL SERVER ERROR	SEPP config-mgr-svc shall send the response when an internal error has occurred.
Response Body as per Data Model defined	C	1	200 - OK	SEPP config-mgr-svc shall send the response when the request is successful.
ProblemDetails	C	1	404 -NOT FOUND	SEPP config-mgr-svc shall send the response when the requested entry is not present in the database.

Error Codes and Recovery Steps

The following tables list the various SEPP error codes and the recovery steps:

Table 3-2 Error Codes and Recovery Steps

Error Code	Error Text	Command/Method	Description	Recovery Steps
SEPP-COM-DB-ERROR-0002	Remote Sepp record not found	GET remotesepp/{name} DELETE remotesepp/{name} PATCH remotesepp/{name}	This error is observed if the provided name in the request is not present.	Verify that Remote SEPP name given in request parameter is present.
SEPP-COM-DB-ERROR-0003	N32F Context not found record not found	GET handshakestatus/fqdn/{fqdn} GET handshakestatus/name/{name}	This error is observed if the context with given name or fqdn in the request parameter is not found.	Verify that context with given name or fqdn in request parameter is present in DB.
SEPP-COM-DB-ERROR-0005	Database connection is down	Can be thrown from any Method (Generic Exception if application is unable to connect with Database)	This error is observed if the application is unable to make the connection with Database.	Verify that DB is up and running.
SEPP-COM-DB-ERROR-0007	SQL Grammer exception	PUT remoteseppset/{name}	This error is observed if there is some corruption in the Database related to Remote SEPP Set table (This error gets generated for all the commands that is trying to access corrupted table or Database).	Recheck the corruption in database and re-install SEPP
SEPP-COM-DB-ERROR-0008	Constraint violation exception of database table columns	PUT {remoteSepp}	This error is observed when one of the mandatory parameters required for remote SEPP are not present.	Verify that mandatory parameter like name, seppfqdn are present.
SEPP-COM-DB-ERROR-0009	Unsupported security capability list exception	POST /remotesepp PUT remotesepp/{name} PATCH remotesepp/{name}	This error is observed if unsupported securityCapabilityList is provided in request.	Make sure securityCapabilityList provided is supported by SEPP. Allowed security capability list is 'TLS' and 'TLS and PRINS.
SEPP-COM-DB-ERROR-0010	Update not allowed on table entry exception	PUT remotesepp/{name} PATCH remotesepp/{name}	This error is observed if given parameter to update is same as configured one or trying to update mandatory parameter.	Verify that the parameter provided in the request to update is different from the configured one or not while updating any mandatory parameter like name, seppfqdn etc.
SEPP-COM-DB-ERROR-0011	Remote Sepp already present	POST /remotesepp	This error is observed if Remote SEPP with same seppfqdn is already present in DB.	Verify that seppfqdn provided in configuration request is not already present in DB.
SEPP-COM-DB-ERROR-0012	Mandatory Parameter Update Not Allowed	PUT remotesepp/{name} PATCH remotesepp/{name}	This error is observed if the user is trying to update the value of mandatory parameter which is not allowed.	Verify that some of the mandatory parameters are not allowed to be updated. Those allowed should match with the value in the request.
SEPP-COM-ERROR-0013	Invalid PLMN List in Request	POST /remotesepp PUT /remotesepp	This error is observed if the user has configured PLMN in incorrect format in PLMNID List.	Verify the entered PLMN (mcc and mnc) in PLMNID List while configuring Remote Sepp.
SEPP-COM-DB-ERROR-0020	Remote Sepp Set not found	GET RemoteSeppSet/{name}	This error is observed if the Remote Sepp Set is not present	Give the Remote SEPP Set name that exist in the database
SEPP-COM-DB-ERROR-0021	Remote Sepp Set associated with Remote SEPP	DELETE RemoteSepp/{name} PUT RemoteSepp/{name} PATCH RemoteSepp/{name}	This error is observed if Remote SEPP is associated with Remote Sepp Set, it gives this error on deletion.	Disassociate the Remote Sepp from Remote SEPP Set by executing the Remote SEPP Set PUT command. Or Do the following: Delete the Remote Sepp Set Delete the Remote Sepp Create Remote Sepp Set again
SEPP-COM-DB-ERROR-0022	Remote Sepp Set already exists	POST RemoteSeppSet/{name}	This error is observed if Remote Sepp Set already exists and same entry is added again.	Use a unique name in Remote Sepp Set
SEPP-COM-DB-ERROR-0023	Remote Sepp Set configuration error	PUT RemoteSeppSet/{name} POST RemoteSeppSet/{name}	Remote Sepp Set does not exists Requested Remote Sepp does not exists At least one Remote Sepp should be associated with Remote Sepp Set Configured Domains are different between peers Remote Sepp Set exists with same domain Configured PLMNs are different between peers Remote Sepp Set exists with same PLMN Configured PLMNs/Domains are different between peers Associating Remote Sepp should be unique for Remote Sepp Set	Give the Remote SEPP Set name that exists in the database Give the Remote SEPP name that exists in the database Give at least a single Remote Sepp name while creating a Remote Sepp Set Check that each Remote Sepp has same Domain when creating a Remote Sepp Set Check that each Remote Sepp has same PLMN when creating a Remote Sepp Set Check that each Remote Sepp name is unique while creating a Remote Sepp Set
SEPP-COM-xx-ERROR-0101	Config Not Acceptable	PUT POST PATCH /v1/remotesepp/	This error is observed if PLMNIdList is empty or PLMNIdList size is greater than max size allowed or if domain is null.	Verify that PLMNIdList is not empty or PLMNIdList size is not greater than max size allowed or if domain is not null.
SEPP-COM-xx-ERROR-0102	Mandatory Parameter Missing	POST /remotesepp POST /remoteseppset	This error is observed if mandatory parameter is missing in request.	Verify that all mandatory parameter for configuration of Remote SEPP or Remote SEPP Set is present
SEPP-COM-xx-ERROR-0103	Connection could not be established on N32c interface	POST /remotesepp DELETE remotesepp/{name} PUT remotesepp/{name} PATCH remotesepp/{name}	This error is observed if n32c service is down or not up and running.	Verify that n32c service is up and running
SEPP-COM-xx-ERROR-0104	Invalid Value for Parameter		This Error occurs when user enters the invalid value for Enum Field in SEPP.
SEPP-COM-SVR-ERROR-0404	Unable to connect to EGW to sync config	PUT peerconfiguration/peersetconfiguration /routesconfiguration	This error is observed if config mgr is not able to update peer/peerset/routes configurations at EGW.	Verify that common configuration server is up and running.
SEPP-CN32FSEPP-ERROR-0013	PLMN ID Validation In Header Failed	NA	This error is observed if PLMN ID is not matched in header of the incoming request on CN32F microservice.	Verify if correct regex is configured against the header identifier in Header Configuration. Verify if MCC & MNC combination is present in the Remote PLMN ID List.
SEPP-CN32FSEPP-ERROR-0014	PLMN ID Validation In Body Failed	NA	This error is observed if PLMN ID is not matched in body of the incoming request on CN32F microservice.	Verify if correct regex is configured against the body IE in Body IE Configuration. Verify f MCC & MNC combination is present in the Remote PLMN ID List.
SEPP-PN32FSEPP-ERROR-0016	PLMN ID Validation In Header Failed	NA	This error is observed if PLMN ID is not matched in header of the incoming request on PN32F microservice.	Verify if correct regex is configured against the header identifier in Header Configuration. Verify if MCC & MNC combination is present in the helm based local PLMN ID list.
SEPP-PN32FSEPP-ERROR-0017	PLMN ID Validation In Body Failed	NA	This error is observed if PLMN ID is not matched in body of the incoming request on PN32F microservice.	Verify if correct regex is configured against the body IE in Body IE Configuration. Verify if MCC & MNC combination is present in the helm based local PLMN ID list.
SEPP-SECURITY-PLMN-HEADER-ERROR-0015	PLMN ID Validation In Header Failed	NA	Verify if correct regex is configured against the header in header Configuration. Verify if MCC & MNC combination is present in the helm based local PLMN ID list.	if PLMN ID is not matched in the header of the incoming request. Metrics can be checked for the details for which it has failed.
SEPP-SECURITY-PLMN-BODY-ERROR-0016	PLMN ID Validation in body failed	NA	Verify if correct regex is configured against the body IE in Body IE Configuration. Verify if MCC and MNC combination is present in the helm based local PLMN ID list.	if PLMN ID is not matched in the body of the incoming request. Metrics can be checked for the details for which it has failed.

The following are the error codes of Mediation feature:

Table 3-3 Mediation Error Codes and Recovery Steps

Error Code	Error Text	Command/ Method	Description	Recovery Steps
SEPP-MEDIATION-ERROR-001	Mediation Trigger Rule Not Found.	GET x`/sepp-mediation-trigger-rule-list/{triggerRuleListName} DELETE /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Requested Trigger Rule List does not exist.	Give the Trigger Rule List Name that exist in Database.
SEPP-MEDIATION-ERROR-002	Unsupported Trigger Points	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Trigger Points provided in configuration request body is not supported.	Provide valid Trigger Points in Configuration Request. Valid Trigger Points : N32_Egress_Request N32_Ingress_Response N32_Ingress_Request N32_Egress_Response
SEPP-MEDIATION-ERROR-003	Mediation Trigger Mandatory Parameter Update Not Allowed.	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Trigger Rule List mandatory parameter like TriggerRuleListName can not be updated.	Make sure TriggerRuleListName you are providing in configuration request url is same as name in request url path.
SEPP-MEDIATION-ERROR-004	Trigger Rule is Mandatory Parameter.	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	If MediationAllEnabled is false and there is no TriggerRules or empty TriggerRules in request.	If MediationAllEnabled is false, then make sure there is TriggerRules Provided in Request.
SEPP-MEDIATION-ERROR-005	Invalid Error Status Code	PUT /mediation/feature	Invalid Http Status Code is provided is Error Configuration Request.	Make sure to provide valid HTTP Status Code in statusCode field in Error Configuration Request
SEPP-MEDIATION-ERROR-006	Invalid ResourceURI and HTTPMethod Error	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	ResourceURI and HttpMethod provided in Request is not valid.	Make sure to provide ResourceURI and HttpMethod combination that is configuration for SEPP, already present in Database.
SEPP-MEDIATION-ERROR-007	DELETE Not Allowed	DELETE /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Trigger Rule List Delete Not Allowed.	Ensure you are deleting only that Trigger Rule List that is not associated with Remote SEPP Set.
SEPP-MEDIATION-ERROR-008	Trigger Rule Configuration Error	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Error in Configuration of Trigger Rule List.	Ensure there is no duplicated ResourceURI and Method in request body.
SEPP-MEDIATION-ERROR-009	Mediation Trigger Rules Configuration Mandatory Parameter Missing Error	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Mediation Trigger Rules Configuration Mandatory Parameter Missing in configuration request.	Ensure all mandatory parameters are present in Mediation Trigger Rule Configuration request.
SEPP-MEDIATION-ERROR-010	Multiple Local Trigger Rule List Configuration Error	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Multiple Local Trigger Rule List Configuration is Not Allowed.	Make sure we are not configuring another Local Trigger Rule List if there is already one configured in DB. Only one Local Trigger Rule List can be configured for SEPP.
SEPP-MEDIATION-ERROR-011	Mediation Feature Mandatory Parameter Error	PUT /mediation/feature	Mediation Feature Configuration Mandatory Parameter is missing.	Make sure if FeatureEnabled is true in request then all field Error Configuration is present in request.
SEPP-MEDIATION-ERROR-012	Mediation Local Trigger Rule IPX Mode Error	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	In IPX Mode SEPP allow only 2 Trigger Points (N32 Ingress Request, N32 Egress Response) in local TRL configuration.	Make sure there is not any invalid Trigger Points like N32 Egress Request or N32 Ingress Response in Local Trigger Rule Configuration Request.
SEPP-MEDIATION-ERROR-013	Mediation service is not available	PUT /mediation/feature	Mediation Service Not deployed.	Before Enabling Mediation Feature through API , make sure Mediation Service is being deployed for SEPP.
SEPP-MEDIATION-ERROR-014	Invalid Error Action	PUT /mediation/feature	Invalid Error Action in Mediation Feature Configuration request.	Make sure to provide valid ErrorAction in Mediation Feature Configuration request in ErrorConfiguration section.

The following are the error codes of Cat-1 feature:

Table 3-4 Cat-1 Error Codes and Recovery Steps

Error Code	Error Text	Description	Recovery
SEPP-SECURITY-ERROR-001	Service API not in allowed list	If resource URI and Http Method is not matched as per the configured allowed list on SEPP.	Verify whether the proper Resource URI and HTTP method is configured in the CNC Console GUI. Go to the Security Countermeasure section, check under the Service API Allowed List for that particular Allowed list name for N32 Egress or N32 Ingress Direction.
SEPP-CN32FSEPP-ERROR-0012	Service API Validation Failed	This error occurs on CN32F microservice. If resource URI and Http Method is not matched as per the configured allowed list on SEPP.	Verify whether the proper Resource URI and HTTP method is configured in the CNC Console GUI. Go to the Security Countermeasure section, check under the Service API Allowed List for that particular Allowed list name for N32 Egress or N32 Ingress Direction.
SEPP-PN32FSEPP-ERROR-0015	Service API Validation Failed	This error occurs on PN32F microservice. If resource URI and Http Method is not matched as per the configured allowed list on SEPP.	Verify whether the proper Resource URI and HTTP method is configured in the CNC Console GUI. Go to the Security Countermeasure section, check under the Service API Allowed List for that particular Allowed list name for N32 Egress or N32 Ingress Direction.

The following are the error codes of Cat-2 Network ID Validation feature:

Table 3-5 Cat-2 Network ID Error Codes and Recovery Steps

Error Code	Error Text	Description	Recovery
SEPP-CN32FSEPP-ERROR-0013	Network ID Validation In Header Failed	Check whether PLMN ID is not matched in header of the incoming request on CN32F microservice.	Verify if correct regex is configured against the header identifier in Header Configuration. Verify if MCC and MNC combination is present in the visitor or target PLMN ID List based on the associated SEPP configuration
SEPP-CN32FSEPP-ERROR-0014	Network ID Validation In Body Failed	Check whether PLMN ID is not matched in body of the incoming request on CN32F microservice.	Verify if correct regex is configured against the body IE in Body IE Configuration. Verify f MCC and MNC combination is present in the home or visitor PLMN ID List based on configuration of associated SEPP.
SEPP-PN32FSEPP-ERROR-0016	Network ID Validation In Header Failed	Check whether PLMN ID is not matched in header of the incoming request on PN32F microservice.	Verify if correct regex is configured against the header identifier in Header Configuration. Verify if MCC and MNC combination is present in the target or visitor PLMN ID list based on the associated SEPP configuration.
SEPP-PN32FSEPP-ERROR-0017	Network ID Validation In Body Failed	Check whether PLMN ID is not matched in body of the incoming request on PN32F microservice.	Verify if correct regex is configured against the body IE in Body IE Configuration. Verify if MCC and MNC combination is present in the target or visitor PLMN ID list based on the associated SEPP configuration.

Table 3-6 Cat-3 Previous Location Check Error Codes

Error Code	Error Text	Description	Recovery
SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-ERROR-0019 or SEPP-PN32FSEPP-ERROR-0018	Previous Location Check Validation Failed	This error code is observed only on PN32F microservice. If Serving Network ID is not matching against the serving network ID coming from UDR response, to check whether UE authentication is success. This error also occurs if the authentication from UDR is false.	Verify whether the correct regex is configured against the serving network identifier in either Header or Body Configuration. Verify whether the MCC and MNC combination is present in the Serving Network ID. Verify whether the identifier coming in the ingress request on PN32F microservice is also same as the serving Network name coming as part of the UDR response if UE authentication is successful.
SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-EXCEPTION-0020	Previous Location Check Validation Failed Due To Exception	This error will occur if the system is not able to extract the SUPI, or if the incoming message doesn't contain SUPI, or if there are any sort of connectivity issues with NRF or UDR.	Verify if correct regex for UE ID in Header or Body configuration screen is configured due to which correct UE ID value is extracted. Verify if the incoming message has SUPI. Verify whether the FQDN or IP fetched for the UDR as part of NRF discovery call is reachable. Verify if UDR discovery procedure from NRF is successful. Verify if the SUPI received in the Ingress request message is part of the SUPI range received in UDR profile from discovery response from NRF.

Error Code

Error Text

Description

Recovery

SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-ERROR-0019 or

SEPP-PN32FSEPP-ERROR-0018

Previous Location Check Validation Failed

This error code is observed only on PN32F microservice.

If Serving Network ID is not matching against the serving network ID coming from UDR response, to check whether UE authentication is success.

This error also occurs if the authentication from UDR is false.

Verify whether the correct regex is configured against the serving network identifier in either Header or Body Configuration.
Verify whether the MCC and MNC combination is present in the Serving Network ID.
Verify whether the identifier coming in the ingress request on PN32F microservice is also same as the serving Network name coming as part of the UDR response if UE authentication is successful.

SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-EXCEPTION-0020

Previous Location Check Validation Failed Due To Exception

This error will occur if the system is not able to extract the SUPI, or if the incoming message doesn't contain SUPI, or if there are any sort of connectivity issues with NRF or UDR.

Verify if correct regex for UE ID in Header or Body configuration screen is configured due to which correct UE ID value is extracted.
Verify if the incoming message has SUPI.
Verify whether the FQDN or IP fetched for the UDR as part of NRF discovery call is reachable.
Verify if UDR discovery procedure from NRF is successful.
Verify if the SUPI received in the Ingress request message is part of the SUPI range received in UDR profile from discovery response from NRF.