Troubleshooting SEPP

3 Troubleshooting SEPP

This section provides information to troubleshoot the common errors which can be encountered during the installation and upgrade of SEPP:

Note:

kubectl commands might vary based on the platform deployment. Replace kubectl with Kubernetes environment-specific command line tool to configure Kubernetes resources through kube-api server. The instructions provided in this document are as per the Oracle Communications Cloud Native Environment (OCCNE) version of kube-api server.

Caution:

User, computer and applications, and character encoding settings may cause an issue when copy-pasting commands or any content from PDF. PDF reader version also affects the copy-pasting functionality. It is recommended to verify the copy-pasted content, especially when hyphens or any special characters are part of the copied content.

3.1 Generic Troubleshooting Scenarios

The following are the generic troubleshooting scenarios:

3.1.1 Generic Checklist

Environment Verification

The following sections provide generic checklist for troubleshooting tips:

a. Deployment related tips

Perform the following checks before the deployment:

Are OCSEPP deployment, pods, and services created, running, and available? .
To check this, run the following command:
```
# kubectl -n get deployments,pods,svc
```

Inspect the output and check the following columns:

AVAILABLE of deployment
READY, STATUS, and RESTARTS of pod
PORT(S) of service

b. Is the correct image used and the correct environment variables set in the deployment?

To check this, run the following the command:

# kubectl -n <namespace> get deployment <deployment-name> -o yaml

c. Inspect the output, check the environment and image.

# kubectl -n seppsvc get deployment sepp-release-1-n32-egress-gateway -o yaml

d. Check if the microservices can access each other through REST interface.

To check this, run following command:

# kubectl -n <namespace> exec <pod name> -- curl <uri>

3.1.2 The environment is not working as expected

Problem:

The environment is not working as expected.

Solution:

Check if kubectl is installed and working as expected.
Check if kubectl version command works: This must display the versions of client and server.
Check if $ kubectl create namespace testcommand works.
Check if kubectl delete namespace test command works.
Check if Helm is installed and working as expected.
Check if helm version command works: This must display the versions of client and server.

3.1.3 Kubernetes Node Failure

Problem

Kubernetes nodes goes down.

Error Code/Error Message

"NotReady" status is displayed against the Kubernetes node.

Symptom

On running the command kubectl get nodes, "NotReady" status is displayed.

Solution

Following is the procedure to identify the kubernetes nodes failure:

Run the following command to describe the node:
```
kubectl describe node <kubernete_node_name>
```
Check the nodes utilization by running the following command:
```
kubectl top nodes
```

3.1.4 SEPP Installation Verification

Problem: The SEPP installation is not successful.

Solution:

Verify if SEPP specific pods are working as expected by running the following command:

kubectl get pods -o wide -n <ocsepp _namespace>

Check whether all the pods are up and running.

Sample output:

NAME     TYPE           CLUSTER-IP     EXTERNAL-IP  PORT(S)                     AGE
ocsepp-release-appinfo-55b8d4f687-wqtgj                      1/1   Running   0     141m
ocsepp-release-cn32c-svc-64cd9c555c-ftd8z                    1/1   Running   0     113m
ocsepp-release-cn32f-svc-dd886fbcc-xr2z8                     1/1   Running   0     4m4s
ocsepp-release-config-mgr-svc-6c8ddf4c4f-lb4zj               1/1   Running   0     141m
ocsepp-release-n32-egress-gateway-5b575bbf5f-z5bbx           2/2   Running   0     131m
ocsepp-release-n32-ingress-gateway-76874c967b-btp46          2/2   Running   0     131m
ocsepp-release-ocpm-config-65978858dc-t4t5k                  1/1   Running   0     141m
ocsepp-release-performance-67d76d9d58-llwmt                  1/1   Running   0     141m
ocsepp-release-plmn-egress-gateway-6dc4759cc7-wn6r8          2/2   Running   0     31m
ocsepp-release-plmn-ingress-gateway-56c9b45658-hfcxx         2/2   Running   0     131m
ocsepp-release-pn32c-svc-57774fdc4-2qpvx                     1/1   Running   0     141m
ocsepp-release-pn32f-svc-586cd87c7b-pxk6m                    1/1   Running   0     3m47s
ocsepp-release-sepp-nrf-client-nfdiscovery-65747884cd-qblqn  1/1   Running   0     141m
ocsepp-release-sepp-nrf-client-nfmanagement-5dd6ff98d6-cr7s7 1/1   Running   0     141m
ocsepp-release-nf-mediation-74bd4dc799-d9ks2                 1/1   Running   0     141m

If status of any pod is shown as ImagePullBackOff or ErrImagePull, then it can be due to:
1. Incorrect ImageName provided in ocsepp_custom_values_<versions>.yaml.
  Then, double check the image name and tags in ocsepp_custom_values_<versions>.yaml.
2. Docker registry is incorrectly configured.
  Then, check docker registry is properly configured in all master and worker nodes.
If RESTARTS count of the pods is continuously increasing, then it can happen due to the following reasons:
1. MySQL primary and secondary hosts may not be configured properly in ocsepp_custom_values_<versions>.yaml
2. MySQL servers may not be configured properly. For more information about the MySQL configuration, see theSEPP Predeployment Configuration section in Cloud Native Core. Security Edge Protection Proxy Installation, Upgrade ,and Fault Recovery Guide.

3.1.5 Debugging General CNE

Problem: The environment is not working as expected

Solution:

Run the command kubectl get events -n <ocsepp_namespace>to get all the events related to a particular namespace.

3.1.6 Collecting the SEPP Logs to Check the Error Scenarios

Problem: The error scenarios are checked by collecting the SEPP logs.

Solution:

Run the following commands to get the logs from SEPP specific pods:

Run the following command to get the pods details:
```
$ kubectl -n get pods
```
Run the following command to collect the logs from the specific pods or containers:
```
kubectl get pods -n <ocsepp_namespace> 
```
Collect the logs from the pod and redirect to file by running kubectl logs <pod_name> -n <ocsepp_namespace> > <Log File>

Example:

kubectl logs - seppsvc-cn32f-svc-57cff5665c-skk41 -n seppsvc > seppsvc_logs1.log

3.1.7 Helm Error During the Rollback

Problem

The Helm rollback causes failure and displays the following error:

"Duplicate value: "cnc-metrics" && cannot patch"

Symptom

The error indicates that the Helm is not able to merge the current and rollback charts.

Solution

Run the helm rollback command again with -–force to resolve the issue.

Note:

If the rollback is to be performed using –force, take the backup of configmap data as the config map data can be cleaned by Helm.

3.1.8 Upgrade or Rollback Failure

When Security Edge Protection Proxy (SEPP) upgrade or rollback fails, perform the following procedure:

Check the pre or post-upgrade or rollback hook logs in Kibana as applicable.
Users can filter upgrade or rollback logs using the pod name filter

example: ocsepp-release-update-db

Check the pod logs in Kibana to analyze the cause of failure.

After detecting the cause of failure, do the following:

For upgrade failure:

If the cause of upgrade failure is database or network connectivity issue, contact your system administrator. When the issue is resolved, rerun the upgrade command.
If the upgrade failure occurs during the preupgrade phase, resolve the issue, then perform a upgrade. Do not perform rollback because SEPP deployment remains in the source or older release.
If the upgrade failure occurs during the postupgrade phase, for example, post upgrade hook failure due to target release pod not moving to ready state, then perform a rollback.

For rollback failure:

If the cause of rollback failure is database or network connectivity issue, contact your system administrator. When the issue is resolved, rerun the rollback command.

If the issue persists, contact My Oracle Support.

3.1.9 Helm Test Failure

Following are the troubleshooting steps if helm test is not getting initiated:

Run the following command to get the cn32f-svc name:
```
kubectl get svc -n namespace
```
Example:
```
kubectl get svc -n seppsvc
```
Add the following property under the global section in the custom-values.yaml file.
```
.global.seppServiceAccountName = cn32f-svc name 
```
Upgrade to the same app version using updated custom-values.yaml file for changes to be updated in the installation.
Run helm test.

3.1.10 Helm Rollback Failure with the Configmap with the Name not Found Error

Problem

Helm rollback results in failure and displays the following error:

Rollback "ocsepp" failed: no ConfigMap with the name "rss-ratelimit-map" found

Symptom

The above-mentioned error indicates that Helm is not able to merge current and rollback charts.

Solution

Run the Helm rollback again with --force to resolve the issue.

Note:

If the rollback is to be performed using –force, take the backup of configmap data as the config map data can be cleaned by Helm.

3.1.11 Continuous Restart of coherence-svc Pods

Problem: Helm install might fail if the coherence-svc pod is restarting repeatedly.

When you run kubectl get pods -n <ocsepp_namespace>, the coherence-svc pods restart count increases continuously.

Solution: Delete the coherence-svc pod using kubectl delete po -n <namespace> <coherence pod> and the pod will be up and running.

3.1.12 IllegalReferenceCount Exception Occurence in Logs of Ingress and Egress Gateways

Problem: In some environments, there can be IllegalReferenceCount exception in the Ingress or Egress logs which results in an unexpected traffic drop. This is visible if the incoming traffic at Gateway is not equal to the outgoing traffic.

Solution:

Check the per second occurrence of IlleagalReferenceCount exception on the Gateway pods.
Update the following configurations in the Config Map of the affected Gateway:
```
nettyInboundExceptions:
  exceptions:
    - io.netty.util.IllegalReferenceCountException
  count: 1000 //Update this to a value less than the per second occurrence
  timePeriod: 1
```
This resets the HTTP2 connection when the count reaches the configured value for this exception within the given time period.

3.1.13 False Message while Doing the Helm Uninstall

Problem: The user gets the following false message while doing the Helm uninstall:


These resources were kept due to the resource policy:
[ConfigMap] egress-ratelimit-map
[ConfigMap] rss-ratelimit-map
release "ocsepp-release" uninstalled.

Solution:

Run the following command to reverify whether the SEPP uninstallation is successful and the config maps are deleted:

kubectl get cm -n <namespace>

Output:


[seppuser@thrust6-bastion-1 ~]$ kubectl get cm -n <namespace>
NAME                 DATA   AGE
istio-ca-root-cert   1      87m
kube-root-ca.crt     1      87m
[seppuser@thrust6-bastion-1 ~]$

Note:

The listed output should not have 'egress-ratelimit-map' and ' rss-ratelimit-map'.

3.2 Feature Specific Troubleshooting Scenarios

The following are the feature specific troubleshooting scenarios:

3.2.1 Cat-2 Network ID Validation Feature

The following are the troubleshooting scenarios of Cat-2 Network ID Validation feature:

The incoming request is rejected at CN32F microservice

Problem:

Incoming request gets rejected with error code configured ( 406 - default error code ) at CN32F microservice.

Solution:

Search for error code SEPP-CN32FSEPP-ERROR-0013 or SEPP-CN32FSEPP-ERROR-0014 in CN32F logs.
Verify if correct regex is configured under Header or Body IE tab under Cat 2 – Network ID Validation Section of Security Countermeasure tab under SEPP.
Verify whether PLMN ID sent in the headers is part of PLMN ID List based on the associated SEPP configured.

The incoming request is rejected at PN32F microservice

Problem:

Incoming request gets rejected with error code configured (406 - default error code) at PN32F microservice.

Solution:

Search for error code SEPP-PN32FSEPP-ERROR-0016 or SEPP-PN32FSEPP-ERROR-0017 in PN32F logs.
Verify if correct regex is configured under Header or Body IE tab under Cat 2 – Network ID Validation Section of Security Countermeasure tab under SEPP.
Verify whether PLMN ID sent in the headers is present in the PLMN ID List based on the associated SEPP configured.

Invalid PLMN ID in Header configurations

Problem:

Invalid PLMN ID in Header configurations is received on SEPP.

Solution:

Verify if correct regex is configured against the header identifier in Header Configuration.
If error is thrown from CN32F microservices, verify if MCC and MNC combination is present in the PLMN ID List based on the associated SEPP configurations.
If error is thrown from PN32F microservices, verify proper configurations PLMN ID list based on the associated SEPP configurations.

Invalid PLMN ID in body configurations

Problem:

Invalid PLMN ID in body configurations is received on SEPP.

Solution:

Verify if correct regex is configured against the body IE in Body IE Configuration.
If the error is from CN32F microservices, verify f MCC and MNC combination is present in the PLMN ID List based on associated SEPP configured.
If the error is from PN32F microservices, verify if MCC and MNC combination is present in the PLMN ID list based on associated SEPP configured.

3.2.2 Cat-1 Service API Validation Feature

The incoming request is rejected at CN32F:

Problem: The incoming request is rejected with the configured status code (default status code is 406) at CN32F microservice.

Solution:

Search for error code SEPP-CN32FSEPP-ERROR-0012 in CN32F logs.
Verify whether the proper HTTP method and Resource URI combination is sent in the request in CN32F logs.
Check allowed list name configured against Remote SEPP Set.
Verify whether the correct Resource URI and HTTP method is configured in the CNC Console GUI. Go to the Security Countermeasure section, check under the Service API Allowed List for that particular Allowed list name for N32 Egress or N32 Ingress Direction.
If the user is configuring a new Resource URI, ensure to configure the correct regular expression.

3.2.3 Overload Control Feature

Problem:

Incoming request does not get rejected with error code configured in CNC Console (429 - default error code) at N32 Ingress Gateway.

Solution:

Check whether the feature is enabled using the API: curl -XGET http://<config-server>:<port>/sepp/nf-common-component/v1/igw/n32/ocpolicymapping
Check if the correct policy is applied using the API: curl -XGET http://<config-server>:port/sepp/nf-common-component/v1/igw/n32/ocpolicymapping
Check the svcName parameter to verify whether the release name is correct or not for pn32f-svc.

Problem:

Scenario 1:

Feature is configured and enabled using REST API, still request is not getting rejected with the configured error code.

Solution

Fetch the current load level for N32 Ingress Gateway using the following API:

curl 'http://<release-name>-n32-ingress-gateway:80/igw/load-level?svcName=<relase-name>-pn32f-svc'--http2-prior-knowledge

If the above API has the output "Normal", check CPU and memory thresholds defined using the API:
```
curl -XGET http://<config-server>:port/sepp/nf-common-component/v1/perf-info/overloadLevelThreshold
```
Check the CPU and memory statistics from Grafana or Prometheus to check the current CPU and memory usage.
Use the cgroup_cpu_nanoseconds and cgroup_memory_bytes metrics for the service mapping.
Either of the metrics value should reach the "onsetvalue" for a particular threshold level (defined in step 2) to be applied and feature to run.

Scenario 2:

The API mentioned in step 1 for scenario 1 returns the following error:

{"type":null,"title":"Service
          Unavailable","status":503,"detail":"Load level
          for service ocsepp-release-pn32f-svc is not Configured at Ingress-Gateway","instance":null,"cause":"Load level for service ocsepp-release-pn32f-svc is not Configured at
          Ingress-Gateway","invalidParams":null}

Solution

Check the ocsepp_custom_values_<version>.yaml file.
In the Perf-info section, check the tagNamespace value. The value must be either "namespace" or "kubernetes_namespace" depending on the CNE version used.
Check the configMap.prometheus value. This should map to Prometheus IP and port or service IP path used to access Prometheus.
If any of the above have been incorrectly set, change and re-deploy SEPP.

Scenario 3:

The API mentioned in step 1 for scenario 1 returns "Connection refused" error.

Solution

Run the following command:

kubectl get svc -n <namespace> | grep n32-ingress-gateway

If the output does not have port 80 present in service, do the following:
1. In the ocsepp_custom_values_<version>.yaml, set the enableIncomingHttp to true in the N32 Ingress Gateway section.
2. Re-deploy SEPP or upgrade the N32-Ingress-gateway service.
3. Verify that the port 80 is enabled by running the step 1.

3.2.4 Troubleshooting Steps for Rate Limiting Feature

Problem

Request not getting rejected with configured code.

Solution

Check rateLimiting.enabled. This parameter must be set to True.
Check globalIngressRateLimiting.enabled. This parameter must be set to True.
For Egress rate limiting, check egressRateLimiting.enabled must be set to True.

Problem

Request not getting rejected with configured error code.

Solution

In Ingress Gateway check for errorCodeProfiles in ocsepp_custom_values_<version>.yaml file.
Check profile name: ERR_1200.
Change the error code from 503 to desired value.
Upgrade or re-deploy SEPP

3.2.5 Message Feed Feature

The following are the troubleshooting scenarios of Message Feed feature:

Problem:

Messages of same transaction are getting copied to different partitions.

Solution:

Verify the configurations on all four gateways. keybasedKafkaProducer parameter should be set to true on all 4 gateways (n32-egress-gateway, plmn-ingress-gateway, n32-egress-gateway, and plmn-egress-gateway).

Problem:

The feature is not working in SASL_SSL or SSL mode.

Solution:

Verify the Data Director configurations and secrets in the Oracle Communications Network Analytics Data Director Installation, Upgrade, and Fault Recovery Guide.

Problem

Message Copy feature not copying JSON data to Data Director (DD).

Solution

Check whether the feature is enabled or not.
Check if copyPayload is set to false. If yes, set to True.
After re-deploying OCSEPP, verify if the data is copied at DD.

Problem

Message copy not copying data to DD as incorrect IP Port.

Solution

Check whether the feature is enabled or not.
Check whether the security enabled or not.
If security set to false, check whether DD Unreachable<GW> has been raised.
If yes , then Kafka.bootstrapAddress parameter must be set to correct listener IP and port
After re-deploying OCSEPP, verify whether data is copied at DD.

Problem

Message copy not copying data to DD as topic name incorrect.

Solution

Check whether the feature is enabled or not.
Check whether the security enabled or not.
If security set to false, check whether the DDUnreachable<GW> has been raised.
If not, check the topicName parameter. This topic should be created in DD so that data copied can be seen on DD.
After creating topic, verify if data is copied at DD.

Problem

Message copy not copying data to DD (security enabled) (Case A).

Solution

Check whether the feature is enabled or not.
Check whether the security enabled or not.
If security set to true ,check whether the DDUnreachable<GW> has been raised
If yes, then Kafka.bootstrapAddress parameter must be set to correct listener IP and security port for DD.
After re-deploying OCSEPP, verify if data is copied at DD.

Problem

Message copy not copying data to DD (security enabled) (Case B).

Solution

Check whether the feature is enabled or not.
Check whether the security enabled or not.
If security set to true, check whether the DDUnreachable<GW> has been raised
If not , then check the security configurations for DD.
Check the following parameters:
1. userName: must be the same as used to configure DD.
2. password: Check the secret name and Namespace details if correct.
After re-deploying OCSEPP, verify if data is copied at DD.

Note:

All the values must be checked in ocsepp_custom_values_<version>.yaml file, as this is a Helm based feature.

3.2.6 Hosted SEPP

Problem:

The feature is Enabled and Consumer Remote SEPP Set not found (Default error code = 400).

Error Code or Error Message

Consumer Remote SEPP Set not found.

Solution:

Check whether the following error is displayed in logs. The error is displayed if allowed P-RSS Validation is enabled and no consumer RSS is configured.

{"instant":{"epochSecond":1668703429,"nanoOfSecond":698428472},"thread":"reactor-http-epoll-4","level":"ERROR","loggerName":"com.oracle.cgbu.cne.ocsepp.cn32f.handler.Cn32fSeppHandler","message":"HostedSEPPException: Request not allowed as source remote sepp set not found","contextMap":{"ocLogId":"1668703429683_71_ocsepp-release-mohit-plmn-ingress-gateway-7b86f4855c-ph9xj"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":15,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"22-11-17 16:43:49.698+0000","processId":"1","ocLogId":"1668703429683_71_ocsepp-release-mohit-plmn-ingress-gateway-7b86f4855c-ph9xj"}

Verify if consumer Remote SEPP Set is present on Hosted SEPP.

Problem:

The feature is enabled and destination Roaming Partner Set is null (Error code = 400) or Remote SEPP Set is not found (Error code = 404).

Error Code or Error Message

Destination RPS is null

Solution:

Above error is displayed if Allowed P-RSS Validation is enabled and no producer Remote SEPP Set is configured.
Verify if the producer Remote SEPP Set is present on Hosted SEPP.

Error Code or Error Message

destinationRPS not present

Problem:

The feature is enabled and destination Roaming Partner Set not present (Default error code = 400).

Solution:

Following logs is displayed and the following error is displayed if destination Sepp Set is not configured in allowedProducerRemoteSeppSets of Consumer SEPP Set:

{"instant":{"epochSecond":1668705561,"nanoOfSecond":940785763},"thread":"reactor-http-epoll-3","level":"ERROR","loggerName":"com.oracle.cgbu.cne.ocsepp.cn32f.handler.Cn32fSeppHandler","message":"HostedSEPPException: Request not allowed as remote sepp set psepp not present in allowed list","contextMap":{"ocLogId":"1668705561928_135_ocsepp-release-mohit-plmn-ingress-gateway-7b86f4855c-ph9xj"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":15,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"22-11-17 17:19:21.940+0000","processId":"1","ocLogId":"1668705561928_135_ocsepp-release-mohit-plmn-ingress-gateway-7b86f4855c-ph9xj"}

Verify whether the destinationRPS is present in allowedProducerRemoteSeppSets configured At RSS of Consumer.
If it is present, then wait for cache refresh to take place as configured.

3.2.7 Steering of Roaming (SOR) Feature

The following are the troubleshooting scenarios of Steering of Roaming (SOR) feature:

SOR feature is not enabled.

Problem:

SOR feature is not enabled.

Solution:

Verify the following scenarios:

Check whether the SOR feature is enabled at CNC Console or REST API.
Check the Remote SEPP Set, validate SOR is enabled for the given RSS.
Check Roaming Hub is disabled, and SEPP is deployed in SEPP Mode.

SOR feature is enabled at Global or RSS level but SOR is still disabled.

Problem:

SOR is enabled at Global or RSS level but SOR is still disabled.

Solution:

Verify the following scenarios:

Check Remote SEPP Set Configuration, check the SOR list name associated with RSS.
Verify the method plus URI that is passed in the message request exists in the SOR List.

SOR is configured with Retry as true and server header value is provided but retry is not working.

Problem:

SOR is configured with Retry as true and server header value is provided but retry is not working.

Solution:

Verify that the server header value given at the time of configuration matches the value that reaches in server header in the response.
Example: SOR server header value is configured as SOR-sorfqdn.com, and message request is sent, error response is received with server header value as 'server': 'SOR-sorfqdn.com'.
In this case, if retry is true then retry will be performed.
If server header value is not matched, even when retry is true, retry will not be performed.

3.2.8 Rate Limiting for Ingress Roaming Signaling per Remote SEPP Set Feature

The following are the troubleshooting scenarios of Rate Limiting for Ingress Roaming Signaling per Remote SEPP Set feature:

Problem: Unable to see reject list for traffic sent

Solution:

Check in ocsepp_custom_values_<version>.yaml file if the rssRateLimiter.enabled parameter for N32 Ingress Gateway is set to true.
Check whether the feature is enabled on Options screen under Remote SEPP Set, which is under Ingress Rate Limiting, at CNC Console by checking whether Remote SEPP Set Ingress Rate Limiting Enabled parameter is set to true.
Check whether the feature is enabled on RSS level by checking RSS Ingress Rate Limiting Enabled parameter is set to true on Remote SEPP Set screen of CNC Console for the particular PLMN traffic.
Check whether the the header configured in Originating Network ID Header parameter on Options screen under Remote SEPP Set, which is under Ingress Rate Limiting, at CNC Console, is being sent in traffic.

Problem: Unable to see status code in traffic set on Ingress Rate Limiting at CNC Console

Solution:

Check whether the status code configured is present in the RSS by checking Error configuration under Ingress Rate Limiting option in Remote SEPP Set screen for which the PLMN is being extracted.
Change the error code in RSS by editing Error Configuration under Ingress Rate Limiting option in Remote SEPP Set screen.

Problem: Unable to see error detail in traffic set on Ingress Rate Limiting at CNC Console

Solution:

Check whether the status code configured is present in the RSS by checking Error configuration under Ingress Rate Limiting option in Remote SEPP Set screen for which the PLMN is being extracted.
Change the error code in RSS by editing Error Configuration under Ingress Rate Limiting option in Remote SEPP Set screen.

Problem: Status code set to a different code in Error Configuration but Status code 429 is seen in rejected requests

Solution:

Check if status code set on CNC Console is a valid HTTP Status code or in the series of 3xx. By default, these will be modified to 429.

Problem: Server header observed in logs

Solution:

Server Header is added for the following Status codes - 408, 404, 400, and 429.

3.2.9 Cat-3 Previous Location Check feature

Problem:

Ingress request message gets rejected and displays the error code configured in the CNC Console (406 - default error code) at PN32F microservice.

Solution:

Search for the error codes SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-ERROR-0019 or SEPP-PN32FSEPP-ERROR-0018 or SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-EXCEPTION-0020 in PN32F microservice logs.
Verify if the correct regex is configured in Header or Body IE configuration for UE ID and Serving Network ID under Cat 3 – Previous Location Check Section under Security Countermeasure of SEPP CNC Console.
Verify if the MCC and MNC from serving network configured in either Header or Body is matching with the serving network name. The MCC and MNC values are part of the UDR response. Check whether the UDR response is success.
SUPI must be present in the incoming message, if it is configured for Cat-3 Previous Location Check.
UDR discovery procedure must be successful.
Coherence service must be up and running.
SUPI must be part of the IMSI range coming as part of the UDR profile received in UDR discovery response.
FQDN or IP of UDR must be reachable.
Proper DNS resolutions must be done for UDR discovery call, pn32f-svc for subscription use case.

3.2.10 Cat-3 Time check for Roaming Subscribers

The following are the troubleshooting scenarios of Cat-3 Time check for Roaming Subscribers feature:

Problem:

The Ingress Request message is rejected and shows the error code configured in the CNC Console (406 - default error code) at the PN32F microservice.

Solution:

Ensure that SUPI or SUCI is included in the incoming message.
Confirm that the UDR discovery procedure is successful.
Confirm that the UDM discovery procedure is successful.
Ensure that the Coherence service is up and running.
Check that SUPI is within the IMSI range provided in the UDR profile from the UDR discovery response.
Verify that the FQDN or IP of the UDR is reachable.
Make sure proper DNS resolution is done for the UDR discovery call, as well as for the pn32f-svc in the subscription use case.
Look for the error codes SEPP-UNAUTHENTICATED-LOCATION-TIME-CHECK-VALIDATION-ERROR-0021 or SEPP-UNAUTHENTICATED-LOCATION-TIME-CHECK-VALIDATION-EXCEPTION-0022 in the PN32F microservice logs.
Check if Cat-3 Time Check for Roaming Subscribers is enabled at both global and remote levels. If it's disabled at either level, the feature won't work.
Ensure that supiOrSuci and servingNetworkName (containing MCC and MNC) are present in the /nausf-auth/v1/ue-authentications Request JSON body.
Verify that the UDR response is successful, and check if the servingNetworkName in the /nausf-auth/v1/ue-authentications Request JSON body differs from the servingNetworkName in the UDR response.

Detailed Debugging Steps of the Feature

Following are the detailed Debugging steps of the feature:

Feature Configurations:

Verify the configurations on CNC Console for Cat-3 Time check for Roaming Subscribers feature.

Perform the following procedure to verify the Cat-3 Time check for Roaming Subscribers feature configurations:

From the left navigation menu, navigate to SEPP and then click Security Countermeasure.
Click Cat-3 Time Location Check under Security Counter Measure, Unauthenticated Location page appears underneath.
Click Unauthenticated Location under Security Countermeasure. The Option appears underneath.
Click Option, the option screen appears at the right pane. The Cat-3 Time check for Roaming Subscribers feature details are available on the screen.
Click Edit icon to modify the Option. The Edit Option page appears.
Set the Cat 3 Time Check Unauthenticated Location Enabled to True.
To access the Remote SEPP Set screen, click SEPP and then click Remote SEPP Set.
To enable the feature, the user also needs to enable the Cat3-Time Location Check - Unauthenticated Location Enabled parameter available at Remote SEPP Set.

Verify the configurations on REST API for Cat-3 Time check for Roaming Subscribers feature.

Using /sepp-configuration/v1/security-counter-measure/time-location-check/unauthenticated-location API, configure timeUnAuthenticatedCheckValidationEnabled to true along with other mandatory parameters to enable the feature at global level.
Using /sepp-configuration/v1/remoteseppset API, configure messageFilterOnTimeUnAuthCheckEnabled to true along with other mandatory parameters to enable the feature at RSS level.

Check Metrics for Error or Exception Scenario Analysis:

Evaluate the metric ocsepp_time_unauthenticated_location_validation_requests_total to check the total number of validation requests received.
Evaluate the metrics ocsepp_time_unauthenticated_location_validation_failure_total and ocsepp_time_unauthenticated_location_exception_failure_total to identify any failures or exceptions encountered during the validation process.
Evaluate the metrics ocsepp_time_unauthenticated_location_blacklist_requests_total to obtain information about requests that have been blacklisted.

Confirm Measurement Units:

If a request is being rejected, it might be due to a mismatch in the units of measurement used. Ensure to use the correct units when configuring the parameters.

From the left navigation menu, navigate to SEPP and then click Security Countermeasure.
Click Cat-3 Time Location Check under Security Counter Measure, Unauthenticated Location page appears underneath.
Click Unauthenticated Location under Security Countermeasure. The Option appears underneath.
Click Option, the option screen appears at the right pane. The Cat-3 Time check for Roaming Subscribers feature details are available on the screen.
Click Edit icon to modify the Option. The Edit Option page appears.
Ensure to set theBlocklist Refresh Timer Value and Blocklist Refresh Time Unit to the correct value.
Ensure to set the Cache Refresh Timer (milliseconds) to the milliseconds.

Check UDM Availability:

Verify that the FQDN or IP address of the UDR is reachable from the network. This can be checked through configuration details if verbosity is disabled.
Aditionally, check the ocsepp_time_unauthenticated_location_exception_failure exception metric.

Validate Velocity Configuration:

If a request is being rejected, it may be due to a mismatch in the travel time calculated by the Cat-3 Time check for Roaming Subscribers feature. This can happen if the velocity used for the calculation is not aligned with the expected measurement unit. Ensure that the configured velocity aligns with the unit of measurement for Average Flight Velocity, which should be set in kilometers per hour (km/h). This ensures that the calculations are consistent and that the request passes the validation.
To check velocity related configuration using CNC Console:
1. From the left navigation menu, navigate to SEPP and then click Security Countermeasure.
2. Click Cat-3 Time Location Check under Security Counter Measure, Unauthenticated Location page appears underneath.
3. Click Unauthenticated Location under Security Countermeasure. The Option appears underneath.
4. Click Option, the option screen appears at the right pane. The Cat-3 Time check for Roaming Subscribers feature details are available on the screen.
5. Click Edit icon to modify the Option. The Edit Option page appears.
6. Ensure that theAverage Flight Velocity (km/hr) is set in kilometers per hour (km/hr).
To check velocity related configurations using REST API, in the /sepp-configuration/v1/security-counter-measure/time-location-check/unauthenticated-location REST API,configure the parameter avgFlightVelocity in kilometers per hour (km/hr).

Ensure UDM Availability:

Check that the UDM is present and reachable in the network. The UDM discovery procedure must be successful for the feature to function correctly. This can be checked through configuration details if verbosity is disabled.
Aditionally, check the ocsepp_time_unauthenticated_location_exception_failure exception metric.

Check SUPI and SUCI in Incoming Requests:

Ensure that the incoming request includes a valid SUPI or SUCI.
If the incoming request contains a SUCI, the message will be forwarded to the UDM, which must be correctly discovered through NRF.
This can be checked using problem details if verbose is disabled or through logs.
Aditionally, check the ocsepp_time_unauthenticated_location_exception_failure exception metric.

Validate Country Specific Parameters:

Confirm that the country related parameters such as longitude, latitude, and MCC are correctly configured to ensure accurate location validation.

3.2.11 Cat-0 SBI Message Schema Validation Feature

The following are the troubleshooting scenarios for Cat-0 SBI Message Schema Validation feature:

Problem:

The incoming request gets rejected at CN32F and PN32F microservices.

Solution:

Check the logs or metrics (ocsepp_message_validation_on_body_failure and ocsepp_message_validation_on_header_failure) to find the request has failed for which resource URI and HTTP method, do the following:

If there is a request body failure, the following logs can be find by searching the text "Message validation failed for request body for request" :

{"instant":{"epochSecond":1680084693,"nanoOfSecond":192915132},"thread":"reactor-http-epoll-1","level":"ERROR","loggerName":"com.oracle.cgbu.cne.ocsepp.cn32f.handler.Cn32fSeppHandler","message":"OUT:
          Cn32fSeppHandler::Message validation failed for request body for request:
            /nausf-auth/v1/ue-authentications for method: POST","contextMap":{"ocLogId":"1680084693177_151_ocsepp-release-plmn-ingress-gateway-77c69f7bbc-2fxvg"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":16,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"23-03-29 10:11:33.192+0000","processId":"7","ocLogId":"1680084693177_151_ocsepp-release-shafali-plmn-ingress-gateway-77c69f7bbc-2fxvg"}

If there is a request query parameters failure, the following logs can be find by searching the text "Message validation failed for request query parameter(s) for request" :

{"instant":{"epochSecond":1678638067,"nanoOfSecond":537933800},"thread":"reactor-http-nio-4","level":"ERROR","loggerName":"com.oracle.cgbu.cne.ocsepp.cn32f.handler.Cn32fSeppHandler","message":"OUT:
        Cn32fSeppHandler:: Message  validation failed for
            request query parameter(s) for request:
            //nnssf-nsselection/v2/network-slice-information for method:
        GET","contextMap":{"ocLogId":"1678638061928_34_"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":22,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"23-03-12 21:51:07.537+0530","processId":"37136","ocLogId":"1678638061928_34_ocsepp-release-plmn-ingress-gateway-77c69f7bbc-2fxvg"}

From the left navigation menu, navigate to SEPP and then click Security Countermeasure. Click Cat 0 - SBI Message Schema Validation feature under Security Countermeasure, the Message Validation List appears underneath. Do the following:
1. Search for the problematic resource URI and can get the corresponding schema.
2. Compare the request body or request query parameter value(s) against the corresponding schema and ensure that either the request is complaint with its schema or existing schema needs updation.

If the user wants to know the detailed causes of message validation failures user can generate the debug logs, search for the configured error code and title or text "Error in Request Body" or "Error in Request Parameter(s)" and can get the following logs:

Request body failure case log:

 {"instant":{"epochSecond":1678392753,"nanoOfSecond":435438500},"thread":"reactor-http-nio-4","level":"DEBUG","loggerName":"com.oracle.cgbu.cne.ocsepp.webfluxlog.LoggingResponseDecorator","message":"LoggingResponseDecorator::getBody()  Response {\"title\":\"Message validation
        failed\",\"status\":406,\"detail\":\"Message
        validation for request /nausf-auth/v1/ue-authentications failed for  remote sepp set:
          RS\",\"instance\":\"/nausf-auth/v1/ue-authentications\",\"cause\":\"Error
          in Request Body\",\"invalidParams\":[\"requestBody.traceData.traceDepth:
        should be valid to any of the schemas  string\",\"requestBody.resynchronizationInfo.rand:
        does not match the  regex pattern ^[A-Fa-f0-9]{32}$\",\"requestBody.servingNetworkName:
        does  not match the regex pattern
        ^5G:mnc[0-9]{3}[.]mcc[0-9]{3}[.]3gppnetwork[.]org(:[A-F0-9]{11})?$\",\"requestBody.traceData.eventList:
        is missing but it is required\",\"requestBody.supiOrSuci: is missing  but it is
        required\"]}","contextMap":{"ocLogId":"1678392742101_34_"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":22,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"23-03-10 01:42:33.435+0530","processId":"23320","ocLogId":"1678392742101_34_"}

Request query parameter failure case log:

{"instant":{"epochSecond":1680685967,"nanoOfSecond":535845100},"thread":"reactor-http-nio-4","level":"DEBUG","loggerName":"com.oracle.cgbu.cne.ocsepp.webfluxlog.LoggingResponseDecorator","message":"LoggingResponseDecorator::getBody() Response {\"title\":\"Message validation
              failed\",\"status\":406,\"detail\":\"Message
              validation for request /nudm-sdm/v2/imsi-987654000000001 failed for  remote sepp set:
                RS\",\"instance\":\"/nudm-sdm/v2/imsi-987654000000001\",\"cause\":\"Error
                in Request Parameter(s)\",\"invalidParams\":[\"supported-features:
              does not match the regex pattern  ^[A-Fa-f0-9]*$\",\"parameters.dataset-names: is
              missing but it is  required\"]}","contextMap":{"ocLogId":"1680685963154_34_"},"endOfBatch":true,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":22,"threadPriority":5,"instanceType":"prod","vendor":"oracle","ts":"23-04-05 14:42:47.535+0530","processId":"13060","ocLogId":"1680685963154_34_"}

On the basis of failure reasons, the user can either correct the request body or request query parameter values or user can update the schema as mentioned in the step 2.

3.2.12 Configuration Failure in Remote SEPP and Remote SEPP Set

Problem:

Configuration operations (Add/ Delete/ Modify) failure in Remote SEPP and Remote SEPP Set, but user receives a 200 OK response code.

Solution:

The user must check the value of metrics ocsepp_configmgr_routefailure_total before and after the configuration operations (Edit/Add/Delete).
An increment in the counter indicates that the operation needs to be triggered again.

3.2.13 Aspen Service Mesh

Problem: SEPP Deployment fails in ASM mode.

Solution:

Check whether istio enabled flag set in namespace. If not, run the following command and deploy again:
```
kubectl label ns seppsvc istio-injection=enabled
```
Check PeerAuthentication is STRICT or PERMISSIVE . If it is set to STRICT, then change to PERMISSIVE and deploy again.
Run the following command to check the IP and host name in Service Entry for Kube-api-server:
```
kubectl get svc
```
Sample Output:
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
```

Check whether the SEPP is able to connect to Database. If cnDBTier is deployed in another namespace, create the DestinationRule(DR) as given below and deploy again:


apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: ocsepp-db-service-dr
  namespace: <ocsepp-namespace>
spec:
  exportTo:
  - "."
  host: <db-service-fqdn>.<db-namespace>.svc.<domain>
  trafficPolicy:
    tls:
      mode: DISABLE

Check whether the Service Account has all the Roles and RoleBindings permissions to access all the resources. If not, give all permissions as given below:
```
...
verbs:
- '*'
```

Problem: nrf-client-nfmanagement and nrf-client-nfdiscovery pods are visible unhealthy on OSO.

Solution:

The nrfClientCommonServicePort parameter must be updated with the value 9091.

The port parameter must be updated with the value 9091 in the startupProbe, readinessProbe, and livenessProbe under nrf-client-nfmanagement section.


        startupProbe:
          httpGet:
            path: /actuator/health
            port: 9091
          initialDelaySeconds: 60
          periodSeconds: 15
          timeoutSeconds: 10
          successThreshold: 1
          failureThreshold: 10
        readinessProbe:
          httpGet:
            path: /actuator/health
            port: 9091
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 10
          successThreshold: 1
          failureThreshold: 10
        livenessProbe:
          httpGet:
            path: /actuator/health
            port: 9091
          initialDelaySeconds: 15
          periodSeconds: 10
          timeoutSeconds: 10
          successThreshold: 1
          failureThreshold: 10

The port parameter must be updated with the value 9091 in the startupProbe, readinessProbe, and livenessProbe under nrf-client-nfdiscovery section.


        startupProbe:
          httpGet:
            path: /actuator/health
            port: 9091
          initialDelaySeconds: 60
          periodSeconds: 15
          timeoutSeconds: 10
          successThreshold: 1
          failureThreshold: 10
        readinessProbe:
          httpGet:
            path: /actuator/health
            port: 9091
          initialDelaySeconds: 10
          periodSeconds: 10
          timeoutSeconds: 10
          successThreshold: 1
          failureThreshold: 10
        livenessProbe:
          httpGet:
            path: /actuator/health
            port: 9091
          initialDelaySeconds: 15
          periodSeconds: 10
          timeoutSeconds: 10
          successThreshold: 1
          failureThreshold: 10

In the nrf-client-nfmanagement and nrf-client-nfdiscoverysection, add the value 9091 to the istioExcludePorts parameter.
```
istioExcludePorts: 53, 9091
```

3.2.14 Rate Limiting for Egress Roaming Signaling per PLMN feature

The following are the troubleshooting scenarios of Rate Limiting for Egress Roaming Signaling per PLMN feature:

Problem: Unable to see the discarded messages for the traffic sent.

Solution:

Check ocsepp_custom_values_<version>.yaml file for the following:
- In SEPP mode, check whether the egressRateLimiter.enabled parameter is set to true in the PLMN Ingress Gateway section of ocsepp_custom_values_<version>.yaml file.
- In Roaming Hub mode, check whether the egressRateLimiter.enabled parameter is set to true in the N32 Ingress Gateway section of ocsepp_custom_values_roaming_hub_<version>.yaml file.

CNC Console

In the CNC Console GUI, from the left navigation menu, navigate to SEPP and click Rate Limiting.
Select Egress Rate Limiting which is defined under Rate Limiting.
The Option and EgressRateLimitingList appears underneath.
Click Option. The option screen appears at the right pane. Check whether Egress Rate Limiting Enabled is true.
Check whether request's PLMN ID is present in any of the EgressRateLimitingList and Egress Rate Limiting Enabled is set totrue.

REST API

Check whether the egressRateLimitingEnabled is set to True using the REST APIs. For more details, see the Egress Rate Limiting Option Configuration and Egress Rate Limiting List Configuration REST APIs sections in the Cloud Native Core, Security Edge Protection Proxy REST Specification Guide.

Problem: Traffic is being forwarded even if tokens for the Egress Rate Limiting List are exhausted.

Solution:

The 3gpp-Sbi-Message-Priority header of the request must be verified before the message is dropped. If the priority in the header is less than (not equal to) Discard Message Priority property of the message in the Egress Rate Limiting List, then the message is not dropped.
If the 3gpp-Sbi-Message-Priority header is not present, then the priority is checked in the route configuration. If a value for 3gpp-Sbi-Message-Priority is present in the route configuration, then the above mentioned condition is considered and the same solution is applied.
If the priority is unknown for the request, 24 is considered as the default value for the request priority, then, the same condition as above is applied.

Problem: Status code is set to a different code in Error Configuration, but status code 429 is seen in rejected requests.

Solution: Check if status code set on CNC Console is a valid HTTP Status code or in the series of 3xx. By default, it should be 429.

Problem: The server header observed in response or logs.

Solution: The server header is observed in the response or logs, if the user cofigured error code is present in the Helm custom values. By default the status codes 400,404,408, and 429 are configured in Helm custom values.

3.2.15 Separate Port Configurations for N32c and N32f on the Egress Routes

The following are the troubleshooting scenarios for separate port configurations for n32c and n32f on the Egress routes feature:

Scenario: The Remote SEPP is changed with new fields of N32F configuration, and the traffic is not proper after changing the profile.

Solution:

Check whether the Remote Partner Set is created. If not, create the Remote Partner Set. For more information about API path, see Oracle Communications Cloud Native Core, Security Edge Protection Proxy REST Specification Guide.
Check if the configuration is stored inside the database correctly. Get the Remote SEPP profile using the following command and verify the configuration:
```
curl -X GET 'http://<ocsepp-release-name>-config-mgr-svc::9091/sepp-configuration/v1/remotesepp/<name>'
```
1. Check if N32fFQDN is correct; whether it is mapped to correct DNS entry and is reachable.
2. Check the N32fAddress, whether it is correctly mapped to the service. To confirm that the IP is mapped to correct service, run the following command:
```
kubectl get endpoints -n <namespace>
```
3. Check whether the IP is mapped correctly to the intended service.
4. Check if N32fPort is configured correctly.
Verify in the database that the routes at n32-egress-gateway are updated according to the new configuration done at Remote SEPP. Run the following command to get the routes created:
```
curl  -X GET 'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/peerconfiguration'#Sample output[{"id":"psepp1","apiPrefix":"","Host":"<n32f-fqdn/IP>", "port": "8888"}]
```
1. Check in the above output if the Host and port parameter are the N32f IP and FQDN and port respectively.
2. Run the following command to check that the ID is mapped correctly in the peer set:
```
curl  -X GET 'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/peersetconfiguration'#Sample output[{"id":"RSS-2","httpConfiguration":[{"priority":1,"peerIdentifier":"psepp1"}]}]
```
3. Run the following command to check in routes configuration that the peerset Id is mapped correctly in the peerSetIndentifier parameter:
```
curl  -X GET  'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/routesconfiguration'
```

3.2.16 Alternate Routing based on the DNS SRV Record for Home Network Functions

The following are the troubleshooting scenarios of alternate routing based on the DNS SRV Record for home network functions feature:

Problem: Virtual FQDNs are configured, but incoming request doesn't match any configured route.

Solution: Verify the routes and the matching criteria (URI and header) associated with each route. If the request is not matching any route, then the request will be routed via the configured default route.

Problem: The incoming requests are not routed according to the configuration defined at plmn-egress-gateway.

Solution:

Get all the routes by using GET API.

Example:

curl -X 'GET' \
  'http://<config-mgr-svc-ip>:<port>/sepp/nf-common-component/v1/egw/plmn/routesconfiguration'

Verify whether the Order id of the each route is configured correctly. Lower the order id, higher will be priority of routes.
User must reconfigure the routes by using REST APIs.

Problem:The incoming requests are not routed to the target FQDNs associated with the virtual FQDNs in the DNS service.

Solution:

You can run below commands for debugging:

Check if alternate-route-svc is up and running.
Use "dig" command to verify if virtual FQDN is resolvable. Example dig -t srv "virtualFqdn". This command should return the list of the target FQDNs associated with the virtual FQDN.

Problem: Configurational issues at plmn-egress-gateway.

Solution: If the user faces difficulty while updating DNS SRV records, the configuration must be cleared in the following order:

Routes Configuration
Peerset Configuration
Peer Configuration

The order for the configurtions must be as follows:

sbiroutingerrorcriteriasets
sbiroutingerroractionsets
Peer Configuration
Peerset Configuration
Routes Configuration

3.2.17 Load Sharing among Multiple Remote SEPP Nodes

The following are the troubleshooting scenarios of load sharing among multiple Remote SEPP nodes feature:

Problem: The Remote SEPP is changed with virtualHost, and the traffic is not working properly after changing the profile

Solution:

Check whether the Remote Partner Set is created. If not, create the Remote Partner Set. For more information about API path, see "Remote Partner Set" section of Oracle Communications Cloud Native Core, Security Edge Protection Proxy REST Specification Guide.
Check if the configuration is stored inside the database correctly. Get the Remote SEPP profile using the following command and verify the configuration:
```
curl -X GET 'http://<ocsepp-release-name>-config-mgr-svc::9091/sepp-configuration/v1/remotesepp/<name>'
```
1. Check if the virtualHost is correct; whether it is mapped to correct DNS entry and is reachable.
2. To verify that the virtual route is created at N32 egress gateway correctly, run the following command:
```
curl  -X GET 'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/peerconfiguration'
#Sample output[{"id":"psepp1","apiPrefix":"","virtualHost":"<virtualHost>"}]
```
3. Check the above output, and whether the virtualHost is mapped to the virtualHost configuration in the Remote SEPP.
4. Check if ID is configured correctly in the peer-set:
```
curl  -X GET 'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/peersetconfiguration'
#Sample output[{"id":"RSS-2","httpConfiguration":[{"priority":1,"peerIdentifier":"psepp1"}]}]
```
Verify that the peerset Id in routes configuration is mapped correctly in peerSetIndentifier parameter:
```
curl  -X GET  'http://<ocsepp-release-name>-config-mgr-svc:9090/sepp/nf-common-component/v1/egw/n32/routesconfiguration'
```

Problem: Check whether target host is mapped against virtual host correctly.

Solution:

Run the following curl from config mgr pod:

curl --noproxy "*"--http2-prior-knowledge -X GET -H 'Accept: application/json'-H "Content-Type: application/json"'http://<ocsepp-release-name>-alternate-route:80/lookup?fqdn=<virtualhost>g&scheme=http'

Sample output

[{"target":"ocsepp-release-adity-n32-ingress-gateway.sepp3.3gppnetwork.org","port":443,"ttl":60,"type":"SRV","dclass":"IN","priority":10,"weight":10000},{"target":"ocsepp-release-adity-n32-ingress-gateway.sepp2.3gppnetwork.org","port":443,"ttl":60,"type":"SRV","dclass":"IN","priority":10,"weight":10000}]bash-4.4$

3.2.18 5G SBI Message Mediation Support

For troubleshooting the mediation rules using Drools Rule Language (DRL) related scenarios, see "Error Messages for Mediation Rule Configuration" section in Cloud Native Core, Service Communication Proxy Troubleshooting Guide.

3.2.19 Support for TLS 1.3

The following are the troubleshooting scenarios of the feature:

Problem: Handshake is not established between SEPPs.

Solution:

Check whether both the Remote Partner profiles posted are correct.
Check the logs for N32 Ingress Gateway and N32 Egress gateway.

If the error logs have the SSL exception, do the following:

Check the TLS version of both SEPPs, if both support different and single TLS versions, (that is, SEPP1 supports TLS 1.2 only and SEPP2 supports TLS 1.3 only or vice versa), handshake fails. Ensure that the TLS version is same for both SEPPs or revert to default config for both SEPPs.
The TLS version communication supported are:

Table 3-1 TLS Version

Client TLS Version	Server TLS Version	Negotiated TLS Version
TLS1.2+1.3	TLS1.2+1.3	TLSv1.3
TLSv1.3	TLSv1.3	TLSv1.3
TLSv1.3	TLSv1.2+1.3	TLSv1.3
TLSv1.2+1.3	TLSv1.3	TLSv1.3
TLSv1.2	TLSv1.2+1.3	TLSv1.2
TLSv1.2+1.3	TLSv1.2	TLSv1.2

Check the cipher suites being supported by both SEPPs, it should be either the same or should have common cipher suites present. If not, revert to default configuration.

Problem: Pods not coming up after populating the clientDisabledExtension or serverDisabledExtension parameter.

Solution:

Check the values given in the Helm parameters. The values listed cannot be added in these parameters:
- supported_versions
- key_share
- supported_groups
- signature_algorithms
- pre_shared_key

If any of the above values is present, remove them or revert to default configuration for the pod to come up.

Problem: Pods not coming up after populating clientSignatureSchemes parameter.

Solution:

Check the values given in the Helm parameters.
Value listed below should not be removed from these parameters:
- rsa_pkcs1_sha512
- rsa_pkcs1_sha384
- rsa_pkcs1_sha256
If any of the above values is not present, add them or revert to default configuration for the pod to come up.

3.2.20 SEPP Deployment on OCI

The following is the troubleshooting scenario of SEPP Deployment on OCI:

Title: SEPP OCI metric dashboard does not display data.

Problem: The message "Query cannot result in more than 2000 streams" is displayed on the dashboard panel, instead of displaying the data.

Solution: The customer must add label filters to the query of the panels, which are giving the error.

Example of the default query:

oc_ingressgateway_http_requests_total[1m]{k8namespace="cndb-test2"}.sum()

Example of the query with the label filter:

oc_ingressgateway_http_requests_total[1m]{app = "plmn-ingress-gateway", k8namespace="cndb-test2"}.sum()

Here, the label filter app = "plmn-ingress-gateway" is added to the query.

3.2.21 Georedundancy Support

The following are the troubleshooting scenarios for the Georedundancy Support feature:

Problem: One of the cnDBTier site data is not reflected on other sites.

Solution:

In the CNC Console GUI, from the left navigation menu, navigate to SEPP and then click Georeplication Status.
If the Replication Status is Down, then user need to perform Recovering a Failed Site procedure.
For more information on how to perform Recovering a Failed Site procedure, see "Recovering a Failed Site" section in Oracle Communications Cloud Native Core, Security Edge Protection Proxy User Guide.

Problem: Traffic is failing at C SEPP.

Solution:

Verify the following on the Grafana dashboard:

If the error is coming from alternate-route service, then check DNS configuration.
If the error is coming on n32-egress-gateway service, then verify routing configuration in Remote SEPP Set.

Problem: Traffic is failed on one of the producer SEPP instances even with equal weights and priorities.

Solution:

Verify n32-ingress-gateway pod of P SEPP is up and running.
Verify n32-egress-gateway logs at C SEPP and n32-ingress-gateway logs at P SEPP to identity the reason for call drop.

Problem: DNS SRV configuration is not reflecting.

Solution:

Verify the DNS settings and run service restart.

Restart the pods on C SEPP:


<release>-alternate-route
<release>-n32-egress-gateway

Problem: cnDBTier health APIs are not working.

Solution:

Check the SEPP and cnDBTier compatibility in SEPP User Guide. If the health APIs are supported from cnDBTier 24.1.x onwards. For the earlier versions of cnDBTier, the health APIs were not supported.

For more details, see 'Support for cnDBTier APIs in CNC Console' section of Oracle Communications Cloud Native Core, Security Edge Protection Proxy User Guide.

Problem: NDB health status Screen is blank.

Solution: Enable the following parameters in cnDBTier yaml file and run the helm upgrade.

pvchealth:

      enable:

        all: true

        mgm: true

        ndb: true

        api: true

3.2.22 Support for Originating Network Id Header Validation, Insertion, and Transposition

The following are the troubleshooting scenarios of the feature:

Problem: The incoming SBI request does not have any of the headers 3gpp-Sbi-Originating-Network-Id or 3gpp-Sbi-Asserted-Plmn-Id and the headers is not added by the feature (applicate only in SEPP mode).

Solution:

Check whether the feature is enabled or not global and remotely.
Check whether the metric ocsepp_originating_id_header_added_total is raised. The metric is for a successful header addition with useful information like the added header name, its value, the remote sepp, and the request URI.
Check whether the metrics ocsepp_originating_header_addition_failed_total (at CSEPP) and ocsepp_originating_header_add_or_transpose_failed (at PSEPP) are raised. The metrics are raised with the incoming request path, if the addition of missing header is failed due to any internal unforeseen error.
Check for the error log: "Error while adding missing originating network id header at consumer sepp" at C SEPP and log message: "Error while adding/transposing missing originating network id header at producer sepp" at P SEPP. The log is printed with the incoming request path, if the addition of missing header is failed due to any internal unforeseen error.

Problem: The header transposition is not working at P SEPP.

Solution:

Check whether the feature is enabled or not global and remotely.
Check whether the metric ocsepp_originating_id_header_transposed_total is raised. The metric is for a successful header addition with useful information like the added header name, its value, the remote sepp, and the request URI.
Check whether the metric ocsepp_originating_header_add_or_transpose_failed_total is raised. The metric is raised with the incoming request path at PSEPP in case header transposition is failed due to any internal unforeseen error.
Check for the error log: "Error while adding/transposing missing originating network id header at producer sepp" at P SEPP. The log is printed with the incoming request path, if the header transposition is failed due to any internal unforeseen error.

3.2.23 Proactive status updates on SEPP

The following are the troubleshooting scenarios of Proactive status updates on SEPP feature:

Problem: Peer marked as unhealthy or oc_egressgateway_peer_health_status is 1 even when peer is up or metric is not pegged.

Solution:

Ensure the Remote Peer is UP and running.
Ensure the flag seppPeerHealthCheck is set to "true" in ocsepp_custom_values_<version>.yaml file. If not, change it to true and run the helm upgrade.
Ensure in the Remote SEPP profile thehealthApiPath and healthApiMethod parameters are present. If not, follow the steps to enable the feature on the CNC Console console.
Check the dimension statusCode of metric oc_egressgateway_peer_health_ping_response_total. If the code is expected code from peer and not present in seppPeerHealthCheckCodes custom values, then add in the custom values and run helm upgrade.
Ensure in the API Peer Configuration, the enable flag is set to "false". If not, change to false by running the REST API.

Problem: Peer is not sending configured response to health API requests.

Solution:

Ensure microservice is up and running
Ensure in the ocsepp_custom_values_<version>.yaml file the flag healthCheckMonitoring enabled is "true". If not, change to true and run Helm upgrade.
Check on the CNC Console to see if the feature is enabled.
Check the expected request Method and URI are same as configured on CNC Console GUI.
Check the configured response code on CNC Console.

3.2.24 Multiple SEPP instances on Shared cnDBTier Cluster

The following are the trouble scenarios of multiple SEPP instances on shared cnDBTier cluster feature:

Problem: After the cnDBTier upgrade, if the SEPP pods are stuck in an "Unready" state and the logs show a database connection failure.

Solution:

For cnDBTier release 23.4.0 and 24.2.0, the user should check the plugin using the following query:
```
SELECT user, host, plugin FROM mysql.user;
```
If the plugin value is "mysql_native_password," use an ALTER query to change the plugin from mysql_native_password to caching_sha2_password, and then proceed with the upgrade.
```
ALTER USER 'seppuser1'@'%' IDENTIFIED WITH caching_sha2_password BY 'NextGenCne1';
```

Problem: If a geo-replication failure occurs and disaster recovery is needed due to a fatal error, the unhealthy site will be reinstalled. Then the SEPP instances associated with the restored site are unable to connect to cnDBTier and are showing an access restriction message.

{"instant":{"epochSecond":1724681253,"nanoOfSecond":107976541},"thread":"main","level":"ERROR","loggerName":"com.zaxxer.hikari.pool.HikariPool","message":"HikariPool-1 - Exception during pool initialization.","thrown":{"message":"Access denied for user 'seppuser1'@'aclmx0466-ilom.us.oracle.com' (using password: YES)"

Solution:

Create the necessary NF-specific user accounts and grants to match the NF users and grants of the working site in the reinstalled cnDBTier cluster, if those user accounts do not already exist. When disaster recovery is performed due to a fatal error, only the SEPP databases are replicated on the restored cluster.

Note:

NF-specific user accounts and grants must be created manually because they are not replicated.

Problem: NF-specific roles are not visible in the CNC Console GUI.

Solution:

Ensure the flag instanceLevelAuthorizationEnabled is set to "true" in cncc_custom_values_<version>.yaml file and run the Helm upgrade.

Problem: The roles that are assigned roles are not applied to the user.

Solution:

Follow the correct combination of NF-specific roles as outlined in the Oracle Communications Cloud Native Configuration Console User Guide.

3.2.25 Cat-1 NRF Service API Query Parameters Validation Feature

The following are the troubleshooting scenarios of Cat-1 NRF service API query parameters validation feature:

Problem:

The Ingress Request message is rejected and shows the error code (406 - default error code) with the message "Query Param Validation failed."

Solution:

Enable Required Features:
- The Cat-1 Service API Validation feature must be enabled for the Cat-1 Service API Query Parameters Validation feature to function accurately.
- If the request fails, make sure both the Cat-1 Service API Validation feature and the Cat-1 Service API Query Parameters Validation feature are enabled.
List Name Configuration:
- Check that the list name set at the RSS level matches the list for query parameter validation.
- Ensure the same list name is selected across all lists configured for query parameter validation.
NF Pair Validation:
- Verify the pair of Network Functions (NFs) in the request. Make sure they match the configuration set of values for this feature.
Correct API Request Format:
- The request must be a NRF discovery request in the following format:
```
nnrf-disc/v1/nf-instances?requester-nf-type={}&target-nf-type={}
```
- The requester-nf-type={} and target-nf-type={} should have the configured values for NFs.
- Use only the GET method with this API, as the feature supports this method only.
- Ensure both requester-nf-type and target-nf-type are included in the request.

Here are the detailed debugging steps to follow:

Feature Configurations:

Verify CNC Console Configurations for Cat-1 Service API Query Parameters Validation feature:
- In the CNC Console GUI, from the left navigation menu, navigate to SEPP and then click Security Countermeasure.
- Click Cat 1 -Service API Validation under Security Countermeasure. Option, Service API Allowed List , and Cat-1 Query Parameter Validation List appears.
- Click Option, the Options page appears on the right pane.
- Ensure that Enable Cat 1-Query Parameter Validation parameter and Enable Cat 1 - Service API Validation are set to True to enable the feature.
- Add any additional configurations in the Cat 1 - Service API Query Param Validation List page.
Verify Configurations via REST API for Cat-1 Service API Query Parameters Validation feature:
- Use the following API to configure the "queryParamValidationEnabled" parameter to 'true':
```
/sepp-configuration/v1/security-counter-measure/feature
```
- Use this API to configure other mandatory parameters for the feature:
```
/sepp-configuration/v1/security-counter-measure/service-api-query-param-validation-list
```

Check the following metrics and alerts for error scenario analysis:

Successful Requests:
To analyze all successful requests for the feature, check the metric:
ocsepp_security_service_api_query_param_validation_success_total
Failure Requests:
To analyze all failed requests for the feature, check the metric:

ocsepp_security_service_api_query_param_validation_failure_total
To analyze the error scenario, check the following alerts:
SEPPN32fServiceApiQueryParamValidationFailureAlertWarn

SEPPN32fServiceApiQueryParamValidationFailureAlertMinor

SEPPN32fServiceApiQueryParamValidationFailureAlertMajor

SEPPN32fServiceApiQueryParamValidationFailureAlertCritical

Problem:

Unable to save configurations for the Cat-1 Service API Query Parameters Validation successfully.

Solution:

NF Configuration:
- Ensure that one of the following eight Network Functions (NFs) is configured: NRF, UDM, AMF, SMF, AUSF, PCF, SEPP, and SCP.
Correct NF Type Names and Values:
- Verify that the requester-nf-type and target-nf-type are correctly named and assigned valid values.
Validation List Configuration:
- Confirm that the CAT1 Service API Validation screen already contains the list you intend to configure for the SCM CAT1 Query Parameter Validation feature.
Mandatory Parameters:
- Ensure that all required parameters are configured, including:
  - Name and value of requester-nf-type and target-nf-type
  - Resource URI
  - Method
  - List name
- The request must be an NRF discovery request in the following format:
```
/nnrf-disc/v1/nf-instances?requester-nf-type={}&target-nf-type={}
```
- Only the GET method should be used with this API, as this feature supports only the GET method.
Avoid Redundant Entries:

Duplicate entries for the name and value of requester-nf-type and target-nf-type are not allowed in the configuration.

3.2.26 Integrating SEPP with 5G Network Intelligence Fabric (5GNIF) feature

The following are the troubleshooting scenarios of integrating SEPP with 5G Network Intelligence Fabric (5GNIF) feature:

Problem: 5GNIF Discovery Request Not Sent to NRF.

Solution:

Verify feature enablement at the Helm level:
- Check whether the 5GNIF feature is enabled in the Helm configuration:
  - Open the ocsepp_custom_values_<version>.yaml file.
  - Navigate to the nif section of config-mgr-svc.
  - Confirm that the parameter enableNif is set to true.
You can also validate this directly from the deployment using:
```
kubectl describe deploy <release-name>-config-mgr-svc -n <namespace> | grep ENABLE_NIF
```
- Ensure the output shows ENABLE_NIF is set to true.
- If not, update the value in the custom values yaml file and perform a Helm upgrade to apply the change.
Confirm feature enablement through REST API:
Use the following curl command to check if 5GNIF is enabled in the runtime configuration:
```
curl 'http://<config-mgr-svc>:<port>/sepp-configuration/v1/nif/options' -X GET
```
- The returned JSON should show "enabled": true.
If this is not the case, refer to the relevant configuration section for corrective action.
Check discovery delay settings:
Make sure the discovery process is not being delayed due to configuration:
- Verify the values of nifDiscoveryInitialDelay and nifDiscoveryScheduledDelay.
- The first discovery request will only be triggered after the configured nifDiscoveryInitialDelay.
Verify HTTP enablement for NIF:
Ensure that HTTP is enabled for NIF discovery:
- Run the following command:
```
kubectl describe deploy <release-name>-config-mgr-svc -n <namespace> | grep HTTP_ENABLED_NIF
```
- Check that httpEnabledNif is set to true in the ocsepp_custom_values_<version>.yaml file.
  If not, update the value and perform a Helm upgrade to apply the change.

Problem: No Peers Visible on PLMN Egress Gateway (CNCC Screen / REST API) Even After 5GNIF is Registered on NRF.

Solution:

If the PLMN Egress Gateway does not show any NIF peers, even though 5GNIF is successfully registered on the NRF, follow these steps to troubleshoot and resolve the issue:

Verify That the 5GNIF feature is enabled: Ensure the feature is enabled in both the Helm deployment and runtime configuration, as outlined in Scenario 1.
Check the 5GNIF Discovery Name Configuration:
Confirm that the nifDiscoveryName in config-mgr-svc matches the nfType set in the NFProfile:
- Run the following command:
```
kubectl describe deploy <release-name>-config-mgr-svc -n <namespace> | grep NIF_DISCOVERY_NAME
```
- If the discovery name differs from the nfType in the NFProfile, update the value in the ocsepp_custom_values_<version>.yaml file:
```
config-mgr-svc:
  nif:
    nifDiscoveryName: <correct_nfType_value>
```
- Perform a Helm upgrade to apply the changes.
Validate PLMN Egress Gateway Configuration for header handling:
Ensure the PLMN Egress Gateway is correctly configured to accept requests where certain headers are absent:
- Check the configMap for the following parameter: sepp.headerAbsentPredicate: true
- If it’s set to false or missing, update ocsepp_custom_values_<version>.yaml under the plmn-egress-gateway section:
```
plmn-egress-gateway:
  sepp:
    headerAbsentPredicate: true
```
- Perform a Helm upgrade to apply the configuration.
Verify NRF Client Configuration Parameters.
In the nrfclient section of the configuration, verify the following parameters are set correctly:
- primaryNrfApiRoot
- enableVirtualNrfResolution
- virtualNrfFqdn
- virtualNrfScheme
Also, ensure DNS entries are properly configured and resolvable for the NRF FQDN.
Check the NRF Route Format in Egress Gateway:
Ensure the correct nrf_route is created with the appropriate structure. It should look like the following:
```
[
  {
    "id": "nrf_route",
    "uri": "egress://request.uri",
    "order": 1,
    "filters": [
      {
        "args": null,
        "name": "DefaultRouteRetry"
      }
    ],
    "predicates": [
      {
        "args": {
          "pattern": "/nnrf-*/**"
        },
        "name": "Path"
      },
      {
        "args": {
          "headerName": "oc-xfcc-dns"
        },
        "name": "HeaderAbsent"
      }
    ]
  }
]
```
- Ensure that the route ID, URI, filters, and predicates match exactly.
- Missing or misconfigured routes can prevent proper peer registration.

Problem: 500 Internal Server Error When Sending Traffic.

Solution:

Check routes Configuration:
If the nifReject route is present, it means all NIF peers have been removed.
Verify Peer Configuration: If the peer list is empty, do the following:
- Ensure all 5GNIF instances are in the REGISTERED state in the NRF. Re-register if needed.
- If the feature was recently disabled and re-enabled, wait for the configured nifDiscoveryInitialDelay, then recheck the peer list.

Problem: SBI Request Returning 404 (N32F Context Not Found).

Solution:

- Run the following command:
```
kubectl describe deploy <release-name>-config-mgr-svc -n <namespace> | grep SAN_HEADER_NAME
```
- Ensure SAN_HEADER_NAME is set to oc-xfcc-dns. If not, update the ocsepp_custom_values_<version>.yaml:
```
configs:
  sanHeaderName: "oc-xfcc-dns"
```
- Perform a Helm upgrade after making the change.
Check Header Predicate in PLMN Egress Gateway
- In the configMap, verify:
- If it's missing or false, update the ocsepp_custom_values_<version>.yaml under plmn-egress-gateway:
```
plmn-egress-gateway:
  sepp:
    headerAbsentPredicate: true
```
- Perform a Helm upgrade.
Verify Feature Configuration (SoR):

Ensure Steering of Roaming (SoR) is not enabled.
If SoR was previously enabled, then go to CNCC > Configurations > Gateways > EGW > PLMN Egress Gateway > Routes Configuration, and Remove all SoR routes and Disable SoR.

Problem: Alternate Routing Issues.

Solution:

Check Current Routing Criteria and Actions:

Run the following curl commands to inspect current error handling configurations:

curl http://127.0.0.1:9090/sepp/nf-common-component/v1/egw/plmn/sbiroutingerrorcriteriasets

curl http://127.0.0.1:9090/sepp/nf-common-component/v1/egw/plmn/sbiroutingerroractionsets

Validate Server Header:
Ensure the server header in the response matches the expected pattern defined in sbiroutingerrorcriteriasets (Exmple: matches .*NIF.*).

Check Response Codes

If the response code is not one of 500, 503, or 504, you’ll need to update the configuration to include the response code that caused the issue.
In the ocsepp_custom_values_<version>.yaml, update the following under nif.nifRoutingErrorCriteriaSets:

[{
  "id": "nif_criteria_1",
  "method": ["GET", "POST", "PUT", "DELETE", "PATCH"],
  "response": {
    "statuses": [
      {"status": [500, 503, 504, 502], "statusSeries": "5xx"}
    ],
    "headersMatchingScript": "headerCheck,server,via,.*NIF.*"
  }
},
{
  "id": "nif_criteria_0",
  "method": ["GET", "POST", "PUT", "DELETE", "PATCH"],
  "exceptions": [
    "java.util.concurrent.TimeoutException",
    "java.net.SocketException",
    "java.net.SocketTimeoutException",
    "java.net.UnknownHostException",
    "java.net.ConnectException",
    "java.net.NoRouteToHostException"
  ]
}]

[{
  "id": "nif_criteria_1",
  "method": ["GET", "POST", "PUT", "DELETE", "PATCH"],
  "response": {
    "statuses": [
      {"status": [400, 404], "statusSeries": "4xx"},
      {"status": [500, 503, 504], "statusSeries": "5xx"}
    ],
    "headersMatchingScript": "headerCheck,server,via,.*NIF.*"
  }
},
{
  "id": "nif_criteria_0",
  "method": ["GET", "POST", "PUT", "DELETE", "PATCH"],
  "exceptions": [
    "java.util.concurrent.TimeoutException",
    "java.net.SocketException",
    "java.net.SocketTimeoutException",
    "java.net.UnknownHostException",
    "java.net.ConnectException",
    "java.net.NoRouteToHostException"
  ]
}]

Validate EnvoyFilter:

Check that the EnvoyFilter responsible for handling headers is correctly applied:

kubectl get envoyfilter -n <namespace>


NAME                 AGE
serverheaderfilter   22h

Run the following command:

kubectl describe envoyfilter serverheaderfilter -n <namespace>

Sample output:

kubectl describe envoyfilter serverheaderfilter -nsepp-1
Name:         serverheaderfilter
Namespace:    sepp-1
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: servicemesh
              meta.helm.sh/release-namespace: sepp-1
API Version:  networking.istio.io/v1alpha3
Kind:         EnvoyFilter
Metadata:
  Creation Timestamp:  2025-07-23T12:11:15Z
  Generation:          1
  Resource Version:    306427514
  UID:                 b4bb357d-a8ac-48e7-937d-629ba10ebfc5
Spec:
  Config Patches:
    Apply To:  NETWORK_FILTER
    Match:
      Listener:
        Filter Chain:
          Filter:
            Name:  envoy.filters.network.http_connection_manager
    Patch:
      Operation:  MERGE
      Value:
        typed_config:
          @type:                         type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          server_header_transformation:  PASS_THROUGH
  Workload Selector:
    Labels:
      app.kubernetes.io/part-of:  ocsepp

3.2.27 LCI and OCI Header Support Feature

The following are the troubleshooting scenarios of LCI and OCI header support feature:

Problem: Unable to see LCI header in the response when the feature is enabled

Solution:

Confirm that the feature is enabled in the Helm configuration.
In the perf-info section, verify that the parameter configmapPerformance.prometheus is properly configured to match the Prometheus service deployed in the cluster. Ensure the perf-info service can report CPU load for the cn32f-svc and pn32f-svc microservices.
Ensure that any consumer NF Identity (OAuth token, User-Agent, or Via header) is included in the request.
Check the localLciHeaderValidity: if multiple requests occur within the validity period without breaching configured thresholds, the SEPP will not add LCI headers.

Problem: Unable to see OCI header in the response when the feature is enabled

Solution:

Confirm that the feature is enabled in the Helm configuration.
In the perf-info section, verify that the parameter configmapPerformance.prometheus is properly configured according to the Prometheus service deployed in the cluster. Confirm that the perf-info service reports CPU load for the cn32f-svc and pn32f-svc microservices.
Ensure that any consumer NF Identity (OAuth token, User-Agent, or Via header) is included in the request.
Verify that overloadConfigRange is correctly set in Helm and that the SEPP is in an overloaded state that triggers OCI header generation.
Check the validity period: if multiple requests happen within this period without breaching the configured thresholds, the SEPP will not add OCI headers.

3.3 HTTP Response Codes and Error Codes

The following are the HTTP Response Codes:

Table 3-2 HTTP Response Codes and Error Codes

Data Type	Mandat ory(M)/Optional(O)/Conditi onal(C )	Cardinality	Response Code	Description
ProblemDetails	C	1	400 - BAD REQUEST	SEPP config-mgr-svc shall send the response when the Request body received is not as per defined Data Model.
ProblemDetails	C	1	500 - INTERNAL SERVER ERROR	SEPP config-mgr-svc shall send the response when an internal error has occurred.
Response Body as per Data Model defined	C	1	200 - OK	SEPP config-mgr-svc shall send the response when the request is successful.
ProblemDetails	C	1	404 -NOT FOUND	SEPP config-mgr-svc shall send the response when the requested entry is not present in the database.

Error Codes and Recovery Steps

The following tables list the various SEPP error codes and the recovery steps:

Table 3-3 Error Codes and Recovery Steps

Error Code	Error Text	Command/Method	Description	Recovery Steps
SEPP-COM-DB-ERROR-0002	Remote Sepp record not found	GET remotesepp/{name} DELETE remotesepp/{name} PATCH remotesepp/{name}	This error is observed if the provided name in the request is not present.	Verify that Remote SEPP name given in request parameter is present.
SEPP-COM-DB-ERROR-0003	N32F Context not found record not found	GET handshakestatus/fqdn/{fqdn} GET handshakestatus/name/{name}	This error is observed if the context with given name or fqdn in the request parameter is not found.	Verify that context with given name or fqdn in request parameter is present in DB.
SEPP-COM-DB-ERROR-0005	Database connection is down	Can be thrown from any Method (Generic Exception if application is unable to connect with Database)	This error is observed if the application is unable to make the connection with Database.	Verify that DB is up and running.
SEPP-COM-DB-ERROR-0007	SQL Grammer exception	PUT remoteseppset/{name}	This error is observed if there is some corruption in the Database related to Remote SEPP Set table (This error gets generated for all the commands that is trying to access corrupted table or Database).	Recheck the corruption in database and re-install SEPP
SEPP-COM-DB-ERROR-0008	Constraint violation exception of database table columns	PUT {remoteSepp}	This error is observed when one of the mandatory parameters required for remote SEPP are not present.	Verify that mandatory parameter like name, seppfqdn are present.
SEPP-COM-DB-ERROR-0009	Unsupported security capability list exception	POST /remotesepp PUT remotesepp/{name} PATCH remotesepp/{name}	This error is observed if unsupported securityCapabilityList is provided in request.	Make sure securityCapabilityList provided is supported by SEPP. Allowed security capability list is 'TLS' and 'TLS and PRINS.
SEPP-COM-DB-ERROR-0010	Update not allowed on table entry exception	PUT remotesepp/{name} PATCH remotesepp/{name}	This error is observed if given parameter to update is same as configured one or trying to update mandatory parameter.	Verify that the parameter provided in the request to update is different from the configured one or not while updating any mandatory parameter like name, seppfqdn etc.
SEPP-COM-DB-ERROR-0011	Remote Sepp already present	POST /remotesepp	This error is observed if Remote SEPP with same seppfqdn is already present in DB.	Verify that seppfqdn provided in configuration request is not already present in DB.
SEPP-COM-DB-ERROR-0012	Mandatory Parameter Update Not Allowed	PUT remotesepp/{name} PATCH remotesepp/{name}	This error is observed if the user is trying to update the value of mandatory parameter which is not allowed.	Verify that some of the mandatory parameters are not allowed to be updated. Those allowed should match with the value in the request.
SEPP-COM-ERROR-0013	Invalid PLMN List in Request	POST /remotesepp PUT /remotesepp	This error is observed if the user has configured PLMN in incorrect format in PLMNID List.	Verify the entered PLMN (mcc and mnc) in PLMNID List while configuring Remote Sepp.
SEPP-COM-DB-ERROR-0020	Remote Sepp Set not found	GET RemoteSeppSet/{name}	This error is observed if the Remote Sepp Set is not present	Give the Remote SEPP Set name that exist in the database
SEPP-COM-DB-ERROR-0021	Remote Sepp Set associated with Remote SEPP	DELETE RemoteSepp/{name} PUT RemoteSepp/{name} PATCH RemoteSepp/{name}	This error is observed if Remote SEPP is associated with Remote Sepp Set, it gives this error on deletion.	Disassociate the Remote Sepp from Remote SEPP Set by executing the Remote SEPP Set PUT command. Or Do the following: Delete the Remote Sepp Set Delete the Remote Sepp Create Remote Sepp Set again
SEPP-COM-DB-ERROR-0022	Remote Sepp Set already exists	POST RemoteSeppSet/{name}	This error is observed if Remote Sepp Set already exists and same entry is added again.	Use a unique name in Remote Sepp Set
SEPP-COM-DB-ERROR-0023	Remote Sepp Set configuration error	PUT RemoteSeppSet/{name} POST RemoteSeppSet/{name}	Remote Sepp Set does not exists Requested Remote Sepp does not exists At least one Remote Sepp should be associated with Remote Sepp Set Configured Domains are different between peers Remote Sepp Set exists with same domain Configured PLMNs are different between peers Remote Sepp Set exists with same PLMN Configured PLMNs/Domains are different between peers Associating Remote Sepp should be unique for Remote Sepp Set	Give the Remote SEPP Set name that exists in the database Give the Remote SEPP name that exists in the database Give at least a single Remote Sepp name while creating a Remote Sepp Set Check that each Remote Sepp has same Domain when creating a Remote Sepp Set Check that each Remote Sepp has same PLMN when creating a Remote Sepp Set Check that each Remote Sepp name is unique while creating a Remote Sepp Set
SEPP-COM-xx-ERROR-0101	Config Not Acceptable	PUT POST PATCH /v1/remotesepp/	This error is observed if PLMNIdList is empty or PLMNIdList size is greater than max size allowed or if domain is null.	Verify that PLMNIdList is not empty or PLMNIdList size is not greater than max size allowed or if domain is not null.
SEPP-COM-xx-ERROR-0102	Mandatory Parameter Missing	POST /remotesepp POST /remoteseppset	This error is observed if mandatory parameter is missing in request.	Verify that all mandatory parameter for configuration of Remote SEPP or Remote SEPP Set is present
SEPP-COM-xx-ERROR-0103	Connection could not be established on N32c interface	POST /remotesepp DELETE remotesepp/{name} PUT remotesepp/{name} PATCH remotesepp/{name}	This error is observed if n32c service is down or not up and running.	Verify that n32c service is up and running
SEPP-COM-xx-ERROR-0104	Invalid Value for Parameter		This Error occurs when user enters the invalid value for Enum Field in SEPP.
SEPP-COM-SVR-ERROR-0404	Unable to connect to EGW to sync config	PUT peerconfiguration/peersetconfiguration /routesconfiguration	This error is observed if config mgr is not able to update peer/peerset/routes configurations at EGW.	Verify that common configuration server is up and running.
SEPP-CN32FSEPP-ERROR-0013	PLMN ID Validation In Header Failed	NA	This error is observed if PLMN ID is not matched in header of the incoming request on CN32F microservice.	Verify if correct regex is configured against the header identifier in Header Configuration. Verify if MCC & MNC combination is present in the Remote PLMN ID List.
SEPP-CN32FSEPP-ERROR-0014	PLMN ID Validation In Body Failed	NA	This error is observed if PLMN ID is not matched in body of the incoming request on CN32F microservice.	Verify if correct regex is configured against the body IE in Body IE Configuration. Verify f MCC & MNC combination is present in the Remote PLMN ID List.
SEPP-PN32FSEPP-ERROR-0016	PLMN ID Validation In Header Failed	NA	This error is observed if PLMN ID is not matched in header of the incoming request on PN32F microservice.	Verify if correct regex is configured against the header identifier in Header Configuration. Verify if MCC & MNC combination is present in the helm based local PLMN ID list.
SEPP-PN32FSEPP-ERROR-0017	PLMN ID Validation In Body Failed	NA	This error is observed if PLMN ID is not matched in body of the incoming request on PN32F microservice.	Verify if correct regex is configured against the body IE in Body IE Configuration. Verify if MCC & MNC combination is present in the helm based local PLMN ID list.
SEPP-SECURITY-PLMN-HEADER-ERROR-0015	PLMN ID Validation In Header Failed	NA	Verify if correct regex is configured against the header in header Configuration. Verify if MCC & MNC combination is present in the helm based local PLMN ID list.	if PLMN ID is not matched in the header of the incoming request. Metrics can be checked for the details for which it has failed.
SEPP-SECURITY-PLMN-BODY-ERROR-0016	PLMN ID Validation in body failed	NA	Verify if correct regex is configured against the body IE in Body IE Configuration. Verify if MCC and MNC combination is present in the helm based local PLMN ID list.	if PLMN ID is not matched in the body of the incoming request. Metrics can be checked for the details for which it has failed.

The following are the error codes of Mediation feature:

Table 3-4 Mediation Error Codes and Recovery Steps

Error Code	Error Text	Command/ Method	Description	Recovery Steps
SEPP-MEDIATION-ERROR-001	Mediation Trigger Rule Not Found.	GET x`/sepp-mediation-trigger-rule-list/{triggerRuleListName} DELETE /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Requested Trigger Rule List does not exist.	Give the Trigger Rule List Name that exist in Database.
SEPP-MEDIATION-ERROR-002	Unsupported Trigger Points	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Trigger Points provided in configuration request body is not supported.	Provide valid Trigger Points in Configuration Request. Valid Trigger Points : N32_Egress_Request N32_Ingress_Response N32_Ingress_Request N32_Egress_Response
SEPP-MEDIATION-ERROR-003	Mediation Trigger Mandatory Parameter Update Not Allowed.	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Trigger Rule List mandatory parameter like TriggerRuleListName can not be updated.	Make sure TriggerRuleListName you are providing in configuration request url is same as name in request url path.
SEPP-MEDIATION-ERROR-004	Trigger Rule is Mandatory Parameter.	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	If MediationAllEnabled is false and there is no TriggerRules or empty TriggerRules in request.	If MediationAllEnabled is false, then make sure there is TriggerRules Provided in Request.
SEPP-MEDIATION-ERROR-005	Invalid Error Status Code	PUT /mediation/feature	Invalid Http Status Code is provided is Error Configuration Request.	Make sure to provide valid HTTP Status Code in statusCode field in Error Configuration Request
SEPP-MEDIATION-ERROR-006	Invalid ResourceURI and HTTPMethod Error	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	ResourceURI and HttpMethod provided in Request is not valid.	Make sure to provide ResourceURI and HttpMethod combination that is configuration for SEPP, already present in Database.
SEPP-MEDIATION-ERROR-007	DELETE Not Allowed	DELETE /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Trigger Rule List Delete Not Allowed.	Ensure you are deleting only that Trigger Rule List that is not associated with Remote SEPP Set.
SEPP-MEDIATION-ERROR-008	Trigger Rule Configuration Error	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Error in Configuration of Trigger Rule List.	Ensure there is no duplicated ResourceURI and Method in request body.
SEPP-MEDIATION-ERROR-009	Mediation Trigger Rules Configuration Mandatory Parameter Missing Error	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Mediation Trigger Rules Configuration Mandatory Parameter Missing in configuration request.	Ensure all mandatory parameters are present in Mediation Trigger Rule Configuration request.
SEPP-MEDIATION-ERROR-010	Multiple Local Trigger Rule List Configuration Error	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	Multiple Local Trigger Rule List Configuration is Not Allowed.	Make sure we are not configuring another Local Trigger Rule List if there is already one configured in DB. Only one Local Trigger Rule List can be configured for SEPP.
SEPP-MEDIATION-ERROR-011	Mediation Feature Mandatory Parameter Error	PUT /mediation/feature	Mediation Feature Configuration Mandatory Parameter is missing.	Make sure if FeatureEnabled is true in request then all field Error Configuration is present in request.
SEPP-MEDIATION-ERROR-012	Mediation Local Trigger Rule IPX Mode Error	PUT /sepp-mediation-trigger-rule-list/{triggerRuleListName}	In IPX Mode SEPP allow only 2 Trigger Points (N32 Ingress Request, N32 Egress Response) in local TRL configuration.	Make sure there is not any invalid Trigger Points like N32 Egress Request or N32 Ingress Response in Local Trigger Rule Configuration Request.
SEPP-MEDIATION-ERROR-013	Mediation service is not available	PUT /mediation/feature	Mediation Service Not deployed.	Before Enabling Mediation Feature through API , make sure Mediation Service is being deployed for SEPP.
SEPP-MEDIATION-ERROR-014	Invalid Error Action	PUT /mediation/feature	Invalid Error Action in Mediation Feature Configuration request.	Make sure to provide valid ErrorAction in Mediation Feature Configuration request in ErrorConfiguration section.

The following are the error codes of Cat-1 feature:

Table 3-5 Cat-1 Error Codes and Recovery Steps

Error Code	Error Text	Description	Recovery
SEPP-SECURITY-ERROR-001	Service API not in allowed list	If resource URI and Http Method is not matched as per the configured allowed list on SEPP.	Verify whether the proper Resource URI and HTTP method is configured in the CNC Console GUI. Go to the Security Countermeasure section, check under the Service API Allowed List for that particular Allowed list name for N32 Egress or N32 Ingress Direction.
SEPP-CN32FSEPP-ERROR-0012	Service API Validation Failed	This error occurs on CN32F microservice. If resource URI and Http Method is not matched as per the configured allowed list on SEPP.	Verify whether the proper Resource URI and HTTP method is configured in the CNC Console GUI. Go to the Security Countermeasure section, check under the Service API Allowed List for that particular Allowed list name for N32 Egress or N32 Ingress Direction.
SEPP-PN32FSEPP-ERROR-0015	Service API Validation Failed	This error occurs on PN32F microservice. If resource URI and Http Method is not matched as per the configured allowed list on SEPP.	Verify whether the proper Resource URI and HTTP method is configured in the CNC Console GUI. Go to the Security Countermeasure section, check under the Service API Allowed List for that particular Allowed list name for N32 Egress or N32 Ingress Direction.

The following are the error codes of Cat-2 Network ID Validation feature:

Table 3-6 Cat-2 Network ID Error Codes and Recovery Steps

Error Code	Error Text	Description	Recovery
SEPP-CN32FSEPP-ERROR-0013	Network ID Validation In Header Failed	Check whether PLMN ID is not matched in header of the incoming request on CN32F microservice.	Verify if correct regex is configured against the header identifier in Header Configuration. Verify if MCC and MNC combination is present in the visitor or target PLMN ID List based on the associated SEPP configuration
SEPP-CN32FSEPP-ERROR-0014	Network ID Validation In Body Failed	Check whether PLMN ID is not matched in body of the incoming request on CN32F microservice.	Verify if correct regex is configured against the body IE in Body IE Configuration. Verify f MCC and MNC combination is present in the home or visitor PLMN ID List based on configuration of associated SEPP.
SEPP-PN32FSEPP-ERROR-0016	Network ID Validation In Header Failed	Check whether PLMN ID is not matched in header of the incoming request on PN32F microservice.	Verify if correct regex is configured against the header identifier in Header Configuration. Verify if MCC and MNC combination is present in the target or visitor PLMN ID list based on the associated SEPP configuration.
SEPP-PN32FSEPP-ERROR-0017	Network ID Validation In Body Failed	Check whether PLMN ID is not matched in body of the incoming request on PN32F microservice.	Verify if correct regex is configured against the body IE in Body IE Configuration. Verify if MCC and MNC combination is present in the target or visitor PLMN ID list based on the associated SEPP configuration.

Table 3-7 Cat-3 Previous Location Check Error Codes

Error Code	Error Text	Description	Recovery
SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-ERROR-0019 or SEPP-PN32FSEPP-ERROR-0018	Previous Location Check Validation Failed	This error code is observed only on PN32F microservice. If Serving Network ID is not matching against the serving network ID coming from UDR response, to check whether UE authentication is success. This error also occurs if the authentication from UDR is false.	Verify whether the correct regex is configured against the serving network identifier in either Header or Body Configuration. Verify whether the MCC and MNC combination is present in the Serving Network ID. Verify whether the identifier coming in the ingress request on PN32F microservice is also same as the serving Network name coming as part of the UDR response if UE authentication is successful.
SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-EXCEPTION-0020	Previous Location Check Validation Failed Due To Exception	This error will occur if the system is not able to extract the SUPI, or if the incoming message doesn't contain SUPI, or if there are any sort of connectivity issues with NRF or UDR.	Verify if correct regex for UE ID in Header or Body configuration screen is configured due to which correct UE ID value is extracted. Verify if the incoming message has SUPI. Verify whether the FQDN or IP fetched for the UDR as part of NRF discovery call is reachable. Verify if UDR discovery procedure from NRF is successful. Verify if the SUPI received in the Ingress request message is part of the SUPI range received in UDR profile from discovery response from NRF.

Error Code

Error Text

Description

Recovery

SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-ERROR-0019 or

SEPP-PN32FSEPP-ERROR-0018

Previous Location Check Validation Failed

This error code is observed only on PN32F microservice.

If Serving Network ID is not matching against the serving network ID coming from UDR response, to check whether UE authentication is success.

This error also occurs if the authentication from UDR is false.

Verify whether the correct regex is configured against the serving network identifier in either Header or Body Configuration.
Verify whether the MCC and MNC combination is present in the Serving Network ID.
Verify whether the identifier coming in the ingress request on PN32F microservice is also same as the serving Network name coming as part of the UDR response if UE authentication is successful.

SEPP-PREVIOUS-LOCATION-CHECK-VALIDATION-EXCEPTION-0020

Previous Location Check Validation Failed Due To Exception

This error will occur if the system is not able to extract the SUPI, or if the incoming message doesn't contain SUPI, or if there are any sort of connectivity issues with NRF or UDR.

Verify if correct regex for UE ID in Header or Body configuration screen is configured due to which correct UE ID value is extracted.
Verify if the incoming message has SUPI.
Verify whether the FQDN or IP fetched for the UDR as part of NRF discovery call is reachable.
Verify if UDR discovery procedure from NRF is successful.
Verify if the SUPI received in the Ingress request message is part of the SUPI range received in UDR profile from discovery response from NRF.