8 Troubleshooting Provisioning Gateway

This section describes how to troubleshoot Provisioning Gateway when provisioning fails.

Verifying Provisioning Gateway Installation

  1. Run the following command to verify if the Provisioning Gateway services are up and running:

    kubectl get pods -n <provw-namespace>

    Figure 8-1 Sample: Provisioning Gateway Pod Status

    img/provgw-pod-status.png

    Note:

    All the pods in the above image are running. The pod, service, and deployment names are prepended by helm release name, which is used at the time of helm install --name <release name>.
  2. If any pod is not running, then check the auditor service logs using the steps given in the following section.

Checking Auditor Service Logs

To view the auditor service logs, run the following command:
$ kubectl logs <provgw-auditor-service pod> -n <provgw-namespace>

Note:

Alternatively, use Kibana to view logs.
To view the auditor service logs directly on the pods, run the following command:
kubectl exec -it provgw-auditor-service-0 -n <namespace> bash

Figure 8-2 Verifying Auditor Service Logs

Verifying Auditor Service Logs

To change the logging level for auditor service using helm:

  1. Open the latest provgw_value.yaml file used to install or upgrade Provisioning Gateway.
  2. Change the logging level root value under provgw to 'INFO'.
    auditor-service:
  ...
  ...
  ...
  logging:
     level:
       root: "WARN"

    Note:

    The supported logging level values are DEBUG, INFO, WARN, and ERROR.
  3. Run the following command to change the log level:
    helm upgrade provgw ocudr-helm-repo/provgw  -f <updated values.yaml with logging level as INFO> --version <helm version>

Checking PROVGW-SERVICE Logs

In the SLF mode, PROVGW-SERVICE dumps all the response status from active UDR in all the mentioned segments inside values.yaml. In the UDR mode, PROVGW-SERVICE dumps all the response from a single mentioned UDR inside values.yaml. To view logs, run the following command:

kubectl logs <provgw-service pod> -n <provgw-namespace>

Alternatively, run the following command to check the logs directly on the pods:

kubectl exec –it <provgw-service pod> -n <provgw-namespace> bash

A sample image is as follows:

Figure 8-3 PROVGW-SERVICE Logs

PROVGW-SERVICE Logs

Check the logs in the application.log file.

Changing PROVGW-SERVICE Logging Level

Note:

Redeploy the setup for the changes to take effect.
To change the PROVGW-SERVICE logging level using Helm:
  1. Open the provgw_value.yaml file used during Provisioning Gateway installation.
  2. Under provgw-service, change the value of logging level root attribute to "INFO" or "DEBUG".
    Extract from provgw_values.yaml

provgw-service:
...
...
...
    logging:
        level:
          root:"WARN"

    Other logging level values are DEBUG, INFO, WARN, ERROR.

Checking Provgw-CONFIG Logs

Run the following command to view logs:

$ kubectl logs <nudr-config pod> -n <ocudr-namespace>

Note:

You can also use Kibana to view logs.
Alternatively, you can check logs directly on pods as shown in the following image.

Figure 8-4 Checking Logs on Pods

Checking Logs on Pods

Changing provgw-service usageMode from "SLF" to "UDR"

To change the provgw-service usageMode from "SLF" to "UDR":
  1. Open the global values.yaml file used during UDR installation.
  2. Change the usageMode value in global section from "SLF" to "UDR".
  3. Change the udrIp mentioned inside the provgw-service inside soapService tag.
    Sample snippet from provgw-service values.yaml
    usageMode: "UDR"
provgw-service:
...
...
...
soapService:
    port: 62001
    udrIp: ocudr-ingressgateway.ocudr #Mention the udr-ingress provisioning ip
    udrSignallingIp: ocudr-ingressgateway-sig.ocudr #Mention the udr-ingress signalling ip
    convertToSec: true
    secEntity: |
      - name: QuotaEntity
        elementString: usage
        innerElemenString: quota
        outerFields:
          - version
          - quota
          - SequenceNumber
        innerFields:
          - name
          - cid
          - time
          - totalVolume
          - inputVolume
          - outputVolume
          - serviceSpecific
          - nextResetTime
          - Type
          - grantedTotalVolume
          - grantedInputVolume
          - grantedOutputVolume
          - grantedTime
          - grantedServiceSpecific
          - QuotaState
          - RefInstanceId
      - name: DynamicQuotaEntity
        elementString: definition
        innerElemenString: DynamicQuota
        outerFields:
          - version
          - DynamicQuota
          - SequenceNumber
        innerFields:
          - Type
          - name
          - InstanceId
          - Priority
          - InitialTime
          - InitialTotalVolume
          - InitialInputVolume
          - InitialOutputVolume
          - InitialServiceSpecific
          - activationdatetime
          - expirationdatetime
          - purchaseddatetime
          - Duration
          - InterimReportingInterval
      - name: Subscriber
        elementString: subscriber
        outerFields:
          - IMSI
          - MSISDN
          - NAI
          - EXTID
          - ACCOUNTID
          - BillingDay
          - Entitlement
          - Tier
          - SequenceNumber
      - name: StateEntity
        elementString: state
        innerElemenString: property
        outerFields:
          - version
          - property
          - SequenceNumber
        innerFields:
          - name
          - value

Debug Pod Creation Failure

A pod creation may fail due to any one of the following reasons:
  • Incorrect Pod Image: Check if any pod is in the ImagePullBackOff state. If it is there, it means the image name used for one of the pods is not correct. Check the following values in the values.yaml file.
    global:
  dockerRegistry: ocudr-registry.us.oracle.com:5000/provgw
 
provgw-service:
  image:
    name: provgateway
    tag: 25.2.100
 
prov-ingressgateway:
  image:
    name: ocingress_gateway
    tag: 25.2.104
 
prov-egressgateway:
  image:
    name: provgw/ocegress_gateway
    tag: 25.2.104
 
auditor-service:
  image:
    name: provgw/auditor_service
    tag: 25.2.100

    After updating the values.yaml file, run the following command to install helm:

    helm install –-name <release-name> --namespace <release-namespace>

    This helps you to purge the old setup and reinstall or upgrade the provisioning gateway instance.

  • Resource Allocation Failure: Check if any pod is in the Pending state. If yes, then run the following command for those pods:

    kubectl describe <provgw-service pod id> --n <provgw-namespace>

    Check the output for the Insufficient CPU warning. This warning means there are not sufficient CPUs to start the pod. To resolve this warning, either increase the number of CPUs as a hardware or reduce the number of CPUs allotted to a pod in the values.yaml file.
    provgw-service:
  resources:
    limits:
      cpu: 2
      memory: 2Gi
    requests:
      cpu: 2
      memory: 2Gi
 
prov-ingressgateway:
  resources:
    limits:
      cpu: 2
      memory: 2Gi
      initServiceCpu: 1
      initServiceMemory: 1Gi
      updateServiceCpu: 1
      updateServiceMemory: 1Gi
    requests:
      cpu: 2
      memory: 2Gi
      initServiceCpu: 1
      initServiceMemory: 1Gi
      updateServiceCpu: 1
      updateServiceMemory: 1Gi
 
prov-egressgateway:
  resources:
    limits:
      cpu: 2
      memory: 2Gi
      initServiceCpu: 1
      initServiceMemory: 1Gi
      updateServiceCpu: 1
      updateServiceMemory: 1Gi
    requests:
      cpu: 2
      memory: 2Gi
      initServiceCpu: 1
      initServiceMemory: 1Gi
      updateServiceCpu: 1
      updateServiceMemory: 1Gi
 
auditor-service:
  resources:
    limits:
      cpu: 2
      memory: 2Gi
    requests:
      cpu: 2
      memory: 2Gi

    After updating the values.yaml file, run the following command to install helm:

    helm install –-name <release-name> --namespace <release-namespace>

Debugging UDR Registration with Provisioning Gateway in SLF Mode

Before deploying Provisioning Gateway, you need to ensure that UDR pods are in the running state and FQDNs of UDRs are correct.

In the helm charts, UDR FQDNs information is available in the following format:

Figure 8-5 UDR FQDN Info

UDR FQDN Info
On startup, Provisioning Gateway (provgw) registers with the preferred FQDN configured for each segment irrespective of its active state, and then, it starts verifying the status of the preferred FQDN. If the preferred FQDN is not available (down), then it checks the status of non-preferred FQDN.
  • If the non-preferred FQDN is in active state, provgw registers with the non-preferred FQDN.
  • SLF switchover to non-preferred FQDN in a segment is done, If the preferred FQDN respond with a 503 status code and no proper response body or if the proper response body has a corresponding Error Code Profile and is set under Provisioning Gateway Global Configurations. Proper response body means the response must have status code, title, cause, and so on. A common improper response body is no healthy upstream.
  • If the non-preferred FQDN is also not available (down) then it keeps the preferred one as the active SLF in that segment.
If provgw registers with non-preferred FQDN, then it will keep on checking the preferred FQDN and tries to fall back to the preferred FQDN, once it is restored. Provgw update the logs if the preferred SLF in the segment is not available. If the preferred FQDN is active, then provgw does not try to validate the non-preferred FQDN availability. Non-preferred FQDN is considered when the preferred one is down.
Before deploying Provisioning Gateway, you need to ensure that:
  • All the pods of UDR are in running state
  • FQDN of the UDR is correctly mentioned
  • The preferred FQDN is considered as active when all the FQDN’s are down in one segment
  • The active FQDN is checked every 15 seconds (this value is configurable)
In Provisioning Gateway (ProvGw),
  • The GET requests sent to UDR for updating the active FQDN, are not dumped in the ProvGw logs.
  • There is only one provgw-service that receives all the requests from Ingress Gateway. If any request fails:
    • With 404 status code without Problem Details information:, then there may be issues with the routeConfig on the ingressgateway custom values file.
      {"title":"404 NOT_FOUND","status":404,"detail":"udr001.oracle.com: ingressgateway: NOT_FOUND: OUDR-IGWSIG-E183"}

      You must check the custom values.yaml file for the essential route configurations. If the essential route configurations are not present you must add the route configurations.

    • With 503 status code with SERVICE_UNAVAILABLE in Problem Details:, then it means that the nudrdrservice pod is not reachable due to some reason.
      {"title":"Service Unavailable","status":503,"detail":"udr001.oracle.com: ingressgateway: Service Unavailable: OUDR-IGWSIG-E003","cause":"Encountered unknown host exception at IGW"}

      Verify this error in the ocudr-ingressgateway-sig and ocudr-ingressgateway-prov pod logs for errors or exception. Check the ocudr-nudr-drservice and ocudr-nudr-dr-provservice pod status and fix the issue.

    • Try to find the root cause using metrics as follows:
      • If the count of oc_ingressgateway_http_requests_total measurement increases, then check the content of incoming requests. You need to ensure that the incoming JSON data blob is as per specification.
      • If the udr_rest_request measurement increases more than one per request then ensure that UDR is working fine and Ingress Gateway of UDR is not down.
  • To debug HTTPS related issues, see the Troubleshooting Unified Data Repository section in the Oracle Communications Cloud Native Core, Unified Data Repository Installation, Upgrade, and Fault Recovery Guide.

    Figure 8-6 ProvGw - HTTPS Port Exposure

    img/provgw-https-port-exposure.png
  • To debug HPA Issues, see the Troubleshooting Unified Data Repository section in the Oracle Communications Cloud Native Core, Unified Data Repository Installation, Upgrade, and Fault Recovery Guide.

Debugging Provisioning Gateway with Service Mesh Failures

There are some known failure scenarios that you may encounter while installing Provisioning Gateway with service mesh. The scenarios along with their solutions are as follows:
  • Istio-Proxy side car container not attached to Pod: This particular failure arise when istio injection is not enabled on UDR installed namespace. Run the following command to verify the same:

    kubectl get namespace -L istio-injection

    Figure 8-7 Verifying Istio-Proxy

    img/troubleshooting-asm-istio-proxy.png

    To enable the istio injection, run the following command:

    kubectl label --overwrite namespace <nf-namespace> istio-injection=enabled

    If any of the hook pods is stuck in the NotReady state and not getting cleared after completion, then check whether the following configuration is set to true under global section. And also ensure the URL configured for istioSidecarQuitUrl is correct.

    Figure 8-8 service mesh related configuration

    service mesh related configuration
    There can be cases where Prometheus does not scrap metrics from nudr-nrf-client-service. In these cases, ensure the following annotation is present under nudr-nrf-client-service.

    Figure 8-9 When Prometheus Does Not Scrap Metric

    When Prometheus Does Not Scrap Metric
  • If Provisioning System outside service mesh is not able to contact Provisioning Gateway service through its Ingress Gateway, then:
    • Exclude the HTTP container port traffic for prov-ingressgateway from istio side car.
    • Ensure the following configuration is enabled under prov-ingressgateway section.

      Figure 8-10 Enabling istioIngressTlsSupport

      Enabling istioIngressTlsSupport
  • If there are issues in viewing Provisioning Gateway metrics on OSO Prometheus then add the following annotation to all the deployments for the NF.

    Figure 8-11 Annotation to View Provisioning Gateway Metrics

    Annotation to View Provisioning Gateway Metrics
  • When vDBTier is used as backend, there are connectivity issues, nudr-preinstall communicates with database (can be seen from error logs on pre-install hook pod), then make the destination rule and service entry for mysql-connectivity-service on occne-infra namespace. For information about creating destination rule and service entry, see Oracle Communications Cloud Native Core, Unified Data Repository Installation, Upgrade, and Fault Recovery Guide, where ASM specific changes are mentioned.

Troubleshooting provgw-service through Metrics

If provgw-service requests fail, try to find the root cause from metrics as well. Some of the troubleshooting tips are as follows:

  • If the count of oc_ingressgateway_http_requests_total measurement increases, check the content of incoming request and make sure that incoming JSON data blob is proper and as per the specification.
  • If on one request, the udr_rest_request measurement increases more than once then ensure:
    • UDRs are working fine
    • Ingress Gateways of UDRs are not down

Troubleshooting provgw-config through Metrics

Some of the troubleshooting tips are as follows:
  • Measurements of metrics like provgw_config_total_requests_total{Method='GET'}, provgw_config_total_requests_total{Method='POST'}, provgw_config_total_requests_total{Method='PUT'} gives us information about the total requests pegged for the GET, POST, and PUT methods respectively.
  • If the count of provgw_config_total_responses_total{Method='GET/POST/PUT',StatusCode="400/404/405/500"}” measurement increases, then it means the requests are not being processed and resulting in failures.

Troubleshooting provgw-config through Configdb

If provgw-config requests fail, try to find the root cause from configdb. Some of the troubleshooting tips are as follows:

  • If you get a BAD REQUEST for GET API, then make sure all the tables shown below is present in configdb table.

    Figure 8-12 Configdb Table


    Configdb Table

  • If all the table are present and you are getting a BAD REQUEST for GET API, then you must verify the configuration item table shown below.

    Figure 8-13 Configuration Item Table


    Configuration Item Table

  • If you get a BAD REQUEST and NOT FOUND for import and export API, then you must verify the import and export table shown below.

    Figure 8-14 Import and Export Data


    Import and Export Data

Troubleshooting Auditor Service issues Using Logs

If Auditor Service fail, try to find the root cause as follows:
  • Make sure that files are transferred completely to both segment folders from Subscriber Export Tool pod.
  • Make sure that you do not have the old files with the same subscriber data as the new files at any given time. The old files must be deleted manually from both the segment folders.
  • When the auditor service pod starts, audit is scheduled based on the configurations. You may observe warning or error messages in the logs.
  • If the message shows as "No audit eligible exported CSV file present for", you must place the CSV files in both the segment folders. The audit will be unsuccessful if the CSV files are placed in only one segment folder.
  • If the message shows as "More than configured discrepancies. Cannot process audit". You must the correct the files. By default only 100000 discrepancies are allowed, if the discrepancies are more than the default value, check the files on both sites where the subscriber dump has taken place.
  • Only one audit process must be running at a given point of time. If you schedule a second audit when the first audit is in progress, the application responds with an error stating "Audit is in progress, so not launching scheduler".
  • If messages shows as "Audit process stopped due to audit pause enabled", You must set the auditPause parameter flag to false to resume the audit.

Troubleshooting Lifecycle Management (LCM) Based Automation Issues

Perform the following steps if there are issues related to Lifecycle Management (LCM) Based Automation feature:
  • Make sure that the autoCreateResources.enabled and autoCreateResources.serviceaccounts.enabled flags are enabled.
  • During upgrade, if a new service account name is provided in the serviceAccountName parameter with autoCreateResources.enabled and autoCreateResources.serviceaccounts.enabled flags enabled, then a new service account name is created. If the service account name is not created, then you must check the configuration and the flags again.
  • During upgrade, you must use a different service account name when you upgrading from manual to automation. If you use same service account name when upgrading from manual to automation, then Helm does not allow upgrade due to ownership issues.
  • To use the OSO alerts automation feature, you must follow the installation steps of the oso-alr-config Helm chart by providing the alert file that needs to be applied to the namespace. For more information, see Oracle Communications Cloud Native Core, Operations Services Overlay Installation and Upgrade Guide. If the alert file is not provided during Helm installation, you can provide the alert file during upgrade procedure. Alert file can be applied to the namespace during Helm installation or upgrade procedure.
  • If an incorrect data is added to the alert file, you can clear the entire data in the alert file by providing the empty alert file (ocslf_alertrules_empty_<version>.yaml). For more information, see "OSO Alerts Automation" in Configuring Alerts section .
  • If you provide the service account name during the upgrade but the feature is disabled, the nudr-pre-upgrade hook fails because it cannot find the service account. If the upgrade fails, the rollback to the previous version will be unsuccessful due to the missing alternate route service account, resulting in an error message indicating that the service account for the alternate route service is not found. To address this issue, it is necessary to manually create the service account after the initial upgrade failure, then continue with the upgrade, this will also ensures a successful rollback.

Export the Configuration Data for 25.1.1xx, 25.1.2xx, and 25.2.1xx Releases

Perform the following steps when exporting the configuration data from 25.1.1xx or 25.1.2xx release version and importing the data to 25.2.1xx or higher release version:
  • When data is exported from 25.1.1xx or 25.1.2xx release version and imported to 25.2.1xx or higher release version, the imported API report is expected to have following response.
    {"report":[{"name":"Auditor Service","status":"1 / 1 records
    imported  successfully."},{"name":"ProvGateway Service","status":"1 / 1 records
    imported successfully."},{"name":"Egressgateway Logging","status":"1 / 1  records
    imported successfully."},{"name":"Ingressgateway  Logging","status":"1 / 1 records
    imported  successfully."},{"name":"ProvGateway Service
    Logging","status":"400"},{"name":"Auditor Service
    Logging","status":"400"},{"name":"SEC Configuration","status":"1 / 1  records
    imported successfully."},{"name":"ProvGateway Config
    Logging","status":"400"},{"name":"Global Configurations","status":"1 / 1  records
    imported successfully."}]}
  • All the logging level import status is 400, since there is a modification in logger package name from "springframework" to "micronaut" in 25.2.100 release onwards. The exported zip file has "springframework" as the package name. Due to this, you will get "400 schema validation failure" internally in 25.2.1xx release. The logging level import status responds with 400, which is as expected.
  • After the import is completed, you must update the following log level API's once as required:
    1. http://<IP>:<PORT>/provgw-config/v1/provgw-service/logging
    2. http://<IP>:<PORT>/provgw-config/v1/auditor-service/logging
    3. http://<IP>:<PORT>/provgw-config/v1/provgw-config/logging

8.1 Debugging Service Mesh related Issues

If the provisioning system outside service mesh is not able to contact provgw service through its ingress gateway, then exclude the http container port traffic for prov-ingressgateway from istio sidecar. Also, you need to enable the following configuration under prov-ingressgateway section.

Figure 8-15 Debugging Service Mesh related Issue

img/provgw-service-mesg-debug.png
If there are issues in viewing ProvGw metrics on OSO Prometheus then add the following highlighted annotation to all the deployments of NF.

Figure 8-16 Annotation: Issue in Viewing ProvGw Metric on OSO Prometheus

img/when-issue-viewing-provgw-metric.png

8.2 Using Debug Tools

Overview

Debug Tools are a collection of third party troubleshooting tools that you can use to troubleshoot run time errors. The collection includes:
  • tcpdump
  • ip
  • netstat
  • jq
  • curl
  • ping
  • nmap
  • dig

Enabling Debug Tool

To enable a debug tool container, edit the custom-values.yaml of provgw at
  • Global Level: Enable the extraContainers parameter and configure the extraContainersTpl value.

    Sample template is as follows:

    extraContainers: ENABLED
extraContainersVolumesTpl: |
  - name: debug-tools-dir
    emptyDir:
      medium: Memory
      sizeLimit: "4Gi"
extraContainersTpl: |
  - command:
      - /bin/sleep
      - infinity
    image: {{ .Values.global.dockerRegistry }}/debug-tools:25.2.101
    imagePullPolicy: IfNotPresent
    name: {{ printf "%s-tools-%s" (include "getprefix" .) (include "getsuffix" .) | trunc 63 | trimPrefix "-" | trimSuffix "-"  }}
    resources:
      requests:
        ephemeral-storage: "512Mi"
        cpu: "0.5"
        memory: "4Gi"
      limits:
        ephemeral-storage: "512Mi"
        cpu: "0.5"
        memory: "4Gi"
    securityContext:
      allowPrivilegeEscalation: true
      capabilities:
        drop:
        - ALL
        add:
        - NET_RAW
        - NET_ADMIN
      runAsUser: 1002
    volumeMounts:
    - mountPath: /tmp/tools
      name: debug-tools-dir
  • Service Level: Set the extraContainers: USE_GLOBAL_VALUE/ENABLED parameter under each microservice section.

The default user ID of the debug tool container is '7000'.

Default value: uid=7000(debugtool) gid=7000(debugtool) groups=7000(debugtool)

You can retain this default value. To override this value, add the following under securityContext in extraContainersTpl.

runAsUser: <user-id>

Configurable Options

Parameter Description
command String array used for container command
image Docker image name
imagePullPolicy Image Pull Policy
name Name of the container
resources Compute Resources required by this container
resources.limits Limits describes the maximum amount of compute resources allowed
resources.requests Requests describes the minimum amount of compute resources required
resources.limits.cpu CPU limits
resources.limits.memory Memory limits
resources.limits.ephemeral-storage Ephemeral Storage limits
resources.requests.cpu CPU requests
resources.requests.memory Memory requests
resources.requests.ephemeral-storage Ephemeral Storage requests
securityContext Security options the container should run with
securityContext.allowPrivilegeEscalation AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This tool directly controls if the no_new_privs flag is set on the container process
secuirtyContext.readOnlyRootFilesystem Whether this container has a read-only root file system. Default is false
securityContext.capabilities The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container run time
securityContext.capabilities.drop Removed capabilities
secuirtyContext.capabilities.add Added capabilities
securityContext.runAsUser The UID to run the entry point of the container process

Accessing Debug Tool

After Installation, the debug tool gets injected into the pods.

Sample Debug Tool Container Injection
[root@master ~]# kubectl get pods -n provgw
NAME                                                READY   STATUS    RESTARTS   AGE
provgw-prov-ingressgateway-7cfd68cbbd-6t5rm         2/2     Running   0          2d6h
provgw-provgw-service-54b7c4996c-rfdvz              2/2     Running   0          3d6h
provgw-prov-egressgateway-54b7c4996c-sx8lt          2/2     Running   0          2d5h

Run the following command to enter the debug tool container:

kubectl exec -it <pod name> -c <container name> -n <namespace> bash

Example: kubectl exec -it provgw-prov-ingressgateway-7cfd68cbbd-6t5rm -c tools -n provgw bash

Copying Output Files from Container to Host

Run the following command to copy output files from container to host:

kubectl cp -c <container name> <pod name>:<file location in container> -n <namespace> <destination location>

Example: kubectl cp -c tools provgw-prov-ingressgateway-7cfd68cbbd-6t5rm:/tmp/capture.pcap -n provgw /tmp/

8.2.1 Debugging Issues Related to Debug Tool

Some of the issues that you may encounter when Debug container is enabled are:
  • No pod is spawned when helm install command is executed with debug container related configurations enabled. You can do a describe on the replica set of a particular replicaset to see if there is error related missing PodSecurityPolicy for debug tool.

    Example: kubectl describe rs ocudr-nudr-drservice -n myudr

    Figure 8-17 Debug Tool Issue

    img/provgw-troubleshooting-debug-tool.png
    In this case, create a required PodSecurityPolicy and associate it with a Rolebinding. Do a describe on Rolebinding and check if it has the serviceaccount associated with the deployment.

    Figure 8-18 Creating Pod Security Policy

    img/provgw-troubeshoot-debug-tool.png
  • If any of the pod does not include debug container to it, make sure the extraContainer configuration is set to ENABLED/USE_GLOBAL_VALUE under the respective section which has configuration values related to pod deployment.

8.3 Log Attribute Details

The following table lists the log attribute details for UDR. These details are applicable to Provisioning Gateway.

Table 8-1 Log Attribute Details

Log Attribute Details Example Value Data Type Source
thread Thread Name Internal by Spring boot XNIO-1 task-1 String log4j
level Log Level of the log printed WARN String log4j
loggerName Class which printed the log ocudr.udr.services.service.DbHandler String log4j
message Outputs the application supplied message Subscriber does not exist String Application
endOfBatch Log4j2 Internal false boolean log4j
loggerFqcn Log4j2 Internal org.apache.logging.slf4j.Log4jLogger String log4j
instant Epoch time {"epochSecond":1599703750,"nanoOfSecond":210064000} Object log4j
threadId Outputs the ID of the thread that generated the logging event, set internally by Log4j2 23 Integer log4j
threadPriority Thread Priority set internally by Log4j2 5 Integer log4j
messageTimestamp Timestamp when log was printed 21-02-17 07:36:06.343+0000 String Application
application NF application name ocudr String Application
engVersion Engineering version of software 1.10.20 String Application
mktgVersion Marketing version of software 1.10.20.0.0 String Application
microservice Microservice name ocudr-nudr-drservice String Application
vendor Vendor name Oracle String Application
subscriberId SubscriberId for which request received msisdn-1111111113 String Application
resourceId Request Uri nudr-group-id-map/v1/nf-group-ids String Application
resultCode Response statusCode 404 String Application
ocLogId Inter NF logId for tracing 1613547369374_225_ocudr-ingressgateway-6f585c76d4-tp622 String Application
sbiCorrelationHeader SBI Correlation Header for request received msisdn-1111111113 String Application
requestType request type received GET String Application
kubernetes.container_name Container name generating log nudr-dr-service String fluentd
kubernetes.namespace_name Namespace of service ocudr String fluentd
kubernetes.pod_name Pod name ocudr-nudr-drservice-7f8c47f5c9-flkmz String fluentd
kubernetes.container_image Container image cne-170-ga-bastion-1:5000/ocudr/nudr_datarepository_service:1.9.50 String fluentd
kubernetes.container_image_id Container image ID cne-170-ga-bastion-1:5000/ocudr/nudr_datarepository_service@sha256:1141f245a3a437f1423496aebf616a6d3315e22ad09904a868bf1b471759b616 String fluentd
kubernetes.pod_id POD id 6dfa91f8-2d0a-4d8c-a339-381ce98264df String fluentd
kubernetes.host Worker node name cne-170-ga-k8s-node-8 String fluentd
kubernetes.namespace_id Unique namespace ID assigned by K8 d932a8ae-54e9-4df1-8e30-e0335f0b1303 String fluentd
labels All the labels on pod that generate the logs

"labels": {

"pod-template-hash": "7f8c47f5c9",

"app_kubernetes_io/instance": "ocudr",

"app_kubernetes_io/managed-by": "Tiller",

"app_kubernetes_io/name": "nudr-drservice",

"app_kubernetes_io/part-of": "ocudr",

"app_kubernetes_io/version": "1.6.0.0.0",

"helm_sh/chart": "nudr-drservice-1.9.50",

"io_kompose_service": "nudr-drservice"

}

 object fluentd
originHost Diameter client fqdn diamcli1.oracle.com String

Application

Note: Only in diameterproxy and diameter-gateway

originRealm Diameter client realm oracle.com String

Application

Note: Only in diameter-gateway

serviceIndications Diameter service indications for GET operations "serviceIndications" : [ "CamiantUserData", "CamiantStateData"] Array

Application

Note: Only in diameterproxy

Example: Log Under Analysis
{"instant":{"epochSecond":1613547366,"nanoOfSecond":343417698},"thread":"XNIO-1 task-1","level":"WARN","loggerName":"ocudr.udr.services.service.DbHandler","message":"Subscriber does not exist","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":46,"threadPriority":5,"messageTimestamp":"21-02-17 07:36:06.343+0000","application":"ocudr","sbiCorrelationHeader":"msisdn-1111111113","engVersion":"1.10.20","mktgVersion":"1.10.20.0.0","microservice":"ocudr-nudr-drservice","vendor":"Oracle","subscriberId":"msisdn-1111111113","resourceId":"nudr-group-id-map/v1/nf-group-ids","resultCode":"404","ocLogId":"1613547369374_225_ocudr-ingressgateway-6f585c76d4-tp622","requestType":"GET"}
Example: Log From Kibana
{
  "_index": "logstash-2021-02-19",
  "_type": "_doc",
  "_id": "yyQXuHcBwFjE8wmhickN",
  "_version": 1,
  "_score": 0,
  "_source": {
    "stream": "stdout",
    "docker": {
      "container_id": "b1b78faa1043132f77148f16777e60b1db8ae30e9e6b5c2f5af45248063f7d6a"
    },
    "kubernetes": {
      "container_name": "nudr-drservice",
      "namespace_name": "bharathudr1",
      "pod_name": "bharathudr1-nudr-drservice-7bd66864c6-fw7cq",
      "container_image": "cne-172-bastion-1:5000/ocudr/nudr_datarepository_service:ocLogIdTest1",
      "container_image_id": "cne-172-bastion-1:5000/ocudr/nudr_datarepository_service@sha256:24875dad7fd363bcb8ec300b007491b2259796e152446fd3d707b17f62fcd6b4",
      "pod_id": "ea92c753-48af-4b31-9356-1ce24f27025e",
      "host": "cne-172-k8s-node-10",
      "labels": {
        "pod-template-hash": "7bd66864c6",
        "app_kubernetes_io/instance": "bharathudr1",
        "app_kubernetes_io/managed-by": "Helm",
        "app_kubernetes_io/name": "nudr-drservice",
        "app_kubernetes_io/part-of": "ocudr",
        "app_kubernetes_io/version": "1.6.0.0.0",
        "helm_sh/chart": "nudr-drservice-1.10.20",
        "io_kompose_service": "nudr-drservice"
      },
      "master_url": "https://10.233.0.1:443/api",
      "namespace_id": "260e3d4a-d455-4afc-9cff-9aaf8eba79d1"
    },
    "instant": {
      "epochSecond": 1613701232,
      "nanoOfSecond": 431593919
    },
    "thread": "XNIO-1 task-1",
    "level": "WARN",
    "loggerName": "ocudr.udr.services.service.DbHandler",
    "message": "Subscriber does not exist",
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
    "threadId": 46,
    "threadPriority": 5,
    "messageTimestamp": "21-02-19 02:20:32.431+0000",
    "application": "bharathudr1",
    "engVersion": "1.10.20",
    "mktgVersion": "1.6.0.0.0",
    "microservice": "bharathudr1-nudr-drservice",
    "vendor": "Oracle",
    "subscriberId": "imsi-100000002",
    "sbiCorrelationHeader": "imsi-100000002",
    "resourceId": "nudr-group-id-map-prov/v1/slf-group",
    "resultCode": "404",
    "ocLogId": "1613701211520_2220_bharathudr1-ingressgateway-7d7659c58b-5k6b2",
    "requestType": "GET",
    "@timestamp": "2021-02-19T02:20:32.432795705+00:00",
    "tag": "kubernetes.var.log.containers.bharathudr1-nudr-drservice-7bd66864c6-fw7cq_bharathudr1_nudr-drservice-b1b78faa1043132f77148f16777e60b1db8ae30e9e6b5c2f5af45248063f7d6a.log"
  },
  "fields": {
    "@timestamp": [
      "2021-02-19T02:20:32.432Z"
    ],
    "timestamp": []
  },
  "highlight": {
    "resultCode": [
      "@kibana-highlighted-field@404@/kibana-highlighted-field@"
    ],
    "message": [
      "@kibana-highlighted-field@Subscriber@/kibana-highlighted-field@ @kibana-highlighted-field@does@/kibana-highlighted-field@ @kibana-highlighted-field@not@/kibana-highlighted-field@ @kibana-highlighted-field@exist@/kibana-highlighted-field@"
    ],
    "kubernetes.namespace_name": [
      "@kibana-highlighted-field@bharathudr1@/kibana-highlighted-field@"
    ]
  }
}