1. Cloud Native Unified Data Repository Installation and Upgrade Guide
  2. Troubleshooting Unified Data Repository
  3. Generic Checklist
  4. Verifying OCUDR Microservices Logs

Verifying OCUDR Microservices Logs

In this section, you will learn to check logs of the following microservices:
  • OCUDR-NUDR-DRSERVICE
  • NRF-CLIENT-SERVICE
  • NUDR-NOTIFY-SERVICE
  • NUDR-CONFIG-SERVICE
  • NUDR-CONFIG-SERVER
  • NUDR-DIAMETERPROXY Service

Checking Logs in OCUDR-NUDR-DRSERVICE

OCUDR-NUDR-DRSERVICE dumps all the header while processing messages. User should search for "Before Request/After Request" header in the messages.
If nudr-drservice requests are failing, check the count of udr_schema_operations_failure_total measurement. If this count is increasing:
  • Check the content of incoming requests
  • Ensure that the incoming json data blob is proper
  • Connectivity between microservices are mysql DB nodes
  • Try not to insert duplicate keys
  • Ensure DB nodes have enough resources available
To view logs, execute the following command:

kubectl logs -f <nudr-drservice pod> -n <ocudr-namespace>

To check logs directly on the pods, execute the following command:

kubectl exec -it ocudr-nudr-drservice-779c67b9f-sjcmv bash

To change logging level in the ocudr-nudr-drservice using helm:
  1. Open the latest ocudr_value.yaml file that is used at the time of ocudr installation/upgrade.
  2. Change the value of "logging level root" attribute under "ocudr" to "INFO".

    Note:

    OCUDR supports logging level values: DEBUG, INFO, WARN and ERROR.
  3. Execute the following helm upgrade command to change the log level:

    helm upgrade ocudr ocudr-helm-repo/ocudr -f <updated values.yaml with logging level as INFO> --version <helm version>

Checking Logs in NUDR-NRF-CLIENT-SERVICE

If the count of udr_nrf_livenessProbe_failure_total measure increases, you need to ensure that helm charts configuration for “nudr-nrf-client-service” is correct and NRF server is up and running fine.

If nudr-nrf-client-service is not able to register with NRF and there is a difference between “udr_nrf_registration_requests_total” and “udr_nrf_registration_success_total”, then you need to ensure that helm charts configuration for “nudr-nrf-client-service” are correct.

If nudr-nrf-client-service is not able to de-register with NRF and there is a difference between “udr_nrf_deregistration_requests_total” and “udr_nrf_deregistration_success_total”, then you need to ensure that helm charts configuration for “nudr-nrf-client-service” are correct.

To view the NUDR-NRF-CLIENT-SERVICE logs, execute the following command:

kubectl logs <nrf-client-pod pod> -n <ocudr-namespace>

To check logs directly on the pods, refer to the screen given below:

Figure 5-5 NRF-Client-Service Logs

img/nrf-client-service-logs.png
To change logging level in the nrf-client-service using helm:
  1. Open the latest ocudr_value.yaml file that is used at the time of ocudr installation/upgrade.
  2. Change the value of "logging level root" attribute under "nrfclient" to "INFO".

    Note:

    nudr-nrf-client-service supports logging level values: DEBUG, INFO, WARN and ERROR.
  3. Execute the following helm upgrade command to change the log level:

    helm upgrade ocudr ocudr-helm-repo/ocudr -f <updated values.yaml with logging level as INFO> --version <helm version>

Checking Logs in NUDR-NOTIFY-SERVICE

Measurements like nudr_notif_notifications_ack_2xx_total, nudr_notif_notifications_ack_4xx_total, and nudr_notif_notifications_ack_5xx_total gives information about the response code returned in the notification response. If the count of nudr_notif_notifications_send_fail_total measurement increases, then you need to ensure that the notification server mentioned in the NOTIFICATION_URI during subscription request is up and running.

To view the NUDR-NOTIFY-SERVICE logs, execute the following command:

kubectl logs <nudr-notify-service pod> -n <ocudr-namespace>

To check logs directly on the pods, refer to the screen given below:

Figure 5-6 NUDR-NOTIFY-SERVICE Logs

img/nudr-notify-service-logs.png
To change logging level in the nudr-notify-service using helm:
  1. Open the latest ocudr_value.yaml file that is used at the time of ocudr installation/upgrade.
  2. Change the value of "logging level root" attribute under "ocudr" to "INFO".

    Note:

    nudr-notify-service supports logging level values: DEBUG, INFO, WARN and ERROR.
  3. Execute the following helm upgrade command to change the log level:

    helm upgrade ocudr ocudr-helm-repo/ocudr -f <updated values.yaml with logging level as INFO> --version <helm version>

Checking Logs in NUDR-CONFIG-SERVICE

To view logs, execute the following command:

kubectl logs <nudr-config pod> -n <ocudr-namespace>

To check logs directly on the pods, refer to the screen given below:

Figure 5-7 NUDR-CONFIG-SERVICE Logs

img/nudr-config-service-logs.png
To change logging level in the ocudr-nudr-config service using helm:
  1. Open the latest ocudr_value.yaml file that is used at the time of ocudr installation/upgrade.
  2. Change the value of "logging level root" attribute under "ocudr" to "INFO".

    Note:

    OCUDR supports logging level values: DEBUG, INFO, WARN and ERROR.
  3. Execute the following helm upgrade command to change the log level:

    helm upgrade ocudr ocudr-helm-repo/ocudr -f <updated values.yaml with logging level as INFO> --version <helm version>

Checking Logs in NUDR-CONFIG-SERVER

To view logs, execute the following command:

kubectl logs <nudr-config-server pod> -n <ocudr-namespace>

To change logging level in the ocudr-nudr-config-server service using helm:
  1. Open the latest ocudr_value.yaml file that is used at the time of ocudr installation/upgrade.
  2. Change the value of "logging level root" attribute under "ocudr" to "INFO".

    Note:

    OCUDR supports logging level values: DEBUG, INFO, WARN and ERROR.
  3. Execute the following helm upgrade command to change the log level:

    helm upgrade ocudr ocudr-helm-repo/ocudr -f <updated values.yaml with logging level as INFO> --version <helm version>

Checking Logs in NUDR-DIAMETERPROXY Service

Debug errors from ocudr-nudr-diameterproxy:

  • If diameterproxy rejects any request or you are not able to send any request from seagull machines, it means the dictionary file is not loaded correctly to the application. You need to check the dictionary path and change it, if required and redeploy the diameterproxy service. (The dictionary file path should be "/home/udruser/app/diameter").
  • If diameterproxy answers CEA message with DIAMETER_UNKNOWN_PEER, it means client peer is not configured correctly. To resolve this, configure client peer of nudr-diameterproxy service.
  • If diameterproxy answers CEA message success and other SH message response as DIAMETER_UNABLE_TO_COMPLY, it means the dr-service pod is not up and running or sent sh message is invalid. You can check dr-service failure using nudr_diameterproxy_rest_failure_res_msgs_total metrics name and invalid sh message, if nudr_diameterproxy_total_requests_total metric is not increasing .
  • If there are many error logs in diameterproxy micro service stating connection refused with some IP Address and port, it means specified server peer in helm charts is not running and diameterproxy retries to connect with that peer.
  • If you are not getting any PNR messages then check whether dr-service and notify-service is up and running. You need to ensure that server peer configuration is correct.
To view NUDR-DIAMETERPROXY service logs, execute the following command:

kubectl logs <nudr-diameterproxy pod> -n <ocudr-namespace>

To change logging level in the ocudr-nudr-diameterproxy service using helm:
  1. Open the latest ocudr_value.yaml file that is used at the time of ocudr installation/upgrade.
  2. Change the value of "logging level root" attribute under "ocudr" to "INFO".

    Note:

    OCUDR supports logging level values: DEBUG, INFO, WARN and ERROR.
  3. Execute the following helm upgrade command to change the log level:

    helm upgrade ocudr ocudr-helm-repo/ocudr -f <updated values.yaml with logging level as INFO> --version <helm version>

Note:

You can use kibana also to view logs.