2 Logs
This chapter explains the process to retrieve the logs and status that can be used for effective troubleshooting.
2.1 Log Levels
Logs register system events along with their date and time of occurrence. They also provide important details about a chain of events that could have led to an error or problem.
A log level helps in defining the severity level of a log message. For NEF, the log
level of a microservice can be set to any one of the following valid values:
- TRACE: A log level that describes events, as a step by step execution of code. This can be ignored during the standard operation, but may be useful during extended debugging sessions.
- DEBUG: A log level used for events during software debugging when more granular information is needed.
- INFO: A standard log level indicating that something has happened, an application has entered a certain state, etc.
- WARN: A log level indicates that something unexpected has happened in the application, a problem, or a situation that might disturb one of the processes. But this does not mean that the application has failed. The WARN level should be used in situations that are unexpected, but the code can continue to work.
- ERROR: A log level that should be used when an application hits an issue preventing one or more functionalities from functioning.
Using this information, the logs can be filtered based on the system requirements. For instance, if you want to filter the critical information about your system from the informational log messages, set a filter to view messages with only WARN log level in Kibana.
The following table provides log level details that may be helpful to handle different NEF debugging issues:
Table 2-1 Log Levels
| Scenarios | Pod | Logs to be searched | Log Level |
|---|---|---|---|
| Registration with NRF Successful | nrf-client-service | Register completed successfully / "nfServiceStatus":"REGISTERED" | INFO |
| Heartbeat message log | nrf-client-service | Update completed successfully | INFO |
| NRF configurations reloading | nrf-client-service | NRF client config reloaded | INFO |
| Check for exiting NF Instance Entry | nrf-client-service | No registered NF instance exists | WARN |
| Started Application | nrf-client-service | Successful application start | INFO |
| Started Application | ocnef-monitoringevents | Successful application start | INFO |
| NRF Client Config Initialized | nrf-client-service | Initialize NRF client configuration | INFO |
| FQDN/BASEURL/livenessProbeUrl Improper | nrf-client-service | response=<503,java.net.UnknownHostException | WARN |
| nudr-drservice liveness probe failure | nrf-client-service | NFService liveness probe failed | WARN |
| SQL Exception during start up | ocnef-monitoringevents | java.sql.SQLException | WARN |
| DB connection pool Established | ocnef-monitoringevents | HikariPool-1 - Start completed | INFO |
| Error Code Mapping configurations loaded | ocnef-monitoringevents | Loaded Error Code Mapping Configuration | INFO |
| Error Code Mapping configurations loaded | ocnef-monitoringevents | Loaded Error Reason Mapping Configuration | INFO |
| Error Code Mapping configurations loaded | ocnef-monitoringevents | Loaded Error Title Mapping Configuration | INFO |
| Error Code Mapping configurations loaded | ocnef-monitoringevents | Loaded Error Type Mapping Configuration | INFO |
| Check if Ports successfully listening | ocnef-monitoringevents | Netty started on port | INFO |
| Check for message received | ocnef-monitoringevents | Entering SubscriptionsApiController | DEBUG |
| Check for message exception | ocnef-monitoringevents | Exception when executing | DEBUG |
| URI Pattern not supported | ocnef-monitoringevents | None match pattern found for URL | WARN |
| Check if Ports successfully listening | nrf-client-service | Undertow started on port(s) | INFO |
| Pod exit | ocnef-monitoringevents | HikariPool-1 - Shutdown completed | INFO |
| DB username/ DB password invalid | ocnef-monitoringevents | Access denied for user | WARN |
| Registration with NRF failed | nrf-client-service | Register failed | ERROR |
| De registration with NRF successful | nrf-client-service | Deregister completed successfully | INFO |
| De registration with NRF failed | nrf-client-service | Deregister failed | ERROR |
| NF Profile update failed | nrf-client-service | Update failed | ERROR |
2.2 Collecting Logs
This section describes the steps to collect logs from PODs and containers. Perform the following steps:
- Run the following command to get the PODs details:
kubectl -n <namespace_name> get pods - Collect the logs from the specific pods or containers:
kubectl logs <podname> -n <namespace> -c <containername> - Store the log in a file using the following command:
kubectl logs <podname> -n <namespace> <filename> - (Optional) You can also use the following commands for the log stream with file redirection starting with last 100 lines of log:
kubectl logs <podname> -n <namespace> -f --tail <number of lines> > <filename>
For more information on how to collect the logs, see Oracle Communication Cloud Native Core, Data Collector Guide.
2.3 Understanding Logs
This chapter explains the logs you need to look into to handle different NEF debugging issues.
For more information on how to collect the logs, see Oracle Communication Cloud Native Core, Data Collector Guide.
This section provides log level attribute details for following services:
- monitoringevents
- fivegcagent
- expgw-apirouter
- expgw-afmgr
Sample Logs
Sample log statement monitoringevents:
{
"instant":
{
"epochSecond":1645710158,
"nanoOfSecond":717564000
},
"thread":"reactor-http-nio-2",
"level":"DEBUG",
"loggerName":"com.oracle.cne.nef.efc.svc.me.api.SubscriptionsApiController",
"message":"processSubscriptionPost: IN",
"endOfBatch":false,
"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger",
"contextMap":{},"threadId":49,
"threadPriority":5,
"messageTimestamp":"2022-02-24T19:12:38.717+0530",
"application":"${env:CONFIGMAP_NAME}",
"microservice":"${env:CONFIGMAP_NAME}",
"vendor":"oracle",
"namespace":"${env:K8S_NAMESPACE}",
"node":"${env:K8S_NODE}",
"pod":"${env:POD_NAME}",
"instanceType":"prod"
}
Sample log statement fivegcagent:
{
"instant":
{
"epochSecond":1645710159,
"nanoOfSecond":347655000},
"thread":"reactor-http-nio-4",
"level":"DEBUG",
"loggerName":"com.oracle.cne.nef.efc.common.fivegcagent.api.FivegcAgentApiController",
"message":"Entering FivegcAgentApiController::processReceiverMessage",
"endOfBatch":false,
"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger",
"contextMap":{},
"threadId":65,
"threadPriority":5,
"messageTimestamp":"2022-02-24T19:12:39.347+0530",
"application":"${env:CONFIGMAP_NAME}",
"microservice":"${env:CONFIGMAP_NAME}",
"vendor":"oracle",
"namespace":"${env:K8S_NAMESPACE}",
"node":"${env:K8S_NODE}",
"pod":"${env:POD_NAME}",
"instanceType":"prod"
}Sample log statement expgw-apirouter:
{
"instant":
{
"epochSecond":1645679731,
"nanoOfSecond":795110229
},
"thread":"main",
"level":"WARN",
"loggerName":"io.opentracing.contrib.spring.tracer.configuration.TracerAutoConfiguration",
"message":"Tracer bean is not configured! Switching to NoopTracer",
"endOfBatch":false,
"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog",
"contextMap":{},
"threadId":1,
"threadPriority":5,
"messageTimestamp":"2022-02-24T05:15:31.795+0000",
"application":"ocnef-expgw-apirouter",
"microservice":"ocnef-expgw-apirouter",
"vendor":"oracle",
"namespace":"nef",
"node":"prowl-k8s-node-10",
"pod":"ocnef-expgw-apirouter-664b9bc79-5rv5n",
"instanceType":"prod"
}Sample log statement expgw-afmgr:
{
"instant":
{
"epochSecond":1645710925,
"nanoOfSecond":734244575
},
"thread":"main",
"level":"INFO",
"loggerName":"com.zaxxer.hikari.HikariDataSource",
"message":"HikariPool-1 - Starting...",
"endOfBatch":false,
"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger",
"contextMap":{},
"threadId":1,"threadPriority":5,
"messageTimestamp":"2022-02-24T13:55:25.734+0000",
"application":"ocnef-expgw-afmgr",
"microservice":"ocnef-expgw-afmgr",
"vendor":"oracle",
"namespace":"nef",
"node":"prowl-k8s-node-1",
"pod":"ocnef-expgw-afmgr-85b7fc5cb5-x9lfm",
"instanceType":"prod"
}Table 2-2 Log Attribute Details
| Log Attribute | Details | Sample Value | Data Type |
|---|---|---|---|
| instant | Epoch time
Note: It is group of two values epochSecond and nanoOfSecond |
{"epochSecond":1604655402,"nanoOfSecond":946649000} | Object |
| thread | Logging Thread Name | "XNIO-1 task-1" | String |
| level | Log Level of the printed log | "WARN" | String |
| loggerName | Class or Module which printed the log | "com.oracle.cne.nef.efc.svc.me.api.SubscriptionsApiController" |
String |
| message | Message related to the log providing brief details
Note: Indicates that no NFProfiles found for mentioned search query |
"processSubscriptionPost: IN" |
String |
| endOfBatch |
Log4j2 Internal Default from log4j2: false |
false | boolean |
| loggerFqcn |
Log4j2 Internal Fully Qualified class name of logger module |
org.apache.logging.log4j.spi.AbstractLogger | String |
| threadId | Thread Id generated internally by Log4j2 | 47 | Integer |
| messageTimestamp | Timestamp of log from application container. Format: yyyy-MM-dd'T'HH:mm:ss.SSSZ | "2020-11-06T09:36:42.946+0000" | String |
This section provides log level attribute details for ingress gateway service:
Sample log statement ingressgateway:
{
"thread":"ingress-h2c-epoll-3",
"level":"DEBUG",
"loggerName":"ocpm.cne.gateway.filters.PreGatewayFilter",
"message":"Exiting PreGatewayFilter",
"endOfBatch":false,
"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger",
"instant":{"epochSecond":1604650229,
"nanoOfSecond":4993000},
"contextMap":
{
"hostname":"ocnrf-ingressgateway-69f6544b8d-cdbgx",
"ingressTxId":"ingress-tx-1087436877",
"ocLogId":"1604650229002_72_ocnrf-ingressgateway-69f6544b8d-cdbgx"
},
"threadId":72,
"threadPriority":5,
"messageTimestamp":"2020-11-06 08:10:29.004",
"ocLogId":"1604650229002_72_ocnrf-ingressgateway-69f6544b8d-cdbgx",
"pod":"ocnrf-ingressgateway-69f6544b8d-cdbgx",
"processId":"1",
"instanceType":"prod",
"ingressTxId":"ingress-tx-1087436877"
}Table 2-3 Log Attribute Details for ingressgateway
| Log Attribute | Details | Sample Value | Data Type | Notes |
|---|---|---|---|---|
| thread | Logging Thread Name | "ingress-h2c-epoll-3" | String | |
| level | Log Level of the log printed | "DEBUG" | String | |
| loggerName | Class/Module which printed the log | "ocpm.cne.gateway.filters.PreGatewayFilter" | String | |
| message | Message related to the log providing brief details | "Exiting PreGatewayFilter" | String | Indicates that the method PreGatewayFilter is being exited. |
| endOfBatch |
Log4j2 Internal Default from log4j2: false |
false | boolean | |
| loggerFqcn |
Log4j2 Internal Fully Qualified class name of logger module |
org.apache.logging.log4j.spi.AbstractLogger | String | |
| instant | Epoch timestamp | {"epochSecond":1604650229,"nanoOfSecond":4993000} | Object | It is group of two values epochSecond and nanoOfSecond |
| contextMap | contents of log4j ThreadContext map | {"hostname":"ocnrf-ingressgateway-69f6544b8d-cdbgx", "ingressTxId":"ingress-tx-1087436877", "ocLogId":"1604650229002_72_ocnrf-ingressgateway-69f6544b8d-cdbgx"} | Object | |
| threadId | Thread Id generated internally by Log4j2 | 72 | Integer | |
| threadPriority | Thread Priority set internally by Log4j2 | 5 | Integer | |
| messageTimestamp | Timestamp of log from application container. Format: yyyy-MM-dd'T'HH:mm:ss.SSSZ | "2020-11-06 08:10:29.004" | String | |
| ocLogId | End to End Log Identifier across the NRF microservices. | "1604650229002_72_ocnrf-ingressgateway-69f6544b8d-cdbgx" | String | Helps to correlate the logs across the microservices in NRF application |
| pod | Pod Name | "ocnrf-ingressgateway-69f6544b8d-cdbgx" | String | |
| processId | Process ID internally assigned | "1" | String | |
| instanceType | Instance type | "prod" | String | |
| ingressTxId | Transaction id that is added to log4j ThreadContext map and is unique to every transaction | "ingress-tx-1087436877" | String |
This section provides log level attribute details for the appinfo service:
Sample log statement appinfo:
{
"name": "gunicorn.access",
"message": "127.0.0.1 - - [29/Oct/2020:18:39:35 +0000]\"GET /v1/liveness HTTP/1.1\" 200 16 \"-\"\"kube-probe/1.17+\"",
"level": "INFO",
"filename":"glogging.py",
"lineno": 344,
"module": "glogging",
"func":"access",
"thread":"MainThread",
"created": "2020-10-29T18:39:35.799448"
}Table 2-4 Log Attribute Details for egressgateway
| Log Attribute | Details | Sample Value | Data Type |
|---|---|---|---|
| name | Logger name | "gunicorn.access" | String |
| message | Message related to the log providing brief details | "message": "127.0.0.1 - - [29/Oct/2020:18:39:35 +0000] \"GET /v1/liveness HTTP/1.1\" 200 16 \"-\" \"kube-probe/1.17+\"" | String |
| level | Log Level of the log printed | "INFO" | String |
| filename | File name which generates the log | "glogging.py" | String |
| lineno | Line number | 344 | Integer |
| module | Python module name | "glogging" | String |
| func | Function Name | "func" | String |
| thread | Thread Name | "MainThread" | String |
| messageTimestamp | Timestamp of log from application container. Format: yyyy-MM-dd'T'HH:mm:ss.SSSZ | "2020-10-29T18:39:35.799448" | String |
Common Useful log attributes
The below log attributes will be available only through Kibana. These attribute names are part of Kubernetes labels which are added in each NEF POD.
Table 2-5 Common useful log attributes
| Log Attribute | Details | Sample Value | Data Type | Notes |
|---|---|---|---|---|
| application | NF Type | "ocnef" | String | complete attribute name in Kibana:
kubernetes.labels.application |
| microservice |
Kubernetes service name Format: <Helm-Release-Name>-<Microservice-Name> |
(assuming the <Helm-Release-Name> is "nefnorth" "nefnorth-monitoringevents" "nefnorth-fivegcagent" "nefnorth-expgw-afmgr" "nefnorth-expgw-apirouter" "nefnorth-appinfo" "nefnorth-ext-ingress-gateway" "nefnorth-ext-egress-gateway" |
String | |
| engVersion | Engineering version | "1.9.0" | String | |
| mktgVersion | Marketing version | "1.9.1.0.0" | String | |
| vendor | Vendor Name | "Oracle" | Integer |