2 Logs

This chapter explains the process to retrieve the logs and status that can be used for effective troubleshooting. Logs register system events along with their date and time of occurrence. They also provide important details about a chain of events that could have led to an error or problem.

2.1 Log Levels

This section provides information on log levels supported by Oracle Communications Cloud Native Core, Network Slice Selection Function (NSSF).

A log level helps in defining the severity level of a log message. Using this information, the logs can be filtered based on the system requirements. For example, if you want to filter the critical information about your system from the informational log messages, set a filter to view messages with only WARN log level in Kibana.

Supported Log Levels

For NSSF, you can set the log level for a microservice to any of the following valid values:

• TRACE: This log level describes events showing step-by-step execution of your code. You may ignore this log during the standard operation. However, it may be useful during extended debugging sessions.
• DEBUG: This log level is useful while debugging an event and when more granular information is required.
• INFO: This is the standard log level indicating an event has occurred. For example, when the application enters a certain state.
• WARN: This log level indicates that some unexpected behavior has occurred in the application, but the application has not failed. Even with this unexpected behavior, the code can continue running.
• ERROR: This log level indicates an issue has occurred in the application that prevents one or more functionalities from working correctly.

Configuring Log Levels

Use the logLevel parameter to view and update the logging level configurations in the ocnssf-custom-values.yaml file. For example, see the following snippet from the ocnssf-custom-values.yaml file for Helm test hook related configuration.

test:
    config:
      logLevel: WARN
      timeout: 120   

Examples with various Log Level Values

The following section provides examples for log messages containing different log level values:

For level ERROR:

{
  "instant": {
    "epochSecond": 1635408023,
    "nanoOfSecond": 739237045
  },
  "thread": "XNIO-1 task-1",
  "level": "ERROR",
  "loggerName": "com.oracle.cgbu.cne.nssf.nsavailability.service.NsAvailabilityService",
  "message": "Unsupported PLMN/S received , supported plmn list: [Plmn [mcc=100, mnc=101], Plmn [mcc=100, mnc=02], Plmn [mcc=101, mnc=101], Plmn [mcc=102, mnc=102], Plmn [mcc=100, mnc=100], Plmn [mcc=200, mnc=200], Plmn [mcc=100, mnc=102], Plmn [mcc=100, mnc=001], Plmn [mcc=103, mnc=103]]",
  "endOfBatch": false,
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "contextMap": {
    "ocLogId": "1635408023716_15768_ocnssf-ingress-gateway-6d554c596-q86hp"
  },
  "threadId": 1446,
  "threadPriority": 5,
  "ts": "2021-10-28 08:00:23.739+0000",
  "ocLogId": "1635408023716_15768_ocnssf-ingress-gateway-6d554c596-q86hp",
  "pod": "ocnssf-nsavailability-5f858b7745-mk2pv",
  "processId": "1",
  "vendor": "Oracle",
  "application": "ocnssf",
  "engVersion": "1.0.0",
  "mktgVersion": "1.0.0.0.0",
  "microservice": "nsavailability",
  "namespace": "ocnssf",
  "node_name": "remote-setup-kamal"
}

For level INFO:

{
  "instant": {
    "epochSecond": 1635408080,
    "nanoOfSecond": 653030110
  },
  "thread": "scheduling-1",
  "level": "INFO",
  "loggerName": "com.oracle.common.scheduler.ReloadConfig",
  "message": "All configurations intact, no updation in the configuration",
  "endOfBatch": false,
  "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
  "contextMap": {},
  "threadId": 28,
  "threadPriority": 5,
  "ts": "2021-10-28 08:01:20.653+0000",
  "ocLogId": "${ctx:ocLogId}",
  "pod": "ocnssf-nsavailability-5f858b7745-mk2pv",
  "processId": "1",
  "vendor": "Oracle",
  "application": "ocnssf",
  "engVersion": "1.0.0",
  "mktgVersion": "1.0.0.0.0",
  "microservice": "nsavailability",
  "namespace": "ocnssf",
  "node_name": "remote-setup-kamal"
}

2.2 Collecting Logs

This section describes how to collect logs from pods and containers. The steps are as follows:

Collect the NSSF Logs to check the error scenarios

Problem: The error scenarios are checked by collecting the NSSF logs.

Solution:

Perform the following steps to get the logs from nssf:

1. Run the following command to get the pod details:

   kubectl -n <ocnssf_namespace> get pods

2. Collect the logs from the specific pods or containers:

   kubectl logs <podname> -n <ocnssf_namespace> -c <containername>

3. Store the log in a file using the following command:

   kubectl logs <podname> -n <ocnssf_namespace> > <filename>

   Example:

   kubectl logs ocnssf-nsselection-57cff5665c-skk4l -n ocnssf > ocnssf_logs1.log

4. (Optional) Run the following command for the log stream with file redirection starting with last 100 lines of log:

   kubectl logs <podname> -n <ocnssf_namespace> -f --tail <number of lines> > <filename>

For more information on kubectl commands, see Kubernetes website.

2.3 Understanding Logs

This section provides information on how to read logs for various services of NSSF in Kibana.

NSSF Log Attributes

The log message format is same for all the NSSF services. The following table describes key attributes of a log message:

Table 2-1 NSSF Log Attributes

Attribute Name	Description	Details
instant	Timestamp details for when the log event occurred. Note: Contains both epoch seconds and nanoseconds for precision.	Data Type: Object
thread	Name of the thread executing the log. Note: Useful in multi-threaded applications.	Data Type: String
level	Log severity level. Note: Common levels: DEBUG, INFO, WARN, ERROR.	Data Type: String
loggerName	Fully qualified logger class name. Note: Identifies the logging component.	Data Type: String
message	Log message details. Note: Describes the issue being logged.	Data Type: String
endOfBatch	Indicates if this is the last log in batch processing. Note: Often used in batch logging frameworks.	Data Type: Boolean
loggerFqcn	Fully qualified class name of the logger implementation. Note: Helps identify the logging backend.	Data Type: String
contextMap	Contextual key-value pairs (if any). Note: Stores contextual metadata.	Data Type: Object
threadId	Numeric ID of the thread executing the log. Note: Helps trace logs in multi-threaded processes.	Data Type: Integer
threadPriority	Priority level of the thread. Note: Helps in debugging thread behavior.	Data Type: Integer
ts	Timestamp when log entry was created. Note: Provides human-readable log time.	Data Type: String (ISO 8601)
ocLogId	Unique identifier for log entry. Note: Used for tracing logs.	Data Type: String
pod	Name of the Kubernetes pod. Note: Helps track logs in containerized environments.	Data Type: String
processId	Process ID of the application instance. Note: Identifies application instance.	Data Type: String
vendor	Vendor information. Note: Identifies software vendor.	Data Type: String
application	Application name. Note: Identifies application logging the message.	Data Type: String
engVersion	Engineering version of the application. Note: Engineering release version.	Data Type: String
mktgVersion	Marketing version of the application. Note: Marketing release version.	Data Type: String
microservice	Name of the microservice component. Note: Helps identify the specific service.	Data Type: String
namespace	Kubernetes namespace. Note: Groups services within Kubernetes.	Data Type: String
node_name	Name of the Kubernetes node where the pod is running. Note: Helps in tracking node-level deployment.	Data Type: String

Sample log statement of nsconfig:

{
    "instant": {
        "epochSecond": 1645187968,
        "nanoOfSecond": 864222104
    },
    "thread": "main",
    "level": "INFO",
    "loggerName": "com.oracle.cgbu.cne.nssf.nsconfig.Application",
    "message": "Started Application in 21.489 seconds (JVM running for 23.688)",
    "endOfBatch": false,
    "loggerFqcn": "org.apache.commons.logging.LogAdapter$Log4jLog",
    "contextMap": {},
    "threadId": 1,
    "threadPriority": 5,
    "ts": "2022-02-18 12:39:28.864+0000",
    "ocLogId": "${ctx:ocLogId}",
    "pod": "ocnssf-nsconfig-85bd697b74-kxrfv",
    "processId": "1",
    "vendor": "Oracle",
    "application": "ocnssf",
    "engVersion": "22.1.1",
    "mktgVersion": "22.1.1.0.0",
    "microservice": "nsconfig",
    "namespace": "nssf",
    "node_name": "bumblebee-k8s-node-15"
}

appinfo Log Attributes

The log message format is same for all the NSSF services. The following table describes key attributes of a log message:

Table 2-2 appinfo Log Attributes

Attribute Name	Description	Details
name	Logger name that identifies the source of the log. Note: Helps categorize logs from different components.	Data Type: String
message	The main log message providing relevant details about the event. Note: Typically contains request details in access logs.	Data Type: String
level	Severity level of the log. Note: Common levels include DEBUG, INFO, WARN, ERROR.	Data Type: String
filename	Name of the file where the log was generated. Note: Helps trace the log source in the codebase.	Data Type: String
lineno	Line number in the source file where the log was generated. Note: Useful for debugging.	Data Type: Integer
module	Python module name associated with the log. Note: Helps identify the exact module in multi-module applications.	Data Type: String
func	Name of the function where the log was generated. Note: Useful for debugging and tracing log sources.	Data Type: String
thread	Name of the thread executing the log. Note: Important for multi-threaded applications.	Data Type: String
messageTimestamp	Timestamp when the log was created, following the format yyyy-MM-dd'T'HH:mm:ss.SSSZ. Note: Provides an accurate event timeline.	Data Type: String

Sample log statement of appinfo:

{
  "name": "gunicorn.access",
  "message": "::ffff:10.244.1.106 - - [21/May/2022:13:54:29 +0000] \"GET /status/category/sepp HTTP/1.1\" 200 7 \"-\" \"okhttp/3.14.9\"",
  "level": "INFO",
  "filename": "glogging.py",
  "lineno": 349,
  "module": "glogging",
  "func": "access",
  "thread": "MainThread",
  "messageTimestamp": "2022-05-21T13:54:29.385+0000"
}

perf-info Log Attributes

Table 2-3 perf-info Log Attributes

Attribute Name	Description <br> Note	Details
name	Logger name or category. Note: Identifies the logging component.	Data Type: String
message	Log message details. Note: Describes the issue being logged.	Data Type: String
level	Log severity level. Note: Common levels: DEBUG, INFO, WARN, ERROR.	Data Type: String
filename	Name of the source file where the log was generated. Note: Helps trace log source.	Data Type: String
lineno	Line number in the source file. Note: Useful for debugging.	Data Type: Integer
module	Python module where the log originated. Note: Indicates module name.	Data Type: String
func	Function name that generated the log. Note: Helps track function-level logging.	Data Type: String
thread	Name of the thread handling the request. Note: Useful in multi-threaded applications.	Data Type: String
messageTimestamp	Timestamp when the log entry was created. Note: Provides human-readable log time.	Data Type: String (ISO 8601)

Sample log statement of perf-info:

{
  "name": "stat_helper",
  "message": "Failed to reach prometheus",
  "level": "ERROR",
  "filename": "stat_helper.py",
  "lineno": 106,
  "module": "stat_helper",
  "func": "get_db_param",
  "thread": "MainThread",
  "messageTimestamp": "2022-05-21T13:57:39.639+0000"
}

config-server Log Attributes

Table 2-4 config-server Log Attributes

Attribute Name	Description <br> Note	Details
instant	Timestamp details for when the log event occurred. Note: Contains both epoch seconds and nanoseconds for precision.	Data Type: Object
thread	Name of the thread executing the log. Note: Useful in multi-threaded applications.	Data Type: String
level	Log severity level. Note: Common levels: DEBUG, INFO, WARN, ERROR.	Data Type: String
loggerName	Fully qualified logger class name. Note: Identifies the logging component.	Data Type: String
message	Log message details. Note: Describes the issue being logged.	Data Type: String
endOfBatch	Indicates if this is the last log in a batch processing. Note: Often used in batch logging frameworks.	Data Type: Boolean
loggerFqcn	Fully qualified class name of the logger implementation. Note: Helps identify the logging backend.	Data Type: String
threadId	Numeric ID of the thread executing the log. Note: Helps trace logs in multi-threaded processes.	Data Type: Integer
threadPriority	Priority level of the thread. Note: Helps in debugging thread behavior.	Data Type: Integer
messageTimestamp	Timestamp when the log entry was created. Note: Provides human-readable log time.	Data Type: String (ISO 8601)

Sample log statement of config-server:

{
    "instant": {
        "epochSecond": 1653140996,
        "nanoOfSecond": 895472496
    },
    "thread": "main",
    "level": "INFO",
    "loggerName": "ocnssf.cne.common.metrics.cgroup.CgroupMetricsHelper",
    "message": "Creating cgroup metric finder",
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
    "threadId": 1,
    "threadPriority": 5,
    "messageTimestamp": "2022-05-21T13:49:56.895+0000"
}

nrf-client-nfdiscovery Log Attributes

Table 2-5 nrf-client-nfdiscovery Log Attributes

Attribute Name	Description <br> Note	Details
instant	Timestamp details for when the log event occurred. Note: Contains both epoch seconds and nanoseconds for precision.	Data Type: Object
thread	Name of the thread executing the log. Note: Useful in multi-threaded applications.	Data Type: String
level	Log severity level. Note: Common levels: DEBUG, INFO, WARN, ERROR.	Data Type: String
loggerName	Fully qualified logger class name. Note: Identifies the logging component.	Data Type: String
message	Log message details. Note: Describes the issue being logged.	Data Type: String
endOfBatch	Indicates if this is the last log in a batch processing. Note: Often used in batch logging frameworks.	Data Type: Boolean
loggerFqcn	Fully qualified class name of the logger implementation. Note: Helps identify the logging backend.	Data Type: String
threadId	Numeric ID of the thread executing the log. Note: Helps trace logs in multi-threaded processes.	Data Type: Integer
threadPriority	Priority level of the thread. Note: Helps in debugging thread behavior.	Data Type: Integer
source	Details of the Java class, method, and file where the log was generated. Note: Contains exact file, method, and line number for debugging.	Data Type: Object
messageTimestamp	Timestamp when the log entry was created. Note: Provides human-readable log time.	Data Type: String (ISO 8601)

Sample log statement of nrf-client-nfdiscovery:

{
    "instant": {
        "epochSecond": 1653141021,
        "nanoOfSecond": 819399951
    },
    "thread": "main",
    "level": "WARN",
    "loggerName": "com.oracle.cgbu.cnc.nrf.util.NrfClientProperties",
    "message": "getHttpsProxyPort():Invalid httpsProxyPort",
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
    "threadId": 1,
    "threadPriority": 5,
    "source": {
        "class": "com.oracle.cgbu.cnc.nrf.util.NrfClientProperties",
        "method": "getHttpsProxyPort",
        "file": "NrfClientProperties.java",
        "line": 260
    },
    "messageTimestamp": "2022-05-21T13:50:21.819+0000"
}

nrf-client-nfmanagement Log Attributes

Table 2-6 nrf-client-nfmanagement Log Attributes

Attribute Name	Description	Details
instant	Timestamp details for when the log event occurred. Note: Contains both epoch seconds and nanoseconds for precision.	Data Type: Object
thread	Name of the thread executing the log. Note: Useful in multi-threaded applications.	Data Type: String
level	Log severity level. Note: Common levels: DEBUG, INFO, WARN, ERROR.	Data Type: String
loggerName	Fully qualified logger class name. Note: Identifies the logging component.	Data Type: String
message	Log message details. Note: Describes the issue being logged.	Data Type: String
endOfBatch	Indicates if this is the last log in batch processing. Note: Often used in batch logging frameworks.	Data Type: Boolean
loggerFqcn	Fully qualified class name of the logger implementation. Note: Helps identify the logging backend.	Data Type: String
threadId	Numeric ID of the thread executing the log. Note: Helps trace logs in multi-threaded processes.	Data Type: Integer
threadPriority	Priority level of the thread. Note: Helps in debugging thread behavior.	Data Type: Integer
source	Details of the Java class, method, and file where the log was generated. Note: Contains exact file, method, and line number for debugging.	Data Type: Object
messageTimestamp	Timestamp when the log entry was created. Note: Provides human-readable log time.	Data Type: String (ISO 8601)

Sample log statement of nrf-client-nfmanagement:

{
    "instant": {
        "epochSecond": 1653141225,
        "nanoOfSecond": 57167090
    },
    "thread": "taskScheduler-2",
    "level": "WARN",
    "loggerName": "com.oracle.cgbu.cnc.nrf.NRFManagement",
    "message": "NfServices is not present inNfProfile.",
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.slf4j.Log4jLogger",
    "threadId": 27,
    "threadPriority": 5,
    "source": {
        "class": "com.oracle.cgbu.cnc.nrf.NRFManagement",
        "method": "setPerformance",
        "file": "NRFManagement.java",
        "line": 1758
    },
    "messageTimestamp": "2022-05-21T13:53:45.057+0000"
}

Ingress Gateway Log Attributes

Table 2-7 Ingress Gateway Log Attributes

Attribute Name	Description	Details
thread	Logging Thread Name.	Data Type: String
level	Log Level of the log printed.	Data Type: String
loggerName	Class or Module which printed the log.	Data Type: String
message	Message related to the log providing brief details. Note: Indicates that the method PreGatewayFilter is being exited.	Data Type: String
endOfBatch	Log4j2 Internal default flag. Note: Default from Log4j2: false.	Data Type: Boolean
loggerFqcn	Log4j2 Internal - Fully Qualified class name of the logger module.	Data Type: String
instant	Epoch timestamp. Note: It consists of two values - epochSecond and nanoOfSecond for high precision.	Data Type: Object
contextMap	Contents of Log4j ThreadContext map.	Data Type: Object
threadId	Thread ID generated internally by Log4j2.	Data Type: Integer
threadPriority	Thread Priority set internally by Log4j2.	Data Type: Integer
messageTimestamp	Timestamp of log from application container. Note: Format: yyyy-MM-dd'T'HH:mm:ss.SSSZ.	Data Type: String
ocLogId	End-to-End Log Identifier across the NSSF microservices. Note: Helps to correlate logs across microservices in NSSF application.	Data Type: String
pod	Pod Name.	Data Type: String
processId	Process ID internally assigned.	Data Type: String
instanceType	Instance type.	Data Type: String
ingressTxId	Transaction ID that is added to Log4j ThreadContext map and is unique to every transaction.	Data Type: String

Sample log statement of Ingress Gateway:

{
    "thread": "ingress-h2c-epoll-3",
    "level": "DEBUG",
    "loggerName": "ocnssf.cne.gateway.filters.PreGatewayFilter",
    "message": "Exiting PreGatewayFilter",
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
    "instant": {
        "epochSecond": 1604650229,
        "nanoOfSecond": 4993000
    },
    "contextMap": {
        "hostname": "ocnssf-ingressgateway-69f6544b8d-cdbgx",
        "ingressTxId": "ingress-tx-1087436877",
        "ocLogId": "1604650229002_72_ocnssf-ingressgateway-69f6544b8d-cdbgx"
    },
    "threadId": 72,
    "threadPriority": 5,
    "messageTimestamp": "2020-11-06 08:10:29.004",
    "ocLogId": "1604650229002_72_ocnssf-ingressgateway-69f6544b8d-cdbgx",
    "pod": "ocnssf-ingressgateway-69f6544b8d-cdbgx",
    "processId": "1",
    "instanceType": "prod",
    "ingressTxId": "ingress-tx-1087436877"
}

Egress Gateway Log Attributes

Table 2-8 Egress Gateway Log Attributes

Attribute Name	Description	Details
thread	Logging Thread Name.	Data Type: String
level	Log Level of the log printed.	Data Type: String
loggerName	Class or Module which printed the log.	Data Type: String
message	Message related to the log providing brief details. Note: Contains property name and value information.	Data Type: String
endOfBatch	Log4j2 Internal default flag. Note: Default from Log4j2: false.	Data Type: Boolean
loggerFqcn	Log4j2 Internal - Fully Qualified class name of the logger module.	Data Type: String
instant	Epoch timestamp. Note: It consists of two values - epochSecond and nanoOfSecond for high precision.	Data Type: Object
contextMap	Elements in Log4j ThreadContext map.	Data Type: Object
threadId	Thread ID generated internally by Log4j2.	Data Type: Integer
threadPriority	Thread Priority set internally by Log4j2.	Data Type: Integer
messageTimestamp	Timestamp of log from application container. Note: Format: yyyy-MM-dd'T'HH:mm:ss.SSSZ.	Data Type: String
ocLogId	End-to-End Log Identifier across the NSSF microservices. Note: Helps to correlate logs across microservices in NSSF application.	Data Type: String
pod	Name of the egress pod.	Data Type: String
processId	Process ID internally assigned.	Data Type: String
instanceType	Instance type.	Data Type: String
egressTxId	Transaction ID that is added to Log4j ThreadContext map and is unique to every transaction.	Data Type: String

Sample log statement of Egress Gateway:

{
    "thread": "main",
    "level": "DEBUG",
    "loggerName": "ocnssf.cne.gateway.config.ScpDynamicBeanConfiguration",
    "message": "Property name: server.port and value: 8080",
    "endOfBatch": false,
    "loggerFqcn": "org.apache.logging.log4j.spi.AbstractLogger",
    "instant": {
        "epochSecond": 1604564777,
        "nanoOfSecond": 135977000
    },
    "contextMap": {},
    "threadId": 1,
    "threadPriority": 5,
    "messageTimestamp": "2020-11-05 08:26:17.135",
    "ocLogId": "1604650229002_72_ocnssf-ingressgateway-69f6544b8d-cdbgx",
    "pod": "ocnssf-egressgateway-69f6544b8d-cdbgx",
    "processId": "1",
    "instanceType": "prod",
    "egressTxId": "egress-tx-1087436877"
}