2 Logs

This chapter explains the process to retrieve the logs and status that can be used for effective troubleshooting.

2.1 Log Levels

Logs register system events along with their date and time of occurrence. They also provide important details about a chain of events that could have led to an error or problem.

A log level helps in defining the severity level of a log message. For OCSEPP, the log level of a microservice can be set to any one of the following valid values:

  • TRACE: A log level that describes events, as a step by step execution of code. This can be ignored during the standard operation, but may be useful during extended debugging sessions.
  • DEBUG: A log level used for events during software debugging when more granular information is needed.
  • INFO: A standard log level indicating that something has happened, an application has entered a certain state, etc.
  • WARN: A log level indicates that something unexpected has happened in the application, a problem, or a situation that might disturb one of the processes. But this does not mean that the application has failed. The WARN level should be used in situations that are unexpected, but the code can continue to work.
  • ERROR: A log level that should be used when an application hits an issue preventing one or more functionalities from functioning.

Using this information, the logs can be filtered based on the system requirements. For instance, if you want to filter the critical information about your system from the informational log messages, set a filter to view messages with only WARN log level in Kibana.

2.2 Collecting Logs

This section describes the steps to collect logs from PODs and containers. Perform the following steps:

  1. Run the following command to get the PODs details:
    $ kubectl -n <namespace_name> get pods
  2. Collect the logs from the specific pods or containers:

    From the pod:

    
$ kubectl logs <podname> -n <namespace>
    From the container:
    
$ kubectl logs <podname> -c <container name> -n <namespace>

    Example:

    From the pod:
    $ kubectl logs ocsepp-release-xxxxxxxxxx-xxxxx -n seppsvc
    From the container:
    $ kubectl logs ocsepp-release-n32-egress-gateway-xxxxx -c n32-egress-gateway -n
        seppsvc
  3. Store the log in a file using the following command:
    $ kubectl logs <podname> -n <namespace>  > <filename>
    Example:
    $ kubectl logs ocsepp-release-xxxxxxxxxx-xxxxx -n seppsvc > logs.txt
  4. (Optional) You can also use the following commands for the log stream with file redirection starting with last 100 lines of log:
    $ kubectl logs <podname> -n <namespace> -f --tail <number of lines> > <filename>
    Example:
    $ kubectl logs ocsepp-release-xxxxxxxxxx-xxxxx -n seppsvc -f --tail 100 > logs.txt

For more information on kubectl commands, see Kubernetes website.

2.3 Using Logs

This section explains the logs you need to look at, to handle different SEPP debugging issues.

For more information on how to collect the logs, see Oracle Communication Cloud Native Core Data Collector Guide.

This section provides log level attribute details for the following services:
  • config-mgr-svc
  • cn32c-svc
  • pn32c-svc
  • cn32f-svc
  • pn32f-svc
  • nrf-client-nfmanagement
  • nrf-client-nfdiscovery
  • app-info
  • perf-info
  • config-server
  • n32-ingress-gateway
  • n32-egress-gateway
  • plmn-ingress-gateway
  • plmn-egress-gateway
  • nf-mediation

Sample Logs

Sample log statement config-mgr-svc :

{"instant":{"epochSecond":1636703617,"nanoOfSecond":449636327},"thread":"XNIO-1 task-4","level":"DEBUG","loggerName":"org.springframework.web.servlet.DispatcherServlet","message":"Completed 200 OK","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":40,"threadPriority":5,"ts":"21-11-12 07:53:449.037+0000","instanceType":"prod","processId":"1","ocLogId":"${ctx:ocLogId}","vendor":"oracle"}
Sample log statement cn32c-svc :
{"instant":{"epochSecond":1636456524,"nanoOfSecond":315917989},"thread":"sepp-cn32c-thread-2","level":"DEBUG","loggerName":"com.oracle.cgbu.cne.ocsepp.client.Http2Client","message":"Http2 Client trying to connect with URL: http://ocsepp-release-config-mgr-svc:9090/cn32c/Handshake-success","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":27,"threadPriority":5,"ts":"21-11-09 11:15:315.024+0000","instanceType":"prod","processId":"1","ocLogId":"${ctx:ocLogId}","vendor":"oracle"}

Sample log statement pn32c-svc :

{"instant":{"epochSecond":1636455735,"nanoOfSecond":301984153},"thread":"main","level":"INFO","loggerName":"com.oracle.cgbu.cne.ocsepp.pn32c.Pn32cApplication","message":"Starting Pn32cApplication using Java 16.0.1 on ocsepp-release-pn32c-svc-7fb7d866c6-sczjg with PID 1(/ocsepp-pn32c-svc.jar started by seppuser in /)","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":1,"threadPriority":5,"ts":"21-11-09 11:02:301.015+0000","instanceType":"prod","processId":"1","ocLogId":"${ctx:ocLogId}","vendor":"oracle"}

Sample log statement cn32f-svc:

{"instant":{"epochSecond":1636457129,"nanoOfSecond":526138937},"thread":"main","level":"INFO","loggerName":"com.oracle.cgbu.cne.ocsepp.cn32f.Cn32fApplication","message":"Starting Cn32fApplication using Java 16.0.1 on ocsepp-release-cn32f-svc-9b8c6d7c6-dgmqv with PID 1 (/ocsepp-cn32f-svc.jar started by seppuser in /)","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":1,"threadPriority":5,"ts":"21-11-09 11:25:526.029+0000","instanceType":"prod","processId":"1","ocLogId":"${ctx:ocLogId}","vendor":"oracle"}
Sample log statement pn32f-svc :
{"instant":{"epochSecond":1636457721,"nanoOfSecond":692849682},"thread":"main","level":"INFO","loggerName":"com.oracle.cgbu.cne.ocsepp.pn32f.Pn32fApplication","message":"Starting Pn32fApplication using Java 16.0.1 on ocsepp-release-pn32f-svc-85b4b9fd9d-gzp6h with PID 1 (/ocsepp-pn32f-svc.jar started by seppuser in /)","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","threadId":1,"threadPriority":5,"ts":"21-11-09 11:35:692.021+0000","instanceType":"prod","processId":"1","ocLogId":"${ctx:ocLogId}","vendor":"oracle"}
Sample log statement - nrf-client-nfmanagement
{"instant":{"epochSecond":1653141225,"nanoOfSecond":57167090},"thread":"taskScheduler-2","level":"WARN","loggerName":"com.oracle.cgbu.cnc.nrf.NRFManagement","message":"NfServices is not present inNfProfile.","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":27,"threadPriority":5,"source":{"class":"com.oracle.cgbu.cnc.nrf.NRFManagement","method":"setPerformance","file":"NRFManagement.java","line":1758},"messageTimestamp":"2022-05-21T13:53:45.057+0000"}
Sample log statement - nrf-client-nfdiscovery
{"instant":{"epochSecond":1653141021,"nanoOfSecond":819399951},"thread":"main","level":"WARN","loggerName":"com.oracle.cgbu.cnc.nrf.util.NrfClientProperties","message":"getHttpsProxyPort():Invalid httpsProxyPort","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":1,"threadPriority":5,"source":{"class":"com.oracle.cgbu.cnc.nrf.util.NrfClientProperties","method":"getHttpsProxyPort","file":"NrfClientProperties.java","line":260},"messageTimestamp":"2022-05-21T13:50:21.819+0000"}
Sample log statement - app-info
{"name": "gunicorn.access", "message": "::ffff:10.244.1.106 - - [21/May/2022:13:54:29 +0000] \"GET /status/category/sepp HTTP/1.1\" 200 7 \"-\" \"okhttp/3.14.9\"", "level": "INFO", "filename": "glogging.py", "lineno": 349, "module": "glogging", "func": "access", "thread": "MainThread", "messageTimestamp": "2022-05-21T13:54:29.385+0000"}Sample log statement - perf-info
Sample log statement - perf-info
{"name": "stat_helper", "message": "Failed to reach prometheus", "level": "ERROR", "filename": "stat_helper.py", "lineno": 106, "module": "stat_helper","func": "get_db_param", "thread": "MainThread", "messageTimestamp":"2022-05-21T13:57:39.639+0000"}
Sample log statement - config-server
{"instant":{"epochSecond":1653140996,"nanoOfSecond":895472496},"thread":"main","level":"INFO","loggerName":"ocpm.cne.common.metrics.cgroup.CgroupMetricsHelper","message":"Creating cgroup metric finder","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","threadId":1,"threadPriority":5,"messageTimestamp":"2022-05-21T13:49:56.895+0000"}
Sample log statement- nf-mediation
{"instant":{"epochSecond":1660282861,"nanoOfSecond":184032848},"thread":"nf-mediation-thread-16","level":"RULE_TRAIL","loggerName":"com.oracle.cgbu.ocmediation.ruleengine.DroolsRuleEngine","message":"Rule Execution Started","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":43,"threadPriority":5,"ts":"22-08-12 05:41:184.001+0000","namespace":"psepp","node_name":"slave2","pod":"ocsepp-release-nf-mediation-74bd4dc799-d9ks2","instanceType":"prod","processId":"1","ocLogId":"1660282860545_289_ocsepp-release-n32-ingress-gateway-6b744d6b8f-z5qqq","vendor":"oracle"}
Log Attribute Details for n32f, n32c, and config-mgr-svc

Table 2-1 Log Attribute Details for n32f, n32c, and config-mgr-svc

Log Attribute Details Sample Value Data Type
instant Epoch time.

It is the group of two values epochSecond and nanoOfSecond

 {"epochSecond":1604655402,"nanoOfSecond":946649000} Object
thread Logging Thread Name "reactor-http-epoll-2" String
level Log Level of the log printed "DEBUG" String
loggerName Class or module which printed the log "com.oracle.cgbu.cne.ocsepp.pn32f.iointerface.Pn32fSeppAsyncIbInterface" String
message Message related to the log providing brief details.

Indicates that no NFProfiles found for mentioned search query

 "{LoggingRequestDecorator::getBody() Query target-nf-type=AUSF&requester-nf-type=SEPP}" String
endOfBatch

Log4j2 Internal

Default from log4j2: false

 false Boolean
loggerFqcn

Log4j2 Internal

Fully Qualified class name of logger module

 org.apache.logging.log4j.spi.AbstractLogger String
threadId Thread Id generated internally by Log4j2 32 Integer
threadPriority Thread Priority set internally by Log4j2 5 Integer
messageTimestamp Timestamp of log from application container. Format: yyyy-MM-dd'T'HH:mm:ss.SSSZ

Timestamp can be filtered using the following label :ts -> logs in containertimestamp -> logs on Kibana

 "messageTimestamp":"2023-09-01T03:01:24.607+0000" String
instanceType Instance details. Example: dev, prod, qa.

Note: Part of container logs but not in Kibana.

 prod String
processId Process ID internally assigned.

Note: Part of container logs but not in Kibana.

 "1" String
ocLogId End to End Log Identifier across the OCSEPP microservices.

Helps to correlate the logs across the microservices in OCSEPP application.

Note: Part of container logs but not in Kibana.

 ${ctx:ocLogId} String
vendor Vendor Name "oracle" String

Sample Logs for Ingress Gateway

This section provides log level attribute details for following service:

  • n32-ingress-gateway
  • plmn-ingress-gateway
Sample log statement n32-ingress-gateway:
{"instant":{"epochSecond":1643968884,"nanoOfSecond":549874972},"thread":"ingress-h2-epoll-2","level":"DEBUG","loggerName":"org.springframework.cloud.gateway.handler.RoutePredicateHandlerMapping","message":"Route matched: n32c2","endOfBatch":false,"loggerFqcn":"org.apache.commons.logging.LogAdapter$Log4jLog","contextMap":{"ocLogId":"1643968884534_142_ocsepp-release-chandra-n32-ingress-gateway-6dfb6fc446-9phs6"},"threadId":142,"threadPriority":5,"messageTimestamp":"2022-02-04T10:01:24.549+0000","ocLog      Id":"1643968884534_142_ocsepp-release-chandra-n32-ingress-gateway-6dfb6fc446-9phs6","pod":"${ctx:hostname}","processId":"1","instanceType":"prod","ingressTxId":"${ctx:ingressTxId}"}^

Sample log statement plmn-ingress-gateway:
{"instant":{"epochSecond":1643971167,"nanoOfSecond":783195870},"thread":"pool-11-thread-6","level":"DEBUG","loggerName":"com.oracle.common.scheduler.ReloadConfig","message":"Config server URL: http://ocsepp-release-chandra-config-mgr-svc:9090/config/igw/plmn/22.2.2/1","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","contextMap":{},"threadId":79,"threadPriority":5,"messageTimestamp":"2022-02-04T10:39:27.783+0000","ocLogId":"${ctx:ocLogId}","pod":"${ctx:hostname}","processId":"1","instanceType":"prod","ingressTxId":"${ctx:ingressTxId}"}

Table 2-2 Log Attribute Details for Ingress Gateway

Log Attribute Details Sample Value Data Type
thread Logging Thread Name "ingress-h2c-epoll-3" String
level Log Level of the log printed "DEBUG" String
loggerName Class/Module which printed the log "ocpm.cne.gateway.filters.PreGatewayFilter" String
message Message related to the log providing brief details.

Indicates that the method PreGatewayFilter is being exited. 		"Exiting PreGatewayFilter" String
endOfBatch

Log4j2 Internal

Default from log4j2: false

 false boolean
loggerFqcn

Log4j2 Internal

Fully Qualified class name of logger module

 org.apache.logging.log4j.spi.AbstractLogger String
instant Epoch timestamp

It is group of two values epochSecond and nanoOfSecond

 {"epochSecond":1604650229,"nanoOfSecond":4993000} Object
contextMap contents of log4j ThreadContext map {"hostname":"ocsepp-ingressgateway-69f6544b8d-cdbgx", "ingressTxId":"ingress-tx-1087436877", "ocLogId":"1604650229002_72_ocsepp-ingressgateway-69f6544b8d-cdbgx"} Object
threadId Thread Id generated internally by Log4j2 72 Integer
threadPriority Thread Priority set internally by Log4j2 5 Integer
messageTimestamp Timestamp of log from application container. Format: yyyy-MM-dd'T'HH:mm:ss.SSSZ "2020-11-06 08:10:29.004" String
ocLogId End to End Log Identifier across the OCSEPP microservices.

Helps to correlate the logs across the microservices

 "1604650229002_72_ocsepp-ingressgateway-69f6544b8d-cdbgx" String
pod Pod Name "ocsepp-ingressgateway-69f6544b8d-cdbgx" String
processId Process ID internally assigned "1" String
instanceType Instance type "prod" String
ingressTxId Transaction id that is added to log4j ThreadContext map and is unique to every transaction "ingress-tx-1087436877" String
Egress Gateway

This section provides log level attribute details for following services:

  • n32-egress-gateway
  • plmn-egress-gateway
Sample log statement n32-egress-gateway:
{"instant":{"epochSecond":1643968532,"nanoOfSecond":801113787},"thread":"scheduling-1","level":"DEBUG","loggerName":"ocpm.cne.gateway.config.DynamicRouteConfiguration","message":"Validated the following route sucessfully: RoutesConfiguration [id=n32d, uri=https://ocsepp.com, order=81, predicates=[PredicateDefinition{name='Path',
      args={pattern=/*/n32c-handshake/**}}], filters=null, metadata={}, httpRuriOnly=null,
      httpsTargetOnly=null, sbiRoutingConfiguration=null]","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","contextMap":{},"threadId":72,"threadPriority":5,"messageTimestamp":"2022-02-04T09:55:32.801+0000","ocLogId":"${ctx:ocLogId}","pod":"${ctx:hostname}","processId":"1","instanceType":"prod","egressTxId":"${ctx:egressTxId}"}

Sample log statement plmn-egress-gateway:

{"instant":{"epochSecond":1643971148,"nanoOfSecond":705331370},"thread":"scheduling-1","level":"INFO","loggerName":"com.oracle.common.metrics.ConfigClientMetrics","message":"Pegged ConfigClient Response metric with releaseVersion 22.2.2, configVersion 1 and updated
      parameter false","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","contextMap":{},"threadId":72,"threadPriority":5,"messageTimestamp":"2022-02-04T10:39:08.705+0000","ocLogId":"${ctx:ocLogId}","pod":"${ctx:hostname}","processId":"1","instanceType":"prod","egressTxId":"${ctx:egressTxId}"}

Table 2-3 Log Attribute Details for Egress Gateway

Log Attribute Details Sample Value Data Type
thread Logging Thread Name "main" String
level Log Level of the log printed "DEBUG" String
loggerName Class/Module which printed the log "ocpm.cne.gateway.config.DynamicRouteConfiguration"

String
message Message related to the log providing brief details "Property name: server.port and value: 8080" String
endOfBatch

Log4j2 Internal

Default from log4j2: false

 false boolean
loggerFqcn

Log4j2 Internal

Fully Qualified class name of logger module

 org.apache.logging.log4j.spi.AbstractLogger String
instant Epoch timestamp.

It is group of two values epochSecond and nanoOfSecond

 {"epochSecond":1604564777,"nanoOfSecond":135977000 Object
contextMap Elements in log4j ThreadContext map {} Object
threadId Thread Id generated internally by Log4j2 1 Integer
threadPriority Thread Priority set internally by Log4j2 5 Integer
messageTimestamp Timestamp of log from application container. Format: yyyy-MM-dd'T'HH:mm:ss.SSSZ "2020-11-05 08:26:17.135" String
ocLogId End to End Log Identifier across the OCSEPP microservices. Helps to correlate the logs across the microservices in OCSEPP application "1604650229002_72_ocsepp-ingressgateway-69f6544b8d-cdbgx" String
pod Name of the egress pod "ocsepp-egressgateway-69f6544b8d-cdbgx" String
processId Process ID internally assigned "1" String
instanceType Instance type "prod" String
egressTxId Transaction id that is added to log4j ThreadContext map and is unique to every transaction "egress-tx-1087436877" String

Nf-Mediation service

This section provides log level attribute details for mediation service:

Sample log statement mediation service:

{"instant":{"epochSecond":1661413301,"nanoOfSecond":856094544},"thread":"Thread-0","level":"RULE_TRAIL","loggerName":"com.oracle.cgbu.ocmediation.ruleengine.DroolsRuleEngine","message":"Mediation Rule files reloading successful","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger","threadId":15,"threadPriority":5,"ts":"22-08-25 07:41:856.041+0000","namespace":"gwnrf","node_name":"cnejac0106.jacvla.morrisville.us.lab.oracle.com","pod":"ocsepp-release-seppsvc-nf-mediation-7679c47c77-zbqd2","instanceType":"prod","processId":"1","ocLogId":"${ctx:ocLogId}","vendor":"oracle"}

Log Attribute Details for Nf-Mediation service

Table 2-4 Log Attribute Details for Nf-Mediation service

Log Attribute Details Sample Value Data Type
instant

Epoch time.

It is the group of two values epochSecond and nanoOfSecond

 {"epochSecond":1604655402,"nanoOfSecond":946649000} Object
thread Logging Thread Name " Thread-0" String
level Log Level of the log printed "RULE_TRAIL" String
loggerName Class or module which printed the log " com.oracle.cgbu.ocmediation.ruleengine.DroolsRuleEngine" String
message Message related to the log providing brief details. "Mediation Rule files reloading successful" String
endOfBatch

Log4j2 Internal

Default from log4j2: false

 false Boolean
loggerFqcn

Log4j2 Internal

Fully Qualified class name of logger module

 org.apache.logging.log4j.spi.AbstractLogger String
threadId Thread Id generated internally by Log4j2 15 Integer
threadPriority Thread Priority set internally by Log4j2 5 Integer
namespace Namespace for which log is generated “gwnrf”
node_name Name of the worker node on which this pod is allocated “cnejac0106.jacvla.morrisville.us.lab.oracle.com”
pod Name of the pod which generated these logs “ocsepp-release-seppsvc-nf-mediation-7679c47c77-zbqd2”
ts

Timestamp of log from application container. Format: yyyy-MM-dd'T'HH:mm:ss.SSSZ

Timestamp can be filtered using the following label :ts -> logs in containertimestamp -> logs on Kibana

 "ts":"22-02-04 03:49:162.039+0000" String
instanceType

Instance details. Example: dev, prod, qa.

Note: Part of container logs but not in Kibana.

 prod String
processId

Process ID internally assigned.

Note: Part of container logs but not in Kibana.

 "1" String
ocLogId

End to End Log Identifier across the OCSEPP microservices.

Helps to correlate the logs across the microservices in OCSEPP application.

Note: Part of container logs but not in Kibana.

 ${ctx:ocLogId} String
vendor Vendor Name "oracle" String

Common Useful log attributes

The following log attributes are available only through Kibana. These attribute names are part of Kubernetes Labels which are added in SEPPs each POD.

Table 2-5 Common useful log attributes

Log Attribute Details Sample Value Data Type
engVersion Engineering version "23.3.0" String
mktgVersion Marketing version "23.3.0.0.0" String
vendor Vendor Name "Oracle" String