10 CNC Console Logs

This section describes about the cncc logs. It contains the following topics:

Log Formats and Details

Types of Logs

Examples of Logs

Accessing the Logs

10.1 Log Formats

This section provides information about the log formats.

Log4j JSON Format

CNCC Message Format

Log4j JSON Format

Following is the log format in JSON:

{
     "thread": <threadName>,
     "level": <log_level>,
     "loggerName": <name_of_the_logging_class>,
     "message": <message>,
     "instant": <timestamp_in_miliseconds>,
     "messageTimestamp": <timestamp_in_readable_format>,
     "threadId": <threadId>,
     "threadPriority": <threadPriority>,
     "pod": <name_of_the_pod>,
     "processId": <processId>,
     "contextMap": <context_map>,
     "ocLogId": <unique_trace_id_for_every_request>,
     "instanceType": <instanceType>,
     "ingressTxId": <IngressTransactionId>
 }

Table 10-1 Log Details

Name Description Example
thread Name of the thread. "thread": "reactor-http-epoll-1"
level Level of the log. It can be: Log level (INFO, WARN, DEBUG, TRACE) "level": "INFO"
loggerName Name of the class that generated the log. "loggerName": "ocpm.cne.gateway.cncc.GatewayApplication"
messageTimestamp Time represented in human readable format and in UTC. Format is date:yyyy-MM-dd'T'HH:mm:ss.SSSZ EFK friendly and also follows Oracle Standards. "messageTimestamp": 2020-07-04'T'12:00:40.702Z
message Information about the event. "message": "Started Application....."

By default, all messages are in simple string except Audit Log, Security Log which are represented in CNC Console Message Format..

instant The Date and Time the event occurred in epoch second and nano seconds. "instant": { "epochSecond": 1590045388, "nanoOfSecond": 339789000}
processId Linux process Identifier (for a multi-process host. Linux process Identifier (for a multi-process host).
threadId Id of the thread. "threadId":"43"
threadPriority Priority assigned to the thread "threadPriority": 5
pod Name of the pods where the log is generated "cncc-mcore-ingress-gateway-77df795fb5-wv2sb"
contextMap It holds information added to threadContext. "contextMap": { "hostname": "cncc-mcore-ingress-gateway-77df795fb5-wv2sb", "ingressTxId": "ingress-tx-1460885598"}
ocLogId It contains the trace id that is uniquely generated for every request of the format "<timestamp(in milliseconds)>_<thread Id>_<POD name>" It contains the trace id that is uniquely generated for every request of the format "<timestamp (in milliseconds)>_<thread Id>_<POD name>"
instanceType Static tag which implies that instance type is production "instanceType": "prod"
ingressTxId It contains id of the format "ingress-tx-<Random no>" to track every transaction "ingressTxId": ingress-tx-1904660570

CNC Console Message Format

Table 10-2 CNC Console Message Format

Name Description Example Possible Values
logType Indicates whether it is Security Log or Audit Log. logType=AUDIT AUDIT

SECURITY
type Indicates nature or action of the log. type=REQUEST For Security Log: REQUEST, RESPONSE

For Audit Log: LOGIN, ACCESS_RESOURCE, ACCESS_RESOURCE_ERROR, LOGOUT
resourceType Indicates what is the resource being requested for. resourceType=SCP CM_SERVICE (For default route)

CNC Console (For User Login Activity)

SCP

UDR

NRF

PCF

(all CNC Console supported NFs)
userId Id of the user who triggered request or action. userId=3314f54f-08bf-489d-b395-27bf56da1262 NA
username Name of the user username= "user1" NA
status HTTP status of the response. status=200 OK NA
operationType HTTP method of the request. operationType=GET NA
scheme Indicates the scheme of the request. scheme=http NA
remoteAddress The remote address that is associated with the request. It also means the remote address to where this request is connected when available. remoteAddress=/192.168.219.64:53587 NA
localAddress The local address that is associated with the request. It also means the local address to where this request is connected when available. localAddress=cncc-mcore-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075 NA
resourcePath Request URI resourcePath=/soothsayer/v1/canaryrelease/ NA
queryParams Query parameters associated with request. queryParams={form_id=9, page=1, view_id=78} NA
headers Headers associated with request or response. headers={Accept=*/*, X-Requested-With=XMLHttpRequest, User-Agent=Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0, Connection=keep-alive, Host=cncc-core-ingress-gateway.cncc.svc.cluster.local:30075, Accept-Language=en-US,en;q=0.5, Accept-Encoding=gzip, deflate, DNT=1, Content-Type=application/json; charset=utf-8} NA
payload Payload or Data associated with request or response. payload=[{"serviceName":"n5g-eir-eic","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5} NA
authenticationType This indicates whether user is requesting resource logged in using CNC Console or directly accessing through postman or curl. authenticationType=OAUTH OAUTH -> User is logged in through CNC Console application and accessing resource.

JWT -> User is accessing resource directly through postman or curl.

10.2 Types of Logs

The CNC Console logs can be categorized into following types:

Regular logs

These logs contain error messages, warnings, or other events written within the application that provide logical, high level information about the application and ongoing events.

Example:

{"level": "INFO","message": "Started GatewayApplication in 10.748 seconds (JVM running for 12.825)"}
{"level": "INFO","message": "Creating plain httpClient"}
{"level": "INFO","message": "Creating plain restTemplate"}
{"level": "ERROR","message": "Can't get cfgs of topic public.dynamic.datamodel,  exception is:\n
javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection
        refused)"}

Audit Logs

Note:

CNC Console IAM Audit logs are not applicable for OCI Deployment.

These logs contain user related information and the activity within the system.

The following events are logged in CNC Console Core:

  • Log in: A user has logged in.
  • Access Resource: A user is accessing a particular NF resource.
  • Access Resource Error: A user is denied from accessing a particular NF resource.
  • Logout: A user has logged out.

    Note:

    The user can find the CNC Console Core User Activity logs as part of cncc-core-ingress-gateway and are represented in CNC Console message format.

The following events are logged in CNC Console IAM:
  • Login events
  • Log in: An admin user has logged in.
  • Register: An admin user has registered.
  • Logout: An admin user has logged out.
  • Code to Token: An application or a client has exchanged a code for a token.
  • Refresh Token: An application or a client has refreshed a token.

    Account events

  • Update Email: The email address for an account has changed.
  • Update Profile: The profile for an account has changed.
  • Send Password Reset: A password reset email has been sent.
  • Update Password: The password for an account has changed.

    Note:

    The user can find the CNC Console IAM User Activity logs as part of cncc-iam-0, represented in Keycloak format. These events are provided by keycloak and documented under Keycloak Auditing End Events.
    Logging Error Logs are recorded by keycloak container as :
    ^[[0m^[[33m10:12:57,388 WARN  [org.keycloak.events] (default task-3) type=LOGIN_ERROR, realmId=master, clientId=security-admin-console,
      userId=ef58d62e-a0a8-4f4e-bcc6-abccf917641c, ipAddress=192.168.203.108, error=invalid_user_credentials, auth_method=openid-connect, auth_type=code,
      redirect_uri=http://10.75.240.33:30085/cncc/auth/admin/master/console/,
      code_id=3e6d822a-9e82-4660-bb01-a814f7ae8f97, username=admin,
      authSessionParentId=3e6d822a-9e82-4660-bb01-a814f7ae8f97,
    authSessionTabId=2ak6Xwal-28

Security Logs

Note:

M-CNCC IAM Security logs are not applicable for OCI Deployment.

The security logs contain the header, payload, method, scheme, URI information for all the requests and corresponding responses.

Disabling Security Logs

By default Security Log will be enabled for M-CCNC IAM, M-CNCC Core, and A-CNCC Core. You can disable this by setting securityLogEnabled flag to false in custom values.yaml file. 

# CNC Console configuration
cncc:
  # Enable security logs for CNC Console
  securityLogEnabled: false

Header Information

At all the log levels, sensitive information like Cookies are masked.

Note:

The user can find the Security logs:
  • For M-CNCC Core and A-CNCC Core, these are logged as part of cncc-mcore-ingress-gateway or cncc-acore-ingress-gateway and are represented in CNC Console message format.
  • For M-CNCC IAM, these are logged as part of cncc-iam-ingress-gateway and are represented in CNCC message format.

Log Levels

Note:

M-CNCC IAM ingress-gateway default log levels are not applicable for OCI deployment.

Default log levels set for M-CNCC Core and A-CNCC Core:


ingress-gateway:
  log:
    level:
      cncc:
        root: WARN
        audit: INFO
        security: INFO

Default log levels set for M-CNCC IAM:


ingress-gateway:
  log:
    level:
      cncc:
        root: WARN
        security: INFO
Updating M-CNCC IAM Kubernetes Cluster (KC) log level

Note:

M-CNCC IAM KC logs levels are not applicable for OCI Deployment.
  • By default the log level of M-CNCC IAM KC is set to WARN, org.keycloak.events:DEBUG
    • This means the root log-level is set to WARN and the org.kecyalok.events package is set to DEBUG
  • In the level label, set the log level. Following are the different options available to set the log level:
    • Log level
      • TRACE
      • DEBUG
      • INFO
      • WARN
      • ERROR
      • FATAL
    • level: OFF

      No logs will appear

  • Sample M-CNCC IAM KC log configuration: 
    kc:    
  preferIpv6Stack:      
    enabled: false
  log:       
    level: WARN,org.keycloak.events:DEBUG

Supported Headers for Logging

Header Header values (regex)
Content-Type ^application/x-www-form-urlencoded.*
^application/json.*
^application/problem+json.*
Accept ^application/json.*
^application/ld+json.*
^application/xml.*
^multipart/form-data.*
Role of supporting headers in CNC Console Audit and Security logs
  • At INFO level, only those request and response that match the supporting headers and values are logged.
  • At DEBUG level, no supporting headers used and all request and response are logged.
  • At ERROR / WARN, no supporting headers used and only error or warnings are logged.

    Note:

    Any failure in authorizing a request will always be logged irrespective of the supported header configuration.

10.3 Examples of Logs

This section lists the examples of audit and security logs.

Examples of Audit Logs

Examples of Security Logs

Examples of Audit Logs

CNC Console Core

Only message part of the JSON log is shown in the example.

  • User successfully logging into CNC Console Core 
    
logType=AUDIT, type=LOGIN, resourceType=CNCC,
userId=186f6f2a-ba6a-4812-8a18-b906a5f9e3f6, username=user,
operationType=GET, remoteAddress=/192.168.219.64:53587,
localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
resourcePath=/login/oauth2/code/cncc-iam,
authenticationType=OAUTH
  • User accessing SCP resource having SCP_READ role 
    
logType=AUDIT, type=ACCESS_RESOURCE, resourceType=SCP,
userId=186f6f2a-ba6a-4812-8a18-b906a5f9e3f6, username=user,
operationType=GET, remoteAddress=/192.168.219.64:53587,
localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
resourcePath=/soothsayer/v1/canaryrelease/,
 authenticationType=OAUTH
  • User updating(PATCH) SCP resource having SCP_WRITE role 
    
logType=AUDIT, type=ACCESS_RESOURCE, resourceType=SCP,
userId=186f6f2a-ba6a-4812-8a18-b906a5f9e3f6, username=user,
operationType=PATCH, remoteAddress=/192.168.219.64:53587,
localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
resourcePath=/soothsayer/v1/canaryrelease/n5g-eir-eic,
authenticationType=OAUTH
  • User accessing NRF resource without having NRF_READ role 
    
logType=AUDIT, type=ACCESS_RESOURCE_ERROR, resourceType=NRF,
 userId=186f6f2a-ba6a-4812-8a18-b906a5f9e3f6, username=user,
status=403 FORBIDDEN, operationType=GET,
remoteAddress=/192.168.219.64:53587,
localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
resourcePath=/nrf-configuration/v1/system-options,
authenticationType=OAUTH
  • User successful logout 
    
logType=AUDIT, type=LOGOUT, resourceType=CNCC,
userId=186f6f2a-ba6a-4812-8a18-b906a5f9e3f6, username=user,
operationType=POST, remoteAddress=/192.168.219.64:53587,
localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
resourcePath=/logout, authenticationType=OAUTH

CNC Console IAM:

Note:

  • The defined CNC Console IAM Audit log examples are not applicable for OCI Deployment.
  • The CNC Console IAM Security log examples are not applicable for OCI Deployment.
  • To see M CNCC-IAM KC Audit event logs, set the org.keycloak.events package log to DEBUG in the custom_values.yaml. Refer Updating M-CNCC IAM KC loglevel for log level configuration.
  • Login Error when password entered was wrong 
    04:56:35,890 WARN  [org.keycloak.events] (default task-22)
                              type=LOGIN_ERROR, realmId=master, clientId=security-admin-console,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              error=invalid_user_credentials, auth_method=openid-connect,
                              auth_type=code, redirect_uri=http://10.75.225.28:31373/cncc/auth/admin/master/console/, code_id=5aca4960-eecf-406b-a7eb-92e249c2beeb,
                              username=admin,
                              authSessionParentId=5aca4960-eecf-406b-a7eb-92e249c2beeb,
                              authSessionTabId=8sruELA1WWs
  • Login with correct credential 
    04:57:24,581 DEBUG  [org.keycloak.events] (default task-22)
                              type=LOGIN, realmId=master, clientId=security-admin-console,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              auth_method=openid-connect, auth_type=code, redirect_uri=http://10.75.225.28:31373/cncc/auth/admin/master/console/, consent=no_consent_required,
                              code_id=5aca4960-eecf-406b-a7eb-92e249c2beeb, username=admin,
                              authSessionParentId=5aca4960-eecf-406b-a7eb-92e249c2beeb,
                              authSessionTabId=8sruELA1WWs
  • User created 
    04:58:41,804 DEBUG  [org.keycloak.events] (default task-22)
                              operationType=CREATE, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=USER,
                              resourcePath=users/070911f5-c397-42c1-b5a4-cd92fa435a33
  • Deleted user 
    05:00:08,226 DEBUG  [org.keycloak.events] (default task-22)
                              operationType=DELETE, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=USER,
                              resourcePath=users/2b931bbb-7f97-4f04-9f75-e0d0974ab73d
  • Admin Role removed for a user 
    05:01:07,781 DEBUG  [org.keycloak.events] (default task-22)
                              operationType=DELETE, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=REALM_ROLE_MAPPING,
                              resourcePath=users/08fc0058-133b-4288-9165-14c96c5dcd7a/role-mappings/realm
  • Admin Role added for a user 
    05:01:33,664 DEBUG  [org.keycloak.events] (default task-27)
                              operationType=CREATE, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=REALM_ROLE_MAPPING,
                              resourcePath=users/08fc0058-133b-4288-9165-14c96c5dcd7a/role-mappings/realm
  • Realm setting update 
    05:02:29,222 DEBUG  [org.keycloak.events] (default task-26)
                              operationType=UPDATE, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=REALM, resourcePath=null

  • Logout all session on keycloak

    05:05:02,383 DEBUG  [org.keycloak.events] (default task-29)
                              operationType=ACTION, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=REALM, resourcePath=logout-all

Examples of Security Logs

Representation for IAM and Core are same as these logs are part of ingress-gateway. Only message part of the JSON log is shown in the example.

CNC Console Core

  • SCP request 
    logType=SECURITY, type=REQUEST, resourceType=SCP,
                                    userId=3314f54f-08bf-489d-b395-27bf56da1262, username=user,
                                    operationType=GET, scheme=http,
                                    remoteAddress=/192.168.219.64:53587,
                                    localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
                                    resourcePath=/soothsayer/v1/canaryrelease/, queryParams={},
                                    headers={Accept=*/*, X-Requested-With=XMLHttpRequest,
                                    User-Agent=Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0)
                                    Gecko/20100101 Firefox/68.0, Connection=keep-alive,
                                    Host=cncc-core-ingress-gateway.cncc.svc.cluster.local:30075,
                                    Accept-Language=en-US,en;q=0.5, Accept-Encoding=gzip, deflate,
                                    DNT=1, Content-Type=application/json; charset=utf-8},
                                    payload={}, authenticationType=OAUTH
  • SCP response 
    logType=SECURITY, type=RESPONSE, resourceType=SCP,
                                    userId=3314f54f-08bf-489d-b395-27bf56da1262, username=user,
                                    status=200 OK, operationType=GET, scheme=http,
                                    resourcePath=/soothsayer/v1/canaryrelease/,
                                    headers={transfer-encoding=chunked, Connection=keep-alive,
                                    Transfer-Encoding=chunked, Content-Type=application/json,
                                    Date=Sat, 04 Jul 2020 11:58:20 GMT},
                                    payload=[{"serviceName":"n5g-eir-eic","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"namf-comm","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"namf-evts","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"namf-loc","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"namf-mt","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"nausf-auth","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"nausf-sorprotection","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5}}],
                                    authenticationType=OAUTH

CNC Console IAM

The logs for IAM Ingress Gateway contain the username and user ID for all actions performed by CNCC-IAM admin users post-login, alongside other relevant information.

  • Request
    logType=SECURITY, type=REQUEST, userId=2811c05e-bc84-4597-8014-3a88ef7535c2, username=admin, operationType=GET, scheme=http, remoteAddress=/10.233.118.0:52597, localAddress=/10.233.80.179:8081, resourcePath=/cncc/auth/admin/realms/master/users/0931c7a1-34a6-45f1-9db5-5cd8a043fefb, queryParams={userProfileMetadata=true}, headers={Connection=keep-alive, User-Agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36, uidToken=07338817-dOac-4b18-948d-e3d5c0c91879, Host=10.121.42.53, Accept-Encoding=gzip, deflate, svcName=cncc-iam-kc-http.amanp-ns.svc.thrust2a, ocLogId=1733063953903_110_cncc-iam-ingress-gateway-5f7d9b9d87-dt6vj,
accept=application/json, text/plain, */*, sbi-timer-publish-headers=false, content-type-application/json, Accept-Language=en-GB,
en; q=0.9,
sbi-timer-feature=false}, payload={}, authenticationType=UNKNOWN", "endOfBatch": false, "loggerFqcn": "org-apache. logging.
10g4j.internal.DefaultLogBuilder", "threadId":112, "threadPriority":5, "messageTimestamp":"2024-12-01T14:39:13.910+0000"' ,
"processId": "1", "ingressTxId":"ingress-tx-1093411995", "pod": "cncc-iam-ingress-gateway-5f7d9b9d87-dt6vj" ,
"ocLogId":"1733063953903 110 cncc-iam-ingress-qateway-5f7d9b9d87-dt6vi" "instanceType":"prod" "RequestId": ''}
  • Response 
    "message":" logType=SECURITY, type=RESPONSE, status=200 0K, userId=2811c05e-bc84-4597-8014-3a88ef7535c2, username=admin, operationType=GET, schemeshttp, resourcePath=/cncc/auth/admin/ realms/ master/clients/a6f486d4-0d0f-468d-a609-f655e9f1c492, headers={transfer-encoding=chunked, content-length=772, Cache-Control=no-cache, Content-Type=application/json;charset=UTF-8, Referrer-Policy=no-referrer, Strict-Transport-Security=max-age=31536000; includeSubDomains, X-Content-Type-Options=nosniff, X-Frame-Options=SAMEORIGIN, X-XSS-Protection=1; mode=block, NettyLatency=1733064487125, RequestMethod=GET}, payload= {\"id\":\"a6f486d4-0d0f-468d-a609-f655e9f1c492\", \"clientId\" :\"cncc-realm\", \"name\": \"cncc Realm\" ,
\"surrogateAuthRequired)": false, \"enabled)": true, \"alwaysDisplayInConsole\": false, \"clientAuthenticatorType\" :\"client-secret\", \"redirectUris\": [1, \"webOrigins\": [1, \"notBefore\":0, \"beareronly)": true, \"consentRequired\": false, \"standardF lowEnabled\": true, \"implicitFlowEnabled\": false, \"directAccessGrantsEnabled\": false, \"serviceAccountsEnabled\": false, \"publicClient\": false, \"frontchannelLogout)": false, \"attributes)": {}, \"authenticationFlowBindingOverrides\": {},\"fullScopeAllowed\": true, \"nodeReRegistrationTimeout\":0, \"defaultClientScopes\": [\"web-origins\", \"roles\", \"profile\", \"email\"], \"optionalClientScopes\": [\"address\", \"phone\", \"offline_access\", \"microprofile-jwt\"], \"access\":{\"view|": true, \"configure)": true, \"manage)": true}}, authenticationType=UNKNOWN", "endofBatch": false, "loggerFqcn": "org-apache. logging. log4j. internal.DefaultLogBuilder", "threadId":122, "threadPriority":5, "messageTimestamp": "2024-12-01T14:48:07.152+0000", "processId": "1" ,
"ingressTxId" :"ingress-tx-577080018", "pod":"cncc-iam-ingress-gateway-5f7d9b9d87-dt6vj", "ocLogId" :"'","instanceType": "prod" ,
"RequestId" : ""}

10.4 Accessing logs

This section gives information about how to access the logs.