1. Cloud Native Core Console User Guide
  2. CNC Console Logs

10 CNC Console Logs

This section describes about the cncc logs. It contains the following topics:

Log Formats and Details

Types of Logs

Configuring the Logs

Examples of Logs

Accessing the Logs

Log Formats

This section provides information about the log formats.

Log4j JSON Format

CNCC Message Format

Log4j JSON Format

Following are the log format and fields. All logs are represented in JSON format.

{
     "thread": <threadName>,
     "level": <log_level>,
     "loggerName": <name_of_the_logging_class>,
     "message": <message>,
     "instant": <timestamp_in_miliseconds>,
     "logTimestamp": <timestamp_in_readable_format>,
     "threadId": <threadId>,
     "threadPriority": <threadPriority>,
     "pod": <name_of_the_pod>,
     "processId": <processId>,
     "contextMap": <context_map>,
     "ocLogId": <unique_trace_id_for_every_request>,
     "instanceType": <instanceType>,
     "ingressTxId": <IngressTransactionId>
 }

Table 10-1 Log Details

Name Description Example
thread Name of the thread "thread": "reactor-http-epoll-1"
level Level of the log. It can be: Log level (INFO, WARN, DEBUG, TRACE) "level": "INFO"
loggerName Name of the class that generated the log "loggerName": "ocpm.cne.gateway.cncc.GatewayApplication"
logTimestamp Time represented in human readable format and in UTC. Format is date:yyyy-MM-dd'T'HH:mm:ss.SSSZ EFK friendly and also follows Oracle Standards "logTimestamp": 2020-07-04'T'12:00:40.702Z
message Information about the event "message": "Started Application....."

By default, all messages are in simple string except Audit Log, Security Log which are represented in CNCC Message Format.

instant The Date and Time the event occurred in epoch second and nano seconds "instant": { "epochSecond": 1590045388, "nanoOfSecond": 339789000}
processId Linux process Identifier (for a multi-process host) Linux process Identifier (for a multi-process host).
threadId Id of the thread "threadId":"43"
threadPriority Priority assigned to the thread "threadPriority": 5
pod Name of the pods where the log is generated "cncc-mcore-ingress-gateway-77df795fb5-wv2sb"
contextMap It holds information added to threadContext. "contextMap": { "hostname": "cncc-mcore-ingress-gateway-77df795fb5-wv2sb", "ingressTxId": "ingress-tx-1460885598"}
ocLogId It contains the trace id that is uniquely generated for every request of the format "<timestamp(in milliseconds)>_<thread Id>_<POD name>" It contains the trace id that is uniquely generated for every request of the format "<timestamp (in milliseconds)>_<thread Id>_<POD name>"
instanceType Static tag which implies that instance type is production "instanceType": "prod"
ingressTxId It contains id of the format "ingress-tx-<Random no>" to track every transaction "ingressTxId": ingress-tx-1904660570

CNCC Message Format

Table 10-2 CNCC Message Format

Name Description Example Possible Values
logType Indicates whether it is Security Log or Audit Log logType=AUDIT AUDIT

SECURITY
type Indicates nature or action of the log type=REQUEST For Security Log: REQUEST, RESPONSE

For Audit Log: LOGIN, ACCESS_RESOURCE, ACCESS_RESOURCE_ERROR, LOGOUT
resourceType Indicates what is the resource being requested for resourceType=SCP CM_SERVICE (For default route)

CNCC (For User Login Activity)

SCP

UDR

NRF

PCF

(all CNCC supported NFs)
userId Id of the user who triggered request or action userId=3314f54f-08bf-489d-b395-27bf56da1262 NA
username Name of the user username= "user1" NA
status HTTP status of the response. status=200 OK NA
operationType HTTP method of the request operationType=GET NA
scheme Indicates the scheme of the request scheme=http NA
remoteAddress The remote address that is associated with the request. It also means the remote address to where this request is connected when available. remoteAddress=/192.168.219.64:53587 NA
localAddress The local address that is associated with the request. It also means the local address to where this request is connected when available. localAddress=cncc-mcore-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075 NA
resourcePath Request uri resourcePath=/soothsayer/v1/canaryrelease/ NA
queryParams Query parameters associated with request queryParams={form_id=9, page=1, view_id=78} NA
headers Headers associated with request or response headers={Accept=*/*, X-Requested-With=XMLHttpRequest, User-Agent=Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0, Connection=keep-alive, Host=cncc-core-ingress-gateway.cncc.svc.cluster.local:30075, Accept-Language=en-US,en;q=0.5, Accept-Encoding=gzip, deflate, DNT=1, Content-Type=application/json; charset=utf-8} NA
payload Payload or Data associated with request or response payload=[{"serviceName":"n5g-eir-eic","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5} NA
authenticationType This indicates whether user is requesting resource logged in using CNCC or directly accessing through postman or curl. authenticationType=OAUTH OAUTH -> User is logged in through CNC Console application and accessing resource

JWT -> User is accessing resource directly through postman or curl

Types of Logs

The CNC Console logs can be categorized into following types:

Regular logs

These logs contain error messages, warnings, or other events written within the application that provide logical, high level information about the application and ongoing events.

Example:

{"level": "INFO","message": "Started GatewayApplication in 10.748 seconds (JVM running for 12.825)"}
{"level": "INFO","message": "Creating plain httpClient"}
{"level": "INFO","message": "Creating plain restTemplate"}
{"level": "ERROR","message": "Can't get cfgs of topic public.dynamic.datamodel,  exception is:\n
javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection
        refused)"}

Audit Logs

These logs contain user related information and the activity within the system.

The following events are logged in CNC Console Core:

  • Log in: A user has logged in.
  • Access Resource: A user is accessing a particular NF resource.
  • Access Resource Error: A user is denied from accessing a particular NF resource.
  • Logout: A user has logged out.

    Note:

    The user can find the CNCC Core User Activity logs as part of cncc-core-ingress-gateway and are represented in CNCC message format.

    The following events are logged in CNCC IAM:

    Login events

  • Log in: An admin user has logged in.
  • Register: An admin user has registered.
  • Logout: An admin user has logged out.
  • Code to Token: An application or a client has exchanged a code for a token.
  • Refresh Token: An application or a client has refreshed a token.

    Account events

  • Update Email: The email address for an account has changed.
  • Update Profile: The profile for an account has changed.
  • Send Password Reset: A password reset email has been sent.
  • Update Password: The password for an account has changed.

    Note:

    The user can find the CNCC IAM User Activity logs as part of cncc-iam-0, represented in Keycloak format. These events are provided by keycloak and documented under Keycloak Auditing End Events.
    Logging Error Logs are recorded by keycloak container as :
    ^[[0m^[[33m10:12:57,388 WARN  [org.keycloak.events] (default task-3) type=LOGIN_ERROR, realmId=master, clientId=security-admin-console,
      userId=ef58d62e-a0a8-4f4e-bcc6-abccf917641c, ipAddress=192.168.203.108, error=invalid_user_credentials, auth_method=openid-connect, auth_type=code,
      redirect_uri=http://10.75.240.33:30085/cncc/auth/admin/master/console/,
      code_id=3e6d822a-9e82-4660-bb01-a814f7ae8f97, username=admin,
      authSessionParentId=3e6d822a-9e82-4660-bb01-a814f7ae8f97,
    authSessionTabId=2ak6Xwal-28

Security Logs

The security logs contain the header, payload, method, scheme, URI information for all the requests and corresponding responses.

Disabling Security Logs

By default Security Log will be enabled for M-CCNC IAM, M-CNCC Core and A-CNCC Core. You can disable this by setting securityLogEnabled flag to false in custom values.yaml file. 

# CNCC configuration
cncc:
  # Enable security logs for CNCC
  securityLogEnabled: false

Header Information

At all the log levels, sensitive information like Cookies are masked.

Note:

The user can find the Security logs :
  • For M-CNCC Core and A-CNCC Core, these are logged as part of cncc-mcore-ingress-gateway or cncc-acore-ingress-gateway and are represented in CNCC message format.
  • For M-CNCC IAM, these are logged as part of cncc-iam-ingress-gateway and are represented in CNCC message format.

Log Levels

Default log levels set for M-CNCC Core and A-CNCC Core:


ingress-gateway:
  log:
    level:
      cncc:
        root: WARN
        audit: INFO
        security: INFO

Default log levels set for M-CNCC IAM:


ingress-gateway:
  log:
    level:
      cncc:
        root: WARN
        security: INFO

Supported Headers for Logging

Header Header values (regex)
Content-Type ^application/x-www-form-urlencoded.*
^application/json.*
^application/problem+json.*
Accept ^application/json.*
^application/ld+json.*
^application/xml.*
^multipart/form-data.*
Role of supporting headers in CNCC Audit and Security logs
  • At INFO level, only those request and response that match the supporting headers and values are logged.
  • At DEBUG level, no supporting headers used and all request and response are logged.
  • At ERROR / WARN, no supporting headers used and only error or warnings are logged.

    Note:

    Any failure in authorizing a request will always be logged irrespective of the supported header configuration.

Examples of Logs

This section lists the examples of audit and security logs.

Examples of Audit Logs

Examples of Security Logs

Examples of Audit Logs

CNCC Core

Only message part of the JSON log is shown in the example.

  • User successfully logging into CNCC Core 
    
logType=AUDIT, type=LOGIN, resourceType=CNCC,
userId=186f6f2a-ba6a-4812-8a18-b906a5f9e3f6, username=shreb,
operationType=GET, remoteAddress=/192.168.219.64:53587,
localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
resourcePath=/login/oauth2/code/cncc-iam,
authenticationType=OAUTH
  • User accessing SCP resource having SCP_READ role 
    
logType=AUDIT, type=ACCESS_RESOURCE, resourceType=SCP,
userId=186f6f2a-ba6a-4812-8a18-b906a5f9e3f6, username=shreb,
operationType=GET, remoteAddress=/192.168.219.64:53587,
localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
resourcePath=/soothsayer/v1/canaryrelease/,
 authenticationType=OAUTH
  • User updating(PATCH) SCP resource having SCP_WRITE role 
    
logType=AUDIT, type=ACCESS_RESOURCE, resourceType=SCP,
userId=186f6f2a-ba6a-4812-8a18-b906a5f9e3f6, username=shreb,
operationType=PATCH, remoteAddress=/192.168.219.64:53587,
localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
resourcePath=/soothsayer/v1/canaryrelease/n5g-eir-eic,
authenticationType=OAUTH
  • User accessing NRF resource without having NRF_READ role 
    
logType=AUDIT, type=ACCESS_RESOURCE_ERROR, resourceType=NRF,
 userId=186f6f2a-ba6a-4812-8a18-b906a5f9e3f6, username=shreb,
status=403 FORBIDDEN, operationType=GET,
remoteAddress=/192.168.219.64:53587,
localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
resourcePath=/nrf-configuration/v1/system-options,
authenticationType=OAUTH
  • User successful logout 
    
logType=AUDIT, type=LOGOUT, resourceType=CNCC,
userId=186f6f2a-ba6a-4812-8a18-b906a5f9e3f6, username=shreb,
operationType=POST, remoteAddress=/192.168.219.64:53587,
localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
resourcePath=/logout, authenticationType=OAUTH

CNCC IAM:

  • Login Error when password entered was wrong 
    04:56:35,890 WARN  [org.keycloak.events] (default task-22)
                              type=LOGIN_ERROR, realmId=master, clientId=security-admin-console,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              error=invalid_user_credentials, auth_method=openid-connect,
                              auth_type=code, redirect_uri=http://10.75.225.28:31373/cncc/auth/admin/master/console/, code_id=5aca4960-eecf-406b-a7eb-92e249c2beeb,
                              username=admin,
                              authSessionParentId=5aca4960-eecf-406b-a7eb-92e249c2beeb,
                              authSessionTabId=8sruELA1WWs
  • Login with correct credential 
    04:57:24,581 INFO  [org.keycloak.events] (default task-22)
                              type=LOGIN, realmId=master, clientId=security-admin-console,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              auth_method=openid-connect, auth_type=code, redirect_uri=http://10.75.225.28:31373/cncc/auth/admin/master/console/, consent=no_consent_required,
                              code_id=5aca4960-eecf-406b-a7eb-92e249c2beeb, username=admin,
                              authSessionParentId=5aca4960-eecf-406b-a7eb-92e249c2beeb,
                              authSessionTabId=8sruELA1WWs
  • User created 
    04:58:41,804 INFO  [org.keycloak.events] (default task-22)
                              operationType=CREATE, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=USER,
                              resourcePath=users/070911f5-c397-42c1-b5a4-cd92fa435a33
  • Deleted user 
    05:00:08,226 INFO  [org.keycloak.events] (default task-22)
                              operationType=DELETE, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=USER,
                              resourcePath=users/2b931bbb-7f97-4f04-9f75-e0d0974ab73d
  • Admin Role removed for a user 
    05:01:07,781 INFO  [org.keycloak.events] (default task-22)
                              operationType=DELETE, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=REALM_ROLE_MAPPING,
                              resourcePath=users/08fc0058-133b-4288-9165-14c96c5dcd7a/role-mappings/realm
  • Admin Role added for a user 
    05:01:33,664 INFO  [org.keycloak.events] (default task-27)
                              operationType=CREATE, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=REALM_ROLE_MAPPING,
                              resourcePath=users/08fc0058-133b-4288-9165-14c96c5dcd7a/role-mappings/realm
  • Realm setting update 
    05:02:29,222 INFO  [org.keycloak.events] (default task-26)
                              operationType=UPDATE, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=REALM, resourcePath=null

  • Logout all session on keycloak

    05:05:02,383 INFO  [org.keycloak.events] (default task-29)
                              operationType=ACTION, realmId=master,
                              clientId=819ce4a5-ddbd-4717-908f-a204bdabc808,
                              userId=d7cde46f-15e1-4ff8-a2cb-c5825e481438, ipAddress=192.168.219.64,
                              resourceType=REALM, resourcePath=logout-all

Examples of Security Logs

Representation for IAM and Core are same as these logs are part of ingress-gateway. Only message part of the JSON log is shown in the example.

CNCC Core

  • SCP request 
    logType=SECURITY, type=REQUEST, resourceType=SCP,
                                    userId=3314f54f-08bf-489d-b395-27bf56da1262, username=shreb,
                                    operationType=GET, scheme=http,
                                    remoteAddress=/192.168.219.64:53587,
                                    localAddress=cncc-core-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30075,
                                    resourcePath=/soothsayer/v1/canaryrelease/, queryParams={},
                                    headers={Accept=*/*, X-Requested-With=XMLHttpRequest,
                                    User-Agent=Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0)
                                    Gecko/20100101 Firefox/68.0, Connection=keep-alive,
                                    Host=cncc-core-ingress-gateway.cncc.svc.cluster.local:30075,
                                    Accept-Language=en-US,en;q=0.5, Accept-Encoding=gzip, deflate,
                                    DNT=1, Content-Type=application/json; charset=utf-8},
                                    payload={}, authenticationType=OAUTH
  • SCP response 
    logType=SECURITY, type=RESPONSE, resourceType=SCP,
                                    userId=3314f54f-08bf-489d-b395-27bf56da1262, username=shreb,
                                    status=200 OK, operationType=GET, scheme=http,
                                    resourcePath=/soothsayer/v1/canaryrelease/,
                                    headers={transfer-encoding=chunked, Connection=keep-alive,
                                    Transfer-Encoding=chunked, Content-Type=application/json,
                                    Date=Sat, 04 Jul 2020 11:58:20 GMT},
                                    payload=[{"serviceName":"n5g-eir-eic","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"namf-comm","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"namf-evts","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"namf-loc","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"namf-mt","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"nausf-auth","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5},{"serviceName":"nausf-sorprotection","canaryReleaseFlag":true,"apiFullVersion":"2.0.0","canaryTraffic":5}}],
                                    authenticationType=OAUTH

CNCC IAM

  • Request 
    logType=SECURITY, type=REQUEST, operationType=GET, scheme=http,
                                    remoteAddress=/192.168.219.64:53587,
                                    localAddress=cncc-iam-ingress-gateway.cncc.svc.cluster.local/<unresolved>:30085,
                                    resourcePath=/cncc/auth/admin/master/console/config,
                                    queryParams={}, headers={Accept=application/json,
                                    User-Agent=Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0)
                                    Gecko/20100101 Firefox/68.0, Referer=http://cncc-iam-ingress-gateway.cncc.svc.cluster.local:30085/cncc/auth/admin/master/console/, Connection=keep-alive,
                                    Host=cncc-iam-ingress-gateway.cncc.svc.cluster.local:30085,
                                    Accept-Language=en-US,en;q=0.5, Accept-Encoding=gzip, deflate,
                                    DNT=1}, payload={},
                              authenticationType=NONE
  • Response 
    logType=SECURITY, type=RESPONSE, status=200 OK,
                                    operationType=GET, scheme=http,
                                    resourcePath=/cncc/auth/admin/master/console/config,
                                    headers={transfer-encoding=chunked, Cache-Control=no-cache,
                                    X-XSS-Protection=1; mode=block, X-Frame-Options=SAMEORIGIN,
                                    Date=Mon, 06 Jul 2020 10:54:16 GMT, Connection=keep-alive,
                                    Strict-Transport-Security=max-age=31536000; includeSubDomains,
                                    X-Content-Type-Options=nosniff, Content-Type=application/json,
                                    Content-Length=211},
                                      payload={\"realm\":\"master\",\"auth-server-url\":\"http://cncc-iam-ingress-gateway.cncc.svc.cluster.local:30085/cncc/auth/\",\"ssl-required\":\"none\",\"resource\":\"security-admin-console\",\"public-client\":true,\"confidential-port\":0},
                                    authenticationType=NONE

Accessing logs

This section gives information about how to access the logs.

The CNCC application logs can be accessed in following ways:

  1. Run the following command to view logs of a cncc application pod: 
    kubectl logs -f -n <cncc_namespace> <pod_name> -c <container_name>

    Example:

    CNCC Core:
    $ kubectl logs -f -n cncc cncc-mcore-ingress-gateway-77df795fb5-wv2sb -c ingress-gateway (Security & Audit Log)
    CNCC IAM: 
    $ kubectl logs -f -n cncc cncc-iam-ingress-gateway-77df795fb5-wv2sb -c ingress-gateway (Security Log)

    $ kubectl logs -f -n cncc cncc-iam-0 (Audit Log)
  2. CNCC uses cloud native supported logging framework to view the logs.

    Example : Elasticsearch, Fluentd, and Kibana (EFK) can be used here with CNC Console to view the logs as follows:

    Figure 10-1 Log View

    img/kibana.png