7 KPIs and Traces

This chapter provides information about newly introduced and updated Key Performance Indicators (KPIs) and Traces.

7.1 Key Performance Indicators

This section provides information about Key Performance Indicators (KPIs) for Service Communication Proxy (SCP).

In the Grafana SCP dashboard, you can select a namespace from the dropdown menu.

If the dropdown menu does not populate with options, perform the following steps:
  1. Navigate to Settings and then to Variables and Data Source.
  2. Ensure the correct Data Source is selected.
  3. Verify that namespaces are being populated correctly by running the query.

7.1.1 Platform KPIs

The following table provides information on Platform metrics.

Table 7-1 SCP CPU Usage

Field Details
Description The total SCP CPU utilization per pod.
Expression sum(rate(container_cpu_usage_seconds_total{image!="",namespace="ocscp"}[2m])) by (pod,container)

Table 7-2 SCP Memory Usage

Field Details
Description The total SCP memory utilization per pod.
Expression (avg_over_time(container_memory_working_set_bytes{namespace="ocscp", container=~"istio-proxy|scp.*."}[2m]))

Table 7-3 SCP Memory Usage Percentage

Field Details
Description The total percentage of SCP memory utilization per pod.
Expression round( 100 * avg_over_time(container_memory_working_set_bytes{image!="", container_name!="POD",namespace="ocscp", container=~"scp-nrfproxy"}[2m]) / (8*1073741824) )

Note: Here, scp-nrfproxy is taken as an example; the same holds true for all the other microservices. The hardcoded memory value of 8GB (for nrfproxy) must be replaced in the Grafana dashboard file with the value that is assigned in the deployment environment, and this holds true for all the other microservices.

Table 7-4 SCP CPU Usage Percentage

Field Details
Description The total percentage of SCP CPU utilization per pod.
Expression sum(rate(container_cpu_usage_seconds_total{image!="",namespace="ocscp",container="scpc-audit"}[2m])) by (pod,container)*100/3

Note: Here, scp-worker is taken as an example; the same holds true for all the other microservices. The hardcoded CPU value of 8 (for scp-worker) must be replaced in the Grafana dashboard file with the value that is assigned in the deployment environment, and this holds true for all the other microservices.

Table 7-5 CPU Usage

Field Details
Description The total percentage of SCP CPU utilization by primary and worker pod.
Expression sum (rate (container_cpu_usage_seconds_total{image!="",node=~"k8s.*",namespace="ocscp"}[2m])) by (node)

Table 7-6 Memory Usage

Field Details
Description The total percentage of SCP memory utilization by primary and worker pod.
Expression sum (container_memory_working_set_bytes{id="/",node=~"k8s.*",namespace="ocscp"}) by (node)

Table 7-7 Prometheus CPU Utilization

Field Details
Description CPU utilization of Prometheus pods.
Expression sum(rate(container_cpu_usage_seconds_total{pod="occne-prometheus-server-5f48d747b8-lx4zn",container="prometheus-server",namespace="ocscp"} [5m]))

Table 7-8 Prometheus Memory Utilization

Field Details
Description Memory Utilization of Prometheus Pods.
Expression container_memory_usage_bytes{container="prometheus-server",namespace="ocscp"}

Table 7-9 Samples_scraped_time

Field Details
Description Total number of Metrics samples scraped by Prometheus in give time.
Expression avg_over_time(scrape_samples_scraped{namespace="ocscp"}[10m])

7.1.2 Application KPIs

The following table provides information on Application KPIs.

Table 7-10 Ingress Request Rate

Field Details
Description Total number of ingress requests sent to SCP by consumer NFs.
Expression sum(irate(ocscp_metric_http_rx_req_total{namespace="ocscp", ocscp_consumer_host!="OCSCP-INITIATED-MSG"}[5m])) by (ocscp_nf_type,ocscp_service_instance_id)

Table 7-11 Egress Request Rate

Field Details
Description Total number of egress requests sent to producer NFs by SCP.
Expression sum(irate(ocscp_metric_http_tx_req_total{namespace="ocscp"}[2m])) by (ocscp_nf_type)

Table 7-12 Ingress Response Rate

Field Details
Description Total number of ingress responses sent to SCP by producer NFs.
Expression sum(irate(ocscp_metric_http_rx_res_total{namespace="ocscp"}[2m])) by (ocscp_response_code_class, ocscp_producer_host)

Table 7-13 Egress Response Rate

Field Details
Description Total number of egress responses sent to consumer NFs by SCP.
Expression sum(irate(ocscp_metric_http_tx_res_total{namespace="ocscp", ocscp_consumer_host!="OCSCP-INITIATED-MSG"}[2m]))

Table 7-14 Response Processing Time

Field Details
Description The rate of processing time by SCP for ingress responses into the time buckets.
Expression sum(irate(ocscp_metric_response_processing_time_total{namespace="ocscp"}[2m]))by(ocscp_processing_time)

Table 7-15 Request Processing Time

Field Details
Description The rate of processing time by SCP for ingress requests into the time buckets.
Expression sum(irate(ocscp_metric_request_processing_time_total{namespace="ocscp"}[2m]))by(ocscp_processing_time)

Table 7-16 Transaction Success Rate

Field Details
Description The rate of successful transaction response sent by SCP for a request.
Expression ((sum(irate(ocscp_metric_http_tx_res_total{ocscp_response_code_class="2", kubernetes_namespace=~"$namespace"}[2m])))/(sum(irate(ocscp_metric_http_tx_res_total{kubernetes_namespace=~"$namespace"}[2m]))))*100

Table 7-17 Response Code Rate

Field Details
Description The rate of different response codes class on SCP.
Expression sum(irate(ocscp_metric_http_rx_res_total{kubernetes_namespace="$namespace"}[2m])) by (ocscp_response_code_class)

Table 7-18 scp-worker replicas

Field Details
Description The total number of ready replicas for the scp-worker service.
Expression kube_replicaset_status_ready_replicas{replicaset =~ '.*scp-worker.*',namespace="$namespace"}

Table 7-19 NF Notifications Success

Field Details
Description The total number of notifications successfully processed by SCP.
Expression sum(ocscp_notification_nf_profile_processing_success_total)

Table 7-20 NF Notifications Failures

Field Details
Description The total number of notifications failed to be processed by SCP.
Expression sum(ocscp_notification_nf_profile_processing_failure_total)

Table 7-21 Audit Requests Total

Field Details
Description The total number of audit requests sent by SCP.
Expression sum(ocscp_audit_tx_req_total{namespace="ocscp"})

Table 7-22 Audit Responses Total

Field Details
Description The total number of audit responses received by SCP.
Expression sum(ocscp_audit_rx_res_total{namespace="ocscp"})

Table 7-23 NF Subscription Success

Field Details
Description The total number of successful subscriptions on SCP per NF type.
Expression sum(ocscp_subscription_nf_success_total)

Table 7-24 NF Subscription Failures

Field Details
Description The total number of failed subscriptions on SCP per NF type.
Expression sum(ocscp_subscription_nf_failure_total)

Table 7-25 NRF Registration Success

Field Details
Description The total number of successful NRF registrations on SCP.
Expression sum(ocscp_subscription_nrf_registration_success_total)

Table 7-26 NRF Registration Failure

Field Details
Description The total number of failed NRF registrations on SCP.
Expression sum(ocscp_subscription_nrf_registration_failure_total)

Table 7-27 NRF Heartbeat Success

Field Details
Description The total number of successful NRF Heartbeats on SCP.
Expression sum(ocscp_subscription_nrf_heartbeat_success_total{namespace="ocscp"})

Table 7-28 NRF Heartbeat Failures

Field Details
Description The total number of failed NRF Heartbeats on SCP.
Expression sum(ocscp_subscription_nrf_heartbeat_failures_total{namespace="ocscp"})

7.1.3 Feature KPIs

The following table provides information on Feature KPIs.

Table 7-29 Access Token Count in Response

Field Details
Description The rate of responses contains the 3gpp-Sbi-Access-Token header.
Expression sum(irate(ocscp_worker_access_token_forwarded_in_tx_res_total{namespace="ocscp"}[2m]))

Table 7-30 Received Response count for nnrrf-oauth2 service

Field Details
Description Rate of nnrf-oaut2 service responses from NRF.
Expression sum(irate(ocscp_metric_http_rx_res_total{namespace="ocscp",ocscp_nf_service_type="nnrf-oauth2",ocscp_response_code="200"}[2m]))by(pod)

Table 7-31 Coherence Cache Size for AccessToken

Field Details
Description Number of access tokens in the cache.
Expression coherence_cache_size{namespace="ocscp",name="accessTokenMap"}

Table 7-32 Access Token Count in Transmitted Request

Field Details
Description Rate of SCP sending the 3gpp-Sbi-Access-Token header in the request.
Expression sum(irate(ocscp_worker_access_token_forwarded_in_tx_req_total{namespace="ocscp"}[2m]))

Table 7-33 SCP Worker - Discovery Tx Requests Rate to NRF Proxy

Field Details
Description Rate of worker sends an on-demand discovery request to nrfproxy.
Expression sum(irate(ocscp_metric_ondemand_discovery_tx_req_total{namespace="ocscp"}[2m]))

Table 7-34 SCP Worker - Discovery Rx Response Rate from NRF Proxy

Field Details
Description Rate of worker receives on-demand discovery response from nrfproxy.
Expression sum(irate(ocscp_metric_ondemand_discovery_rx_res_total{namespace="ocscp"}[2m]))

Table 7-35 NRF Proxy - Discovery Tx Requests Rate to NRF

Field Details
Description Rate of discovery requests sent to the NRF by nrfproxy.
Expression sum(irate(ocscp_nrf_nf_discovery_tx_req_total{namespace="ocscp"}[2m]))by(pod)

Table 7-36 NRF Proxy - Discovery Rx Response Rate from NRF

Field Details
Description Rate of discovery responses received on nrfproxy from NRF.
Expression sum(irate(ocscp_nrf_nf_discovery_rx_res_total{namespace="ocscp"}[2m]))by(pod)

Table 7-37 SCP-Worker - Local Discovery Requests Resolution Rate(Cache Hits)

Field Details
Description Rate of worker was able to resolve discovery request using local cache.
Expression sum(irate(ocscp_metric_ondemand_discovery_cache_match_total{namespace="ocscp"}[2m]))by(pod)

Table 7-38 NRF Proxy Generated Error response Rate

Field Details
Description Rate of error responses generated by nrfproxy (excluding nrf-returned errors).
Expression sum(irate(ocscp_nrfproxy_generated_error_response_total{namespace="ocscp"}[2m]))by(pod)

Table 7-39 SCP LCI Tx Rate

Field Details
Description Rate of messages forwarded with SCP's LCI header.
Expression sum(irate(ocscp_metric_scp_lci_tx_total{namespace=~"ocscp"}[2m]))

Table 7-40 Per NF Load

Field Details
Description The current load of peer NF.
Expression ocscp_nf_load{namespace=~"ocscp"}

Table 7-41 Peer LCI Rx Rate

Field Details
Description Rate of messages received with NF's LCI header.
Expression sum(irate(ocscp_metric_nf_lci_rx_total{namespace=~"ocscp"}[2m]))

Table 7-42 Peer LCI Tx Rate

Field Details
Description Rate of messages forwarded with NF's LCI header.
Expression sum(irate(ocscp_metric_nf_lci_tx_total{namespace=~"ocscp"}[2m]))

Table 7-43 Ingress Rate Limit - Request Throttle Rate

Field Details
Description Number of requests throttled due to ingress rate limitations.
Expression sum(irate(ocscp_metric_ingress_rate_limiting_throttle_req_total{namespace="ocscp"}[2m]))by(ocscp_consumer_fqdn, ocscp_configured_ingress_rate ,ocscp_ingress_rl_bucketkey)

Table 7-44 Ingress Rate Limit - Rate Limit Not Applied Rate

Field Details
Description Number of requests routed without ingress rate limiting treatment.
Expression sum(irate(ocscp_metric_ingress_rate_limiting_not_applied_req_total{namespace="ocscp"}[2m]))by(ocscp_consumer_fqdn,ocscp_cause)

Table 7-45 Egress Rate Limit - Request Throttle Rate

Field Details
Description Number of requests throttled due to egress rate limitating.
Expression sum(irate(ocscp_metric_egress_rate_limiting_throttle_req_total{namespace="ocscp"}[2m]))by(ocscp_nf_type,ocscp_producer_host, ocscp_nf_service_type, ocscp_configured_egress_rate, ocscp_egress_rl_bucketkey, ocscp_nf_instance_id)

Table 7-46 Egress Rate Limit - Rate Limit Not Applied Rate

Field Details
Description Number of requests routed without egress rate limiting treatment.
Expression sum(irate(ocscp_metric_egress_rate_limiting_not_applied_req_total{namespace="ocscp"}[2m]))by(ocscp_nf_type,ocscp_producer_host, ocscp_cause)

Table 7-47 SCP Cache Size

Field Details
Description Coherence cache size for different caches.
Expression coherence_cache_size

Table 7-48 SCP Cache Hits

Field Details
Description Number of coherence cache hits.
Expression sum(irate(coherence_cache_hits{namespace="ocscp"}[2m])) by (name)

Table 7-49 Message Request Rate Per Rate Limit Bucket Key

Field Details
Description Rate of request received per rate limit key.
Expression sum(irate(ocscp_metric_message_current_request_rate_total{namespace="ocscp"}[2m]))by (bucketKey)

Table 7-50 SCP OCI Conveyance Rate

Field Details
Description Number of messages with SCP’s OCI conveyed to peers.
Expression sum(irate(ocscp_worker_oci_conveyance_total{namespace=~"ocscp"}[2m]))

Table 7-51 Peer OCI Conveyance Rate

Field Details
Description Number of messages with a peer’s OCI conveyed to peers.
Expression sum(irate(ocscp_worker_peer_oci_conveyance_total{namespace=~"ocscp"}[2m]))

Table 7-52 OCI Enforced Rate

Field Details
Description Number of messages on which OCI is enforced that are either rerouted to alternate NFs or an error response is sent.
Expression sum(irate(ocscp_worker_oci_enforced_total{namespace=~"ocscp"}[2m]))

Table 7-53 OCI Unsupported Rx Rate

Field Details
Description Number of messages received with unsupported scope.
Expression sum(irate(ocscp_worker_rx_unsupported_oci_scope_total{namespace=~"ocscp"}[2m]))

Table 7-54 SCP Worker - Mediation Tx Requests Rate

Field Details
Description The total number of messages that forwarded to the mediation service from SCP.
Expression sum(irate(ocscp_mediation_http_tx_total{namespace="ocscp"}[2m])) by (pod)

Table 7-55 SCP Worker -Mediation Rx Response Rate

Field Details
Description The total number of messages that the mediation service sends back as a response to SCP.
Expression sum(irate(ocscp_mediation_http_rx_total{namespace="ocscp"}[2m])) by (pod)

Table 7-56 SCP Worker - Mediation Upstream Service Time

Field Details
Description This indicates the total time taken by the mediation to process the request.
Expression sum(irate(ocscp_mediation_upstream_service_time_ms_total{namespace="ocscp"}[2m])) by (ocscp_med_upstream_service_time)

7.2 Traces

The following table provides information of traces for SCP.

Table 7-57 SCP Traces

Field Name Description
component The software package, framework, library, or module that generated the associated Span.

Request or Response Type

: common
node_id Local information.

Request or Response Type

: common
method HTTP method of the request for the associated Span. Example: "GET","POST".

Request or Response Type

: request,common
scheme Url scheme is http.

Request or Response Type

: request
authority Authority give you details about registered name or server address, along with optional port and user information.

Request or Response Type

: request
path A path consists of a sequence of path segments separated by a slash ("/") character.

Request or Response Type

: request
3gpp-sbi-message-priority This header shall be included in HTTP/2 messages when a priority for the message needs to be conveyed.

Request or Response Type

: request
x-forwarded-for To identify the originating IP address of a client.

Request or Response Type

: request
x-forwarded-proto To determine the protocol used between the client and SCP.

Request or Response Type

: request
x-envoy-internal service wants to know whether a request is internal origin or not.

Request or Response Type

: request
via It is used for tracking message forwards, avoiding request loops, and identifying the protocol capabilities of senders along the request/response chain.

Request or Response Type

: request
x-request-id The x-request-idheader is used to uniquely identify a request as well as perform stable access logging and tracing.

Request or Response Type

: common
payload http request body.

Request or Response Type

: request, common
status To determine the http request has been succeeded or not.

Request or Response Type

: response
content-type The Content-Type entity header is used to indicate the media type of the resource.

Request or Response Type

: response
content-length The Content-Length header indicates the size of the entity body in the message, in bytes.

Request or Response Type

: response
server The Server response-header field contains information about the software used by the origin server to handle the request.

Request or Response Type

: response
date The time and date, when the request is processed.

Request or Response Type

: response
x-envoy-upstream-service-time Contains the time in milliseconds spent by the upstream host processing the request.

Request or Response Type

: response
location To provide information about the location of a newly created resource.

Request or Response Type

: response
payload http response body.

Request or Response Type

: response
http.url Specifies the request's URL.

Request or Response Type

: request, common
downstream_cluster A downstream host connects to Envoy, sends requests, and receives responses.

Request or Response Type

: common
user_agent When your browser connects to a website, it includes a User-Agentfield in its HTTPheader.

Request or Response Type

: request, common
http.protocol The communication between client and server over HTTP/2.

Request or Response Type

: common
request_size HTTP header size.

Request or Response Type

: common
upstream_cluster An upstream host receives connections and requests from Envoy and returns responses.

Request or Response Type

: common
http.ststus_code HTTP response status code for the associated Span. Example: 200, 503, 404

Request or Response Type

: common
response_size HTTP header size.

Request or Response Type

: common
response_flag Additional details about the response or connection, if any.

Request or Response Type

:common
span.kind Specifies the role of the span in a RPC communication. In case of HTTP communication it is searching client and server values for this tag.

Request or Response Type

: common
error True, if and only if, the application considers the operation represented by the Span to have failed.

Request or Response Type

: common
x-request-id The x-request-idheader is used by Envoy to uniquely identify a request as well as perform stable access logging and tracing

Request or Response Type

: request
x-b3-traceid The x-b3-traceidHTTP header is used by the Zipkin tracer in Envoy. The TraceId is 64-bit in length and indicates the overall ID of the trace. Every span in a trace shares this ID.

Request or Response Type

: request
x-b3-spanid The x-b3-spanidHTTP header is used by the Zipkin tracer in Envoy. The SpanId is 64-bit in length and indicates the position of the current operation in the trace tree.

Request or Response Type

: request
x-b3-sampled The x-b3-sampledHTTP header is used by the Zipkin tracer in Envoy. When the sampled flag is either not specified or set to 1, the span will be reported to the tracing system.

Request or Response Type

: request
ueidentitytype NF type.

Request or Response Type

: request
ueidentityvalue SUPI range for NF type.

Request or Response Type

: request
x-envoy-expected-rq-timeout-ms This is the time in milliseconds the router expects the request to be completed.

Request or Response Type

: request