7 KPIs and Traces
This chapter provides information about newly introduced and updated Key Performance Indicators (KPIs) and Traces.
7.1 Key Performance Indicators
7.1.1 Platform KPIs
Table 7-1 SCP CPU Usage
| Field | Details |
|---|---|
| Description | The total SCP CPU utilization per pod. |
| Expression | sum(rate(container_cpu_usage_seconds_total{image!="",namespace="ocscp"}[2m])) by (pod,container) |
Table 7-2 SCP Memory Usage
| Field | Details |
|---|---|
| Description | The total SCP memory utilization per pod. |
| Expression | (avg_over_time(container_memory_working_set_bytes{namespace="ocscp", container=~"istio-proxy|scp.*."}[2m])) |
Table 7-3 SCP Memory Usage Percentage
| Field | Details |
|---|---|
| Description | The total percentage of SCP memory utilization per pod. |
| Expression | round( 100 *
avg_over_time(container_memory_working_set_bytes{image!="",
container_name!="POD",namespace="ocscp",
container=~"scp-nrfproxy"}[2m]) / (8*1073741824) )
Note: Here, scp-nrfproxy is taken as an example; the same holds true for all the other microservices. The hardcoded memory value of 8GB (for nrfproxy) must be replaced in the Grafana dashboard file with the value that is assigned in the deployment environment, and this holds true for all the other microservices. |
Table 7-4 SCP CPU Usage Percentage
| Field | Details |
|---|---|
| Description | The total percentage of SCP CPU utilization per pod. |
| Expression | sum(rate(container_cpu_usage_seconds_total{image!="",namespace="ocscp",container="scpc-audit"}[2m]))
by (pod,container)*100/3
Note: Here, scp-worker is taken as an example; the same holds true for all the other microservices. The hardcoded CPU value of 8 (for scp-worker) must be replaced in the Grafana dashboard file with the value that is assigned in the deployment environment, and this holds true for all the other microservices. |
Table 7-5 CPU Usage
| Field | Details |
|---|---|
| Description | The total percentage of SCP CPU utilization by master and worker pod. |
| Expression | sum (rate (container_cpu_usage_seconds_total{image!="",node=~"k8s.*",namespace="ocscp"}[2m])) by (node) |
Table 7-6 Memory Usage
| Field | Details |
|---|---|
| Description | The total percentage of SCP memory utilization by master and worker pod. |
| Expression | sum (container_memory_working_set_bytes{id="/",node=~"k8s.*",namespace="ocscp"}) by (node) |
Table 7-7 Prometheus CPU Utilization
| Field | Details |
|---|---|
| Description | CPU utilization of Prometheus pods. |
| Expression | sum(rate(container_cpu_usage_seconds_total{pod="occne-prometheus-server-5f48d747b8-lx4zn",container="prometheus-server",namespace="ocscp"} [5m])) |
Table 7-8 Prometheus Memory Utilization
| Field | Details |
|---|---|
| Description | Memory Utilization of Prometheus Pods. |
| Expression | container_memory_usage_bytes{container="prometheus-server",namespace="ocscp"} |
Table 7-9 Samples_scraped_time
| Field | Details |
|---|---|
| Description | Total number of Metrics samples scraped by Prometheus in give time. |
| Expression | avg_over_time(scrape_samples_scraped{namespace="ocscp"}[10m]) |
7.1.2 Application KPIs
The following table provides information on Application KPIs.
Table 7-10 Ingress Request Rate
| Field | Details |
|---|---|
| Description | Total number of ingress requests sent to SCP by consumer NFs. |
| Expression | sum(irate(ocscp_metric_http_rx_req_total{namespace="ocscp", ocscp_consumer_host!="OCSCP-INITIATED-MSG"}[5m])) by (ocscp_nf_type,ocscp_service_instance_id) |
Table 7-11 Egress Request Rate
| Field | Details |
|---|---|
| Description | Total number of egress requests sent to producer NFs by SCP. |
| Expression | sum(irate(ocscp_metric_http_tx_req_total{namespace="ocscp"}[2m])) by (ocscp_nf_type) |
Table 7-12 Ingress Response Rate
| Field | Details |
|---|---|
| Description | Total number of ingress responses sent to SCP by producer NFs. |
| Expression | sum(irate(ocscp_metric_http_rx_res_total{namespace="ocscp"}[2m])) by (ocscp_response_code_class, ocscp_producer_fqdn) |
Table 7-13 Egress Response Rate
| Field | Details |
|---|---|
| Description | Total number of egress responses sent to consumer NFs by SCP. |
| Expression | sum(irate(ocscp_metric_http_tx_res_total{namespace="ocscp", ocscp_consumer_host!="OCSCP-INITIATED-MSG"}[2m])) |
Table 7-14 Response Processing Time
| Field | Details |
|---|---|
| Description | The rate of processing time by SCP for ingress responses into the time buckets. |
| Expression | sum(irate(ocscp_metric_response_processing_time_total{namespace="ocscp"}[2m]))by(ocscp_processing_time) |
Table 7-15 Request Processing Time
| Field | Details |
|---|---|
| Description | The rate of processing time by SCP for ingress requests into the time buckets. |
| Expression | sum(irate(ocscp_metric_request_processing_time_total{namespace="ocscp"}[2m]))by(ocscp_processing_time) |
Table 7-16 Transaction Success Rate
| Field | Details |
|---|---|
| Description | The rate of successful transaction response sent by SCP for a request. |
| Expression | ((sum(irate(ocscp_metric_http_tx_res_total{ocscp_response_code_class="2", kubernetes_namespace=~"$namespace"}[2m])))/(sum(irate(ocscp_metric_http_tx_res_total{kubernetes_namespace=~"$namespace"}[2m]))))*100 |
Table 7-17 Response Code Rate
| Field | Details |
|---|---|
| Description | The rate of different response codes class on SCP. |
| Expression | sum(irate(ocscp_metric_http_rx_res_total{kubernetes_namespace="$namespace"}[2m])) by (ocscp_response_code_class) |
Table 7-18 scp-worker replicas
| Field | Details |
|---|---|
| Description | The total number of ready replicas for the scp-worker service. |
| Expression | kube_replicaset_status_ready_replicas{replicaset =~ '.*scp-worker.*',namespace="$namespace"} |
Table 7-19 NF Notifications Success
| Field | Details |
|---|---|
| Description | The total number of notifications successfully processed by SCP. |
| Expression | sum(ocscp_notification_nf_profile_processing_success_total) |
Table 7-20 NF Notifications Failures
| Field | Details |
|---|---|
| Description | The total number of notifications failed to be processed by SCP. |
| Expression | sum(ocscp_notification_nf_profile_processing_failure_total) |
Table 7-21 Audit Requests Total
| Field | Details |
|---|---|
| Description | The total number of audit requests sent by SCP. |
| Expression | sum(ocscp_audit_tx_req_total{namespace="ocscp"}) |
Table 7-22 Audit Responses Total
| Field | Details |
|---|---|
| Description | The total number of audit responses received by SCP. |
| Expression | sum(ocscp_audit_rx_res_total{namespace="ocscp"}) |
Table 7-23 NF Subscription Success
| Field | Details |
|---|---|
| Description | The total number of successful subscriptions on SCP per NF type. |
| Expression | sum(ocscp_subscription_nf_success_total) |
Table 7-24 NF Subscription Failures
| Field | Details |
|---|---|
| Description | The total number of failed subscriptions on SCP per NF type. |
| Expression | sum(ocscp_subscription_nf_failure_total) |
Table 7-25 NRF Registration Success
| Field | Details |
|---|---|
| Description | The total number of successful NRF registrations on SCP. |
| Expression | sum(ocscp_subscription_nrf_registration_success_total) |
Table 7-26 NRF Registration Failure
| Field | Details |
|---|---|
| Description | The total number of failed NRF registrations on SCP. |
| Expression | sum(ocscp_subscription_nrf_registration_failure_total) |
Table 7-27 NRF Heartbeat Success
| Field | Details |
|---|---|
| Description | The total number of successful NRF Heartbeats on SCP. |
| Expression | sum(ocscp_subscription_nrf_heartbeat_success_total{namespace="ocscp"}) |
Table 7-28 NRF Heartbeat Failures
| Field | Details |
|---|---|
| Description | The total number of failed NRF Heartbeats on SCP. |
| Expression | sum(ocscp_subscription_nrf_heartbeat_failures_total{namespace="ocscp"}) |
7.1.3 Feature KPIs
Table 7-29 Access Token Count in Response
| Field | Details |
|---|---|
| Description | The rate of responses contains the 3gpp-Sbi-Access-Token header. |
| Expression | sum(irate(ocscp_worker_access_token_forwarded_in_tx_res_total{namespace="ocscp"}[2m])) |
Table 7-30 Received Response count for nnrrf-oauth2 service
| Field | Details |
|---|---|
| Description | Rate of nnrf-oaut2 service responses from NRF. |
| Expression | sum(irate(ocscp_metric_http_rx_res_total{namespace="ocscp",ocscp_nf_service_type="nnrf-oauth2",ocscp_response_code="200"}[2m]))by(pod) |
Table 7-31 Coherence Cache Size for AccessToken
| Field | Details |
|---|---|
| Description | Number of access tokens in the cache. |
| Expression | coherence_cache_size{namespace="ocscp",name="accessTokenMap"} |
Table 7-32 Access Token Count in Transmitted Request
| Field | Details |
|---|---|
| Description | Rate of SCP sending the 3gpp-Sbi-Access-Token header in the request. |
| Expression | sum(irate(ocscp_worker_access_token_forwarded_in_tx_req_total{namespace="ocscp"}[2m])) |
Table 7-33 SCP Worker - Discovery Tx Requests Rate to NRF Proxy
| Field | Details |
|---|---|
| Description | Rate of worker sends an on-demand discovery request to nrfproxy. |
| Expression | sum(irate(ocscp_metric_ondemand_discovery_tx_req_total{namespace="ocscp"}[2m])) |
Table 7-34 SCP Worker - Discovery Rx Response Rate from NRF Proxy
| Field | Details |
|---|---|
| Description | Rate of worker receives on-demand discovery response from nrfproxy. |
| Expression | sum(irate(ocscp_metric_ondemand_discovery_rx_res_total{namespace="ocscp"}[2m])) |
Table 7-35 NRF Proxy - Discovery Tx Requests Rate to NRF
| Field | Details |
|---|---|
| Description | Rate of discovery requests sent to the NRF by nrfproxy. |
| Expression | sum(irate(ocscp_nrf_nf_discovery_tx_req_total{namespace="ocscp"}[2m]))by(pod) |
Table 7-36 NRF Proxy - Discovery Rx Response Rate from NRF
| Field | Details |
|---|---|
| Description | Rate of discovery responses received on nrfproxy from NRF. |
| Expression | sum(irate(ocscp_nrf_nf_discovery_rx_res_total{namespace="ocscp"}[2m]))by(pod) |
Table 7-37 SCP-Worker - Local Discovery Requests Resolution Rate(Cache Hits)
| Field | Details |
|---|---|
| Description | Rate of worker was able to resolve discovery request using local cache. |
| Expression | sum(irate(ocscp_metric_ondemand_discovery_cache_match_total{namespace="ocscp"}[2m]))by(pod) |
Table 7-38 NRF Proxy Generated Error response Rate
| Field | Details |
|---|---|
| Description | Rate of error responses generated by nrfproxy (excluding nrf-returned errors). |
| Expression | sum(irate(ocscp_nrfproxy_generated_error_response_total{namespace="ocscp"}[2m]))by(pod) |
Table 7-39 SCP LCI Tx Rate
| Field | Details |
|---|---|
| Description | Rate of messages forwarded with SCP's LCI header. |
| Expression | sum(irate(ocscp_metric_scp_lci_tx_total{namespace=~"ocscp"}[2m])) |
Table 7-40 Per NF Load
| Field | Details |
|---|---|
| Description | The current load of peer NF. |
| Expression | ocscp_nf_load{namespace=~"ocscp"} |
Table 7-41 Peer LCI Rx Rate
| Field | Details |
|---|---|
| Description | Rate of messages received with NF's LCI header. |
| Expression | sum(irate(ocscp_metric_nf_lci_rx_total{namespace=~"ocscp"}[2m])) |
Table 7-42 Peer LCI Tx Rate
| Field | Details |
|---|---|
| Description | Rate of messages forwarded with NF's LCI header. |
| Expression | sum(irate(ocscp_metric_nf_lci_tx_total{namespace=~"ocscp"}[2m])) |
Table 7-43 Ingress Rate Limit - Request Throttle Rate
| Field | Details |
|---|---|
| Description | Number of requests throttled due to ingress rate limitations. |
| Expression | sum(irate(ocscp_metric_ingress_rate_limiting_throttle_req_total{namespace="ocscp"}[2m]))by(ocscp_consumer_fqdn, ocscp_configured_ingress_rate ,ocscp_ingress_rl_bucketkey) |
Table 7-44 Ingress Rate Limit - Rate Limit Not Applied Rate
| Field | Details |
|---|---|
| Description | Number of requests routed without ingress rate limiting treatment. |
| Expression | sum(irate(ocscp_metric_ingress_rate_limiting_not_applied_req_total{namespace="ocscp"}[2m]))by(ocscp_consumer_fqdn,ocscp_cause) |
Table 7-45 Egress Rate Limit - Request Throttle Rate
| Field | Details |
|---|---|
| Description | Number of requests throttled due to egress rate limitating. |
| Expression | sum(irate(ocscp_metric_egress_rate_limiting_throttle_req_total{namespace="ocscp"}[2m]))by(ocscp_nf_type,ocscp_producer_fqdn, ocscp_nf_service_type, ocscp_configured_egress_rate, ocscp_egress_rl_bucketkey, ocscp_nf_instance_id) |
Table 7-46 Egress Rate Limit - Rate Limit Not Applied Rate
| Field | Details |
|---|---|
| Description | Number of requests routed without egress rate limiting treatment. |
| Expression | sum(irate(ocscp_metric_egress_rate_limiting_not_applied_req_total{namespace="ocscp"}[2m]))by(ocscp_nf_type,ocscp_producer_fqdn, ocscp_cause) |
Table 7-47 SCP Cache Size
| Field | Details |
|---|---|
| Description | Coherence cache size for different caches. |
| Expression | coherence_cache_size |
Table 7-48 SCP Cache Hits
| Field | Details |
|---|---|
| Description | Number of coherence cache hits. |
| Expression | sum(irate(coherence_cache_hits{namespace="ocscp"}[2m])) by (name) |
Table 7-49 Message Request Rate Per Rate Limit Bucket Key
| Field | Details |
|---|---|
| Description | Rate of request received per rate limit key. |
| Expression | sum(irate(ocscp_metric_message_current_request_rate_total{namespace="ocscp"}[2m]))by (bucketKey) |
Table 7-50 SCP OCI Conveyance Rate
| Field | Details |
|---|---|
| Description | Number of messages with SCP’s OCI conveyed to peers. |
| Expression | sum(irate(ocscp_worker_oci_conveyance_total{namespace=~"ocscp"}[2m])) |
Table 7-51 Peer OCI Conveyance Rate
| Field | Details |
|---|---|
| Description | Number of messages with a peer’s OCI conveyed to peers. |
| Expression | sum(irate(ocscp_worker_peer_oci_conveyance_total{namespace=~"ocscp"}[2m])) |
Table 7-52 OCI Enforced Rate
| Field | Details |
|---|---|
| Description | Number of messages on which OCI is enforced that are either rerouted to alternate NFs or an error response is sent. |
| Expression | sum(irate(ocscp_worker_oci_enforced_total{namespace=~"ocscp"}[2m])) |
Table 7-53 OCI Unsupported Rx Rate
| Field | Details |
|---|---|
| Description | Number of messages received with unsupported scope. |
| Expression | sum(irate(ocscp_worker_rx_unsupported_oci_scope_total{namespace=~"ocscp"}[2m])) |
Table 7-54 SCP Worker - Mediation Tx Requests Rate
| Field | Details |
|---|---|
| Description | The total number of messages that forwarded to the mediation service from SCP. |
| Expression | sum(irate(ocscp_mediation_http_tx_total{namespace="ocscp"}[2m])) by (pod) |
Table 7-55 SCP Worker -Mediation Rx Response Rate
| Field | Details |
|---|---|
| Description | The total number of messages that the mediation service sends back as a response to SCP. |
| Expression | sum(irate(ocscp_mediation_http_rx_total{namespace="ocscp"}[2m])) by (pod) |
Table 7-56 SCP Worker - Mediation Upstream Service Time
| Field | Details |
|---|---|
| Description | This indicates the total time taken by the mediation to process the request. |
| Expression | sum(irate(ocscp_mediation_upstream_service_time_ms_total{namespace="ocscp"}[2m])) by (ocscp_med_upstream_service_time) |