5 Metrics
CNE uses Prometheus to collect metrics from CNE services. These metrics are used by the alert rules defined by CNE to detect abnormal conditions. This section provides details about CNE metrics.
5.1 Cloud Native Load Balancer Metrics
CNE provides custom CNLB metrics that can be used to observe the cluster and monitor the cluster health from Prometheus, these metrics can be used in Grafana for interactive monitoring of the cluster.
Egress Network Metrics
Table 5-1 cnlb_egr_outpkt
| Field | Details |
|---|---|
| Description | Total outgoing packets from the egress IP. |
| Type | Counter |
| Dimensions | egress_ip:
|
Table 5-2 cnlb_egr_inpkt
| Field | Details |
|---|---|
| Description | Total incoming packets to the egress IP. |
| Type | Counter |
| Dimensions | egress_ip:
|
Table 5-3 cnlb_egr_inbytes
| Field | Details |
|---|---|
| Description | Total incoming bytes to the egress IP. |
| Type | Counter |
| Dimensions | egress_ip:
|
Table 5-4 cnlb_egr_outbytes
| Field | Details |
|---|---|
| Description | Total outgoing bytes from the egress IP. |
| Type | Counter |
| Dimensions | egress_ip:
|
Internal Multus (Interface) Metrics
Table 5-5 cnlb_interface_int_inpkt
| Field | Details |
|---|---|
| Description | Total incoming packets on internal multus master interface. |
| Type | Counter |
| Dimensions |
|
Table 5-6 cnlb_interface_int_outpkt
| Field | Details |
|---|---|
| Description | Total outgoing packets from internal multus master interface. |
| Type | Counter |
| Dimensions |
|
Table 5-7 cnlb_interface_ext_inpkt
| Field | Details |
|---|---|
| Description | Total incoming packets on external multus master interface. |
| Type | Counter |
| Dimensions |
|
Table 5-8 cnlb_interface_ext_outpkt
| Field | Details |
|---|---|
| Description | Total outgoing packets from external multus master interface. |
| Type | Counter |
| Dimensions |
|
Service Ingress IPVS & Connection Metrics
Table 5-9 connections_total
| Field | Details |
|---|---|
| Description | Total connection count for backend_ip (active tracked for each backend/service). |
| Type | Gauge |
| Dimensions |
|
Table 5-10 cnlb_inbyte
| Field | Details |
|---|---|
| Description | Total incoming bytes from remote to service IP. |
| Type | Gauge |
| Dimensions |
|
Table 5-11 cnlb_inpkt
| Field | Details |
|---|---|
| Description | Total incoming packets received from remote to service IP (scrape interval). |
| Type | Gauge |
| Dimensions |
|
Table 5-12 cnlb_outbyte
| Field | Details |
|---|---|
| Description | Total outgoing bytes from app pod to remote. |
| Type | Gauge |
| Dimensions |
|
Table 5-13 cnlb_outpkt
| Field | Details |
|---|---|
| Description | Total outgoing packets sent from app pod to remote. |
| Type | Gauge |
| Dimensions |
|
5.2 CNE Metrics
CNE offers custom metrics that enable more detailed monitoring of node behavior. These metrics are gathered from cluster nodes and provide insights into each vCPU as well as every network interface on each node.
Mpstat Metrics
These metrics are collected by saving the output of the mpstat
command in a list and processing it as needed. After Prometheus scrapes the gauges,
the list is cleared, retaining only the most recent value. Each metric displays data
for every vCPU on the selected host.
Table 5-14 cne_metrics_mpstat_idle_average
| Field | Details |
|---|---|
| Description | Captures the %idle value every second, stores it on a list and returns the average of it. |
| Type | Gauge |
| Dimensions | host:
|
Table 5-15 cne_metrics_mpstat_idle_low_watermark
| Field | Details |
|---|---|
| Description | Captures the %idle value every second and stores it on a list and returns the lowest value. |
| Type | Gauge |
| Dimensions | host:
|
Table 5-16 cne_metrics_mpstat_isoft_average
| Field | Details |
|---|---|
| Description | Captures the %soft value every second, stores it on a list and returns the average of it. |
| Type | Gauge |
| Dimensions | host:
|
Table 5-17 cne_metrics_mpstat_soft_high_watermark
| Field | Details |
|---|---|
| Description | Captures the %idle value every second and stores it on a list and returns the highest value. |
| Type | Gauge |
| Dimensions | host:
|
Network Metrics
These metrics are collected by storing the output of the ip -s -s link ls
{interface} command in a list. The Gauge value is updated using the
delta between the most recent entry and the previous one in the list. After
Prometheus scrapes the gauges, the list is cleared, keeping only the latest value.
Each metric displays data for every network interface on the selected host.
Table 5-18 cne_metrics_network_bytes_sent
| Field | Details |
|---|---|
| Description | Captures the delta value for TX: bytes sent by each interface every second. |
| Type | Gauge |
| Dimensions | host:
|
Table 5-19 cne_metrics_network_packets_sent
| Field | Details |
|---|---|
| Description | Captures the delta value for TX: packets sent by each interface every second. |
| Type | Gauge |
| Dimensions | host:
|
Table 5-20 cne_metrics_network_errors_sent
| Field | Details |
|---|---|
| Description | Captures the delta value for TX: errors sent by each interface every second. |
| Type | Gauge |
| Dimensions | host:
|
Reference: For more information on how to visualize these metrics on Grafana, see unresolvable-reference.html#GUID-F15443A7-84E2-4B24-B3AE-1986AB4C4484.