Prometheus Metrics Processor

Overview

The Unified Assurance Prometheus Metrics Processor microservice is part of the Prometheus metric collection pipeline. The Prometheus Metrics Processor is responsible for filtering the metrics collected by Prometheus server and then inserting them into the Metric database.

Prerequisites

  1. A microservice cluster must be setup. Refer to Microservice Cluster Setup.
  2. Kube Prometheus Stack deployed with override configuration:
  3. Uninstall prometheus if it’s already running: a1helm uninstall prometheus -n a1-monitoring
  4. Reinstall prometheus with override: a1helm install prometheus assure1/kube-prometheus-stack -n a1-monitoring --set global.imageRegistry=$WEBFQDN -f <overide-file-name>
    Contents of override file: 
    prometheus:
  prometheusSpec:
    remoteWrite:
    - name: prometheus-metrics-processor
      url: "http://prometheus-metrics-processor.a1-monitoring.svc.cluster.local:8185/write"
      metadataConfig:
        send: false

Setup

su - assure1
export NAMESPACE=a1-monitoring
export WEBFQDN=<Primary Presentation Web FQDN> 
a1helm install prometheus-metrics-processor assure1/prometheus-metrics-processor -n $NAMESPACE --set global.imageRegistry=$WEBFQDN

Default Configuration

Name Value Possible Values Notes
LOG_LEVEL INFO FATAL, ERROR, WARN, INFO, DEBUG Logging level used by application.
FILTER_FILE_LOCATION core/default/collection/metric/prometheus Text SVN Path where filter file is located.

Configurations can be changed by passing the values to the a1helm install prefixed with the configData parent key.

Example of setting the log level to DEBUG

a1helm install ... --set configData.LOG_LEVEL=DEBUG

Debugging

The prometheus-metrics-processor consists of two components: prometheus-metrics-processor and telegraf. To see logs for each of the components, run the following command (replacing appropriate parameters).

For prometheus-metrics-processor logs:

a1k logs <Pod Name> -n <Namespace> prometheus-metrics-processor

For telegraf logs:

a1k logs <Pod Name> -n <Namespace> telegraf

The <Pod Name> and <Namespace> values are available in the get pods command:

a1k get pods --all-namespaces

Schema

Prometheus metrics processor expects filters in the following schema format:

{
  "Filters": {
    "metrics": {
      "filterType": "keep",
      "values": [
        "some_metric_name"
      ]
    },
    "labels": {
      "filterType": "keep",
      "values": {
        "some_label": "some_label_value"
      }
    },
    "regex": [
      {
        "field": "some_label",
        "pattern": "regex_pattern.*",
        "filterType": "keep"
      }
    ]
  }
}

Filters

metrics

This is applied to the name of the metrics.

Name Required Possible Values Notes
filterType yes Text The action to be applied on the metric when name is matched. Can be either keep or discard
values yes Array This holds the name of the metrics on which the filter is to be applied. Metric name should be specified without prom_ prefix.

Example

{
    "metrics": {
        "filterType": "discard",
        "values": [
          "some_metric_name",
          "some_other_metric_name"
        ]
    }
}

labels

This is applied on the labels present in the metrics.

Name Required Possible Values Notes
filterType yes Text The action to be applied on the metric when label is matched. Can be either keep or discard
values yes Object A key-value binding map associated with metric labels.

Example

{
    "labels": {
        "filterType": "discard",
        "values": {
            "some_label": "some_label_value"
        }
    }
}

regex

This is applied to the name or labels present in the metrics.

Name Required Possible Values Notes
filterType yes Text The action to be applied on the metric when regular expression is matched. Can be either keep or discard
pattern yes Text Regex pattern which will be matched on the metric name or label value.
field yes Text Should be set to:
- Name if regex is to be matched with the name of the metric
- Key of the label when regex is to be matched with the value of the label.

Example

{
    "regex": [
      {
        "field": "Name",
        "pattern": "regex_pattern.*",
        "filterType": "discard"
      },
      {
        "field": "some_label_key",
        "pattern": "regex_pattern.*",
        "filterType": "keep"
      }
    ]
}

Note: