Metric Sink

The Metric Sink microservice takes metrics that have been normalized and inserts them into the Metric database or the Observability Analytics database, depending on your configuration.

This microservice is part of the Metric microservice pipeline. See Understanding Microservice Pipelines in Unified Assurance Concepts for conceptual information.

Autoscaling is enabled by default for this microservice. You can optionally disable autoscaling when you deploy the microservice. See Configuring Autoscaling.

This microservice provides additional Prometheus monitoring metrics. See Metric Sink Self-Monitoring Metrics.

Metric Sink Prerequisites

Before deploying the microservice, confirm that the following prerequisites are met:

  1. A microservice cluster is set up. See Microservice Cluster Setup.

  2. The Apache Pulsar microservice is deployed. See Pulsar.

Deploying Metric Sink

There are two versions of the Metric Sink microservice, one that publishes to the Observability Analytics database, and one that publishes to the Metric database. You use the same Helmchart or Microservices UI card for each, with different configurations.

To deploy the Observability Analytics version of the microservice, run the following commands:

su - assure1
export NAMESPACE=<namespace>
export WEBFQDN=<WebFQDN>
a1helm install <microservice-release-name> assure1/metric-sink -n  $NAMESPACE --set global.imageRegistry=$WEBFQDN --set opensearch.connect=true --set configData.STREAM_INPUT=persistent://assure1/metric/sinkos

In the commands:

To deploy the Metric database version of the microservice, run the following commands:

su - assure1
export NAMESPACE=<namespace>
export WEBFQDN=<WebFQDN> 
a1helm install <microservice-release-name> assure1/metric-sink -n $NAMESPACE --set global.imageRegistry=$WEBFQDN

For <microservice-release-name>, Oracle recommends using the microservice name, without -os prefix to indicate the version (metric-sink), unless you are deploying multiple instances of this version of the microservice to the same cluster.

You can also use the Unified Assurance UI to deploy microservices. See Deploying a Microservice by Using the UI for more information. You use the Metric Sink card and modify the opensearch.connect parameter according to the version of the microservice you are deploying.

Changing Metric Sink Configuration Parameters

When running the install command, you can optionally change default configuration parameter values by including them in the command with additional --set arguments. You can add as many additional --set arguments as you need.

For example:

Default Metric Sink Configuration

The following table describes the default configuration parameters found in the Helm chart under configData for the microservice.

Name Default Value Possible Values Notes
LOG_LEVEL INFO FATAL, ERROR, WARN, INFO, DEBUG The logging level for the microservice.
STREAM_INPUT persistent://assure1/metric/sink Text, 255 characters The path to the Pulsar topic to get data from. The topic at the end of the path may be any text value. Use persistent://assure1/metric/sinkos to publish to Observability Analytics when you have set opensearch.connect to true.
FLUSH_INTERVAL 10 Integer The interval, in seconds, to insert a batch of metrics into the Observability Analytics database.
Used by the Observability Analytics version of the microservice only.
In the legacy Metric database version, this is controlled by the telegraf.config.agent.flush_interval parameter.
BATCH_SIZE 10 Integer The maximum number of messages to include in a batch. Used by the Observability Analytics version of the microservice only.
In the legacy Metric database version, this is controlled by the telegraf.config.agent.metric_batch_size parameter.
RETRY_LIMIT 3 Integer The number of times to retry reaching the database before failing over to the secondary database. Used by both versions of the microservice.

Note:

Although you deploy both versions of the microservice based on the same Helmchart, some parameters in the default chart apply to one version only. For example, only the legacy Metric database version uses Telgraf, so the parameters under telegraf do not apply to the Observability Analytics database version.

About Batch Timing

The microservice sends a batch of metrics to the Observability Analytics database when either FLUSH_INTERVAL or BATCH_SIZE is reached.

If FLUSH_INTERVAL is reached before BATCH_SIZE, the partial batch is sent.

If BATCH_SIZE is reached before FLUSH_INTERVAL, the batch is sent immediately. The timer for FLUSH_INTERVAL continues without resetting, independent of the batch that was just sent.

The same behavior applies to the legacy Metric database version, using the Telegraf batch parameters.

Metric Sink Autoscaling Configuration

Autoscaling is supported for the Metric Sink microservice. See Configuring Autoscaling for general information and details about the standard autoscaling configurations.

The Metric Sink microservice also supports the additional configurations described in the following table.

Name Default Value Possible Values Notes
thresholds.backlogSize 1000 Integer If there are more items in the Metric database backlog than this threshold, autoscaling starts additional processes. Used by the legacy Metric database version of the microservice.
thresholds.totalMetricsProcessed 400 Integer If the average of total metrics processed in five minutes for the Metric database exceeds this threshold, pods will be scaled. Used by the legacy Metric database version of the microservice.
thresholds.pcombacklogsize 1000 Integer If there are more items in the Observability Analytics backlog than this threshold, autoscaling starts additional processes. Used by the Observability Analytics version of the microservice.
thresholds.totalPerformanceMetricsProcessed 400 Integer If the average of total metrics processed in five minutes for the Observability Analytics database exceeds this threshold, pods will be scaled. Used by the Observability Analytics version of the microservice.

Metric Sink Self-Monitoring Metrics

The Metric Sink microservice exposes the self-monitoring metrics described in the following table to Prometheus.

Metric Name Type Description
metric_sink_backlog_size Gauge The number of items pending in the Metric database backlog. Used by the legacy Metric database version of the microservice only.
total_metrics_processed Counter The number of metrics processed for the Metric database. Used by the legacy Metric database version of the microservice.
processing_time_per_metric Gauge The processing time per metric, from receiving it, to sending it, to acknowledging it. Used by the legacy Metric database version of the microservice only.
metric_sink_os_backlog_size Gauge The number of items pending in the Observability Analytics database backlog. Used by the Observability Analytics version of the microservice only.
total_performance_metrics_processed Counter The number of metrics processed for the Observability Analytics database. Used by the Observability Analytics version of the microservice only.

Note:

Metric names in the database include a prefix that indicates the service that inserted them. The prefix is prom_ for metrics inserted by Prometheus. For example, total_metrics_processed is stored as prom_total_metrics_processed in the database.

Sample JSON Metric Schema

The following example shows the expected JSON file schema formats for the legacy Metric database version of Metric Sink.

{
    "@timestamp":  "2021-05-06T18:40:00.000Z",
    "measurement": "iot_test",
    "tags": {
        "host":     "some_host",
        "instance": "some_instance"
    },
    "fields": {
        "availability": 1,
        "value":        123456789000,
        "utilization":  0.1,
        "other":        1
    }
}

Although you are responsible for defining the proper format, the Metric Sink microservice tries to match the current Metric schema. For example, it cleans up slash marks (), replaces dashes (-) and spaces with underscores (_) and capitalizes letters as needed.

Debugging Metric Sink

The Metric Sink microservice consists of two components: metric-sink and telegraf. To see logs for each of the components, run the following commands (replacing appropriate parameters).

For metric-sink logs:

a1k logs <pod_name> -n <namespace> metric-sink

For telegraf logs, when using the legacy Metric database version of the microservice:

a1k logs <pod_name> -n <namespace> telegraf

You can get pod names and namespaces by running the following command:

a1k get pods --all-namespaces