4 Using Oracle Coherence Metrics

Oracle Coherence Metrics enables an alternative means to monitor a running Coherence cluster than JMX MBeans.

This chapter includes the following sections:

• About Coherence Metrics
  As of version 12.2.1.4, Coherence provides a metrics endpoint which extends the metrics capability by allowing scraping of metrics via metrics gathering systems such as Prometheus.
• Enabling Coherence Metrics Endpoint
  The Coherence Metrics endpoint requires COHERENCE_HOME/lib/coherence-metrics.jar library as well as its third-party dependencies.
• Using Metrics System Properties
  The metrics system properties are defined in the metrics-http-config.xml file in the coherence-metrics.jar.
• Configuring Prometheus to Scrape Coherence Metrics Endpoint
  Configuring a scraping job to scrape the Coherence Metrics endpoints for a Coherence cluster is done using the static_configs parameter of a Prometheus configuration file.
• Visualizing Metrics in Grafana
  These dashboards provide detailed insight into your Coherence cluster by using the collected metrics and are a valuable tool in monitoring and managing the health of you cluster.
• Querying for Coherence Metrics
  The metrics endpoint supports Prometheus and JSON formats. It also supports querying for specific metrics by name and tags.
• Adding Custom Metrics
  You can add your own metrics if you annotate the MBeans that you add to the Coherence Registry.

About Coherence Metrics

As of version 12.2.1.4, Coherence provides a metrics endpoint which extends the metrics capability by allowing scraping of metrics via metrics gathering systems such as Prometheus. The Coherence metrics endpoints are enabled on each Coherence Cache server, and a Prometheus scraping job is configured with the location of these endpoints.

If an application requires the Coherence Metrics endpoints to be secure, it is possible to configure the Coherence Metrics endpoint to be secure and to configure the Prometheus scraping job with the necessary security information. See Securing Oracle Coherence Metrics in Securing Oracle Coherence.

The Coherence Metrics endpoint supports content-encoding of gzip. By default, the metrics endpoint returns metrics in text, or Prometheus format, but can also return metrics in JSON format if the Accepts: application/json HTTP header is set.

To secure Oracle Coherence Metrics endpoint, see Securing Oracle Coherence Metrics in Securing Oracle Coherence.

Enabling Coherence Metrics Endpoint

The Coherence Metrics endpoint requires COHERENCE_HOME/lib/coherence-metrics.jar library as well as its third-party dependencies.

To enable the Coherence Metrics endpoint for a Coherence Cache server, add the Coherence Metrics module and its dependencies to the classpath, and set the system property coherence.metrics.http.enabled to true when starting the Coherence cache server.

Note:

The Coherence distribution does not include the third-party dependencies.

It is a best practice to manage dependencies using Maven. It is assumed that the latest Oracle Coherence artifacts are installed in your local Maven repository using the Oracle Maven Synchronization Plug-in as described in Populating the Maven Repository Manager. To generate a classpath containing third party libraries, run the following Maven command with the provided pom.xml below. Add the generated classpath to start server script.

mvn dependency:build-classpath

Note:

• If running with JDK 11, add -P jdk11 to the mvn command line to get additional libraries that are no longer part of JDK 11.
• When copying this pom.xml for your use, update the coherence <version> after <groupId>, to match the coherence patch version that you are using.

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <artifactId>metrics</artifactId>
  <groupId>metrics-deps</groupId>
  <version>14.1.1-0-0</version>
  <name>Coherence Metrics dependencies</name>
  <packaging>pom</packaging>
  <dependencies>
    <dependency>
      <groupId>com.oracle.coherence</groupId>
      <artifactId>coherence-metrics</artifactId>
      <version>${project.version}</version>
    </dependency>
  </dependencies>
  <profiles>
    <profile>
      <id>jdk11</id>
      <properties>
        <com.sun.xml.bind.version>2.3.0</com.sun.xml.bind.version>
        <javax.activation.version>1.1.1</javax.activation.version>
      </properties>
      <dependencies>
	<dependency>
          <groupId>com.sun.xml.bind</groupId>
          <artifactId>jaxb-core</artifactId>
          <version>${com.sun.xml.bind.version}</version>
        </dependency>
        <dependency>
          <groupId>com.sun.xml.bind</groupId>
          <artifactId>jaxb-impl</artifactId>   
          <version>${com.sun.xml.bind.version}</version>
        </dependency>
        <dependency>
          <groupId>javax.activation</groupId>
          <artifactId>activation</artifactId>
          <version>${javax.activation.version}</version>
        </dependency>
        <dependency>
          <groupId>javax.xml.bind</groupId>
          <artifactId>jaxb-api</artifactId>
          <version>${com.sun.xml.bind.version}</version>
        </dependency>
      </dependencies>
    </profile>
  </profiles>
</project>

Using Metrics System Properties

The metrics system properties are defined in the metrics-http-config.xml file in the coherence-metrics.jar.

This section contains the following topics:

• Coherence Metrics System Properties
  The following Coherence Metrics system properties are defined in the default metrics-http-config.xml file in the coherence-metrics.jar.
• Other Metrics System Properties
  The other metrics system property includes:

Coherence Metrics System Properties

The following Coherence Metrics system properties are defined in the default metrics-http-config.xml file in the coherence-metrics.jar.

Table 4-1 Coherence Metrics system properties

System Property	Description	Default
coherence.metrics.http.enabled	Determines if metrics endpoint enabled	false
coherence.metrics.http.address	Specifies the address to listen on	0.0.0.0 (All interfaces)
coherence.metrics.http.port	Specifies the port to listen on (0 indicates the ephemeral port)	9612
coherence.metrics.http.auth	Configures the authentication mechanism for the HTTP server basic, auth, basic+auth or not set	not set
coherence.metrics.http.provider	Defines the id of a <socket-provider> in coherence operational override file	none

You can override the default metrics-http-config.xml by placing an override file with same file name in classpath before the coherence-metrics.jar.

Note:

If multiple Coherence cache servers with metrics endpoint enabled are started on same machine, the coherence.metrics.http.port must be unique for each server. If not, the subsequent ones with same coherence.metrics.http.port value will get an address already in use and will result in an error with starting up the MetricsHttpProxy service.

The Coherence cache server will start up but the Coherence Metrics endpoint will only be available on one Coherence cache server.

Other Metrics System Properties

The other metrics system property includes:

Table 4-2 Other metrics system properties

System Property	Description	Default
coherence.metrics.extended	Includes extended information (type and/or description) when publishing metrics.	false

The extended information for Prometheus metrics is #Help, which contains the metrics name. If more data follows, the #Help is considered the docstring for that metric name. For Coherence Metrics, the docstring is typically the JMX MBean attribute description from which the metric value was derived. There is one or no #Help per metrics value in a Prometheus response.

Configuring Prometheus to Scrape Coherence Metrics Endpoint

Configuring a scraping job to scrape the Coherence Metrics endpoints for a Coherence cluster is done using the static_configs parameter of a Prometheus configuration file. You need to specify the IP address and coherence.metrics.http.port for each Coherence Metrics endpoint in the static_configs of a Prometheus configuration file. Once configured and started, the Prometheus server will scrape the configured static targets at the configured interval, collecting metrics data for the Coherence cluster.

See Prometheus configuration documentation.

Visualizing Metrics in Grafana

After you have configured Prometheus to scrape metrics, you can visualize these metrics using the pre-built Grafana dashboards available in the Coherence Operator. See coherence-operator.

These dashboards provide detailed insight into your Coherence cluster by using the collected metrics and are a valuable tool in monitoring and managing the health of you cluster.

For more information about the dashboards and how to load them, see the Coherence Operator documentation.

For an example of the Coherence Summary Dashboard, see Use dashboards.

Note:

Oracle recommends you to use the minimum Grafana version 8.5.13 to view the dashboards. At this point, due to the many issues with Grafana 9.x, do not use this version yet.

Querying for Coherence Metrics

The metrics endpoint supports Prometheus and JSON formats. It also supports querying for specific metrics by name and tags.

This section contains the following topics:

• Basic Querying
  The coherence.metrics.http.port is assumed to be set to 9612 for the below listed basic querying examples.
• Querying for Specific Metrics
  To query for specific metrics append the metric name to the URL.

Basic Querying

The coherence.metrics.http.port is assumed to be set to 9612 for the below listed basic querying examples.

http://localhost:9612/metrics

By default, this will produce Prometheus' simple text-based exposition format. If the caller uses the Accepts: application/json header, then JSON is returned. The endpoint also supports content negotiation by appending the required type to the url.

http://localhost:9612/metrics/.txt returns Prometheus data whereas http://localhost:9612/metrics/.json returns JSON data.

The coherence.metrics.extended system property controls the inclusion of additional help or descriptions. By default, the coherence.metrics.extended is set to false and additional information is not included. By adding the extended=true query parameter, additional data can be included.

http://localhost:9612/metrics?extended=true (this applies to both Prometheus and JSON formats).

Querying for Specific Metrics

To query for specific metrics append the metric name to the URL.

To query for number of members in a cluster:

http://localhost:9612/metrics/Coherence.Cluster.Size

To query for all cache size metrics:

http://localhost:9612/metrics/Coherence.Cache.Size

To query for a metric with matching tags add the tag key and value as query parameters. To query for the size metric for a specific cache where the cache name is cache_name.

http://localhost:9612/metrics/Coherence.Cache.Size?name=cache_name

Additionally, if the cache (for example near cache) is included so that there are front and back tiers, then the request could be further restricted.

http://localhost:9612/metrics/Coherence.Cache.Size?name=cache_name&tier=back

http://localhost:9612/metrics/Coherence.Cache.Size?name=cache_name&tier=front

Adding Custom Metrics

You can add your own metrics if you annotate the MBeans that you add to the Coherence Registry.

For example:

import com.tangosol.net.management.annotation.MetricsValue;
import com.tangosol.net.management.annotation.MetricsTag;   

public interface CustomMBean
    {
    @MetricsValue("custom_value")
    long getValueOne();

    @MetricsValue
    long getValueTwo();

    long getValueThree();

    @MetricsTag("custom_tag")
    String getTagValueOne();

    @MetricsTag
    String getTagValueTwo();
    }

• The method getValueOne is annotated with @MetricValue. Hence, it will be a metric and the name of the metric will be custom_value.

• The method getValueTwo is annotated with @MetricValue. Hence, it will be a metric. The annotation does not specify a name and the name will be generated from the attribute name (in this case value_two).

• The method getValueThree is not annotated. Hence, though being an MBean attribute it will not be used as a metric or as a tag.

• The method getTagValueOne is annotated with @MetricTag. Hence, it will be a metric tag and the name of the tag will be custom_tag.

• The method getTagValueTwo is annotated with @MetricTag. Hence, it will be a metric tag. The annotation does not specify a name and the name will be generated from the attribute name (in this case tag_value_two).

For annotations to be detected by Coherence when registering the MBean, the annotations must be wrapped in an AnnotatedStandardMBean.

For example:

Registry registry   = CacheFactory.ensureCluster().getManagement();
Custom   custom     = new Custom();
String   sMBeanName = registry.ensureGlobalName(“type=Custom,name=bar”);

registry.register(sMBeanName, new AnnotatedStandardMBean(custom, CustomMBean.class));

The code above will register the Mbean with the Object name similar to Coherence:type=Custom,name=bar,nodeId=1

The metrics name produced will be prefixed with the value from the type key in the ObjectName, in this case Custom. This will produce the following metrics:

• Custom_custom_value from method getValueOne
• Custom_value_two from method getValueTwo

Both metrics will have common tags such as cluster name, nodeId, machine, role, site, and so on, as well as the custom tags name=bar from the ObjectName, custom_tag from the annotated method getTagValueOne and tag_value_two from the annotated method getTagValueTwo. The method getTagValueTwo is annotated with @MetricTag. Hence it will be a metric tag. The annotation does not specify a name and the name will be generated from the attribute name (in this case tag_value_two).