Expose TimesTen Metrics Automatically

Let's walk through an example showing how the TimesTen Operator automatically exports and exposes TimesTen metrics. In this example, let's create a TimesTenClassic object and observe how the TimesTen Operator automatically creates the objects needed to automatically export and expose TimesTen metrics to Prometheus.

Let's assume you have installed the Prometheus Operator in your Kubernetes cluster and there is a Prometheus server running in your namespace. Let's also assume you have started the TimesTen Operator following the steps in About Deploying the TimesTen Operator and there are no TimesTenClassic or TimesTenScaleout objects deployed in your namespace.

Create a TimesTenClassic object.

vi samplepublish.yaml

apiVersion: timesten.oracle.com/v1
kind: TimesTenClassic
metadata:
  name: samplepublish
spec:
  ttspec:
    storageClassName: oci
    storageSize: 250G
    image: container-registry.oracle.com/timesten/timesten:22.1.1.19.0
    imagePullSecret: sekret
    prometheus:
      port: 6666

In this example, the port on which the TimesTen exporter listens is 6666.

Deploy the TimesTenClassic object.

kubectl create -f samplepublish.yaml

The output is the following:

timestenclassic.timesten.oracle.com/samplepublish created

Wait a few minutes, then confirm the TimesTenClassic object is in the Normal state.

kubectl get ttc samplepublish

The output is similar to the following:

NAME            STATE    ACTIVE            AGE
samplepublish   Normal   samplepublish-0   3m56s

Confirm the TimesTen Operator provisioned an exporter container in each TimesTen Pod

kubectl get pods

The output is similar to the following:

NAME                                  READY   STATUS    RESTARTS   AGE
...
samplepublish-0                       3/3     Running   0          7m33s
samplepublish-1                       3/3     Running   0          7m33s
...

The TimesTen Operator automatically provisioned 3 containers in each TimesTen Pod. One of these containers is the exporter container, which is running the TimesTen exporter.

Confirm the TimesTen Operator automatically created the appropriate Kubernetes Secrets.
```
kubectl get secrets
```
The output is similar to the following:
```
NAME                                             TYPE                             DATA   AGE
...
samplepublish-metrics                            Opaque                           1      15m
samplepublish-metrics-client                     Opaque                           3      15m
```
By default, TimesTen metrics are served using https. As a result, the TimesTen Operator automatically created an Oracle Wallet, the certificates, and the Kubernetes Secrets needed for https. For more information, see About Transport Layer Security (mutual TLS) Certificates for TimesTen Metrics.
Confirm the appropriate files are in the samplepublish-metrics-client Kubernetes Secret.
```
kubectl describe secret samplepublish-metrics-client
```
The output is similar to the following:
```
Name:         samplepublish-metrics-client
Namespace:    mynamespace
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
ca.crt:      1461 bytes
client.crt:  1273 bytes
client.key:  1675 bytes
```
The TimesTen Operator automatically created the ca.crt, client.crt, and client.key files. The samplepublish-metrics-client Kubernetes Secret holds these files. See About Transport Layer Security (mutual TLS) Certificates for TimesTen Metrics.

Confirm the TimesTen Operator automatically created the samplepublish PodMonitor object.

kubectl describe podmonitor samplepublish

The output is similar to the following:

Name:         samplepublish
Namespace:    mynamespace
Labels:       app=samplepublish
              database.timesten.oracle.com=samplepublish
Annotations:  <none>
API Version:  monitoring.coreos.com/v1
Kind:         PodMonitor
Metadata:
  Creation Timestamp:  2023-09-25T15:36:08Z
  Generation:          1
  Owner References:
    API Version:           timesten.oracle.com/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  TimesTenClassic
    Name:                  samplepublish
    UID:                   d0f46bf7-b1d8-4499-876c-51410a469772
  Resource Version:        284346942
  UID:                     5a61ec9e-df7d-4a98-be47-dce1e7c3d217
Spec:
  Namespace Selector:
  Pod Metrics Endpoints:
    Bearer Token Secret:
      Key:
    Interval:  15s
    Path:      /metrics
    Port:      exporter
    Scheme:    https
    Tls Config:
      Ca:
        Secret:
          Key:   ca.crt
          Name:  samplepublish-metrics-client
      Cert:
        Secret:
          Key:   client.crt
          Name:  samplepublish-metrics-client
      Key Secret:
        Key:        client.key
        Name:       samplepublish-metrics-client
      Server Name:  samplepublish.samplepublish.mynamespace.svc.cluster.local
  Selector:
    Match Labels:
      database.timesten.oracle.com:  samplepublish
Events:                              <none>

Let's look at the important information in this PodMonitor object:

There is an app=samplepublish label. If there are Pods with a label that matches app=samplepublish, Prometheus scrapes metrics from them. The TimesTen Pods contain the app=samplepublish label. Prometheus will therefore scrape metrics from these Pods. We will see this later.
Prometheus scrapes metrics from the /metrics endpoint.
Metrics are exposed using https.
The TimesTen Operator placed the samplepublish-metrics and samplepublish-metrics-client Kubernetes Secrets in the PodMonitor object. These Secrets and their contents are used by the Prometheus Operator.

The Prometheus Operator edits the Prometheus server configuration files based on the information in this PodMonitor object.

Confirm the TimesTen Operator automatically created the appropriate Kubernetes Service.

kubectl describe service samplepublish

The output is similar to the following:

Name:              samplepublish
Namespace:         mynamespace
Labels:            app=samplepublish
Annotations:       <none>
Selector:          app=samplepublish
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              cs  6625/TCP
TargetPort:        6625/TCP
Endpoints:         10.244.0.120:6625,10.244.1.144:6625
Port:              exporter  6666/TCP
TargetPort:        6666/TCP
Endpoints:         10.244.0.120:6666,10.244.1.144:6666
Session Affinity:  None
Events:            <none>

The TimesTen exporter listens on port 6666.

Confirm there is a Prometheus server running in your namespace.
```
kubectl get pods
```
The output is similar to the following:
```
NAME                                  READY   STATUS    RESTARTS   AGE
prometheus-sampleprometheusserver-0   2/2     Running   2          18d
...
```
The prometheus-sampleprometheusserver-0 Prometheus server is running in your namespace.

Confirm the Prometheus Operator edited the Prometheus server configuration file based on the information in the samplepublish PodMonitor object.

 kubectl exec prometheus-sampleprometheusserver-0 -c prometheus -- cat /etc/prometheus/config_out/prometheus.env.yaml

The output is similar to the following:

global:
  evaluation_interval: 30s
  scrape_interval: 30s
  external_labels:
    prometheus: mynamespace/sampleprometheusserver
    prometheus_replica: prometheus-sampleprometheusserver-0
scrape_configs:
- job_name: podMonitor/mynamespace/samplepublish/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
      - mynamespace
  scrape_interval: 15s
  metrics_path: /metrics
  scheme: https
  tls_config:
    insecure_skip_verify: false
    ca_file: /etc/prometheus/certs/secret_mynamespace_samplepublish-metrics-client_ca.crt
    cert_file: /etc/prometheus/certs/secret_mynamespace_samplepublish-metrics-client_client.crt
    key_file: /etc/prometheus/certs/secret_mynamespace_samplepublish-metrics-client_client.key
    server_name: samplepublish.samplepublish.mynamespace.svc.cluster.local
  relabel_configs:
  - source_labels:
    - job
    target_label: __tmp_prometheus_job_name
  - action: drop
    source_labels:
    - __meta_kubernetes_pod_phase
    regex: (Failed|Succeeded)
  - action: keep
    source_labels:
    - __meta_kubernetes_pod_label_database_timesten_oracle_com
    - __meta_kubernetes_pod_labelpresent_database_timesten_oracle_com
    regex: (samplepublish);true
  - action: keep
    source_labels:
    - __meta_kubernetes_pod_container_port_name
    regex: exporter
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_pod_container_name
    target_label: container
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - target_label: job
    replacement: mynamespace/samplepublish
  - target_label: endpoint
    replacement: exporter
  - source_labels:
    - __address__
    target_label: __tmp_hash
    modulus: 1
    action: hashmod
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
  metric_relabel_configs: []

Prometheus has the information it needs to scrape TimesTen metrics.

Review some of the TimesTen metrics: In your browser, go to your Prometheus server.

In the Prometheus server search bar, type a TimesTen metric. For example, timesten_databases. Next, click Execute.

The output is similar to the following:

timesten_databases{container="exporter", endpoint="exporter", 
instance="10.244.0.120:6666", instancename="instance1", job="mynamespace/samplepublish", 
namespace="mynamespace", pod="samplepublish-0"} 1

timesten_databases{container="exporter", endpoint="exporter", 
instance="10.244.1.144:6666", instancename="instance1", job="mynamespace/samplepublish", 
namespace="mynamespace", pod="samplepublish-1"} 1

There is one TimesTen database in the samplepublish-0 Pod and one TimesTen database in the samplepublish-1 Pod as evidenced by the value of 1 for the metric.

In the Prometheus server search bar, type a second TimesTen metric. For example, timesten_database_loaded. Next, click Execute.

The output is similar to the following:

timesten_database_loaded{container="exporter", dsn="samplepublish", 
endpoint="exporter", instance="10.244.0.120:6666", instancename="instance1", 
job="mynamespace/samplepublish", namespace="mynamespace", 
pod="samplepublish-0"} 1

timesten_database_loaded{container="exporter", dsn="samplepublish", 
endpoint="exporter", instance="10.244.1.144:6666", instancename="instance1", 
job="mynamespace/samplepublish", namespace="mynamespace", 
pod="samplepublish-1"} 1

The samplepublish database in the samplepublish-0 Pod and the samplepublish database in the samplepublish-1 Pod are both loaded into memory as evidenced by the value of 1 for the metric.

Congratulations! You successfully created TimesTen databases whose metrics are collected by Prometheus.