Demonstrate How to Expose TimesTen Kubernetes Operator Metrics

Let's walk through an example illustrating how the TimesTen Kubernetes Operator exposes metrics about its own functionality as well as the status of TimesTenClassic or TimesTenScaleout objects. Let's assume Prometheus is installed in your Kubernetes cluster and you have a Prometheus server running in your namespace.

Let's make the following decisions:

https or http?: Let's choose https. The METRICS_SCHEME environment variable determines if metrics should be made available by https or http. A setting of "1" (default) indicates https.
Create a ServiceMonitor object?: Let's have the TimesTen Operator create a ServiceMonitor object. This object contains the information needed by Prometheus to configure the TimesTen Operator as a scrape target. The CREATE_SERVICEMONITOR environment variable determines if the TimesTen Operator should create a ServiceMonitor object. A setting of "1" (default) indicates the TimesTen Operator should create the object.
Expose TimesTen Operator metrics outside of the TimesTen Operator Pods?: Let's have the metrics exposed outside of the TimesTen Operator Pods so that Prometheus can scrape metrics from it. The EXPOSE_METRICS environment variable determines if metrics are exposed outside of the TimesTen Operator Pods. A setting of "1" (default) indicates metrics should be exposed. This means that the TimesTen Operator will create a Kubernetes Service (called timesten-operator) that allows the /metrics endpoint to be available to other Pods in the Kubernetes cluster.

Since we have chosen the defaults for the TimesTen Operator environment variables, you do not have to modify these variables in the operator.yaml YAML manifest file. For more information about the operator.yaml YAML manifest file, see Customize the TimesTen Operator.

Start the TimesTen Operator following the steps in About Deploying the TimesTen Operator.

(Optional) Review the TimesTen Operator deployment running in your namespace.

kubectl describe deployment timesten-operator

Output.

Name:                   timesten-operator
Namespace:              mynamespace
CreationTimestamp:      Sat, 23 Sep 2023 20:48:48 +0000
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               name=timesten-operator
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           name=timesten-operator
  Service Account:  timesten-operator
  Containers:
   timesten-operator:
    Image:       container-registry.oracle.com/timesten/timesten:22.1.1.19.0
    Ports:       8081/TCP, 8080/TCP
    Host Ports:  0/TCP, 0/TCP
    Command:
      /timesten/operator/operator/timesten-operator
    Liveness:   http-get http://:probes/healthz delay=10s timeout=10s period=30s #success=1 #failure=3
    Readiness:  http-get http://:probes/healthz delay=10s timeout=10s period=10s #success=1 #failure=1
    Environment:
      WATCH_NAMESPACE:         (v1:metadata.namespace)
      POD_NAME:                (v1:metadata.name)
      OPERATOR_NAME:          timesten-operator
      EXPOSE_METRICS:         1
      METRICS_SCHEME:         https
      EXPOSE_PROBES:          1
      CREATE_SERVICEMONITOR:  1
    Mounts:                   <none>
  Volumes:                    <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   timesten-operator-7f77c749fd (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  73s   deployment-controller  Scaled up replica set timesten-operator-7f77c749fd to 1

The EXPOSE_METRICS and CREATE_SERVICEMONITOR TimesTen Operator environment variables are set to 1 (defaults) and the METRICS_SCHEME is set to https (default).

Confirm the TimesTen Operator created the appropriate Kubernetes Secrets.
```
 kubectl get secrets
```
Output:
```
NAME                                             TYPE                             DATA   AGE
...
timesten-operator-metrics                        Opaque                           1      11m
timesten-operator-metrics-client                 Opaque                           3      11m
```
The TimesTen Operator created the timesten-operator-metrics Kubernetes Secret. This Secret contains the certificates needed by the TimesTen Operator for https/TLS. This Secret is used internally by the TimesTen Operator.

The TimesTen Operator also created the timesten-operator-metrics-client Kubernetes Secret. This Secret contains the files that a Prometheus server needs to scrape metrics from the TimesTen Operator Pods.
Confirm the appropriate files are in the timesten-operator-metrics-client Kubernetes Secret.
```
kubectl describe secret timesten-operator-metrics-client
```
Output:
```
Name:         timesten-operator-metrics-client
Namespace:    mynamespace
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
ca.crt:      1465 bytes
client.crt:  1277 bytes
client.key:  1675 bytes
```
The Secret contains the following files:
- ca.crt: The Certificate Authority file needed by the client to authenticate the self-signed certificate used by the TimesTen Operator.
- client.crt: The client certificate that the TimesTen Operator uses to authenticate clients that are attempting to scrape metrics from it.
- client.key: The private key that is associated with the client.crt file.

Confirm the TimesTen Operator created the timesten-operator ServiceMonitor object.

kubectl describe servicemonitor timesten-operator

Output:

Name:         timesten-operator
Namespace:    mynamespace
Labels:       app=timesten-operator
              database.timesten.oracle.com=timesten-operator
Annotations:  deployment.kubernetes.io/revision: 1
API Version:  monitoring.coreos.com/v1
Kind:         ServiceMonitor
Metadata:
  Creation Timestamp:  2023-09-23T20:49:33Z
  Generation:          1
  Owner References:
    API Version:           apps/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Deployment
    Name:                  timesten-operator
    UID:                   3084af2d-90ca-49d1-874a-cde1174f50b5
  Resource Version:        282625424
  UID:                     6337b813-9d62-43b0-8594-60e387e7a30d
Spec:
  Endpoints:
    Bearer Token Secret:
      Key:
    Interval:  15s
    Path:      /metrics
    Port:      metrics
    Scheme:    https
    Tls Config:
      Ca:
        Secret:
          Key:   ca.crt
          Name:  timesten-operator-metrics-client
      Cert:
        Secret:
          Key:   client.crt
          Name:  timesten-operator-metrics-client
      Key Secret:
        Key:        client.key
        Name:       timesten-operator-metrics-client
      Server Name:  timesten-operator.mynamespace.svc.cluster.local
  Namespace Selector:
  Selector:
    Match Labels:
      App:  timesten-operator
Events:     <none>

Let's look at the important information in this ServiceMonitor object:

There is an app=timesten-operator label. If there are Pods with a label that matches app=timesten-operator, Prometheus scrapes metrics from them. The TimesTen Operator Pod contains the app=timesten-operator label. Prometheus will therefore scrape metrics from it. We will see this later.
Prometheus scrapes metrics from the /metrics endpoint.
Metrics are exposed using https.
The TimesTen Operator placed the timesten-operator-metrics and timesten-operator-metrics-client Kubernetes Secrets in the ServiceMonitor object. These Secrets are used by the Prometheus Operator.

The Prometheus Operator edits the Prometheus server configuration files based on the information in this ServiceMonitor object.

Confirm the TimesTen Operator created the appropriate Kubernetes Service.

kubectl describe service timesten-operator

Output:

Name:              timesten-operator
Namespace:         mynamespace
Labels:            app=timesten-operator
                   database.timesten.oracle.com=timesten-operator
Annotations:       deployment.kubernetes.io/revision: 1
Selector:          name=timesten-operator
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.169.59
IPs:               10.96.169.59
Port:              metrics  8080/TCP
TargetPort:        8080/TCP
Endpoints:         10.244.8.180:8080
Port:              probe  8081/TCP
TargetPort:        8081/TCP
Endpoints:         10.244.8.180:8081
Session Affinity:  None
Events:            <none>

Confirm there is a Prometheus server running in your namespace.

kubectl get pods

Output.

NAME                                  READY   STATUS    RESTARTS   AGE
prometheus-sampleprometheusserver-0   2/2     Running   0          10d
...

The prometheus-sampleprometheusserver-0 Prometheus server is running in your namespace.

Confirm the Prometheus Operator edited the Prometheus server configuration file based on the information in the timesten-operator ServiceMonitor object.

 kubectl exec prometheus-sampleprometheusserver-0 -c prometheus -- cat /etc/prometheus/config_out/prometheus.env.yaml

Output.

global:
  evaluation_interval: 30s
  scrape_interval: 30s
  external_labels:
    prometheus: mynamespace/sampleprometheusserver
    prometheus_replica: prometheus-sampleprometheusserver-0
scrape_configs:
- job_name: serviceMonitor/mynamespace/timesten-operator/0
  honor_labels: false
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - mynamespace
  scrape_interval: 15s
  metrics_path: /metrics
  scheme: https
  tls_config:
    insecure_skip_verify: false
    ca_file: /etc/prometheus/certs/secret_mynamespace_timesten-operator-metrics-client_ca.crt
    cert_file: /etc/prometheus/certs/secret_mynamespace_timesten-operator-metrics-client_client.crt
    key_file: /etc/prometheus/certs/secret_mynamespace_timesten-operator-metrics-client_client.key
    server_name: timesten-operator.mynamespace.svc.cluster.local
  relabel_configs:
  - source_labels:
    - job
    target_label: __tmp_prometheus_job_name
  - action: keep
    source_labels:
    - __meta_kubernetes_service_label_app
    - __meta_kubernetes_service_labelpresent_app
    regex: (timesten-operator);true
  - action: keep
    source_labels:
    - __meta_kubernetes_endpoint_port_name
    regex: metrics
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Node;(.*)
    replacement: ${1}
    target_label: node
  - source_labels:
    - __meta_kubernetes_endpoint_address_target_kind
    - __meta_kubernetes_endpoint_address_target_name
    separator: ;
    regex: Pod;(.*)
    replacement: ${1}
    target_label: pod
  - source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: service
  - source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod
  - source_labels:
    - __meta_kubernetes_pod_container_name
    target_label: container
  - action: drop
    source_labels:
    - __meta_kubernetes_pod_phase
    regex: (Failed|Succeeded)
  - source_labels:
    - __meta_kubernetes_service_name
    target_label: job
    replacement: ${1}
  - target_label: endpoint
    replacement: metrics
  - source_labels:
    - __address__
    target_label: __tmp_hash
    modulus: 1
    action: hashmod
  - source_labels:
    - __tmp_hash
    regex: 0
    action: keep
  metric_relabel_configs: []

Prometheus has the information it needs to scrape TimesTen Operator metrics.

Review then deploy a TimesTenClassic object.

Review.

cat sample.yaml

Output.

apiVersion: timesten.oracle.com/v1
kind: TimesTenClassic
metadata:
  name: sample
spec:
  ttspec:
    storageClassName: oci
    storageSize: 250G
    image: container-registry.oracle.com/timesten/timesten:22.1.1.19.0
    imagePullSecret: sekret

Deploy:

kubectl create -f sample.yaml

Output:

timestenclassic.timesten.oracle.com/sample created

Wait a few minutes, then confirm the sample TimesTenClassic object is in the Normal state.
```
kubectl get ttc sample
```
Output:
```
NAME     STATE    ACTIVE     AGE
sample   Normal   sample-0   2m37s
```

Review some of the TimesTen Operator metrics. In your browser, go to your Prometheus server.

In the Prometheus server search bar, type a TimesTen Operator metric. For example, timesten_classic_state_normal. Next, click Execute.

Output.

timesten_classic_state_normal{container="timesten-operator", 
endpoint="metrics", instance="10.244.8.180:8080", job="timesten-operator", 
name="sample", namespace="mynamespace", pod="timesten-operator-7f77c749fd-2lt5x", 
service="timesten-operator"} 1

There is one TimesTenClassic object (sample) and it is in the Normal state.

In the Prometheus server search bar, type a second TimesTen Operator metric. For example, timesten_classic_state_not_normal. Next, click Execute.
Output.
```
timesten_classic_state_not_normal{container="timesten-operator", 
endpoint="metrics", instance="10.244.8.180:8080", job="timesten-operator", 
name="sample", namespace="mynamespace", pod="timesten-operator-7f77c749fd-2lt5x", 
service="timesten-operator"} 0
```
There is one TimesTenClassic object (sample) and it is in either Normal or Initializing state. A value of 0 indicates it is not in any other state.

In the Prometheus server search bar, type a third TimesTen Operator metric. For example, timesten_classic_state. Next, click Execute.

Output.

timesten_classic_state{container="timesten-operator", endpoint="metrics", 
instance="10.244.8.180:8080", job="timesten-operator", name="sample", 
namespace="mynamespace", pod="timesten-operator-7f77c749fd-2lt5x", 
service="timesten-operator", state="Initializing"} 0

timesten_classic_state{container="timesten-operator", endpoint="metrics", 
instance="10.244.8.180:8080", job="timesten-operator", name="sample", 
namespace="mynamespace", pod="timesten-operator-7f77c749fd-2lt5x", 
service="timesten-operator", state="Normal"} 1

The sample TimesTenClassic object is no longer in the Initializing state. It is now in the Normal state.

Congratulations! You successfully walked through an example demonstrating how TimesTen Operator metrics are exposed, scraped, and published.