O&M Solution: Using OpenTelemetry to monitor Kubernetes applications in OCI

Introduction

In today's cloud-native landscape, Observability is crucial for ensuring the reliability, performance, and scalability of applications built with Kubernetes and microservices. Observability provides a comprehensive view of an application's behavior, allowing developers to identify performance bottlenecks, troubleshoot issues, and optimize their applications.

Oracle offers several solutions for automatic instrumentation to support Observability. The Oracle Application Performance Monitoring (APM) Java Agent is one such solution that provides deep visibility into Java-based applications. Additionally, the OpenTelemetry Operator is another key offering that simplifies the process of instrumenting applications with OpenTelemetry.

This solution playbook presents a fully OpenTelemetry (OTel) based approach of sending telemetry data from Kubernetes applications to OCI Observability and Management (O&M) APM and Monitoring Services. This approach standardize the use of OTel components such as the Agent, Collector, and Operator, providing flexibility and choice in selecting a solution that best fits specific needs. Furthermore the playbook solution highlights the process of sending logs to Logging Analytics.

Choice is important when designing an Observability solution

It is well understood that a comprehensive observability solution is essential, though the specific approach may vary based on requirements. Some organizations may opt for a single-vendor solution, while others may prefer a vendor-neutral strategy requiring integration with other systems to ensure a consistent monitoring approach. Oracle provides several solutions that can be combined to meet these requirements. This playbook solution examines the OpenTelemetry approach of sending telemetry data to OCI services.

Instrument your applications using APM Agents

The OCI Application Performance Monitoring (APM) service offers multiple agents to instrument applications. These can be configured to send telemetry data directly to APM without the need of an OTel collector, if needed:

APM Java Agent: For automatic instrumentation of Java applications
APM .NET Agent: For .NET applications running on Windows
APM Browser Agent: For monitoring performance in user browsers
APM Java Tracer: Provides more control over tracing of java applications

Oracle's observability solutions include a range of agents, support for OpenTelemetry, and integration with open-source tracing tools such as Jaeger and Zipkin (as detailed here). Additionally, metrics can be sent to the OCI Monitoring service, while logs can be forwarded to Logging Analytics Services from both Kubernetes and applications.

This combination enables a comprehensive view of application performance, health, and troubleshooting, ensuring a unified monitoring approach across diverse environments. By leveraging these capabilities, organisations can gain deeper insights, enhance reliability, and optimise their observability strategy.

OpenTelemetry approach to collecting and exporting telemetry data

OpenTelemetry offers a set of tools that work together to collect and export telemetry data. The key components in the ecosystem include:

OpenTelemetry Agent: Agents reside within Kubernetes pods or containers, automatically capturing telemetry data from applications. Multiple language-specific agents are available, along with an auto-instrumentation option, with the choice is depending on the applications environment. This playbook solution considers the auto-instrumentation.
OpenTelemetry Collector: The Collector is optional when sending data to the backend, however it offers great advantages dependent on the setup. It can function as a central hub for receiving telemetry data from agents, filtering out or enriching telemetry data to multiple backends, enabling flexible data routing and processing based specific requirements.
OpenTelemetry Operator: The Operator simplifies the management and deployment of OpenTelemetry components in Kubernetes. It automates tasks such as installation, scaling, and upgrades.

Description — OTel toolset sending telemetry data to oTel2OCI-BW.png

Installing the OpenTelemetry toolset in the Kubernetes environment

Integrating OpenTelemetry into your Kubernetes environment involves several steps. There are multiple methods for installing OpenTelemetry Operators, Collectors and Agents. The following is just one example. For additional options and detailed information, please refer to this document.

Step1: Install the OpenTelemetry operator

The Operator automates the lifecycle management for instrumentation. In most cases a prerequisite of the cert-manager is required.

Install Cert-Manger
Run the following command to install cert-manager:

kubectl apply -f https://github.com/cert-manager/certmanager/releases/latest/download/cert-manager.yaml

Install the OpenTelemetry Operator
Once the cert-manager is installed, deploy the OTel Operator manifest:

kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

Verify Operator Status
Ensure the Operator pod is running before proceeding:

kubectl get pods -n opentelemetry-operator-system

Step 2: Configure the OpenTelemetry Collector

The Collector acts as the intermediary between applications and O&M Services. Below an example of an OpenTelemetry-collector.yaml file which creates a OpenTelemtryCollector Custom Resource (CR) that defines the configuration for this collector.

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector  # The type of resource we are creating
metadata:
  name: OpenTelemetry-collector  # Name of the collector (can be anything descriptive)
  namespace: my-app  # Namespace where the app is installed (adjust as needed)
spec:
  mode: deployment  # Deployment mode for the collector
  serviceAccount: default
  config: >
    receivers:  # Defines where to receive telemetry data
      otlp:  # Receiver for OpenTelemetry Protocol messages
        protocols:
          grpc:  # Enable gRPC protocol
          http:  # Enable HTTP protocol
    processors:  # Modify or enrich telemetry data before sending it to exporters
      batch:  # Batch processor for efficiency
    exporters:
      otlphttp/ociapm:  # Exporter for sending traces to OCI APM
        endpoint: "https://123456789abcdefgh.apm-agt.us-ashburn-1.oci.oraclecloud.com/20200101/opentelemetry/"  # Endpoint URL
          headers:
            Authorization: "dataKey ABCDEFGH123456789"  # APM private key
      otlphttp/ocimetrics:  # Exporter for sending metrics to OCI Metrics
        endpoint: "https://123456789abcdefgh.apm-agt.us-ashburn-1.oci.oraclecloud.com/20200101/observations/metric?dataFormat=otlp-metric&dataFormatVersion=1"  # Endpoint URL
          headers:  # Authorization header
            Authorization: "dataKey ABCDEFGH123456789"
          tls:
			  insecure: false  # Disable insecure connections
    service:
      pipelines:
        traces:  # Pipeline configuration for traces
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp/ociapm]
        metrics:  # Pipeline configuration for metrics
          receivers: [otlp]
          processors: [batch]
          exporters: [otlphttp/ocimetrics]
  resources:  # Resource limits and requests
    limits:
      cpu: "1"  # CPU limit
      memory: "2Gi"  # Memory limit
    requests:
      cpu: "0.5"  # CPU request
      memory: "1Gi"  # Memory request

Note: Endpoint URL is where data is uploaded. The authorisation data key allows APM to accept telemetry data. Follow these instructions for finding endpoints.

Step 3: Deploy the Collector

Once the configuration is prepared, deploy the collector using the following command:

kubectl apply -f OpenTelemetry-collector.yaml

After executing this command, Kubernetes will create the necessary resources based on the OpenTelemetry-collector.yaml definition.

Verify Collector Deployment

Check if the OpenTelemetry Collector is running:

kubectl get pods -n

You should see a pod with a name similar to OpenTelemetry-collector-xxxx in a Running state.

Step 4: Instrument the application with the Agent

To automatically instrument an application for collecting traces and metrics, a Custom Resource (CR) can be used in kubernetes. Below is an example YAML configuration (my-app-instrumentation.yaml) for instrumenting a Java application.


apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-app-instrumentation  # Name of the instrumentation resource
  namespace: my-app  # Namespace where your application is running
spec:
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.5.0  # Java auto-instrumentation image

  env:
    exporting
    - name: OPENTELEMETRY_TRACES_EXPORTER  # Enable OTLP (OpenTelemetry Protocol) for trace
      value: otlp
    - name: OPENTELEMETRY_EXPORTER_OTLP_ENDPOINT  # Endpoint where traces will be sent (see note below)
      value: "http://OpenTelemetry-collector.my-app:4318"
    - name: OPENTELEMETRY_EXPORTER_OTLP_PROTOCOL  # Define the protocol used for trace exporting
      value: "http/protobuf"
    - name: OPENTELEMETRY_METRICS_EXPORTER  # Enable OTLP for metrics exporting
      value: otlp
    - name: OPENTELEMETRY_EXPORTER_OTLP_METRICS_ENDPOINT  # Endpoint where metrics will be sent
      value: "http://OpenTelemetry-collector.my-app:4317"
    - name: OPENTELEMETRY_EXPORTER_OTLP_METRICS_PROTOCOL  # Define the protocol used for metrics exporting
      value: grpc
    - name: OPENTELEMETRY_SERVICE_NAME  # Define a unique service name to identify telemetry data from this application
      value: "my-java-app"

Notes: OPENTELEMETRY _EXPORTER_OTLP_ENDPOINT may vary depending on the setup. To construct the endpoint, the following command can be used:

kubectl get services -n <namespace>

From the output, the service name, namespace and port should be identified. The format for the endpoint will be <service-name>. <namespace>.<port>
Once the YAML configuration is ready, it can be applied with the following command:

kubectl apply -f my-app-instrumentation.yaml

To check if the instrumentation has been applied, use the following:

kubectl get instrumentation -n my-app

Step 5: Instrument the application with the Agent

After installing the OTel toolset, the final step is to integrate the agent into the application using annotation. Annotation can be applied at different levels such as Deployments, StatefulSets and Namespaces. In this example, the annotation will be applied to a StatefulSets.

The annotation will instruct Kubernetes to automatically inject the OpenTelemetry agent into the application. This process ensures that the application will send traces and metrics to the configured OpenTelemetry Collector for monitoring and analysis.

kubectl edit statefulset <statefulset-name> -n <namespace>

Under metadata, add the annotation:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"

Restart the StatefulSet to apply the changes:

kubectl rollout restart statefulset <statefulset-name> -n <namespace>

Or restart the pods where auto-injection should be applied:

kubectl delete pod <your-pod-name> -n <your-namespace-name>

The setup is complete, traces/spans and metrics are now visible in OCI O&M services.

Completing Observability with logs for Kubernetes cluster visibility

The setup so far has shown how to collect traces, spans and metrics. Logs can also be sent to OCI O&M Services to provide complete visibility into Kubernetes cluster. This can be achieved by using the OCI Management Agent and Fluentd. For easy to follow instructions, please refer to this document .

Observability is essential in cloud-native application environments, and there are various ways to achieve it. Oracle provides a range of agents for specific technologies, such as, Java, .NET, and browsers, while supporting open-source tracing solutions. The OpenTelemetry toolset – including the Agent, Collector, and Operator, offers a flexible,vendor-neutral alternative for those seeking standardisation and control.

The best path for your organisation depends on your specific requirements and preferences. It's important to explore both Oracle’s and OpenTelemetry’s solutions to find the one that best fits the organisation's needs. Regardless of the chosen solution for sending telemetry data, OCI's Observability and Management Services will enhance troubleshooting, monitoring and performance, providing valuable insights into applications.

Next steps

For those looking to explore Oracle’s Application Performance Monitoring (APM) capabilities, a free domain for APM is available, allowing teams to experience its powerful features firsthand. Whether leveraging Oracle’s native agents or the OpenTelemetry ecosystem, OCI provides a robust observability framework tailored to various use cases.

Acknowledgements

Primary authors: Mahesh Sharma, Dr. Juergen Fleischer

More Learning Resources

Explore other labs on docs.oracle.com/learn or access more free learning content on the Oracle Learning YouTube channel. Additionally, visit education.oracle.com/learning-explorer to become an Oracle Learning Explorer.

For product documentation, visit Oracle Help Center.

Title and Copyright Information