OpenTelemetry

Modern computing has seen more distributed systems scaled and more services added, such as with microservices and serverless applications. With this growth, service ownership across the system is allocated to different individuals, or even organizations. It is increasingly difficult to observe how these services depend on each and affect other services without a unified observability framework. This becomes clear after a deployment or during an outage when issue identification requires speed and accuracy.

OpenTelemetry is such a unified observability framework. It is a popular open-source observability framework for instrumenting, generating, collecting, and exporting telemetry data. It provides a common specification and protocol so that multiple services can furnish a unified version of traces, metrics, and logs. OpenTelemetry relies on components that participate in the observability model to be instrumented, which means the code emits traces, metrics, and logs.

Numerous managed ODP.NET and ODP.NET Core APIs have been instrumented to support OpenTelemetry observability and standards. Developers and operators can then customize the ODP.NET metrics collected.

ODP.NET OpenTelemetry Traces

Traces record the request paths taken either by an application or end user as they propagate through a multi-service architecture. Traces in OpenTelemetry are defined implicitly by their spans. A trace can be thought of as a directed acyclic graph of spans where the edges between spans are defined as parent/child relationships.

A span, or activity, represents an operation within a transaction. Each span encapsulates the following state:

Display name
Start timestamp and duration
Attributes – list of key-value pairs
Events – a tuple (timestamp, name, attributes) and can number from zero or more
The parent span identifier
Links to causally related spans through SpanContext, which can number from zero or more

ODP.NET spans become child spans if the application creates a (parent) span before calling ODP.NET instrumented APIs.

ODP.NET OpenTelemetry publishes the following activity tags, or attributes.

Table 3-41 ODP.NET OpenTelemetry activity tags

Attribute Name	Description
`db.system`	Database name. `Oracle` is the attribute value for Oracle databases.
`db.name`	Database being accessed by the command. The attribute value is the same as the `OracleConnection` `DatabaseName` property.
`db.user`	Database user
`db.statement`	Database statement being executed
`otel.status_description`	Status description of the trace
`otel.status_code`	Status code of the trace
`exception.stacktrace`	Exception stack trace
`exception.type`	Exception type
`exception.message`	Exception message
`Activity.DisplayName`	Activity display name or the name of the operation being instrumented.
`server.port`	Server port number
`db.odp.sql_id`	Oracle database SQL statement identifier value, equivalent to `SQL_ID`
`server.address`	Name of the database host

ODP.NET OpenTelemetry Trace Sample Exporter Visualization:


Activity.TraceId:          ea0203e89f932940e0ffd0c6e3ee0913
Activity.SpanId:           fdffbc4cbd46442a
Activity.TraceFlags:           Recorded
Activity.ParentSpanId:    2d0e9a180b2cc6f1
Activity.ActivitySourceName: Oracle.ManagedDataAccess.Core
Activity.DisplayName: Connect myhostname:1521:dbview
Activity.Kind:        Client
Activity.StartTime:   2024-04-08T07:00:13.8434743Z
Activity.Duration:    00:00:00.3927843
Activity.Tags:
    db.system: oracle
    db.user: bloguser
    db.name: dbview
    server.address: myhostname
    server.port: 1521
StatusCode : Ok
Resource associated with Activity:
    service.name: DemoApp
    service.version: 1.0.0
    service.instance.id: d16fd8e5-8c49-4489-900b-55af928044c6


Activity.TraceId:          ea0203e89f932940e0ffd0c6e3ee0913
Activity.SpanId:           2d0e9a180b2cc6f1
Activity.TraceFlags:           Recorded
Activity.ParentSpanId:    a5d9f8d14f7c7363
Activity.ActivitySourceName: Oracle.ManagedDataAccess.Core
Activity.DisplayName: Open myhostname:1521:dbview
Activity.Kind:        Client
Activity.StartTime:   2024-04-08T07:00:13.7765477Z
Activity.Duration:    00:00:00.5161881
Activity.Tags:
    db.system: oracle
    db.user: bloguser
    db.name: dbview
    server.address: myhostname
    server.port: 1521
StatusCode : Ok
Resource associated with Activity:
    service.name: DemoApp
    service.version: 1.0.0
    service.instance.id: d16fd8e5-8c49-4489-900b-55af928044c6

Activity.TraceId:          ea0203e89f932940e0ffd0c6e3ee0913
Activity.SpanId:           60c7fd751cfda4b7
Activity.TraceFlags:           Recorded
Activity.ParentSpanId:    44c746737d1eb69c
Activity.ActivitySourceName: Oracle.ManagedDataAccess.Core
Activity.DisplayName: SendExecuteRequest myhostname:1521:dbview
Activity.Kind:        Client
Activity.StartTime:   2024-04-08T07:00:14.3763571Z
Activity.Duration:    00:00:00.0207091
Activity.Tags:
    db.system: oracle
    server.address: myhostname
    server.port: 1521
    db.name: dbview
    db.user: bloguser
    db.statement: select count(*) from blogs
    db.odp.sql_id: 28b6f34d0ft8x
StatusCode : Ok
Resource associated with Activity:
    service.name: DemoApp
    service.version: 1.0.0
    service.instance.id: d16fd8e5-8c49-4489-900b-55af928044c6


Activity.TraceId:          ea0203e89f932940e0ffd0c6e3ee0913
Activity.SpanId:           44c746737d1eb69c
Activity.TraceFlags:           Recorded
Activity.ParentSpanId:    a5d9f8d14f7c7363
Activity.ActivitySourceName: Oracle.ManagedDataAccess.Core
Activity.DisplayName: ExecuteNonQuery myhostname:1521:dbview
Activity.Kind:        Client
Activity.StartTime:   2024-04-08T07:00:14.3369387Z
Activity.Duration:    00:00:00.1049166
Activity.Tags:
    db.system: oracle
    server.address: myhostname
    server.port: 1521
    db.name: dbview
    db.user: bloguser
    db.statement: select count(*) from blogs
    db.odp.sql_id: 28b6f34d0ft8x
StatusCode : Ok
Resource associated with Activity:
    service.name: DemoApp
    service.version: 1.0.0
    service.instance.id: d16fd8e5-8c49-4489-900b-55af928044c6

Activity.TraceId:          ea0203e89f932940e0ffd0c6e3ee0913
Activity.SpanId:           e503c191cb27b09c
Activity.TraceFlags:           Recorded
Activity.ParentSpanId:    a5d9f8d14f7c7363
Activity.ActivitySourceName: Oracle.ManagedDataAccess.Core
Activity.DisplayName: Close myhostname:1521:dbview
Activity.Kind:        Client
Activity.StartTime:   2024-04-08T07:00:14.4862863Z
Activity.Duration:    00:00:00.0122342
Activity.Tags:
    db.system: oracle
    server.address: myhostname
    server.port: 1521
    db.name: dbview
    db.user: bloguser
StatusCode : Ok
Resource associated with Activity:
    service.name: DemoApp
    service.version: 1.0.0
    service.instance.id: d16fd8e5-8c49-4489-900b-55af928044c6

Activity.TraceId:          ea0203e89f932940e0ffd0c6e3ee0913
Activity.SpanId:           a5d9f8d14f7c7363
Activity.TraceFlags:           Recorded
Activity.ActivitySourceName: TEST
Activity.DisplayName: SampleTestActivity
Activity.Kind:        Internal
Activity.StartTime:   2024-04-08T07:00:13.6475302Z
Activity.Duration:    00:00:00.9192665
Resource associated with Activity:
    service.name: DemoApp
    service.version: 1.0.0
    service.instance.id: d16fd8e5-8c49-4489-900b-55af928044c6

ODP.NET OpenTelemetry supports both dynamic and manual instrumentation. The next section describes how to set up each instrumentation method.

Dynamic and Manual Instrumentation

ODP.NET OpenTelemetry requires an exporter, such Jaeger or Zipkin, to visualize and analyze traces. These exporters are available as NuGet packages and will also include the OpenTelemetry SDK as a NuGet dependency.

ODP.NET OpenTelemetry manual instrumentation has the following requirements:

Oracle.ManagedDataAccess.OpenTelemetry package available from NuGet Gallery.
Exporter NuGet package, such as Console or Zipkin, which is used for instrumentation visualization.
OpenTelemetry SDK, which will be installed automatically with an exporter NuGet package.

Once the ODP.Net OpenTelemetry nuget package is installed, ODP.NET OpenTelemetry is enabled by invoking the TracerProviderBuilder AddOracleDataProviderInstrumentation extension method upon application startup. It accepts various options for configuring OpenTelemetry instrumentation.

Code sample: Enabling ODP.NET OpenTelemetry instrumentation with default options

using OpenTelemetry.Trace;

Sdk.CreateTracerProviderBuilder()
.AddOracleDataProviderInstrumentation()  // ODP.NET extension method 
.AddConsoleExporter()
.Build();

Code sample: Enabling ODP.NET OpenTelemetry instrumentation with all options enabled

using OpenTelemetry.Trace;

Sdk.CreateTracerProviderBuilder()
.AddOracleDataProviderInstrumentation(o =>
        {
          o.EnableConnectionLevelAttributes = true;
          o.RecordException = true;
          o.InstrumentOracleDataReaderRead = true;
          o.SetDbStatementForText = true;
        })
.AddConsoleExporter()
.Build();

After enabling ODP.NET OpenTelemetry, whenever a provider instrumented API is called, OpenTelemetry traces will be generated and sent to the configured exporter.

By default, instrumentation for potentially sensitive data is disabled, such as for SQL statements.

A second manual instrumentation method allows enabling ODP.NET OpenTelemetry without adding the ODP.NET OpenTelemetry NuGet package and without calling the AddOracleDataProviderInstrumentation() extension method. It requires adding the TracerProviderBuilder call AddSource(“Oracle.ManagedDataAccess.Core”) for ODP.NET Core or AddSource(“Oracle.ManagedDataAccess”) for managed ODP.NET.

ODP.NET OpenTelemetry instrumentation will then be enabled using default options. The defaults must be used as there is no way to modify option values. Changing these values require using ODP.NET OpenTelemetry NuGet package directly.

OpenTelemetry.Api NuGet package is installed automatically if the application adds any OpenTelemetry exporter NuGet packages, such as OpenTelemetry.Exporter.Console or OpenTelemetry.Exporter.Zipkin. OpenTelemetry.Api is necessary to call the AddSource() method.

Code sample: Enabling ODP.NET OpenTelemetry manual instrumentation via AddSource


Sdk.CreateTracerProviderBuilder()
         .AddSource("Oracle.ManagedDataAccess.Core")
         .AddConsoleExporter()
         .Build();

Through this instrumentation method, it is possible to enable and disable ODP.NET OpenTelemetry tracing programmatically at runtime dynamically. To enable, set OracleConfiguration OpenTelemetryTracing to true (default). To disable, set it to false.

OpenTelemetryTracing is available as a managed ODP.NET .NET configuration file setting as well. As a .NET config setting, it is read upon app startup only. Changes to this setting's value after startup are ignored.

Automatic Instrumentation

OpenTelemetry automatic instrumentation enables ODP.NET tracing without requiring code changes. Existing .NET apps can turn on tracing without having to recompile.

To use it, add the OpenTelemetry.AutoInstrumentation NuGet package to your project. Older package versions require setting environment variables to enable ODP.NET tracing. For example, the following environment variable settings turn on ODP.NET Core tracing using the console exporter.


OTEL_DOTNET_AUTO_TRACES_ADDITIONAL_SOURCES=Oracle.ManagedDataAccess.Core
OTEL_DOTNET_AUTO_TRACES_CONSOLE_EXPORTER_ENABLED=true

ODP.NET automatic instrumentation is on by default, without requiring environment variables, when using a newer OpenTelemetry.AutoInstrumentation NuGet package.

Instrumented ODP.NET APIs

The list of managed ODP.NET and ODP.NET Core APIs that are instrumented to support OpenTelemetry is as follows. Every ODP.NET database round trip originating from one of these APIs is instrumented as well.

OracleCommand
- ExecuteNonQuery()
- ExecuteNonQueryAsync(CancellationToken cancellationToken)
- ExecuteNonQueryAsync()
- All ExecuteReader overloads
- All ExecuteReaderAsync overloads
- ExecuteScalar()
- ExecuteScalarAsync(CancellationToken cancellationToken)
- ExecuteStream()
- ExecuteToStream(Stream)
- ExecuteXmlReader()
- All ExecuteXmlReaderAsync overloads
OracleConnection
- Close()
- Open()
- OpenAsync()
OracleDataAdapter
- All Fill overloads
OracleDataReader
- Read()
- ReadAsync()