2 DTrace Concepts
The topics in this section explore DTrace at a conceptual level and describe components and terminology. Topics are general and can help you to understand what DTrace is and how it works.
About DTrace
DTrace is a powerful tracing tool that's available in Oracle Linux for use with the Unbreakable Enterprise Kernel (UEK). DTrace has low overhead and is safe to use on production systems to analyze what a system is doing in real time.
DTrace lets you examine the behavior of user programs and the OS, to understand how the system works, to track down performance problems, and to find the causes of aberrant behavior. DTrace can collect or print stack traces, function arguments, timestamps, and statistical aggregates by using probes that can be runtime events or source-code locations.
Unlike many tracing tools, DTrace is fully programmable. You can collect data for one event and store it for use when another event is triggered. You can choose what information you want to gather and how to report it. DTrace programs have a familiar syntax that draws on the C programming language.
DTrace v2 is a reimplementation of DTrace that uses existing Linux kernel tracing facilities, such as eBPF, which didn't exist when DTrace was first ported to Linux. The new implementation removes DTrace dependencies on specialized kernel patches, but retains syntax compatibility with earlier implementations of DTrace to deliver a mature tracing tool based on modern technology. Furthermore, DTrace v2 also maintains functional compatibility with earlier implementations of DTrace, so that you can perform the same actions using either version of DTrace.
DTrace v2 is available with UEK R6 and later kernels and is implemented as a user space application on Oracle Linux 8 and Oracle Linux 9.
DTrace is developed as an open source project available under the Universal Permissive License (UPL), Version 1.0. You can access source code and more information at https://github.com/oracle/dtrace-utils.
DTrace Components and Terminology
Probes
DTrace works by using probes that identify particular instrumentation in the kernel or within a user space application, or which can be used to identify interval counters or performance event counters. Events such as when particular code is run or when a specific counter is incremented cause a probe to fire and DTrace can perform functions that are bound to the event in a program or script. For example, a probe can fire when a particular file is opened and a DTrace program can print information related to the event that can be useful for debugging or resolving an issue. Equally, at the moment that DTrace starts or ends any tracing activity, the BEGIN and END probes dedicated to these actions always fire.
You can list all the available probes on a system by typing the following command:
sudo dtrace -lOutput is displayed to show each of the different values that are used to reference a probe correctly:
ID PROVIDER MODULE FUNCTION NAME
1 dtrace BEGIN
2 dtrace END
3 dtrace ERROR
4 syscall vmlinux read entry
5 syscall vmlinux read return
6 syscall vmlinux write entry
7 syscall vmlinux write return
...See List and Enable Probes for more information on how to list and enable specific probes.
Probes are made available by providers, which group particular kinds of instrumentation together. If a provider is related to source code, its probes might also include information about the piece of code that the probe relates to in a module and a function identifier. Therefore, a probe is identified by a probe description, grouped into four fields:
- provider
-
The name of the DTrace provider that the probe belongs to.
- module
-
If the probe corresponds to a specific program location, the name of the kernel module, library, or user-space program in which the probe is found. Some probes might be associated with a module name that isn't tied to a particular source location in cases where they relate to more abstract tracepoints.
- function
-
If the probe corresponds to a specific program location, the name of the program function in which the probe is found.
- name
-
The name that provides some idea of the probe's semantic meaning, such as
BEGINorEND.
When referencing a probe, write all four parts of the probe description separated by colons:
provider:module:function:nameNote that some probes don't have a module or function identifier when they're listed. When providing the complete probe description for these probes, you must still include the empty fields:
dtrace:::BEGIN Probes aren't required to have a module and function. The dtrace BEGIN,
END and ERROR probes are good examples of this because
these probes don't correspond to any specific instrumented program function or location.
Instead, these probes are used for more abstract concepts, such as the idea of the end a
tracing request. Other probes, such as those made available by the Profile Provider or the CPC Provider, also don't include
module or function identifiers in their descriptions.
D Programs
You can bind a set of processing instructions called statements to one or more DTrace probes, so that when a probe fires, the specified statements are run to perform some required functionality. The set of enabled probes, the statements, and any conditions that might be evaluated when the probe fires, can all be collated into a D program.
A program can consist of several probe descriptions that decide which probes can trigger some
functionality within the D program. Probe descriptions are followed by a set of
processing instructions, called a clause, that describes what to do when the
selected probe fires. Conditional expressions, called predicates, can be inserted
between the probe descriptions and the clause to control the conditions under which the
actions within the clause are run. For example, a program might be designed to fire for
all system calls and to count these for a particular application. The program would
consist of a probe description for the syscall:::entry probe, a
predicate to limit processing to match either a process ID or the name of an executable,
and a clause that performed the count() function to gather information
about each system call function. The resulting D program might be:
/* Probe descriptions */
syscall:::entry
/* Predicate */
/execname=='date'/
/* Clause */
{
@reads[probefunc]=count();
}When the script is run it shows each system call that's made by the date command and provides the count value for each, as follows:
dtrace: description 'syscall:::entry ' matched 344 probes
Wed 22 Feb 11:54:51 GMT 2023
exit_group 1
lseek 1
mmap 1
write 1
openat 2
read 2
brk 3
close 4
newfstat 4The program probe description matches all system call functions at the entry point. The
program predicate evaluates a built-in variable, execname,
against a string using an operator. The clause includes an aggregation,
@reads, that's used to gather data about the firing probe. In this
case, the aggregation stores a counter that increments every time the probe fires and
the predicate resolves. The counter is implemented by the count()
function and stores count values for each system call probe function. See D Program Syntax Reference
for more information on program structure and syntax.
Aggregations
Aggregations can be used to reduce large bodies of data to smaller, meaningful statistical metrics. Many common functions that are used to understand a set of data are aggregating functions. These functions include the following:
-
Counting the number of elements in the set.
-
Computing the minimum value of the set.
-
Computing the maximum value of the set.
-
Summing all the elements in the set.
-
Creating a histogram of the values in the set, as quantized into certain bins.
Although you could code an application to calculate an aggregation for a set of data, when many probes are firing concurrently, they can overwrite each other's updates to the aggregating variable or the calculation can become a serial bottleneck.
DTrace aggregation functions apply to the data as it's traced, so that the dataset doesn't need to be stored and the aggregation is always available as events occur. In this way, aggregation functions are more efficient and exact, and avoid overwrites. See Aggregations for more information.
Speculation
While predicates can be used to filter out uninteresting events, they're only useful if you already know which events you need to filter. Because DTrace is often used to help debug particular system behaviors, DTrace includes a set of speculation functions that can be used to trace data speculatively.
Speculation is used to trace quantities temporarily until particular information is known, at which case the data can be discarded or committed. By performing speculative tracing you can trace data until you know whether it's useful. For example, to trace data about events that might trigger a particular return code or error, you could speculatively trace all events and discard the trace data if it doesn't match the return code that you're interested in. See Speculation for more information.
Buffers
As DTrace probes fire, the kernel writes data into various buffers that are read
by the dtrace user-space utility, which prints requested data.
The generation of trace data by the kernel and the processing of that data by the
dtrace utility operate asynchronously. The processing of the trace data can
be tuned by setting buffer options and refresh rates. Buffer sizes can be tuned with options
such as aggsize, bufsize, and nspec.
The various options that control buffer sizing and policies are described in DTrace Runtime and Compile-time Options Reference.
Stability
DTrace is a tracing tool that takes advantage of the probes that are included in code that can change over time. DTrace and the D compiler include features to dynamically compute and describe the stability of the D programs that you create. You can use these DTrace stability features to inform you of the stability attributes of D programs or to produce compile-time errors when a program has inappropriate interface dependencies.
DTrace provides two types of stability attributes for entities such as built-in variables, functions, and probes: a stability level and an architectural dependency class. The DTrace stability level helps you to assess risk when developing scripts and tools that are based on DTrace by indicating how likely an interface or DTrace entity might change in a future release or patch. The DTrace dependency class indicates whether an interface is common to all Oracle Linux platforms and processors or whether it's associated with a particular architecture. The two types of attributes that are used to describe interfaces can vary independently.
Applications that depend only on stable interfaces are likely to continue to function reliably on future minor releases and are unlikely to be broken by interim patches. Less stable interfaces can be used for experimentation, prototyping, tuning, and debugging on the current system. Use less stable with the understanding that they might change and become incompatible or even be dropped or replaced with alternatives in future minor releases.
Interfaces can be common to all Oracle Linux platforms and processors or might be associated with a particular system architecture. Dependency classes help indicate architecture dependencies and are orthogonal to stability levels. For example, a DTrace interface can be stable, but only available on x86_64 microprocessors. Or, the interface can be unstable, but common to all Oracle Linux platforms.
See DTrace Stability Reference for more information about the different stability levels and dependency classes.