Chapter 1 About DTrace

DTrace provides dynamic tracing, which is the ability to instrument a running operating system kernel.

DTrace enables you to associate actions, such as collecting or printing stack traces, function arguments, timestamps, and statistical aggregates, with probes, which can be runtime events or source-code locations. The D language is powerful, yet simple. DTrace is dynamic, has low overhead, and is safe to use on production systems. It enables you to examine the behavior of user programs and the operating system, to understand how your system works, to track down performance problems, and to locate the causes of aberrant behavior.

DTrace is a kernel framework that dynamically traces data into buffers that are read by consumers. On Oracle Linux, you will probably only use one consumer, the dtrace command-line utility, which contains the D language that grants you full access to the framework's power.

This guide is largely a reference manual. For information about how to use DTrace and step-by-step examples, see Oracle® Linux: DTrace Tutorial.

1.1 Getting Started With DTrace

Note

Most uses of DTrace require root privileges.

Prior to installing the dtrace_utils package, ensure that you are subscribed to the ULN channel that corresponds to the UEK kernel that you are running. For example, if you are running Oracle Linux 7 with UEK R5, the dtrace_utils package is available in the ol7_UEKR5 channel. For more information about subscribing to channels on ULN, see Oracle® Linux: Unbreakable Linux Network User's Guide for Oracle Linux 6 and Oracle Linux 7.

For information about updating your Oracle Linux or UEK release, see the documentation at https://docs.oracle.com/en/operating-systems/linux.html.

Install the dtrace-utils package:

# yum install dtrace-utils

If you want to implement a libdtrace consumer:

# yum install dtrace-utils-devel

If you want to develop a DTrace provider:

# yum install dtrace-modules-provider-headers

To confirm that dtrace is properly installed on your system and that you have all of the required privileges, use the dtrace -l command. Running this command should load any of the required kernel modules and the output should indicate any available probes.

Note

The dtrace-utils package installs dtrace in /usr/sbin/dtrace. Make sure your path detects this path instead of the similarly named utility that is located in /usr/bin/dtrace, which is installed by the systemtap-sdt-devel package.

A provider is a set of probes with a particular kind of instrumentation.

Note

To use a provider's probes, the kernel module that supports that provider must be loaded. Typically, dtrace automatically handles this for you. Upon first use, it will load the dtrace module and all of the modules that are listed in /etc/dtrace-modules, which the system administrator can edit.

In some cases, the kernel module that supports the desired provider must be loaded manually, for example:

# more /etc/dtrace-modules
sdt
systrace
profile
fasttrap
# modprobe sdt
# modprobe systrace
# modprobe profile
# modprobe fasttrap

These required modules are different from the modules, if any, that are instrumented by the provider's probes and are found in the dtrace -l output. For example, while the module that is required to support proc probes is sdt, the module that these probes instrument is vmlinux, as shown in the following output:

# dtrace -l -P proc
   ID   PROVIDER    MODULE          FUNCTION NAME
  197       proc   vmlinux          _do_fork lwp-create
  198       proc   vmlinux          _do_fork create
  225       proc   vmlinux           do_exit lwp-exit
  226       proc   vmlinux           do_exit exit
  275       proc   vmlinux   do_sigtimedwait signal-clear
...

You dynamically assign actions to be taken at probes, which can be runtime events or source-code locations. Every probe in DTrace has two names: a unique integer ID, which is assigned as the probes are loaded, and a human-readable string name. You can start learning about DTrace by building some very simple requests that use the probe named BEGIN. The BEGIN probe fires once each time you start a new tracing request.

Use the dtrace command with the -n option to enable a probe by specifying its name:

# dtrace -n BEGIN
dtrace: description 'BEGIN' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0      1                           :BEGIN 
^C
#

The default output of the previous example displays the following information: the probes that were matched, column headers, and then one row each time a probe fires. The default per row is the CPU where the probe fired and information about which probe fired. DTrace remains paused, waiting for other probes to fire. To exit, press Ctrl-C.

You can construct DTrace requests by using arbitrary numbers of probes and actions. For example, create a simple request using two probes by adding the END probe to the command shown in the previous example. The END probe fires once when tracing is completed.

Type the following command, and then press Ctrl-C in your shell again, after you see the line of output for the BEGIN probe:

# dtrace -n BEGIN -n END 
dtrace: description 'BEGIN' matched 1 probe
dtrace: description 'END' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0      1                           :BEGIN 
^C
  1      2                             :END

Pressing Ctrl-C to exit dtrace triggers the END probe. The dtrace command reports this probe firing before exiting.

In addition to constructing DTrace experiments on the command line, you can also write DTrace experiments in text files by using the D programming language.

In a text editor, create a new file named hello.d and type your first D program:

BEGIN
{
  trace("hello, world");
  exit(0);
}

After you save the program, you can run it by using the dtrace -s command, as shown in the following example:

# dtrace -s hello.d
dtrace: script 'hello.d' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0      1                           :BEGIN   hello, world   
#

The dtrace command printed the same output as the previous example, followed by the text, ”hello, world”. However, unlike the previous example, you did not have to wait and then press Ctrl-C. These changes were the result of the actions that you specified for the BEGIN probe in hello.d.

To understand what happened, let us explore the structure of your D program in more detail.

  • Each D program consists of a series of clauses, and each clause describes one or more probes to enable, as well as an optional set of actions to perform when the probes fires.

  • The actions are listed as a series of statements that are enclosed in braces ({}) that follow the probe name. Each statement ends with a semicolon (;).

  • The first statement uses the trace() function to indicate that DTrace should record the specified argument, the string, ”hello, world”, when the BEGIN probe fires and then print it out.

  • The second statement uses the exit() function to indicate that DTrace should cease tracing and exit the dtrace command.

DTrace provides a set of useful functions such as trace() and exit() for you to call in your D programs.

To call a function, you specify its name, followed by a parenthesized list of arguments. See Chapter 4, Actions and Subroutines for the complete set of D functions.

If you are familiar with the C programming language, you probably have noticed that DTrace's D programming language is very similar to C. Indeed, D is derived from a large subset of C, combined with a special set of functions and variables to help make tracing easy. These features are described in more detail in subsequent chapters. If you have written a C program previously, you should be able to immediately transfer most of your knowledge to building tracing programs in D. If you have never written a C program, learning D is still relatively easy. By the end of this chapter, you will understand all of the syntax. First, let us take a step back from language rules and learn more about how DTrace works. Then, later in this guide, you will learn how to build more interesting D programs.

1.2 Providers and Probes

In the preceding examples, you learned how to use two simple probes named BEGIN and END. DTrace probes come in sets that are called providers, each of which performs a particular kind of instrumentation to create probes. When you use DTrace, each provider is given an opportunity to publish the probes that it can provide to the DTrace framework. You can then enable and bind your tracing actions to any of the probes that have been published.

You can list all of the available probes on your system by typing the following command:

# dtrace -l 
   ID   PROVIDER            MODULE                          FUNCTION NAME
    1     dtrace                                                     BEGIN
    2     dtrace                                                     END
    3     dtrace                                                     ERROR
    4    syscall           vmlinux                              read entry
    5    syscall           vmlinux                              read return
    6    syscall           vmlinux                             write entry
    7    syscall           vmlinux                             write return
    ...

Note that it might take some time for all of the output to be displayed.

To count all of the probes, type the following command:

# dtrace -l | wc -l
4097

Note that you might observe a different total on your system, as the number of probes can vary, depending on the following: your operating platform, the software you have installed, and the provider modules you have loaded. Note also that this output is not the complete list. As will be described later, some providers offer the ability to create new probes on-the-fly, based on your tracing requests, which makes the actual number of DTrace probes virtually unlimited. Notice that each probe has the two names previously mentioned: an integer ID and a human-readable name. The human-readable name is composed of four parts that are displayed as separate columns in the dtrace output and are as follows:

provider

A name of the DTrace provider that is publishing this probe.

module

If this probe corresponds to a specific program location, the name of the kernel module, library, or user-space program in which the probe is located.

function

If this probe corresponds to a specific program location, the name of the program function in which the probe is located.

name

A name that provides some idea of the probe's semantic meaning, such as BEGIN or END.

When writing the full human-readable name of a probe, write all four parts of the name separated by colons like this:

provider:module:function:name

Notice that some of the probes in the list do not have a module and function, such as the BEGIN and END probes that were used previously. Some probes leave these two fields blank because these probes do not correspond to any specific instrumented program function or location. Instead, these probes refer to a more abstract concept, such as the idea of the end of your tracing request.

By convention, if you do not specify all of the fields of a probe name, DTrace matches your request to all of the probes with matching values in the parts of the name that you do specify. In other words, when you used the probe name BEGIN in the previous exercise, you were actually directing DTrace to match any probe with the name field BEGIN, regardless of the value of the provider, module, and function fields. Because there is only one probe matching that description, the result is the same. You now know that the true name of the BEGIN probe is dtrace:::BEGIN, which indicates that this probe is provided by the DTrace framework itself and is not specific to any function. Therefore, the hello.d program could be written as follows and would produce the same result:

dtrace:::BEGIN
{
  trace("hello, world");
  exit(0);
}