13.2 Adding Probes to an Application

13.2.1 Defining Providers and Probes
13.2.2 Adding Probes to Application Code
13.2.3 Testing if a Probe Is Enabled
13.2.4 Building Applications with Probes
13.2.5 Using Statically Defined Probes

DTrace probes for libraries and executables are defined in an ELF section in the corresponding application binary. This section describes how to define your probes, add them to your application source code, and augment your application's build process to include the DTrace probe definitions.

13.2.1 Defining Providers and Probes

You define DTrace probes in a .d source file, which is then used when compiling and linking your application. First, select an appropriate name for your user application provider. In a .d source file, add a provider definition similar to the following example:

provider myserv
{
  ...
};

Next, add a definition for each probe and the corresponding arguments. The following example defines the two probes discussed in Section 13.1, “Choosing the Probe Points”. The first probe has two arguments, both of type char *, and the second probe has no arguments. The D compiler converts two consecutive underscores (__) to a dash (-) in the probe name.

provider myserv
{
  probe query__receive(char *, char *);
  probe query__respond();
};

You can add stability attributes to your provider definition so that consumers of your probes understand the likelihood of change in future versions of your application. See Chapter 15, Stability for more information on the DTrace stability attributes. Stability attributes are defined as shown in the following example:

#pragma D attributes Evolving/Evolving/Common provider myserv provider
#pragma D attributes Private/Private/Unknown provider myserv module
#pragma D attributes Private/Private/Unknown provider myserv function
#pragma D attributes Evolving/Evolving/Common provider myserv name
#pragma D attributes Evolving/Evolving/Common provider myserv args

provider myserv
{
  probe query__receive(char *, char *);
  probe query__respond();
};

13.2.2 Adding Probes to Application Code

When you have defined your probes in a .d file, you need to augment your source code to indicate the locations that should trigger your probes. Consider the following example C application source code:

void main_look(void)
{
  ...
  query = wait_for_new_query();
  process_query(query);
  ...
}

There are two methods for adding probes to an application:

  • By using the DTRACE_PROBE macros that are defined in /usr/include/sys/sdt.h. This method is compatible with C but not with C++.

  • By using the -h option to dtrace to generate a header file based on the probe definitions. For example, the following command generates the header file myserv.h, which contains macro definitions that correspond to the probe definitions in myserv.d:

    # dtrace -h -s myserv.d

    This method is preferred as the coding is easier to implement and understand. The method is also compatible with both C and C++. In addition, as the generated macros depend on the types that you define in the provider definition, the compiler can perform type checking on them.

To add a probe site by using a DTRACE_PROBE macro defined in /usr/include/sys/sdt.h:

#include <sys/sdt.h>
...
void main_look(void)
{
  ...
  query = wait_for_new_query();
  DTRACE_PROBE2(myserv, query__receive, query->clientname, query->msg);
  process_query(query);
  ...
}

The suffix 2 in the macro name DTRACE_PROBE2 refers the number of arguments that are passed to the probe. The first two arguments to the probe macro are the provider name and probe name and must correspond to your D provider and probe definitions. The remaining macro arguments are the arguments assigned to the DTrace arg0..9 variables (or the args[] array) when the probes fires. Your application source code can contain multiple references to the same provider and probe name. If multiple references to the same probe are present in your source code, any of the macro references causes the probe to fire.

Alternatively, you can add a probe site by using the MYSERV_QUERY_RECEIVE macro that dtrace -h defines in myserv.h:

#include "myserv.h"
...
void main_look(void)
{
  ...
  query = wait_for_new_query();
  MYSERV_QUERY_RECEIVE(query->clientname, query->msg);
  process_query(query);
  ...
}

In this case, the name of the macro encodes the provider name and probe name.

13.2.3 Testing if a Probe Is Enabled

The computational overhead of a DTrace probe is usually equivalent to a few no-op instructions. However, setting up probe arguments can be expensive, particularly in the case of dynamic languages where the code has to determine the name of a class or method at runtime.

In addition to the probe macro, the dtrace -h command creates an is-enabled probe macro for each probe that you specify in the provider definition. To ensure that your program computes the arguments to a DTrace probe only when required, you can use the is-enabled probe test to verify whether the probe is currently enabled, for example:

if (MYSERV_QUERY_RECEIVE_ENABLED())
  MYSERV_QUERY_RECEIVE(query->clientname, query->msg);

If the probe arguments are computationally expensive to calculate, the slight overhead incurred by performing the is-enabled probe test is more than offset when the probe is not enabled.

13.2.4 Building Applications with Probes

You must augment the build process for your application to include the DTrace provider and probe definitions. A typical build process takes each source file and compiles it to create a corresponding object file. The compiled object files are then linked together to create the finished application binary, as shown in the following example:

src1.o: src1.c
    gcc -c src1.c

src2.o: src2.c
    gcc -c src2.c

myserv: src1.o src2.o
    gcc -o myserv src1.o src2.o

If you have included DTrace probe definitions in your application, you need to add appropriate Makefile rules to your build process to execute the dtrace command.

The following example shows how you might modify the rules if you inserted probes by using DTRACE_PROBE macros that are defined in /usr/include/sys/sdt.h:

src1.o: src1.c
    gcc -c src1.c

src2.o: src2.c
    gcc -c src2.c

myserv.o: myserv.d src1.o src2.o
    dtrace -G -s myserv.d src1.o src2.o

myserv: myserv.o
    gcc -Wl,--export-dynamic,--strip-all -o myserv myserv.o src1.o src2.o

The dtrace command post-processes the object files created by the preceding compiler commands and generates the object file myserv.o from myserv.d and the other object files. The -G option is used to link provider and probe definitions with a user application.

The -Wl,--export-dynamic link options to gcc are required to support symbol lookup in a stripped executable at runtime, for example, by running ustack().

If you inserted probes in the source code by using the macros defined in a header file that was created by dtrace -h, you need to include that command as well:

myserv.h: myserv.d
    dtrace -h -s myserv.d

src1.o: src1.c myserv.h
    gcc -c src1.c

src2.o: src2.c myserv.h
    gcc -c src2.c

myserv.o: myserv.d src1.o src2.o
    dtrace -G -s myserv.d src1.o src2.o

myserv: myserv.o
    gcc -Wl,--export-dynamic,--strip-all -o myserv myserv.o src1.o src2.o

The rules in the Makefile now also take account of the dependency of the header file on the probe definition.

13.2.5 Using Statically Defined Probes

The DTrace helper device (/dev/dtrace/helper) allows a user-space application that contains USDT probes to send probe provider information to DTrace.

To allow access to the probes in a DTrace-enabled, user-space program, enable the fasttrap provider by loading the fasttrap module:

# modprobe fasttrap

If the program to be traced is run by a user other than root, change the mode of the DTrace helper device to allow the user to record tracing information, for example:

# chmod 666 /dev/dtrace/helper

Alternatively, if the acl package is installed on your system, you can use an ACL rule to limit access to a specific user, for example:

# setfacl -m u:guest:rw /dev/dtrace/helper
# ls -l /dev/dtrace
total 0
crw-rw----  1 root root 10, 16 Sep 26 10:38 dtrace
crw-rw----+ 1 root root 10, 17 Sep 26 10:38 helper
drwxr-xr-x  2 root root     80 Sep 26 10:38 provider
# getfacl /dev/dtrace/helper
getfacl: Removing leading '/' from absolute path names
# file: dev/dtrace/helper
# owner: root
# group: root
user::rw-
user:guest:rw-
group::rw-
mask::rw-
other::---
Note

You must change the mode on the device before the user runs the program.

The full name of a probe in a user application takes the usual providerPID:module:function:name form, where:

provider

The name of the provider as defined in the provider definition file.

PID

The process ID of the running executable.

module

The name of the executable.

function

The name of the function where the probe is located.

name

The name of the probe as defined in the provider definition file with any two consecutive underscores (__) replaced by a dash (-).

For example, for a myserv process with a PID of 1173, the full name of the query-receive probe would be myserv1173:myserv:main_look:query-receive.

The following simple example shows how to invoke a traced process from dtrace itself.

# dtrace -c ./myserv -qs /dev/stdin <<EOF
  $target:::query-receive
    {
      printf("%s:%s:%s:%s %s %s\n", probeprov, probemod, probefunc, probename,
                                    stringof(args[0]), stringof(args[1]));
    }
  $target:::query-respond
    {
      printf("%s:%s:%s:%s\n", probeprov, probemod, probefunc, probename);
    }
EOF

myserv1173:myserv:main_look:query-receive foo1 msg1
myserv1173:myserv:process_query:query-respond
myserv1173:myserv:main_look:query-receive bar2 msg1
myserv1173:myserv:process_query:query-respond
...
Note

For the query-receive probe, we use stringof() to cast args[0] and args[1] to type string. Otherwise, we would see a DTrace compilation error similar to the following:

dtrace: failed to compile script /dev/stdin: line 7:
printf( ) argument #5 is incompatible with conversion #4 prototype:
	conversion: %s
	 prototype: char [] or string (or use stringof)
	  argument: char *

If we had defined the probe arguments as type string instead of char * in the probe definition file, we would see a compilation warning, for example:

In file included from src1.c:5:
myserv.h:39: warning: parameter names (without types) in function declaration

In this case, casting the probe arguments to type string would no longer be required.

The following script illustrates the complete process of instrumenting, compiling and tracing a simple user-space program:

#!/bin/bash

# Define the probes
cat > prov.d <<EOF
provider myprog
{
  probe dbquery__entry(char *);
  probe dbquery__result(int);
};
EOF

# Create the C program
cat > test.c <<EOF
#include <stdio.h>
#include "prov.h"

int
main(void)
{
        char *query = "select value from table where name = 'foo'";
        /* If the dbquery-entry probe is enabled, trigger it */
        if (MYPROG_DBQUERY_ENTRY_ENABLED())
                MYPROG_DBQUERY_ENTRY(query);
        /* Pretend to run query and obtain result */
        sleep(1);
        int result = 42;
        /* If the dbquery-result probe is enabled, trigger it */
        if (MYPROG_DBQUERY_RESULT_ENABLED())
                MYPROG_DBQUERY_RESULT(result);
        return (0);
}
EOF

# Create the Makefile
cat > Makefile <<EOF
prov.h: prov.d
        dtrace -h -s prov.d

test.o: test.c prov.h
        gcc -c test.c

prov.o: prov.d test.o
        dtrace -G -s prov.d test.o

test: prov.o
        gcc -o test prov.o test.o
EOF

# Make the executable
make test

# Trace the program
dtrace -c ./test -qs /dev/stdin <<EOF
$target:::dbquery-entry
{
        self->ts = timestamp;
        printf("Query = %s\n", stringof(args[0]));
}

$target:::dbquery-result
{
        printf("Query time = %d microseconds; Result = %d\n",
            (timestamp - self->ts) / 1000, args[0]);
}
EOF

The output from running this script shows the compilation steps as well as the results of tracing the program:

# chmod +x testscript
# ./testscript
dtrace -h -s prov.d
gcc -c test.c
dtrace -G -s prov.d test.o
gcc -o test prov.o test.o
Query = select value from table where name = 'foo'
Query time = 1000481 microseconds; Result = 42