Statically Defined Tracing of User Applications

13 Statically Defined Tracing of User Applications

DTrace provides a facility for user application developers to define customized probes in application code to augment the capabilities of the pid provider. These static probes impose little to no overhead when disabled and are dynamically enabled like all other DTrace probes. You can use static probes to describe application semantics to users of DTrace without exposing or requiring implementation knowledge of your applications. This chapter describes how to define static probes in user applications and how to use DTrace to enable such probes in user processes.

Note:

DTrace supports statically defined tracing of user applications for both 32-bit and 64-bit binaries.

For information about using static probes with kernel modules, see Statically Defined Tracing of Kernel Modules.

Choosing the Probe Points

DTrace enables developers to embed static probe points in application code, including both complete applications and shared libraries. You can enable these probes wherever the application or library is running, either in development or production. You should define probes that have a semantic meaning that is readily understood by your DTrace user community. For example, you could define query-receive and query-respond probes for a web server that correspond to a client that is submitting a request and the web server that is responding to the request. These example probes are easily understood by most DTrace users and correspond to the highest level abstractions for the application, rather than lower-level implementation details. DTrace users can use these probes to understand the time distribution of requests. If your query-receive probe presented the URL request strings as an argument, a DTrace user could determine which requests were generating the most disk I/O by combining this probe with the io provider.

You should also consider the stability of the abstractions you describe when choosing probe names and locations. For example, will the probe persist in future releases of the application even if the implementation changes? Does the probe make sense on all system architectures or is it specific to a particular instruction set? This chapter discusses how these decisions can guide your static tracing definitions.

Adding Probes to an Application

DTrace probes for libraries and executables are defined in an ELF section in the corresponding application binary. The following topics are discussed in more detail in this section: defining probes, adding probes to your application source code, and augmenting your application's build process to include the DTrace probe definitions.

Defining Providers and Probes

You define DTrace probes in a .d source file, which is then used when compiling and linking your application. First, select an appropriate name for your user application provider. The provider name that you choose is appended with the process identifier for each process that is executing your application code. For example, if you chose the provider name myserv for a web server that was executing as process ID 1203, the DTrace provider name that corresponds to this process would be myserv1203. In a .d source file, you would add a provider definition similar to the one that is shown the following example:

provider myserv
{
  ...
};

Next, add a definition for each probe and the corresponding arguments. The following example defines the two probes that are discussed in Choosing the Probe Points. The first probe has two arguments, both of type char *. The second probe has no arguments. The D compiler converts two consecutive underscores (__) to a dash (-) in the probe name:

provider myserv
{
  probe query__receive(char *, char *);
  probe query__respond();
};

You can add stability attributes to your provider definition so that consumers of your probes understand the likelihood of change in future versions of your application. See DTrace Stability Features for more information on DTrace stability attributes.

The following example illustrates how stability attributes are defined:

#pragma D attributes Evolving/Evolving/Common provider myserv provider
#pragma D attributes Private/Private/Unknown provider myserv module
#pragma D attributes Private/Private/Unknown provider myserv function
#pragma D attributes Evolving/Evolving/Common provider myserv name
#pragma D attributes Evolving/Evolving/Common provider myserv args

provider myserv
{
  probe query__receive(char *, char *);
  probe query__respond();
};

Adding Probes to Application Code

After you have defined your probes in a .d file, you then need to augment your source code to indicate the locations that should trigger your probes. Consider the following example C application source code:

void main_look(void)
{
  ...
  query = wait_for_new_query();
  process_query(query);
  ...
}

To add probes to an application, use the -h option to the dtrace command, which generates a header file based on the probe definitions. For example, the following command generates the header file myserv.h, which contains macro definitions corresponding to the probe definitions in myserv.d:

# dtrace -h -s myserv.d

This method is recommended, as the coding is easier to implement and understand. The method is also compatible with both C and C++. In addition, because the generated macros depend on the types that you define in the provider definition, the compiler can perform type checking on them.

For example, you can add a probe site by using the MYSERV_QUERY_RECEIVE macro that dtrace -h defines in myserv.h:

#include "myserv.h"
...
void main_look(void)
{
  ...
  query = wait_for_new_query();
  MYSERV_QUERY_RECEIVE(query->clientname, query->msg);
  process_query(query);
  ...
}

In the previous example, the name of the macro encodes both the provider name and the probe name.

Testing if a Probe Is Enabled

The computational overhead of a DTrace probe is usually equivalent to a few no-op instructions. However, setting up probe arguments can be expensive, particularly in the case of dynamic languages, where the code has to determine the name of a class or the method at runtime.

In addition to the probe macro, the dtrace -h command creates an is-enabled probe macro for each probe that you specify in the provider definition. To ensure that your program computes the arguments to a DTrace probe only when required, you can use the is-enabled probe test to verify whether the probe is currently enabled, for example:

if (MYSERV_QUERY_RECEIVE_ENABLED())
  MYSERV_QUERY_RECEIVE(query->clientname, query->msg);

If the probe arguments are computationally expensive to calculate, the slight overhead that is incurred by performing the is-enabled probe test is more than offset when the probe is not enabled.

Building Applications With Probes

You must augment the build process for your application to include the DTrace provider and probe definitions. A typical build process takes each source file and compiles it to create a corresponding object file. The compiled object files are then linked to each other to create the finished application binary, as shown in the following example:

src1.o: src1.c
    gcc -c src1.c

src2.o: src2.c
    gcc -c src2.c

myserv: src1.o src2.o
    gcc -o myserv src1.o src2.o

If you included DTrace probe definitions in your application, you need to add appropriate Makefile rules to your build process to execute the dtrace command.

The dtrace command post-processes the object files that are created by the preceding compiler commands and generates the object file myserv.o from myserv.d and the other object files. The -G option is used to link provider and probe definitions with a user application.

The -Wl,--export-dynamic link options to gcc are required to support symbol lookup in a stripped executable at runtime, for example, by running ustack().

If you inserted probes in the source code by using the macros that were defined in a header file created by dtrace -h, you need to include that command in the Makefile:

myserv.h: myserv.d
    dtrace -h -s myserv.d

src1.o: src1.c myserv.h
    gcc -c src1.c

src2.o: src2.c myserv.h
    gcc -c src2.c

myserv.o: myserv.d src1.o src2.o
    dtrace -G -s myserv.d src1.o src2.o

myserv: myserv.o
    gcc -Wl,--export-dynamic,--strip-all -o myserv myserv.o src1.o src2.o

The rules in the Makefile take into account the dependency of the header file on the probe definition.

Using Statically Defined Probes

The DTrace helper device (/dev/dtrace/helper) enables a user-space application that contains USDT probes to send probe provider information to DTrace.

If the program that is to be traced is run by a user other than root, change the mode of the DTrace helper device to allow the user to record tracing information:

# chmod 666 /dev/dtrace/helper

Alternatively, if the acl package is installed on your system, you can use an ACL rule to limit access to a specific user, as shown in the following example:

# setfacl -m u:guest:rw /dev/dtrace/helper
# ls -l /dev/dtrace
total 0
crw-rw----  1 root root 10, 16 Sep 26 10:38 dtrace
crw-rw----+ 1 root root 10, 17 Sep 26 10:38 helper
drwxr-xr-x  2 root root     80 Sep 26 10:38 provider
# getfacl /dev/dtrace/helper
getfacl: Removing leading '/' from absolute path names
# file: dev/dtrace/helper
# owner: root
# group: root
user::rw-
user:guest:rw-
group::rw-
mask::rw-
other::---

Note:

You must change the mode on the device before the user runs the program.

The full name of a probe in a user application takes the usual provider PID : module : function : name form, where:

provider: Is the name of the provider, as defined in the provider definition file.
PID: Is the process ID of the running executable.
module: Is the name of the executable.
function: Is the name of the function where the probe is located.
name: Is the name of the probe, as defined in the provider definition file with any two consecutive underscores (__) replaced by a dash (-).

For example, for a myserv process with a PID of 1173, the full name of the query-receive probe would be myserv1173:myserv:main_look:query-receive.

The following simple example shows how to invoke a traced process from dtrace:

# dtrace -c ./myserv -qs /dev/stdin <<EOF
                     $target:::query-receive
    {
      printf("%s:%s:%s:%s %s %s\n", probeprov, probemod, probefunc, probename,
                                    stringof(args[0]), stringof(args[1]));
    }
  $target:::query-respond
    {
      printf("%s:%s:%s:%s\n", probeprov, probemod, probefunc, probename);
    }
EOF

myserv1173:myserv:main_look:query-receive foo1 msg1
myserv1173:myserv:process_query:query-respond
myserv1173:myserv:main_look:query-receive bar2 msg1
myserv1173:myserv:process_query:query-respond
...

Note:

For the query-receive probe, stringof() is used to cast args[0] and args[1] to type string. Otherwise, a DTrace compilation error similar to the following is displayed:

dtrace: failed to compile script /dev/stdin: line 7:
printf( ) argument #5 is incompatible with conversion #4 prototype:
	conversion: %s
	 prototype: char [] or string (or use stringof)
	  argument: char *

If the probe arguments were defined as type string instead of char * in the probe definition file, a compilation warning similar to the following would be displayed:

In file included from src1.c:5:
myserv.h:39: warning: parameter names (without types) in function declaration

In this case, casting the probe arguments to the type string would no longer be required.

The following script illustrates the complete process of instrumenting, compiling and tracing a simple user-space program. Save it in a file named testscript:

#!/bin/bash

# Define the probes
cat > prov.d <<EOF
provider myprog
{
  probe dbquery__entry(char *);
  probe dbquery__result(int);
};
EOF

# Create the C program
cat > test.c <<EOF
#include <stdio.h>
#include "prov.h"

int
main(void)
{
        char *query = "select value from table where name = 'foo'";
        /* If the dbquery-entry probe is enabled, trigger it */
        if (MYPROG_DBQUERY_ENTRY_ENABLED())
                MYPROG_DBQUERY_ENTRY(query);
        /* Pretend to run query and obtain result */
        sleep(1);
        int result = 42;
        /* If the dbquery-result probe is enabled, trigger it */
        if (MYPROG_DBQUERY_RESULT_ENABLED())
                MYPROG_DBQUERY_RESULT(result);
        return (0);
}
EOF



test.o: test.c prov.h
  gcc -c test.c

prov.o: prov.d test.o
  dtrace -G -s prov.d test.o

test: prov.o
  gcc -o test prov.o test.o
EOF

# Make the executable
make test

# Trace the program
dtrace -c ./test -qs /dev/stdin <<EOF
myprog\$target:::dbquery-entry
{
        self->ts = timestamp;
        printf("Query = %s\n", stringof(args[0]));
}

myprog\$target:::dbquery-result
{
        printf("Query time = %d microseconds; Result = %d\n",
            (timestamp - self->ts) / 1000, args[0]);
}
EOF

The output from running this script shows the compilation steps, as well as the results of tracing the program:

# chmod +x testscript
# ./testscript
dtrace -h -s prov.d
gcc -c test.c
dtrace -G -s prov.d test.o
gcc -o test prov.o test.o
Query = select value from table where name = 'foo'
Query time = 1000481 microseconds; Result = 42