Solaris Dynamic Tracing Guide

Data Recording Actions

The data recording actions comprise the core DTrace actions. Each of these actions records data to the principal buffer by default, but each action may also be used to record data to speculative buffers. See Chapter 11, Buffers and Buffering for more details on the principal buffer. See Chapter 13, Speculative Tracing for more details on speculative buffers. The descriptions in this section refer only to the directed buffer, indicating that data is recorded either to the principal buffer or to a speculative buffer if the action follows a speculate().

`trace()`

void trace(expression)

The most basic action is the trace() action, which takes a D expression as its argument and traces the result to the directed buffer. The following statements are examples of trace() actions:

trace(execname);
trace(curlwpsinfo->pr_pri);
trace(timestamp / 1000);
trace(`lbolt);
trace("somehow managed to get here");

`tracemem()`

void tracemem(address, size_t nbytes)

The tracemem() action takes a D expression as its first argument, address, and a constant as its second argument, nbytes. tracemem() copies the memory from the address specified by addr into the directed buffer for the length specified by nbytes.

`printf()`

void printf(string format, ...)

Like trace(), the printf() action traces D expressions. However, printf() allows for elaborate printf(3C)-style formatting. Like printf(3C), the parameters consists of a format string followed by a variable number of arguments. By default, the arguments are traced to the directed buffer. The arguments are later formatted for output by dtrace(1M) according to the specified format string. For example, the first two examples of trace() from trace() could be combined in a single printf():

printf("execname is %s; priority is %d", execname, curlwpsinfo->pr_pri);

For more information on printf(), see Chapter 12, Output Formatting.

`printa()`

void printa(aggregation)
void printa(string format, aggregation)

The printa() action enables you to display and format aggregations. See Chapter 9, Aggregations for more detail on aggregations. If a format is not provided, printa() only traces a directive to the DTrace consumer that the specified aggregation should be processed and displayed using the default format. If a format is provided, the aggregation will be formatted as specified. See Chapter 12, Output Formatting for a more detailed description of the printa() format string.

printa() only traces a directive that the aggregation should be processed by the DTrace consumer. It does not process the aggregation in the kernel. Therefore, the time between the tracing of the printa() directive and the actual processing of the directive depends on the factors that affect buffer processing. These factors include the aggregation rate, the buffering policy and, if the buffering policy is switching, the rate at which buffers are switched. See Chapter 9, Aggregations and Chapter 11, Buffers and Buffering for detailed descriptions of these factors.

`stack()`

void stack(int nframes)
void stack(void)

The stack() action records a kernel stack trace to the directed buffer. The kernel stack will be nframes in depth. If nframes is not provided, the number of stack frames recorded is the number specified by the stackframes option. For example:

# dtrace -n uiomove:entry'{stack()}'
  CPU     ID                    FUNCTION:NAME
    0   9153                    uiomove:entry 
                genunix`fop_write+0x1b
                namefs`nm_write+0x1d
                genunix`fop_write+0x1b
                genunix`write+0x1f7

    0   9153                    uiomove:entry 
                genunix`fop_read+0x1b
                genunix`read+0x1d4

    0   9153                    uiomove:entry 
                genunix`strread+0x394
                specfs`spec_read+0x65
                genunix`fop_read+0x1b
                genunix`read+0x1d4
   ...

The stack() action is a little different from other actions in that it may also be used as the key to an aggregation:

# dtrace -n kmem_alloc:entry'{@[stack()] = count()}'
dtrace: description 'kmem_alloc:entry' matched 1 probe
^C

                rpcmod`endpnt_get+0x47c
                rpcmod`clnt_clts_kcallit_addr+0x26f
                rpcmod`clnt_clts_kcallit+0x22
                nfs`rfscall+0x350
                nfs`rfs2call+0x60
                nfs`nfs_getattr_otw+0x9e
                nfs`nfsgetattr+0x26
                nfs`nfs_getattr+0xb8
                genunix`fop_getattr+0x18
                genunix`cstat64+0x30
                genunix`cstatat64+0x4a
                genunix`lstat64+0x1c
                  1

                genunix`vfs_rlock_wait+0xc
                genunix`lookuppnvp+0x19d
                genunix`lookuppnat+0xe7
                genunix`lookupnameat+0x87
                genunix`lookupname+0x19
                genunix`chdir+0x18
                  1

                rpcmod`endpnt_get+0x6b1
                rpcmod`clnt_clts_kcallit_addr+0x26f
                rpcmod`clnt_clts_kcallit+0x22
                nfs`rfscall+0x350
                nfs`rfs2call+0x60
                nfs`nfs_getattr_otw+0x9e
                nfs`nfsgetattr+0x26
                nfs`nfs_getattr+0xb8
                genunix`fop_getattr+0x18
                genunix`cstat64+0x30
                genunix`cstatat64+0x4a
                genunix`lstat64+0x1c
                  1
    ...

`ustack()`

void ustack(int nframes, int strsize)
void ustack(int nframes)
void ustack(void)

The ustack() action records a user stack trace to the directed buffer. The user stack will be nframes in depth. If nframes is not provided, the number of stack frames recorded is the number specified by the ustackframes option. While ustack() is able to determine the address of the calling frames when the probe fires, the stack frames will not be translated into symbols until the ustack() action is processed at user-level by the DTrace consumer. If strsize is specified and non-zero, ustack() will allocate the specified amount of string space, and use it to perform address-to-symbol translation directly from the kernel. This direct user symbol translation is currently available only for Java virtual machines, version 1.5 and higher. Java address-to-symbol translation annotates user stacks that contain Java frames with the Java class and method name. If such frames cannot be translated, the frames will appear only as hexadecimal addresses.

The following example traces a stack with no string space, and therefore no Java address-to-symbol translation:

# dtrace -n syscall::write:entry'/pid == $target/{ustack(50, 0); 
    exit(0)}' -c "java -version"
dtrace: description 'syscall::write:entry' matched 1 probe
java version "1.5.0-beta3"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-beta3-b58)
Java HotSpot(TM) Client VM (build 1.5.0-beta3-b58, mixed mode)
dtrace: pid 5312 has exited
CPU     ID                    FUNCTION:NAME
  0     35                      write:entry 
              libc.so.1`_write+0x15
              libjvm.so`__1cDhpiFwrite6FipkvI_I_+0xa8
              libjvm.so`JVM_Write+0x2f
              d0c5c946
              libjava.so`Java_java_io_FileOutputStream_writeBytes+0x2c
              cb007fcd
              cb002a7b
              cb002a7b
              cb002a7b
              cb002a7b
              cb002a7b
              cb002a7b
              cb002a7b
              cb002a7b
              cb002a7b
              cb002a7b
              cb002a7b
              cb002a7b
              cb002a7b
              cb000152
              libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_
                          pnMmethodHandle_pnRJavaCallArguments_
                          pnGThread__v_+0x187
              libjvm.so`__1cCosUos_exception_wrapper6FpFpnJJavaValue_
                          pnMmethodHandle_pnRJavaCallArguments_
                          pnGThread__v2468_v_+0x14
              libjvm.so`__1cJJavaCallsEcall6FpnJJavaValue_nMmethodHandle_
                          pnRJavaCallArguments_pnGThread __v_+0x28
              libjvm.so`__1cRjni_invoke_static6FpnHJNIEnv__pnJJavaValue_
                          pnI_jobject_nLJNICallType_pnK_jmethodID_pnSJNI_
                          ArgumentPusher_pnGThread__v_+0x180
              libjvm.so`jni_CallStaticVoidMethod+0x10f
              java`main+0x53d

Notice that the C and C++ stack frames from the Java virtual machine are presented symbolically using C++ “mangled” symbol names, and the Java stack frames are presented only as hexadecimal addresses. The following example shows a call to ustack() with a non-zero string space:

# dtrace -n syscall::write:entry'/pid == $target/{ustack(50, 500); exit(0)}'
      -c "java -version"
dtrace: description 'syscall::write:entry' matched 1 probe
java version "1.5.0-beta3"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-beta3-b58)
Java HotSpot(TM) Client VM (build 1.5.0-beta3-b58, mixed mode)
dtrace: pid 5308 has exited
CPU     ID                    FUNCTION:NAME
  0     35                      write:entry 
              libc.so.1`_write+0x15
              libjvm.so`__1cDhpiFwrite6FipkvI_I_+0xa8
              libjvm.so`JVM_Write+0x2f
              d0c5c946
              libjava.so`Java_java_io_FileOutputStream_writeBytes+0x2c
              java/io/FileOutputStream.writeBytes
              java/io/FileOutputStream.write
              java/io/BufferedOutputStream.flushBuffer
              java/io/BufferedOutputStream.flush
              java/io/PrintStream.write
              sun/nio/cs/StreamEncoder$CharsetSE.writeBytes
              sun/nio/cs/StreamEncoder$CharsetSE.implFlushBuffer
              sun/nio/cs/StreamEncoder.flushBuffer
              java/io/OutputStreamWriter.flushBuffer
              java/io/PrintStream.write
              java/io/PrintStream.print
              java/io/PrintStream.println
              sun/misc/Version.print
              sun/misc/Version.print
              StubRoutines (1)
              libjvm.so`__1cJJavaCallsLcall_helper6FpnJJavaValue_
                          pnMmethodHandle_pnRJavaCallArguments_pnGThread
                          __v_+0x187
              libjvm.so`__1cCosUos_exception_wrapper6FpFpnJJavaValue_
                          pnMmethodHandle_pnRJavaCallArguments_pnGThread
                          __v2468_v_+0x14
              libjvm.so`__1cJJavaCallsEcall6FpnJJavaValue_nMmethodHandle
                          _pnRJavaCallArguments_pnGThread__v_+0x28
              libjvm.so`__1cRjni_invoke_static6FpnHJNIEnv__pnJJavaValue_pnI
                          _jobject_nLJNICallType_pnK_jmethodID_pnSJNI
                          _ArgumentPusher_pnGThread__v_+0x180
              libjvm.so`jni_CallStaticVoidMethod+0x10f
              java`main+0x53d
              8051b9a

The above example output demonstrates symbolic stack frame information for Java stack frames. There are still some hexadecimal frames in this output because some functions are static and do not have entries in the application symbol table. Translation is not possible for these frames.

The ustack() symbol translation for non-Java frames occurs after the stack data is recorded. Therefore, the corresponding user process might exit before symbol translation can be performed, making stack frame translation impossible. If the user process exits before symbol translation is performed, dtrace will emit a warning message, followed by the hexadecimal stack frames, as shown in the following example:

  dtrace: failed to grab process 100941: no such process
                c7b834d4
                c7bca85d
                c7bca1a4
                c7bd4374
                c7bc2628
                8047efc

Techniques for mitigating this problem are described in Chapter 33, User Process Tracing.

Finally, because the postmortem DTrace debugger commands cannot perform the frame translation, using ustack() with a ring buffer policy always results in raw ustack() data.

The following D program shows an example of ustack() that leaves strsize unspecified:

syscall::brk:entry
/execname == $$1/
{
	@[ustack(40)] = count();
}

To run this example for the Netscape web browser, .netscape.bin in default Solaris installations, use the following command:

# dtrace -s brk.d .netscape.bin
dtrace: description 'syscall::brk:entry' matched 1 probe
^C
                libc.so.1`_brk_unlocked+0xc
                88143f6
                88146cd
                .netscape.bin`unlocked_malloc+0x3e
                .netscape.bin`unlocked_calloc+0x22
                .netscape.bin`calloc+0x26
                .netscape.bin`_IMGCB_NewPixmap+0x149
                .netscape.bin`il_size+0x2f7
                .netscape.bin`il_jpeg_write+0xde
                8440c19
                .netscape.bin`il_first_write+0x16b
                8394670
                83928e5
                .netscape.bin`NET_ProcessHTTP+0xa6
                .netscape.bin`NET_ProcessNet+0x49a
                827b323
                libXt.so.4`XtAppProcessEvent+0x38f
                .netscape.bin`fe_EventLoop+0x190
                .netscape.bin`main+0x1875
                   1

                libc.so.1`_brk_unlocked+0xc
                libc.so.1`sbrk+0x29
                88143df
                88146cd
                .netscape.bin`unlocked_malloc+0x3e
                .netscape.bin`unlocked_calloc+0x22
                .netscape.bin`calloc+0x26
                .netscape.bin`_IMGCB_NewPixmap+0x149
                .netscape.bin`il_size+0x2f7
                .netscape.bin`il_jpeg_write+0xde
                8440c19
                .netscape.bin`il_first_write+0x16b
                8394670
                83928e5
                .netscape.bin`NET_ProcessHTTP+0xa6
                .netscape.bin`NET_ProcessNet+0x49a
                827b323
                libXt.so.4`XtAppProcessEvent+0x38f
                .netscape.bin`fe_EventLoop+0x190
                .netscape.bin`main+0x1875
                  1
    ...

`jstack()`

void jstack(int nframes, int strsize)
void jstack(int nframes)
void jstack(void)

jstack() is an alias for ustack() that uses the jstackframes option for the number of stack frames the value specified by , and for the string space size the value specified by the jstackstrsize option. By default, jstacksize defaults to a non-zero value. This means that use of jstack() results in a stack with Java frame translation in place.