Solaris Dynamic Tracing Guide

Destructive Actions

Some DTrace actions are destructive in that they change the state of the system in some well-defined way. Destructive actions may not be used unless they have been explicitly enabled. When using dtrace(1M), you can enable destructive actions using the -w option. If an attempt is made to enable destructive actions in dtrace(1M) without explicitly enabling them, dtrace will fail with a message similar to the following example:


dtrace: failed to enable 'syscall': destructive actions not allowed

Process Destructive Actions

Some destructive actions are destructive only to a particular process. These actions are available to users with the dtrace_proc or dtrace_user privileges. See Chapter 35, Security for details on DTrace security privileges.

stop()

void stop(void)

The stop() action forces the process that fires the enabled probe to stop when it next leaves the kernel, as if stopped by a proc(4) action. The prun(1) utility may be used to resume a process that has been stopped by the stop() action. The stop() action can be used to stop a process at any DTrace probe point. This action can be used to capture a program in a particular state that would be difficult to achieve with a simple breakpoint, and then attach a traditional debugger like mdb(1) to the process. You can also use the gcore(1) utility to save the state of a stopped process in a core file for later analysis.

raise()

void raise(int signal)

The raise() action sends the specified signal to the currently running process. This action is similar to using the kill(1) command to send a process a signal. The raise() action can be used to send a signal at a precise point in a process's execution.

copyout()

void copyout(void *buf, uintptr_t addr, size_t nbytes)

The copyout() action copies nbytes from the buffer specified by buf to the address specified by addr in the address space of the process associated with the current thread. If the user-space address does not correspond to a valid, faulted-in page in the current address space, an error will be generated.

copyoutstr()

void copyoutstr(string str, uintptr_t addr, size_t maxlen)

The copyoutstr() action copies the string specified by str to the address specified by addr in the address space of the process associated with the current thread. If the user-space address does not correspond to a valid, faulted-in page in the current address space, an error will be generated. The string length is limited to the value set by the strsize option. See Chapter 16, Options and Tunables for details.

system()

void system(string program, ...) 

The system() action causes the program specified by program to be executed as if it were given to the shell as input. The program string may contain any of the printf()/printa() format conversions. Arguments must be specified that match the format conversions. Refer to Chapter 12, Output Formatting for details on valid format conversions.

The following example runs the date(1) command once per second:


# dtrace -wqn tick-1sec'{system("date")}'
 Tue Jul 20 11:56:26 CDT 2004
 Tue Jul 20 11:56:27 CDT 2004
 Tue Jul 20 11:56:28 CDT 2004
 Tue Jul 20 11:56:29 CDT 2004
 Tue Jul 20 11:56:30 CDT 2004

The following example shows a more elaborate use of the action, using printf() conversions in the program string along with traditional filtering tools like pipes:

#pragma D option destructive
#pragma D option quiet

proc:::signal-send
/args[2] == SIGINT/
{
	printf("SIGINT sent to %s by ", args[1]->pr_fname);
	system("getent passwd %d | cut -d: -f5", uid);
}

Running the above script results in output similar to the following example:


# ./whosend.d
SIGINT sent to MozillaFirebird- by Bryan Cantrill
SIGINT sent to run-mozilla.sh by Bryan Cantrill
^C
SIGINT sent to dtrace by Bryan Cantrill

The execution of the specified command does not occur in the context of the firing probe – it occurs when the buffer containing the details of the system() action are processed at user-level. How and when this processing occurs depends on the buffering policy, described in Chapter 11, Buffers and Buffering. With the default buffering policy, the buffer processing rate is specified by the switchrate option. You can see the delay inherent in system() if you explicitly tune the switchrate higher than its one-second default, as shown in the following example:

#pragma D option quiet
#pragma D option destructive
#pragma D option switchrate=5sec

tick-1sec
/n++ < 5/
{
	printf("walltime  : %Y\n", walltimestamp);
	printf("date      : ");
	system("date");
	printf("\n");
}

tick-1sec
/n == 5/
{
	exit(0);
}

Running the above script results in output similar to the following example:


# dtrace -s ./time.d
 walltime  : 2004 Jul 20 13:26:30
date      : Tue Jul 20 13:26:35 CDT 2004

walltime  : 2004 Jul 20 13:26:31
date      : Tue Jul 20 13:26:35 CDT 2004

walltime  : 2004 Jul 20 13:26:32
date      : Tue Jul 20 13:26:35 CDT 2004

walltime  : 2004 Jul 20 13:26:33
date      : Tue Jul 20 13:26:35 CDT 2004

walltime  : 2004 Jul 20 13:26:34
date      : Tue Jul 20 13:26:35 CDT 2004

Notice that the walltime values differ, but the date values are identical. This result reflects the fact that the execution of the date(1) command occured only when the buffer was processed, not when the system() action was recorded.

Kernel Destructive Actions

Some destructive actions are destructive to the entire system. These actions must obviously be used extremely carefully, as they will affect every process on the system and any other system implicitly or explicitly depending upon the affected system's network services.

breakpoint()

void breakpoint(void)

The breakpoint() action induces a kernel breakpoint, causing the system to stop and transfer control to the kernel debugger. The kernel debugger will emit a string denoting the DTrace probe that triggered the action. For example, if one were to do the following:


# dtrace -w -n clock:entry'{breakpoint()}'
dtrace: allowing destructive actions
dtrace: description 'clock:entry' matched 1 probe

On Solaris running on SPARC, the following message might appear on the console:


dtrace: breakpoint action at probe fbt:genunix:clock:entry (ecb 30002765700)
Type  'go' to resume
ok

On Solaris running on x86, the following message might appear on the console:


dtrace: breakpoint action at probe fbt:genunix:clock:entry (ecb d2b97060)
stopped at      int20+0xb:      ret
kmdb[0]:

The address following the probe description is the address of the enabling control block (ECB) within DTrace. You can use this address to determine more details about the probe enabling that induced the breakpoint action.

A mistake with the breakpoint() action may cause it to be called far more often than intended. This behavior might in turn prevent you from even terminating the DTrace consumer that is triggering the breakpoint actions. In this situation, set the kernel integer variable dtrace_destructive_disallow to 1. This setting will disallow all destructive actions on the machine. Apply this setting only in this particular situation.

The exact method for setting dtrace_destructive_disallow will depend on the kernel debugger that you are using. If using the OpenBoot PROM on a SPARC system, use w!:


ok 1 dtrace_destructive_disallow w!
ok

Confirm that the variable has been set using w?:


ok dtrace_destructive_disallow w?
1
ok

Continue by typing go:


ok go

If using kmdb(1) on x86 or SPARC systems, use the 4–byte write modifier (W) with the / formatting dcmd:


kmdb[0]: dtrace_destructive_disallow/W 1
dtrace_destructive_disallow:    0x0             =       0x1
kmdb[0]:

Continue using :c:


kadb[0]: :c

To re-enable destructive actions after continuing, you will need to explicitly reset dtrace_destructive_disallow back to 0 using mdb(1):


# echo "dtrace_destructive_disallow/W 0" | mdb -kw
dtrace_destructive_disallow:    0x1             =       0x0
#

panic()

void panic(void)

The panic() action causes a kernel panic when triggered. This action should be used to force a system crash dump at a time of interest. You can use this action together with ring buffering and postmortem analysis to understand a problem. For more information, see Chapter 11, Buffers and Buffering and Chapter 37, Postmortem Tracing respectively. When the panic action is used, a panic message appears that denotes the probe causing the panic. For example:


  panic[cpu0]/thread=30001830b80: dtrace: panic action at probe
  syscall::mmap:entry (ecb 300000acfc8)

  000002a10050b840 dtrace:dtrace_probe+518 (fffe, 0, 1830f88, 1830f88,
    30002fb8040, 300000acfc8)
    %l0-3: 0000000000000000 00000300030e4d80 0000030003418000 00000300018c0800
    %l4-7: 000002a10050b980 0000000000000500 0000000000000000 0000000000000502
  000002a10050ba30 genunix:dtrace_systrace_syscall32+44 (0, 2000, 5,
    80000002, 3, 1898400)
    %l0-3: 00000300030de730 0000000002200008 00000000000000e0 000000000184d928
    %l4-7: 00000300030de000 0000000000000730 0000000000000073 0000000000000010

  syncing file systems... 2 done
  dumping to /dev/dsk/c0t0d0s1, offset 214827008, content: kernel
  100% done: 11837 pages dumped, compression ratio 4.66, dump
  succeeded
  rebooting...

syslogd(1M) will also emit a message upon reboot:


  Jun 10 16:56:31 machine1 savecore: [ID 570001 auth.error] reboot after panic:
  dtrace: panic action at probe syscall::mmap:entry (ecb 300000acfc8)

The message buffer of the crash dump also contains the probe and ECB responsible for the panic() action.

chill()

void chill(int nanoseconds)

The chill() action causes DTrace to spin for the specified number of nanoseconds. chill() is primarily useful for exploring problems that might be timing related. For example, you can use this action to open race condition windows, or to bring periodic events into or out of phase with one another. Because interrupts are disabled while in DTrace probe context, any use of chill() will induce interrupt latency, scheduling latency, dispatch latency. Therefore, chill() can cause unexpected systemic effects and it should not used indiscriminately. Because system activity relies on periodic interrupt handling, DTrace will refuse to execute the chill() action for more than 500 milliseconds out of each one-second interval on any given CPU. If the maximum chill() interval is exceeded, DTrace will report an illegal operation error, as shown in the following example:


# dtrace -w -n syscall::open:entry'{chill(500000001)}'
dtrace: allowing destructive actions
dtrace: description 'syscall::open:entry' matched 1 probe
dtrace: 57 errors
CPU     ID                    FUNCTION:NAME
dtrace: error on enabled probe ID 1 (ID 14: syscall::open:entry): \
  illegal operation in action #1

This limit is enforced even if the time is spread across multiple calls to chill(), or multiple DTrace consumers of a single probe. For example, the same error would be generated by the following command:


# dtrace -w -n syscall::open:entry'{chill(250000000); chill(250000001);}'