Speculative Tracing

About Speculative Tracing

In DTrace, the primary mechanism for filtering out uninteresting events is the predicate mechanism, which is described in more detail in D Program Structure. Predicates are useful when you know whether a probe event is of interest at the time that it fires. For example, if you are only interested in activity that is associated with a certain process or a certain file descriptor, you know when the probe fires if it is associated with the process or file descriptor of interest. Note that in other situations, you might not know whether a given probe event is of interest until some time after the probe fires.

Take the example of a system call that is occasionally failing with a common error code such as EIO or EINVAL. In this instance, you might want to examine the code path leading to the error condition. To capture the code path, you could enable every probe, but only if the failing call can be isolated in such a way that a meaningful predicate can be constructed. If the failures are sporadic or non-deterministic, you would be forced to trace all of the events that might be interesting, then later post-process the data to filter out the events that were not associated with the failing code path. In this case, even though the number of interesting events might be reasonably small, the number of events that must be traced is very large, making post-processing difficult.

In such situations, you can use speculative tracing facility to tentatively trace data at one or more probe locations. You can then decide to commit the data to the principal buffer at another probe location. The result is that your trace data only contains the output that is of interest; no post-processing is required and the DTrace overhead is minimized.

Speculation Interfaces

The following table describes DTrace speculation functions.

Table 7-1 DTrace Speculation Functions

Function	Args	Description
`speculation`	None	Returns an identifier for a new speculative buffer.
`speculate`	ID	Denotes that the remainder of the clause should be traced to the speculative buffer specified by ID.
`commit`	ID	Commits the speculative buffer that is associated with ID.
`discard`	ID	Discards the speculative buffer that is associated with ID.

Creating a Speculation

The speculation function allocates a speculative buffer and returns a speculation identifier. The speculation identifier should be used in subsequent calls to the speculate function. Speculative buffers are a finite resource. If no speculative buffer is available when speculation is called, an ID of zero is returned and a corresponding DTrace error counter is incremented. An ID of zero is always invalid, but it can be passed to the speculate, commit and discard functions. If a call to speculation fails, dtrace generates a message similar to the following:

dtrace: 2 failed speculations (no speculative buffer space available)

The number of speculative buffers defaults to one but can be optionally tuned higher. See Speculation Options and Tuning.

Using a Speculation

To use a speculation, an identifier that is returned from speculation must be passed to the speculate function in a clause prior to any data-recording actions. All subsequent data-recording actions in a clause containing a speculate are speculatively traced. The D compiler generates a compile-time error if a call to speculate follows data-recording actions in a D probe clause. Therefore, clauses might contain speculative tracing or non-speculative tracing requests, but not both.

Aggregating actions, destructive actions, and the exit action may never be speculative. Any attempt to take one of these actions in a clause containing a speculate results in a compile-time error. Also, a speculate may not follow a speculate. Only one speculation is permitted per clause. A clause that contains only a speculate speculatively traces the default action, which is defined to trace only the enabled probe ID. See Actions and Subroutines for a description of the default action.

Typically, you assign the result of speculation to a thread-local variable and then use that variable as a subsequent predicate to other probes, as well as an argument to speculate, as shown in the following example:

syscall::openat:entry
{
  self->spec = speculation();
}

syscall:::
/self->spec/
{
  speculate(self->spec);
  printf("this is speculative");
}

Committing a Speculation

You commit speculations by using the commit function. When a speculative buffer is committed, its data is copied into the principal buffer. If there is more data in the specified speculative buffer than there is available space in the principal buffer, no data is copied and the drop count for the buffer is incremented. If the buffer has been speculatively traced on more than one CPU, the speculative data on the committing CPU is copied immediately, while speculative data on other CPUs is copied some time after the commit. Thus, some time might elapse between a commit that begins on one CPU, while the data is being copied from speculative buffers to principal buffers on all CPUs. This length of time is guaranteed to be no longer than the time dictated by the cleaning rate. See Speculation Options and Tuning.

A committing speculative buffer is not made available to subsequent speculation calls until each per-CPU speculative buffer has been completely copied into its corresponding per-CPU principal buffer. Similarly, subsequent calls to speculate to the committing buffer are silently discarded, and subsequent calls to commit or discard silently fail. Finally, a clause containing a commit cannot contain a data recording action. However, a clause can contain multiple commit calls to commit disjoint buffers.

Discarding a Speculation

You discard speculations by using the discard function. When a speculative buffer is discarded, its contents are also discarded. If the speculation has only been active on the CPU calling discard, the buffer is immediately available for subsequent calls to speculation. If the speculation has been active on more than one CPU, the discarded buffer will be available for subsequent speculation some time after the call to discard. The length of time between a discard on one CPU and the buffer being made available for subsequent speculations is guaranteed to be no longer than the time that is dictated by the cleaning rate. If, at the time speculation is called, no buffer is available because all speculative buffers are currently being discarded or committed, dtrace generates a message similar to the following:

dtrace: 905 failed speculations (available buffer(s) still busy)

You can reduce the likelihood of all buffers being unavailable by tuning the number of speculation buffers or the cleaning rate. See Speculation Options and Tuning.

Example of a Speculation

One potential use for speculations is to highlight a particular code path. The following example shows the entire code path under the open() system call when the call fails. Type the following source code and save it in a file named specopen.d:

#!/usr/sbin/dtrace -Fs

syscall::open:entry
{
  /*
   * The call to speculation() creates a new speculation. If this fails,
   * dtrace will generate an error message indicating the reason for
   * the failed speculation(), but subsequent speculative tracing will be
   * silently discarded.
   */
  self->spec = speculation();
  speculate(self->spec);

  /*
   * Because this printf() follows the speculate(), it is being
   * speculatively traced; it will only appear in the data buffer if the
   * speculation is subsequently committed.
   */
  printf("%s", copyinstr(arg0));
}

syscall::open:return
/self->spec/
{
  /*
   * To balance the output with the -F option, we want to be sure that
   * every entry has a matching return. Because we speculated the
   * open entry above, we want to also speculate the open return.
   * This is also a convenient time to trace the errno value.
   */
  speculate(self->spec);
  trace(errno);
}

syscall::open:return
/self->spec && errno != 0/
{
  /*
   * If errno is non-zero, we want to commit the speculation.
   */
  commit(self->spec);
  self->spec = 0;
}

syscall::open:return
/self->spec && errno == 0/
{
  /*
   * If errno is not set, we discard the speculation.
   */
  discard(self->spec);
  self->spec = 0;
}

Running the previous script produces output similar to the following:

# ./specopen.d
dtrace: script ’./specopen.d’ matched 4 probes
CPU FUNCTION
  1  => open                                  /var/ld/ld.config
  1  <= open                                          2
  1  => open                                  /images/UnorderedList16.gif
  1  <= open                                          4
...

Speculation Options and Tuning

If a speculative buffer is full when a speculative tracing action is attempted, no data is stored in the buffer and a drop count is incremented. In this situation, dtrace generates a message similar to the following:

dtrace: 38 speculative drops

Speculative drops do not prevent the full speculative buffer from being copied into the principal buffer when it is committed. Similarly, speculative drops can occur even if drops were experienced on a speculative buffer that were ultimately discarded. Speculative drops can be reduced by increasing the speculative buffer size, which is tuned by using the specsize option. The specsize option can be specified with any size suffix. The resizing policy of this buffer is dictated by the bufresize option.

Speculative buffers might be unavailable when speculation is called. If buffers that have not yet been committed or discards exist, dtrace generates a message similar to the following:

dtrace: 1 failed speculation (no speculative buffer available)

You can reduce the likelihood of failed speculations of this nature by increasing the number of speculative buffers by specifying the nspec option. The value of nspec defaults to 1.

Also, speculation can fail if all speculative buffers are busy. In this case, an error message similar to the following is displayed:

dtrace: 1 failed speculation (available buffer(s) still busy)

This error message indicates that speculation was called after commit was called for a speculative buffer, but before that buffer was actually committed on all CPUs. You can reduce the likelihood of failed speculations of this nature by increasing the rate at which CPUs are cleaned by using the cleanrate option. The value of cleanrate defaults to 101.