Go to main content

Oracle® Solaris 11.4 DTrace (Dynamic Tracing) Guide

Exit Print View

Updated: September 2020
 
 

Processing of Aggregations in DTrace

In addition to the basic functions, the libdtrace library also provides finer-grained control for certain operations. For example, the dtrace_work() function is composed of calls to the dtrace_status(), dtrace_aggregate_snap(), and dtrace_consume() functions. A consumer can call these three functions separately. In some cases, all the three functions are not called. This section discusses some of the other useful functions provided by the API.

Aggregation Walkers

The libdtrace library provides control over how to handle the data from the aggregations. By default, dtrace outputs the elements of an aggregation sorted by the values stored in the aggregation. The API provides a number of options to print the values in aggregations. It also provides the ability to process the aggregations in alternate ways rather than printing the keys and values. The available functions to sort the output order are:

dtrace_aggregate_walk_keysorted() Function

This function walks aggregations in order sorted by key. For example, a five-second count of system calls prints by using this function and displays output similar to the following example.

brk                                                4
close                                              2
fchmod                                             1
getpid                                             1
gtime                                              5
ioctl                                            650
lstat                                              1
lwp_cond_wait                                      2
lwp_park                                          53
lwp_sigmask                                        2
mkdir                                              1
mmap                                               1
nanosleep                                          3
open                                               4
p_online                                         256
pollsys                                          178
portfs                                            40
pset                                               3
read                                              26
rename                                             1
schedctl                                           1
sysconfig                                          3
write                                             27
yield                                              1

Note -  This result also can be achieved by setting the undocumented aggsortkey option in a D script.

dtrace_aggregate_walk_valsorted() Function

This function walks aggregations in order sorted by value. This aggregation is the default behavior of dtrace and the dtrace_aggregate_print() function. This function displays output similar to the following example.

fchmod                                               1
getpid                                               1
lstat                                                1
mkdir                                                1
mmap                                                 1
rename                                               1
schedctl                                             1
yield                                                1
close                                                2
lwp_cond_wait                                        2
lwp_sigmask                                          2
nanosleep                                            3
pset                                                 3
sysconfig                                            3
brk                                                  4
open                                                 4
gtime                                                5
read                                                26
write                                               27
portfs                                              40
lwp_park                                            53
pollsys                                            178
p_online                                           256
ioctl                                              650

dtrace_aggregate_walk_keyrevsorted() Function

This function walks aggregations in reverse order sorted by key. This function displays output similar to the following example.

yield                                                1
write                                               27
sysconfig                                            3
schedctl                                             1
rename                                               1
read                                                26
pset                                                 3
portfs                                              40
pollsys                                            178
p_online                                           256
open                                                 4 
nanosleep                                            3
mmap                                                 1
mkdir                                                1
lwp_sigmask                                          2
lwp_park                                            53
lwp_cond_wait                                        2
lstat                                                1
ioctl                                              650
gtime                                                5
getpid                                               1
fchmod                                               1
close                                                2
brk                                                  4

Note -  This result also can be achieved by setting both the aggsortkey and aggsortrev options in a D program.

dtrace_aggregate_walk_valrevsorted() Function

This function walks aggregations in reverse order sorted by value. This function displays output similar to the following example.

ioctl                                              650
p_online                                           256
pollsys                                            178
lwp_park                                            53
portfs                                              40
write                                               27
read                                                26
gtime                                                5
open                                                 4
brk                                                  4
sysconfig                                            3
pset                                                 3
nanosleep                                            3
lwp_sigmask                                          2
lwp_cond_wait                                        2
close                                                2
yield                                                1
schedctl                                             1
rename                                               1
mmap                                                 1
mkdir                                                1
lstat                                                1
getpid                                               1
fchmod                                               1

Note -  This aggregation also can be achieved by setting the aggsortrev option in a D program.

This function appear to process data from separate aggregations. For example, the function prints the data from aggregation A before printing the data from aggregation B. The data from all aggregations are processed, but the function sorts the data by aggregation variable ID first before sorting by value in reverse order. The remaining functions change this order so that data from separate aggregations might intermingle.

dtrace_aggregate_walk_keyvarsorted() Function

This function walks aggregations sorted by key first and then by the aggregation's variable ID. For example, the following D program would print the minimum, average, and maximum latency for the specified system calls:

syscall::p*:entry
{
       self->ts = timestamp;
}
syscall::p*:return
/ self->ts /
{
       @c[probefunc] = min(timestamp - self->ts);
       @d[probefunc] = avg(timestamp - self->ts);
       @e[probefunc] = max(timestamp - self->ts);  
       self->ts = 0;
}

When the D program is processed by using dtrace_aggregate_walk_keyvarsorted(), output similar to the following example is displayed. This function sorts the system call names first, and then sorts the entries with the same system call name, and then by the aggregation's variable ID. These IDs are assigned in the order in which the aggregation first appears in the D program, so, for example, the ID for @c is less than the ID for @d, which is less than the ID for @e. The system calls are grouped on consecutive lines and the minimum, average, and maximum values print in that order.

p_online                                           968
p_online                                          1051
p_online                                          9685
pollsys                                           7161
pollsys                                      120515277
pollsys                                     4159836122
portfs                                            1668
portfs                                            2583
portfs                                            6948
pset                                              1165
pset                                              1911
pset                                              3369

Note -  The data is not sorted by value, though this example might give that impression.

dtrace_aggregate_walk_valvarsorted() Function

This function walks aggregations in order sorted by value first and then by the aggregation's variable ID. Because the data stored by the aggregating actions derive the values for the aggregation, DTrace only performs value comparisons between aggregations from the same aggregating action. For example, DTrace only compares min() values to min() values and max() values to max() values. DTrace also compares aggregations based on the number of records stored for the aggregation. This policy might yield slightly unexpected results. For example, when the data is processed by using the dtrace_aggregate_walk_valvarsorted() function, output is generated similar to the following example.

p_online                                            968
pset                                               1165
portfs                                             1668
pollsys                                            7161
pset                                               3369
portfs                                             6948
p_online                                           9685
pollsys                                      4159836122
p_online                                           1051
pset                                               1911
portfs                                             2583
pollsys                                       120515277

Note -  The values are grouped as min(), max(), and avg() values.

dtrace_aggregate_walk_keyvarrevsorted() Function

This function walks aggregations in reverse order sorted first by key and then by the aggregation's variable ID. This function displays output similar to the following example.

pset                                                3369
pset                                                1911
pset                                                1165
portfs                                              6948
portfs                                              2583
portfs                                              1668
pollsys                                       4159836122
pollsys                                        120515277
pollsys                                             7161
p_online                                            9685
p_online                                            1051
p_online                                             968

dtrace_aggregate_walk_valvarrevsorted() Function

This function walks aggregations in reverse order sorted by value and then by aggregation variable ID. This function displays an output similar to the following:

pollsys                                         120515277
portfs                                               2583
pset                                                 1911
p_online                                             1051
pollsys                                        4159836122
p_online                                             9685
portfs                                               6948
pset                                                 3369
pollsys                                              7161
portfs                                               1668
pset                                                 1165
p_online                                              968

These functions can be used in two ways. First, they can be passed as the third argument to the dtrace_aggregate_print() function to control how the data from aggregations prints. For example, you can replace the call to the dtrace_aggregate_print() function with the following code to print the data by using the dtrace_aggregate_walk_keysorted() function:

if (dtrace_aggregate_print(g_dtp, stdout,
    dtrace_aggregate_walk_keysorted) == -1)
        fatal("failed to print aggregation");

A second use is to call the aggregation functions directly and specify an alternate function to handle each aggregation record. This method is useful if you want custom output for the data or if you want to do something other than output the data.

The DTrace stddev() aggregating action calculates the standard deviation over a set of samples but because of the limitations of DTrace, the values are given as integers. In some cases, this level of precision might be insufficient. For example, consider the following D program:

BEGIN
{ 
          @c["foo"] = stddev(1); 
          @c["foo"] = stddev(2);
          @c["foo"] = stddev(3);
          @c["foo"] = stddev(4);
          @c["foo"] = stddev(5);
          @c["bar"] = stddev(6);
          @c["bar"] = stddev(8);
          @c["bar"] = stddev(10);
          @c["bar"] = stddev(12);
          @c["bar"] = stddev(14);
          @c["baz"] = stddev(17); 
          @c["baz"] = stddev(20);
          @c["baz"] = stddev(23);
          @c["baz"] = stddev(26);  
          @c["baz"] = stddev(29);
          exit(0);
}

The default dtrace output for this program would appear as follows:

foo                    1
bar                    2
baz                    4

The values in the output are rounded to the closest integer value. However, the actual values are 1.414, 2.828, and 4.243. To get better approximation of the correct values, you can write a custom DTrace consumer that uses aggregation to process the raw data.

Aggregation stores a small amount of data in the aggregation buffer, and this data is copied from the kernel periodically by the consumer. Some aggregations, such as count(), min(), max(), and sum(), store a single value. For example, count() stores a running count, and sum() stores a running sum. No further processing of this data is needed before the output is displayed. The avg() aggregation stores two values, a count of the number of data points and the sum of those data points. Before displaying the output, the final sum is divided by the final count to yield the average. Similarly, stddev() stores three values: the count, the sum, and the sum of the squares of the values. The standard deviation is computed from these values. For information, see DTrace Buffers and Buffering.

If you know how the stddev() aggregation is implemented, you can implement a function to extract the raw aggregation data and use it to calculate the standard deviation with more precision. Because the data is stored by the stddev() aggregation is a superset of the data stored for the avg() aggregation, you can also report the count of data points and their average.

Example 36  Using the walk() Function

This example shows the walk() function.

static int
walk(const dtrace_aggdata_t *data, void *arg)
{
        dtrace_aggdesc_t *aggdesc = data->dtada_desc;
        dtrace_recdesc_t *namerec, *datarec;
        char *name;
        uint64_t count, sum, sumsquares;  
        double avg, avgsquares, stddev;
        int i; 
        namerec = &aggdesc->dtagd_rec[1]; 
        name = data->dtada_data + namerec->dtrd_offset;
        datarec = &aggdesc->dtagd_rec[2];
        count = *((uint64_t *)(data->dtada_data + datarec->dtrd_offset));
        sum = *((uint64_t *)(data->dtada_data + datarec->dtrd_offset) + 1);
        sumsquares = *((uint64_t *)(data->dtada_data + datarec->dtrd_offset)+ 2); 
        avg = (double)sum / count;
        avgsquares = (double)sumsquares / count;
        stddev = sqrt (avgsquares - avg * avg); 
        printf("%10s %10lu %11.3f %11.3f\n", name, count, avg, stddev);
        return (DTRACE_AGGWALK_NEXT);
}

The walk() function is passed as an argument to the dtrace_aggregate_walk_keysorted() function, as shown in the following example:

printf("%10s %10s %11s %11s\n", "NAME", "COUNT", "AVG", "STDDEV");

if (dtrace_aggregate_walk_keysorted(g_dtp, walk, NULL) == -1)
        fatal("aggregation walk failed");

When the D program is run, the consumer generates the following output:

NAME           COUNT           AVG            STDDEV
bar                5        10.000             2.828
baz                5        23.000             4.243
foo                5         3.000             1.414

The walk() function uses dtrace_aggdata_t, dtrace_aggdesc_t, and dtrace_recdesc_t data structures. The function passes a pointer to the dtrace_aggdata_t structure, which describes the data for a single entry in an aggregation.

struct dtrace_aggdata {
      dtrace_hdl_t *dtada_handle;                         /* handle to DTrace library */
      dtrace_aggdesc_t *dtada_desc;                       /* aggregation description */ 
      dtrace_eprobedesc_t *dtada_edesc;                   /* enabled probe description */
      dtrace_probedesc_t *dtada_pdesc;                    /* probe description */ 
      caddr_t dtada_data;                                 /* pointer to raw data */
      uint64_t dtada_normal;                              /* the normal -- 1 for denorm */
      size_t dtada_size;                                  /* total size of the data */
      caddr_t dtada_delta;                                /* delta data, if available */
      caddr_t *dtada_percpu;                              /* per CPU data, if avail */
      caddr_t *dtada_percpu_delta;                        /* per CPU delta, if avail */
};

Example 36, Using the walk Function uses of two of the fields. It first pulls the aggregation description out of the structure by using the dtada_desc member. It later accesses the raw data stored in the aggregation by using the dtada_data member.

The aggregation description is contained in the dtrace_aggdesc_t data structure.

typedef struct dtrace_aggdesc {
        DTRACE_PTR(char, dtagd_name);           /* not filled in by kernel */
        dtrace_aggvarid_t dtagd_varid;          /* not filled in by kernel */
        int dtagd_flags;                        /* not filled in by kernel */
        dtrace_aggid_t dtagd_id;                /* aggregation ID */
        dtrace_epid_t dtagd_epid;               /* enabled probe ID */
        uint32_t dtagd_size;                    /* size in bytes */
        int dtagd_nrecs;                        /* number of records */
        uint32_t dtagd_pad;                     /* explicit padding */
        dtrace_recdesc_t dtagd_rec[1];          /* record descriptions */
} dtrace_aggdesc_t;

Example 36, Using the walk Function uses only the dtagd_rec array. This is an array of descriptions of the records for this entry. You can use these record descriptions to access the name and the data associated with this entry. Though only a single entry is statically-allocated for the dtagd_rec array, the array dynamically allocates to contain dtagd_nrecs entries.

The record descriptions are contained in the dtrace_recdesc_t data structure:

typedef struct dtrace_recdesc {
        dtrace_actkind_t dtrd_action;                 /* kind of action */
        uint32_t dtrd_size;                           /* size of record */
        uint32_t dtrd_offset;                         /* offset in ECB's data */
        uint16_t dtrd_alignment;                      /* required alignment */
        uint16_t dtrd_format;                         /* format, if any */
        uint64_t dtrd_arg;                            /* action argument */ 
        uint64_t dtrd_uarg;                           /* user argument */
} dtrace_recdesc_t;

Example 36, Using the walk Function uses only the dtrd_offset member. The consumer deals with the single stddev() action type and you do not require any of the other members. For example, if you want to include min() and max() aggregation data, you must examine the dtrd_action member to determine which data is contained in the current aggregation entry.

The definitions of these data structures are in the /usr/include/dtrace.h and /usr/include/sys/dtrace.h files. The following diagram shows the interaction between these data structures in this consumer.

Figure 12  Interactions Between the Data Structures

image:Graphic shows Consumer Data Structure

This figure shows that sumsquares is stored in two pieces. To avoid overflow, the stddev() aggregation stores and operates on the value as a 128-bit value. Note that the Example 36, Using the walk Function is only an approximation to how the stddev() aggregation works, because it assumes that the sum of the squares never exceeds the maximum 64-bit value.

Periodic Processing of Aggregation

In addition to different ways to process aggregations after the data collection is complete, you can also process aggregation data periodically as the data is being collected. You can use the dtrace_aggregate_snap() and dtrace_aggregate_clear() functions to process aggregation data periodically.

The dtrace_work() function transfers the existing aggregation data from the kernel, clears the in-kernel buffers, and adds the data to the copies of the aggregation data, which is maintained in user space. However, if a consumer wants to generate only aggregation data, it is more efficient to call the dtrace_aggregate_snap() function and skip the call to the dtrace_consume() function. You must also use the dtrace_status() function because the deadman timer fires if the aggregation is not performed periodically.

The dtrace_aggregate_clear() function clears the aggregate data associated with a DTrace handle. Although the function does not free the data structures holding this data, it zeros those parts of the data structures that hold aggregation data. This distinction is important because the entries in an aggregation remains with zeroed values. For example, suppose you modify the Example 35, Embedding DTrace in a Consumer by replacing the work() loop with the following code. You can also use dtrace_getopt() to determine the rate at which the dtrace_status() and dtrace_aggregate_snap() functions are called. You can also use aggrate.

if (dtrace_getopt(g_dtp, "statusrate", &statustime) == -1)
        fatal("failed to get 'statusrate'");

for (int i = 0; i < 10; i++) {
        usleep(statustime / 1000);

        if (dtrace_status(g_dtp) == -1)    
                fatal("dtrace_status()");

        if (dtrace_aggregate_snap(g_dtp) != 0)
                fatal("failed to add to aggregate");

        if (dtrace_aggregate_print(g_dtp, stdout, NULL) == -1)
                fatal("failed to print aggregation");

        dtrace_aggregate_clear(g_dtp);
}

If you run this consumer by using a D program that counts system calls, you can see the output for each iteration of the loop. The output for the final iteration of the loop would be similar to the following example.

brk                           0
close                         0
fchmod                        0
fcntl                         0
getdents64                    0
getpid                        0
lstat                         0
mkdir                         0
p_online                      0
pread                         0
pset                          0
rename                        0
stat64                        0
statvfs64                     0
sysconfig                     0
yield                         0
gtime                         1
lwp_cond_wait                 1
open                          1
writev                        1
clock_gettime                 2
nanosleep                     2
setitimer                     2
lwp_sigmask                  18
portfs                       22
read                         28
lwp_park                     30
write                        55
ioctl                        63
pollsys                     102

The zeroed entries in this output correspond to system calls that have been made since the D program is executing but are not called during the final loop. The dtrace_aggregate_clear() function zeros the data for the entries but does not remove the entries themselves. The dtrace_aggregate_print() function prints the value for every entry, including the entries with a zero value.

Per-CPU Data for Aggregations

DTrace offers the option to gather per-CPU data for aggregations. This capability can be useful when the combined aggregation data does not provide sufficient resolution. For example, the intrstat command uses per-CPU aggregation data to report statistics about CPUs that handle interrupts for each device. You can enable the collection of per-CPU aggregation data by setting the aggpercpu option.

When per-CPU aggregation data is collected, the dtada_percpu array in the dtrace_aggdata structure references the location to store the collected data. The dtada_data member of that structure references the location to store the total aggregation data. The following figure shows the per-CPU aggregation.

Figure 13  Using Per-CPU Data for Aggregations

image:Graphic shows the use of Per-CPU Data for Aggregations

You do not need an offset to index these buffers because the first two fields in the dtada_data buffer are not duplicated in the per-CPU buffers. You can add the following code to the end of the Example 36, Using the walk Function, to examine the data in the per-CPU buffers.

if (!data->dtada_percpu) 
        fatal("No per-cpu data\n");

for (i = 0; i < g_max_cpus; i++) {
        if (!g_present[i])
                continue;

        count = *((uint64_t *)(data->dtada_percpu[i]) + 0);
        sum = *((uint64_t *)(data->dtada_percpu[i]) + 1);
        sumsquares = *((uint64_t *)(data->dtada_percpu[i]) + 2);

        avg = (double)sum / count;
        avgsquares = (double)sumsquares / count;
        stddev = sqrt (avgsquares - avg * avg);

        if (count)
                printf("%11s %2d %10lu %17.3f %17.3f\n", "CPU", i,
                    count, avg, stddev);
        else
                printf("%11s %2d %10lu %17s %17s\n", "CPU", i,
                    count, "-", "-");
}
printf("\n");

The variable g_max_cpus is set to make a call to the sysconf() function. Because the value might be larger than the number of CPUs present, the entries in the g_present array are set to indicate whether a particular CPU is present. The function iterates over the set of possible CPU IDs. If a CPU is present, the function extracts and processes this data from the per-CPU buffer.

When this version of the consumer is run by using a D program to measure the standard deviation of system call latency, the output displays the overall values and the per-CPU breakdowns; as shown in the following example.

         NAME          COUNT          AVG         STDDEV
          brk             30     3811.167       3460.861
        CPU 0             16     3350.438       2969.729
        CPU 1             14     4337.714       3881.644
  
clock_gettime              3     1694.000        501.488   
        CPU 0              3     1694.000        501.488
        CPU 1              0            -              -

        close              5     5091.800        1613.886
        CPU 0              0            -               -
        CPU 1              5     5091.800        1613.886

       fchmod              1     3994.000           0.000
        CPU 0              0            -               -
        CPU 1              1     3994.000           0.000

        fcntl              3     1445.333         482.400
        CPU 0              0            -               -
        CPU 1              3     1445.333         482.400

         fsat              3     31520.000        6722.756  
        CPU 0              0             -               -
        CPU 1              3     31520.000        6722.756

      fstat64              3      2520.667         537.064 
        CPU 0              0             -               -
        CPU 1              3      2520.667         537.064

Joining Data From Multiple Aggregations

DTrace offers the option to process data from multiple aggregations with similar keys, for example, processing statistical data on system call latency, which is the minimum, maximum, average, and standard deviation of time spent in system calls. The printa() action enables you to print multiple aggregations, as shown in the following example.

# cat syscall-latency-stats.d
#!/usr/sbin/dtrace -qs

#pragma D option aggsortpos=2

syscall:::entry
{
        self->ts = timestamp;
}
syscall:::return
/ self->ts /
{
        this->lat = timestamp - self->ts;
        @m[probefunc] = min(this->lat);
        @M[probefunc] = max(this->lat);
        @a[probefunc] = avg(this->lat);
        self->ts = 0;
}
END
{
        printa("%-20s min: %12@d max:%12@d avg:%12@d\n", @m, @M, @a);
}#
# ./syscall-latency-stats.d
^C
[ ... ]
close             min:             19559 max:         38758 avg:         29158
schedctl          min:             36407 max:         36407 avg:         36407
write             min:              5156 max:        170056 avg:         87716
send              min:             97028 max:         97028 avg:         97028
connect           min:            169528 max:        169528 avg:        169528
lwp_cond_wait     min:             75977 max:    1001221741 avg:      47341037
read              min:              1253 max:    1000786548 avg:      55212840
lwp_park          min:              2275 max:    2000410123 avg:     521297430
pollsys           min:              2611 max:    5000232030 avg:     545102592
#

A custom consumer can use the dtrace_aggregation_walk_joined() function to process multiple aggregations similar to this DTrace script. This function has the following signature:

int dtrace_aggregate_walk_joined(dtrace_hdl_t *,
    dtrace_aggvarid_t *, int, dtrace_aggregate_walk_joined_f *, void *)

The arguments to the dtrace_aggregate_walk_joined() function are as follows:

  • DTrace handle

  • Array of dtrace_aggvarid_t – The identifiers for the aggregations to be joined

  • Number of elements in that array – The number of aggregations to be joined

  • Function to process each entry in the aggregation

  • Private argument to be passed

The dtrace_aggregate_walk_joined() function bundles aggregations data with the values containing the same key. The specified function is called on each bundle. The function syntax is as follows:

typedef int dtrace_aggregate_walk_joined_f(const dtrace_aggdata_t **,
    const int, void *);

The arguments to the dtrace_aggregate_walk_joined_f() function are:

  • Array of dtrace_aggdata_t pointers. The bundle of data with similar values to be processed.

  • Number of elements in the bundle

  • Private argument originally passed to the dtrace_aggregate_walk_joined() function

The dtrace_aggregate_walk_joined() function is similar to the walk() function that is passed to the other aggregation walkers except that the function passes an array of pointers to aggregation data rather than a single pointer. This function processes the aggregation data from each of the elements in this array.

The following example shows the dtrace_aggregate_walk_joined() function. This function processes the data to gather statistics on system call latency and then prints the data after the data is extracted.

In this example, the key for this bundle is stored in the zeroth element of data, but the values start at data[1]. The value in data[1] contains the same value as data[0].

static int
walk_joined(const dtrace_aggdata_t **data, const int naggs, void *arg)
{
        dtrace_aggdesc_t *aggdesc;
        dtrace_recdesc_t *keyrec, *datarec;
        char *syscall;
        int64_t stats[4], *avgdata;
        int i;

        aggdesc = data[0]->dtada_desc; 
        keyrec = &aggdesc->dtagd_rec[1];
        syscall = data[0]->dtada_data + keyrec->dtrd_offset;

        for (i = 1; i < naggs; i++) {
                aggdesc = data[i]->dtada_desc;
                datarec = &aggdesc->dtagd_rec[2];

                switch (datarec->dtrd_action) {
                case DTRACEAGG_MIN:
                case DTRACEAGG_MAX:
                        stats[i] = *((int64_t *)(data[i]->dtada_data +
                            datarec->dtrd_offset));
                        break;
                case DTRACEAGG_AVG:
                        avgdata = (int64_t *)(data[i]->dtada_data +
                            datarec->dtrd_offset);
                        stats[i] = avgdata[0] ? avgdata[1] / avgdata[0] : 0;
                        break;
                default:
                        fatal("Incorrect record type in walk_joined()\n");
                        break;
                }       
         }

         printf("%-20s min: %12lld max: %12lld avg: %12lld\n", syscall,
             stats[1], stats[2], stats[3]);

         return (DTRACE_AGGWALK_NEXT);
}

For the call to dtrace_aggregate_walk_joined(), you must know the IDs for the aggregations you wish to process. In the following example, the chewrec() function shows how to extract the aggregation IDs when processing the record for the printa() action.

static int 
chewrec(const dtrace_probedata_t *data, const dtrace_recdesc_t *rec,
    void *arg)
{
       dtrace_actkind_t act;
       dtrace_eprobedesc_t *epd = data->dtpda_edesc;
       dtrace_aggvarid_t aggvars[3];
       const void *buf;
       int i, nagv;

       if (rec == NULL)
               return (DTRACE_CONSUME_NEXT);
       act = rec->dtrd_action;
       buf = data->dtpda_data - rec->dtrd_offset;
 
       if (act == DTRACEACT_EXIT)
               return (DTRACE_CONSUME_NEXT);

       if (act == DTRACEACT_PRINTA) {
               for (nagv = 0, i = 0; i < 3; i++) {
                       const dtrace_recdesc_t *nrec = &rec[i];

                       if (nrec->dtrd_uarg != rec->dtrd_uarg)
                               break;

                       aggvars[nagv++] = *((dtrace_aggvarid_t *)
                       ((caddr_t)buf + nrec->dtrd_offset));
                }
                if (nagv == 3)
                       if (dtrace_aggregate_walk_joined(g_dtp, aggvars, nagv,
                            walk_joined, NULL) == -1)
                               fatal("dtrace_aggregate_walk_joined failed");
        }
        return (DTRACE_CONSUME_NEXT);
}