3. Scripting With the D Language
Using the copyin() and copyinstr() Subroutines
Eliminating dtrace Interference
User Function Boundary Tracing
Several DTrace providers implement probes that correspond to existing performance monitoring tools:
The vminfo provider implements probes that correspond to the vmstat(1M) tool
The sysinfo provider implements probes that correspond to the mpstat(1M) tool
The io provider implements probes that correspond to the iostat(1M) tool
The syscall provider implements probes that correspond to the truss(1) tool
You can use the DTrace facility to extract the same information that the bundled tools provide, but with greater flexibility. The DTrace facility provides arbitrary kernel information that is available at the time that the probes fire. The DTrace facility enables you to receive information such as process identification, thread identification, and stack traces.
The sysinfo provider makes available probes that correspond to the sys kernel statistics. These statistics provide the input for system monitoring utilities such as mpstat. The sysinfo provider probes fire immediately before the sys named kstat increments. The probes that are provided by the sysinfo provider are in the following list.
Probe that fires whenever a buffer is about to be asynchronously written out to a device.
Probe that fires whenever a buffer is physically read from a device. bread fires after the buffer has been requested from the device, but before blocking pending its completion.
Probe that fires whenever a buffer is about to be written out to a device, whether synchronously or asynchronously.
Probe that fires when the periodic system clock has made the determination that a CPU is idle. Note that this probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The cpu_t argument (arg2) indicates the CPU that has been deemed idle.
Probe that fires when the periodic system clock has made the determination that a CPU is executing in the kernel. This probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The cpu_t argument (arg2) indicates the CPU that has been deemed to be executing in the kernel.
Probe that fires when the periodic system clock has made the determination that a CPU is executing in user mode. This probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The cpu_t argument (arg2) indicates the CPU that has been deemed to be running in user-mode.
Probe that fires when the periodic system clock has made the determination that a CPU is otherwise idle, but some threads are waiting for I/O on the CPU. This probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The cpu_t argument (arg2) indicates the CPU that has been deemed waiting on I/O.
Probe that fires whenever a CPU enters the idle loop.
Probe that fires whenever an interrupt thread blocks.
Probe that fires whenever a running thread is forced to involuntarily give up the CPU.
Probe that fires whenever a buffer is logically read from a device.
Probe that fires whenever a buffer is logically written to a device
Probe that fires whenever a kernel module is loaded.
Probe that fires whenever a kernel module is unloaded.
Probe that fires whenever a msgsnd(2) or msgrcv(2) system call is made, but before the message queue operations have been performed.
Probe that fires whenever an attempt is made to acquire an owned adaptive lock. If this probe fires, one of the lockstat provider's adaptive-block or adaptive-spin probes also fires.
Probe that fires whenever a name lookup is attempted in the filesystem.
Probe that fires whenever a thread is created.
Probe that fires whenever a raw I/O read is about to be performed.
Probe that fires whenever a raw I/O write is about to be performed.
Probe that fires whenever a new process cannot be created because the system is out of process table entries.
Probe that fires whenever a CPU switches from executing one thread to executing another.
Probe that fires after each successful read, but before control is returned to the thread that is performing the read. A read can occur through the read(2), readv(2) or pread(2) system calls. arg0 contains the number of bytes that were successfully read.
Probe that fires whenever an attempt is made to read-lock a reader or writer when the lock is held by a writer or desired by a writer. If this probe fires, the lockstat provider's rw-block probe also fires.
Probe that fires whenever an attempt is made to write-lock a reader or writer lock when the lock is held by readers or by another writer. If this probe fires, the lockstat provider's rw-block probe also fires.
Probe that fires whenever a semop(2) system call is made, but before any semaphore operations have been performed.
Probe that fires whenever an exec(2) system call is made.
Probe that fires whenever a fork(2) system call is made.
Probe that fires whenever a read, readv, or pread system call is made.
Probe that fires whenever a vfork(2) system call is made.
Probe that fires whenever a write(2), writev(2), or pwrite(2) system call is made.
Probe that fires whenever a processor trap occurs. Note that some processors, in particular UltraSPARC variants, handle some lightweight traps through a mechanism that does not cause this probe to fire.
Probe that fires whenever a directory block is read from the UFS file system. See ufs(7FS) for details on UFS.
Probe that fires whenever an inode is retrieved. See ufs(7FS) for details on UFS.
Probe that fires after an in-core inode without any associated data pages has been made available for reuse. See ufs(7FS) for details on UFS.
Probe that fires after an in-core inode with associated data pages has been made available for reuse. This probe fires after the associated data pages have been flushed to disk. See ufs(7FS) for details on UFS.
Probe that fires when the periodic system clock has made the determination that a CPU is otherwise idle but some threads are waiting for I/O on the CPU. This probe fires in the context of the system clock and therefore fires on the CPU running the system clock. The cpu_t argument (arg2) indicates the CPU that is described as waiting for I/O. No semantic difference between wait_ticks_io and cpu_ticks_wait; wait_ticks_io exists solely for historical reasons.
Probe that fires after each successful write, but before control is returned to the thread performing the write. A write can occur through the write, writev, or pwrite system calls. arg0 contains the number of bytes that were successfully written.
Probe that fires whenever a cross-call is about to be made. A cross-call is the operating system's mechanism for one CPU to request immediate work of another CPU.
Example 4-1 Using the quantize Aggregation Function With the sysinfo Probes
The quantize aggregation function displays a power-of-two frequency distribution bar graph of its argument. The following example uses the quantize function to determine the size of the read calls that are performed by all processes on the system over a period of ten seconds. The arg0 argument for the sysinfo probes states the amount by which to increment the statistic. This value is 1 for most sysinfo probes. Two exceptions are the readch and writech probes. For these probes, the arg0 argument is set to the actual number of bytes that are read or are written, respectively.
# cat -n read.d 1 #!/usr/sbin/dtrace -s 2 sysinfo:::readch 3 { 4 @[execname] = quantize(arg0); 5 } 6 7 tick-10sec 8 { 9 exit(0); 10 } # dtrace -s read.d dtrace: script 'read.d' matched 5 probes CPU ID FUNCTION:NAME 0 36754 :tick-10sec bash value ---------- Distribution ---------- count 0 | 0 1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 13 2 | 0 file value ---------- Distribution ---------- count -1 | 0 0 | 2 1 | 0 2 | 0 4 | 6 8 | 0 16 | 0 32 | 6 64 | 6 128 |@@ 16 256 |@@@@ 30 512 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 199 1024 | 0 2048 | 0 4096 | 1 8192 | 1 16384 | 0 grep value ---------- Distribution ---------- count -1 | 0 0 |@@@@@@@@@@@@@@@@@@@ 99 1 | 0 2 | 0 4 | 0 8 | 0 16 | 0 32 | 0 64 | 0 128 | 1 256 |@@@@ 25 512 |@@@@ 23 1024 |@@@@ 24 2048 |@@@@ 22 4096 | 4 8192 | 3 16384 | 0
Example 4-2 Finding the Source of Cross-Calls
In this example, consider the following output form the mpstat(1M) command:
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 2189 0 1302 14 1 215 12 54 28 0 12995 13 14 0 73 1 3385 0 1137 218 104 195 13 58 33 0 14486 19 15 0 66 2 1918 0 1039 12 1 226 15 49 22 0 13251 13 12 0 75 3 2430 0 1284 220 113 201 10 50 26 0 13926 10 15 0 75
The values in the xcal and syscl columns are atypically high, reflecting a possible drain on system performance. The system is relatively idle and is not spending an unusual amount of time waiting for I/O. The numbers in the xcal column are scaled per second and are read from the xcalls field of the sys kstat. To see which executables are responsible for the cross-calls, enter the following dtrace command:
# dtrace -n 'xcalls {@[execname] = count()}' dtrace: description 'xcalls ' matched 3 probes ^C find 2 cut 2 snmpd 2 mpstat 22 sendmail 101 grep 123 bash 175 dtrace 435 sched 784 xargs 22308 file 89889 #
This output indicates that the bulk of the cross calls are originating from file(1) and xargs(1) processes. You can find these processes with the pgrep(1) and ptree(1) commands.
# pgrep xargs 15973 # ptree 15973 204 /usr/sbin/inetd -s 5650 in.telnetd 5653 -sh 5657 bash 15970 /bin/sh ./findtxt configuration 15971 cut -f1 -d: 15973 xargs file 16686 file /usr/bin/tbl /usr/bin/troff /usr/bin/ul /usr/bin/vgrind /usr/bin/catman
This output indicates that the xargs and file commands form part of a custom user shell script. To locate this script, you can perform the following commands:
# find / -name findtxt /usrs1/james/findtxt # cat /usrs1/james/findtxt #!/bin/sh find / -type f | xargs file | grep text | cut -f1 -d: > /tmp/findtxt$$ cat /tmp/findtxt$$ | xargs grep $1 rm /tmp/findtxt$$ #
This script runs many process concurrently. A large amount of interprocess communication is happening through pipes. The number of pipes makes the script resource intensive. The script attempts to find every text file on the system and then searches each file for a specific text.