Chapter 16 Debugging
This chapter describes how to debug a device driver. This includes how to set up a tip(1) connection to the test machine, how to prepare for a crash, and how to use debugging tools to test and code device drivers.
Machine Configuration
Setting Up a tip(1) Connection
A serial connection can be made between a test system (the machine executing the code to be debugged) and a host system using tip(1). This connection enables a window on the host system, called a tip window, to be used as the console of the test machine. See tip(1) for additional information.
Note -
A second machine is not required to debug a Solaris 7 device driver. It is only required for the use of tip(1).
Using a tip window confers the following advantages:
-
Interactions with the test system or kadb can be monitored. For example, the window can keep a log of the session for use if the driver crashes the test system.
-
The test machine can be accessed remotely by logging into a host machine (often called a tip host) and using tip(1) to connect to the test machine.
Setting Up the Host System
To set up the host system, do the following:
-
Connect the host system to the test machine using serial port A on both machines. This connection must be made with a null modem cable.
-
On the host system, make an entry in /etc/remote for the connection if it is not already there (see remote(4)).
The terminal entry must match the serial port being used. The Solaris 7 operating environment comes with the correct entry for serial port B, but a terminal entry must be added for serial port A:
debug:\ :dv=/dev/term/a:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:
Note -The baud rate must be set to 9600.
-
In a shell window on the host, run tip(1) and specify the name of the entry:
test% tip debugconnected
The shell window is now a tip window connected to the console of the test machine.
Caution - Do not use L1-A (for SPARC machines) or Control-ALT-D (for x86 machines) on the host machine to send a break to stop the test machine. This action actually stops the host machine. To send a break to the test machine, type ~# in the tip window. Commands such as this are recognized only if they are the first characters on a line, so press the Return key or Control-U first if there is no effect.
Setting Up the Test System for SPARC Platforms
A quick way to set up the test machine is to unplug the keyboard before turning the machine on. The machine then automatically uses serial port A as the console.
Another way to set up the test machine is to use boot PROM commands to make serial port A the console. On the test machine, at the boot PROM ok prompt, direct console I/O to the serial line. To make the test machine always come up with serial port A as the console, set the environment variables input-device and output-device.
ok setenv input-device ttyaok setenv output-device ttya
The eeprom command can also be used to make serial port A the console. As root user, execute the following commands to make the input-device and output-device parameters point to serial port A.
eeprom input-device=ttya eeprom output-device=ttya
Executing the eeprom commands causes the console to switch to serial port A during reboot.
Setting Up the Test System for x86 Platforms
On x86 platforms, use the eeprom command to make serial port A the console. The procedure for this is the same as for SPARC platform and is discussed above. Executing the eeprom commands causes the console to switch to serial port A (COM1) during reboot.
Note -
Unlike SPARC machines, where the tip connection maintains console control throughout the boot process, x86 machines don't transfer console control to the tip connection until an early stage in the boot process.
Preparing for Disasters
It is possible for a driver to render the system incapable of booting. To avoid system reinstallation in this event, some advance work must be done.
Critical System Files
A number of driver-related system files are difficult, if not impossible, to reconstruct. Files such as /etc/name_to_major,/etc/driver_aliases, /etc/driver_classes, and /etc/minor_perm can be corrupted if the driver crashes the system during installation (see add_drv(1M)).
To be safe, once the test machine is in the proper configuration, make a backup copy of the root file system. If you plan on modifying the /etc/system file, make a backup copy of the file before modifying it.
Booting an Alternate Kernel
A kernel other than /platform/`uname -i`/kernel/unix can be booted by specifying it as the boot file. In fact, backup copies of all the system drivers in /platform/* can be made and used if the original drivers fail (this is probably more useful if more than one driver is being debugged). For example:
# cp -r /platform/sun4c/kernel /platform/sun4c/kernel.orig
To boot the original system, boot kernel.orig/unix.
Note -
During testing, the new driver should be placed in /platform/sun4c/kernel (and not in /kernel or /usr/kernel) so that, the driver is not loaded if the system is booted out of kernel.orig. Alternatively, the module path can be changed by booting with the ask (-a) option.
ok boot kernel.orig/unix ... Rebooting with command: kernel.orig/unix Boot device: /sbus/esp@0,800000/sd@1,0 File and args:kernel.orig/unix SunOS Release 5.7 Version Generic [UNIX(R) System V Release 4.0] Copyright (c) 1983-1998, Sun Microsystems, Inc. ...
For more complete control, boot with the ask (-a) option; this allows alternate boot parameters to be specified (such as /dev/null or /etc/system.orig if that is the saved original system file that was copied earlier).
ok boot -a ... Rebooting with command: disk1 -a Boot device: /sbus/esp@0,800000/sd@1,0 File and args: -a Enter filename [/kernel/unix]: kernel.orig/unixEnter default directory for modules [/platform/SUNW,Sun_4_75/kernel.orig /kernel /usr/kernel]:<CR>SunOS Release 5.7 Version Generic [UNIX(R) System V Release 4.0] Copyright (c) 1983-1998, Sun Microsystems, Inc. Name of system file [etc/system]: etc/system.origroot filesystem type [ufs]: <CR>Enter physical name of root device [/sbus@1,f8000000/esp@0,800000/sd@1,0:a]: <CR>
Booting Off the Network or CD-ROM
If the system is attached to a network, the test machine can be added as a client of a server. If a problem occurs, the system can be booted off the network. The local disks can then be mounted and fixed. Alternatively, the system can be booted directly from the Solaris 7 CD-ROM.
Re-creating /devices and /dev
If the /devices or /dev directories are damaged--most likely to occur if the driver crashes during attach(9E))--they may be recreated by booting the system and running fsck(1M) to repair the damaged root file system. The root file system can then be mounted and /devices re-created by running drvconfig(1M) and specifying the /devices directory on the mounted disk. The /dev directory can be repaired by running devlinks(1M), disks(1M), tapes(1M), and ports(1M) on the dev directory of the mounted disk.
On SPARC, for example, if the damaged disk is /dev/dsk/c0t3d0s0, and an alternate boot disk is /dev/dsk/c0t1d0s0, do the following:
ok boot disk1 ... Rebooting with command: disk1 Boot device: /sbus/esp@0,800000/sd@1,0 File and args: SunOS Release 5.7 Version Generic [UNIX(R) System V Release 4.0] Copyright (c) 1983-1998, Sun Microsystems, Inc. ... # fsck /dev/dsk/c0t3d0s0** /dev/dsk/c0t3d0s0 ** Last Mounted on / ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 1478 files, 9922 used, 29261 free(141 frags, 3640 blocks, 0.4% fragmentation) # mount /dev/dsk/c0t3d0s0 /mnt# drvconfig -r /mnt/devices# devlinks -r /mnt# disks -r /mnt# tapes -r /mnt# ports -r /mnt
Caution - Fixing /devices and /dev may allow the system to boot, but other parts of the system may still be corrupted. This may only be a temporary fix to allow saving information (such as system core dumps) before reinstalling the system.
Booting Off a Backup Root Partition
One way to recover from disaster is to have another bootable root file system. Use format(1M) to make a partition the exact size of the original, then use dd(1M) to copy it. After making a copy, run fsck(1M) on the new file system to ensure its integrity.
Later, if the system cannot boot from the original root partition, boot the backup partition and use dd(1M) to copy the backup partition onto the original one. If the system will not boot but the root file system is undamaged (just the boot block or boot program was destroyed), boot off the backup partition with the ask (-a) option, then specify the original file system as the root file system.
Coding Hints
During development, debugging the driver should be a constant consideration. Because the driver is operating much closer to the hardware, and without the protection of the operating system, debugging kernel code is more difficult than debugging user-level code. For example, a stray pointer access can crash the entire system. This section provides some information that may be used to make the driver easier to debug.
Process Layout for Sun4m, Sun4c, Sun4d, and x86 Platforms
On the Sun4m, Sun4c, Sun4d, and x86 platforms, a Solaris 7 process looks like this:
On these platforms, the kernel and user programs share the same context. The system portion of a process's virtual address space occupies the high end of memory, and the user portion occupies the lower end of memory.
The Solaris 7 system defines a KERNELBASE for each platform. KERNELBASE can be used when debugging drivers to determine the address space. Addresses below KERNELBASE probably refer to user addresses, while addresses above refer to kernel addresses. Table 16-1 lists the values of KERNELBASE for each of these platforms.
Table 16-1 KERNELBASE Values|
Platform |
Value |
|---|---|
|
Sun4c |
OxF0000000 |
|
Sun4m |
OxF0000000 |
|
Sun4d |
OxE0000000 |
|
x86 |
OxE0000000 |
Process Layout for Sun4u Platforms
On Sun4u platforms, there are separate kernel and user contexts. The process layout looks like this:
On this platform, KERNELBASE is set to Ox10000000. USERLIMIT defines the upper limit of the user context; it is set to OxF0000000. Addresses below KERNELBASE probably refer to user addresses, while addresses above USERLIMIT probably refer to kernel addresses.
System Support
The system provides a number of routines that can aid in debugging; these are documented in Section 9 of the Solaris 2.7 Reference Manual.
cmn_err()()
cmn_err(9F) is used to print messages to the console from within the device driver. cmn_err(9F) provides additional format characters (such as %b) to print device register bits. See cmn_err(9F) and "Printing Messages"for more information.
Note -
Though printf() and uprintf() currently exist, they should not be used if the driver is to be Solaris DDI-compliant.
ASSERT()()
void ASSERT(int expression)
ASSERT(9F) can be used to ensure that a condition is true at some point in the program. It is a macro whose use varies depending upon whether the compilation symbol DEBUG is defined. If DEBUG is not defined, the macro expands to nothing and the expression is not evaluated. If DEBUG is defined, the expression is evaluated and, if the value is zero, a message is printed to the system console and the system panics.
For example, if a driver pointer should be non-NULL and is not, the following assertion could be used to check the code:
ASSERT(ptr != NULL);
If the driver is compiled with DEBUG defined and the assertion fails, a message is printed to the console and the system panics:
panic: assertion failed: ptr != NULL, file: driver.c, line: 56
Note -
Because ASSERT(9F) uses DEBUG, it is suggested that any conditional debugging code also be based on DEBUG, rather than on a driver symbol (such as MYDEBUG). Otherwise, for ASSERT(9F) to function properly, DEBUG must be defined whenever MYDEBUG is defined.
Assertions are an extremely valuable form of active documentation.
mutex_owned()()
int mutex_owned(kmutex_t *mp);
A significant portion of driver development involves properly handling multiple threads. Comments should always be used when a mutex is acquired; they are even more useful when an apparently necessary mutex is not acquired. To determine if a mutex is held by a thread, use mutex_owned(9F) within ASSERT(9F):.
void helper(void)
{
/* this routine should always be called with the mutex held */
ASSERT(mutex_owned(&xsp->mu));
...
}
Future releases of the Solaris operating system may only support the use of mutex_owned(9F) within ASSERT(9F) by not defining mutex_owned(9F) unless the preprocessor symbol DEBUG is defined.
Conditional Compilation and Variables
Debugging code can be placed in a driver by conditionally compiling code based on a preprocessor symbol such as DEBUG or by using a global variable. Conditional compilation has the advantage that unnecessary code can be removed in the production driver. Using a variable allows the amount of debugging output to be chosen at runtime. This can be accomplished by setting a debugging level at runtime with an I/O control or through a debugger. Commonly, these two methods are combined.
The following example relies on the compiler to remove unreachable code (the code following the always-false test of zero), and also provides a local variable that can be set in /etc/system or patched by a debugger.
#ifdef DEBUG comments on values of xxdebug and what they do static int xxdebug; #define dcmn_err if (xxdebug) cmn_err #else #define dcmn_err if (0) cmn_err #endif ... dcmn_err(CE_NOTE, "Error!\n");
This method handles the fact that cmn_err(9F) has a variable number of arguments. Another method relies on the macro having one argument, a parenthesized argument list for cmn_err(9F), which the macro removes. It also removes the reliance on the optimizer by expanding the macro to nothing if DEBUG is not defined.
#ifdef DEBUG comments on values of xxdebug and what they do static int xxdebug; #define dcmn_err(X) if (xxdebug) cmn_err X #else #define dcmn_err(X) /* nothing */ #endif ... /* Note:double parentheses are required when using dcmn_err. */ dcmn_err((CE_NOTE, "Error!"));
This can be extended in many ways, such as by having different messages from cmn_err(9F) depending on the value of xxdebug, but be careful not to obscure the code with too much debugging information.
Another common scheme is to write an xxlog()() function, which uses vsprintf(9F) or vcmn_err(9F) to handle variable argument lists.
volatile and _depends_on
volatile is a keyword that must be used when declaring any variable that will reference a device register. If this is not done, the optimizer might optimize important accesses away. This is important; neglecting to use volatile can result in bugs that are difficult to track down. See "volatile" for more information.
Note -
_depends_on must not be declared a static variable; if it is, the compiler might optimize it out of the device driver code.
Debugging Tools
This section describes some programs and files that can be used to debug the driver at runtime.
/etc/system
The /etc/system file is read once while the kernel is booting. It is used to set various kernel options. After this file is modified, the system must be rebooted for the changes to take effect. If a change in the file causes the system not to work, boot with the ask (-a) option and specify /dev/null as the system file.
Add the following set commands to the /etc/system file:
-
To set module variables, specify the module name and the variable:
set module_name:variable=value
-
To set the variable xxdebug in driver xx, use the following set command:
set xx:xxdebug=1
-
To set a kernel integer variable, omit the module name. Other assignments are also supported, such as bitwise OR'ing a value into an existing value:
set moddebug | 0x80000000
See system(4) for more information.
Note -
Most kernel variables are not guaranteed to be present in subsequent releases.
moddebug
moddebug is a kernel variable that controls the module loading process. The possible values are:
|
0x80000000 |
Prints messages to the console when loading or unloading modules. |
|
0x40000000 |
Gives more detailed error messages. |
|
0x20000000 |
Prints more detail when loading or unloading (such as including the address and size). |
|
0x00001000 |
No autounloading drivers: the system will not attempt to unload the device driver when the system resources become low. |
|
0x00000080 |
No autounloading streams: the system will not attempt to unload the streams module when the system resources become low. |
|
0x00000010 |
No autounloading of drivers of any type. Module loading is disabled. |
|
0x00000004 |
Not acceptable to page out symbol table. Prevents kernel from (possibly) paging out the driver's symbol table. kadb requires access to the symbol table to operate properly. |
|
0x00000001 |
If running with kadb, moddebug causes a breakpoint to be executed and a return to kadb immediately before each module's _init(9E) routine is called. Also generates additional debug messages when the module's _info and _fini routines are executed. |
modload and modunload
Because the kernel automatically loads needed modules, and unloads unused ones, these two commands are now obsolete. However, they can be used for debugging.
modload(1M) can be used to force a module into memory. The kernel might subsequently unload it, but modload(1M) may be used to ensure that the driver has no unresolved references when loaded.
modunload(1M) can be used to unload a module, given a module ID (which can be determined with modinfo(1M)). Unloading a module does not necessarily remove it from memory. To unload all unloadable modules and forcibly remove them from memory (so that they will be reloaded from the actual object file), use module ID zero:
# modunload -i 0
Note -
A future release might not include modload(1M) and modunload(1M).
Saving System Core Dumps
When the system panics, it writes the memory image to the dump device. The dump device by default is the primary swap device. The dump is a system core dump, similar to core dumps generated by applications. On rebooting after a panic, savecore(1M) checks the dump device for a crash dump. If one is found, it makes a copy of the kernel that was running (called unix.n) and dumps a core file (called vmcore.n) in the core image directory which by default is /var/crash/machine_name.
In Solaris 7, crash dump is enabled by default. The dumpadm(1M) command is used to configure system crash dumps. The dumpadm(1M) command should be used to verify that crash dumps are enabled and to determine the location of the directory where core files are saved. See dumpadm(1M) for more information.
Note -
dumpadm(1M) should be used to verify that the system is correctly configured for obtaining crash dumps.
When savecore(1M) runs, it makes a copy of the kernel that was running (called unix.n) and dumps a core file (called vmcore.n) in the specified directory, normally /var/crash/machine_name. There must be enough space in /var/crash to contain the core dump or it will be truncated. Because the file contains holes, it will appear larger than actual size; avoid copying it. adb(1) can then be used on the core dump and the saved kernel.
Note -
savecore(1M) can be prevented from filling the file system if there is a file called minfree in the directory in which the dump will be saved. This file contains a number of kilobytes to remain free after savecore(1M) has run. However, if not enough space is available, the core file is not saved.
adb and kadb
adb(1) can be used to debug applications or the kernel, though it cannot debug the kernel interactively (such as by setting breakpoints). To interactively debug the kernel, use kadb(1M). Both adb(1) and kadb(1M) share a common command set.
Starting adb
The command for starting adb(1) to debug a kernel core dump is:
% adb -k /var/crash/`hostname`/unix.n /var/crash/`hostname`/vmcore.n
Note -
For best results, use adb on the same architecture (such as Sun4m) that generated the core image.
To start adb(1) on a live system, type (as root):
# adb -k /dev/ksyms /dev/mem
/dev/ksyms is a special driver that provides an image of the kernel's symbol table to adb(1).
When < adb(1) responds with physmem xxx, it is ready for a command.
Note -
If the -p option of adb is used, an input prompt is displayed.
Starting kadb
The system must be booted under kadb(1M) before kadb(1M) can be used.
ok boot kadb ... Boot device: /sbus/esp@0,800000/sd@3,0 File and args: kadb kadb: kernel/unix Size: 191220+114284+12268 Bytes /platform/SUNW,Sun_4_75/kernel/unix loaded - 0x70000 bytes used SunOS Release 5.7 Version Generic [UNIX(R) System V Release 4.0] Copyright (c) 1983-1998, Sun Microsystems, Inc. ...
By default, kadb(1M) boots (and debugs) kernel/unix. It can be passed a file name as an argument to boot a different kernel, or -d can be passed to have kadb(1M) prompt for the kernel name. The -d flag also causes kadb(1M) to provide a prompt after it has loaded the kernel, so breakpoints can be set.
ok boot kadb -d ... Boot device: /sbus/esp@0,800000/sd@3,0 File and args: kadb -d kadb: kernel/unix kadb: kernel/unix Size: 191220+114284+12268 Bytes /platform/SUNW,Sun_4_75/kernel/unix loaded - 0x70000 bytes used kadb[0]:
Note -
Modules are dynamically loaded. Consequently, driver symbols are not generally available until the driver is loaded. To set breakpoints in modules that have not been loaded, use deferred breakpoints. For information on deferred breakpoints, see "Breakpoints".
At this point you can set breakpoints or continue with the :c command.
kadb(1M) passes any kernel flags to the booted kernel. For example, the flags -r, -s, and -a can be passed to kernel/unix with the command:
boot kadb -ras
Once the system is booted, sending a break passes control to kadb(1M). A break is generated with L1+A (on the console of SPARC machines) , or by Crtl+Alt+D (on the console of x86 machines) or ~# (if the console is connected through a tip window).
... The system is ready. test console login: ~stopped at 0xfbd01028: ta 0x7d kadb[0]:
The number in brackets is the CPU that kadb(1M) is currently executing on; the remaining CPUs are halted. The CPU number is zero on a uniprocessor.
Caution - Before rebooting or turning off the power, always halt the system cleanly (with init 0 or shutdown). Buffers may not be flushed otherwise. If the shutdown must occur from the boot PROM prompt, make sure to flush buffers with sync.
To return control to the operating system, use :c.
kadb[0]: :c test console login:
Exiting
To exit either adb(1M) or kadb(1M), use $q.
kadb[0]: $qType `go' to resume ok
On SPARC machines, kadb(1M) can be resumed by typing go at the ok prompt. On x86 machines, kadb(1M) cannot be resumed.
Caution - No other commands can be performed from the PROM if the system is to be resumed. PROM commands other than go may change system state that the Solaris 7 operating environment depends upon.
Staying at the kadb(1M) prompt for too long may cause the system to lose track of the time of day, and cause network connections to time out.
Commands
The general form of an adb(1M) or kadb(1M) command is:
[ address ] [ ,count ] command [;]
If address is omitted, the current location is used (`.' could also be used to represent the current location). The address can be a kernel symbol. If count is omitted, it defaults to 1.
Commands to adb consist of a verb followed by a modifier or list of modifiers. Verbs can be:
|
? |
Prints locations starting at address in the executable. |
|
/ |
Prints locations starting at address in the core file. |
|
= |
Prints the value of address itself. |
|
> |
Assigns a value to a variable or register. |
|
< |
Reads a value from a variable or register. |
|
RETURN |
Repeats the previous command with a count of 1. Increments `.' (the current location). |
With ?, /, and =, output format specifiers can be used. Lowercase letters normally print 2 bytes, uppercase letters print 4 bytes:
|
o, O |
2-, 4-byte octal |
|
g |
8-byte octal |
|
G |
8-byte unsigned octal |
|
d,D |
2-, 4-byte decimal |
|
e |
8-byte decimal |
|
E |
8-byte unsigned decimal |
|
x,X |
2-, 4-byte hexadecimal |
|
J |
8-byte hexadecimal |
|
K |
4-byte hexadecimal for 32-bit programs, 8-byte hexadecimal for 64-bit programs |
|
u,U |
2-, 4-byte unsigned decimal |
|
f,F |
4-, 8-byte floating point |
|
c |
Prints the addressed character. |
|
C |
Prints the addressed character using ^ escape notation. |
|
s |
Prints the addressed string. |
|
S |
Prints the addressed string using ^ escape notation. |
|
i |
Prints as machine instructions (disassemble). |
|
a |
Prints the value of `.' in symbolic form. |
|
w,W |
2-, 4-byte write |
Note -
Understand exactly what sizes the objects are, and what effects changing them might have, before making any changes.
For example, to set a bit in the moddebug variable when debugging the driver, first examine the value of moddebug, then set it to the desired bit.
kadb[0]: moddebug/Xmoddebug: moddebug: 1000 kadb[0]: moddebug/W 0x80001000moddebug: 0x1000 = 0x80001000
Routines can be disassembled with the `i' command. This is useful when tracing crashes, since the only information may be the program counter at the time of the crash. For example, to print the first four instructions of the kmem_alloc function:
kadb[0]: kmem_alloc,4?i kmem_alloc: kmem_alloc: save %sp, -0x60, %sp sub %i0, 0x1, %l6 sra %l6, 0x3, %i5 tst %i5
To show the addresses also, specify symbolic notation with the `a' command.
kadb[0]: kmem_alloc,4?ai kmem_alloc: kmem_alloc: save %sp, -0x60, %sp kmem_alloc+4: sub %i0, 0x1, %l6 kmem_alloc+8: sra %l6, 0x3, %i5 kmem_alloc+0xc: tst %i5
Register Identifiers
Machine or kadb(1M) internal registers are identified with the `<` command, followed by the register of interest. On SPARC machines, the following register names are recognized:
|
. |
dot, the current location |
|
i0-7 |
Input registers to current function |
|
o0-7 |
Output registers for current function |
|
l0-7 |
Local registers |
|
g0-7 |
Global registers |
|
psr |
Processor Status Register |
|
tbr |
Trap Base Register |
|
wim |
Window Invalid Mask |
For more information on how these registers are normally used, see the System V Application Binary Interface, SPARC Processor Supplement.
On x86 machines, the following register names are recognized:
|
ebp |
Stack frame base register |
|
esp |
Stack pointer |
|
kesp |
Kernel stack pointer |
|
cs |
Code segment |
|
ds |
Data segment |
|
ss |
Stack segment |
|
eip |
Program pointer or instruction pointer |
|
efl |
Status flags register |
Note -
The remaining examples in this chapter are for use on SPARC machines only. For specific register information relating to x86 machines, see the System V Application Binary Interface, x86 Processor Supplement.
The following command displays the PSR as a 4-byte hexadecimal value:
kadb[0]: <psr=X
400cc3
Display and Control Commands
The following commands display and control the status of adb(1)/ kadb(1M):
|
$b |
Display all breakpoints |
|
$c |
Display stack trace |
|
$d |
Change default radix to value of dot |
|
$q |
Quit |
|
$r |
Display registers |
|
$M |
Display built-in macros |
`$c' is useful with crash dumps: it shows the call trace and arguments at the time of the crash. It is also useful in kadb(1M) when a breakpoint is reached, but is usually not useful if kadb(1M) is entered at a random time. The number of arguments to print can be passed following the `$c' (`$c 2' for two arguments).
Breakpoints
In kadb(1M), breakpoints can be set that will automatically drop back into kadb when reached. The standard form of a breakpoint command is:
[module_name#] addr [, count]:b [command]
addr is the address at which the program will be stopped and the debugger will receive control, count is the number of times that the breakpoint address occurs before stopping, and command is almost any adb(1) command.
The optional module_name specifies deferred breakpoints that are set when the module is loaded. module_name identifies a particular module that contains addr. If the module has been loaded, kadb will try to set a regular breakpoint; if the module is not loaded, kadb will set a deferred breakpoint. When the module is loaded, kadb will try to resolve the location of the breakpoint and convert the breakpoint to a regular breakpoint.
Other breakpoint commands are:
|
:c |
Continue execution |
|
:d |
Delete breakpoint |
|
:s |
Single step |
|
:e |
Single step, but step over function calls |
|
:u |
Stop after return to caller of current function |
|
:z |
Delete all breakpoints |
The following example sets a breakpoint in scsi_transport(9F), a commonly used routine. Upon reaching the breakpoint, `$c' is used to get a stack trace. The top of stack is the first function printed. Note that kadb(1M) does not know how many arguments were passed to the function; it always prints six.
kadb[0]: scsi_transport:bkadb[0]: :c test console login: root Password: breakpoint at: scsi_transport: save %sp, -0x60, %sp kadb[0]: $c scsi_transport(0xf5ad0d78,0x0,0x5,0xf5ad0e1c,0xf5ad0ec8,0x0) sdstrategy(0xf5d4b8a8,0xf5ad0e1c,0x0,0x2c1d0,0xf5ad0d78,0x0) + 3c0 bdev_strategy(0xf5d4b8a8,0xf5cd3dcc,0x8,0x40,0xf0319e60,0x1000) + d0 ufs_getpage_miss(0x0,0xf6133840,0x0,0x0,0xfbecb8e8,0x1) + 23c ufs_getpage(0x0,0x1000,0x0,0x0,0x0,0x1) + 640 segmap_fault(0xf5b34000,0xf5d76f98,0x0,0x1000,0x0,0x1) + 120 segmap_getmapflt(0xf0ba4000,0xf0ba4000,0x10000,0x0,0x6c267,0x0) + 4b0 rdip(0xf61337b8,0x0,0xf6133840,0xf5a74938,0x0,0xf5cda000) + 328 ufs_read(0xf61338a4,0xfbecbaf0,0x0,0xf5a74938,0xf5ca7e80,0xf61337b8) + 118 read(0x4) + 274 syscall_trap(?) + 18c Syssize(0x4) + d3c0 Syssize(0x26ad4,0x27400,0x0,0xef66855c,0xef661f58,0x11e78) + d2bc Syssize(0x27b88,0x27400,0xeffffe74,0x26968,0x27f94,0x27a84) + bc28 kadb[0]: :s stopped at: scsi_transport+4: mov %i0, %i1 kadb[0]: $b breakpoints count bkpt type len command 1 scsi_transport :b instr 4 kadb[0]: scsi_transport:d kadb[0]: :c
Conditional Breakpoints
This is the general syntax of conditional breakpoints:
address,count:b command
In this example, address is the address at which to set the breakpoint. count is the number of times the breakpoint should be ignored (note that 0 means break only when the command returns zero). command is the adb(1) command to execute.
Breakpoints can also be set to occur only if a certain condition is met. By providing a command, the breakpoint will be taken only if the count is reached or the command returns zero. For example, a breakpoint that occurs only on certain I/O controls could be set in the driver's ioctl(9E) routine. Here is an example of breaking only in the sdioctl() routine if the DKIOGVTOC (get volume table of contents) I/O control occurs.
kadb[0]: sdioctl+4,0:b <i1-0x40Bkadb[0]: $bbreakpoints count bkpt command 0 sdioctl+4 <i1-0x40B kadb[0]: :c
Adding four to sdioctl skips to the second instruction in the routine, bypassing the save instruction that establishes the stack. The `<i1' refers to the first input register, which is the second parameter to the routine (the cmd argument of ioctl(9E)). The count of zero is impossible to reach, so it stops only when the command returns zero, which is when `i1 - 0x40B' is true. This means i1 contains 0x40B (the value of the ioctl(9E) command, determined by examining the ioctl definition).
To force the breakpoint to be reached, the prtvtoc(1M) command is used. It is known to issue this I/O control:
# prtvtoc /dev/rdsk/c0t3d0s0breakpoint sdioctl+4: st %i5, [%fp + 0x58] breakpoint sdioctl+4: mov %i0, %o0 kadb[0]: $csdioctl(0x800018,0x40b,0xeffffc04,0x5,0xff34dc68,0xf03f0c00) + 4 ioctl(0xf03f0c90,0xf03f0c00,0x3,0x18,0xff445f5c,0xff7a09e0) + 270 syscall_ap(0x3) + 6c syscall_trap(?) + 150 Syssize(0x3) + 20458 Syssize(0x3,0x22f70,0xeffffc04,0x1,0x4,0xefffff6c) + fc14 Syssize(0xefffff6c,0xffffffff,0x1,0x1,0x3,0x22f70) + f5b4 Syssize(0x2,0xeffffec4,0xeffffed0,0x22c00,0x0,0x1) + ebe4
kadb(1M) cannot always determine where the bottom of the stack is. In the previous example, the call to Syssize is not part of the stack.
Macros
adb(1) and kadb(1M) support macros. adb(1) macros are in /usr/lib/adb and /usr/platform/`uname -i`/lib/adb for 32-bit programs and in /usr/lib/adb/sparcv9 and /usr/platform/`uname -i`/lib/adb/sparcv9 for 64-bit programs. kadb(1M) macros are builtin and can be displayed with $M. Most of the existing macros are for private kernel structures. New macros for adb can be created with adbgen(1M).
Macros are used in the form:
[ address ] $<macroname
threadlist is a useful macro that displays the stacks of all the threads in the system. Because this macro does not take an address and can generate excessive output, be ready to use Control-S and Control-Q to start or stop if necessary--this is another good reason to use a tip window. Control-C can be used to abort the listing.
kadb[0]: $<threadlist
============== thread_id f0244020
p0:
p0: process args= sched
t0:
t0: lwp proc wchan
f02a3e78 f02a8078 0
t0+0x34: sp pc
f0243af0 sched+0x4e0
?(?) + 0
main(0x0,0xf02b7e20,0xf02a800c,0x0,0x6e,0x0)
afsrbuf(?) + 1b8
exitto(0xf0040000,0xf02a0f58,0x3c,0xf02d3c40,0xfbd7d668,0x0)
============== thread_id fbe01ea0
p0:
p0: process args= sched
0xfbe01ea0: lwp proc wchan
0 f02a8078 0
0xfbe01ed4: sp pc
fbe1fe68 debug_enter+0xb0
?(?) + 0
abort_sequence_enter(0x0,0x740000,0xc4,0x0,0x0,0x40000e7)
zsa_xsint(0xf5dd9000,0x80,0xf5dd906c,0x44,0xff0113,0x0) + 244
zs_high_intr(0xf5dd9000) + 204
_level1(0x0) + 13bc
idle(0xf02a0fac,0x0,0xf02a8078,0xf02a8078,0xf004684c,0x4400ae1) + 50
============== thread_id fbe22ea0
p0:
p0: process args= sched
0xfbe22ea0: lwp proc wchan
0 f02a8078 f5caf024
0xfbe22ed4: sp pc
fbe22d80 cv_wait+0x64
?(?) + 0
cv_wait(0xf5caf024,0xf5caf010,0xffdf1175,0x6ea7e,0xf5caf030,0x40000000)
callout_thread(0xf5caf010,0xf5caf024,0xf02a8078,0xf02a8078,0x1000,0x4400ae5)+38
...
Another useful macro is thread. Given a thread ID, this macro prints the corresponding thread structure. This can be used to look at a certain thread found with the threadlist macro, to look at the owner of a mutex, or to look at the current thread.
kadb[0]: <g7$<thread
0xfbe01ea0:0xfbe01ea0: link stk startpc bound
0 fbe01e40 f0076d40 f02a0fac
0xfbe01eb0: affcnt bind_cpu flag procflag
1 -1 8 0
0xfbe01eb8: schedflag preempt preempt_lk state
3 1 0 4
0xfbe01ec0: pri epri pc sp
-1 0 f005bdf8 fbe1fe68
0xfbe01ecc: wchan0 wchan sobj_ops cid
0 0 0 0
0xfbe01edc: clfuncs cldata ctx lofault
f02aeed8 0 0 0
0xfbe01eec: onfault nofault swap lock
0 0 fbe01000 ff
0xfbe01efa: delay_cv cpu intr did
0 f02a0fac 0 1
0xfbe01f08: tnf_tpdp tid alarmid
f5a75c80 0 0 realitimer
0xfbe01f14: interval.sec interval.usec value.sec value.usec
0 0 0 0
0xfbe01f24: itimerid sigqueue sig
0 0 0 0
0xfbe01f34: hold forw back
0 0 0 0
0xfbe01f44: lwp procp next prev
0 f02a8078 fbe04ea0 f0244020
0xfbe01f58: trace why what dslot pollstate
0 0 0 0 0
0xfbe01f68: cred lbolt sysnum pctcpu
0 0 0 0
0xfbe01f88: lockp oldspl pre_sys disp_queue
f02a1014 e0 0 f02a0ff0
...
Note -
No type information is kept in the kernel, so using a macro on an inappropriate object results in garbage output.
Macros do not necessarily output all the fields of the structures, nor is the output necessarily in the order given in the structure definition. Occasionally, memory may need to be dumped for certain structures and then matched with the structure definition in the kernel header files.
Caution - Drivers should never reference header files and structures not listed in Section 9S of the Solaris 2.7 Reference Manual. However, examining non-DDI-compliant structures (such as thread structures) can be useful in debugging drivers.
Example: adb on a Core Dump
During the development of the example ramdisk driver, the system crashes with a data fault when running mkfs(1M).
test# mkfs -F ufs -o nsect=8,ntrack=8,free=5 /devices/pseudo/ramdisk:0,raw 1024BAD TRAP mkfs: Data fault kernel read fault at addr=0x4, pme=0x0 Sync Error Reg 80<INVALID> pid=280, pc=0xff2f88b0, sp=0xf01fe750, psr=0xc0, context=2 g1-g7: ffffff98, 8000000, ffffff80, 0, f01fe9d8, 1, ff1d4900 Begin traceback... sp = f01fe750 Called from f0098050,fp=f01fe7b8,args=1180000 f01fe878 ff1ed280 ff1ed280 2 ff2f8884 Called from f0097d94,fp=f01fe818,args=ff24fd40 f01fe878 f01fe918 0 0 ff2c9504 Called from f0024e8c,fp=f01fe8b0,args=f01fee90 f01fe918 2 f01fe8a4 f01fee90 3241c Called from f0005a28,fp=f01fe930,args=f00c1c54 f01fe98c 1 f00b9d58 0 3 Called from 15c9c,fp=effffca0,args=5 3241c 200 0 0 7fe00 End traceback... panic: Data fault
When the system comes up, it saves the kernel and the core file, which can then be examined with adb(1):
# cd /var/crash/test# lsbounds unix.0 vmcore.0 # adb -k unix.0 vmcore.0physmem 1ece
The first step is to examine the stack to determine where the system was when it crashed:
$ccomplete_panic(0x0,0x1,0xf00b6c00,0x7d0,0xf00b6c00,0xe3) + 114 do_panic(0xf00be7ac,0xf0269750,0x4,0xb,0xb,0xf00b6c00) + 1c die(0x9,0xf0269704,0x4,0x80,0x1,0xf00be7ac) + 5c trap(0x9,0xf0269704,0x4,0x80,0x1,0xf02699d8) + 6b4
This stack trace is not helpful initially, as the ramdisk routines are not on the stack trace. However, there is a useful bit of information: the call to trap(). The first argument to trap() is the trap type. The second argument to trap() is a pointer to a regs structure containing the state of the registers at the time of the trap. See The SPARC Architecture Manual, Version 9 for more information.
0xf0269704$<regs0xf0269704: psr pc npc c0 ff2dd8b0 ff2dd8b4 0xf0269710: y g1 g2 g3 e0000000 ffffff98 8000000 ffffff80 0xf0269720: g4 g5 g6 g7 0 f02699d8 1 ff22c800 0xf0269730: o0 o1 o2 o3 f02697a0 ff080000 19000 ef709000 0xf0269740: o4 o5 o6 o7 8000 0 f0269750 7fffffff
Note that the program counter (pc) in the previous example was ff2dd8b0 when the trap occurred. The next step is to determine which routine it is in.
ff2dd8b0/ird_write+0x2c: ld [%o2 + 0x4], %o3
The pc corresponds to rd_write(), which is a routine in the ramdisk driver. The bug is in the ramdisk write routine, and occurs during an load (ld) instruction. This load instruction is dereferencing the value of o2+4, so the next step is to determine the value of o2.
Note -
Using the $r command to examine the registers is inappropriate because the registers have been reused in the trap routine. Instead, examine the value of o2 from the regs structure.
o2 has the value 19000 in the regs structure. Valid kernel addresses are constrained to be above KERNELBASE by the ABI, so this is probably a user address. The ramdisk does not deal with user addresses; consequently, the ramdisk write routine should not dereference an address below KERNELBASE.
To match the assembly language with the C code, the routine is disassembled up to the problem instruction. Each instruction is 4 bytes in size, so 2c/4 or 0xb additional instructions should be displayed:
rd_write,c/ird_write: rd_write: sethi %hi(0xfffffc00), %g1 add %g1, 0x398, %g1 ! ffffff98 save %sp, %g1, %sp st %i0, [%fp + 0x44] st %i1, [%fp + 0x48] st %i2, [%fp + 0x4c] ld [%fp + 0x44], %o0 call getminor nop st %o0, [%fp - 0x4] ld [%fp - 0x8], %o2 ld [%o2 + 0x4], %o3
The crash occurs a few instructions after a call to getminor(9F). If the ramdisk.c file is examined, the following lines stand out in rd_write:
int instance = getminor(dev); rd_devstate_t *rsp; if (uiop->uio_offset >= rsp->ramsize) return (EINVAL);
Notice that rsp is never initialized. This is the problem. It is fixed by including the correct call to ddi_get_soft_state(9F) (as the ramdisk driver uses the soft state routines to do state management):
int instance = getminor(dev); rd_devstate_t *rsp = ddi_get_soft_state(rd_state, instance); if (uiop->uio_offset >= rsp->ramsize) return (EINVAL);
Note -
Many data fault panics are the result of bad pointer references.
Example: kadb on a Deadlocked Thread
The next problem is that the system does not panic, but the mkfs(1M) command hangs and cannot be aborted. Though a core dump can be forced--by sending a break and then using sync from the OBP or using `g 0' from SunMon--in this case kadb(1M) will be used. After logging in remotely and using ps (which indicated that only the mkfs(1M) process was hung, not the entire system) the system is shut down and booted using kadb(1M) .
ok boot kadb -d Boot device: /sbus/esp@0,800000/sd@3,0 File and args: kadb -d kadb:kernel/unix Size: 673348+182896+46008 bytes /platform/SUNW,Sun_4_75/kernel/unix ... kadb[0]::cSunOS Release 5.7 Version Generic [UNIX(R) System V Release 4.0] Copyright (c) 1983-1998, Sun Microsystems, Inc. ...
After the rest of the kernel has loaded, moddebug is patched to see if loading is the problem. Because it got as far as rd_write() before, loading is probably not the problem, but it will be checked anyway.
# ~stopped at 0xfbd01028: ta 0x7d kadb[0]: moddebug/Xmoddebug: moddebug: 0 kadb[0]: moddebug/W 0x80000000moddebug: 0x0 = 0x80000000 kadb[0]: :c
modload(1M) is used to load the driver, to separate module loading from the real access:
# modload /home/driver/drv/ramdisk
It loads without errors, so loading is not the problem. The condition is recreated with mkfs(1M).
# mkfs -F ufs -o nsect=8,ntrack=8,free=5 /devices/pseudo/ramdisk@0:c,raw 1024ramdisk0: misusing 524288 bytes of memory
mkfs(1M) hangs. At this point, kadb(1M) is entered and the stack examined:
~stopped at Syslimit+0x1028: ta 0x7d kadb[0]: $c Syslimit() + 1028 debug_enter(0x0,0x740000,0xc4,0x0,0x0,0x40000e7) + c0 zsa_xsint(0xf5dd9000,0x80,0xf5dd906c,0x44,0xff0113,0x0) + 244 zs_high_intr(0xf5dd9000) + 204 _level1(0x1) + 13bc idle(0xf02a0fac,0x0,0xf02a8078,0xf02a8078,0xf004684c,0x4400ae1) + 50
In the previous example, the presence of idle on the current thread stack indicates that this thread is not the cause of the deadlock. To determine the deadlocked thread, the entire thread list is checked:
kadb[0]: $<threadlist
============== thread_id f0244020
p0:
p0: process args= sched
t0:
t0: lwp proc wchan
f02a3e78 f02a8078 0
t0+0x34: sp pc
f0243af0 sched+0x4e0
?(?) + 0
main(0x0,0xf02b7e20,0xf02a800c,0x0,0x6e,0x0)
afsrbuf(?) + 1b8
exitto(0xf0040000,0xf02a0f58,0x3c,0xf02d3c40,0xfbd7d668,0x0)
============== thread_id fbe01ea0
p0:
p0: process args= sched
0xfbe01ea0: lwp proc wchan
0 f02a8078 0
0xfbe01ed4: sp pc
fbe1fe68 debug_enter+0xb0
?(?) + 0
abort_sequence_enter(0x0,0x740000,0xc4,0x0,0x0,0x40000e7)
zsa_xsint(0xf5dd9000,0x80,0xf5dd906c,0x44,0xff0113,0x0) + 244
zs_high_intr(0xf5dd9000) + 204
_level1(0xf02d40ec) + 13bc
idle(0xf02a0fac,0x0,0xf02a8078,0xf02a8078,0xf004684c,0x4400ae1) + 50
...
============== thread_id f6350a80
0xf6285518: process args= mkfs -o nsect=8,ntrack=8,free=5 /devices/pseudo/
ramdisk@0:c,raw 1024
0xf6350a80: lwp proc wchan
f634baa0 f6285518 f5ef6fd0
0xf6350ab4: sp pc
fbedc9e0 biowait+0xe4
?(?) + 0
biowait(0xf5ef6f68,0xf5ef6f68,0xf604ee00,0xf591b56c,0xf6306430,0xf591b550)
physio(0xf5ef6f68,0xf02d596c,0x200,0x100,0xf591b550,0xfbedcb50) + 364
write(0x200) + 250
Of all the threads, only one has a stack trace that references the ramdisk driver. It happens to be the last one. It seems that the process running mkfs(1M) is blocked in biowait(9F). After a call to physio(9F), biowait(9F) takes a buf(9S) structure as a parameter. The next step is to examine the buf(9S) structure:
kadb[0]: f5ef6f68$<buf
0xf5ef6f68: flags
4129
0xf5ef6f6c: forw back av_forw av_back
0 0 0 0
0xf5ef6f80: bcount bufsize error edev
512 0 0 1180000
0xf5ef6f84: addr blkno resid proc
f5f66250 3ff 0 f6285518
0xf5ef6fac: iodone vp pages
0 0 0
The resid field is 0, which indicates that the transfer is complete. physio(9F) is still blocked, however. The reference for physio(9F) in the Solaris 2.7 Reference Manual AnswerBook points out that biodone(9F) should be called to unblock biowait(9F). This is the problem; rd_strategy() did not call biodone(9F). Adding a call to biodone(9F) before returning fixes this problem.
Testing
Once a device driver is functional, it should be thoroughly tested before it is distributed. In addition to the testing done to traditional UNIX device drivers, Solaris 7 drivers require testing of Solaris 7 features, such as dynamic loading and unloading of drivers and multithreading.
Configuration Testing
A driver's ability to handle multiple configurations is as important part of the test process. Once the driver is working on a simple, or default, configuration, additional configurations should be tested. Depending upon the device, this may be accomplished by changing jumpers or DIP switches. If the number of possible configurations is small, all of them should be tried. If the number is large, various classes of possible configurations should be defined, and a sampling of configurations from each class should be tested. The designation of such classes depends on how the different configuration parameters might interact, which in turn depends on the device and on how the driver was written.
For each configuration, the basic functions must be tested, which include loading, opening, reading, writing, closing, and unloading the driver. Any function that depends upon the configuration deserves special attention. For example, changing the base memory address of device registers is not likely to affect the behavior of most driver functions; if the driver works well with one address, it is likely to work as well with a different address, provided the configuration code enables it to work at all. On the other hand, a special I/O control call might have different effects depending upon the particular device configuration.
Loading the driver with varying configurations ensures that the probe(9E) and attach(9E) entry points can find the device at different addresses. For basic functional testing, using regular UNIX commands such as cat(1) or dd(1M) is usually sufficient for character devices. Mounting or booting may be required for block devices.
Functionality Testing
After a driver has been completely tested for configuration, all of its functionality should be thoroughly tested. This requires exercising the operation of all the driver's entry points. In addition to the basic functional tests done in configuration testing, full functionality testing requires testing the rest of the entry points and functions to obtain confidence that the driver can correctly perform all its functions.
Many drivers will require custom applications to test functionality, but basic drivers for devices such as disks, tapes, or asynchronous boards can be tested using standard system utilities. All entry points should be tested in this process, including devmap(9E), poll(9E) and ioctl(9E), if applicable. The ioctl(9E) tests might be quite different for each driver, and for nonstandard devices a custom testing application will be required.
Error Handling
A driver may perform correctly in an ideal environment, but fail to handle cases where a device encounters an error or an application specifies erroneous operations or sends bad data to the driver. Therefore, an important part of driver testing is the testing of its error handling.
All of a driver's possible error conditions should be exercised, including error conditions for actual hardware malfunctions. Some hardware error conditions might be difficult to induce, but an effort should be made to cause them or to simulate them if possible. It should always be assumed that all of these conditions will be encountered in the field. Cables should be removed or loosened, boards should be removed, and erroneous user application code should be written to test those error paths.
Stress, Performance, and Interoperability Testing
To help ensure that the driver performs well, it should be subjected to vigorous stress testing. Running single threads through a driver will not test any of the locking logic and might not test condition variable waits. Device operations should be performed by multiple processes at once to cause several threads to execute the same code simultaneously. The way to do this depends upon the driver; some drivers will require special testing applications, but starting several UNIX commands in the background will be suitable for others. It depends upon where the particular driver uses locks and condition variables. Testing a driver on a multiprocessor machine is more likely to expose problems than testing on a single-processor machine.
Interoperability between drivers must also be tested, particularly because different devices can share interrupt levels. If possible, configure another device at the same interrupt level as the one being tested. Then stress-test the driver to determine if it correctly claims its own interrupts and otherwise operates according to expectations. Stress tests should be run on both devices at once. Even if the devices do not share an interrupt level, this test can still be valuable; for example, if serial communication devices start to experience errors while a network driver is being tested, this could indicate that the network driver is causing the rest of the system to encounter interrupt latency problems.
Driver performance under these stress tests should be measured using UNIX performance-measuring tools. This can be as simple as using the time(1) command along with commands used for stress tests.
DDI/DKI Compliance Testing
To ensure compatibility with later releases and reliable support for the current release, every driver should be Solaris 7 DDI/DKI compliant. One way to determine if the driver is compliant is by inspection. The driver can be visually inspected to ensure that only kernel routines and data structures specified in Sections 9F and 9S of the Solaris 2.7 Reference Manual are used.
The Solaris 7 Driver Developer Kit (DDK) includes a DDI compliance tool (DDICT) that checks device driver C source code for non-DDI/DKI compliance and issues either error or warning messages when it finds non-compliant code. For best results, all drivers should be written to pass DDICT.
Installation and Packaging Testing
Drivers are delivered to customers in packages. A package can be added and removed from the system using a standard mechanism (see the Application Packaging Guide).
Test that the driver has been correctly packaged to ensure that the end user will be able to add it to and remove it from a system. In testing, the package should be installed and removed from every type of media on which it will be released and on several system configurations. Packages must not make unwarranted assumptions about the directory environment of the target system. Certain valid assumptions, however, may be made about where standard kernel files are kept. It is a good idea to test adding and removing of packages on newly-installed machines that have not been modified for a development environment. It is a common packaging error for a package to use a tool or file that exists only in a development environment, or only on the driver writer's own development system. For example, no tools from Source Compatibility package, SUNWscpu, should be used in driver installation programs.
The driver installation must be tested on a minimal Solaris system without any of the optional packages installed.
Testing Specific Types of Drivers
Because each type of device is different, it is difficult to describe how to test them all specifically. This section provides some information about how to test certain types of standard devices.
Tape Drivers
Tape drivers should be tested by performing several archive and restore operations. The cpio(1) and tar(1) commands may be used for this purpose. The dd(1M) command can be used to write an entire disk partition to tape, which can then be read back and written to another partition of the same size, and the two copies compared. The mt(1) command will exercise most of the I/O controls that are specific to tape drivers (see mtio(7I)); all the options should be attempted. The error handling of tape drivers can be tested by attempting various operations with the tape removed, attempting writes with the write protect on, and removing power during operations. Tape drivers typically implement exclusive-access open(9E) calls, which should be tested by having a second process try to open the device while a first process already has it open.
Disk Drivers
Disk drivers should be tested in both the raw and block device modes. For block device tests, a new file system should be created on the device and mounted. Multiple file operations can be performed on the device at this time.
Note -
The file system uses a page cache, so reading the same file over and over again will not really exercise the driver. The page cache can be forced to retrieve data from the device by memory-mapping the file (with mmap(2)), and using msync(2) to invalidate the in-memory copies.
Another (unmounted) partition of the same size can be copied to the raw device and then commands such as fsck(1M) can be used to verify the correctness of the copy. The new partition can also be mounted and compared to the old one on a file-by-file basis.
Asynchronous Communication Drivers
Asynchronous drivers can be tested at the basic level by setting up a login line to the serial ports. A good start is if a user can log in on this line. To sufficiently test an asynchronous driver, however, all the I/O control functions must be tested, and many interrupts at high speed must occur. A test involving a loopback serial cable and high data transfer rates will help determine the reliability of the driver. Running uucp(1C) over the line also provides some exercise; however, since uucp(1C) performs its own error handling, it is important to verify that the driver is not reporting excessive numbers of errors to the uucp(1C) process.
These types of devices are usually STREAMS based.
Network Drivers
Network drivers may be tested using standard network utilities. ftp(1) and rcp(1) are useful because the files can be compared on each end of the network. The driver should be tested under heavy network loading, so that various commands can be run by multiple processes. Heavy network loading means:
-
Traffic to the test machine is heavy.
-
Traffic among all machines on the network is heavy.
Network cables should be unplugged while the tests are executing to ensure that the driver recovers gracefully from the resulting error conditions. Another important test is for the driver to receive multiple packets in rapid succession (back-to-back packets). In this case, a relatively fast host on a lightly loaded network should send multiple packets in quick succession to the test machine. It should be verified that the receiving driver does not drop the second and subsequent packets.
These types of devices are usually STREAMS based.
- © 2010, Oracle Corporation and/or its affiliates