Writing Device Drivers

Debugging Tools

This section describes some programs and files that can be used to debug the driver at runtime.


The /etc/system file is read once while the kernel is booting. It is used to set various kernel options. After this file is modified, the system must be rebooted for the changes to take effect. If a change in the file causes the system not to work, boot with the ask (-a) option and specify /dev/null as the system file.

Add the following set commands to the /etc/system file:

See system(4) for more information.

Note -

Most kernel variables are not guaranteed to be present in subsequent releases.


moddebug is a kernel variable that controls the module loading process. The possible values are:


Prints messages to the console when loading or unloading modules. 


Gives more detailed error messages. 


Prints more detail when loading or unloading (such as including the address and size). 


No autounloading drivers: the system will not attempt to unload the device driver when the system resources become low. 


No autounloading streams: the system will not attempt to unload the streams module when the system resources become low. 


No autounloading of drivers of any type. Module loading is disabled. 


Not acceptable to page out symbol table. Prevents kernel from (possibly) paging out the driver's symbol table. kadb requires access to the symbol table to operate properly.


If running with kadb, moddebug causes a breakpoint to be executed and a return to kadb immediately before each module's _init(9E) routine is called. Also generates additional debug messages when the module's _info and _fini routines are executed.

modload and modunload

Because the kernel automatically loads needed modules, and unloads unused ones, these two commands are now obsolete. However, they can be used for debugging.

modload(1M) can be used to force a module into memory. The kernel might subsequently unload it, but modload(1M) may be used to ensure that the driver has no unresolved references when loaded.

modunload(1M) can be used to unload a module, given a module ID (which can be determined with modinfo(1M)). Unloading a module does not necessarily remove it from memory. To unload all unloadable modules and forcibly remove them from memory (so that they will be reloaded from the actual object file), use module ID zero:

# modunload -i 0

Note -

A future release might not include modload(1M) and modunload(1M).

Saving System Core Dumps

When the system panics, it writes the memory image to the dump device. The dump device by default is the primary swap device. The dump is a system core dump, similar to core dumps generated by applications. On rebooting after a panic, savecore(1M) checks the dump device for a crash dump. If one is found, it makes a copy of the kernel that was running (called unix.n) and dumps a core file (called vmcore.n) in the core image directory which by default is /var/crash/machine_name.

In Solaris 7, crash dump is enabled by default. The dumpadm(1M) command is used to configure system crash dumps. The dumpadm(1M) command should be used to verify that crash dumps are enabled and to determine the location of the directory where core files are saved. See dumpadm(1M) for more information.

Note -

dumpadm(1M) should be used to verify that the system is correctly configured for obtaining crash dumps.

When savecore(1M) runs, it makes a copy of the kernel that was running (called unix.n) and dumps a core file (called vmcore.n) in the specified directory, normally /var/crash/machine_name. There must be enough space in /var/crash to contain the core dump or it will be truncated. Because the file contains holes, it will appear larger than actual size; avoid copying it. adb(1) can then be used on the core dump and the saved kernel.

Note -

savecore(1M) can be prevented from filling the file system if there is a file called minfree in the directory in which the dump will be saved. This file contains a number of kilobytes to remain free after savecore(1M) has run. However, if not enough space is available, the core file is not saved.

adb and kadb

adb(1) can be used to debug applications or the kernel, though it cannot debug the kernel interactively (such as by setting breakpoints). To interactively debug the kernel, use kadb(1M). Both adb(1) and kadb(1M) share a common command set.

Starting adb

The command for starting adb(1) to debug a kernel core dump is:

% adb -k /var/crash/`hostname`/unix.n /var/crash/`hostname`/vmcore.n

Note -

For best results, use adb on the same architecture (such as Sun4m) that generated the core image.

To start adb(1) on a live system, type (as root):

# adb -k /dev/ksyms /dev/mem

/dev/ksyms is a special driver that provides an image of the kernel's symbol table to adb(1).

When < adb(1) responds with physmem xxx, it is ready for a command.

Note -

If the -p option of adb is used, an input prompt is displayed.

Starting kadb

The system must be booted under kadb(1M) before kadb(1M) can be used.

ok boot kadb
Boot device: /sbus/esp@0,800000/sd@3,0   File and args: kadb
kadb: kernel/unix
Size: 191220+114284+12268 Bytes
/platform/SUNW,Sun_4_75/kernel/unix loaded - 0x70000 bytes used
SunOS Release 5.7 Version Generic [UNIX(R) System V Release 4.0]
Copyright (c) 1983-1998, Sun Microsystems, Inc.

By default, kadb(1M) boots (and debugs) kernel/unix. It can be passed a file name as an argument to boot a different kernel, or -d can be passed to have kadb(1M) prompt for the kernel name. The -d flag also causes kadb(1M) to provide a prompt after it has loaded the kernel, so breakpoints can be set.

ok boot kadb -d
Boot device: /sbus/esp@0,800000/sd@3,0   File and args: kadb -d
kadb: kernel/unix
kadb: kernel/unix
Size: 191220+114284+12268 Bytes
/platform/SUNW,Sun_4_75/kernel/unix loaded - 0x70000 bytes used

Note -

Modules are dynamically loaded. Consequently, driver symbols are not generally available until the driver is loaded. To set breakpoints in modules that have not been loaded, use deferred breakpoints. For information on deferred breakpoints, see "Breakpoints".

At this point you can set breakpoints or continue with the :c command.

kadb(1M) passes any kernel flags to the booted kernel. For example, the flags -r, -s, and -a can be passed to kernel/unix with the command:

boot kadb -ras

Once the system is booted, sending a break passes control to kadb(1M). A break is generated with L1+A (on the console of SPARC machines) , or by Crtl+Alt+D (on the console of x86 machines) or ~# (if the console is connected through a tip window).


The system is ready.

test console login: ~stopped at 0xfbd01028: ta  0x7d

The number in brackets is the CPU that kadb(1M) is currently executing on; the remaining CPUs are halted. The CPU number is zero on a uniprocessor.

Caution - Caution -

Before rebooting or turning off the power, always halt the system cleanly (with init 0 or shutdown). Buffers may not be flushed otherwise. If the shutdown must occur from the boot PROM prompt, make sure to flush buffers with sync.

To return control to the operating system, use :c.

kadb[0]: :c
test console login: 


To exit either adb(1M) or kadb(1M), use $q.

kadb[0]: $qType `go' to resume

On SPARC machines, kadb(1M) can be resumed by typing go at the ok prompt. On x86 machines, kadb(1M) cannot be resumed.

Caution - Caution -

No other commands can be performed from the PROM if the system is to be resumed. PROM commands other than go may change system state that the Solaris 7 operating environment depends upon.

Staying at the kadb(1M) prompt for too long may cause the system to lose track of the time of day, and cause network connections to time out.


The general form of an adb(1M) or kadb(1M) command is:

	[ address ] [ ,count ] command [;]

If address is omitted, the current location is used (`.' could also be used to represent the current location). The address can be a kernel symbol. If count is omitted, it defaults to 1.

Commands to adb consist of a verb followed by a modifier or list of modifiers. Verbs can be:

Prints locations starting at address in the executable.

Prints locations starting at address in the core file.

Prints the value of address itself.

Assigns a value to a variable or register. 

Reads a value from a variable or register. 


Repeats the previous command with a count of 1. Increments `.' (the current location). 

With ?, /, and =, output format specifiers can be used. Lowercase letters normally print 2 bytes, uppercase letters print 4 bytes:

o, O 

2-, 4-byte octal 

8-byte octal 

8-byte unsigned octal 


2-, 4-byte decimal 

8-byte decimal 

8-byte unsigned decimal 


2-, 4-byte hexadecimal 

8-byte hexadecimal 

4-byte hexadecimal for 32-bit programs, 8-byte hexadecimal for 64-bit programs 


2-, 4-byte unsigned decimal 


4-, 8-byte floating point 

Prints the addressed character. 

Prints the addressed character using ^ escape notation. 

Prints the addressed string. 

Prints the addressed string using ^ escape notation. 

Prints as machine instructions (disassemble). 

Prints the value of `.' in symbolic form. 


2-, 4-byte write 

Note -

Understand exactly what sizes the objects are, and what effects changing them might have, before making any changes.

For example, to set a bit in the moddebug variable when debugging the driver, first examine the value of moddebug, then set it to the desired bit.

kadb[0]: moddebug/Xmoddebug:
moddebug:       1000
kadb[0]: moddebug/W 0x80001000moddebug:       0x1000 = 0x80001000

Routines can be disassembled with the `i' command. This is useful when tracing crashes, since the only information may be the program counter at the time of the crash. For example, to print the first four instructions of the kmem_alloc function:

kadb[0]: kmem_alloc,4?i
kmem_alloc: save    %sp, -0x60, %sp
sub     %i0, 0x1, %l6
sra     %l6, 0x3, %i5
tst     %i5

To show the addresses also, specify symbolic notation with the `a' command.

kadb[0]: kmem_alloc,4?ai
kmem_alloc:     save    %sp, -0x60, %sp
kmem_alloc+4:   sub     %i0, 0x1, %l6
kmem_alloc+8:   sra     %l6, 0x3, %i5
kmem_alloc+0xc: tst     %i5

Register Identifiers

Machine or kadb(1M) internal registers are identified with the `<` command, followed by the register of interest. On SPARC machines, the following register names are recognized:


dot, the current location


Input registers to current function 


Output registers for current function 


Local registers 


Global registers 


Processor Status Register 


Trap Base Register 


Window Invalid Mask 

For more information on how these registers are normally used, see the System V Application Binary Interface, SPARC Processor Supplement.

On x86 machines, the following register names are recognized:


Stack frame base register 


Stack pointer 


Kernel stack pointer 


Code segment 


Data segment 


Stack segment 


Program pointer or instruction pointer 


Status flags register 

Note -

The remaining examples in this chapter are for use on SPARC machines only. For specific register information relating to x86 machines, see the System V Application Binary Interface, x86 Processor Supplement.

The following command displays the PSR as a 4-byte hexadecimal value:

kadb[0]: <psr=X

Display and Control Commands

The following commands display and control the status of adb(1)/ kadb(1M):


Display all breakpoints 


Display stack trace 


Change default radix to value of dot 




Display registers 


Display built-in macros 

`$c' is useful with crash dumps: it shows the call trace and arguments at the time of the crash. It is also useful in kadb(1M) when a breakpoint is reached, but is usually not useful if kadb(1M) is entered at a random time. The number of arguments to print can be passed following the `$c' (`$c 2' for two arguments).


In kadb(1M), breakpoints can be set that will automatically drop back into kadb when reached. The standard form of a breakpoint command is:

	[module_name#] addr [, count]:b [command]

addr is the address at which the program will be stopped and the debugger will receive control, count is the number of times that the breakpoint address occurs before stopping, and command is almost any adb(1) command.

The optional module_name specifies deferred breakpoints that are set when the module is loaded. module_name identifies a particular module that contains addr. If the module has been loaded, kadb will try to set a regular breakpoint; if the module is not loaded, kadb will set a deferred breakpoint. When the module is loaded, kadb will try to resolve the location of the breakpoint and convert the breakpoint to a regular breakpoint.

Other breakpoint commands are:


Continue execution 


Delete breakpoint 


Single step 


Single step, but step over function calls 


Stop after return to caller of current function 


Delete all breakpoints 

The following example sets a breakpoint in scsi_transport(9F), a commonly used routine. Upon reaching the breakpoint, `$c' is used to get a stack trace. The top of stack is the first function printed. Note that kadb(1M) does not know how many arguments were passed to the function; it always prints six.

kadb[0]: scsi_transport:bkadb[0]: :c

 test console login: root
  breakpoint at:
  scsi_transport: save    %sp, -0x60, %sp
  kadb[0]: $c
  sdstrategy(0xf5d4b8a8,0xf5ad0e1c,0x0,0x2c1d0,0xf5ad0d78,0x0) + 3c0
  bdev_strategy(0xf5d4b8a8,0xf5cd3dcc,0x8,0x40,0xf0319e60,0x1000) + d0
  ufs_getpage_miss(0x0,0xf6133840,0x0,0x0,0xfbecb8e8,0x1) + 23c
  ufs_getpage(0x0,0x1000,0x0,0x0,0x0,0x1) + 640
  segmap_fault(0xf5b34000,0xf5d76f98,0x0,0x1000,0x0,0x1) + 120
  segmap_getmapflt(0xf0ba4000,0xf0ba4000,0x10000,0x0,0x6c267,0x0) + 4b0
  rdip(0xf61337b8,0x0,0xf6133840,0xf5a74938,0x0,0xf5cda000) + 328
  ufs_read(0xf61338a4,0xfbecbaf0,0x0,0xf5a74938,0xf5ca7e80,0xf61337b8) + 118
  read(0x4) + 274
  syscall_trap(?) + 18c
  Syssize(0x4) + d3c0
  Syssize(0x26ad4,0x27400,0x0,0xef66855c,0xef661f58,0x11e78) + d2bc
  Syssize(0x27b88,0x27400,0xeffffe74,0x26968,0x27f94,0x27a84) + bc28

  kadb[0]: :s
  stopped at:
  scsi_transport+4:               mov     %i0, %i1
  kadb[0]: $b
  count   bkpt            type      len   command
  1       scsi_transport  :b instr  4
  kadb[0]: scsi_transport:d
  kadb[0]: :c  

Conditional Breakpoints

This is the general syntax of conditional breakpoints:

	address,count:b command

In this example, address is the address at which to set the breakpoint. count is the number of times the breakpoint should be ignored (note that 0 means break only when the command returns zero). command is the adb(1) command to execute.

Breakpoints can also be set to occur only if a certain condition is met. By providing a command, the breakpoint will be taken only if the count is reached or the command returns zero. For example, a breakpoint that occurs only on certain I/O controls could be set in the driver's ioctl(9E) routine. Here is an example of breaking only in the sdioctl() routine if the DKIOGVTOC (get volume table of contents) I/O control occurs.

kadb[0]: sdioctl+4,0:b <i1-0x40Bkadb[0]: $bbreakpoints
count		bkpt			command
0		sdioctl+4			<i1-0x40B
kadb[0]: :c

Adding four to sdioctl skips to the second instruction in the routine, bypassing the save instruction that establishes the stack. The `<i1' refers to the first input register, which is the second parameter to the routine (the cmd argument of ioctl(9E)). The count of zero is impossible to reach, so it stops only when the command returns zero, which is when `i1 - 0x40B' is true. This means i1 contains 0x40B (the value of the ioctl(9E) command, determined by examining the ioctl definition).

To force the breakpoint to be reached, the prtvtoc(1M) command is used. It is known to issue this I/O control:

# prtvtoc /dev/rdsk/c0t3d0s0breakpoint      sdioctl+4:      st      %i5, [%fp + 0x58]
breakpoint sdioctl+4: mov %i0, %o0
kadb[0]: $csdioctl(0x800018,0x40b,0xeffffc04,0x5,0xff34dc68,0xf03f0c00) + 4
ioctl(0xf03f0c90,0xf03f0c00,0x3,0x18,0xff445f5c,0xff7a09e0) + 270
syscall_ap(0x3) + 6c
syscall_trap(?) + 150
Syssize(0x3) + 20458
Syssize(0x3,0x22f70,0xeffffc04,0x1,0x4,0xefffff6c) + fc14
Syssize(0xefffff6c,0xffffffff,0x1,0x1,0x3,0x22f70) + f5b4
Syssize(0x2,0xeffffec4,0xeffffed0,0x22c00,0x0,0x1) + ebe4

kadb(1M) cannot always determine where the bottom of the stack is. In the previous example, the call to Syssize is not part of the stack.


adb(1) and kadb(1M) support macros. adb(1) macros are in /usr/lib/adb and /usr/platform/`uname -i`/lib/adb for 32-bit programs and in /usr/lib/adb/sparcv9 and /usr/platform/`uname -i`/lib/adb/sparcv9 for 64-bit programs. kadb(1M) macros are builtin and can be displayed with $M. Most of the existing macros are for private kernel structures. New macros for adb can be created with adbgen(1M).

Macros are used in the form:

	[ address ] $<macroname

threadlist is a useful macro that displays the stacks of all the threads in the system. Because this macro does not take an address and can generate excessive output, be ready to use Control-S and Control-Q to start or stop if necessary--this is another good reason to use a tip window. Control-C can be used to abort the listing.

kadb[0]: $<threadlist

                ============== thread_id        f0244020
p0:             process args=   sched
t0:             lwp             proc            wchan
                f02a3e78        f02a8078        0
t0+0x34:        sp              pc
                f0243af0        sched+0x4e0
?(?) + 0
afsrbuf(?) + 1b8

                ============== thread_id        fbe01ea0
p0:             process args=   sched
0xfbe01ea0:     lwp             proc            wchan
                0               f02a8078        0
0xfbe01ed4:     sp              pc
                fbe1fe68        debug_enter+0xb0
?(?) + 0
zsa_xsint(0xf5dd9000,0x80,0xf5dd906c,0x44,0xff0113,0x0) + 244
zs_high_intr(0xf5dd9000) + 204
_level1(0x0) + 13bc
idle(0xf02a0fac,0x0,0xf02a8078,0xf02a8078,0xf004684c,0x4400ae1) + 50

                ============== thread_id        fbe22ea0
p0:             process args=   sched
0xfbe22ea0:     lwp             proc            wchan
                0               f02a8078        f5caf024
0xfbe22ed4:     sp              pc
                fbe22d80        cv_wait+0x64
?(?) + 0

Another useful macro is thread. Given a thread ID, this macro prints the corresponding thread structure. This can be used to look at a certain thread found with the threadlist macro, to look at the owner of a mutex, or to look at the current thread.

kadb[0]: <g7$<thread
0xfbe01ea0:0xfbe01ea0:     link            stk             startpc         bound
                0               fbe01e40        f0076d40        f02a0fac
0xfbe01eb0:     affcnt          bind_cpu        flag            procflag
                1               -1              8               0
0xfbe01eb8:     schedflag       preempt         preempt_lk      state
                3               1               0               4
0xfbe01ec0:     pri             epri            pc              sp
                -1              0               f005bdf8        fbe1fe68
0xfbe01ecc:     wchan0          wchan           sobj_ops        cid
                0               0               0               0
0xfbe01edc:     clfuncs         cldata          ctx             lofault
                f02aeed8        0               0               0
0xfbe01eec:     onfault         nofault         swap            lock
                0               0               fbe01000        ff
0xfbe01efa:     delay_cv        cpu             intr            did
                0               f02a0fac        0               1
0xfbe01f08:     tnf_tpdp        tid             alarmid
                f5a75c80        0               0               realitimer
0xfbe01f14:     interval.sec    interval.usec   value.sec       value.usec
                0               0               0               0
0xfbe01f24:     itimerid        sigqueue        sig
                0               0               0               0
0xfbe01f34:     hold                            forw            back
                0               0               0               0
0xfbe01f44:     lwp             procp           next            prev
                0               f02a8078        fbe04ea0        f0244020
0xfbe01f58:     trace           why     what    dslot           pollstate
                0               0       0       0               0
0xfbe01f68:     cred            lbolt           sysnum          pctcpu
                0               0               0               0
0xfbe01f88:     lockp           oldspl          pre_sys         disp_queue
                f02a1014        e0              0               f02a0ff0

Note -

No type information is kept in the kernel, so using a macro on an inappropriate object results in garbage output.

Macros do not necessarily output all the fields of the structures, nor is the output necessarily in the order given in the structure definition. Occasionally, memory may need to be dumped for certain structures and then matched with the structure definition in the kernel header files.

Caution - Caution -

Drivers should never reference header files and structures not listed in Section 9S of the Solaris 2.7 Reference Manual. However, examining non-DDI-compliant structures (such as thread structures) can be useful in debugging drivers.

Example: adb on a Core Dump

During the development of the example ramdisk driver, the system crashes with a data fault when running mkfs(1M).

test# mkfs -F ufs -o nsect=8,ntrack=8,free=5 /devices/pseudo/ramdisk:0,raw 1024BAD TRAP
mkfs: Data fault
kernel read fault at addr=0x4, pme=0x0
Sync Error Reg 80<INVALID>
pid=280, pc=0xff2f88b0, sp=0xf01fe750, psr=0xc0, context=2
g1-g7: ffffff98, 8000000, ffffff80, 0, f01fe9d8, 1, ff1d4900
Begin traceback... sp = f01fe750
Called from f0098050,fp=f01fe7b8,args=1180000 f01fe878 ff1ed280 ff1ed280 2 ff2f8884
Called from f0097d94,fp=f01fe818,args=ff24fd40 f01fe878 f01fe918 0 0 ff2c9504
Called from f0024e8c,fp=f01fe8b0,args=f01fee90 f01fe918 2 f01fe8a4 f01fee90 3241c
Called from f0005a28,fp=f01fe930,args=f00c1c54 f01fe98c 1 f00b9d58 0 3
Called from 15c9c,fp=effffca0,args=5 3241c 200 0 0 7fe00
End traceback...
panic: Data fault

When the system comes up, it saves the kernel and the core file, which can then be examined with adb(1):

# cd /var/crash/test# lsbounds    unix.0    vmcore.0
# adb -k unix.0 vmcore.0physmem 1ece

The first step is to examine the stack to determine where the system was when it crashed:

$ccomplete_panic(0x0,0x1,0xf00b6c00,0x7d0,0xf00b6c00,0xe3) + 114
do_panic(0xf00be7ac,0xf0269750,0x4,0xb,0xb,0xf00b6c00) + 1c
die(0x9,0xf0269704,0x4,0x80,0x1,0xf00be7ac) + 5c
trap(0x9,0xf0269704,0x4,0x80,0x1,0xf02699d8) + 6b4

This stack trace is not helpful initially, as the ramdisk routines are not on the stack trace. However, there is a useful bit of information: the call to trap(). The first argument to trap() is the trap type. The second argument to trap() is a pointer to a regs structure containing the state of the registers at the time of the trap. See The SPARC Architecture Manual, Version 9 for more information.

0xf0269704$<regs0xf0269704: psr					pc			npc
			c0		ff2dd8b0			ff2dd8b4
0xf0269710: y						g1			g2			g3
			e0000000			ffffff98			8000000			ffffff80
0xf0269720: g4						g5			g6			g7
			0			f02699d8			1			ff22c800
0xf0269730: o0						o1			o2			o3
			f02697a0			ff080000			19000			ef709000
0xf0269740: o4						o5			o6			o7
			8000			0			f0269750			7fffffff

Note that the program counter (pc) in the previous example was ff2dd8b0 when the trap occurred. The next step is to determine which routine it is in.

ff2dd8b0/ird_write+0x2c: ld [%o2 + 0x4], %o3

The pc corresponds to rd_write(), which is a routine in the ramdisk driver. The bug is in the ramdisk write routine, and occurs during an load (ld) instruction. This load instruction is dereferencing the value of o2+4, so the next step is to determine the value of o2.

Note -

Using the $r command to examine the registers is inappropriate because the registers have been reused in the trap routine. Instead, examine the value of o2 from the regs structure.

o2 has the value 19000 in the regs structure. Valid kernel addresses are constrained to be above KERNELBASE by the ABI, so this is probably a user address. The ramdisk does not deal with user addresses; consequently, the ramdisk write routine should not dereference an address below KERNELBASE.

To match the assembly language with the C code, the routine is disassembled up to the problem instruction. Each instruction is 4 bytes in size, so 2c/4 or 0xb additional instructions should be displayed:

rd_write:			sethi		%hi(0xfffffc00), %g1
			add		%g1, 0x398, %g1		 ! ffffff98
			save		%sp, %g1, %sp
			st		%i0, [%fp + 0x44]
			st		%i1, [%fp + 0x48]
			st		%i2, [%fp + 0x4c]
			ld		[%fp + 0x44], %o0
			call		getminor
			st		%o0, [%fp - 0x4]
			ld		[%fp - 0x8], %o2
			ld		[%o2 + 0x4], %o3

The crash occurs a few instructions after a call to getminor(9F). If the ramdisk.c file is examined, the following lines stand out in rd_write:

int instance = getminor(dev);
 rd_devstate_t *rsp;

 if (uiop->uio_offset >= rsp->ramsize)
 	return (EINVAL);

Notice that rsp is never initialized. This is the problem. It is fixed by including the correct call to ddi_get_soft_state(9F) (as the ramdisk driver uses the soft state routines to do state management):

int instance = getminor(dev);
 rd_devstate_t *rsp = ddi_get_soft_state(rd_state, instance);
if (uiop->uio_offset >= rsp->ramsize)
 	return (EINVAL);

Note -

Many data fault panics are the result of bad pointer references.

Example: kadb on a Deadlocked Thread

The next problem is that the system does not panic, but the mkfs(1M) command hangs and cannot be aborted. Though a core dump can be forced--by sending a break and then using sync from the OBP or using `g 0' from SunMon--in this case kadb(1M) will be used. After logging in remotely and using ps (which indicated that only the mkfs(1M) process was hung, not the entire system) the system is shut down and booted using kadb(1M) .

ok boot kadb -d
Boot device: /sbus/esp@0,800000/sd@3,0   File and args: kadb -d
Size: 673348+182896+46008 bytes
/platform/SUNW,Sun_4_75/kernel/unix ...
kadb[0]::cSunOS Release 5.7 Version Generic [UNIX(R) System V Release 4.0]
Copyright (c) 1983-1998, Sun Microsystems, Inc.

After the rest of the kernel has loaded, moddebug is patched to see if loading is the problem. Because it got as far as rd_write() before, loading is probably not the problem, but it will be checked anyway.

# ~stopped at 0xfbd01028: ta 0x7d
kadb[0]: moddebug/Xmoddebug:
moddebug: 0
kadb[0]: moddebug/W 0x80000000moddebug: 0x0 = 0x80000000
kadb[0]: :c

modload(1M) is used to load the driver, to separate module loading from the real access:

# modload /home/driver/drv/ramdisk

It loads without errors, so loading is not the problem. The condition is recreated with mkfs(1M).

# mkfs -F ufs -o nsect=8,ntrack=8,free=5 /devices/pseudo/ramdisk@0:c,raw 1024ramdisk0: misusing 524288 bytes of memory

mkfs(1M) hangs. At this point, kadb(1M) is entered and the stack examined:

~stopped        at      Syslimit+0x1028:                ta      0x7d
kadb[0]: $c
Syslimit() + 1028
debug_enter(0x0,0x740000,0xc4,0x0,0x0,0x40000e7) + c0
zsa_xsint(0xf5dd9000,0x80,0xf5dd906c,0x44,0xff0113,0x0) + 244
zs_high_intr(0xf5dd9000) + 204
_level1(0x1) + 13bc
idle(0xf02a0fac,0x0,0xf02a8078,0xf02a8078,0xf004684c,0x4400ae1) + 50

In the previous example, the presence of idle on the current thread stack indicates that this thread is not the cause of the deadlock. To determine the deadlocked thread, the entire thread list is checked:

kadb[0]: $<threadlist

                ============== thread_id        f0244020
p0:             process args=   sched
t0:             lwp             proc            wchan
                f02a3e78        f02a8078        0
t0+0x34:        sp              pc
                f0243af0        sched+0x4e0
?(?) + 0
afsrbuf(?) + 1b8

                ============== thread_id        fbe01ea0
p0:             process args=   sched
0xfbe01ea0:     lwp             proc            wchan
                0               f02a8078        0
0xfbe01ed4:     sp              pc
                fbe1fe68        debug_enter+0xb0
?(?) + 0
zsa_xsint(0xf5dd9000,0x80,0xf5dd906c,0x44,0xff0113,0x0) + 244
zs_high_intr(0xf5dd9000) + 204
_level1(0xf02d40ec) + 13bc
idle(0xf02a0fac,0x0,0xf02a8078,0xf02a8078,0xf004684c,0x4400ae1) + 50

                ============== thread_id        f6350a80
0xf6285518:     process args=   mkfs -o nsect=8,ntrack=8,free=5 /devices/pseudo/
                ramdisk@0:c,raw 1024
0xf6350a80:     lwp             proc            wchan
                f634baa0        f6285518        f5ef6fd0
0xf6350ab4:     sp              pc
                fbedc9e0        biowait+0xe4
?(?) + 0
physio(0xf5ef6f68,0xf02d596c,0x200,0x100,0xf591b550,0xfbedcb50) + 364
write(0x200) + 250

Of all the threads, only one has a stack trace that references the ramdisk driver. It happens to be the last one. It seems that the process running mkfs(1M) is blocked in biowait(9F). After a call to physio(9F), biowait(9F) takes a buf(9S) structure as a parameter. The next step is to examine the buf(9S) structure:

kadb[0]:  f5ef6f68$<buf
0xf5ef6f68:     flags
0xf5ef6f6c:     forw            back            av_forw         av_back
                0               0               0               0
0xf5ef6f80:     bcount          bufsize         error           edev
                512             0               0               1180000
0xf5ef6f84:     addr            blkno           resid           proc
                f5f66250        3ff             0               f6285518
0xf5ef6fac:     iodone          vp              pages
                0               0               0

The resid field is 0, which indicates that the transfer is complete. physio(9F) is still blocked, however. The reference for physio(9F) in the Solaris 2.7 Reference Manual AnswerBook points out that biodone(9F) should be called to unblock biowait(9F). This is the problem; rd_strategy() did not call biodone(9F). Adding a call to biodone(9F) before returning fixes this problem.