Writing Device Drivers

The kadb Kernel Debugger

kadb(1M) is a kernel debugger with facilities for disassembly, breakpoints, watch points, data display, and stack tracing. This section provides a tutorial on some of the features of kadb. For further information, consult the kadb(1M) man page.

Starting kadb

In order to start up kadb, the system must be booted with kadb(1M) enabled:

ok boot kadb
Rebooting with command: boot kadb
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a
File and args: kadb
kadb: kernel/sparcv9/unix
Size: 499808+109993+132503 Bytes
/platform/sun4u/kernel/sparcv9/unix loaded - 0x11e000 bytes used
SunOS Release 5.8 Version Generic 64-bit
Copyright 1983-2000 Sun Microsystems, Inc.  All rights reserved

By default, kadb(1M) boots (and debugs) kernel/unix, or kernel/sparcv9/unix on a system capable of running a 64-bit kernel. To boot kadb with an alternate kernel, pass the -D flag to boot, as follows:

ok boot kadb -D kernel.test/unix
Rebooting with command: boot kadb -D kernel.test/unix
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a  File
and args: kadb -D kernel.test/unix
kadb: kernel.test/unix
Size: 482384+67201+88883 Bytes
/platform/sun4u/kernel.test/unix loaded - 0xfe000 bytes used
SunOS Release 5.8 Version dacf-fixes:11/13/99 32-bit
Copyright 1983-2000 Sun Microsystems, Inc.  All rights reserved.

In this example, the 32-bit version of the alternate kernel kernel.test was booted. Another option is to pass kadb the -d flag, which causes kadb to prompt for the kernel name. The -d flag also causes kadb(1M) to provide a prompt after it has loaded the kernel, so breakpoints can be set.

ok boot kadb -d
Rebooting with command: boot kadb -d
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a  File
and args: kadb -d
kadb: kernel.test/unix
Size: 482384+67201+88883 Bytes
/platform/sun4u/kernel.test/unix loaded - 0xfc000 bytes used
stopped at      _start:         sethi   %hi(0x10006c00), %g1

At this point you can set breakpoints or continue execution with the :c command.

Note -

Kernel modules are dynamically loaded. Consequently, driver symbols are not available until the driver is loaded. To set breakpoints in modules that have not been loaded, use deferred breakpoints. For information on deferred breakpoints, refer to "Breakpoints".

kadb(1M) passes any kernel flags to the booted kernel. For example, to boot an alternate kernel and pass the -r flag:

ok boot kadb -r -D kernel.test/unix
Rebooting with command: boot kadb -r -D kernel.test/unix
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a
File and args: kadb -r -D kernel.test/unix
kadb: kernel.test/unix
Size: 482384+67201+88883 Bytes
/platform/sun4u/kernel.test/unix loaded - 0xfe000 bytes used
SunOS Release 5.8 Version Generic 32-bit
Copyright 1983-2000 Sun Microsystems, Inc.  All rights reserved.
obpsym: symbolic debugging is available.
Read 208377 bytes from misc/forthdebug
configuring IPv4 interfaces: le0.
Hostname: test
Configuring /dev and /devices

After the system is booted, sending a break passes control to kadb(1M). A break is generated with STOP-A (on the console of SPARC machines), or wtih F1-A (on the console of IA machines), or by using ~# (if the console is connected through a tip window).


The system is ready.

test console login: ~
stopped at      edd000d8:       ta      %icc,%g0 + 125

The number in brackets is the CPU that kadb(1M) is currently executing on; the remaining CPUs are halted. The CPU number is zero on a uniprocessor system.

Caution - Caution -

Before rebooting or turning off the power, always halt the system cleanly (with init 0 or shutdown). Buffers might not be flushed otherwise. If the shutdown must occur from the boot PROM prompt, make sure to flush buffers using the sync command at the ok prompt.

To return control to the operating system, use :c.

kadb[0]: :c
test console login: 


To exit kadb(1M), use $q. On SPARC machines, this will exit to the ok prompt. On IA machines, you will be prompted to reboot the system.

kadb[0]: $qType `go' to resume

kadb(1M) can be resumed by typing go at the ok prompt.

Caution - Caution -

No other commands should be performed from the PROM if the system is to be resumed. PROM commands other than go can change system state that the Solaris 8 operating environment depends upon.

Staying at the kadb(1M) prompt for too long can cause the system to lose track of the time of day, and can cause network connections to time out.


The general form of a kadb command is:

    [ address ] [ ,count ] command [;]

If address is omitted, the current location is used (`.' could also be used to represent the current location). The address can be a kernel symbol. If count is omitted, it defaults to 1.

Commands to kadb consist of a verb followed by a modifier or list of modifiers. Verbs can be:

Prints locations starting at address in the kernel address space

Prints the value of address itself

Assigns a value to a debugger variable or machine register 

Reads a value from a debugger variable or machine register 


Repeats the previous command with a count of 1. Increments `.' (the current location) 

With / and =, output format specifiers can be used. Lowercase letters normally print 2 bytes, uppercase letters print 4 bytes:

o, O 

2-, 4-byte octal 

8-byte octal 

8-byte unsigned octal 

d, D 

2-, 4-byte decimal 

8-byte decimal 

8-byte unsigned decimal 

x, X 

2-, 4-byte hexadecimal 

8-byte hexadecimal 

4-byte hexadecimal for 32-bit programs, 8-byte hexadecimal for 64-bit programs. Use this format specifier to examine pointers. 

u, U 

2-, 4-byte unsigned decimal 

Prints the addressed character 

Prints the addressed character using ^ escape notation 

Prints the addressed string 

Prints the addressed string using ^ escape notation 

Prints as machine instructions (disassemble) 

Prints the value of `.' in symbolic form 

w, W 

2-, 4-byte write 

8-byte write 

Caution - Caution -

When using w, W or Z to modify a kernel variable, make sure that the size of the variable matches the size of the write you are performing. If you specify an incorrect size you could corrupt neighboring data.

For example, to set a bit in the moddebug variable when debugging a driver, first examine the value of moddebug, then set it to the desired bit.

kadb[0]: moddebug/Xmoddebug:
moddebug:       1000
kadb[0]: moddebug/W 0x80001000moddebug:       0x1000 = 0x80001000

Routines can be disassembled with the `i' command. This is useful when tracing crashes, since the only information might be the program counter at the time of the crash. For example, to print the first four instructions of the kmem_alloc function:

kadb[0]: kmem_alloc,4/i
kmem_alloc: save    %sp, -0x60, %sp
sub     %i0, 0x1, %l6
sra     %l6, 0x3, %i5
tst     %i5

Specify symbolic notation with the `a' command, to show the addresses:

kadb[0]: kmem_alloc,4/ai
kmem_alloc:     save    %sp, -0x60, %sp
kmem_alloc+4:   sub     %i0, 0x1, %l6
kmem_alloc+8:   sra     %l6, 0x3, %i5
kmem_alloc+0xc: tst     %i5

Register Identifiers

You can discover what machine registers are available on your processor architecture using the $r command. This example shows the output of $r on a SPARC system with the sun4u architecture:

kadb[0]: $r
g0    0                                 l0      0
g1    100130a4      debug_enter         l1      edd00028
g2    10411c00      tsbmiss_area+0xe00  l2      10449c90
g3    10442000      ti_statetbl+0x1ba   l3      1b
g4    3000061a004                       l4      10474400
g5    0                                 l5      3b9aca00
g6    0                                 l6      0
g7    2a10001fd40                       l7      0
o0    0                                 i0      0
o1    c                                 i1      10449e50
o2    20                                i2      0
o3    300006b2d08                       i3      10
o4    0                                 i4      0
o5    0                                 i5      b0
sp    2a10001b451                       fp      2a10001b521
o7    1001311c      debug_enter+0x78    i7      1034bb24
y     0
tstate: 1604  (ccr=0x0, asi=0x0, pstate=0x16, cwp=0x4)
pstate: ag:0 ie:1 priv:1 am:0 pef:1 mm:0 tle:0 cle:0 mg:0 ig:0
winreg: cur:4 other:0 clean:7 cansave:1 canrest:5 wstate:14
tba   0x10000000
pc    edd000d8 edd000d8:        ta      %icc,%g0 + 125
npc   edd000dc edd000dc:        nop

kadb exports each of these registers as a debugger variable with the same name. Reading from the variable fetches the current value of the register. Writing to the variable changes the value of the associated machine register. For example, you can change the value of the '%o0' register:

kadb[0]: <o0=K
kadb[0]: 0x1>o0
kadb[0]: <o0=K

Display and Control Commands

The following commands display and control the status of kadb(1M):


Display all breakpoints 


Display stack trace 


Change default radix to value of dot 




Display registers 


Display built-in macros 

`$c' is useful when a breakpoint is reached, but is usually not useful if kadb(1M) is entered at a random time. The number of arguments to print can be passed following the `$c' (`$c 2' for two arguments).


In kadb(1M), breakpoints can be set. When reached, the kernel will automatically drop back into kadb. The standard form of a breakpoint command is:

    [module_name#] addr [, count]:b [command]

addr is the address at which the program will be stopped and the debugger will receive control, count is the number of times that the breakpoint address occurs before stopping, and command is almost any adb(1) command.

The optional module_name specifies deferred breakpoints that are set when the module is loaded. module_name identifies a particular module that contains addr. If the module has been loaded, kadb will try to set a regular breakpoint; if the module is not loaded, kadb will set a deferred breakpoint. When the module is loaded, kadb will try to resolve the location of the breakpoint and convert the breakpoint to a regular breakpoint.

Other breakpoint commands are:


Continue execution 


Delete breakpoint 


Single step 


Single step, but step over function calls 


Stop after return to caller of current function 


Delete all breakpoints 

The following example sets a breakpoint in scsi_transport(9F), a commonly used routine. Upon reaching the breakpoint, '$c' is used to print a stack trace. The top of the stack is displayed first. Note that kadb does not know how many arguments were passed to each function.

stopped at      edd000d8:       ta      %icc,%g0 + 125
kadb[0]: scsi_transport:b
kadb[0]: :c
test console login: root
breakpoint at:
scsi_transport: save    %sp, -0x60, %sp
kadb[0]: $c
sdstrategy(1019c8c0,702bb61c,0,0,702bb578,70cad7b8) + 704
bdev_strategy(1042a808,70cad7b8,705f3efc,40,10597900,2000) + 98
ufs_getpage_miss(70cad7b8,0,10597900,0,0,4023ba8c) + 2b0
ufs_getpage(0,0,0,0,2000,4023ba8c) + 7c0
segvn_fault(4023ba8c,2000,ff3b0000,0,0,0) + 7c8
as_fault(1,ff3b0000,70d98030,2000,0,ff3b0000) + 49c
pagefault(0,0,70df8048,705c7450,0,ff3b0000) + 4c
trap(10080,10000,ff3c4ea4,70df8048,ff3b0000,1) + db4
kadb[0]: $b
count   bkpt            type      len   command
1       scsi_transport  :b instr  4
kadb[0]: scsi_transport:d
kadb[0]: :c

Conditional Breakpoints

Breakpoints can also be set to occur only if a certain condition is met. By providing a command, the breakpoint will be taken only if the count is reached or the command returns zero. For example, a breakpoint that occurs only on certain I/O controls could be set in the driver's ioctl(9E) routine. This is the general syntax of conditional breakpoints:

    address,count:b command

In this example, address is the address at which to set the breakpoint. count is the number of times the breakpoint should be ignored (note that 0 means break only when the command returns zero). command is the kadb(1M) command to execute.

Here is an example of breaking only in the sdioctl() routine if the DKIOGVTOC (get volume table of contents) I/O control occurs.

kadb[0]: sdioctl+4,0:b <i1-0x40B
kadb[0]: $b
count   bkpt            type      len   command
0       sdioctl+4       :b instr  4     <i1-0x40B
kadb[0]: :c     

Adding four to sdioctl skips to the second instruction in the routine, bypassing the save instruction that establishes the stack. The `<i1' refers to the first input register, which is the second parameter to the routine (the cmd argument of ioctl(9E)). The count of zero is impossible to reach, so it stops only when the command returns zero, which is when `i1 - 0x40B' is true. This means i1 contains 0x40B (the value of the ioctl(9E) command, determined by examining the ioctl definition).

To force the breakpoint to be reached, the prtvtoc(1M) command is used. It is known to issue this I/O control:

# prtvtoc /dev/rdsk/c0t0d0s0breakpoint at:
sdioctl+4:      mov     %i5, %l0
kadb[0]: $c
sdioctl(800000,40b,ffbefb54,100005,704a3ce8,4026bc7c) + 4
ioctl(3,40b,70ca27b8,40b,ffbefb54,0) + 1e0


kadb(1M) supports macros that are used for displaying kernel data structures. kadb(1M) macros can be displayed with $M. Macros are used in the form:

    [ address ] $<macroname

threadlist is a useful macro that displays the stacks of all the threads in the system.

kadb[0]: $<threadlist

                ============== thread_id        10404000
                process args    sched

t0+0xa8:        lwp             procp           wchan
                1041b810        10424688        0
                pc              sp
                sched+0x4f4     10403be8
_start(10006ef4,1041adb0,1041adb0,1041adb0,10462910,50) + 15c


                ============== thread_id        40043e60
                process args    sched

40043f08:       lwp             procp           wchan
                0               10424688        10473c56
                pc              sp
                cv_wait+0x60    40043c08
ufs_thread_idle(10471e80,0,10473c5c,10424688,81010100,0) + bc
thread_start(0,0,0,0,0,0) + 4

Another useful macro is thread. Given a thread ID, this macro prints the corresponding thread structure. This can be used to look at a certain thread found with the threadlist macro, to look at the owner of a mutex, or to look at the current thread, as shown here:

kadb[0]: <g7$<thread
70e87ac0:       link            stk             startpc
                0               4026bc80        0
70e87acc:       bound_cpu       affinitycnt     bind_cpu
                0               0               -1
70e87ad4:       flag    proc_flag       schedflag
                0       4               3
70e87ada:       preempt preempt_lk      state
                0       0               4
70e87ae0:       pri     epri
                40      0
                pc              sp
                10098350        4026b618
70e87aec:       wchan0          wchan           sobj_ops
                0               0               0
70e87af8:       cid             clfuncs         cldata
                1               10470ffc        702c0488
70e87b04:       ctx             lofault         onfault
                0               0               0

Note -

No type information is kept in the kernel, so using a macro on an inappropriate address results in garbage output.

Macros do not necessarily output all the fields of the structures, nor is the output necessarily in the order given in the structure definition. Occasionally, memory needs to be dumped for certain structures and then matched with the structure definition in the kernel header files.

Caution - Caution -

Drivers should never reference header files and structures not listed in man pages section 9S: DDI and DKI Data Structures. However, examining non-DDI-compliant structures (such as thread structures) can be useful in debugging drivers.

Output Pager

Some kadb commands (like $<threadlist) output lots of data, which can scroll off of the screen very rapidly. kadb provides a simple output pager to remedy this problem. The pager command is lines::more, where lines represents the number of lines to print before pausing the console output. Keep in mind that this does not take into account lines that wrap because they are wider than the terminal width. Here is an example usage:

kadb[0]: 0t10::more
kadb[0]: $<threadlist 

                ============== thread_id        10408000
                process args    sched

t0+0x128:       lwp             procp           wchan
            10429ed0         104393e8         0
                pc              sp
                sched+0x4e4     104071f1
_start(10007588,104292e0,104292e0,104292e0,1043b8b0,10429360) + 200

                ============== thread_id        2a10001fd40
                process args    sched
--More-- <SPACE>

Pressing the space bar at the "--More--" prompt pages the output by the number of lines specified to ::more (in this case, 10). Pressing "Return" prints only the next line of output. You can abort the output and return to the kadb prompt by typing Ctrl-C. To disable the pager, issue '0::more' at the kadb prompt.