The kadb Kernel Debugger
kadb is a kernel debugger with facilities for disassembly, breakpoints, watch points, data display, and stack tracing. This section provides a tutorial on some of the features of kadb. For further information, consult the kadb(1M) man page.
Starting kadb
In order to start up kadb, the system must be booted with kadb(1M) enabled:
ok boot kadb ... Rebooting with command: boot kadb Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a File and args: kadb kadb: kernel/sparcv9/unix Size: 499808+109993+132503 Bytes /platform/sun4u/kernel/sparcv9/unix loaded - 0x11e000 bytes used SunOS Release 5.9 Version Generic 64-bit Copyright 1983-2002 Sun Microsystems, Inc. All rights reserved ....
By default, kadb(1M) boots (and debugs) kernel/unix, or kernel/sparcv9/unix on a system capable of running a 64-bit kernel. To boot kadb with an alternate kernel, pass the -D flag to boot, as follows:
ok boot kadb -D kernel.test/unix ... Rebooting with command: boot kadb -D kernel.test/unix Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a File and args: kadb -D kernel.test/unix kadb: kernel.test/unix Size: 482384+67201+88883 Bytes /platform/sun4u/kernel.test/unix loaded - 0xfe000 bytes used SunOS Release 5.9 Version dacf-fixes:11/13/99 32-bit Copyright 1983-2002 Sun Microsystems, Inc. All rights reserved. ...
In this example, the 32-bit version of the alternate kernel kernel.test was booted. Another option is to pass kadb the -d flag, which causes kadb to prompt for the kernel name. The -d flag also causes kadb(1M) to provide a prompt after it has loaded the kernel, so breakpoints can be set.
ok boot kadb -d ... Rebooting with command: boot kadb -d Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a File and args: kadb -d kadb: kernel.test/unix Size: 482384+67201+88883 Bytes /platform/sun4u/kernel.test/unix loaded - 0xfc000 bytes used stopped at _start: sethi %hi(0x10006c00), %g1 kadb[0]:
At this point you can set breakpoints or continue execution with the :c command.
Note –
Kernel modules are dynamically loaded. Consequently, driver symbols are not available until the driver is loaded. To set breakpoints in modules that have not been loaded, use deferred breakpoints. For information on deferred breakpoints, refer to Breakpoints.
kadb(1M) passes any kernel flags to the booted kernel. For example, to boot an alternate kernel and pass the -r flag:
ok boot kadb -r -D kernel.test/unix ... Rebooting with command: boot kadb -r -D kernel.test/unix Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a File and args: kadb -r -D kernel.test/unix kadb: kernel.test/unix Size: 482384+67201+88883 Bytes /platform/sun4u/kernel.test/unix loaded - 0xfe000 bytes used SunOS Release 5.9 Version Generic 32-bit Copyright 1983-2002 Sun Microsystems, Inc. All rights reserved. obpsym: symbolic debugging is available. Read 208377 bytes from misc/forthdebug configuring IPv4 interfaces: le0. Hostname: test Configuring /dev and /devices
After the system is booted, sending a break passes control to kadb(1M). A break is generated with STOP-A (on the console of SPARC machines), or with F1-A (on the console of IA machines), or by using ~# (if the console is connected through a tip window).
... The system is ready. test console login: ~# stopped at edd000d8: ta %icc,%g0 + 125 kadb[0]:
The number in brackets is the CPU that kadb(1M) is currently executing on; the remaining CPUs are halted. The CPU number is zero on a uniprocessor system.
Caution – Before rebooting or turning off the power, always halt the system cleanly (with init 0 or shutdown). Buffers might not be flushed otherwise. If the shutdown must occur from the boot PROM prompt, make sure to flush buffers using the sync command at the ok prompt.
To return control to the operating system, use :c.
kadb[0]: :c test console login:
Exiting kadb
To exit kadb(1M), use $q. On SPARC machines, this will exit to the ok prompt. On IA machines, you will be prompted to reboot the system.
kadb[0]: $q Type `go' to resume ok
kadb(1M) can be resumed by typing go at the ok prompt.
Caution – No other commands should be performed from the PROM if the system is to be resumed. PROM commands other than go can change system state that the Solaris 9 operating environment depends upon.
Staying at the kadb(1M) prompt for too long can cause the system to lose track of the time of day, and can cause network connections to time out.
kadb Command Usage
The general form of a kadb command is:
[ address ] [ ,count ] command [;]
If address is omitted, the current location is used (`.' could also be used to represent the current location). The address can be a kernel symbol. If count is omitted, it defaults to 1.
Commands to kadb consist of a verb followed by a modifier or list of modifiers. Verbs can be:
|
/ |
Prints locations starting at address in the kernel address space |
|
= |
Prints the value of address itself |
|
> |
Assigns a value to a debugger variable or machine register |
|
< |
Reads a value from a debugger variable or machine register |
|
RETURN |
Repeats the previous command with a count of 1. Increments `.' (the current location) |
With / and =, output format specifiers can be used. Lowercase letters normally print 2 bytes, uppercase letters print 4 bytes. The values of these specifiers are:
|
o, O |
2-, 4-byte octal |
|
g |
8-byte octal |
|
G |
8-byte unsigned octal |
|
d, D |
2-, 4-byte decimal |
|
e |
8-byte decimal |
|
E |
8-byte unsigned decimal |
|
x, X |
2-, 4-byte hexadecimal |
|
J |
8-byte hexadecimal |
|
K |
4–byte hexadecimal for 32–bit programs, 8–byte hexadecimal for 64–bit programs. Use this format specifier to examine pointers. |
|
u, U |
2-, 4-byte unsigned decimal |
|
c |
Prints the addressed character |
|
C |
Prints the addressed character using ^ escape notation |
|
s |
Prints the addressed string |
|
S |
Prints the addressed string using ^ escape notation |
|
i |
Prints as machine instructions (disassemble) |
|
a |
Prints the value of `.' in symbolic form |
|
w, W |
2-, 4-byte write |
|
Z |
8-byte write |
Caution – When using w, W or Z to modify a kernel variable, make sure that the size of the variable matches the size of the write you are performing. If you specify an incorrect size you could corrupt neighboring data.
For example, to set a bit in the moddebug variable when debugging a driver, first examine the value of moddebug, then set it to the desired bit.
kadb[0]: moddebug/X moddebug: moddebug: 1000 kadb[0]: moddebug/W 0x80001000 moddebug: 0x1000 = 0x80001000
Routines can be disassembled with the `i' command. This is useful when tracing crashes, since the only information might be the program counter at the time of the crash. For example, to print the first four instructions of the kmem_alloc function:
kadb[0]: kmem_alloc,4/i kmem_alloc: kmem_alloc: save %sp, -0x60, %sp sub %i0, 0x1, %l6 sra %l6, 0x3, %i5 tst %i5
Specify symbolic notation with the `a' command, to show the addresses:
kadb[0]: kmem_alloc,4/ai kmem_alloc: kmem_alloc: save %sp, -0x60, %sp kmem_alloc+4: sub %i0, 0x1, %l6 kmem_alloc+8: sra %l6, 0x3, %i5 kmem_alloc+0xc: tst %i5
Register Identifiers
You can discover what machine registers are available on your processor architecture using the $r command. This example shows the output of $r on a SPARC system with the sun4 architecture:
kadb[0]: $r
g0 0 l0 0
g1 100130a4 debug_enter l1 edd00028
g2 10411c00 tsbmiss_area+0xe00 l2 10449c90
g3 10442000 ti_statetbl+0x1ba l3 1b
g4 3000061a004 l4 10474400
ecc_syndrome_tab+0x80
g5 0 l5 3b9aca00
g6 0 l6 0
g7 2a10001fd40 l7 0
o0 0 i0 0
o1 c i1 10449e50
o2 20 i2 0
o3 300006b2d08 i3 10
o4 0 i4 0
o5 0 i5 b0
sp 2a10001b451 fp 2a10001b521
o7 1001311c debug_enter+0x78 i7 1034bb24
zsa_xsint+0x2c4
y 0
tstate: 1604 (ccr=0x0, asi=0x0, pstate=0x16, cwp=0x4)
pstate: ag:0 ie:1 priv:1 am:0 pef:1 mm:0 tle:0 cle:0 mg:0 ig:0
winreg: cur:4 other:0 clean:7 cansave:1 canrest:5 wstate:14
tba 0x10000000
pc edd000d8 edd000d8: ta %icc,%g0 + 125
npc edd000dc edd000dc: nop
kadb exports each of these registers as a debugger variable with the same name. Reading from the variable fetches the current value of the register. Writing to the variable changes the value of the associated machine register. For example, you can change the value of the '%o0' register:
kadb[0]: <o0=K
0
kadb[0]: 0x1>o0
kadb[0]: <o0=K
1
Display and Control Commands
The following commands display and control the status of kadb(1M):
|
$b |
Display all breakpoints |
|
$c |
Display stack trace |
|
$d |
Change default radix to value of dot |
|
$q |
Quit |
|
$r |
Display registers |
|
$M |
Display built-in macros |
`$c' is useful when a breakpoint is reached, but is usually not useful if kadb(1M) is entered at a random time. The number of arguments to print can be passed following the `$c' (`$c 2' for two arguments).
Breakpoints
In kadb(1M), breakpoints can be set. When reached, the kernel will automatically drop back into kadb. The standard form of a breakpoint command is:
[module_name#] addr [, count]:b [command]
addr is the address at which the program will be stopped and the debugger will receive control, count is the number of times that the breakpoint address occurs before stopping, and command is almost any adb(1) command.
The optional module_name specifies deferred breakpoints that are set when the module is loaded. module_name identifies a particular module that contains addr. If the module has been loaded, kadb will try to set a regular breakpoint; if the module is not loaded, kadb will set a deferred breakpoint. When the module is loaded, kadb will try to resolve the location of the breakpoint and convert the breakpoint to a regular breakpoint.
Other breakpoint commands are:
|
:c |
Continue execution |
|
:d |
Delete breakpoint |
|
:s |
Single step |
|
:e |
Single step, but step over function calls |
|
:u |
Stop after return to caller of current function |
|
:z |
Delete all breakpoints |
The following example sets a breakpoint in scsi_transport(9F), a commonly used routine. Upon reaching the breakpoint, '$c' is used to print a stack trace. The top of the stack is displayed first. Note that kadb does not know how many arguments were passed to each function.
stopped at edd000d8: ta %icc,%g0 + 125 kadb[0]: scsi_transport:b kadb[0]: :c test console login: root breakpoint at: scsi_transport: save %sp, -0x60, %sp kadb[0]: $c scsi_transport(702bb578,1000,1,10000,0,702bb7fe) sdstrategy(1019c8c0,702bb61c,0,0,702bb578,70cad7b8) + 704 bdev_strategy(1042a808,70cad7b8,705f3efc,40,10597900,2000) + 98 ufs_getpage_miss(70cad7b8,0,10597900,0,0,4023ba8c) + 2b0 ufs_getpage(0,0,0,0,2000,4023ba8c) + 7c0 segvn_fault(4023ba8c,2000,ff3b0000,0,0,0) + 7c8 as_fault(1,ff3b0000,70d98030,2000,0,ff3b0000) + 49c pagefault(0,0,70df8048,705c7450,0,ff3b0000) + 4c trap(10080,10000,ff3c4ea4,70df8048,ff3b0000,1) + db4 kadb[0]: $b breakpoints count bkpt type len command 1 scsi_transport :b instr 4 kadb[0]: scsi_transport:d kadb[0]: :c
Conditional Breakpoints
Breakpoints can also be set to occur only if a certain condition is met. By providing a command, the breakpoint will be taken only if the count is reached or the command returns zero. For example, a breakpoint that occurs only on certain I/O controls could be set in the driver's ioctl(9E) routine. This is the general syntax of conditional breakpoints:
address,count:b command
In this example, address is the address at which to set the breakpoint. count is the number of times the breakpoint should be ignored (note that 0 means break only when the command returns zero). command is the kadb(1M) command to execute.
Here is an example of breaking only in the sdioctl() routine if the DKIOGVTOC (get volume table of contents) I/O control occurs.
kadb[0]: sdioctl+4,0:b <i1-0x40B kadb[0]: $b breakpoints count bkpt type len command 0 sdioctl+4 :b instr 4 <i1-0x40B kadb[0]: :c
Adding four to sdioctl skips to the second instruction in the routine, bypassing the save instruction that establishes the stack. The `<i1' refers to the first input register, which is the second parameter to the routine (the cmd argument of ioctl(9E)). The count of zero is impossible to reach, so it stops only when the command returns zero, which is when `i1 - 0x40B' is true. This means i1 contains 0x40B (the value of the ioctl(9E) command, determined by examining the ioctl definition).
To force the breakpoint to be reached, the prtvtoc(1M) command, which is known to issue the following I/O control, is used:
# prtvtoc /dev/rdsk/c0t0d0s0 breakpoint at: sdioctl+4: mov %i5, %l0 kadb[0]: $c sdioctl(800000,40b,ffbefb54,100005,704a3ce8,4026bc7c) + 4 ioctl(3,40b,70ca27b8,40b,ffbefb54,0) + 1e0
kadbMacros
kadb(1M) supports macros that are used for displaying kernel data structures. kadb macros can be displayed with $M. Macros are used in the form:
[ address ] $<macroname
Note –
Neither the information displayed by these macros, nor the format in which the information is displayed constitute an interface. Therefore, the information and format can change at any time.
Device Macros
This section describes the kadb(1M) macros that can be used to retrieve information from the kernel about a device during a debugging session.
|
devinfo_brief |
Prints a brief summary of a dev_info_t structure; the name and addr fields are shown in string form and are used in the /devices path to the node |
|
devinfo |
Prints devinfo_brief information for a dev_info_t structure followed by an in-order dump of all fields |
|
devinfo.parent |
Prints devinfo_brief information for the specified node and its parents all the way back to the root of the devinfo tree |
|
devinfo.sibling |
Prints devinfo_brief information for all siblings of the specified node; you see all sibling nodes if you start at the parent's child |
|
devinfo.minor |
Prints the ddi_minor_data structure for all minor name nodes associated with the specified device node |
|
devinfo.prop |
Prints all the property lists associated with the specified device node |
|
devinfo_major |
Prints devinfo_brief information for all device nodes associated with the specified driver major number |
|
devt |
Breaks a dev_t into its major and minor parts and prints the values in hexadecimal and decimal |
|
devnames |
Prints the contents of the specified devnames structure |
|
devnames_major |
Prints the contents of the devnames structure for the specified major number |
|
bus_op |
Prints the contents of a bus_ops structure |
The devinfo commands, such as devinfo and devinfo_brief, return a node_state value, which may be one of the following:
- DS_LINKED
-
This device node has been linked into the kernel's device tree, but the system has not yet found a driver for it.
- DS_BOUND
-
The node is bound to a driver, but the driver's probe(9E) routine has not yet been called.
- DS_INITIALIZED
-
The parent nexus has assigned a bus address for the driver and completed implementation-specific initializations, but the driver's probe(9E) routine has not yet been called.
- DS_PROBED
-
The driver's probe(9E) routine returned successfully.
- DS_ATTACHED
-
The driver's attach(9E) routine returned successfully.
- DS_READY
-
The device is fully configured.
Examples: Use of Device Macros
The examples in this section show how some of the device-related macros can be used to debug a driver.
The following ls -l command shows how the /devices path fits into the devinfo structure:
# ls -l /dev/ttya
lrwxrwxrwx 1 root root 6 Aug 17 17:50 /dev/ttya -> term/a
# ls -l /dev/term/a
lrwxrwxrwx 1 root root 32 Aug 17 17:42 /dev/term/a ->
../../devices/obio/zs@0,100000:a
Example 18–1 shows how to obtain information about the /dev/ttya device from the devinfo structure and other kernel structures, beginning with devinfo_major.
Example 18–1 The devinfo_major Macro
# ls -lL /dev/ttya
crw-rw-rw- 1 root sys 29, 0 Aug 17 17:42 /dev/ttya
# ~stopped at:
edd000d8: ta %icc,%g0 + 125
kadb[0]: 0t29$<devinfo_major
============== devinfo f5f0dcd0
binding_name
0xf5bb38f8: zs
node_name
0xf5bb38f8: zs
addr
0xf5f8d6a0: 0,100000
node_state
6 DS_READY
major (hex)
1d
instance
0
0xf5f0dcd0: parent child sibling
f5f0f380 0 f5f0dba8
============== devinfo f5f0dba8
binding_name
0xf5bb3a18: zs
node_name
0xf5bb3a18: zs
addr
0xf5f90560: 0,0
node_state
6 DS_READY
major (hex)
1d
instance
1
0xf5f0dba8: parent child sibling
f5f0f380 0 f5f0da80
Example 18–2 shows the use of the devinfo_brief macro. It also shows the use of top_devinfo. top_devinfo is a pointer to the root of the devinfo tree (rootnexus)—all other devinfo nodes are descendants of this node. If you want to look at the device tree top-down in the debugger, this is where you start.
Example 18–2 The devinfo_brief Macro
kadb[0]: *top_devinfo$<devinfo_brief
============== devinfo f5f0fde8
binding_name
0xf5bc6a30: SUNW,SPARCstation-10
node_name
0xf5bc6af8: SUNW,SPARCstation-10
addr
0xf5f90950:
node_state
6 DS_READY
major (hex)
1
0xf5f0fde8: parent child sibling
0 f5f0fcc0 0
Thread Macros
threadlist is a useful macro that displays the stacks of all the threads in the system.
Example 18–3 The threadlist Macro
kadb[0]: $<threadlist
============== thread_id 10404000
p0+0x300:
process args sched
t0+0xa8: lwp procp wchan
1041b810 10424688 0
t0+0x24:
pc sp
sched+0x4f4 10403be8
?(10404000,1040c000,2,10424604,0,6e)
_start(10006ef4,1041adb0,1041adb0,1041adb0,10462910,50) + 15c
...
============== thread_id 40043e60
p0+0x300:
process args sched
40043f08: lwp procp wchan
0 10424688 10473c56
40043e84:
pc sp
cv_wait+0x60 40043c08
?(10473c56,10473c5c,0,40043cd0,40043e60,10093084)
ufs_thread_idle(10471e80,0,10473c5c,10424688,81010100,0) + bc
thread_start(0,0,0,0,0,0) + 4
...
Another useful macro is thread. Given a thread ID, this macro prints the corresponding thread structure. This can be used to look at a certain thread found with the threadlist macro, to look at the owner of a mutex, or to look at the current thread, as shown here:
Example 18–4 The thread Macro
kadb[0]: <g7$<thread
70e87ac0: link stk startpc
0 4026bc80 0
70e87acc: bound_cpu affinitycnt bind_cpu
0 0 -1
70e87ad4: flag proc_flag schedflag
0 4 3
70e87ada: preempt preempt_lk state
0 0 4
70e87ae0: pri epri
40 0
70e87ae4:
pc sp
10098350 4026b618
70e87aec: wchan0 wchan sobj_ops
0 0 0
70e87af8: cid clfuncs cldata
1 10470ffc 702c0488
70e87b04: ctx lofault onfault
0 0 0
...
Note –
No type information is maintained by kadb, so using a macro on an inappropriate address results in garbage output.
Macros do not necessarily output all the fields of the structures, nor is the output necessarily in the order given in the structure definition. Occasionally, memory needs to be dumped for certain structures and then matched with the structure definition in the kernel header files.
Caution – Drivers should never reference system header files or structures not listed in man pages section 9S: DDI and DKI Data Structures. However, examining non-DDI-compliant structures (such as thread structures) can be useful in debugging drivers.
kadb Output Pager
Some kadb commands (like $<threadlist) output lots of data, which can scroll off of the screen very rapidly. kadb provides a simple output pager to remedy this problem. The pager command is lines::more, where lines represents the number of lines to print before pausing the console output. Keep in mind that this does not take into account lines that wrap because they are wider than the terminal width. Here is an example usage:
kadb[0]: 0t10::more
kadb[0]: $<threadlist
============== thread_id 10408000
p0+0x4c0:
process args sched
t0+0x128: lwp procp wchan
10429ed0 104393e8 0
t0+0x38:
pc sp
sched+0x4e4 104071f1
?(10408000,10414c00,2,104393e8,10439308,0)
_start(10007588,104292e0,104292e0,104292e0,1043b8b0,10429360) + 200
============== thread_id 2a10001fd40
p0+0x4c0:
process args sched
--More-- <SPACE>
...
Pressing the space bar at the “--More--” prompt pages the output by the number of lines specified to ::more (in this case, 10). Pressing "Return" prints only the next line of output. You can abort the output and return to the kadb prompt by typing Ctrl-C. To disable the pager, issue '0::more' at the kadb prompt.
Example: kadb on a Deadlocked Thread
This example shows how kadb can be used to debug a driver bug. This example was taken from the development of the ramdisk sample driver. This driver exports physical memory as a virtual disk. In this case, the dd(1M) command hangs while trying to copy some data onto the device and cannot be aborted. Though a crash dump could be forced, for illustrative purposes, kadb(1M) will be used. After logging into the system remotely, ps was used to determine that the system was still running; and only the dd(1M) command is hung.
At this point, the system is rebooted with kadb, which can now be entered by typing STOP-A on the system console. After the rest of the kernel has loaded, moddebug is patched to see if loading is the problem:
stopped at: edd000d8: ta %icc,%g0 + 125 kadb[0]: moddebug/X moddebug: moddebug: 0 kadb[0]: moddebug/W 0x80000000 moddebug: 0x0 = 0x80000000 kadb[0]: :c
modload(1M) is used to load the driver, to separate module loading from the real access:
# modload /home/driver/drv/ramdisk
It loads without errors, so loading is not the problem. The condition is recreated with dd(1M):
# dd if=/dev/zero of=/devices/pseudo/ramdisk@0:c,raw
dd(1M) hangs. At this point, kadb(1M) is entered and the stack examined:
stopped at: edd000d8: ta %icc,%g0 + 125 kadb[0]: $c intr_vector() + 7dcfc0d8 debug_enter(0,0,10431e50,10,1,b0) + 78 zsa_xsint(80,7044a06c,44,7044a000,ff0113,0) + 278 zs_high_intr(7044a000,1,1,1042f78c,10424680,100949d0) + 20c sbus_intr_wrapper(704dfad4,0,702bd048,7029cec0,630,10260250) + 30 current_thread(4001fe60,1041a550,10424698,10424698,10150f08,0) + 180 idle(1040b6c0,0,0,1041a550,704d6a98,0) + 54 thread_start(0,0,0,0,0,0) + 4
The presence of idle on the current thread stack indicates that this thread is not the cause of the deadlock. To determine the deadlocked thread, the entire thread list is checked:
kadb[0]: $<threadlist
...
============== thread_id 70cef120
70c8b1c0:
process args dd if=/dev/zero of=/devices/pseudo/ramdisk@0:c,raw
70cef1c8: lwp procp wchan
70fa9080 70c8aec0 70691fc8
70cef144:
pc sp
sema_p+0x290 40313a78
?(70691fc8,10424680,1,1042b99c,10460f8c,70691fc8)
biowait(70691f60,1041a6c4,70691f60,70c385d0,40313bcc,705c73a0) + 8c
default_physio(1042e8fc,200,129,100,70eb5b54,705c73a0) + 3bc
write(2002,70aac1d0,70f9f9ac,200,4,200) + 23c
...
Of all the threads, only one has a stack trace which references the ramdisk driver. It seems that the process running dd(1M) is blocked in biowait(9F). biowait(9F)'s first parameter is a buf(9S) structure. The next step is to examine this structure:
kadb[0]: 70691f60$70691f60$
70691f60: flags forw back
204129 0 0
70691f6c: av_forw av_back bcount
0 0 512
70691fa0: bufsize error edev
0 0 1180000
70691f7c: un.b_addr _b_blkno resid
710e8000 0 0
70691f94: proc iodone vp
70c8aec0 0 0
70691f98: pages
0
The resid field is 0, which indicates that the transfer is complete. physio(9F) is still blocked, however. The reference for physio(9F) in the Solaris 9 Reference Manual Collection points out that biodone(9F) should be called to unblock biowait(9F). This is the problem; rd_strategy() did not call biodone(9F). Adding a call to biodone(9F) before returning fixes this problem.
- © 2010, Oracle Corporation and/or its affiliates