adb and kadb
adb(1) can be used to debug applications or the kernel, though it cannot debug the kernel interactively (such as by setting breakpoints). To interactively debug the kernel, use kadb(1M). Both adb(1) and kadb(1M) share a common command set.
Starting adb
The command for starting adb(1) to debug a kernel core dump is:
% adb -k /var/crash/`hostname`/unix.n /var/crash/`hostname`/vmcore.n
Note -
For best results, use adb on the same architecture (such as Sun4m) that generated the core image.
To start adb(1) on a live system, type (as root):
# adb -k /dev/ksyms /dev/mem
/dev/ksyms is a special driver that provides an image of the kernel's symbol table to adb(1).
When < adb(1) responds with physmem xxx, it is ready for a command.
Note -
If the -p option of adb is used, an input prompt is displayed.
Starting kadb
The system must be booted under kadb(1M) before kadb(1M) can be used.
ok boot kadb ... Boot device: /sbus/esp@0,800000/sd@3,0 File and args: kadb kadb: kernel/unix Size: 191220+114284+12268 Bytes /platform/SUNW,Sun_4_75/kernel/unix loaded - 0x70000 bytes used SunOS Release 5.7 Version Generic [UNIX(R) System V Release 4.0] Copyright (c) 1983-1998, Sun Microsystems, Inc. ...
By default, kadb(1M) boots (and debugs) kernel/unix. It can be passed a file name as an argument to boot a different kernel, or -d can be passed to have kadb(1M) prompt for the kernel name. The -d flag also causes kadb(1M) to provide a prompt after it has loaded the kernel, so breakpoints can be set.
ok boot kadb -d ... Boot device: /sbus/esp@0,800000/sd@3,0 File and args: kadb -d kadb: kernel/unix kadb: kernel/unix Size: 191220+114284+12268 Bytes /platform/SUNW,Sun_4_75/kernel/unix loaded - 0x70000 bytes used kadb[0]:
Note -
Modules are dynamically loaded. Consequently, driver symbols are not generally available until the driver is loaded. To set breakpoints in modules that have not been loaded, use deferred breakpoints. For information on deferred breakpoints, see "Breakpoints".
At this point you can set breakpoints or continue with the :c command.
kadb(1M) passes any kernel flags to the booted kernel. For example, the flags -r, -s, and -a can be passed to kernel/unix with the command:
boot kadb -ras
Once the system is booted, sending a break passes control to kadb(1M). A break is generated with L1+A (on the console of SPARC machines) , or by Crtl+Alt+D (on the console of x86 machines) or ~# (if the console is connected through a tip window).
... The system is ready. test console login: ~stopped at 0xfbd01028: ta 0x7d kadb[0]:
The number in brackets is the CPU that kadb(1M) is currently executing on; the remaining CPUs are halted. The CPU number is zero on a uniprocessor.
Caution - Before rebooting or turning off the power, always halt the system cleanly (with init 0 or shutdown). Buffers may not be flushed otherwise. If the shutdown must occur from the boot PROM prompt, make sure to flush buffers with sync.
To return control to the operating system, use :c.
kadb[0]: :c test console login:
Exiting
To exit either adb(1M) or kadb(1M), use $q.
kadb[0]: $qType `go' to resume ok
On SPARC machines, kadb(1M) can be resumed by typing go at the ok prompt. On x86 machines, kadb(1M) cannot be resumed.
Caution - No other commands can be performed from the PROM if the system is to be resumed. PROM commands other than go may change system state that the Solaris 7 operating environment depends upon.
Staying at the kadb(1M) prompt for too long may cause the system to lose track of the time of day, and cause network connections to time out.
Commands
The general form of an adb(1M) or kadb(1M) command is:
[ address ] [ ,count ] command [;]
If address is omitted, the current location is used (`.' could also be used to represent the current location). The address can be a kernel symbol. If count is omitted, it defaults to 1.
Commands to adb consist of a verb followed by a modifier or list of modifiers. Verbs can be:
|
? |
Prints locations starting at address in the executable. |
|
/ |
Prints locations starting at address in the core file. |
|
= |
Prints the value of address itself. |
|
> |
Assigns a value to a variable or register. |
|
< |
Reads a value from a variable or register. |
|
RETURN |
Repeats the previous command with a count of 1. Increments `.' (the current location). |
With ?, /, and =, output format specifiers can be used. Lowercase letters normally print 2 bytes, uppercase letters print 4 bytes:
|
o, O |
2-, 4-byte octal |
|
g |
8-byte octal |
|
G |
8-byte unsigned octal |
|
d,D |
2-, 4-byte decimal |
|
e |
8-byte decimal |
|
E |
8-byte unsigned decimal |
|
x,X |
2-, 4-byte hexadecimal |
|
J |
8-byte hexadecimal |
|
K |
4-byte hexadecimal for 32-bit programs, 8-byte hexadecimal for 64-bit programs |
|
u,U |
2-, 4-byte unsigned decimal |
|
f,F |
4-, 8-byte floating point |
|
c |
Prints the addressed character. |
|
C |
Prints the addressed character using ^ escape notation. |
|
s |
Prints the addressed string. |
|
S |
Prints the addressed string using ^ escape notation. |
|
i |
Prints as machine instructions (disassemble). |
|
a |
Prints the value of `.' in symbolic form. |
|
w,W |
2-, 4-byte write |
Note -
Understand exactly what sizes the objects are, and what effects changing them might have, before making any changes.
For example, to set a bit in the moddebug variable when debugging the driver, first examine the value of moddebug, then set it to the desired bit.
kadb[0]: moddebug/Xmoddebug: moddebug: 1000 kadb[0]: moddebug/W 0x80001000moddebug: 0x1000 = 0x80001000
Routines can be disassembled with the `i' command. This is useful when tracing crashes, since the only information may be the program counter at the time of the crash. For example, to print the first four instructions of the kmem_alloc function:
kadb[0]: kmem_alloc,4?i kmem_alloc: kmem_alloc: save %sp, -0x60, %sp sub %i0, 0x1, %l6 sra %l6, 0x3, %i5 tst %i5
To show the addresses also, specify symbolic notation with the `a' command.
kadb[0]: kmem_alloc,4?ai kmem_alloc: kmem_alloc: save %sp, -0x60, %sp kmem_alloc+4: sub %i0, 0x1, %l6 kmem_alloc+8: sra %l6, 0x3, %i5 kmem_alloc+0xc: tst %i5
Register Identifiers
Machine or kadb(1M) internal registers are identified with the `<` command, followed by the register of interest. On SPARC machines, the following register names are recognized:
|
. |
dot, the current location |
|
i0-7 |
Input registers to current function |
|
o0-7 |
Output registers for current function |
|
l0-7 |
Local registers |
|
g0-7 |
Global registers |
|
psr |
Processor Status Register |
|
tbr |
Trap Base Register |
|
wim |
Window Invalid Mask |
For more information on how these registers are normally used, see the System V Application Binary Interface, SPARC Processor Supplement.
On x86 machines, the following register names are recognized:
|
ebp |
Stack frame base register |
|
esp |
Stack pointer |
|
kesp |
Kernel stack pointer |
|
cs |
Code segment |
|
ds |
Data segment |
|
ss |
Stack segment |
|
eip |
Program pointer or instruction pointer |
|
efl |
Status flags register |
Note -
The remaining examples in this chapter are for use on SPARC machines only. For specific register information relating to x86 machines, see the System V Application Binary Interface, x86 Processor Supplement.
The following command displays the PSR as a 4-byte hexadecimal value:
kadb[0]: <psr=X
400cc3
Display and Control Commands
The following commands display and control the status of adb(1)/ kadb(1M):
|
$b |
Display all breakpoints |
|
$c |
Display stack trace |
|
$d |
Change default radix to value of dot |
|
$q |
Quit |
|
$r |
Display registers |
|
$M |
Display built-in macros |
`$c' is useful with crash dumps: it shows the call trace and arguments at the time of the crash. It is also useful in kadb(1M) when a breakpoint is reached, but is usually not useful if kadb(1M) is entered at a random time. The number of arguments to print can be passed following the `$c' (`$c 2' for two arguments).
Breakpoints
In kadb(1M), breakpoints can be set that will automatically drop back into kadb when reached. The standard form of a breakpoint command is:
[module_name#] addr [, count]:b [command]
addr is the address at which the program will be stopped and the debugger will receive control, count is the number of times that the breakpoint address occurs before stopping, and command is almost any adb(1) command.
The optional module_name specifies deferred breakpoints that are set when the module is loaded. module_name identifies a particular module that contains addr. If the module has been loaded, kadb will try to set a regular breakpoint; if the module is not loaded, kadb will set a deferred breakpoint. When the module is loaded, kadb will try to resolve the location of the breakpoint and convert the breakpoint to a regular breakpoint.
Other breakpoint commands are:
|
:c |
Continue execution |
|
:d |
Delete breakpoint |
|
:s |
Single step |
|
:e |
Single step, but step over function calls |
|
:u |
Stop after return to caller of current function |
|
:z |
Delete all breakpoints |
The following example sets a breakpoint in scsi_transport(9F), a commonly used routine. Upon reaching the breakpoint, `$c' is used to get a stack trace. The top of stack is the first function printed. Note that kadb(1M) does not know how many arguments were passed to the function; it always prints six.
kadb[0]: scsi_transport:bkadb[0]: :c test console login: root Password: breakpoint at: scsi_transport: save %sp, -0x60, %sp kadb[0]: $c scsi_transport(0xf5ad0d78,0x0,0x5,0xf5ad0e1c,0xf5ad0ec8,0x0) sdstrategy(0xf5d4b8a8,0xf5ad0e1c,0x0,0x2c1d0,0xf5ad0d78,0x0) + 3c0 bdev_strategy(0xf5d4b8a8,0xf5cd3dcc,0x8,0x40,0xf0319e60,0x1000) + d0 ufs_getpage_miss(0x0,0xf6133840,0x0,0x0,0xfbecb8e8,0x1) + 23c ufs_getpage(0x0,0x1000,0x0,0x0,0x0,0x1) + 640 segmap_fault(0xf5b34000,0xf5d76f98,0x0,0x1000,0x0,0x1) + 120 segmap_getmapflt(0xf0ba4000,0xf0ba4000,0x10000,0x0,0x6c267,0x0) + 4b0 rdip(0xf61337b8,0x0,0xf6133840,0xf5a74938,0x0,0xf5cda000) + 328 ufs_read(0xf61338a4,0xfbecbaf0,0x0,0xf5a74938,0xf5ca7e80,0xf61337b8) + 118 read(0x4) + 274 syscall_trap(?) + 18c Syssize(0x4) + d3c0 Syssize(0x26ad4,0x27400,0x0,0xef66855c,0xef661f58,0x11e78) + d2bc Syssize(0x27b88,0x27400,0xeffffe74,0x26968,0x27f94,0x27a84) + bc28 kadb[0]: :s stopped at: scsi_transport+4: mov %i0, %i1 kadb[0]: $b breakpoints count bkpt type len command 1 scsi_transport :b instr 4 kadb[0]: scsi_transport:d kadb[0]: :c
Conditional Breakpoints
This is the general syntax of conditional breakpoints:
address,count:b command
In this example, address is the address at which to set the breakpoint. count is the number of times the breakpoint should be ignored (note that 0 means break only when the command returns zero). command is the adb(1) command to execute.
Breakpoints can also be set to occur only if a certain condition is met. By providing a command, the breakpoint will be taken only if the count is reached or the command returns zero. For example, a breakpoint that occurs only on certain I/O controls could be set in the driver's ioctl(9E) routine. Here is an example of breaking only in the sdioctl() routine if the DKIOGVTOC (get volume table of contents) I/O control occurs.
kadb[0]: sdioctl+4,0:b <i1-0x40Bkadb[0]: $bbreakpoints count bkpt command 0 sdioctl+4 <i1-0x40B kadb[0]: :c
Adding four to sdioctl skips to the second instruction in the routine, bypassing the save instruction that establishes the stack. The `<i1' refers to the first input register, which is the second parameter to the routine (the cmd argument of ioctl(9E)). The count of zero is impossible to reach, so it stops only when the command returns zero, which is when `i1 - 0x40B' is true. This means i1 contains 0x40B (the value of the ioctl(9E) command, determined by examining the ioctl definition).
To force the breakpoint to be reached, the prtvtoc(1M) command is used. It is known to issue this I/O control:
# prtvtoc /dev/rdsk/c0t3d0s0breakpoint sdioctl+4: st %i5, [%fp + 0x58] breakpoint sdioctl+4: mov %i0, %o0 kadb[0]: $csdioctl(0x800018,0x40b,0xeffffc04,0x5,0xff34dc68,0xf03f0c00) + 4 ioctl(0xf03f0c90,0xf03f0c00,0x3,0x18,0xff445f5c,0xff7a09e0) + 270 syscall_ap(0x3) + 6c syscall_trap(?) + 150 Syssize(0x3) + 20458 Syssize(0x3,0x22f70,0xeffffc04,0x1,0x4,0xefffff6c) + fc14 Syssize(0xefffff6c,0xffffffff,0x1,0x1,0x3,0x22f70) + f5b4 Syssize(0x2,0xeffffec4,0xeffffed0,0x22c00,0x0,0x1) + ebe4
kadb(1M) cannot always determine where the bottom of the stack is. In the previous example, the call to Syssize is not part of the stack.
Macros
adb(1) and kadb(1M) support macros. adb(1) macros are in /usr/lib/adb and /usr/platform/`uname -i`/lib/adb for 32-bit programs and in /usr/lib/adb/sparcv9 and /usr/platform/`uname -i`/lib/adb/sparcv9 for 64-bit programs. kadb(1M) macros are builtin and can be displayed with $M. Most of the existing macros are for private kernel structures. New macros for adb can be created with adbgen(1M).
Macros are used in the form:
[ address ] $<macroname
threadlist is a useful macro that displays the stacks of all the threads in the system. Because this macro does not take an address and can generate excessive output, be ready to use Control-S and Control-Q to start or stop if necessary--this is another good reason to use a tip window. Control-C can be used to abort the listing.
kadb[0]: $<threadlist
============== thread_id f0244020
p0:
p0: process args= sched
t0:
t0: lwp proc wchan
f02a3e78 f02a8078 0
t0+0x34: sp pc
f0243af0 sched+0x4e0
?(?) + 0
main(0x0,0xf02b7e20,0xf02a800c,0x0,0x6e,0x0)
afsrbuf(?) + 1b8
exitto(0xf0040000,0xf02a0f58,0x3c,0xf02d3c40,0xfbd7d668,0x0)
============== thread_id fbe01ea0
p0:
p0: process args= sched
0xfbe01ea0: lwp proc wchan
0 f02a8078 0
0xfbe01ed4: sp pc
fbe1fe68 debug_enter+0xb0
?(?) + 0
abort_sequence_enter(0x0,0x740000,0xc4,0x0,0x0,0x40000e7)
zsa_xsint(0xf5dd9000,0x80,0xf5dd906c,0x44,0xff0113,0x0) + 244
zs_high_intr(0xf5dd9000) + 204
_level1(0x0) + 13bc
idle(0xf02a0fac,0x0,0xf02a8078,0xf02a8078,0xf004684c,0x4400ae1) + 50
============== thread_id fbe22ea0
p0:
p0: process args= sched
0xfbe22ea0: lwp proc wchan
0 f02a8078 f5caf024
0xfbe22ed4: sp pc
fbe22d80 cv_wait+0x64
?(?) + 0
cv_wait(0xf5caf024,0xf5caf010,0xffdf1175,0x6ea7e,0xf5caf030,0x40000000)
callout_thread(0xf5caf010,0xf5caf024,0xf02a8078,0xf02a8078,0x1000,0x4400ae5)+38
...
Another useful macro is thread. Given a thread ID, this macro prints the corresponding thread structure. This can be used to look at a certain thread found with the threadlist macro, to look at the owner of a mutex, or to look at the current thread.
kadb[0]: <g7$<thread
0xfbe01ea0:0xfbe01ea0: link stk startpc bound
0 fbe01e40 f0076d40 f02a0fac
0xfbe01eb0: affcnt bind_cpu flag procflag
1 -1 8 0
0xfbe01eb8: schedflag preempt preempt_lk state
3 1 0 4
0xfbe01ec0: pri epri pc sp
-1 0 f005bdf8 fbe1fe68
0xfbe01ecc: wchan0 wchan sobj_ops cid
0 0 0 0
0xfbe01edc: clfuncs cldata ctx lofault
f02aeed8 0 0 0
0xfbe01eec: onfault nofault swap lock
0 0 fbe01000 ff
0xfbe01efa: delay_cv cpu intr did
0 f02a0fac 0 1
0xfbe01f08: tnf_tpdp tid alarmid
f5a75c80 0 0 realitimer
0xfbe01f14: interval.sec interval.usec value.sec value.usec
0 0 0 0
0xfbe01f24: itimerid sigqueue sig
0 0 0 0
0xfbe01f34: hold forw back
0 0 0 0
0xfbe01f44: lwp procp next prev
0 f02a8078 fbe04ea0 f0244020
0xfbe01f58: trace why what dslot pollstate
0 0 0 0 0
0xfbe01f68: cred lbolt sysnum pctcpu
0 0 0 0
0xfbe01f88: lockp oldspl pre_sys disp_queue
f02a1014 e0 0 f02a0ff0
...
Note -
No type information is kept in the kernel, so using a macro on an inappropriate object results in garbage output.
Macros do not necessarily output all the fields of the structures, nor is the output necessarily in the order given in the structure definition. Occasionally, memory may need to be dumped for certain structures and then matched with the structure definition in the kernel header files.
Caution - Drivers should never reference header files and structures not listed in Section 9S of the Solaris 2.7 Reference Manual. However, examining non-DDI-compliant structures (such as thread structures) can be useful in debugging drivers.
- © 2010, Oracle Corporation and/or its affiliates