kmdb - in situ kernel debugger
ok boot [device-specifier] -k | -K [-d] [boot-flags]
ok boot [device-specifier] kmdb [-d] [boot-flags]
kernel$ /platform/i86pc/kernel/$ISADIR/unix -K [-d] [boot-flags]
kmdb is an interactive kernel debugger which implements the user interface and functionality of mdb(1) in a live kernel context. kmdb provides features that allow for the control of kernel execution and for the inspection and modification of live kernel state. kmdb can be loaded at the beginning of a boot session or after the system is booted.
This man page describes the features and functionality that are unique to kmdb or different in kmdb as compared to mdb(1). For more information on mdb(1) or further details on the features and functionality implemented by kmdb, see the mdb(1) man page and the Oracle Solaris Modular Debugger Guide.
When requested, the kernel runtime linker (krtld) loads kmdb prior to the transfer of control to the kernel. If the –d flag is used, the debugger gains control of the system prior to the execution of the initial function in the unix object. If –d is not used, kmdb is loaded but does not gain control until such time as it is explicitly entered. See the Debugger Entry section below. For a list of the boot commands which cause kmdb to be loaded at boot, see the SYNOPSIS section above. See eeprom(8) for an example of the use of that command on a SPARC machine to specify that kmdb is always loaded upon boot.
Boot-loaded kmdb can be unloaded only by means of a system reboot.
Some features of kmdb rely on the presence of kernel services and are not immediately available to boot-loaded kmdb. In particular, the loading and unloading of dmods is not available until the module subsystem is initialized. Requests are queued until they can be processed. Similarly, translation of virtual addresses to physical addresses is not be available until the VM system has been initialized. Attempted translations fail until translation facilities are available.
kmdb can also be loaded after the system has booted, using the –K flag to mdb(1). When loaded in this fashion, it will immediately gain control of the system. Run-time-loaded kmdb can be unloaded using the –U flag to mdb(1) or from within the debugger with the –u flag to the ::quit dcmd.
When loaded, kmdb attempts to determine the proper terminal type in use on the system console. If the system being debugged has an attached keyboard and local display that are both used for the system console, kmdb uses the terminal type appropriate for the machine: 'sun' for SPARC; 'sun-color' for x86. When a serial console is in use, boot-loaded kmdb defaults to a terminal type 'vt100'. Run-time-loaded kmdb defaults to the terminal type requested by mdb(1). mdb(1) requests the terminal type specified by the value of the TERM environment variable unless overridden by the –T flag. ::term can be used to view the current terminal type.
Debugger entry can be requested explicitly or implicitly. Implicit entry, encountered when breakpoints or other execution control features are used, is discussed in the Execution Control section.
The primary means for explicit debugger entry is with the keyboard abort sequence for systems with local consoles and the BREAK character for those with serial consoles. The abort sequence is STOP-A or Shift-Pause for SPARC systems with local consoles, and F1–A or Shift-Pause for x86 systems with local consoles. See kbd(1) for a discussion of the abort sequence and for instructions on disabling it.
If the kernel panics and kmdb is loaded, by default, the panic routine enters kmdb for live debugging. If a dump device is specified, and you enter ::cont, the debugger exits and a crash dump is performed. To prevent the kernel from entering kmdb when panicking, you can set the nopanicdebug variable to 1. Set the nopanicdebug variable to 1 using kmdb or including the following a line in /etc/system:
set nopanicdebug = 1
This can be useful if you want to keep kmdb loaded, but always want a panic to trigger a crash dump without entering the debugger.
For boot-loaded kmdb, this behaviour can be modified by using the –K option of the boot command. This prevents the initial entry to kmdb, if there is a valid dump device configuration. However, if there is no dump device and deferred dump is not online, the system will enter kmdb at panic time to allow for debugging where the dump would otherwise be lost. For more information, see the boot(8) man page.
For the most part, the execution control facilities provided by the kmdb command for the kernel, mirror those provided by the mdb(1) process target. Breakpoints (::bp), watchpoints (::wp), ::continue, and the various flavors of ::step can be used.
In contrast to the unlimited user process watchpoints supplied by the kernel, kmdb is restricted to a set of CPU watchpoints that limit the number, size, and type of watchpoints allowed. The ::wp command does not allow a watchpoint to be created, if it is incompatible with the watchpoints supported by the hardware.
The thread spin functionality is also available in kmdb, to debug and reproduce the race, or hang conditions where the race window is very narrow. The thread spin functionality allows the kmdb command to instruct the thread(s) to go into a spin state at the desired point.
As with mdb(1), kmdb is installed with a number of subsystem-specific debugger modules, or dmods. The dmods are loaded and unloaded automatically with the loading and unloading of the subsystems that they support. The dmods can also be explicitly loaded and unloaded using ::load and ::unload.
kmdb uses kernel facilities to load and unload dmods and must resume system execution to perform each requested action. When a dmod load or unload is complete, the system is stopped and the debugger is automatically re-entered. For a dmod load, processing is completed when the load of a requested dmod succeeds or fails. Status messages are provided in either case.
Some functionality is specific to an individual processor type. An example of such functionality is the branch tracing provided by various x86 processors. Access to these processor-specific features is provided with processor-specific dcmds that are present only on systems that support them. The availability of processor-specific support is indicated in the output of the ::status dcmd. The debugger relies on the kernel to determine the processor type. Even though the debugger might provide support for a given processor type, the support is not exposed until the kernel has progressed to the point at which processor identification has completed.
The debugger provides access to a set of macros that are precompiled into the debugger. Only the precompiled macros are available . Unlike with mdb(1), the $< dcmd may not be used to load macros from arbitrary locations. Use the $M command to list the available macros.
This section lists dcmds that are unique to kmdb or those with behavior that differs in kmdb as compared to mdb(1).
Set a breakpoint at the specified locations. The ::bp dcmd sets a breakpoint at each address or symbol specified, including an optional address specified by an explicit expression preceding the dcmd, and each string or immediate value following the dcmd. The arguments can be symbol names or immediate values denoting a particular virtual address of interest.
If a symbol name is specified, the name may refer to a symbol that cannot yet be evaluated. It might consist of an object name and function name in a load object that has not yet been opened. In such a case, the breakpoint is deferred and is not active in the target until an object matching the given name is loaded. The breakpoint is automatically enabled when the load object is opened.
The –d, –D, –e, –s, –t, –T, –c, and –n options have the same meaning as they do for the ::evset dcmd. See mdb(1) for a description of ::evset. If the :b form of the dcmd is used, a breakpoint is set only at the virtual address specified by the expression preceding the dcmd. The arguments following the :b dcmd are concatenated together to form the callback string. If this string contains meta-characters, it must be quoted.
Display the last branches taken by the CPU. This dcmd is supported only on x86 systems, and is available only when processor-specific support is detected and enabled. The number and type of branches displayed is dependent on the capabilities of the branch tracing facilities provided by the CPU. When the –v option is used, the instructions prior to a given branch are displayed.
Call the specified function using the specified arguments. The called function must be listed as a function in the symbol table for a loaded module. String arguments are passed by reference. When the call completes, the return value of the function is displayed.
This dcmd must be used with extreme caution. The kernel will not be resumed when the call is made. The function being called may not make any assumptions regarding the availability of any kernel services, and must not perform operations or calls that may block. The user must also beware of any side-effects introduced by the called function, as kernel stability might be affected.
Display the current general purpose register set for the specified CPU, in the format used by ::regs.
Print a C stack backtrace for the specified CPU. The backtrace displayed is for the point at which the specified CPU entered or was stopped by the debugger.
Read len bytes from the I/O port specified by addr. The value of the –L option, if provided, takes precedence over the value of the repeat count. The read length must be 1, 2, or 4 bytes, and the port address must have the same alignment as the length.
Write value to the len-byte I/O port specified by addr. The value of the –L option, if provided, takes precedence over the value of the repeat count. The write length must be 1, 2, or 4 bytes and the port address must have the same alignment as the length.
Causes the debugger to exit. When the –u option is used, the system is resumed and the debugger is unloaded. The –u option may not be used if the debugger was loaded at boot. When the –u option is not used, SPARC systems will exit to the boot PROM ok prompt. The go command can be used to re-enter the debugger. On x86 systems, a prompt is displayed that requests permission to reboot the machine.
Step the target one instruction. The optional over argument is used to step over subroutine calls. When the optional out argument is specified, the target program continues until control returns from the current function.
The optional branch argument is available only on x86 systems when processor-specific support is detected and enabled. When ::step branch is specified, the target program continues until the next branching instruction is encountered.
On SPARC systems, the ::step dcmd may not be used to step 'ta' instructions. Similarly, it may not be used on x86 systems to step 'int' instructions. If the step results in a trap that cannot be resolved by the debugger, a message to that effect is printed and the step will fail.
Controls the target thread spinning. The arm subcommand can be used to send the thread which hit the breakpoint to a spinning state. Manual invocation of the resume subcommand is needed to resume the spinning thread, while the list subcommand lists all spinning threads.
Use the specified CPU as the representative. Stack traces, general purpose register dumps, and similar functionality use the new representative CPU as the data source. Full execution control functionality is available on the new representative CPU.
Display the current terminal type.
Set a watchpoint at the specified address, interpreted by default as a virtual address. If the –p option is used, the address is interpreted as a physical address. On x86 platforms, watchpoints can be set on I/O ports using the –i option. When the –i option is used, the address is interpreted as that of an I/O port.
The length in bytes of the watched region can be set by specifying an optional repeat count preceding the dcmd. If no length is explicitly set, the default is one byte. The ::wp dcmd allows the watchpoint to be configured to trigger on any combination of read (–r option), write (–w option), or execute (–x option) access.
The –d, –D, –e, –s, –t, –T, –c, and –n options have the same meaning as they do for the ::evset dcmd. See mdb(1) for a description of ::evset. The :a dcmd sets a read access watchpoint at the specified address. The :p dcmd sets an execute access watchpoint at the specified address. The :w dcmd sets a write access watchpoint at the specified address. The arguments following the :a, :p, and :w dcmds are concatenated together to form the callback string. If the string contains meta-characters, it must be quoted.
See attributes(7) for descriptions of the following attributes:
The memory region available to the debugger is allocated when the debugger is loaded, and is fixed at that point. If dcmds attempt to allocate more memory than is available, they will, if possible, be terminated. The debugger will attempt to recover gracefully from an out-of-memory situation, but may be unable to, and may be forced to terminate the system. This constraint is especially acute on 32-bit x86 systems.
System performance will be negatively impacted by the loading of kmdb, as the debugger will consume kernel memory and other limited system resources.
To troubleshoot a panic() on a SPARC machine, it can be useful to use eeprom(8) to specify that the system always load kmdb upon booting. Following a panic, the system starts to reboot, in so doing clearing the panic stack from the console. By booting into kmdb, one can capture and interpret the panic stack. See eeprom(8) for an example of specifying that kmdb load upon boot.