Part I Designing Device Drivers for the Solaris Platform
1. Overview of Solaris Device Drivers
2. Solaris Kernel and Device Tree
5. Managing Events and Queueing Tasks
7. Device Access: Programmed I/O
10. Mapping Device and Kernel Memory
14. Layered Driver Interface (LDI)
Part II Designing Specific Kinds of Device Drivers
15. Drivers for Character Devices
18. SCSI Host Bus Adapter Drivers
19. Drivers for Network Devices
Part III Building a Device Driver
21. Compiling, Loading, Packaging, and Testing Drivers
22. Debugging, Testing, and Tuning Device Drivers
Enable the Deadman Feature to Avoid a Hard Hang
Testing With a Serial Connection
To Set Up the Host System for a tip Connection
Setting Up a Target System on the SPARC Platform
Setting Up a Target System on the x86 Platform
Loading and Unloading Test Modules
Setting kmem_flags Debugging Flags
Avoiding Data Loss on a Test System
To Boot With an Alternate Kernel
Consider Alternative Back-Up Plans
Recovering the Device Directory
Using the kmdb Kernel Debugger
Booting kmdb With an Alternate Kernel on the SPARC Platform
Booting kmdb With an Alternate Kernel on the x86 Platform
kmdb Macros for Driver Developers
Using the mdb Modular Debugger
Getting Started With the Modular Debugger
Useful Debugging Tasks With kmdb and mdb
Exploring System Registers With kmdb
Writing Debugger Commands With mdb
Obtaining Kernel Data Structure Information
Obtaining Device Tree Information
Kernel Statistics Structure Members
Kernel Statistics for Solaris Ethernet Drivers
DTrace for Dynamic Instrumentation
23. Recommended Coding Practices
B. Summary of Solaris DDI/DKI Services
C. Making a Device Driver 64-Bit Ready
This section describes two debuggers that can be applied to device drivers. Both debuggers are described in detail in the Oracle Solaris Modular Debugger Guide.
The kmdb(1) kernel debugger provides typical runtime debugger facilities, such as breakpoints, watch points, and single-stepping. The kmdb debugger supersedes kadb, which was available in previous releases. The commands that were previously available from kadb are used in kmdb, in addition to new functionality. Where kadb could only be loaded at boot time, kmdb can be loaded at any time. The kmdb debugger is preferred for live, interactive debugging due to its execution controls.
The mdb(1) modular debugger is more limited than kmdb as a real-time debugger, but mdb has rich facilities for postmortem debugging.
The kmdb and mdb debuggers mostly share the same user interface. Many debugging techniques therefore can be applied with the same commands in both tools. Both debuggers support macros, dcmds, and dmods. A dcmd (pronounced dee-command) is a routine in the debugger that can access any of the properties of the current target program. A dcmd can be dynamically loaded at runtime. A dmod, which is short for debugger module, is a package of dcmds that can be loaded to provide non-standard behavior.
Both mdb and kmdb are backward-compatible with legacy debuggers such as adb and kadb. The mdb debugger can execute all of the macros that are available to kmdb as well as any legacy user-defined macros for adb. See the Solaris Modular Debugger Guide for information about where to find standard macro sets.
Postmortem analysis offers numerous advantages to driver developers. More than one developer can examine a problem in parallel. Multiple instances of the debugger can be used simultaneously on a single crash dump. The analysis can be performed offline so that the crashed system can be returned to service, if possible. Postmortem analysis enables the use of user-developed debugger functionality in the form of dmods. Dmods can bundle functionality that would be too memory-intensive for real-time debuggers, such as kmdb.
When a system panics while kmdb is loaded, control is passed to the debugger for immediate investigation. If kmdb does not seem appropriate for analyzing the current problem, a good strategy is to use :c to continue execution and save the crash dump. When the system reboots, you can perform postmortem analysis with mdb on the saved crash dump. This process is analogous to debugging an application crash from a process core file.
Note - In earlier versions of the Solaris operating system, adb(1) was the recommended tool for postmortem analysis. In the current Solaris operating system, mdb(1) is the recommended tool for postmortem analysis. The mdb() feature set surpasses the set of commands from the legacy crash(1M) utility. The crash utility is no longer available in the Solaris operating system.
The kmdb debugger is an interactive kernel debugger that provides the following capabilities:
Control of kernel execution
Inspection of the kernel state
Live modifications to the code
This section assumes that you are already familiar with the kmdb debugger. The focus in this section is on kmdb capabilities that are useful in device driver design. To learn how to use kmdb in detail, refer to the kmdb(1) man page and to the Oracle Solaris Modular Debugger Guide. If you are familiar with kadb, refer to the kadb(1M) man page for the major differences between kadb and kmdb.
The kmdb debugger can be loaded and unloaded at will. Instructions for loading and unloading kmdb are in the Solaris Modular Debugger Guide. For safety and convenience, booting with an alternate kernel is highly encouraged. The boot process is slightly different between the SPARC platform and the x86 platform, as described in this section.
Note - By default, kmdb uses the CPU ID as the prompt when kmdb is running. In the examples in this chapter [0] is used as the prompt unless otherwise noted.
Use either of the following commands to boot a SPARC system with both kmdb and an alternate kernel:
boot kmdb -D kernel.test/sparcv9/unix boot kernel.test/sparcv9/unix -k
Use either of the following commands to boot an x86 system with both kmdb and an alternate kernel:
b kmdb -D kernel.test/unix b kernel.test/unix -k
Use the bp command to set a breakpoint, as shown in the following example.
Example 22-7 Setting Standard Breakpoints in kmdb
[0]> myModule`myBreakpointLocation::bp
If the target module has not been loaded, then an error message that indicates this condition is displayed, and the breakpoint is not created. In this case you can use a deferred breakpoint. A deferred breakpoint activates automatically when the specified module is loaded. Set a deferred breakpoint by specifying the target location after the bp command. The following example demonstrates a deferred breakpoint.
Example 22-8 Setting Deferred Breakpoints in kmdb
[0]>::bp myModule`myBreakpointLocation
For more information on using breakpoints, see the Solaris Modular Debugger Guide. You can also get help by typing either of the following two lines:
> ::help bp > ::bp dcmd
The kmdb(1M) debugger supports macros that can be used to display kernel data structures. Use $M to display kmdb macros. Macros are used in the form:
[ address ] $<macroname
Note - Neither the information displayed by these macros nor the format in which the information is displayed, constitutes an interface. Therefore, the information and format can change at any time.
The kmdb macros in the following table are particularly useful to developers of device drivers. For convenience, legacy macro names are shown where applicable.
Table 22-1 kmdb Macros
|
The ::devinfo dcmd displays a node state that can have one of the following values:
The driver's attach(9E) routine returned successfully.
The node is bound to a driver, but the driver's probe(9E) routine has not yet been called.
The parent nexus has assigned a bus address for the driver. The implementation-specific initializations have been completed. The driver's probe(9E) routine has not yet been called at this point.
The device node has been linked into the kernel's device tree, but the system has not yet found a driver for this node.
The driver's probe(9E) routine returned successfully.
The device is fully configured.
The mdb(1) modular debugger can be applied to the following types of files:
Live operating system components
Operating system crash dumps
User processes
User process core dumps
Object files
The mdb debugger provides sophisticated debugging support for analyzing kernel problems. This section provides an overview of mdb features. For a complete discussion of mdb, refer to the Oracle Solaris Modular Debugger Guide.
Although mdb can be used to alter live kernel state, mdb lacks the kernel execution control that is provided by kmdb. As a result kmdb is preferred for runtime debugging. The mdb debugger is used more for static situations.
Note - The prompt for mdb is >.
The mdb debugger provides an extensive programming API for implementing debugger modules so that driver developers can implement custom debugging support. The mdb debugger also provides many usability features, such as command-line editing, command history, an output pager, and online help.
Note - The adb macros should no longer be used. That functionality has largely been superseded by the dcmds in mdb.
The mdb debugger provides a rich set of modules and dcmds. With these tools, you can debug the Solaris kernel, any associated modules, and device drivers. These facilities enable you to perform tasks such as:
Formulate complex debugging queries
Locate all the memory allocated by a particular thread
Print a visual picture of a kernel STREAM
Determine what type of structure a particular address refers to
Locate leaked memory blocks in the kernel
Analyze memory to locate stack traces
Assemble dcmds into modules called dmods for creating customized operations
To get started, switch to the crash directory and type mdb, specifying a system crash dump, as illustrated in the following example.
Example 22-9 Invoking mdb on a Crash Dump
% cd /var/crash/testsystem % ls bounds unix.0 vmcore.0 % mdb unix.0 vmcore.0 Loading modules: [ unix krtld genunix ufs_log ip usba s1394 cpc nfs ] > ::status debugging crash dump vmcore.0 (64-bit) from testsystem operating system: 5.10 Generic (sun4u) panic message: zero dump content: kernel pages only
When mdb responds with the > prompt, you can run commands.
To examine the running kernel on a live system, run mdb from the system prompt as follows.
Example 22-10 Invoking mdb on a Running Kernel
# mdb -k Loading modules: [ unix krtld genunix ufs_log ip usba s1394 ptm cpc ipc nfs ] > ::status debugging live kernel (64-bit) on testsystem operating system: 5.10 Generic (sun4u)
This section provides examples of useful debugging tasks. The tasks in this section can be performed with either mdb or kmdb unless specifically noted. This section assumes a basic knowledge of the use of kmdb and mdb. Note that the information presented here is dependent on the type of system used. A Sun Blade 100 workstation running the 64-bit kernel was used to produce these examples.
Caution - Because irreversible destruction of data can result from modifying data in kernel structures, you should exercise extreme caution. Do not modify or rely on data in structures that are not part of the Solaris DDI. See the Intro(9S) man page for information on structures that are part of the Solaris DDI. |
The kmdb debugger can display machine registers as a group or individually. To display all registers as a group, use $r as shown in the following example.
Example 22-11 Reading All Registers on a SPARC Processor With kmdb
[0]: $r g0 0 l0 0 g1 100130a4 debug_enter l1 edd00028 g2 10411c00 tsbmiss_area+0xe00 l2 10449c90 g3 10442000 ti_statetbl+0x1ba l3 1b g4 3000061a004 l4 10474400 ecc_syndrome_tab+0x80 g5 0 l5 3b9aca00 g6 0 l6 0 g7 2a10001fd40 l7 0 o0 0 i0 0 o1 c i1 10449e50 o2 20 i2 0 o3 300006b2d08 i3 10 o4 0 i4 0 o5 0 i5 b0 sp 2a10001b451 fp 2a10001b521 o7 1001311c debug_enter+0x78 i7 1034bb24 zsa_xsint+0x2c4 y 0 tstate: 1604 (ccr=0x0, asi=0x0, pstate=0x16, cwp=0x4) pstate: ag:0 ie:1 priv:1 am:0 pef:1 mm:0 tle:0 cle:0 mg:0 ig:0 winreg: cur:4 other:0 clean:7 cansave:1 canrest:5 wstate:14 tba 0x10000000 pc edd000d8 edd000d8: ta %icc,%g0 + 125 npc edd000dc edd000dc: nop
The debugger exports each register value to a variable with the same name as the register. If you read the variable, the current value of the register is returned. If you write to the variable, the value of the associated machine register is changed. The following example changes the value of the %o0 register from 0 to 1 on an x86 machine.
Example 22-12 Reading and Writing Registers on an x86 Machine With kmdb
[0]> <eax=K c1e6e0f0 [0]> 0>eax [0]> <eax=K 0 [0]> c1e6e0f0>eax
If you need to inspect the registers of a different processor, you can use the ::cpuregs dcmd. The ID of the processor to be examined can be supplied as either the address to the dcmd or as the value of the -c option, as shown in the following example.
Example 22-13 Inspecting the Registers of a Different Processor
[0]> 0::cpuregs %cs = 0x0158 %eax = 0xc1e6e0f0 kmdbmod`kaif_dvec %ds = 0x0160 %ebx = 0x00000000
The following example switches from processor 0 to processor 3 on a SPARC machine. The %g3 register is inspected and then cleared. To confirm the new value, %g3 is read again.
Example 22-14 Retrieving the Value of an Individual Register From a Specified Processor
[0]> 3::switch [3]> <g3=K 24 [3]> 0>g3 [3]> <g3 0
The ::findleaks dcmd provides powerful, efficient detection of memory leaks in kernel crash dumps. The full set of kernel-memory debugging features must be enabled for ::findleaks to be effective. For more information, see Setting kmem_flags Debugging Flags. Run ::findleaks during driver development and testing to detect code that leaks memory, thus wasting kernel resources. See Chapter 9, Debugging With the Kernel Memory Allocator, in Oracle Solaris Modular Debugger Guide for a complete discussion of ::findleaks.
Note - Code that leaks kernel memory can render the system vulnerable to denial-of-service attacks.
The mdb debugger provides a powerful API for implementing debugger facilities that you customize to debug your driver. The Solaris Modular Debugger Guide explains the programming API in detail.
The SUNWmdbdm package installs sample mdb source code in the directory /usr/demo/mdb. You can use mdb to automate lengthy debugging chores or help to validate that your driver is behaving properly. You can also package your mdb debugging modules with your driver product. With packaging, these facilities are available to service personnel at a customer site.
The Solaris kernel provides data type information in structures that can be inspected with either kmdb or mdb.
Note - The kmdb and mdb dcmds can be used only with objects that contain compressed symbolic debugging information that has been designed for use with mdb. This information is currently available only for certain Solaris kernel modules. The SUNWzlib package must be installed to process the symbolic debugging information.
The following example demonstrates how to display the data in the scsi_pkt structure.
Example 22-15 Displaying Kernel Data Structures With a Debugger
> 7079ceb0::print -t 'struct scsi_pkt' { opaque_t pkt_ha_private = 0x7079ce20 struct scsi_address pkt_address = { struct scsi_hba_tran *a_hba_tran = 0x70175e68 ushort_t a_target = 0x6 uchar_t a_lun = 0 uchar_t a_sublun = 0 } opaque_t pkt_private = 0x708db4d0 int (*)() *pkt_comp = sd_intr uint_t pkt_flags = 0 int pkt_time = 0x78 uchar_t *pkt_scbp = 0x7079ce74 uchar_t *pkt_cdbp = 0x7079ce64 ssize_t pkt_resid = 0 uint_t pkt_state = 0x37 uint_t pkt_statistics = 0 uchar_t pkt_reason = 0 }
The size of a data structure can be useful in debugging. Use the ::sizeof dcmd to obtain the size of a structure, as shown in the following example.
Example 22-16 Displaying the Size of a Kernel Data Structure
> ::sizeof struct scsi_pkt sizeof (struct scsi_pkt) = 0x58
The address of a specific member within a structure is also useful in debugging. Several methods are available for determining a member's address.
Use the ::offsetof dcmd to obtain the offset for a given member of a structure, as in the following example.
Example 22-17 Displaying the Offset to a Kernel Data Structure
> ::offsetof struct scsi_pkt pkt_state offsetof (struct pkt_state) = 0x48
Use the ::print dcmd with the -a option to display the addresses of all members of a structure, as in the following example.
Example 22-18 Displaying the Relative Addresses of a Kernel Data Structure
> ::print -a struct scsi_pkt { 0 pkt_ha_private 8 pkt_address { ... } 18 pkt_private ... }
If an address is specified with ::print in conjunction with the -a option, the absolute address for each member is displayed.
Example 22-19 Displaying the Absolute Addresses of a Kernel Data Structure
> 10000000::print -a struct scsi_pkt { 10000000 pkt_ha_private 10000008 pkt_address { ... } 10000018 pkt_private ... }
The ::print, ::sizeof and ::offsetof dcmds enable you to debug problems when your driver interacts with the Solaris kernel.
Caution - This facility provides access to raw kernel data structures. You can examine any structure whether or not that structure appears as part of the DDI. Therefore, you should refrain from relying on any data structure that is not explicitly part of the DDI. |
Note - These dcmds should be used only with objects that contain compressed symbolic debugging information that has been designed for use with mdb. Symbolic debugging information is currently available for certain Solaris kernel modules only. The SUNWzlib (32-bit) or SUNWzlibx (64-bit) decompression software must be installed to process the symbolic debugging information. The kmdb debugger can process symbolic type data with or without the SUNWzlib or SUNWzlibx packages.
The mdb debugger provides the ::prtconf dcmd for displaying the kernel device tree. The output of the ::prtconf dcmd is similar to the output of the prtconf(1M) command.
Example 22-20 Using the ::prtconf Dcmd
> ::prtconf 300015d3e08 SUNW,Sun-Blade-100 300015d3c28 packages (driver not attached) 300015d3868 SUNW,builtin-drivers (driver not attached) 300015d3688 deblocker (driver not attached) 300015d34a8 disk-label (driver not attached) 300015d32c8 terminal-emulator (driver not attached) 300015d30e8 obp-tftp (driver not attached) 300015d2f08 dropins (driver not attached) 300015d2d28 kbd-translator (driver not attached) 300015d2b48 ufs-file-system (driver not attached) 300015d3a48 chosen (driver not attached) 300015d2968 openprom (driver not attached)
You can display the node by using a macro, such as the ::devinfo dcmd, as shown in the following example.
Example 22-21 Displaying Device Information for an Individual Node
> 300015d3e08::devinfo 300015d3e08 SUNW,Sun-Blade-100 System properties at 0x300015abdc0: name='relative-addressing' type=int items=1 value=00000001 name='MMU_PAGEOFFSET' type=int items=1 value=00001fff name='MMU_PAGESIZE' type=int items=1 value=00002000 name='PAGESIZE' type=int items=1 value=00002000 Driver properties at 0x300015abe00: name='pm-hardware-state' type=string items=1 value='no-suspend-resume'
Use ::prtconf to see where your driver has attached in the device tree, and to display device properties. You can also specify the verbose (-v) flag to ::prtconf to display the properties for each device node, as follows.
Example 22-22 Using the ::prtconf Dcmd in Verbose Mode
> ::prtconf -v DEVINFO NAME 300015d3e08 SUNW,Sun-Blade-100 System properties at 0x300015abdc0: name='relative-addressing' type=int items=1 value=00000001 name='MMU_PAGEOFFSET' type=int items=1 value=00001fff name='MMU_PAGESIZE' type=int items=1 value=00002000 name='PAGESIZE' type=int items=1 value=00002000 Driver properties at 0x300015abe00: name='pm-hardware-state' type=string items=1 value='no-suspend-resume' ... 300015ce798 pci10b9,5229, instance #0 Driver properties at 0x300015ab980: name='target2-dcd-options' type=any items=4 value=00.00.00.a4 name='target1-dcd-options' type=any items=4 value=00.00.00.a2 name='target0-dcd-options' type=any items=4 value=00.00.00.a4
Another way to locate instances of your driver is the ::devbindings dcmd. Given a driver name, the command displays a list of all instances of the named driver as demonstrated in the following example.
Example 22-23 Using the ::devbindings Dcmd to Locate Driver Instances
> ::devbindings dad 300015ce3d8 ide-disk (driver not attached) 300015c9a60 dad, instance #0 System properties at 0x300015ab400: name='lun' type=int items=1 value=00000000 name='target' type=int items=1 value=00000000 name='class_prop' type=string items=1 value='ata' name='type' type=string items=1 value='ata' name='class' type=string items=1 value='dada' ... 300015c9880 dad, instance #1 System properties at 0x300015ab080: name='lun' type=int items=1 value=00000000 name='target' type=int items=1 value=00000002 name='class_prop' type=string items=1 value='ata' name='type' type=string items=1 value='ata' name='class' type=string items=1 value='dada'
A common problem when debugging a driver is retrieving the soft state for a particular driver instance. The soft state is allocated with the ddi_soft_state_zalloc(9F) routine. The driver can obtain the soft state through ddi_get_soft_state(9F). The name of the soft state pointer is the first argument to ddi_soft_state_init(9F)). With the name, you can use mdb to retrieve the soft state for a particular driver instance through the ::softstate dcmd:
> *bst_state::softstate 0x3 702b7578
In this case, ::softstate is used to fetch the soft state for instance 3 of the bst sample driver. This pointer references a bst_soft structure that is used by the driver to track state for this instance.
You can use both kmdb and mdb to modify kernel variables or other kernel state. Kernel state modification with mdb should be done with care, because mdb does not stop the kernel before making modifications. Groups of modifications can be made atomically by using kmdb, because kmdb stops the kernel before allowing access by the user. The mdb debugger is capable of making single atomic modifications only.
Be sure to use the proper format specifier to perform the modification. The formats are:
w – Writes the lowest two bytes of the value of each expression to the target beginning at the location specified by dot
W – Writes the lowest 4 bytes of the value of each expression to the target beginning at the location specified by dot
Z – Write the complete 8 bytes of the value of each expression to the target beginning at the location specified by dot
Use the ::sizeof dcmd to determine the size of the variable to be modified.
The following example overwrites the value of moddebug with the value 0x80000000.
Example 22-24 Modifying a Kernel Variable With a Debugger
> moddebug/W 0x80000000 moddebug: 0 = 0x80000000