Writing Device Drivers

Chapter 18 Debugging

Debugging is the process of finding and eliminating faults from software. Almost every device driver writer will be faced with a difficult bug at some point in the development process. This chapter presents an overview of the tools available to make this process easier. This chapter provides information on the following subjects:

Machine Configuration

Before you begin developing a Solaris driver, set up your test platform for this purpose. Testing on a separate system is safest. This section explains how to set up a pair of machines for development, and how to prepare a test system for disaster recovery.

Setting Up a tip Connection

A serial connection can be made between a test system (the machine executing the code to be debugged) and a host system using tip(1). This connection enables a window on the host system, called a tip window, to be used as the console of the test machine. See the tip(1) man page for additional information.

Using a tip window confers the following advantages:


Note –

A tip connection (and a second machine) is not required to debug a Solaris 9 device driver, but is recommended.


To Set Up the Host System

To set up the host system, do the following:

  1. Connect the host system to the test machine using serial port A on both machines.

    This connection must be made with a null modem cable.

  2. On the host system, make an entry in /etc/remote for the connection if it is not already there (see the remote(4) man page for details).

    The terminal entry must match the serial port being used. The Solaris 9 operating environment comes with the correct entry for serial port B, but a terminal entry must be added for serial port A:

    debug:\
            :dv=/dev/term/a:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:


    Note –

    The baud rate must be set to 9600.


  3. In a shell window on the host, run tip(1) and specify the name of the entry:

    % tip debug
    connected

    The shell window is now a tip window connected to the console of the test machine.


    Caution – Caution –

    Do not use STOP-A for SPARC machines or F1-A for IA (Intel architecture) machines on the host machine to send a break to stop the test machine. This action actually stops the host machine. To send a break to the test machine, type ~# in the tip window. Commands such as this are recognized only if they are the first characters on a line, so press the Return key or Control-U first if there is no effect.


Setting Up the Test System for SPARC Platforms

A quick way to set up the test machine is to unplug the keyboard before turning on the machine. The machine then automatically uses serial port A as the console.

Another way to set up the test machine is to use boot PROM commands to make serial port A the console. On the test machine, at the boot PROM ok prompt, direct console I/O to the serial line. To make the test machine always come up with serial port A as the console, set the environment variables input-device and output-device.

ok setenv input-device ttya
ok setenv output-device ttya

The eeprom command can also be used to make serial port A the console. As superuser, execute the following commands to make the input-device and output-device parameters point to serial port A.

# eeprom input-device=ttya
# eeprom output-device=ttya

Executing the eeprom commands causes the console to be redirected to serial port A at each subsequent system boot.

Setting Up the Test System for IA Platforms

On IA platforms, use the eeprom command to make serial port A the console. The procedure for this is the same as for SPARC platform and is discussed above. Executing the eeprom commands causes the console to switch to serial port A (COM1) during reboot.


Note –

Unlike SPARC machines, where the tip connection maintains console control throughout the boot process, IA machines don't transfer console control to the tip connection until an early stage in the boot process.


Preparing for Disasters

Under certain circumstances, a driver can render the system incapable of booting. To avoid system reinstallation in this event, some advance work must be done.

Back Up Critical System Files

A number of driver-related system files are difficult, if not impossible, to reconstruct. Files such as /etc/name_to_major,/etc/driver_aliases, /etc/driver_classes, and /etc/minor_perm can be corrupted if the driver crashes the system during installation (see the add_drv(1M) man page).

To be safe, after the test machine is in the proper configuration, make a backup copy of the root file system. If you plan on modifying the /etc/system file, make a backup copy of the file before modifying it.

Prepare and Boot an Alternate Kernel

A good strategy to avoid rendering a system inoperable is to make a copy of the kernel and associated binaries, and to boot that instead of the default kernel. To do so, make a copy of the drivers in /platform/* as follows:

# cp -r /platform/`uname -i`/kernel /platform/`uname -i`/kernel.test

When developing your driver, place it in /platform/`uname -i`/kernel.test/drv and boot that kernel instead of the default kernel:

# reboot -- kernel.test/unix

or from the PROM:

ok boot kernel.test/unix

This results in the test kernel and drivers being booted:

Rebooting with command: boot kernel.test/unix
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a  File and args:
kernel.test/unix
SunOS Release 5.9 Version Generic 32-bit
Copyright 1983-2002 Sun Microsystems, Inc.  All rights reserved.
...

Alternately, the module path can be changed by booting with the ask (-a) option:

ok boot -a

This results in a series of prompts which you can use to configure the way the kernel boots:

Rebooting with command: boot -a
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a  File and args: -a
Enter filename [kernel/sparcv9/unix]: kernel.test/sparcv9/unix
Enter default directory for modules
[/platform/sun4u/kernel.test /kernel /usr/kernel]: <CR>
Name of system file [etc/system]: <CR>
SunOS Release 5.9 Version Generic 64-bit
Copyright 1983-2002 Sun Microsystems, Inc.  All rights reserved.
root filesystem type [ufs]: <CR>
Enter physical name of root device
[/sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a]: <CR>

Prepare Other Backup Plans

If the system is attached to a network, the test machine can be added as a client of a server. If a problem occurs, the system can be booted off the network. The local disks can then be mounted and fixed. Alternatively, the system can be booted directly from the Solaris 9 CD-ROM.

Another way to recover from disaster is to have another bootable root file system. Use format(1M) to make a partition the exact size of the original, then use dd(1M) to copy it. After making a copy, run fsck(1M) on the new file system to ensure its integrity.

Later, if the system cannot boot from the original root partition, boot the backup partition and use dd(1M) to copy the backup partition onto the original one. If the system will not boot but the root file system is undamaged (just the boot block or boot program was destroyed), boot off the backup partition with the ask (-a) option, then specify the original file system as the root file system.

Saving System Crash Dumps

When the system panics, it writes an image of kernel memory to the dump device. The dump device by default is the most suitable swap device. The dump is a system crash dump, similar to core dumps generated by applications. On rebooting after a panic, savecore(1M) checks the dump device for a crash dump. If one is found, it makes a copy of the kernel's symbol table (called unix.n) and dumps a core file (called vmcore.n) in the core image directory which by default is /var/crash/machine_name. There must be enough space in /var/crash to contain the core dump or it will be truncated. mdb(1) can then be used on the core dump and the saved kernel.

In the Solaris 9 operating system, crash dump is enabled by default. The dumpadm(1M) command is used to configure system crash dumps. Use the dumpadm(1M) command to verify that crash dumps are enabled and to determine the location of the directory where core files are saved. See the dumpadm(1M) man page for more information.


Note –

savecore(1M) can be prevented from filling the file system if there is a file called minfree in the directory in which the dump will be saved. This file contains a number of kilobytes to remain free after savecore(1M) has run. However, if not enough space is available, the core file is not saved.


Disaster Recovery

If the /devices or /dev directories are damaged—most likely to occur if the driver crashes during attach(9E)—they can be re-created by booting the system and running fsck(1M) to repair the damaged root file system. The root file system can then be mounted. Re-create /dev and /devices by running devfsadm(1M) and specifying the /devices directory on the mounted disk.

On SPARC, for example, if the damaged disk is /dev/dsk/c0t3d0s0, and an alternate boot disk is /dev/dsk/c0t1d0s0, do the following:

ok boot disk1
...
Rebooting with command: boot kernel.test/unix
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@31,0:a  File and args:
kernel/unix
SunOS Release 5.9 Version Generic 32-bit
Copyright 1983-2002 Sun Microsystems, Inc.  All rights reserved.
...
# fsck /dev/dsk/c0t3d0s0** /dev/dsk/c0t3d0s0
** Last Mounted on /
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
1478 files, 9922 used, 29261 free
     (141 frags, 3640 blocks, 0.4% fragmentation)
# mount /dev/dsk/c0t3d0s0 /mnt
# devfsadm -r /mnt

Caution – Caution –

Fixing /devices and /dev may allow the system to boot, but other parts of the system can still be corrupted. This is only a temporary fix to allow saving information (such as system crash dumps) before reinstalling the system.


Runtime Debugging Tools

This section describes some of the mechanisms that can be used to debug drivers at runtime. Runtime debugging is typically performed during driver development; this process is substantially simplified if you have followed the coding practices described in the previous section. Although the kadb debugger is a runtime debugging tool, it is treated at length in a separate section, The kadb Kernel Debugger.

/etc/system File

The /etc/system file serves several purposes, but for driver development, the most important is that it allows you to set the value of kernel variables at boot time. This can be used to toggle different behaviors in a driver, or to enable certain debugging features made available by the kernel.

/etc/system is read only once, while the kernel is booting. After this file is modified, the system must be rebooted for the changes to take effect. If a change in the file causes the system not to work, boot with the ask (-a) option and specify /dev/null as the system file.

The set command is used to change the value of module or kernel variables:

See the system(4) man page for more information.


Note –

Most kernel variables are not guaranteed to be present in subsequent releases.


Controlling Module Loading with moddebug

moddebug is a kernel variable that controls the module loading process. The possible values are:

0x80000000

Prints messages to the console when loading or unloading modules. 

0x40000000

Gives more detailed error messages. 

0x20000000

Prints more detail when loading or unloading (such as including the address and size). 

0x00001000

No auto-unloading drivers: the system will not attempt to unload the device driver when the system resources become low. 

0x00000080

No auto-unloading streams: the system will not attempt to unload the streams module when the system resources become low. 

0x00000010

No auto-unloading of kernel modules of any type. 

0x00000001

If running with kadb, moddebug causes a breakpoint to be executed and a return to kadb immediately before each module's _init(9E) routine is called. Also generates additional debug messages when the module's _info and _fini routines are executed.

kmem_flags

kmem_flags is a kernel variable used to enable debugging features in the kernel's memory allocator. Setting kmem_flags to 0xf enables the allocator's debugging features. These include runtime checks to find:

The “Debugging With the Kernel Memory Allocator” section of the Solaris Modular Debugger Guide describes how the kernel memory allocator can be used to determine the root cause of these problems.


Note –

Testing and developing with kmem_flags set to 0xf is extremely valuable because it can detect latent memory corruption bugs. Because setting kmem_flags to 0xf changes the internal behavior of the kernel memory allocator, you should thoroughly test without kmem_flags as well.


modload, modunload, and modinfo Commands

The kernel automatically loads needed modules and unloads unused ones, so modload(1M), modunload(1M), and modinfo(1M) are not very useful for system administration. However, they can be useful when debugging and stress testing driver load/unload scenarios.

modload can be used to force a module into memory. The kernel might subsequently unload the module, but modload can be used to verify that the driver has no unresolved references when loaded. Keep in mind that loading a driver does not mean that the driver will attach. A driver that loads successfully will have its _info(9E) entrypoint called, but will not necessarily attach.

You can use modinfo to confirm that your driver is loaded. Here is an example:

$ modinfo
 Id Loadaddr   Size Info Rev Module Name
  6 101b6000    732   -   1  obpsym (OBP symbol callbacks)
  7 101b65bd  1acd0 226   1  rpcmod (RPC syscall)
  7 101b65bd  1acd0 226   1  rpcmod (32-bit RPC syscall)
  7 101b65bd  1acd0   1   1  rpcmod (rpc interface str mod)
  8 101ce8dd  74600   0   1  ip (IP Streams module)
  8 101ce8dd  74600   3   1  ip (IP Streams device)
...

$ modinfo | grep mydriver
169 781a8d78   13fb   0   1  mydriver (Test Driver 1.5)

The number in the info field is the major number chosen for the driver. modunload can be used to unload a module, given a module ID (which can be found in the leftmost column of modinfo output). A common bug is that a driver refuses to unload, even after a modunload is issued. Note that a driver will not unload if the system thinks the driver is busy. This occurs when the driver fails detach(9E), either because the driver really is busy, or because the detach entry point is implemented incorrectly.

To remove all currently unused modules from memory, run modunload with a module ID of 0:

# modunload -i 0

The kadb Kernel Debugger

kadb is a kernel debugger with facilities for disassembly, breakpoints, watch points, data display, and stack tracing. This section provides a tutorial on some of the features of kadb. For further information, consult the kadb(1M) man page.

Starting kadb

In order to start up kadb, the system must be booted with kadb(1M) enabled:

ok boot kadb
...
Rebooting with command: boot kadb
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a
File and args: kadb
kadb: kernel/sparcv9/unix
Size: 499808+109993+132503 Bytes
/platform/sun4u/kernel/sparcv9/unix loaded - 0x11e000 bytes used
SunOS Release 5.9 Version Generic 64-bit
Copyright 1983-2002 Sun Microsystems, Inc.  All rights reserved
....

By default, kadb(1M) boots (and debugs) kernel/unix, or kernel/sparcv9/unix on a system capable of running a 64-bit kernel. To boot kadb with an alternate kernel, pass the -D flag to boot, as follows:

ok boot kadb -D kernel.test/unix
...
Rebooting with command: boot kadb -D kernel.test/unix
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a  File
and args: kadb -D kernel.test/unix
kadb: kernel.test/unix
Size: 482384+67201+88883 Bytes
/platform/sun4u/kernel.test/unix loaded - 0xfe000 bytes used
SunOS Release 5.9 Version dacf-fixes:11/13/99 32-bit
Copyright 1983-2002 Sun Microsystems, Inc.  All rights reserved.
...

In this example, the 32-bit version of the alternate kernel kernel.test was booted. Another option is to pass kadb the -d flag, which causes kadb to prompt for the kernel name. The -d flag also causes kadb(1M) to provide a prompt after it has loaded the kernel, so breakpoints can be set.

ok boot kadb -d
...
Rebooting with command: boot kadb -d
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a  File
and args: kadb -d
kadb: kernel.test/unix
Size: 482384+67201+88883 Bytes
/platform/sun4u/kernel.test/unix loaded - 0xfc000 bytes used
stopped at      _start:         sethi   %hi(0x10006c00), %g1
kadb[0]: 

At this point you can set breakpoints or continue execution with the :c command.


Note –

Kernel modules are dynamically loaded. Consequently, driver symbols are not available until the driver is loaded. To set breakpoints in modules that have not been loaded, use deferred breakpoints. For information on deferred breakpoints, refer to Breakpoints.


kadb(1M) passes any kernel flags to the booted kernel. For example, to boot an alternate kernel and pass the -r flag:

ok boot kadb -r -D kernel.test/unix
...
Rebooting with command: boot kadb -r -D kernel.test/unix
Boot device: /sbus@1f,0/espdma@e,8400000/esp@e,8800000/sd@0,0:a
File and args: kadb -r -D kernel.test/unix
kadb: kernel.test/unix
Size: 482384+67201+88883 Bytes
/platform/sun4u/kernel.test/unix loaded - 0xfe000 bytes used
SunOS Release 5.9 Version Generic 32-bit
Copyright 1983-2002 Sun Microsystems, Inc.  All rights reserved.
obpsym: symbolic debugging is available.
Read 208377 bytes from misc/forthdebug
configuring IPv4 interfaces: le0.
Hostname: test
Configuring /dev and /devices

After the system is booted, sending a break passes control to kadb(1M). A break is generated with STOP-A (on the console of SPARC machines), or with F1-A (on the console of IA machines), or by using ~# (if the console is connected through a tip window).

...

The system is ready.

test console login: ~#
stopped at      edd000d8:       ta      %icc,%g0 + 125
kadb[0]: 

The number in brackets is the CPU that kadb(1M) is currently executing on; the remaining CPUs are halted. The CPU number is zero on a uniprocessor system.


Caution – Caution –

Before rebooting or turning off the power, always halt the system cleanly (with init 0 or shutdown). Buffers might not be flushed otherwise. If the shutdown must occur from the boot PROM prompt, make sure to flush buffers using the sync command at the ok prompt.


To return control to the operating system, use :c.

kadb[0]: :c
test console login: 

Exiting kadb

To exit kadb(1M), use $q. On SPARC machines, this will exit to the ok prompt. On IA machines, you will be prompted to reboot the system.

kadb[0]: $q
Type `go' to resume
ok 

kadb(1M) can be resumed by typing go at the ok prompt.


Caution – Caution –

No other commands should be performed from the PROM if the system is to be resumed. PROM commands other than go can change system state that the Solaris 9 operating environment depends upon.


Staying at the kadb(1M) prompt for too long can cause the system to lose track of the time of day, and can cause network connections to time out.

kadb Command Usage

The general form of a kadb command is:

    [ address ] [ ,count ] command [;]

If address is omitted, the current location is used (`.' could also be used to represent the current location). The address can be a kernel symbol. If count is omitted, it defaults to 1.

Commands to kadb consist of a verb followed by a modifier or list of modifiers. Verbs can be:

Prints locations starting at address in the kernel address space

Prints the value of address itself

Assigns a value to a debugger variable or machine register 

Reads a value from a debugger variable or machine register 

RETURN 

Repeats the previous command with a count of 1. Increments `.' (the current location) 

With / and =, output format specifiers can be used. Lowercase letters normally print 2 bytes, uppercase letters print 4 bytes. The values of these specifiers are:

o, O 

2-, 4-byte octal 

8-byte octal 

8-byte unsigned octal 

d, D 

2-, 4-byte decimal 

8-byte decimal 

8-byte unsigned decimal 

x, X 

2-, 4-byte hexadecimal 

8-byte hexadecimal 

4–byte hexadecimal for 32–bit programs, 8–byte hexadecimal for 64–bit programs. Use this format specifier to examine pointers. 

u, U 

2-, 4-byte unsigned decimal 

Prints the addressed character 

Prints the addressed character using ^ escape notation 

Prints the addressed string 

Prints the addressed string using ^ escape notation 

Prints as machine instructions (disassemble) 

Prints the value of `.' in symbolic form 

w, W 

2-, 4-byte write 

8-byte write 


Caution – Caution –

When using w, W or Z to modify a kernel variable, make sure that the size of the variable matches the size of the write you are performing. If you specify an incorrect size you could corrupt neighboring data.


For example, to set a bit in the moddebug variable when debugging a driver, first examine the value of moddebug, then set it to the desired bit.

kadb[0]: moddebug/X
moddebug:
moddebug:       1000
kadb[0]: moddebug/W 0x80001000
moddebug:       0x1000 = 0x80001000

Routines can be disassembled with the `i' command. This is useful when tracing crashes, since the only information might be the program counter at the time of the crash. For example, to print the first four instructions of the kmem_alloc function:

kadb[0]: kmem_alloc,4/i
kmem_alloc:
kmem_alloc: save    %sp, -0x60, %sp
sub     %i0, 0x1, %l6
sra     %l6, 0x3, %i5
tst     %i5

Specify symbolic notation with the `a' command, to show the addresses:

kadb[0]: kmem_alloc,4/ai
kmem_alloc:    
kmem_alloc:     save    %sp, -0x60, %sp
kmem_alloc+4:   sub     %i0, 0x1, %l6
kmem_alloc+8:   sra     %l6, 0x3, %i5
kmem_alloc+0xc: tst     %i5

Register Identifiers

You can discover what machine registers are available on your processor architecture using the $r command. This example shows the output of $r on a SPARC system with the sun4 architecture:

kadb[0]: $r
         
g0    0                                 l0      0
g1    100130a4      debug_enter         l1      edd00028
g2    10411c00      tsbmiss_area+0xe00  l2      10449c90
g3    10442000      ti_statetbl+0x1ba   l3      1b
g4    3000061a004                       l4      10474400
ecc_syndrome_tab+0x80
g5    0                                 l5      3b9aca00
g6    0                                 l6      0
g7    2a10001fd40                       l7      0
o0    0                                 i0      0
o1    c                                 i1      10449e50
o2    20                                i2      0
o3    300006b2d08                       i3      10
o4    0                                 i4      0
o5    0                                 i5      b0
sp    2a10001b451                       fp      2a10001b521
o7    1001311c      debug_enter+0x78    i7      1034bb24
zsa_xsint+0x2c4
y     0
tstate: 1604  (ccr=0x0, asi=0x0, pstate=0x16, cwp=0x4)
pstate: ag:0 ie:1 priv:1 am:0 pef:1 mm:0 tle:0 cle:0 mg:0 ig:0
winreg: cur:4 other:0 clean:7 cansave:1 canrest:5 wstate:14
tba   0x10000000
pc    edd000d8 edd000d8:        ta      %icc,%g0 + 125
npc   edd000dc edd000dc:        nop

kadb exports each of these registers as a debugger variable with the same name. Reading from the variable fetches the current value of the register. Writing to the variable changes the value of the associated machine register. For example, you can change the value of the '%o0' register:

kadb[0]: <o0=K
                 0
kadb[0]: 0x1>o0
kadb[0]: <o0=K
                 1

Display and Control Commands

The following commands display and control the status of kadb(1M):

$b

Display all breakpoints 

$c

Display stack trace 

$d

Change default radix to value of dot 

$q

Quit 

$r

Display registers 

$M

Display built-in macros 

`$c' is useful when a breakpoint is reached, but is usually not useful if kadb(1M) is entered at a random time. The number of arguments to print can be passed following the `$c' (`$c 2' for two arguments).

Breakpoints

In kadb(1M), breakpoints can be set. When reached, the kernel will automatically drop back into kadb. The standard form of a breakpoint command is:

    [module_name#] addr [, count]:b [command]

addr is the address at which the program will be stopped and the debugger will receive control, count is the number of times that the breakpoint address occurs before stopping, and command is almost any adb(1) command.

The optional module_name specifies deferred breakpoints that are set when the module is loaded. module_name identifies a particular module that contains addr. If the module has been loaded, kadb will try to set a regular breakpoint; if the module is not loaded, kadb will set a deferred breakpoint. When the module is loaded, kadb will try to resolve the location of the breakpoint and convert the breakpoint to a regular breakpoint.

Other breakpoint commands are:

:c

Continue execution 

:d

Delete breakpoint 

:s

Single step 

:e

Single step, but step over function calls 

:u

Stop after return to caller of current function 

:z

Delete all breakpoints 

The following example sets a breakpoint in scsi_transport(9F), a commonly used routine. Upon reaching the breakpoint, '$c' is used to print a stack trace. The top of the stack is displayed first. Note that kadb does not know how many arguments were passed to each function.

stopped at      edd000d8:       ta      %icc,%g0 + 125
kadb[0]: scsi_transport:b
kadb[0]: :c
test console login: root
breakpoint at:
scsi_transport: save    %sp, -0x60, %sp
kadb[0]: $c
scsi_transport(702bb578,1000,1,10000,0,702bb7fe)
sdstrategy(1019c8c0,702bb61c,0,0,702bb578,70cad7b8) + 704
bdev_strategy(1042a808,70cad7b8,705f3efc,40,10597900,2000) + 98
ufs_getpage_miss(70cad7b8,0,10597900,0,0,4023ba8c) + 2b0
ufs_getpage(0,0,0,0,2000,4023ba8c) + 7c0
segvn_fault(4023ba8c,2000,ff3b0000,0,0,0) + 7c8
as_fault(1,ff3b0000,70d98030,2000,0,ff3b0000) + 49c
pagefault(0,0,70df8048,705c7450,0,ff3b0000) + 4c
trap(10080,10000,ff3c4ea4,70df8048,ff3b0000,1) + db4
kadb[0]: $b
breakpoints
count   bkpt            type      len   command
1       scsi_transport  :b instr  4
kadb[0]: scsi_transport:d
kadb[0]: :c

Conditional Breakpoints

Breakpoints can also be set to occur only if a certain condition is met. By providing a command, the breakpoint will be taken only if the count is reached or the command returns zero. For example, a breakpoint that occurs only on certain I/O controls could be set in the driver's ioctl(9E) routine. This is the general syntax of conditional breakpoints:

    address,count:b command

In this example, address is the address at which to set the breakpoint. count is the number of times the breakpoint should be ignored (note that 0 means break only when the command returns zero). command is the kadb(1M) command to execute.

Here is an example of breaking only in the sdioctl() routine if the DKIOGVTOC (get volume table of contents) I/O control occurs.

kadb[0]: sdioctl+4,0:b <i1-0x40B
kadb[0]: $b
breakpoints
count   bkpt            type      len   command
0       sdioctl+4       :b instr  4     <i1-0x40B
kadb[0]: :c     

Adding four to sdioctl skips to the second instruction in the routine, bypassing the save instruction that establishes the stack. The `<i1' refers to the first input register, which is the second parameter to the routine (the cmd argument of ioctl(9E)). The count of zero is impossible to reach, so it stops only when the command returns zero, which is when `i1 - 0x40B' is true. This means i1 contains 0x40B (the value of the ioctl(9E) command, determined by examining the ioctl definition).

To force the breakpoint to be reached, the prtvtoc(1M) command, which is known to issue the following I/O control, is used:

# prtvtoc /dev/rdsk/c0t0d0s0
breakpoint at:
sdioctl+4:      mov     %i5, %l0
kadb[0]: $c
sdioctl(800000,40b,ffbefb54,100005,704a3ce8,4026bc7c) + 4
ioctl(3,40b,70ca27b8,40b,ffbefb54,0) + 1e0

kadbMacros

kadb(1M) supports macros that are used for displaying kernel data structures. kadb macros can be displayed with $M. Macros are used in the form:

    [ address ] $<macroname

Note –

Neither the information displayed by these macros, nor the format in which the information is displayed constitute an interface. Therefore, the information and format can change at any time.


Device Macros

This section describes the kadb(1M) macros that can be used to retrieve information from the kernel about a device during a debugging session.

devinfo_brief

Prints a brief summary of a dev_info_t structure; the name and addr fields are shown in string form and are used in the /devices path to the node

devinfo

Prints devinfo_brief information for a dev_info_t structure followed by an in-order dump of all fields

devinfo.parent

Prints devinfo_brief information for the specified node and its parents all the way back to the root of the devinfo tree

devinfo.sibling

Prints devinfo_brief information for all siblings of the specified node; you see all sibling nodes if you start at the parent's child

devinfo.minor

Prints the ddi_minor_data structure for all minor name nodes associated with the specified device node

devinfo.prop

Prints all the property lists associated with the specified device node 

devinfo_major

Prints devinfo_brief information for all device nodes associated with the specified driver major number

devt

Breaks a dev_t into its major and minor parts and prints the values in hexadecimal and decimal

devnames

Prints the contents of the specified devnames structure

devnames_major

Prints the contents of the devnames structure for the specified major number

bus_op

Prints the contents of a bus_ops structure

The devinfo commands, such as devinfo and devinfo_brief, return a node_state value, which may be one of the following:

DS_LINKED

This device node has been linked into the kernel's device tree, but the system has not yet found a driver for it.

DS_BOUND

The node is bound to a driver, but the driver's probe(9E) routine has not yet been called.

DS_INITIALIZED

The parent nexus has assigned a bus address for the driver and completed implementation-specific initializations, but the driver's probe(9E) routine has not yet been called.

DS_PROBED

The driver's probe(9E) routine returned successfully.

DS_ATTACHED

The driver's attach(9E) routine returned successfully.

DS_READY

The device is fully configured.

Examples: Use of Device Macros

The examples in this section show how some of the device-related macros can be used to debug a driver.

The following ls -l command shows how the /devices path fits into the devinfo structure:

# ls -l /dev/ttya
lrwxrwxrwx   1 root   root   6 Aug 17 17:50 /dev/ttya -> term/a
# ls -l /dev/term/a
lrwxrwxrwx   1 root   root   32 Aug 17 17:42 /dev/term/a -> 
    ../../devices/obio/zs@0,100000:a

Example 18–1 shows how to obtain information about the /dev/ttya device from the devinfo structure and other kernel structures, beginning with devinfo_major.


Example 18–1 The devinfo_major Macro

# ls -lL /dev/ttya
crw-rw-rw-   1 root     sys    29,  0 Aug 17 17:42 /dev/ttya
# ~stopped      at:
edd000d8:       ta      %icc,%g0 + 125
kadb[0]: 0t29$<devinfo_major
                ============== devinfo  f5f0dcd0
                                binding_name
0xf5bb38f8:                       zs
                                node_name
0xf5bb38f8:                       zs
                                addr
0xf5f8d6a0:                       0,100000
                                node_state
                                  6                 DS_READY
                                major (hex)
                                  1d
                                instance
                                  0

0xf5f0dcd0:     parent          child           sibling
                f5f0f380        0               f5f0dba8

                ============== devinfo  f5f0dba8
                                binding_name
0xf5bb3a18:                       zs
                                node_name
0xf5bb3a18:                       zs
                                addr
0xf5f90560:                       0,0
                                node_state
                                  6                 DS_READY
                                major (hex)
                                  1d
                                instance
                                  1

0xf5f0dba8:     parent          child           sibling
                f5f0f380        0               f5f0da80


Example 18–2 shows the use of the devinfo_brief macro. It also shows the use of top_devinfo. top_devinfo is a pointer to the root of the devinfo tree (rootnexus)—all other devinfo nodes are descendants of this node. If you want to look at the device tree top-down in the debugger, this is where you start.


Example 18–2 The devinfo_brief Macro

kadb[0]: *top_devinfo$<devinfo_brief

                ============== devinfo  f5f0fde8
                                binding_name
0xf5bc6a30:                       SUNW,SPARCstation-10
                                node_name
0xf5bc6af8:                       SUNW,SPARCstation-10
                                addr
0xf5f90950:
                                node_state
                                  6                 DS_READY
                                major (hex)
                                  1

0xf5f0fde8:     parent          child           sibling
                0               f5f0fcc0        0

Thread Macros

threadlist is a useful macro that displays the stacks of all the threads in the system.


Example 18–3 The threadlist Macro

kadb[0]: $<threadlist

                ============== thread_id        10404000
p0+0x300:
                process args    sched

t0+0xa8:        lwp             procp           wchan
                1041b810        10424688        0
t0+0x24:
                pc              sp
                sched+0x4f4     10403be8
?(10404000,1040c000,2,10424604,0,6e)
_start(10006ef4,1041adb0,1041adb0,1041adb0,10462910,50) + 15c

...

                ============== thread_id        40043e60
p0+0x300:
                process args    sched

40043f08:       lwp             procp           wchan
                0               10424688        10473c56
40043e84:
                pc              sp
                cv_wait+0x60    40043c08
?(10473c56,10473c5c,0,40043cd0,40043e60,10093084)
ufs_thread_idle(10471e80,0,10473c5c,10424688,81010100,0) + bc
thread_start(0,0,0,0,0,0) + 4
...

Another useful macro is thread. Given a thread ID, this macro prints the corresponding thread structure. This can be used to look at a certain thread found with the threadlist macro, to look at the owner of a mutex, or to look at the current thread, as shown here:


Example 18–4 The thread Macro

kadb[0]: <g7$<thread
70e87ac0:       link            stk             startpc
                0               4026bc80        0
70e87acc:       bound_cpu       affinitycnt     bind_cpu
                0               0               -1
70e87ad4:       flag    proc_flag       schedflag
                0       4               3
70e87ada:       preempt preempt_lk      state
                0       0               4
70e87ae0:       pri     epri
                40      0
70e87ae4:
                pc              sp
                10098350        4026b618
70e87aec:       wchan0          wchan           sobj_ops
                0               0               0
70e87af8:       cid             clfuncs         cldata
                1               10470ffc        702c0488
70e87b04:       ctx             lofault         onfault
                0               0               0
...


Note –

No type information is maintained by kadb, so using a macro on an inappropriate address results in garbage output.


Macros do not necessarily output all the fields of the structures, nor is the output necessarily in the order given in the structure definition. Occasionally, memory needs to be dumped for certain structures and then matched with the structure definition in the kernel header files.


Caution – Caution –

Drivers should never reference system header files or structures not listed in man pages section 9S: DDI and DKI Data Structures. However, examining non-DDI-compliant structures (such as thread structures) can be useful in debugging drivers.


kadb Output Pager

Some kadb commands (like $<threadlist) output lots of data, which can scroll off of the screen very rapidly. kadb provides a simple output pager to remedy this problem. The pager command is lines::more, where lines represents the number of lines to print before pausing the console output. Keep in mind that this does not take into account lines that wrap because they are wider than the terminal width. Here is an example usage:

kadb[0]: 0t10::more
kadb[0]: $<threadlist 

                ============== thread_id        10408000
p0+0x4c0:
                process args    sched

t0+0x128:       lwp             procp           wchan
            10429ed0         104393e8         0
t0+0x38:
                pc              sp
                sched+0x4e4     104071f1
?(10408000,10414c00,2,104393e8,10439308,0)
_start(10007588,104292e0,104292e0,104292e0,1043b8b0,10429360) + 200

                ============== thread_id        2a10001fd40
p0+0x4c0:
                process args    sched
--More-- <SPACE>
         
...

Pressing the space bar at the “--More--” prompt pages the output by the number of lines specified to ::more (in this case, 10). Pressing "Return" prints only the next line of output. You can abort the output and return to the kadb prompt by typing Ctrl-C. To disable the pager, issue '0::more' at the kadb prompt.

Example: kadb on a Deadlocked Thread

This example shows how kadb can be used to debug a driver bug. This example was taken from the development of the ramdisk sample driver. This driver exports physical memory as a virtual disk. In this case, the dd(1M) command hangs while trying to copy some data onto the device and cannot be aborted. Though a crash dump could be forced, for illustrative purposes, kadb(1M) will be used. After logging into the system remotely, ps was used to determine that the system was still running; and only the dd(1M) command is hung.

At this point, the system is rebooted with kadb, which can now be entered by typing STOP-A on the system console. After the rest of the kernel has loaded, moddebug is patched to see if loading is the problem:

stopped at:
edd000d8:       ta      %icc,%g0 + 125
kadb[0]: moddebug/X
moddebug:
moddebug:       0
kadb[0]: moddebug/W 0x80000000
moddebug:       0x0             =       0x80000000
kadb[0]: :c

modload(1M) is used to load the driver, to separate module loading from the real access:

# modload /home/driver/drv/ramdisk

It loads without errors, so loading is not the problem. The condition is recreated with dd(1M):

# dd if=/dev/zero of=/devices/pseudo/ramdisk@0:c,raw

dd(1M) hangs. At this point, kadb(1M) is entered and the stack examined:

stopped at:
edd000d8:       ta      %icc,%g0 + 125
kadb[0]: $c
intr_vector() + 7dcfc0d8
debug_enter(0,0,10431e50,10,1,b0) + 78
zsa_xsint(80,7044a06c,44,7044a000,ff0113,0) + 278
zs_high_intr(7044a000,1,1,1042f78c,10424680,100949d0) + 20c
sbus_intr_wrapper(704dfad4,0,702bd048,7029cec0,630,10260250) + 30
current_thread(4001fe60,1041a550,10424698,10424698,10150f08,0) + 180
idle(1040b6c0,0,0,1041a550,704d6a98,0) + 54
thread_start(0,0,0,0,0,0) + 4

The presence of idle on the current thread stack indicates that this thread is not the cause of the deadlock. To determine the deadlocked thread, the entire thread list is checked:

kadb[0]: $<threadlist
...
                ============== thread_id        70cef120
70c8b1c0:
                process args    dd if=/dev/zero of=/devices/pseudo/ramdisk@0:c,raw

70cef1c8:       lwp             procp           wchan
                70fa9080        70c8aec0        70691fc8
70cef144:
                pc              sp
                sema_p+0x290    40313a78
?(70691fc8,10424680,1,1042b99c,10460f8c,70691fc8)
biowait(70691f60,1041a6c4,70691f60,70c385d0,40313bcc,705c73a0) + 8c
default_physio(1042e8fc,200,129,100,70eb5b54,705c73a0) + 3bc
write(2002,70aac1d0,70f9f9ac,200,4,200) + 23c
...

Of all the threads, only one has a stack trace which references the ramdisk driver. It seems that the process running dd(1M) is blocked in biowait(9F). biowait(9F)'s first parameter is a buf(9S) structure. The next step is to examine this structure:

kadb[0]:  70691f60$70691f60$
70691f60:       flags           forw            back
                204129          0               0
70691f6c:       av_forw         av_back         bcount
                0               0               512
70691fa0:       bufsize         error           edev
                0               0               1180000
70691f7c:       un.b_addr       _b_blkno        resid
                710e8000        0                0
70691f94:       proc            iodone          vp
                70c8aec0        0               0
70691f98:       pages
                0

The resid field is 0, which indicates that the transfer is complete. physio(9F) is still blocked, however. The reference for physio(9F) in the Solaris 9 Reference Manual Collection points out that biodone(9F) should be called to unblock biowait(9F). This is the problem; rd_strategy() did not call biodone(9F). Adding a call to biodone(9F) before returning fixes this problem.

Post-Mortem Debugging

When kadb is running and the system panics, control is passed to the debugger so that you can investigate the source of the problem. However, kadb is not always the best tool for problem analysis; frequently it is easier to use ':c' to continue execution and allow the system to save a crash dump. When the system reboots, you can perform post-mortem analysis on the saved crash dump. This process is analogous to debugging an application crash from a process core file.

Post-mortem analysis offers several advantages to driver developers: it allows more than one developer to examine a problem in parallel; it allows developers to retrieve information on a problem that occurred in production at a customer site, where it is not acceptable to debug interactively; it is necessary to perform certain types of advanced kernel analysis, such as checking for kernel memory leaks.

Getting Started With the Modular Debugger

The modular debugger, mdb, provides sophisticated debugging support for analyzing kernel problems. This section provides an overview of mdb features. For a more complete discussion of mdb, refer to the Solaris Modular Debugger Guide.

mdb command syntax is compatible with the kadb syntax and mdbcan execute all of the kadb (and legacy adb) macros. These are stored in /usr/lib/adb and in /usr/platform/`uname -i`/lib/adb for 32-bit kernels; and in /usr/lib/adb/sparcv9 and /usr/platform/`uname -i`/lib/adb/sparcv9 for 64-bit kernels.

In addition to macro files, mdb supports debugger commands (or dcmds). These dcmds can be dynamically loaded at runtime from a set of debugger modules. mdb provides a first-class programming API for implementing debugger modules so that driver developers can implement their own custom debugging support. mdb also provides a host of usability features, such as command line editing, command history, an output pager, and online help.

mdb provides a rich set of modules and dcmds for debugging the Solaris kernel and associated modules and device drivers. Some of the activities these facilities enable you to do include:


Note –

In earlier versions of the Solaris operating environment, adb(1) was the recommended tool for post-mortem analysis. In the Solaris 9 operating environment, mdb(1) is the recommended tool for post-mortem analysis. It provides an upward-compatible syntax and feature set that surpass the set of commands available from the legacy crash(1M) utility, which has been removed from Solaris 9.


To get started, type mdb and supply it with a system crash dump:

% cd /var/crash/testsystem
% ls
bounds     unix.0    vmcore.0
% mdb unix.0 vmcore.0
Loading modules: [ unix krtld genunix ufs_log ip usba s1394 cpc nfs ]
> ::status
debugging crash dump vmcore.0 (64-bit) from testsystem
operating system: 5.9 Generic (sun4u)
panic message: zero
dump content: kernel pages only

When mdb responds with the '>' prompt, it is ready for commands. To examine the running kernel on a live system, type:

# mdb -k
Loading modules: [ unix krtld genunix ufs_log ip usba s1394 ptm cpc ipc nfs ]
> ::status
debugging live kernel (64-bit) on testsystem
operating system: 5.9 Generic (sun4u)

Important mdb Commands

This section provides a tutorial for some of the mdb debugger commands most applicable to driver authors. Note that the information presented here is dependent on the type of system used. A Sun Blade™ 100 workstation running the 64-bit kernel was used to produce these examples.

The Solaris Modular Debugger Guide provides details about each debugger command discussed here, as well as more information about all aspects of mdb. Online help is available from within mdb using the ::help built-in command.

Displaying Data Structures with mdb

mdb provides a powerful facility for displaying kernel data structures, so that earlier kadb(1) and mdb(1) debugger macros are no longer needed. Starting in Solaris 9, the operating system kernel maintains a highly compressed database of data structure type information in nonpageable system memory. This means that when a crash occurs, this type information is saved as part of the crash dump.

Here is an example of using the kernel type information to display all of the fields of a scsi_pkt structure:


Example 18–5 Displaying Kernel Information with mdb

> 7079ceb0::print -t 'struct scsi_pkt'
{
    opaque_t pkt_ha_private = 0x7079ce20
    struct scsi_address pkt_address = {
        struct scsi_hba_tran *a_hba_tran = 0x70175e68
        ushort_t a_target = 0x6
        uchar_t a_lun = 0
        uchar_t a_sublun = 0
    }
    opaque_t pkt_private = 0x708db4d0
    int (*)() *pkt_comp = sd_intr
    uint_t pkt_flags = 0
    int pkt_time = 0x78
    uchar_t *pkt_scbp = 0x7079ce74
    uchar_t *pkt_cdbp = 0x7079ce64
    ssize_t pkt_resid = 0
    uint_t pkt_state = 0x37
    uint_t pkt_statistics = 0
    uchar_t pkt_reason = 0
}

Each data structure member is presented along with its type. Nested structures are expanded for easy viewing. ::print can also decode arrays and unions.

It is frequently helpful to discover the size of a particular kernel data structure; doing so is simple, using the ::sizeof dcmd:


> ::sizeof 'struct scsi_pkt'
sizeof (struct scsi_pkt) = 0x58

You can also locate the offset of a field within a data structure:


> ::offsetof 'struct scsi_pkt' pkt_state
offsetof (pkt_state) = 0x48

The -a option may be used to view the offset of each member of a data structure; if no address is specified to ::print, the output begins at address 0, providing the offset of each field:


Example 18–6 mdb: Viewing Data Members

> ::print -at 'struct scsi_pkt'

{
    0 opaque_t pkt_ha_private
    8 struct scsi_address pkt_address {
        8 struct scsi_hba_tran *a_hba_tran
        10 ushort_t a_target
        12 uchar_t a_lun
        13 uchar_t a_sublun
    }
    18 opaque_t pkt_private
    20 int (*)() *pkt_comp
    28 uint_t pkt_flags
    2c int pkt_time
    30 uchar_t *pkt_scbp
    38 uchar_t *pkt_cdbp
    40 ssize_t pkt_resid
    48 uint_t pkt_state
    4c uint_t pkt_statistics
    50 uchar_t pkt_reason
}

The ::print, ::sizeof and ::offsetof facilities make it possible to more rapidly debug problems which arise when your driver interacts with the Solaris kernel.


Caution – Caution –

This facility provides access to “raw” kernel data structures. You may examine any structure whether it appears as part of the DDI or not; therefore, refrain from relying on any data structure that is not explicitly part of the DDI.



Note –

These dcmds may only be used with objects that contain compressed symbolic debugging information designed for use with mdb. This information is currently only available for certain Solaris kernel modules. The SUNWzlib (32-bit) or SUNWzlibx (64-bit) decompression software must be installed in order to process the symbolic debugging information.


Navigating the Device Tree with mdb

mdb provides the ::prtconf dcmd to display the kernel device tree. The output of this dcmd is similar to the output of the prtconf(1M) command:

> ::prtconf
300015d3e08      SUNW,Sun-Blade-100
    300015d3c28      packages (driver not attached)
        300015d3868      SUNW,builtin-drivers (driver not attached)
        300015d3688      deblocker (driver not attached)
        300015d34a8      disk-label (driver not attached)
        300015d32c8      terminal-emulator (driver not attached)
        300015d30e8      obp-tftp (driver not attached)
        300015d2f08      dropins (driver not attached)
        300015d2d28      kbd-translator (driver not attached)
        300015d2b48      ufs-file-system (driver not attached)
    300015d3a48      chosen (driver not attached)
    300015d2968      openprom (driver not attached)
    ...

The node can then be displayed using a macro (such as the $<devinfo_brief macro) or the ::devinfo dcmd:

> 300015d3e08::devinfo
300015d3e08      SUNW,Sun-Blade-100
        System properties at 0x300015abdc0:
            name='relative-addressing' type=int items=1
                value=00000001
            name='MMU_PAGEOFFSET' type=int items=1
                value=00001fff
            name='MMU_PAGESIZE' type=int items=1
                value=00002000
            name='PAGESIZE' type=int items=1
                value=00002000
        Driver properties at 0x300015abe00:
            name='pm-hardware-state' type=string items=1
                value='no-suspend-resume'


> 300015d3e08$<devinfo_brief

                ============== devinfo  300015d3e08
                                binding_name
0x300013b3058:                    SUNW,Sun-Blade-100
                                node_name
0x300013b3118:                    SUNW,Sun-Blade-100
                                addr
0x300015b1760:
                                node_state
                                  6                 DS_READY
                                major (hex)
                                  1
0x300015d3e08:  parent          child           sibling
                0               300015d3c28     0

Use ::prtconf to see where your driver has attached in the device tree, and to display device properties. You can also specify the verbose (-v) flag to ::prtconf to display the properties for each device node:

> ::prtconf -v
DEVINFO          NAME                                              
300015d3e08      SUNW,Sun-Blade-100
        System properties at 0x300015abdc0:
            name='relative-addressing' type=int items=1
                value=00000001
            name='MMU_PAGEOFFSET' type=int items=1
                value=00001fff
            name='MMU_PAGESIZE' type=int items=1
                value=00002000
            name='PAGESIZE' type=int items=1
                value=00002000
        Driver properties at 0x300015abe00:
            name='pm-hardware-state' type=string items=1
                value='no-suspend-resume'
        ...
        300015ce798      pci10b9,5229, instance #0
                Driver properties at 0x300015ab980:
                    name='target2-dcd-options' type=any items=4
                        value=00.00.00.a4
                    name='target1-dcd-options' type=any items=4
                        value=00.00.00.a2
                    name='target0-dcd-options' type=any items=4
                        value=00.00.00.a4
 		...

Another way to locate instances of your driver is the ::devbindings dcmd. Given a driver name, it displays a list of all instances of the named driver:

> ::devbindings dad
300015ce3d8      ide-disk (driver not attached)
300015c9a60      dad, instance #0
        System properties at 0x300015ab400:
            name='lun' type=int items=1
                value=00000000
            name='target' type=int items=1
                value=00000000
            name='class_prop' type=string items=1
                value='ata'
            name='type' type=string items=1
                value='ata'
            name='class' type=string items=1
                value='dada'
	...
300015c9880      dad, instance #1
        System properties at 0x300015ab080:
            name='lun' type=int items=1
                value=00000000
            name='target' type=int items=1
                value=00000002
            name='class_prop' type=string items=1
                value='ata'
            name='type' type=string items=1
                value='ata'
            name='class' type=string items=1
                value='dada'
	...

Retrieving Driver Soft State Information

A common problem when debugging a driver is retrieving the soft state for a particular driver instance. The soft state is allocated with the ddi_soft_state_zalloc(9F) routine and obtained by drivers using ddi_get_soft_state(9F). If you know the name of the soft state pointer (the first argument to ddi_soft_state_init(9F)), mdb enables you to retrieve the soft state for a particular driver instance using the ::softstate dcmd:

> *bst_state::softstate 0x3
702b7578

In this case, ::softstate is used to fetch the soft state for instance 3 of the bst sample driver. This pointer points to a bst_soft structure used by the driver to track state for this instance.

Detecting Kernel Memory Leaks

The ::findleaks dcmd provides powerful and efficient detection of memory leaks in kernel crash dumps. The full set of kernel-memory debugging features should be enabled for ::findleaks to be effective. For more information see kmem_flags. Running ::findleaks during driver development and testing can detect code which is leaking memory and wasting kernel resources. See “Debugging With the Kernel Memory Allocator” in the Solaris Modular Debugger Guide for a complete discussion of ::findleaks.


Note –

Use ::findleaks to detect and eliminate kernel memory leaks caused by your code. Code that leaks kernel memory can render the system vulnerable to denial-of-service attacks.


Writing Debugger Commands with mdb

mdb provides a powerful API for implementing new debugger facilities that you can use to debug your driver. The Solaris Modular Debugger Guide explains the programming API in more detail, and the SUNWmdbdm package installs sample mdb source code in the directory /usr/demo/mdb. You can use mdb to automate lengthy debugging chores or help to validate that your driver is behaving properly. You can also package your mdb debugging modules with your driver product so that these facilities will be available to service personnel at a customer site.