Displays a kernel stack trace of the current context or of a specified PID or task. In the case of a dump that followed a kernel panic, the command traces the functions that were called leading up to the panic. For example:

crash> bt
PID: 10651  TASK: d1347000  CPU: 1   COMMAND: "insmod"
 #0 [d1547e44] die at c010785a
 #1 [d1547e54] do_invalid_op at c0107b2c
 #2 [d1547f0c] error_code (via invalid_op) at c01073dc
...

You can use the -l option to display the line number of the source file that corresponds to each function call in a stack trace.

crash> bt -l 1
PID: 1      TASK: ffff88007d032040  CPU: 1   COMMAND: "init"
 #0 [ffff88007d035878] schedule at ffffffff8144fdd4
    /usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/kernel/sched.c: 3091
 #1 [ffff88007d035950] schedule_hrtimeout_range at ffffffff814508e4
    /usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/arch/x86/include/asm/current.h: 14
 #2 [ffff88007d0359f0] poll_schedule_timeout at ffffffff811297d5
    /usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/arch/x86/include/asm/current.h: 14
 #3 [ffff88007d035a10] do_select at ffffffff81129d72
    /usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/fs/select.c: 500
 #4 [ffff88007d035d80] core_sys_select at ffffffff8112a04c
    /usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/fs/select.c: 575
 #5 [ffff88007d035f10] sys_select at ffffffff8112a326
    /usr/src/debug/kernel-2.6.32/linux-2.6.32.x86_64/fs/select.c: 615
 #6 [ffff88007d035f80] system_call_fastpath at ffffffff81011cf2
    /usr/src/debug////////kernel-2.6.32/linux-2.6.32.x86_64/arch/x86/kernel/entry_64.S:
    488
    RIP: 00007fce20a66243  RSP: 00007fff552c1038  RFLAGS: 00000246
    RAX: 0000000000000017  RBX: ffffffff81011cf2  RCX: ffffffffffffffff
    RDX: 00007fff552c10e0  RSI: 00007fff552c1160  RDI: 000000000000000a
    RBP: 0000000000000000   R8: 0000000000000000   R9: 0000000000000200
    R10: 00007fff552c1060  R11: 0000000000000246  R12: 00007fff552c1160
    R13: 00007fff552c10e0  R14: 00007fff552c1060  R15: 00007fff552c121f
    ORIG_RAX: 0000000000000017  CS: 0033  SS: 002b

bt is probably the most useful crash command. It has a large number of options that you can use to examine a kernel stack trace. For more information, enter help bt.

Displays character and block device data. The -d and -i options display disk I/O statistics and I/O port usage. For example:

crash> dev
CHRDEV    NAME                 CDEV        OPERATIONS      
   1      mem            ffff88007d2a66c0  memory_fops
   4      /dev/vc/0      ffffffff821f6e30  console_fops
   4      tty            ffff88007a395008  tty_fops
   4      ttyS           ffff88007a3d3808  tty_fops
   5      /dev/tty       ffffffff821f48c0  tty_fops
...
BLKDEV    NAME                GENDISK      OPERATIONS      
   1      ramdisk        ffff88007a3de800  brd_fops
 259      blkext              (none)     
   7      loop           ffff880037809800  lo_fops
   8      sd             ffff8800378e9800  sd_fops
   9      md                  (none)     
...
crash> dev -d
MAJOR GENDISK            NAME       REQUEST QUEUE      TOTAL ASYNC  SYNC   DRV
    8 0xffff8800378e9800 sda        0xffff880037b513e0    10     0    10     0
   11 0xffff880037cde400 sr0        0xffff880037b50b10     0     0     0     0
  253 0xffff880037902c00 dm-0       0xffff88003705b420     0     0     0     0
  253 0xffff880037d5f000 dm-1       0xffff88003705ab50     0     0     0     0
crash> dev -i
    RESOURCE        RANGE    NAME
ffffffff81a9e1e0  0000-ffff  PCI IO
ffffffff81a96e30  0000-001f  dma1
ffffffff81a96e68  0020-0021  pic1
ffffffff81a96ea0  0040-0043  timer0
ffffffff81a96ed8  0050-0053  timer1
ffffffff81a96f10  0060-0060  keyboard
...

Displays information about files that are open in the current context or in the context of a specific PID or task. For example:

crash> files 12916
PID: 12916  TASK: ffff8800276a2480  CPU: 0   COMMAND: "firefox"
ROOT: /    CWD: /home/guest
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff88001c57ab00 ffff88007ac399c0 ffff8800378b1b68 CHR  /null
  1 ffff88007b315cc0 ffff88006046f800 ffff8800604464f0 REG  /home/guest/.xsession-errors
  2 ffff88007b315cc0 ffff88006046f800 ffff8800604464f0 REG  /home/guest/.xsession-errors
  3 ffff88001c571a40 ffff88001d605980 ffff88001be45cd0 REG  /home/guest/.mozilla/firefox
  4 ffff88003faa7300 ffff880063d83440 ffff88001c315bc8 SOCK
  5 ffff88003f8f6a40 ffff88007b41f080 ffff88007aef0a48 FIFO
...

Displays the tasks that reference a specified file name or inode address as the current root directory, current working directory, open file descriptor, or that memory map the file. For example:

crash> fuser /home/guest
 PID         TASK        COMM             USAGE
 2990  ffff88007a2a8440  "gnome-session"  cwd 
 3116  ffff8800372e6380  "gnome-session"  cwd 
 3142  ffff88007c54e540  "metacity"       cwd 
 3147  ffff88007aa1e440  "gnome-panel"    cwd 
 3162  ffff88007a2d04c0  "nautilus"       cwd 
 3185  ffff88007c00a140  "bluetooth-appl  cwd 
...

Displays interrupt request queue data. For example:

crash> irq 0
    IRQ: 0
 STATUS: 400000 ()
HANDLER: ffffffff81b3da30            <ioapic_chip>
         typename: ffffffff815cdaef  "IO-APIC"
          startup: ffffffff8102a513  <startup_ioapic_irq>
         shutdown: ffffffff810aef92  <default_shutdown>
           enable: ffffffff810aefe3  <default_enable>
          disable: ffffffff810aeecc  <default_disable>
              ack: ffffffff8102a43d  <ack_apic_edge>
             mask: ffffffff81029be1  <mask_IO_APIC_irq>
...

Displays the state of the kernel memory subsystems. For example:

crash> kmem -i
              PAGES        TOTAL      PERCENTAGE
 TOTAL MEM   512658         2 GB         ----
      FREE    20867      81.5 MB    4% of TOTAL MEM
      USED   491791       1.9 GB   95% of TOTAL MEM
    SHARED   176201     688.3 MB   34% of TOTAL MEM
   BUFFERS     8375      32.7 MB    1% of TOTAL MEM
    CACHED   229933     898.2 MB   44% of TOTAL MEM
      SLAB    39551     154.5 MB    7% of TOTAL MEM

TOTAL SWAP  1032190       3.9 GB         ----
 SWAP USED     2067       8.1 MB    0% of TOTAL SWAP
 SWAP FREE  1030123       3.9 GB   99% of TOTAL SWAP

kmem has a large number of options. For more information, enter help kmem.

Displays the kernel message buffer in chronological order. This is the same data that dmesg displays but the output can include messages that never made it to syslog or disk.

Displays machine-specific information such as the cpuinfo structure and the physical memory map.

Displays information about the currently installed kernel modules. The -s and -S options load debug data (if available) from the specified module object files to enable symbolic debugging.

Displays information about currently mounted file systems.

Displays network-related information.

Displays information about processes. For example:

crash> ps Xorg crash bash
   PID    PPID  CPU       TASK        ST  %MEM     VSZ    RSS  COMM
   2679   2677   0  ffff88007cbcc400  IN   4.0  215488  84880  Xorg
> 13362  11853   0  ffff88007b25a500  RU   6.9  277632 145612  crash
   3685   3683   1  ffff880058714580  IN   0.1  108464   1984  bash
  11853  11845   1  ffff88001c6826c0  IN   0.1  108464   1896  bash

Translates a page table entry (PTE) to the physical page address and page bit settings. If the PTE refers to a swap location, the command displays the swap device and offset.

Displays the list of tasks that are on the run queue of each CPU.

Displays signal-handling information for the current context or for a specified PID or task.

Displays information about the configured swap devices.

Displays the contents of the task_struct for the current context or for a specified PID or task.

Displays the entries in the timer queue in chronological order.

Displays the virtual memory data, including the addresses of mm_struct and the page directory, resident set size, and total virtual memory size for the current context or for a specified PID or task.

Translates a user or kernel virtual address to a physical address. The command also displays the PTE translation, vm_area_struct data for user virtual addresses, mem_map page data for a physical page, and the swap location or file location if the page is not mapped.

Displays tasks that are blocked on a specified wait queue.