9.2.7 Guidelines for Examining a Dump File

The steps for debugging a memory dump from a kernel crash vary widely according to the problem. The following guidelines suggest some basic investigations that you can try:

  • Use bt to trace the functions that led to the kernel panic.

  • Use bt -a to trace the active task on each CPU. There is often a relationship between the panicking task on one CPU and the running tasks on the other CPUs. If the listed command is cpu_idle or swapper, no task was running on a CPU.

  • Use bt -l to display the line number of the source files corresponding to each function call in the stack trace.

  • Use kmem -i to obtain a summary of memory and swap usage. Look for a SLAB value greater than 500 MB and a SWAP USED value greater than 0%.

  • Use ps | grep UN to check for processes in the TASK_UNINTERRUPTIBLE state (D state), usually because they are waiting on I/O. Such processes contribute to the load average and cannot be killed.

  • Use files to display the files that a process had open.

You can shell indirection operators to save output from a command to a file for later analysis or to pipe the output through commands such as grep, for example:

crash> foreach files > files.txt
crash> foreach bt | grep bash
PID: 3685   TASK: ffff880058714580  CPU: 1   COMMAND: "bash"
PID: 11853  TASK: ffff88001c6826c0  CPU: 0   COMMAND: "bash"