Debugging a Core File

If the program that dumped core was dynamically linked with any shared libraries, it is best to debug the core file in the same operating environment in which it was created. dbx has limited support for the debugging of “mismatched” core files (for example, core files produced on a system running a different version or patch level of the Solaris Operating System.

Debugging a Core File in the Same Operating Environment

To debug a core file, type:

$ dbx program_name core

or

$ dbxtool program_name core

If you type the following, dbx determines the program_name from the core file:

$ dbx - core

or

$ dbxtool - core

You can also debug a core file using the debug command when dbx is already running:

(dbx) debug -c core program_name

You can substitute - for the program name and dbx will attempt to extract the program name from the core file. dbx might not find the executable if its full path name is not available in the core file. If dbx does not find the executable, specify the complete path name of the binary when you tell dbx to load the core file.

If the core file is not in the current directory, you can specify its path name (for example, /tmp/core).

Use the where command (see where Command) to determine where the program was executing when it dumped core.

When you debug a core file, you can also evaluate variables and expressions to see the values they had at the time the program crashed, but you cannot evaluate expressions that make function calls. You cannot single step. You can set breakpoints and then rerun the program.

If Your Core File Is Truncated

If you have problems loading a core file, check whether you have a truncated core file. If you have the maximum allowable size of core files set too low when the core file is created, then dbx cannot read the resulting truncated core file. In the C shell, you can set the maximum allowable core file size using the limit command (see the limit(1) man page). In the Bourne shell and Korn shell, use the ulimit command (see the limit(1) man page). You can change the limit on core file size in your shell start-up file, re-source the start-up file, and then rerun the program that produced the core file to produce a complete core file.

If the core file is incomplete, and the stack segment is missing, then stack trace information is not available. If the runtime linker information is missing, then the list of loadobjects is not available. In this case, you get an error message about librtld_db.so not being initialized. If the list of LWPs is missing, then no thread information, lwp information, or stack trace information is available.If you run the where command, you get an error saying the program was not “active.”

Debugging a Mismatched Core File

Sometimes a core file is created on one system (the core-host) and you want to load the core file on another machine (the dbx-host) to debug it. However, two problems with libraries might arise when you do so:

• The shared libraries used by the program on the core-host might not be the same libraries as those on the dbx-host. To get proper stack traces involving the libraries, you’ll want to make these original libraries available on the dbx-host.

• dbx uses system libraries in /usr/lib to help understand the implementation details of the run time linker and threads library on the system. It may also be necessary to provide these system libraries from the core-host so that dbx can understand the runtime linker data structures and the threads data structures.

The user libraries and system libraries can change in patches as well as major Solaris operating environment upgrades, so this problem can even occur on the same host, if, for example, a patch was installed after the core file was collected, but before running dbx on the core file.

dbx might display one or more of the following error messages when you load a “mismatched” core file:

dbx: core file read error: address 0xff3dd1bc not available
dbx: warning: could not initialize librtld_db.so.1 -- trying libDP_rtld_db.so
dbx: cannot get thread info for 1 -- generic libthread_db.so error
dbx: attempt to fetch registers failed - stack corrupted
dbx: read of registers from (0xff363430) failed -- debugger service failed

Eliminating Shared Library Problems

To Eliminate the Library Problems and Debug a “mismatched” Core File

1. Set the dbx environment variable core_lo_pathmap to on.
2. Use the pathmap command to tell dbx where the correct libraries for the core file are located.
3. Use the debug command to load the program and the core file.

   For example, assuming that the root partition of the core-host has been exported over NFS and can be accessed using /net/core-host/ on the dbx-host machine, you would use the following commands to load the program prog and the core file prog.core for debugging:

   (dbx) dbxenv core_lo_pathmap on
   (dbx) pathmap /usr /net/core-host/usr
   (dbx) pathmap /appstuff /net/core-host/appstuff
   (dbx) debug prog prog.core

   If you are not exporting the root partition of the core-host, you must copy the libraries by hand. You need not re-create the symbolic links. (For example, you need not make a link from libc.so to libc.so.1; just make sure libc.so.1 is available.)

Things to Remember

Keep the following things in mind when debugging a mismatched core file:

• The pathmap command does not recognize a pathmap for ’/’ so you cannot use the following command:

  pathmap / /net/core-host

• The single-argument mode for the pathmap command does not work with load object pathnames, so use the two argument from-path to-path mode.

• Debugging the core file is likely to work better if the dbx-host has either the same or a more recent version of the Solaris operating environment than the core-host, though this is not always necessary.

• The system libraries that you might need are:

  • For the runtime linker:

    /usr/lib/ld.so.1

    /usr/lib/librtld_db.so.1

    /usr/lib/64/ld.so.1

    /usr/lib/64/librtld_db.so.1

  • For the threads library, depending on which implementation of libthread you are using:

    /usr/lib/libthread_db.so.1

    /usr/lib/64/libthread_db.so.1

    You will need the 64-bit versions of the xxx_db.so libraries if dbx is running on a 64-bit capable version of the Solaris OS since these system libraries are loaded and used as part of dbx, not as part of the target program.

    The ld.so.1 libraries are part of the core file image like libc.so or any other library, so you need the 32-bit ld.so.1 library or 64-bit ld.so.1 library that matches the program that created the core file.

• If you are looking at a core file from a threaded program, and the where command does not display a stack, try using lwp commands. For example:.

  (dbx) where
  current thread: t@0
  [1] 0x0(), at 0xffffffff
  (dbx) lwps
  o>l@1 signal SIGSEGV in _sigfillset()
  (dbx) lwp l@1
  (dbx) where
  =>[1] _sigfillset(), line 2 in "lo.c"
    [2] _liblwp_init(0xff36291c, 0xff2f9740, ...
    [3] _init(0x0, 0xff3e2658, 0x1, ...
  ...

  The -setfp and -resetfp options of the lwp command are useful when the frame pointer (fp) of the LWP is corrupted. These options work when debugging a core file, where assign $fp=... is unavailable.

  The lack of a thread stack can indicate a problem with thread_db.so.1 Therefore, you might also want to try copying the proper libthread_db.so.1 library from the core-host.