Debugging a Mismatched Core File

Sometimes a core file is created on one system (the core-host) and you want to load the core file on another machine (the dbx-host) to debug it. However, two problems with libraries might arise when you do so:

• The shared libraries used by the program on the core-host might not be the same libraries as those on the dbx-host. To get proper stack traces involving the libraries, make these original libraries available on the dbx-host.

• dbx uses system libraries in /usr/lib to help understand the implementation details of the runtime linker and threads library on the system. You might also have to provide these system libraries from the core-host so that dbx can understand the runtime linker data structures and the threads data structures.

The user libraries and system libraries can change in patches as well as major Oracle Solaris operating system upgrades, so this problem can even occur on the same host, if, for example, a patch was installed after the core file was collected but before running dbx on the core file.

dbx might display one or more of the following error messages when you load a mismatched core file:

dbx: core file read error: address 0xff3dd1bc not available
dbx: warning: could not initialize librtld_db.so.1 -- trying libDP_rtld_db.so
dbx: cannot get thread info for 1 -- generic libthread_db.so error
dbx: attempt to fetch registers failed - stack corrupted
dbx: read of registers from (0xff363430) failed -- debugger service failed

Keep the following things in mind when debugging a mismatched core file:

• The pathmap command does not recognize a pathmap for ’/’ so you cannot use the following command:

  pathmap / /net/core-host

• The single-argument mode for the pathmap command does not work with load object path names, so use the two argument from-path to-path mode.

• Debugging the core file is likely to work better if the dbx-host has either the same or a more recent version of the Oracle Solaris operating system than the core-host, though this setup is not always necessary.

• The system libraries that you might need are as follows:

  • For the runtime linker:

    /usr/lib/ld.so.1

    /usr/lib/librtld_db.so.1

    /usr/lib/64/ld.so.1

    /usr/lib/64/librtld_db.so.1

  • For the threads library, depending on which implementation of libthread you are using:

    /usr/lib/libthread_db.so.1

    /usr/lib/64/libthread_db.so.1

    You will need the 64-bit versions of the xxx_db.so libraries if dbx is running on a 64-bit capable version of the Oracle Solaris OS since these system libraries are loaded and used as part of dbx, not as part of the target program.

    The ld.so.1 libraries are part of the core file image like libc.so or any other library, so you need the 32-bit ld.so.1 library or 64-bit ld.so.1 library that matches the program that created the core file.

• If you are looking at a core file from a threaded program and the where command does not display a stack, try using lwp commands. For example:.

  (dbx) where
  current thread: t@0
  [1] 0x0(), at 0xffffffff
  (dbx) lwps
  o>l@1 signal SIGSEGV in _sigfillset()
  (dbx) lwp l@1
  (dbx) where
  =>[1] _sigfillset(), line 2 in "lo.c"
    [2] _liblwp_init(0xff36291c, 0xff2f9740, ...
    [3] _init(0x0, 0xff3e2658, 0x1, ...
  ...

  The –setfp and –resetfp options of the lwp command are useful when the frame pointer (fp) of the LWP is corrupted. These options work when debugging a core file, where assign $fp=... is unavailable.

  The lack of a thread stack can indicate a problem with thread_db.so.1 Therefore, you might also want to try copying the proper libthread_db.so.1 library from the core-host.

To Eliminate Shared Library Problems and Debug a Mismatched Core File

1. Set the dbxenv variable core_lo_pathmap to on.
2. Use the pathmap command to indicate where the correct libraries for the core file are located.
3. Use the debug command to load the program and the core file.

   For example, assuming that the root partition of the core-host has been exported over NFS and can be accessed using /net/core-host/ on the dbx-host machine, you would use the following commands to load the program prog and the core file prog.core for debugging:

   (dbx) dbxenv core_lo_pathmap on
   (dbx) pathmap /usr /net/core-host/usr
   (dbx) pathmap /appstuff /net/core-host/appstuff
   (dbx) debug prog prog.core

   If you are not exporting the root partition of the core-host, you must copy the libraries by hand. You need not re-create the symbolic links. For example, you need not make a link from libc.so to libc.so.1; just make sure libc.so.1 is available.