Examining a Core File on Solaris

Once you have obtained a core file, run the pstack and pmap Solaris utilities on the file. The pmap utility shows the process map, which includes a list of virtual addresses, where the dynamic libraries are loaded, and where the variables are declared. The pstack utility shows the process stack. For each thread in the process, it describes the exact stack of instruction the thread was executing at the moment when the process died or when the pstack command was executed.

# pstack core-file

# pmap core-file

If the results of the pstack utility are almost empty, all of the lines in the output look as follows:

0002c3cc ???????? (1354ea0, f3400, 1354ea0, 868, 2fc, 1353ff8)

In this case, make sure to run pstack on the machine where the core file was generated.

You can also use the mdb command instead of the pstack command to know the stack of the core. Run the mdb command as follows:

# mdb $path-to-executable $path-to-core
$C to show the core stack
$q to quit

The output of the mdb and the pstack commands provide helpful information about the process stack at the time of the crash. The mdb $C command output provides the exact thread that caused the crash.

On Solaris 9, the first thread of the pstack output often contains the thread responsible for the crash. On Solaris 10, use mdb to find the crashing thread or, if using the pstack command, analyze the stack by looking for threads that do not contain lwp-park, poll, and pollsys.

For example, the following core process stack occurs during the call of a plug-in function:

core '/local/dsInst/logs/core' of 18301:    ./ns-slapd \
-D /local/dsInst -i /local/dsInst
-----------------  lwp# 13 / thread# 25  --------------------
 ff2b3148 strlen   (0, fde599fb, 0, fbed1, 706d2d75, fde488a8) + 1c
 ff307ef8 sprintf  (7fffffff, fde488a0, fde599d8, fde599ec, 706d2d75, fde599fc) \
+ 3c
 fde47cf8 ???????? (1354ea0, 850338, fde59260, e50243, 923098, 302e3800) + f8
 fde429cc ???????? (1354ea0, 3, 440298, 154290, 345c10, 154290) + 614
 ff164018 plugin_call_exop_plugins (1354ea0, 8462a0, d0c, ff1e7c70, ff202a94, \
1353ff8) + d0
 0002c3cc ???????? (1354ea0, f3400, 1354ea0, 868, 2fc, 1353ff8)
 00025e08 ???????? (0, 1353ff8, fdd02a68, f3400, f3000, fbc00)
 fef47d18 _pt_root (362298, fe003d10, 0, 5, 1, fe401000) + a4
 fed5b728 _thread_start (362298, 0, 0, 0, 0, 0) + 40

When analyzing process stacks from cores, concentrate on the operations in the middle of the thread. Processes at the bottom are too general and processes at the top are too specific. The commands in the middle of the thread are specific to the Directory Server and can thus help you identify at which point during processing the operation failed. In the above example, we see the plugin_call_exop_plugins process call indicates a problem calling an external operation in the custom plug-in.

If the problem is related to the Directory Server, you can use the function call that seems like the most likely cause of the problem to search on SunSolve for known problems associated with this function call. SunSolve is located at http://sunsolve.sun.com/.

If you do locate a problem related to the one you are experiencing, confirm that it applies to the version of Directory Server that you are running. To get information about the version you are running, use the following command:

# dsadm -V

If after doing a basic analysis of your core files you cannot identify the problem, collect the binaries and libraries using the pkgapp script and contact the Oracle Support Center.