test

Sun Java System Directory Server Enterprise Edition 6.2 Troubleshooting Guide

Analyzing the Directory Proxy Server 5.x Core Data on Solaris

Once you have obtained a core file, run the pstack and pmap Solaris utilities on the file.

Run the pstack utility as follows:


# pstack core-file > /tmp/pstack.txt

For example, the pstack utility creates the following output:


core '/var/core/core_dps-dr-zone1_ldapfwd_0_0_1156942096_3167' of 3167: ./ldapfwd 
-t /var/opt/mps/serverroot/dps-dps-dr-zone1/etc/tailor.txt
-----------------  lwp# 1 / thread# 1  --------------------
 fedc0b6c __pollsys (ffbff680, 2, ffbff610, 0, 0, 1770) + 8
 fed5cea8 poll     (ffbff680, 2, 1770, 10624c00, 0, 0) + 7c
 ff19c610 _pr_poll_with_poll (1770, ffbff680, 927c0, ffbff91c, 2, 1) + 42c
 00039504 __1cLCAI_LF_CmgrDrun6F_v_ (75, 115d74, 11202c, 2, 88c00, 116510) + 1a8
 00062070 ldapfwdMain (0, ffbffa84, c, 9952c, feb60740, feb60780) + 1c
 0002f968 _start   (0, 0, 0, 0, 0, 0) + 108
-----------------  lwp# 3 / thread# 3  --------------------
 fedc0b6c __pollsys (fea19b70, 3, fea19b00, 0, 0, 3e8) + 8
 fed5cea8 poll     (fea19b70, 3, 3e8, 10624c00, 0, 0) + 7c
 ff19c610 _pr_poll_with_poll (3e8, fea19b70, 186a0, fea19e8c, 3, 2) + 42c
 0005de38 __1cIidarPollEwait6MI_i_ (fea19e6c, 186a0, 16e360, 1, 0, 1) + 20
 000639ec __1cJHeartbeatJheartbeat6Fpv_1_ (158f40, 1, 14da70, faa36, 16e360, faa5d) + 114
 fedbfd9c _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 136734 / thread# 136734  --------------------
 fedbfe40 __lwp_park (0, 0, 116710, 0, 0, 1) + 14
 00076548 __1cMCAI_LF_MutexHacquire6M_v_ (116708, fd2bc, 0, 46, fea24800, 1000) + 34
 00076158 __1cPCAI_LF_RefCountGAddRef6M_v_ (116708, 156510, 2400, 26dc, 11667c, 800) + 24
 0006ddb0 __1cVCAI_LF_ReferralServer2t5B6MpnPCAI_LF_ConnPair_pnVCAI_LF_RequestMessage_pnNCAI_LF_Server
_ipcibbi_v_ (197198, fe7790d8, 1541c0, 156510, 185, 1565b8) + 50
 00046324 __1cPCAI_LF_ConnPairYstart_referral_operation6MipCpnNCAI_LF_Server_i_i_ (fe7790d8, 1, 1565b8
, 156510, 1, 197198) + 26c
 0004f2a0 __1cPCAI_LF_ConnPairLsend_result6MrnOCAI_LF_Message_rnVCAI_LF_RequestMessage__v_ (fe7790d8, 
156eb8, 1541c0, 2400, 0, 0) + 354
 00044758 __1cPCAI_LF_ConnPairOinner_activity6MpnOCAI_LF_Message__v_ (fe7790d8, 156eb8, 11202c, f4f27,
 fe7790f8, 1) + 114c
 00045c24 __1cPCAI_LF_ConnPairDrun6M_v_ (fe7797c8, 198383, 170c78, fe7790d8, fe779e54, fe779740) + 6cc
 00046cd0 CAI_LF_StartFunction (157500, 11202c, 1, 0, 46ab4, 1) + 21c
 fedbfd9c _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 136735 / thread# 136735  --------------------
 fedc0b6c __pollsys (fe9e8d98, 3, fe9e8d28, 0, 0, 1d4c0) + 8
 fed5cea8 poll     (fe9e8d98, 3, 1d4c0, 10624c00, 0, 0) + 7c
 ff19c610 _pr_poll_with_poll (1d4c0, fe9e8d98, b71b00, fe9e9e74, 3, 2) + 42c
 0005de38 __1cIidarPollEwait6MI_i_ (fe9e9e54, b71b00, 1, f4f27, fe9e90f8, 12d8d0) + 20
 00045cbc __1cPCAI_LF_ConnPairDrun6M_v_ (fe9e97c8, 17bb43, 130ff8, fe9e90d8, fe9e9e54, fe9e9740) + 764
 00046cd0 CAI_LF_StartFunction (157500, 11202c, 1, 0, 46ab4, 1) + 21c
 fedbfd9c _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 136738 / thread# 136738  --------------------
 fedc0b6c __pollsys (fe9a8d98, 3, fe9a8d28, 0, 0, 1d4c0) + 8
 fed5cea8 poll     (fe9a8d98, 3, 1d4c0, 10624c00, 0, 0) + 7c
 ff19c610 _pr_poll_with_poll (1d4c0, fe9a8d98, b71b00, fe9a9e74, 3, 2) + 42c
 0005de38 __1cIidarPollEwait6MI_i_ (fe9a9e54, b71b00, 1, f4f27, fe9a90f8, 0) + 20
 00045cbc __1cPCAI_LF_ConnPairDrun6M_v_ (fe9a97c8, 198123, 130fd8, fe9a90d8, fe9a9e54, fe9a9740) + 764
 00046cd0 CAI_LF_StartFunction (157500, 11202c, 1, 0, 46ab4, 1) + 21c
 fedbfd9c _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 136788 / thread# 136788  --------------------
 00197d68 ???????? (116708, 1, 156510, 0, 0, 1000)
 0006dee4 __1cVCAI_LF_ReferralServer2T6M_v_ (155780, 1000, fedecbc0, fea25c00, fe779780, 0) + 34
 000707c4 __SLIP.DELETER__A (155780, 1, 4, a6c, 1, 2400) + 4
 00046a94 CAI_LF_ReferralStartFunction (155780, fe6da000, 0, 0, 46a64, 1) + 30
 fedbfd9c _lwp_start (0, 0, 0, 0, 0, 0)

You can also use the mdb or adb command instead of the pstack command to show the stack of the core. The mdb command is a modular debugger and the adb command is a general purpose debugger that is part of Solaris. Run the mdb command as follows:


# mdb $path-to-executable $path-to-core
$C to show the core stack
$q to quit

The output of the mdb and the pstack commands provide helpful information about the process stack at the time of the crash. The mdb $C command output provides the exact thread that was executing at the time of the crash.

On Solaris 8 and 9, the first thread of the pstack output often contains the thread responsible for the crash. On Solaris 10, use mdb to find the crashing thread or, if using the pstack command, analyze the stack by looking for threads that do not contain lwp-park, poll, and pollsys.

On Solaris, you can also use the Solaris dbx symbolic debugger, which is a developer tool available free from http://sun.com/. The dbx tool provides symbolic debugging, and includes variables that can be manipulated. For example, the dbx debugger produces the following output:


current thread: t@2482121
=>[1] 0x0(0x156138, 0x0, 0xff000000, 0xfedefad4, 0x2, 0x2), at 0xffffffffffffffff
  [2] CAI_LF_RefCount::Release(0x116708, 0x1, 0x156138, 0x0, 0x0, 0x1000), at 0x7629c
  [3] CAI_LF_ReferralServer::~CAI_LF_ReferralServer(0x241270, 0x1000, 0xfedecbc0, 0xfe097c00, 
0xfe8d9780, 0x0), at 0x6dee4
  [4] __SLIP.DELETER__A(0x241270, 0x1, 0x4, 0xa6c, 0x1, 0x2400), at 0x707c4
  [5] CAI_LF_ReferralStartFunction(0x241270, 0xfe81a000, 0x0, 0x0, 0x46a64, 0x1), at 0x46a94