test

Sun Gathering Debug Data for Sun Java System Application Server

ProcedureTo Collect Debug Data on a Hung or Unresponsive Application Server Process

A process hang is defined as one of the Application Server processes not responding to requests while the appserv process is still running. On Solaris systems, you can use the appserver_8_hang script to gather debug data in hang situations. See 2.3.1 Running the appserver_8_hang.sh Debugging Script for detailed information about this script.

Before You Begin

Make sure that you collect all the data over the same time frame in which the problem occurs. See 2.2 Configuring Solaris OS to Generate Core Files if a core file is not generated.

Collect the following information for process hang problems. Run the commands in the order listed below when the problem occurs. Be sure to specify the time when the process hang happened and affected processes, if possible.

  1. Collect the general system information as explained in To Collect Required Debug Data for Any Application Server Problem.

  2. If your Application Server uses JDK 1.4.2 or higher, verify that the -XX:+PrintClassHistogram Application Server JVM option is enabled.

    For more information, see http://docs.sfbay.sun.com/source/819-0215/jvm.html.

  3. For the Solaris platform, use the ptree command on any of the following processes, depending on the component for which you want to collect data.

    DAS

    ps -ef | grep appservDAS

    Node agent or instance

    ps -ef | grep appserv

    • Sample DAS Output


      ptree 21293
21293 /export/home/as81ur2-02/lib/appservDAS domain1
 21299 /bin/sh /export/home/as81ur2-02/imq/bin/imqbrokerd -javahom
  21309 /export/home/pablo/as81ur2-02_caixa/jdk/bin/java -server -cp /export/
      Note –

      Collect data on the lowest PID process for the DAS, which in this example is 21293, or the highest PID for the instance, which is in this example is 178.

    • Sample Instance Output


      ptree 171
171 ./bin/sh /export/home/as82p1/nodeagents/draco.spain.sun.com/instac
 173 /export/home/as82p1/lib/appservLauncher /export/home/as82
  178 /export/home/as82p1/lib/appserv instacia_draco

  4. Run the netstat command and save the output.

    Solaris, HP-UX, Linux

    netstat -an | grep appserv-port

    Windows

    netstat -an

  5. For Solaris and Application Server 8, run the appserv_8_hang.sh script.

    The appserv_8_hang.sh script captures the debug data for Solaris and Application Server 8. After running this script, run the pkg_app script on one of the core files generated by the appserv_8_hang.

  6. Run the following commands and save the output.

    Solaris

    ps -aux | server-root

    vmstat 5 5

    iostat -xtopuptime

    HP-UX

    ps -aux | server-root

    vmstat 5 5

    iostat -xtopsar

    Linux

    ps -aux | server-root

    vmstat 5 5

    Windows

    Obtain the appserv or appservDAS process PID:


    C:\windbg-root>tlist.exe

    Obtain the process details of the appserv or appserveDAS running process PID:


    C:\windbg-root>tlist.exe appserv_PID

  7. Get the swap information.

    Solaris

    swap -l

    HP-UX

    swapinfo

    Linux

    free

    Windows

    Already provided in C:\report.txt.

  8. (Solaris only) If you are able to isolate the hung process, collect the following debug data for that process.

    Using the PID obtained in Step 3, above, get a series of five of the following commands (one every ten seconds):


    pstack appserv-pid
pmap -x appserv-pid

    Additionally, collect the outputs from the following commands:


    prstat -L -p appserv-pid
pfiles appserv-pid
pmap appserv-pid

  9. Get a Java Stack trace from the either appserv or appservDAS process.


    ptree 171
171 ./bin/sh /export/home/as82p1/nodeagents/draco.spain.sun.com/instac
 173 /export/home/as82p1/lib/appservLauncher /export/home/as82
  178 /export/home/as82p1/lib/appserv instacia_draco
kill -3 178

    The kill -3 dumps a Java stack trace for the server.log file.

  10. Gather core files and the output of the following commands.

    If a process hangs, it is helpful to compare several core files to review the state of the threads over time. Rename the core file, wait for approximately one minute, then rerun the following commands. In this way you can collect a series of core files without subsequent files overwriting each other. Do this three times to obtain three core files.

    Note –

    For HP-UX, you need the PHKL_31876 and PHCO_32173 patches to use the gcore command. If you cannot install these patches, use the HP-UX /opt/langtools/bin/gdb command from version 3.2 or later, or use the dumpcore command.

    • Solaris 


      gcore -o /tmp/appserver-core appserv-pid
pstack /tmp/appserver-core

    • HP-UX 


      # cd server-root/lib
gcore -p appserv-pid
(gdb) attach appserv-pid
Attaching to process appserv-pid
No executable file name was specified
(gdb) dumpcore
Dumping core to the core file core.appserv-pid
(gdb) quit
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: , process appserv-pid

    • Linux 


      # cd server-root/lib
gdb
(gdb) attach appserv-pid
Attaching to process appserv-pid
No executable file name was specified
(gdb) gcore
Saved corefile core.appserv-pid
(gdb)backtrace
(gdb)quit

    • Windows

      Get the appserv process PID:


      C:\windbg-root>tlist.exe

      Generate a crash dump on the appserv running process PID:


      C:\windbg-root>adplus.vbs -hang -p appserv-pid -o C:\crashdump_dir
      Note –

      For Windows, provide the complete generated folder under C:\crashdump_dir.

  11. For Solaris, archive the result of the pkg_app script (at least one core file is required).


    ./pkg_app.ksh application_pid corefile
    Note –

    Make sure that the appropriate limitations are set by using the ulimit command, and that the user ID is not nobody. Also check the coreadm command for additional control. See 2.2 Configuring Solaris OS to Generate Core Files if a core file is not generated.