test

Sun Gathering Debug Data for Sun Java System Web Server

1.5 Types of Web Server Debug Data

This section describes the various kinds of debug data that you need to provide to the Sun Support Center. The procedure to obtain debug data based on the kind of problem you are experiencing is described in-detail.

This section contains the following topics:

ProcedureTo Gather General Debug Data for Any Web Server Problem

To report problems described in this technical note, you need to gather some basic information. Basic information includes System details and date and time when the problem occurred. Follow these steps to gather the basic information.

  1. Note the day(s) and time(s) the problem occurred.

  2. Provide a graphical representation of your deployment. Include all hosts and IP addresses, host names, operating system versions, role they perform, and other important systems such as load balancers, firewalls, and so forth.

  3. Note the Version of the operating system.

    Solaris

    uname -a

    HP-UX

    uname -r

    Linux

    more /etc/redhat-release

    Windows

    C:\Program Files\Common files\Microsoft Shared\MSInfo\msinfo32.exe /report C:\report.txt

  4. Note the patch level.

    Solaris

    showrev -p

    HP-UX

    swlist

    Linux

    rpm -qa

    Windows

    Already provided in the C:\report.txt file above.

  5. Note the version of Web Server.

    If a configured JDK is used instead of the default JRE then provide the output of the command java -version.

    Web Server version is indicated in the error log of the instance during the start.

    • Start Instance Script

      UNIX (Solaris and HP-UX) and Linux

      cd server-root/web-identifier/start

    • Error logs

      UNIX (Solaris and HP-UX) and Linux

      cd server-root/web-identifier/logs/errors

      Windows

      cd server-root\web-identifier\logs\errors

    • Access logs

      UNIX (Solaris and HP-UX) and Linux

      cd server-root/web-identifier/logs/access

      Windows

      cd server-root\web-identifier\logs\access

  6. Create a tar file of the Web Server configuration directory.

    • Sun Java System Web Server :

      UNIX (Solaris and HP-UX) and Linux

      cd server-root/web-identifier/configCreate a tar file of the server-root/config directory.

      Windows

      cd server-root\web-identifier\configCreate a compressed file of the server-root\config directory.

    Note –

    If possible, provide an explorer (SUNWexplo) of the machine where the problem occurs. For UNIX and Linux systems, the customer can use the script webinfo. For more information on how to run the webinfo script, see To Run the webinfo Script.

ProcedureTo Gather Debug Data on Web Server Installation Problems

Follow these steps if you are unable to complete the installation or if you get a “failed” installation status for Web Server.

  1. See the following troubleshooting information:

    • Sun Java Enterprise System 5:

      http://docs.sun.com/app/docs/doc/820-0464

    • Sun Java Enterprise System 2005Q4:

      Troubleshooting

      http://docs.sun.com/app/docs/doc/819-2328

    • Sun Java Enterprise System 2005Q1:

      http://docs.sun.com/source/819-0056/troubleshooting.html

    • Sun Java Enterprise System 2004Q2:

      http://docs.sun.com/source/817-5760/troubleshooting.html

    • Sun Java Enterprise System 2003Q4:

      http://docs.sun.com/source/816-6874/std-troubleshooting.html

    If the problem persists after using this troubleshooting information, then continue with this procedure to gather the necessary data for the Sun Support Center.

  2. Gather the general system information as explained in To Gather General Debug Data for Any Web Server Problem.

  3. Specify whether this is a first-time installation or a Hot Fix installation on a pre-existing Web Server.

  4. Get the installation logs.

    • Sun Java System Web Server (Web Server 7.0):

      Web Server 7.0 log files mostly reside in the server-root/log directory. However, the initial configuration log files reside in the server-root/install directory, which also contains information on the initial configuration.

      Solaris

      /var/sadm/install/logsThe log file names start with Java_Enterprise_System*_install.Bdatetime, where date and time correspond to the failing installing (for example, B12161532).

      HP-UX and Linux

      /var/opt/sun/install/logsThe log file names start with Java_Enterprise_System*_install.Bdatetime, where date and time correspond to the failing installing (for example, B12161532).

      Windows

      C:\DocumentsandSettings\current-user\LocalSettings\TempThe log file names start with MSI*.log (usually a text file). The asterisk (*) represents a random number in the Temp directory for each MSI based setup.

    • Sun Java System Web Server (Web Server 6.1):

      Web Server 6.1 log files mostly reside in the server-root/log directory. However, the initial configuration log files reside in the server-root/install directory, which also contains information on the initial configuration.

      Solaris

      /var/sadm/install/logsThe log file names start with Java_Enterprise_System*_install.Bdatetime, where date and time correspond to the failing installing (for example, B12161532).

      HP-UX and Linux

      /var/opt/sun/install/logsThe log file names start with Java_Enterprise_System*_install.Bdatetime, where date and time correspond to the failing installing (for example, B12161532).

      Windows

      C:\DocumentsandSettings\current-user\LocalSettings\TempThe log file names start with MSI*.log (usually a text file). The asterisk (*) represents a random number in the Temp directory for each MSI based setup.

    • iPlanet Web Server (Web Server 6.0):

      Rerun the installation with the following command and save the output file.

      Solaris

      truss -ealf -rall -wall -vall -o /tmp/install-web.truss ./setup

      HP-UX

      tusc -v -fealT -rall -wall -o /tmp/install-web.tusc ./setup

      Linux

      strace -fv -o /tmp/install-web.strace ./setup

      Windows

      Use DebugView tool. You can download this tool from http://www.sysinternals.com/Utilities/DebugView.html

ProcedureTo Gather Debug Data on Web Server Startup Problem

Follow these steps if you are unable to start a Web Server instance.

  1. Gather the general system information as explained in To Gather General Debug Data for Any Web Server Problem.

  2. Run the netstat command and save the output.

    UNIX (Solaris and HP-UX) and Linux

    netstat -an | grep web-port

    Windows

    netstat -an

  3. Run the following command on the Web Server start script and provide the resultant file.

    Solaris

    truss -eafl -wall -vall -rall -o /tmp/web-start.truss ./start

    HP-UX

    tusc -v -fealT -rall -wall -o /tmp/web-start.tusc ./start

    Linux

    strace -fv -o /tmp/web-start.strace ./start

    Windows

    Use DebugView tool. You can download this tool from http://www.sysinternals.com/Utilities/DebugView.html

  4. If logs file does not contain any error message about the problem, do the following:

    Edit and add the following line to the configuration file to get more debug information during the start.

    UNIX (Solaris and HP-UX) and Linux

    Edit the file server-root/web-identifier/config/server.xml and change the loglevel to finest: loglevel=finest in server.xml.

    Windows

    Edit the file server-root\web-identifier\config\server.xml and change the loglevel to finest: loglevel=finest in server.xml.

ProcedureTo Gather Debug Data on a Hung or Unresponsive Web Server Process

A process hang is defined as one of the Web Server processes not responding to requests while the httpd process is still running.

Before You Begin

Make sure that you collect all the data over the same time frame in which the problem occurs. See 1.6 Configuring Solaris to Generate Core Files if a core file is not generated.

Gather the following information for process hang problems. Run the commands in the order when the problem occurs. Be sure to specify the time when the process hanged and list the affected processes, if possible.

  1. Gather the general system information as explained in To Gather General Debug Data for Any Web Server Problem.

  2. For Solaris, use the ptree command on the uxwdog process to find about the process.

    Note –

    If you are using Web Server 6.1 or Web Server 7.0, instead of the uxwdog process, use the webservd-wdog process.

    Output


    ptree 11449
11449 ./uxwdog -d /prods/crypto/60SP6/https-sun/config
 11450 ns-httpd -d /prods/crypto/60SP6/https-sun/config
  11451 ns-httpd -d /prods/crypto/60SP6/https-sun/config
    Note –

    Gather the data on the highest PID process, which in this example is 11451. The Web Process is either ns-httpd or webservd, depending on the Web Server version.

  3. Run the netstat command and save the output.

    UNIX (Solaris and HP-UX) and Linux

    netstat -an | grep web-port

    Windows

    netstat -an

  4. (For Solaris), wshang script captures the debug data.

    The wshang script is available at: http://www.sun.com/bigadmin/scripts/indexSjs.html

    Run the script pkg_app on one of the core file generated by the wshang script. For more information on how to run thewshang script, see To Run the wshang Script.

  5. Run the following commands and save the output.

    Solaris
    ps -aux
server-root
vmstat 5 5
iostat [ -t ] [ interval [ count ] ]
top
uptime
    HP-UX
    ps -aux
server-root
vmstat 5 5
iostat [ -t ] [ interval [ count ] ]
top
sar
    Linux
    ps -aux
server-root
vmstat 5 5
top
uptime
sar
    Windows

    Obtain the WEB process PID:

    C:\windbg-root>tlist.exe

    Obtain the process details of the WEB running process PID:

    C:\windbg-root>tlist.exe web-pid

  6. Get the swap information.

    Solaris

    swap -l

    HP-UX

    swapinfo

    Linux

    free

    Windows

    Already provided in C:\report.txt as described in To Gather General Debug Data for Any Web Server Problem.

  7. If the Web Server uses a Directory Server, provide the access, errors and audit logs of the Directory Server used by the Web Server.

    • Access log

      UNIX (Solaris and HP-UX) and Linux

      server-root/slapd-identifier/logs/access

      Windows

      server-root\slapd-identifier\logs\access

    • Errors log

      UNIX (Solaris and HP-UX) and Linux

      server-root/slapd-identifier/logs/errors

      Windows

      server-root\slapd-identifier\logs\errors

    • Audit log

      UNIX (Solaris and HP-UX) and Linux

      server-root/slapd-identifier/logs/audit

      Windows

      server-root\slapd-identifier\logs\audit

    Note –

    The paths of these logs files are specified by the following parameters in the dse.1dif file.nsslapd-accesslog,nsslap-errorlog, and nsslapd-auditlog

    The dse.1dif file is located in the config directory.

    UNIX (Solaris and HP-UX) and Linux

    server-root/slapd-identifier/config/dse.ldif

    Windows

    server-root\slapd-identifier\config\dse.ldif

  8. (For Solaris) If you are able to isolate the hanging process, get the following debug data for that process. Otherwise, get the following data for each of the Web Server processes.

    For Solaris

    Using the PID obtained in Step 3, get a series of five of the following commands (one every 10 seconds) :

    pstack web-pid

    pmap -x web-pid

    Additionally, get the outputs of the following commands:

    prstat -L -p web-pid

    pfiles web-pid

    pmap web-pid

  9. Search for any core file that could have been dumped by one of the Web Server processes. If you find one, see To Gather Debug Data on Web Server Crashed Process.

  10. Get the output of the following command.

    Solaris

    truss -ealf -rall -wall -vall -o /tmp/WEBProc-PID -p web-pid

    HP-UX

    tusc -v -fealT -rall -wall -o /tmp/WEBProc-PID -p web-pid

    Linux

    strace -fv -o /tmp/WEBProc-PID.strace -p web-pid

    Windows

    Use DebugView tool. You can download this tool from http://www.sysinternals.com/Utilities/DebugView.html

    Note –

    Wait for a minute after launching the appropriate command (truss, strace, tusc, or DebugView) then stop it by pressing Control+C in the terminal where you launched the command.

  11. Get core files and the output of the following commands.

    If a process hangs, it is helpful to compare several core files to review the state of the threads over time. Make a copy of the core file to a new name, wait for approximately one minute then rerun the following commands, so that the core files are not overwritten. Do this three times to obtain three core files.

    Note –

    For HP-UX, you need PHKL_31876 and PHCO_32173 patches to use the gcore command. If you cannot install these patches, use the HP-UX /opt/langtools/bin/gdb command from version 3.2 and later, or the dumpcore command.

    Solaris

    cd server-root/bin/https/bin;gcore -o /tmp/web-core web-pid;pstack /tmp/web-core

    HP-UX

    # cd server-root/bin/https/bin
gcore -p web-pid
(gdb) attach web-pid
Attaching to process web-pid
No executable file name was specified
(gdb) dumpcore
Dumping core to the core file core.web-pid
(gdb) quit
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: , process web-pid
    Note –

    The core.web-pid should be generated in the web-identifier/config directory.

    Linux

    # cd server-root/bin/https/bin
gdb
(gdb) attach web-pid
Attaching to process web-pid
No executable file name was specified
(gdb) gcore
Saved corefile core.web-pid

(gdb)backtrace
(gdb)quit
    Windows

    Get the WEB process PID:

    C:\windbg-root>tlist.exe

    Generate a crash dump on the WEB running process PID:

    C:\windbg-root>adplus.vbs -hang -p web-pid -o C:\crashdump_dir

    Note –

    For Windows, provide the complete generated folder under C:\crashdump_dir.

  12. For Solaris, Archive the result of the script pkg_app (at least one core file is required).

    ./pkg_app.ksh -c [pid-of-application or corefile] -p <full path to process binary of webservd>

    The Sun Support Center requires the output from the pkg_app script to properly analyze the core file(s). For more information on how to run the pkg_app script, see To Run the pkg_app Script

    Note –

    Make sure that the appropriate limitations are set by using the ulimit command, and that the user is not nobody. Also check the coreadm command for additional control. See 1.6 Configuring Solaris to Generate Core Files if a core file is not generated.

    Note –

    If you are using Web Server 6.1 or Web Server 7.0, do not proceed further with the next step.

  13. For UNIX and Linux, If JVM is used for the Web applications, provide the JVM Stack traces during a hang situation.

    A series of three to five Stack traces will be required.

    To enable thread dumps for version 6.0, perform the following steps:

    1. Edit the configuration file

      server-root/https-host/obj.conf

    2. Modify the following line

      Init fn="NSServletLateInit" LateInit=yes

      to

      Init LateInit="yes" fn="NSServletInit" CatchSignals="yes" Signals=SIGQUIT

    3. Add or modify the following line in /server-root/https-host/jvm12.conf

      jvm.printerrors=1

    4. Restart Web Server.

    Note –

    When a problem occurs during a restart, issuing a kill —3 against the process dumps the stack traces into the Web Server errors log.

ProcedureTo Gather Debug Data on Web Server Crashed Process

Use this task to gather data when a Web Server process has stopped (crashed) unexpectedly. Run all the commands on the actual machine where the core file(s) were generated.

  1. Gather the general system information as explained in To Gather General Debug Data for Any Web Server Problem.

  2. Try to restart Web Server.

  3. If the Web Server is using a Directory Server, provide the access, errors and audit logs of the Directory Server used by the Web Server

    • Access log

      UNIX (Solaris and HP-UX) and Linux

      server-root/slapd-identifier/logs/access

      Windows

      server-root\slapd-identifier\logs\access

    • Errors log

      UNIX (Solaris and HP-UX) and Linux

      server-root/slapd-identifier/logs/errors

      Windows

      server-root\slapd-identifier\logs\errors

    • Audit log

      UNIX (Solaris and HP-UX) and Linux

      server-root/slapd-identifier/logs/audit

      Windows

      server-root\slapd-identifier\logs\audit

    Note –

    The paths of these logs files are specified by the following parameters in the dse.1dif file.nsslapd-accesslog,nsslap-errorlog, and nsslapd-auditlog

    The dse.1dif file is located in the config directory.

    UNIX (Solaris and HP-UX) and Linux

    server-root/slapd-identifier/config/dse.ldif

    Windows

    server-root\slapd-identifier\config\dse.ldif

  4. Get the output of the following commands.

    Solaris

    ps -aux | server-rootvmstat 5 5iostat -xtopuptime

    HP-UX

    ps -aux | server-rootvmstat 5 5iostat -xtopsar

    Linux

    ps -aux | server-rootvmstat 5 5topuptimesar

    Windows

    Obtain the WEB process PID:

    C:\windbg-root>tlist.exe

    Obtain process details of the WEB running process PID:

    C:\windbg-root>tlist.exe web-pid

  5. Get the swap information.

    Solaris

    swap -l

    HP-UX

    swapinfo

    Linux

    free

    Windows

    Already provided in C:\report.txt as described in To Gather General Debug Data for Any Web Server Problem.

  6. Get the system logs.

    Solaris and Linux

    /var/adm/messages/var/log/syslog

    HP-UX

    /var/adm/syslog/syslog.log

    Windows

    Event log files:Start-> Settings-> Control Panel —> Event Viewer-> Select LogThen click Action-> Save log file as and type the name for the resulting file.

  7. Get core files (called “Crash Dumps” in Windows).

    Solaris

    See 1.6 Configuring Solaris to Generate Core Files if a core file was not generated.

    Linux

    Core dumps are turned off by default in the /etc/profile file. You can make user-specific changes by editing your ~/.bash_profile file. Look for the following line:

    ulimit -S -c 0 > /dev/null 2>&1

    You can either comment out the entire line to set no limit on the size of the core files or set your own maximum size.

    Windows

    Generate a crash dump during a crash of Web Server by using the following commands:

    Get the WEB process PID :

    C:\windbg-root>tlist.exe

    Generate a crash dump when the WEB process crashes, by executing the following commands:

    C:\windbg-root>adplus.vbs -crash -FullOnFirst -p web-pid -o C:\crashdump_dir

    The adplus.vbs command monitors web-pid until it crashes and generates the dmp file. Provide the complete generated folder under C:\crashdump_dir.

    Note –

    If you have not installed the Debugging Tools for Windows, you can use the drwtsn32 -i command to select Dr. Watson as the default debugger. Use the drwtsn32 command, check all options, and choose the path for crash dumps. Then provide the dump and the drwtsn32.log files.

  8. (Solaris) For each core file, provide the output of the following commands.

    cd server-root/bin/https/bin
file corefile
pstack corefile
pmap corefile
pflags corefile

  9. (Solaris) Archive the result of the script pkg_app (one core file is sufficient).

    ./pkg_app.ksh Pid-of-application corefile
    Note –

    The Sun Support Center must have the output from the pkg_app script to properly analyze the core file(s). For more information on how to run the pkg_app script, see To Run the pkg_app Script.

    All these commands must be executed on the actual machine where the core file(s) were generated.