test

Sun Gathering Debug Data for Sun Java System Web Proxy Server

1.5 What Web Proxy Server Debug Data Should You Collect?

This section describes the various kinds of debug data that you need to provide to the Support center. The procedure to obtain debug data based on the kind of problem you are experiencing is described in-detail.

This section contains the following tasks:

Procedure1.5.1 To Collect Required Debug Data for Any Web Proxy Server Problem

To report problems described in this technical note, you need to collect some basic information. Basic information includes system details and date and time when the problem occurred. Follow these steps to collect the basic information.

  1. Note the day(s) and time(s) the problem occurred.

  2. Provide a graphical representation of your deployment. Include all hosts and IP addresses, host names, operating system versions, role they perform, and other important systems such as load balancers, firewalls, and so on.

  3. Note the version of the operating system.

    Solaris

    uname -a

    HP-UX

    uname -r

    Linux

    more /etc/redhat-release

    Windows

    C:\Program Files\Common files\Microsoft Shared\MSInfo\msinfo32.exe /report C:\report.txt

  4. Note the patch level.

    Solaris

    showrev -p

    HP-UX

    swlist

    Linux

    rpm -qa

    Windows

    Already provided in the C:\report.txt file.

  5. Note the version of Web Proxy Server.

    If a configured JDK is used instead of the default JRE then provide the output of the command java -version.

    Web Proxy Server Version is indicated in the error log of the instance during the start.

    • Start Instance Script

      UNIX and Linux

      server-root/proxy-identifier/start

    • Error logs

      UNIX and Linux

      server-root/proxy-identifier/logs/errors

      Windows

      server-root\proxy-identifier\logs\errors

    • Access logs

      UNIX and Linux

      server-root/proxy-identifier/logs/access

      Windows

      server-root\proxy-identifier\logs\access

  6. Create a tar file of the Web Proxy Server configuration directory.

    • Sun Java System Web Proxy Server :

      UNIX and Linux

      server-root/proxy-identifier/config Create a tar file of the server-root/proxy-identifier/configdirectory.

      Windows

      server-root/proxy-identifier\config Create a tar file of the server-rootproxy-identifier\config directory.

    Note –

    If possible, provide an explorer (SUNWexplo) of the machine where the problem occurs.

Procedure1.5.2 To Collect Debug Data on Web Proxy Server Installation Problems

If you are unable to complete the installation or if you get a failed status for the installation of Web Proxy Server, follow these steps.

  1. Collect the general system information as explained in To Collect Required Debug Data for Any Web Proxy Server Problem.

  2. Specify if this is a first-time installation or a Hot Fix installation on a pre-existing Web Proxy Server instance.

  3. Get the installation logs.

    Rerun the installation with the following command and save the resultant file.

    Solaris

    truss -ealf -rall -wall -vall -o /tmp/install-proxy.truss ./ns-setup

    HP-UX

    tusc -v -feaIT -rall -wall -o /tmp/install-proxy.tusc.out ./ns-setup

    Linux

    strace -fv -o /tmp/install-proxy.strace.out ./setup

    Windows

    Use Debug View: http://www.sysinternals.com/Utilities/DebugView.html

Procedure1.5.3 To Collect Debug Data on a Hung or Unresponsive Web Proxy Server Process

A process hang is defined as one of the Web Proxy Server processes not responding to requests anymore while the process is still running locally. Web Proxy Server's processes are: ns-proxy.

If Web Proxy Server has a cache, there will be two parent processes and one of these processes has one child process. The other process has all the ns-proxy processes as its child processes.

Before You Begin

Make sure that you collect all the data over the same time frame in which the problem occurs. See Configuring Solaris to Generate Core Files if a core file is not generated.

Collect the following information for process hang problems. Run the commands in order when the problem occurs. Be sure to specify the time when the process hanged and list the affected processes, if possible.

  1. Collect the general system information as explained in To Collect Required Debug Data for Any Web Proxy Server Problem.

  2. Run the netstat command and save the output.

    UNIX and Linux

    netstat -an | grep proxy-server-port

    Windows

    netstat -an

  3. Run the following commands and save the output.

    Solaris

    ps -aux | grep server-rootvmstat 5 5iostat -xtopuptime

    HP-UX

    ps -aux | grep server-rootvmstat 5 5iostat -xtopsar

    Linux

    ps -aux | grep server-rootvmstat 5 5topuptimesar

    Windows

    Obtain the Proxy process PID: C:\windbg-root>tlist.exe

    Obtain process details of the Proxy running process PID: C:\windbg-root>tlist.exe proxy-pid

    Note –

    Install the debugging tools to use the debug command. You can download the same at http://www.microsoft.com/whdc/devtools/debugging/default.mspx. Install the latest version of debugging tools and the OS symbols for the version of Windows that you are using. You must add the environment variable _NT_SYMBOL_PATH.

  4. Get the swap information.

    Solaris

    swap -l

    HP-UX

    swapinfo

    Linux

    free

    Windows

    Already provided in C:\report.txt as described in To Collect Required Debug Data for Any Web Proxy Server Problem.

  5. If the Web Proxy Server uses a Directory Server, provide the access, errors and audit logs of the Directory Server used by the Web Proxy Server.

    • Access log

      UNIX and Linux

      server-root/slapd-identifier/logs/access

      Windows

      server-root\slapd-identifier\logs\access

    • Errors log

      UNIX and Linux

      server-root/slapd-identifier/logs/errors

      Windows

      server-root\slapd-identifier\logs\errors

    • Audit log

      UNIX and Linux

      server-root/slapd-identifier/logs/audit (if enabled)

      Windows

      server-root\slapd-identifier\logs\audit (if enabled)

    Note –

    The paths of these logs files are specified by the following parameters in the dse.ldif file: nsslapd-accesslog, nsslap-errorlog, and nsslapd-auditlog

    The dse.1dif file is located in the config directory:

    UNIX and Linux: server-root/slapd-identifier/config/dse.ldif

    Windows: server-root\slapd-identifier\config\dse.ldif

  6. Provide network trace files between components, such as these:

    • Browser and Proxy Server

    • Proxy Server and Firewall

    • Proxy Server and Directory Server

    • Firewall and the Web

    Here are examples of commands on the proxy server side:

    Solaris

    snoop -V -vvv -d <interface> -o /tmp/proxy-snoop-web <IP_WEB_SERVER>

    HP-UX

    tcpdump -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

    Note –

    tcpdump for HP-UX is available at: http://hpux.connect.org.uk. You can also use the native command nettl.

    Linux

    tethereal -V -F snoop -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

    Note –

    You can use the graphical user interface for tethereal or use the command tcpdump. tethereal is available at: http://www.ethereal.com.

    Windows

    tethereal -vvv -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

    Note –

    You can use either the graphical user interface or the command for tethereal. tethereal is available at: http://www.ethereal.com.

    Note –

    Clearly indicate IP and hostname for each component. This will help to read the network trace files correctly .

  7. (Solaris OS only) If you are able to isolate the hanging process, get the following debug data for that process. Otherwise, get the following data for each of the Web Proxy Server processes.

    For Windows System

    Get the following data for the process ns-proxy.exe.

    For Solaris only

    Get a series of ten of the following commands (one every second for ten seconds):

    pstack proxy-pid and pmap -x proxy-pid

    Additionally, get the outputs of the following commands: prstat -L -p proxy-pidpfiles proxy-pid pmap proxy-pid

  8. Get the output of the following command.

    Solaris

    truss -ealf -rall -wall -vall -o /tmp/truss.out -p proxy-pid

    HP-UX

    tusc -v -fealT -rall -wall -o /tmp/tusc.out -p proxy-pid

    Linux

    strace -fv -o /tmp/strace.out -p proxy-pid

    Windows

    Use DebugView: http://www.sysinternals.com/Utilities/DebugView.html

    Note –

    Wait for one minute after launching the appropriate command (truss, strace, tusc, or DebugView), then stop it by pressing Control-C in the terminal where you launched the command.

  9. Get core files and the output of the following commands.

    In a process hang situation, it is helpful to compare several core files to review the state of the threads over time. To not overwrite a core file, copy that core file with a new name, wait approximately one minute then rerun the following commands. Do this three times to obtain three core files.

    Note –

    For HP-UX, you need the following two patches to use the gcore command: PHKL_31876 and PHCO_32173. If you cannot install these patch, use the HP-UX /opt/langtools/bin/gdb command from version 3.2 and later, or the dumpcore command.

    Solaris

    cd server-root/bin/proxygcore -o /tmp/proxy-core proxy-pidpstack /tmp/proxy-core

    HP-UX

    # cd server-root/bin/https/bin
gcore -p proxy-pid
(gdb) attach proxy-pid
Attaching to process proxy-pid
No executable file name was specified
(gdb) dumpcore
Dumping core to the core file core.proxy-pid
(gdb) quit
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: , process proxy-pid
    Note –

    The file core.<proxy-pid> is generated in the proxy-instance/config directory.

    Linux

    # cd server-root/bin/https/bin
gdb
(gdb) attach proxy-pid
Attaching to process proxy-pid
No executable file name was specified
(gdb) gcore
Saved corefile core.proxy-pid

(gdb)backtrace
(gdb)quit
    Windows

    Get the Proxy process PID:

    C:\windbg-root>tlist.exe

    Generate a crash dump on the Proxy running process PID:

    C:\windbg-root>adplus.vbs -hang -p proxy-pid -o C:\crashdump_dir

    Note –

    Provide the complete generated folder under C:\crashdump_dir.

  10. (Solaris OS only) Archive the result of the script pkg_app (one core file is sufficient). See To Run the pkg_app Script.

    ./pkg_app.ksh proxy-pid corefile

    The Sun Support Center must have the output from the pkg_app script to properly analyze the core file(s).

    Note –

    Make sure the appropriate limitations are set by using the ulimit command, and that the user is not nobody. Also check the coreadm command for additional control. See Configuring Solaris to Generate Core Files if a core file is not generated.

Procedure1.5.4 To Collect Debug Data on a Web Proxy Server Crashed Process

Use this task to collect data when a Web Proxy Server process has stopped (crashed) unexpectedly. Run all the commands on the actual machine where the core file(s) were generated.

  1. Collect the general system information as explained in To Collect Required Debug Data for Any Web Proxy Server Problem.

  2. Note whether you can restart Web Proxy Server.

  3. If the Web Server is using a Directory Server, provide the access, errors and audit logs of the Directory Server used by the Web Server

    • Access log

      UNIX - Linux

      server-root/slapd-identifier/logs/access

      Windows

      server-root\slapd-identifier\logs\access

    • Errors log

      UNIX - Linux

      server-root/slapd-identifier/logs/errors

      Windows

      server-root\slapd-identifier\logs\errors

    • Audit log

      UNIX - Linux

      server-root/slapd-identifier/logs/audit (if enabled)

      Windows

      server-root\slapd-identifier\logs\audit (if enabled)

    Note –

    The paths of these logs files are specified by the following parameters in the dse.ldif file: nsslapd-accesslog, nsslap-errorlog, and nsslapd-auditlog

    The dse.1dif file is located in the config directory:

    UNIX and Linux: server-root/slapd-identifier/config/dse.ldif

    Windows: server-root\slapd-identifier\config\dse.ldif

  4. Check if the problem is reproducible. If yes, provide a test case for reproducing the problem.

  5. Get the output of the following commands.

    Solaris

    ps -aux | grep server-rootvmstat 5 5iostat -xtopuptime

    HP-UX

    ps -aux | grep server-rootvmstat 5 5iostat -xtopsar

    Linux

    ps -aux | grep server-rootvmstat 5 5topuptimesar

    Windows

    Obtain the PROXY process PID: C:\windbg-root>tlist.exe

    Obtain process details of the PROXY running process PID: C:\windbg-root>tlist.exe proxy-pid

    Note –

    Install the debugging tools to use the debug command. You can download the same at http://www.microsoft.com/whdc/devtools/debugging/default.mspx. Install the latest version of debugging tools and the OS symbols for the version of Windows that you are using.

  6. Get the swap information.

    Solaris

    swap -l

    HP-UX

    swapinfo

    Linux

    free

    Windows

    Already provided in C:\report.txt as described in To Collect Required Debug Data for Any Web Proxy Server Problem.

  7. Get the system logs.

    Solaris and Linux

    /var/adm/messages/var/log/syslog

    HP-UX

    /var/adm/syslog/syslog.log

    Windows

    Event log files:Start> Settings>Control Panel> Event Viewer> Select LogThen click Action> Save log file as

  8. Get core files (called “Crash Dumps” by Windows).

    Solaris

    See Configuring Solaris to Generate Core Files if a core file was not generated.

    Linux

    Core dumps are turned off by default in the /etc/profile file. You can make user-specific changes by editing your ~/.bash_profile file. Look for the following line:

    ulimit -S -c 0 > /dev/null 2>&1

    You can either comment out the entire line to set no limit on the size of the core files or set your own maximum size.

    Windows

    Generate a crash dump during a crash of Web Proxy Server by using the following commands:

    Get the Proxy process PID : C:\windbg-root>tlist.exeGenerate a crash dump when the Proxy process crashes: C:\windbg-root>adplus.vbs -crash -FullOnFirst -p proxy-pid -o C:\crashdump_dir

    The adplus.vbs command watches proxy-pid until it crashes and will generate the dmp file. Provide the complete generated folder under C:\crashdump_dir.

    Note –

    If you have not installed the Debugging Tools for Windows, you can use the drwtsn32.exe -i command to select Dr. Watson as the default debugger. Use the drwtsn32.exe command, check all options, and choose the path for crash dumps. Then provide the dump and the drwtsn32.log files.

  9. (Solaris OS only) For each core file, provide the output of the following commands. See To Run the pkg_app Script 

    file corefile
pstack corefile
pmap corefile
pflags corefile

  10. (Solaris OS only) Archive the result of the script pkg_app (one core file is sufficient).

    ./pkg_app.ksh proxy-pid corefile
    Note –

    The Sun Support Center must have the output from the pkg_app script to properly analyze the core file(s).

  11. Provide network trace files between components, such as these:

    • Browser and Proxy Server

    • Proxy Server and Firewall

    • Proxy Server and Directory Server

    • Firewall and the Web

    Here are examples of commands on the proxy server side:

    Solaris OS

    snoop -V -vvv -d <interface> -o /tmp/proxy-snoop-web <IP_WEB_SERVER>

    HP-UX

    tcpdump -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

    Note –

    tcpdump for HP-UX is available at: http://hpux.connect.org.uk. You can also use the native command nettl.

    Linux

    tethereal -V -F snoop -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

    Note –

    You can use the graphical user interface for tethereal or use the command tcpdump. tethereal is available at: http://www.ethereal.com.

    Windows

    tethereal -vvv -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

    Note –

    You can use either the graphical user interface or the command for tethereal. tethereal is available at: http://www.ethereal.com.

    Note –

    Clearly indicate IP and hostname for each component. This will help to read the network trace files correctly .

    Note –

    The commands listed in this step must be executed on the machine where the core files were generated.

Procedure1.5.5 To Collect Debug Data on Web Proxy Server Authentication Problems

Use this task to collect data when Web Proxy Server is experiencing authentication problems.

A Web Proxy Server authentication problem is when Proxy Server prohibits access when it should not, or the inability of the Proxy Server to authenticate a user correctly while using the Directory Server for authentication.

  1. Collect the general system information as explained in To Collect Required Debug Data for Any Web Proxy Server Problem.

  2. Provide all the files under the following directories:

    UNIX and Linux

    server-root/proxy-identifier/conifgserver-root/userdbserver-root/httpaclserver-root/adminacl

    Windows

    server-rootproxy-identifier\conifgserver-root\userdbserver-root\httpaclserver-root\adminacl

  3. If the Web Proxy Server uses a Directory Server, provide the access, errors and audit logs of the Directory Server used by the Web Proxy Server.

    • Access log

      UNIX and Linux

      server-root/slapd-identifier/logs/access

      Windows

      server-root\slapd-identifier\logs\access

    • Errors log

      UNIX and Linux

      server-root/slapd-identifier/logs/errors

      Windows

      server-root\slapd-identifier\logs\errors

    • Audit log

      UNIX and Linux

      server-root/slapd-identifier/logs/audit (if enabled)

      Windows

      server-root\slapd-identifier\logs\audit (if enabled)

    Note –

    The paths of these logs files are specified by the following parameters in the dse.ldif file: nsslapd-accesslog, nsslap-errorlog, and nsslapd-auditlog

    The dse.1dif file is located in the config directory:

    UNIX and Linux: server-root/slapd-identifier/config/dse.ldif

    Windows: server-root\slapd-identifier\config\dse.ldif

  4. Provide network trace files between components, such as these:

    • Browser and Proxy Server

    • Proxy Server and Firewall

    • Proxy Server and Directory Server

    • Firewall and the Web

    Here are examples of commands on the proxy server side:

    Solaris

    snoop -V -vvv -d <interface> -o /tmp/proxy-snoop-web <IP_WEB_SERVER>

    HP-UX

    tcpdump -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

    Note –

    tcpdump for HP-UX is available at: http://hpux.connect.org.uk. You can also use the native command nettl.

    Linux

    tethereal -V -F snoop -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

    Note –

    You can use the graphical user interface for tethereal or use the command tcpdump. tethereal is available at: http://www.ethereal.com.

    Windows

    tethereal -vvv -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

    Note –

    You can use either the graphical user interface or the command for tethereal. tethereal is available at: http://www.ethereal.com.

    Note –

    Clearly indicate IP and hostname for each component. This will help to read the network trace files correctly .

Procedure1.5.6 To Collect Debug Data on Web Proxy Server Protocol Problems

Use this task to collect data when Web Proxy Server 's functioning is not in conformance with an RFC.

  1. Collect the general system information as explained in To Collect Required Debug Data for Any Web Proxy Server Problem.

  2. Details of the RFC and the specific section that you believe the Proxy Server is not conforming to.

  3. Check if the Proxy Server is displaying the same problem with a different browser. You can check in browsers like Internet Explorer, Mozilla, Netscape and so on.

  4. Provide network trace files between components, such as these:

    • Browser and Proxy server

    • Proxy Server and Firewall

    • Proxy Server and Directory Server

    • Firewall and the Web

    Here are examples of commands on the proxy server side:

    Solaris

    snoop -V -vvv -d <interface> -o /tmp/proxy-snoop-web <IP_PROXY_SERVER>

    HP-UX

    tcpdump -i <interface> -w /tmp/proxy-snoop-web host <IP_PROXY_SERVER>

    Note –

    tcpdump for HP-UX is available at: http://hpux.connect.org.uk. You can also use the native command nettl.

    Linux

    tethereal -V -F snoop -i <interface> -w /tmp/proxy-snoop-web host <IP_PROXY_SERVER>

    Note –

    You can use the graphical user interface for tethereal or use the command tcpdump. tethereal is available at: http://www.ethereal.com.

    Windows

    tethereal -vvv -i <interface> -w /tmp/proxy-snoop-web host <IP_PROXY_SERVER>

    Note –

    You can use either the graphical user interface or the command for tethereal. tethereal is available at: http://www.ethereal.com.

    Note –

    Clearly indicate IP and hostname for each component. This will help to read the network trace files correctly .