1.5.3 To Collect Debug Data on a Hung or Unresponsive Web Proxy Server Process (Sun Gathering Debug Data for Sun Java System Web Proxy Server)

Sun Gathering Debug Data for Sun Java System Web Proxy Server

1.5.3 To Collect Debug Data on a Hung or Unresponsive Web Proxy Server Process

A process hang is defined as one of the Web Proxy Server processes not responding to requests anymore while the process is still running locally. Web Proxy Server's processes are: ns-proxy.

If Web Proxy Server has a cache, there will be two parent processes and one of these processes has one child process. The other process has all the ns-proxy processes as its child processes.

Before You Begin

Make sure that you collect all the data over the same time frame in which the problem occurs. See Configuring Solaris to Generate Core Files if a core file is not generated.

Collect the following information for process hang problems. Run the commands in order when the problem occurs. Be sure to specify the time when the process hanged and list the affected processes, if possible.

Collect the general system information as explained in To Collect Required Debug Data for Any Web Proxy Server Problem.

Run the netstat command and save the output.

UNIX and Linux

netstat -an | grep proxy-server-port

Windows

netstat -an

Run the following commands and save the output.

Solaris

ps -aux | grep server-rootvmstat 5 5iostat -xtopuptime

HP-UX

ps -aux | grep server-rootvmstat 5 5iostat -xtopsar

Linux

ps -aux | grep server-rootvmstat 5 5topuptimesar

Windows

Obtain the Proxy process PID: C:\windbg-root>tlist.exe

Obtain process details of the Proxy running process PID: C:\windbg-root>tlist.exe proxy-pid

Note –
Install the debugging tools to use the debug command. You can download the same at http://www.microsoft.com/whdc/devtools/debugging/default.mspx. Install the latest version of debugging tools and the OS symbols for the version of Windows that you are using. You must add the environment variable _NT_SYMBOL_PATH.

Get the swap information.

Solaris

swap -l

HP-UX

swapinfo

Linux

free

Windows

Already provided in C:\report.txt as described in To Collect Required Debug Data for Any Web Proxy Server Problem.

If the Web Proxy Server uses a Directory Server, provide the access, errors and audit logs of the Directory Server used by the Web Proxy Server.
- Access log
  
  UNIX and Linux
  
  server-root/slapd-identifier/logs/access
  
  Windows
  
  server-root\slapd-identifier\logs\access
- Errors log
  
  UNIX and Linux
  
  server-root/slapd-identifier/logs/errors
  
  Windows
  
  server-root\slapd-identifier\logs\errors
- Audit log
  
  UNIX and Linux
  
  server-root/slapd-identifier/logs/audit (if enabled)
  
  Windows
  
  server-root\slapd-identifier\logs\audit (if enabled)
Note –
The paths of these logs files are specified by the following parameters in the dse.ldif file: nsslapd-accesslog, nsslap-errorlog, and nsslapd-auditlog

The dse.1dif file is located in the config directory:

UNIX and Linux: server-root/slapd-identifier/config/dse.ldif

Windows: server-root\slapd-identifier\config\dse.ldif

Provide network trace files between components, such as these:
- Browser and Proxy Server
- Proxy Server and Firewall
- Proxy Server and Directory Server
- Firewall and the Web
Here are examples of commands on the proxy server side:

Solaris

snoop -V -vvv -d <interface> -o /tmp/proxy-snoop-web <IP_WEB_SERVER>

HP-UX

tcpdump -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

Note –
tcpdump for HP-UX is available at: http://hpux.connect.org.uk. You can also use the native command nettl.

Linux

tethereal -V -F snoop -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

Note –
You can use the graphical user interface for tethereal or use the command tcpdump. tethereal is available at: http://www.ethereal.com.

Windows

tethereal -vvv -i <interface> -w /tmp/proxy-snoop-web host <IP_WEB_SERVER>

Note –
You can use either the graphical user interface or the command for tethereal. tethereal is available at: http://www.ethereal.com.

Note –
Clearly indicate IP and hostname for each component. This will help to read the network trace files correctly .

(Solaris OS only) If you are able to isolate the hanging process, get the following debug data for that process. Otherwise, get the following data for each of the Web Proxy Server processes.

For Windows System

Get the following data for the process ns-proxy.exe.

For Solaris only

Get a series of ten of the following commands (one every second for ten seconds):

pstack proxy-pid and pmap -x proxy-pid

Additionally, get the outputs of the following commands: prstat -L -p proxy-pidpfiles proxy-pid pmap proxy-pid

Get the output of the following command.

Solaris

truss -ealf -rall -wall -vall -o /tmp/truss.out -p proxy-pid

HP-UX

tusc -v -fealT -rall -wall -o /tmp/tusc.out -p proxy-pid

Linux

strace -fv -o /tmp/strace.out -p proxy-pid

Windows

Use DebugView: http://www.sysinternals.com/Utilities/DebugView.html

Note –
Wait for one minute after launching the appropriate command (truss, strace, tusc, or DebugView), then stop it by pressing Control-C in the terminal where you launched the command.

Get core files and the output of the following commands.

In a process hang situation, it is helpful to compare several core files to review the state of the threads over time. To not overwrite a core file, copy that core file with a new name, wait approximately one minute then rerun the following commands. Do this three times to obtain three core files.

Note –

For HP-UX, you need the following two patches to use the gcore command: PHKL_31876 and PHCO_32173. If you cannot install these patch, use the HP-UX /opt/langtools/bin/gdb command from version 3.2 and later, or the dumpcore command.

Solaris

cd server-root/bin/proxygcore -o /tmp/proxy-core proxy-pidpstack /tmp/proxy-core

HP-UX

# cd server-root/bin/https/bin
gcore -p proxy-pid
(gdb) attach proxy-pid
Attaching to process proxy-pid
No executable file name was specified
(gdb) dumpcore
Dumping core to the core file core.proxy-pid
(gdb) quit
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: , process proxy-pid

Note –

The file core.<proxy-pid> is generated in the proxy-instance/config directory.

Linux

# cd server-root/bin/https/bin
gdb
(gdb) attach proxy-pid
Attaching to process proxy-pid
No executable file name was specified
(gdb) gcore
Saved corefile core.proxy-pid

(gdb)backtrace
(gdb)quit

Windows

Get the Proxy process PID:

C:\windbg-root>tlist.exe

Generate a crash dump on the Proxy running process PID:

C:\windbg-root>adplus.vbs -hang -p proxy-pid -o C:\crashdump_dir

Note –

Provide the complete generated folder under C:\crashdump_dir.

(Solaris OS only) Archive the result of the script pkg_app (one core file is sufficient). See To Run the pkg_app Script.
```
./pkg_app.ksh proxy-pid corefile
```
The Sun Support Center must have the output from the pkg_app script to properly analyze the core file(s).

Note –
Make sure the appropriate limitations are set by using the ulimit command, and that the user is not nobody. Also check the coreadm command for additional control. See Configuring Solaris to Generate Core Files if a core file is not generated.