test

Sun Gathering Debug Data for Sun Java System Directory Proxy Server 5

Chapter 1 Sun Gathering Debug Data for Sun Java System Directory Proxy Server

This technical note describes how to use SunTM Gathering Debug Data (Sun GDD or GDD) to collect data that the Sun Support Center requires in order to debug problems with Sun JavaTM System Directory Proxy Server software.

By collecting this data before you open a Service Request, you can reduce substantially the time needed to analyze and resolve the problem. For more information on how this document and associated scripts can help you in better dealing with Directory Proxy Server problems, see:

http://www.sun.com/services/gdd/index.html

This document is intended for anyone who needs to open a Service Request about Directory Proxy Server software with the Sun Support Center.

1.1 Technical Note Revision History

Version 

Date 

Description of Changes 

1.0 

December 2006 

Initial release of this technical note. 

1.1 

January 2007 

Updated To Configure Solaris OS to Generate Core Files.

1.2 About This Technical Note

This document covers Sun Java System Directory Proxy Server 5 on all supported platforms.

You can use this document in all types of environments, including test, pre-production, and production. Verbose debugging is not used so as to avoid performance impact, except when it is deemed necessary. In some cases, it is possible that the problem could disappear when you configure logging for debug mode. However, this is the minimum to understand the problem. In the majority of cases, the debug data described in this document is sufficient to analyze the problem.

This document does not provide workarounds nor techniques or tools to analyze debug data. It provides some troubleshooting, but you should not use this guide as an approach to troubleshooting Directory Proxy Server problems.

If your problem does not conveniently fit into any of the specific categories, supply the general information described in To Collect Required Debug Data For Any Directory Proxy Server Problem and clearly explain your problem.

If the information you initially provide is not sufficient to find the root cause of the problem, Sun will ask for more details, as needed.

1.2.1 Prerequisites for Collecting Directory Proxy Server Debug Data

The prerequisites for collecting debug data for Directory Proxy Server are as follows.

1.2.2 Variables Used in This Technical Note

The following describes the variables used in the procedures in this document. Gather the values of the variables if you don't already know them before you try to do the procedures.

server-root

The base file system location for Directory Proxy Server or Directory Server.

win-dbg-root

The base file system location for Windows Debugging Tools.

Many paths specified in this document use the forward slash format of UNIX. If you are running software on a Windows system, use the equivalent backslash format.

1.3 Overview of Collecting Debug Data For Directory Proxy Server

    Collecting debug data for any Directory Proxy Server problem involves these basic operations:

  1. Collecting basic problem and system information.

  2. Collecting specific problem information.

  3. Creating a tar.gz file of all the information and uploading the file to the Sun Support Center.

  4. Creating a Service Request with the Sun Support Center.

1.4 Creating a Service Request with the Sun Support Center

When you create a Service Request with the Sun Support Center, either online or by phone, provide the following information:

Upload your debug data archive file to one of the following locations:

Note –

When opening a Service Request by phone with the Sun Support Center, provide a summary of the problem. Also provide the details in a text file named Description.txt. Be sure to include Description.txt in the archive along with the rest of your debug data.

For more information on how to upload files, see: http://supportfiles.sun.com/show?target=faq

1.5 What Directory Proxy Server Debug Data Should You Collect?

This section describes the kinds of debug data you need to provide based on the problem you are experiencing.

This section contains the following tasks:

ProcedureTo Collect Required Debug Data For Any Directory Proxy Server Problem

All problems described in this technical note need basic information collected about when the problem occurred and about the system having the problem. Use this task to collect that basic information.

  1. Note the time or times the problem occurred.

  2. If possible, collect explorer output from SUNWexplo software on the system where the problem occurred.

  3. Provide graphical representation of your deployment.

    The graphical representation of your deployment is key to understanding the context of the problem. Show the following in the graphical representation.

    • All computers involved, with their IP addresses, hostnames, roles in the deployment, operating systems, and versions used.

    • All other relevant systems, including load balancers, firewalls, and so forth.

  4. Note the operating system version.

    Solaris OS

    uname -a

    HP-UX

    uname -r

    Red Hat

    cat /etc/redhat-release

    Windows

    C:\Program Files\Common files\Microsoft Shared\MSInfo\msinfo32.exe /report C:\report.txt

  5. Note the patch level.

    Solaris OS

    showrev -p

    HP-UX

    swlist

    Red Hat

    rpm -qa

    Windows

    Already provided in C:\report.txt.

  6. Note the version of Directory Proxy Server.

  7. To determine the version information, use the ldapfwd -v command.

    server-root/bin/dps/server/bin/ldapfwd -v

    On UNIX and Linux systems, you might see the following error.

    ld.so.1: ldapfwd: fatal: libnss3.so: open failed: No such file or directory

    If you see the error, set LD_LIBRARY_PATH to include Directory Proxy Server libraries in your load path. For example, if you use sh, use the following command.

    export LD_LIBRARY_PATH=server-root/lib

  8. Collect the Directory Proxy Server configuration file generated with the dpsconfig2ldif command.

    UNIX and Linux

    server-root/bin/dps_utilities/dpsconfig2ldif -t server-root/dps-serverID/etc/tailor.txt -o /tmp/config.ldif

    Windows

    server-root\bin\dps_utilities\dpsconfig2ldif -t server-root\dps-serverID\etc\tailor.txt -o %TEMP%\config.ldif

  9. Collect Directory Proxy Server trace logging information.

    1. Access the Directory Proxy Server Console.

    2. Select the Configuration tab.

    3. Select the Logs node.

    4. In the Log detail drop-down list, select Detailed trace.

    5. Save your work.

    6. Restart Directory Proxy Server.

    7. After reproducing the problem, collect the server-root/dps-serverID/logs/fwd.log file.

  10. Collect Directory Server version information with the nsslapd -V command for directories accessed through the proxy.

    server-root/bin/slapd/server/ns-slapd -D instance-dir -V

  11. Collect Directory Server access, errors, and audit logs for directories accessed through the proxy.

    When you find more than one log file of each type, collect each of the log files. If possible, provide logs that show server operation before the problem occurred, and also logs written as the problem occurred.

    By default, you find these logs in the following locations.

    server-root/slapd-serverID/logs/access

    server-root/slapd-serverID/logs/errors

    server-root/slapd-serverID/logs/audit (if enabled)

    If these log files are not in the default locations, examine the Directory Server configuration file, server-root/slapd-serverID/config/dse.ldif, to find the paths to the logs. The paths are specified as the values of attributes nsslapd-accesslog, nsslapd-errorlog, and nsslapd-auditlog.

ProcedureTo Collect Required Debug Data For Directory Proxy Server Installation Problems

This procedure describes what data to collect when you cannot complete Directory Proxy Server installation.

Before You Begin

If the problem concerns a general installation failure for Java Enterprise System, first check the installation troubleshooting chapter in the Installation Guide for your version of Java Enterprise System.

  1. For compressed archive installations, collect installation output showing system calls.

    Reinstall while using the appropriate command for your system from the following list. Collect the output of the command that is displayed during installation.

    Solaris OS

    truss -ealf -rall -wall -vall -o /tmp/install-dps-truss.out ./setup

    HP-UX

    tusc -v -fealT -rall -wall -o /tmp/install-dps-tusc.out ./setup

    Red Hat

    strace -fv -o /tmp/install-dps-strace.out ./setup

    Windows

    Use DebugView.

    DebugView is available at http://www.sysinternals.com/Utilities/DebugView.html.

  2. For Java Enterprise System installations, collect installation error logs.

    The log file is named after the date and time that the installation failed. For example, a log file for an installation that failed on December 16 at 3:32 p.m. would have a name like Java_Enterprise_System*_install.B12161532.

    On Solaris systems, installation logs are located under /var/sadm/install/logs.

    On Red Hat and HP-UX systems, installation logs are located under /var/opt/sun/install/logs.

    On Windows systems, installation logs are located under C:\Documents and Settings\current-user\Local Settings\Temp.

ProcedureTo Collect Required Debug Data For an Unresponsive or Hung Directory Proxy Server Process

This procedure describes what data to collect when Directory Proxy Server is still running, but is no longer responding to client application requests.

Collect the data describe in this procedure while the server is hanging.

  1. Note the time during which the hang is seen to occur.

  2. Collect information about the port used during the hang.

    UNIX and Linux

    netstat -an | grep dps-port

    On Solaris OS, also collect the output of pfiles pid, where pid is the process ID of the unresponsive server process.

    Windows

    netstat -an

  3. Collect statistics about the system running Directory Proxy Server.

    Solaris OS

    ps -aux | grep server-root

    vmstat 5 5

    iostat -x

    top

    uptime

    HP-UX

    ps -aux | grep server-root

    vmstat 5 5

    iostat -x

    top

    sar

    Red Hat

    ps -aux | grep server-root

    vmstat 5 5

    top

    uptime

    sar

    Windows

    Get the process ID using the tlist.exe command, then get process details using the same command.

    win-dbg-root\tlist.exe pid

  4. Collect swap information.

    Solaris OS

    swap -l

    HP-UX

    swapinfo

    Red Hat

    free

    Windows

    Already provided in C:\report.txt.

  5. On Solaris systems, collect output from pstack and pmap five times, once every ten seconds.

    pstack pid

    pmap -x pid

  6. Get output showing system calls during the hang, by letting each of the commands listed here run for about a minute, then stopping them by typing Control-C.

    Solaris OS

    truss -ealf -rall -wall -vall -o /tmp/truss.out -p pid

    HP-UX

    tusc -v -fealT -rall -wall -o /tmp/truss.out -p pid

    Red Hat

    strace -fv -o /tmp/strace.out -p pid

    Windows

    Use Debug View.

    DebugView is available at http://www.sysinternals.com/Utilities/DebugView.html.

  7. Collect core files or crash dumps, and related command output.

    When the server is hanging, attempt to get several core files that show the state of the server threads over time. You can do this by generating a core file, changing the name of the core file, waiting 30 seconds to a minute, and generating another core file. Repeat the process at least once to get a minimum of three sets of core files and related data.

    Solaris OS

    cd server-root/bin/dps/server/bin

    gcore -o /tmp/dps-core pid

    pstack /tmp/dps-core

    For each core file on Solaris OS, collect output from the following commands.

    cd server-root/bin/dps/server/bin

    file core-file

    pstack core-file

    pmap core-file

    pflags core-file

    For at least one of the core files on Solaris OS, collect output from the pkg_app script.

    ./pkg_app.ksh pid core-file

    Here, pid is the server process ID number. See To Run the pkg_app Script for instructions on using pkg_app.

    HP-UX

    cd server-root/bin/dps/server/bin

    If you have the patches PHKL_31876 and PHCO_32173 patches installed, generate the core file using the gcore command.

    gcore -p pid

    Otherwise, use the gdb command to generate the core file.

    gdb

    (gdb) attach pid

    Attaching to process pid

    No executable file name was specified

    (gdb) dumpcore

    Dumping core to the core file core.pid

    (gdb) quit

    The program is running. Quit anyway (and detach it)? (y or n) y

    Detaching from program: , process pid

    Red Hat

    cd server-root/bin/dps/server/bin

    gdb

    (gdb) attach pid

    Attaching to process pid

    No executable file was specified

    (gdb) gcore

    Saved corefile core.pid

    (gdb) backtrace

    (gdb) quit

    Windows

    win-dbg-root\tlist.exe

    win-dbg-root\adplus.vbs -hang -p pid -o C:\dump-dir

    Collect everything in the folder under C:\dump-dir.

ProcedureTo Collect Required Debug Data For a Crashed Directory Proxy Server Process

This procedure describes what data to collect when a Directory Proxy Server process unexpectedly dies.

  1. Try to restart Directory Proxy Server and record the results.

  2. Include the test case to reproduce the problem.

  3. Collect statistics about the system running Directory Proxy Server.

    Solaris OS

    ps -aux | grep server-root

    vmstat 5 5

    iostat -x

    top

    uptime

    HP-UX

    ps -aux | grep server-root

    vmstat 5 5

    iostat -x

    top

    sar

    Red Hat

    ps -aux | grep server-root

    vmstat 5 5

    top

    uptime

    sar

    Windows

    Get the process ID using the tlist.exe command, then get process details using the same command.

    win-dbg-root\tlist.exe pid

  4. Collect swap information.

    Solaris OS

    swap -l

    HP-UX

    swapinfo

    Red Hat

    free

    Windows

    Already provided in C:\report.txt.

  5. Collect system logs.

    Solaris OS

    /var/adm/messages

    /var/log/syslog

    HP-UX

    /var/adm/syslog/syslog.log

    Red Hat

    /var/log/messages

    /var/log/syslog

    Windows

    Retrieve the Event log files.

    From the Control Panel, open the Event Viewer. Select the event log. Then select Action > Save log file as to save the log.

  6. Collect core files.

    For instructions on preparing your system to produce core files or crash dumps in the event of a crash, see 1.6 Configuring the Operating System to Generate Core Files.

    1. For each core file on Solaris OS, collect output from the following commands.

      cd server-root/bin/dps/server/bin

      file core-file

      pstack core-file

      pmap core-file

      pflags core-file

    2. For at least one of the core files on Solaris OS, collect output from the pkg_app script.

      ./pkg_app.ksh pid core-file

      Here, pid is the server process ID number. See To Run the pkg_app Script for instructions on using pkg_app.

ProcedureTo Collect Required Debug Data For Directory Proxy Server Console Problems

This section describes what data to collect when part of the Directory Proxy Server Console fails to comply with what you expect.

  1. Collect screen shots of the affected screen or screens.

    The screen shots should show the problem you are experiencing.

  2. Provide step by step instructions for reproducing the problem.

    If needed, also provide test case data.

  3. Collect debug traces for Directory Proxy Server Console when you reproduce the problem, saved in a console-debug.txt file.

    startconsole -D 9 > console-debug.txt

  4. Collect Administration Server access and error logs.

    The access log is server-root/admin-serv/logs/access.

    The error log is server-root/admin-serv/logs/error.

1.6 Configuring the Operating System to Generate Core Files

Core files and crash dumps are generated when a process or application terminates abnormally. You can also generate core files and crash dumps to help diagnose why a process does not respond to client application requests.

This section includes the following procedures:

ProcedureTo Configure Solaris OS to Generate Core Files

This procedure shows you how to use the coreadm command to configure the system so that all process core files are placed in a single system directory. This means it is easier to track problems by examining core files in a specific directory whenever a Solaris OS process or daemon terminates abnormally.

Before You Begin

Make sure that the /var file system where the core files are generated has sufficient space. Once you configure Solaris OS to generate core files as shown here, all processes that crash write a core file to the /var/cores directory.

  1. Run the following commands as root.

    mkdir -p /var/cores
coreadm -g /var/cores/%f.%n.%p.%t.core -e global -e global-setid -e log -d process -d proc-setid

    In this command:

    -g

    Specifies the global core file name pattern. Unless a per-process pattern or setting overrides it, core files are stored in the specified directory with a name such as program.node.pid.time.core, for example: mytest.myhost.1234.1102010309.core.

    -e

    Specifies options to enable. The preceding command enables:

    • Use of the global (that is, system-wide) core file name pattern (and thereby location)

    • Capability of setuid programs to also dump core as per the same pattern

    • Generation of a syslog message by any attempt to dump core (successful or not)

    -d

    Specifies options to disable. The preceding command disables:

    • Core dumps per the per-process core file pattern

    • Per-process core dumps of setuid programs

    The preceding command stores all core dumps in a central location with names identifying what process dumped core and when. These changes only impact processes started after you run the coreadm command. Use the coreadm -u command after the preceding command to apply the settings to all existing processes.

  2. Display the core configuration.


    # coreadmglobal core file pattern: /var/cores/%f.%n.%p.%t.core
      init core file pattern: core
           global core dumps: enabled
      per-process core dumps: disabled
     global setid core dumps: enabled
per-process setid core dumps: disabled
    global core dump logging: enabled

    See the coreadm man page for further information.

  3. Set the file size as large as possible, using the ulimit command.


    # ulimit -c unlimited
# ulimit -a
       coredump(blocks) unlimited

    See the ulimit(1) man page for details.

  4. Verify that applications can in fact generate core files.


    # cd /var/cores
# sleep 100000 &
[1] pid
# kill -8 pid
# ls

    You should see a core file for the sleep process you killed.

ProcedureTo Configure Red Hat Linux to Generate Core Files

On Red Hat systems, you can enable core files to be generated on a per user basis.

  1. Open the ~/.bash_profile file for the server user in a text editor.

  2. Search for a line using the ulimit command as follows.

    ulimit -S -c 0 > /dev/null 2>&1

  3. Either comment out the line, or set your own limit for core file size.

ProcedureTo Configure Windows to Generate Crash Dumps

The native debug tool on Windows systems, Dr. Watson, allows you to generate crash dumps.

However, Dr. Watson does not allow generation of crash dumps on a running process. To generate crash dumps from a running process, install the Debugging Tools. The Debugging Tools are freely available from the Windows web site at http://www.microsoft.com/whdc/devtools/debugging/default.mspx.

  1. You can use Dr. Watson for crash dumps generated when a process dies.

    1. Use the drwtsn32 -i command to make Dr. Watson the default debugger.

    2. Open Dr. Watson with the drwtsn32 -i command.

    3. Check all options.

    4. Choose the path where crash dumps are generated.

      When providing crash dumps, collect both the dmp and drwtsn32.log files.

  2. Use the Window Debugging Tools to generate crash dumps of a running process.

    1. Make sure you install the latest version of the Debugging Tools and OS Symbols for your version of Windows.

    2. Set the _NT_SYMBOL_PATH for your environment.

  3. Enable generation of a crash dump for your application.

    Get the process ID of the application using the tlist.exe command, then enable the crash dump.

    win-dbg-root\tlist.exe

    win-dbg-root\adplus.vbs -crash -FullOnFirst -p pid -o C:\dump-dir

    The adplus.vbs command tracks the application with process ID pid. The adplus.vbs command generates a dmp file in the event of a crash.

  4. When collecting crash dump information, take the complete folder generated under C:\dump-dir.

ProcedureTo Run the pkg_app Script

The pkg_app script packages an executable and all of its shared libraries into one compressed tar file given the process ID of the application, and optionally the name of the core file to be opened. The files are stripped of their directory paths, and are stored under a relative directory named app/ with their names only, allowing them to be unpacked in one directory.

On Solaris 9 and Solaris 10, the list of files is derived from the core file rather than the process image if it is specified. You must still provide the process ID of the running application to assist in path resolution.

Two scripts are created to facilitate opening the core file when the tar file is unpacked.

  1. Copy the script to a temporary directory on the system where the server is installed.

  2. Become superuser.

  3. Run the pkg_app script in one of the following ways.

    • ./pkg_app server-pid core-file

    • ./pkg_app server-pid

      The pkg_app script prompts for the core-file name.

    • ./pkg_app core-file

1.7 Reporting Problems

Use the following email aliases to report problems with this document and its associated scripts:

gdd-feedback@sun.com

For feedback on this document

gdd-issue-tracker@sun.com

To report problems in gathering debug data

1.8 Documentation, Support, and Training

The Sun web site provides information about the following additional resources:

1.9 Third-Party Web Site References

Third-party URLs are referenced in this document and provide additional, related information.

Note –

Sun is not responsible for the availability of third-party web sites mentioned in this document. Sun does not endorse and is not responsible or liable for any content, advertising, products, or other materials that are available on or through such sites or resources. Sun will not be responsible or liable for any actual or alleged damage or loss caused or alleged to be caused by or in connection with use of or reliance on any such content, goods, or services that are available on or through such sites or resources.

1.10 Searching Sun Product Documentation

Besides searching for Sun product documentation from the docs.sun.com web site, you can use a search engine of your choice by typing the following syntax in the search field:

search-term site:docs.sun.com

For example, to search for Directory Server, type the following:

"Directory Server" site:docs.sun.com

To include other Sun web sites in your search, such as java.sun.com, www.sun.com, and developers.sun.com, use sun.com in place of docs.sun.com in the search field.

1.11 Sun Welcomes Your Comments

Sun is interested in improving its documentation and welcomes your comments and suggestions. To share your comments, go to http://docs.sun.com and click Send Comments. In the online form, provide the full document title and part number. The part number is a 7-digit or 9-digit number that can be found on the book's title page or in the document's URL. For example, the part number of this book is 820-0436-10.