Sun Gathering Debug Data for Sun Java System Directory Server 5

1.5 What Directory Server Debug Data Should You Collect?

This section describes the kinds of debug data you need to provide based on the problem you are experiencing.

This section contains the following tasks:

ProcedureTo Collect Required Debug Data For Any Directory Server Problem

All problems described in this technical note need basic information collected about when the problem occurred and about the system having the problem. Use this task to collect that basic information.

  1. Note the time or times the problem occurred.

  2. If possible, collect explorer output from SUNWexplo software on the system where the problem occurred.

  3. Provide graphical representation of your deployment.

    The graphical representation of your deployment is key to understanding the context of the problem. Show the following in the graphical representation.

    • All computers involved, with their IP addresses, hostnames, roles in the deployment, operating systems, and versions used.

    • All other relevant systems, including load balancers, firewalls, and so forth.

  4. If possible, collect a copy of the database on spare disk space or backup media to be used later if necessary.

    Database files and the transaction log are often indispensable for diagnosing deadlocks involving database locks.

  5. Note the operating system version.

    Solaris OS

    uname -a

    HP-UX

    uname -r

    Red Hat

    cat /etc/redhat-release

    Windows

    C:\Program Files\Common files\Microsoft Shared\MSInfo\msinfo32.exe /report C:\report.txt

  6. Note the patch level.

    Solaris OS

    showrev -p

    HP-UX

    swlist

    Red Hat

    rpm -qa

    Windows

    Already provided in C:\report.txt.

  7. Collect Directory Server version information.

    server-root/bin/slapd/server/ns-slapd -D instance-dir -V

    The server executable is slapd.exe on Windows systems.

  8. Collect the Directory Server configuration file, server-root/slapd-serverID/config/dse.ldif.

  9. Collect Directory Server access, errors, and audit logs.

    By default, you find these logs in the following locations:

    server-root/slapd-serverID/logs/access

    server-root/slapd-serverID/logs/errors

    server-root/slapd-serverID/logs/audit (if enabled)

    If these log files are not in the default locations, examine the Directory Server configuration file, server-root/slapd-/serverID/config/dse.ldif, to find the paths to the logs. The paths are specified as the values of attributes nsslapd-accesslog, nsslapd-errorlog, and nsslapd-auditlog.

ProcedureTo Collect Required Debug Data For Directory Server Installation Problems

This section describes what data to collect when you cannot complete Directory Server installation.

Before You Begin

If the problem concerns a general installation failure for Java Enterprise System, first check the installation troubleshooting chapter in the Installation Guide for your version of Java Enterprise System.

  1. For compressed archive installations, collect installation output showing system calls.

    Reinstall while using the appropriate command for your system from the following list. Collect the output of the command that is displayed during installation.

    Solaris OS

    truss -ealf -rall -wall -vall -o /tmp/install-directory-truss.out ./setup

    HP-UX

    tusc -v -fealT -rall -wall -o /tmp/install-directory-tusc.out ./setup

    Red Hat

    strace -fv -o /tmp/install-directory-strace.out ./setup

    Windows

    Use DebugView.

    DebugView is available at http://www.sysinternals.com/Utilities/DebugView.html.

  2. For Java Enterprise System installations, collect installation error logs.

    The log file is named after the date and time that the installation failed. For example, a log file for an installation that failed on December 16 at 3:32 p.m. would have a name like Java_Enterprise_System*_install.B12161532.

    On Solaris systems, installation logs are located under /var/sadm/install/logs.

    On Red Hat and HP-UX systems, installation logs are located under /var/opt/sun/install/logs.

    On Windows systems, installation logs are located under C:\Documents and Settings\current-user\Local Settings\Temp.

ProcedureTo Collect Required Debug Data For an Unresponsive or Hung Directory Server Process

This procedure describes what data to collect when Directory Server is still running, but is no longer responding to client application requests.

Collect the data describe in this procedure while the server is hanging.

  1. Note the time during which the hang is seen to occur.

  2. Collect information about the port used during the hang.

    UNIX and Linux

    netstat -an | grep ds-port

    Windows

    netstat -an

  3. Collect statistics about the system running Directory Server.

    Solaris OS

    ps -aux | grep server-root

    vmstat 5 5

    iostat -x

    top

    uptime

    HP-UX

    ps -aux | grep server-root

    vmstat 5 5

    iostat -x

    top

    sar

    Red Hat

    ps -aux | grep server-root

    vmstat 5 5

    top

    uptime

    sar

    Windows

    Get the process ID using the tlist.exe command, then get process details using the same command.

    win-dbg-root\tlist.exe pid

  4. Collect swap information.

    Solaris OS

    swap -l

    HP-UX

    swapinfo

    Red Hat

    free

    Windows

    Already provided in C:\report.txt.

  5. On Solaris systems, collect output from pstack and pmap five times, once every ten seconds.

    pstack pid

    pmap -x pid

  6. Get output showing system calls during the hang, by letting each of the commands listed here run for about a minute, then stopping them by typing Control-C.

    Solaris OS

    truss -ealf -rall -wall -vall -o /tmp/truss.out -p pid

    HP-UX

    tusc -v -fealT -rall -wall -o /tmp/truss.out -p pid

    Red Hat

    strace -fv -o /tmp/strace.out -p pid

    Windows

    Use Debug View.

    DebugView is available at http://www.sysinternals.com/Utilities/DebugView.html.

  7. Collect core files or crash dumps, and related command output.

    When the server is hanging, attempt to get several core files that show the state of the server threads over time. You can do this by generating a core file, changing the name of the core file, waiting 30 seconds to a minute, and generating another core file. Repeat the process at least once to get a minimum of three sets of core files and related data.

    Solaris OS

    cd server-root/bin/slapd/server

    gcore -o /tmp/directory-core pid

    pstack /tmp/directory-core

    For each core file on Solaris OS, collect output from the following commands.

    cd server-root/bin/slapd/server

    file core-file

    pstack core-file

    pmap core-file

    pflags core-file

    For at least one of the core files on Solaris OS, collect output from the pkg_app script.

    ./pkg_app.ksh pid /tmp/directory-core

    Here, pid is the server process ID number. See To Run the pkg_app Script for instructions on using pkg_app.

    HP-UX

    cd server-root/bin/slapd/server

    If you have the patches PHKL_31876 and PHCO_32173 patches installed, generate the core file using the gcore command.

    gcore -p pid

    Otherwise, use the gdb command to generate the core file.

    gdb

    (gdb) attach pid

    Attaching to process pid

    No executable file name was specified

    (gdb) dumpcore

    Dumping core to the core file core.pid

    (gdb) quit

    The program is running. Quit anyway (and detach it)? (y or n) y

    Detaching from program: , process pid

    Red Hat

    cd server-root/bin/slapd/server

    gdb

    (gdb) attach pid

    Attaching to process pid

    No executable file was specified

    (gdb) gcore

    Saved corefile core.pid

    (gdb) backtrace

    (gdb) quit

    Windows

    win-dbg-root\tlist.exe

    win-dbg-root\adplus.vbs -hang -p pid -o C:\dump-dir

    Collect everything in the folder under C:\dump-dir.

  8. Collect output from idsktune.

    The idsktune command provides information on system parameters, patch level, tuning recommendations, and so forth. The command is described in the product documentation.

    server-root/bin/slapd/server/idsktune

ProcedureTo Collect Required Debug Data For a Crashed Directory Server Process

This procedure describes what data to collect when a Directory Server process unexpectedly dies.

  1. Try to restart Directory Server and record the results.

  2. Collect statistics about the system running Directory Server.

    Solaris OS

    ps -aux | grep server-root

    vmstat 5 5

    iostat -x

    top

    uptime

    HP-UX

    ps -aux | grep server-root

    vmstat 5 5

    iostat -x

    top

    sar

    Red Hat

    ps -aux | grep server-root

    vmstat 5 5

    top

    uptime

    sar

    Windows

    Get the process ID using the tlist.exe command, then get process details using the same command.

    win-dbg-root\tlist.exe pid

  3. Collect swap information.

    Solaris OS

    swap -l

    HP-UX

    swapinfo

    Red Hat

    free

    Windows

    Already provided in C:\report.txt.

  4. Collect system logs.

    Solaris OS

    /var/adm/messages

    /var/log/syslog

    HP-UX

    /var/adm/syslog/syslog.log

    Red Hat

    /var/log/messages

    /var/log/syslog

    Windows

    Retrieve the Event log files.

    From the Control Panel, open the Event Viewer. Select the event log. Then select Action > Save log file as to save the log.

  5. Collect core files.

    For instructions on preparing your system to produce core files or crash dumps in the event of a crash, see 1.6 Configuring the Operating System to Generate Core Files.

    1. For each core file on Solaris OS, collect output from the following commands.

      cd server-root/bin/slapd/server

      file core-file

      pstack core-file

      pmap core-file

      pflags core-file

    2. For at least one of the core files on Solaris OS, collect output from the pkg_app script.

      ./pkg_app.ksh pid core-file

      Here, pid is the server process ID number. See To Run the pkg_app Script for instructions on using pkg_app.

ProcedureTo Collect Required Debug Data For Directory Server Replication Problems

This procedure describes what data to collect when experiencing inconsistencies and other issues involving Directory Server replication.

Directory Server commands used here are described in the product documentation.

  1. Collect the hostname, IP address, and serverID for each server instance in the replication topology.

  2. Collect a copy of the schema folder, server-root/slapd-serverID/config/schema, and all the files in the folder.

  3. Collect access logs configured to show replication information.

    Before you collect access logs as described in To Collect Required Debug Data For Any Directory Server Problem, adjust the log level to keep replication information.

    You can use the Console to adjust the access log level.

    Alternatively, you can use the ldapmodify command as follows.


    $ ldapmodify -h host -p port -D "cn=Directory Manager" -w password
    dn: cn=config
    changetype: modify
    replace: nsslapd-infolog-area  # nsslapd-errorlog-level in 5.1
    nsslapd-infolog-area: 8192

    To return to the default log level, use the following command.


    $ ldapmodify -h host -p port -D "cn=Directory Manager" -w password
    dn: cn=config
    changetype: modify
    replace: nsslapd-infolog-area  # nsslapd-errorlog-level in 5.1
    nsslapd-infolog-area: 0

    You must restart Directory Server for the change to take effect.

  4. Provide a listing of the changelog directory.

    UNIX and Linux

    ls -la changelog-dir

    Windows

    dir changelog-dir

    If you cannot find the change log, examine the Directory Server configuration file, server-root/slapd-serverID/config/dse.ldif, to find the path. The path is specified as the values of attribute nsslapd-changelogdir.

  5. Collect output from the insync command.

    The insync command indicates the state of synchronization between a master replica and one or more consumer replicas. The following command shows the state over a period of 30 seconds.

    server-root/shared/bin/insync -s "cn=Directory Manager:password@hostname1:ldap-port" -c "cn=Directory Manager:password@hostname2:ldap-port" 30

  6. Collect output from the repldisc command.

    The repldisc command displays the replication topology, building a graph of all known replica, then showing the result as a matrix.

    server-root/shared/bin/repldisc -D "cn=Directory Manager" -w password -b base-dn -s host:ldap-port

    Here, base-dn is the DN of the replicated suffix.

ProcedureTo Collect Required Debug Data For Directory Server Schema Problems

This procedure describes what data to collect when experiencing Directory Server schema violation errors.

  1. Collect a copy of the schema folder, server-root/slapd-serverID/config/schema, and all the files in the folder.

  2. Indicate whether you have the same problem both in the Console and on the command line.

  3. Provide a list of the last changes made to the schema files.

  4. If the schema violation occurred during an add, modify, delete, or replace operation, provide an LDIF representation of the changes and a list of the commands used.