Gather Additional Information for Specific Problems

If you are experiencing issues related to the following, specific problem areas, you must gather some additional information. Be sure to read the pertinent sections before calling Support.

• Installation Problems

• Hung Systems

• Performance Issues

• System Crashes

Installation Problems

If you are having problems with your Waveset installation, you must have the following information available for Support:

• Information about your repository configuration

  You can type the lh setRepo -c -v command in a terminal window to view detailed information about your repository configuration.

• Database information

  You should be able to answer the following questions about your database:

  • What type of database is installed?

  • What database version, including any patches, are you using?

  • What JDBC driver is being used to connect?

  ---

  Note –

  If you do not know this information, consult your database administrator.

  ---

For upgrade issues, Support asks for this information on a case-by-case basis:

• Current level. What version of Waveset are you upgrading from?

• Target level. What version of Waveset are you upgrading to?

Hung Systems

If your system is hung, try to collect following information:

• Thread dump. A thread dump creates a stack trace for every running thread. Support can use thread dumps to determine the state of the system.

  To collect a thread dump, send a SIGQUIT signal to the Java process running Waveset, as follows:

  • In UNIX: Type kill -3 pid.

    Where pid is the process ID of the JVM running Waveset (usually the same as Application Server).

  • In Windows: Press CTRL-BREAK in the JVM console.

• Heap dump. A heap dump contains the entire status of what is in memory for a Java process.

  To force a heap dump, type the jmap -dump:file=filenamepid command.

  Where:

  • filename is the heap dump file to be created. Be aware that this file can be very large.

  • pid is the process ID of the JVM running Waveset (usually the same as Application Server).

• Screen shots or saved HTML pages. Use these tools to preserve information about the last action you performed before the system hung. You might capture information about the state of the system that is worth noting and saving for review.

• Recent activity report. If you can restart Waveset, you can retrieve an activity report that might help you determine what was happening in the system before the hang occurred.

  Select the Reports tab in Waveset; and generate a Recent Systems Messages report and a Today's Activities report.

Performance Issues

Instructions for accessing the Waveset debug pages mentioned in this section are described in Accessing the Debug Pages.

If you are having performance problems, try gathering the following information for Support:

• Call Timer data. Waveset's Call Timer debug page (debug/callTimer.jsp) enables you to collect call timer statistics for different methods, which can be useful performance data.

  Use the following steps to collect the most-relevant data:

  1. Open the Clear Timing Statistics page and click Start Timing and Tracing to enable tracing and timing.

     ---

     Note –

     Waveset can only collect call timing statistics when tracing is enabled.

     ---

  2. Return to Waveset and perform the action that is causing performance issues.

  3. Refresh the Call Timer debug page.

  4. Verify that you captured performance data.

     The Show Timings table should list the methods for which statistics are available and the methods' aggregate call timer statistics. (These statistics are not broken down by caller).

  5. Click the Export button to export the data to a calltimer.xml file.

  6. Save calltimer.xml and deliver the file to Support for analysis.

  ---

  Note –

  For more information about the Call Timer debug page, see Control Timings (callTimer.jsp).

  ---

• Java Profiler data. Java profilers are third-party applications that attach to running Java processes and track all operations.

  Support might ask you to run a memory profiler if you are having extreme or low-level performance issues in Waveset or in your custom code. Running Java profilers can cause additional performance degradation, but they provide useful insight about what is happening at a low level. If running a profiler is necessary, your Support representative will give you detailed instructions.

• Main/Run Loop Tracing data. You can use Waveset Trace to capture the state of your system but to be useful, the trace must be targeted where the problem occurs. Identifying this location can be difficult, so consult your Support team if you need help.

  For common performance issues, it's likely that Support already knows which area is affected. For example, following are several major Waveset systems and which areas to trace:

  • Scheduler

    • com.waveset.task.Scheduler

    • com.waveset.task.TaskManager

  • Caching

    • com.waveset.server.AdminCache

    • com.waveset.server.ListCache

    • com.waveset.server.SubjectAuthCache

  • Reconciler

    • com.waveset.recon.ReconResults

    • com.waveset.recon.ReconTask

• Process Level Profiling data. You must sometimes gather profiling data using native tools on the server or resource having issues.

  • For UNIX: Invoke the top command from the command line to view a list of all processes with their CPU and memory consumption.

  • For Windows: Type Control—Alt—Del, click Task Manager to open the Windows Task Manager dialog. Select the Processes tab to view a list of all processes with their CPU and memory consumption. Alternatively, you can launch the Windows Task Manager by right-clicking the Task Bar and selecting Task Manager.

• Query Timer data. This utility facilitates connecting to Waveset's back-end repository and allows you to issue JDBC SQL statements directly to the database. As the name suggests, this utility times statements issued to the repository SQL database.

  When you have performance issues, the Query Timer can check the performance of specific SQL statements or check connectivity using the JDBC connection that Waveset uses from a command line. The Query Timer uses the JDBC driver, user, and password configured for Waveset.

  ---

  Note –

  The Query Timer utility has been hard coded to use the default Configurator authentication. If you modified Configurator, you will not be able to run the Query Timer in your environment.

  ---

  You invoke the Query Timer from the command line, by using a $WSHOME/bin/lh script. The following example is for a UNIX system. If necessary, modify the script appropriately for Windows.

  sh ${WSHOME}/bin/lh com.waveset.repository.QueryTimer [ sql statement ]

  Issue this script with no options to perform a query that summarizes the statistics of each object type in the repository. You can use this query to determine the size of the objects in the database.

  ---

  Note –

  If you are running Waveset 7.0, this script outputs additional information for every 1000th object of each object type in the repository, such as:

  • Type

  • numObjects

  • numAttrVals

  • objectBytes

  • attrBytes

  • objIdxBytes

  • attrIdxBytes

  ---

• Show Timings data. The Method Timings page (debug/Show_Timings.jsp) enables you to determine at a glance which processes are being run most frequently. You might use this page instead of the Call Timings page if you are only interested in a quick view of the processes being run, and where an in-depth investigation is not likely to be necessary.

  For the best results, click the Clear ALL button and then perform the action that is causing poor performance. After checking the page results, save the HTML page from the JSP and send the page to Support.

  ---

  Note –

  See Method Timings (Show_Timings.jsp) for more information.

  ---

System Crashes

If you are reporting a system crash, try to collect the following information for Support:

• Stack traces. When Java crashes, a stack trace is sometimes printed to Standard Error. If a stack trace is available, forward the trace to Oracle Support.

• Java core files. If the JVM dumps a core file, the file might be useful to Oracle Support or Sustaining. Although core files are large, collect and store them in a safe place so you can deliver them to Support upon request.

• Memory issues. If your system is running out of memory, try to gather the following information for Support before your program crashes:

  • Heap dump. You can automate heap dump generation by adding the -XX:+HeapDumpOnOutOfMemoryError option to the application server's JVM options. You can also force a heap dump on a running process as described in Hung Systems.

  • Histogram. A histogram is a list of class objects that are occupying memory for the Java application.

    You must collect histograms at set intervals over a period of time to track memory growth over time. For example, starting a reconcile consumes all system memory in ten hours, so collecting a histogram every hour results in ten histograms, or ten points of data from which to track memory growth. Save the output from each histogram to a different file, with an appropriate timestamp, and deliver this information to Support.

    To collect histograms for Support, type jmap -histo pid in a terminal window. Where pid is the Process ID of the JVM running Waveset (usually the same as application server).

• Exception tracing. Support can frequently use an exception trace to determine the cause of a fatal error. However, Waveset's exception tracing is disabled by default. To collect a useful exception trace for Support, you must enable exception tracing and then coerce Waveset into crashing again in the same way.

  To enable exception tracing, use the instructions in Tracing Exceptions.