Before You Begin Tracing or Troubleshooting
Review the information in the following sections before tracing or troubleshooting Waveset:
Intended Audience
This chapter is intended for application server and database administrators, front-line support engineers, and partners who are responsible for maintaining Waveset in a deployment environment.
Before troubleshooting problems with Waveset, you must
-
Be familiar with Java 5.0 (required for Oracle Waveset 8.1.1)
-
Understand the components that you are trying to troubleshoot
Important Notes About Tracing and Troubleshooting Waveset
Before you start tracing or troubleshooting Waveset, note the following:
-
Tracing affects system performance. To ensure optimal performance, use the minimum tracing level needed or turn off tracing after debugging the systems.
-
Do not enable tracing for the com.waveset class. The com.waveset class is verbose and has many classes, so tracing this class might cause your server to hang.
-
Do not configure Waveset to trace at the advanced level unless instructed to do so by Oracle Support.
-
If you set exception.trace in the Waveset.properties file to debug a specific problem, do not use it for an extended period of time and be sure you disable it after debugging. Setting exception.trace in the Waveset.properties file can significantly affect system performance.
-
By default, Waveset returns stack trace information as HTML comments when certain errors occur. Generally, a stack trace is not visible to the end-user, but the trace can be seen by viewing the page source in the browser.
Stack trace information can be very useful when diagnosing a problem, but exposes information about the execution of the web application that could be considered a security risk. Production deployments should consider turning off stack trace information rendering by configuring the following setting in Waveset.properties:
ui.web.disableStackTraceComments=true
You must perform this change manually on each Waveset server.
Note –Beginning with Waveset version 8.1.1, the default setting for ui.web.disableStackTraceComments is true. In previous releases, the default setting for ui.web.disableStackTraceComments was false.
However, because version 8.1.1 is an upgrade and not an install, the new setting does not automatically take place because Waveset.properties is not overridden during an upgrade.
Before Calling Support
Before calling Oracle Technical Support or Sustaining for assistance, you must prepare for the call by collecting as much information as possible about your environment and the problem. Instructions are provided in the following sections:
Note –
Unless specifically noted otherwise, the commands described in the following sections will work from a properly sourced terminal on UNIX or Windows.
Identify the Problem Area
Try using one or more of the following methods to narrow the problem area:
-
Use web browser search tools to research the problem.
-
Use resource-specific support options when available. Your problem might be related to a resource and fixed by using a resource patch.
-
Remove Waveset from the equation by using an appropriate client tool when possible. For example, by using a Java LDAP browser.
Gather Required Information
You must have the following information about your environment and Waveset installation available for all calls to your Support or Sustaining representative.
Gather Additional Information for Specific Problems
If you are experiencing issues related to the following, specific problem areas, you must gather some additional information. Be sure to read the pertinent sections before calling Support.
Installation Problems
If you are having problems with your Waveset installation, you must have the following information available for Support:
-
Information about your repository configuration
You can type the lh setRepo -c -v command in a terminal window to view detailed information about your repository configuration.
-
Database information
You should be able to answer the following questions about your database:
-
What type of database is installed?
-
What database version, including any patches, are you using?
-
What JDBC driver is being used to connect?
Note –If you do not know this information, consult your database administrator.
-
Note –
For upgrade issues, Support asks for this information on a case-by-case basis:
-
Current level. What version of Waveset are you upgrading from?
-
Target level. What version of Waveset are you upgrading to?
Hung Systems
If your system is hung, try to collect following information:
-
Thread dump. A thread dump creates a stack trace for every running thread. Support can use thread dumps to determine the state of the system.
To collect a thread dump, send a SIGQUIT signal to the Java process running Waveset, as follows:
-
In UNIX: Type kill -3 pid.
Where pid is the process ID of the JVM running Waveset (usually the same as Application Server).
-
In Windows: Press CTRL-BREAK in the JVM console.
-
-
Heap dump. A heap dump contains the entire status of what is in memory for a Java process.
To force a heap dump, type the jmap -dump:file=filenamepid command.
Where:
-
filename is the heap dump file to be created. Be aware that this file can be very large.
-
pid is the process ID of the JVM running Waveset (usually the same as Application Server).
-
-
Screen shots or saved HTML pages. Use these tools to preserve information about the last action you performed before the system hung. You might capture information about the state of the system that is worth noting and saving for review.
-
Recent activity report. If you can restart Waveset, you can retrieve an activity report that might help you determine what was happening in the system before the hang occurred.
Select the Reports tab in Waveset; and generate a Recent Systems Messages report and a Today's Activities report.
Performance Issues
Note –
Instructions for accessing the Waveset debug pages mentioned in this section are described in Accessing the Debug Pages.
If you are having performance problems, try gathering the following information for Support:
-
Call Timer data. Waveset's Call Timer debug page (debug/callTimer.jsp) enables you to collect call timer statistics for different methods, which can be useful performance data.
-
Open the Clear Timing Statistics page and click Start Timing and Tracing to enable tracing and timing.
Note –Waveset can only collect call timing statistics when tracing is enabled.
-
Return to Waveset and perform the action that is causing performance issues.
-
Refresh the Call Timer debug page.
-
Verify that you captured performance data.
The Show Timings table should list the methods for which statistics are available and the methods' aggregate call timer statistics. (These statistics are not broken down by caller).
-
Click the Export button to export the data to a calltimer.xml file.
-
Save calltimer.xml and deliver the file to Support for analysis.
Use the following steps to collect the most-relevant data:
Note –For more information about the Call Timer debug page, see Control Timings (callTimer.jsp).
-
-
Java Profiler data. Java profilers are third-party applications that attach to running Java processes and track all operations.
Support might ask you to run a memory profiler if you are having extreme or low-level performance issues in Waveset or in your custom code. Running Java profilers can cause additional performance degradation, but they provide useful insight about what is happening at a low level. If running a profiler is necessary, your Support representative will give you detailed instructions.
-
Main/Run Loop Tracing data. You can use Waveset Trace to capture the state of your system but to be useful, the trace must be targeted where the problem occurs. Identifying this location can be difficult, so consult your Support team if you need help.
For common performance issues, it's likely that Support already knows which area is affected. For example, following are several major Waveset systems and which areas to trace:
-
Process Level Profiling data. You must sometimes gather profiling data using native tools on the server or resource having issues.
-
For UNIX: Invoke the top command from the command line to view a list of all processes with their CPU and memory consumption.
-
For Windows: Type Control—Alt—Del, click Task Manager to open the Windows Task Manager dialog. Select the Processes tab to view a list of all processes with their CPU and memory consumption. Alternatively, you can launch the Windows Task Manager by right-clicking the Task Bar and selecting Task Manager.
-
-
Query Timer data. This utility facilitates connecting to Waveset's back-end repository and allows you to issue JDBC SQL statements directly to the database. As the name suggests, this utility times statements issued to the repository SQL database.
When you have performance issues, the Query Timer can check the performance of specific SQL statements or check connectivity using the JDBC connection that Waveset uses from a command line. The Query Timer uses the JDBC driver, user, and password configured for Waveset.
Note –The Query Timer utility has been hard coded to use the default Configurator authentication. If you modified Configurator, you will not be able to run the Query Timer in your environment.
You invoke the Query Timer from the command line, by using a $WSHOME/bin/lh script. The following example is for a UNIX system. If necessary, modify the script appropriately for Windows.
sh ${WSHOME}/bin/lh com.waveset.repository.QueryTimer [ sql statement ]Issue this script with no options to perform a query that summarizes the statistics of each object type in the repository. You can use this query to determine the size of the objects in the database.
Note –If you are running Waveset 7.0, this script outputs additional information for every 1000th object of each object type in the repository, such as:
-
Type
-
numObjects
-
numAttrVals
-
objectBytes
-
attrBytes
-
objIdxBytes
-
attrIdxBytes
-
-
Show Timings data. The Method Timings page (debug/Show_Timings.jsp) enables you to determine at a glance which processes are being run most frequently. You might use this page instead of the Call Timings page if you are only interested in a quick view of the processes being run, and where an in-depth investigation is not likely to be necessary.
For the best results, click the Clear ALL button and then perform the action that is causing poor performance. After checking the page results, save the HTML page from the JSP and send the page to Support.
Note –See Method Timings (Show_Timings.jsp) for more information.
System Crashes
If you are reporting a system crash, try to collect the following information for Support:
-
Stack traces. When Java crashes, a stack trace is sometimes printed to Standard Error. If a stack trace is available, forward the trace to Oracle Support.
-
Java core files. If the JVM dumps a core file, the file might be useful to Oracle Support or Sustaining. Although core files are large, collect and store them in a safe place so you can deliver them to Support upon request.
-
Memory issues. If your system is running out of memory, try to gather the following information for Support before your program crashes:
-
Heap dump. You can automate heap dump generation by adding the -XX:+HeapDumpOnOutOfMemoryError option to the application server's JVM options. You can also force a heap dump on a running process as described in Hung Systems.
-
Histogram. A histogram is a list of class objects that are occupying memory for the Java application.
You must collect histograms at set intervals over a period of time to track memory growth over time. For example, starting a reconcile consumes all system memory in ten hours, so collecting a histogram every hour results in ten histograms, or ten points of data from which to track memory growth. Save the output from each histogram to a different file, with an appropriate timestamp, and deliver this information to Support.
To collect histograms for Support, type jmap -histo pid in a terminal window. Where pid is the Process ID of the JVM running Waveset (usually the same as application server).
-
-
Exception tracing. Support can frequently use an exception trace to determine the cause of a fatal error. However, Waveset's exception tracing is disabled by default. To collect a useful exception trace for Support, you must enable exception tracing and then coerce Waveset into crashing again in the same way.
To enable exception tracing, use the instructions in Tracing Exceptions.
Related Documentation and Web Sites
In addition to the information provided in this chapter, consult the documents and web sites listed in this section for information related to tracing and troubleshooting Waveset.
Recommended Reading
See the following documents for information related to tracing and troubleshooting Waveset:
-
Testing Your Customized Form in Oracle Waveset 8.1.1 Deployment Reference for the recommended method of tracing XPRESS functions.
-
The article titled Using JConsole to Monitor Applications for information about using JConsole to monitor applications that run on the Java platform. This article is available from the following URL, http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html.
Useful Web Sites
The following table describes web sites related to troubleshooting Waveset.
Table 5–1 Useful Web Sites
Process Overview
Generally, you can identify and resolve problems in your deployment if you follow the steps shown in the following illustration.
Figure 5–1 Tracing and Troubleshooting a Problem
See Tracing Waveset Objects and Activities for information about how to enable tracing for different Waveset objects and activities.
See Troubleshooting and Fixing Common Problems for information about how to troubleshoot problems that commonly occur for those objects and activities.
- © 2010, Oracle Corporation and/or its affiliates