|
| |
| Sun Java System Application Server 7 2004Q2 Update 1 Standard and Enterprise Edition Troubleshooting Guide | |
Chapter 1
Troubleshooting ApproachThis chapter provides a description of the tools, methods, and information sources available for troubleshooting the Application Server. Guidelines for evaluating and investigating a problem are included.
The following sections are contained in this chapter:
Identifying the ProblemJ2EE application servers are typically deployed in complex and highly sophisticated operating environments. The Sun Java System Application Server covers a broad range of technologies, including Java, Java servlets, XML, JSP, JDBC data sources, EJB technology, and more. Other products and tools associated with the Application Server are LDAP, Web Server, Sun ONE Message Queue, deployment and migration tools, and so on. Understanding and diagnosing complex issues involving so many disparate components requires thorough knowledge and a careful diagnostic process.
Gathering any or all of the following information will help you classify the problem, so you can more easily search for a solution.
Note
Operating system utilities such as pkginfo and showrev on Solaris and rpm on Linux are helpful in gathering system information.
- What are the exact version numbers of the operating system and products installed?
- Have any patches been applied? If so, specify product and operating system patch numbers.
- How is the system configured?
- What system resources does the system have (memory, disk, swap space, and so on)?
- How many application servers, web servers, and directory servers are installed?
- How is the web server connected to Sun ONE Application Server? On the same machine or not?
- How is the Sun ONE Application Server connected to the directory server?
- Are Sun ONE Application Servers in a cluster or not?
- Was any upgrade done? If so, what were source and target versions?
- Was a migration done? If so, what were source and target versions?
- Have any new applications been deployed?
- Is SSL enabled or not?
- What versions of the HADB and the backend database are being used?
- What JDBC driver is being used to access the database?
- What JDK version is being used?
- What are the JVM heap, stack, and garbage collection-related parameters set to?
- What are the JVM options?
- Are the interoperating component versions in compliance with the compatibility matrix specified in the release notes?
After gathering this information, you will want to do the following:
- Collect web server error and access log data (web server instance-specific). Refer to Server Logs for information on specific logs.
- Collect any Sun ONE Application Server stack traces.
Note
A fresh set of logs associated with the specific problem should be run. This avoids scanning gigabytes of irrelevant log information.
- Determine the sequence of events that occurred when the problem first appeared, including any steps the user may have already taken in attempting to resolve the problem.
Seeking a SolutionAfter you have identified the problem and formulated a preliminary hypothesis of what may be wrong, you are ready to do some investigation.
The following topics are addressed in this section:
Evaluate Messages
Messages generally include information about the attempted action, the outcome of the action, and, if applicable, the cause of jeopardy or failure.
Types of Messages
The log files contain the following general types of message entries:
- Information—These messages mark normal completion of particular tasks.
- Warning—These messages mark non-critical failures. Warning messages generally contain information about the cause and the nature of the failure, and also provide possible remedies.
- Error—These messages mark critical failures that cause status to be reported as Failed. Error messages generally provide detailed information about the nature and the cause of the problem that occurred.
Error Messages
A problem is often accompanied by an error message that prevents the user from proceeding.
- In some cases, the message is very clear about what is wrong and what needs to be done to fix it. For example, you start up a domain using the asadmin start-domain command, then inadvertently issue the same command a few moments later, after the domain has been started. You get the following message:
- Sometimes an error message gives only general information on the problem or solution, or suggests multiple possibilities. For example:
[16/Jun/2003:22:20:50] SEVERE ( 2204): WEB0200: Configuration error in web module [JAXBProjectStudio] (while initializing virtual server [server1])
com.iplanet.ias.config.ConfigException: Failed to load deployment descriptor for: JAXBProjectStudio cause: java.io.FileNotFoundException:
In this case, the problem is not as obvious, or there could be multiple things wrong. You will have to consider various possibilities and perhaps a number of solutions. If your proposed fix is time consuming or costly, you should take steps to ensure that the fix is likely to be correct before actually doing anything.
- Some error messages are either not helpful, or give you no guidance. For example:
[23/Jun/2003:16:50:45] WARNING ( 1972): for host 127.0.0.1 trying to GET /SupplierServiceClient1/SupplierServiceClient1_SOAP.html, send-file reports: HTTP4144: error sending D:/Sun/studio5_se/appserver7/domains/domain1/server1/applications/j2ee-modules/Su pplierServiceClient1_1/SupplierServiceClient1_SOAP.html (Overlapped I/O operation is in progress.) status=1:5
For descriptions of all the Application Server error messages, refer to the Sun Java System Application Server Error Messages Reference at the following location:
Examine Log Files
A number of the Application Server subsystems create log files and log their events to these files. The primary purpose of these log files is to provide troubleshooting information.
In addition to the message text, a logged message provides the following information:
- Date and time of the event
- Log level for the event—Application Server-specified log level ID or name
- Process identifier (PID)—PID of the Application Server process
- (Optional) Virtual server identifier (VSID)—VSID that generated the message
- Message identifier (MID)—subsystem and a four digit integer
- Message data
The specific logs associated with each Application Server problem area are discussed in the associated chapters of this manual.
Log Levels
The Sun Java System Application Server has many log levels that can be set in the Administration interface (FINEST, FINER, FINE, CONFIG, INFO, WARNING, SEVERE, ALERT, and FATAL). All messages are logged when the log level is set to FINEST and only serious error messages appear if the log level is set to FATAL.
All messages with a log level less than the default level of INFO (FINEST, FINER, FINE, and CONFIG) provide information related to debugging and must be specifically enabled. Instructions for doing this are contained in the Sun Java System Application Server Administrator’s Guide.
In addition to the standard JDK log levels, the Application Server has added log levels designed to map more intuitively to the Admin Server log file (server.log) and to tightly integrate with Solaris. The log levels ALERT and FATAL are specific to Application Server and are not implemented in the JDK1.4 logging API.
Log Options
The Administration interface provides the following two logging options:
When these options are set, stdout and stderr messages are written to the server.log file. (The event log is a syslog daemon on Solaris.)
If the above options are not set:
Client Side Logging
The Application Client Container (ACC) has its own log service and can only log to a local file. The ACC typically runs in its own process, on a different host from the Sun ONE Application Server. It has its own logging infrastructure and its own log file. The sun-acc.xml file contains the ACC configuration. Refer to the Sun Java System Application Server Developer’s Guide to Clients for more information.
Obtaining a Thread Dump
This section explains how to obtain a thread dump for Application Server 7. By default, the server dumps a core file and restarts with the -Xrs java-option flag in the server.xml file.
- Verify that the server.xml file for your server instance does not include the -Xrs java-option flag. Remove the -Xrs java-option flag if it exists.
- If the option was changed, restart your server instance.
- Run the following command on the application server instance:
kill -3 pid
The kill command will redirect the thread dump to the server.log file for the instance.
See if the Problem has been Solved Before
A good initial step is to scan this Troubleshooting Guide to see if the problem is addressed here. If so, select the appropriate solution. Many of the solutions contain references to other documents in the Application Server document collection if you need to seek more details, explanations, or examples.
Additional resources for researching an existing problem are available within Sun Microsystems:
Search the Product Documentation
Start by reading the Release Notes for the version of the product you are troubleshooting.
The documentation for this Application Server product release is here:
A description of the Application Server manuals can be found in Using the Documentation.
Search the Knowledge Base
The Knowledge Base is a collection of articles on product issues that may provide information helpful to you in troubleshooting. To access the Knowledge Base:
- Go to SunSolve.
http://sunsolve.sun.com/pub-cgi/show.pl?target=home
- Under SunSolve Collections, click the Search Collections link.
- Select the checkbox for Free SunONE Articles.
- Click Next.
- Enter your search criteria.
- Click Go.
Search or Participate in the Online Forum
You can browse directly in any of the online forums, or log in and register to start posting your own messages. The Application Server online forum is here:
Contact Support
When necessary, gather together the information you have acquired and Contact Product Support, as described on (more...) .