A Java application might stop running for several reasons. The most common reason is that the application finished running or was halted normally. Other reasons might be Java application errors, exceptions that cannot be handled, and irrecoverable Java errors like OutOfMemoryError. Occasionally, a JRockit JVM crash might occur, which means that the JVM encountered a problem from which it could not recover gracefully.
This chapter describes how to diagnose and troubleshoot JVM crashes. It includes information about the following topics:
The first step in diagnosing and resolving a JVM crash is to classify the crash; that is, try to determine where and why the crash occurred.
When the JRockit JVM crashes, it creates a snapshot of the state of the computer and the JVM process at the time of the crash, and writes the state information into the following crash files:
.dump file: The .dump file is a text file that is like an executive summary of the complete memory image and the environment in which the JVM was running at the time of the crash. This file is produced by the JVM itself when it crashes and is useful for classifying crashes; it can also sometimes be used for identifying problems that have already been fixed. This file rarely reveals enough information to determine the cause of the problem.
.core file: The .core file is a binary crash file produced on UNIX-like systems such as Linux and Solaris. By default, the .core file captures complete information about the entire JVM process at the time of the crash.
.mdmp file (or a minidump file): The .mdmp file is the Windows equivalent of the .core file described earlier.
If you have a support agreement with Oracle, you can contact Oracle Support for troubleshooting JRockit JVM problems. The binary crash file (.core or .mdmp) is essential when you contact Oracle Support.
For more information about crash files, see Chapter 8, "About Crash Files."
Table 6-1 lists the symptoms you can look for and the probable crash types corresponding to those symptoms.
Table 6-1 Crash Symptoms and Crash Types
| Symptom | For Troubleshooting Information, See |
|---|---|
|
The |
|
|
The size of the |
|
|
The |
|
|
On Linux only: The |
Section 6.4, "Crash Caused by Unsupported Linux Configuration" |
|
On Linux only: You are using a nonstandard or unsupported Linux configuration. |
Section 6.4, "Crash Caused by Unsupported Linux Configuration" |
|
You observe symptoms other than those listed in this table. |
The JVM reserves virtual memory for many purposes; for example, the Java heap, Java methods, thread stacks, and JVM-internal data structures. In addition, native (JNI) code can also allocate memory. The process size consists of all the memory reserved by the JVM and any third-party libraries running inside the process, and is subject to operating system limitations.
If the virtual memory allocation of the JVM process exceeds the operating system limitations, the JVM runs out of virtual memory, which may cause it to crash.
This section contains the following topics:
Before you start troubleshooting an out-of-virtual-memory error, you must verify that the error is indeed due to the JVM process running out of virtual memory.
Table 6-2 shows the maximum virtual memory available to a single process on various 32-bit operating systems. Virtual memory is practically unlimited on 64-bit platforms.
Table 6-2 Approximate Maximum Virtual Memory Available to IA32 Architectures
| Operating System | Approximate Maximum Virtual Memory |
|---|---|
|
Windows |
2 GB |
|
Windows /3GB Startup Option |
3 GB |
|
Linux (normally) |
3 GB |
You can verify whether an out-of-virtual-memory error has occurred in the following ways:
Look in the text crash file.
The text crash file .dump might indicate that memory allocations have failed. This is a strong indication that the JRockit JVM process has run out of virtual memory. For more information about the text crash file, see Chapter 8, "About Crash Files."
Check the size of the binary crash file.
When the JRockit JVM crashes, it generates a binary crash file (.core or .mdmp). By default, the binary crash file contains a copy of the entire JVM process.
Check the size of the binary crash file to determine whether the JVM process has indeed run out of virtual memory.
Verify that the binary crash file size has not been limited with the command-line option -XXdumpSize or with the operating system command ulimit (Linux and Solaris only). Use the command ulimit -a to verify that the crash file size is unlimited on Linux and Solaris. If the size of the binary crash file has been limited, you cannot use it to verify that the JVM process has run out of virtual memory.
Verify whether the size of the binary crash file is larger than the size of the heap. This is a sanity check to ensure that the binary crash file has not been truncated (due to limited disk space, for example).
Check whether the size of the binary crash file is close to the maximum process size allowed by the operating system (see Table 6-2).
After verifying that the JVM process has run out of virtual memory, you can start troubleshooting the problem as described in this section.
Ensure that you run the latest JRockit JVM release. The memory usage problems that are causing the JVM crash might have been fixed in the latest JRockit JVM release.
The Java heap is only a part of the total memory usage of the JVM. If the Java heap is too large, the JVM may fail to start or run out of virtual memory when Java methods are compiled and optimized or when native libraries are loaded. If this happens, you should try lowering the maximum heap size.
On Windows 2000 Advanced Server and Datacenter, Windows 2003, Windows XP, and subsequent Windows versions, you have the option of starting the operating system with the /3GB option (by specifying the option in the BOOT.INI file). This option changes the maximum virtual memory process size from 2 GB to 3 GB.
Check the JNI code you are using for memory leaks. Incorrectly written or used JNI code might be causing memory leaks, which results in the Java process growing until it reaches the maximum virtual memory size on the platform.
Recording the virtual memory usage can help in diagnosing the out-of-virtual-memory problem.
This section describes how you can collect virtual memory usage statistics on Windows and Linux.
Recording Virtual Memory Usage on Windows
Use the Windows tool perfmon to record the PrivateBytes process counter. Collect information about the amount of reserved virtual memory for the JVM process. To do this:
Open Performance Monitor, which you can find in the administrative tools.
Click + to open the Add Counters dialog box.
In the Performance Object list, select Process.
From the Process list, select the Private Bytes counter.
Select the process to monitor and click Add.
Recording Virtual Memory Usage on Linux
Create a script that records the virtual memory usage at regular intervals.
For example: top -b -n 10 > virtualmemory.log
This script runs top every ten seconds and puts the resulting data in the file, virtualmemory.log.
The virtual memory usage for all the running processes can be found in the VIRT column in the log file. To view only the current status, type top and press [Shift]-[M] to sort the output by memory usage. This usually puts the JVM processes at the top of the output.
Creating a recording such as virtualmemory.log can be useful as it enables you to check whether the JRockit JVM process is actually growing and provide evidence about the growth to Oracle Support.
If the solutions provided in this section do not help you resolve the problem, contact Oracle Support, as described in Chapter 9, "Contacting Oracle for Support."
A stack overflow crash occurs when the JRockit JVM cannot handle a stack overflow error gracefully. According to the J2SE Javadoc, a gracefully handled java.lang.StackOverflowError is a java.lang.VirtualMachineError thrown to indicate that the JVM is broken or has run out of resources necessary for it to continue operating.
For more information, see the following J2SE java.lang Javadocs:
Java SE 6:
Class StackOverflowError
http://java.sun.com/javase/6/docs/api/java/lang/StackOverflowError.html
Class VirtualMachineError
http://java.sun.com/javase/6/docs/api/java/lang/VirtualMachineError.html
J2SE 5.0:
Class StackOverflowError
http://java.sun.com/j2se/1.5.0/docs/api/java/lang/StackOverflowError.html
Class VirtualMachineError
http://java.sun.com/j2se/1.5.0/docs/api/java/lang/VirtualMachineError.html
The JRockit JVM .dump file includes information about the number of stack overflow errors thrown.
When the JRockit JVM crashes due to stack overflow, the text crash file (.dump) shows Error Message: Stack overflow near the top of the file. Other indications might be an extremely long stack trace in the crash file or no stack trace at all. If the .dump file shows something like StackOverFlow: 2 StackOverFlowErrors occurred, this is an indication that the crash might be triggered by a previous stack overflow problem.
This section describes possible solutions for stack overflow errors.
Often, a stack overflow error is caused by the application being coded to require stack space that exceeds the memory limits of the JRockit JVM.
Examine the stack trace in the .dump file to determine whether the Java code can be changed to use less stack space. For example, the application code might contain recursive method calls. Deep recursions can cause StackOverflow errors.
If it is not possible to change the stack requirements of the application, you can change the thread stack size by using the -Xss command-line option.
The -XX:+CheckStacks command-line option makes the JRockit JVM more robust against stack overflow errors. It usually prevents the JVM from dumping and throwing a java.lang.StackOverflowError.
Note that the -XX:+CheckStacks command-line option results in a slight performance penalty because the JVM touches pages on the stack.
If the solutions provided in this section do not help you resolve the problem, contact Oracle Support, as described in Chapter 9, "Contacting Oracle for Support."
If your application crashes when you run the JRockit JVM on Linux, even if the stack trace indicates a reason for the crash, you should ensure that you run the JRockit JVM on a supported Linux configuration.
Do the following:
Verify whether the version of your operating system is supported for the JRockit JVM.
For more information, see Oracle JRockit JDK Supported Configurations at http://www.oracle.com/technology/software/products/ias/files/fusion_certification.html.
Verify whether the correct glibc binary is installed.
Linux on IA32 must be configured to use the glibc compiled for the i686 architecture; otherwise, the JRockit JVM might crash.
You can check the version of the glibc that is installed by running the following command:
rpm -q --queryformat '\n%{NAME} %{VERSION} %{RELEASE} %{ARCH}\n' glibc
If the output shows something like glibc 2.3.4 2.25 i386, you are using an unsupported glibc binary.
Upgrade glibc to a version that is compiled for the i686 architecture.
Examine the thread library.
If you have a .core file in gdb, you can get a hint about the thread library you are using by running the command, info shared.
Look at the path of the loaded libpthreadx.so file.
If the file is in /lib/, check the result of the rpm command. If the output shows i386, you are using an unsupported glibc.
Upgrade glibc to a version that is compiled for the i686 architecture.
If solutions provided in this section do not help you resolve the problem, contact Oracle Support, as described in Chapter 9, "Contacting Oracle for Support."
A JVM crash could be caused by a programming error in the JRockit JVM or by errors in third-party library code.
Identifying and troubleshooting a JVM crash can help you find a temporary workaround until the problem is solved in the JRockit JVM. This may also help Oracle Support to identify and fix the problem faster.
The process of diagnosing and troubleshoot JVM crashes varies depending on whether the crash occurred during code generation or garbage collection.
This section describes how to identify and troubleshoot a JVM crash that occurs during code generation. It contains the following topics:
Section 6.5.1.1, "Identify the Method that Might Have Caused the Code-Generation Crash"
Section 6.5.1.2, "Verify Whether the Crash is Due to Optimization Problems"
Section 6.5.1.3, "Exclude the Problem Method from the Optimization Process"
Section 6.5.1.4, "Check Whether the Problem is Caused by an External Instrumentation Tool"
If the JVM crashed while generating code, the most common cause could be a miscompiled method. If the JRockit JVM miscompiles a method, either the JVM crashes or the behavior of the method is different from the expected behavior.
The text crash file identifies the method that was being compiled at the time of the crash. The method is identified in a line that starts with Method, located near the beginning of the .dump file.
The JRockit JVM might miscompile methods due to a problem in the optimizing compiler.
To determine whether optimization is the cause of the crash, disable optimization by specifying the -XnoOpt command-line option and then restart the application, as shown in the following example:
java -XnoOpt myApplication
If the JRockit JVM runs the application as expected, the crash is caused by code optimization.
The JRockit JVM might miscompile methods due to a problem in the optimizing compiler.
If the JRockit JVM stopped crashing after you disabled global optimization, you can exclude only the offending methods (that you identified earlier) from the optimization process by using optimization directives. If the application runs successfully without the offending method being optimized, this workaround should solve the problem.
Specifying Optimization Directives
You can control how the JRockit JVM optimizes code by specifying directives in a control file and then specifying the control file by using the -XX:OptFile=filename command-line option.
Note:
-XX:OptFile is an internal diagnostic option. So to use it, you should include -XX:+UnlockDiagnosticVMOptions as well on the command line, before -XX:OptFile.Example 6-1 is for a control file containing a single optimization directive.
Example 6-1 Directive to Perform Optimized Compilation for Specific Methods
{
match: [ "java/lang/FloatingDecimal.dtoa*", "java/lang/Object.*"],
jit: { preset : opt, } ,
}
The directive in Example 6-1 directs the JRockit JVM to optimize the methods matching the patterns specified by the match keyword when the JVM compiles the method for the first time.
Example 6-2 Directives to Restrict Optimization to Specific Methods
[
{
match: ["java/lang/FloatingDecimal.dtoa*", "java/lang/Object.*"],
jit: { preset : opt, } ,
},
{
match: "*",
hotspot : { enable : false },
}
]
The directive in Example 6-2 directs the JRockit JVM to allow optimization (hotspot) for only the java/lang/FloatingDecimal.dtoa* and java/lang/Object.* methods.
Example 6-3 Directive to Control Method Inlining
{
match: "java.lang.*",
inline: ["+java.util.*", "-com.sun.*", "sun.*" ],
}
The directive in Example 6-3 directs the JRockit JVM to do the following:
Allow inlining of method calls from java.lang.* to java.util.*
Forbid inlining of method calls from java.lang.* to com.sun.*
Allow, if applicable, inlining of method calls from java.lang.* to sun.*
Enabling Optimization Directives
After creating the control file containing the required optimization directives, specify the pathname of control file by using the -XX:OptFile=filename command-line option.
Note:
-XX:OptFile is an internal diagnostic option. So to use it, you should include -XX:+UnlockDiagnosticVMOptions as well on the command line, before -XX:OptFile.Verifying the Behavior of the Optimization Directives
To verify whether the directives work as expected, use the -Xverbose:opt command-line option and check the output; the methods that you excluded from the optimization process should not appear in the output. For more information about -Xverbose:opt, see the Oracle JRockit Command-Line Reference.
Guidelines for Creating Optimization Directives
Use the match keyword to specify the patterns of classes and methods to which the directive must be applied. The value that you specify can be either a string containing a single pattern or an array of patterns.
Use the jit keyword to specify any task for the JVM to perform when it compiles a method for the first time.
Use the inline keyword to specify method calls that should be inlined.
Enclose each directive in a pair of braces.
The control file can contain multiple directives, in which case:
Adjacent directives must be separated by a comma.
All of the directives must be enclosed in an array.
If you determine that a miscompiled method is not the reason for the JVM crash, investigate whether any external instrumentation tool you are using (for example JProbe or OptimizeIt) is causing the problem. These tools can alter bytecode, which can cause unexpected behavior.
To eliminate the possibility of the problem being caused by external instrumentation tools, disable the tools and then run the application.
If the JVM continues to crash after disabling the external tool, the problem is not caused by that tool.
If the application runs as expected, consider using a different tool or running without the tool.
If solutions provided in this section do not help you resolve the problem, contact Oracle Support, as described in Chapter 9, "Contacting Oracle for Support."
For code generation crashes, you must provide the following data to Oracle Support:
The .core or .mdmp file
The .dump file
The .class file containing the method that was being generated
Source code for the class containing the method that was being generated
This section describes how to identify and troubleshoot a JVM crash that occurs during garbage collection. It contains the following topics:
You can identify a garbage collection crash by looking at the stack trace in the text crash file (.dump).
If garbage collection functions are shown in the stack trace or if the thread that caused the crash is a garbage collection thread, the crash probably occurred during garbage collection.
Garbage collection functions in the stack trace are identified by prefixes such as mm, gc, yc, and oc.
The problem might have been fixed in the latest release of the JRockit JVM. Upgrade to the latest release and check whether the problem continues.
The garbage collection mode that you are using might be causing problems that you could avoid by changing to another garbage collection mode. For example, if you are using -Xgc:pausetime try switching to -Xgc:throughput. Note though, that if you change the garbage collection mode, you will not receive the same performance profile from the JRockit JVM.
If you are using the deterministic garbage collection mode, you cannot change to another garbage collection and yet retain the guarantees provided by deterministic garbage collection. In such a case, instead of changing the garbage collection mode, contact Oracle Support.
For more information, see the Oracle JRockit Performance Tuning Guide.
Bugs in heap compaction can sometimes cause problems leading to crashes during garbage collection. You can disable compaction by using the -XXcompaction:enable=false command-line option.
Note that using the -XXcompaction:enable=false option can lead to heap fragmentation; use it only for troubleshooting purposes. If the heap becomes too fragmented, you might encounter out-of-memory errors.
Erroneous inlining might cause broken code, which makes the garbage collector lose track of live objects. You can disable inlining by using the optimization directives as described in Specifying Optimization Directives.
You might be experiencing garbage collection crashes because the nonoptimizing JIT compiler is generating broken code that makes the garbage collector lose track of live objects. Use the -XX:+PreOpt command at startup to use the optimizing compiler for everything. Note that using the optimizing compiler can slow down the JVM startup.
If the solutions provided in this section do not help you solve the problem, contact Oracle Support as described in Chapter 9, "Contacting Oracle for Support." When you contact Oracle Support, you must include the following information:
For crashes in garbage collection, include a complete .core or .mdmp file, otherwise the support staff won't be able to resolve your issue. Verify that the .core or .mdmp file is at least as big as the Java heap.
If you can reproduce the crash, include the steps you used to reproduce it.
If you tried using another garbage collector indicate whether one garbage collector worked better than another or if crashes continued regardless of the collector used.
Include information about any workaround you attempted.