This chapter contains the following sections:
Set Up Java for Troubleshooting
Set up the Java environment and command-line options to enable gathering relevant data for troubleshooting.
To set up Java, perform the following:
- Update the Java version: Use the latest Java version to avoid spending time on troubleshooting issues in Java that were fixed. Often, a problem caused by a bug in the Java runtime is fixed in the latest update release. Working with the latest Java version helps avoid some known and common issues.
- Set up the Java environment to debug: Consider the following scenarios while setting up a bigger Java application, starting an application with a launcher script, or running distributed Java on several machines.
- Make it easy to change the Java version: Using the latest Java version helps avoid many runtime issues. If your application starts by running a script, ensure that you have to update the Java path in only one place. If you run in a distributed system, then think about easy ways to change the Java versions across all of the machines.
- Make it easy to change the Java command-line options: Sometimes, while troubleshooting, you may want to change Java options; for example, to add a verbose output, to turn off a feature, or to tune Java for better performance. Prepare the systems for these changes.In a Java application that is running remotely, for example in a testing framework or a cloud solution, ensure that you can still change the Java flags easily. Sometimes, the application takes command-line parameters, or you may want to try a flag quickly to reproduce a problem. Prepare the systems to make these changes easy.
Enable Options and Flags for JVM Troubleshooting
Set up JVM options and flags to enable gathering relevant data for troubleshooting.
The data you gather depends on the system and what data you would use in case you run into problems. Consider gathering the following data.
- Enable core files: If Java crashes, for example due to a segmentation fault, the OS saves to disk a core file (complete dump of the memory). On Linux and Solaris, core files are sometimes disabled by default. To enable core files on Linux/Solaris, it is usually enough to run the
ulimit -c unlimitedbefore starting the application command. Some systems may have different ways to handle these limits.
Note:The core files take up a lot of disk space, especially when run with a large Java heap.
To decide whether to enable core files, consider what you would do if you had a crash in your system. Would you want to see a core file? Many Java users won't have much use for a core file. However, if you would want to debug a possible crash either in a native debugger such as
gdbor by using the Serviceability Agent, then ensure that you enable core files before the starting the application.
Many times, crashes are hard to reproduce; therefore, enable core files before the starting the application.
- Add -XX:+HeapDumpOnOutOfMemoryError to the JVM flags: The
-XX:+HeapDumpOnOutOfMemoryErrorflag saves a Java Heap dump to disk if the applications runs into an
Like core files, heap dumps can be very large, especially when run with a big Java heap.
Again, think about what you would do if the application runs into an
OutOfMemoryError. Would you want to inspect the heap at the time of the error? In that case, turn flag by default so that you get this data if the application runs into an unexpected
- Run a continuous Java flight recording: Set up Java to run with a continuous flight recording.
Continuous flight recordings are a circular buffer of JFR events. If the application runs into an issue, you can dump the data from the last hour of the run. The JFR events can be helpful to debug a wide range of issues from memory leaks to network errors, high CPU usage, thread blocks, and so on.
The overhead of running with a continuous flight recording is very low. See How to Produce a Flight Recording for producing a continuous Java Flight Recording.
- Add -verbosegc to the JVM command-line: The flag
-verbosegclogs basic information about Java Garbage Collector. This log helps you find the following:
Does garbage collection run for a long time?
Does the free memory decrease over time?
The garbage collector log helps diagnose issues when the application throws an
OutOFMemoryErroror the application runs into performance issues; therefore, turning on the
-verbosegcflag by default helps troubleshoot issues.
Note:Use log rotation so that an application restart doesn't delete the previous logs. Since JDK7, the flags
NumberOfGCLogFilescan be used to set up for log rotation. For a description of these flags, see Debugging Options for Java HotSpot VM.
- Print Java version and JVM flags: Before filing a bug on Java or seeking help from a forum, have the basic information handy in the log files. For example, it's helpful to print the Java version and the JVM flags used.
If your application starts with a script, run
java -versionto print the Java version and print the command line before executing it. Another alternative is to add
-showversionto the JVM arguments.
- Set up JMC JMX for remote monitoring: JMX can be used to connect to a Java application remotely using tools such as Mission Control or Visual VM. Unless you can run these tools on the same machine that is running your application, setting this up can be helpful later on to monitor the application, send diagnostic commands, manage flight recordings, and so on. There is no performance overhead if you enable JMX.
Another alternative, is to enable JMX after a Java application has started is to use the diagnostic command
jcmd <pid> help ManagementAgent.startfor a list of flags that can be sent with the command.
See The jcmd Utility.
Gather Relevant Data
If your application runs into a problem and you want to debug the problem further, ensure that you collect any relevant data before restarting the system, especially if restarting will remove previous files.
- It is important to gather the following files:
Core files for crash issues.
hs_errprinted text file for Java crashes.
Log files: Java and application logs.
Java heap dumps for
Java flight recordings (if enabled). If the problem didn't terminate the application, dump the continuous recordings.
- If the application stopped responding, then gather the following files:
Stack traces: Take several stack traces using
jcmd <pid> Thread.printbefore restarting the system.
Dump flight recordings (if enabled).
Force a core file: If the application can't be closed properly, then stop the application, and force a core file using
kill -6 <pid>on Linux or Solaris systems.