1 Prepare Java for Troubleshooting

This chapter provides some guidelines for setting up both Java and a Java application for better troubleshooting techniques. These proactive Java setups help debug and narrow down issues with Java and the application. Not all suggestions apply to every application.

This chapter contains the following sections:

• Set Up Java for Troubleshooting

• Enable Options and Flags for JVM Troubleshooting

• Gather Relevant Data

Set Up Java for Troubleshooting

Set up the Java environment and command-line options to enable gathering relevant data for troubleshooting.

To set up Java, perform the following:

1. Update the Java version: Use the latest Java version to avoid spending time on troubleshooting issues in Java that were fixed. Often, a problem caused by a bug in the Java runtime is fixed in the latest update release. Working with the latest Java version helps avoid some known and common issues.
2. Set up the Java environment to debug: Consider the following scenarios while setting up a bigger Java application, starting an application with a launcher script, or running distributed Java on several machines.
   1. Make it easy to change the Java version: Using the latest Java version helps avoid many runtime issues. If your application starts by running a script, ensure that you have to update the Java path in only one place. If you run in a distributed system, then think about easy ways to change the Java versions across all of the machines.
   2. Make it easy to change the Java command-line options: Sometimes, while troubleshooting, you may want to change Java options; for example, to add a verbose output, to turn off a feature, or to tune Java for better performance. Prepare your systems for these changes.
      In a Java application that is running remotely, for example in a testing framework or a cloud solution, ensure that you can still change the Java flags easily. Sometimes, the application takes command-line parameters, or you may want to try a flag quickly to reproduce a problem. Prepare the systems to make these changes easy.

Enable Options and Flags for JVM Troubleshooting

Set up JVM options and flags to enable gathering relevant data for troubleshooting.

The data you gather depends on the system and what data you would use in case you run into problems. Consider gathering the following data.

1. Enable core files: If Java crashes, for example due to a segmentation fault, the OS saves to disk a core file (complete dump of the memory). On Linux, core files are sometimes disabled by default. To enable core files on Linux, it is usually enough to run the ulimit -c unlimited before starting the application command. Some systems may have different ways to handle these limits.

   Note:

   The core files take up a lot of disk space, especially when enabled.

   To decide whether to enable core files, consider what you would do if you had a crash in your system. Would you want to see a core file? Many Java users won't have much use for a core file. However, if you would want to debug a possible crash either in a native debugger such as gdb or by using the Serviceability Agent, then ensure that you enable core files before the starting the application.

   Many times, crashes are hard to reproduce; therefore, enable core files before the starting the application.

2. Add -XX:+HeapDumpOnOutOfMemoryError to the JVM flags: The -XX:+HeapDumpOnOutOfMemoryError flag saves a Java heap dump to disk if the applications runs into an OutOfMemoryError.

   Like core files, heap dumps can be very large, especially when run with a big Java heap.

   Again, think about what you would do if the application runs into an OutOfMemoryError. Would you want to inspect the heap at the time of the error? In that case, enable this option so that you get this data if the application runs into an unexpected OutOfMemoryError. (The option is disabled by default.)

3. Run a continuous flight recording: Set up Java to run with a continuous flight recording.

   Continuous flight recordings are a circular buffer of Flight Recorder events. If the application runs into an issue, you can dump the data from the last hour of the run. The Flight Recorder events can be helpful to debug a wide range of issues from memory leaks to network errors, high CPU usage, thread blocks, and so on.

   The overhead of running with a continuous flight recording is very low. See Produce a Flight Recording.

4. Add -verbosegc to the JVM command-line: The flag -verbosegc logs basic information about Java Garbage Collector. This log helps you find the following:

   • Does garbage collection run for a long time?

   • Does the free memory decrease over time?

   The garbage collector log helps diagnose issues when the application throws an OutOFMemoryError or the application runs into performance issues; therefore, turning on the -verbosegc flag helps troubleshoot issues.

   Note:

   Use log rotation so that an application restart doesn't delete the previous logs. Since JDK7, the flags UseGClogFileRotation and NumberOfGCLogFiles can be used to set up for log rotation. For a description of these flags, see Debugging Options for Java HotSpot VM.
5. Print Java version and JVM flags: Before filing a bug on Java or seeking help from a forum, have the basic information handy in the log files. For example, it's helpful to print the Java version and the JVM flags used.

   If your application starts with a script, run java -version to print the Java version and print the command line before running it. Another alternative is to add -XX+PrintCommandLineFlags and -showversion to the JVM arguments.

6. Set up JMX for remote monitoring: JMX can be used to connect to a Java application remotely using tools such as JDK Mission Control or Visual VM. Unless you can run these tools on the same machine that is running your application, setting this up can be helpful later on to monitor the application, send diagnostic commands, manage flight recordings, and so on. There is no performance overhead if you enable JMX.

   Another alternative, is to enable JMX after a Java application has started is to use the diagnostic command ManagementAgent.start. Run jcmd <pid> help ManagementAgent.start for a list of flags that can be sent with the command.

   See The jcmd Utility.

Gather Relevant Data

If your application runs into a problem and you want to debug the problem further, ensure that you collect any relevant data before restarting the system, especially if restarting will remove previous files.

• It is important to gather the following files:

  • Core files for crash issues.

  • hs_err printed text file for Java crashes.

  • Log files: Java and application logs.

  • Java heap dumps for -XX:+HeapDumpOnOutOfMemoryError.

  • Flight recordings (if enabled). If the problem didn't terminate the application, dump the continuous recordings.

• If the application stopped responding, then gather the following files:

  • Stack traces: Take several stack traces using jcmd <pid> Thread.print before restarting the system.

  • Dump flight recordings (if enabled).

  • Force a core file: If the application can't be closed properly, then stop the application, and force a core file using kill -6 <pid> on Linux systems.

Make a Java Application Easier to Debug

Using a logging framework is a good way to enable future debugging.

If you run into problems in a specific module, you should be able to enable logging in that module. It is also good to specify different levels of logging, for example info, debug, and trace. For more information about Java logging, see Java Logging Overview.