Pre-General Availability: 2017-05-23

1 Prepare Java for Troubleshooting

This chapter provides some guidelines for setting up both Java and a Java application for better troubleshooting techniques; in case you run into problems at a later stage. These proactive Java setups help debug and narrow down issues with Java and the application. Not all suggestions apply to every application.

This chapter contains the following sections:

Set Up Java for Troubleshooting

Set up the Java environment and command-line options to enable gathering relevant data for troubleshooting.

To set up Java perform the following:

  1. Update the Java version: As a first step, use the latest Java version to avoid spending time on troubleshooting issues in Java that have been fixed. Often, a problem caused by a bug in the Java runtime is fixed in the latest update release. Working on the latest Java version helps avoid some known and common issues.
  2. Set up the Java environment to debug: Consider the following scenarios while setting up a bigger Java application, starting an application with a launcher script, or running distributed Java on several machines.
    1. Make it easy to change the Java version: Using the latest Java version helps avoid many runtime issues. If your application starts by running a script, make sure that you have to update the Java path in only one place. If you run in a distributed system, think about easy ways to change the Java versions across all of the machines.
    2. Make it easy to change the Java command-line options: Sometimes while troubleshooting you may want to change Java options; for example, to add a verbose output, to turn off a feature, or to tune Java for better performance. Make sure to prepare the systems for these changes.
      In a Java application that is running remotely, for example in a testing framework or a cloud solution, make sure you can still change the Java flags easily. Sometimes the application takes command-line parameters or you may want to try a flag quickly to reproduce a problem. Prepare the systems to make these changes easy.

Enable Options/Flags for JVM Troubleshooting

Set up JVM options/flags to enable gathering relevant data for troubleshooting.

The data you gather depends on the system and what data you actually would use in case you ran into problems. Consider gathering the following data before you run into a problem.

  1. Enable core files: If Java crashes, for example due to a segmentation fault, the OS saves to disk a core file (complete dump of the memory). On Linux and Solaris, core files are sometimes disabled by default. To enable core files on Linux/Solaris, it is usually enough to run the command ulimit -c unlimited on the command line before starting the application (some systems may have different ways to handle these limits).

    Note:

    The core files take up a lot of disk space, especially when run with a large Java heap.

    To decide whether to enable core files, consider what you would do if you had a crash in your system. Would you want to see a core file? Many Java users won't have much use for a core file. However, if you would want to debug a possible crash either in a native debugger such as gdb or by using the Serviceability Agent, then make sure that you enable core files before the start of application.

    Many times crashes may be hard to reproduce; therefore, enable core files before the start of application.

  2. Add -XX:+HeapDumpOnOutOfMemoryError to the JVM flags: The -XX:+HeapDumpOnOutOfMemoryError flag saves a Java Heap dump to disk if the applications runs into an OutOfMemoryError.

    Like core files, heap dumps can be very large, especially when run with a big Java heap.

    Again, think about what you would do if the application runs into an OutOfMemoryError. Would you want to inspect the heap at the time of the error? In that case, turn onthis flag by default so that you get this data if the application runs into an unexpected OutOfMemoryError.

  3. Run a continuous Java flight recording: The Java Flight Recorder (JFR) is a commercial feature. You can use it for free on developer desktops/laptops, and for evaluation purposes in test, development, and production environments. However, to enable JFR on a production server, you need a commercial license.

    Set up Java to run with a continuous flight recording. Continuous flight recordings are a circular buffer of JFR events. If the application runs into an issue, you can dump the data from the last hour of the run. The JFR events can be extremely helpful to debug a wide range of issues from memory leaks to network errors, high CPU usage, thread blocks and so on.

    The overhead of running with a continuous flight recording is very low. See How to Produce a Flight Recording for producing a continuous Java Flight Recording.

  4. Add -verbosegc to the JVM command-line: The flag -verbosegc logs basic information about Java Garbage Collector. This log helps you find the following:
    • Does garbage collection run for a long time?

    • Does the free memory decrease over time?

    The garbage collector log helps diagnose issues when the application throws an OutOFMemoryError or the application runs into performance issues; therefore, turning on the -verbosegc flag on by default helps troubleshoot issues.

    Note:

    Use log rotation so that an application restart doesn't delete the previous logs. Since JDK7, the flags UseGClogFileRotation and NumberOfGCLogFiles can be used to set up for log rotation. For description on these flags, see Debugging Options for Java HotSpot VM.
  5. Print Java version and JVM flags: Prior to filing a bug on Java or seeking help from a forum, have the basic information handy in the log files. For example, it's helpful to print out the Java version and the JVM flags used.

    If your application starts with a script, simply run java -version to print the Java version and print the command line before executing it. Another alternative is to add -XX+PrintCommandLineFlags and -showversion to the JVM arguments.

  6. Set up JMC JMX for remote monitoring: JMX can be used to connect to a Java application remotely using tools such as Mission Control or Visual VM. Unless you can run these tools on the same machine that is running your application, setting this up can be helpful later on to monitor the application, send diagnostic commands, manage flight recordings and so on. There is no performance overhead in enabling JMX.

    Another alternative to enable JMX after a Java application has started is to use the diagnostic command ManagementAgent.start. Run jcmd <pid> help ManagementAgent.start for a list of flags that can be sent with the command.

    See The jcmd Utility.

Gather Relevant Data

If your application runs into a problem and you want to debug the problem further, make sure that you collect any relevant data before restarting the system, especially if restarting will remove previous files.

  • It is important to gather the following files:
    • Core files for crash issues.

    • hs_err printed text file for Java crashes.

    • Log files: Java and application logs.

    • Java heap dumps for -XX:+HeapDumpOnOutOfMemoryError.

    • Java flight recordings (if enabled) - If the problem didn't terminate the application, dump the continuous recordings.

  • If the application has stopped responding, then gather the following files:
    • Stack traces: Take several stack traces using jcmd <pid> Thread.print before restarting the system.

    • Dump flight recordings (if enabled).

    • Force a core file: If the application can't be closed properly, then stop the application and force a core file using kill -6 <pid> on Linux or Solaris systems.

Make a Java Application Easier to Debug

Using a logging framework is a good way to enable future debugging.

If you run into problems in a specific module, you should be able to enable logging in that module. It is also good to specify different levels of logging, for example info, debug and trace.