7 Flight Recorder

Flight Recorder (JFR) is a profiling and event collection framework built into the JDK.

Flight Recorder allows Java administrators and developers to gather detailed low-level information about how a JVM and Java applications are behaving. You can use JMC to visualize the data collected by JFR. Flight Recorder and JMC together create a complete toolchain to continuously collect low-level and detailed runtime information enabling after-the-fact incident analysis.

The advantages of using JFR are:

  • It records data about JVM events as they occur, with a timestamp.
  • Recording events with JFR enables you to preserve the execution states to analyze issues. You can access the data anytime to better understand problems and resolve them.
  • JFR can record a large amount of data on production systems while keeping the overhead of the recording process low.
  • It is most suited for recording latencies. It records situations where the application is not executing as expected and provide details on the bottlenecks.
  • It provides insight into how programs interact with execution environment as a whole, ranging from hardware, operating systems, JVM, JDK, and the Java application environment.

Flight recordings can be started when the application is started or while the application is running. The data is recorded as time-stamped data points called events. Events are categorized as follows:

  • Duration events: occurs at a particular duration with specific start time and stop time.
  • Instant events: occurs instantly and gets logged immediately, for example, a thread gets blocked.
  • Sample events: occurs at regular intervals to check the overall health of the system, for example, printing heap diagnostics every minute.
  • Custom events: user defined events created using JMC or APIs.

In addition, there are predefined events that are enabled in a recording template. Some templates only save very basic events and have virtually no impact on performance. Other templates may come with slight performance overhead and may also trigger garbage collections to gather additional data. The following templates are provided with Flight Recorder in the <JDK_ROOT>/lib/jfr directory:

  • default.jfc: Collects a predefined set of data with low overhead.
  • profile.jfc: Provides more data than the default.jfc template, but with overhead and impact on performance.

Flight Recorder produces following types of recordings:

  • Time fixed recordings: A time fixed recording is also known as a profiling recording that runs for a set amount of time, and then stops. Usually, a time fixed recording has more events enabled and may have a slightly bigger performance effect. Events that are turned on can be modified according to your requirements. Time fixed recordings will be automatically dumped and opened.

    Typical use cases for a time fixed recording are as follows:

    • Profile which methods are run the most and where most objects are created.

    • Look for classes that use more and more heap, which indicates a memory leak.

    • Look for bottlenecks due to synchronization and many more such use cases.

  • Continuous recordings: A continuous recording is a recording that is always on and saves, for example, the last six hours of data. During this recording, JFR collects events and writes data to the global buffer. When the global buffer fills up, the oldest data is discarded. The data currently in the buffer is written to the specified file whenever you request a dump, or if the dump is triggered by a rule.

    A continuous recording with the default template has low overhead and gathers a lot of useful data. However, this template doesn't gather heap statistics or allocation profiling.

Start a Flight Recording

Follow these steps to start a flight recording using JMC.

  1. Find your JVM in the JVM Browser.
  2. Right-click the JVM and select Start Flight Recording...

    The Start Flight Recording window opens.

  3. Click Browse to find a suitable location and file name to save the recording.
  4. Select either Time fixed recording (profiling recording), or Continuous recording. For continuous recordings, you can specify the maximum size or maximum age of events you want to save.
  5. Select the flight recording template in the Event settings drop-down list. Templates define the events that you want to record. To create your own templates, click Template Manager. However, for most use cases, select either the Continuous template (for very low overhead recordings) or the Profiling template (for more data and slightly more overhead).
  6. Click Finish to start the recording or click Next to modify the event options defined in the selected template.
  7. Modify the event options for the flight recording. The default settings provide a good balance between data and performance. You can change these settings based on your requirement.

    For example:

    • The Threshold value is the length of event recording. By default, synchronization events above 10 ms are collected. This means, if a thread waits for a lock for more than 10 ms, an event is saved. You can lower this value to get more detailed data for short contentions.
    • The Thread Dump setting gives you an option to perform periodic thread dumps. These are normal textual thread dumps.
  8. Click Finish to start the recording or click Next to modify the event details defined in the selected template.
  9. Modify the event details for the selected flight recording template. Event details define whether the event should be included in the recording. For some events, you can also define whether a stack trace should be attached to the event, specify the duration threshold (for duration events) and a request period (for requestable events).
  10. Click Back if you want to modify any of the settings set in the previous steps or click Finish to start the recording.
    The new flight recording appears in the Progress View.

    Note:

    Expand the node in the JVM Browser to view the recordings that are running. Right-click any of the recordings to dump, dump whole, dump last part, edit, stop, or close the recording. Stopping a profiling recording will still produce a recording file and closing a profiling recording will discard the recording.

Note:

You can set up JMC to automatically start a flight recording if a condition is met using the Triggers tab in the JMX console. For more information, see Triggers Tab.

Save Current Buffers into a Flight Recording

JFR saves the recorded data to files with the .jfr extension. These JFR recordings are binary files for viewing in the JMC. You can manually dump the current contents of the global buffer to a recording file.

  1. Right-click a continuous recording in the JVM Browser and then select Dump.
    The Dump Recording dialog opens.
  2. Click the Browse button and select the path and file name for the recording.
  3. Select to dump the whole recording, only the last part of the recording, or a specified interval of recording.
  4. Click Finish to create the recording dump file.

Analyze a Flight Recording Using JMC

Once the flight recording file opens in the JMC, you can look at a number of different areas like code, memory, threads, locks and I/O and analyze various aspects of runtime behavior of your application.

The recording file is automatically opened in the JMC when a timed recording finishes or when a dump of a running recording is created. You can also open any recording file by double-clicking it or by opening it through the File menu. The flight recording opens in the Automated Analysis Results page. This page helps you to diagnose issues quicker. For example, if you’re tuning the garbage collection, or tracking down memory allocation issues, then you can use the memory view to get a detailed view on individual garbage collection events, allocation sites, garbage collection pauses, and so on. You can visualize the latency profile of your application by looking at I/O and Threads views, and even drill down into a view representing individual events in the recording.

View Automated Analysis Results Page

The Flight Recorder extracts and analyzes the data from the recordings and then displays color-coded report logs on the Automated Analysis Results page.

By default, results with yellow and red scores are displayed to draw your attention to potential problems. If you want to view all results in the report, click the Show OK Results button (a tick mark) on the top-right side of the page. Similarly, to view the results as a table, click the Table button.

The benchmarks are mainly divided into problems related to the following:

Clicking on a heading in the report, for example, Java Application, displays a corresponding page.

Note:

You can select a respective entry in the Outline view to navigate between the pages of the automated analysis.
Analyze the Java Application

Java Application dashboard displays the overall health of the Java application.

Concentrate on the parameters having yellow and red scores. The dashboard provides exact references to the problematic situations. Navigate to the specific page to analyze the data and fix the issue.

Threads

The Threads page provides a snapshot of all the threads that belong to the Java application. It reveals information about an application’s thread activity that can help you diagnose problems and optimize application and JVM performance.

Threads are represented in a table and each row has an associated graph. Graphs can help you to identify the problematic execution patterns. The state of each thread is presented as a Stack Trace, which provides contextual information of where you can instantly view the problem area. For example, you can easily locate the occurrence of a deadlock.

Lock Instances

Lock instances provides further details on threads specifying the lock information, that is, if the thread is trying to take a lock or waiting for a notification on a lock. If a thread has taken any lock, the details are shown in the stack trace.

Memory

One way to detect problems with application performance to is to see how it uses memory during runtime.

In the Memory page, the graph represents heap memory usage of the Java application. Each cycle consists of a Java heap growth phase that represents the period of heap memory allocations, followed by a short drop that represents garbage collection, and then the cycle starts over. The important inference from the graph is that the memory allocations are short-lived as garbage collector pushes down the heap to the start position at each cycle.

Select the Garbage Collection check box to see the garbage collection pause time in the graph. It indicates that the garbage collector stopped the application during the pause time to do its work. Long pause times lead to poor application performance, which needs to be addressed.

Select the Alloc Total check box to see how much memory is allocated per second. You can also view this as a percentage value in the Total Allocation (%) column.

Method Profiling

Method Profiling page enables you to see how often a specific method is run and for how long it takes to run a method. The bottlenecks are determined by identifying the methods that take a lot of time to execute.

As profiling generates a lot of data, it is not turned on by default. Start a new recording and select Profiling - on server in the Event settings drop-down menu. Do a time fixed recording for a short duration. JFR dumps the recording to the file name specified. Open the Method Profiling page in JMC to see the top allocations. Top packages and classes are displayed. Verify the details in the stack trace. Inspect the code to verify if the memory allocation is concentrated on a particular object. JFR points to the particular line number where the problem persists.

JVM Internals

The JVM Internals page provides detailed information about the JVM and its behavior.

One of the most important parameters to observe is Garbage Collections. Garbage collection is a process of deleting unused objects so that the space can be used for allocation of new objects. The Garbage Collections page helps you to better understand the system behavior and garbage collection performance during runtime.

The graphs shows the heap usage as compared to the pause times and how it varies during the specified period. The page also lists all the garbage collection events that occurred during the recording. Observe the longest pause times against the heap. The pause time indicates that garbage collections are taking longer during application processing. It implies that garbage collections are freeing less space on the heap. This situation can lead to memory leaks.

For effective memory management, see the Compilations page, which provides details on code compilation along with duration. In large applications, you may have many compiled methods, and memory can be exhausted, resulting in performance issues.

The TLAB Allocations page provides information about all object allocations. It provides information about Allocations in TLAB and Allocations outside TLAB. The Allocation tab provides three types of visualization, specific to the TLAB events: Allocation by Class (which class instances are being allocated), Allocation by Thread (which threads allocate most of the objects), and Allocation Profile (aggregated stack trace tree of all the events).

A new tab By Top Methods is added to the TLAB Allocations (Thread Local Allocation Buffers) page in addition to the existing By Threads tab to classify the Item Histograms against Top Methods. Both tabs now have Alloc in TLABs (%) and Alloc Outside TLABs (%) columns, which provides estimated allocation size of TLAB as percentage. These updates will make it easier to view relevant areas of allocation pressure.

Environment

The Environment page provides information about the environment in which the recording was made. It helps to understand the CPU usage, memory, and operating system that is being used.

See the Processes page to understand concurrent processes running and the competing CPU usage of these processes. The application performance will be affected if many processes use CPU and other system resources.

Check the Event Browser page to see the statistics of all the event types. It helps you to focus on the bottlenecks and take appropriate action to improve application performance.

You can create Custom Pages using the Event Browser page. Select the required event type from Event Type Tree and click the Create a new page using the select event type button in the top right corner of the page. The custom page is listed as a new event page below the event browser page.

JMC Agent

JMC agent is used to instrument or inject custom JFR events to your running JVM or applications without the need to restart the applications. You don't need to program the JFR instrumentation in the source code of the application to use JMC agent.

Here are some of the features and advantages of JMC agent:

  • You can use XML configuration to define the inject events
  • You can dynamically add events at runtime
  • JVM agent is extremely versatile; you can dynamically load or sideload the agent anytime
  • It causes minimal footprint; you can issue only event-related function calls
  • It is suitable for production use when the source is not available

Using JMC agent plugin, you can manage predefined configurations and also view live information about the resulting transformations.

Click JMC Agent from JVM Browser tab to Start JMC Agent. The prerequisites to start the agent are:

  • Build the Agent JAR: See JMC Agent Readme to to build the agent.jar. In the Start JMC Agent dialog, browse and connect to the Agent JAR file. Use the latest version of the code for building the agent.
  • Define the JFR event in an xml file: You can either create an xml file manually or use JMC Agent Preset Manager, which is the user interface to create the XML file.

Note:

  • If the target application that you need to instrument is running on JDK 11 or later, then run your application with the JVM argument: --add-opens java.base/jdk.internal.misc=ALL-UNNAMED.
  • If the target application that you need to instrument is running on JDK 8, then run your application with the JVM argument -XX:+UnlockCommercialFeatures -XX:+FlightRecorder.

Agent Live Config: Right-click the JMC Agent to open Agent Live Config console. You can use this page to view the Global Configurations applied to the agent, along with the Event List and Event Details.

JMC Agent Preset Manager: Preset Manager will help you to create, edit, and modify the configuration templates. Click Windows and select JMC Agent Preset Manager. You can Add an xml file with only required options, Edit the file, or Import Files to manage the configurations.

JMC Agent Plugin

JMC agent plugin integrates JMC agent features into JDK Mission Control. You can use the plugin to start the JMC agent and connect to a local JVM through the JMX API.

Support for LZ4 Compressed Recordings

JMC 9.1 now supports working with JFR recordings that are compressed using LZ4 format. LZ4 is an extremely fast decoder, which provides compression speed of more than 500 MB/s per core and is scalable with multi-cores CPU.

You can compress the JFR recordings that are in zip or gzipped format. Use the file utility to know the file type. Compress the data (.jfr) file using LZ4 utility or record using compress=true.

Example code to compress JRF recording: 

$ lz4 <un_compressed_filename>.jfr <compressed_filename>.jfr

Output: Compressed 8182601 bytes into 3626010 bytes ==> 44.31%

Websocket Server to Access JFR Stack Trace

When you select JFR events in JMC, you can access their associated stack trace as JSON data through a user-defined port. You can programmatically control the visualization directly in the browser. It allows you to develop or alter visualization using data visualization tools.

The websocket server port is disabled by default. To enable, click Windows, Preferences, JDK Mission Control, and then Flight Recorder. In the Flight Recorder Setting dialog, provide the port number in the Websocket server port field. Set 0 to disable the port.