This document will help you understand how the Real-Time Garbage Collector (RTGC) works, focusing on the most important configuration parameters and use cases.

Technical Documentation: Links to all the Java RTS technical documents

Contents

Introduction

Real-Time Garbage Collection Principles

Sun's Real-Time Garbage Collector

Java RTS Thread Types and Their Priorities

Real-Time Garbage Collection Scheduling

Tuning the RTGC

Normal Mode: Recycling Memory Without Disrupting Any Real-Time Threads

Tuning the Normal Priority of the RTGC
Tuning the Targeted Minimum Free Memory
Tuning the Initial Minimum Free Memory
Tuning the Start Time in Normal Mode for the RTGC

Boosted Mode: Boosting the RTGC Priority

Activating Auto-Tuning for Boosted Mode
Tuning the Boost Priority
Tuning the Boost Time

Deterministic Mode: Ensuring Determinism for Time-Critical Threads

Improving Determinism by Using Multiprocessors

Description of the Analytical Auto-Tuning Mechanism

Description of the "Sliding-Value" Auto-Tuning Mechanism

Summary of the RTGC Scheduling

Finding the Right Configuration Parameters

Additional Note on Pause Times

Using MBeans to Monitor and Configure the RTGC

Command-Line Options

RTGC Basic Options
RTGC Advanced Options
RTGC Expert Options
RTGC Obsolete Options

Examples of Printing RTGC Information

Introduction

Sun Java™ Real-Time System 2.2 (Java RTS) supports two garbage collectors:

• The Real-Time Garbage Collector (RTGC), introduced in Java RTS 2.0
• The non-real-time serial garbage collector of the Java HotSpot™ virtual machine

The other garbage collectors featured in the Java SE version of the HotSpot virtual machine are not supported.

The default garbage collector is the real-time garbage collector. This RTGC can be configured to execute in a deterministic manner.

The RTGC might exhibit lower throughput than non-real-time collectors, particularly on uniprocessors. For applications that are more sensitive to the collector's throughput than to the pause times caused by the collector's execution, the RTGC can be turned off with the -XX:-UseRTGC option. In this case, the non-real-time, serial collector is used, and all threads except NoHeapRealtimeThreads might suffer from pause times caused by garbage collection.

See also the Practical Introduction to Achieving Determinism document for a detailed description of example programs that allow you to begin using Java RTS, including the RTGC, quickly and easily.

The RTGC's activity can be configured with a set of special parameters. To guarantee determinism, some minimum configuration is necessary. Further tuning can be performed by trying different values of various parameters, in various combinations. The parameters are categorized by the level of RTGC configuration that might be needed, depending on application requirements:

• Basic-level configuration is probably sufficient for most applications and is a good way to get started.
• Advanced-level configuration represents the next step for applications that do not achieve determinism with the basic configuration.
• Expert-level configuration might be necessary in certain cases. Use care with this level of tuning, being sure to thoroughly test your application. This level of configuration can also be helpful as a temporary troubleshooting mechanism.

All the parameters are listed in the tables at the end of this document, in the section Command-Line Options.

Real-Time Garbage Collection Principles

Any real-time garbage collector is expected to recycle memory without disrupting the temporal determinism of time-critical tasks. In addition, a real-time GC must recycle memory fast enough so that these tasks are not blocked if the memory is exhausted when they attempt to allocate more.

This means that a real-time GC must be granted enough processor cycles to complete its work on time. On the other hand, granting it too many processor cycles reduces the application's overall throughput. Therefore, if application throughput is a concern, users must properly configure real-time GCs.

Work-based real-time collectors execute when allocations are performed, regardless of the priority of the threads that are executing. Their configuration defines how much time is granted to the GC for each allocation.

Time-based real-time collectors execute at specified times, and at top priority. Their configuration defines when and how often the GC should run. For instance these real-time GCs can be scheduled as periodic threads.

However, most of the available real-time GCs do not allow threads to preempt the GC. The real-time application could be frozen for a considerable amount of time. The better the real-time GC, the smaller this time can get. Unfortunately, this approach does not scale well on multiprocessors.

Another frequent drawback of real-time GCs is that, to ensure determinism, users must analyze the memory behavior of all the threads of their applications. Adding a new low-priority thread that allocates a lot of memory might force the user to change all the configuration parameters of the real-time GC. It might even result in an application for which the GC cannot ensure determinism, requiring the addition of more memory or CPU resources. For instance, this prevents the use of these real-time GCs for a server on which servlets are dynamically added and removed, unless the server has enough memory and CPUs.

Sun's Java RTS and its Real-Time Garbage Collector (RTGC) avoid these two pitfalls.

[Contents]

Sun's Real-Time Garbage Collector

The important point about the RTGC provided with Java RTS is that it is fully concurrent, and thus can be preempted at any time. There is no need to run the RTGC at the highest priority, and there is no stop-the-world phase where all the application's threads are suspended during the RTGC execution. On a multiprocessor, one CPU can be doing some GC work while an application thread is making some progress on another CPU.

Thus, the RTGC offered by Java RTS is very flexible. While other real-time GCs usually either have to be executed as incremental, periodic, high-priority activities or else induce an allocation-time overhead, the Java RTS RTGC can execute according to many different scheduling policies.

In addition, instead of trying to ensure determinism for all the threads of the application, the RTGC considers that the criticality of an application's threads is based on their priority and ensures hard real-time behavior only for real-time threads at the critical level, while trying to offer soft real-time behavior for real-time threads below the critical level. This reduces the total overhead of the RTGC and ensures that the determinism is not impacted by the addition of new low-priority application threads. In addition, this makes the configuration easier because there is no need to study the allocation behavior of an application in its entirety in order to configure the RTGC. Determinism is accomplished by considering only the critical tasks.

Note that the RTGC does not use heap generations. Therefore each RTGC run recycles the entire heap.

By setting only two parameters, namely a memory threshold and a priority boundary, you can ensure that threads running at critical priority will not be disrupted by garbage collection pauses. The big advantage of this approach is that these parameters are independent of the non-critical part of the application. You do not need to reconfigure the RTGC when you add a new non-critical component or when the machine load changes.

The RTGC tries to recycle memory fast enough for the non-critical real-time threads, but without offering any guarantees for them. If the non-critical load increases, the RTGC might fail to recycle memory fast enough for all the threads. However, this will not disrupt the critical threads, as long as the memory threshold is correctly set for the application. Only the non-critical real-time threads will be blocked and temporarily suffer from some jitter, due to memory recycling.

The RTGC implements auto-tuning mechanisms to try to find the best balance between determinism and throughput. The expert-level parameters can be used to control this auto-tuning in order to improve this balance.

[Contents]

Java RTS Thread Types and Their Priorities

The function of the RTGC is based on the criticality of the application threads. Critical threads are those that must execute within well-defined time limits so that their response times are predictable (that is, deterministic), whereas non-critical threads do not have these constraints. Java RTS real-time threads can be critical or non-critical, whereas non-real-time threads (java.lang.Thread instances) are non-critical by definition.

The RTGC starts running at its normal priority, and its priority can be boosted to a higher level if memory falls below a certain level. (This is explained in the next section.) Both of these priority values are configurable with command-line options.

The critical boundary is the priority boundary between critical and non-critical threads. By default, the critical boundary is the same value as the boosted RTGC priority, but this also is configurable.

The thread types and their priorities are summarized as follows:

• No-heap real-time threads. These threads are instances of the Java RTS class javax.realtime.NoHeapRealtimeThread (NHRT), and they exhibit "hard" real-time behavior. These threads are critical by definition and must execute within defined time limits (a few tens of microseconds). By definition, these threads do not allocate memory from the heap and therefore do not depend on garbage collection completion in order to run properly. It is the developer's responsibility to ensure that the priority of the NHRTs is above the RTGC boosted priority so that they are not preempted by the RTGC. (The RTGC boosted priority is specified with the RTGCBoostedPriority parameter.) We recommend running the NHRTs at a priority higher than the priority of the critical real-time threads. Keep in mind that these threads should not use all the CPU power, because that would prevent the RTGC from recycling memory that might be needed for other threads in the application.

• Critical real-time threads. These threads are instances of the Java RTS class javax.realtime.RealtimeThread (RTT), and they exhibit "hard" real-time behavior. By definition, their priority is above the critical boundary and the RTGC boosted priority, and therefore they are not preempted by the RTGC. These threads execute within defined time limits (a few hundreds of microseconds), provided that the RTGC is correctly configured. These threads are not blocked on memory allocation, thanks to the memory reserved for them with the RTGCCriticalReservedBytes parameter. However, these threads must not use all the CPU power, because that would prevent the RTGC from recycling memory, resulting in memory exhaustion that would cause threads to block on allocation.

• Non-critical real-time threads. These threads are also instances of the javax.realtime.RealtimeThread class (RTT), but they exhibit "soft" real-time behavior. By definition, their priority is at or below the critical boundary. The non-critical threads do not necessarily have to execute within defined time limits. Based on their priorities relative to the RTGC priorities, these threads can be considered as high-importance, medium-importance, or low-importance.

  By default, the priority levels of these threads are below the RTGC boosted priority. If the critical boundary is configured to be above the RTGC boosted priority, then some non-critical threads could have a priority above the RTGC boosted priority, thereby allowing them to preempt the RTGC at its boosted priority. These are considered to be high-importance non-critical threads. However, since they are still non-critical threads, they would be blocked on memory allocation when the RTGC goes into its deterministic mode, as explained in the next section.

  The threads with priority below the RTGC boosted priority but above the RTGC normal priority are considered to be medium-importance non-critical threads.

  In addition, the normal priority of the RTGC can be set high enough to allow the RTGC (running at this normal priority) to preempt the low-importance non-critical threads, if necessary.

• Non-real-time threads. These are instances of the java.lang.Thread class (JLT) and are non-critical by definition.

In all cases, it is the programmer's responsibility to set the correct priority to reflect the level of criticality of any thread in the Java RTS application. Note that this is the base priority of the thread. Threads that share locks with more critical threads can be automatically boosted from time to time, via priority inheritance, to a higher priority, and the RTGC will take into consideration the change to this higher priority.

The figure below shows the priority levels for the RTGC and for the different thread types.

Figure 1 Priority Levels for RTGC and Threads

By default, the "normal" priority of the RTGC is above that of non-real-time threads (JLTs), but at the lowest priority for real-time threads. When the Java RTS VM determines that memory is getting low, the RTGC priority is boosted to a higher level, but still lower than that of the critical threads.

Note: The figure also shows the priority level of the NHRT threads, but these threads do not depend on garbage collection activity since they do not allocate memory from the heap.

[Contents]

Real-Time Garbage Collection Scheduling

The most important property for the RTGC is the priority at which it runs. Our real-time garbage collector dynamically changes its priority to try to balance throughput and determinism.

For the parallel (multi-processor) version of the RTGC, the number of threads supporting RTGC activity is also important. See the section Improving the Determinism by Using Multiprocessors.

The RTGC can function in three different modes during a garbage-collection run, based on the remaining free memory available:

• Normal mode. When free memory goes below the startup memory threshold, the RTGC runs in normal mode, and at its normal priority.

• Boosted mode. When free memory goes below the boosted memory threshold, the RTGC priority is boosted to a higher level. This is called boosted mode.

• Deterministic mode. When free memory goes below the critical reserved bytes threshold, the RTGC runs at its boosted priority level. In addition, memory allocation causes non-critical threads (threads with priority below the critical boundary) to block. This is deterministic mode.

The startup and boosted memory thresholds are calculated, and the critical reserved bytes threshold is specified by a parameter, as explained a little further along.

The figure below shows the levels of heap memory, where the top represents a full heap of free memory. As memory is consumed, the first threshold to be reached is the startup memory threshold, then the boosted memory threshold, and finally the critical reserved bytes threshold. If free memory continues to fall, an out-of-memory situation could occur.

Figure 2 Free Memory Levels

The figure below shows how the RTGC is scheduled (with one CPU), based on free memory thresholds. As free heap memory goes down, the priority of the RTGC goes up.

Figure 3 RTGC Scheduling on One CPU

The RTGC starts its next run at its initial (normal) priority when free memory goes below the startup memory threshold. Java RTS uses its auto-tuning mechanisms to calculate this threshold. Lower-priority threads are preempted by higher-priority threads. Since the RTGC is running at a lower priority, it can be preempted by non-critical real-time threads. This is called the normal mode.

Java RTS also calculates a boosted memory threshold, either with auto-tuning (disabled by default) or based on a parameter. When memory reaches this threshold, the RTGC is boosted to a higher priority (specified with the RTGCBoostedPriority parameter), meaning that fewer threads can preempt the RTGC. This is called the boosted mode.

Not shown in the figure above is a further implication of the fine tuning of the RTGC. If the boosted RTGC value is equal to the critical boundary value, then only critical threads can preempt the RTGC. However, the critical boundary could be configured to be higher than the boosted RTGC priority. In this case, threads with priority below the critical boundary but higher than the RTGC boosted priority, despite being non-critical threads, could possibly preempt the RTGC.

If the free memory falls below the critical reserved bytes threshold (specified by the RTGCCriticalReservedBytes parameter), the non-critical threads are blocked on allocation, waiting for the RTGC to recycle memory. This guarantees that the critical RT threads, and only the critical RT threads, will be able to allocate memory from the reserved amount. This mode is called the deterministic mode, as it assures determinism for the critical threads. If the free memory remains below RTGCCriticalReservedBytes, the non-critical threads will eventually fail with an OutOfMemoryError.

The figure also shows the priority level of the NHRT threads, but these threads are not involved in garbage collection since they do not allocate memory from the heap.

[Contents]

Tuning the RTGC

The principle behind the RTGC is the balance between determinism and throughput. With a finite amount of system resources (CPU time and memory in particular), Java RTS must ensure determinism for the critical threads, while ensuring that the other threads also are able to execute in a timely manner. Therefore, the RTGC must recycle enough memory for the allocation requests, while not consuming all the CPU cycles. This delicate balance is configured with several parameters. Since the RTGC continuously auto-tunes its own operation, only two of these parameters need to concern the basic user:

• You must configure the RTGCCriticalReservedBytes parameter in order to guarantee determinism for hard real-time. The default value for this parameter is zero, which would reserve no memory at all for the critical threads when the RTGC is boosted to its higher priority. This is the only parameter that you are required to configure for hard real-time.

• The RTGCCriticalBoundary parameter is set to a default value, but you can configure this value in relation to the priority values for the other threads in your application.

The remaining parameters should be used in advanced-level or expert-level tuning.

Note: The default behavior of Java RTS is to allow the RTGC to use all the CPU power, because determinism is our primary goal. However, with a bad configuration or with applications that have a large percentage of reachable objects, the RTGC might run continuously to try to ensure determinism. Therefore, by default Java RTS forces a wait period between two consecutive RTGC runs in order to help prevent the system from being blocked due to continuous garbage collection. The expert-level RTGCWaitDuration parameter can be used to configure this wait period.

All the parameters are listed in the tables at the end of this document in the section Command-Line Options.

Normal Mode: Recycling Memory Without Disrupting Any Real-Time Threads

Java RTS uses one of two auto-tuning mechanisms to determine when the RTGC starts its next run. Some advanced-level configuration might be needed, for example, to tune the value of the RTGCNormalPriority parameter. The rest of the parameters are used in expert-level tuning.

In the Real-Time Specification for Java, the impact of the GC on real-time threads is implementation-dependent. With Java RTS, the threads running at the highest priorities can completely avoid garbage collection pauses. However, the RTGC must start soon enough to ensure that it completes before memory is exhausted. This works as long as the allocation behavior is relatively stable, because the RTGC must be started soon enough to handle the worst case allocation.

In normal mode, the RTGC runs at a lower priority than the real-time threads. Unfortunately, this does not scale well to larger applications. As a general rule, the more real-time threads are running (and allocating memory), the longer an RTGC cycle lasts. The amount of memory allocated during a cycle can quickly increase. To ensure determinism, we must take into account the worst case allocation behavior, which could possibly cause the RTGC to run continuously. And even this might not be sufficient to guarantee that memory would be recycled fast enough.

However, real-world applications normally contain a mix of critical and non-critical tasks. For some of the non-critical tasks, timing is important but not critical. You might be willing to miss a few deadlines when there is an allocation burst. The gain is that this allows the RTGC to start much later, improving the global throughput.

With Java RTS, we can configure the behavior of the non-critical threads independently from the behavior of critical ones. We consider that the normal behavior for real-time threads is to run at a higher priority than the RTGC, while the JLTs run at a lower priority than the RTGC.

The RTGC starts running at RTGCNormalPriority (which defaults to the minimal real-time priority). The startup memory threshold determines when the RTGC starts. The idea is to start it early enough so that it completes its cycle before reaching the boosted memory threshold, which would increase the RTGC priority to its boosted level, creating jitter for the non-critical real-time threads. The auto-tuning takes into account the allocation performed during the last RTGC cycles to try to start the RTGC just on time, thus maximizing the throughput while avoiding pause times, assuming that the allocation behavior is stable.

The value of the startup memory threshold is determined as the maximum of the following values (which are explained further on in this section):

• Memory threshold value calculated as a result of one of the tuning methods.
• Initial (decreased) minimum free memory.
• Minimum free memory as specified by NormalMinFreeBytes.

As mentioned earlier, Java RTS uses one of two auto-tuning mechanisms to determine the startup memory threshold:

• Analytical auto-tuning. This mechanism is new in Java RTS 2.2 and activated by default. It is described in detail in Description of the Analytical Auto-Tuning Mechanism.

• "Sliding-value" auto-tuning. This is the auto-tuning mechanism that was used before Java RTS 2.2, and it is deactivated by default, in favor of the new analytical mechanism. You can activate this sliding-value mechanism by deactivating the default analytical auto-tuning, by specifying -XX:-UseAnalyticalPolicy. See Description of the Sliding-Value Auto-Tuning Mechanism for a detailed description.

The expert-level tuning provides opportunities to fine-tune the RTGC functionality for normal mode.

Tuning the Normal Priority of the RTGC

By default, all real-time threads run at a priority higher than the initial (normal) priority of the RTGC, which is represented by the RTGCNormalPriority parameter. Increasing this value allows the RTGC to complete faster because there would be fewer threads at a priority higher than the RTGC that would preempt the RTGC and allocate memory during a RTGC cycle. This improves determinism for the higher-priority threads, as well as global throughput. Make sure that RTGCNormalPriority is set such that only the RealtimeThreads sensitive to pause times run at a higher priority than the RTGC.

Tuning the Targeted Minimum Free Memory

The overall goal of the auto-tuning is to start the RTGC soon enough so that free memory never falls below a specified level. This level is expressed as a percentage of the heap, and is indicated by the value of the parameter RTGCTargetedMinFreePercent. The default value is 5% of the heap. In advanced-level tuning, you can change this value in order to specify a different targeted percentage of the heap. For example, -XX:RTGCTargetedMinFreePercent=20 means that the target is 20% of the heap.

Tuning the Initial Minimum Free Memory

When the Java RTS VM starts up, the RTGC auto-tuning mechanism needs to learn about the behavior of the application in terms of memory allocation. At the beginning of this learning period, Java RTS overestimates the startup memory threshold by assigning a rather large default value to the initial minimum free memory level. The RTGC will start if memory falls below this level. The initial minimum free memory is expressed as a percentage of the heap and is indicated by the value of the parameter RTGCInitialMinFreePercent. At each GC cycle, the value of this reduced by a percentage indicated by the value of the parameter RTGCInitialDecreaseFactor. In this way, for each GC cycle, this initial minimum free memory level will be reduced until it is no longer a factor in the calculation. Ideally, this should occur at the end of the learning period of the RTGC.

The default value of RTGCInitialMinFreePercent is 30 and the default value of RTGCInitialDecreaseFactor is 50. This means that the, for the first GC cycle, the initial minimum free memory limit is 30% of the heap; for the second cycle, the limit is reduced by 50% to be 15% of the heap; for the third GC cycle, the limit is reduced by 50% to be 7.5% of the heap; and so on, until the limit is negligible.

In order to avoid exhaustion of the memory during the learning period during the early GC cycles, you can increase the default value of the initial free memory limit. You can also change the value of the decrease factor so that the limit decreases faster or slower, depending on the behavior of the application.

Both the analytical and the sliding-value auto-tuning mechanisms use these parameters.

Tuning the Start Time in Normal Mode for the RTGC

If you use the "sliding-value" auto-tuning mechanism for normal mode (-XX:-UseAnalyticalPolicy), the startup memory threshold is calculated using a set of parameters that can be fine-tuned in expert-level tuning:

• NormalMinFreeBytes
• NormalSlideFactor
• NormalSafetyMargin

See the section Description of the "Sliding-Value" Auto-Tuning Mechanism below for a detailed explanation of this mechanism.

Boosted Mode: Boosting the RTGC Priority

The boosted memory threshold determines when the RTGC priority is boosted to a higher level. When the RTGC priority is boosted, non-critical threads can be preempted by RTGC threads, but the RTGC can be preempted by critical threads. Therefore this can cause non-critical threads to pause for a long time, unless in a multiprocessor environment. See the section Improving the Determinism by Using Multiprocessors.

The expert-level tuning provides opportunities to fine-tune the RTGC functionality for boosted mode.

Activating Auto-Tuning for Boosted Mode

As of Java RTS 2.2, auto-tuning is off by default for boosted mode, and the boosted memory threshold is determined by the value of the parameter BoostedMinFreeBytes. You can turn on auto-tuning for boosted mode by specifying -XX:+RTGCTuneBoostedMode. The "sliding-value" auto-tuning mechanism will be used, as described in Description of the "Sliding-Value" Auto-Tuning Mechanism.

Tuning the Boost Priority

You can change the default value of the boosted priority with the expert-level RTGCBoostedPriority parameter.

Tuning the Boost Time

As mentioned earlier, if auto-tuning is off for boosted mode, the boosted memory threshold is determined by the value of the parameter BoostedMinFreeBytes. If auto-tuning is on for boosted mode, then the auto-tuning mechanism uses the following parameters to calculate this threshold:

• BoostedMinFreeBytes
• BoostedSlideFactor
• BoostedSafetyMargin

See the section Description of the "Sliding-Value" Auto-Tuning Mechanism below for a detailed explanation of this mechanism.

Deterministic Mode: Ensuring Determinism for Time-Critical Threads

The real-time threads that run at a priority higher than the critical boundary are called the critical real-time threads. The RTGC must ensure that their worst case pause time is very low (at worst a few hundreds of microseconds).

To guarantee this determinism, you must specify the RTGCCriticalReservedBytes parameter. You can tune the critical boundary with the RTGCCriticalBoundary parameter.

When the free memory goes below the limit specified by RTGCCriticalReservedBytes, Java RTS blocks the non-critical threads from allocating memory to prevent them from disrupting the critical threads. If the free memory remains below this threashold after a few RTGC runs, then OutOfMemoryError conditions might start being thrown for these non-critical threads.

Note that the critical threads continue running at a priority higher than the RTGC and can preempt it at any time.

If RTGCCriticalReservedBytes is too low, a critical thread might block when the memory is full, waiting for the RTGC to free some memory. If RTGCCriticalReservedBytes is too high, the RTGC will run more often, preventing the lower priority threads from running. Hence, this reduces their global throughput.

[Contents]

Improving the Determinism by Using Multiprocessors

The RTGC is fully concurrent, that is, application threads can run concurrently with the RTGC. Therefore, we can improve the determinism of non-critical threads on multiprocessors by specifying how many worker threads the RTGC can use. This is both simpler and safer than depending on parameter tuning alone.

The drawback of parameter tuning alone is that when the RTGC parameters are underestimated, all non-time-critical threads might be suspended from time to time. If they are overestimated, the JLTs will be suspended very often. At worst, they will not make any progress because the RTGC could consume all the CPU not used by the real-time threads.

You can specify the initial (normal) and boosted number of parallel worker threads for the RTGC with the RTGCNormalWorkers and RTGCBoostedWorkers options, respectively. Thus, when the RTGC runs, there should still be some CPU cycles not used by the RTGC and the critical threads. (It is assumed that critical threads use only a small part of the CPU power.) Hence, non-critical real-time threads should still be able to make some progress. Even JLTs could get some CPU cycles.

Note: The values of the RTGCNormalWorkers and RTGCBoostedWorkers options specify the maximum number of RTGC worker threads that can execute in parallel (in normal mode and boosted mode, respectively) performing CPU-intensive garbage-collection related tasks. These maximum values roughly correspond to the worst case CPU usage of the RTGC in these two modes. There are other RTGC-related physical threads (some on them using the same worker infrastruture), but their impact on CPU consumption is negligible and is not taken into account for the configuration parameters.

When the real-time load is low, RTGCNormalWorkers threads executing the RTGC at RTGCNormalPriority will be enough to cope with allocation performed on the remaining processors. However, if the allocation rate increases or if more real-time threads are running, the RTGC at RTGCNormalPriority, even if running continuously, might not recycle memory fast enough. When this happens, the RTGC priority is boosted, and the number of RTGC threads is increased to RTGCBoostedWorkers.

On multiprocessors, as on uniprocessors, the VM performs auto-tuning to try to maximize throughput. It will first try to complete the RTGC on time by using RTGCNormalWorkers threads running at RTGCNormalPriority. Even if the RTGC is boosted to RTGCBoostedPriority, the non-critical threads still get some CPU power.

Expert-level tuning should focus on the boosted memory threshold, which defines when the RTGC threads start running at RTGCBoostedPriority. This is done through BoostedMinFreeBytes. If boosted-mode tuning is active (-XX:+RTGCTuneBoostedMode), then BoostedSlideFactor, BoostedSafetyMargin are used. (See the section Description of the Auto-Tuning Mechanism.) The big advantage to this configuration is that overestimating the threshold (for example, small slide factors and big safety margins) is not dangerous. Even if this configuration makes the RTGC run continuously on RTGCBoostedWorkers processors, there will always be a few processors available for the lower priority threads. The impact on determinism depends on the ratio between RTGCBoostedWorkers and the total number of processors.

The following example shows a simple configuration to easily achieve determinism (though without optimizing throughput) by limiting the number of processors the RTGC can use at any one time and forcing the RTGC to run continuously. Since the RTGC usually needs about 25% of the CPU power in order to guarantee determinism, we allow the RTGC to use two processors (parallel worker threads), assuming an 8-processor system. We also force garbage collection to run almost continuously by specifying the NormalMinFreeBytes option to be equal to the total heap size. (As explained in the section Description of the Auto-Tuning Mechanism, the NormalMinFreeBytes option specifies the amount of free memory below which the RTGC starts its next cycle.)

-Xms1G -Xmx1G
-XX:NormalMinFreeBytes=1G
-XX:RTGCNormalWorkers=2

When making estimations, consider the following:

• If you underestimate the boosted number of worker threads, that is, if RTGCBoostedWorkers is too low, the RTGC might not recycle memory fast enough, even if it runs continuously at RTGCBoostedPriority. This means that the memory will often decrease down to RTGCCriticalReservedBytes and that the non-critical threads will block.

• If you overestimate the boosted number of worker threads, you have to be more careful with the BoostedSlideFactor and BoostedSafetyMargin because the RTGC could be granted too many CPU cycles, reducing the global throughput.

• If you assign values to the boosting parameters that are too low, this will cause the RTGC to be boosted too late and the remaining processors will become idle when the memory reaches RTGCCriticalReservedBytes. Note that this is a temporary problem that happens only when the allocation behavior changes and it would only slightly impact the overall throughput. Setting RTGCBoostedWorkers to the total number of processors prevents processors from sitting idle in deterministic mode, but causes the boosted mode to be more disruptive to non-critical threads, as stated above.

As a summary, this expert-level tuning is a parameterized way to offer more determinism for the real-time non-critical threads. In addition, it does not block the machine, avoiding the case where the RTGC runs continuously on all the processors, at very high priority.

[Contents]

Description of the Analytical Auto-Tuning Mechanism

This section describes the analytical auto-tuning mechanism that Java RTS uses by default to calculate the startup memory threshold, which represents the optimal conditions to start the RTGC in normal mode. If you turn off this auto-tuning mechanism, by specifying -XX:-UseAnalyticalPolicy, then Java RTS will use the "sliding-value" auto-tuning policy for normal mode, as described in the next section.

The analytical auto-tuner is more precise than the sliding-value auto-tuner, because it is based on average drops and standard deviations. It also monitors the memory consumption to detect bursts. If there is a burst, it will try to evaluate a worst case memory drop for that bursty rate and start the RTGC accordingly. It will also memorize the real drop at that rate to gradually learn how the application behaves. This should ensure that the memory is less likely to be exhausted at any time, for a small throughput hit. This auto-tuning mechanism should prevent the RTGC from staring too early when the allocation behavior changes.

[Contents]

Description of the "Sliding-Value" Auto-Tuning Mechanism

This section describes in detail the "sliding-value" auto-tuning mechanism. The same basic mechanism is used for both normal mode and boosted mode:

• If the analytical auto-tuning is turned off for normal mode (with the -XX:-UseAnalyticalPolicy parameter), then Java RTS uses this sliding-value auto-tuning mechanism for normal mode to determine the optimal conditions to start the RTGC. This condition is referred to as the startup memory threshold.

• If the -XX:+RTGCTuneBoostedMode parameter is set, then Java RTS uses this sliding-value auto-tuning mechanism for boosted mode to determine the optimal conditions to boost the RTGC priority. This condition is referred to as the boosted memory threshold.

Note: The sliding-value mechanism performs the same calculations for both normal mode or boosted mode, but with different parameters; the parameter names begin with Normal and Boosted, respectively. Therefore, we drop Normal and Boosted from the parameter names used in the calculations below.

Expert-level tuning might be used to tune these threshold values, by changing the parameter values.

The VM does not try to use the worst case allocation behavior to ensure the RTGC will complete on time, because this would run the RTGC too often after the first period of allocation bursts. Hence, the auto-tuning mechanism uses a depreciation mechanism to compute a "sliding value" at the end of each GC cycle. This sliding value is the higher of its depreciated previous value and the memory that was allocated during the current GC cycle. A safety margin is then applied to this sliding value to compute the next memory threshold. The RTGC will start (or be boosted to its higher priority) when the free memory goes below that threshold.

The RTGC uses the amount of memory that was allocated previously in order to estimate the next threshold. It is expected that this value will "slide" to a stable amount. (This is why we call it a "sliding value.")

The following formulas are used to calculate the startup memory threshold and the boosted memory threshold:

SlidingValuenew = MAX(current, (SlidingValueprevious * (100 - SlideFactor) / 100))

Threshold = MAX(MinFreeBytes, (SlidingValuenew * (100 + SafetyMargin) / 100))

This calculation is performed at the end of each RTGC cycle. In the formulas, the variables have the following meanings:

• current represents the amount of memory allocated during the current GC cycle.

• SlidingValue represents a prediction of the amount of memory that will be allocated in the next GC cycle.

• SlideFactor is an integer value converted to a percentage by which we depreciate the previous sliding value, that is, the sliding value calculated during the previous GC cycle.

• SafetyMargin is an integer value converted to a percentage which we add to the current sliding value, in order to reduce unnecessary boosts for small variations in the memory level. For example, in the calculation for the startup memory threshold, if the value of NormalSafetyMargin is not big enough, the RTGC can start too late and have to be boosted later, causing jitter for the soft real-time threads.

• MinFreeBytes is the user-estimated amount of free memory below which the RTGC should always start its next run (or be boosted). (This is the parameter NormalMinFreeBytes or BoostedMinFreeBytes.) This user-estimated value is used in calculating the threshold.

• Threshold is the final value for the startup memory threshold or the boosted memory threshold, that is, the free memory threshold below which the RTGC will start its next run or be boosted to its higher priority.

The configuration of SlideFactor, SafetyMargin, and MinFreeBytes is at the expert level.

The first formula above uses the sliding value calculated during the previous GC cycle. For the initial cycle of the RTGC, the sliding value is zero, as there was no previous GC cycle. With the slide factor applied, the sliding value is still zero. Therefore, the amount of memory allocated during this first cycle becomes the new sliding value.

For subsequent GC cycles, the sliding value calculated during the previous GC cycle is depreciated by the slide factor percentage. This result is compared with the amount of memory allocated during the current GC cycle, and the higher of the two becomes the new sliding value, which represents the amount of memory we predict will be allocated during the next cycle.

Then, in the second formula above, this new sliding value, with the safety margin percentage applied, is compared with the user-estimated free memory threshold (MinFreeBytes), and the higher of the two becomes the memory threshold that will actually be used to determine startup or priority boosting of the RTGC.

The slide factor represents the percentage of memory allocated during the previous cycles that will not be considered in the calculation of the prediction of the allocation needs during the next garbage-collection cycle. In other words, this can be considered as the speed with which allocation bursts are forgotten.

As an example, let's say that you have set the SlideFactor parameter to 40. In this case, at the end of each GC cycle, the sliding value is reduced by 40% before being compared to the memory allocated during the cycle. If a large amount of memory is allocated during the cycle, only 60% of that amount will be used to determine when to start the next cycle or boost the priority. If the slide factor is increased to 80%, then only 20% of the allocation is used in the calculation, and the effect of the allocation burst is more quickly "forgotten." Therefore, by increasing the SlideFactor parameter, you ensure that the RTGC will not continue consuming a lot of CPU cycles after a period of allocation bursts.

As mentioned above, the sliding value is initially zero. It also goes back to zero if no allocations are performed during a few GC cycles.

Note that, if the sliding value is zero for the calculation of the memory thresholds (normal or boosted), the RTGC could start or be boosted too late and cause jitter for the first few cycles (the time needed for the application to reach a steady state). If you are concerned with this "initial learning phase," you can specify a MinFreeBytes threshold, which should roughly correspond to the average behavior of the application.

The SafetyMargin parameter is a percentage of the calculated sliding value that is added to the value before we compare it to the free memory threshold. In this way, the RTGC ignores small variations in the level of memory allocations. This represents an estimation of the variation of allocation rate that can be supported without creating unacceptable jitter.

Significance for Normal Mode

Decreasing NormalSafetyMargin or NormalMinFreeBytes improves throughput by starting the RTGC later. Unfortunately, the RTGC might start too late, particularly at application startup or when the allocation rate increases. In this case the RTGC priority would be boosted, allowing the RTGC to preempt the non-critical real-time threads. Increasing NormalSafetyMargin or NormalMinFreeBytes avoids this situation.

Note that if NormalSafetyMargin is too big, the RTGC might run continuously, and threads at a priority lower than RTGCNormalPriority might be prevented from running. This might also happen if the real-time threads often have allocation bursts and the NormalSlideFactor is too low.

Increasing NormalSlideFactor allows the RTGC to forget more quickly the allocation bursts, improving the throughput but increasing the likelihood of jitter during the next allocation burst.

For boosted mode, these tuning options are mainly useful in a multiprocessing enviroment. See the section Improving the Determinism by Using Multiprocessors.

[Contents]

Summary of the RTGC Scheduling

The VM auto-tunes the memory thresholds to try to start or boost the RTGC as late as possible (to maximize throughput) while trying not to have to increase its priority or block the non-critical threads (which impacts their determinism).

The RTGC can go through three different phases.

• By monitoring how much memory is allocated when the RTGC is running at RTGCNormalPriority, the VM tries to start it on time to complete at that priority using the number of processors specified by RTGCNormalWorkers. By default, the RTGC uses all these processors at priority 11, blocking the JLTs.

• If it fails to complete on time, the RTGC is boosted to RTGCBoostedPriority, and the non-critical threads suffer from jitter. If RTGCBoostedWorkers is set to a value lower than the total number of processors, then the highest priority non-critical threads might be able to use the remaining processors to make some progress. Otherwise, the RTGC tries to use all the processors. The VM tries to boost the RTGC so that it should complete before reaching RTGCCriticalReservedBytes. This is also done by monitoring how many bytes are allocated while running.

• If the RTGC does not complete fast enough, the RTGCCriticalReservedBytes memory threshold is hit. In that case, the priority of the RTGC is not modified but the RTGC prevents the non-critical threads from allocating additional bytes. If memory is completely exhausted, a critical thread will block when it tries to allocate memory.
  You must ensure that this amount of memory will be sufficient for the critical threads.

[Contents]

Finding the Right Configuration Parameters

As mentioned above, in order to guarantee determinism, you must configure the RTGCCriticalReservedBytes parameter. This is the only parameter that you are required to configure for hard real-time.

You might also want to configure the RTGCCriticalBoundary parameter. For the rest, Java RTS includes several auto-tuning mechanisms to try to lighten the configuration burden, as detailed in Description of the Auto-Tuning Mechanism.

Java RTS provides a garbage collector API and a few MBeans to gather additional information. See Using MBeans to Monitor and Configure the RTGC.

However, the simplest way to figure what is happening is to use the -XX:+PrintGC command-line option (and optionally -Xloggc:<filename> to redirect the output to a file). After each GC cycle, the RTGC prints information that was gathered during the cycle. For more advanced tuning or debugging, the -XX:+RTGCPrintStatistics command-line option provides additional detailed information. See Examples of Printing RTGC Information for sample output from these options.

With respect to critical threads, the most important number to look at in the PrintGC output is the "worst" non-fragmented (that is, allocatable) memory, which represents the lowest level of allocatable memory since the VM startup. If this value falls to 0, this means that a critical thread was blocked during an allocation. This is a clear indication that you must increase RTGCCriticalReservedBytes. On the other hand, if RTGCCriticalReservedBytes is too high, the RTGC will run too often at RTGCBoostedPriority and block non-critical threads. This decreases the throughput. At worst, it will run continuously and prevent the non-critical threads from making any progress. This might even block your system. However, you will still be able to see the RTGC messages.

The PrintGC output also provides the "minimum" level of non-fragmented memory during this last cycle. You can compare this minimum value with the overall (worst) minimum.

For soft real-time threads, that is, non-critical real-time threads, the most important information is whether or not the RTGC "completed without boosting." If not, all the threads below the RTGC boosted priority may have suffered from jitter during the GC cycle. This jitter is not necessarily an error. It results from the configured balance between throughput and determinism. The default auto-tuning tries to avoid this boosting. However, on multiprocessors, we recommend configuring RTGCBoostedWorkers and BoostedMinFreeBytes in order to optimize the behavior when the auto-tuning fails. This can allow high-priority non-critical threads to continue running on the remaining CPUs even in boosted mode.

Note that JLTs can also continue to run concurrently with the RTGC. This is especially interesting if RTGCNormalWorkers is tuned to a value high enough so that the RTGC is rarely boosted but low enough to leave several CPUs for the JLTs when the RTGC runs in normal mode.

With expert-level tuning, you can try to improve the determinism by ensuring that the worst case is taken into account for a longer time (by reducing the slide factors in the auto-tuning mechanism) and/or that the RTGC can cope with quicker changes in the allocation rates (by increasing the safety margins or the minimum threshold). To optimize the throughput, you can try to tune the RTGC so that the minimum free memory is nearly equal to the memory threshold that would have caused the RTGC to enter the next mode. If RTGCBoostedWorkers is set, you should also determine whether the RTGCCriticalReservedBytes boundary was reached by checking the RTGC output; if the boundary was reached, the RTGC prints the number of blocked threads.

In addition, if a non-critical thread blocked on allocation, the "worst" non-fragmented memory will be no more than the value of RTGCCriticalReservedBytes.

[Contents]

Additional Note on Pause Times

The RTGC is truly concurrent. The only time it prevents a thread from running is when it has to look at this particular thread's Java stack, where the local variables and operands are stored. Hence, the main potential source of pause time for a given thread is caused by the scanning of its stack. Since a thread is not impacted by the scanning of the other thread stacks, the pause time for a thread is smaller than with non-concurrent GCs.

To reduce this pause time even further, threads should execute in compiled mode. See the Java RTS Compilation Guide for further details on how to compile the methods executed by time-critical code. In general, the pause time primarily depends on the stack's depth. Pause times are very small for threads that do not extensively perform recursive method calls.

Do not forget that there are a lot of other jitter sources that might look like RTGC pause times. Java RTS provides solutions to avoid them. For instance, the Java RTS Compilation Guide covers the issues related to late class loading and initialization. Remember also that only real-time priorities provide real-time guarantees. For priorities lower than real-time, the threads are scheduled with a time-sharing policy to reduce the risks of starvation in the non real-time part of the application. Hence, a thread at a priority lower than real-time can be preempted at any time to let lower priority threads make some progress.

[Contents]

Using MBeans to Monitor and Configure the RTGC

Java RTS provides a garbage collector API, useable from the application code, which allows you to dynamically change the RTGC parameters either locally from your Java application or remotely through management beans.

For additional information, look in the Javadoc™ for the FullyConcurrentGarbageCollector class. The MBeans are registered in the realtime.rtgc namespace and can be viewed with, for example, the JConsole tool. We have also extended the LastGCInfo attribute of the garbage collector MBean.

To see a description of the attributes and operations of these MBeans, refer to the corresponding Javadocs:

• CriticalThreadsMXBean to monitor and change the behavior of critical threads.
• ImportantThreadsMXBean to monitor and change the behavior of real-time non-critical threads.
• RealtimeMemoryManagerMXBean to monitor the CPU overhead and get detailed information about each GC cycle.

[Contents]

Command-Line Options

All these parameters are intended for a production environment.

These parameters are also listed in the Java RTS Options document.

Note that the RTGC checks the logical consistency of the parameter values and adjusts them accordingly. For example, if you specify a value for RTGCBoostedPriority that is higher than the value for RTGCCriticalBoundary, then the RTGC will set the value of RTGCBoostedPriority equal to that of RTGCCriticalBoundary, with an accompanying message.

Caution: The "-XX" options are not part of the official Java API and can vary from one release to the other.

RTGC Basic Options

Parameter	Default	Env	Description
RTGCCriticalReservedBytes	0	prod	Free memory threshold, in bytes, under which only critical threads can allocate memory. To guarantee determinism, configure this parameter. See Tuning the RTGC.
RTGCCriticalBoundary	OS-dependent	prod	The priority boundary between critical and non-critical threads. Threads with priority above this boundary are considered to be critical threads, while threads with priority at this boundary or below are non-critical threads. The value defaults to the middle of the real-time range, which is greater than the default priority for real-time threads. This default value is OS-dependent, as explained in "Real-Time Priorities" in the Technical Information document.
PrintGC	false	prod	Print information gathered whenever the GC runs. See Examples of Printing RTGC Information for sample output.
UseRTGC	true	prod	Use the new real-time garbage collector instead of the non-real-time, serial collector.

RTGC Advanced Options

Parameter	Default	Env	Description
RTGCNormalPriority	11	prod	Normal priority for the RTGC. It defaults to 11 to try to offer soft real-time behavior for all the real-time threads.
RTGCNormalWorkers	(number of CPUs) / 2	prod	Initial number of parallel RTGC worker threads.
RTGCBoostedWorkers	(number of CPUs) / 2	prod	Boosted number of parallel RTGC worker threads.

The number of CPUs available to the RTGC is the number of CPUs available to the VM that have not been specifically assigned to running RealtimeThread or NoHeapRealtimeThread instances. The minimum possible value is one.

RTGC Expert Options

Parameter	Default	Env	Description
RTGCPrintStatistics	false	prod	Print additional information gathered whenever the RTGC runs, for advanced tuning or debugging. See Examples of Printing RTGC Information for sample output.
RTGCWaitDuration	10	prod	Number of milliseconds between two consecutive runs of the RTGC, unless memory is exhausted or an explicit GC is requested.
UseAnalyticalPolicy	true	prod	When true, use analytical auto-tuning (using statistics) for normal mode. When false, use "sliding value" auto-tuning for normal mode.
RTGCTuneBoostedMode	false	prod	When true, turn on auto-tuning ("sliding value") for boosted mode.
RTGCBoostedPriority	RTGCCriticalBoundary	prod	Priority at which the RTGC activity should execute in order to guarantee determinism. When free memory falls below the boosted memory threshold, the RTGC priority is boosted to this level. If this parameter is not specified, then its default value is the value of the RTGCCriticalBoundary parameter.

The following expert-level options are used by both the analytical and the "sliding value" auto-tuning mechanisms for normal mode.

Parameter	Default	Env	Description
RTGCTargetedMinFreePercent	5	prod	Minimum free memory level targeted by auto-tuning to start the RTGC in normal mode. The value is expressed as a percentage of the heap.
RTGCInitialMinFreePercent	30	prod	Initial free memory level used by auto-tuning to start the RTGC in normal mode. The value is expressed as a percentage of the heap.
RTGCInitialDecreaseFactor	50	prod	In auto-tuning, the percentage by which to decrease the value of RTGCInitialMinFreePercent at the end of each GC cycle.
NormalMinFreeBytes	0	prod	Minimum (estimated) free heap memory level, in bytes, below which the RTGC should start its next cycle, at priority RTGCNormalPriority.

The following expert-level options are used by the "sliding value" auto-tuning mechanism for normal mode.

Parameter	Default	Env	Description
NormalSlideFactor	20	prod	Speed at which allocation bursts are forgotten in normal mode. An integer from 0 to 100, which will be converted to a percentage.
NormalSafetyMargin	10	prod	Variation of the allocation rate that is accepted without creating jitter in normal mode. An integer from 0 to 100, which will be converted to a percentage.

The following expert-level options are used by the "sliding value" auto-tuning mechanism for boosted mode.

Parameter	Default	Env	Description
BoostedMinFreeBytes	0	prod	Minimum (estimated) free heap memory level, in bytes, below which the RTGC should be boosted to priority RTGCBoostedPriority.
BoostedSlideFactor	10	prod	Speed at which allocation bursts are forgotten in boosted mode. An integer from 0 to 100, which will be converted to a percentage.
BoostedSafetyMargin	20	prod	Variation of the allocation rate that is accepted without creating jitter in boosted mode. An integer from 0 to 100, which will be converted to a percentage.

RTGC Obsolete Options

The following options are obsolete and should not be used. They will be completely removed in a future version of Java RTS.

Parameter	Explanation
RTGCCriticalPriority (obsolete)	This parameter was previously used to indicate the priority at which the RTGC should execute in order to guarantee determinism, that is, it represented the division in priority levels between critical threads and non-critical threads. This parameter has been replaced by two new parameters: RTGCCriticalBoundary and RTGCBoostedPriority. These new parameters allow finer RTGC tuning. For this release, if this obsolete parameter is set, and the two new parameters are not set, then the two new parameters will be set to the value of this obsolete parameter.
RTGCMaxCPUs (obsolete)	This parameter was previously used to indicate the maximum number of parallel RTGC threads that could be used. This parameter has been replaced by two new parameters: RTGCNormalWorkers and RTGCBoostedWorkers. These new parameters allow finer RTGC tuning. For this release, if this obsolete parameter is set, and the two new parameters are not set, then the two new parameters will be set to the value of this obsolete parameter.

[Contents]

Examples of Printing RTGC Information

This section first provides examples of the output from the XX:+PrintGC and the XX:+RTGCPrintStatistics command-line options. Then it presents the output from a bad (non-determinstic) configuration, with an analysis. Finally, it shows the output that results from correcting the configuration.

Command-Line Output

When the XX:+PrintGC command-line option is specified, Java RTS prints information gathered during the last GC cycle, as shown below:

[GC 2047K->1925K(2048K, non-fragmented: current 122K / min 1K / worst 0K, 
blocked threads: max 1 / still blocked 0 requesting 0 bytes, 
dark matter: 75K in 177 blocks smaller than 2048 bytes) 
<boosted during 62887 microseconds> {CPU load: 1.05 recycling / 0.04 since last GC}
, 0.0635765 secs] 

With the XX:+RTGCPrintStatistics command-line option, Java RTS prints additional information about the GC cycle, as shown in the following example:

[GC 2047K->1925K(2048K, non-fragmented: current 122K / min 1K / worst 0K, 
blocked threads: max 1 / still blocked 0 requesting 0 bytes, 
dark matter: 75K in 177 blocks smaller than 2048 bytes) 
<boosted during 62887 microseconds> {CPU load: 1.05 recycling / 0.04 since last GC}
GC cycle stats:
 RTGC completed in deterministic mode (3 workers at priority 40)
 End free bytes: 105152 bytes
 Min free bytes: 1024 bytes
 Next boosted threshold: 72960 bytes
 Next normal threshold: 139840 bytes
 Allocations during the GC:
    in deterministic mode: 60896 bytes 
                     (worst 60896 bytes, average 60896 bytes)
    in boosted+deterministic modes: 60896 bytes 
                     (worst 60896 bytes, average 60896 bytes)
    in normal+boosted+deterministic modes: 60896 bytes 
                     (worst 60896 bytes, average 60896 bytes)
 Total CPU cost in nanoseconds: 66093516
     Pacing CPU cost: 100667 (0 %)
     Serial CPU cost: 5858823 (8 %)
     Parallel worker_0 CPU cost: 10220120 (15 %)
     Parallel worker_1 CPU cost: 46879782 (70 %)
     Parallel worker_2 CPU cost: 2175946 (3 %)
 Bytes allocated by critical threads:
    in deterministic mode: 0 bytes
    total for this cycle: 0 bytes
    grand total: 0 bytes
 Minimum RTGCCriticalReservedBytes: 0 bytes
, 0.0635765 secs] 

The output is largely self-explanatory. However, note the following points:

• The values that are noted "worst" represent values since VM startup, whereas the "min" and "max" values occurred during the last GC cycle.
• Because the RTGC does not use heap generations, the "FullGC" value does not appear in the output. In fact, the entire heap is recycled in each RTGC run.
• The "non-fragmented" bytes represent memory that is allocatable. If the "min" or "worst" value falls to zero, this means that threads were probably suspended due to memory exhaustion.
• The "dark matter" represents memory that is fragmented and therefore cannot be allocated.
• The output also indicates whether the RTGC completed without boosting, whether the RTGC priority was boosted during the run (and for how long), or whether memory fell below the RTGCCriticalReservedBytes threshold.
• The "CPU load recycling" value shows the usage of the available CPU time for the RTGC during that run. For example, if this value is 1.05 using 2 processors, this could mean the RTGC used 90% of one processor and 15% of the second one, or 80% of the first and 25% of the second, and so forth.
• The "CPU load since last GC" value provides the usage of the available CPU time for garbage collection during the entire GC cycle, that is, from the end of the previous GC run to the end of this GC run. For example, on a four-way machine, a value of 0.04 means that the collection took 0.04/4=1% of the total CPU time (for all the available CPUs) from the end of the previous GC run to the end of this run.
• The longer output provides the mode in which the RTGC completed the cycle: normal, boosted, or deterministic. It also provides the amount of memory that was allocated during these three modes, that is, only during deterministic mode, during either boosted or deterministic mode, and during all three modes together.

[Contents]

Sample Output from a "Bad" Configuration

The following test output from XX:+RTGCPrintStatistics shows that the Java RTS VM was not configured to be deterministic.

[ 63823K->18950K(65536K, non fragmented: current 46585K / min 1050K / worst 0K,
blocked threads: max 0 / still blocked 0 requesting 0 bytes,
dark matter: 17572K in 16299 blocks smaller than 2048 bytes)
<completed without boosting> {CPU load: 2.22 recycling / 0.06 since last GC}
GC cycle stats:
 RTGC completed in normal mode (4 CPUs at priority 15)
 End free bytes: 47695680 bytes
 Min free bytes: 17111136 bytes
 Next boosted threshold: 171960 bytes
 Next normal threshold: 1676650 bytes
 Allocations during the GC:
    in deterministic mode: 0 bytes 
                    (worst 243072 bytes, average 126566 bytes)
    in boosted+deterministic modes: 0 bytes 
                    (worst 243072 bytes, average 126566 bytes)
    in normal+boosted+deterministic modes: 1367904 bytes 
                    (worst 1461152 bytes, average 841890 bytes)
 Total CPU cost: 231007716 nanoseconds
     Pacing CPU cost: 285081 (0 %)
     Serial CPU cost: 6322420 (2 %)
     Parallel worker_0 CPU cost: 89724588 (38 %)
     Parallel worker_1 CPU cost: 43886333 (18 %)
     Parallel worker_2 CPU cost: 46492001 (20 %)
     Parallel worker_3 CPU cost: 44295704 (19 %)
 Bytes allocated by critical threads:
    in deterministic mode: 0 bytes
    total for this cycle: 16128 bytes
    grand total: 5322592 bytes
 Minimum RTGCCriticalReservedBytes: 9216 bytes
, 0.1054441 secs]

The value of the "worst" non-fragmented memory on the first line of the output shows that at least one thread blocked during allocation. Since the "grand total" for "Bytes allocated by critical threads" is not zero, that means that there were, in fact, some critical threads, and therefore, RTGCCriticalReservedBytes must be set to a value at least equal to the amount of memory needed by these critical threads. The "Minimum RTGCCriticalReservedBytes" helps in estimating it. To guarantee the determinism of the critical threads, you should add a very large safety margin to this value. In this case, since the "worst" value was zero, the RTGCCriticalReservedBytes value was set too low.

[Contents]

Sample Output from a Corrected Configuration

The following output from XX:+RTGCPrintStatistics shows what happens in the same test when we set RTGCCriticalReservedBytes to 1 Mb. Determinism has now been ensured for the critical threads.

[ 62798K->16857K(65536K, non fragmented: current 48678K / min 2068K / worst 1016K,
blocked threads: max 0 / still blocked 0 requesting 0 bytes, 
dark matter: 15426K in 16298 blocks smaller than 2048 bytes) 
<completed without boosting> {CPU load: 2.24 recycling / 0.06 since last GC}
GC cycle stats:
 RTGC completed in normal mode (4 CPUs at priority 15)
 End free bytes: 49825664 bytes
 Min free bytes: 15991968 bytes
 Next boosted threshold: 1266976 bytes
 Next normal threshold: 2853616 bytes
 Allocations during the GC:
        in deterministic mode: 0 bytes
	                (worst 277600 bytes, average 160552 bytes)
        in boosted+deterministic modes: 0 bytes
	                (worst 277600 bytes, average 160552 bytes)
        in normal+boosted+deterministic modes: 1442400 bytes
	                (worst 1450688 bytes, average 889055 bytes)
 Total CPU cost: 244487405 nanoseconds
        Pacing CPU cost: 290305 (0 %)
        Serial CPU cost: 6557720 (2 %)
        Parallel worker_0 CPU cost: 46866569 (19 %)
        Parallel worker_1 CPU cost: 93580338 (38 %)
        Parallel worker_2 CPU cost: 50525021 (20 %)
        Parallel worker_3 CPU cost: 46592857 (19 %)
 Bytes allocated by critical threads:
        in deterministic mode: 0 bytes
        total for this cycle: 15616 bytes
        grand total: 4763680 bytes
 Minimum RTGCCriticalReservedBytes: 17440 bytes
, 0.1102827 secs]

With RTGCCriticalReservedBytes set to 1 Mb, the "worst" non-fragmented memory level is 1016 Kb. Therefore, memory was never exhausted, and determinism has been assured for the critical threads. However, since "worst" is lower than the value of RTGCCriticalReservedBytes, some non-critical threads might have blocked on allocation.

[Contents]