This document is a guide to help troubleshoot problems that might occur in applications that are developed with Sun Java™ Real-Time System (Java RTS). The guide concentrates on issues that are specific to the Java RTS VM. For problems with applications that are developed on the Java SE 5.0 platform, with the Java HotSpot™ VM, see the Java 2 Platform Standard Edition 5.0 Trouble-Shooting and Diagnostic Guide (PDF).

Note that some procedures are platform-specific: Solaris™ Operating System (Solaris OS) or Linux Operating System.

[Java RTS Readme: links to all Java RTS technical documents]

Contents

Introduction

Application Determinism, Jitter, Missed Deadlines
Information to Collect
How to Determine the Source of the Problem
Other Helpful Documentation for Troubleshooting
Blogs

Troubleshooting Tools and Options

Overview
Command-Line Options
Balancing Determinism and Debugging
Debugging and Profiling
DTrace Probe Provider (jrts) for Java RTS (Solaris OS only)
Thread Scheduling Visualizer
jmap Utility for Java RTS
jstack Utility for Java RTS
Java VisualVM with Java RTS

Tuning Java RTS

Compilation Tuning
Real-Time Garbage Collector Tuning

General Troubleshooting Tips

Glossary of Terms

Introduction

This document represents an accumulation of information that can help you resolve problems that might occur with your Java RTS application. A number of specialized tools have been developed, and this activity is on-going. Certain problems can be resolved by correctly tuning the application. And finally, the document presents some troubleshooting tips.

This document is under continual improvement and extension, and all comments, suggestions, and ideas are welcome on our Evaluation Feedback Form.

Application Determinism, Jitter, Missed Deadlines

The hallmark of a real-time application is its determinism, which means that deadlines must not be missed. Determinism can be measured in terms of jitter, where jitter is defined as the variation in response time or execution time of a real-time application. The response time is calculated from the time you send a request to a real-time server to the time you receive an answer. This time is an accumulation of the following times:

• The time to send the request from your non-real-time JVM. Consider what happens if the GC occurs in the middle of your send call.
• Network latencies. What if users automatically do their downloads at night, when your benchmark is running?
• The variation in the time required to compute the answer, for example, if you must access a database.
• Preemption by other processes or threads running at the same time on the server machine.
• Blocking due to synchronization with other activities.
• Network latency for the reply.
• The time to detect the answer.

You can use the -XX:+LogJitterProneEvents command-line option to obtain a log file of various events that can cause jitter. See Command-Line Options.

If you are not able to find the cause for the jitter, then it might be the case that the application code is indeed not deterministic. This can be due to non-deterministic optimizations, or to calls non-deterministic code (for example, JDK libraries, I/O, and so forth).

The document A Practical Introduction to Achieving Determinism is part of a "Getting Started" package that shows how to quickly and easily achieve determinism. This document contains a discussion of the usual sources of jitter, as well as a description of how to use the sample programs and script that are provided in the package to achieve determinism by eliminating these sources of jitter. Take a look at this document to get ideas for troubleshooting jitter issues.

[Contents]

Information to Collect

This section suggests some of the information to collect in order to troubleshoot a problem.

• Core file, if one exists. To force a core dump, use the gcore command. (Solaris OS only)
• Output from the command-line options, for example, -XX:+PrintGC. See Command-Line Options.
• Output from DTrace probes. See DTrace Probe Provider (jrts) for Java RTS (Solaris OS only).
• Log files.
• Ouput from the command uname -v (OS version, hardware platform, and so forth)
• Java RTS version, as shown in the output from the command java -version.
• Runtime options used when starting the VM.
• Settings of environment variables, for example, LD_LIBRARY_PATH.

[Contents]

How to Determine the Source of the Problem

Before starting to troubleshoot a possible problem, make a attempt to categorize the problem you are experiencing. This will help you identify the cause of the problem and ultimately find a solution. It is also important to determine what is not a problem in order to clarify the situation.

One of the first issues to consider is your configuration.

• Make sure that you have the correct version of Java RTS, of the OS platform, and so forth. Refer to the Java RTS Release Notes.

• Consider the compilation mode that you are using. See the section Compilation Tuning.

• Are you using the RTGC? Have you performed the minimum configuration that is recommended? See the section Real-Time Garbage Collector Tuning.

The next important step is to check how you measured the different variables in your system. In particular, the measurement itself must not create additional jitter.

• Use the right clock. See "Clocks" section of the Technical Information document.

• Alternatively, use DTrace timestamps. (Solaris OS only)

• Finally, check what you measured.

The following tips can help simplify and verify your troubleshooting procedure:

• First, assure that your application works for a simple example. You must be sure that you know what your application threads are supposed to do.

• Protect your system from external sources of jitter (for example, isolated network, low loaded machine, processor sets, and so forth).

• Simplify what you measure (one request at a time).

• Provide several measurement points in order to distinguish the various issues:

  • Java measurement beginning and end of your answer computation code.
  • DTrace measurement of wakeup_time, end_of_read, start_writing. (Solaris OS only)

• Use timestamps to corrolate the events.

• Check if anything disturbed your measurement.

• Distinguish preemption time from blocking time and from execution time:

  • Use DTrace to monitor the worker thread once it awakens. (Solaris OS only)
  • Identify the processes and threads that ran instead of your thread.

[Contents]

Other Helpful Documentation for Troubleshooting

In addition to this document, the following documents will be helpful:

• Java RTS Garbage Collection Guide

• Java RTS Compilation Guide

• Monitoring and Management for the Java Platform (J2SE 5.0)

• J2SE 5.0 Troubleshooting and Diagnostic Guide. Although this guide concerns applications running on a HotSpot VM, and not on a Java RTS VM, it can provide ideas for solving certain types of problems.

• Sun Java Real-Time System - FAQ

• D-I-Y: Diagnose-It-Yourself (PDF) - JavaOne^SM 2008 presentation (PDF file) on monitoring and troubleshooting Java RTS. This presentation contains examples of troubleshooting analysis using DTrace, TSV, jstack, and jmap.

[Contents]

Blogs

Individual blog sites and blog entries have proven to be very helpful by providing ideas and tips for solving problems. Here are a few such blogs:

• Greg Bollella's Real-Time blogs: http://blogs.sun.com/bollellaRT/
• Bertrand Delsart's blogs: http://blogs.sun.com/delsart/
• David Holmes' blogs: http://blogs.sun.com/dholmes/
• Nikolay Igotti's blogs describing how to use x86 debugging regs to catch memory accesses
  • http://blogs.sun.com/nike/entry/memory_debugger_for_linux
  • http://blogs.sun.com/nike/entry/memory_debugger_for_windows
• David Therkelsen's blogs: http://blogs.sun.com/therk/
• Roland Westrelin's blogs: http://blogs.sun.com/roland/

[Contents]

Troubleshooting Tools and Options

This section introduces several tools that can be used to troubleshoot problems in your Java RTS application. Some command-line options are also suggested.

Overview

Java SE 5.0 (also called J2SE 1.5) provides a set of experimental and unsupported tools for monitoring and management as well as tools for troubleshooting. However, the design of the the Real-Time Garbage Collector (RTGC) and the no-heap real-time threads (NHRTs) introduced differences between Java RTS and HotSpot, for example, in the memory model, the stack design, and other issues. For this reason, some of the options of some of the tools cannot work with Java RTS applications that use the RTGC or NHRTs. The Java RTS team is continually improving the architecture of the system in order to be able to take full advantage of these tools.

Here is a summary of the current availability of the Java SE 5.0 tools with Java RTS:

• jconsole works as expected. The jconsole command launches a graphical console tool that enables you to monitor and manage Java applications and virtual machines on a local or remote machine. See the jconsole man page for J2SE 1.5.

• jinfo works as expected. This command-line utility prints configuration information for a given Java process or core file. This configuration information includes Java system properties and command-line flags that were used to start the VM. See the jinfo man page for J2SE 1.5.

• jps works as expected. This command-line utility lists the Java RTS virtual machines on the target system. See the jps man page for J2SE 1.5.

• With Java RTS, jmap can extract memory information from a core file. See the subsection jmap Utility for Java RTS.

• With Java RTS, jstack can extract thread stack information from a core file. See the subsection jstack Utility for Java RTS.

• With Java RTS, Java™ VisualVM can perform local and remote monitoring and management. See the subsection Java VisualVM with Java RTS.

• The other tools are provided "as is" because most of the features work with Java RTS if you do not use the RTGC or NHRTs. These tools will be improved in future releases.

For related information, see the Monitoring and Management for the Java Platform guide (Java SE 5.0) and the Java 2 Platform Standard Edition 5.0 Trouble-Shooting and Diagnostic Guide (PDF).

In addition, Java RTS provides these specialized tools:

• Management beans for the real-time garbage collector API allow you to dynamically monitor and control certain RTGC elements. The platform MBeans can be used to remotely get information about the VM, for example through JConsole. See Java RTS Garbage Collection Guide.

• (Solaris OS only) A separate DTrace probe provider, called jrts, furnishes DTrace probes that allow you to observe, debug, and tune your Java RTS real-time application. See the subsection DTrace Probe Provider (jrts) for Java RTS (Solaris OS only).

• The Thread Scheduling Visualizer (TSV) allows you to record and visualize thread scheduling events. See the subsection Thread Scheduling Visualizer.

[Contents]

Command-Line Options

The following command-line options are useful for troubleshooting:

• -XX:+PrintGC and -XX:+PrintGCTimeStamps - Print information whenever the GC runs. Note that, because the RTGC does not use heap generations, the "FullGC" value does not appear in the output. In fact, the entire heap is recycled with each RTGC garbage collection run.

• -XX:+RTGCPrintStatistics - Prints detailed information when the RTGC runs. See output examples in the Garbage Collection Guide.

• -XX:-UseRTGC - Disables the RTGC. Debugging in this mode is easier and can help detect a lot of issues (particularly when using RTSJ extensions).

• -XX:+LogJitterProneEvents - Produces a file named hs_jitter_pid<pid>.log containing various time-stamped events that can cause jitter.

  The format of an entry in the log file is: <timestamp>, <priority>, <thread ID>, <event name>, <thread name> if available, and <event details> if any.

  The following is a list of the event names (in quotes) followed by the event detail if any (in quotes), with a short explanation:

  • "Entering Steady Mode" / "Exiting Steady Mode": Entering or exiting a user-defined "deterministic" section of a real-time application. You can specify various sections of "steady mode" code in your application with the "enter" and "exit" methods of the SteadyMode class in the extended RTSJ package. See item "Jitter in Deterministic Code" in General Troubleshooting Tips for a discussion.
  • "C-Heap malloc" "addr s t": Allocation of VM internal buffer of type t, of s bytes, at address addr.
  • "C-Heap free" "addr": Freeing of VM internal buffer at address addr.
  • "Oop allocation failed for critical thread": Failed to allocate a contiguous block for a critical thread. Split allocation will be tried.
  • "Split oop allocation failed for critical thread": Not enough memory for this request. The critical thread must wait for the RTGC to complete.
  • "field patching" "bc at bci bci in method caller for class cl": Compiled code patching for bytecode bc at bytecode index bci in method caller for class cl.
  • "klass patching" "bc at bci bci in method caller for class cl": Compiled code patching for bytecode bc at bytecode index bci in method caller for class cl.
  • "invokevirtual patching" "bc at bci bci in method caller for method callee": Compiled code patching for bytecode bc at bytecode index bci in method caller for method callee.
  • "invokeinterface patching" "bc at bci bci in method caller for method callee": Compiled code patching for bytecode bc at bytecode index bci in method caller for method callee.
  • "patching call" "bc at bci bci in method caller for method callee": Compiled code patching for bytecode bc at bytecode index bci in method caller for method callee.
  • "class initialization attempt" "cl": Attempted initialization of class cl.
  • "Actual Class Initialization" "cl": Successful initialization of class cl.
  • "Calling Class Static Initializer" "cl": Static initializer for class cl is called.
  • "interpreted method" "m": Entering interpreted execution of method m.

  The following is sample output from the LogJitterProneEvents option. Note that some lines have been broken into two for displayability.

  1213625734174692917, 3, 18: C-Heap malloc, N/A, 0x0826a060 16 CHeapObj-new
  1213625734177445726, 3, 18: class initialization attempt, 
      Thread-0, com.sun.rtsjx.Finalizer
  1213625734177482742, 3, 18: Actual Class Initialization, Thread-0,
      com.sun.rtsjx.Finalizer
  1213625734177510434, 3, 18: Calling Class Static Initializer, Thread-0,
      com.sun.rtsjx.Finalizer
  1213625734843048306, 30, 21: C-Heap malloc, N/A, 0x0826a2d0 160 CHeapObj-new
  1213625734843341384, 30, 21: interpreted method, RealtimeServerThread-0,
      virtual void javax.realtime.AsyncEventHandler.run1()
  1213625734843631236, 30, 21: interpreted method, RealtimeServerThread-0,
      virtual jboolean javax.realtime.AsyncEventHandler.setupReleaseEnv()
  1213625734843796212, 30, 21: interpreted method, RealtimeServerThread-0,
      virtual void TCK.AbsoluteTimeTest$1.handleAsyncEvent()
  1213625734844012649, 30, 21: interpreted method, RealtimeServerThread-0,
      virtual void java.lang.Object.notify()
  1213625734844128823, 30, 21: C-Heap malloc, N/A, 0x0826a3a0 12 CHeapObj-new
  1213625734844224219, 30, 21: interpreted method, RealtimeServerThread-0,
      virtual jboolean javax.realtime.AsyncEventHandler.setupReleaseEnv()
  1213625734845736678, 3, 18: C-Heap malloc, N/A, 0x0826a3e0 12 CHeapObj-new
  

See the Java RTS Command-Line Options page for a list of all the -XX command-line options for Java RTS, and the Java HotSpot VM Options page for a list of all the HotSpot -XX command-line options.

[Contents]

Balancing Determinism and Debugging

The use of some of the monitoring tools or supporting library calls in Java RTS can lead to determinism issues. For instance, Java RTS must suspend the application to take a snapshot of the thread states and look for deadlocks. By default, determinism is considered to be more important than obtaining all the information that HotSpot is able to provide. However, during application development you might be more interested in the debugging functionalities than in determinism.

Therefore, Java RTS provides mechanisms to select between a very deterministic deployment mode and a less deterministic development or debugging mode. This currently impacts only the stack trace information and the thread state snapshots. Both are controlled by a single parameter that can be changed in two ways:

• Dynamically with a monitoring bean: JRTSDebugMXBean.

• At VM startup with a command-line option (turned on by default): -XX:-RTSJDisableAtomicThreadDump.

[Contents]

• The DTrace probe provider for Java RTS, called jrts, is the least intrusive profiling tool. (Solaris OS only)

• The platform MBeans can be used to get information about the VM, for example, application stack traces, the real-time heap state. You can use the JConsole GUI to see this information.

• Some command-line options provide verbose output to analyze specific issue, for example, -PrintGC and -XX:+RTGCPrintStatistics. See the Command-Line Options section.

Other profilers are usually so intrusive that they are of little use in a real-time environment.

Java RTS supports the Java Platform Debugger Architecture (JPDA), which includes the following interfaces: the Java Debug Interface (JDI), the Java Debug Wire Protocol (JDWP), and the Java™ Virtual Machine Tool Interface (JVMTI).

Since the Java Virtual Machine Profile Interface is a deprecated interface as of Java SE 5.0, it is not supported in Java RTS; instead, use the JVMTI.

The NetBeans™ Integrated Development Environment (IDE) implements these interfaces for debugging, and can be used to debug applications running on Java RTS. This release of Java RTS is compatible with NetBeans 5.5.1.

Note: Java RTS does not support NetBeans profiling because some of the methods break real-time behavior.

JPDA provides a comprehensive set of features for debugging an application, at the expense of impairing its temporal behavior during the debugging session. Once the debugger is attached to the Java RTS virtual machine, it is possible to stop, inspect, and resume any type of schedulable that is being executed, that is java.lang.Thread, RealtimeThread, NoHeapRealtimeThread and AsyncEventHandler. It is also possible to debug schedulables executing in scoped memory.

However, once the debugger is attached, the temporal behavior of the application is no longer guaranteed, and there is therefore no need to support the Real-Time Garbage Collector (RTGC). In addition, most debugging operations require the suspension of threads. These suspensions impact all the running threads, including the NoHeapRealtimeThreads and the high-priority RealtimeThreads. Therefore, when debug options are specified on the command line, Java RTS automatically activates the non-real-time mark-and-sweep GC.

[Contents]

DTrace Probe Provider (jrts) for Java RTS (Solaris OS only)

The DTrace probe provider for Java RTS is called jrts. With these probes you can observe, debug, and tune your Java RTS real-time application. Without any modification to the running application, you can monitor activities in several categories, including priority changes, memory allocation contexts, thread execution and interruption, event handlers and server threads, compilation, symbol and method resolution, real-time garbage collection, class loading and initialization, and VM operations. Finally, a special class allows you to fire a general-purpose probe in order to track any event within the application that you choose.

The document Java RTS DTrace Provider contains a detailed description of these probes and how to use them, as well as several sample scripts.

[Contents]

Thread Scheduling Visualizer

In troubleshooting a multi-threaded real-time application, it can be very useful to see the actual scheduling of all the threads. The Thread Scheduling Visualizer (TSV) is a set of tools that provide an easy way to record and visualize thread scheduling for Java RTS. The document Java RTS Thread Scheduling Visualizer contains a detailed description of this tool set, including how to record scheduling events using customizable DTrace scripts that are provided, as well as how to use the Visualizer GUI to get a graphical view of the events. You can download the TSV tool set from the Java RTS Evaluation Downloads page.

The following examples show the types of debugging information TSV can provide (taken from a real-life experience):

• A 1 millisecond period was too short because the thread was taking longer than 1 millisecond to create the interfering threads and reach wait-for-next-period.
• The deadline miss handler ran only once, which reminded the user to call schedulePeriodic() in the handler.
• The blocker threads were completing almost immediately, which turned out to be caused by the programmer adding milliseconds to a nanosecond time, causing the program to check for the passing of 50 nanoseconds instead of 50 milliseconds.
• The critical threads had to have priorities above the RTGC critical value because the RTGC threads were preempting the critical threads when they woke up periodically.
• The deadline miss handler had too low a priority with respect to the blocker threads.
• TSV showed at what point the JIT compiler thread was executing - and that it was not interfering with the critical threads' execution.
• A high priority thread had boosted a low priority thread, which took longer and longer to release the lock requested by the high priority thread. Armed with this information, the user was able to identify the lock and fix the code using a different data structure, with more deterministic critical sections.

Before using TSV with Java RTS, ensure that you have the required resource access privileges, as described in the Java RTS Installation Guide. Otherwise, TSV fails with messages such as "invalid probe specifier" or other messages indicating problems with the set of probes.

[Contents]

jmap Utility for Java RTS

The jmap command-line utility for Java RTS prints shared object memory maps or heap memory details of a given core file.

When no option is used, jmap prints shared object mappings for each shared object loaded in the target VM: start address, size of the mapping, and full path of the shared object file.

Each option produces different output: a heap summary, a heap dump in binary format, a heap histogram, or permanent generation statistics.

See the jmap man page for Java RTS.

Note that, in the sample output listings below, some lines have been broken into two or more for displayability, and some lines have been removed to reduce the volume of the output for this document.

jmap output: Heap summary with the -heap option

The sample output below is from the jmap command specifying an executable and a core file, and with the -heap option.

% jmap -heap <Java RTS install dir>/bin/java core.5626
Attaching to core core.4921 from executable 
/net/mackdrive/export/binaries/solaris-sparc/bin/java, 
please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_13-b05

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 67108864 (64.0MB)
   NewSize          = 655360 (640.0KB)
   MaxNewSize       = 4294967295 (4095.99MB)
   OldSize          = 1441792 (1.375MB)
   NewRatio         = 12
   SurvivorRatio    = 32
   PermSize         = 67108864 (64.0MB)
   MaxPermSize      = 67108864 (64.0MB)

Real-Time Parameters Configuration:
   UseRTGC                      = 1
   ImmortalSize                 = 33554432 (32.0MB)
   ImmortalPhysicalReservedSize = 1048576 (1.0MB)
   ScopedSize                   = 100663296 (96.0MB)
   ScopedPhysicalReservedSize   = 1048576 (1.0MB)
   ScopedMemoryAllocGrain       = 16384 (16.0KB)
   RTGCCriticalReservedBytes    = 0 (0B)
   RTGCWaitDuration             = 10
   RTGCCriticalBoundary         = 40
   RTGCBoostedWorkers           = 2
   RTGCBoostedPriority          = 40
   BoostedMinFreeBytes          = 0 (0B)
   BoostedSlideFactor           = 10
   BoostedSafetyMargin          = 20
   RTGCNormalWorkers            = 2
   RTGCNormalPriority           = 11
   NormalMinFreeBytes           = 0 (0B)
   NormalSlideFactor            = 20
   NormalSafetyMargin           = 10

Heap Usage:
RT Collected Heap:
   capacity = 67108864 (64.0MB)
   used     = 338944 (331.0KB)
   free     = 66769920 (63.67MB)
   0.50% used

Non-heap memory areas:
Scoped memory block 1:
   capacity = 16777216 (16.0MB)
   used     = 7850224 (7.48MB)
   free     = 8926992 (8.51MB)
   46.79% used
Scoped memory block 2:
   capacity = 16777216 (16.0MB)
   used     = 7794224 (7.43MB)
   free     = 8982992 (8.56MB)
   46.45% used
Immortal memory block:
   capacity = 33554432 (32.0MB)
   used     = 16377608 (15.61MB)
   free     = 17176824 (16.38MB)
   48.80% used
Unallocated scoped memory chunk:
   size = 33554432 (32.0MB)
Total unallocated scoped memory:
   size = 33554432 (32.0MB)
   

jmap output: Heap dump in binary format with the -heap:format=b option

The sample output below is from the jmap command specifying an executable and a core file, and with the -heap:format=b option. The dump is written in hprof binary format to a file named heap.bin in the working directory. This file can then be examined with the HAT or jhat tool.

% jmap -heap:format=b <Java RTS install dir>/bin/java core.4921
Attaching to core core.4921 from executable 
/net/mackdrive/export/binaries/solaris-sparc/bin/java, 
please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_13-b05

heap written to heap.bin

jmap output: Heap histogram with the -histo option

The sample output below is from the jmap command specifying an executable and a core file, and with the -histo option. For each Java class the utility prints the memory size in bytes, the number of objects, and the fully qualified class name. The memory value includes both heap and non-heap (immortal and scoped areas).

% jmap -histo <Java RTS install dir>/bin/java core.4921
Attaching to core core.4921 from executable 
/net/mackdrive/export/binaries/solaris-sparc/bin/java, 
please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_13-b05
Iterating over heap. This may take a while...

Object Histogram:

Size	Count	Class description
-------------------------------------------------------
840464	9	long[]
481168	858	int[]
219168	1826	char[]
154584	574	byte[]
120360	4	double[]
64704	2696	Fibonacci1
64704	2696	Fibonacci2
56888	547	java.lang.Class
56064	1752	java.lang.String
55192	837	java.lang.Object[]
51160	824	short[]

[...lines removed to reduce output ... ]

jmap output: Permstat statistics with the -permstat option

The sample output below is from the jmap command specifying an executable and a core file, and with the -permstat option. The command prints statistics for the permanent generation of the heap. For each class loader, the utility prints the number and size of the classes it has loaded, its parent loader, its aliveness, and its type. A total line appears at the end.

% jmap -permstat <Java RTS install dir>/bin/java core.4921
Attaching to core core.4921 from executable 
/net/mackdrive/export/binaries/solaris-sparc/bin/java, 
please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 1.5.0_13-b05
finding class loader instances ..done.
computing per loader stat ..done.
please wait.. computing liveness.done.

class_loader   classes bytes   parent_loader   alive?  type

<bootstrap>    470     1353216   null          live    <internal>
0xf44205a8     40      190064  0xf4419af0      live    
                          sun/misc/Launcher$AppClassLoader@0xf05a6bb0
0xf4419af0     0       0         null          live    
                          sun/misc/Launcher$ExtClassLoader@0xf05811f8

total = 3      510     1543280   N/A     alive=3, dead=0     N/A    

[Contents]

jstack Utility for Java RTS

The jstack command-line utility for Java RTS attaches to the specified core file and prints the stack traces of all threads that were attached to the virtual machine at the time the core file was produced.

Some private symbols of the JVM are not recognized by jstack but can be viewed with dbx or the Solaris OS pmap tool.

See the jstack man page for Java RTS.

Each thread is specified as an instance of java.lang.Thread, javax.realtime.RealtimeThread, or javax.realtime.NoHeapRealtimeThread. For each thread, the integer thread identifer and the thread state are printed. For each Java frame, the utility prints the full class name, method name, bci (byte code index), line number (if available), and whether the code was interpreted or compiled.

Note that, in the sample output listings below, some lines have been broken into two or more for displayability, and some lines have been removed to reduce the volume of the output for this document.

jstack output: Thread stack with no options

Below is sample output from the jstack command specifying an executable and a core file, but with no options.

% jstack <Java RTS install dir>/bin/java core.4921
Attaching to core core.4921 from executable 
/net/mackdrive/export/binaries/solaris-sparc/bin/java, 
please wait...
Debugger attached successfully.
Client compiler detected.

[...lines removed to reduce output ... ]

java.lang.Thread t@38: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=121
                                                (Interpreted frame)

[...lines removed to reduce output ... ]

javax.realtime.RealtimeThread t@46: (state = IN_JAVA)
 - Fibonacci2.recursiveFib(int) @bci=17, line=26 (Interpreted frame)
 - Fibonacci2.recursiveFib(int) @bci=18, line=26 (Interpreted frame)
 - Fibonacci2.recursiveFib(int) @bci=18, line=26 (Interpreted frame)
 - Fibonacci2.computeFib() @bci=17, line=35 (Interpreted frame)
 - Deterministic.computeFibs(int, int) @bci=35, line=99 
                                                (Interpreted frame)
 - Deterministic.run() @bci=19, line=124 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=713 (Interpreted frame)

[...lines removed to reduce output ... ]

jstack output: Thread stack with mixed mode (-m) option

With the -m option (mixed mode), jstack prints both Java and native frames of all threads. Frames prefixed with "*" are Java frames, and other are native frames. Below is sample output from the jstack -m command. Note that some lines have been broken into two for displayability, and some lines have been removed to reduce the volume of the output for this document.

% jstack -m <Java RTS install dir>/bin/java core.4921
Attaching to core core.4921 from executable 
/net/mackdrive/export/binaries/solaris-sparc/bin/java, 
please wait...
Debugger attached successfully.
Client compiler detected.

[...lines removed to reduce output ... ]


----------------- t@1 -----------------
0xff340ec8	ioctl + 0x8
0xfe919134	JVM_MonitorWait + 0x31c
0xf880e11c	* java.lang.Object.wait(long) bci:0 (Interpreted frame)
0xf88062c8	* java.lang.Thread.join(long) bci:38 line:1302 
                                                    (Interpreted frame)
0xf88062c8	* java.lang.Thread.join() bci:2 line:1355 
                                                    (Interpreted frame)
0xf88062c8	* Deterministic.main(java.lang.String[]) 
                                  bci:1428 line:409 (Interpreted frame)
0xf8800278	<StubRoutines>
0xfe8feba4	void JavaCalls::call_helper(JavaValue*,methodHandle*,
                                   JavaCallArguments*,Thread*) + 0x674
0xfe9e5308	jni_CallStaticVoidMethod + 0x3d8
0x00012224	main + 0x115c
0x000110b0	_start + 0x108
----------------- t@2 -----------------
0xff340ec8	ioctl + 0x8
0xff340368	_lwp_start

[...lines removed to reduce output ... ]

[Contents]

Java VisualVM with Java RTS

Java™ VisualVM is a graphical tool for the monitoring, management, and profiling of Java applications. The tool can analyze a running process or a core file. An application developed to run with the Java RTS VM can have certain constraints that interfere with the activity of Java VisualVM. In particular, the profiling function of Java VisualVM does not work with a Java RTS application because the profiling uses dynamic instrumentation of the application, creating overhead, and thus adversely affecting the determinism of a real-time application.

Java VisualVM must run with Java SE 6 (JDK 6). However, it can display information on any application running with Java RTS.

The following functions of Java VisualVM work with Java RTS:

• Obtain high-level information about a local or remote running process, for example, JVM flags.

• Obtain graphical statistics about the heap allocation, permanent generation, threads, and classes of a local or remote running process.

• Obtain a thread dump or heap dump of a local or remote running process.

• Read a core file that was produced on the host where Java VisualVM is running.

For a detailed description of Java VisualVM and how to use it, see http://java.sun.com/javase/6/docs/technotes/guides/visualvm/index.html

[Contents]

Tuning Information

For a non-real-time environment, the primary performance criterion is the throughput of the system. However, in a real-time environment, execution of time-critical code must be deterministic, that is, predictable, and this means reducing the worst-case execution time to an acceptable level.

To avoid unpredictable delays, consider tuning the compilation modes for your application or the activity of the Real-Time Garbage Collector, as described in the subsections that follow.

Compilation Tuning

Interpreter and compiler activity can cause unpredictable delays, as follows:

• Interpreted code. The interpreter might require several executions of the bytecode before reaching a steady state, and the programmer cannot control when this will happen.

• Run-time compilation. If compilation interrupts the execution of a section of time-critical code, determinism is seriously compromised.

• On-demand symbol resolution. When an unresolved symbol is encountered during the execution of the code, the VM uses some extra, non-deterministic, code to resolve the symbol.

Java RTS currently supports two compilation modes: Just-In-Time Compilation (JIT) and Initialization Time Compilation (ITC). JIT compilation is the usual compilation mode of a Java virtual machine. With the ITC scheme in Java RTS, you can specify certain time-critical classes and methods to be preloaded, preinitialized, or precompiled. You can tune this compilation behavior to improve your application's determinism. As a general rule, you should preload time-critical classes that contain symbols that are likely to be referenced during execution, you should preinitialize classes containing time-critical code that will be executed, and you should precompile time-critical methods that will be executed. To facilitate your work, Java RTS can automatically generate lists of these classes and methods for you.

Moreover, you can configure different compilation modes for different thread types, that is, instances of java.lang.Thread, javax.realtime.RealtimeThread, and javax.realtime.NoHeapRealtimeThread classes. This possibility allows you to tune the trade-off between throughput and determinism.

Note that, by default, JIT compilation is performed asynchronously. Java code is executed interpreted until one of the internal counters triggers a request to compile the method. The method continues running interpreted while the JIT compilation is handled by the compilation thread. Once the compilation is complete, the application thread continues execution with the JIT-compiled code. However, the compilation thread runs at the first real-time priority, so if the CPUs are all busy doing work at a higher real-time priority the compilation thread will not get a chance to run, code will never get compiled, and threads will keep running interpreted. Therefore, be sure that all critical methods are precompiled, as described above.

The Java RTS Compilation Guide contains a very detailed description of all these tuning mechanisms and the relevant command-line parameters.

In addition, the document Practical Introduction to Achieving Determinism describes in detail a sample program that can be made deterministic by correctly configuring the compilation mode, among other variables.

[Contents]

Real-Time Garbage Collector Tuning

Java RTS introduces a new garbage collector, the Real-Time Garbage Collector (RTGC). The RTGC is the default GC in Java RTS.

The RTGC advances determinism, but at the cost of global throughput. If your application is more sensitve to the collector's throughput than to the pause times caused by the collector's execution, then you can disable the RTGC with the command-line option -XX:-UseRTGC. In this case, the non-real-time serial garbage collector is used.

The RTGC can be configured to a very advanced degree, allowing you to precisely tune factors such as when garbage collection should start and at what priority relative to other threads. Once again, this tuning allows you to balance throughput against determinism. The Java RTS Garbage Collection Guide contains a very detailed description of all these tuning mechanisms and the relevant command-line parameters.

However, there is one parameter that should be configured: RTGCCriticalReservedBytes. This parameter specifies a amount of memory that is held in reserve for the use of critical threads. Since the default value for this parameter is zero, threads that are time-critical could be blocked, thereby compromising the determinism of the application.

The RTGC has an auto-tuning mechanism that dynamically tries to find the best balence between throughput and determinism. Many of the factors that are used in this process are command-line parameters that you can configure in order to adapt the RTGC activity to your needs. Again, the Java RTS Garbage Collection Guide describes this process in great detail.

In addition, the document Practical Introduction to Achieving Determinism describes in detail a sample program that can be made deterministic by introducing the use of the RTGC, among other variables.

[Contents]

General Troubleshooting Tips

This section presents some general tips and hints to use in troubleshooting.

• No access to real-time resources

  The following warning messages indicate that you do not have the level of privileges necessary to gain access to the real-time system resources:

  Solaris OS messages:

  Warning: insufficient effective privileges; temporal behavior is not
           predictable
  Warning: You don't have the privilege to use high-resolution timers
  Warning: Switching to the low resolution (10ms) timers
  Warning: Disabling the cyclic driver as the high-resolution timers
           are not used
  Warning: unable to lock pages into
  Warning: Cannot use the real-time scheduling class. memory
  

  Linux OS messages:

  Warning: unable to lock pages into memory
  Warning: Cannot use the real-time scheduling class.
  

  You must have been granted the appropriate privileges in order to gain access to the system resources that are necessary for ensuring the deterministic behavior of a real-time application. The Java RTS Installation Guide describes this procedure for each operating system:

  • Solaris OS: Granting Resource Access Privileges to Java RTS Users (Solaris OS)
  • Linux OS: Granting Resource Access Privileges to Java RTS Users (Linux OS)

• Failure due to access checks or assignment checks

  Programs that fail because of access checks or assignment checks are particularly hard to debug. Such situations are further complicated when an exception is replaced by a ThrowBoundaryError when it tries to escape its scope. The following options can be useful in these conditions:

  • -XX:+RTSJTraceAccessChecks
  • -XX:+VMStackTraceAtRTSJException
  • -XX:+RTSJIgnoreThrowBoundaryError (see caution note below)

  Caution: If you activate the RTSJIgnoreThrowBoundaryError flag, do so only temporarily to obtain information that is hidden because of the replacement of an exception by a ThrowBoundaryError. Activation of this flag might lead to core dumps or other undefined behavior since the concerned object might be deleted while it is still in use. Always disable this option before making a bug report.

• OutOfMemoryError

  Note that, with Java RTS, OutOfMemoryError (OOME) might not mean out of heap memory, but out of current allocation context memory. The message part of the exception gives an indication of the type of memory that was exhausted.

  Static initializers are executed in ImmortalMemory. Hence, any object that is allocated in an initialization phase becomes immortal and is never collected. If that object refers to other heap objects, they are also prevented from being collected. This can cause an eventual OOME in Immortal space or Java heap space. One solution is to change the code to use non-static variables.

• Jitter in Deterministic Code

  Real-time applications do not require being deterministic 100% of the time. During certain time-critical phases, the application must be determinstic, that is, unbounded jitter in latency or execution time is a failure of the real-time application. When the application is in this phase, we call it "steady mode." On the other hand, during other phases of the application, for example, initialization, reconfiguration, test, and so forth, unbounded jitter can be tolerated.

  When debugging a real-time application, you can specify various sections of "steady mode" code in your application with the "enter" and "exit" methods of the SteadyMode class in the extended RTSJ package. Then, if the -XX:+LogJitterProneEvents parameter is specified on the command line, these "enter" and "exit" events are written, with a time stamp, to a special log file. This can help you determine if jitter is occurring in an area of code that is supposed to be determinstic.

  See Command-Line Options for a description of the -XX:+LogJitterProneEvents parameter.

• Throughput penalty due to cross calls

  When a compiled method calls an interpreted method, or vice-versa, this is known as a "cross call." Such calls might produce a significant (and unexpected) throughput penalty.

  Within HotSpot, methods that call each other are compiled at approximately the same time, resulting in optimal performance. With ITC, users must be more careful.

  Unlike other flavors of threads, NoHeapRealtimeThread threads (NHRTs) cannot be subjected to JIT compilation. Therefore, if some methods executed by NHRTs are not in the user-provided ITC compilation list, both determinism and throughput might be compromised. For example, a given list could produce a set of methods whose execution is slower than the same set of entirely interpreted methods.

  For other threads, JIT compilation (if not disabled) will take care of this issue in the long term, but the application will suffer from jitter and throughput problems in the initialization phase if the ITC list leads to too many cross calls.

  For detailed information about JIT, ITC, and the ITC compilation list, see the Compilation Guide.

• Useful information about thread creation

  In HotSpot, a native thread is not created until the java.lang.Thread object is start()ed. Likewise, for RealtimeThreads and NoHeapRealtimeThreads, the native thread is created at start time. However, some RTTs and NHRTs are created to populate the server pools used by the Asynchronous Event Handler. (They are created when the AsyncEventHandler class is initialized, before the first fire() can be done.) Therefore, you might see created but not started NHRTs listed in the thread dump.

• Process seems to be hanging

  If a process seems to be hanging, it could be blocked or in a deadlock situation or looping or consuming CPU without making apparent forward progress. Try taking the following steps:

  • First use prstat -L -p <pid> to see which threads are using the CPU.
  • If a process is still hanging, add the -XX:+PrintGC option to see whether the GC or the application might be looping.
  • Use pstack -L or dbx to generate stack traces for all the threads.
  • Use the jrts DTrace probes. See the section DTrace Probe Provider (jrts) for Java RTS (Solaris OS only).

• Signal Handling

  Java RTS supports the following signals from the POSIXSignalHandler API:

  INT QUIT ILL TRAP EMT FPE BUS SEGV SYS PIPE ALRM TERM USR1 USR2 CLD CONT TTIN TTOU IOT TSTP

  However, some restrictions apply. We see the POSIXSignalHandler API as a way to communicate from outside of the VM to the Java RTS application. Thus, you can attach handlers to those signals and be notified when signals are sent from outside of the VM. When the signal is sent from the VM itself, not all those signals are supported.

[Contents]

Glossary of Terms

Determinism - The predictability of a real-time application.

ITC - Initialization Time Compilation.

JIT - Just-In-Time compilation.

Jitter - The variation in response time or execution time of a real-time application.

JLT - An instance of the java.lang.Thread class.

NHRT - An instance of the javax.realtime.NoHeapRealtimeThread class.

RTGC - Java RTS Real-Time Garbage Collector

RTT - An instance of the javax.realtime.RealtimeThread class.

[Contents]