This document is a guide to help troubleshoot problems that might occur in applications that are developed with Sun Java Real-Time System (Java RTS). The guide concentrates on issues that are specific to the Java RTS VM. For problems with applications that are developed on the Java SE 5.0 platform, with the Java HotSpot VM, see the Java 2 Platform Standard Edition 5.0 Trouble-Shooting and Diagnostic Guide (PDF). Note that some procedures are platform-specific: Solaris Operating System (Solaris OS) or Linux Operating System. [Java RTS Readme: links to all Java RTS technical documents] ContentsIntroductionApplication Determinism, Jitter, Missed DeadlinesTroubleshooting Tools and Options IntroductionThis document represents an accumulation of information that can help you resolve problems that might occur with your Java RTS application. A number of specialized tools have been developed, and this activity is on-going. Certain problems can be resolved by correctly tuning the application. And finally, the document presents some troubleshooting tips. This document is under continual improvement and extension, and all comments, suggestions, and ideas are welcome on our Evaluation Feedback Form. Application Determinism, Jitter, Missed DeadlinesThe hallmark of a real-time application is its determinism, which means that deadlines must not be missed. Determinism can be measured in terms of jitter, where jitter is defined as the variation in response time or execution time of a real-time application. The response time is calculated from the time you send a request to a real-time server to the time you receive an answer. This time is an accumulation of the following times:
You can use the If you are not able to find the cause for the jitter, then it might be the case that the application code is indeed not deterministic. This can be due to non-deterministic optimizations, or to calls non-deterministic code (for example, JDK libraries, I/O, and so forth). The document A Practical Introduction to Achieving Determinism is part of a "Getting Started" package that shows how to quickly and easily achieve determinism. This document contains a discussion of the usual sources of jitter, as well as a description of how to use the sample programs and script that are provided in the package to achieve determinism by eliminating these sources of jitter. Take a look at this document to get ideas for troubleshooting jitter issues. Information to CollectThis section suggests some of the information to collect in order to troubleshoot a problem.
How to Determine the Source of the ProblemBefore starting to troubleshoot a possible problem, make a attempt to categorize the problem you are experiencing. This will help you identify the cause of the problem and ultimately find a solution. It is also important to determine what is not a problem in order to clarify the situation. One of the first issues to consider is your configuration.
The next important step is to check how you measured the different variables in your system. In particular, the measurement itself must not create additional jitter.
The following tips can help simplify and verify your troubleshooting procedure:
Other Helpful Documentation for TroubleshootingIn addition to this document, the following documents will be helpful:
BlogsIndividual blog sites and blog entries have proven to be very helpful by providing ideas and tips for solving problems. Here are a few such blogs:
Troubleshooting Tools and OptionsThis section introduces several tools that can be used to troubleshoot problems in your Java RTS application. Some command-line options are also suggested. OverviewJava SE 5.0 (also called J2SE 1.5) provides a set of experimental and unsupported tools for monitoring and management as well as tools for troubleshooting. However, the design of the the Real-Time Garbage Collector (RTGC) and the no-heap real-time threads (NHRTs) introduced differences between Java RTS and HotSpot, for example, in the memory model, the stack design, and other issues. For this reason, some of the options of some of the tools cannot work with Java RTS applications that use the RTGC or NHRTs. The Java RTS team is continually improving the architecture of the system in order to be able to take full advantage of these tools. Here is a summary of the current availability of the Java SE 5.0 tools with Java RTS:
For related information, see the Monitoring and Management for the Java Platform guide (Java SE 5.0) and the Java 2 Platform Standard Edition 5.0 Trouble-Shooting and Diagnostic Guide (PDF). In addition, Java RTS provides these specialized tools:
Command-Line OptionsThe following command-line options are useful for troubleshooting:
See the Java RTS Command-Line Options page for a list of all the -XX command-line options for Java RTS, and the Java HotSpot VM Options page for a list of all the HotSpot -XX command-line options. Balancing Determinism and DebuggingThe use of some of the monitoring tools or supporting library calls in Java RTS can lead to determinism issues. For instance, Java RTS must suspend the application to take a snapshot of the thread states and look for deadlocks. By default, determinism is considered to be more important than obtaining all the information that HotSpot is able to provide. However, during application development you might be more interested in the debugging functionalities than in determinism. Therefore, Java RTS provides mechanisms to select between a very deterministic deployment mode and a less deterministic development or debugging mode. This currently impacts only the stack trace information and the thread state snapshots. Both are controlled by a single parameter that can be changed in two ways:
Debugging and ProfilingAs mentioned above, monitoring and profiling a real-time application can impact its determinism. However, the following tools can be used to monitor a real-time application, even in a production environment:
Other profilers are usually so intrusive that they are of little use in a real-time environment. Java RTS supports the Java Platform Debugger Architecture (JPDA), which includes the following interfaces: the Java Debug Interface (JDI), the Java Debug Wire Protocol (JDWP), and the Java Virtual Machine Tool Interface (JVMTI). Since the Java Virtual Machine Profile Interface is a deprecated interface as of Java SE 5.0, it is not supported in Java RTS; instead, use the JVMTI. The NetBeans Integrated Development Environment (IDE) implements these interfaces for debugging, and can be used to debug applications running on Java RTS. This release of Java RTS is compatible with NetBeans 5.5.1. Note: Java RTS does not support NetBeans profiling because some of the methods break real-time behavior. JPDA provides a comprehensive set of features for debugging an application, at the expense of impairing its temporal behavior during the debugging session. Once the debugger is attached to the Java RTS virtual machine, it is possible to stop, inspect, and resume any type of schedulable that is being executed, that is java.lang.Thread, RealtimeThread, NoHeapRealtimeThread and AsyncEventHandler. It is also possible to debug schedulables executing in scoped memory. However, once the debugger is attached, the temporal behavior of the application is no longer guaranteed, and there is therefore no need to support the Real-Time Garbage Collector (RTGC). In addition, most debugging operations require the suspension of threads. These suspensions impact all the running threads, including the NoHeapRealtimeThreads and the high-priority RealtimeThreads. Therefore, when debug options are specified on the command line, Java RTS automatically activates the non-real-time mark-and-sweep GC. DTrace Probe Provider (jrts) for Java RTS (Solaris OS only)The DTrace probe provider for Java RTS is called jrts. With these probes you can observe, debug, and tune your Java RTS real-time application. Without any modification to the running application, you can monitor activities in several categories, including priority changes, memory allocation contexts, thread execution and interruption, event handlers and server threads, compilation, symbol and method resolution, real-time garbage collection, class loading and initialization, and VM operations. Finally, a special class allows you to fire a general-purpose probe in order to track any event within the application that you choose. The document Java RTS DTrace Provider contains a detailed description of these probes and how to use them, as well as several sample scripts. Thread Scheduling VisualizerIn troubleshooting a multi-threaded real-time application, it can be very useful to see the actual scheduling of all the threads. The Thread Scheduling Visualizer (TSV) is a set of tools that provide an easy way to record and visualize thread scheduling for Java RTS. The document Java RTS Thread Scheduling Visualizer contains a detailed description of this tool set, including how to record scheduling events using customizable DTrace scripts that are provided, as well as how to use the Visualizer GUI to get a graphical view of the events. You can download the TSV tool set from the Java RTS Evaluation Downloads page. The following examples show the types of debugging information TSV can provide (taken from a real-life experience):
Before using TSV with Java RTS, ensure that you have the required resource access privileges, as described in the Java RTS Installation Guide. Otherwise, TSV fails with messages such as "invalid probe specifier" or other messages indicating problems with the set of probes. jmap Utility for Java RTSThe When no option is used, jmap prints shared object mappings for each shared object loaded in the target VM: start address, size of the mapping, and full path of the shared object file. Each option produces different output: a heap summary, a heap dump in binary format, a heap histogram, or permanent generation statistics. See the Note that, in the sample output listings below, some lines have been broken into two or more for displayability, and some lines have been removed to reduce the volume of the output for this document. jmap output: Heap summary with the -heap optionThe sample output below is from the % jmap -heap <Java RTS install dir>/bin/java core.5626 Attaching to core core.4921 from executable /net/mackdrive/export/binaries/solaris-sparc/bin/java, please wait... Debugger attached successfully. Client compiler detected. JVM version is 1.5.0_13-b05 Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 67108864 (64.0MB) NewSize = 655360 (640.0KB) MaxNewSize = 4294967295 (4095.99MB) OldSize = 1441792 (1.375MB) NewRatio = 12 SurvivorRatio = 32 PermSize = 67108864 (64.0MB) MaxPermSize = 67108864 (64.0MB) Real-Time Parameters Configuration: UseRTGC = 1 ImmortalSize = 33554432 (32.0MB) ImmortalPhysicalReservedSize = 1048576 (1.0MB) ScopedSize = 100663296 (96.0MB) ScopedPhysicalReservedSize = 1048576 (1.0MB) ScopedMemoryAllocGrain = 16384 (16.0KB) RTGCCriticalReservedBytes = 0 (0B) RTGCWaitDuration = 10 RTGCCriticalBoundary = 40 RTGCBoostedWorkers = 2 RTGCBoostedPriority = 40 BoostedMinFreeBytes = 0 (0B) BoostedSlideFactor = 10 BoostedSafetyMargin = 20 RTGCNormalWorkers = 2 RTGCNormalPriority = 11 NormalMinFreeBytes = 0 (0B) NormalSlideFactor = 20 NormalSafetyMargin = 10 Heap Usage: RT Collected Heap: capacity = 67108864 (64.0MB) used = 338944 (331.0KB) free = 66769920 (63.67MB) 0.50% used Non-heap memory areas: Scoped memory block 1: capacity = 16777216 (16.0MB) used = 7850224 (7.48MB) free = 8926992 (8.51MB) 46.79% used Scoped memory block 2: capacity = 16777216 (16.0MB) used = 7794224 (7.43MB) free = 8982992 (8.56MB) 46.45% used Immortal memory block: capacity = 33554432 (32.0MB) used = 16377608 (15.61MB) free = 17176824 (16.38MB) 48.80% used Unallocated scoped memory chunk: size = 33554432 (32.0MB) Total unallocated scoped memory: size = 33554432 (32.0MB) jmap output: Heap dump in binary format with the -heap:format=b optionThe sample output below is from the % jmap -heap:format=b <Java RTS install dir>/bin/java core.4921 Attaching to core core.4921 from executable /net/mackdrive/export/binaries/solaris-sparc/bin/java, please wait... Debugger attached successfully. Client compiler detected. JVM version is 1.5.0_13-b05 heap written to heap.bin jmap output: Heap histogram with the -histo optionThe sample output below is from the % jmap -histo <Java RTS install dir>/bin/java core.4921 Attaching to core core.4921 from executable /net/mackdrive/export/binaries/solaris-sparc/bin/java, please wait... Debugger attached successfully. Client compiler detected. JVM version is 1.5.0_13-b05 Iterating over heap. This may take a while... Object Histogram: Size Count Class description ------------------------------------------------------- 840464 9 long[] 481168 858 int[] 219168 1826 char[] 154584 574 byte[] 120360 4 double[] 64704 2696 Fibonacci1 64704 2696 Fibonacci2 56888 547 java.lang.Class 56064 1752 java.lang.String 55192 837 java.lang.Object[] 51160 824 short[] [...lines removed to reduce output ... ] jmap output: Permstat statistics with the -permstat optionThe sample output below is from the % jmap -permstat <Java RTS install dir>/bin/java core.4921 Attaching to core core.4921 from executable /net/mackdrive/export/binaries/solaris-sparc/bin/java, please wait... Debugger attached successfully. Client compiler detected. JVM version is 1.5.0_13-b05 finding class loader instances ..done. computing per loader stat ..done. please wait.. computing liveness.done. class_loader classes bytes parent_loader alive? type <bootstrap> 470 1353216 null live <internal> 0xf44205a8 40 190064 0xf4419af0 live sun/misc/Launcher$AppClassLoader@0xf05a6bb0 0xf4419af0 0 0 null live sun/misc/Launcher$ExtClassLoader@0xf05811f8 total = 3 510 1543280 N/A alive=3, dead=0 N/A jstack Utility for Java RTSThe Some private symbols of the JVM are not recognized by See the Each thread is specified as an instance of Note that, in the sample output listings below, some lines have been broken into two or more for displayability, and some lines have been removed to reduce the volume of the output for this document. jstack output: Thread stack with no optionsBelow is sample output from the % jstack <Java RTS install dir>/bin/java core.4921 Attaching to core core.4921 from executable /net/mackdrive/export/binaries/solaris-sparc/bin/java, please wait... Debugger attached successfully. Client compiler detected. [...lines removed to reduce output ... ] java.lang.Thread t@38: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=474 (Interpreted frame) - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=121 (Interpreted frame) [...lines removed to reduce output ... ] javax.realtime.RealtimeThread t@46: (state = IN_JAVA) - Fibonacci2.recursiveFib(int) @bci=17, line=26 (Interpreted frame) - Fibonacci2.recursiveFib(int) @bci=18, line=26 (Interpreted frame) - Fibonacci2.recursiveFib(int) @bci=18, line=26 (Interpreted frame) - Fibonacci2.computeFib() @bci=17, line=35 (Interpreted frame) - Deterministic.computeFibs(int, int) @bci=35, line=99 (Interpreted frame) - Deterministic.run() @bci=19, line=124 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=713 (Interpreted frame) [...lines removed to reduce output ... ] jstack output: Thread stack with mixed mode (-m) optionWith the % jstack -m <Java RTS install dir>/bin/java core.4921 Attaching to core core.4921 from executable /net/mackdrive/export/binaries/solaris-sparc/bin/java, please wait... Debugger attached successfully. Client compiler detected. [...lines removed to reduce output ... ] ----------------- t@1 ----------------- 0xff340ec8 ioctl + 0x8 0xfe919134 JVM_MonitorWait + 0x31c 0xf880e11c * java.lang.Object.wait(long) bci:0 (Interpreted frame) 0xf88062c8 * java.lang.Thread.join(long) bci:38 line:1302 (Interpreted frame) 0xf88062c8 * java.lang.Thread.join() bci:2 line:1355 (Interpreted frame) 0xf88062c8 * Deterministic.main(java.lang.String[]) bci:1428 line:409 (Interpreted frame) 0xf8800278 <StubRoutines> 0xfe8feba4 void JavaCalls::call_helper(JavaValue*,methodHandle*, JavaCallArguments*,Thread*) + 0x674 0xfe9e5308 jni_CallStaticVoidMethod + 0x3d8 0x00012224 main + 0x115c 0x000110b0 _start + 0x108 ----------------- t@2 ----------------- 0xff340ec8 ioctl + 0x8 0xff340368 _lwp_start [...lines removed to reduce output ... ] Java VisualVM with Java RTSJava VisualVM is a graphical tool for the monitoring, management, and profiling of Java applications. The tool can analyze a running process or a core file. An application developed to run with the Java RTS VM can have certain constraints that interfere with the activity of Java VisualVM. In particular, the profiling function of Java VisualVM does not work with a Java RTS application because the profiling uses dynamic instrumentation of the application, creating overhead, and thus adversely affecting the determinism of a real-time application. Java VisualVM must run with Java SE 6 (JDK 6). However, it can display information on any application running with Java RTS. The following functions of Java VisualVM work with Java RTS:
For a detailed description of Java VisualVM and how to use it, see http://java.sun.com/javase/6/docs/technotes/guides/visualvm/index.html Tuning InformationFor a non-real-time environment, the primary performance criterion is the throughput of the system. However, in a real-time environment, execution of time-critical code must be deterministic, that is, predictable, and this means reducing the worst-case execution time to an acceptable level. To avoid unpredictable delays, consider tuning the compilation modes for your application or the activity of the Real-Time Garbage Collector, as described in the subsections that follow. Compilation TuningInterpreter and compiler activity can cause unpredictable delays, as follows:
Java RTS currently supports two compilation modes: Just-In-Time Compilation (JIT) and Initialization Time Compilation (ITC). JIT compilation is the usual compilation mode of a Java virtual machine. With the ITC scheme in Java RTS, you can specify certain time-critical classes and methods to be preloaded, preinitialized, or precompiled. You can tune this compilation behavior to improve your application's determinism. As a general rule, you should preload time-critical classes that contain symbols that are likely to be referenced during execution, you should preinitialize classes containing time-critical code that will be executed, and you should precompile time-critical methods that will be executed. To facilitate your work, Java RTS can automatically generate lists of these classes and methods for you. Moreover, you can configure different compilation modes for different
thread types, that is, instances of Note that, by default, JIT compilation is performed asynchronously. Java code is executed interpreted until one of the internal counters triggers a request to compile the method. The method continues running interpreted while the JIT compilation is handled by the compilation thread. Once the compilation is complete, the application thread continues execution with the JIT-compiled code. However, the compilation thread runs at the first real-time priority, so if the CPUs are all busy doing work at a higher real-time priority the compilation thread will not get a chance to run, code will never get compiled, and threads will keep running interpreted. Therefore, be sure that all critical methods are precompiled, as described above. The Java RTS Compilation Guide contains a very detailed description of all these tuning mechanisms and the relevant command-line parameters. In addition, the document Practical Introduction to Achieving Determinism describes in detail a sample program that can be made deterministic by correctly configuring the compilation mode, among other variables. Real-Time Garbage Collector TuningJava RTS introduces a new garbage collector, the Real-Time Garbage Collector (RTGC). The RTGC is the default GC in Java RTS. The RTGC advances determinism, but at the cost of global throughput. If your application
is more sensitve to the collector's throughput than to the pause times caused by
the collector's execution, then you can disable the RTGC with the
command-line option The RTGC can be configured to a very advanced degree, allowing you to precisely tune factors such as when garbage collection should start and at what priority relative to other threads. Once again, this tuning allows you to balance throughput against determinism. The Java RTS Garbage Collection Guide contains a very detailed description of all these tuning mechanisms and the relevant command-line parameters. However, there is one parameter that should be configured:
The RTGC has an auto-tuning mechanism that dynamically tries to find the best balence between throughput and determinism. Many of the factors that are used in this process are command-line parameters that you can configure in order to adapt the RTGC activity to your needs. Again, the Java RTS Garbage Collection Guide describes this process in great detail. In addition, the document Practical Introduction to Achieving Determinism describes in detail a sample program that can be made deterministic by introducing the use of the RTGC, among other variables. General Troubleshooting TipsThis section presents some general tips and hints to use in troubleshooting.
Glossary of Terms
|