This chapter describes how to get and use Oracle JRockit JVM thread dumps. For basic information about threads and thread synchronization, see Oracle JRockit Introduction to the JDK.
A thread dump is a snapshot of the state of all threads that are part of the process. The state of each thread is presented with a stack trace, which shows the contents of a thread's stack. Some of the threads belong to the Java application you are running, while others are JVM internal threads.
A thread dump reveals information about an application's thread activity that can help you diagnose problems and better optimize application and JVM performance; for example, thread dumps automatically show the occurrence of a deadlock. Deadlocks bring some or all of an application to a complete halt.
This chapter contains the following sections:
To create a thread dump from a process, do either of the following:
Press Ctrl-Break while the process is running (or send SIGQUIT to the process on Linux and Solaris).
Enter the following at the command line:
bin\jrcmd.exe pid print_threads
The thread dump is displayed at the command line.
Note:
For more information about thejrcmd command and Ctrl-Break handlers, see Section 4, "Running Diagnostic Commands."This section describes the typical contents of a thread dump. An example thread dump from beginning to end is shown. Example 3-1, Example 3-2, Example 3-3, Example 3-4, and Example 3-5 show the components of a thread dump). Next, information about the main thread is shown, then all the JVM internal threads, followed by all other Java application threads (if there are any). Finally, information about lock chains is shown.
The example thread dump is taken from a program that creates three threads that are quickly forced into a deadlock. The application threads Thread-0, Thread-1, and Thread-2 correspond to three different classes in the Java code.
The thread dump starts with the date and time of the dump, and the version number of the JRockit JVM used. See Example 3-1.
Example 3-2 shows the stack trace of the main application thread. There is a thread information line, followed by information about locks and a trace of the thread's stack at the moment of the thread dump.
Example 3-2 Main Thread in the Thread Dump
"Main Thread" id=1 idx=0x2 tid=48652 prio=5 alive, in native, waiting -- Waiting for notification on: util/repro/Thread1@0x01226528[fat lock] at jrockit/vm/Threads.waitForSignal(J)Z(Native Method) at java/lang/Object.wait(J)V(Native Method) at java/lang/Thread.join(Thread.java:1095) ^-- Lock released while waiting: util/repro/Thread1@0x01226528[fat lock] at java/lang/Thread.join(Thread.java:1148) at util/repro/DeadLockExample.main(DeadLockExample.java:23) at jrockit/vm/RNI.c2java(IIII)V(Native Method) -- end of trace
After the name and other identification information, the different status messages of the main application thread are printed. The main application thread in Example 3-2 is a running thread (alive). It is either executing JVM internal code or user-defined JNI code (in native). It is currently waiting for an object to be released (waiting). If a thread is waiting for a notification about a lock (by calling Object.wait()), this is indicated at the top of the stack trace (Waiting for notification on).
For each thread, the JRockit JVM prints the following information:
If the thread is trying to take a lock (to enter a synchronized block), but the lock is already held by another thread, this is indicated at the top of the stack trace, (Blocked trying to get lock).
If the thread is waiting for a notification about a lock (by calling Object.wait()), this is indicated at the top of the stack trace (Waiting for notification).
If the thread has taken locks, this is shown in the stack trace. After a line in the stack trace describing a function call is a list of the locks taken by the thread in that function. This is described as ^-- Holding lock where the ^-- is a reminder that the lock is taken in the function written above the line with the lock.
The semantics for waiting (for notification) for an object in Java is somewhat complex. First, to enter a synchronized block, you must take the lock for the object, and then you call wait() on that object. In the wait method, the lock is released before the thread goes to sleep waiting for a notification. When the thread receives a notification, wait() re-takes the lock before returning. If a thread has taken a lock, and is waiting for notification about that lock, the line in the stack trace that describes when the lock was taken is not shown as (Holding lock); it is shown as (Lock released while waiting.)
All locks are described as Classname@0xLockID[LockType]; for example:
java/lang/Object@0x105BDCC0[thin lock]
Classname@0xLockID describes the object to which the lock belongs. The classname is an exact description, the fully qualified classname of the object. LockID, is a temporary ID that is valid for a single thread dump. If a thread A holds a lock java/lang/Object@0x105BDCC0, and a thread B is waiting for a lock java/lang/Object@0x105BDCC0 in a single thread dump, then it is the same lock. For subsequent thread dumps, LockID might be different, even if a thread holds the same lock. LockType describes the JVM internal type of the lock (fat, thin, recursive, or lazy). The status of active locks (monitors) is also shown in stack traces.
The lines with the lock information might not be correct due to compiler optimizations. This means two things:
If a thread, in the same function, takes lock A first and then lock B, the order in which they are printed is unspecified.
If a thread, in method edit() calls method save(), and takes a lock A in save(), the lock might be printed as being taken in edit().
Typically, this is not be a problem. The order of the lock lines never move much from their correct positions. Also, lock lines will never be missing— all locks taken by a thread are shown in the thread dump.
Example 3-3 shows the traces of JVM internal threads. The threads have been marked as daemon threads, noted by their daemon state indicators. Daemon threads are either JVM internal threads (as in this case) or threads marked as daemon threads by java.lang.Thread.setDaemon().
Example 3-3 First and Last Thread in a List of JVM Internal Threads
"(Signal Handler)" id=2 idx=0x4 tid=48668 prio=5 alive, in native, daemon [...] "(Sensor Event Thread)" id=10 idx=0x1c tid=48404 prio=5 alive, in native, daemon
Lock information and stack traces are not printed for the JVM internal threads in Example 3-3. This is the default setting.
If you want to see stack traces for the JVM internal threads, then use the parameter nativestack=true when you run the print_threads command. At the command line, enter the following:
bin\jrcmd.exe <pid> print_threads nativestack=true
Typically, you would be interested in the threads of the Java application you are running (including the main thread). All Java application threads, except the main thread, are shown near the end of the thread dump. Example 3-4 shows the stack traces of three different application threads.
Example 3-4 Additional Application Threads
"Thread-0" id=11 idx=0x1e tid=48408 prio=5 alive, in native, blocked -- Blocked trying to get lock: java/lang/Object@0x01226300[fat lock] at jrockit/vm/Threads.waitForSignal(J)Z(Native Method) at jrockit/vm/Locks.fatLockBlockOrSpin(ILjrockit/vm/ObjectMonitor;II)V(Unknown Source) at jrockit/vm/Locks.lockFat(Ljava/lang/Object;ILjrockit/vm/ObjectMonitor;Z)Ljava/lang/Object;(Unknown Source) at jrockit/vm/Locks.monitorEnterSecondStage(Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source) at jrockit/vm/Locks.monitorEnter(Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source) at util/repro/Thread1.run(DeadLockExample.java:34) ^-- Holding lock: java/lang/Object@0x012262F0[thin lock] ^-- Holding lock: java/lang/Object@0x012262F8[thin lock] at jrockit/vm/RNI.c2java(IIII)V(Native Method) -- end of trace "Thread-1" id=12 idx=0x20 tid=48412 prio=5 alive, in native, blocked -- Blocked trying to get lock: java/lang/Object@0x012262F8[thin lock] at jrockit/vm/Threads.sleep(I)V(Native Method) at jrockit/vm/Locks.waitForThinRelease(Ljava/lang/Object;I)I(Unknown Source) at jrockit/vm/Locks.monitorEnterSecondStage(Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source) at jrockit/vm/Locks.monitorEnter(Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source) at util/repro/Thread2.run(DeadLockExample.java:48) at jrockit/vm/RNI.c2java(IIII)V(Native Method) -- end of trace "Thread-2" id=13 idx=0x22 tid=48416 prio=5 alive, in native, blocked -- Blocked trying to get lock: java/lang/Object@0x012262F8[thin lock] at jrockit/vm/Threads.sleep(I)V(Native Method) at jrockit/vm/Locks.waitForThinRelease(Ljava/lang/Object;I)I(Unknown Source) at jrockit/vm/Locks.monitorEnterSecondStage(Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source) at jrockit/vm/Locks.monitorEnter(Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source) at util/repro/Thread3.run(DeadLockExample.java:65) ^-- Holding lock: java/lang/Object@0x01226300[fat lock] at jrockit/vm/RNI.c2java(IIII)V(Native Method) -- end of trace
All three threads are in a blocked state (indicated by blocked), which means that they are all trying to enter synchronized blocks. Thread-0 is trying to take the lock Object@0x01226300[fat lock], but this lock is held by Thread-2. Both Thread-2 and Thread-1 are trying to take Object@0x012262F8[thin lock] but this lock is held by Thread-0. This means that Thread-0 and Thread-2 form a deadlock, while Thread-1 is blocked.
JRockit JVM automatically detects deadlocked, blocked, and open lock chains among the running threads. Example 3-5 shows all the lock chains created by the threads T1, T2, T3, T4, and T5. This information can be used to tune and troubleshoot your Java code.
Example 3-5 Deadlocked and Blocked Lock Chains
Circular (deadlocked) lock chains ================================= Chain 6: "Dead T1" id=16 idx=0x48 tid=3648 waiting for java/lang/Object@0x01225018 held by: "Dead T3" id=18 idx=0x50 tid=900 waiting for java/lang/Object@0x01225010 held by: "Dead T2" id=17 idx=0x4c tid=3272 waiting for java/lang/Object@0x01225008 held by: "Dead T1" id=16 idx=0x48 tid=3648 Blocked lock chains =================== Chain 7: "Blocked T2" id=20 idx=0x58 tid=3612 waiting for java/lang/Object@0x01225310 held by: "Blocked T1" id=19 idx=0x54 tid=2500 waiting for java/lang/Object@0x01224B60 held by: "Open T3" id=13 idx=0x3c tid=1124 in chain 1 Open lock chains ================ Chain 1: "Open T5" id=15 idx=0x44 tid=4048 waiting for java/lang/Object@0x01224B68 held by: "Open T4" id=14 idx=0x40 tid=3380 waiting for java/lang/Object@0x01224B60 held by: "Open T3" id=13 idx=0x3c tid=1124 waiting for java/lang/Object@0x01224B58 held by: "Open T2" id=12 idx=0x38 tid=3564 waiting for java/lang/Object@0x01224B50 held by: "Open T1" id=11 idx=0x34 tid=2876 (active)
This section describes the different statuses or states of a thread in a thread dump:
Table 3-0 describes the life states that a thread can show in a thread dump.
| State | Description |
|---|---|
|
|
This is a typical, running thread. Virtually all threads in the thread dump will be (alive). |
|
|
The thread was requested to start running by |
|
|
This thread has finished its |
Table 3-2 describes the run states that a thread can show in a thread dump.
| State | Description |
|---|---|
|
|
This thread tried to enter a synchronized block, but the lock was taken by another thread. This thread is blocked until the lock gets released. |
|
|
This is the same state as |
|
|
This thread called |
|
|
This thread called |
|
|
This thread called |
|
|
The thread's execution was suspended by |
Note:
Each state in Table 3-2 represents a scenario where the thread is currently blocked (that is, ineligible for scheduling by the operating system) for some (ideally temporary) reason.Java execution threads that are not currently blocked (that is, eligible for scheduling by the operating system to run) will not have any of the above states.
Table 3-3 describes the special states that a thread can show in a thread dump. Note that these states are not mutually exclusive.
Table 3-3 Special Thread States
| State | Description |
|---|---|
|
|
The user called |
|
|
This is either JVM internal thread or a thread that was marked as a daemon thread by |
|
|
This thread is executing native code: either user-supplied JNI code or JVM internal code. |
|
|
This thread is executing JVM internal code and has marked itself as |
|
|
This thread is executing JVM internal code and has tried to take a JVM internal lock. The thread is blocked because that lock is held by another thread. |
|
|
This thread is executing JVM internal code and is waiting for notification from another thread about a JVM internal lock. |
Note:
None of the above states in Table 3-3 by themselves represent a situation where a Java execution thread would be blocked from making progress. By design, when a thread dump is collected, all Java execution threads must be paused momentarily in a known state in order to collect the necessary data (stack traces). As a result, in a thread dump, every Java execution thread will be in one of these states—in native, native_blocked, or native_waiting. As soon as the thread dump is finished, these threads will be able to continue making progress (as long as these threads are not also in one of the states listed in Table 3-2). For analyzing the Java application behavior, these threads should not be considered blocked.This section contains information about using thread dumps for troubleshooting and diagnostics.
To use thread dumps for troubleshooting, beyond detecting deadlocks, you must take several thread dumps from the same process. For analyzing the behavior over longer durations, a better option would be to combine thread dumps with other diagnostic tools, such as the JRockit Flight Recorder, which is part of Oracle JRockit Mission Control.
The Oracle JRockit JVM automatically analyzes the thread dump information and detects whether there exists any circular (deadlocked) or blocked lock chains in it. A blocked lock chain is not a deadlock: it indicates contention.
To detect more than deadlocks in your threads, you must make several consecutive thread dumps. This lets you detect the occurrence of contention, when multiple threads are trying to get the same lock. Contention might create long open lock chains that, while not deadlocked, will degrade performance.
If you discover (in a set of consecutive thread dumps) that one or more threads in your application are temporarily stuck waiting for a lock to be released, then review the code of your Java application to see if the synchronization (serialization) is necessary or if the threads can be organized differently.
By making several consecutive thread dumps, you can get an overview of the parts of your Java application that are used the most. Click the Threads tab in JRockit Management Console for more detailed information about the workload on the different parts of your application.