Skip Headers
Oracle® JRockit JDK Tools Guide
Release R28

Part Number E15061-06
Go to Documentation Home
Home
Go to Table of Contents
Contents
Go to Feedback page
Contact Us

Go to previous page
Previous
Go to next page
Next
PDF · Mobi · ePub

3 Using Thread Dumps

This chapter describes how to get and use Oracle JRockit JVM thread dumps. For basic information about threads and thread synchronization, see Oracle JRockit Introduction to the JDK.

A thread dump is a snapshot of the state of all threads that are part of the process. The state of each thread is presented with a stack trace, which shows the contents of a thread's stack. Some of the threads belong to the Java application you are running, while others are JVM internal threads.

A thread dump reveals information about an application's thread activity that can help you diagnose problems and better optimize application and JVM performance; for example, thread dumps automatically show the occurrence of a deadlock. Deadlocks bring some or all of an application to a complete halt.

This chapter contains the following sections:

3.1 Creating Thread Dumps

To create a thread dump from a process, do either of the following:

The thread dump is displayed at the command line.

Note:

For more information about the jrcmd command and Ctrl-Break handlers, see Section 4, "Running Diagnostic Commands."

3.2 Reading Thread Dumps

This section describes the typical contents of a thread dump. An example thread dump from beginning to end is shown. Example 3-1, Example 3-2, Example 3-3, Example 3-4, and Example 3-5 show the components of a thread dump). Next, information about the main thread is shown, then all the JVM internal threads, followed by all other Java application threads (if there are any). Finally, information about lock chains is shown.

The example thread dump is taken from a program that creates three threads that are quickly forced into a deadlock. The application threads Thread-0, Thread-1, and Thread-2 correspond to three different classes in the Java code.

3.2.1 The Beginning of the Thread Dump

The thread dump starts with the date and time of the dump, and the version number of the JRockit JVM used. See Example 3-1.

Example 3-1 Initial Information in a Thread Dump

===== FULL THREAD DUMP ===============
Wed Feb 21 13:46:45 2007
Oracle JRockit(R) R28.0.0-109-73164-1.5.0_08-20061129-1428-windows-ia32

3.2.2 Stack Trace for the Main Application Thread

Example 3-2 shows the stack trace of the main application thread. There is a thread information line, followed by information about locks and a trace of the thread's stack at the moment of the thread dump.

Example 3-2 Main Thread in the Thread Dump

"Main Thread" id=1 idx=0x2 tid=48652 prio=5 alive, in native, waiting
-- Waiting for notification on: util/repro/Thread1@0x01226528[fat lock]
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)
at java/lang/Thread.join(Thread.java:1095)
^-- Lock released while waiting: util/repro/Thread1@0x01226528[fat lock]
at java/lang/Thread.join(Thread.java:1148)
at util/repro/DeadLockExample.main(DeadLockExample.java:23)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace

After the name and other identification information, the different status messages of the main application thread are printed. The main application thread in Example 3-2 is a running thread (alive). It is either executing JVM internal code or user-defined JNI code (in native). It is currently waiting for an object to be released (waiting). If a thread is waiting for a notification about a lock (by calling Object.wait()), this is indicated at the top of the stack trace (Waiting for notification on).

3.2.3 Locks and Lock Chains

For each thread, the JRockit JVM prints the following information:

  • If the thread is trying to take a lock (to enter a synchronized block), but the lock is already held by another thread, this is indicated at the top of the stack trace, (Blocked trying to get lock).

  • If the thread is waiting for a notification about a lock (by calling Object.wait()), this is indicated at the top of the stack trace (Waiting for notification).

  • If the thread has taken locks, this is shown in the stack trace. After a line in the stack trace describing a function call is a list of the locks taken by the thread in that function. This is described as ^-- Holding lock where the ^-- is a reminder that the lock is taken in the function written above the line with the lock.

The semantics for waiting (for notification) for an object in Java is somewhat complex. First, to enter a synchronized block, you must take the lock for the object, and then you call wait() on that object. In the wait method, the lock is released before the thread goes to sleep waiting for a notification. When the thread receives a notification, wait() re-takes the lock before returning. If a thread has taken a lock, and is waiting for notification about that lock, the line in the stack trace that describes when the lock was taken is not shown as (Holding lock); it is shown as (Lock released while waiting.)

All locks are described as Classname@0xLockID[LockType]; for example:

java/lang/Object@0x105BDCC0[thin lock]

Classname@0xLockID describes the object to which the lock belongs. The classname is an exact description, the fully qualified classname of the object. LockID, is a temporary ID that is valid for a single thread dump. If a thread A holds a lock java/lang/Object@0x105BDCC0, and a thread B is waiting for a lock java/lang/Object@0x105BDCC0 in a single thread dump, then it is the same lock. For subsequent thread dumps, LockID might be different, even if a thread holds the same lock. LockType describes the JVM internal type of the lock (fat, thin, recursive, or lazy). The status of active locks (monitors) is also shown in stack traces.

3.2.3.1 Presentation of Locks Out of Order

The lines with the lock information might not be correct due to compiler optimizations. This means two things:

  • If a thread, in the same function, takes lock A first and then lock B, the order in which they are printed is unspecified.

  • If a thread, in method edit() calls method save(), and takes a lock A in save(), the lock might be printed as being taken in edit().

Typically, this is not be a problem. The order of the lock lines never move much from their correct positions. Also, lock lines will never be missing— all locks taken by a thread are shown in the thread dump.

3.2.4 JVM Internal Threads

Example 3-3 shows the traces of JVM internal threads. The threads have been marked as daemon threads, noted by their daemon state indicators. Daemon threads are either JVM internal threads (as in this case) or threads marked as daemon threads by java.lang.Thread.setDaemon().

Example 3-3 First and Last Thread in a List of JVM Internal Threads

"(Signal Handler)" id=2 idx=0x4 tid=48668 prio=5 alive, in native, daemon
[...]
"(Sensor Event Thread)" id=10 idx=0x1c tid=48404 prio=5 alive, in native, daemon

Lock information and stack traces are not printed for the JVM internal threads in Example 3-3. This is the default setting.

If you want to see stack traces for the JVM internal threads, then use the parameter nativestack=true when you run the print_threads command. At the command line, enter the following:

bin\jrcmd.exe <pid> print_threads nativestack=true

3.2.5 Other Java Application Threads

Typically, you would be interested in the threads of the Java application you are running (including the main thread). All Java application threads, except the main thread, are shown near the end of the thread dump. Example 3-4 shows the stack traces of three different application threads.

Example 3-4 Additional Application Threads

"Thread-0" id=11 idx=0x1e tid=48408 prio=5 alive, in native, blocked
-- Blocked trying to get lock: java/lang/Object@0x01226300[fat lock]
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
at jrockit/vm/Locks.fatLockBlockOrSpin(ILjrockit/vm/ObjectMonitor;II)V(Unknown Source)
at jrockit/vm/Locks.lockFat(Ljava/lang/Object;ILjrockit/vm/ObjectMonitor;Z)Ljava/lang/Object;(Unknown Source)
at
jrockit/vm/Locks.monitorEnterSecondStage(Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source)
at jrockit/vm/Locks.monitorEnter(Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at util/repro/Thread1.run(DeadLockExample.java:34)
^-- Holding lock: java/lang/Object@0x012262F0[thin lock]
^-- Holding lock: java/lang/Object@0x012262F8[thin lock]
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace

"Thread-1" id=12 idx=0x20 tid=48412 prio=5 alive, in native, blocked
-- Blocked trying to get lock: java/lang/Object@0x012262F8[thin lock]
at jrockit/vm/Threads.sleep(I)V(Native Method)
at jrockit/vm/Locks.waitForThinRelease(Ljava/lang/Object;I)I(Unknown Source)
at jrockit/vm/Locks.monitorEnterSecondStage(Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source)
at jrockit/vm/Locks.monitorEnter(Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at util/repro/Thread2.run(DeadLockExample.java:48)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace

"Thread-2" id=13 idx=0x22 tid=48416 prio=5 alive, in native, blocked
-- Blocked trying to get lock: java/lang/Object@0x012262F8[thin lock]
at jrockit/vm/Threads.sleep(I)V(Native Method)
at jrockit/vm/Locks.waitForThinRelease(Ljava/lang/Object;I)I(Unknown Source)
at jrockit/vm/Locks.monitorEnterSecondStage(Ljava/lang/Object;I)Ljava/lang/Object;(Unknown Source)
at jrockit/vm/Locks.monitorEnter(Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at util/repro/Thread3.run(DeadLockExample.java:65)
^-- Holding lock: java/lang/Object@0x01226300[fat lock]
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace

All three threads are in a blocked state (indicated by blocked), which means that they are all trying to enter synchronized blocks. Thread-0 is trying to take the lock Object@0x01226300[fat lock], but this lock is held by Thread-2. Both Thread-2 and Thread-1 are trying to take Object@0x012262F8[thin lock] but this lock is held by Thread-0. This means that Thread-0 and Thread-2 form a deadlock, while Thread-1 is blocked.

3.2.6 Lock Chains

JRockit JVM automatically detects deadlocked, blocked, and open lock chains among the running threads. Example 3-5 shows all the lock chains created by the threads T1, T2, T3, T4, and T5. This information can be used to tune and troubleshoot your Java code.

Example 3-5 Deadlocked and Blocked Lock Chains

Circular (deadlocked) lock chains
=================================
Chain 6:
"Dead T1" id=16 idx=0x48 tid=3648 waiting for java/lang/Object@0x01225018 held by:
"Dead T3" id=18 idx=0x50 tid=900 waiting for java/lang/Object@0x01225010 held by:
"Dead T2" id=17 idx=0x4c tid=3272 waiting for java/lang/Object@0x01225008 held by:
"Dead T1" id=16 idx=0x48 tid=3648
Blocked lock chains
===================
Chain 7:
"Blocked T2" id=20 idx=0x58 tid=3612 waiting for java/lang/Object@0x01225310 held by:
"Blocked T1" id=19 idx=0x54 tid=2500 waiting for java/lang/Object@0x01224B60 held by: 
"Open T3" id=13 idx=0x3c tid=1124 in chain 1
Open lock chains
================
Chain 1:
"Open T5" id=15 idx=0x44 tid=4048 waiting for java/lang/Object@0x01224B68 held by:
"Open T4" id=14 idx=0x40 tid=3380 waiting for java/lang/Object@0x01224B60 held by:
"Open T3" id=13 idx=0x3c tid=1124 waiting for java/lang/Object@0x01224B58 held by:
"Open T2" id=12 idx=0x38 tid=3564 waiting for java/lang/Object@0x01224B50 held by:
"Open T1" id=11 idx=0x34 tid=2876 (active)

3.3 Thread Status in Thread Dumps

This section describes the different statuses or states of a thread in a thread dump:

3.3.1 Life States

Table 3-0 describes the life states that a thread can show in a thread dump.

Table 3-1 Thread Life States

State Description

alive

This is a typical, running thread. Virtually all threads in the thread dump will be (alive).

not started

The thread was requested to start running by java.lang.Thread.start(), but the actual OS process has not yet started or executed far enough to pass control to the JRockit JVM. It is unlikely to see this value. A java.lang.Thread object that is created, but has not had start() executed, will not show in the thread dump.

terminated

This thread has finished its run() and has also notified any threads joining on it, but it is still kept in the JVM internal thread structure for running threads. It is unlikely to see this value. A thread that has been terminated for a time longer than a few milliseconds will not show in the thread dump.


3.3.2 Run States

Table 3-2 describes the run states that a thread can show in a thread dump.

Table 3-2 Thread Run States

State Description

blocked

This thread tried to enter a synchronized block, but the lock was taken by another thread. This thread is blocked until the lock gets released.

blocked (on thin lock)

This is the same state as blocked, but the lock in question is a thin lock.

waiting

This thread called Object.wait() on an object. The thread will remain there until some other thread sends a notification to that object.

sleeping

This thread called java.lang.Thread.sleep().

parked

This thread called java.util.concurrent.locks.LockSupport.park().

suspended

The thread's execution was suspended by java.lang.Thread.suspend() or a JVMTI agent call.


3.3.3 Special States

Table 3-3 describes the special states that a thread can show in a thread dump. Note that these states are not mutually exclusive.

Table 3-3 Special Thread States

State Description

interrupted

The user called java.lang.Thread.interrupt() on this thread.

daemon

This is either JVM internal thread or a thread that was marked as a daemon thread by java.lang.Thread.setDaemon().

in native

This thread is executing native code: either user-supplied JNI code or JVM internal code.

in suspend critical mode

This thread is executing JVM internal code and has marked itself as suspend critical. Garbage collection is stopped for a specified time period.

native_blocked

This thread is executing JVM internal code and has tried to take a JVM internal lock. The thread is blocked because that lock is held by another thread.

native_waiting

This thread is executing JVM internal code and is waiting for notification from another thread about a JVM internal lock.


3.4 Troubleshooting with Thread Dumps

This section contains information about using thread dumps for troubleshooting and diagnostics.

To use thread dumps for troubleshooting, beyond detecting deadlocks, you must take several thread dumps from the same process. For analyzing the behavior over longer durations, a better option would be to combine thread dumps with other diagnostic tools, such as the JRockit Flight Recorder, which is part of Oracle JRockit Mission Control.

3.4.1 Detecting Deadlocks

The Oracle JRockit JVM automatically analyzes the thread dump information and detects whether there exists any circular (deadlocked) or blocked lock chains in it. A blocked lock chain is not a deadlock: it indicates contention.

3.4.2 Detecting Processing Bottlenecks

To detect more than deadlocks in your threads, you must make several consecutive thread dumps. This lets you detect the occurrence of contention, when multiple threads are trying to get the same lock. Contention might create long open lock chains that, while not deadlocked, will degrade performance.

If you discover (in a set of consecutive thread dumps) that one or more threads in your application are temporarily stuck waiting for a lock to be released, then review the code of your Java application to see if the synchronization (serialization) is necessary or if the threads can be organized differently.

3.4.3 Viewing the Run-time Profile of an Application

By making several consecutive thread dumps, you can get an overview of the parts of your Java application that are used the most. Click the Threads tab in JRockit Management Console for more detailed information about the workload on the different parts of your application.