Using JRockit JDK

Monitoring Thread Activity With Thread Dumps

Thread dumps, or "thread stack traces," reveal information about an application's activity that can help you diagnose problems and better optimize application and JVM performance; for example, thread dumps can show the occurrence of "deadlock" conditions, which can seriously impact application performance.

You can create a thread dump by invoking a control break (usually by pressing Ctrl-Break or Ctrl-\ or SIGQUIT on linux). This section provides information on working with thread dumps. It includes information on these subjects:

Lock Information in Thread Dumps

When printing stack traces with Control-Break or SIGQUIT on linux, BEA JRockit also shows the status of active locks (monitors). For each thread, BEA JRockit prints the following information if the thread is in a waiting state:

If the thread is trying to take a lock (to enter a synchronized block), but the lock is already held by another thread, this is indicated at the top of the stack trace, as "Blocked trying to get lock".
If the thread is waiting for a notification on a lock (by calling Object.wait()), this is indicated at the top of the stack trace as "Waiting for notification".
If the thread has taken any locks, this is shown in the stack trace. After a line in the stack trace describing a function call is a list of the locks taken by the thread in that function. This is described as ^-- Holding lock (where the ^-- serves as a reminder that the lock is taken in the function written above the line with the lock).

Caution: The lines with the lock information might not always be correct, due to compiler optimizations. This means two things:

If a thread, in the same function, takes first lock A and then lock B, the order in which they are printed is unspecified.
Sometimes, if a thread, in method foo() calls method bar(), and takes a lock A in bar(), the lock might be printed as being taken in foo().

Normally, this shouldn't be a problem. The order of the lock lines should never move much from their correct position. Also, lock lines will never be missing—you can be assured that all locks taken by a thread are shown in the stack dump.

The semantics for waiting (for notification) on an object in Java is somewhat complex. First you must take the lock for the object, and then you call wait() on that object. In the wait method, the lock is released before the thread actually goes to sleep waiting for a notification. When it receives a notification, wait re-takes the lock before returning. So, if a thread has taken a lock, and is waiting (for notification) on that lock, the line in the stack trace that describes when the lock was taken is not shown as "Holding lock," but as "Lock released while waiting."

All locks are described as Classname@0xLockID[LockType]; for example:

java/lang/Object@0x105BDCC0[thin lock]

Where:

Classname@0xLockID describe the object the to which the lock belongs. The classname is an exact description, the fully qualified class name of the object. LockID, on the other hand, is a temporary ID which is only valid for a single thread stack dump. That is, you can trust that if a thread A holds a lock java/lang/Object@0x105BDCC0, and a thread B is waiting for a lock java/lang/Object@0x105BDCC0, in a single thread stack dump, then it is the same lock. If you do any subsequent stack dumps however, LockID is not comparable and, even if a thread holds the same lock, it might have a different LockID and, conversely, the same LockID does not guarantee that it holds the same lock.
LockType describes the kind of BEA JRockit internal lock type the lock is. Currently, three kinds of locks exist:

fat locks: locks with a history of contention (several threads trying to take the lock simultaneously), or that have been waited on (for notification).
thin locks: locks that have had no contention.
recursive locks: locks occur when a thread takes a lock it already holds.

Listing 3-1 shows an example of what a stack trace for a single thread can look like.

Listing 3-1 Example: Stack Trace for a Single Thread

"Open T1" prio=5 id=0x680 tid=0x128 waiting
   -- Waiting for notification on: java/lang/Object@0x1060FFC8[fat lock]
   at jrockit/vm/Threads.waitForSignalWithTimeout(Native Method)@0x411E39C0
   at jrockit/vm/Locks.wait(Locks.java:1563)@0x411E3BE5
   at java/lang/Thread.sleep(Thread.java:244)@0x41211045
   ^-- Lock released while waiting: java/lang/Object@0x1060FFC8[fat lock]
   at test/Deadlock.loopForever(Deadlock.java:67)@0x412304FC
   at test/Deadlock$LockerThread.run(Deadlock.java:57)@0x4123042E
   ^-- Holding lock: java/lang/Object@0x105BDCC0[recursive]
   ^-- Holding lock: java/lang/Object@0x105BDCC0[thin lock]
   at java/lang/Thread.startThreadFromVM(Thread.java:1690)@0x411E5F73
   --- End of stack trace

Detecting Deadlocks

After the normal stack dumps, BEA JRockit performs a deadlock detection. This is done by finding "lock chains" in the Java application. If a lock chain is found to be circular, the application is considered caught in a deadlock.

What is a "Lock Chain"?

Although they appear somewhat complex, lock chains are fairly straightforward. Informally, lock chains can be described as a sequence of threads, each waiting for a lock held by the next thread in the chain. An open lock chain ends with a thread that is not trying to take a lock, but is instead doing actual work or possibly waiting for some external event. A circular chain is a deadlock—it will never be resolved. A closed chain depends on another lock chain, and is in effect deadlocked if the other chain is deadlocked, and open if the other chain is open. Closed chains mean that several threads are trying to take the same lock.

Formal Definition of a Lock Chain

If thread Tb holds lock Lb, and thread Ta is trying to take lock Lb, then they form the lock chain Ta->Tb.

If Ta->Tb is a lock chain, and thread Tc is holding the lock Lc, which thread Tb is trying to take, then Ta->Tb->Tc is also a lock chain.

If Ta->...->Tn is a lock chain, and there exist no lock La held by Ta and a thread Tx such that Tx is trying to take La, then the lock chain is starting at Ta.

If Ta->...->Tn is a lock chain starting at Ta, and there exist no lock Lx such that thread Tn is trying to take Lx, then Ta->...->Tn is an open lock chain, ending on Tn.

If Ta->...->Tn is a lock chain starting at Ta, and thread Tn is trying to take lock Lo, which is held by a thread To, and thread To is involved in a separate, complete lock chain, then Ta->...->Tn is a closed lock chain, ending on Tn.

If Ta->...->Tn is a lock chain, and thread Tn is trying to take lock La held by thread Ta, then Ta->...->Tn is a circular (deadlocked) lock chain.

A lock chain is complete if it is either an open, closed or circular lock chain.

From the definitions follows that all threads that are trying to take a lock belong to exactly one complete lock chain

Lock Chain Dump

BEA JRockit will find all complete lock chains, and will group them into open, closed and circular lock chains. All open and closed lock chains will be printed from their starting elements to their end. Circular lock chains has neither a start nor an end—BEA JRockit will chose an element arbitrarily and treat it like the start.

The division between a closed lock chain and the other lock chain is arbitrary. Closed chains arise whenever two different threads are blocked trying to take the same lock; for example: Thread A holds lock Lock A while Thread B is waiting for Lock A; Thread C is also waiting for Lock A. BEA JRockit will interpret this in one of the following ways:

B > A as an open lock chain and C > A as a closed lock chain.
C > A as an open lock chain and B > A as a closed lock chain.

A deadlocked lock chain can never be resolved, and the application will be stuck waiting indefinitely. If you have long (but open) lock chains, your application might be spending unnecessary time waiting for locks.