User Guide

Tracing Thread Activity With Stack Dumps

Stack dumps, or "stack traces," reveal information about an application's thread activity that can help you diagnose problems and better optimize application and JVM performance; for example, stack dumps can show the occurrence of "deadlock" conditions, which can seriously impact application performance.

Stack dumps usually occur when certain errors are thrown. You can also create a stack dump by invoking a control break (usually by pressing Ctrl-Break or Ctrl-\; or SIGQUIT on Linux). This section provides information on working with stack dumps. It includes information on these subjects:

Monitoring Information in Stack Dumps

When printing stack traces with Control-Break, BEA JRockit also shows the status of active locks (monitors). For each thread, BEA JRockit prints the following information if the thread is in a waiting state:

If the thread is trying to take a lock (to enter a synchronized block), but the lock is already held by another thread, this is indicated at the top of the stack trace, as "Blocked trying to get lock".

If the thread is waiting on a notification on a lock (by calling Object.wait()), this is indicated at the top of the stack trace as "Waiting for notification".

If the thread has taken any locks, this is shown in the stack trace. After a line in the stack trace describing a function call is a list of the locks taken by the thread in that function. This is described as ^-- Holding lock (where the ^-- serves as a reminder that the lock is taken in the function written above the line with the lock).

Caution: The lines with the lock information might not always be correct, due to compiler optimizations. This means two things:

If a thread, in the same function, takes first lock A and then lock B, the order in which they are printed is unspecified.
Sometimes, if a thread, in method foo() calls method bar(), and takes a lock A in bar(), the lock might be printed as being taken in foo().

Normally, this shouldn't be a problem. The order of the lock lines should never move much from their correct position. Also, lock lines will never be missing—you can be assured that all locks taken by a thread are shown in the stack dump.

The semantics for waiting (for notification) on an object in Java is somewhat complex. First you must take the lock for the object, and then you call wait() on that object. In the wait method, the lock is released before the thread actually goes to sleep waiting for a notification. When it receives a notification, wait re-takes the lock before returning. So, if a thread has taken a lock, and is waiting (for notification) on that lock, the line in the stack trace that describes when the lock was taken is not shown as "Holding lock," but as "Lock released while waiting."

All locks are described as Classname@0xLockID[LockType]; for example:

java/lang/Object@0x105BDCC0[thin lock]

Where:

Classname@0xLockID describe the object the to which the lock belongs. The classname is an exact description, the fully qualified class name of the object. LockID, on the other hand, is a temporary ID which is only valid for a single thread stack dump. That is, you can trust that if a thread A holds a lock java/lang/Object@0x105BDCC0, and a thread B is waiting for a lock java/lang/Object@0x105BDCC0, in a single thread stack dump, then it is the same lock. If you do any subsequent stack dumps however, LockID is not comparable and, even if a thread holds the same lock, it might have a different LockID and, conversely, the same LockID does not guarantee that it holds the same lock.
LockType describes the kind of BEA JRockit internal lock type the lock is. Currently, three kinds of locks exist:

fat locks: locks with a history of contention, or that have been waited on (for notification).
thin locks: locks that have no contention (several threads trying to take the lock simultaneously).
recursive locks: locks occur when a thread takes a lock it already holds.

Listing C-7 shows an example of what a stack trace for a single thread can look like.

Listing C-7 Example: Stack Trace for a Single Thread

"Open T1" prio=5 id=0x680 tid=0x128 waiting
   -- Waiting for notification on: java/lang/Object@0x1060FFC8[fat lock]
   at jrockit/vm/Threads.waitForSignalWithTimeout(Native Method)@0x411E39C0
   at jrockit/vm/Locks.wait(Locks.java:1563)@0x411E3BE5
   at java/lang/Thread.sleep(Thread.java:244)@0x41211045
   ^-- Lock released while waiting: java/lang/Object@0x1060FFC8[fat lock]
   at test/Deadlock.loopForever(Deadlock.java:67)@0x412304FC
   at test/Deadlock$LockerThread.run(Deadlock.java:57)@0x4123042E
   ^-- Holding lock: java/lang/Object@0x105BDCC0[recursive]
   ^-- Holding lock: java/lang/Object@0x105BDCC0[thin lock]
   at java/lang/Thread.startThreadFromVM(Thread.java:1690)@0x411E5F73
   --- End of stack trace

Detecting Deadlocks

After the normal stack dumps, BEA JRockit performs a deadlock detection. This is done by finding "lock chains" in the Java application. If a lock chain is found to be circular, the application is considered caught in a deadlock.

What is a "Lock Chain"?

Although they appear somewhat complex, lock chains are fairly straightforward; they can be defined as follows:

Threads A and B form a lock chain if Thread A holds a lock that Thread B is trying to take. If A is not trying to take a lock, then the lock chain is "open."
If A->B is a lock chain, and B->C is a lock chain, then A->B->C is a more complete lock chain.
If a Thread D doesn't exist, meaning lock chain C->D doesn't exist, then A->B->C is a complete and open lock chain.

Lock Chain Types

BEA JRockit analyzes the threads and forms complete lock chains. There are three possible kinds of lock chains: Open, Deadlock and Closed lock chains.

Open Chains

Open lock chains represent a straight dependency, as described in What is a "Lock Chain"?. Thread A is waiting for B which is waiting for C, and so on.

Deadlock Chains

Deadlock (circular) chains are similar to an open lock chain, except that the first element is waiting for the last element, in the simplest case: A is waiting for B, which is waiting for A. Note that a deadlocked chain has no head. BEA JRockit selects an arbitrary thread to display as the first element in the chain.

Closed Chains

Closed chains are like open chains, but the first element in the chain is waiting for a lock in another chain. This other chain may be open, deadlocked or closed. If the other chain is deadlocked, then the closed chain is also deadlocked. Note that the division between a closed chain and the other chain is arbitrary.

Closed chains arise whenever two different threads are blocked trying to take the same lock; for example: Thread A holds lock Lock A while Thread B is waiting for Lock A; Thread C is also waiting for Lock A. BEA JRockit will interpret this in one of the following ways:

B —>A as an open lock chain.
C —>A as a closed lock chain.
C —>A as an open lock chain.
B —>A as a closed lock chain.

The only item you might find of interest is if you have a deadlocked lockchain. This can never be resolved, and the application will be stuck waiting indefinitely. Also, if you have long (but open) lock chains, your application might be spending unnecessary time waiting for locks.