Generating Thread Dumps

What is a thread dump?

Thread dumps are the basic diagnostic tool for JVMs. Thread dumps present a partial state of each Java thread in a running JVM. I say partial because it doesn’t present all information that captures the state of a Java thread. The state of a running Java thread is stored in a series of data structures that captures activation frames, program counter, execution stack, method’s local variable storage, and others. Each activation frame contains a return value, method arguments, address of a method being called, address of the calling instruction, and others. The thread dump produces a stack trace of each Java thread, which shows the method name of each method called by the thread and the Java class name & line number that called it. A header also gives the Java thread name, state, stack address range, and native thread id. Sometimes the stack traces will also include information about Monitor locks the thread has acquired or is attempting to acquire.

Below is an example of a single thread’s stack trace in a thread dump from a JVM running the Recursion.java program:

“main” prio=1 tid=0x090800a8 nid=0x6552 waiting on condition [0xbfdbc000..0xbfdbc8b8]
at java.lang.Thread.sleep(Native Method)
at Recursion.add2(Recursion.java:27)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.add2(Recursion.java:24)
at Recursion.add1(Recursion.java:9)
at Recursion.main(Recursion.java:38)

This program intentionally creates an infinite recursion situation that alternately calls the methods Recursion.add1() and Recursion.add2(). A depth counter is included to make the main thread sleep for sixty seconds after twenty five method activations. That is when this stack trace was produced.

Generating A Thread Dump

Over the years, I’ve found the following methods helpful in generating thread dumps. Each one has situations where it will not work for one reason or another. For example, if signals are disabled (-Xrs flag), a SIGQUIT signal will not generate a thread dump. So, you’ll have to rely on one of the other methods in this case.

On an IBM & Sun JVM on a Unix-like OS, you can issue a “kill -QUIT pid” or “kill -3 pid” command to generate a thread dump. A Sun JVM will write the thread dump to standard out. An IBM JVM will create a file the working directory of the java process called javacore.date.pid.txt.

For a Sun JVM, it is possible to generate a thread dump, with the jstack command. This has the the added benefit of writing the thread dump to standard out of the jstack process, not the running JVM that produced the thread dump. If you have a large standard out log, this may be the better approach. Filtering out thread dumps from large standard out logs can be a nucense.

If your JVM has remote debugging enabled by adding something similar to “-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5002” the java arguments, a remote debugger can be used to generate a thread dump. This article explains how to use jdb to generate thread dumps.

If the JVM is running Websphere, the following wsadmin snippet can be used to generate a thread dump:

wsadmin> set jvm_handle [$AdminControl completeObjectName type=JVM,process=yourServerName,*]

wsadmin> $AdminControl invoke $jvm dumpThreads

If the JVM is running Weblogic, the following Weblogic Scripting Tool commands will generate a thread dump for an application server:

connect(user, password, server_url)
threaddump(path_to_dump_file)

If the JVM is running JBoss, you can generate a thread dump following these instructions.

One of these methods should be able to produce a thread dump in your situation. Not all of these are usable in every situation, which is why I presented so many options. And, there is an exception to every rule–including this one. So, eventually, someone will encounter a situation there requires more creativity. Complex middleware environments create interesting challenges to troubleshooting and basic diagnostics.

It is also possible to generate a thread dump programatically in your java code by calling Thread.dumpStack() on each Java thread in the JVM. This requires you to keep track of all application-level, Java threads. I don’t believe you’ll be able to see the system-level, Java daemon threads that the JVM spawns.

If the JVM is severely impaired (meaning, it’s problems exist below the Java application level), it is possible it will not be able to produce a thread dump. If this is the case, your options may be limited this time around. But, most adverse conditions encountered will happen more than once.

Example

The full thread dump from the Sun JVM running the Recursion.java program looks like:

Full thread dump Java HotSpot(TM) Client VM (1.5.0_16-b02 mixed mode):

“Low Memory Detector” daemon prio=1 tid=0x090f4668 nid=0x655a runnable [0x00000000..0x00000000]

“CompilerThread0” daemon prio=1 tid=0x090f30c0 nid=0x6559 waiting on condition [0x00000000..0xaa1be908]

“Signal Dispatcher” daemon prio=1 tid=0x090f1d20 nid=0x6558 waiting on condition [0x00000000..0x00000000]

“JDWP Event Helper Thread” daemon prio=1 tid=0x090f0ac0 nid=0x6557 runnable [0x00000000..0x00000000]

“JDWP Transport Listener: dt_socket” daemon prio=1 tid=0x090eef40 nid=0x6556 runnable [0x00000000..0x00000000]

“Finalizer” daemon prio=1 tid=0x090dfc70 nid=0x6555 in Object.wait() [0xaa6cc000..0xaa6cd120]
at java.lang.Object.wait(Native Method)
– waiting on <0xaa850af8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
– locked <0xaa850af8> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

“Reference Handler” daemon prio=1 tid=0x090dde80 nid=0x6554 in Object.wait() [0xaa74d000..0xaa74dfa0]
at java.lang.Object.wait(Native Method)
– waiting on <0xaa850a00> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
– locked <0xaa850a00> (a java.lang.ref.Reference$Lock)

“VM Thread” prio=1 tid=0x090db1c0 nid=0x6553 runnable

“VM Periodic Task Thread” prio=1 tid=0x090f5cc8 nid=0x655b waiting on condition

Analysis

I’ll save the analysis of thread dumps for a future article.

In the meantime, this tutorial from Sun might help.

References

[1] Websphere wsadmin tool threaddump command
[2] http://download.oracle.com/docs/cd/E12840_01/wls/docs103/pdf/config_scripting.pdf
[3] http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jstack.html
[4] http://www.jboss.org/community/docs/DOC-9804
[5] http://blogs.sun.com/watt/resource/jvm-options-list.html
[6] http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/
[7] http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Thread.html