Capturing JVM TCP Traffic

Introduction

Any time you have a distributed application, there is network communication involved. Capturing this network traffic can be instrumental in diagnosing and solving problems. Generally, to capturing network traffic, superuser privledges, timing, and a bit of luck is needed. Often, the appropriate privledges will require the cooperation of an external group. Ideally, there would be a way to capture network traffic that doesn’t require the level of effort just described. Of course, one has to be careful with these things, being able to monitor network traffic is not particularly desirable from a security perspective. However, we are assuming this is a development environment–no harm, no foul.

This article describes strategies for capturing all TCP traffic coming into and out of a java process.

Accomplishing Our Task With Platform Specific Solutions

Solaris

On Solaris 2.8 and above, the truss command will capture all data sent to a process or sent from a process that it is tracing using the “-rall” and “-wall” options. So, for example, to capture all input and output from a java program, the following command would be used:

truss -o /var/tmp/truss.out -f -rall -wall -l java …

The ‘-o’ parameter specifies where output should be written. The ‘-f’ parameter says follow forked children. The ‘-l’ parameter reports which LWP made a system call (and read/wrote data).

Linux

The modern Linux systems, the strace command can do basically the same thing with the following:

strace -o /var/tmp/strace.out -f -e read=all -e write=all java …

The ‘-o’ and ‘-f’ parameters do the same as with the truss example. The “-e read=all” tells strace to dump all data from read() and recv() system calls. The “-e write=all” tells strace to dump all data from write() and send() system calls.

Using strace on Linux and truss on Solaris (or OpenSolaris), it is possible to capture all data coming into a process or leaving a process via TCP sockets. Although, it can be massaged to achieve the desired result. This works well enough on Solaris and Linux, but the output isn’t in the best format.

But, what does one do with other systems were these tools aren’t available? Ideally, there would be a JVM-level mechanism to generate this information.

A JVM Mechanism To Capture TCP Traffic

Unfortunately, there doesn’t seem to be such a facility in most of the major JVM implementations.

The exception to this is the SSL network debugging parameter, “-Djavax.net.debug=all”. Among other things, this will dump the data (encrypted and unencrypted) from every packet sent or recieved via SSLSocket or SSLServerSocket objects. The use of this parameter is discussed here. This works great if the data you want to capure is being passed over an SSL connection. But, that isn’t always the case.

So, the remainder of this article is going to introduce the basic pieces needed to build your own TCP traffic dumper for a JVM that will work on any platform/OS.

Basic Java TCP Networking

Server

Java server-side TCP networking consists mainly of using ServerSocket objects obtained from a ServerSocketFactory object. ServerSocket objects use SocketInputStream and SocketOutputStream to handle I/O for TCP sockets.

Client

Java client-side TCP networking consists of using Socket objects obtained from a SocketFactory object. Socket objects also use SocketInputStream and SocketOutputStream to handle I/O for TCP sockets.

Notice that SocketInputStream and SocketOutputStream are not defined in the regular Java API documentation. These are internal classes of the Sun and IBM distributions of the JDK–probably others. www.docjar.com publishes the details of this class on the internet. These details are based upon the OpenJDK7 classes, but the basic idea is still there.

A General TCP Capturing Tool For Java

So, for many of the common JVM implementations currently available, the non-public SocketInputStream and SocketOutputStream serve as “gateway” objects that all TCP communication coming into or out of a JVM pass through. This fact makes these two objects a good place to capture data flowing through TCP sockets.

The code for these two classes is distributed with the Sun JDKs. The author used the souces distributed with JDK 1.5.

java.net.SocketInputStream

This object has several methods that can be used for reading data:

private native int socketRead0(FileDescriptor fd, byte b[], int off, int len, int timeout)
public int read(byte b[])
public int read(byte b[], int off, int length)
public int read()

To simplify things, notice that socketRead0() is a native method. We will not be instrumenting this method in Java code. However, the first read() method listed above calls the second read() method. The second read() method calls socketRead0(). So, to capture data passing through any combination of the first two read() methods, some logic that prints out the contents of the array, b, between position off and position off + n. Again, note, the variable names are coming from the java.net.SocketInputStream code distributed with Sun JDK 1.5. This code is not posted here, but is easily obtainable.

The third read() method also calls the second read() method. So, what was described in the last paragraph also captures this data. Likewise, in the end, all data passes through the socketRead0() native method via the second read() method.

To capture when a socket is closed, put a System.out.println(“Closing Socket”) call (possibly capturing a file descriptor) in the public void close() method.

To capture when a socket is opened, put a System.out.println(“Opening Socket”) call in the public constructors of the java.net.SocketInputStream class.

Note, when a new socket is opened, a one SocketInputStream and one SocketOutputStream objects will be created. Using this information, one could infer when the socket was created.

java.net.SocketOutputStream

The SocketOutputStream provides several methods for writing data to an established TCP socket connection:

private native void socketWrite0(FileDescriptor fd, byte[] b, int off, int len)
private void socketWrite(byte b[], int off, int len)
public void write(int b)
public void write(byte b[])
public void write(byte b[], int off, int len)

Once again, there is a native method that ultimately is called by every other method in this list: socketWrite0(). The Java method socketWrite() is a wrapper around socketWrite0(), which provides some error checking and ensures that the file descriptor is locked when the data is actually written to the socket (i.e., is thread safe).

Each write() method calls socketWrite(). So, to capture all data that passes through this object, implementing logic to print out the contents of the b array in the socketWrite() method between positions off and off + len are required.

To capture when a SocketOutputStream is opened or closed, adding a line to the constructor or close() should be sufficient.

Deployment

You’ve just modified two JDK classes that belong to the java.net package. In order to have a JVM successfully use these classes either the new classes must replace the original classes in rt.jar or the new classes must be placed in the Boot Classpath. This is because java.* classes can only be loaded by the Primordial Classloader.

Let’s assume that the classes described above have been successfully compiled and placed into a JAR file called TCPDebug.jar. To place this JAR file in the bootclasspath, use the following:

java -Xbootclasspath/p:/path/to/TCPDebug.jar TesterClass …

The “-Xbootclasspath/p:” option prepends TCPDebug.jar to the Primordial Classloader’s classpath. This means that the newly created debug versions of SocketInputStream and SocketOutputStream will beloaded instead of the original versions of the classes. Using this method to pick up custom java.* classes prevents corrupting the JRE libraries.

Closing Thoughts

Using these custom classes will result in all TCP traffic coming into or out of the JVM being written to Standard Out (or where ever this file descriptor is redirected). Note, the formatting will only be as neat as you make it.

The author’s first version didn’t produce pretty results, but it helped solve a problem that had gone unsolved for days.

Happy Debugging.