On a quest for missing stacktraces

September 5, 2013 by Nikita Salnikov-Tarnovski

One comment to our recent blog post brought back some memories about a specific experience. The type of experience I wish I had not experienced. Long time before we founded Plumbr I was debugging an application that gave me an exception every once in a blue moon. The level of detail was astounding:

java.lang.NullPointerException

That was it – no stacktrace to work with. And the application was guarded by a truly evil sysadmin who would not let me within 100 yards from the deployment. So the stage was set for numerous all-nighters trying to understand what was causing the error. I bet most of our readers know what I am talking about here.

The first suspect was obvious. The summer intern who never really got exception handling and who could have easily swallowed the exception instead of rethrowing or proper logging. So I was expecting to find something like this in the code somewhere:

System.out.println(e);

But no matter how thoroughly I grep’ed through my source code I could not find anything resembling such carelessness. I do not recall much from the following day besides answering a phone call every few hours and giving the “progress report”. Unfortunately the progress was nowhere in sight. Unless I would count in the numerous updates done to production in a desperate attempt of getting something a bit more meaningful about the possible causes.

But I do remember my faith in Java being tested when I started suspecting something I had always thought being impossible. Namely that Java is capable of literally eating your Exception stacktraces.

And indeed the JIT can and will do it. Need proof? Run the following code:

for (int i = 0; i < 100_000; i++) {
	try {
		args[0].toString();
	} catch (Exception e) {
		if (e.getStackTrace().length == 0) {
			System.out.format("Java ate my stacktrace after iteration #%d %n", i);
			break;
		}
	}
}

In my MB Pro equipped with the latest stable JDK 7 hotspot the exception stacktrace gets eaten at the iteration #12,288. But why on earth is this happening and how could one avoid it?

The simple answer is that it is one of the JIT optimizations applied during runtime. As one of our recent posts described, creating Exceptions is darn expensive. So, for performance purposes, when the JIT discovers that an exception is being thrown several times, the code may be recompiled. After recompilation, the compiler may choose a faster tactic using preallocated exceptions that do not provide a stack trace.

Hopefully the vast majority of you will never have to deal with a situation like this. After all, when exceptions are created at the pace triggering JIT to switch to preallocated exceptions, you must be doing something else wrong. After all, exceptions should occur only on exceptional situations.

But for those who do, the solution is simple. You can disable the use of preallocated exceptions by adding -XX:-OmitStackTraceInFastThrow to your JVM startup parameters. So did I and found the culprit after the next JVM restart.

If you liked the blog post, you could consider subscribing to our Twitter feed. I promise to keep the marketroids away from this blog for as long as I possibly can. And keep providing interesting insights from the Java performance landscape.

Can't figure out what causes your OutOfMemoryError? Read more

ADD COMMENT

COMMENTS

Thanks for the article :) Good observation, that exception creation site is not always the throw site

iNikem

Object obj = null;

for (int i = 0; i < 1_000_000; i++) {

try {

obj.toString();

} catch (Exception e) {

}

}

try {

obj.toString();

} catch (Exception e) {

int length = e.getStackTrace().length;

if (length == 0) {

System.err.println("No Stack Trace");

} else {

System.out.println("Stack Trace length is " + length);

}

}

The above piece of code prints Stack Trace length is 1

But if you change the obj reference type to String/Integer it prints as No Stack Trace

For some type of reference types I get the output as No Stack Trace and for some other reference types and also Interface as reference types I am getting the “Stack Trace length is 1″ as the output.

Need to figure out why this behavior. Please do let me know if you have the reason for this :)

Regards,
Chandan

Chandan

Need to dig into OpenJDK source. I have no explanation for this from the top of my head…

iNikem

I just hope I remember this when i come across a similar situation… you’ll have saved me probably 2 days of uninspired code-searching. -

Mike

Can't figure out what causes your OutOfMemoryError? Read more

Latest
Recommended
You cannot predict the way you die
When debugging a situation where systems are failing due to the lack of resources, you can no longer count on anything. Seemingly unrelated changes can trigger completely different messages and control flows within the JVM. Read more
Tuning GC - it does not have to be that hard
Solving GC pauses is a complex task. If you do not believe our words, check out the recent LinkedIn experience in garbage collection optimization. It is a complex and tedious task, so we are glad to report we have a whole lot simpler solution in mind Read more
Building a nirvana
We have invested a lot into our continuous integration / delivery infrastructure. As of now we can say that the Jenkins-orchestrated gang consisting of Ansible, Vagrant, Gradle, LiveRebel and TestNG is something an engineer can call a nirvana. Read more
Creative way to handle OutOfMemoryError
Wish to spend a day troubleshooting? Or make enemies among sysops? Registering pkill java to OutOfMemoryError events is one darn good way to achieve those goals. Read more