Blog

Our series of posts about solving the OutOfMemoryError in our hypothetical production system nearing the conclusion. We have covered a number of different ways to attack OutOfMemoryErrors so far - profilers, JDK bundled tools and heap dump analyzers. Our today’s mercenaries will be Application Performance Management tools, or APMs.

Let us continue our series of posts about solving the OutOfMemoryError in our hypothetical production system.

We have described  different methods to tackle the problem, and today's post concentrates on what you can learn from heap dumps. Spoiler alert: with a bit of luck, you can get very close to solving the OOM :)

Here's the news for you: Java memory leaks have attitude, and you need to know your way around to successfully nail and solve them. While this blog post is written in response to people who became confused while using our product for discovering memory leaks, it should serve some food for thought for other leak hunters as well.

I got an email a couple of days ago. The author had tried Plumbr on a self-made leaking test application and, to his disappointment, instead of getting a beautiful leak report he got an OutOfMemoryError. He was not the first one sending us such comments, and from the fair amount of feedback we've gathered so far we've begun seeing some patterns in how people use Plumbr.

When we talk to people about our solution for discovering memory leaks  –  Plumbr – we immediately get positive feedback. But when we add Java into the equation, the initial excitement is often complemented with questions: “Are there memory leaks in Java? Isn't Java a garbage-collected language?”

In this post I will explain why memory leaks are in fact a common problem for Java applications.

In the last days of January, the number of Plumbr users exceeded an important milestone - we now have more than 1,000 users! As we started offering Plumbr as a public beta in the beginning of November last year, we can say that „collecting“ the first 1,000 customers took us 3 months.

Let's dive into the numbers (we are big fans of factual proof!), and elaborate on:

• What did we do to get here?
• What have we learned from the customers and where are we planning to head with product development?
• What are we going to do to reach 10,000? 100,000? Bazillion?

We recently celebrated crossing the magic threshold of 1000 registered users.  Around 10% of those one thousand early adopters have given us feedback in some form. There is a separate blog post in progress summarizing the first 1000, and our future plans (subscribe to our twitter feed to be notified!).

But this announcement is actually dedicated to the next step in the product maturity. Fair amount of users have reported false alarms: they say that Plumbr is too sensitive and claims objects to be leaking even if those objects occupy less than 1% of heap. We have good news for you: based on the hundreds of samples of Java application object lifecycles we have gathered, the detection algorithm has been tweaked to be more „suspicious“, ie. it now avoids most of the false alerts by collecting some more evidence before announcing the verdict.

We encourage all users to upgrade.

On our quest to solve every memory leak in the Java world we get in touch with many teams who struggle with performance issues of their applications. Every developer tries to solve these differently. Some begin with vmstat or top, others with a CPU profiler or a DB load monitoring tool. Many of them turn to different APM products, like  CA APM, AppDynamics, dynaTrace, etc.

However, if you properly ask the application, it will normally give you all the information you need for performance tuning. And you can get the info without relying on indirect metrics of the operating system or the JVM.

How do I know that the application is actually suffering from memory leaks? Where and how can I find the cause of the possible memory leak? An experienced developer usually begins its troubleshooting by answering the aforementiond fundamental questions.

1. First - you should find evidence hidden somewhere in the logs. If you are lucky you can have a peek at production logs. In most cases there is a pack of angry Kerberos - like guys guarding the access - both from operations and security units of the corporation.
2. Lets assume that you can bypass the walls separating you from the vital information. Is it anyhow helpful? Most likely not. You find your good'ol friend Exception in thread "main" java.lang.OutOfMemoryError: while grepping through logs, but nothing interesting seems to precede its creation. There might be hints from monitoring logs that performance degradation started minutes before the actual error. But besides this - business as usual.