Plumbr Blog

The first half of 2013 has kept us quite busy. I’m happy to announce that today we are ready to release the first significant result of our work – Plumbr version 3.0.

What has changed? The most visible changes are related to usability. Before 3.0, Plumbr was packaged as a desktop/server software that you needed to install on every workstation separately and the user interface (UI) was limited to that workstation. From today on, Plumbr is offered as a Software-as-a-Servise (SaaS) solution. This means that all your JVMs, performance metrics and leak reports are now gathered in one easily accessible place.

Read on for a more detailed description of the new UI and a sneak peak into Plumbr future releases!

Some of you have been there. You have added -Xmx option to your startup scripts and sat back relaxed knowing that there is no way your Java process is going to eat up more memory than your fine-tuned option had permitted. And then you were up for a nasty surprise. Either by yourself by checking a process table in your development / test box or if things got really bad then by operations who calls you in the middle of the night telling that the 4G memory you had asked for the production is exhausted. And that the application just died.

So what the heck is happening under the hood? Why is the process consuming more memory than you allocated? Is it a bug or something completely normal? Bear with me and I will guide you through what is happening.

3.0 release is changing the way you experience Plumbr in action. We are also about to simplify the pricing in a week along the final 3.0 release. Both of the changes are based on your feedback, thus we expect you to be happy with the changes.

In order to discover all the new look and feel, register yourself to get your hands of the latest release and access the new user interface. If you are an existing user, generate yourself a password using the same email address you used during the registration.

This is going to be another story sharing our recent experience with memory-related problems. The case is extracted from a recent customer support case, where we had an issue with an application dying with OutOfMemoryError messages in logs. After running the application with Plumbr attached we were sure we were not facing a memory leak this time. But something was still terribly wrong.

Lessons learned?

• Do not let integration details cross system boundaries

• Redundant data will cost you. Remove it whenever you can.

• Primitives are your friends. Know thy tools and learn Trove if you already haven’t

• Be aware of the optimizations provided by your JVM

This is the second post in the series describing our development infrastructure. We started with describing the external goals related to multi-platform support. In the first post we also reasoned why we need to test on different platforms separately. We concluded the post with the fact that instead of “support as many configurations as possible”, a lot more feasible goal would be “support as many users as possible”. From this customer-facing goal we now extracted internally used development goals:

• Automate builds. Plumbr release has to be built without any manual intervention.
• Automate tests. Plumbr release has to be verified automatically using different testing techniques ranging from unit to acceptance tests.
• Automate infrastructure. Build infrastructure has to be able to launch and destroy the server instances used to build and test Plumbr automatically.
• Provide transparency. Whether it is a functionality implemented or bug fixed – we need to know in which versions this change is present. If an exception is thrown, we need to be able to map the obfuscated stacktrace back to the actual source code using the correct version of the obfuscation map.

This post is part of our open culture. We do not believe into closed processes, so we decided to share some interesting concepts about our development infrastructure. Or more precisely – how do we support multiple platforms in Plumbr development.

Lets start with our goals in mind. Among other goals we are aiming towards, the one adding vasts amounts of complexity is the number of platforms we need to support. If you have thought that for Java it will all be about Write Once, Run Anywhere then – make a second guess.

It all falls apart the very same minute you create something truly low-level and optimized to the last possible bit. Suddenly all kind of unexpected small details start to matter, such as the garbage collector selection, minor differences in the classloader implementations or the way JVM vendors have decided to interpret certain parts of the JVM specification.

I have just recently returned from JavaOne Russia conference. JavaOne is one of the biggest and the most known conferences for Java developers. The event held in Moscow draw the attention of about 3,000 participants. I was selected to present their my talk about “Multi-platform Java application development”. I will convert that talk into a blog post in near future, stay tuned.

But what I would like to share with you today are my impressions of this conference. Being a speaker gave me a really good opportunity to see the event from the different angles:

This is going to be a rather evil post – something you will be googling when you really wish to make someone’s life a misery. In the world of Java development memory leaks are just the type of bugs you would introduce in this case. Days or even weeks of sleepless nights in the office are guaranteed for your victim.

We will describe two leaks in this post. Both of them are easy to understand and reproduce. The leaks originate from the real world case studies, but for the sake of clarity we have extracted the demo cases to be shorter and simpler for you to grasp.

But rest assured – after we have both seen and fixed hundreds of leaks – cases similar to those demoed this are more common than you might have expected.

We have recently posted data about application server market share we gathered from the free Plumbr deployments. And it resonated well – via different channels we got hundreds of comments and opinions on how to interpret the data.

But one of the arguments in its different forms kept coming after us through every channel. Whether it took the form of “Tomcat is not an application server” or “This data is irrelevant as it is not focused on real application servers such as Weblogic or WebSphere”, it just kept surfacing.

I made us wonder – why does Java community have so different opinion abou what actually is an application server. So we decided to shed some light upon the issue. Looking into the most obvious source – namely Wikipedia – and things do not look too bad:

The post is the third in the series where we publish statistical data about the Java installations. The dataset used originates from the free Plumbr installations out there totalling 1,024 different environments we have collected during the past six months.

First post in the series analyzed the foundation – on what OS the JVM is run, whether it is a 32 or 62-bit infrastructure and what JVM vendor and version were used. Second post focused on the different application servers used. The one you are reading now sheds some light upon the heap sizes used by Java applications.

Two weeks ago we got hacked. Which was something we have anticipated for the long time – after all, you cannot expect to run a reasonable business and not be a target of some malicious attacks. So as we finally reached our “goal” we had to face the consequences implied.

Full disclosure: no customer data was exposed during the attack and no Plumbr executables were harmed during the attack.

We have both the good and the bad news for you in this version upgrade. The good news is that – we have significantly improved permgen leak detection based on data gathered from the hundreds of reports we have detected since the initial version was launched back in July 2012. Based on the data and feedback we can now say this functionality is matured and stable.

The sad news is also related to the maturity of this functionality – we announce that starting from April 8th, 2013 the permgen leak reports will no longer be available for free. So – take the chance while it is still possible, download Plumbr and get rid of all your existing permgen leaks without us charging you a dime.

The post is the second post in the series where we publish statistical data about the Java installations. The dataset used originates from the free Plumbr installations out there totalling 1,024 different environments we have collected during the past six months.

First post in the series analyzed the foundation – on what OS the JVM is run, whether it is a 32 or 62-bit infrastructure and what JVM vendor and version were used. In this post we are going to focus on the application servers used.

It proved to be a bit more challenging task than originally expected – the best shot we had towards the goal was to extract it from the bootstrap classpath. With queries similar to “grep -i tomcat classpath.log”.

A week ago I was asked to fix a problematic webapp suffering from memory leaks. How hard can it be, I thought – considering that I have both seen and fixed hundreds of leaks over the past two years or so.

But this one proved to be a challenge. 12 hours later I had discovered no less than five leaks in the application and had managed to fix four of them. I figured it would be an experience worth sharing. For the impatient ones – all in all I found leaks from

• MySQL drivers launching background threads
• java.sql.DriverManager not unloaded on redeploys
• BoneCP loading resources from the wrong classloaders
• Datasource registered into a JNDI tree blocking unloading
• Connection pool using finalizers tied to Google’s implementation of reference queue running in a separate thread

We have now been discovering leaks for more than a year with Plumbr. Throughout this time we have gathered anonymous statistics from the free Plumbr installations out there. And as we are good guys, we decided to share our discoveries with you.

The post is going to be the first in the forthcoming series. We start with the environments used: if you are interested which is the most popular JVM vendor or JVM version, whether 32bit is more popular architecture than 64bit or whether Windows 8 is more popular than Windows XP – this will all be covered in our post. In the next series we are analyzing the application server market shares and different configuration settings on the JVMs.

Every once in awhile each and every one of us is trying to measure performance of certain parts of the code. It can be either a fully functional application measured against specific number of user transactions per minute or a microbenchmark. Which you have written to prove that your fellow developer is wrong.

Especially in the latter case it is beneficial to be aware of the pitfalls skewing your benchmark results. Looking into the benchmarks I have created ~seven years ago, I have to admit I have been proving my point on several cases I was actually incorrect. My apologies, dear friends. But I have learned something along the way and think it is worth sharing. So that you would not end up in the same stupid situation.

We have all been there. Looking at the poorly designed code while listening to the author’s explanations about how one should never sacrifice performance over design. And you just cannot convince the author to get rid of his 500-line methods because chaining method calls would destroy the performance.

Well, it might have been true in 1996 or so. But since then JVM has evolved to be an amazing piece of software. One way to find out about it is to start looking more deeply into optimizations carried out by the virtual machine. The arsenal of techniques applied by the JVM is quite extensive, but lets look into one of them in more details. Namely method inlining. It is easiest to explain via the following sample:

We are dealing with complex stuff in Plumbr. Hacking JVM internals, bytecode manipulation, data mining – we are all over it. But some of us also teach young developers. Who are doing their first steps in the Java wilderness. And year after year we see those youngsters run into the same problems over and over again. And part of those problems seems to be related with the cryptic ways Java has chosen to explain developers about the problem at hand.

The article will explain in simple terms the Amdahl’s law. We are going to demonstrate via a case study how throughput and latency are changing when you change the number of threads performing the tasks. We also help to draw right conclusions in the context of your own performance tuning task at hand.

First of all, let’s refresh our memory on the definitions.

• Throughput – number of computing tasks closed per time unit. Example – 1,000 credit card payments in a minute.
• Latency – delay between invoking the operation and getting the response. Example – maximum time taken to process a credit card transaction is 25ms.

It will start nicely. Like most of the stories. You discover a new concept and are amazed by it’s powers. And then equipped with this new hammer suddenly everything starts to look like a nail. From what we have experienced in past months, java.lang.ThreadLocal makes one hell of a hammer.

I guess it all boils down to the very concept of how ThreadLocal works. It’s easiest to grasp the concept via the scoping analogy. In the very same way your Spring beans can be for example container, session or request scoped. ThreadLocal equips you with the possibility to declare objects in Thread scope.

You can set any object to ThreadLocal and this object will have both global and local scope in the thread accessing this object. It might be complex to grasp at first but let me explain:

tl;dr; depends on your application.

But for those who wish to have some insight about how to squeeze out most from all those expensive cores you have purchased for your production site – bear with me and I will shed some light on the mysteries surrounding multi-threaded Java applications.

The content is “optimized” towards the most typical Java EE application, which has got a web frontend allowing end users to initiate a lot of small transactions within the application. And significant part of each transaction is kept waiting for some external resource. Such as a query to return from the database or from any other integrated data source. But most of the content is also relevant for other applications. Such as computation-heavy modeling applications or data-chugging batch processes.

Is this really something you should bother? Is there something fundamentally wrong with java.util.ArrayList and java.util.HashMap? For most of the source code out there the answer is  – no; those implementations are perfectly OK. But as always, the devil is in the details. And there exist situations when either the feature set built into the Collections API is not sufficient enough or you find the overhead provided by the standard collections to be way too high for your likings.

In the past years we have continuously stumbled upon the very same problems.  And thought it would be nice to share the experience – hopefully it can save somebody a day or two. Either via avoiding yet another bidirectional Map implementation or understanding why his HashSet consumes 10x more memory than intended.

This one is going to be one of the rare non-technical post. But we just closed our first year and did an internal review of what did we manage to deliver throughout 2012. So our technical readers – do not be afraid. We continue with in-depth technical articles throughout 2013.

When we were done with Plumbr 2012 review, it looked like something we could also openly publish. So here we go, hopefully those of you building your own tech company can learn a thing or two from the next 900 words.

Modern technology is wonderful. Looking back to the early 2000’s, it seems ridiculous. We literally had to wait for months to get our hands on a new staging environment. Young developers, you should be grateful for virtualization and IaaS! You can now get happily up and running on a brand new application server within minutes. Or, so the platform providers promise.

We wanted to test this promise and see whether it’s really that easy to build a virtualized environment. And, of course, run our very own Plumbr leak detector on it.

This story goes back. For weeks or even decades, depending on how you mark the starting date. Anyhow, few weeks ago one of our customers had problems with interpreting a leak reported by Plumbr. Quoting his words “It seems that Java itself is broken”.

As a matter of fact, the customer was right. Java was indeed broken. But let’s check the case and see what we can learn from it. Lets start by looking into the report generated by Plumbr.

We have worked hard during the past months and can now proudly present you the results. In a form of new Plumbr release. In the 2.0 version we are shipping you a complete rewrite on the core algorithms with significant leak detection improvements. The quality of leak detection algorithms is significantly increased: we now find 88% of all leaks present in the applications.

This number might need a bit more insight from us. What we do behind the closed doors is a lot of (machine) learning. Free Plumbr versions send us anonymous statistics about the object creation and destruction patterns. We used this data as a training set for supervised machine learning to improve our leak detection algorithm.

JVMs are considered smart nowadays. Not much configuration is expected – just set the maximum heap to use in the startup scripts and you should be good to go. All other default settings are fine. Or so some of us mistakenly thought. There is a lot going on during runtime which cannot be automatically adjusted for performance, so I am going to walk you through what and when to tweak throughout a case study I recently faced.

This is a question I have faced several times during my career in enterprise software development. Every once in awhile I had to hand out recommendations for configuring a specific new environment. And more often than not, part of the question at hand was related to “Should I use 32- or 64-bit JVM”. To be honest, in the beginning I just flipped the coin. Instead of giving a reasoned answer. Sorry, bros. But later on I gathered some insights about when and why to choose which configuration and thought to share it with you.

We are now announcing the public availability of Plumbr 1.4. In this version we focused on compatibility and stability issues.

The biggest news in the release is Solaris support – Plumbr now works both on x86 and SPARC architectures. We have conducted a month of beta testing and can now proudly present the results.

All of you who contacted us via different support channels asking for the Solaris support were already notified, but if you are one of the silent guys then – go ahead and download the newest release.

Another important improvement is JDK 7 support in Mac OS X – if you were among the early JDK 7 adopters on Mac OS X and had issues when loading native libraries, we now have fixed the problem.

The story starts with our decision to go to Devoxx.  With a booth. By a caravan. Driving 2,200 kilometers. But this is not going to be a travel story, do not worry.

During the travel we taught our marketing guys to be prepared for doing demos in our booth.  After all, what’s the benefit of dragging all the droids to Antwerp if they are not even good for a demo or two.

As we had ~30 hours of driving time ahead of us, we thought we have enough time to get the droids up to speed.

First stop – verifying the required infrastructure. One of the droids had managed to install a JDK and Eclipse all by himself. So far so good. But when firing up the Eclipse and adding our demo project resulted in CPU usage jumping to 100% and Eclipse becoming completely unresponsive.

Remember the last time when you were laughed at by C-developers? That the Java is so slooooow that they would never even consider using a language like this? In many ways, the concept still holds. But for it’s typical usage – in the backbones of a large enterprise – Java performance can definitely stand against many contestants. And this is possible mostly thanks to the magical JIT. Before jumping into explaining Just-In-Time compilation tricks, lets dig into background a bit.

As you might remember – Java is an interpreted language. Java compiler known by most users, javac, does not compile java source files directly into processor instructions like C compilers do. Instead it produces bytecode, machine independent binary format governed by specification. This bytecode is interpreted during runtime by JVM.This is the main reason why Java is so successful in cross-platform – you can write and build the program in one platform and run it on several others.

Java is a cross platform runtime. You write your code on a platform of your choice, test it on a different platform and then push it into production on a third platform. It just works. You do not think about the differences under the hood – JVM implementations take care of it. On worst case scenario you need to tweak some minor differences once in a blue moon. For example when building Plumbr we have only a single Ant target that takes care of producing a Plumbr executable. On any platform we supported so far.

In addition, as computer science history states, the C – language was a first truly portable language. In Plumbr we also have a bunch of native code written in C, which interoperates with JVM internals. In contrast to an Ant target we build the native part of Plumbr with a makefile. It contains several conditional branches supplying suitable flags for different compilers and linkers. We use gcc on Linux or Mac for example (with different flags, however). On Microsoft platforms we use ‘cl’ which takes 29, I kid you not, command line parameters to compile a dll. Then we have bitness issue, meaning that we have to build 32 and 64 bit versions of our native code. All this means we have a single jar file and six native libraries in total to build, test and distribute. Or we used to have before today.

The term ‘Cobra effect’ stems from an anecdote set at the time of British rule of colonial India. The British government was concerned about the number of venomous cobra snakes. The Government therefore offered a reward for every dead snake. Initially this was a successful strategy as large numbers of snakes were killed for the reward. Eventually however Indians began to breed cobras for the income.

When this was realized the reward was canceled, but the cobra breeders set the snakes free and the wild cobras consequently multiplied. The apparent solution for the problem made the situation even worse.

So how is Java heap size related with Colonial India and poisonous snakes? Bear with me and I’ll guide you through the analogy using a story from a real life as a reference.

We are going to Devoxx this year. In case you haven’t heard of it – this is the biggest Java event in Europe with approximately 3000 attendants. The event is held in Antwerp, Belgium from 12th to 16th November.

The good part with all this Devoxx fuss is that – we have some Devoxx passes waiting for you. If you haven’t bought yours – the conference is sold out and this might be your last chance to attend. To get your pass, you need to follow those steps:

Another day in the office. Except that you have just realized that the application at your guard seems to take forever to respond to requests. Operations which usually take just few hundred milliseconds to complete are now slower than snails on Xanax. This weird situation gradually worsens throughout the next hours.

You manage to discover that the CPU usage is unusually high. Before you have time to dig further you have to surrender to the boss breathing down your neck. You restart the application on his demand. Only to discover the Xanax guys to return within half an hour. This time you decide to stand your ground and find out the cause.

Monitoring tools show high CPU usage and quick growth in memory consumption. In conjunction with quickly decreasing throughput. You plot the data on a graph and see the following trend taking shape:

It has now been three months since the last major release. We have not been sitting on our hands – quite the contrary. The main focus of the release was to reduce performance overhead posed by our solution.

Numerous customers have asked us about the actual overhead our tool poses. And the second subset of our customers was not happy with the performance impact we posed so far. So we listened and started measuring. And optimizing. And measuring again. As the topic proved to be far more difficult and interesting than we originally imagined, we also covered the lessons learned in our blog posts – available here, here and here.

Without further ado, here are the results

You have an optimization task at hand. Which is great, now you can do something interesting besides implementing yet another invoice processing screen. The optimization task is (hopefully) linked to some non-functional requirement, such as the number of transactions your application has to process per second or the maximum time per transaction allowed.

Now, you have an identified the part of your application as the source of the performance problem. This part of your application uses a particular algorithm. You are considering switching this to another implementation and want to measure the performance impact. As discussed in one of our previous articles, you wisely choose the number of user transactions per second as the metrics to measure. You run your stress test with old implementation, write down the number of operations per second achieved, then run the very same stress test with new implementation, write down new number of operations per second.

I have spent a lot of time recently measuring Plumbr’s overhead and figuring out some ways of reducing it. When I started the measurements, I was frustrated to the extent that I had to let my feelings out in a form of a previous blog post. But now I am back in my constructive mode and ready to share more insights on the subject.

How do you measure memory overhead of a Java agent? The answer, as seen below, seems trivial at first:

“You measure memory consumption of the application with that agent and without it, then compare”.

As it turns out, even the “measure” part is not that obvious, not to speak of “comparing”.

 You create an amazing app. You put it into production. And you notice that you just do not have enough memory available. Even when all your measurements (maybe taken with the help of our small utility) show that you should be just fine.

We are planning to publish a series of blog posts investigating where the heap disappears, and kick off with the infrastructure edition. The hypothesis for today – could my application server be the greedy bastard consuming all my precious memory?

To find it out, we ran a small set of tests on five application servers. Ladies and gentlemen, let us introduce the contestants:

• Oracle Weblogic 12.1.1
• Glassfish 3.1.1.2
• RedHat JBoss AS 7.1.1
• Apache Tomcat 7.0.30
• Jetty 8.1.7

“What’s your overhead?”. We often hear this question when we talk about Plumbr. Wikipedia describes overhead as “any combination of excess or indirect computation time, memory, bandwidth, or other resources that are required to attain a particular goal”. When you dig further in the Internet, you see mostly CPU overhead and memory overhead being mentioned.

That is why we can probably translate the question posed in the title into “how much more memory and/or how many more CPU cycles are utilized by my application when run with Plumbr attached?”. After having spent four weeks on trying to measure Plumbr overhead as precisely as possible, I have got a bunch of numbers under my belt. Unfortunately though, it seems that none of them makes much sense in the real life. Let’s elaborate on this a bit more, to see the reasons behind my conclusion.

Let’s start with a quick rewind to the early days of my career as a Java developer. I wanted to eliminate Garbage Collection (GC) pauses from a test that I was conducting. Lo and behold I was annoyed when I discovered that it couldn’t be done. Back then I left the issue for a “design error” and moved on with my life. Angry at James Gosling or whoever was responsible for the decision.

A couple of days ago I ran into a situation that reminded me of those old times. Luckily, the years gone by have accumulated some insight about JVM internals in me, and I thought I’d share my current thoughts in a form of a blog post.

What looks like as an easy task can in reality become somewhat complicated. There is a whole lot of different aspects you have to bear in mind when calculating the memory footprint of your objects:

• Do I need to measure shallow or retained heap size?
• Do I make the calculations for 32 or 64bit architecture?
• Am I running on x86, SPARC, POWER or on something even beyond imagination?
• Do I use compressed or uncompressed ordinary object pointers?
• [enter something else you are afraid or do not completely understand here]

Bearing all those aspects in mind when trying to estimate the size of your data structures is simply unreasonable when trying to meet yet another deadline. So we went ahead and packaged the code published by Java Champion Heinz Kabutz as a java agent and provided an easy way to add it to your application.

What is the size of a particular data structure?
“Can I fit all these objects into my ehCache?”

This article is the second post in the series where we try to answer those questions. The last post explained the difference between retained and shallow sizes of an object. In the article we also offered an example of how to calculate retained heap size of a data structure. In today’s article we will expand on what we called “simple” in the previous post. Namely - what is and how to measure shallow heap used by an object.

We are confident that we have created a good product. Plumbr has detected hundreds of leaks in very different Java apps all around the world. However, we really like to put our money where our mouth is, so we thought it’s time to raise the bar even higher. From now on:

If Plumbr fails to detect an existing memory leak in your application, our team will find it for you!

As simple as that. If our tool is not good enough then we have to make our hands dirty. No hidden tricks here.

How much memory will I need? This is a question you might have asked yourself (or others) when building a solution, creating a data structure or choosing an algorithm. Will this graph of mine fit in my 3G heap if it contains 1,000,000 edges and I use a HashMap to store it? Can I use the standard Collections API while building my custom caching solution or is the overhead posed by them too much?

Apparently, the answer to the simple question is a bit more complex. In this post we’ll take a first peek at it and see how deep the rabbit hole actually is.

The answer to the question in the headline comes in several parts.

By now we have published seven articles in the Solving OutOfMemoryError series and covered different aspects of the problem, as well as tested different possible solutions. Before wrapping this all up, we decided to briefly summarize our experience with the different tools out there.

What we discovered was that…

We are often asked ‘How does Plumbr work internally?’ – ever since we made the tool publicly available. As the number of Plumbr users is growing, the number of these questions is also rising, and we have finally reached a point where it is more efficient to write the answer down than recite it over and over again. In the following article I will describe how memory leaks can be detected in general and which approaches does Plumbr use internally to do its job.

In my latest post I explained the reasons that can cause the java.lang.OutOfMemoryError: PermGen space crashes. Now it is time to talk about possible solutions to the problem. Or, more precisely, about what the Internet suggests for possible solutions. Unfortunately, I can only say that I felt my inner Jamie Hyneman from MythBusters awakening when going through the different “expert opinions” on the subject.

Oracle says it’s 9,000,000. Wikipedia claims it’s 10,000,000. And the guys from NumberOf.net seem to be the most precise – they know that there are exactly 9,007,346 Java developers out there.

Nice numbers. I have used those articles as reference points while speaking about the potential market size for our memory leak detection tool. But something in these numbers has bothered me for years – there is no trustworthy and public analysis behind those numbers. Its just conjured up from thin air. So I finally thought I would do something about it and try to figure it out for good.

What follows is a practical introduction to a specific type of memory problems in Java applications. Namely – we will analyze the errors that cause the java.lang.OutOfMemoryError: PermGen space symptom in the stack trace. 

First of all we will go through the core concepts required to understand the subject – and explain what objects, classes, classloaders and the JVM memory model are. If you are familiar with the basic concepts, you can jump directly to the next section where I will describe two typical cases for the error in question alongside with hints and suggestions for solving it.

It’s been only a month since we released 1.0. But we already have something new and cool to ship – as of this version we also find the leaks in the JVM permanent generation.

PermGen leaks are a specific type of a memory leak the often happen during the redeploy phase.

I guess you all have faced the dreaded java.lang.OutOfMemoryError: PermGen space  at some point during your career. And I bet most of you have struggled to find the cause and fix for this. Worry no more – Plumbr now provides you the tools needed to find and fix those errors.

Yes, we were invited by the Y Combinator for the interview, but didn’t have time to write about it back then. Right after we came back we did the final product and infrastructure improvements, and started testing for the official 1.0 release. Now, after having released the product, analyzed the feedback, issued the first invoices, we can sit back for a moment and relive the adventure once more. The article will hopefully be useful for future applicants by sharing our experience from the trip.



TL;DR: An Estonian startup who applied to YC. Was invited to YC interview. Got rejected.  Still spent a week in the Valley and got tons of interesting meetings, ideas and contacts in return.

I would like to describe a recent case from my career as troubleshooter, which again reminds me that one should not put all his eggs into one basket. And that one should look around and use every possible tool from his toolbox, not just his favorite hammer. And that wrong and unverified assumptions are the most common reasons for long debugging sessions. And mostly of all: you should never jump to conclusions too fast!

We in the Plumbr team decided that it is. Over the last few months in beta our customers have solved more than 200 memory leaks with the help of Plumbr.

The sad truth is that nobody buys software solely based on a slick sales pitch. Everybody wants to get their hands on the piece of software first to find out whether they like it or not. And herein lies a pretty complicated problem – you want your users to fall in love with your product.  At the same time you somehow want to limit the access to the product so that you can monetize it later.

By and large you have two options here – either hand out a fully functional version of your software for a limited period of time and force users to convert to paid plans after the period passes or you somehow limit the functionality / capacity of the free version available. As most of the software industry tends to go with time-limited trials, so did we. But we soon realized we could do better.

In this post we will describe how we reached the conclusion that time-limited trials might not suit us and what we did to overcome the obstacles.

Our series of posts about solving the OutOfMemoryError in our hypothetical production system nearing the conclusion. We have covered a number of different ways to attack OutOfMemoryErrors so far – profilers, JDK bundled tools and heap dump analyzers. Our today’s mercenaries will be Application Performance Management tools, or APMs.

We are writing in from Tallinn Airport, on our way to San Francisco to the YCombinator interview. And we want to announce a major user interface update of our memory leak detector tool.

Throughout the decade of being responsible for delivering various IT projects I have involuntarily developed  a reflex – whenever I hear someone saying “Our application has a memory leak“, I immediately get shivers running down my spine.

Let us continue our series of posts about solving the OutOfMemoryError in our hypothetical production system.

We have described  different methods to tackle the problem, and today’s post concentrates on what you can learn from heap dumps. Spoiler alert: with a bit of luck, you can get very close to solving the OOM

When I ask our users what they think about Plumbr, most of the replies fall into 3 categories:

Here’s the news for you: Java memory leaks have attitude, and you need to know your way around to successfully nail and solve them. While this blog post is written in response to people who became confused while using our product for discovering memory leaks, it should serve some food for thought for other leak hunters as well.

I got an email a couple of days ago. The author had tried Plumbr on a self-made leaking test application and, to his disappointment, instead of getting a beautiful leak report he got an OutOfMemoryError. He was not the first one sending us such comments, and from the fair amount of feedback we’ve gathered so far we’ve begun seeing some patterns in how people use Plumbr.

When we talk to people about our solution for discovering memory leaks  –  Plumbr – we immediately get positive feedback. But when we add Java into the equation, the initial excitement is often complemented with questions: “Are there memory leaks in Java? Isn’t Java a garbage-collected language?”

In this post I will explain why memory leaks are in fact a common problem for Java applications.

In the last days of January, the number of Plumbr users exceeded an important milestone – we now have more than 1,000 users! As we started offering Plumbr as a public beta in the beginning of November last year, we can say that „collecting“ the first 1,000 customers took us 3 months.

Let’s dive into the numbers (we are big fans of factual proof!), and elaborate on:

• What did we do to get here?
• What have we learned from the customers and where are we planning to head with product development?
• What are we going to do to reach 10,000? 100,000? Bazillion?

We recently celebrated crossing the magic threshold of 1000 registered users.  Around 10% of those one thousand early adopters have given us feedback in some form. There is a separate blog post in progress summarizing the first 1000, and our future plans (subscribe to our twitter feed to be notified!).

But this announcement is actually dedicated to the next step in the product maturity. Fair amount of users have reported false alarms: they say that Plumbr is too sensitive and claims objects to be leaking even if those objects occupy less than 1% of heap. We have good news for you: based on the hundreds of samples of Java application object lifecycles we have gathered, the detection algorithm has been tweaked to be more „suspicious“, ie. it now avoids most of the false alerts by collecting some more evidence before announcing the verdict.

We encourage all users to upgrade.

Is it possible to explain what Plumbr is in just a few minutes? Yes, it is!

We created a screencast that walks you through installation and usage of Plumbr – it shows you how to attach it to you Java server, what should you to to make discovery of memory leaks possible and how to interpret the Plumbr report.

For those of you too lazy to press the Play button or who want to get more background information, we added some comments also below the video. Have fun watching and if you have any additional questions, leave them in the comments!

In our last blog post on the “Solving OutOfMemoryError” series we looked at profilers and concluded that profiling alone doesn’t help you solve memory leaks.

Today we will talk about the command line tools that are bundled with the Oracle JDK and can be used to find memory leaks. The benefit of knowing the bundled tooling is obvious: they are available everywhere where Oracle’s Java is installed, i.e. often also in these test and production environments where other fancy tools are either not installed.

It is about time to continue our Solving OutOfMemoryError blog post series. In retrospect, so far we have covered: Part 1 described the Story of solving an OutOfMemoryError through the eyes of a Developer, Part 2 explained how the Ops usually tackle the OutOfMemoryError problem, and Part 3  started looking at where to start solving the OutOfMemoryError. A couple of next posts will now look at the existing tools that you can use to find a Java memory leak.

We are happy to announce Plumbr 1.0RC3 – a new release that greatly improves speed and leak detection accuracy, includes a demo application for fast demoing of Plumbr and fixes some smaller bugs.  We are a big step closer to finally releasing the first official Plumbr 1.0, and perhaps also removing the BETA tag from our logo

On our quest to solve every memory leak in the Java world we get in touch with many teams who struggle with performance issues of their applications. Every developer tries to solve these differently. Some begin with vmstat or top, others with a CPU profiler or a DB load monitoring tool. Many of them turn to different APM products, like  CA APM, AppDynamics, dynaTrace, etc.

However, if you properly ask the application, it will normally give you all the information you need for performance tuning. And you can get the info without relying on indirect metrics of the operating system or the JVM.

First of all a big thanks to everyone who rushed in over the last days – the first public version of Plumbr turned out to be extremely popular! Thanks for keeping us up the days and nights, collecting feedback and improving the product.

Java Collection API is one of the most used APIs in the Java world. It provides a convenient and pretty solid way to implement and use some every-day data structures. But are they suitable and efficient in all situations?

In this article I will investigate one case from our own experience where Java Collection API turned out to be a huge waste of memory in a pretty trivial usage scenario. 

Starting from today, the memory leak detector Plumbr is available to everyone as a public beta. This release marks a major milestone for us. After months of development we can now proudly present the first public release!

This means that:

• The leak detection algorithm works – we have discovered more than 20 leaks in our private beta customers’ projects
• We are confident that Plumbr doesn’t crash your application and doesn’t create any noticeable overhead
• We have verified that Plumbr works with JDK 6 and 7 versions on different vendors (Hotspot, jRockit)
• Plumbr is available for everyone with a 45-day free evaluation licence

How do I know that the application is actually suffering from memory leaks? Where and how can I find the cause of the possible memory leak? An experienced developer usually begins its troubleshooting by answering the aforementiond fundamental questions.

Lets see a typical flow when trying to answer those two simple questions.

1. First – you should find evidence hidden somewhere in the logs. If you are lucky you can have a peek at production logs. In most cases there is a pack of angry Kerberos – like guys guarding the access – both from operations and security units of the corporation.
2. Lets assume that you can bypass the walls separating you from the vital information. Is it anyhow helpful? Most likely not. You find your good’ol friend Exception in thread "main" java.lang.OutOfMemoryError: while grepping through logs, but nothing interesting seems to precede its creation. There might be hints from monitoring logs that performance degradation started minutes before the actual error. But besides this – business as usual.

You are a developer. You should not be solving problems occurring in production once in a blue moon. But here you are again, trying to figure out what has gone wrong again. Why are you here and what could be done differently to avoid you solving these problems?

After all – your operations is also full of competent people. They have tried their best to pinpoint the root cause, by pulling different tricks from their book of knowledge. What we have faced is that the methods used might not always be the most beneficial:

Another day in the office. Except that you get another call from the operations saying they now have to restart your production application every three hours. And they seem to be somewhat annoyed with this. You have a feeling that their emotion is in correlation to the number of phone calls customer support is getting lately – which has skyrocketed in the past few weeks.

I’m sure you have been there. And I am also sure you wish you would never have to be there again.

All you have is a symptom: an OutOfMemoryError. You have no clue what causes it. You cannot reproduce the problem. You cannot see the log files in production. But you have to solve it. Not that you have the task in your job contract – you are just a developer – the reasons are rather related to your engineering pride and being a responsible human being.

And here we go again. Sleepless nights or weeks. Struggling with authorization issues and bureaucracy. Getting ignorant people find the correct data for you. Great. Just great. Having the boss breathe down on your neck does not help much either. “When will it be fixed?” I bet you’d rather estimate when the P versus NP problem will be solved.