What is a PermGen leak?

July 11, 2012 by Nikita Salnikov-Tarnovski

What follows is a practical introduction to a specific type of memory problems in Java applications. Namely – we will analyze the errors that cause the java.lang.OutOfMemoryError: PermGen space symptom in the stack trace.

First of all we will go through the core concepts required to understand the subject – and explain what objects, classes, classloaders and the JVM memory model are. If you are familiar with the basic concepts, you can jump directly to the next section where I will describe two typical cases for the error in question alongside with hints and suggestions for solving it.

Objects, Classes and ClassLoaders

Well, I will not start with the very basics. I guess if you have already found us, you should be familiar with the concept that everything in Java is an Object. And that all Objects are specified by their Class. So every object has a reference to an instance of java.lang.Class describing the structure of that object’s class.

But what actually happens under the hood, when you create a new object in your code? For example if you write something truly complicated like

Person boss = new Person()

The Java Virtual Machine (JVM) needs to understand the structure of the object to create. To achieve this, the JVM looks for the class called Person. And if the Person class is accessed for the first time during this particular execution of the program, it has to be loaded by the JVM, normally from the corresponding Person.class file. The process of seeking for the Person.class file on the drive, loading it into memory and parsing it’s structure is called class loading. Ensuring proper class loading process is the responsibility of a ClassLoader. ClassLoaders are instances of java.lang.ClassLoader class and each and every class in a Java program has to be loaded by some ClassLoader. As a result we now have the following relationships:

relationships

As you can see from the next diagram every classloader holds references to all the classes it has loaded. For the purpose of our article these relationships are very interesting.

references

Remember this image, we will need it later.

Permanent Generation

Almost every JVM nowadays uses a separate region of memory, called the Permanent Generation (or PermGen for short), to hold internal representations of java classes. PermGen is also used to store more information – find out the details from this post if you are interested – but for our article it is safe to assume that only the class definitions are being stored in PermGen. The default size of this region on my two machines running java 1.6 is not a very impressive 82MB.

As I have explained in one of my earlier posts, a memory leak in Java is a situation where some objects are no longer used by an application, but the Garbage Collector fails to recognize them as unused. This leads to the OutOfMemoryError if those unused objects contribute to the heap usage significantly enough that the next memory allocation request by the application cannot be fulfilled.

The root cause of java.lang.OutOfMemoryError: PermGen space is exactly the same: the JVM needs to load the definition of a new class but there is not enough space in PermGen to do it – there are already too many classes stored there. A possible reason for this could be your application or server using too many classes for the current size of PermGen not to be able to accommodate them. Another commone reason could be a memory leak.

Permanent Generation Leak

But still, how on earth it is possible to leak something in PermGen? It holds definitions of java classes and they cannot become unused, can they? Actually, they can. In case of a Java web application deployed into an application server all those classes in your EAR/WAR become worthless when the application is undeployed. The JVM continues to run as the application server is still alive, but a whole bunch of class definitions are not in use anymore. And they should be removed from PermGen. If not, we will have memory leak in the PermGen area.

As a nice sample on the reasons – the Tomcat developers have set up a Wiki page describing different leaks found and fixed in the Apache Tomcat versions 6.0.24 and above.

Leaking Threads

One possible scenario for a classloader leak is through long running threads. This happens when your application or – as often was the case in my experience – a 3rd party library used by your application, starts some long running thread. An example of this could be a timer thread whose job is to execute some code periodically.

If the intended lifespan of this thread is not fixed, we are heading directly into a trouble. When any part of your application ever starts a thread, you must make sure that it is not going to outlive the application. In typical cases the developer either is not aware of this responsibility or simply forgets to write the clean-up code.

Otherwise, if some thread continues to run after the application is undeployed, it will, usually, hold a reference to a classloader of the web application it was started by, called context classloader. Which in turn means that all classes of the undeployed application continue to be held in memory. Remedy? If it is your application that starts new threads, you should shut them down during undeployment using a servlet context listener. If it is a 3rd party library, you should search for its own specific shutdown hook. Or file a bug report if there is none.

Leaking Drivers

Another typical case of a leak can be caused by database drivers. We have encountered this leak in our own demo application that we ship with Plumbr. It is a slightly modified Pet Clinic application shipped along with Spring MVC. Let us highlight some things that happen when this application is being deployed to the server.

  • The server creates a new instance of java.lang.Classloader and starts to load the application’s classes using it.
  • Since the PetClinic uses a HSQL database, it loads the corresponding JDBC driver, org.hsqldb.jdbcDriver
  • This class, being a good-mannered JDBC driver, registers itself with java.sql.DriverManager during initialization, as required per JDBC specification. This registration includes storing inside a static field of DriverManager a reference to an instance of org.hsqldb.jdbcDriver.

Now, when the application is undeployed from the application server, the java.sql.DriverManager will still hold that reference, as there is no code in the HSQLDB library nor in the Spring framework nor in the application to remove that! As was explained above, a jdbcDriver object still holds a reference to an org.hsqldb.jdbcDriver class, which in turn holds a reference to the instance of java.lang.Classloader used to load the application. This classloader now still reference all the classes of the application. In case of our particular demo application, during application startup almost 2000 classes are loaded, occupying roughly 10MB in PermGen. Which means that it takes about 5-10 redeploys to fill the PermGen with default size to reach the java.lang.OutOfMemoryError: PermGen space crash.

How to fix that? One possibility is to write a servlet context listener, which de-registers the HSQLDB driver from DriverManager during application shutdown. This is pretty straightforward. But remember – you will have to write the corresponding code in every application using the driver.

Download our latest version of Plumbr with our demo application and play with it to find out how the leak occurs, how Plumbr finds it and how do we explain the cause.

Conclusion

There are many reasons why your application might encounter a java.lang.OutOfMemoryError: PermGen space. The root cause for the majority of them is some reference to an object or a class loaded by the application’s class loader that has died after that. Or a direct link to the class loader itself. The actions you need to take for remedy are quite similar for most of these causes. Firstly, find out where that reference is being held. Secondly, add a shutdown hook to your web application to remove the reference during application’s undeployment. You can do that by either using a servlet context listener or by using the API provided by your 3rd party library.

Finding those leaking references has never been easy. We ourselves have spent countless hours trying to trace down why some applications require 20MB of PermGen on each redeploy. But as of version 1.1,, Plumbr will show you the cause of the leak and give you a hint on how to fix it. If you want to try it out, register and download the tool. If you are running an older version of Plumbr, we strongly recommend downloading an upgrade.

 

Can't figure out what causes your OutOfMemoryError? Read more

ADD COMMENT

COMMENTS

Hi,nThanks for the great explanation. From this,i understand that permgen OOMs can occur only if the server remains running and the app is re-deployed multiple times. Can the permgen space continue to go even if server is restarted?

permgen-victim

Yes – for example when your application generates classes on the fly then those also end up in permgen and can cause the permgen space to overflow.

Ivo Mägi

Yes, we use Hibernate, Axis, JAXB etc. which would create classes on the fly. There’s a huge jump in the permgen space usage after the server has restarted (ie all initializations complete). Can we track the trigger for this using Plumbr?

permgen-victim

Really good article. Thanks a lot.

Xin Han

Very comphrehensive..very well written..this is an article that’s absolutely worth keeping in the references bookmarks!nnThanks

Guest

Excelent post!! nnBest regards

Marcus Andrade

One of the classic manifestation of PermGen leak is Java.lang.OutOfMemoryError:PermGen Space in Tomcat, caused by either ThreadLocal variables or JDBC drivers and can easily reproduced by redeploying the web app couple of time. If any class outside of web-app class loader holds reference of class loaded by this classloader it prevents not only that class but all others from being GC’d and result in permgen leak.

Javin Paul

Yes, you are correct, I was just trying to say you can easily see these issues if you are working in tomcat and redeploying your app quite frequent.

Javin Paul

A very well explained informative article. Thanks for sparing time and sharing your knowledge.

Nilesh Subscribe

Good One – Crisp and Comphrehensive

Rahul

thank you sir, i always get confused with PermGen and PermSize…

Nitesh Chauhan

Can't figure out what causes your OutOfMemoryError? Read more

Latest
Recommended
You cannot predict the way you die
When debugging a situation where systems are failing due to the lack of resources, you can no longer count on anything. Seemingly unrelated changes can trigger completely different messages and control flows within the JVM. Read more
Tuning GC - it does not have to be that hard
Solving GC pauses is a complex task. If you do not believe our words, check out the recent LinkedIn experience in garbage collection optimization. It is a complex and tedious task, so we are glad to report we have a whole lot simpler solution in mind Read more
Building a nirvana
We have invested a lot into our continuous integration / delivery infrastructure. As of now we can say that the Jenkins-orchestrated gang consisting of Ansible, Vagrant, Gradle, LiveRebel and TestNG is something an engineer can call a nirvana. Read more
Creative way to handle OutOfMemoryError
Wish to spend a day troubleshooting? Or make enemies among sysops? Registering pkill java to OutOfMemoryError events is one darn good way to achieve those goals. Read more