Java "OutOfMemory: permgen" errors on Unix/Linux install

Ignite is not monitoring some/all of my database instances, and I see "out of memory" errors for something called "permgen" in the Ignite logs.  This article explains what this means and what you can do about it.

Background

According to Sun, it appears to be either a memory bloating issue, or a bug in permanent generation garbage collecting, or perhaps both. 

Fixes are documented in 1.5.0.23.5 (and up) and 1.6.0.19.6 (and up). The 1.6 fix is in the “for business” release, so it is not clear that it is in the free version, but we are hopeful that they’ve rolled this fix into the later versions (1.6.0.20 and higher).

• http://bugs.sun.com/view_bug.do?bug_id=4957990
• http://java.sun.com/javase/javaseforbusiness/docs/6/overview.html

Perm Gen space typically defaults to 64m, but it can be increased using –XX:PermSize=128m (or higher).

Solution

The following steps should resolve this issue:

1. Install JDK 1.6.0.20 or higher
   1. Download and install the JDK
   2. Make a copy of <Ignite Home>/iwc/tomcat/ignite_config/java_loc.txt
   3. Update java_loc.txt with the full path including the java executable for the newly-installed JDK
      
      Note: java_loc.txt will override any system setting to JAVA_HOME for Ignite only when it’s started or shutdown.

2.  Increase permanent generation size to 128m, as follows:
   1. Open <Ignite Home>/iwc/tomcat/bin/catalina.sh
   2. Search for line that looks like:
      JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
   3. Insert the following line after the above line:
      JAVA_OPTS="$JAVA_OPTS -XX:PermSize=128m"

3. Restart Ignite PI
   1. Run shutdown.sh and then startup.sh
       

Considerations

If you can’t immediately upgrade to the new java version, try increasing the PermSize to 128m as this may allow more time between the fatal errors. Also, consider stopping/starting Ignite on a weekly basis so to reduce the chances of getting memory errors. Another way to reset Ignite without a regular schedule, may be to query the heartbeat parameter in conprm periodically. If the heartbeat hasn’t changed in some interval of time, then stop / start Ignite again.