Java "OutOfMemory: permgen" errors on Windows install

Ignite is not monitoring some/all of my database instances, and I see "out of memory" errors for something called "permgen" in the Ignite logs.  This article explains what this means and what you can do about it.

Background

According to Sun, it appears to be either a memory bloating issue, or a bug in permanent generation garbage collecting, or perhaps both. 

Fixes are documented in 1.5.0.23.5 (and up) and 1.6.0.19.6 (and up). The 1.6 fix is in the “for business” release, so it is not clear that it is in the free version, but we are hopeful that they’ve rolled this fix into the later versions (1.6.0.20 and higher).

• http://bugs.sun.com/view_bug.do?bug_id=4957990
• http://java.sun.com/javase/javaseforbusiness/docs/6/overview.html

Perm Gen space typically defaults to 64m, but it can be increased using –XX:PermSize=128m (or higher).

Solution

The following steps should resolve this issue:

1. Install JDK 1.6.0.20 or higher
   1. Download and install the JDK
   2. Make a copy of <Ignite Home>\iwc\tomcat\ignite_config\java_loc.txt
   3. Update java_loc.txt with the full path including the java executable for the newly-installed JDK
      
      Note: java_loc.txt will override any system setting to JAVA_HOME for Ignite only when it’s started or shutdown.

2. Increase permanent generation size to 128m, for the Ignite PI Service, you will need to edit the registry.

1. As the same user that installed Ignite, open a Command Window from the Start menu
2. At the command prompt, enter>  regedit
3. Navigate to the following 'Options' in regedit:

              HKEY_LOCAL_MACHINE>SOFTWARE>Wow6432Node>

                    Apache Software Foundation>Procrun 2.0>IgnitePI>Java>Options

d. Open up the Option and at the bottom of the list, insert this line:

--XX:MaxPermSize=128m

e. See Example below:

3. Click OK and then exit out of regedit.
4. Restart the Ignite PI Service
   1. Select Start Menu > Control Panel > Administrative Tools > Services 
   2. Look for the service, "Ignite PI Server" and stop/start or restart it.

Verify Permgen Changes

To validate that the changes were made, you will need to bring up jconsole.  This section describes how to bring up jconsole, connect and verify the changes.

1. Bring up jconsole
   1. In Windows Explorer, navigate to <ignite dir>\Ignite PI\iwc\jre\bin
   2. Locate jconsole.exe and run it.
   3. In a screen, like the example below, choose the 'Remote Process' button and enter the following connection string:

                 service:jmx:rmi://localhost:11099/jndi/rmi://localhost:11099/idc

2.  After connecting, select the memory tab and choose the "Memory Pool Perm Gen" setting in the drop down box:

3. Check the "Max:" setting at the bottom of the screen.  It should be set at 131,072 kbytes.

When Upgrading Ignite or Re-creating the Ignite Service

It is important to know that if you upgrade Ignite to a new version, or if you recreate the Ignite PI service by running the createWindowsService.bat in Ignite's install directory, these changes will be lost and you will have to reapply the MaxPermSize settings.

Considerations

If you can’t immediately upgrade to the new java version, try increasing the PermSize to 128m as this may allow more time between the fatal errors. Also, consider stopping/starting Ignite on a weekly basis so to reduce the chances of getting memory errors. Another way to reset Ignite without a regular schedule, may be to query the heartbeat parameter in conprm periodically. If the heartbeat hasn’t changed in some interval of time, then stop / start Ignite again.