OutOfMemoryError: PermGen space

[mkieboom]

Hi there,

Within our company we have Alfresco running within VMWare for a few months now however still we have problems with Permgen space error from Java.
The error occurs every 2 or 3 days while the virtual machine is definetely powerfull enough to run Alfresco.

I can not find any website or blog with the right solution to the problem.
Can anyone help me with this problem? The wiki from Alfresco also did not give me the right solution.
http://wiki.alfresco.com/wiki/ASR_Performance_Optimization

Kind regards,


Marcel Kieboom

export JAVA_OPTS='-Xms2G -Xmx4G -Xss1024k -XX:MaxPermSize=256m -XX:NewSize=1G -server'
export JAVA_OPTS='-Xms512m -Xmx768m -Xss768k -XX:MaxPermSize=256m -XX:NewSize=256m -server'
export JAVA_OPTS='-Xms512m -Xmx1024m -Xss1024k -XX:MaxPermSize=256m -XX:NewSize=256m -server'
export JAVA_OPTS='-Xms1G -Xmx2G -Xss1024k -XX:MaxPermSize=256m -XX:NewSize=512m -server'
export JAVA_OPTS="${JAVA_OPTS} -Dalfresco.home=${ALF_HOME} -Dcom.sun.management.jmxremote"
export JAVA_OPTS='-Xms768m -Xmx1536m -Xss1m -XX:MaxPermSize=128m -Xcomp -Xbatch -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=384m -XX:CMSInitiatingOccupancyFraction=80 -server'
export JAVA_OPTS='-Xms768m -Xmx1536m -Xss1024k -XX:MaxPermSize=256m -Xcomp -Xbatch -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=512m -XX:CMSInitiatingOccupancyFraction=80 -server'
export JAVA_OPTS='-Xms768m -Xmx1536m -Xss1024k -XX:MaxPermSize=382m -Xcomp -Xbatch -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:NewSize=512m -XX:CMSInitiatingOccupancyFraction=80 -server'
‍‍‍‍‍‍‍‍‍‍

The code above shows the settings I made / changed where the last line is the current setting.

Alfresco Community v4.0.0 (3979) schema 5025

Here some pieces of the log file when the error occured:

=====================================================================================================================================================

Exception in thread "http-8080-12" java.lang.OutOfMemoryError: PermGen space
Exception in thread "JobAcquisitionThread" java.lang.OutOfMemoryError: PermGen space
Exception in thread "http-8080-62" java.lang.OutOfMemoryError: PermGen space
Exception in thread "http-8080-31" java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI RenewClean-[192.168.1.213:50502,org.alfresco.util.remote.server.socket.HostConfigurableSocketFactory@0]" java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI RenewClean-[192.168.1.213:50501,org.alfresco.util.remote.server.socket.HostConfigurableSocketFactory@0]" java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI RenewClean-[192.168.1.213:50504,org.alfresco.util.remote.server.socket.HostConfigurableSocketFactory@0]" java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI RenewClean-[192.168.1.213:50506,org.alfresco.util.remote.server.socket.HostConfigurableSocketFactory@0]" java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI RenewClean-[192.168.1.213:50508]" java.lang.OutOfMemoryError: PermGen space
2012-05-29 13:27:19,199  ERROR [extensions.webscripts.AbstractRuntime] [http-8080-12] Exception from executeScript - redirecting to status template error: 04290002 Wrapped Exception (with status template): 042923501 Failed to execute script 'classpath*:alfresco/templates/webscripts/org/alfresco/slingshot/documentlibrary/doclist.get.js': 042923500 PermGen space
 2012-05-29 13:27:19,200  ERROR [extensions.webscripts.AbstractRuntime] [http-8080-31] Exception from executeScript - redirecting to status template error: 04290003 Wrapped Exception (with status template): 042923506 Failed to execute script 'classpath*:alfresco/templates/webscripts/org/alfresco/repository/site/site.get.js': 042923504 Exception from transactional callback: org.alfresco.repo.transaction.RetryingTransactionAdvice$1@c87c354
 2012-05-29 13:28:55,095  ERROR [job.executor.JobExecutorThread] [AlfrescoJbpmJobExecutor:192.168.1.213:1] exception in job executor thread. waiting 90000 milliseconds
 2012-05-29 13:29:00,056  ERROR [transaction.SpringAwareUserTransaction.trace] [Finalizer] Detected first UserTransaction which is being garbage collected without a commit() or rollback()
 2012-05-29 13:29:00,056  ERROR [transaction.SpringAwareUserTransaction.trace] [Finalizer] Logging of transaction call stack is now enabled and will affect performance
 Exception in thread "RMI RenewClean-[192.168.1.213:50505,org.alfresco.util.remote.server.socket.HostConfigurableSocketFactory@0]" java.lang.OutOfMemoryError: PermGen space
Exception in thread "RMI RenewClean-[192.168.1.213:55574]" java.lang.OutOfMemoryError: PermGen space
Exception in thread "AlfrescoJbpmJobExecutor:192.168.1.213:1" java.lang.OutOfMemoryError: PermGen space
2012-05-29 13:29:22,231  ERROR [quartz.simpl.SimpleThreadPool] [DefaultScheduler_Worker-3] Error while executing the Runnable:
 java.lang.OutOfMemoryError: PermGen space
2012-05-29 13:27:53,380  INFO  [webscripts.connector.RemoteClient] [http-8080-62] Error status 499 Read timed out
 2012-05-29 13:29:25,649  ERROR [extensions.webscripts.AbstractRuntime] [http-8080-18] Exception from executeScript - redirecting to status template error: (class: org/springframework/extensions/webscripts/WebScriptException, method: getStatusModel signature: ()Ljava/util/Map;) Illegal constant pool index
 2012-05-29 13:29:30,546  INFO  [webscripts.connector.RemoteClient] [http-8080-27] Error status 499 Read timed out
 2012-05-29 13:29:33,411  ERROR [surf.render.RenderService] [http-8080-18] The following error occurred attemping to render a sub-component
 java.lang.OutOfMemoryError: PermGen space
2012-05-29 13:29:35,794  ERROR [transaction.SpringAwareUserTransaction.trace] [Finalizer] UserTransaction being garbage collected without a commit() or rollback(). NOTE: Prior to transaction call stack logging.
 2012-05-29 13:29:35,791  ERROR [extensions.webscripts.AbstractRuntime] [http-8080-27] Exception from executeScript - redirecting to status template error: (class: org/springframework/extensions/webscripts/WebScriptException, method: getStatusModel signature: ()Ljava/util/Map;) Illegal constant pool index
 2012-05-29 13:29:40,776  ERROR [surf.render.RenderService] [http-8080-27] The following error occurred attemping to render a sub-component
 java.lang.OutOfMemoryError: PermGen space
2012-05-29 13:29:58,409  ERROR [quartz.simpl.SimpleThreadPool] [DefaultScheduler_Worker-7] Error while executing the Runnable:
 java.lang.OutOfMemoryError: PermGen space
2012-05-29 13:30:03,703  ERROR [extensions.webscripts.AbstractRuntime] [http-8080-34] Exception from executeScript - redirecting to status template error: PermGen space
 java.lang.OutOfMemoryError: PermGen space
2012-05-29 13:30:13,451  ERROR [transaction.SpringAwareUserTransaction.trace] [Finalizer] UserTransaction being garbage collected without a commit() or rollback(). NOTE: Prior to transaction call stack logging.
 2012-05-29 13:30:13,451  ERROR [transaction.SpringAwareUserTransaction.trace] [Finalizer] UserTransaction being garbage collected without a commit() or rollback(). NOTE: Prior to transaction call stack logging.

 
=====================================================================================================================================================

2012-05-23 09:00:00,091  INFO  [security.sync.ChainingUserRegistrySynchronizer] [DefaultScheduler_Worker-6] Synchronize User Creation and Association: Completed batch of 3 entries
 2012-05-23 09:00:00,125  INFO  [security.sync.ChainingUserRegistrySynchronizer] [DefaultScheduler_Worker-6] Finished synchronizing users and groups with user registry 'Synchronize'
 2012-05-23 09:00:00,125  INFO  [security.sync.ChainingUserRegistrySynchronizer] [DefaultScheduler_Worker-6] 3 user(s) and 0 group(s) processed
 Exception in thread "AlfrescoJbpmJobExecutor:192.168.1.213:1" Exception in thread "DefaultScheduler_QuartzSchedulerThread" Exception in thread "http-8080-59" 2012-05-23 09:21:49,674  ERROR [extensions.webscripts.AbstractRuntime] [http-8080-59] Exception from executeScript - redirecting to status template error: PermGen space
 2012-05-23 09:22:16,704  ERROR [transaction.SpringAwareUserTransaction.trace] [Finalizer] Detected first UserTransaction which is being garbage collected without a commit() or rollback()
 2012-05-23 09:22:16,705  ERROR [transaction.SpringAwareUserTransaction.trace] [Finalizer] Logging of transaction call stack is now enabled and will affect performance

=====================================================================================================================================================
 2012-05-22 13:42:18,526  ERROR [post.lookup.PostLookup] [DefaultScheduler_Worker-3] Exception during update of posts
 java.lang.OutOfMemoryError: PermGen space
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2818)
        at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1159)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1647)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526)
        at org.alfresco.repo.activities.post.lookup.PostLookup.execute(PostLookup.java:165)
        at org.alfresco.repo.activities.post.lookup.PostLookupJob.execute(PostLookupJob.java:50)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Exception in thread "JobAcquisitionThread" java.lang.OutOfMemoryError: PermGen space
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2444)
        at java.lang.Class.getMethod0(Class.java:2687)
        at java.lang.Class.getMethod(Class.java:1620)
        at org.apache.commons.logging.LogFactory.directGetContextClassLoader(LogFactory.java:896)
        at org.apache.commons.logging.LogFactory$1.run(LogFactory.java:862)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.commons.logging.LogFactory.getContextClassLoaderInternal(LogFactory.java:859)
        at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:423)
        at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685)
        at org.springframework.transaction.support.TransactionTemplate.<init>(TransactionTemplate.java:66)
        at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:38)
        at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
        at org.activiti.engine.impl.jobexecutor.JobAcquisitionThread.run(JobAcquisitionThread.java:63)
2012-05-22 13:43:05,467  ERROR [post.lookup.PostLookup] [DefaultScheduler_Worker-7] Exception during update of posts
 java.lang.OutOfMemoryError: PermGen space
=====================================================================================================================================================
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[bisana]

This happened to me, when I did stopped the service and then restarted the service
When you stop 4.0.d , there are some java instance which is not stopped, this they are going to solve in the next release.
What I do after ,

 /opt/alfresco-4.0.d/alfresco.sh stop‍

I check the process using

 ps aux ‍

and kill the process
and then start the service by 

 /opt/alfresco-4.0.d/alfresco.sh start‍

Try out this and let us know the status
Thanks
Joseph John

[heiko_robert]

Hi Marcel,

which OS / which Tomcat (from Alfrsco install, from distribtion) do you use? Why do you export the settings several times and where did you make your changes?

You can see the effective java parameters by

ps aux|grep java‍

MaxPermSize=256m should be OK for a vanilla Alfresco installation. Did you install additional AMPs? You could increase this parameter to 512m and try again.

We don't use the Alfresco installer and it's start/stop-scripts. The (tomcat-)init.d-scripts of the latest ubuntu/debian distribution do a good job, also for restart.

Regards
Heiko

[mkieboom]

Hi there,


Thank you for both your replies.

>>Hi Joseph,
It also happens after a complete reboot of the ubuntu machine and a fresh startup of Alfresco, so at that moment I think it can not be the case that there are old processes which are still running.

>>Heiko,
It is not the case that I export multiple settings although I changed the settings over a period of time and each of them gives me the same problem. The last line of the exports is the one which is currently set.
I installed 4.0.0.c community edition (Alfresco Community v4.0.0 (3979) schema 5025)

It operates on: Ubuntu 10.04.4 LTS
Server version: Apache Tomcat/6.0.35
Server built:   Nov 28 2011 11:20:06
Server number:  6.0.35.0
OS Name:        Linux
OS Version:     2.6.32-38-generic
Architecture:   amd64
JVM Version:    1.6.0_20-b20
JVM Vendor:     Sun Microsystems Inc.

I use a alfresco startup script I'm not sure if this is still needed ?


And just when I wrote this it happened again:

 Exception in thread "JobAcquisitionThread" java.lang.OutOfMemoryError: PermGen space
2012-05-30 14:39:12,426  ERROR [activities.feed.AbstractFeedGenerator] [DefaultScheduler_Worker-10] Exception during generation of feeds
 2012-05-30 14:39:17,287  ERROR [quartz.core.JobRunShell] [DefaultScheduler_Worker-10] Job DEFAULT.feedGeneratorJobDetail threw an unhandled Exception:
 java.lang.OutOfMemoryError: PermGen space
2012-05-30 14:39:19,796  ERROR [quartz.simpl.SimpleThreadPool] [DefaultScheduler_Worker-10] Error while executing the Runnable:
 java.lang.OutOfMemoryError: PermGen space
2012-05-30 14:39:24,763  ERROR [transaction.support.TransactionSynchronizationUtils] [http-8080-31] TransactionSynchronization.afterCompletion threw exception
 java.lang.OutOfMemoryError: PermGen space
Exception in thread "http-8080-31" java.lang.OutOfMemoryError: PermGen space
2012-05-30 14:39:29,624  ERROR [extensions.webscripts.AbstractRuntime] [http-8080-31] Exception from executeScript - redirecting to status template error: 04305444 Exception in Transaction.
 2012-05-30 14:39:49,425  ERROR [activities.feed.AbstractFeedGenerator] [DefaultScheduler_Worker-6] Exception during generation of feeds
 2012-05-30 14:39:52,159  ERROR [quartz.core.JobRunShell] [DefaultScheduler_Worker-6] Job DEFAULT.feedGeneratorJobDetail threw an unhandled Exception:
 java.lang.OutOfMemoryError: PermGen space
2012-05-30 14:39:54,300  ERROR [quartz.simpl.SimpleThreadPool] [DefaultScheduler_Worker-6] Error while executing the Runnable:
 java.lang.OutOfMemoryError: PermGen space
2012-05-30 14:40:00,228  ERROR [activities.feed.AbstractFeedGenerator] [DefaultScheduler_Worker-7] Exception during generation of feeds
 2012-05-30 14:40:02,632  ERROR [quartz.core.JobRunShell] [DefaultScheduler_Worker-7] Job DEFAULT.feedGeneratorJobDetail threw an unhandled Exception:
 java.lang.OutOfMemoryError: PermGen space
2012-05-30 14:40:04,934  ERROR [quartz.simpl.SimpleThreadPool] [DefaultScheduler_Worker-7] Error while executing the Runnable:
 java.lang.OutOfMemoryError: PermGen space
OpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGTERM to handler- the VM may need to be forcibly terminated
Exception in thread "MessageDispatcher" java.lang.OutOfMemoryError: PermGen space
May 30, 2012 2:40:37 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I hope to be able to solve this problem because it is pretty annoying.

Kind regards,


Marcel Kieboom

[bisana]

JVM Vendor: Sun Microsystems Inc.

for me when I am using Java which came along with alfresco installation
for me when I give java -version I get

[alfresco@r12docs ~]$ java -version
java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)‍‍‍‍

Are u getting the same
I assume that the problem is because of two  java installation
Before installing alfresco you had Java installed already  and after installing Alfresco you have another Java instance
I guess this is the problem, please check and let us know

[mward]

Have you tried using jconsole to inspect the vm? On the VM Summary tab you can check that the PermSize arguments are being passed to the VM.

It is worth checking just to make sure that your settings are taking effect - if for example you were running tomcat as a different user to the one where the JAVA_OPTS are being exported then you won't see the correct settings.

Also, on the Memory tab, change the chart to CMS Perm Gen and see how much is being used.

[mkieboom]

Bisana, I did not install the version you mentioned however I installed the openJDK version

java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.13) (6b20-1.9.13-0ubuntu1~10.04.1)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)
‍‍‍‍‍

I did not use the jconsole to analyze the Java VM, so I will do that and analyze it.

Thanks a lot!
I will post any results…

[mrogers]

Open JDK will have different optional command line options to the sun JVM - especially around the issues of memory management.    I suspect you are trying to use sun specific config with a different JVM.

[bisana]

removing your Sun java excetuable may solve this issue
Let us try out this for troubleshooting, may this will work, try out after stopping your existing Alfresco 

whereis java ‍

you may get the result contain

/usr/bin/java‍

go to /usr/bin
rename java to javaold
again try giving "java -version", u should not be getting any result, if you are getting any result means, you have "java"  executable in your path

Now try installing alfresco, . Please give "Yes" for java installation
After the installation over
start alfreso and see the log files, I think this procedure will work, since your sun java is removed from the path
prior to alfresco installation

[mkieboom]

Hi,

Open JDK will have different optional command line options to the sun JVM - especially around the issues of memory management.    I suspect you are trying to use sun specific config with a different JVM.

I have installend OpenJDK 6 so this could be the problem, however where can I find the differences between Sun Java and OpenJDK6 configs,  or what could be wrong with my export settings above?

removing your Sun java excetuable may solve this issue

Sorry but I don't have Sun Java running it is the OpenJDK.
I know this is a bit vague with my post above however this was directly from the server

Kind regards,


Marcel Kieboom