3.2 New Install and GC overhead limit exceeded

[sergisat]

Hello, my new dev server is Ubuntu Server 9.04 64 bits in a virtual machine:

4vcpus
16GB RAM
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)
Mysql in another server

New installation of Alfresco-Community-3.2-Linux-x86-Install, I try to copy files from 5 pc's over CIFS and after 15 minutes I get this errors:

INFO: Server startup in 268752 ms
09:04:35,454  ERROR [hibernate.transaction.JDBCTransaction] JDBC rollback failed
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Connection.close() has already been called. Invalid operation in this state.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
        at com.mysql.jdbc.Util.getInstance(Util.java:381)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)
        at com.mysql.jdbc.ConnectionImpl.getMutex(ConnectionImpl.java:3032)
        at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4606)
        at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
        at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
        at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
        at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:597)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.doRollbackOnCommitException(AbstractPlatformTransactionManager.java:803)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:697)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314)
        at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:467)
        at org.alfresco.filesys.alfresco.AlfrescoDiskDriver.endTransaction(AlfrescoDiskDriver.java:227)
        at org.alfresco.jlan.server.SrvSession.endTransaction(SrvSession.java:523)
        at org.alfresco.jlan.smb.server.SMBSrvSession.sendResponseSMB(SMBSrvSession.java:1574)
        at org.alfresco.jlan.smb.server.NTProtocolHandler.procNTCreateAndX(NTProtocolHandler.java:5820)
        at org.alfresco.jlan.smb.server.NTProtocolHandler.runProtocol(NTProtocolHandler.java:306)
        at org.alfresco.jlan.smb.server.SMBSrvSession.runHandler(SMBSrvSession.java:1354)
        at org.alfresco.jlan.smb.server.SMBSrvSession.processPacket(SMBSrvSession.java:1446)
        at org.alfresco.jlan.smb.server.nio.NIOCIFSThreadRequest.runRequest(NIOCIFSThreadRequest.java:106)
        at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:141)
        at java.lang.Thread.run(Thread.java:619)
09:04:42,710  WARN  [jdbc.support.SQLErrorCodesFactory] Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is java.sql.SQLException: Connection com.mysql.jdbc.JDBC4Connection@44f7bb41 is closed.
Caused by: java.sql.SQLException: Connection com.mysql.jdbc.JDBC4Connection@44f7bb41 is closed.
        at org.apache.commons.dbcp.DelegatingConnection.checkOpen(DelegatingConnection.java:354)
        at org.apache.commons.dbcp.DelegatingConnection.getMetaData(DelegatingConnection.java:310)
        at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.getMetaData(PoolingDataSource.java:234)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:50)
        at $Proxy5.getMetaData(Unknown Source)
        at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:189)
        at org.springframework.jdbc.support.JdbcUtils.extractDatabaseMetaData(JdbcUtils.java:224)
        at org.springframework.jdbc.support.SQLErrorCodesFactory.getErrorCodes(SQLErrorCodesFactory.java:215)
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.setDataSource(SQLErrorCodeSQLExceptionTranslator.java:133)
        at org.springframework.jdbc.support.SQLErrorCodeSQLExceptionTranslator.<init>(SQLErrorCodeSQLExceptionTranslator.java:98)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.getDefaultJdbcExceptionTranslator(HibernateTransactionManager.java:741)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:715)
        at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:604)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.doRollbackOnCommitException(AbstractPlatformTransactionManager.java:803)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:697)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314)
        at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:467)
        at org.alfresco.filesys.alfresco.AlfrescoDiskDriver.endTransaction(AlfrescoDiskDriver.java:227)
        at org.alfresco.jlan.server.SrvSession.endTransaction(SrvSession.java:523)
        at org.alfresco.jlan.smb.server.SMBSrvSession.sendResponseSMB(SMBSrvSession.java:1574)
        at org.alfresco.jlan.smb.server.NTProtocolHandler.procNTCreateAndX(NTProtocolHandler.java:5820)
        at org.alfresco.jlan.smb.server.NTProtocolHandler.runProtocol(NTProtocolHandler.java:306)
        at org.alfresco.jlan.smb.server.SMBSrvSession.runHandler(SMBSrvSession.java:1354)
        at org.alfresco.jlan.smb.server.SMBSrvSession.processPacket(SMBSrvSession.java:1446)
        at org.alfresco.jlan.smb.server.nio.NIOCIFSThreadRequest.runRequest(NIOCIFSThreadRequest.java:106)
        at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:141)
        at java.lang.Thread.run(Thread.java:619)
09:04:42,711  ERROR [orm.hibernate3.HibernateTransactionManager] Commit exception overridden by rollback exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
09:04:56,753  ERROR [repo.transaction.AlfrescoTransactionSupport] After completion (rolled-back) TransactionalCache exception
org.alfresco.error.AlfrescoRuntimeException: 06080174 Failed to add txn caches to manager
        at org.alfresco.repo.cache.TransactionalCache.getTransactionData(TransactionalCache.java:221)
        at org.alfresco.repo.cache.TransactionalCache.afterRollback(TransactionalCache.java:643)
        at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.afterCompletion(AlfrescoTransactionSupport.java:824)
        at org.springframework.transaction.support.TransactionSynchronizationUtils.invokeAfterCompletion(TransactionSynchronizationUtils.java:133)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.invokeAfterCompletion(AbstractPlatformTransactionManager.java:904)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerAfterCompletion(AbstractPlatformTransactionManager.java:879)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:782)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:730)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:332)
        at org.alfresco.util.transaction.SpringAwareUserTransaction.completeTransactionAfterThrowing(SpringAwareUserTransaction.java:562)
        at org.alfresco.util.transaction.SpringAwareUserTransaction.rollback(SpringAwareUserTransaction.java:532)
        at org.alfresco.filesys.alfresco.AlfrescoDiskDriver.endTransaction(AlfrescoDiskDriver.java:216)
        at org.alfresco.jlan.server.SrvSession.endTransaction(SrvSession.java:523)
        at org.alfresco.jlan.smb.server.SMBSrvSession.processPacket(SMBSrvSession.java:1514)
        at org.alfresco.jlan.smb.server.nio.NIOCIFSThreadRequest.runRequest(NIOCIFSThreadRequest.java:106)
        at org.alfresco.jlan.server.thread.ThreadRequestPool$ThreadWorker.run(ThreadRequestPool.java:141)
        at java.lang.Thread.run(Thread.java:619)
Caused by: net.sf.ehcache.ObjectExistsException: Cache org.alfresco.storeAndNodeIdTransactionalCache_bd188298-f852-4ed9-973e-f19254ae888e_updates already exists
        at net.sf.ehcache.CacheManager.addCacheNoCheck(CacheManager.java:689)
        at net.sf.ehcache.CacheManager.addCache(CacheManager.java:683)
        at net.sf.ehcache.CacheManager.addCache(CacheManager.java:640)
        at org.alfresco.repo.cache.TransactionalCache.getTransactionData(TransactionalCache.java:216)
        … 16 more
Exception in thread "DefaultScheduler_QuartzSchedulerThread" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "NetBIOS Name Server" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "FileStateReaper" java.lang.OutOfMemoryError: GC overhead limit exceeded
09:21:14,767  ERROR [transaction.SpringAwareUserTransaction.trace] Detected first UserTransaction which is being garbage collected without a commit() or rollback()
09:21:14,767  ERROR [transaction.SpringAwareUserTransaction.trace] Logging of transaction call stack is now enabled and will affect performance
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

My alfresco.sh is:

#!/bin/sh
# Start or stop Alfresco server
# Set the following to where Tomcat is installed
ALF_HOME=/opt/Alfresco
cd "$ALF_HOME"
APPSERVER="${ALF_HOME}/tomcat"
export JAVA_HOME="/usr"
# Set any default JVM values
#export JAVA_OPTS='-Xms128m -Xmx512m -XX:MaxPermSize=128m -server'
export JAVA_OPTS='-Xms512m -Xmx1024m -XX:MaxPermSize=1024m -server'
export JAVA_OPTS="${JAVA_OPTS} -Dalfresco.home=${ALF_HOME} -Dcom.sun.management.jmxremote"
# Following only needed for Sun JVMs before to 1.5 update 8
#export JAVA_OPTS="${JAVA_OPTS} -XX:CompileCommand=exclude,org/apache/lucene/index/IndexReader\$1,doBody -XX:CompileCommand=exclude,org/
alfresco/repo/search/impl/lucene/index/IndexInfo\$Merger,mergeIndexes -XX:CompileCommand=exclude,org/alfresco/repo/search/impl/lucene/in
dex/IndexInfo\$Merger,mergeDeletions"
#
if [ "$1" = "start" ]; then
  "${APPSERVER}/bin/startup.sh"
#  if [ -r ./virtual_start.sh ]; then
#    sh ./virtual_start.sh
#  fi
#  if [ -r "$ALF_HOME"/start_oo.sh ]; then
#    sh "$ALF_HOME"/start_oo.sh
#  fi
elif [ "$1" = "stop" ]; then
  "${APPSERVER}/bin/shutdown.sh"
#  if [ -r ./virtual_start.sh ]; then
#    sh ./virtual_stop.sh
#  fi
#  if [ -r "$ALF_HOME"/start_oo.sh ]; then
#    killall soffice.bin
#  fi
fi
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

My /opt/Alfresco/tomcat/shared/classes/alfresco-global.properties

###############################
## Common Alfresco Properties #
###############################

#
# Sample custom content and index data location
#————-
#dir.root=/opt/Alfresco/alf_data
dir.root=/mnt/alfresco

#
# Sample database connection properties
#————-
db.name=alfrescopru32
db.username=alfrescopru32
db.password=password
db.host=server_name_mysql
db.port=3306

#
# External locations
#————-
#ooo.exe=/opt/Alfresco/program/soffice
#img.root=/usr/local
#swf.exe=/opt/Alfresco/bin/pdf2swf

#Modificaciones para ImageMagick
img.root=/usr

#Modificacion para pdf2swf (swftools)
swf.exe=/usr/bin/pdf2swf

#
# MySQL connection
#————-
db.driver=org.gjt.mm.mysql.Driver
#db.url=jdbc:mysql://${db.host}:${db.port}/${db.name}
db.url=jdbc:mysql://server_name_mysql:3306/alfrescopru32
hibernate.dialect=org.hibernate.dialect.MySQLInnoDBDialect
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[steve]

Hi,

I am investigating these issues and hopefully will have some results next week.
What would help is if you explicitly state your use cases for me to help replicate your issues.

Currently, I have the following:

5 clients connect via CIFS and load data.
5 clients connect via FTP and load data?

Are there any others?

Thanks,

Steve

[pescha]

Well after many travails, I have finally figured out a solution…I think.  I'm crossing my fingers.

First, get a copy of VisualGC.  I can't stress this enough.  While running Alfresco, you really have to find out what the garbage collectors are doing, and in particular, the size of the survivor spaces.  VisualGC helps you see what is going on with the garbage collectors with easy to follow diagrams.

If you are using the concurrent garbage collector, it looks like the survivor space sizes remain static.  Be sure to size the survivor space adequately so that there is not too much overflow into the OldGen.  FullGC's should only happen once in a while.

Good luck, and send me a msg if you need help with your particular set up.

[dc41]

Has any progress been made on this issue? I've been off on another project, but has anyone found a setting that works, or will each install require it's own tweaking as pescha has done?

Thanks

[steve]

Hi,

We have replicated this and it is logged as ETHREEOH-2538
It is targetted to be fixed in Alfresco 3.2 Enterprise and will be merged to the HEAD codeline for you to get from svn, nightly builds or from the next Community release.

Steve

[dc41]

We have replicated this and it is logged as ETHREEOH-2538
It is targetted to be fixed in Alfresco 3.2 Enterprise and will be merged to the HEAD codeline for you to get from svn, nightly builds or from the next Community release.

Thanks Steve!

[bcampeau]

Can anyone advise on the current status of this bug fix? We're currently using Alfresco 3.2 Community Edition in a production environment. Although it passed our tests, this memory leak is manifesting itself almost daily under full production load. We'd like to know if a fix is imminent or if we need to begin creating a contingency plan.

Thanks.

Bradford Campeau-Laurion
Director of Web Production
Forbes.com

[dc41]

Can anyone advise on the current status of this bug fix? We're currently using Alfresco 3.2 Community Edition in a production environment. Although it passed our tests, this memory leak is manifesting itself almost daily under full production load. We'd like to know if a fix is imminent or if we need to begin creating a contingency plan.

Great question. Are our only two choices to ride the bleeding edge and use the nightly builds or just wait for the next release?

[mrogers]

You can check up on the status of issues in JIRA.    And community members are welcome to provide fixes.  

This particular issue is in status "Open".   There is no fix yet.

[jager13579]

Has there been any progress with a fix for the community edition? I see the Jira issue is for the Enterprise and it says it's resolved but I'm not sure if this has been extended to the community edition yet and if it is part of a stable release or just the trunk.

We are running in a production environment moving in files in 3.2 via cifs and really need this issue to be resolved. Any help would be appreciated.

Thanks.

[muralie39]

Please provide solutions..  i am struggling with same problem..i am using 3.3.4