topic Re: OutOfMemory/GC overhead limit error & performance tuning? in Alfresco Archive

OutOfMemory/GC overhead limit error & performance tuning?

dynamolalit — Tue, 16 Nov 2010 09:43:32 GMT

Hi,I am using alfresco 3.2 over tomcat 6 & jdk 1.6. Till sometime back , everything was working fine, but from last 15 days or so, i am getting java.lang.OutOfMemoryError as below:10:44:11,769 ERROR [org.alfresco.web.ui.common.Utils] Failed to get completed tasks: java.lang.OutOfMemoryError: GC

Re: OutOfMemory/GC overhead limit error & performance tuning?

hap234 — Tue, 16 Nov 2010 11:31:56 GMT

What is the size that you have set for JVM in alfresco.bat/alfresco.sh file?

set JAVA_OPTS=-Xms128m -Xmx512m -Xss64k -server

Increase 128 to 256 and 512 to 1024, if you have enough RAM available in your server and check.

Best Regards,
Hetal Patel

Re: OutOfMemory/GC overhead limit error & performance tuning?

dynamolalit — Wed, 17 Nov 2010 09:43:26 GMT

Hi Hetal,

Here are my JVM settings:

set JAVA_OPTS= -server -Xms1G -Xmx1G -XXermSize=256m -XX:MaxPermSize=256m -Dalfresco.home=%ALF_HOME% -Dcom.sun.management.jmxremote

Still its same. ops:

Re: OutOfMemory/GC overhead limit error & performance tuning?

muralie39 — Mon, 22 Nov 2010 12:50:10 GMT

Hi,

can you do with threads.

add the following properties;
db.pool.max=300
db.pool.initial=10

alfresco.bat
set JAVA_OPTS=-Xms2G -Xmx4G -Xss1024k -XX:MaxPermSize=512m -server -Dalfresco.home=%ALF_HOME% -Dcom.sun.management.jmxremote
the Xss is per each thread

tomcat/server.xml

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="260" minSpareThreads="4"/>

try this one…..

Re: OutOfMemory/GC overhead limit error & performance tuning?

dynamolalit — Tue, 23 Nov 2010 04:32:28 GMT

Hi Murali,

Thanks for response.

I have updated db.pool in alfresco-global.properties for 300, alfresco.bat for 1024K as well as tomcat/server.xml for thread pool. Also i updated MySql db for max connections to 350 but still it gets java.lang.OutOfMemoryError: GC overhead limit exceeded Error but after longer time.Its like workaround only. My server is 32bit machine & windows 2003 based. I have 8 GB of RAM at my exposure. How much can i use or same?

Also,I will updated Xss settings in alfresco.bat & check it for sure.

Re: OutOfMemory/GC overhead limit error & performance tuning?

muralie39 — Tue, 23 Nov 2010 05:44:43 GMT

Hi Dynamolalit,

please post the log

Re: OutOfMemory/GC overhead limit error & performance tuning?

dynamolalit — Tue, 23 Nov 2010 05:47:31 GMT

Hi Dynamolalit,

please post the log

Sure i will soon.

Re: OutOfMemory/GC overhead limit error & performance tuning?

dynamolalit — Thu, 25 Nov 2010 06:56:39 GMT

Hi Murali,

Here is a snapshot of logs.


12:04:16,577 ERROR [org.alfresco.web.ui.common.Utils] A system error happened during the operation: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
12:04:37,984 ERROR [org.alfresco.web.ui.common.Utils] Failed to get pooled tasks: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
12:05:56,377 DEBUG [com.xxxx.alfresco.km.evaluator.KmSpaceEvaluator] Current space is : General
12:05:56,393 DEBUG [com.xxxx.alfresco.km.evaluator.KmSpaceEvaluator] kMFlag in KmSpaceEvaluator : true
12:05:56,393 DEBUG [com.xxxx.alfresco.km.evaluator.KmSpaceEvaluator]  For All other users on JKM 
12:05:56,393 DEBUG [com.xxxx.alfresco.km.evaluator.KmSpaceEvaluator] result in KmSpaceEvaluator() : false
12:07:09,599 ERROR [org.hibernate.transaction.JDBCTransaction] JDBC rollback failed
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Connection.close() has already been called. Invalid operation in this state.
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
   at com.mysql.jdbc.Util.getInstance(Util.java:381)
   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)
   at com.mysql.jdbc.ConnectionImpl.getMutex(ConnectionImpl.java:3032)
   at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4606)
   at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
   at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
   at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
   at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
   at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:597)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:753)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:730)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:332)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.completeTransactionAfterThrowing(SpringAwareUserTransaction.java:562)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.rollback(SpringAwareUserTransaction.java:532)
   at org.alfresco.web.bean.workflow.WorkflowBean.getPooledTasks(WorkflowBean.java:177)
   at sun.reflect.GeneratedMethodAccessor3979.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.myfaces.el.PropertyResolverImpl.getProperty(PropertyResolverImpl.java:457)
   at org.apache.myfaces.el.PropertyResolverImpl.getValue(PropertyResolverImpl.java:85)
   at org.apache.myfaces.el.ELParserHelper$MyPropertySuffix.evaluate(ELParserHelper.java:535)
   at org.apache.commons.el.ComplexValue.evaluate(ComplexValue.java:145)
   at org.apache.commons.el.UnaryOperatorExpression.evaluate(UnaryOperatorExpression.java:156)
   at org.apache.myfaces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:386)
   at javax.faces.component.UIComponentBase.isRendered(UIComponentBase.java:1078)
   at org.apache.myfaces.shared_impl.renderkit.RendererUtils.renderChild(RendererUtils.java:408)
   at org.apache.myfaces.shared_impl.renderkit.RendererUtils.renderChildren(RendererUtils.java:400)
   at org.apache.myfaces.shared_impl.renderkit.RendererUtils.renderChild(RendererUtils.java:417)
   at org.apache.myfaces.shared_impl.renderkit.RendererUtils.renderChildren(RendererUtils.java:400)
   at org.apache.myfaces.shared_impl.renderkit.RendererUtils.renderChild(RendererUtils.java:417)
   at org.apache.myfaces.shared_impl.renderkit.html.HtmlGridRendererBase.renderChildren(HtmlGridRendererBase.java:229)
   at org.apache.myfaces.shared_impl.renderkit.html.HtmlGridRendererBase.encodeEnd(HtmlGridRendererBase.java:101)
   at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:539)
   at javax.faces.webapp.UIComponentTag.encodeEnd(UIComponentTag.java:498)
   at javax.faces.webapp.UIComponentTag.doEndTag(UIComponentTag.java:366)
   at org.apache.myfaces.shared_impl.taglib.UIComponentBodyTagBase.doEndTag(UIComponentBodyTagBase.java:57)
   at org.apache.jsp.jsp.dashboards.layouts.single_002dcolumn_jsp._jspService(single_002dcolumn_jsp.java:376)
   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:630)
   at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:535)
   at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:472)
   at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
   at org.apache.jsp.jsp.dashboards.container_jsp._jspService(container_jsp.java:484)
   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:630)
   at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
   at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
   at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
   at org.apache.myfaces.context.servlet.ServletExternalContextImpl.dispatch(ServletExternalContextImpl.java:419)
   at org.apache.myfaces.application.jsp.JspViewHandlerImpl.renderView(JspViewHandlerImpl.java:211)
   at org.apache.myfaces.lifecycle.RenderResponseExecutor.execute(RenderResponseExecutor.java:41)
   at org.apache.myfaces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:132)
   at javax.faces.webapp.FacesServlet.service(FacesServlet.java:140)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at com.keembay.alfresco.web.app.servlet.CASAuthenticationFilter.doFilter(CASAuthenticationFilter.java:149)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.alfresco.repo.web.filter.beans.NullFilter.doFilter(NullFilter.java:74)
   at sun.reflect.GeneratedMethodAccessor729.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.management.subsystems.ChainingSubsystemProxyFactory$1.invoke(ChainingSubsystemProxyFactory.java:122)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
   at $Proxy220.doFilter(Unknown Source)
   at org.alfresco.repo.web.filter.beans.BeanProxyFilter.doFilter(BeanProxyFilter.java:88)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at edu.yale.its.tp.cas.client.filter.CASFilter.doFilter(CASFilter.java:317)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
   at java.lang.Thread.run(Thread.java:619)
12:07:09,646 ERROR [org.alfresco.util.transaction.SpringAwareUserTransaction] Application exception overridden by rollback exception
java.lang.Exception
   at org.alfresco.util.transaction.SpringAwareUserTransaction.rollback(SpringAwareUserTransaction.java:532)
   at org.alfresco.web.bean.workflow.WorkflowBean.getPooledTasks(WorkflowBean.java:177)
   at sun.reflect.GeneratedMethodAccessor3979.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.myfaces.el.PropertyResolverImpl.getProperty(PropertyResolverImpl.java:457)
   at org.apache.myfaces.el.PropertyResolverImpl.getValue(PropertyResolverImpl.java:85)
   at org.apache.myfaces.el.ELParserHelper$MyPropertySuffix.evaluate(ELParserHelper.java:535)
   at org.apache.commons.el.ComplexValue.evaluate(ComplexValue.java:145)
   at org.apache.commons.el.UnaryOperatorExpression.evaluate(UnaryOperatorExpression.java:156)
   at org.apache.myfaces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:386)
   at javax.faces.component.UIComponentBase.isRendered(UIComponentBase.java:1078)
   at org.apache.myfaces.shared_impl.renderkit.RendererUtils.renderChild(RendererUtils.java:408)
   at org.apache.myfaces.shared_impl.renderkit.RendererUtils.renderChildren(RendererUtils.java:400)
   at org.apache.myfaces.shared_impl.renderkit.RendererUtils.renderChild(RendererUtils.java:417)
   at org.apache.myfaces.shared_impl.renderkit.RendererUtils.renderChildren(RendererUtils.java:400)
   at org.apache.myfaces.shared_impl.renderkit.RendererUtils.renderChild(RendererUtils.java:417)
   at org.apache.myfaces.shared_impl.renderkit.html.HtmlGridRendererBase.renderChildren(HtmlGridRendererBase.java:229)
   at org.apache.myfaces.shared_impl.renderkit.html.HtmlGridRendererBase.encodeEnd(HtmlGridRendererBase.java:101)
   at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:539)
   at javax.faces.webapp.UIComponentTag.encodeEnd(UIComponentTag.java:498)
   at javax.faces.webapp.UIComponentTag.doEndTag(UIComponentTag.java:366)
   at org.apache.myfaces.shared_impl.taglib.UIComponentBodyTagBase.doEndTag(UIComponentBodyTagBase.java:57)
   at org.apache.jsp.jsp.dashboards.layouts.single_002dcolumn_jsp._jspService(single_002dcolumn_jsp.java:376)
   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:630)
   at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:535)
   at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:472)
   at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968)
   at org.apache.jsp.jsp.dashboards.container_jsp._jspService(container_jsp.java:484)
   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:630)
   at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
   at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
   at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
   at org.apache.myfaces.context.servlet.ServletExternalContextImpl.dispatch(ServletExternalContextImpl.java:419)
   at org.apache.myfaces.application.jsp.JspViewHandlerImpl.renderView(JspViewHandlerImpl.java:211)
   at org.apache.myfaces.lifecycle.RenderResponseExecutor.execute(RenderResponseExecutor.java:41)
   at org.apache.myfaces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:132)
   at javax.faces.webapp.FacesServlet.service(FacesServlet.java:140)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at com.keembay.alfresco.web.app.servlet.CASAuthenticationFilter.doFilter(CASAuthenticationFilter.java:149)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.alfresco.repo.web.filter.beans.NullFilter.doFilter(NullFilter.java:74)
   at sun.reflect.GeneratedMethodAccessor729.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.management.subsystems.ChainingSubsystemProxyFactory$1.invoke(ChainingSubsystemProxyFactory.java:122)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
   at $Proxy220.doFilter(Unknown Source)
   at org.alfresco.repo.web.filter.beans.BeanProxyFilter.doFilter(BeanProxyFilter.java:88)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at edu.yale.its.tp.cas.client.filter.CASFilter.doFilter(CASFilter.java:317)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
   at java.lang.Thread.run(Thread.java:619)
12:07:09,646 ERROR [org.alfresco.web.ui.common.Utils] Failed to get pooled tasks: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] User name to get statistics for : sankalp.x.kumar
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] User name to get statistics for : sankalp.x.kumar
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] Size of resultset in webscript : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] totalNoOfContent for sankalp.x.kumar : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] approvedContent for  sankalp.x.kumar : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] rejectedContent for  sankalp.x.kumar : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] underReviewContent for sankalp.x.kumar : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] ratingToDisplay for  sankalp.x.kumar : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] Size of resultset in webscript : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] totalNoOfContent for sankalp.x.kumar : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] approvedContent for  sankalp.x.kumar : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] rejectedContent for  sankalp.x.kumar : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] underReviewContent for sankalp.x.kumar : 0
12:07:23,224 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] ratingToDisplay for  sankalp.x.kumar : 0
12:10:06,590 ERROR [org.alfresco.web.ui.common.Utils] Failed to get pooled tasks: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
12:10:42,873 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] User name to get statistics for : sankalp.x.kumar
12:10:42,888 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] Size of resultset in webscript : 0
12:10:42,888 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] totalNoOfContent for sankalp.x.kumar : 0
12:10:42,888 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] approvedContent for  sankalp.x.kumar : 0
12:10:42,888 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] rejectedContent for  sankalp.x.kumar : 0
12:10:42,888 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] underReviewContent for sankalp.x.kumar : 0
12:10:42,888 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] ratingToDisplay for  sankalp.x.kumar : 0
12:11:44,953 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] User name to get statistics for : psrana
12:11:44,984 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] Size of resultset in webscript : 0
12:11:44,984 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] totalNoOfContent for psrana : 0
12:11:44,984 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] approvedContent for  psrana : 0
12:11:45,000 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] rejectedContent for  psrana : 0
12:11:45,000 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] underReviewContent for psrana : 0
12:11:48,281 DEBUG [com.xxxx.alfresco.km.webscripts.GetUserStatisticsWebScript] ratingToDisplay for  psrana : 0
12:13:31,176 ERROR [org.alfresco.web.ui.common.Utils] Failed to get pooled tasks: java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: OutOfMemory/GC overhead limit error & performance tuning?

muralie39 — Fri, 26 Nov 2010 09:16:18 GMT

Hi dynamolalit,

do you have jconsole.exe?
please start the jconsole & check with your memory.

Is your production server is down because of this issuse.

add the -XX:-UseGCOverheadLimit preperty in alfresco.bat file

http://stackoverflow.com/questions/2863984/duration-of-excessive-gc-time-in-java-lang-outofmemoryerror-gc-overhead-limit-e

Re: OutOfMemory/GC overhead limit error & performance tuning?

sharifu — Mon, 28 May 2012 15:57:57 GMT

I am getting this same problem on 4.0d when i try to do a full re=index


2012-05-28 16:53:49,649  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread4] Reindexer     4 failed with error: Error during run as..
 2012-05-28 16:53:49,650  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread4] java.lang.RuntimeException: Error during run as.
   at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:531)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocumentsImpl(ADMLuceneIndexerImpl.java:680)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocuments(ADMLuceneIndexerImpl.java:646)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2015)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2011)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl$1.execute(AbstractLuceneIndexerImpl.java:715)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.doInReadthroughTransaction(AbstractLuceneIndexerImpl.java:702)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.readDocuments(ADMLuceneIndexerImpl.java:2010)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1057)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   at sun.util.calendar.Gregorian.newCalendarDate(Gregorian.java:67)
   at java.util.GregorianCalendar.<init>(GregorianCalendar.java:575)
   at java.util.GregorianCalendar.<init>(GregorianCalendar.java:552)
   at org.springframework.extensions.surf.util.ISO8601DateFormat.parse(ISO8601DateFormat.java:200)
   at org.alfresco.service.cmr.repository.datatype.DefaultTypeConverter$13.convert(DefaultTypeConverter.java:193)
   at org.alfresco.service.cmr.repository.datatype.DefaultTypeConverter$13.convert(DefaultTypeConverter.java:188)
   at org.alfresco.service.cmr.repository.datatype.TypeConverter.convert(TypeConverter.java:112)
   at org.alfresco.repo.domain.node.NodePropertyValue$ValueType$8.convert(NodePropertyValue.java:229)
   at org.alfresco.repo.domain.node.NodePropertyValue.getValue(NodePropertyValue.java:1088)
   at org.alfresco.repo.domain.node.NodePropertyHelper.makeSerializableValue(NodePropertyHelper.java:658)
   at org.alfresco.repo.domain.node.NodePropertyHelper.collapsePropertiesWithSameQNameAndListIndex(NodePropertyHelper.java:579)
   at org.alfresco.repo.domain.node.NodePropertyHelper.convertToPublicProperties(NodePropertyHelper.java:407)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.cacheNodesNoBatch(AbstractNodeDAOImpl.java:3926)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.cacheNodes(AbstractNodeDAOImpl.java:3839)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.cacheNodes(AbstractNodeDAOImpl.java:3815)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl$ChildAssocRefBatchingQueryCallback.done(AbstractNodeDAOImpl.java:3058)
   at org.alfresco.repo.domain.node.ibatis.NodeDAOImpl.selectChildAssocs(NodeDAOImpl.java:1095)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getChildAssocs(AbstractNodeDAOImpl.java:3075)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getChildAssocs(DbNodeServiceImpl.java:1721)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getChildAssocs(DbNodeServiceImpl.java:1664)
   at org.alfresco.repo.node.AbstractNodeServiceImpl.getChildAssocs(AbstractNodeServiceImpl.java:734)
   at sun.reflect.GeneratedMethodAccessor275.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:104)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy8.getChildAssocs(Unknown Source)
   at sun.reflect.GeneratedMethodAccessor275.invoke(Unknown Source)












2012-05-28 17:07:47,597  INFO  [node.index.FullIndexRecoveryComponent] [main]    10 % complete.
 2012-05-28 17:08:24,226  INFO  [node.index.FullIndexRecoveryComponent] [main]    20 % complete.
 2012-05-28 17:09:11,427  INFO  [node.index.FullIndexRecoveryComponent] [main]    30 % complete.
 2012-05-28 17:09:38,895  INFO  [node.index.FullIndexRecoveryComponent] [main]    40 % complete.
 2012-05-28 17:10:03,422  INFO  [node.index.FullIndexRecoveryComponent] [main]    50 % complete.
 2012-05-28 17:10:27,570  INFO  [node.index.FullIndexRecoveryComponent] [main]    60 % complete.
 2012-05-28 17:10:52,077  INFO  [node.index.FullIndexRecoveryComponent] [main]    70 % complete.
 2012-05-28 17:11:18,632  INFO  [node.index.FullIndexRecoveryComponent] [main]    80 % complete.
 2012-05-28 17:12:06,464  INFO  [node.index.FullIndexRecoveryComponent] [main]    90 % complete.
 2012-05-28 17:18:01,084  ERROR [transaction.support.TransactionSynchronizationUtils] [indexTrackerThread3] TransactionSynchronization.afterCompletion threw exception
 java.lang.OutOfMemoryError: GC overhead limit exceeded
   at org.alfresco.repo.cache.EhCacheAdapter.put(EhCacheAdapter.java:108)
   at org.alfresco.repo.cache.TransactionalCache$NewCacheBucket.doPostCommit(TransactionalCache.java:889)
   at org.alfresco.repo.cache.TransactionalCache.afterCommit(TransactionalCache.java:750)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.afterCompletion(AlfrescoTransactionSupport.java:800)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.invokeAfterCompletion(TransactionSynchronizationUtils.java:168)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.invokeAfterCompletion(AbstractPlatformTransactionManager.java:996)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerAfterCompletion(AbstractPlatformTransactionManager.java:971)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:799)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent.reindexTransaction(AbstractReindexComponent.java:777)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable$1.execute(AbstractReindexComponent.java:986)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
2012-05-28 17:18:03,878  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread3] Reindexer   871 failed with error: Already value [TransactionSychronizationImpl[ txnId=90614901-1958-446b-a493-fede49b3544b, daos=0, integrity=0, indexers=0, resources={org.alfresco.repo.search.IndexerComponentREAD_THROUGH=true, RetryingTransactionHelper.ActiveTxn=UserTransaction[object=org.alfresco.util.transaction.SpringAwareUserTransaction@5869c42b, status=0]}]] for key [txnSynch] bound to thread [indexTrackerThread3].
 2012-05-28 17:18:03,879  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread3] java.lang.IllegalStateException: Already value [TransactionSychronizationImpl[ txnId=90614901-1958-446b-a493-fede49b3544b, daos=0, integrity=0, indexers=0, resources={org.alfresco.repo.search.IndexerComponentREAD_THROUGH=true, RetryingTransactionHelper.ActiveTxn=UserTransaction[object=org.alfresco.util.transaction.SpringAwareUserTransaction@5869c42b, status=0]}]] for key [txnSynch] bound to thread [indexTrackerThread3]
   at org.springframework.transaction.support.TransactionSynchronizationManager.bindResource(TransactionSynchronizationManager.java:179)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport.rebindSynchronization(AlfrescoTransactionSupport.java:514)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport.access$300(AlfrescoTransactionSupport.java:50)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.resume(AlfrescoTransactionSupport.java:662)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.doResumeSynchronization(AbstractPlatformTransactionManager.java:680)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.resume(AbstractPlatformTransactionManager.java:632)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:1017)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:804)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent.reindexTransaction(AbstractReindexComponent.java:777)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable$1.execute(AbstractReindexComponent.java:986)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)

 2012-05-28 17:18:04,381  INFO  [node.index.FullIndexRecoveryComponent] [main]    100 % complete.
 2012-05-28 17:19:54,322  INFO  [node.index.FullIndexRecoveryComponent] [main] Index recovery completed.
 2012-05-28 17:19:54,334  INFO  [node.index.AVMFullIndexRecoveryComponent] [main] Rebuilding indexes for 1 AVM Stores
 2012-05-28 17:19:54,334  INFO  [node.index.AVMFullIndexRecoveryComponent] [main]     Rebuilding index for sitestore
 2012-05-28 17:19:54,500  INFO  [node.index.AVMFullIndexRecoveryComponent] [main]   Reindex   100% complete
 2012-05-28 17:19:54,500  INFO  [node.index.AVMFullIndexRecoveryComponent] [main] Finished rebuilding indexes for AVM Stores












2012-05-28 17:23:26,480  ERROR [quartz.core.JobRunShell] [DefaultScheduler_Worker-7] Job DEFAULT.ftsIndexerJobDetail threw an unhandled Exception: 
 org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.error.AlfrescoRuntimeException: 04280006 Exception in Transaction.
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.indexCompleted(FullTextSearchIndexerImpl.java:107)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:258)
   at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:46)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.error.AlfrescoRuntimeException: 04280006 Exception in Transaction.
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.indexCompleted(FullTextSearchIndexerImpl.java:107)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:240)
   … 3 more
Caused by: org.alfresco.error.AlfrescoRuntimeException: 04280006 Exception in Transaction.
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:473)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:278)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:218)
   … 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   at java.nio.CharBuffer.wrap(CharBuffer.java:350)
   at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:291)
   at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
   at sun.nio.cs.StreamDecoder.read0(StreamDecoder.java:107)
   at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:93)
   at java.io.InputStreamReader.read(InputStreamReader.java:151)
   at org.alfresco.repo.search.impl.lucene.MultiReader.read(MultiReader.java:86)
   at java.io.BufferedReader.read1(BufferedReader.java:185)
   at java.io.BufferedReader.read(BufferedReader.java:261)
   at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:443)
   at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:629)
   at org.apache.lucene.analysis.standard.StandardTokenizer.next(StandardTokenizer.java:140)
   at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:47)
   at org.alfresco.repo.search.impl.lucene.analysis.AlfrescoStandardFilter.next(AlfrescoStandardFilter.java:62)
   at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:91)
   at org.apache.lucene.analysis.LowerCaseFilter.next(LowerCaseFilter.java:34)
   at org.apache.lucene.analysis.StopFilter.next(StopFilter.java:120)
   at org.apache.lucene.analysis.ISOLatin1AccentFilter.next(ISOLatin1AccentFilter.java:37)
   at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:47)
   at org.alfresco.repo.search.impl.lucene.analysis.MLTokenDuplicator.buildIterator(MLTokenDuplicator.java:116)
   at org.alfresco.repo.search.impl.lucene.analysis.MLTokenDuplicator.next(MLTokenDuplicator.java:95)
   at org.alfresco.repo.search.impl.lucene.analysis.MLTokenDuplicator.next(MLTokenDuplicator.java:109)
   at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:91)
   at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:134)
   at org.apache.lucene.index.DocFieldConsumersPerField.processFields(DocFieldConsumersPerField.java:36)
   at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:234)
   at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:765)
   at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:743)
   at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1917)
   at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1895)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.updateFullTextSearch(ADMLuceneIndexerImpl.java:1960)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl$1.execute(FullTextSearchIndexerImpl.java:232)
2012-05-28 17:23:26,482  ERROR [quartz.core.ErrorLogger] [DefaultScheduler_Worker-7] Job (DEFAULT.ftsIndexerJobDetail threw an exception.
 org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.error.AlfrescoRuntimeException: 04280006 Exception in Transaction.]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.error.AlfrescoRuntimeException: 04280006 Exception in Transaction.
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.indexCompleted(FullTextSearchIndexerImpl.java:107)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:258)
   at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:46)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   … 1 more
Caused by: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.error.AlfrescoRuntimeException: 04280006 Exception in Transaction.
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.indexCompleted(FullTextSearchIndexerImpl.java:107)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:240)
   … 3 more
Caused by: org.alfresco.error.AlfrescoRuntimeException: 04280006 Exception in Transaction.
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:473)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:278)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:218)
   … 3 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   at java.nio.CharBuffer.wrap(CharBuffer.java:350)
   at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:291)
   at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
   at sun.nio.cs.StreamDecoder.read0(StreamDecoder.java:107)
   at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:93)
   at java.io.InputStreamReader.read(InputStreamReader.java:151)
   at org.alfresco.repo.search.impl.lucene.MultiReader.read(MultiReader.java:86)
   at java.io.BufferedReader.read1(BufferedReader.java:185)
   at java.io.BufferedReader.read(BufferedReader.java:261)
   at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:443)
   at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:629)
   at org.apache.lucene.analysis.standard.StandardTokenizer.next(StandardTokenizer.java:140)
   at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:47)
   at org.alfresco.repo.search.impl.lucene.analysis.AlfrescoStandardFilter.next(AlfrescoStandardFilter.java:62)
   at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:91)
   at org.apache.lucene.analysis.LowerCaseFilter.next(LowerCaseFilter.java:34)
   at org.apache.lucene.analysis.StopFilter.next(StopFilter.java:120)
   at org.apache.lucene.analysis.ISOLatin1AccentFilter.next(ISOLatin1AccentFilter.java:37)
   at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:47)
   at org.alfresco.repo.search.impl.lucene.analysis.MLTokenDuplicator.buildIterator(MLTokenDuplicator.java:116)
   at org.alfresco.repo.search.impl.lucene.analysis.MLTokenDuplicator.next(MLTokenDuplicator.java:95)
   at org.alfresco.repo.search.impl.lucene.analysis.MLTokenDuplicator.next(MLTokenDuplicator.java:109)
   at org.apache.lucene.analysis.TokenStream.next(TokenStream.java:91)
   at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:134)
   at org.apache.lucene.index.DocFieldConsumersPerField.processFields(DocFieldConsumersPerField.java:36)
   at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:234)
   at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:765)
   at org.apache.lucene.index.DocumentsWriter.addDocument(DocumentsWriter.java:743)
   at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1917)
   at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1895)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.updateFullTextSearch(ADMLuceneIndexerImpl.java:1960)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl$1.execute(FullTextSearchIndexerImpl.java:232)







012-05-28 17:23:41,955  ERROR [impl.lucene.ADMLuceneIndexerImpl] [DefaultScheduler_Worker-6] FTS index of workspace://SpacesStore/b4389c9d-6e9f-426e-b90e-0a920b3150e6 failed. Reindexing without FTS
 org.alfresco.error.AlfrescoRuntimeException: 04280008 Exception in Transaction.
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:473)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.doInReadthroughTransaction(AbstractLuceneIndexerImpl.java:702)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.readDocuments(ADMLuceneIndexerImpl.java:2010)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.updateFullTextSearch(ADMLuceneIndexerImpl.java:1948)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl$1.execute(FullTextSearchIndexerImpl.java:232)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl$1.execute(FullTextSearchIndexerImpl.java:220)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:278)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:218)
   at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:46)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   at com.mysql.jdbc.MysqlIO.unpackField(MysqlIO.java:765)
   at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:416)
   at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2581)
   at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1757)
   at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2171)
   at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568)
   at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2113)
   at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1364)
   at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
   at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
   at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
   at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:39)
   at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:55)
   at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:41)
   at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:120)
   at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:75)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:75)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:69)
   at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:40)
   at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.mybatis.spring.SqlSessionTemplate$SqlSessionInterceptor.invoke(SqlSessionTemplate.java:338)
   at $Proxy6.selectOne(Unknown Source)
   at org.mybatis.spring.SqlSessionTemplate.selectOne(SqlSessionTemplate.java:154)
   at org.alfresco.repo.domain.node.ibatis.NodeDAOImpl.selectNodeByNodeRef(NodeDAOImpl.java:408)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl$NodesCacheCallbackDAO.findByValue(AbstractNodeDAOImpl.java:886)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl$NodesCacheCallbackDAO.findByValue(AbstractNodeDAOImpl.java:842)
   at org.alfresco.repo.cache.lookup.EntityLookupCache.getByValue(EntityLookupCache.java:396)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getNodeRefStatus(AbstractNodeDAOImpl.java:931)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getNodeStatus(DbNodeServiceImpl.java:195)
   at sun.reflect.GeneratedMethodAccessor273.invoke(Unknown Source)
May 28, 2012 5:24:00 PM org.apache.catalina.core.StandardContext start
SEVERE: Error listenerStart
May 28, 2012 5:24:00 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/alfresco] startup failed due to previous errors
2012-05-28 17:24:01,851  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'Replication' subsystem, ID: [Replication, default]
 2012-05-28 17:24:01,852  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'Replication' subsystem, ID: [Replication, default]
 2012-05-28 17:24:02,057  ERROR [impl.lucene.ADMLuceneIndexerImpl] [DefaultScheduler_Worker-6] FTS index of workspace://SpacesStore/8d767ef6-04e5-4561-beac-8f2fbcea9153 failed. Reindexing without FTS
 org.alfresco.error.AlfrescoRuntimeException: 04280009 Failed to get from EhCache as state invalid: 
  state: STATUS_SHUTDOWN
   key: workspace://SpacesStore
   at org.alfresco.repo.cache.EhCacheAdapter.get(EhCacheAdapter.java:93)
   at org.alfresco.repo.cache.TransactionalCache.getSharedCacheValue(TransactionalCache.java:326)
   at org.alfresco.repo.cache.TransactionalCache.get(TransactionalCache.java:392)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getAllRootNodes(AbstractNodeDAOImpl.java:706)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getAllRootNodes(DbNodeServiceImpl.java:289)
   at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:104)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy8.getAllRootNodes(Unknown Source)
   at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy9.getAllRootNodes(Unknown Source)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.getAllParents(ADMLuceneIndexerImpl.java:1739)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocumentsImpl(ADMLuceneIndexerImpl.java:771)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocuments(ADMLuceneIndexerImpl.java:646)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2015)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2011)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl$1.execute(AbstractLuceneIndexerImpl.java:715)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.doInReadthroughTransaction(AbstractLuceneIndexerImpl.java:702)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.readDocuments(ADMLuceneIndexerImpl.java:2010)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.updateFullTextSearch(ADMLuceneIndexerImpl.java:1948)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl$1.execute(FullTextSearchIndexerImpl.java:232)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl$1.execute(FullTextSearchIndexerImpl.java:220)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:278)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:218)
   at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:46)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: java.lang.IllegalStateException: The org.alfresco.cache.node.allRootNodesCache Cache is not alive.
   at net.sf.ehcache.Cache.checkStatus(Cache.java:2282)
   at net.sf.ehcache.Cache.get(Cache.java:1325)
   at net.sf.ehcache.Cache.get(Cache.java:1306)
   at org.alfresco.repo.cache.EhCacheAdapter.get(EhCacheAdapter.java:81)
   … 37 more
2012-05-28 17:24:02,085  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'Subscriptions' subsystem, ID: [Subscriptions, default]
 2012-05-28 17:24:02,086  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'Subscriptions' subsystem, ID: [Subscriptions, default]
 2012-05-28 17:24:02,086  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'googledocs' subsystem, ID: [googledocs, default]
 2012-05-28 17:24:02,086  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'googledocs' subsystem, ID: [googledocs, default]
 2012-05-28 17:24:02,086  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'email' subsystem, ID: [email, inbound]
 2012-05-28 17:24:02,087  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'email' subsystem, ID: [email, inbound]
 2012-05-28 17:24:02,104  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'fileServers' subsystem, ID: [fileServers, default]
 2012-05-28 17:24:02,585  ERROR [quartz.core.JobRunShell] [DefaultScheduler_Worker-6] Job DEFAULT.ftsIndexerJobDetail threw an unhandled Exception: 
 org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.error.AlfrescoRuntimeException: 04280010 Failed to get from EhCache as state invalid: 
  state: STATUS_SHUTDOWN
   key: N.P.NodeVersionKey [nodeId=65878, version=887]
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.indexCompleted(FullTextSearchIndexerImpl.java:107)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:258)
   at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:46)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.error.AlfrescoRuntimeException: 04280010 Failed to get from EhCache as state invalid: 
  state: STATUS_SHUTDOWN
   key: N.P.NodeVersionKey [nodeId=65878, version=887]
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.indexCompleted(FullTextSearchIndexerImpl.java:107)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:240)
   … 3 more
Caused by: org.alfresco.error.AlfrescoRuntimeException: 04280010 Failed to get from EhCache as state invalid: 
  state: STATUS_SHUTDOWN
   key: N.P.NodeVersionKey [nodeId=65878, version=887]
   at org.alfresco.repo.cache.EhCacheAdapter.get(EhCacheAdapter.java:93)
   at org.alfresco.repo.cache.TransactionalCache.getSharedCacheValue(TransactionalCache.java:326)
   at org.alfresco.repo.cache.TransactionalCache.get(TransactionalCache.java:392)
   at org.alfresco.repo.cache.lookup.EntityLookupCache.getByKey(EntityLookupCache.java:303)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getNodePropertiesCached(AbstractNodeDAOImpl.java:2201)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getNodeProperties(AbstractNodeDAOImpl.java:1766)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getPropertiesImpl(DbNodeServiceImpl.java:1404)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getProperties(DbNodeServiceImpl.java:1395)
   at sun.reflect.GeneratedMethodAccessor274.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:104)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy8.getProperties(Unknown Source)
   at sun.reflect.GeneratedMethodAccessor274.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy9.getProperties(Unknown Source)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocumentsImpl(ADMLuceneIndexerImpl.java:672)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocuments(ADMLuceneIndexerImpl.java:646)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2015)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2011)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl$1.execute(AbstractLuceneIndexerImpl.java:715)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.doInReadthroughTransaction(AbstractLuceneIndexerImpl.java:702)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.readDocuments(ADMLuceneIndexerImpl.java:2010)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.updateFullTextSearch(ADMLuceneIndexerImpl.java:1954)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl$1.execute(FullTextSearchIndexerImpl.java:232)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl$1.execute(FullTextSearchIndexerImpl.java:220)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:278)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:218)
   … 3 more
Caused by: java.lang.IllegalStateException: The org.alfresco.cache.node.propertiesCache Cache is not alive.
   at net.sf.ehcache.Cache.checkStatus(Cache.java:2282)
   at net.sf.ehcache.Cache.get(Cache.java:1325)
   at net.sf.ehcache.Cache.get(Cache.java:1306)
   at org.alfresco.repo.cache.EhCacheAdapter.get(EhCacheAdapter.java:81)
   … 39 more
2012-05-28 17:24:02,586  ERROR [quartz.core.ErrorLogger] [DefaultScheduler_Worker-6] Job (DEFAULT.ftsIndexerJobDetail threw an exception.
 org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.error.AlfrescoRuntimeException: 04280010 Failed to get from EhCache as state invalid: 
  state: STATUS_SHUTDOWN
   key: N.P.NodeVersionKey [nodeId=65878, version=887]]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.error.AlfrescoRuntimeException: 04280010 Failed to get from EhCache as state invalid: 
  state: STATUS_SHUTDOWN
   key: N.P.NodeVersionKey [nodeId=65878, version=887]
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.indexCompleted(FullTextSearchIndexerImpl.java:107)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:258)
   at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:46)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   … 1 more
Caused by: org.alfresco.repo.search.impl.lucene.fts.FTSIndexerException: org.alfresco.error.AlfrescoRuntimeException: 04280010 Failed to get from EhCache as state invalid: 
  state: STATUS_SHUTDOWN
   key: N.P.NodeVersionKey [nodeId=65878, version=887]
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.indexCompleted(FullTextSearchIndexerImpl.java:107)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:240)
   … 3 more
Caused by: org.alfresco.error.AlfrescoRuntimeException: 04280010 Failed to get from EhCache as state invalid: 
  state: STATUS_SHUTDOWN
   key: N.P.NodeVersionKey [nodeId=65878, version=887]
   at org.alfresco.repo.cache.EhCacheAdapter.get(EhCacheAdapter.java:93)
   at org.alfresco.repo.cache.TransactionalCache.getSharedCacheValue(TransactionalCache.java:326)
   at org.alfresco.repo.cache.TransactionalCache.get(TransactionalCache.java:392)
   at org.alfresco.repo.cache.lookup.EntityLookupCache.getByKey(EntityLookupCache.java:303)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getNodePropertiesCached(AbstractNodeDAOImpl.java:2201)
   at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getNodeProperties(AbstractNodeDAOImpl.java:1766)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getPropertiesImpl(DbNodeServiceImpl.java:1404)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getProperties(DbNodeServiceImpl.java:1395)
   at sun.reflect.GeneratedMethodAccessor274.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:104)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy8.getProperties(Unknown Source)
   at sun.reflect.GeneratedMethodAccessor274.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy9.getProperties(Unknown Source)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocumentsImpl(ADMLuceneIndexerImpl.java:672)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocuments(ADMLuceneIndexerImpl.java:646)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2015)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2011)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl$1.execute(AbstractLuceneIndexerImpl.java:715)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.doInReadthroughTransaction(AbstractLuceneIndexerImpl.java:702)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.readDocuments(ADMLuceneIndexerImpl.java:2010)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.updateFullTextSearch(ADMLuceneIndexerImpl.java:1954)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl$1.execute(FullTextSearchIndexerImpl.java:232)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl$1.execute(FullTextSearchIndexerImpl.java:220)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:278)
   at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:218)
   … 3 more
Caused by: java.lang.IllegalStateException: The org.alfresco.cache.node.propertiesCache Cache is not alive.
   at net.sf.ehcache.Cache.checkStatus(Cache.java:2282)
   at net.sf.ehcache.Cache.get(Cache.java:1325)
   at net.sf.ehcache.Cache.get(Cache.java:1306)
   at org.alfresco.repo.cache.EhCacheAdapter.get(EhCacheAdapter.java:81)
   … 39 more







2012-05-28 17:24:02,788  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'fileServers' subsystem, ID: [fileServers, default]
 2012-05-28 17:24:02,797  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'ActivitiesFeed' subsystem, ID: [ActivitiesFeed, default]
 2012-05-28 17:24:02,798  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'ActivitiesFeed' subsystem, ID: [ActivitiesFeed, default]
 2012-05-28 17:24:02,813  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'OOoDirect' subsystem, ID: [OOoDirect, default]
 2012-05-28 17:24:02,815  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'OOoDirect' subsystem, ID: [OOoDirect, default]
 2012-05-28 17:24:02,819  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'Synchronization' subsystem, ID: [Synchronization, default]
 2012-05-28 17:24:02,819  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'Synchronization' subsystem, ID: [Synchronization, default]
 2012-05-28 17:24:02,825  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'email' subsystem, ID: [email, outbound]
 2012-05-28 17:24:02,825  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'email' subsystem, ID: [email, outbound]
 2012-05-28 17:24:02,832  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'imap' subsystem, ID: [imap, default]
 2012-05-28 17:24:02,833  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'imap' subsystem, ID: [imap, default]
 2012-05-28 17:24:03,105  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'Search' subsystem, ID: [Search, managed, lucene]
 2012-05-28 17:24:03,121  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'Search' subsystem, ID: [Search, managed, lucene]
 2012-05-28 17:24:03,161  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'thirdparty' subsystem, ID: [thirdparty, default]
 2012-05-28 17:24:03,161  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'thirdparty' subsystem, ID: [thirdparty, default]
 2012-05-28 17:24:03,181  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'Authentication' subsystem, ID: [Authentication, managed, passthru1]
 2012-05-28 17:24:03,181  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'Authentication' subsystem, ID: [Authentication, managed, passthru1]
 2012-05-28 17:24:03,181  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'Authentication' subsystem, ID: [Authentication, managed, ldap1]
 2012-05-28 17:24:03,182  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'Authentication' subsystem, ID: [Authentication, managed, ldap1]
 2012-05-28 17:24:03,193  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopping 'sysAdmin' subsystem, ID: [sysAdmin, default]
 2012-05-28 17:24:03,193  INFO  [management.subsystems.ChildApplicationContextFactory] [main] Stopped 'sysAdmin' subsystem, ID: [sysAdmin, default]
 May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [SolrWatcherScheduler_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [SolrWatcherScheduler_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [SolrWatcherScheduler_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [Timer-1] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [Timer-2] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [Timer-3] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [Timer-4] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [Timer-5] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [MultiThreadedHttpConnectionManager cleanup] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [PassthruReaper] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [Thread-66] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [Thread-67] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [Timer-9] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [JobLockService1] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/alfresco] appears to have started a thread named [Timer-6] but has failed to stop it. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@6a4b3998]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@70b260ac]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4ee8f601]) and a value of type [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory.ReferenceCountingReadOnlyIndexReader] (value [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory$ReferenceCountingReadOnlyIndexReader@5f1826f8]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@2aa6948e]) and a value of type [org.apache.cxf.bus.CXFBusImpl] (value [org.apache.cxf.bus.CXFBusImpl@1e630c82]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@78ca3a63]) and a value of type [org.alfresco.util.CachingDateFormat.SimpleDateFormatAndResolution[]] (value [[Lorg.alfresco.util.CachingDateFormat$SimpleDateFormatAndResolution;@24bf7789]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@1640d638]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@7ab001f1]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.security.authentication.AuthenticationUtil.ThreadLocalStack] (value [org.alfresco.repo.security.authentication.AuthenticationUtil$ThreadLocalStack@394fb424]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [null] (value [org.alfresco.repo.template.QNameAwareObjectWrapper$1@1fbbc4e1]) and a value of type [null] (value [org.alfresco.repo.template.QNameAwareObjectWrapper$1$1@607aa00d]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@12a1e6f7]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [null] (value [com.sun.xml.bind.v2.ClassFactory$1@43426baf]) and a value of type [java.util.WeakHashMap] (value [{class org.alfresco.repo.audit.model._3.DataExtractors=java.lang.ref.WeakReference@d8619d2, class org.alfresco.repo.audit.model._3.GenerateValue=java.lang.ref.WeakReference@5f0930ed, class org.alfresco.repo.audit.model._3.Audit=java.lang.ref.WeakReference@546b4859, class javax.xml.bind.annotation.W3CDomHandler=java.lang.ref.WeakReference@4f5af65b, class org.alfresco.repo.audit.model._3.DataGenerator=java.lang.ref.WeakReference@5ce3f910, class java.util.ArrayList=java.lang.ref.WeakReference@7af4b914, class org.alfresco.repo.audit.model._3.DataExtractor=java.lang.ref.WeakReference@58a30f1c, class org.alfresco.repo.audit.model._3.PathMap=java.lang.ref.WeakReference@393524ba, class org.alfresco.repo.audit.model._3.AuditPath=java.lang.ref.WeakReference@4ffe4cad, class org.alfresco.repo.audit.model._3.DataGenerators=java.lang.ref.WeakReference@405c1ae2, class org.alfresco.repo.audit.model._3.Application=java.lang.ref.WeakReference@5ee90c70, class org.alfresco.repo.audit.model._3.PathMappings=java.lang.ref.WeakReference@162fbdbe, class org.alfresco.repo.audit.model._3.RecordValue=java.lang.ref.WeakReference@18521053}]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@33adc59e]) and a value of type [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory.ReferenceCountingReadOnlyIndexReader] (value [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory$ReferenceCountingReadOnlyIndexReader@55892293]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@1e3d0cc4]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3a11275e]) and a value of type [org.alfresco.util.CachingDateFormat] (value [yyyy-MM-dd]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c2faa1a]) and a value of type [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory.ReferenceCountingReadOnlyIndexReader] (value [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory$ReferenceCountingReadOnlyIndexReader@5d48afd4]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@4f0fb724]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@10c0c509]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@5fc7da76]) and a value of type [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory.ReferenceCountingReadOnlyIndexReader] (value [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory$ReferenceCountingReadOnlyIndexReader@39d7aa1a]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@7de0aea0]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@29e8caf0]) and a value of type [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory.ReferenceCountingReadOnlyIndexReader] (value [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory$ReferenceCountingReadOnlyIndexReader@7263680e]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [null] (value [org.springframework.extensions.webscripts.processor.FTLTemplateProcessor$NonBlockingObjectWrapper$1@5c44a9cd]) and a value of type [freemarker.template.DefaultObjectWrapper] (value [freemarker.template.DefaultObjectWrapper@1fe8010e]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@389d4e7f]) and a value of type [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory.ReferenceCountingReadOnlyIndexReader] (value [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory$ReferenceCountingReadOnlyIndexReader@6f7b0466]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@5f8c90e2]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.security.authentication.AuthenticationUtil.ThreadLocalStack] (value [org.alfresco.repo.security.authentication.AuthenticationUtil$ThreadLocalStack@57c1670]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@93f303a]) and a value of type [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory.ReferenceCountingReadOnlyIndexReader] (value [org.alfresco.repo.search.impl.lucene.index.ReferenceCountingReadOnlyIndexReaderFactory$ReferenceCountingReadOnlyIndexReader@13773494]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [org.alfresco.repo.policy.BaseBehaviour.StackThreadLocal] (value [org.alfresco.repo.policy.BaseBehaviour$StackThreadLocal@771cea87]) and a value of type [java.util.Stack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [null] (value [com.sun.xml.bind.v2.runtime.Coordinator$1@b698229]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@3be05799]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
May 28, 2012 5:24:03 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/alfresco] created a ThreadLocal with key of type [null] (value [org.alfresco.repo.tagging.TagScopePropertyMethodInterceptor$1@5971c85]) and a value of type [java.lang.Boolean] (value [true]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
log4j:ERROR LogMananger.repositorySelector was null likely due to error in class reloading, using NOPLoggerRepository.
May 28, 2012 5:24:07 PM org.apache.catalina.startup.HostConfig deployDescriptor











‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

this is what i have on my startup script


start_tomcat() {
    is_tomcat_running
    RUNNING=$?
    if [ $RUNNING -eq 1 ]; then
        echo "$0 $ARG: tomcat (pid $PID) already running"
    else
   rm -f $CATALINA_PID
   export JAVA_OPTS="-XX:MaxPermSize=1024m -Xms512m -Xmx768m -XX:-DisableExplicitGC -Dalfresco.home=/opt/indexing -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true"
   export JAVA_HOME=$JRE_HOME
   $TOMCAT_BINDIR/startup.sh 
   if [ $? -eq 0 ];  then
            echo "$0 $ARG: tomcat started"
   else
            echo "$0 $ARG: tomcat could not be started"
            ERROR=1
   fi
    fi
}

daemon_tomcat() {
    export JAVA_OPTS="-XX:MaxPermSize=1024m -Xms512m -Xmx768m -XX:-DisableExplicitGC -Dalfresco.home=/opt/indexing -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true"
    export JAVA_HOME=$JRE_HOME
    $TOMCAT_BINDIR/catalina.sh run
}

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: OutOfMemory/GC overhead limit error & performance tuning?

dynamolalit — Tue, 29 May 2012 06:14:37 GMT

Hi,

Looking into logs, i can see below error

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Ideal way to deal with this kind of situation is to fine tune your jvm settings, may be you need for heap sapce. Refer to alfresco docs for same.

Can use concurrent GC mechanism such as

JAVA_OPTS="-server -Xms1G -Xmx2G -XX:MaxPermSize=512M -Xss1024k -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70"

Refer to http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html for more details.

Regards.

Re: OutOfMemory/GC overhead limit error & performance tuning?

sharifu — Tue, 29 May 2012 09:54:46 GMT

this is what i got after inputting what you mentioned


2012-05-29 10:48:54,336  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread5] Reindexer     5 failed with error: 04290001 Failed to open stream onto channel: 
   accessor: ContentAccessor[ contentUrl=store://2011/10/6/19/59/fd7ef08c-d470-4aa4-a8c6-d5d74ac50071.bin, mimetype=text/plain, size=168558, encoding=UTF-8, locale=en_US].
 2012-05-29 10:48:54,341  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread5] org.alfresco.service.cmr.repository.ContentIOException: 04290001 Failed to open stream onto channel: 
   accessor: ContentAccessor[ contentUrl=store://2011/10/6/19/59/fd7ef08c-d470-4aa4-a8c6-d5d74ac50071.bin, mimetype=text/plain, size=168558, encoding=UTF-8, locale=en_US]
   at org.alfresco.repo.content.AbstractContentReader.getContentInputStream(AbstractContentReader.java:401)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.indexProperty(ADMLuceneIndexerImpl.java:1358)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocumentsImpl(ADMLuceneIndexerImpl.java:762)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocuments(ADMLuceneIndexerImpl.java:646)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2015)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2011)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl$1.execute(AbstractLuceneIndexerImpl.java:715)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.doInReadthroughTransaction(AbstractLuceneIndexerImpl.java:702)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.readDocuments(ADMLuceneIndexerImpl.java:2010)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1057)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: org.alfresco.service.cmr.repository.ContentIOException: 04290000 Failed to open file channel: ContentAccessor[ contentUrl=store://2011/10/6/19/59/fd7ef08c-d470-4aa4-a8c6-d5d74ac50071.bin, mimetype=text/plain, size=168558, encoding=UTF-8, locale=en_US]
   at org.alfresco.repo.content.filestore.FileContentReader.getDirectReadableChannel(FileContentReader.java:234)
   at org.alfresco.repo.content.AbstractContentReader.getReadableChannel(AbstractContentReader.java:271)
   at org.alfresco.repo.content.AbstractContentReader.getContentInputStream(AbstractContentReader.java:375)
   … 33 more
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/contentstore/2011/10/6/19/59/fd7ef08c-d470-4aa4-a8c6-d5d74ac50071.bin (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.alfresco.repo.content.filestore.FileContentReader.getDirectReadableChannel(FileContentReader.java:215)
   … 35 more

 533.830: [GC 533.830: [ParNew: 37544K->3248K(38336K), 0.0190850 secs] 1079904K->1046441K(1481972K) icms_dc=0 , 0.0191900 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
533.996: [GC 533.996: [ParNew: 37360K->2880K(38336K), 0.0216140 secs] 1080553K->1047096K(1481972K) icms_dc=0 , 0.0217410 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
534.150: [GC 534.150: [ParNew: 36992K->2739K(38336K), 0.0217150 secs] 1081208K->1048083K(1481972K) icms_dc=0 , 0.0218300 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
534.309: [GC 534.309: [ParNew: 36851K->2711K(38336K), 0.0188460 secs] 1082195K->1048769K(1481972K) icms_dc=0 , 0.0189490 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
2012-05-29 10:48:55,451  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread2] Reindexer     7 failed with error: 04290002 Failed to flush index.
 2012-05-29 10:48:55,463  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread2] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290002 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/b57d1fc0-fc9c-4f70-bea5-eb6f95f15d0b/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

 534.467: [GC 534.467: [ParNew: 36823K->3251K(38336K), 0.0146450 secs] 1082881K->1049309K(1481972K) icms_dc=0 , 0.0147580 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
534.589: [GC 534.589: [ParNew: 37363K->4216K(38336K), 0.0202520 secs] 1083421K->1050808K(1481972K) icms_dc=0 , 0.0203600 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
534.728: [GC 534.728: [ParNew: 38328K->3089K(38336K), 0.0223170 secs] 1084920K->1050483K(1481972K) icms_dc=0 , 0.0224410 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
534.875: [GC 534.875: [ParNew: 37201K->4224K(38336K), 0.0215070 secs] 1084595K->1052363K(1481972K) icms_dc=0 , 0.0216090 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
535.021: [GC 535.021: [ParNew: 38336K->3518K(38336K), 0.0222590 secs] 1086475K->1052339K(1481972K) icms_dc=0 , 0.0223630 secs] [Times: user=0.03 sys=0.01, real=0.03 secs] 
535.150: [GC 535.151: [ParNew: 37630K->3452K(38336K), 0.0202250 secs] 1086451K->1052946K(1481972K) icms_dc=0 , 0.0203570 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
535.275: [GC 535.275: [ParNew: 37564K->3533K(38336K), 0.0198810 secs] 1087058K->1053908K(1481972K) icms_dc=3 , 0.0199970 secs] [Times: user=0.03 sys=0.01, real=0.02 secs] 
535.354: [GC [1 CMS-initial-mark: 1050375K(1443636K)] 1070844K(1481972K), 0.0230910 secs] [Times: user=0.02 sys=0.00, real=0.03 secs] 
535.377: [CMS-concurrent-mark-start]
535.438: [GC 535.438: [ParNew: 37645K->2698K(38336K), 0.0202340 secs] 1088020K->1053686K(1481972K) icms_dc=6 , 0.0203360 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
535.570: [GC 535.570: [ParNew: 36810K->3847K(38336K), 0.0160300 secs] 1087798K->1054835K(1481972K) icms_dc=6 , 0.0161340 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
535.687: [GC 535.687: [ParNew: 37959K->2538K(38336K), 0.0247490 secs] 1088947K->1054787K(1481972K) icms_dc=8 , 0.0248600 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
535.816: [GC 535.816: [ParNew: 36650K->2217K(38336K), 0.0196780 secs] 1088899K->1055099K(1481972K) icms_dc=8 , 0.0198070 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
535.941: [GC 535.941: [ParNew: 36329K->4193K(38336K), 0.0157050 secs] 1089211K->1057075K(1481972K) icms_dc=8 , 0.0158350 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
536.061: [GC 536.061: [ParNew: 38305K->4224K(38336K), 0.0213470 secs] 1091187K->1057855K(1481972K) icms_dc=8 , 0.0214550 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
536.187: [GC 536.187: [ParNew: 38336K->4028K(38336K), 0.0219150 secs] 1091967K->1058275K(1481972K) icms_dc=8 , 0.0220230 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
536.312: [GC 536.312: [ParNew: 38140K->2724K(38336K), 0.0275350 secs] 1092387K->1058268K(1481972K) icms_dc=9 , 0.0276610 secs] [Times: user=0.05 sys=0.00, real=0.03 secs] 
536.451: [GC 536.451: [ParNew: 36836K->4224K(38336K), 0.0163620 secs] 1092380K->1059833K(1481972K) icms_dc=9 , 0.0164660 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
536.573: [GC 536.573: [ParNew: 38336K->4224K(38336K), 0.0205610 secs] 1093945K->1060462K(1481972K) icms_dc=9 , 0.0206780 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
536.708: [GC 536.708: [ParNew: 38336K->4204K(38336K), 0.0218090 secs] 1094574K->1061097K(1481972K) icms_dc=9 , 0.0219350 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
536.846: [GC 536.847: [ParNew: 38316K->3171K(38336K), 0.0252420 secs] 1095209K->1061357K(1481972K) icms_dc=9 , 0.0253530 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
536.999: [GC 536.999: [ParNew: 37283K->3889K(38336K), 0.0204310 secs] 1095469K->1062698K(1481972K) icms_dc=9 , 0.0205460 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
537.132: [GC 537.132: [ParNew: 38001K->2282K(38336K), 0.0194640 secs] 1096810K->1061786K(1481972K) icms_dc=9 , 0.0195880 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
2012-05-29 10:48:58,235  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread3] Reindexer     8 failed with error: 04290004 Failed to flush index.
 2012-05-29 10:48:58,238  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread3] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290004 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/d82b9a61-d8f7-4d9a-9c61-2b74475f544d/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

 2012-05-29 10:48:58,239  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread4] Reindexer     9 failed with error: 04290003 Failed to flush index.
 2012-05-29 10:48:58,241  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread4] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290003 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/108bbc28-579f-4b30-9c9b-9ba7e51020b2/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

 537.276: [GC 537.276: [ParNew: 36394K->4192K(38336K), 0.0157630 secs] 1095898K->1063696K(1481972K) icms_dc=9 , 0.0158720 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
537.410: [GC 537.410: [ParNew: 38304K->2685K(38336K), 0.0237660 secs] 1097808K->1063267K(1481972K) icms_dc=9 , 0.0238980 secs] [Times: user=0.04 sys=0.01, real=0.03 secs] 
537.547: [GC 537.547: [ParNew: 36797K->3872K(38336K), 0.0211820 secs] 1097379K->1065056K(1481972K) icms_dc=9 , 0.0213200 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
537.685: [GC 537.685: [ParNew: 37984K->2692K(38336K), 0.0218860 secs] 1099168K->1064719K(1481972K) icms_dc=9 , 0.0219870 secs] [Times: user=0.04 sys=0.01, real=0.02 secs] 
537.817: [GC 537.817: [ParNew: 36804K->3310K(38336K), 0.0203470 secs] 1098831K->1066021K(1481972K) icms_dc=9 , 0.0204840 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
537.948: [GC 537.948: [ParNew: 37422K->4223K(38336K), 0.0196730 secs] 1100133K->1067497K(1481972K) icms_dc=9 , 0.0197750 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
538.076: [GC 538.076: [ParNew: 38335K->2732K(38336K), 0.0205740 secs] 1101609K->1066635K(1481972K) icms_dc=9 , 0.0206830 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
538.204: [GC 538.204: [ParNew: 36844K->4040K(38336K), 0.0202740 secs] 1100747K->1068619K(1481972K) icms_dc=9 , 0.0203870 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
538.331: [GC 538.331: [ParNew: 38152K->2687K(38336K), 0.0202270 secs] 1102731K->1067940K(1481972K) icms_dc=9 , 0.0203470 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
538.458: [GC 538.458: [ParNew: 36799K->3937K(38336K), 0.0215430 secs] 1102052K->1069889K(1481972K) icms_dc=9 , 0.0216550 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
538.585: [GC 538.585: [ParNew: 38049K->2584K(38336K), 0.0191110 secs] 1104001K->1069183K(1481972K) icms_dc=9 , 0.0192150 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
538.711: [GC 538.711: [ParNew: 36696K->2868K(38336K), 0.0206450 secs] 1103295K->1070056K(1481972K) icms_dc=9 , 0.0207600 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
538.836: [GC 538.837: [ParNew: 36980K->3163K(38336K), 0.0210290 secs] 1104168K->1071009K(1481972K) icms_dc=9 , 0.0211650 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
538.962: [GC 538.962: [ParNew: 37275K->3250K(38336K), 0.0190350 secs] 1105121K->1071698K(1481972K) icms_dc=9 , 0.0191520 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
539.100: [GC 539.100: [ParNew: 37362K->3548K(38336K), 0.0201790 secs] 1105810K->1072681K(1481972K) icms_dc=9 , 0.0202930 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
539.231: [GC 539.232: [ParNew: 37660K->3211K(38336K), 0.0199510 secs] 1106793K->1072941K(1481972K) icms_dc=9 , 0.0200530 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
539.363: [GC 539.363: [ParNew: 37323K->3110K(38336K), 0.0217650 secs] 1107053K->1073515K(1481972K) icms_dc=9 , 0.0219080 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 
539.500: [GC 539.500: [ParNew: 37222K->4224K(38336K), 0.0153280 secs] 1107627K->1074686K(1481972K) icms_dc=9 , 0.0154320 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
540.035: [GC 540.035: [ParNew: 38336K->3830K(38336K), 0.0288520 secs] 1108798K->1075461K(1481972K) icms_dc=9 , 0.0289820 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
2012-05-29 10:49:01,145  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread5] Reindexer    10 failed with error: 04290006 Failed to flush index.
 2012-05-29 10:49:01,147  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread5] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290006 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/51d11547-929b-48b1-ae9f-133efb555c3f/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

 540.268: [GC 540.268: [ParNew: 37942K->2693K(38336K), 0.0264930 secs] 1109573K->1076091K(1481972K) icms_dc=9 , 0.0265990 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 
2012-05-29 10:49:01,451  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread1] Reindexer    11 failed with error: 04290008 Failed to flush index.
 2012-05-29 10:49:01,452  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread1] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290008 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/6721b43e-2035-4aef-8cfa-abf61901dd8e/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

 540.419: [GC 540.419: [ParNew: 36805K->2852K(38336K), 0.0154110 secs] 1110203K->1076250K(1481972K) icms_dc=9 , 0.0155110 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
540.550: [GC 540.550: [ParNew: 36964K->2580K(38336K), 0.0193940 secs] 1110362K->1076836K(1481972K) icms_dc=9 , 0.0195040 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
540.691: [GC 540.692: [ParNew: 36692K->3900K(38336K), 0.0156010 secs] 1110948K->1078155K(1481972K) icms_dc=9 , 0.0157030 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
540.819: [GC 540.819: [ParNew: 38012K->3712K(38336K), 0.0234860 secs] 1112267K->1078938K(1481972K) icms_dc=9 , 0.0236060 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
540.962: [GC 540.962: [ParNew: 37824K->2536K(38336K), 0.0232800 secs] 1113050K->1078330K(1481972K) icms_dc=9 , 0.0233790 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 
541.156: [GC 541.156: [ParNew: 36648K->2024K(38336K), 0.0180530 secs] 1112442K->1078358K(1481972K) icms_dc=9 , 0.0181770 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
541.361: [GC 541.361: [ParNew: 36136K->3238K(38336K), 0.0149010 secs] 1112470K->1079572K(1481972K) icms_dc=9 , 0.0150200 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
541.574: [GC 541.574: [ParNew: 37350K->4076K(38336K), 0.0196560 secs] 1113684K->1080796K(1481972K) icms_dc=9 , 0.0197670 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
541.787: [GC 541.787: [ParNew: 38188K->2847K(38336K), 0.0192210 secs] 1114908K->1080001K(1481972K) icms_dc=9 , 0.0193270 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
541.996: [GC 541.996: [ParNew: 36959K->4224K(38336K), 0.0228670 secs] 1114113K->1082027K(1481972K) icms_dc=9 , 0.0229720 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
542.244: [GC 542.244: [ParNew: 38336K->3842K(38336K), 0.0223210 secs] 1116139K->1082296K(1481972K) icms_dc=9 , 0.0224380 secs] [Times: user=0.03 sys=0.01, real=0.03 secs] 
542.463: [GC 542.463: [ParNew: 37954K->3940K(38336K), 0.0215140 secs] 1116408K->1083046K(1481972K) icms_dc=9 , 0.0216220 secs] [Times: user=0.04 sys=0.01, real=0.03 secs] 
542.682: [GC 542.682: [ParNew: 38052K->3961K(38336K), 0.0212900 secs] 1117158K->1083707K(1481972K) icms_dc=9 , 0.0213890 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
542.916: [GC 542.916: [ParNew: 38073K->3946K(38336K), 0.0216000 secs] 1117819K->1084313K(1481972K) icms_dc=9 , 0.0217140 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
543.134: [GC 543.134: [ParNew: 38058K->3979K(38336K), 0.0211750 secs] 1118425K->1084996K(1481972K) icms_dc=9 , 0.0212920 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
543.346: [GC 543.346: [ParNew: 38091K->3971K(38336K), 0.0212490 secs] 1119108K->1085641K(1481972K) icms_dc=9 , 0.0213580 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
2012-05-29 10:49:04,535  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread2] Reindexer    12 failed with error: 04290011 Failed to flush index.
 2012-05-29 10:49:04,536  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread2] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290011 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/2d53acc7-87ec-4767-8ad6-af23018bae9c/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

 2012-05-29 10:49:04,537  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread3] Reindexer    13 failed with error: 04290005 Failed to flush index.
 2012-05-29 10:49:04,538  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread3] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290005 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/d517c4c1-2f7b-4cd4-bd7e-519a1f773331/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

 2012-05-29 10:49:04,539  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread4] Reindexer    14 failed with error: 04290007 Failed to flush index.
 2012-05-29 10:49:04,540  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread4] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290007 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/3fef9731-93c3-49a6-a81b-712b56ab2559/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

 2012-05-29 10:49:04,540  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread5] Reindexer    15 failed with error: 04290009 Failed to flush index.
 2012-05-29 10:49:04,541  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread5] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290009 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/52e200d0-67e7-48c6-97a6-78d55a563e63/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

 2012-05-29 10:49:04,542  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread1] Reindexer    16 failed with error: 04290010 Failed to flush index.
 2012-05-29 10:49:04,543  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread1] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290010 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/a7514926-3562-455a-9b1f-86b4bfa7d843/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

 543.823: [GC 543.823: [ParNew: 38083K->4184K(38336K), 0.0211870 secs] 1119753K->1086488K(1481972K) icms_dc=9 , 0.0213230 secs] [Times: user=0.04 sys=0.00, real=0.03 secs] 
544.088: [GC 544.089: [ParNew: 38296K->4224K(38336K), 0.0392680 secs] 1120600K->1089383K(1481972K) icms_dc=9 , 0.0393770 secs] [Times: user=0.06 sys=0.01, real=0.04 secs] 
544.458: [GC 544.458: [ParNew: 38336K->3904K(38336K), 0.0277950 secs] 1123495K->1090060K(1481972K) icms_dc=9 , 0.0279200 secs] [Times: user=0.04 sys=0.01, real=0.03 secs] 
544.712: [GC 544.712: [ParNew: 38016K->3437K(38336K), 0.0207070 secs] 1124172K->1089995K(1481972K) icms_dc=9 , 0.0208310 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
545.086: [GC 545.086: [ParNew: 37549K->3327K(38336K), 0.0189540 secs] 1124107K->1089944K(1481972K) icms_dc=9 , 0.0191630 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 
2012-05-29 10:49:06,414  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread1] Reindexer    41 failed with error: 04290012 Failed to flush index.
 2012-05-29 10:49:06,415  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread1] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290012 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.GeneratedMethodAccessor314.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/5321aa32-6885-4bec-af7e-06eccfe60551/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 22 more

 545.580: [GC 545.581: [ParNew: 37439K->4224K(38336K), 0.0519850 secs] 1124056K->1094639K(1481972K) icms_dc=9 , 0.0521030 secs] [Times: user=0.07 sys=0.00, real=0.05 secs] 
545.841: [GC 545.841: [ParNew: 38336K->4224K(38336K), 0.0249280 secs] 1128751K->1095804K(1481972K) icms_dc=9 , 0.0250420 secs] [Times: user=0.03 sys=0.00, real=0.03 secs] 
545.980: [GC 545.980: [ParNew: 38336K->4188K(38336K), 0.0184720 secs] 1129916K->1095768K(1481972K) icms_dc=9 , 0.0185730 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 
2012-05-29 10:49:07,106  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread4] Reindexer    43 failed with error: 04290013 Failed to flush index.
 2012-05-29 10:49:07,107  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread4] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290013 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.GeneratedMethodAccessor314.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/workspace/SpacesStore/11ba62b1-66f4-46b5-99de-3fe303f99baa/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 22 more

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

with the follwoing settings

start_tomcat() {
    is_tomcat_running
    RUNNING=$?
    if [ $RUNNING -eq 1 ]; then
        echo "$0 $ARG: tomcat (pid $PID) already running"
    else
   rm -f $CATALINA_PID
   export JAVA_OPTS="-server -Xms1G -Xmx2G -XX:MaxPermSize=512M -Xss1024k -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -Dalfresco.home=/opt/indexing -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true"
   export JAVA_HOME=$JRE_HOME
   $TOMCAT_BINDIR/startup.sh 
   if [ $? -eq 0 ];  then
            echo "$0 $ARG: tomcat started"
   else
            echo "$0 $ARG: tomcat could not be started"
            ERROR=1
   fi
    fi
}

daemon_tomcat() {
    #export JAVA_OPTS="-XX:MaxPermSize=1024m -Xms512m -Xmx1024m -XX:-UseGCOverheadLimit -XX:+UseConcMarkSweepGC -XX:-DisableExplicitGC -Dalfresco.home=/opt/indexing -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true"
    export JAVA_OPTS="-server -Xms1G -Xmx2G -XX:MaxPermSize=512M -Xss1024k -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -Dalfresco.home=/opt/indexing -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true"
    export JAVA_HOME=$JRE_HOME
    $TOMCAT_BINDIR/catalina.sh run
}
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: OutOfMemory/GC overhead limit error & performance tuning?

dynamolalit — Wed, 30 May 2012 12:12:24 GMT

Hi,

I can see below error

Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/b57d1fc0-fc9c-4f70-bea5-eb6f95f15d0b/segments_1 (Too many open files)

I think its related to Too many open files. Please refer to http://wiki.alfresco.com/wiki/Too_many_open_files for resolution.

Finally ensure you have all necessary RAM at exposure to alfresco say fair percentage of overall machine RAM and try to fine tune tomcat as well as db also.

Regards.

Re: OutOfMemory/GC overhead limit error & performance tuning?

dynamolalit — Wed, 27 Jun 2012 05:46:56 GMT

Hi All,

After a long period of implementations & observations, i can pin point on following items to get this (OutOfMemory) kind of errors resolved/minimized. Rule of thumb would be to have enough RAM & processing power for your alfresco as well as DB fine tuning.

1. Always use 64 bit machines with Quad Core CPUs to get maximum throughput and expose a good amount of RAM to alfresco installations.
2. Use alfresco 4.0 onwards, if you can use, which basically has a lot of improvements in terms of performance, indexing etc.
3. Optimize DB for best performance especially the number of open connections, connection closing delays etc & vendor specific optimizations.

### database connection properties ###
db.driver=org.gjt.mm.mysql.Driver
db.username=admin
db.password=admin
db.name=alfresco
db.url=jdbc:mysql://localhost:3306/alfresco
hibernate.dialect=org.hibernate.dialect.MySQLInnoDBDialect
db.pool.min=20
db.pool.max=275
db.pool.validate.return=true
db.pool.validate.query=select 1
db.pool.evict.idle.min=550000
db.pool.evict.validate=true
db.pool.abandoned.detect=true
db.pool.abandoned.time=550

4. Optimize JVM for best results like as below. It has following main points.
     Use Concurrent sweep algorithm.
     Use -XX:NewSize for Young Gen.
     User -XX:CMSInitiatingOccupancyFraction for major GC.

JAVA_OPTS="-server -Xms2G -Xmx4G -XX:MaxPermSize=256M -XX:NewSize=1G -Xss1M -verbose:gc -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70"

5. Optimize lucene, if you are using it, to get most of it as it can be a culprit on a number of occasions using configurations such as

lucene.indexer.maxDocsForInMemoryIndex=1000
lucene.indexer.maxDocsForInMemoryMerge=1000

6. I have not used SOLR but you can try out for getting best performance.
7. Clean alfresco temporary (<alf_home>\tomcat\temp) folders periodically to remove lot of temporary files such as related to transformations etc to avoid them taking a lot of disk space which can degrade performance.
8. Finally, monitor performance using tools like Jvisualvm, jmx reader etc from time to time.

Hope this helps.

Regards.