cancel
Showing results for 
Search instead for 
Did you mean: 

OutOfMemory/GC overhead limit error & performance tuning?

dynamolalit
Champ on-the-rise
Champ on-the-rise
Hi,

I am using alfresco 3.2 over tomcat 6 & jdk 1.6. Till sometime back , everything was working fine, but from last 15 days or so, i am getting java.lang.OutOfMemoryError as below:

10:44:11,769 ERROR [org.alfresco.web.ui.common.Utils] Failed to get completed tasks: java.lang.OutOfMemoryError: GC overhead limit exceeded

Here is full details:


10:50:51,480 ERROR [org.alfresco.web.ui.common.Utils] A system error happened during the operation: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
10:50:51,480 ERROR [org.hibernate.transaction.JDBCTransaction] JDBC rollback failed
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Connection.close() has already been called. Invalid operation in this state.
   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
   at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
   at com.mysql.jdbc.Util.getInstance(Util.java:381)
   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:984)
   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
   at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)
   at com.mysql.jdbc.ConnectionImpl.getMutex(ConnectionImpl.java:3032)
   at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:4606)
   at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
   at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:328)
   at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:312)
   at org.hibernate.transaction.JDBCTransaction.rollbackAndResetAutoCommit(JDBCTransaction.java:183)
   at org.hibernate.transaction.JDBCTransaction.rollback(JDBCTransaction.java:162)
   at org.springframework.orm.hibernate3.HibernateTransactionManager.doRollback(HibernateTransactionManager.java:597)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:753)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.rollback(AbstractPlatformTransactionManager.java:730)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.completeTransactionAfterThrowing(TransactionAspectSupport.java:332)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.completeTransactionAfterThrowing(SpringAwareUserTransaction.java:562)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.rollback(SpringAwareUserTransaction.java:532)
   at org.alfresco.web.bean.BrowseBean.queryBrowseNodes(BrowseBean.java:1016)
   at org.alfresco.web.bean.BrowseBean.getNodes(BrowseBean.java:511)
   at sun.reflect.GeneratedMethodAccessor964.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.myfaces.el.PropertyResolverImpl.getProperty(PropertyResolverImpl.java:457)
   at org.apache.myfaces.el.PropertyResolverImpl.getValue(PropertyResolverImpl.java:85)
   at org.apache.myfaces.el.ELParserHelper$MyPropertySuffix.evaluate(ELParserHelper.java:535)
   at org.apache.commons.el.ComplexValue.evaluate(ComplexValue.java:145)
   at org.apache.myfaces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:386)
   at org.alfresco.web.ui.common.component.data.UIRichList.getValue(UIRichList.java:157)
   at org.alfresco.web.ui.common.component.data.UIRichList.getDataModel(UIRichList.java:518)
   at org.alfresco.web.ui.common.component.data.UIRichList.bind(UIRichList.java:454)
   at org.alfresco.web.ui.common.renderer.data.RichListRenderer.encodeChildren(RichListRenderer.java:83)
   at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:527)
   at javax.faces.webapp.UIComponentTag.encodeChildren(UIComponentTag.java:483)
   at javax.faces.webapp.UIComponentTag.doEndTag(UIComponentTag.java:364)
   at org.apache.jsp.jsp.browse.browse_jsp._jspx_meth_a_005frichList_005f0(browse_jsp.java:2697)
   at org.apache.jsp.jsp.browse.browse_jsp._jspx_meth_a_005fpanel_005f5(browse_jsp.java:2585)
   at org.apache.jsp.jsp.browse.browse_jsp._jspService(browse_jsp.java:849)
   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:630)
   at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
   at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
   at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
   at org.apache.myfaces.context.servlet.ServletExternalContextImpl.dispatch(ServletExternalContextImpl.java:419)
   at org.apache.myfaces.application.jsp.JspViewHandlerImpl.renderView(JspViewHandlerImpl.java:211)
   at org.apache.myfaces.lifecycle.RenderResponseExecutor.execute(RenderResponseExecutor.java:41)
   at org.apache.myfaces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:132)
   at javax.faces.webapp.FacesServlet.service(FacesServlet.java:140)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at com.keembay.alfresco.web.app.servlet.CASAuthenticationFilter.doFilter(CASAuthenticationFilter.java:149)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.alfresco.repo.web.filter.beans.NullFilter.doFilter(NullFilter.java:74)
   at sun.reflect.GeneratedMethodAccessor707.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.management.subsystems.ChainingSubsystemProxyFactory$1.invoke(ChainingSubsystemProxyFactory.java:122)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
   at $Proxy224.doFilter(Unknown Source)
   at org.alfresco.repo.web.filter.beans.BeanProxyFilter.doFilter(BeanProxyFilter.java:88)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at edu.yale.its.tp.cas.client.filter.CASFilter.doFilter(CASFilter.java:317)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
   at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:354)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
10:50:54,324 ERROR [org.alfresco.util.transaction.SpringAwareUserTransaction] Application exception overridden by rollback exception
java.lang.Exception
   at org.alfresco.util.transaction.SpringAwareUserTransaction.rollback(SpringAwareUserTransaction.java:532)
   at org.alfresco.web.bean.BrowseBean.queryBrowseNodes(BrowseBean.java:1016)
   at org.alfresco.web.bean.BrowseBean.getNodes(BrowseBean.java:511)
   at sun.reflect.GeneratedMethodAccessor964.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.myfaces.el.PropertyResolverImpl.getProperty(PropertyResolverImpl.java:457)
   at org.apache.myfaces.el.PropertyResolverImpl.getValue(PropertyResolverImpl.java:85)
   at org.apache.myfaces.el.ELParserHelper$MyPropertySuffix.evaluate(ELParserHelper.java:535)
   at org.apache.commons.el.ComplexValue.evaluate(ComplexValue.java:145)
   at org.apache.myfaces.el.ValueBindingImpl.getValue(ValueBindingImpl.java:386)
   at org.alfresco.web.ui.common.component.data.UIRichList.getValue(UIRichList.java:157)
   at org.alfresco.web.ui.common.component.data.UIRichList.getDataModel(UIRichList.java:518)
   at org.alfresco.web.ui.common.component.data.UIRichList.bind(UIRichList.java:454)
   at org.alfresco.web.ui.common.renderer.data.RichListRenderer.encodeChildren(RichListRenderer.java:83)
   at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:527)
   at javax.faces.webapp.UIComponentTag.encodeChildren(UIComponentTag.java:483)
   at javax.faces.webapp.UIComponentTag.doEndTag(UIComponentTag.java:364)
   at org.apache.jsp.jsp.browse.browse_jsp._jspx_meth_a_005frichList_005f0(browse_jsp.java:2697)
   at org.apache.jsp.jsp.browse.browse_jsp._jspx_meth_a_005fpanel_005f5(browse_jsp.java:2585)
   at org.apache.jsp.jsp.browse.browse_jsp._jspService(browse_jsp.java:849)
   at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:374)
   at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:342)
   at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:267)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:630)
   at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
   at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:374)
   at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
   at org.apache.myfaces.context.servlet.ServletExternalContextImpl.dispatch(ServletExternalContextImpl.java:419)
   at org.apache.myfaces.application.jsp.JspViewHandlerImpl.renderView(JspViewHandlerImpl.java:211)
   at org.apache.myfaces.lifecycle.RenderResponseExecutor.execute(RenderResponseExecutor.java:41)
   at org.apache.myfaces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:132)
   at javax.faces.webapp.FacesServlet.service(FacesServlet.java:140)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at com.keembay.alfresco.web.app.servlet.CASAuthenticationFilter.doFilter(CASAuthenticationFilter.java:149)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.alfresco.repo.web.filter.beans.NullFilter.doFilter(NullFilter.java:74)
   at sun.reflect.GeneratedMethodAccessor707.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.management.subsystems.ChainingSubsystemProxyFactory$1.invoke(ChainingSubsystemProxyFactory.java:122)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
   at $Proxy224.doFilter(Unknown Source)
   at org.alfresco.repo.web.filter.beans.BeanProxyFilter.doFilter(BeanProxyFilter.java:88)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at edu.yale.its.tp.cas.client.filter.CASFilter.doFilter(CASFilter.java:317)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
   at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:354)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:619)
10:51:03,152 WARN  [org.alfresco.repo.activities.feed.AbstractFeedGenerator] Still busy …

Server is getting down again n again & i am getting no idea how to fix it. I have done some customizations as web scripts, content mpdel, evaluator etc.

Any idea??
13 REPLIES 13

dynamolalit
Champ on-the-rise
Champ on-the-rise
Hi,

Looking into logs, i can see below error

Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Ideal way to deal with this kind of situation is to fine tune your jvm settings, may be you need for heap sapce. Refer to alfresco docs for same.

Can use concurrent GC mechanism such as

JAVA_OPTS="-server -Xms1G -Xmx2G -XX:MaxPermSize=512M -Xss1024k  -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70"

Refer to http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html for more details.

Regards.

sharifu
Confirmed Champ
Confirmed Champ
this is what i got after inputting what you mentioned


2012-05-29 10:48:54,336  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread5] Reindexer     5 failed with error: 04290001 Failed to open stream onto channel:
   accessor: ContentAccessor[ contentUrl=store://2011/10/6/19/59/fd7ef08c-d470-4aa4-a8c6-d5d74ac50071.bin, mimetype=text/plain, size=168558, encoding=UTF-8, locale=en_US].
2012-05-29 10:48:54,341  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread5] org.alfresco.service.cmr.repository.ContentIOException: 04290001 Failed to open stream onto channel:
   accessor: ContentAccessor[ contentUrl=store://2011/10/6/19/59/fd7ef08c-d470-4aa4-a8c6-d5d74ac50071.bin, mimetype=text/plain, size=168558, encoding=UTF-8, locale=en_US]
   at org.alfresco.repo.content.AbstractContentReader.getContentInputStream(AbstractContentReader.java:401)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.indexProperty(ADMLuceneIndexerImpl.java:1358)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocumentsImpl(ADMLuceneIndexerImpl.java:762)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocuments(ADMLuceneIndexerImpl.java:646)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2015)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:2011)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl$1.execute(AbstractLuceneIndexerImpl.java:715)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.doInReadthroughTransaction(AbstractLuceneIndexerImpl.java:702)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.readDocuments(ADMLuceneIndexerImpl.java:2010)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1057)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: org.alfresco.service.cmr.repository.ContentIOException: 04290000 Failed to open file channel: ContentAccessor[ contentUrl=store://2011/10/6/19/59/fd7ef08c-d470-4aa4-a8c6-d5d74ac50071.bin, mimetype=text/plain, size=168558, encoding=UTF-8, locale=en_US]
   at org.alfresco.repo.content.filestore.FileContentReader.getDirectReadableChannel(FileContentReader.java:234)
   at org.alfresco.repo.content.AbstractContentReader.getReadableChannel(AbstractContentReader.java:271)
   at org.alfresco.repo.content.AbstractContentReader.getContentInputStream(AbstractContentReader.java:375)
   … 33 more
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/contentstore/2011/10/6/19/59/fd7ef08c-d470-4aa4-a8c6-d5d74ac50071.bin (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.alfresco.repo.content.filestore.FileContentReader.getDirectReadableChannel(FileContentReader.java:215)
   … 35 more

533.830: [GC 533.830: [ParNew: 37544K->3248K(38336K), 0.0190850 secs] 1079904K->1046441K(1481972K) icms_dc=0 , 0.0191900 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
533.996: [GC 533.996: [ParNew: 37360K->2880K(38336K), 0.0216140 secs] 1080553K->1047096K(1481972K) icms_dc=0 , 0.0217410 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
534.150: [GC 534.150: [ParNew: 36992K->2739K(38336K), 0.0217150 secs] 1081208K->1048083K(1481972K) icms_dc=0 , 0.0218300 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
534.309: [GC 534.309: [ParNew: 36851K->2711K(38336K), 0.0188460 secs] 1082195K->1048769K(1481972K) icms_dc=0 , 0.0189490 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
2012-05-29 10:48:55,451  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread2] Reindexer     7 failed with error: 04290002 Failed to flush index.
2012-05-29 10:48:55,463  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread2] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290002 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/b57d1fc0-fc9c-4f70-bea5-eb6f95f15d0b/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

534.467: [GC 534.467: [ParNew: 36823K->3251K(38336K), 0.0146450 secs] 1082881K->1049309K(1481972K) icms_dc=0 , 0.0147580 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
534.589: [GC 534.589: [ParNew: 37363K->4216K(38336K), 0.0202520 secs] 1083421K->1050808K(1481972K) icms_dc=0 , 0.0203600 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
534.728: [GC 534.728: [ParNew: 38328K->3089K(38336K), 0.0223170 secs] 1084920K->1050483K(1481972K) icms_dc=0 , 0.0224410 secs] [Times: user=0.03 sys=0.01, real=0.02 secs]
534.875: [GC 534.875: [ParNew: 37201K->4224K(38336K), 0.0215070 secs] 1084595K->1052363K(1481972K) icms_dc=0 , 0.0216090 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
535.021: [GC 535.021: [ParNew: 38336K->3518K(38336K), 0.0222590 secs] 1086475K->1052339K(1481972K) icms_dc=0 , 0.0223630 secs] [Times: user=0.03 sys=0.01, real=0.03 secs]
535.150: [GC 535.151: [ParNew: 37630K->3452K(38336K), 0.0202250 secs] 1086451K->1052946K(1481972K) icms_dc=0 , 0.0203570 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
535.275: [GC 535.275: [ParNew: 37564K->3533K(38336K), 0.0198810 secs] 1087058K->1053908K(1481972K) icms_dc=3 , 0.0199970 secs] [Times: user=0.03 sys=0.01, real=0.02 secs]
535.354: [GC [1 CMS-initial-mark: 1050375K(1443636K)] 1070844K(1481972K), 0.0230910 secs] [Times: user=0.02 sys=0.00, real=0.03 secs]
535.377: [CMS-concurrent-mark-start]
535.438: [GC 535.438: [ParNew: 37645K->2698K(38336K), 0.0202340 secs] 1088020K->1053686K(1481972K) icms_dc=6 , 0.0203360 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
535.570: [GC 535.570: [ParNew: 36810K->3847K(38336K), 0.0160300 secs] 1087798K->1054835K(1481972K) icms_dc=6 , 0.0161340 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
535.687: [GC 535.687: [ParNew: 37959K->2538K(38336K), 0.0247490 secs] 1088947K->1054787K(1481972K) icms_dc=8 , 0.0248600 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
535.816: [GC 535.816: [ParNew: 36650K->2217K(38336K), 0.0196780 secs] 1088899K->1055099K(1481972K) icms_dc=8 , 0.0198070 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
535.941: [GC 535.941: [ParNew: 36329K->4193K(38336K), 0.0157050 secs] 1089211K->1057075K(1481972K) icms_dc=8 , 0.0158350 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
536.061: [GC 536.061: [ParNew: 38305K->4224K(38336K), 0.0213470 secs] 1091187K->1057855K(1481972K) icms_dc=8 , 0.0214550 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
536.187: [GC 536.187: [ParNew: 38336K->4028K(38336K), 0.0219150 secs] 1091967K->1058275K(1481972K) icms_dc=8 , 0.0220230 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
536.312: [GC 536.312: [ParNew: 38140K->2724K(38336K), 0.0275350 secs] 1092387K->1058268K(1481972K) icms_dc=9 , 0.0276610 secs] [Times: user=0.05 sys=0.00, real=0.03 secs]
536.451: [GC 536.451: [ParNew: 36836K->4224K(38336K), 0.0163620 secs] 1092380K->1059833K(1481972K) icms_dc=9 , 0.0164660 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
536.573: [GC 536.573: [ParNew: 38336K->4224K(38336K), 0.0205610 secs] 1093945K->1060462K(1481972K) icms_dc=9 , 0.0206780 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
536.708: [GC 536.708: [ParNew: 38336K->4204K(38336K), 0.0218090 secs] 1094574K->1061097K(1481972K) icms_dc=9 , 0.0219350 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
536.846: [GC 536.847: [ParNew: 38316K->3171K(38336K), 0.0252420 secs] 1095209K->1061357K(1481972K) icms_dc=9 , 0.0253530 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
536.999: [GC 536.999: [ParNew: 37283K->3889K(38336K), 0.0204310 secs] 1095469K->1062698K(1481972K) icms_dc=9 , 0.0205460 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
537.132: [GC 537.132: [ParNew: 38001K->2282K(38336K), 0.0194640 secs] 1096810K->1061786K(1481972K) icms_dc=9 , 0.0195880 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
2012-05-29 10:48:58,235  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread3] Reindexer     8 failed with error: 04290004 Failed to flush index.
2012-05-29 10:48:58,238  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread3] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290004 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/d82b9a61-d8f7-4d9a-9c61-2b74475f544d/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

2012-05-29 10:48:58,239  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread4] Reindexer     9 failed with error: 04290003 Failed to flush index.
2012-05-29 10:48:58,241  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread4] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290003 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/108bbc28-579f-4b30-9c9b-9ba7e51020b2/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

537.276: [GC 537.276: [ParNew: 36394K->4192K(38336K), 0.0157630 secs] 1095898K->1063696K(1481972K) icms_dc=9 , 0.0158720 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
537.410: [GC 537.410: [ParNew: 38304K->2685K(38336K), 0.0237660 secs] 1097808K->1063267K(1481972K) icms_dc=9 , 0.0238980 secs] [Times: user=0.04 sys=0.01, real=0.03 secs]
537.547: [GC 537.547: [ParNew: 36797K->3872K(38336K), 0.0211820 secs] 1097379K->1065056K(1481972K) icms_dc=9 , 0.0213200 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
537.685: [GC 537.685: [ParNew: 37984K->2692K(38336K), 0.0218860 secs] 1099168K->1064719K(1481972K) icms_dc=9 , 0.0219870 secs] [Times: user=0.04 sys=0.01, real=0.02 secs]
537.817: [GC 537.817: [ParNew: 36804K->3310K(38336K), 0.0203470 secs] 1098831K->1066021K(1481972K) icms_dc=9 , 0.0204840 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
537.948: [GC 537.948: [ParNew: 37422K->4223K(38336K), 0.0196730 secs] 1100133K->1067497K(1481972K) icms_dc=9 , 0.0197750 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
538.076: [GC 538.076: [ParNew: 38335K->2732K(38336K), 0.0205740 secs] 1101609K->1066635K(1481972K) icms_dc=9 , 0.0206830 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
538.204: [GC 538.204: [ParNew: 36844K->4040K(38336K), 0.0202740 secs] 1100747K->1068619K(1481972K) icms_dc=9 , 0.0203870 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
538.331: [GC 538.331: [ParNew: 38152K->2687K(38336K), 0.0202270 secs] 1102731K->1067940K(1481972K) icms_dc=9 , 0.0203470 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
538.458: [GC 538.458: [ParNew: 36799K->3937K(38336K), 0.0215430 secs] 1102052K->1069889K(1481972K) icms_dc=9 , 0.0216550 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
538.585: [GC 538.585: [ParNew: 38049K->2584K(38336K), 0.0191110 secs] 1104001K->1069183K(1481972K) icms_dc=9 , 0.0192150 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
538.711: [GC 538.711: [ParNew: 36696K->2868K(38336K), 0.0206450 secs] 1103295K->1070056K(1481972K) icms_dc=9 , 0.0207600 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
538.836: [GC 538.837: [ParNew: 36980K->3163K(38336K), 0.0210290 secs] 1104168K->1071009K(1481972K) icms_dc=9 , 0.0211650 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
538.962: [GC 538.962: [ParNew: 37275K->3250K(38336K), 0.0190350 secs] 1105121K->1071698K(1481972K) icms_dc=9 , 0.0191520 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
539.100: [GC 539.100: [ParNew: 37362K->3548K(38336K), 0.0201790 secs] 1105810K->1072681K(1481972K) icms_dc=9 , 0.0202930 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
539.231: [GC 539.232: [ParNew: 37660K->3211K(38336K), 0.0199510 secs] 1106793K->1072941K(1481972K) icms_dc=9 , 0.0200530 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
539.363: [GC 539.363: [ParNew: 37323K->3110K(38336K), 0.0217650 secs] 1107053K->1073515K(1481972K) icms_dc=9 , 0.0219080 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
539.500: [GC 539.500: [ParNew: 37222K->4224K(38336K), 0.0153280 secs] 1107627K->1074686K(1481972K) icms_dc=9 , 0.0154320 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
540.035: [GC 540.035: [ParNew: 38336K->3830K(38336K), 0.0288520 secs] 1108798K->1075461K(1481972K) icms_dc=9 , 0.0289820 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
2012-05-29 10:49:01,145  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread5] Reindexer    10 failed with error: 04290006 Failed to flush index.
2012-05-29 10:49:01,147  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread5] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290006 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/51d11547-929b-48b1-ae9f-133efb555c3f/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

540.268: [GC 540.268: [ParNew: 37942K->2693K(38336K), 0.0264930 secs] 1109573K->1076091K(1481972K) icms_dc=9 , 0.0265990 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
2012-05-29 10:49:01,451  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread1] Reindexer    11 failed with error: 04290008 Failed to flush index.
2012-05-29 10:49:01,452  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread1] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290008 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/6721b43e-2035-4aef-8cfa-abf61901dd8e/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

540.419: [GC 540.419: [ParNew: 36805K->2852K(38336K), 0.0154110 secs] 1110203K->1076250K(1481972K) icms_dc=9 , 0.0155110 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
540.550: [GC 540.550: [ParNew: 36964K->2580K(38336K), 0.0193940 secs] 1110362K->1076836K(1481972K) icms_dc=9 , 0.0195040 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
540.691: [GC 540.692: [ParNew: 36692K->3900K(38336K), 0.0156010 secs] 1110948K->1078155K(1481972K) icms_dc=9 , 0.0157030 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
540.819: [GC 540.819: [ParNew: 38012K->3712K(38336K), 0.0234860 secs] 1112267K->1078938K(1481972K) icms_dc=9 , 0.0236060 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
540.962: [GC 540.962: [ParNew: 37824K->2536K(38336K), 0.0232800 secs] 1113050K->1078330K(1481972K) icms_dc=9 , 0.0233790 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
541.156: [GC 541.156: [ParNew: 36648K->2024K(38336K), 0.0180530 secs] 1112442K->1078358K(1481972K) icms_dc=9 , 0.0181770 secs] [Times: user=0.03 sys=0.00, real=0.01 secs]
541.361: [GC 541.361: [ParNew: 36136K->3238K(38336K), 0.0149010 secs] 1112470K->1079572K(1481972K) icms_dc=9 , 0.0150200 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
541.574: [GC 541.574: [ParNew: 37350K->4076K(38336K), 0.0196560 secs] 1113684K->1080796K(1481972K) icms_dc=9 , 0.0197670 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
541.787: [GC 541.787: [ParNew: 38188K->2847K(38336K), 0.0192210 secs] 1114908K->1080001K(1481972K) icms_dc=9 , 0.0193270 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
541.996: [GC 541.996: [ParNew: 36959K->4224K(38336K), 0.0228670 secs] 1114113K->1082027K(1481972K) icms_dc=9 , 0.0229720 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
542.244: [GC 542.244: [ParNew: 38336K->3842K(38336K), 0.0223210 secs] 1116139K->1082296K(1481972K) icms_dc=9 , 0.0224380 secs] [Times: user=0.03 sys=0.01, real=0.03 secs]
542.463: [GC 542.463: [ParNew: 37954K->3940K(38336K), 0.0215140 secs] 1116408K->1083046K(1481972K) icms_dc=9 , 0.0216220 secs] [Times: user=0.04 sys=0.01, real=0.03 secs]
542.682: [GC 542.682: [ParNew: 38052K->3961K(38336K), 0.0212900 secs] 1117158K->1083707K(1481972K) icms_dc=9 , 0.0213890 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
542.916: [GC 542.916: [ParNew: 38073K->3946K(38336K), 0.0216000 secs] 1117819K->1084313K(1481972K) icms_dc=9 , 0.0217140 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
543.134: [GC 543.134: [ParNew: 38058K->3979K(38336K), 0.0211750 secs] 1118425K->1084996K(1481972K) icms_dc=9 , 0.0212920 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
543.346: [GC 543.346: [ParNew: 38091K->3971K(38336K), 0.0212490 secs] 1119108K->1085641K(1481972K) icms_dc=9 , 0.0213580 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
2012-05-29 10:49:04,535  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread2] Reindexer    12 failed with error: 04290011 Failed to flush index.
2012-05-29 10:49:04,536  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread2] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290011 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/2d53acc7-87ec-4767-8ad6-af23018bae9c/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

2012-05-29 10:49:04,537  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread3] Reindexer    13 failed with error: 04290005 Failed to flush index.
2012-05-29 10:49:04,538  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread3] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290005 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/d517c4c1-2f7b-4cd4-bd7e-519a1f773331/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

2012-05-29 10:49:04,539  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread4] Reindexer    14 failed with error: 04290007 Failed to flush index.
2012-05-29 10:49:04,540  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread4] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290007 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/3fef9731-93c3-49a6-a81b-712b56ab2559/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

2012-05-29 10:49:04,540  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread5] Reindexer    15 failed with error: 04290009 Failed to flush index.
2012-05-29 10:49:04,541  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread5] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290009 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/52e200d0-67e7-48c6-97a6-78d55a563e63/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

2012-05-29 10:49:04,542  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread1] Reindexer    16 failed with error: 04290010 Failed to flush index.
2012-05-29 10:49:04,543  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread1] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290010 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/a7514926-3562-455a-9b1f-86b4bfa7d843/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 23 more

543.823: [GC 543.823: [ParNew: 38083K->4184K(38336K), 0.0211870 secs] 1119753K->1086488K(1481972K) icms_dc=9 , 0.0213230 secs] [Times: user=0.04 sys=0.00, real=0.03 secs]
544.088: [GC 544.089: [ParNew: 38296K->4224K(38336K), 0.0392680 secs] 1120600K->1089383K(1481972K) icms_dc=9 , 0.0393770 secs] [Times: user=0.06 sys=0.01, real=0.04 secs]
544.458: [GC 544.458: [ParNew: 38336K->3904K(38336K), 0.0277950 secs] 1123495K->1090060K(1481972K) icms_dc=9 , 0.0279200 secs] [Times: user=0.04 sys=0.01, real=0.03 secs]
544.712: [GC 544.712: [ParNew: 38016K->3437K(38336K), 0.0207070 secs] 1124172K->1089995K(1481972K) icms_dc=9 , 0.0208310 secs] [Times: user=0.03 sys=0.00, real=0.02 secs]
545.086: [GC 545.086: [ParNew: 37549K->3327K(38336K), 0.0189540 secs] 1124107K->1089944K(1481972K) icms_dc=9 , 0.0191630 secs] [Times: user=0.02 sys=0.00, real=0.02 secs]
2012-05-29 10:49:06,414  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread1] Reindexer    41 failed with error: 04290012 Failed to flush index.
2012-05-29 10:49:06,415  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread1] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290012 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.GeneratedMethodAccessor314.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/5321aa32-6885-4bec-af7e-06eccfe60551/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 22 more

545.580: [GC 545.581: [ParNew: 37439K->4224K(38336K), 0.0519850 secs] 1124056K->1094639K(1481972K) icms_dc=9 , 0.0521030 secs] [Times: user=0.07 sys=0.00, real=0.05 secs]
545.841: [GC 545.841: [ParNew: 38336K->4224K(38336K), 0.0249280 secs] 1128751K->1095804K(1481972K) icms_dc=9 , 0.0250420 secs] [Times: user=0.03 sys=0.00, real=0.03 secs]
545.980: [GC 545.980: [ParNew: 38336K->4188K(38336K), 0.0184720 secs] 1129916K->1095768K(1481972K) icms_dc=9 , 0.0185730 secs] [Times: user=0.04 sys=0.00, real=0.02 secs]
2012-05-29 10:49:07,106  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread4] Reindexer    43 failed with error: 04290013 Failed to flush index.
2012-05-29 10:49:07,107  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread4] org.alfresco.repo.search.impl.lucene.LuceneIndexException: 04290013 Failed to flush index
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1098)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.GeneratedMethodAccessor314.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy211.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/workspace/SpacesStore/11ba62b1-66f4-46b5-99de-3fe303f99baa/segments_1 (Too many open files)
   at java.io.RandomAccessFile.open(Native Method)
   at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
   at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:682)
   at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:443)
   at org.apache.lucene.index.SegmentInfos.write(SegmentInfos.java:296)
   at org.apache.lucene.index.SegmentInfos.prepareCommit(SegmentInfos.java:761)
   at org.apache.lucene.index.SegmentInfos.commit(SegmentInfos.java:825)
   at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1086)
   at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:626)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.makeDeltaIndexWriter(IndexInfo.java:1079)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeltaIndexWriter(IndexInfo.java:1122)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getDeltaWriter(AbstractLuceneBase.java:225)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1079)
   … 22 more



with the follwoing settings

start_tomcat() {
    is_tomcat_running
    RUNNING=$?
    if [ $RUNNING -eq 1 ]; then
        echo "$0 $ARG: tomcat (pid $PID) already running"
    else
   rm -f $CATALINA_PID
   export JAVA_OPTS="-server -Xms1G -Xmx2G -XX:MaxPermSize=512M -Xss1024k -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -Dalfresco.home=/opt/indexing -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true"
   export JAVA_HOME=$JRE_HOME
   $TOMCAT_BINDIR/startup.sh
   if [ $? -eq 0 ];  then
            echo "$0 $ARG: tomcat started"
   else
            echo "$0 $ARG: tomcat could not be started"
            ERROR=1
   fi
    fi
}

daemon_tomcat() {
    #export JAVA_OPTS="-XX:MaxPermSize=1024m -Xms512m -Xmx1024m -XX:-UseGCOverheadLimit -XX:+UseConcMarkSweepGC -XX:-DisableExplicitGC -Dalfresco.home=/opt/indexing -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true"
    export JAVA_OPTS="-server -Xms1G -Xmx2G -XX:MaxPermSize=512M -Xss1024k -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -Dalfresco.home=/opt/indexing -Dcom.sun.management.jmxremote -Dsun.security.ssl.allowUnsafeRenegotiation=true"
    export JAVA_HOME=$JRE_HOME
    $TOMCAT_BINDIR/catalina.sh run
}

dynamolalit
Champ on-the-rise
Champ on-the-rise
Hi,

I can see below error

Caused by: java.io.FileNotFoundException: /opt/indexing/alf_data/lucene-indexes/archive/SpacesStore/b57d1fc0-fc9c-4f70-bea5-eb6f95f15d0b/segments_1 (Too many open files)

I think its related to Too many open files. Please refer to http://wiki.alfresco.com/wiki/Too_many_open_files for resolution.

Finally ensure you have all necessary RAM at exposure to alfresco say fair percentage of overall machine RAM and try to fine tune tomcat as well as db also.

Regards.

dynamolalit
Champ on-the-rise
Champ on-the-rise
Hi All,

After a long period of implementations & observations, i can pin point on following items to get this (OutOfMemory) kind of errors resolved/minimized. Rule of thumb would be to have enough RAM & processing power for your alfresco as well as DB fine tuning.

1. Always use 64 bit machines with Quad Core CPUs to get maximum throughput and expose a good amount of RAM to alfresco installations.
2. Use alfresco 4.0 onwards, if you can use, which basically has a lot of improvements in terms of performance, indexing etc.
3. Optimize DB for best performance especially the number of open connections, connection closing delays etc & vendor specific optimizations.
### database connection properties ###
db.driver=org.gjt.mm.mysql.Driver
db.username=admin
db.password=admin
db.name=alfresco
db.url=jdbc:mysql://localhost:3306/alfresco
hibernate.dialect=org.hibernate.dialect.MySQLInnoDBDialect
db.pool.min=20
db.pool.max=275
db.pool.validate.return=true
db.pool.validate.query=select 1
db.pool.evict.idle.min=550000
db.pool.evict.validate=true
db.pool.abandoned.detect=true
db.pool.abandoned.time=550
4. Optimize JVM for best results like as below. It has following main points.
     Use Concurrent sweep algorithm.
     Use -XX:NewSize for Young Gen.
     User -XX:CMSInitiatingOccupancyFraction for major GC.
JAVA_OPTS="-server -Xms2G -Xmx4G -XX:MaxPermSize=256M -XX:NewSize=1G -Xss1M -verbose:gc -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70"
5. Optimize lucene, if you are using it, to get most of it as it can be a culprit on a number of occasions using configurations such as
lucene.indexer.maxDocsForInMemoryIndex=1000
lucene.indexer.maxDocsForInMemoryMerge=1000
6. I have not used SOLR but you can try out for getting best performance.
7. Clean alfresco temporary (<alf_home>\tomcat\temp) folders periodically to remove lot of temporary files such as related to transformations etc to avoid them taking a lot of disk space which can degrade performance.
8. Finally, monitor performance using tools like Jvisualvm, jmx reader etc from time to time.

Hope this helps.

Regards.