Too many open files & Failed to merge deletions exceptio

[shay]

Hi,

My alfresco start to throw exceptions and it force me to
restart the alfrsco, it has a several problems:
1. Too many open files
2. Failed to merge deletions

i'm working on alfresco 2.1 with linux

anyone got an idea about what could cause this problem?
Thanks in advance,
Shay.

2008-01-30 21:09:47,409 ERROR [Index merger thread /var/opt/mnt/alfresco/host-specific/mg1/indexes/archive/SpacesStore]: org.alfresco.repo.search.impl.lucene.index.IndexInfo - Failed to merge indexes
java.io.IOException: Cannot read directory /var/opt/mnt/alfresco/host-specific/mg1/indexes/archive/SpacesStore/4e279b25-cf6f-11dc-ba4c-b9ff80d66a94
                at org.apache.lucene.store.FSDirectory.create(FSDirectory.java:242)
                at org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:232)
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo$Merger.mergeIndexes(IndexInfo.java:2943)
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo$Merger.run(IndexInfo.java:2448)
                at java.lang.Thread.run(Thread.java:595)
2008-01-30 21:09:47,409 ERROR [Index merger thread /var/opt/mnt/alfresco/host-specific/mg1/indexes/workspace/SpacesStore]: org.alfresco.repo.search.impl.lucene.index.IndexInfo - Failed to merge deletions
java.io.FileNotFoundException: /var/opt/mnt/alfresco/host-specific/mg1/indexes/workspace/SpacesStore/48718236-cf6f-11dc-ba4c-b9ff80d66a94/IndexInfoDeletions (Too many open files)
                at java.io.FileInputStream.open(Native Method)
                at java.io.FileInputStream.<init>(FileInputStream.java:106)
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeletions(IndexInfo.java:842)
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo$Merger.mergeDeletions(IndexInfo.java:2585)
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo$Merger.run(IndexInfo.java:2444)
                at java.lang.Thread.run(Thread.java:595)
2008-01-30 21:11:28,556 ERROR [http-60080-Processor19]: org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/alfresco].[Faces Servlet] - Servlet.service() for servlet Faces Servlet threw exception
org.alfresco.repo.search.IndexerException: Invalid state MARKED_ROLLBACK
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getTransition(IndexInfo.java:1171)
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo.setStatus(IndexInfo.java:1099)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.setStatus(AbstractLuceneBase.java:256)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.setRollbackOnly(AbstractLuceneIndexerImpl.java:549)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.prepare(AbstractLuceneIndexerImpl.java:484)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.prepare(AbstractLuceneIndexerAndSearcherFactory.java:679)
                at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:634)
                at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:48)
                at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:821)
                at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:637)
                at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:624)
                at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:307)
                at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:117)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)
                at $Proxy2.setProperty(Unknown Source)
                at sun.reflect.GeneratedMethodAccessor45158.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at org.apache.myfaces.el.MethodBindingImpl.invoke(MethodBindingImpl.java:132)
                at javax.faces.component.UICommand.broadcast(UICommand.java:89)
                at javax.faces.component.UIViewRoot._broadcastForPhase(UIViewRoot.java:97)
                at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:171)
                at org.apache.myfaces.lifecycle.InvokeApplicationExecutor.execute(InvokeApplicationExecutor.java:32)
                at org.apache.myfaces.lifecycle.LifecycleImpl.executePhase(LifecycleImpl.java:95)
                at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:70)
                at javax.faces.webapp.FacesServlet.service(FacesServlet.java:139)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
                at org.alfresco.web.app.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:81)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
                at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
                at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
                at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
                at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
                at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
                at java.lang.Thread.run(Thread.java:595)
2008-01-30 21:11:31,050 ERROR [Sess_FTP1317_207.218.157.50]: org.alfresco.repo.search.impl.lucene.index.IndexInfo - Failed building filter reader beneath 3be86eb4-cf6f-11dc-ba4c-b9ff80d66a94
java.io.FileNotFoundException: /var/opt/mnt/alfresco/host-specific/mg1/indexes/workspace/SpacesStore/3be86eb4-cf6f-11dc-ba4c-b9ff80d66a94/IndexInfoDeletions (Too many open files)
                at java.io.FileInputStream.open(Native Method)
                at java.io.FileInputStream.<init>(FileInputStream.java:106)
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeletions(IndexInfo.java:842)
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo.createMainIndexReader(IndexInfo.java:1698)
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getMainIndexReferenceCountingReadOnlyIndexReader(IndexInfo.java:969)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.getReader(AbstractLuceneBase.java:271)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.flushPending(AbstractLuceneIndexerImpl.java:745)
                at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.doPrepare(ADMLuceneIndexerImpl.java:1337)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.prepare(AbstractLuceneIndexerImpl.java:471)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.prepare(AbstractLuceneIndexerAndSearcherFactory.java:679)
                at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:634)
                at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:48)
                at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:821)
                at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:637)
                at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:624)
                at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:307)
                at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:420)
                at org.alfresco.filesys.server.SrvSession.endTransaction(SrvSession.java:627)
                at org.alfresco.filesys.ftp.FTPSrvSession.procStoreFile(FTPSrvSession.java:2350)
                at org.alfresco.filesys.ftp.FTPSrvSession.run(FTPSrvSession.java:4235)
                at java.lang.Thread.run(Thread.java:595)
2008-01-30 21:11:34,815 ERROR [http-60080-Processor21]: org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/alfresco].[Faces Servlet] - Servlet.service() for servlet Faces Servlet threw exception
java.lang.NullPointerException
                at org.apache.lucene.index.IndexFileDeleter.findDeletableFiles(IndexFileDeleter.java:88)
                at org.apache.lucene.index.IndexWriter.mergeSegments(IndexWriter.java:1570)
                at org.apache.lucene.index.IndexWriter.flushRamSegments(IndexWriter.java:1351)
                at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:588)
                at org.alfresco.repo.search.impl.lucene.index.IndexInfo.closeDeltaIndexWriter(IndexInfo.java:796)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.closeDeltaWriter(AbstractLuceneBase.java:231)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.flushPending(AbstractLuceneIndexerImpl.java:821)
                at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.doPrepare(ADMLuceneIndexerImpl.java:1337)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.prepare(AbstractLuceneIndexerImpl.java:471)
                at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.prepare(AbstractLuceneIndexerAndSearcherFactory.java:679)
                at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:634)
                at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:48)
                at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:821)
                at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:637)
                at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:624)
                at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:307)
                at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:117)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:210)
                at $Proxy2.setProperty(Unknown Source)
                at sun.reflect.GeneratedMethodAccessor45158.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at org.apache.myfaces.el.MethodBindingImpl.invoke(MethodBindingImpl.java:132)
                at javax.faces.component.UICommand.broadcast(UICommand.java:89)
                at javax.faces.component.UIViewRoot._broadcastForPhase(UIViewRoot.java:97)
                at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:171)
                at org.apache.myfaces.lifecycle.InvokeApplicationExecutor.execute(InvokeApplicationExecutor.java:32)
                at org.apache.myfaces.lifecycle.LifecycleImpl.executePhase(LifecycleImpl.java:95)
                at org.apache.myfaces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:70)
                at javax.faces.webapp.FacesServlet.service(FacesServlet.java:139)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
                at org.alfresco.web.app.servlet.AuthenticationFilter.doFilter(AuthenticationFilter.java:81)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
                at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
                at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
                at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
                at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
                at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
                at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
                at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
                at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
                at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
                at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
                at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
                at java.lang.Thread.run(Thread.java:595)

[andy]

Hi

See http://wiki.alfresco.com/wiki/Search#File_Handles_and_lucene

Check your file handle use.

Have you got any custom code?

Andy

[captnbob]

I experienced a similar problem when trying to upload a file. I closed the browser and re-opened it and it succeeded.
I was using Firefox 3.5.8.

[savic_prvoslav]

maybe that you have many files opened try
ulimit -n number ( like  2000 or so)