Lucene error : Too Many Open File ??

[#1]

Hello,

We are working with Alfresco 1.4. And I think that our index is corrupted because I can see a lot of error in alfresco log file and we can't add more file since see a dump error.

Error log :

00:03:43,186 ERROR [org.alfresco.repo.search.impl.lucene.LuceneBase2] Errorjava.io.FileNotFoundException: ./alf_data/lucene-indexes/workspace/SpacesStore/0e3909ea-79a7-11db-946d-3bededff6578/IndexInfoDeletions (Too many open files)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:106)        at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeletions(IndexInfo.java:648)
        at org.alfresco.repo.search.impl.lucene.index.IndexInfo.createMainIndexReader(IndexInfo.java:1377)
        at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getMainIndexReferenceCountingReadOnlyIndexReader(IndexInfo.java:748)
        at org.alfresco.repo.search.impl.lucene.LuceneBase2.getSearcher(LuceneBase2.java:156)
        at org.alfresco.repo.search.impl.lucene.LuceneIndexerImpl2.updateFullTextSearch(LuceneIndexerImpl2.java:1772)
        at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:172)
        at sun.reflect.GeneratedMethodAccessor503.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:335)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)
        at $Proxy4.index(Unknown Source)
        at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:36)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:191)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:516)


00:03:43,193 ERROR [org.quartz.core.JobRunShell] Job DEFAULT.ftsIndexerJobDetail threw an unhandled Exception:
org.alfresco.repo.search.impl.lucene.LuceneIndexException: Failed FTS update
        at org.alfresco.repo.search.impl.lucene.LuceneIndexerImpl2.updateFullTextSearch(LuceneIndexerImpl2.java:1897)
        at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:172)
        at sun.reflect.GeneratedMethodAccessor503.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:335)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)
        at $Proxy4.index(Unknown Source)
        at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:36)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:191)        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:516)Caused by: org.alfresco.repo.search.impl.lucene.LuceneIndexException: Failed to open IndexSarcher for ./alf_data/lucene-indexes/workspace/SpacesStore/
        at org.alfresco.repo.search.impl.lucene.LuceneBase2.getSearcher(LuceneBase2.java:172)
        at org.alfresco.repo.search.impl.lucene.LuceneIndexerImpl2.updateFullTextSearch(LuceneIndexerImpl2.java:1772)
        … 14 moreCaused by: java.io.FileNotFoundException: ./alf_data/lucene-indexes/workspace/SpacesStore/0e3909ea-79a7-11db-946d-3bededff6578/IndexInfoDeletions (Too many open files)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:866)
        at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
        at java.lang.Thread.run(Thread.java:595)


00:03:43,186 ERROR [org.alfresco.repo.search.impl.lucene.LuceneBase2] Error
java.io.FileNotFoundException: ./alf_data/lucene-indexes/workspace/SpacesStore/0e3909ea-79a7-11db-946d-3bededff6578/IndexInfoDeletions (Too many open files)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:106)
        at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getDeletions(IndexInfo.java:648)
        at org.alfresco.repo.search.impl.lucene.index.IndexInfo.createMainIndexReader(IndexInfo.java:1377)
        at org.alfresco.repo.search.impl.lucene.index.IndexInfo.getMainIndexReferenceCountingReadOnlyIndexReader(IndexInfo.java:748)
        at org.alfresco.repo.search.impl.lucene.LuceneBase2.getSearcher(LuceneBase2.java:156)
        at org.alfresco.repo.search.impl.lucene.LuceneIndexerImpl2.updateFullTextSearch(LuceneIndexerImpl2.java:1772)
        at org.alfresco.repo.search.impl.lucene.fts.FullTextSearchIndexerImpl.index(FullTextSearchIndexerImpl.java:172)
        at sun.reflect.GeneratedMethodAccessor503.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:335)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)
        at $Proxy4.index(Unknown Source)
        at org.alfresco.repo.search.impl.lucene.fts.FTSIndexerJob.execute(FTSIndexerJob.java:36)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:191)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:516)

We tried to increase the number of files that can be opened simultaneously : ulimit -n 65535

But It's still the same.

If I check the number of file open by lucene, this is what I see :

[root@fidji alfresco]# lsof | grep luc | wc -l
845

With this command, I can see that lucene seams working on deleted file.

[root@fidji alfresco]# lsof | grep luc
java      3930    root  807r      REG        9,0     21298   18677931 /data/alfresco/alf_data/lucene-indexes/workspace/lightWeightVersionStore/d9838a8f-79a6-11db-946d-3bededff6578/_m.cfs (deleted)
java      3930    root  808r      REG        9,0     18657   18678420 /data/alfresco/alf_data/lucene-indexes/workspace/lightWeightVersionStore/f7a4dc0a-79a6-11db-946d-3bededff6578/_j.cfs (deleted)
java      3930    root  809r      REG        9,0     22366   18678493 /data/alfresco/alf_data/lucene-indexes/workspace/lightWeightVersionStore/0f1efd2e-79a7-11db-946d-3bededff6578/_n.cfs (deleted)
java      3930    root  810r      REG        9,0     21296   18564073 /data/alfresco/alf_data/lucene-indexes/workspace/lightWeightVersionStore/286aab1d-79a7-11db-946d-3bededff6578/_m.cfs (deleted)
java      3930    root  811r      REG        9,0      5789   18678520 /data/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/a3bbbacb-79a7-11db-946d-3bededff6578/_4.cfs
java      3930    root  812r      REG        9,0     22409   18678521 /data/alfresco/alf_data/lucene-indexes/workspace/lightWeightVersionStore/34e16e87-79a7-11db-946d-3bededff6578/_n.cfs
java      3930    root  813r      REG        9,0      4019   18678523 /data/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/a4082d21-79a7-11db-946d-3bededff6578/_2.cfs
java      3930    root  814r      REG        9,0     18656   18678536 /data/alfresco/alf_data/lucene-indexes/workspace/lightWeightVersionStore/a4d8eaac-79a7-11db-946d-3bededff6578/_j.cfs
java      3930    root  815r      REG        9,0     23181   18678553 /data/alfresco/alf_data/lucene-indexes/workspace/lightWeightVersionStore/11c3e8ff-79a8-11db-946d-3bededff6578/_o.cfs

What can I do ?
Is it possible to recreate the index ?

Thanks
Regards,
Chris

[derek]

http://forums.alfresco.com/viewtopic.php?p=19038

[arautenberg]

Thx…..it works 😉