topic Re: 32000 lucene index folder in Alfresco Archive

32000 lucene index folder

aktif — Wed, 24 Feb 2016 10:10:56 GMT

Hello,I am using alfresco community 4.0.d on redhat linux with tomcat and mysql. I have 3.3TB data on production environment. When i reach 32000 folders limit in lucenes-indexes/workspace/SpacesStore folder, I restart the tomcat and merge process starts while alfresco restarting. 32000 folders decre

Re: 32000 lucene index folder

afaust — Thu, 25 Feb 2016 08:42:13 GMT

Hello,

the index merge job is not only active during startup but active all the time. The main problem is that it needs to be able to merge, which isn't the case when the index files are "apparently" still in use. There is a programmer error that can cause the Lucene index to swell up like this based on not cleaning up search results properly. This keeps index files marked as "in use" which causes Alfresco to create copies whenever new nodes are indexed. The longer the system runs and the longer those result sets aren't cleared by application code, the more of those folders / copies accumulate. When you restart the system, those "in use" marks are all gone and Alfresco is able to remove all those redundant copies.

Since Lucene is not used in current Alfresco versions (4.0 and up have SOLR, 5.0 removed Lucene completely), there isn't much current documentation on that matter. But <a href="https://alfrescoshare.wordpress.com/2009/11/19/coding-best-practice-lucene-search-query-resultset-close/">an older blog posts</a> highlights the relevant pattern of closing result sets.

Regards
Axel

Re: 32000 lucene index folder

aktif — Fri, 11 Mar 2016 08:59:26 GMT

Thank you for your answer Axel. I find several old alfresco issues about this topic after your post.
I think this problem doesn't fixed in 4.0.d version.
Apperantly there are lots of result sets in alfresco source codes which is not closed properly. This leads to 32000 lucene index directories!

Are there any class/method name list for 4.0.d version to patch?
Or should I review the source codes on my own?

https://alfrescoshare.wordpress.com/2009/11/27/coding-best-practice-lucene-search-query-resultset-close-part2/
https://issues.alfresco.com/jira/browse/ALF-1588
https://issues.alfresco.com/jira/browse/ETHREEOH-3229
https://issues.alfresco.com/jira/browse/MNT-1500
https://issues.alfresco.com/jira/browse/ALFCOM-3683

Re: 32000 lucene index folder

afaust — Fri, 11 Mar 2016 09:34:31 GMT

It might be wise to upgrade to Alfresco 4.2.f (the last version which still has Lucene) before you spent your time looking through code to fix things yourself. While SOLR has been the preferred search sub system since 4.0, Lucene issues still received fixes up to 4.2, so you may be chasing issues that have already been addressed.

Regards
Axel