topic Bad Lucene index distribution among segments in Alfresco Archive

Bad Lucene index distribution among segments

rivarola — Tue, 13 Oct 2015 15:17:07 GMT

Hello,On a server with a bit more than 1 million documents, I have noticed this bad Lucene index segments distribution:09:00:00,440 DEBUG [org.alfresco.repo.search.impl.lucene.index.IndexInfo] Entry List1 Name=cb6b6534-1162-403c-acea-59b5d9c55dba Type=INDEX Status=COMMITTED Docs=2445933 Dele

Re: Bad Lucene index distribution among segments

afaust — Wed, 14 Oct 2015 08:21:16 GMT

Hello,

what precisely do you consider bad with regards to this distribution? Your distribution actually looks ok to me - not perfect but a realistic result of normal operations.

In your case, the only thing that seems off to me is the relation between index segment #1 and #2. Optimally, the difference should be within an order of magnitude, not several or double digit. But this might be the result of a recent round of merging and segment #2 is only now starting to fill up again as new documents / changes come in.
Otherwise the progression is quite decent - you might even make do with fewer segments to improve search performance since you (currently) have very few documents in the three highest numbere segments.

I am not aware of any method to optimize the segment structure apart from performing a full reindex. The only way I can think of involves writing a low-level Lucene tool to basically create a new index by piping the contents from the old into it, resulting in an implicit optimisation by "writing anew", without any actual reindexing that requires access to Alfresco DB / content.

Regards
Axel

Re: Bad Lucene index distribution among segments

rivarola — Wed, 14 Oct 2015 13:46:03 GMT

Hi Alex,
Two things seem strange to me :
- I asked for 10 segments and there are only 8
- the first one is far too big: 200 times the second. Usually the biggest is only 3 times the second.

Philippe

Re: Bad Lucene index distribution among segments

mrogers — Wed, 14 Oct 2015 16:17:19 GMT

You aren't looking for an even spread of documents to segments. Lucene does not use segments to partition the data in a way that say a hash bucket would, it simply creates a segment as and when it needs to persist its in memory state. And a background process to merge together old segments and remove deleted documents from the new segment.

Re: Bad Lucene index distribution among segments

afaust — Thu, 15 Oct 2015 08:09:09 GMT

Hello Philippe,

you have set a "target factor" of 10 which Lucene uses as input in its optimization / distribution but it is a "target", not a guarantee. Much like the MERGE index segments are created / deleted as needed and almost never are exactly the amount specified by the merger target factor, the normal index also goes through creation / deletion cycles. The 200-times larger first segment may very well be the result of Lucene merging the segments previously at positions 1 through 3 into one large segment at position 1, resulting in temporarily 2 fewer segments than the target factor until Lucene re-creates those when needed. With those previous segments, the progression might have been less radical and more in line with your expectations.

Regards
Axel