cancel
Showing results for 
Search instead for 
Did you mean: 

Index Tip Recovery For Large Repositories

callermd
Champ in-the-making
Champ in-the-making
Hi All,

After bashing my head against a wall trying to figure out how to recover our indexes with 300 000 transaction and having it take well over 24 hours I have put together some helpful tips to help any poor souls in a similar situation.  This was done against 2.1 CE.

node.index.FullIndexRecoveryComponent

- Firstly remember to blow away the existing
lucene-indexes
directory AFTER shutting down tomcat.

- Set
index.recovery.mode=FULL

- One of the most frustrating things is that it is really hard to figure out if something is actually happening.  To make your life easier edit your log4j.properties (under FC with WAR install in /var/lib/tomcat5/webapps/alfresco/WEB-INF/classes/log4j.properties) and set 
log4j.logger.org.alfresco.repo.node.index.FullIndexRecoveryComponent=debug

You should now get
Reindexing transaction: xxxxx
messages which will help you figure out what is going on

- I changed
lucene.indexer.batchSize=1000
from its really high value.  This breaks the index recovery into chunks and has the happy effect of flushing to disk the lucene indexes to the lucene-indexes directory.  This flushing makes the log percentage indicator more accurate and seems to prevent nasty risks like the disk running out of space or the JVM running out of heap from being oh so heart breaking

- As per http://www.onjava.com/pub/a/onjava/2003/03/05/lucene.html benchmarks and recommendation for batch benchmarking I jacked the
lucene.indexer.mergeFactor=1000
from 10 .  However I did not benchmark this change in any rigorous fashion.

- After it is done you should set the batch size back as there is an open issue AR-1280 (documented in the file) about something bad happening if you don't

- Also remember to set the
index.recovery.mode=AUTO
instead of FULL.

Cheers!
6 REPLIES 6

andy
Champ on-the-rise
Champ on-the-rise
Hi

I have rasied an enhancement request to merge many small transactions together during index recovery.

The merge factor is not likely to make much difference for small transactions.

Moving to lucene 2.3 is likely to give significantly better performance.

Your perfromance seems very slow. What hardware are you using?
Where is the alf data stored? What DB? Have you configured to do most content transforms atomically - or only "short" ones (the default). Roughly how many files/folders per transaction?

Andy

simon
Champ in-the-making
Champ in-the-making
Hi callermd,

This is an extremely valuable post, thanks! We recently rebuild our indexes (139,071 transactions) and that takes 10 hours on our server so next time we'll try your solution, 10 hours downtime for something like this is not really acceptable. It would be nice if Alfresco could tune this for the following release, hint, hint.

Can you tell us what the performance gain was after your tweaks? How much time did you save after this?

andy
Champ on-the-rise
Champ on-the-rise
Hi

The index rebuild is now multi-threaded.

Andy

callermd
Champ in-the-making
Champ in-the-making
Hi

The index rebuild is now multi-threaded.

Andy

What version will have this functionality?

Thanks!

pmonks
Star Contributor
Star Contributor
Putting a backup process in place (as described at http://wiki.alfresco.com/wiki/Backup_and_Restore) can help avoid these problems.  See http://wiki.alfresco.com/wiki/Backup_and_Restore#Lucene_Index_Restoration specifically.

Cheers,
Peter

oznevo
Champ in-the-making
Champ in-the-making
You should probably add to that:
http://forums.alfresco.com/en/viewtopic.php?f=4&t=15401
We are running on 5-10m transactions…

Oz