Lucene re-index big problem

[javixulo]

Hello,

I'm facing a big problem when trying to re-index a large amount of documents in Alfresco 4.0.

In my Alfresco I've got around 4 million documents, and it always fail with out of memory exception. These are my java_opts:

set JAVA_OPTS=%JAVA_OPTS% -Xms1024m -Xmx3072m -XX:MaxPermSize=512m -Dfile.encoding=UTF-8 -Djava.file.encoding=UTF-8‍

I understand that memory is quite important when re-indexing such a big number of documents, but I believe that 3GB is quite a lot…

Can anybody tell me the relation between number of documents / memory needed?

There is any configuration that can improve the performance (less memory usage) of the re-index operation?

Is it possible to re-index in blocks?

I need to change sth in the configuration so the re-index works :roll:. I would appretiate any help.

Thanks in advance

[abarisone]

Hi,
first of all here you can find the reference http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html

Xms == Xmx will let the JVM free to recalculate the memory in order to save resources.

If the out of memory exception seems due to PermGen space then increase it as much as you can.
If it comes from something like OutOfMemoryError: unable to create new native thread (too many threads, each thread has a large stack), decrease the -Xss parameter (I set 256k).

Regards,
Andrea

[loftux]

You can also have a look at the configuration options here
http://wiki.alfresco.com/wiki/Full-Text_Search_Configuration

[javixulo]

Hi,

First of all, thank u all 

I got outofmemory, but heap memory, not permgen.

I dive into all this parameteres but I'm not very good selecting them, and it's a mess because everytime I try a new configuration I must wait about 2h to get the results… Can u give me an example of a good configuration in order to solve this problem?

And again, thank u all