topic Re: OutOfmemory exception when processing a large number of documents in Nuxeo Forum

OutOfmemory exception when processing a large number of documents

Ahmad_BenMaallem — Thu, 15 Apr 2021 10:10:38 GMT

Hello,

I want to process a large number of documents (around 3 million). The process consists of updating a metadata of each document from another metadata (so I will need to do a read and write from the database).

For that, I was thinking of using elasticsearch's SCROLL API.

The problem is that I will have a "java.lang.OutOfMemoryError: GC overhead limit exceeded" exception in the middle of processing (knowing that I have xmx = xms = 24g in JAVA_OPTS)

I have tried different configuration for the garbage collector but no great effect noticed.

Can someone help me or give me an idea how to process a large batch of documents in nuxeo.

Thank you in advance.

Re: OutOfmemory exception when processing a large number of documents

Ahmad_BenMaallem — Fri, 16 Apr 2021 08:13:13 GMT

Hello,

I have used the bulkAction mechanism in nuxeo and it looks good.

Thank you

Re: OutOfmemory exception when processing a large number of documents

sujoy_debnath — Wed, 21 Apr 2021 05:05:24 GMT

Can you help me how to use bulk upload with custom metadata?

Re: OutOfmemory exception when processing a large number of documents

Ahmad_BenMaallem — Wed, 21 Apr 2021 09:38:53 GMT

Hello,