topic Re: [solved]The more search queries the slower lucene gets in Alfresco Archive

[solved]The more search queries the slower lucene gets

sebp — Tue, 15 Dec 2009 13:47:13 GMT

I want to search my repository for duplicate documents. Therefore, in a first step, I go through it recursively and give each content node a hash sum property. In a second step, to know which document has how many duplicates I have to go through the repository again. For each node it is checked if t

Re: [solved]The more search queries the slower lucene gets

rogier_oudshoor — Thu, 17 Dec 2009 06:27:06 GMT

In Alfresco, most processing is done inside of a transaction. What you're doing, is add more and more information inside a single transaction - increasing the transaction overheid. You can try to break down your batch into multiple transactions and see if it speeds up.

Re: [solved]The more search queries the slower lucene gets

loftux — Thu, 17 Dec 2009 07:22:17 GMT

Side note: Be aware of how Alfresco makes copies. When you copy nodeA pointing to contentA to nodeB, the new nodeB will still point to contentA. That is until either nodeA or nodeB is versioned, at that time each content node will point to a unique content.
So if you hash sum is on file content, you may get false positives. you can probably take this into account checking versionlabel and/or checking if its has the copiedfrom aspect.

Re: [solved]The more search queries the slower lucene gets

sebp — Thu, 17 Dec 2009 15:01:03 GMT

Thanks for your replies. I think the problem was that I let the administrator start the "Update duplicates" action from within the web client ui. So there automatically was a surrounding transaction that was growing bigger and bigger. Now I update the duplicates using a CronScheduledQueryBasedTemplateActionDefinition. If it runs in ISOLATED_TRANSACTIONS mode everything is fine.

Re: [solved]The more search queries the slower lucene gets

ganeshkolhe — Wed, 23 Dec 2009 05:42:46 GMT

Hi sebp,
Can you please share your implementation with us . How you used CronScheduledQueryBasedTemplateActionDefinition and how you have set ISOLATED_TRANSACTIONS mode?

Regards,
Ganesh

Re: [solved]The more search queries the slower lucene gets

sebp — Wed, 23 Dec 2009 08:44:58 GMT

Hi Ganesh,
Have a look at the duplicate finders' manual page http://www.hmedia.de/wiki/doku.php?id=products:duplicates
There you will find the scheduled-action-services-context.xml file where the action is started as a CronScheduledQueryBasedTemplateActionDefinition. You can find more details about scheduled actions and the available transaction modes at http://wiki.alfresco.com/wiki/Scheduled_Actions
The implementation can be found at http://forge.alfresco.com/projects/duplicates/