topic Large Lucene index size prevents cron job from running? in Alfresco Archive https://connect.hyland.com/t5/alfresco-archive/large-lucene-index-size-prevents-cron-job-from-running/m-p/278127#M231257 <HTML><HEAD></HEAD><BODY><SPAN>Hello,</SPAN><BR /><BR /><SPAN>I am currently running into performance problems with a custom bulk import process. Hopefully someone can help me find the bottleneck.</SPAN><BR /><BR /><SPAN>First some background information:</SPAN><BR /><SPAN>We planned to import a large amount of documents into Alfresco 3.4.7 EE, in controlled and monitored batches.</SPAN><BR /><BR /><SPAN>The steps involved are the following:</SPAN><BR /><SPAN>1. Batch bulk import via Peter Monks bulk import tool (in-memory, no streaming of binary data needed)</SPAN><BR /><SPAN>2. Creation of missing Lucene indexes for new content</SPAN><BR /><SPAN>3. Run of a custom scheduled job that moves the new document nodes and adds meta data to it from an external database.</SPAN><BR /><BR /><BR /><SPAN>Additional information about each step:</SPAN><BR /><BR /><SPAN>#1: Works flawlessly, the tool runs stable and fast. </SPAN><BR /><SPAN>The import job registers binary content only, no specific meta-data is being added via xml files. </SPAN><BR /><SPAN>The content to be processed is physically stored in a custom content store folder inside a the Alfresco default content store, with the following structure:</SPAN><BR /><SPAN>{CS BASE FOLDER}/{UNIQUE DOSSIER FOLDER ID 1}/{UNIQUE DOC FOLDER 1}/{FILE 1}</SPAN><BR /><SPAN>{CS BASE FOLDER}/{UNIQUE DOSSIER FOLDER ID 1}/{UNIQUE DOC FOLDER 2}/{FILE 2}</SPAN><BR /><SPAN>{CS BASE FOLDER}/{UNIQUE DOSSIER FOLDER ID 1}/{UNIQUE DOC FOLDER 3}/{FILE 3}</SPAN><BR /><SPAN>{CS BASE FOLDER}/{UNIQUE DOSSIER FOLDER ID 2}/{UNIQUE DOC FOLDER 4}/{FILE 4}</SPAN><BR /><SPAN>{CS BASE FOLDER}/{UNIQUE DOSSIER FOLDER ID 2}/{UNIQUE DOC FOLDER 5}/{FILE 5}</SPAN><BR /><SPAN>{CS BASE FOLDER}/{UNIQUE DOSSIER FOLDER ID 3}/{UNIQUE DOC FOLDER 6}/{FILE 6}</SPAN><BR /><SPAN>{CS BASE FOLDER}/{UNIQUE DOSSIER FOLDER ID 3}/{UNIQUE DOC FOLDER 7}/{FILE 7}</SPAN><BR /><SPAN>Each dossier folder can contain up to 250 document folders (average 125), containing exactly 1 file each.</SPAN><BR /><SPAN>During this phase the in-transaction creation of indexes is DISABLED, the reindexing mode is set to NONE.</SPAN><BR /><BR /><SPAN>#2: Seems to work fine as well, as far as the logs state this process runs through without problems. </SPAN><BR /><SPAN>During this phase the in-transaction creation of indexes is ENABLED, the reindexing mode is set to AUTO.</SPAN><BR /><BR /><SPAN>#3: This job is a custom Java job, scheduled as cron job and wired in through Spring via a custom Alfresco context file.</SPAN><BR /><SPAN>Simplified each run performs the following actions:</SPAN><BR /><SPAN>3.1) retrieve a list of X dossiers and the contained Y documents to be processed from an external migration database (connected via a custom ibatis connection)</SPAN><BR /><SPAN>3.2) iterate over all dossier entities and perform the perform the following actions for each dossier:</SPAN><BR /><SPAN>3.2.1) Check via Lucene (aspect search) whether the current dossier exists in Alfresco; if not create a new space for the dossier</SPAN><BR /><SPAN>3.2.2) If a new dossier space has been created in {3.2.1} and can be found, iterate over all document entities and perform for each:</SPAN><BR /><SPAN>3.2.2.1) Move the document from the original space (as imported in {step 1}) to the new archive dossier location (using FileFolderService.move)</SPAN><BR /><SPAN>3.2.2.2) Modify the type of the document to a custom type</SPAN><BR /><SPAN>3.2.3.3) Add new custom metadata to the document, as provided through the ibatis database entity</SPAN><BR /><SPAN>3.2.4. Persist the changes done to the repository (NodeService.setProperties).</SPAN><BR /><SPAN>Since this job makes use of Lucene while processing the repository changes, this job is being set up to make use of in-transaction indexing, the reindexing mode is set to AUTO.</SPAN><BR /><BR /><BR /><SPAN>So far we processed the import for about 30000 documents.</SPAN><BR /><BR /><SPAN>What we noticed (after enabling debug log settings) is that:</SPAN><BR /><SPAN>A) for the indexer component, we see that the largest commited index contains 400000 documents</SPAN><BR /><SPAN>B) At times the cron job does not run at all, most likely because Alfrescos indexer/merging service is too busy and uses too many threads concurrently for the index merging process</SPAN><BR /><SPAN>C) Also we see a pattern in the run frequencies; e.g. if we set up the job to run once every minute:</SPAN><BR /><SPAN>it does so for 8-9 times, then it stops running for about 3 minutes, later on it even takes breaks of up to 30 minutes.</SPAN><BR /><SPAN>A few hours later the pattern repeats as described, without restarting Alfresco.</SPAN><BR /><BR /><SPAN>Looking a the logs we find a lot of actions being done by the index merger during the breaks mentioned.</SPAN><BR /><BR /><SPAN>So to start with I have two questions I hope you can help us with:</SPAN><BR /><SPAN>1. Which default Alfresco runtime settings could/should we override in order to support the import to run smoothly and stable?</SPAN><BR /><SPAN>2. Is there a limit to the execution of configured quartz/cron jobs, like maximal thread count, cpu, mem, etc.?</SPAN><BR /><SPAN>3. How come the index can grow so quickly and contain that many documents, while (according to Alfresco wiki/forum recommendations) it should ideally only contains a few hundred max?</SPAN><BR /><BR /><SPAN>If required I can submit more details about our configuration, like alfresco-global.properties, alfresco.log and such.</SPAN><BR /><BR /><BR /><SPAN>Any hints or direction welcome!</SPAN><BR /><BR /></BODY></HTML> Thu, 14 Feb 2013 08:47:41 GMT chini 2013-02-14T08:47:41Z Large Lucene index size prevents cron job from running? https://connect.hyland.com/t5/alfresco-archive/large-lucene-index-size-prevents-cron-job-from-running/m-p/278127#M231257 Hello,I am currently running into performance problems with a custom bulk import process. Hopefully someone can help me find the bottleneck.First some background information:We planned to import a large amount of documents into Alfresco 3.4.7 EE, in controlled and monitored batches.The steps involve Thu, 14 Feb 2013 08:47:41 GMT https://connect.hyland.com/t5/alfresco-archive/large-lucene-index-size-prevents-cron-job-from-running/m-p/278127#M231257 chini 2013-02-14T08:47:41Z Re: Large Lucene index size prevents cron job from running? https://connect.hyland.com/t5/alfresco-archive/large-lucene-index-size-prevents-cron-job-from-running/m-p/278128#M231258 <HTML><HEAD></HEAD><BODY><SPAN>Hi</SPAN><BR /><BR /><SPAN>It looks like you are not batching your docs up in transactions and doing each individual operation in a single transaction.</SPAN><BR /><SPAN>Add meta data - move - change type etc etc.</SPAN><BR /><BR /><SPAN>Why do you need the indexer to find the dossier - I assume you know where to make it if you do not find it - so find it by xpath.</SPAN><BR /><BR /><SPAN>You need to work out why you have made so many documents! Are you deleting things but in fact archiving them?</SPAN><BR /><BR /><SPAN>Andy</SPAN></BODY></HTML> Fri, 08 Mar 2013 15:18:31 GMT https://connect.hyland.com/t5/alfresco-archive/large-lucene-index-size-prevents-cron-job-from-running/m-p/278128#M231258 andy 2013-03-08T15:18:31Z