topic Alfresco FULL reindex of 2.2M documents doesn't complete in Alfresco Archive https://connect.hyland.com/t5/alfresco-archive/alfresco-full-reindex-of-2-2m-documents-doesn-t-complete/m-p/248438#M201568 <HTML><HEAD></HEAD><BODY><SPAN>Hi Everyone,</SPAN><BR /><BR /><SPAN>using the 3.3.3 Enterprise version. After a larger crash, in all probability due to us stressing the app too much (we were jamming documents into alfresco through every orifice, and by orifice I mean 'repository interface.') Had to do a full reindex, but it failed to restart. (Out of memory, which we managed to fix by giving it 8 Gigs of Heap) The Repo now houses 2.2 million documents.</SPAN><BR /><BR /><SPAN>We are doing a lot of tests and tried setting all sorts of different parameters but there's apparently nothing that can jiggle the beast back to life.</SPAN><BR /><BR /><SPAN>This is what happens: the reindexing starts, it goes up to the point in the log file where it says it's got to 100%. We don't get the message saying it's completely done though. Stopping and restarting with the indexing in AUTO mode yields the exact same behaviour. </SPAN><BR /><BR /><SPAN>Obviously, alfresco doesn't start up without the complete index. What precisely happens?</SPAN><BR /><BR /><SPAN>The java threads basically lock each other out over time and in the end there's just one doing the job (might be normal behaviour; could someone confirm?). And that one just runs and doesn't stop. It uses between 70%-100% of one CPU core and runs forever(ran from Thursday afternoon until Monday morning).&nbsp; The indexing obviously failed to finish.</SPAN><BR /><BR /><SPAN>I believe though, we've got something interesting we can go on: we took a few thread dumps throughout last week and today.. and they all show the same behaviour:</SPAN><BR /><BR /><SPAN>there's the "main" thread, waiting in the </SPAN><BR /><SPAN> </SPAN><BR /><PRE class="language-none line-numbers"><CODE>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; org.alfresco.repo.node.index.AbstractReindexComponent.waitForAsynchronousReindexing(AbstractReindexComponent.java:1101)<SPAN class="line-numbers-rows"><SPAN>‍</SPAN></SPAN></CODE></PRE><SPAN> </SPAN><BR /><SPAN>method for the other threads to complete.</SPAN><BR /><BR /><SPAN>And then there is the only thread doing any apparent work : "indexTrackerThread1", which is basically stuck here: (it is always the same thread per Alfresco instance that keeps on going: it was indexTrackerThread</SPAN><STRONG>1</STRONG><SPAN> on Thursday, Friday and Monday. The number obviously varies between instances. If we start Alfresco up again, the thread might be&nbsp; called indexTrackerThread5 or some other number)</SPAN><BR /><BR /><BR /><PRE class="language-none line-numbers"><CODE> <BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.NativeThread.current(Native Method)<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:27)<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610)<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:113)<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:136)<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.lucene.index.CompoundFileReader$CSIndexInput.readInternal(CompoundFileReader.java:247)<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:157)<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:116)<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:92)<BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:82)<BR /> <SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN></CODE></PRE><BR /><SPAN>It is basically hanging here. Regardless of the point in time when I take the thread dump, it's hanging here. Here's the source code around this line </SPAN><BR /><PRE class="language-none line-numbers"><CODE>org.apache.lucene.store.NIOFSDirectory:$NIOFSIndexInput.readInternal(NIOFSDirectory.java:113)<SPAN class="line-numbers-rows"><SPAN>‍</SPAN></SPAN></CODE></PRE><SPAN> </SPAN><BR /><SPAN>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </SPAN><PRE class="language-none line-numbers"><CODE>&nbsp; long pos = getFilePointer(); <BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; while (bb.hasRemaining()) {<BR /><BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; int i = channel.read(bb, pos);<BR /><BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (i == -1)<BR /><BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; throw new IOException("read past EOF");<BR /><BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; pos += i;<BR /><BR />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; }<BR /><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN></CODE></PRE><SPAN>I have no idea of how Lucene and the indexer work, but it strikes me as pretty odd that every time I look at the program's progress, it's doing the exact same thing. (for bloody days now)</SPAN><BR /><BR /><SPAN>Could this bit of Lucene code be the issue?</SPAN><BR /><BR /><SPAN>I'd really really appreciate some help here.</SPAN><BR /><BR /><SPAN>Thank you very much. Hope the size of the post doesn't frighten you. </SPAN><BR /><STRONG>Any</STRONG><SPAN> feedback is welcome. Don't hesitate to throw in your two cents.</SPAN><BR /><BR /><SPAN>Max</SPAN></BODY></HTML> Mon, 17 Jan 2011 15:39:15 GMT bnmaxim 2011-01-17T15:39:15Z Alfresco FULL reindex of 2.2M documents doesn't complete https://connect.hyland.com/t5/alfresco-archive/alfresco-full-reindex-of-2-2m-documents-doesn-t-complete/m-p/248438#M201568 Hi Everyone,using the 3.3.3 Enterprise version. After a larger crash, in all probability due to us stressing the app too much (we were jamming documents into alfresco through every orifice, and by orifice I mean 'repository interface.') Had to do a full reindex, but it failed to restart. (Out of mem Mon, 17 Jan 2011 15:39:15 GMT https://connect.hyland.com/t5/alfresco-archive/alfresco-full-reindex-of-2-2m-documents-doesn-t-complete/m-p/248438#M201568 bnmaxim 2011-01-17T15:39:15Z Re: Alfresco FULL reindex of 2.2M documents doesn't complete https://connect.hyland.com/t5/alfresco-archive/alfresco-full-reindex-of-2-2m-documents-doesn-t-complete/m-p/248439#M201569 <HTML><HEAD></HEAD><BODY><SPAN>I don't have any solution for you, but to know more about what Alfresco is looping on, you might want to debug it with Eclipse (Remote Java Application).</SPAN><BR /><SPAN>It is possible only if you started Alfresco with a catalina.sh modified like this:</SPAN><BR /><BR /><PRE class="language-none line-numbers"><CODE>JAVA_OPTS='-Xms128m -Xmx512m -server -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=1044'<SPAN class="line-numbers-rows"><SPAN>‍</SPAN></SPAN></CODE></PRE><BR /><SPAN>If you don't have it, you might want to add it next time you restart.</SPAN><BR /><BR /><SPAN>Hope this helps,</SPAN><BR /><SPAN>Nicolas Raoul</SPAN></BODY></HTML> Tue, 18 Jan 2011 10:12:58 GMT https://connect.hyland.com/t5/alfresco-archive/alfresco-full-reindex-of-2-2m-documents-doesn-t-complete/m-p/248439#M201569 nicolasraoul 2011-01-18T10:12:58Z Re: Alfresco FULL reindex of 2.2M documents doesn't complete https://connect.hyland.com/t5/alfresco-archive/alfresco-full-reindex-of-2-2m-documents-doesn-t-complete/m-p/248440#M201570 <HTML><HEAD></HEAD><BODY><SPAN>Unfortunately debugging is not a real option in our environment just now. </SPAN><BR /><BR /><SPAN>Was expecting a bit of help from someone with a bit more experience regarding indexing and Lucene.</SPAN><BR /><BR /><SPAN>Any other takers?</SPAN></BODY></HTML> Tue, 18 Jan 2011 10:46:16 GMT https://connect.hyland.com/t5/alfresco-archive/alfresco-full-reindex-of-2-2m-documents-doesn-t-complete/m-p/248440#M201570 bnmaxim 2011-01-18T10:46:16Z Re: Alfresco FULL reindex of 2.2M documents doesn't complete https://connect.hyland.com/t5/alfresco-archive/alfresco-full-reindex-of-2-2m-documents-doesn-t-complete/m-p/248441#M201571 <HTML><HEAD></HEAD><BODY><SPAN>Please contact Alfresco support for help with Alfresco Enterprise.</SPAN></BODY></HTML> Tue, 18 Jan 2011 11:04:17 GMT https://connect.hyland.com/t5/alfresco-archive/alfresco-full-reindex-of-2-2m-documents-doesn-t-complete/m-p/248441#M201571 mrogers 2011-01-18T11:04:17Z