Solr tracking stopped working

[aussen2]

We set up a fresh installation of Alfresco Community 4.0.d about three months ago and everything was working fine until recently. Then out of the sudden the following error messages started to appear in our log files about a week ago. We can still find some documents via search but not all. If we delete the Solr index it will reindex a certain amount of documents but again not all. So the tracking still seems to work in general but for some reason breaks after a certain amount of data is indexed.

We checked the SSL configuration and even added the maxSavePostSize="-1" setting to tomcats server.xml file. That didn't solve the problem. We then set the maxSavePostSize to 16384 (four times the default value) and that initially seemed to solve the problem. But then it came back the next day. We also tried running Solr without SSL but that had no effect either. Any help will be highly appreciated.

Here is the error from the log files:

 2012-05-14 05:14:42,849  ERROR [extensions.webscripts.AbstractRuntime] [http-8080-14] Exception from executeScript - redirecting to status template error: 04140014 Wrapped Exception (with status template): null
 org.springframework.extensions.webscripts.WebScriptException: 04140014 Wrapped Exception (with status template): null
        at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:970)
        at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:171)
        at org.alfresco.repo.web.scripts.RepositoryContainer$2.execute(RepositoryContainer.java:393)
        at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
        at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:462)
        at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:500)
        at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:275)
        at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:372)
        at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
        at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:58)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:470)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
May 14, 2012 5:14:42 AM org.alfresco.solr.tracker.CoreTracker trackRepository
SEVERE: Tracking failed
org.alfresco.error.AlfrescoRuntimeException: 04140012 GetNodeMetaData return status is 500
        at org.alfresco.solr.client.SOLRAPIClient.getNodesMetaData(SOLRAPIClient.java:721)
        at org.alfresco.solr.tracker.CoreTracker.indexNode(CoreTracker.java:1879)
        at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1410)
        at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:491)
        at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[andy]

Hi

See the Alfrecsoi and SOLR wiki page.

Use the CHECK and FIX URLs.

Please report any errors logged on the Alfresco side.
There wil be a real cause.

Andy

[aussen2]

Here is the output from /solr/admin/cores?action=CHECK:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst>
</response>
‍‍‍‍‍‍

And the one from /solr/admin/cores?action=FIX:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">32753</int></lst>
</response>
‍‍‍‍‍‍

Nothing special showed up in the log files when calling theses URLs.

[aussen2]

When calling /solr/admin/cores?action=SUMMARY&wt=xml we get the following output indicating that the last commit for the workspace store was 3+ days ago while the archive store seems to work just fine.

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">0</int></lst><lst name="Summary"><lst name="alfresco"><long name="Last Index Commit Time">1336744804210</long><date name="Last Index Commit Date">2012-05-11T14:00:04.21Z</date><str name="Lag">270165 s</str><str name="Duration">P0Y3DT3H2M45.211S</str><bool name="Active">false</bool><long name="Timestamp for last TX on server">1337014969421</long><date name="Date for last TX on server">2012-05-14T17:02:49.421Z</date></lst><lst name="archive"><long name="Last Index Commit Time">1337014969421</long><date name="Last Index Commit Date">2012-05-14T17:02:49.421Z</date><str name="Lag">0 s</str><str name="Duration">-P0Y</str><bool name="Active">false</bool><long name="Timestamp for last TX on server">1337014969421</long><date name="Date for last TX on server">2012-05-14T17:02:49.421Z</date></lst></lst>
</response>
‍‍‍‍‍‍

Could the cause for the tracking error be a corrupted node in the repository? Another search related process started to throw InvalidNodeRefExceptions recently (see https://forums.alfresco.com/en/viewtopic.php?f=52&t=41135#p130275 for details). How would the Solr tracking react if it came along such a node? Would it go on to index all the other nodes or would it just stop indexing?