Something haywire with my vanilla 4.0.a linux install

[sor]

I've searched around, and unfortunately I don't know much about java/tomcat troubleshooting. What I have managed to nail down are the symptoms and some traces that seem to show problems (whether they are correlated or not I am unsure).

At some point as I started adding users and documents to the vanilla install, documents stopped getting indexed. Under 'document library' on my site, I also see that there are documents with various tags, but when I click on the tags it shows no documents. Also, cpu usage is relatively high.

Here are the various traces I've found:


16:49:19,012 ERROR [org.springframework.extensions.webscripts.AbstractRuntime] Exception from executeScript - redirecting to status template error: 09050019 Wrapped Exception (with status template): null
org.springframework.extensions.webscripts.WebScriptException: 09050019 Wrapped Exception (with status template): null
   at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:884)
   at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:171)
   at org.alfresco.repo.web.scripts.RepositoryContainer$2.execute(RepositoryContainer.java:393)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:462)
   at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:500)
   at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:275)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:352)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:189)
   at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:58)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:558)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
   at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException

Oct 5, 2011 4:44:59 PM org.apache.catalina.core.StandardContext loadOnStartup
SEVERE: Servlet /awe threw load() exception
java.lang.ClassNotFoundException: org.json.simple.JSONObject
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1516)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1361)
   at java.lang.Class.getDeclaredMethods0(Native Method)
   at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
   at java.lang.Class.getDeclaredMethods(Class.java:1791)
   at java.beans.Introspector$1.run(Introspector.java:1287)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.beans.Introspector.getPublicDeclaredMethods(Introspector.java:1285)
   at java.beans.Introspector.getTargetMethodInfo(Introspector.java:1151)
   at java.beans.Introspector.getBeanInfo(Introspector.java:402)
   at java.beans.Introspector.getBeanInfo(Introspector.java:168)
   at org.springframework.extensions.webscripts.processor.FTLTemplateProcessor$NonBlockingObjectWrapper.<init>(FTLTemplateProcessor.java:341)
   at org.springframework.extensions.webscripts.processor.FTLTemplateProcessor.initConfig(FTLTemplateProcessor.java:281)
   at org.springframework.extensions.webscripts.processor.FTLTemplateProcessor.init(FTLTemplateProcessor.java:114)
   at org.springframework.extensions.webscripts.processor.AbstractTemplateProcessor.onApplicationEvent(AbstractTemplateProcessor.java:155)
   at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:97)
   at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303)
   at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:911)
   at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:428)
   at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:442)
   at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:458)
   at org.springframework.web.servlet.FrameworkServlet.initWebApplicationContext(FrameworkServlet.java:339)
   at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:306)
   at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:127)
   at javax.servlet.GenericServlet.init(GenericServlet.java:212)
   at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1173)
   at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:993)
   at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4187)
   at org.apache.catalina.core.StandardContext.start(StandardContext.java:4496)
   at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
   at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
   at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:546)
   at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:905)
   at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:740)
   at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:500)
   at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
   at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
   at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
   at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
   at org.apache.catalina.core.StandardHost.start(StandardHost.java:785)
   at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
   at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
   at org.apache.catalina.core.StandardService.start(StandardService.java:519)
   at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
   at org.apache.catalina.startup.Catalina.start(Catalina.java:581)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
   at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)

Oct 5, 2011 4:50:19 PM org.quartz.core.ErrorLogger schedulerError
SEVERE: Job (Solr.CoreTracker-alfresco threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 09050021 GetNodeMetaData return status is 500]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 09050021 GetNodeMetaData return status is 500
   at org.alfresco.solr.client.SOLRAPIClient.getNodesMetaData(SOLRAPIClient.java:701)
   at org.alfresco.solr.tracker.CoreTracker.updateDescendantAuxDocs(CoreTracker.java:1833)
   at org.alfresco.solr.tracker.CoreTracker.indexNode(CoreTracker.java:1654)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1316)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:481)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   … 1 more
2011-10-05 16:50:34,095  ERROR [extensions.webscripts.AbstractRuntime] [http-8443-1] Exception from executeScript - redirecting to status template error: 09050024 Wrapped Exception (with status template): null
org.springframework.extensions.webscripts.WebScriptException: 09050024 Wrapped Exception (with status template): null
   at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:884)
   at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:171)
   at org.alfresco.repo.web.scripts.RepositoryContainer$2.execute(RepositoryContainer.java:393)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:462)
   at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:500)
   at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:275)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:352)
   at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:189)
   at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
   at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:58)
   at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
   at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
   at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
   at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
   at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:558)
   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
   at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
   at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
   at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.NullPointerException
Oct 5, 2011 4:50:34 PM org.quartz.core.JobRunShell run
SEVERE: Job Solr.CoreTracker-alfresco threw an unhandled Exception:
org.alfresco.error.AlfrescoRuntimeException: 09050022 GetNodeMetaData return status is 500
   at org.alfresco.solr.client.SOLRAPIClient.getNodesMetaData(SOLRAPIClient.java:701)
   at org.alfresco.solr.tracker.CoreTracker.updateDescendantAuxDocs(CoreTracker.java:1833)
   at org.alfresco.solr.tracker.CoreTracker.indexNode(CoreTracker.java:1654)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1316)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:481)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Oct 5, 2011 4:50:34 PM org.quartz.core.ErrorLogger schedulerError
SEVERE: Job (Solr.CoreTracker-alfresco threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 09050022 GetNodeMetaData return status is 500]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 09050022 GetNodeMetaData return status is 500
   at org.alfresco.solr.client.SOLRAPIClient.getNodesMetaData(SOLRAPIClient.java:701)
   at org.alfresco.solr.tracker.CoreTracker.updateDescendantAuxDocs(CoreTracker.java:1833)
   at org.alfresco.solr.tracker.CoreTracker.indexNode(CoreTracker.java:1654)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1316)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:481)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)

[alexandra]

We had a similar issue with similar error messages on our Alfresco 4.0 install where we upgraded from 3.4d which was run over SSL. Any progress in solving this?

Oct 18, 2011 10:42:30 AM org.quartz.core.JobRunShell run
SEVERE: Job Solr.CoreTracker-alfresco threw an unhandled Exception:
org.alfresco.error.AlfrescoRuntimeException: 09180026 GetNodeMetaData return status is 500
        at org.alfresco.solr.client.SOLRAPIClient.getNodesMetaData(SOLRAPIClient.java:701)
        at org.alfresco.solr.tracker.CoreTracker.indexNode(CoreTracker.java:1623)
        at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1316)
        at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:481)
        at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Oct 18, 2011 10:42:30 AM org.quartz.core.ErrorLogger schedulerError
SEVERE: Job (Solr.CoreTracker-alfresco threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 09180026 GetNodeMetaData return status is 500]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 09180026 GetNodeMetaData return status is 500
        at org.alfresco.solr.client.SOLRAPIClient.getNodesMetaData(SOLRAPIClient.java:701)
        at org.alfresco.solr.tracker.CoreTracker.indexNode(CoreTracker.java:1623)
        at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1316)
        at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:481)
        at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:216)

[fliot]

Same issue with a fresh 4.0.a

[jaredd]

Having the same issues on Alfresco 4.0a.  It was working fine for two months, until Nov 19th when I started getting errors just like 'sor' and the others.  Now nothing new is getting indexed in the search feature.

[fliot]

Got the issue in 4.0.a, and it was solved with 4.0.b upgrade….

[jaredd]

Got the issue in 4.0.a, and it was solved with 4.0.b upgrade….

We just did an upgrade to 4.0.b.  The issue went away, then came back.  I'll post if I find any fixes.

EDIT: I scrapped the upgrade, and did it again, and now it works.