AlfrescoRuntimeException: GetModelsDiff return stats is 404

[callen]

I have been running 4.2.b since 12/2/12 with no problems. Then yesterday 12/27/12 share refused to allow users to log in. (The server is restarted automatically each night)

The log  seems to indicate an error with the solr index, but I really do not have enough knowledge to diagnose. I have copied the alf_data to another machine, and the problem is duplicated on the 2nd machine, which would seem to indicate the error is not related to the machine or environment, but within the alf_data.

Please help!! – I will post a copy of the log. – We are shut down at the the time.

2012-12-28 19:51:56 Commons Daemon procrun stdout initialized
Listening for transport dt_socket at address: 8000
2012-12-28 19:52:30,511  WARN  [handler.component.SpellCheckComponent] [localhost-startStop-1] No queryConverter defined, using default converter
 2012-12-28 19:52:39,011  WARN  [handler.component.SpellCheckComponent] [localhost-startStop-1] No queryConverter defined, using default converter
 2012-12-28 19:54:18,011  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'sysAdmin' subsystem, ID: [sysAdmin, default]
 2012-12-28 19:54:18,448  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'sysAdmin' subsystem, ID: [sysAdmin, default] complete
 2012-12-28 19:55:35,526  INFO  [extensions.webscripts.TemplateProcessorRegistry] [localhost-startStop-1] Registered template processor Repository Template Processor for extension ftl
 2012-12-28 19:55:35,542  INFO  [extensions.webscripts.ScriptProcessorRegistry] [localhost-startStop-1] Registered script processor Repository Script Processor for extension js
 2012-12-28 19:56:15,714  INFO  [domain.schema.SchemaBootstrap] [localhost-startStop-1] Connecting to database: jdbc:postgresql://localhost:5432/alfresco, UserName=alfresco, PostgreSQL Native Driver
 2012-12-28 19:56:15,714  INFO  [domain.schema.SchemaBootstrap] [localhost-startStop-1] Schema managed by database dialect org.hibernate.dialect.PostgreSQLDialect.
 2012-12-28 19:56:25,698  INFO  [domain.schema.SchemaBootstrap] [localhost-startStop-1] No changes were made to the schema.
 2012-12-28 19:56:29,870  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'Search' subsystem, ID: [Search, managed, solr]
 2012-12-28 19:56:30,729  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'Search' subsystem, ID: [Search, managed, solr] complete
 2012-12-28 19:56:31,886  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'thirdparty' subsystem, ID: [thirdparty, default]
 2012-12-28 19:56:33,807  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'thirdparty' subsystem, ID: [thirdparty, default] complete
 2012-12-28 19:56:33,807  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'OOoDirect' subsystem, ID: [OOoDirect, default]
 2012-12-28 19:56:36,776  WARN  [alfresco.util.OpenOfficeConnectionTester] [localhost-startStop-1] An initial OpenOffice connection could not be established.
 2012-12-28 19:56:36,823  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'OOoDirect' subsystem, ID: [OOoDirect, default] complete
 2012-12-28 19:56:36,839  INFO  [repo.admin.ConfigurationChecker] [localhost-startStop-1] The Alfresco root data directory ('dir.root') is: C:\Alfresco\alf_data
 2012-12-28 19:56:36,964  INFO  [admin.patch.PatchExecuter] [localhost-startStop-1] Checking for patches to apply …
 2012-12-28 19:56:40,323  INFO  [admin.patch.PatchExecuter] [localhost-startStop-1] No patches were required.
 2012-12-28 19:56:40,370  INFO  [repo.module.ModuleServiceImpl] [localhost-startStop-1] Found 3 module(s).
 2012-12-28 19:56:40,761  INFO  [repo.module.ModuleServiceImpl] [localhost-startStop-1] Starting module 'alfresco-mm-repo' version 0.1.5.9.
 2012-12-28 19:56:40,979  INFO  [repo.module.ModuleServiceImpl] [localhost-startStop-1] Starting module 'org.alfresco.module.vti' version 1.2.
 2012-12-28 19:56:41,104  INFO  [repo.module.ModuleServiceImpl] [localhost-startStop-1] Starting module 'org.alfresco.integrations.google.docs' version 2.0.0.
 2012-12-28 19:56:41,182  WARN  [repo.module.ModuleServiceImpl] [localhost-startStop-1] A previously-installed module 'org_alfresco_module_wcmquickstart' (version 1.0) is not present in your distribution.
 2012-12-28 19:56:41,198  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'fileServers' subsystem, ID: [fileServers, default]
 2012-12-28 19:56:44,182  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'Authentication' subsystem, ID: [Authentication, managed, alfrescoNtlm1]
 2012-12-28 19:56:44,886  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'Authentication' subsystem, ID: [Authentication, managed, alfrescoNtlm1] complete
 2012-12-28 19:56:45,151  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'fileServers' subsystem, ID: [fileServers, default] complete
 2012-12-28 19:56:45,151  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'imap' subsystem, ID: [imap, default]
 2012-12-28 19:56:46,339  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'imap' subsystem, ID: [imap, default] complete
 2012-12-28 19:56:46,339  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'email' subsystem, ID: [email, outbound]
 2012-12-28 19:56:46,636  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'email' subsystem, ID: [email, outbound] complete
 2012-12-28 19:56:46,636  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'email' subsystem, ID: [email, inbound]
 2012-12-28 19:56:47,120  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'email' subsystem, ID: [email, inbound] complete
 2012-12-28 19:56:47,120  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'googledocs' subsystem, ID: [googledocs, default]
 2012-12-28 19:56:47,401  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'googledocs' subsystem, ID: [googledocs, default] complete
 2012-12-28 19:56:47,401  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'Subscriptions' subsystem, ID: [Subscriptions, default]
 2012-12-28 19:56:47,479  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'Subscriptions' subsystem, ID: [Subscriptions, default] complete
 2012-12-28 19:56:47,495  INFO  [repo.usage.UserUsageTrackingComponent] [localhost-startStop-1] Disabled - clear non-missing user usages …
 2012-12-28 19:56:48,011  INFO  [repo.usage.UserUsageTrackingComponent] [localhost-startStop-1] Found 0 users to clear
 2012-12-28 19:56:48,011  INFO  [repo.usage.UserUsageTrackingComponent] [localhost-startStop-1] … cleared non-missing usages for 0 users
 2012-12-28 19:56:48,011  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Starting 'Synchronization' subsystem, ID: [Synchronization, default]
 2012-12-28 19:56:48,370  INFO  [management.subsystems.ChildApplicationContextFactory] [localhost-startStop-1] Startup of 'Synchronization' subsystem, ID: [Synchronization, default] complete
 2012-12-28 19:56:49,417  ERROR [web.context.ContextLoader] [localhost-startStop-1] Context initialization failed
 org.alfresco.service.cmr.repository.InvalidNodeRefException: Node does not exist: workspace://SpacesStore/50e3a132-62f0-45e3-832e-e090f1d749e6(null)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getNodePairNotNull(DbNodeServiceImpl.java:178)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getChildAssocs(DbNodeServiceImpl.java:1780)
   at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:601)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:105)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy7.getChildAssocs(Unknown Source)
   at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:601)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy35.getChildAssocs(Unknown Source)
   at org.alfresco.repo.security.authority.AuthorityDAOImpl.isAuthorityContained(AuthorityDAOImpl.java:1003)
   at org.alfresco.repo.security.authority.AuthorityDAOImpl.isAuthorityContained(AuthorityDAOImpl.java:1012)
   at org.alfresco.repo.security.authority.AuthorityDAOImpl.getContainingAuthoritiesInZone(AuthorityDAOImpl.java:802)
   at org.alfresco.repo.security.authority.AuthorityServiceImpl.getContainingAuthoritiesInZone(AuthorityServiceImpl.java:551)
   at org.alfresco.repo.site.SiteServiceImpl.listSitesImpl(SiteServiceImpl.java:1040)
   at org.alfresco.repo.site.SiteServiceImpl.listSites(SiteServiceImpl.java:934)
   at org.alfresco.repo.site.SiteServiceImpl.listSites(SiteServiceImpl.java:911)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   at java.lang.reflect.Method.invoke(Method.java:601)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at net.sf.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:80)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.alfresco.repo.security.permissions.impl.ExceptionTranslatorMethodInterceptor.invoke(ExceptionTranslatorMethodInterceptor.java:46)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.alfresco.repo.audit.AuditMethodInterceptor.invoke(AuditMethodInterceptor.java:159)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.alfresco.repo.transaction.CheckTransactionAdvice.invoke(CheckTransactionAdvice.java:47)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.alfresco.repo.transaction.RetryingTransactionAdvice$1.execute(RetryingTransactionAdvice.java:64)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:433)
   at org.alfresco.repo.transaction.RetryingTransactionAdvice.invoke(RetryingTransactionAdvice.java:67)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy89.listSites(Unknown Source)
   at org.alfresco.repo.site.SiteServiceBootstrap$1.doWork(SiteServiceBootstrap.java:63)
   at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:529)
   at org.alfresco.repo.site.SiteServiceBootstrap.onBootstrap(SiteServiceBootstrap.java:59)
   at org.springframework.extensions.surf.util.AbstractLifecycleBean.onApplicationEvent(AbstractLifecycleBean.java:56)
   at org.alfresco.repo.management.SafeApplicationEventMulticaster.multicastEventInternal(SafeApplicationEventMulticaster.java:209)
   at org.alfresco.repo.management.SafeApplicationEventMulticaster.multicastEvent(SafeApplicationEventMulticaster.java:180)
   at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:303)
   at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:911)
   at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:428)
   at org.springframework.web.context.ContextLoader.createWebApplicationContext(ContextLoader.java:276)
   at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:197)
   at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:47)
   at org.alfresco.web.app.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:63)
   at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)
   at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)
   at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
   at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
   at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
   at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:618)
   at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:963)
   at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1600)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
   at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
   at java.util.concurrent.FutureTask.run(FutureTask.java:166)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
   at java.lang.Thread.run(Thread.java:722)
2012-12-28 19:57:03,557  WARN  [alfresco.util.OpenOfficeConnectionTester] [DefaultScheduler_Worker-2] Error trying to query Open Office version information. OpenOffice.org's ConfigurationRegistry not implemented in this version of OOo. This should not affect the operation of OOo.
 2012-12-28 19:57:03,604  INFO  [alfresco.util.OpenOfficeConnectionTester] [DefaultScheduler_Worker-2] The OpenOffice connection was re-established.
 2012-12-28 19:58:02,682  INFO  [extensions.webscripts.TemplateProcessorRegistry] [localhost-startStop-1] Registered template processor freemarker for extension ftl
 2012-12-28 19:58:03,136  INFO  [extensions.webscripts.ScriptProcessorRegistry] [localhost-startStop-1] Registered script processor javascript for extension js
 2012-12-28 19:58:03,136  INFO  [extensions.webscripts.TemplateProcessorRegistry] [localhost-startStop-1] Registered template processor freemarker for extension ftl
 2012-12-28 19:58:03,182  INFO  [extensions.webscripts.ScriptProcessorRegistry] [localhost-startStop-1] Registered script processor javascript for extension js
 2012-12-28 19:58:17,745  INFO  [extensions.webscripts.DeclarativeRegistry] [localhost-startStop-1] Registered 339 Web Scripts (+0 failed), 352 URLs
 2012-12-28 19:58:17,745  INFO  [extensions.webscripts.DeclarativeRegistry] [localhost-startStop-1] Registered 8 Package Description Documents (+0 failed) 
 2012-12-28 19:58:17,745  INFO  [extensions.webscripts.DeclarativeRegistry] [localhost-startStop-1] Registered 0 Schema Description Documents (+0 failed) 
 2012-12-28 19:58:21,120  INFO  [extensions.webscripts.AbstractRuntimeContainer] [localhost-startStop-1] Initialised Spring Surf Container Web Script Container (in 17910.336ms)
 2012-12-28 19:58:21,167  INFO  [extensions.webscripts.TemplateProcessorRegistry] [localhost-startStop-1] Registered template processor freemarker for extension ftl
 2012-12-28 19:58:21,214  INFO  [extensions.webscripts.ScriptProcessorRegistry] [localhost-startStop-1] Registered script processor javascript for extension js
 2012-12-28 19:58:25,011  ERROR [quartz.core.JobRunShell] [SolrTrackerScheduler_Worker-3] Job Solr.CoreTracker-archive threw an unhandled Exception: 
 org.alfresco.error.AlfrescoRuntimeException: 11280001 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
2012-12-28 19:58:25,042  ERROR [quartz.core.ErrorLogger] [SolrTrackerScheduler_Worker-3] Job (Solr.CoreTracker-archive threw an exception.
 org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 11280001 GetModelsDiff return status is 404]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 11280001 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   … 1 more
2012-12-28 19:58:25,042  ERROR [quartz.core.JobRunShell] [SolrTrackerScheduler_Worker-2] Job Solr.CoreTracker-alfresco threw an unhandled Exception: 
 org.alfresco.error.AlfrescoRuntimeException: 11280000 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
2012-12-28 19:58:25,073  ERROR [quartz.core.ErrorLogger] [SolrTrackerScheduler_Worker-2] Job (Solr.CoreTracker-alfresco threw an exception.
 org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 11280000 GetModelsDiff return status is 404]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 11280000 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   … 1 more
2012-12-28 19:58:25,870  INFO  [web.site.EditionInterceptor] [http-apr-8080-exec-4] Unable to retrieve License information from Alfresco: 404
 2012-12-28 19:58:30,589  ERROR [quartz.core.JobRunShell] [SolrTrackerScheduler_Worker-3] Job Solr.CoreTracker-archive threw an unhandled Exception: 
 org.alfresco.error.AlfrescoRuntimeException: 11280002 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
2012-12-28 19:58:30,589  ERROR [quartz.core.ErrorLogger] [SolrTrackerScheduler_Worker-3] Job (Solr.CoreTracker-archive threw an exception.
 org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 11280002 GetModelsDiff return status is 404]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:227)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 11280002 GetModelsDiff return status is 404
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1036)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   … 1 more‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

here is the catalina log: (slightly different time period)

Dec 28, 2012 12:18:30 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
   at java.security.Provider$Service.getImplClass(Provider.java:1279)
   at java.security.Provider$Service.newInstance(Provider.java:1237)
   at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
   at javax.crypto.Cipher.init(Cipher.java:1348)
   at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
   at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
   at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
   at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
   at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
   at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:300)
   at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
   at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:446)
   at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:140)
   at org.alfresco.encryption.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:168)
   at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:130)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:106)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:364)
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1033)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)

Dec 28, 2012 12:18:30 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
   at java.security.Provider$Service.getImplClass(Provider.java:1279)
   at java.security.Provider$Service.newInstance(Provider.java:1237)
   at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
   at javax.crypto.Cipher.init(Cipher.java:1348)
   at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
   at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
   at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
   at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
   at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
   at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:300)
   at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
   at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:446)
   at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:140)
   at org.alfresco.encryption.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:168)
   at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:130)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:106)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:364)
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1033)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)

Dec 28, 2012 12:18:30 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
   at java.security.Provider$Service.getImplClass(Provider.java:1279)
   at java.security.Provider$Service.newInstance(Provider.java:1237)
   at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
   at javax.crypto.Cipher.init(Cipher.java:1348)
   at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
   at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
   at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
   at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
   at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
   at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:293)
   at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
   at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:494)
   at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:313)
   at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:182)
   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
   at java.lang.Thread.run(Thread.java:722)

Dec 28, 2012 12:18:30 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
   at java.security.Provider$Service.getImplClass(Provider.java:1279)
   at java.security.Provider$Service.newInstance(Provider.java:1237)
   at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
   at javax.crypto.Cipher.init(Cipher.java:1348)
   at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
   at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
   at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
   at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
   at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
   at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:293)
   at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
   at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:494)
   at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:313)
   at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:182)
   at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:216)
   at java.lang.Thread.run(Thread.java:722)

Dec 28, 2012 12:43:30 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1600)
   at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1559)
   at java.security.Provider$Service.getImplClass(Provider.java:1279)
   at java.security.Provider$Service.newInstance(Provider.java:1237)
   at javax.crypto.Cipher.chooseProvider(Cipher.java:845)
   at javax.crypto.Cipher.init(Cipher.java:1348)
   at sun.security.ssl.CipherBox.<init>(CipherBox.java:175)
   at sun.security.ssl.CipherBox.newCipherBox(CipherBox.java:208)
   at sun.security.ssl.CipherSuite$BulkCipher.newCipher(CipherSuite.java:467)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:507)
   at sun.security.ssl.CipherSuite$BulkCipher.isAvailable(CipherSuite.java:485)
   at sun.security.ssl.CipherSuite.isAvailable(CipherSuite.java:190)
   at sun.security.ssl.SSLContextImpl.getApplicableCipherSuiteList(SSLContextImpl.java:342)
   at sun.security.ssl.SSLContextImpl.getDefaultCipherSuiteList(SSLContextImpl.java:300)
   at sun.security.ssl.SSLSocketImpl.init(SSLSocketImpl.java:576)
   at sun.security.ssl.SSLSocketImpl.<init>(SSLSocketImpl.java:446)
   at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:140)
   at org.alfresco.encryption.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:168)
   at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
   at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
   at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
   at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
   at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
   at org.alfresco.httpclient.AbstractHttpClient.executeMethod(AbstractHttpClient.java:130)
   at org.alfresco.httpclient.AbstractHttpClient.sendRemoteRequest(AbstractHttpClient.java:106)
   at org.alfresco.httpclient.HttpClientFactory$HttpsClient.sendRequest(HttpClientFactory.java:364)
   at org.alfresco.solr.client.SOLRAPIClient.getModelsDiff(SOLRAPIClient.java:1033)
   at org.alfresco.solr.tracker.CoreTracker.trackModels(CoreTracker.java:1787)
   at org.alfresco.solr.tracker.CoreTracker.trackRepository(CoreTracker.java:1126)
   at org.alfresco.solr.tracker.CoreTracker.updateIndex(CoreTracker.java:472)
   at org.alfresco.solr.tracker.CoreTrackerJob.execute(CoreTrackerJob.java:45)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:563)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[janv]

That message implies a connectivity error when Solr tries to call back into the Repo (configured by default to use https). You're second log snippet shows failure to start properly, eg.

 Could not load org.bouncycastle.jce.provider.symmetric.AES$ECB ‍

You can also search forum posts for "GetModelsDiff return status is 404" (include the double quotes).

Regards,
Jan

[romschn]

Have you checked if your database is running and your alfresco server is pointing to the correct database?