No live node exists - error halts solr indexing

[loftux]

Using 4.0.d on postgresql indexing with Solr, I see this error in alfresco.log

2012-02-15 08:11:50,393  ERROR [extensions.webscripts.AbstractRuntime] [http-8443-4] Exception from executeScript - redirecting to status template error: 01156612 Wrapped Exception (with status template): No live node exists: 
   ID:        520521
   Cache row: NodeEntity[ ID=520521, version=16, store=workspace://SpacesStore, uuid=64b01e12-dfac-4b22-96d0-bfa30ba7d34e, typeQNameId=32, localeId=15, aclId=null, deleted=true, transaction=TransactionEntity[ ID=1064115, server=null, changeTxnId=f4359d12-f144-43c7-8c17-18697c4eb864, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=anbj01, auditCreated=2012-02-07T08:05:37.520+01:00, auditModifier=anbj01, auditModified=2012-02-07T09:20:18.000+01:00]]
   DB row:    NodeEntity[ ID=520521, version=16, store=workspace://SpacesStore, uuid=64b01e12-dfac-4b22-96d0-bfa30ba7d34e, typeQNameId=32, localeId=15, aclId=null, deleted=true, transaction=TransactionEntity[ ID=1064115, server=null, changeTxnId=f4359d12-f144-43c7-8c17-18697c4eb864, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=anbj01, auditCreated=2012-02-07T08:05:37.520+01:00, auditModifier=anbj01, auditModified=2012-02-07T09:20:18.000+01:00]]
 org.springframework.extensions.webscripts.WebScriptException: 01156612 Wrapped Exception (with status template): No live node exists: 
   ID:        520521
   Cache row: NodeEntity[ ID=520521, version=16, store=workspace://SpacesStore, uuid=64b01e12-dfac-4b22-96d0-bfa30ba7d34e, typeQNameId=32, localeId=15, aclId=null, deleted=true, transaction=TransactionEntity[ ID=1064115, server=null, changeTxnId=f4359d12-f144-43c7-8c17-18697c4eb864, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=anbj01, auditCreated=2012-02-07T08:05:37.520+01:00, auditModifier=anbj01, auditModified=2012-02-07T09:20:18.000+01:00]]
   DB row:    NodeEntity[ ID=520521, version=16, store=workspace://SpacesStore, uuid=64b01e12-dfac-4b22-96d0-bfa30ba7d34e, typeQNameId=32, localeId=15, aclId=null, deleted=true, transaction=TransactionEntity[ ID=1064115, server=null, changeTxnId=f4359d12-f144-43c7-8c17-18697c4eb864, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=anbj01, auditCreated=2012-02-07T08:05:37.520+01:00, auditModifier=anbj01, auditModified=2012-02-07T09:20:18.000+01:00]]‍‍‍‍‍‍‍‍

This error halts Solr indexing, it cannot get past that. I think Solr should be more fault tolerant, but my primary question here is how can this be fixed?
If I do

select * from alf_transaction where change_txn_id='f4359d12-f144-43c7-8c17-18697c4eb864';
id   version   server_id   change_txn_id   commit_time_ms
1064115   1   4   f4359d12-f144-43c7-8c17-18697c4eb864   1328887525021‍‍‍

and then

select * from alf_node where transaction_id = 1064115

id   version   store_id   uuid   transaction_id   node_deleted   type_qname_id   acl_id   audit_creator   audit_created   audit_modifier   audit_modified   audit_accessed   locale_id
530844   4   45   64b01e12-dfac-4b22-96d0-bfa30ba7d34e   1064115   f   34   1649   anbj01   2012-02-07T08:05:37.520+01:00   anbj01   2012-02-07T09:20:18.000+01:00      27
530847   2   45   8b85ff9f-4416-4195-aab8-88ec1dab1288   1064115   f   120   1649   anbj01   2012-02-10T16:24:56.896+01:00   anbj01   2012-02-10T16:24:56.896+01:00      27
530841   5   50   8b85ff9f-4416-4195-aab8-88ec1dab1288   1064115   t   32      anbj01   2012-02-10T16:24:56.896+01:00   anbj01   2012-02-10T16:24:56.896+01:00      15
520521   16   50   64b01e12-dfac-4b22-96d0-bfa30ba7d34e   1064115   t   32      anbj01   2012-02-07T08:05:37.520+01:00   anbj01   2012-02-07T09:20:18.000+01:00      15‍‍‍‍‍‍‍

Is this just a cache issue, so a restart of tomcat would help?
Or is it something corrupted in the db?
Looking at the source http://svn.alfresco.com/repos/alfresco-open-mirror/alfresco/HEAD/root/projects/repository/source/jav...
it is failing in the function trying to "Find an undeleted node" with "ConcurrencyFailureException if the ID doesn't reference a <b>live</b> node".
But what exactly makes this node not live?

[andy]

Hi

I believe this is a cache issue and a restart shuld fix it.

SOLR tracking should not interact with the cache.
Some of this has been fixed recently.

SOLR would retry until it gets pas the issue (it does not skip stuff and try later)
It may do in the future ….

Andy

[loftux]

This was not a cache issue, I had to remove the "no live" node directly from the database.

Solr is definitely exposing underlying issues on the database, I'm not sure if it would be best if Solr (like old lucene) moved past errors, so that indexing can continue, or halt like in this case. It went unnoticed for several days, so there was a very outdated index. But if indexing had moved on, I'm not sure an error in alfresco.log would have been spotted and subsequently fixed.

This is how I fixed it:
I ran these select statements to find out where the node id 520521 is present

select * from alf_child_assoc where child_node_id = 520521;
select * from alf_node_assoc where target_node_id = 520521;
select * from alf_node_assoc where source_node_id = 520521;
select * from alf_node_aspects where node_id = 520521;
select * from alf_node_properties where node_id = 520521;
select * from alf_node where id = 520521;‍‍‍‍‍‍

From there I could see that the node had no properties, it was not a child node to any node, and didn't have any associations.
It did however have a childnode (webpreview). This to me are some remains from a transaction that has gone very wrong, and the post in alf_node can/must be deleted.
So I deleted the rows

delete from alf_node_assoc where source_node_id = 520521;
delete from alf_node where id = 520521‍‍

Then for the now orphan webpreview childnode

 update alf_node set NODE_DELETED = true where id = 530544;
‍‍

From there the indexing kicked in an now works.

[clouseau219]

This was not a cache issue, I had to remove the "no live" node directly from the database.

Solr is definitely exposing underlying issues on the database, I'm not sure if it would be best if Solr (like old lucene) moved past errors, so that indexing can continue, or halt like in this case. It went unnoticed for several days, so there was a very outdated index. But if indexing had moved on, I'm not sure an error in alfresco.log would have been spotted and subsequently fixed.

This is how I fixed it:
I ran these select statements to find out where the node id 520521 is present

select * from alf_child_assoc where child_node_id = 520521;
select * from alf_node_assoc where target_node_id = 520521;
select * from alf_node_assoc where source_node_id = 520521;
select * from alf_node_aspects where node_id = 520521;
select * from alf_node_properties where node_id = 520521;
select * from alf_node where id = 520521;‍‍‍‍‍‍

From there I could see that the node had no properties, it was not a child node to any node, and didn't have any associations.
It did however have a childnode (webpreview). This to me are some remains from a transaction that has gone very wrong, and the post in alf_node can/must be deleted.
So I deleted the rows

delete from alf_node_assoc where source_node_id = 520521;
delete from alf_node where id = 520521‍‍

Then for the now orphan webpreview childnode

 update alf_node set NODE_DELETED = true where id = 530544;
‍‍

From there the indexing kicked in an now works.

Hi,

I have the same problem with a "No live node exists" error.

17:49:05,383 ERROR [org.springframework.extensions.webscripts.AbstractRuntime] Exception from executeScript - redirecting to status template error: 05202404 Wrapped Exception (with status template): No live node exists:
ID: 7518
Cache row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
DB row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
org.springframework.extensions.webscripts.WebScriptException: 05202404 Wrapped Exception (with status template): No live node exists:
ID: 7518
Cache row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
DB row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:970)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:171)
at org.alfresco.repo.web.scripts.RepositoryContainer$2.execute(RepositoryContainer.java:393)
at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:462)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:500)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:275)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:372)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.springframework.dao.ConcurrencyFailureException: No live node exists:
ID: 7518
Cache row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
DB row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getNodeNotNull(AbstractNodeDAOImpl.java:981)
at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getParentAssocsCached(AbstractNodeDAOImpl.java:3586)
at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.prependPaths(AbstractNodeDAOImpl.java:3459)
at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.prependPaths(AbstractNodeDAOImpl.java:3575)
at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getPaths(AbstractNodeDAOImpl.java:3389)
at org.alfresco.repo.solr.SOLRTrackingComponentImpl.getNodesMetadata(SOLRTrackingComponentImpl.java:585)
at sun.reflect.GeneratedMethodAccessor484.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.alfresco.repo.management.subsystems.SubsystemProxyFactory$1.invoke(SubsystemProxyFactory.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy145.getNodesMetadata(Unknown Source)
at org.alfresco.repo.web.scripts.solr.NodesMetaDataGet.executeImpl(NodesMetaDataGet.java:181)
at org.springframework.extensions.webscripts.DeclarativeWebScript.executeImpl(DeclarativeWebScript.java:235)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:64)
… 25 more

So I found your solution, since I'm not a developer and a alfresco newbie I don't understand what you have done. Is it possible to explain me how I can solve the problem?

Thank you for your help

Greetings,
Alfred

[chrisokelly]

This was not a cache issue, I had to remove the "no live" node directly from the database….

Hi,

I have the same problem with a "No live node exists" error.

17:49:05,383 ERROR [org.springframework.extensions.webscripts.AbstractRuntime] Exception from executeScript - redirecting to status template error: 05202404 Wrapped Exception (with status template): No live node exists:…

So I found your solution, since I'm not a developer and a alfresco newbie I don't understand what you have done. Is it possible to explain me how I can solve the problem?

Thank you for your help

Greetings,
Alfred

Hi Alfred,

Not sure if you had solved this already or not, but we just encountered the same issue.
some background on the issue (skip this and go straight to the next paragraph if you just want how I fixed it)
It occurred when a user uploaded 47 large, high res images at once then deleted them almost immediately, having realized they were in the wrong folder (not sure if this is the "cause" exactly, but it was the situation). A restart of the server did not resolve the issue. I followed the steps Loftux posted, however, I had slightly different results, I found the node id from my error logs (5909) only in one table, alf_node. there was no child node. when I tried deleting that row however I got an error about constraints, being that the id was still referenced in alf_child_assocs. The issue was a row in alf_child_assocs which still referenced 5909 as a parent_node_id. I considered deleting this row, but I don't actually know enough about the inner workings of Alfresco's database to be comfortable deleting rows of my own accord.

Following some steps from the wiki, I ran the SOLR FIX action. As I hadn't used any of the SOLR urls before I first had to set up the certificate on my browser. I copied browser.p12 from (my directory structure, YMMV) /opt/alfresco-4.0.d/tomcat/webapps/alfresco/WEB-INF/classes/keystore/browser.p12 on the server to my desktop, then imported it into Firefox (better instructions on the wiki). I then navigated to https://our.alfresco.url:8443/solr/admin/cores?action=FIX in Firefox, accepted the untrusted certificate exception and waited a few minutes for the page to load. Once it had loaded it displayed:

<response><lst name="responseHeader"><int name="status">0</int><int name="QTime">48942</int></lst></response>‍

and the error messages in catalina.out stopped.

I hope this helps, or (better yet) that you had already solved this yourself.

Chris

[throwback]

Chris you sir are a leg-end. I had this exact same problem in the same circumstances- I added a ton of docs to alfresco then deleted them almost immediately (fortunately just a testing server) and then spent a hair raising half hour doing sql queries (along the lines above) to get it indexing again- good to know there is a more elegant solution next time! Guaranteed one of my users is going to do this at some point.

Kind regards,

Iain Bagnall

[dnallsopp]

I think the default location of the browser.p12 certificate should be:

<TOMCAT>/webapps/alfresco/WEB-INF/classes/alfresco/keystore/

not

<TOMCAT>/webapps/alfresco/WEB-INF/classes/keystore/

[dnallsopp]

Thanks very much - but how do you navigate to

https://our.alfresco.url:8443/solr/admin/cores?action=FIX ?

If I paste this URL into the browser then I get the response as you describe, but I can't see any way to navigate to this URL via the links in the Solr Admin web interface…

Or is this action 'hidden' and I just have to know the magic URL?

[chrisokelly]

There's a Solr Admin web interface? Sorry, pasting that URI into the browser was as deep into the SOLR pool as I have ventured so far, someone else might be able to help you though.

[dnallsopp]

Ah, OK! 

The Solr web admin interface is at https://localhost:8443/solr/

Maybe the links from the wiki are just 'magic' links rather than being navigable from the admin interface…