topic Re: No live node exists - error halts solr indexing in Alfresco Archive

No live node exists - error halts solr indexing

loftux — Wed, 15 Feb 2012 07:35:40 GMT

Using 4.0.d on postgresql indexing with Solr, I see this error in alfresco.log2012-02-15 08:11:50,393 ERROR [extensions.webscripts.AbstractRuntime] [http-8443-4] Exception from executeScript - redirecting to status template error: 01156612 Wrapped Exception (with status template): No live node exis

Re: No live node exists - error halts solr indexing

andy — Mon, 20 Feb 2012 11:24:22 GMT

Hi

I believe this is a cache issue and a restart shuld fix it.

SOLR tracking should not interact with the cache.
Some of this has been fixed recently.

SOLR would retry until it gets pas the issue (it does not skip stuff and try later)
It may do in the future ….

Andy

Re: No live node exists - error halts solr indexing

loftux — Wed, 22 Feb 2012 07:58:24 GMT

This was not a cache issue, I had to remove the "no live" node directly from the database.

Solr is definitely exposing underlying issues on the database, I'm not sure if it would be best if Solr (like old lucene) moved past errors, so that indexing can continue, or halt like in this case. It went unnoticed for several days, so there was a very outdated index. But if indexing had moved on, I'm not sure an error in alfresco.log would have been spotted and subsequently fixed.

This is how I fixed it:
I ran these select statements to find out where the node id 520521 is present

select * from alf_child_assoc where child_node_id = 520521;
select * from alf_node_assoc where target_node_id = 520521;
select * from alf_node_assoc where source_node_id = 520521;
select * from alf_node_aspects where node_id = 520521;
select * from alf_node_properties where node_id = 520521;
select * from alf_node where id = 520521;‍‍‍‍‍‍

From there I could see that the node had no properties, it was not a child node to any node, and didn't have any associations.
It did however have a childnode (webpreview). This to me are some remains from a transaction that has gone very wrong, and the post in alf_node can/must be deleted.
So I deleted the rows

delete from alf_node_assoc where source_node_id = 520521;
delete from alf_node where id = 520521‍‍

Then for the now orphan webpreview childnode

 update alf_node set NODE_DELETED = true where id = 530544;
‍‍

From there the indexing kicked in an now works.

Re: No live node exists - error halts solr indexing

clouseau219 — Fri, 22 Jun 2012 07:13:22 GMT

This was not a cache issue, I had to remove the "no live" node directly from the database.

Solr is definitely exposing underlying issues on the database, I'm not sure if it would be best if Solr (like old lucene) moved past errors, so that indexing can continue, or halt like in this case. It went unnoticed for several days, so there was a very outdated index. But if indexing had moved on, I'm not sure an error in alfresco.log would have been spotted and subsequently fixed.

This is how I fixed it:
I ran these select statements to find out where the node id 520521 is present
select * from alf_child_assoc where child_node_id = 520521;
select * from alf_node_assoc where target_node_id = 520521;
select * from alf_node_assoc where source_node_id = 520521;
select * from alf_node_aspects where node_id = 520521;
select * from alf_node_properties where node_id = 520521;
select * from alf_node where id = 520521;‍‍‍‍‍‍
From there I could see that the node had no properties, it was not a child node to any node, and didn't have any associations.
It did however have a childnode (webpreview). This to me are some remains from a transaction that has gone very wrong, and the post in alf_node can/must be deleted.
So I deleted the rows
delete from alf_node_assoc where source_node_id = 520521;
delete from alf_node where id = 520521‍‍
Then for the now orphan webpreview childnode
 update alf_node set NODE_DELETED = true where id = 530544;
‍‍
From there the indexing kicked in an now works.

Hi,

I have the same problem with a "No live node exists" error.

17:49:05,383 ERROR [org.springframework.extensions.webscripts.AbstractRuntime] Exception from executeScript - redirecting to status template error: 05202404 Wrapped Exception (with status template): No live node exists:
ID: 7518
Cache row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
DB row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
org.springframework.extensions.webscripts.WebScriptException: 05202404 Wrapped Exception (with status template): No live node exists:
ID: 7518
Cache row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
DB row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:970)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:171)
at org.alfresco.repo.web.scripts.RepositoryContainer$2.execute(RepositoryContainer.java:393)
at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:462)
at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:500)
at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:275)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:372)
at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:563)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.springframework.dao.ConcurrencyFailureException: No live node exists:
ID: 7518
Cache row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
DB row: NodeEntity[ ID=7518, version=9, store=workspace://SpacesStore, uuid=6e0c356d-6a34-4e88-b69a-d354c4710b14, typeQNameId=32, localeId=1, aclId=null, deleted=true, transaction=TransactionEntity[ ID=19926, server=null, changeTxnId=e9b2f873-b0ae-47aa-a3a8-2886a355e72c, commitTimeMs=null], auditProps=AuditablePropertiesEntity[ auditCreator=haa, auditCreated=2012-03-30T10:01:19.942+02:00, auditModifier=haa, auditModified=2012-05-30T11:12:53.304+02:00]]
at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getNodeNotNull(AbstractNodeDAOImpl.java:981)
at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getParentAssocsCached(AbstractNodeDAOImpl.java:3586)
at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.prependPaths(AbstractNodeDAOImpl.java:3459)
at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.prependPaths(AbstractNodeDAOImpl.java:3575)
at org.alfresco.repo.domain.node.AbstractNodeDAOImpl.getPaths(AbstractNodeDAOImpl.java:3389)
at org.alfresco.repo.solr.SOLRTrackingComponentImpl.getNodesMetadata(SOLRTrackingComponentImpl.java:585)
at sun.reflect.GeneratedMethodAccessor484.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.alfresco.repo.management.subsystems.SubsystemProxyFactory$1.invoke(SubsystemProxyFactory.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy145.getNodesMetadata(Unknown Source)
at org.alfresco.repo.web.scripts.solr.NodesMetaDataGet.executeImpl(NodesMetaDataGet.java:181)
at org.springframework.extensions.webscripts.DeclarativeWebScript.executeImpl(DeclarativeWebScript.java:235)
at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:64)
… 25 more

So I found your solution, since I'm not a developer and a alfresco newbie I don't understand what you have done. Is it possible to explain me how I can solve the problem?

Thank you for your help

Greetings,
Alfred

Re: No live node exists - error halts solr indexing

chrisokelly — Wed, 04 Jul 2012 23:56:13 GMT

This was not a cache issue, I had to remove the "no live" node directly from the database….

Hi,

I have the same problem with a "No live node exists" error.

17:49:05,383 ERROR [org.springframework.extensions.webscripts.AbstractRuntime] Exception from executeScript - redirecting to status template error: 05202404 Wrapped Exception (with status template): No live node exists:…

So I found your solution, since I'm not a developer and a alfresco newbie I don't understand what you have done. Is it possible to explain me how I can solve the problem?

Thank you for your help

Greetings,
Alfred

Hi Alfred,

Not sure if you had solved this already or not, but we just encountered the same issue.
some background on the issue (skip this and go straight to the next paragraph if you just want how I fixed it)
It occurred when a user uploaded 47 large, high res images at once then deleted them almost immediately, having realized they were in the wrong folder (not sure if this is the "cause" exactly, but it was the situation). A restart of the server did not resolve the issue. I followed the steps Loftux posted, however, I had slightly different results, I found the node id from my error logs (5909) only in one table, alf_node. there was no child node. when I tried deleting that row however I got an error about constraints, being that the id was still referenced in alf_child_assocs. The issue was a row in alf_child_assocs which still referenced 5909 as a parent_node_id. I considered deleting this row, but I don't actually know enough about the inner workings of Alfresco's database to be comfortable deleting rows of my own accord.

Following some steps from the wiki, I ran the SOLR FIX action. As I hadn't used any of the SOLR urls before I first had to set up the certificate on my browser. I copied browser.p12 from (my directory structure, YMMV) /opt/alfresco-4.0.d/tomcat/webapps/alfresco/WEB-INF/classes/keystore/browser.p12 on the server to my desktop, then imported it into Firefox (better instructions on the wiki). I then navigated to https://our.alfresco.url:8443/solr/admin/cores?action=FIX in Firefox, accepted the untrusted certificate exception and waited a few minutes for the page to load. Once it had loaded it displayed:

<response><lst name="responseHeader"><int name="status">0</int><int name="QTime">48942</int></lst></response>‍

and the error messages in catalina.out stopped.

I hope this helps, or (better yet) that you had already solved this yourself.

Chris

Re: No live node exists - error halts solr indexing

throwback — Mon, 09 Jul 2012 13:53:31 GMT

Chris you sir are a leg-end. I had this exact same problem in the same circumstances- I added a ton of docs to alfresco then deleted them almost immediately (fortunately just a testing server) and then spent a hair raising half hour doing sql queries (along the lines above) to get it indexing again- good to know there is a more elegant solution next time! Guaranteed one of my users is going to do this at some point.

Kind regards,

Iain Bagnall

Re: No live node exists - error halts solr indexing

dnallsopp — Thu, 02 Aug 2012 13:17:24 GMT

I think the default location of the browser.p12 certificate should be:

<TOMCAT>/webapps/alfresco/WEB-INF/classes/alfresco/keystore/

not

<TOMCAT>/webapps/alfresco/WEB-INF/classes/keystore/

Re: No live node exists - error halts solr indexing

dnallsopp — Thu, 02 Aug 2012 13:33:43 GMT

Thanks very much - but how do you navigate to

https://our.alfresco.url:8443/solr/admin/cores?action=FIX ?

If I paste this URL into the browser then I get the response as you describe, but I can't see any way to navigate to this URL via the links in the Solr Admin web interface…

Or is this action 'hidden' and I just have to know the magic URL?

Re: No live node exists - error halts solr indexing

chrisokelly — Thu, 02 Aug 2012 22:34:59 GMT

There's a Solr Admin web interface? Sorry, pasting that URI into the browser was as deep into the SOLR pool as I have ventured so far, someone else might be able to help you though.

Re: No live node exists - error halts solr indexing

dnallsopp — Mon, 06 Aug 2012 13:08:41 GMT

Ah, OK!

The Solr web admin interface is at https://localhost:8443/solr/

Maybe the links from the wiki are just 'magic' links rather than being navigable from the admin interface…

Re: No live node exists - error halts solr indexing

andy — Tue, 07 Aug 2012 19:37:21 GMT

Hi

In the current enterprise versions:
1) the cause fixed
2) SOLR skips such errors
3) SOLR records any errors (it does not fail the whole TX - just some nodes)
4) Errors can be found and nodes retried

FIX will soon include this

These fixes will be in the next community release (no I do not know when)

We did intitially fail hard so issues were not passed by ….

Andy

Re: No live node exists - error halts solr indexing

pavelm — Wed, 28 Nov 2012 10:25:58 GMT

So I deleted the rows
delete from alf_node_assoc where source_node_id = 520521;
delete from alf_node where id = 520521‍‍
Then for the now orphan webpreview childnode
 update alf_node set NODE_DELETED = true where id = 530544;
‍‍
From there the indexing kicked in an now works.

I don't delete parent node. NODE_DELETED on parent is true.
Enough update child node (thumbnail) alf_node set NODE_DELETED = true where id = 530544;
It works for me.

Re: No live node exists - error halts solr indexing

yogeshnehra — Fri, 28 Dec 2012 16:03:13 GMT

This issue is related to solr, When solr create indexes, Solr make connection with alfresco to get all transactions to create index, By default it ask for all the transactions and alfresco reply with all transaction ids. If some node is deleted, its details go to archive folder. but if there are some reference remain for deleted node then scheduler will unable to clean it and throw this exception. To solve this issue just delete that transaction id. But when you try to delete transaction id from alf_transaction table, it always throw foreign key constraints. To resolve this just take database backup and drop below foreign key constraints and recreate them with cascade effect.

(`alfresco400dev`.`alf_node`, CONSTRAINT `fk_alf_node_txn` FOREIGN KEY (`transaction_id`) REFERENCES `alf_transaction` (`id`))

ALTER TABLE alf_node DROP CONSTRAINT fk_alf_node_txn;

ALTER TABLE alf_node
    ADD CONSTRAINT fk_alf_node_txn FOREIGN KEY (transaction_id)
    REFERENCES alf_node (id)
    ON DELETE CASCADE

(`alfresco400dev`.`alf_child_assoc`, CONSTRAINT `fk_alf_cass_cnode` FOREIGN KEY (`child_node_id`) REFERENCES `alf_node` (`id`))

ALTER TABLE alf_child_assoc DROP CONSTRAINT fk_alf_cass_cnode;

ALTER TABLE alf_child_assoc
    ADD CONSTRAINT fk_alf_cass_cnode FOREIGN KEY (child_node_id)
    REFERENCES alf_node (id)
    ON DELETE CASCADE

(`alfresco400dev`.`alf_node_aspects`, CONSTRAINT `fk_alf_nasp_n` FOREIGN KEY (`node_id`) REFERENCES `alf_node` (`id`))

ALTER TABLE alf_node_aspects DROP CONSTRAINT fk_alf_nasp_n;

ALTER TABLE alf_node_aspects
    ADD CONSTRAINT fk_alf_nasp_n FOREIGN KEY (node_id)
    REFERENCES alf_node (id)
    ON DELETE CASCADE

(`alfresco400dev`.`alf_node_properties`, CONSTRAINT `fk_alf_nprop_n` FOREIGN KEY (`node_id`) REFERENCES `alf_node` (`id`))

ALTER TABLE alf_node_properties DROP CONSTRAINT fk_alf_nprop_n;

ALTER TABLE alf_node_properties
    ADD CONSTRAINT fk_alf_nprop_n FOREIGN KEY (node_id)
    REFERENCES alf_node (id)
    ON DELETE CASCADE

delete from alf_transaction where id = 42057

then configure alfresco for not to provide deleted node for indexing by putting below property -

index.tracking.minRecordPurgeAgeDays=0

Now on solr machine call this url -

http://solrhostname:8080/solr/admin/cores?action=FIX

Now delete solr indexes and recreate them.

Now your problem will disappear (it worked for me).

Re: No live node exists - error halts solr indexing

yogeshnehra — Fri, 28 Dec 2012 16:05:21 GMT

Forgot to mention - Transaction Id used in above reply can be get from exception see bold text below -

NodeEntity[ ID=520521, version=16, store=workspace://SpacesStore, uuid=64b01e12-dfac-4b22-96d0-bfa30ba7d34e, typeQNameId=32, localeId=15, aclId=null, deleted=true, transaction=TransactionEntity[ ID=1064115, server=null, changeTxnId=f4359d12-f144-43c7-8c17-18697c4eb864, commitTimeMs=null]

Re: No live node exists - error halts solr indexing

ltardioli — Wed, 02 Jan 2013 12:46:48 GMT

The chrisokelly's solution works like a charm here!
Thank you!

Re: No live node exists - error halts solr indexing

yanpak — Fri, 22 Feb 2013 10:10:00 GMT

Chrisokelly, many thanks to you! You save my day and… You are man! Certainly!
Thank you!