Lucene Index not getting updated properly outside the cluster

[somil]

HI
We are using alfresco enterprise 4.1.5 in our project. We just upgraded from alfresco 3.4 to 4.1, and since then we are facing an issue with the lucene indexing.
We have three alfresco nodes out of which two are residing inside the cluster and one outside of it.
When ever we are creating a document inside the cluster, the document is not searchable through the lucene query in the third node which lies outside the cluster, even though we can see the new document on the alfresco UI on that node.
When i do an index check for the node I get the following message on the screen:
<blockquote>Alfresco ADM Index Check - transaction (txnId=9284220) is OUT-OF-SYNC with local indexes</blockquote>
Does anybody here know how to fix this issue?

PS. We are not using SOLR tracking as of yet.

[kaynezhang]

This might be caused by more server clock in accurately,Please check the thrid node in your cluster to make sure it si  time-synchronized with the other two
For recovery ,have you tried set

 index.recovery.mode=AUTO‍

in the third node and restart it.

[somil]

the third node is out of the cluster, and the index.recovery.mode=AUTO is already set.

[somil]

Hi, Do you have idea how to solve this issue, syncing indexes in the nodes which are not in cluster?

[mrogers]

Are you actually "syncing" the index (like you would with backup and restore with a tool like rsync) from the clustered index somehow or or trying to build a separate instance on some remote non clustered instance.

In general I'd expect the UI to have problems but lucene to be O.K.   You seem to have it the other way around which is strange.

But lets be clear you have a "broken cluster" so there will be issues and problems.

[somil]

It is a separate instance out of the cluster. The lucene index is rebuilt when I restart the node.
It was working fine before the upgrade to alfresco 4.1.5. We had alfresco 3.4.9 previously and this issue was not present there.

Initially in the UI as well we had seen the stale node issue but then we fixed it by invalidating the nodeCache in the ehcache-custom.xml.

Not sure what is a "broken cluster". Can you elaborate more on this?

[thanak]

Hi Somil,
I'm using the similar architecture - 2 nodes in cluster and 3rd node outside for rebuilding indexes. The fact that 3rd node is not in the cluster (EHCache is not synchronized) means you have a broken cluster. With version 3.x it works ok, but after we upgraded to 4.x some issues appeared with reindexation and DB data. Maybe you will face it as I do, but it depends on environment and Alfresco release so maybe you will be lucky.

To your trouble - if you have a 3rd node outside of cluster you need:
1) Completely disable EHCache on this 3rd node, because you need to use data directly from DB. Nodes in cluster inform each other about changes, but your 3rd node is outside of cluster and it's not receiving these infos.

Try to add this JVM parameter to your app. server:
-Dnet.sf.ehcache.disabled=true

You can verify if EHCache is enabled from log file or do a heapdump - You will see if EHCache is using some heap or not.

2) 3rd node should be just readonly. You need to change these properties:
server.transaction.mode.default=PROPAGATION_REQUIRED, readOnly
server.transaction.allow-writes=false

[somil]

Thanks Tomas for the suggestion. But after applying your suggested configuration, now the server is failing to load at the startup itself. It is throwing the following exception
<blockcode> java.lang.IllegalStateException: dictionaryDAO.init() called, yet no namespace registry for domain

[thanak]

I see, I'm using version 4.0.2.44 ENT but in your release 4.1.5 (if you are still using it) is a change in NamespaceDAOImpl which causing that exception with EHCache disabled.

I attached the NamespaceDAOImpl from 4.0.2.44 so you can try to figure out how to solve this exception. I should also note that in our case we had to upgrade to latest service pack for our release to make 3rd node reindexation works so I'm suggesting you to check if there isn't a newer SP for your release.