Indexing does not work

[ashwini]

Hello There ,

After configuring index.recovery.mode=FULL , reindexing hangs for a while and continue again with below exception.

2012-09-04 04:27:23,416  ERROR [index.AbstractReindexComponent.threads] [indexTrackerThread1] Reindexer   206 failed with error: Possible cache integrity issue during reindexing; nested exception is org.alfresco.service.cmr.repository.InvalidNodeRefException: Node does not exist: workspace://SpacesStore/16a0b417-6aca-4e03-96d2-97debcf93357(Status[changeTxnId=927d124a-41ca-4b1e-9d9b-afd0067edd56, dbTxnId=230497, deleted=true]).
 2012-09-04 04:27:23,418  WARN  [index.AbstractReindexComponent.threads] [indexTrackerThread1] org.springframework.dao.ConcurrencyFailureException: Possible cache integrity issue during reindexing; nested exception is org.alfresco.service.cmr.repository.InvalidNodeRefException: Node does not exist: workspace://SpacesStore/16a0b417-6aca-4e03-96d2-97debcf93357(Status[changeTxnId=927d124a-41ca-4b1e-9d9b-afd0067edd56, dbTxnId=230497, deleted=true])
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl$1.execute(AbstractLuceneIndexerImpl.java:720)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.doInReadthroughTransaction(AbstractLuceneIndexerImpl.java:702)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.readDocuments(ADMLuceneIndexerImpl.java:1980)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1044)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.GeneratedMethodAccessor337.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy215.flush(Unknown Source)
   at org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
Caused by: org.alfresco.service.cmr.repository.InvalidNodeRefException: Node does not exist: workspace://SpacesStore/16a0b417-6aca-4e03-96d2-97debcf93357(Status[changeTxnId=927d124a-41ca-4b1e-9d9b-afd0067edd56, dbTxnId=230497, deleted=true])
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getNodePairNotNull(DbNodeServiceImpl.java:176)
   at org.alfresco.repo.node.db.DbNodeServiceImpl.getPaths(DbNodeServiceImpl.java:2174)
   at sun.reflect.GeneratedMethodAccessor309.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
   at org.alfresco.repo.tenant.MultiTNodeServiceInterceptor.invoke(MultiTNodeServiceInterceptor.java:104)
   at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
   at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
   at $Proxy8.getPaths(Unknown Source)
   at sun.reflect.GeneratedMethodAccessor309.invoke(Unknown Source)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy9.getPaths(Unknown Source)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$3.doWork(ADMLuceneIndexerImpl.java:682)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$3.doWork(ADMLuceneIndexerImpl.java:668)
   at org.alfresco.repo.security.authentication.AuthenticationUtil.runAs(AuthenticationUtil.java:519)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocumentsImpl(ADMLuceneIndexerImpl.java:667)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocuments(ADMLuceneIndexerImpl.java:633)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:1985)
   at org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:1981)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl$1.execute(AbstractLuceneIndexerImpl.java:715)
   … 26 more‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Has anybody faced above issue ?

I have read somewhere that indexing is background process then why I can not access alfresco till it is not done with indexing?

Thanks in advance!!
Ashwini

[ashwini]

Thanks a lot amitha .

I tried deleting entry from alf_node and found foreign key constraint violation with alf_child_assoc error message. So I deleted first entry from alf_child_assoc and then from alf_node. And started server and found another exception :

  2012-09-11 11:09:31,603  ERROR
[index.AbstractReindexComponent.threads] [indexTrackerThread1]  
Reindexer   206 failed with error: null.
  2012-09-11 11:09:31,604  WARN   
[index.AbstractReindexComponent.threads] [indexTrackerThread1] java.lang.NullPointerException
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocumentsImpl(ADMLuceneIndexerImpl.java:803)
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocuments(ADMLuceneIndexerImpl.java:633)
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:1985)
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:1981)
   at
org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl$1.execute(AbstractLuceneIndexerImpl.java:715)
   at
org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at
org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.doInReadthroughTransaction(AbstractLuceneIndexerImpl.java:702)
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.readDocuments(ADMLuceneIndexerImpl.java:1980)
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1044)
   at
org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at
org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy215.flush(Unknown Source)
   at
org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at
org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at
org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at
org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at
org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at
org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at
org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at
org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at
org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at
org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at
org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

before some days even I tried to delete the entry only from alf_child_assoc and got same exception. Do you have any idea why I am getting this exception.

Regards,
Ashwini

[amitabhandari1]

Hi Ashwini,

Did you try to  start your server with AUTO/Validate mode . If not , can you try that .
If server start up , you can  try indexing from web scripts. There are many web-scripts available for performing index.

Thanks,

[ashwini]

Thanks Amita,

Now after removing some entries from database
1. I am not getting any exception when cleanup scheduler run(which runs everyday at 21:00)
2.  indexing completes in around 20-30 min (which was taking before more than 16 hours only for 50000 transactions) with some exceptions like :

  2012-09-11 11:09:31,603  ERROR
[index.AbstractReindexComponent.threads] [indexTrackerThread1]  
Reindexer   206 failed with error: null.
  2012-09-11 11:09:31,604  WARN   
[index.AbstractReindexComponent.threads] [indexTrackerThread1] java.lang.NullPointerException
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocumentsImpl(ADMLuceneIndexerImpl.java:803)
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.createDocuments(ADMLuceneIndexerImpl.java:633)
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:1985)
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl$4.execute(ADMLuceneIndexerImpl.java:1981)
   at
org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl$1.execute(AbstractLuceneIndexerImpl.java:715)
   at
org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:388)
   at
org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerImpl.doInReadthroughTransaction(AbstractLuceneIndexerImpl.java:702)
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.readDocuments(ADMLuceneIndexerImpl.java:1980)
   at
org.alfresco.repo.search.impl.lucene.ADMLuceneIndexerImpl.flushPending(ADMLuceneIndexerImpl.java:1044)
   at
org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.flush(AbstractLuceneIndexerAndSearcherFactory.java:874)
   at sun.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at
org.alfresco.repo.service.StoreRedirectorProxyFactory$RedirectorInvocationHandler.invoke(StoreRedirectorProxyFactory.java:215)
   at $Proxy215.flush(Unknown Source)
   at
org.alfresco.repo.search.IndexerComponent.flushPending(IndexerComponent.java:133)
   at
org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.beforeCommit(AbstractReindexComponent.java:1086)
   at
org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:747)
   at
org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.doBeforeCommit(AlfrescoTransactionSupport.java:727)
   at
org.alfresco.repo.transaction.AlfrescoTransactionSupport$TransactionSynchronizationImpl.beforeCommit(AlfrescoTransactionSupport.java:687)
   at
org.springframework.transaction.support.TransactionSynchronizationUtils.triggerBeforeCommit(TransactionSynchronizationUtils.java:95)
   at
org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerBeforeCommit(AbstractPlatformTransactionManager.java:927)
   at
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:737)
   at
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
   at
org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
   at
org.alfresco.util.transaction.SpringAwareUserTransaction.commit(SpringAwareUserTransaction.java:472)
   at
org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:410)
   at
org.alfresco.repo.node.index.AbstractReindexComponent$ReindexWorkerRunnable.run(AbstractReindexComponent.java:1008)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
   at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
   at java.lang.Thread.run(Thread.java:662)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

And then , when I restart my server with auto mode, in works fine.

Could you please tell me the web script which I can use for indexing.


Thanks a lot for your help.

[amitabhandari1]

Hi Ashwini,

Please check the following webscript:
http://<server-name>/alfresco/service/index/package/org/alfresco/enterprise/repository/admin/indexch...

Thanks,

[ashwini]

Hi Amita,

Please check the following webscript:
http://<server-name>/alfresco/service/index/package/org/alfresco/enterprise/repository/admin/indexcheck

  I have looked for this before. Unfortunately we are not using enterprise edition. Do we have something for community edition.

Thanks ,
Ashwini

[shagan12]

I fixed this issue in 4.2b by opening table alf_activity_post and deleted all rows with status=Error and Pending

[dranakan]

How can I get the node with problems ? (to manually delete records in DB)

I have done a FULL reindex but I don't get the node.

I have setted the log4j to warn (log4j.logger.org.alfresco.repo.node.cleanup=DEBUG), enable the profile (in alfresco-global.properties : db.url=jdbc:mysql://…&profileSQL=true) but I never get the node id.

I ever get this bug and I have done a FULL indexe, get the node and delete in the DB. This time I have not details when I do the FUL reindexe.
I have change scheduler (WEB-INF/classes/alfresco/scheduled-jobs-context.xml, bean "nodeServiceCleanupTrigger").

In my situation, deleting all rows (error and pending) in alf_activity_post has no effect.

[dranakan]

instead of doing the full reindex, it's faster to do this :

Find the nodes with problems : 

SELECT alf_node.id FROM alf_node INNER JOIN alf_child_assoc ON alf_node.id = alf_child_assoc.parent_node_id and alf_node.node_deleted = 1;
‍‍‍

Delete them :

delete from alf_child_assoc where parent_node_id in (1234,5678,…);
‍‍‍

After that, the cleaner can do its job (delete from alf_node where node_deleted = 1 and transaction_id <= (select max(txn.id) from alf_transaction txn where txn.commit_time_ms < 1368857400033))

[heiko_robert]

Hi dranakan,

you shouldn't delete the parent nodes otherwise you have lots of lost nodes without parents and then will get trouble somewhen later and you have to find the lost nodes later!
The parents shoudn't be touched at all because they are just a maker for the DeletedNodeCleanupWorker.

I didn't find the source of the problem but it's allway related to webpreview, doclib. So it's maybe a problem in the removal of the hidden nodes? Maybe AndyH/DavidW reads this and gives a hint? The problem occurs not on a 4.0 system only it continues on a migrated system running 4.2.c.

Anyway - here is the way I just did it on a 4.2.c system (where deleted flag is moved to a doc type):

drop table if exists alf_remove_nodes;
create temporary table alf_remove_nodes as
    select alf_child_assoc.parent_node_id, 
           alf_child_assoc.child_node_id
    from alf_qname, alf_node, alf_child_assoc 
    where
    alf_node.id = alf_child_assoc.parent_node_id and 
    alf_node.type_qname_id = alf_qname.id and
    alf_qname.local_name = 'deleted'
;
select * from alf_remove_nodes;

– finally remove references and then the node itself
delete from alf_node_properties where node_id in (select child_node_id from alf_remove_nodes);
delete from alf_node_aspects where node_id in (select child_node_id from alf_remove_nodes);
delete from alf_child_assoc where child_node_id in (select child_node_id from alf_remove_nodes);
delete from alf_node where id in (select child_node_id from alf_remove_nodes);
– cleanup
drop table if exists alf_remove_nodes;
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[lukasz_turakiew]

Hello,

To find and clean nodes (and its aspects, properties, …) without parents (except root nodes) we can use the following queries (Alfresco 4.0.d, MySQL):

DROP TEMPORARY TABLE IF EXISTS alf_clean_nodes;

CREATE temporary TABLE IF NOT EXISTS alf_clean_nodes (
    select id from alf_node
        where 
            alf_node.type_qname_id != (select id from alf_qname where local_name='store_root')
        and 
            alf_node.node_deleted = false
        and
            alf_node.id not in(select child_node_id from alf_child_assoc)
);

update alf_content_url set orphan_time = (UNIX_TIMESTAMP() * 1000) 
    where id in (
        select content_url_id from alf_content_data
            where id in (
                select long_value from alf_node_properties
                    where
                        (actual_type_n = 3 or actual_type_n = 21)
                    and
                        node_id in (select id from alf_clean_nodes)
            )
    );
            
delete from alf_content_data
    where id in (
        select long_value from alf_node_properties
            where
                node_id in (select id from alf_clean_nodes)
            and
               (qname_id in (select id from alf_qname where local_name='content'))
    );

delete from alf_usage_delta where node_id in (select id from alf_clean_nodes);

delete from alf_node_assoc where source_node_id in (select id from alf_clean_nodes);
delete from alf_node_assoc where target_node_id in (select id from alf_clean_nodes);

delete from alf_child_assoc where parent_node_id in (select id from alf_clean_nodes);
delete from alf_child_assoc where child_node_id in (select id from alf_clean_nodes);

delete from alf_node_aspects where node_id in (select id from alf_clean_nodes);

delete from alf_node_properties where node_id in (select id from alf_clean_nodes);

delete from alf_node where id in (select id from alf_clean_nodes);

DROP TEMPORARY TABLE IF EXISTS alf_clean_nodes;
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Regards,
Lukasz Turakiewicz