Full Reindex stuck on 30%

[leonk]

Hello,

I'm trying to rebuild indexes in 3.4.d version (upgraded from 3.3.g).

1) So I've set:
index.recovery.mode=FULL

in alfresco-global.properties

2)
deleted contents of dir.indexes

3) Increased org.alfresco.cache.node.aspectsTransactionalCache value (cause I was getting messages that it's full during rebuild)
# diff -Naur /srv/home/lkolchin/cache-context.xml.orig /srv/alfresco/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/cache-context.xml
— /srv/home/lkolchin/cache-context.xml.orig   2011-03-25 15:51:10.000000000 +1100
+++ /srv/alfresco/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/cache-context.xml    2011-03-25 15:51:51.000000000 +1100
@@ -210,7 +210,7 @@
          <value>org.alfresco.cache.node.aspectsTransactionalCache</value>
       </property>
       <property name="maxCacheSize">
-         <value>10000</value>
+         <value>500000</value>
       </property>
    </bean>

4) and started Alfresco.
I'm getting this in the log (rebuild stiops at 30%)

10:25:56,322  INFO  [alfresco.config.FixedPropertyPlaceholderConfigurer] Loading properties file from class path resource [alfresco/domain/cache-strategies.properties]
10:25:56,323  INFO  [alfresco.config.FixedPropertyPlaceholderConfigurer] Loading properties file from class path resource [alfresco/module/org.alfresco.module.vti/context/vti.properties]
10:25:56,323  INFO  [alfresco.config.FixedPropertyPlaceholderConfigurer] Loading properties file from URL [file:/srv/alfresco/tomcat/shared/classes/alfresco/extension/custom-vti.properties]
10:25:57,973  WARN  [alfresco.util.OpenOfficeConnectionTester] An initial OpenOffice connection could not be established.
10:25:58,000  INFO  [management.subsystems.ChildApplicationContextFactory] Startup of 'OOoDirect' subsystem, ID: [OOoDirect, default] complete
10:25:58,003  DEBUG [node.index.FullIndexRecoveryComponent] Performing index recovery for type: FULL
10:25:59,268  INFO  [node.index.FullIndexRecoveryComponent] Index recovery started: 46,405 transactions.
10:29:21,745  INFO  [node.index.FullIndexRecoveryComponent]     10 % complete.
10:32:32,317  INFO  [node.index.FullIndexRecoveryComponent]     20 % complete.
10:35:05,364  INFO  [node.index.FullIndexRecoveryComponent]     30 % complete.
‍‍‍‍‍‍‍‍‍‍‍‍

Any suggestion/resolution?

Cheers,
Leon

[loftux]

Did it make it to 100% when the transactional cache had its original value? If so, change it back.
What was the transactional message? If it was a WARN, then it is just that, and can be ignored.

And as an advice, rename the old index directory rather than delete. Alfresco will create a new index directory. That way you can restore your old index should anything go wrong. Once you have a new working index, thats the time to delete the old index.

[leonk]

Did it make it to 100% when the transactional cache had its original value? If so, change it back.
What was the transactional message? If it was a WARN, then it is just that, and can be ignored.

And as an advice, rename the old index directory rather than delete. Alfresco will create a new index directory. That way you can restore your old index should anything go wrong. Once you have a new working index, thats the time to delete the old index.

Hi,

It never made it to 100% in both cases
And yes I did see that it's a warning but changing org.alfresco.cache.node.aspectsTransactionalCache didn't help.

I have this index directory backed up, so getting back is not a problem.

The problem is that it gets stuck ;(

Tomorrow I'll try to rename index dir. and report here.

[andy]

Hi

The version you are using does not include a bug fix for reindexing transactions that contain very large cascade deletes.
The delete from the UI will cascade delete optimally but the index rebuild will not.

Is this likely to be your issue?

Andy

[dranakan]

Hello Andy,

The version you are using does not include a bug fix for reindexing transactions that contain very large cascade deletes.

We can not rebuild (and restore) an Alfresco 3.4d if some files have been deleted ?

[loftux]

I found this entry
http://issues.alfresco.com/jira/browse/ALF-5166
Maybe thats the one Andy is referring to.

I to would like to know the answer to this, because it is a really serious matter if you can in fact break a repository completely if you hit this bug.

Can you work around it by emptying/purging all deleted items before doing reindex?

[leonk]

I reckon Andy and Loftux are right.
The thing is that previously our Alfresco was configured to syncronize users from our AD (~180,000 users) and it's created 180,000 User Homes. Apparently Alfresco has some issues with such large number of User Homes.

So I've reconfigured Alfresco to Authenticate (without Sync.) users only and deleted all empty directories with the following script (courtesy of John Jaquette):

var childList = companyhome.childByNamePath("User Homes").children;
for (var i=0; i<childList.length; i++)
{
   var child = childList[i];
   logger.log(child.name);
   var subchildren = child.children;
  logger.log(subchildren.length);
  if (subchildren.length==0){
     child.remove();
     }
}
‍‍‍‍‍‍‍‍‍‍‍‍‍

If this JIRA is right - http://issues.alfresco.com/jira/browse/ALF-5166

Why won't they attach a fixed .jar or .xml file?

P.S.:
Andy may be you can supply us with fixed components as this issue is fixed for Enterprise ver. (I presume it was tested first on Community nightly build)

[andy]

Hi

The issue only affects very large transactions and particularly those with cascade deletes that are optimised at delete time but not when the index is rebuilt.

There were several parts to the fix.

The issue is fixed in the Enterprise product. It will be in the community release after 3.4.
The easiest fix is via support.

If you are doing a full index rebuild you could delete "delete only transactions" to try and work around the problem.

Better to configure the repo not to check the index and set

index.tracking.minRecordPurgeAgeDays=0‍

and wait for the clean up to run.
It will take out deleted nodes and and empty transactions.
Then set tthe configuration back rebuild your index.

See node-services-context.xml

Andy

[leonk]

Hello Andy,

I've added
index.recovery.mode=NONE
index.tracking.minRecordPurgeAgeDays=0

to alfresco-global.properties (Is that what you meant?) files and restarted Alfresco.

Can you tell me when this "cleanup" will run?
Is there a way to trigger this cleanup manually?
How do I know that it's finished and I can set configs back and rebuild the index?

Update:
On the restart I'm getting lots of those warnings:
Alfresco interface gives me the following message:

WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
WARN : org.alfresco.wcm.client.util.impl.GuestSessionFactoryImpl - WQS unable to connect to repository: Not Found
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I'm unable to connect via Share interface.
Alfresco interface gives me the following message:

HTTP Status 404 -

type Status report

message

description The requested resource () is not available.
Apache Tomcat/6.0.26
‍‍‍‍‍‍‍‍‍‍

Thanks,
Leon