Indexes invalid on every restart

[insumo]

Hi All,

We have an Alfresco Labs3 system that has been running without a problem for a couple of months, but now fails to start up unless a full reindex is performed.

After performing the full reindex, the system will then run as normal, until shut down. On restart, Alfresco will fail to start unless a full reindex is again performed.

12:33:27,760 User:System WARN  [repo.admin.ConfigurationChecker] Starting bootstrap configuration check: ConfigurationChecker[indexRecoveryMode=VALIDATE]
12:33:27,761 User:System INFO  [repo.admin.ConfigurationChecker] The Alfresco root data directory ('dir.root') is: /opt/apps/alfresco/alf_data
12:33:27,773 User:System DEBUG [repo.admin.ConfigurationChecker] Checking index for store: user://alfrescoUserStore
12:33:27,774 User:System DEBUG [repo.admin.ConfigurationChecker] Index missing for store: 
   store: user://alfrescoUserStore
12:33:27,775 User:System DEBUG [repo.admin.ConfigurationChecker] Checking index for store: system://system
12:33:27,856 User:System DEBUG [repo.admin.ConfigurationChecker] Index missing for store: 
   store: system://system
12:33:27,858 User:System DEBUG [repo.admin.ConfigurationChecker] Checking index for store: workspace://lightWeightVersionStore
12:33:27,859 User:System DEBUG [repo.admin.ConfigurationChecker] Index missing for store: 
   store: workspace://lightWeightVersionStore
12:33:27,861 User:System DEBUG [repo.admin.ConfigurationChecker] Checking index for store: workspace://version2Store
12:33:27,862 User:System DEBUG [repo.admin.ConfigurationChecker] Index missing for store: 
   store: workspace://version2Store
12:33:27,863 User:System DEBUG [repo.admin.ConfigurationChecker] Checking index for store: archive://SpacesStore
12:33:27,864 User:System DEBUG [repo.admin.ConfigurationChecker] Index missing for store: 
   store: archive://SpacesStore
12:33:27,866 User:System DEBUG [repo.admin.ConfigurationChecker] Checking index for store: workspace://SpacesStore
12:33:27,867 User:System DEBUG [repo.admin.ConfigurationChecker] Index missing for store: 
   store: workspace://SpacesStore
12:33:27,873 User:System DEBUG [repo.admin.ConfigurationChecker] Checking index for store: avm://sitestore
12:33:27,874 User:System DEBUG [repo.admin.ConfigurationChecker] Index missing for store: 
   store: avm://sitestore
12:33:27,937 User:System ERROR [repo.admin.ConfigurationChecker] CONTENT INTEGRITY ERROR: Indexes not found for 7 stores.
12:33:27,938 User:System INFO  [repo.admin.ConfigurationChecker] You may set 'index.recovery.mode=FULL' if you need to rebuild the indexes.
12:33:27,938 User:System ERROR [repo.admin.ConfigurationChecker] Check config failed… missingStoreIndex:7 versionPropertiesContentAvailable:true
12:33:27,938 User:System ERROR [repo.admin.ConfigurationChecker] Ensure that the 'dir.root' property is pointing to the correct data location.
12:33:27,944 User:System ERROR [web.context.ContextLoader] Context initialization failed
org.alfresco.error.AlfrescoRuntimeException: Ensure that the 'dir.root' property is pointing to the correct data location.
   at org.alfresco.repo.admin.ConfigurationChecker.check(ConfigurationChecker.java:313)
   at org.alfresco.repo.admin.ConfigurationChecker.access$000(ConfigurationChecker.java:72)
   at org.alfresco.repo.admin.ConfigurationChecker$1.execute(ConfigurationChecker.java:178)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:320)
        ….
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The alf_data/lucene-indexes directory contains the following

[application@happy lucene-indexes]$ pwd
/opt/apps/alfresco/alf_data/lucene-indexes
[application@happy lucene-indexes]$ find . -print
.
./avm
./avm/sitestore
./avm/sitestore/cfe0e437-147f-41fc-bb92-309ef8efaf91
./avm/sitestore/cfe0e437-147f-41fc-bb92-309ef8efaf91/segments.gen
./avm/sitestore/cfe0e437-147f-41fc-bb92-309ef8efaf91/segments_3
./avm/sitestore/cfe0e437-147f-41fc-bb92-309ef8efaf91/_0.cfs
./avm/sitestore/IndexInfoBackup
./avm/sitestore/IndexInfo
./archive
./archive/SpacesStore
./archive/SpacesStore/IndexInfoBackup
./archive/SpacesStore/IndexInfo
./user
./user/alfrescoUserStore
./user/alfrescoUserStore/IndexInfoBackup
./user/alfrescoUserStore/4f52ff81-db79-4232-947b-9883886fcef5
./user/alfrescoUserStore/4f52ff81-db79-4232-947b-9883886fcef5/segments.gen
./user/alfrescoUserStore/4f52ff81-db79-4232-947b-9883886fcef5/segments_3
./user/alfrescoUserStore/4f52ff81-db79-4232-947b-9883886fcef5/_0.cfs
./user/alfrescoUserStore/f8dfa036-ed0f-46f0-83ee-b3b7225def61
./user/alfrescoUserStore/f8dfa036-ed0f-46f0-83ee-b3b7225def61/IndexInfoDeletions
./user/alfrescoUserStore/f8dfa036-ed0f-46f0-83ee-b3b7225def61/segments.gen
./user/alfrescoUserStore/f8dfa036-ed0f-46f0-83ee-b3b7225def61/segments_1
./user/alfrescoUserStore/efad2332-4b24-43be-b728-d27fb2f07860
./user/alfrescoUserStore/efad2332-4b24-43be-b728-d27fb2f07860/IndexInfoDeletions
./user/alfrescoUserStore/efad2332-4b24-43be-b728-d27fb2f07860/segments.gen
./user/alfrescoUserStore/efad2332-4b24-43be-b728-d27fb2f07860/segments_1
./user/alfrescoUserStore/8a8022b3-fd1d-4f18-9505-f0cb592193d7
./user/alfrescoUserStore/8a8022b3-fd1d-4f18-9505-f0cb592193d7/IndexInfoDeletions
./user/alfrescoUserStore/8a8022b3-fd1d-4f18-9505-f0cb592193d7/segments.gen
./user/alfrescoUserStore/8a8022b3-fd1d-4f18-9505-f0cb592193d7/segments_1
./user/alfrescoUserStore/3f2dd76e-882f-4d25-95ba-ec9b6f75c281
./user/alfrescoUserStore/3f2dd76e-882f-4d25-95ba-ec9b6f75c281/IndexInfoDeletions
./user/alfrescoUserStore/3f2dd76e-882f-4d25-95ba-ec9b6f75c281/segments.gen
./user/alfrescoUserStore/3f2dd76e-882f-4d25-95ba-ec9b6f75c281/segments_1
./user/alfrescoUserStore/3316f67d-4e01-4842-9dea-f5612b5c6a17
./user/alfrescoUserStore/3316f67d-4e01-4842-9dea-f5612b5c6a17/IndexInfoDeletions
./user/alfrescoUserStore/3316f67d-4e01-4842-9dea-f5612b5c6a17/segments.gen
./user/alfrescoUserStore/3316f67d-4e01-4842-9dea-f5612b5c6a17/segments_1
./user/alfrescoUserStore/edc6dee3-0129-4538-b8f4-ac9eb0b213b6
./user/alfrescoUserStore/edc6dee3-0129-4538-b8f4-ac9eb0b213b6/segments.gen
./user/alfrescoUserStore/edc6dee3-0129-4538-b8f4-ac9eb0b213b6/segments_3
./user/alfrescoUserStore/edc6dee3-0129-4538-b8f4-ac9eb0b213b6/_0.cfs
./user/alfrescoUserStore/IndexInfo
./locks
./system
./system/system
./system/system/IndexInfoBackup
./system/system/6d44939b-e360-475a-842e-cb723b96c636
./system/system/6d44939b-e360-475a-842e-cb723b96c636/IndexInfoDeletions
./system/system/6d44939b-e360-475a-842e-cb723b96c636/segments.gen
./system/system/6d44939b-e360-475a-842e-cb723b96c636/segments_3
./system/system/6d44939b-e360-475a-842e-cb723b96c636/_0.cfs
./system/system/b8f865ad-ebe5-4ec7-9eac-1c01e98f2e41
./system/system/b8f865ad-ebe5-4ec7-9eac-1c01e98f2e41/IndexInfoDeletions
./system/system/b8f865ad-ebe5-4ec7-9eac-1c01e98f2e41/segments.gen
./system/system/b8f865ad-ebe5-4ec7-9eac-1c01e98f2e41/segments_3
./system/system/b8f865ad-ebe5-4ec7-9eac-1c01e98f2e41/_0.cfs
./system/system/56386eb4-b4ac-40e7-a78e-61007a38f1af
./system/system/56386eb4-b4ac-40e7-a78e-61007a38f1af/IndexInfoDeletions
./system/system/56386eb4-b4ac-40e7-a78e-61007a38f1af/segments.gen
./system/system/56386eb4-b4ac-40e7-a78e-61007a38f1af/segments_3
./system/system/56386eb4-b4ac-40e7-a78e-61007a38f1af/_0.cfs
./system/system/d185367f-6d78-4c1b-bf35-652dc9756462
./system/system/d185367f-6d78-4c1b-bf35-652dc9756462/IndexInfoDeletions
./system/system/d185367f-6d78-4c1b-bf35-652dc9756462/segments.gen
./system/system/d185367f-6d78-4c1b-bf35-652dc9756462/segments_3
./system/system/d185367f-6d78-4c1b-bf35-652dc9756462/_0.cfs
./system/system/IndexInfo
./workspace
./workspace/version2Store
./workspace/version2Store/0a79a567-0f65-44a2-89c7-2d9738c718b4
./workspace/version2Store/0a79a567-0f65-44a2-89c7-2d9738c718b4/segments.gen
./workspace/version2Store/0a79a567-0f65-44a2-89c7-2d9738c718b4/segments_3
./workspace/version2Store/0a79a567-0f65-44a2-89c7-2d9738c718b4/_0.cfs
./workspace/version2Store/IndexInfoBackup
./workspace/version2Store/7b4ebb5e-a2c0-4e34-9e8e-7862ccf5d7f6
./workspace/version2Store/7b4ebb5e-a2c0-4e34-9e8e-7862ccf5d7f6/IndexInfoDeletions
./workspace/version2Store/7b4ebb5e-a2c0-4e34-9e8e-7862ccf5d7f6/segments.gen
./workspace/version2Store/7b4ebb5e-a2c0-4e34-9e8e-7862ccf5d7f6/segments_3
./workspace/version2Store/7b4ebb5e-a2c0-4e34-9e8e-7862ccf5d7f6/_0.cfs
./workspace/version2Store/06ef9156-efaf-4fbe-93d0-3708c96db676
./workspace/version2Store/06ef9156-efaf-4fbe-93d0-3708c96db676/segments_5
./workspace/version2Store/06ef9156-efaf-4fbe-93d0-3708c96db676/segments.gen
./workspace/version2Store/06ef9156-efaf-4fbe-93d0-3708c96db676/_0_2.del
./workspace/version2Store/06ef9156-efaf-4fbe-93d0-3708c96db676/_0.cfs
./workspace/version2Store/IndexInfo
./workspace/SpacesStore
./workspace/SpacesStore/IndexInfoBackup
./workspace/SpacesStore/b7691886-bdc0-463c-b32a-967d9cac6aca
./workspace/SpacesStore/b7691886-bdc0-463c-b32a-967d9cac6aca/segments.gen
./workspace/SpacesStore/b7691886-bdc0-463c-b32a-967d9cac6aca/segments_3
./workspace/SpacesStore/b7691886-bdc0-463c-b32a-967d9cac6aca/_0.cfs
./workspace/SpacesStore/ffd0c18d-39ef-4ad9-9c40-55e7cca81778
./workspace/SpacesStore/ffd0c18d-39ef-4ad9-9c40-55e7cca81778/IndexInfoDeletions
./workspace/SpacesStore/ffd0c18d-39ef-4ad9-9c40-55e7cca81778/segments.gen
./workspace/SpacesStore/ffd0c18d-39ef-4ad9-9c40-55e7cca81778/segments_3
./workspace/SpacesStore/ffd0c18d-39ef-4ad9-9c40-55e7cca81778/_0.cfs
./workspace/SpacesStore/b2d2f104-69c2-4ab2-bd72-8167e7f462c3
./workspace/SpacesStore/b2d2f104-69c2-4ab2-bd72-8167e7f462c3/_0_1.del
./workspace/SpacesStore/b2d2f104-69c2-4ab2-bd72-8167e7f462c3/segments.gen
./workspace/SpacesStore/b2d2f104-69c2-4ab2-bd72-8167e7f462c3/segments_4
./workspace/SpacesStore/b2d2f104-69c2-4ab2-bd72-8167e7f462c3/_0.cfs
./workspace/SpacesStore/a4f41ee7-665f-4a46-b1c4-21f80f3b38fd
./workspace/SpacesStore/a4f41ee7-665f-4a46-b1c4-21f80f3b38fd/segments.gen
./workspace/SpacesStore/a4f41ee7-665f-4a46-b1c4-21f80f3b38fd/_0_4.del
./workspace/SpacesStore/a4f41ee7-665f-4a46-b1c4-21f80f3b38fd/segments_7
./workspace/SpacesStore/a4f41ee7-665f-4a46-b1c4-21f80f3b38fd/_0.cfs
./workspace/SpacesStore/1f144d3f-2ed6-4330-9ce4-b3c8eeb4deaa
./workspace/SpacesStore/1f144d3f-2ed6-4330-9ce4-b3c8eeb4deaa/IndexInfoDeletions
./workspace/SpacesStore/1f144d3f-2ed6-4330-9ce4-b3c8eeb4deaa/segments.gen
./workspace/SpacesStore/1f144d3f-2ed6-4330-9ce4-b3c8eeb4deaa/_2.cfs
./workspace/SpacesStore/1f144d3f-2ed6-4330-9ce4-b3c8eeb4deaa/_1.cfs
./workspace/SpacesStore/1f144d3f-2ed6-4330-9ce4-b3c8eeb4deaa/segments_7
./workspace/SpacesStore/1f144d3f-2ed6-4330-9ce4-b3c8eeb4deaa/_0.cfs
./workspace/SpacesStore/IndexInfo
./workspace/lightWeightVersionStore
./workspace/lightWeightVersionStore/IndexInfoBackup
./workspace/lightWeightVersionStore/IndexInfo

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Again, if we set full reindex, Alfresco will start, but on a subsequent restart, we get the same problem.

Any ideas?

Thanks
Steve

[loftux]

Did you delete (or rename if you want to keep a backup) the lucene-indexes folder before doing the full reindexing?
This way you make sure you get a clean index.

[insumo]

Yes, we have removed the indexes, (rm -rf lucene-indexes) and performed a full reindex, and get this problem.

Also, when the reindex mode is set to FULL, it takes around an hour and a half to reindex. When we set to AUTO, we've seen it get to 70% after 15 hours (i.e. over night) at which point we kill it and do a full reindex.

Thanks
Steve

[loftux]

Is dir.root in custom-repository.properties set to have an absolute path?
Maybe set the dir.indexes=${dir.root}/lucene-indexes?
Have you checked file permissions so that the account that you run alfresco under have full access? Check not only the indexes folder, but also tomcat temp and work folders and the contentstore.

[insumo]

Unfortunately, all configs for directory are correct, so I still have no idea why this is behaving as it is. If we remove the alf_data directory and wipe out the database, the install will operate correctly, only without any data.

As this has now dragged on too long, I'll be performing an export to import into a new instance. Can anyone provide any tips on using the command line import/export to pull over all content, including versions? I've exported SpacesStore and Version2Store, but after importing with UPDATE_EXISTING, I end up with two roots for /app:company_home.

13:24:30,602 User:System ERROR [filesys.repo.ContentDiskDriver] Error during create context
org.alfresco.jlan.server.core.DeviceContextException: Multiple possible roots for device: 
   root path: /app:company_home
   results: [workspace://SpacesStore/0ae71b2a-eee0-4082-b35f-23136bda69ad, workspace://SpacesStore/bc2fe280-7e00-4e39-baa2-8d1e0d769581]
   at org.alfresco.filesys.repo.ContentDiskDriver.createContext(ContentDiskDriver.java:382)
‍‍‍‍‍‍‍

Again, any suggestions?

Thanks in advance
Steve

[ofrxnz]

i learned not to export the root directory but to export/import each folder in company home individually….it was easiest for me