Failed to copy Lucene index root due to low disk space

[anndrewtr]

Hi:

We are Alfresco3.2.0 community edition, we are experiencing No space left on device during automatic backup lucene indexing runs on 3AM. data1 folder is having 275GB space for lucene and backup lucene indexing folders. While we got the exception the space used was 87GB by .indexbackup_temp/ and 93GB by backup-lucene-indexes. Below is the error message:

org.alfresco.error.AlfrescoRuntimeException: 042594108 Failed to copy Lucene index root:    Index root: /data1/alf_data/lucene-indexes   Target: /data1/alf_data/backup-lucene-indexes   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1216)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$CoreLockWork$1.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1890)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$CoreLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1884)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.doWithAllWriteLocks(AbstractLuceneIndexerAndSearcherFactory.java:1824)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.backupImpl(AbstractLuceneIndexerAndSearcherFactory.java:1181)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.access$000(AbstractLuceneIndexerAndSearcherFactory.java:1027)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$1.execute(AbstractLuceneIndexerAndSearcherFactory.java:1143)   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:326)   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:233)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.backup(AbstractLuceneIndexerAndSearcherFactory.java:1147)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupJob.execute(AbstractLuceneIndexerAndSearcherFactory.java:1550)   at org.quartz.core.JobRunShell.run(JobRunShell.java:202)   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)Caused by: java.io.IOException: No space left on device   at java.io.FileOutputStream.writeBytes(Native Method)   at java.io.FileOutputStream.write(FileOutputStream.java:260)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copy(AbstractLuceneIndexerAndSearcherFactory.java:1398)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyFile(AbstractLuceneIndexerAndSearcherFactory.java:1345)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1317)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.backupDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1241)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1211)   … 46 more03:34:41,831 ERROR [org.quartz.core.ErrorLogger] Job (DEFAULT.indexBackupJobDetail threw an exception.org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 042594108 Failed to copy Lucene index root:    Index root: /data1/alf_data/lucene-indexes   Target: /data1/alf_data/backup-lucene-indexes]   at org.quartz.core.JobRunShell.run(JobRunShell.java:213)   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)Caused by: org.alfresco.error.AlfrescoRuntimeException: 042594108 Failed to copy Lucene index root:    Index root: /data1/alf_data/lucene-indexes   Target: /data1/alf_data/backup-lucene-indexes   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1216)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$CoreLockWork$1.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1890)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$CoreLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1884)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.doWithAllWriteLocks(AbstractLuceneIndexerAndSearcherFactory.java:1824)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.backupImpl(AbstractLuceneIndexerAndSearcherFactory.java:1181)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.access$000(AbstractLuceneIndexerAndSearcherFactory.java:1027)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$1.execute(AbstractLuceneIndexerAndSearcherFactory.java:1143)   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:326)   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:233)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.backup(AbstractLuceneIndexerAndSearcherFactory.java:1147)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupJob.execute(AbstractLuceneIndexerAndSearcherFactory.java:1550)   at org.quartz.core.JobRunShell.run(JobRunShell.java:202)   … 1 moreCaused by: java.io.IOException: No space left on device   at java.io.FileOutputStream.writeBytes(Native Method)   at java.io.FileOutputStream.write(FileOutputStream.java:260)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copy(AbstractLuceneIndexerAndSearcherFactory.java:1398)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyFile(AbstractLuceneIndexerAndSearcherFactory.java:1345)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1317)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.backupDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1241)   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1211)   … 46 more‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The normal space used is 3% by data1 is getting increased to 100% during lucene backup.

Is there any configuration changes or settings will solve this issue rather than increasing the space.

Any help is appreciated!!! Thanks in advance..

[bhavikp]

I am also facing this issue. Can someone help me?
My alfresco version is 3.1.0 enterprise edition.

Regards
Bhavik.

[marco_altieri]

Hi,

how many documents have you indexed? and how big are they?
The size of the lucene indexes seems to big to me.
I have seen this behavior before on other systems. I have found out that it was necessary to configure lucene to have a better distribution of documents in the available segments.

Cheers,
Marco