Failed to copy Lucene index root due to low disk space

[anndrewtr]

Hi:

We are Alfresco3.2.0 community edition, we are experiencing No space left on device during automatic backup lucene indexing runs on 3AM. data1 folder is having 275GB space for lucene and backup lucene indexing folders. While we got the exception the space used was 87GB by .indexbackup_temp/ and 93GB by backup-lucene-indexes. Below is the error message:

org.alfresco.error.AlfrescoRuntimeException: 042594108 Failed to copy Lucene index root: 
   Index root: /data1/alf_data/lucene-indexes
   Target: /data1/alf_data/backup-lucene-indexes
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1216)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$CoreLockWork$1.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1890)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$CoreLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1884)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.doWithAllWriteLocks(AbstractLuceneIndexerAndSearcherFactory.java:1824)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.backupImpl(AbstractLuceneIndexerAndSearcherFactory.java:1181)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.access$000(AbstractLuceneIndexerAndSearcherFactory.java:1027)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$1.execute(AbstractLuceneIndexerAndSearcherFactory.java:1143)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:326)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:233)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.backup(AbstractLuceneIndexerAndSearcherFactory.java:1147)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupJob.execute(AbstractLuceneIndexerAndSearcherFactory.java:1550)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Caused by: java.io.IOException: No space left on device
   at java.io.FileOutputStream.writeBytes(Native Method)
   at java.io.FileOutputStream.write(FileOutputStream.java:260)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copy(AbstractLuceneIndexerAndSearcherFactory.java:1398)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyFile(AbstractLuceneIndexerAndSearcherFactory.java:1345)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1317)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.backupDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1241)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1211)
   … 46 more
03:34:41,831 ERROR [org.quartz.core.ErrorLogger] Job (DEFAULT.indexBackupJobDetail threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: org.alfresco.error.AlfrescoRuntimeException: 042594108 Failed to copy Lucene index root: 
   Index root: /data1/alf_data/lucene-indexes
   Target: /data1/alf_data/backup-lucene-indexes]
   at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
   at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:529)
Caused by: org.alfresco.error.AlfrescoRuntimeException: 042594108 Failed to copy Lucene index root: 
   Index root: /data1/alf_data/lucene-indexes
   Target: /data1/alf_data/backup-lucene-indexes
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1216)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$CoreLockWork$1.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1890)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$CoreLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1884)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2342)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithFileLock(IndexInfo.java:2262)
   at org.alfresco.repo.search.impl.lucene.index.IndexInfo.doWithWriteLock(IndexInfo.java:2248)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneBase.doWithWriteLock(AbstractLuceneBase.java:334)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$NestingLockWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1861)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory.doWithAllWriteLocks(AbstractLuceneIndexerAndSearcherFactory.java:1824)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.backupImpl(AbstractLuceneIndexerAndSearcherFactory.java:1181)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.access$000(AbstractLuceneIndexerAndSearcherFactory.java:1027)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$1.execute(AbstractLuceneIndexerAndSearcherFactory.java:1143)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:326)
   at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:233)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent.backup(AbstractLuceneIndexerAndSearcherFactory.java:1147)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupJob.execute(AbstractLuceneIndexerAndSearcherFactory.java:1550)
   at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
   … 1 more
Caused by: java.io.IOException: No space left on device
   at java.io.FileOutputStream.writeBytes(Native Method)
   at java.io.FileOutputStream.write(FileOutputStream.java:260)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copy(AbstractLuceneIndexerAndSearcherFactory.java:1398)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyFile(AbstractLuceneIndexerAndSearcherFactory.java:1345)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1317)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.copyDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1312)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.backupDirectory(AbstractLuceneIndexerAndSearcherFactory.java:1241)
   at org.alfresco.repo.search.impl.lucene.AbstractLuceneIndexerAndSearcherFactory$LuceneIndexBackupComponent$BackUpWithAllWriteLocksWork.doWork(AbstractLuceneIndexerAndSearcherFactory.java:1211)
   … 46 more
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

The normal space used is 3% by data1 is getting increased to 100% during lucene backup.

Is there any configuration changes or settings will solve this issue rather than increasing the space.

Any help is appreciated!!! Thanks in advance..

[bhavikp]

I am also facing this issue. Can someone help me?
My alfresco version is 3.1.0 enterprise edition.

Regards
Bhavik.

[marco_altieri]

Hi,

how many documents have you indexed? and how big are they?
The size of the lucene indexes seems to big to me.
I have seen this behavior before on other systems. I have found out that it was necessary to configure lucene to have a better distribution of documents in the available segments.

Cheers,
Marco