linux 32000 files limitation

[paul_lahitte]

I am using alfresco community on redhat linux and lucenes-indexes/workspace/SpacesStore is containing 32001 directories. The systems refuses creating new dir "too many links" and then Alfresco is not starting while it is trying to create anew directory.


the log:java.io.IOException: Cannot create directory: /data/opt/alf_data/lucene-indexes/workspace/SpacesStore/e51ff042-3628-11dd-b952-2b903bc78fd4        at org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:175)        at org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:227)
        at org.alfresco.repo.search.impl.lucene.index.IndexInfo$Merger.mergeIndexes(IndexInfo.java:2943)
        at org.alfresco.repo.search.impl.lucene.index.IndexInfo$Merger.run(IndexInfo.java:2448)        at java.lang.Thread.run(Thread.java:619)
ERROR [org.alfresco.repo.search.impl.lucene.index.IndexInfo] Failed to merge indexes
java.io.IOException: Cannot create directory: /data/opt/alf_data/lucene-indexes/workspace/SpacesStore/e55b2483-3628-11dd-b952-2b903bc78fd4
        at org.apache.lucene.store.FSDirectory.getDirectory(FSDirectory.java:175

Does any one know how to increase this number ??

Thank's

[louise]

A shell script to test limitation (script results more than 100.000 on OSX 10.5):

#!/bin/bash
#
# Simple node limit test script by LouiSe@louise.hu
#
# Known Linux limitation:
#  - Ext2, Ext3: 32000
#  - ReiserFS: 64K
#
TESTDIR="test"
LIMIT=100000
a=1

mkdir ${TESTDIR}

while [ "$a" -le $LIMIT ]
do
    echo "mkdir ${TESTDIR}/$a"
    mkdir "${TESTDIR}/$a"
    let "a+=1"
done‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[andy]

Hi

There should not be this many directories.

Have you got any custom code?
What configuration changes have you made?

If there seems to be any kind of hang then you need to obtain a stack trace.

Andy

[jjf]

We had a batch job which added thousands of users to the Alfresco store using addAuthority, addPerson, etc.  An index was created for every call to these methods.  We switched to packaging these calls as one transaction job.  This seems to only create one index and is much more efficient.

[dwilson]

We had a batch job which added thousands of users to the Alfresco store using addAuthority, addPerson, etc.  An index was created for every call to these methods.  We switched to packaging these calls as one transaction job.  This seems to only create one index and is much more efficient.

This thread is frightening indeed- along with the others like it.  It makes my stomach sink at the thought that my repository might soon hit the inode limit in Linux, and days of production server downtime followed by potential tomcat start-up failure is the only way to fix it.

To Alfresco Engineers - based on jjf's result quoted above are there some session/transaction guidelines we need to be following with our code?  I also have (php) code that batch imports thousands of pieces of content, should I only be saving the session at the end of the entire script to avoid the inode limitation?  Is this problem only related to the user store?  Has this problem been reproduced and fixed/noted in newer releases?

Thanks for the help!
-Dave

[etzapata]

Hi All,

Does anyone managed to fix this? I have huge content (around 380GB) and encounters some errors so I was thinking that reindexing would fix the problem but full reindex (per alfresco.log) did not continue. It stops (or seems to stop) at 60% and never continue even until the next 2 days.

When I looked at catalina.out it says:
08:11:24,237 INFO [node.index.FullRecoveryComponent]   100 % complete.
08:11:24,244 INFO [node.index.FullRecoveryComponent]   Index recovery completed.

[WARNING] Unknown Ptg 2a (42) at cell (350,2)

Could my problem be the same as what is being discussed in this thread?

I would be very grateful for any inputs anyone can share.

Thank you very much.

EriC z.

p.s. I would like to delete a huge single file (3.6GB in size) directly in alf_data/contentstore. Is this okay without affecting anything? I'm sure the DB will not be in sync vs. contentstore but is it fixed somehow with reindex and totally delete the entry in DB?