Many, many open files

[raphael]

Dear users and developers,

I'm currently testing alfresco 1.3 official release (alfresco-community-tomcat-1.3.0.tar.gz) patched as mentioned in another thread concerning CIFS and NTLM [1]. So alfresco is configured to do NTLM passthru for the CIFS server and it just works.

However, I've rapidly faced a problem: both CIFS server and WEB interface stopped responding. After investigation it turns out that alfresco is not able to open a socket and I can observe "too many open files" exceptions in alfresco.log.

So, I've pushed the ulimit -n from 1024 to 8192 and the CIFS server seems to still be up'n running. So what? The number of opened files is rapidly increasing and I doubt this is neither normal nor wanted.

After about 4 hours running, the java process has 2583 open files :

xxx:~# lsof -p 4813 | wc -l
2583
‍‍‍‍

Many of them are in the alf_data and concerns lucene:

~# lsof -p 4813 | grep lucene | wc -l
2487
‍‍‍‍

Part of the output from lsof on the java process:

java    4813 root   32r   REG        8,5   697637 21037419 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_0.cfs
java    4813 root   33r   REG        8,5     8811 21037416 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_j.cfs
java    4813 root   34r   REG        8,5     3527 21037403 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_l.cfs (deleted)
java    4813 root   35r   REG        8,5     9429 21037401 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_h.cfs (deleted)
java    4813 root   36r   REG        8,5     7408 21037750 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_a.cfs (deleted)
java    4813 root   37r   REG        8,5     3959 21037736 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_c.cfs (deleted)
java    4813 root   38r   REG        8,5     5681 21037742 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_e.cfs (deleted)
java    4813 root   39r   REG        8,5     4826 21037740 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_g.cfs (deleted)
java    4813 root   40r   REG        8,5     4543 21051138 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_i.cfs (deleted)
java    4813 root   41r   REG        8,5     4814 21051141 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_k.cfs (deleted)
java    4813 root   42r   REG        8,5   697637 21037419 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_0.cfs
java    4813 root   43r   REG        8,5     8811 21037416 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_j.cfs
java    4813 root   44r   REG        8,5     3527 21037403 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_l.cfs (deleted)
java    4813 root   45r   REG        8,5     9429 21037401 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_h.cfs (deleted)
java    4813 root   46r   REG        8,5     7408 21037750 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_a.cfs (deleted)
java    4813 root   47r   REG        8,5     3959 21037736 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_c.cfs (deleted)
java    4813 root   48r   REG        8,5     5681 21037742 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_e.cfs (deleted)
java    4813 root   49r   REG        8,5     4826 21037740 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_g.cfs (deleted)
java    4813 root   50r   REG        8,5     4543 21051138 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_i.cfs (deleted)
java    4813 root   51r   REG        8,5     4814 21051141 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_k.cfs (deleted)
java    4813 root   52r   REG        8,5   697637 21037419 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_0.cfs
java    4813 root   53r   REG        8,5     8811 21037416 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_j.cfs
java    4813 root   54r   REG        8,5     3527 21037403 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_l.cfs (deleted)
java    4813 root   55r   REG        8,5     9429 21037401 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_h.cfs (deleted)
java    4813 root   56r   REG        8,5     7408 21037750 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_a.cfs (deleted)
java    4813 root   57r   REG        8,5     3959 21037736 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_c.cfs (deleted)
java    4813 root   58r   REG        8,5     5681 21037742 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_e.cfs (deleted)
java    4813 root   59r   REG        8,5     4826 21037740 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_g.cfs (deleted)
java    4813 root   60r   REG        8,5     4543 21051138 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_i.cfs (deleted)
java    4813 root   61r   REG        8,5     4814 21051141 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_k.cfs (deleted)
…
java    4813 root 2508r   REG        8,5   697637 21037419 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_0.cfs
java    4813 root 2509r   REG        8,5     8811 21037416 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_j.cfs
java    4813 root 2510r   REG        8,5    16554 21037646 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_r.cfs
java    4813 root 2511r   REG        8,5     9100 21037699 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_x.cfs
java    4813 root 2512r   REG        8,5     5948 21037642 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_13.cfs
java    4813 root 2513r   REG        8,5     5907 21051150 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_u.cfs
java    4813 root 2514r   REG        8,5     4142 21037683 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_1c.cfs
java    4813 root 2515r   REG        8,5     6031 21051164 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_18.cfs
java    4813 root 2516r   REG        8,5     6019 21051165 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_1a.cfs
java    4813 root 2517r   REG        8,5     4890 21051163 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_1b.cfs
java    4813 root 2518r   REG        8,5   697637 21037419 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_0.cfs
java    4813 root 2519r   REG        8,5     8811 21037416 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_j.cfs
java    4813 root 2520r   REG        8,5    16554 21037646 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_r.cfs
java    4813 root 2521r   REG        8,5     9100 21037699 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_x.cfs
java    4813 root 2522r   REG        8,5     5948 21037642 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_13.cfs
java    4813 root 2523r   REG        8,5     5907 21051150 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_u.cfs
java    4813 root 2524r   REG        8,5     4142 21037683 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_1c.cfs
java    4813 root 2525r   REG        8,5     6031 21051164 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_18.cfs
java    4813 root 2526r   REG        8,5     6019 21051165 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_1a.cfs
java    4813 root 2527r   REG        8,5     4890 21051163 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_1b.cfs
java    4813 root 2528r   REG        8,5   697637 21037419 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_0.cfs
java    4813 root 2529r   REG        8,5     8811 21037416 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_j.cfs
java    4813 root 2530r   REG        8,5    16554 21037646 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_r.cfs
java    4813 root 2531r   REG        8,5     9100 21037699 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_x.cfs
java    4813 root 2532r   REG        8,5     5948 21037642 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_13.cfs
java    4813 root 2533r   REG        8,5     5907 21051150 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_u.cfs
java    4813 root 2534r   REG        8,5     4142 21037683 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_1c.cfs
java    4813 root 2535r   REG        8,5     6031 21051164 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_18.cfs
java    4813 root 2536r   REG        8,5     6019 21051165 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_1a.cfs
java    4813 root 2537r   REG        8,5     4890 21051163 /opt/alfresco/alf_data/lucene-indexes/workspace/SpacesStore/index/_1b.cfs
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Additional information, I've installed the emaillistener from alfresco's forge.

Any idea?

Regards,
Rapha?l
[1] http://forums.alfresco.com/viewtopic.php?t=2139

[derek]

Hi,

The Lucene-based will constantly accessing the indexes, so I'm not too surprised with the list of files you indicate.  It's not that they aren't closed properly, but rather that they are constantly being used.

Does the system behave with the limits increased?

Regards

[raphael]

Hi,

thanks for your fast answer. In fact the number of open files increased regularly while the system is not used at all. After some more investigation this morning, I think I've tracked down the "guilty":

each time the EmailListener launches its schedule I have 10 more opened files and they are never released. Even without any emailListener configured the number of opened files increase by 10 each minute as soon as there is the debug output in the alfresco.log. After emailListener deactivation, I've no problem anymore (currently 141 opened files). I'm gonna try to contact the maintainer of this extension and update this case consequently.

Thanks for all and keep the good work,
Rapha

[raphael]

Hi,

just a small update. One of the developers of email listener has just posted a patch concerning this. It's available from here:
http://forge.alfresco.com/tracker/index.php?func=detail&aid=2&group_id=18&atid=153

(I precise that it seems to solve this issue!)

Cheers,
Raphael