performance problems when too many files in one directory

[lagreen]

Hello,

I have a real performance problem when user/s tries to open directory (either through CIFS or through WEB interface) with more than 1000 files in it (it takes like 20 seconds). I have MySQL DB and Alfresco 3.2.
I did some monitoring tests and tomcat and JVM seems to be running OK. So i think that the weak point is MySQL DB. Maybe i have to rebuild the tables indexes. I serached the forum but did not find anything.

Does somebody have some ideas what to optimize or what to do?

thanx.

[brian_robinson]

One suggestion: check your database connection pool configuration settings.  Here's a reference: http://wiki.alfresco.com/wiki/ASR_Performance_Optimization#Increase_Database_Connection_Pool

This reference is applicable for both DM and ASR installations.

-Brian

[fracat71]

Hi,
i have a similar problem: a space (named as root) contains 40000 spaces (created by a java main external to the Alfresco application).
Let me explain. I have two enviroments (named as A , B).

Env. A:
- ALFRESCO: Community Current version 3.2.0 (r2 2440) schema 3300
- OS: Windows xp 32 bit , RAM 2,5G , Intel Core Duo 2,1 GHz
- AS: Tomcat 6.0.18
- JAVA: Jdk 6.0_11. Heap 986,125MB
- DB :Mysql 5.1.35 Community

Env.B:
- ALFRESCO: Community Current version 3.2.0 (r2 2440) schema 3300
- OS: Linux Oracle  64 bit (RedHat 5.4), RAM 6G
- AS: JBOSS 4.2.2 GA
- JAVA: Jdk 6.0_18.  Heap 4GB
- DB: Oracle 11g

Problems:

Env. A  : when i try to display the folder root i wait for 5/6 minutes and after Alfresco Explorer displays sometime 3500 items , 3800 itmes but never 40000 (note that 3800 has one 0 minus that 40000).
Env. B  : see http://forums.alfresco.com/en/viewtopic.php?f=14&t=26355

I don't know what happen, always strange behaviour.

Bye

[antoninoferraro]

Hello,

I have a very similar performance problem when user/s tries to open a space through WEB interface with more than 1000 documents in it (it takes like 30 seconds after Alfresco Explorer displays sometime less then 1000 items).
The documents are created via WebScripts in the repository everyone with own ACL (that is a permission associated to a detailed user).
If instead I search documents from the search interface through Lucene metadata (example: for contained text or for property document name ), the search is faster.

I have:
ALFRESCO: Alfresco 3.0 Community/ Alfresco 3.0 Enterprise
DB: Oracle 10 DB
AS: JBoss 4.2.1 GA
JAVA: jdk 1.5_22
OS: Red Hat Enterprise Linux 5.3
RAM: 2 GB


Does somebody have some any ideas ?

thank you.

[lagreen]

Thanks brian.robinson for the hint.
I set higher my db connection pool configuration but no improvement. Its really sad that there are not any information about similar issues.

[loftux]

Here is one post on the topic.
http://forums.alfresco.com/en/viewtopic.php?f=4&t=22629

For Alfresco Explorer: Find you document using a search. No one can in practice browse through pages and pages to find documents. Why does page browsing in Epxlorer become slow? It retrieves all the files in the space in one batch, and then have to do security evaluation for each, and determine for each what action items to display. Not the best solution, but that is how it works. Share is better in this regard, with true paging. Still, avoid to many files in one directory.

Why is the number of files in a space different each time? This is because of lucene/query timeouts. It has a maxmum number of ACL check, and a time limit. Whatever comes first stops returning more nodes, and return the already processed batch. http://www.google.com/search?q=site:forums.alfresco.com+system.acl.maxPermissionChecks

If you really need to store many files in one space, create a separate webscript for browsing that space. there is also the possibility to sidestep security check on each file in the search result, thereby improving performance.

[dgenard]

Hi, we did notice that problem as well.
Our solution was to automatically classify the files in sub-folders (in example in a year/month/day structure) so that a single folder will not exceed 500 files. The automatic classification can be done with a rule and a bit of javascript.
Hope this helps.
Denis

[lagreen]

Thanx for the suggestions. I think we will do this sub-folder thing. We allready did this by making month sub-folders, but now i understand that we should make sub-folders by days too. I hope that some day there will be possibility to keep 1000+ files in one directory and the browse time will be good

ok,
wish you all the best.

[gyro_gearless]

At least this is a known issue for 3.2, see for example http://issues.alfresco.com/jira/browse/ALF-2722 and related. There is a fix for 3.3 which might be retrofitted to 3.2, i'm just investigating this… However, partitioning your directories might not cure the problem, as the deeper reason for the problem is perhaps a bogus SQL "OUTER JOIN", which might bite you even on small folders. (I think it rather depends on the total number of nodes in the system)

Cheers
Gyro

[fracat71]

Hi Gyro.Gearless,
So, Enterprise and Community have the same problem. Also previous version?
How can i fix it?

Thanks