topic Re: performance problems when too many files in one directory in Alfresco Archive

performance problems when too many files in one directory

lagreen — Thu, 29 Apr 2010 12:29:27 GMT

Hello,I have a real performance problem when user/s tries to open directory (either through CIFS or through WEB interface) with more than 1000 files in it (it takes like 20 seconds). I have MySQL DB and Alfresco 3.2.I did some monitoring tests and tomcat and JVM seems to be running OK. So i think th

Re: performance problems when too many files in one directory

brian_robinson — Sat, 01 May 2010 01:21:49 GMT

One suggestion: check your database connection pool configuration settings. Here's a reference: http://wiki.alfresco.com/wiki/ASR_Performance_Optimization#Increase_Database_Connection_Pool

This reference is applicable for both DM and ASR installations.

-Brian

Re: performance problems when too many files in one directory

fracat71 — Sat, 01 May 2010 11:00:47 GMT

Hi,
i have a similar problem: a space (named as root) contains 40000 spaces (created by a java main external to the Alfresco application).
Let me explain. I have two enviroments (named as A , B).

Env. A:
- ALFRESCO: Community Current version 3.2.0 (r2 2440) schema 3300
- OS: Windows xp 32 bit , RAM 2,5G , Intel Core Duo 2,1 GHz
- AS: Tomcat 6.0.18
- JAVA: Jdk 6.0_11. Heap 986,125MB
- DB :Mysql 5.1.35 Community

Env.B:
- ALFRESCO: Community Current version 3.2.0 (r2 2440) schema 3300
- OS: Linux Oracle 64 bit (RedHat 5.4), RAM 6G
- AS: JBOSS 4.2.2 GA
- JAVA: Jdk 6.0_18. Heap 4GB
- DB: Oracle 11g

Problems:

http://forums.alfresco.com/en/viewtopic.php?f=14&t=26355

I don't know what happen, always strange behaviour.

Bye

Re: performance problems when too many files in one directory

antoninoferraro — Wed, 05 May 2010 08:16:42 GMT

Hello,

I have a very similar performance problem when user/s tries to open a space through WEB interface with more than 1000 documents in it (it takes like 30 seconds after Alfresco Explorer displays sometime less then 1000 items).
The documents are created via WebScripts in the repository everyone with own ACL (that is a permission associated to a detailed user).
If instead I search documents from the search interface through Lucene metadata (example: for contained text or for property document name ), the search is faster.

I have:
ALFRESCO: Alfresco 3.0 Community/ Alfresco 3.0 Enterprise
DB: Oracle 10 DB
AS: JBoss 4.2.1 GA
JAVA: jdk 1.5_22
OS: Red Hat Enterprise Linux 5.3
RAM: 2 GB

Does somebody have some any ideas ?

thank you.

Re: performance problems when too many files in one directory

lagreen — Wed, 05 May 2010 11:35:45 GMT

Thanks brian.robinson for the hint.
I set higher my db connection pool configuration but no improvement. Its really sad that there are not any information about similar issues.

Re: performance problems when too many files in one directory

loftux — Wed, 05 May 2010 12:10:09 GMT

Here is one post on the topic.
http://forums.alfresco.com/en/viewtopic.php?f=4&t=22629

For Alfresco Explorer: Find you document using a search. No one can in practice browse through pages and pages to find documents. Why does page browsing in Epxlorer become slow? It retrieves all the files in the space in one batch, and then have to do security evaluation for each, and determine for each what action items to display. Not the best solution, but that is how it works. Share is better in this regard, with true paging. Still, avoid to many files in one directory.

Why is the number of files in a space different each time? This is because of lucene/query timeouts. It has a maxmum number of ACL check, and a time limit. Whatever comes first stops returning more nodes, and return the already processed batch. http://www.google.com/search?q=site:forums.alfresco.com+system.acl.maxPermissionChecks

If you really need to store many files in one space, create a separate webscript for browsing that space. there is also the possibility to sidestep security check on each file in the search result, thereby improving performance.

Re: performance problems when too many files in one directory

dgenard — Wed, 05 May 2010 14:43:35 GMT

Hi, we did notice that problem as well.
Our solution was to automatically classify the files in sub-folders (in example in a year/month/day structure) so that a single folder will not exceed 500 files. The automatic classification can be done with a rule and a bit of javascript.
Hope this helps.
Denis

Re: performance problems when too many files in one directory

lagreen — Fri, 07 May 2010 08:30:15 GMT

Thanx for the suggestions. I think we will do this sub-folder thing. We allready did this by making month sub-folders, but now i understand that we should make sub-folders by days too. I hope that some day there will be possibility to keep 1000+ files in one directory and the browse time will be good

ok,
wish you all the best.

Re: performance problems when too many files in one directory

gyro_gearless — Fri, 07 May 2010 11:33:42 GMT

At least this is a known issue for 3.2, see for example http://issues.alfresco.com/jira/browse/ALF-2722 and related. There is a fix for 3.3 which might be retrofitted to 3.2, i'm just investigating this… However, partitioning your directories might not cure the problem, as the deeper reason for the problem is perhaps a bogus SQL "OUTER JOIN", which might bite you even on small folders. (I think it rather depends on the total number of nodes in the system)

Cheers
Gyro

Re: performance problems when too many files in one directory

fracat71 — Fri, 07 May 2010 17:00:56 GMT

Hi Gyro.Gearless,
So, Enterprise and Community have the same problem. Also previous version?
How can i fix it?

Thanks

Re: performance problems when too many files in one directory

kgeis — Mon, 24 May 2010 23:04:30 GMT

I've filed ALF-2960 in response to this. I changed the node->properties and node->aspects associations to fetch via subselect, and that has helped some. Further tuning is in order. It's still not returning results within ten seconds, so I might need to bump up my system.acl.maxPermissionCheckTimeMillis.

Re: performance problems when too many files in one directory

gyro_gearless — Tue, 25 May 2010 09:39:36 GMT

Hi all,

as workaround, we have changed the ResultTransformer in /Repository/source/java/org/alfresco/repo/node/db/hibernate/HibernateNodeDaoServiceImpl.java around line 3126:

    private void cacheNodesNoBatch(Store store, List<String> uuids)
    {
        Criteria criteria = getSession().createCriteria(NodeImpl.class, "node");

        // ————————————————————————————-
        // Kludge AW 2010-05-07: Discard duplicate node ids
        //
        // criteria.setResultTransformer(Criteria.ROOT_ENTITY);
        criteria.setResultTransformer(Criteria.DISTINCT_ROOT_ENTITY);
        // ————————————————————————————-
‍‍‍‍‍‍‍‍‍‍‍

This is perhaps essentially the same as the fix proposed by Alfresco - the problem with OUTER JOIN fetching is a known weakness in Hibernate, so it has it's own workaround :mrgreen:
Results are quite promising: a folder with ~ 600 documents, which was peviously shown in 40 secs now shows up in 4…

Cheers
Gyro

Re: performance problems when too many files in one directory

derek — Tue, 25 May 2010 11:00:15 GMT

Hi.
We've done a similar fix: https://issues.alfresco.com/jira/browse/ALF-2839. There are also a few other related tweaks to speed things up. You'll still be getting 2n+1 performance against cold caches, but it's better than the blow-out of the resultset caused by the X*Y*Z rows returned by the original query.

Re: performance problems when too many files in one directory

fracat71 — Tue, 25 May 2010 11:49:20 GMT

Hi all,
sorry for my question, but what changes solve the problem?
This is mine version (revision is 17458)


  private void cacheNodesNoBatch(Store store, List<String> uuids)
  {
    Criteria criteria = getSession().createCriteria(NodeImpl.class, "node");
    criteria.setResultTransformer(Criteria.ROOT_ENTITY);
    criteria.add(Restrictions.eq("store.id", store.getId()));
    criteria.add(Restrictions.in("uuid", uuids));
    criteria.setCacheMode(CacheMode.PUT);
    criteria.setFlushMode(FlushMode.MANUAL);

    List nodeList = criteria.list();
    List nodeIds = new ArrayList(nodeList.size());
    for (Node node : nodeList)
    {
      Long nodeId = node.getId();
      this.storeAndNodeIdCache.put(node.getNodeRef(), nodeId);
      nodeIds.add(nodeId);
    }

    if (nodeIds.size() == 0)
    {
      return;
    }

    criteria = getSession().createCriteria(ChildAssocImpl.class, "parentAssoc");
    criteria.setResultTransformer(Criteria.ROOT_ENTITY);
    criteria.add(Restrictions.in("child.id", nodeIds));
    criteria.setCacheMode(CacheMode.PUT);
    criteria.setFlushMode(FlushMode.MANUAL);
    List parentAssocs = criteria.list();
    Map parentAssocMap = new HashMap(nodeIds.size() * 2);
    for (ChildAssoc parentAssoc : parentAssocs)
    {
      nodeId = parentAssoc.getChild().getId();
      List parentAssocsOfNode = (List)parentAssocMap.get(nodeId);
      if (parentAssocsOfNode == null)
      {
        parentAssocsOfNode = new ArrayList(3);
        parentAssocMap.put(nodeId, parentAssocsOfNode);
      }
      parentAssocsOfNode.add(parentAssoc);
      if (this.isDebugParentAssocCacheEnabled)
      {
        loggerParentAssocsCache.debug("\nParent associations cache - Adding entry: \n   Node:   " + nodeId + "\n" + "   Assocs: " + parentAssocsOfNode);
      }

    }

    for (Node node : nodeList)
    {
      nodeId = node.getId();
      List parentAsssocsOfNode = (List)parentAssocMap.get(nodeId);
      if (parentAsssocsOfNode == null)
      {
        parentAsssocsOfNode = Collections.emptyList();
      }
      this.parentAssocsCache.put(nodeId, new NodeInfo(node, this.qnameDAO, parentAsssocsOfNode));
    }
  }

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

This is TH solution. See the comment //Kludge AW 2010-05-07: Discard duplicate node ids


    private void cacheNodesNoBatch(Store store, List<String> uuids)
    {
        Criteria criteria = getSession().createCriteria(NodeImpl.class, "node");

        // ————————————————————————————-
        // Kludge AW 2010-05-07: Discard duplicate node ids
        //
        // criteria.setResultTransformer(Criteria.ROOT_ENTITY);
        criteria.setResultTransformer(Criteria.DISTINCT_ROOT_ENTITY);
        // ————————————————————————————-
‍‍‍‍‍‍‍‍‍‍‍‍

This is 3.3 version (revision is 20392).See the comment //We have duplicate nodes, so make sure we only process each node once


    private void cacheNodesNoBatch(Store store, List<String> uuids)
    {
        Criteria criteria = getSession().createCriteria(NodeImpl.class, "node");
        criteria.setResultTransformer(Criteria.ROOT_ENTITY);
        criteria.add(Restrictions.eq("store.id", store.getId()));
        criteria.add(Restrictions.in("uuid", uuids));
        criteria.setCacheMode(CacheMode.PUT);
        criteria.setFlushMode(FlushMode.MANUAL);

        List<Node> nodeList = criteria.list();
        Set<Long> nodeIds = new HashSet<Long>(nodeList.size()*2);
        for (Node node : nodeList)
        {
            // We have duplicate nodes, so make sure we only process each node once
            Long nodeId = node.getId();
            if (!nodeIds.add(nodeId))
            {
                // Already processed
                continue;
            }
            storeAndNodeIdCache.put(node.getNodeRef(), nodeId);
        }        
        
        if (nodeIds.size() == 0)
        {
            // Can't query
            return;
        }
        
        criteria = getSession().createCriteria(ChildAssocImpl.class, "parentAssoc");
        criteria.setResultTransformer(Criteria.ROOT_ENTITY);
        criteria.add(Restrictions.in("child.id", nodeIds));
        criteria.setCacheMode(CacheMode.PUT);
        criteria.setFlushMode(FlushMode.MANUAL);
        List<ChildAssoc> parentAssocs = criteria.list();
        Map<Long, List<ChildAssoc>> parentAssocMap = new HashMap<Long, List<ChildAssoc>>(nodeIds.size() * 2);
        for (ChildAssoc parentAssoc : parentAssocs)
        {
            Long nodeId = parentAssoc.getChild().getId();
            List<ChildAssoc> parentAssocsOfNode = parentAssocMap.get(nodeId);
            if (parentAssocsOfNode == null)
            {
                parentAssocsOfNode = new ArrayList<ChildAssoc>(3);
                parentAssocMap.put(nodeId, parentAssocsOfNode);
            }
            parentAssocsOfNode.add(parentAssoc);
            if (isDebugParentAssocCacheEnabled)
            {
                loggerParentAssocsCache.debug("\n" +
                        "Parent associations cache - Adding entry: \n" +
                        "   Node:   " + nodeId + "\n" +
                        "   Assocs: " + parentAssocsOfNode);
            }
        }
        // Cache NodeInfo for each node
        for (Node node : nodeList)
        {
            Long nodeId = node.getId();
            List<ChildAssoc> parentAsssocsOfNode = parentAssocMap.get(nodeId);
            if (parentAsssocsOfNode == null)
            {
                parentAsssocsOfNode = Collections.emptyList();
            }
            parentAssocsCache.put(nodeId, new NodeInfo(node, qnameDAO, parentAsssocsOfNode));
        }        
    }
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Regards

Re: performance problems when too many files in one directory

fracat71 — Tue, 25 May 2010 14:25:30 GMT

Hi all,
i have applied both changes to my project.
I have two Envs.

- Env. A:
- ALFRESCO: Community Current version 3.2.0 (r2 2440) schema 3300
- OS: Windows xp 32 bit , RAM 2,5G , Intel Core Duo 2,1 GHz
- AS: Tomcat 6.0.18
- JAVA: Jdk 6.0_11. Heap 986,125MB
- DB :Mysql 5.1.35 Community

Env.B:
- ALFRESCO: Community Current version 3.2.0 (r2 2440) schema 3300
- OS: Linux Oracle 64 bit (RedHat 5.4), RAM 6G
- AS: JBOSS 4.2.2 GA
- JAVA: Jdk 6.0_18. Heap 4GB
- DB: Oracle 11g

I have not applied and tested on Env. A where i have problems of perfomance and returned values
I have applied on Env.B and now no excpetion is eraised until 247 elements returned (before the changes every Oracle query on DB has errors ORA-01795 also with 247) , on this Env.B i have not tested with a lot data (40000) like my previuos post.

Regards

Re: performance problems when too many files in one directory

derek — Tue, 25 May 2010 16:58:29 GMT


        // Get nodes and properties
        Criteria criteria = getSession().createCriteria(NodeImpl.class, "node");
        criteria.setResultTransformer(Criteria.ROOT_ENTITY);
        criteria.add(Restrictions.eq("store.id", store.getId()));
        criteria.add(Restrictions.in("uuid", uuids));
        criteria.setFetchMode("aspects", FetchMode.SELECT);                 // Don't join to aspects
        criteria.setCacheMode(CacheMode.PUT);
        criteria.setFlushMode(FlushMode.MANUAL);
        criteria.list();

        // Get nodes and aspects
        criteria = getSession().createCriteria(NodeImpl.class, "node");
        criteria.setResultTransformer(Criteria.ROOT_ENTITY);
        criteria.add(Restrictions.eq("store.id", store.getId()));
        criteria.add(Restrictions.in("uuid", uuids));
        criteria.setFetchMode("properties", FetchMode.SELECT);              // Don't join to properties
        criteria.setCacheMode(CacheMode.PUT);
        criteria.setFlushMode(FlushMode.MANUAL);

        List<Node> nodeList = criteria.list();
        Set<Long> nodeIds = new HashSet<Long>(nodeList.size()*2);
        for (Node node : nodeList)
        {
            // We have duplicate nodes, so make sure we only process each node once
            Long nodeId = node.getId();
            if (!nodeIds.add(nodeId))
            {
                // Already processed
                continue;
            }
            storeAndNodeIdCache.put(node.getNodeRef(), nodeId);
        }        
        
        if (nodeIds.size() == 0)
        {
            // Can't query
            return;
        }
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: performance problems when too many files in one directory

fracat71 — Wed, 26 May 2010 07:33:53 GMT

Hi all,
now i have a lot of versions of the software……

1- Gyro.Gearless :
replace

 criteria.setResultTransformer(Criteria.ROOT_ENTITY);‍

with

 criteria.setResultTransformer(Criteria.DISTINCT_ROOT_ENTITY);‍

2- SVN Head :
replace

for (Node node : nodeList)
    {
      Long nodeId = node.getId();
      this.storeAndNodeIdCache.put(node.getNodeRef(), nodeId);
      nodeIds.add(nodeId);
}
‍‍‍‍‍‍‍

with

 for (Node node : nodeList)
        {
            // We have duplicate nodes, so make sure we only process each node once
            Long nodeId = node.getId();
            if (!nodeIds.add(nodeId))
            {
                // Already processed
                continue;
            }
            storeAndNodeIdCache.put(node.getNodeRef(), nodeId);
        }‍‍‍‍‍‍‍‍‍‍‍

3-derek
add


…..
criteria.setFetchMode("aspects", FetchMode.SELECT);
……
criteria.setFetchMode("properties", FetchMode.SELECT);
…..
‍‍‍‍‍‍‍

Do i have applay all?
I believe that this a blocking bug.

Regards

Re: performance problems when too many files in one directory

derek — Wed, 26 May 2010 11:23:34 GMT

Hi,
1: I'm not convinced on the behaviour of 1 (it says a distinct entity per row, if I remember) and I don't know if it's rolling the related properties and aspects up correctly. We don't have that in our tested code.
2: Yes. That eliminates the duplicates and prevents blow-outs of the subsequent queries.
3: Yes. That prevents blow-out of the resultset from left joins to nodes AND aspects in the same query. We have a Criteria query for nodes-and-properties and a Criteria query for nodes-and-aspects.

Re: performance problems when too many files in one directory

fracat71 — Wed, 26 May 2010 12:29:55 GMT

Hi derek,
thanks for your support.
But i don't understand yuor code it seems to code written twice with a criteria.list() without returned value: is it ok?

        // Get nodes and properties
        Criteria criteria = getSession().createCriteria(NodeImpl.class, "node");
        criteria.setResultTransformer(Criteria.ROOT_ENTITY);
        criteria.add(Restrictions.eq("store.id", store.getId()));
        criteria.add(Restrictions.in("uuid", uuids));
        criteria.setFetchMode("aspects", FetchMode.SELECT);                 // Don't join to aspects
        criteria.setCacheMode(CacheMode.PUT);
        criteria.setFlushMode(FlushMode.MANUAL);
        criteria.list();

        // Get nodes and aspects
        criteria = getSession().createCriteria(NodeImpl.class, "node");
        criteria.setResultTransformer(Criteria.ROOT_ENTITY);
        criteria.add(Restrictions.eq("store.id", store.getId()));
        criteria.add(Restrictions.in("uuid", uuids));
        criteria.setFetchMode("properties", FetchMode.SELECT);              // Don't join to properties
        criteria.setCacheMode(CacheMode.PUT);
        criteria.setFlushMode(FlushMode.MANUAL);

        List<Node> nodeList = criteria.list();
        Set<Long> nodeIds = new HashSet<Long>(nodeList.size()*2);
        for (Node node : nodeList)
        {
            // We have duplicate nodes, so make sure we only process each node once
            Long nodeId = node.getId();
            if (!nodeIds.add(nodeId))
            {
                // Already processed
                continue;
            }
            storeAndNodeIdCache.put(node.getNodeRef(), nodeId);
        }

        if (nodeIds.size() == 0)
        {
            // Can't query
            return;
        }

Note that i'm testing 1+2 on OracleDB with 250 elements and seems to work( fast and without errors).
Now i'm testing 1+2 on MysqlDB a folder with 1000 subfolders (every subfolder has others 5 subsubfolders) and the explorer view is working(2/3 seconds to display) but the delete is very very very slow (15 minutes and is working……untill now).
The log4j says:

14:18:29,218 User:admin WARN  [org.alfresco.storeAndNodeIdTransactionalCache] Transactional update cache 'org.alfresco.storeAndNodeIdTransactionalCach
e' is full (10000).‍‍

Regards

Re: performance problems when too many files in one directory

derek — Wed, 26 May 2010 12:43:14 GMT

You might be better off without the batch code completely. On the Enterprise release we have also added code to check the presence of nodes in the cache before adding it to the list to batch-fetch. The cache being full is a problem and you should increase the cache size to accommodate the types of operations you wish to perform.