topic Re: Performance, large transaction in Alfresco Archive

Performance, large transaction

buurd — Thu, 10 Jun 2010 13:04:51 GMT

Hi!I'm running a large importjob that are going to put about 100 000 documents into our Alfresco-installation. But there are some problems that makes me a little bit worried.1) We add all files into import-maps, they are named with unique names from the old system2) We add metadata-files from the ol

Re: Performance, large transaction

mrogers — Fri, 11 Jun 2010 08:24:57 GMT

The issue is in the title. Its due to the size of your transactions. So if you try to write 100K documents in a single transaction then you are going to have problems. As you have already found if you choose smaller transaction boundaries then things will work better and the locking will be less.

The transaction locking that happens is probably at database level so if you post details of your database and configuration there may be some advice.

Re: Performance, large transaction

buurd — Tue, 15 Jun 2010 08:06:44 GMT

The problem isn't in the batchjob, we handle it by limit the size of the transaction, even though it might be nice to be able to force a commit in the javascript for each file imported.

The problem is the action that we might expect a user to perform, delete a space. It also cause lockups. How to work around that? People are going to import zip-files with a lot of documents and then discover that it was the wrong version, or the wrong locations and then delete them. That operation can't stop other users to write to file, right?

We have support on the issue now so I'll report back when we have a solution.

Re: Performance, large transaction

buurd — Wed, 16 Jun 2010 07:44:08 GMT

Some more information about the issue. It seems like it is versioning that is the problem. We have a default rule that makes sure that everything gets versioned, unless for those spaces where we manually removes the rule.

If we run the batch to copy files into a directory where the rule is so add versionable directly we get a long delay, not if the rule is added in the background. Same thing for files that already are versioned, there it seems like the "copy" of the file get the versioning directly and thereby case the delay.

If there are no versioning there are no delay and the number of files needed to cause a delay seems to be larger we tried with.