topic Re: Script Node move in Alfresco Archive

Script Node move

smcardle — Wed, 13 Feb 2013 22:08:25 GMT

Hi AllI'm trying to get to the bottom of an issue in a JavaScript webscript controller.We have thousands of spaces to move to a new parent structure. The current structure has maybe 50 parent spaces each of which has several thousand sub spaces to move under a new top level space. The following is a

Re: Script Node move

afaust — Thu, 14 Feb 2013 08:54:16 GMT

Hello,

which version of Alfresco are you using? Are you using Lucene indexing?

Regards
Axel

Re: Script Node move

lista — Sun, 17 Feb 2013 23:02:02 GMT

I don't think that's the right approach, having that many nodes under a single node will result in poor performance. You should really re-think if you need this.

Re: Script Node move

smcardle — Mon, 18 Feb 2013 22:01:27 GMT

Hi Axel

Thanks for responding.

Yes we are using Lucene indexing.

Regards

Re: Script Node move

smcardle — Mon, 18 Feb 2013 22:05:42 GMT

Hi Lista

Unfortunately we can't change this structure and I would not have thought that this would cause performance issues.

Each client has multiple child folders containing different document types. It's just that we have thousands of Clients under a Clients node.

There is no other logical grouping we can apply without significant rework in the underlying webscripts.

Steve

Re: Script Node move

smcardle — Mon, 18 Feb 2013 22:09:13 GMT

Sorry Axel.

Also forgot to mention this is a 4.2.c community install

Re: Script Node move

mitpatoliya — Tue, 19 Feb 2013 06:39:30 GMT

Steve
I think you can optimize it like this


try {
            var existingclient=clients.childByNamePath(child.name)
                                       
               if(null!=existingclient) {
                  // Should not be required as move should return true or false.
                  // This does stop move throwing exceptions and causing the whole transaction to abort
                  remove = false;
                  notMovedClients.push(child.name);
                  continue;
               }
               
               else {
                  var moved = child.move(clients);
                  movedClients.push(child.name);
               }
            }catch(e) {
               // The move should not throw an exception but it does so catch it
               // and treat it as a failure to move
               remove = false;
               notMovedClients.push(child.name);
            }
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Also you want to keep track of the clients which are moved that is adding overhead in this script

One more suggestion if it gets much more complected you can always go for JAVA backed webscript.

Re: Script Node move

afaust — Tue, 19 Feb 2013 09:18:07 GMT

Hello,

ok, this explains some of the duration. When you are using Lucene you have to keep in mind that changes you do in a transaction are indexed before the transaction completes. If you move a couple of thousand client folders (with how many documents?) in one web script call, the indexer takes quite a bit of time to reindex all moved folders AND the contained structures. The move itself probably completed in just a few minutes, but the indexing part is the one that really takes a long time. You can't optimize this away by rewriting your script, but you may want to think about disabling indexing while you perform the move and then running a re-index afterwards (which should be faster, since it can run in parallel).

As far as the structure of your folders goes: If you stick to DB-based paged navigation or direct ID lookups, you should be fine concerning the navigation performance. But if remotely possible, I'd personally try to find a way to sub-divide the clients further, e.g. have subfolders for clients whose ID begins with 101, 102 … etc. - not so much for performance sake, but for administration / navigation in Alfresco views.

Regards
Axel

Re: Script Node move

smcardle — Tue, 19 Feb 2013 21:50:52 GMT

Thanks Axel

I had considered the lucene indexing and was in-fact working on switching it off prior to migration just to test that.

We don't have an issue with alfresco views as we never use the Alfresco UI in deployment. The whole repository is accessed via webscripts and we never do general queries i.e. get all clients.

I'll get back with results for disabling the indexing

Thanks

Steve

Re: Script Node move

smcardle — Wed, 20 Feb 2013 23:40:46 GMT

Hi Axel.

How do I completely disable indexing ?

lucene.indexer.contentIndexingEnabled=false
This property controls whether or not the content of the documents is indexed. If false, content is not indexed.

The content may not be indexed but it seems that everything else is…..

This has had little effect on the time it takes to run the migration of clients from the branch structure as the documents currently stored under a client are very small, text files in the order of 100-200 bytes only

So, my webscript is throwing thousands of exceptions during the move of a branch containing thousands of clients.

Basically, I want to switch indexing off completely, mover the clients from the branched into their new client structure, re-enable indexing, stop alfresc and set properties for a full re-index and viola…..

Only I can't do this due to large numbers of failures during the move that then rolls back the transaction

What in the webscript could be causing the exceptions to occur?

Regards

Steve

Re: Script Node move

afaust — Thu, 21 Feb 2013 16:37:43 GMT

Hello,

could you provide examples of your exceptions?

You have not yet turned indexing off completely in the same transaction as your move operation - this can only be achieved by using

index.tracking.disableInTransactionIndexing=true‍

.

Regards
Axel

Re: Script Node move

zladuric — Thu, 21 Feb 2013 20:04:24 GMT

One thing that I'm wondering is if you're just accessing via webscripts, why is it that you have to get the stuff under one folder? It seems to me that lucene would find whatever you want, wherever it may be.