topic Re: Bulk uploading using web scripts in Alfresco Archive

Bulk uploading using web scripts

mrsaqib — Fri, 10 Feb 2012 10:45:26 GMT

Hi , I have written a web script which gets called each minute (cron job). That script starts uploading files from the sub-folder of one of my defined directory, each sub-folder contains around 2000 files to process. Now, as the script is being called automatically after each minute despite of wheth

Re: Bulk uploading using web scripts

openpj — Sat, 11 Feb 2012 10:41:34 GMT

I suggest to work on a transactional environment on WebScripts, that is by default if you set the value of the property "transaction" (in the descriptor file) to "required" or "requiresnew".

If you would like to work in a similar way of Java threads you could set "requiresnew" and the Spring context will create a new transaction for each WebScript call.

Remember that if you set the value of the property "transaction" to "none" this means that you are working without any transactional support, this means that you could corrupt the repository with some critical operations. But it could be useful this setting if you need to explicitly manage the transactions using the TransactionService of Alfresco deciding when the container must execute a rollback of all the operations. This is possible only using the Java API of Alfresco (Foundation Services API).

By default WebScripts are transactional (if the transaction property has a "requiresnew" or "required" value) for ALL the operations that are executed inside of it, this means that if only one of the operation fails, all the changes will be rollbacked.

Hope this helps :wink:

Re: Bulk uploading using web scripts

mrogers — Mon, 13 Feb 2012 11:38:12 GMT

The problem with adding a new job every minute is that if each job takes longer than a minute you will eventually have a problem. You may also have contention between the multiple jobs so your throughput drops. However some degree of parallel processing is essential for the highest throughput.

If the traffic tends to even out then this may not be a problem.

If I were building it it I'd go for a single multi-threaded process or consider some sort of "throttle" somwhere.

Re: Bulk uploading using web scripts

mrsaqib — Mon, 13 Feb 2012 11:46:36 GMT

Yeah My Job (1 execution of script) takes more than minute, it depends upon the no of files but in most of the cases if takes many minutes. but each execution of job are having different data to process but eventually uploading to same alfresco repository.
I am having no issue till now regarding any conflict between jobs etc. but definitely i am concerned with the approach, i want to make it even better (efficient).