topic Re: Document batch-upload in Alfresco Archive

Document batch-upload

mikechenosky — Sat, 23 Jun 2007 00:11:15 GMT

Hi, does Alfresco support batch-upload of documents to its document library?if so, how?If not, can someone suggest how to get around it?Thanks,mike

Re: Document batch-upload

mikeh — Sat, 23 Jun 2007 01:50:57 GMT

Hi,

FTP/CIFS/WEBDAV can all be used for batch update - also a user posted the code in these forums for a ZIP file import action where you upload a single ZIP containing all the files you want and it gets exploded out into the Alfresco repository.

Please see the wiki for CIFS and/or FTP configuration.

Thanks,
Mike

Re: Document batch-upload

kevinr — Sat, 23 Jun 2007 09:07:22 GMT

Batch import via ZIP file upload has been added to the web-client UI for 2.1.

Thanks,

Kevin

Re: Document batch-upload

gerald_quimpo — Thu, 19 Jun 2008 05:54:12 GMT

Hi Kevin, or anyone else,

Batch import via ZIP file upload has been added to the web-client UI for 2.1.

How do I do that (operationally, in the UI)? I've been looking around and it's not clear to me how to do it.

Gerald

Re: Document batch-upload

tommorris — Thu, 19 Jun 2008 13:27:58 GMT

Off the top of my head, you can do this for WCM projects.
When you browse your sandbox for a particular web-project, the 'Create' icon drop-down list has a 'Bulk Import' option.
I think kevnr might mean that. Did you mean bulk import for regular DM spaces?

I did notice that the code for the regular ACP import action, can also import ZIP files…
So you can try a rule that kick starts that action for ZIP file-types.

Tom

Re: Document batch-upload

gerald_quimpo — Thu, 19 Jun 2008 22:20:04 GMT

Did you mean bulk import for regular DM spaces?

yes, what I meant was bulk uploading many documents (in my prototype/test,
it's around 10,000 small (less than 512 bytes each) XML files. I don't need aspects,
title, description, etc to be set. I just need to push many files into the repository.

I find that writing files to alfresco via the CIFS or NFS share is very slow. In my
prototype/test, writing to a raw disk filesystem takes 7 seconds or so. Writing the
same files to CIFS takes 34 minutes or so. Deleting those files (or, actually,
the folder in which they reside and specifying all files and folders inside it to be
deleted too) is also very slow (I'm not timing it, but it's been around 3 minutes now,
while I've been writing/editing this post).

This is just my desktop development server, one drive, 2G of RAM, tomcat and
alfresco have 1G of that. but the overhead is very high. I'd be happy with writes,
reads and deletes being 5 times slower than to the raw filesystem. Even
10 times slower would be OK (my 10,000 files would be written in 70 sec).
But almost 300 times slower is a problem. Is this normal? Or are there well known
performance tuning tips that I might missed?

I did notice that the code for the regular ACP import action, can also import ZIP files…
So you can try a rule that kick starts that action for ZIP file-types.

Ok. I'll try to look into that. In fact, though, I'm eventually going to have to do this
in a program (users will upload a single large file, the program, or rule+action will
take that large file and split it into many individual records/xml documents). So
I'm looking at the web interface mainly as a proof of concept, that high speed
bulk uploading is possible at all. Then I'd wrap whatever the right technique is
in code.

Thanks for your reply!

Gerald

Re: Document batch-upload

ttsherpa — Sat, 21 Mar 2009 07:43:14 GMT

Hi everyone,

I'm quite new to alfresco, but I have a testing installation that seems to work, and while doing a batch upload of several thousands very small text files into Alfresco Enterprise in one of our LInux servers I notice the extremely slow speed you mention (100 times slower than a raw file write!!!!). Is this normal? Did you find any solution or explanation to the problem? I use FTP but I guess that similar bad results were obtained with zip files in the web interface, or with cifs.

What do you think?

TIA

Re: Document batch-upload

mdutoo — Tue, 26 May 2009 17:18:31 GMT

Hi

Don't be so sure, I didn't do a comparison on Alfresco 3, but on 2.x CIFS was notoriously faster than FTP…

Re: Document batch-upload

lfcohen — Thu, 30 Jul 2009 10:09:03 GMT

We are also facing severe problems. We use WebDAV and need to migrate a lot's of files and folders, basically, a full structure into our Alfresco.

Not only the transfer is painfully slow even on the local network where the server is located, as well as doing it remotely.

And a major problem is when we try to send a structure with folders and nested folders and more nested folders, regardless if they have files or not, we get "internal server error 500" and major problems!

We are using both the native WebDAV "client" on Mac and Windows, as well as CrossFTP (using WebDEV protocol) on both platforms as well..

Any clue what might be doing this ?????

Thanks for any feedback!

Leonardo

Re: Document batch-upload

mmurphy — Fri, 31 Jul 2009 21:07:51 GMT

I'm giving Alfresco a trial run and was hoping to give a recommendation to use if for our company's internal document management and collaboration. To say my experience so far has been "frustrating" would be an understatement. Anyway, I'm trucking along hoping it will all come together.

My problem as it relates to this thread is almost exactly the same as described above. FTP and CIFS are both very slow copying files. 200 MB took almost 30 minutes on a Gig Ethernet network. I tried copying the files to the server in windows then connected to the localhost share and copied the files that way into the repository. Local drive into the repository took about the same amount of time as over the network, close to 30 minutes.

This really isn't acceptable for a production environment. I have almost 2 TB of data to potentially upload into this repository, A task that will take about 2 and a half years at this rate.

Re: Document batch-upload

mikeh — Fri, 31 Jul 2009 21:23:19 GMT

It's unlikely the network is ever going to be the limiting factor. What sort of server are you running Alfresco on? Also what database. Is the database local?

Alfresco scales very well, but you need to make sure you've got the hardware to match. http://www.alfresco.com/media/releases/2008/01/unisys-benchmark/

Did you see this wiki ariticle? http://wiki.alfresco.com/wiki/JVM_Tuning

Thanks,
Mike

Re: Document batch-upload

sergisat — Fri, 07 Aug 2009 10:23:17 GMT

I have some problems too, I tested with Alfresco3Stable and with new Alfresco 3.2 in a vmware esx 3.5 host with 128GB RAM and the storage is stored in NetApp filer.

The virtual machine configuration is 4vcpu and 16Gb RAM with Centos 5.3 x86_64 / Ubuntu server 9.04 x86_64.
My database is Mysql 5.1.36 Community in another virtual machine with 4vcpu and 16Gb RAM in Centos 5.3 x86_64 .

My jvm configuration line is:
JAVA_OPTS="$JAVA_OPTS -XX:NewRatio=2 -server -Xcomp -Xbatch -Xss1024K -Xms2G -Xmx2G -XX:NewSize=1G -XX:+UseConcMarkSweepGC -XX:+CMSIncre
mentalMode -XX:MaxPermSize=256M -XX:CMSInitiatingOccupancyFraction=80"

Alfresco is slowly copying files over CIFS/FTP and more slowly deleting files over Alfresco Explorer.

Re: Document batch-upload

mrogers — Fri, 07 Aug 2009 10:43:10 GMT

Is your alfresco running well in vmware?

Have you seen this thread http://forums.alfresco.com/en/viewtopic.php?f=8&t=19990&p=66193&hilit=vmware#p66193

Have you installed the Microsoft loopback adapter?

Re: Document batch-upload

sergisat — Fri, 07 Aug 2009 10:56:12 GMT

I dont use Microsoft operating system for alfresco, I installed in Centos 5.3 and in Ubuntu Server 9.04.

Alfresco is working slowly in uploads/deleting files and if I explore nodes with 1500 elements is slowly too. I have folders in alfresco with 1500 elements and is more slowly to see. If folder have around 100 elements is more fast.

Re: Document batch-upload

loftux — Fri, 07 Aug 2009 11:46:58 GMT

@sergisat
Maybe you can try to change the Batch size for indexing to defer the indexing. See http://wiki.alfresco.com/wiki/Full-Text_Search_Configuration
Are you storing the index on the NetApp? I do not know how fast a NetApp is compared to local disk, but the index storage location needs to be on a fast disk.
Have you looked at the mysql load? As you probably know :wink: Alfresco also needs to do db updates to store a file, maybe you have the bottle-neck there? Or the communcation in between.
Your hardware for both Alfresco tomcat and the db server looks well sized, looks like it should be able to handle this load (such as indexing while saving).

Re: Document batch-upload

usuf — Fri, 26 Feb 2010 21:43:37 GMT

I am having the same problem when uploading a directory with multiple files through webdav (painfully slow on a local Alf & local DB) . Has anyone found a way to speed this up?