topic Re: 44 million document require one year for load on alfresco!! in Alfresco Archive

44 million document require one year for load on alfresco!!

necipakca — Mon, 11 Apr 2011 15:34:14 GMT

Hi,I am trying to develop alfresco,I have added approximately 120 custom models,And i am trying to migration for move on alfresco our old system documents.There are approximately 44,000,000 document and when i try to load its on alfresco using java client, it requires approximately one year for comp

Re: 44 million document require one year for load on alfresco!!

mrogers — Mon, 11 Apr 2011 15:45:45 GMT

You probably want to have some sort of upload tool. There's one developed by Peter Monks that uploads directly from filesystem to an alfresco repo.

There was a presentation on this type of requirement at the last alfresco DevCon. The other keys points I remember from that talk were were to turn indexing off on the server doing the uploading and to adjust the server to optimise the write use case. You probably also need to consider whether you can upload in parallel.

Re: 44 million document require one year for load on alfresco!!

necipakca — Tue, 12 Apr 2011 11:06:04 GMT

Thanks mrogers,
Firstly,
Our all documents are picture jpg file so I think they won't be indexing and i did not use indexing for metadatas of my custom models too.
Secondly,
I did try parallel upload but all threads are waiting each others so there is no differences abouth one thread or multi thread because threads are waiting by alfresco.I think alfresco use queue for upload.Why? i couldnt understand.

I will examine to OpenMigrate , i hope it can solve my problem.

Best Regards

Re: 44 million document require one year for load on alfresco!!

jneeve — Tue, 10 May 2011 20:49:26 GMT

We multi-thread many of our uploads and operations for nightly batch processing (using the Alfresco java api) and I can process about 5000 documents in under 20 minutes.. (in our case these arent even physical files, just binary files stored in a seperate database)

If i needed to make it even faster I have looked at Peter's tool and it appears it should be a faster way..

It might help to post your server specs and memory settings..