topic Setup for import 1 mio files in Alfresco Archive

Setup for import 1 mio files

ustraub — Mon, 12 Aug 2013 10:18:07 GMT

Hi,I want to import several millions of files into Alfresco. The file names are numerical values like 1000001, 1000002, etc. The files are later accessed by their file name only. The directory where they will be stored in Alfresco is irrelevant for the later access from outside Alfresco. Alfresco i

Re: Setup for import 1 mio files

mrogers — Mon, 12 Aug 2013 10:43:56 GMT

Some answers.

Content ingestion is concurrent with retries.

Yes multiple threads will help.

You may also like to look at the bulk upload tool.

Re: Setup for import 1 mio files

ustraub — Mon, 12 Aug 2013 11:22:05 GMT

Hi,

what do you mean by "Content ingestion is concurrent with retries."?

If I understand you correctly your recommendation is to use configuration (2) with 10 threads? Theoretically it should be at best 10 times faster than configuration (1) with 10 threads (in practice a factor 5 would be great), is that correct?.
Configuration (1) with 1 thread would as fast as with 10 threads, correct?

What performance do you expect with your bulk upload tool?
Currently we have 0,3 sec per file uploaded with configuration (1) and 1 thread (and it gets worse with more threads).

Regards
U.Straub

Re: Setup for import 1 mio files

mdutoo — Wed, 09 Oct 2013 11:16:50 GMT

Hi

In addition to the bulk upload tool, there is the Alfresco ETL Connector for Talend, which uses an ETL-optimized version of the native bulk import services (per file transaction & error code…) :

http://knowledge.openwide.fr/Main/AlfrescoETLConnector

Regards

Re: Setup for import 1 mio files

jego — Fri, 11 Oct 2013 07:28:19 GMT

There is also the migration-center to import files from filesystem into alfresco. You can download a free version here:
http://www.migration-center.com/free-versions/

Thanks,
Jens

Re: Setup for import 1 mio files

lista — Tue, 22 Oct 2013 17:35:00 GMT

The second approach is mandatory, no question about it.
http://forums.alfresco.com/forum/developer-discussions/technical-architecture-discussion/folder-structure-alfresco-10112013