topic Re: Batch import of metadata / aspects on Alfresco? in Alfresco Archive

Batch import of metadata / aspects on Alfresco?

trelofysikos — Fri, 19 Dec 2008 09:17:52 GMT

Hi,I would like to batch import documents on the Alfresco repository.I m able to do this with CIFS/ FTP/ WEBDAV.These documents have certain metadata which i store in a database.How can I give properties / aspects easily(btach) on my files?Is it possible to batch import metadata or aspects on Files

Re: Batch import of metadata / aspects on Alfresco?

pmonks — Thu, 01 Jan 2009 16:39:54 GMT

I'd suggest using a Web Script (http://wiki.alfresco.com/wiki/Web_Scripts) for uploading the metadata. Your batch import process would upload files as usual, then call a Web Script to set the metadata on those files.

Cheers,
Peter

Re: Batch import of metadata / aspects on Alfresco?

mdutoo — Tue, 26 May 2009 17:22:43 GMT

Hi

Beyond webscripted import as pmonks said, you can use the Open Source ETL tool Talend 3.1 and its Alfresco connector to import data (taken e.g. from XLS, XML, databases or anything Talend can look into) into Alfresco as any kind of Alfresco content (documents, folders, custom…).

Otherwise there is an ACP generator (but I didn't test it) or you could even develop a specific automated action, though that probably would amount to more work.

Some links :
Talend 3.1 including the Alfresco connector is available at http://www.talend.com/download.php
Its Alfresco connector has been contributed on the Alfresco forge at http://forge.alfresco.com/projects/etlconnector/
More information at http://knowledge.openwide.fr/bin/view/Main/AlfrescoETLConnector

Re: Batch import of metadata / aspects on Alfresco?

pmonks — Tue, 26 May 2009 20:25:39 GMT

That reminds me - OpenMigrate (http://www.tsgrp.com/Open_Source/OpenMigrate/open-migrate.jsp) does something like this too. One of my colleagues used it quite extensively for a Hummingbird -> Alfresco migration and thought it was pretty good.

Cheers,
Peter

Re: Batch import of metadata / aspects on Alfresco?

mdutoo — Wed, 27 May 2009 08:51:18 GMT

Interesting, I didn't know about it, thanks for the feedback.

Re: Batch import of metadata / aspects on Alfresco?

groutal — Mon, 15 Jun 2009 14:20:56 GMT

Hi Mdutoo, thank you for the information. About Talend Open Studio, it is an open source ETL tool for data integration experts with a user-friendly GUI, easy to learn for a non-technical user. What distinguishes Talend, when it comes to business users, is the tMap component which allows the user to get a graphical and functional view of integration processes. For more information: http://www.talend.com/

Re: Batch import of metadata / aspects on Alfresco?

trelofysikos — Wed, 22 Jul 2009 11:31:37 GMT

thanks for the feedback guys.

I gave talend a wuick try and thought it was a bit confusing. If anyone can help me with how to import a certain tree structure with metadata in alfresco i would be gratefull.

Also i'm now downloading openMigrate which seems promising. Anyone ever used this for batch import of documents + metadata in alfresco ?

Re: Batch import of metadata / aspects on Alfresco?

groutal — Mon, 03 Aug 2009 13:36:13 GMT

Hello trelofysikos,

With Talend you can perform many functions and if you are trying to load metadata in Alfresco, you can perform that operation easily. Talend has connectors to Alfresco so you can import metadata from Alfresco.
About importing a tree structure, you should be able to do that with Talend.

If you have any questions, feel free to ask, we will answer the best we can.

Re: Batch import of metadata / aspects on Alfresco?

itbeb — Mon, 03 Aug 2009 17:31:03 GMT

What does one do if you want to migrate ordinary Windows file system docs to Alfresco and want to keep the dates in tact, i.e. you want the created and modified dates to stay the same when you upload it to Alfresco and not to change to the current date?

Re: Batch import of metadata / aspects on Alfresco?

trelofysikos — Wed, 02 Sep 2009 12:52:20 GMT

Hello trelofysikos,

With Talend you can perform many functions and if you are trying to load metadata in Alfresco, you can perform that operation easily. Talend has connectors to Alfresco so you can import metadata from Alfresco.
About importing a tree structure, you should be able to do that with Talend.

If you have any questions, feel free to ask, we will answer the best we can.

Hey Groutal,

can you give me a simple example of one file+metadata import to alfresco using Talend ?
like a step by step tutorial ?

Re: Batch import of metadata / aspects on Alfresco?

groutal — Fri, 04 Sep 2009 09:19:33 GMT

Hey,

Absolutely! Check out the Talend tutorials available on the website: http://www.talendforge.org/tutorials/menu.php

Re: Batch import of metadata / aspects on Alfresco?

bwakkie — Wed, 16 Sep 2009 19:37:08 GMT

ps as far as I understood does Talend only support alfresco version 2.x if I read this correctly.

Re: Batch import of metadata / aspects on Alfresco?

groutal — Mon, 21 Sep 2009 12:58:32 GMT

You might want to check the component page on the Talendforge website. The latest update on the component was made on Talend version 3.1.

Component page:
http://www.talendforge.org/components/index.php

Re: Batch import of metadata / aspects on Alfresco?

freebsdboy — Thu, 08 Oct 2009 17:06:40 GMT

This is rather strange.. the process and hoops to go through to migrate data. Anyone who deploys Alfresco needs to do this.

There is no prebuild acp file creator? or no one has shared theirs?

I'm walking through the steps to create the acp or metadata extract using Talend, a very cool product.
Using tutorial http://www.talendforge.org/tutorials/tutorial.php?idTuto=3#step53

But the turotial is generic. So what creates the import file on a Linux server?
a good old ls -l /tmp/importmetadata.file
or
find . -print | xargs ls -gilds >/tmp/importmetatdata.file

Re: Batch import of metadata / aspects on Alfresco?

rliu — Fri, 09 Oct 2009 04:57:51 GMT

freebsdboy,

You are absolutely correct. Most customers using Alfresco will most likely be migrating documents, metadata, content, etc. from an existing CMS or some form of document management system. My approach was to re-engineer the exported Alfresco Content Package (ACP):

1. Define the content model as needed.
2. Deploy to environment where you can create some test content or upload a document of some kind that would be representative of the migrated content.
3. Do an export of your test data.
4. Review the schema from the XML that was generated from the export.
5. Create a program that can generate an XML that reflects the XML file that was exported.
6. Package into an ACP.
7. Import into Alfresco.

This was not the easiest of solutions, but I think it was a good approach. I wrote a program that generated the XML and other content items. Build a stable repeatable process along the way as it'll require a lot of patience and testing. My ACP file that I used to import worked in various Alfresco Labs 3.0 stack. I was able to package 600 documents in its respective space. Though, 600 was all I had, I believe this approach can be repeated for larger volume.

Best of luck!

Re: Batch import of metadata / aspects on Alfresco?

freebsdboy — Fri, 09 Oct 2009 20:00:29 GMT

Care to share the work you've done?

Hey this is open source - we all shouldn't have to redo the same thing.

Re: Batch import of metadata / aspects on Alfresco?

pmonks — Fri, 09 Oct 2009 21:17:01 GMT

It's not an all-singing, all-dancing content migration tool (OpenMigrate is a better choice if that's the requirement), but I've put a simple filesystem import process into Google Code at http://code.google.com/p/alfresco-bulk-filesystem-import/. This package is intended to simply and efficiently handle the basic use case of importing a large number or size of files & folders from local disk into Alfresco.

Please take the time to read the readme file [1] before using it. The current version is still under heavy development and has a number of major functional gaps (eg. no support for loading metadata yet, although that work is in progress) that will preclude it for specific use cases, but for the basic case of quickly loading in a large number of assets it already works well.

Cheers,
Peter

[1] http://code.google.com/p/alfresco-bulk-filesystem-import/source/browse/trunk/README.txt

Re: Batch import of metadata / aspects on Alfresco?

norgan — Sat, 10 Oct 2009 17:56:16 GMT

Hi Rich liu,
what where the major problems you had to solve ? Would this offer itself to be written as to be so generic to be of public use ?

i think about something like XSD transformation or atleast a XML defintion of the dataformat, which is to be imported (worst case - from an excel file or CSV file) Maybe this could go with Talend or become an alfresco forge project ?

Regards, Norgan

Re: Batch import of metadata / aspects on Alfresco?

rliu — Tue, 13 Oct 2009 04:53:13 GMT

It'll take a little time to generify the code and make it extensible. As soon as I have something I feel worthy of publishing, I will provide it to the community. In the meantime, the actions taken as I described should allow you to migrate data into Alfresco (inclusive of custom properties).

Re: Batch import of metadata / aspects on Alfresco?

ajkalbente — Mon, 19 Oct 2009 12:23:38 GMT

Inform us on what is going on with the project! Thanks.