topic Re: Aspect (or property) with Geometry, Geo-Spatial Data type. in Alfresco Archive

Aspect (or property) with Geometry, Geo-Spatial Data type.

steve_tekell — Thu, 22 Jan 2009 21:54:14 GMT

Is it feasible to add geo-spatial information into Alfresco (or any JCR)?I would like to associate content items with spatial data - points, polygons, etc. Something stored in a spatial (geometry) typed column, so that it can be properly indexed and queried. There would then be some UI (google map

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

stevereiner — Fri, 23 Jan 2009 00:37:09 GMT

Although not the more general geo-spatial data type support you mentioned, there is some geo-location support in the Calais Integration Alfresco server extension (and auto geo-tagging/search UI in FlexSpaces including a google map). If geo-location info is available from Calais, it will be stored in an aspect on the semantic tags (categories) that can be automatically associated with docs based on content.

http://forums.alfresco.com/en/viewtopic.php?f=32&t=15766

http://integratedsemantics.org/2008/12/08/calais-integration-for-alfresco-geo-tagging-flexspaces-part-2/

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

steve_tekell — Mon, 26 Jan 2009 20:18:32 GMT

It looks like you are storing lat/lon as decimals (an existing data type) and there's no way to store anything other than a point.

So am I correct to think it's not feasible to add a new datatype with a corresponding sql type into Alfresco (or any JCR)?

Geometry types in the database is a requirement for me. I need more than point data, and I need to be able to efficiently query for something like 'all publications within or intersecting a given polygon between given dates with certain authors'. So while I see how I could build the UI creating/viewing geo-spatial data, there's no way extend the model in Alfresco to store it properly.

Although not the more general geo-spatial data type support you mentioned, there is some geo-location support in the Calais Integration Alfresco server extension (and auto geo-tagging/search UI in FlexSpaces including a google map). If geo-location info is available from Calais, it will be stored in an aspect on the semantic tags (categories) that can be automatically associated with docs based on content.

http://forums.alfresco.com/en/viewtopic.php?f=32&t=15766

http://integratedsemantics.org/2008/12/08/calais-integration-for-alfresco-geo-tagging-flexspaces-part-2/

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

mrogers — Mon, 26 Jan 2009 22:09:52 GMT

To answer your question yes it is feasible to add geo-spatial types. However as you have no doubt concluded the current version of Alfresco does not support these types and there would be issues with database support and the query engine.

This would make an interesting project for an extension to Alfresco.

How many people would make use geo-spatial searching if Alfresco supported it?

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

steve_tekell — Fri, 30 Jan 2009 17:36:05 GMT

I am little confused by your reply.

When I say "feasible" I didn't mean can Alfresco be changed to support geo-spatial types in a few years with Alfresco 4 or 5.
I mean can I, through existing extensions points in Alfresco 3, add support for sql geometry types.
For that matter, can I add support for any new data types?

Can this really be accomplished just as an extension?
I don't want to manage a fork or Alfresco to do this.

As to how many people will want to query spatial information in Alfresco, I dunno, but the examples I am familiar with are in science and government. Here there is a very significant unfulfilled need. The question is not whether or not will we will do this, but how - with or without Alfresco.

Now, forget querying for a moment, just having the ability to store spatial data in standard sql geometry types like any other properties in a aspect would be a good start.

To answer your question yes it is feasible to add geo-spatial types. However as you have no doubt concluded the current version of Alfresco does not support these types and there would be issues with database support and the query engine.

This would make an interesting project for an extension to Alfresco.

How many people would make use geo-spatial searching if Alfresco supported it?

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

mbailen — Mon, 08 Jun 2009 18:34:23 GMT

Hi,

Has anybody made any movement on this topic (integrating PostGIS geo-spatial data with Alfresco)?

If no other solution is available, we are considering storing the geo-spatial data in a table outside of Alfresco with ID's pointing back to the documents. I was hoping for something more elegant.

Thanks, Mark

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

bnordgren — Tue, 01 Dec 2009 19:09:53 GMT

Also interested in geospatial aspects/properties.

In my organization (US Forest Service research), some people are just starting to ask for it, essentially implementing it in their legacy custom built document indexer (it's not really a content management system). However, I suspect its one of those things that would catch on like wildfire if it was available, and it seems to be something that no one else offers. Querying by the area referenced in the document: not possible if you're not the author or if you happen to use two different keywords for the same location.

+1 on the hibernate spatial, and retaining database independence.

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

bnordgren — Tue, 01 Dec 2009 20:52:29 GMT

Note: I just located a Lucene spatial contrib too. After poking around a bit in Alfresco's development website, it seems that if one wants to search for documents using any parameter, Lucene must be the thing doing the searching, right?

The downside of the Lucene extension is that it appears to be based on GeoHashing, which is essentially an algorithm for converting lat/lon points into a text representation. It cannot handle anything other than points. Further, I'm not clear on whether the hash can be directly used to determine proximity (e.g., does the document refer to a point which is within "x" distance of my search point P)? I'd think you'd have to convert ALL of the hashes back to coordinates, then do the calculation, which would seem to defeat the purpose of using a hash in the first place. Also, the spatial contrib seems to implement their own distance calculation and map projection algorithms instead of using a library for that.

Spatially aware databases usually maintain their own spatial indices for use with queries. If the metadata is stored in PostGIS, I see no advantage in using a Lucene spatial index.

So I guess the questions relevant to this forum are:

What is the component which needs to understand spatial objects if Alfresco is to implement spatial queries (Lucene? Hibernate? both?)

Can we just let the spatial database do the spatial query and somehow merge it with the results of the Lucene search for the other properties?

Being unaware of how Lucene works, am I to understand that it copies the properties out of the RDBMS containing the metadata, then creates indices and executes searches against its own copy? What guarantees that Lucene is in sync with the RDBMS?

Are there any recommendations as to how someone interested in this topic might proceed in the most expeditious manner possible, avoiding common new-developer pitfalls? (Note: no promises)

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

mbailen — Tue, 01 Dec 2009 21:58:44 GMT

Glad to see some more interest in this topic. I am dealing with similar issues and if we stick with Alfresco, we will need to implement a solution by summer of 2010. Until it becomes a priority for the Alfresco team, I don't see them adding spatial types to their architecture (whether it uses Lucene or not).

We have white-boarded a solution, but will likely not attempt it until the spring. Our initial plan was to avoid mucking about with Alfresco code. We intended on using an aspect (string) to hold the WKT of all geo-spatial entries in the DMS. These entries (document ID's and WKT) would then be dumped into an external PostGIS table (daily? to avoiding syncing issues). And then our searches would become two-phased. First narrow down the search through Alfresco (by keywords or other criteria) and then pass the document id's and WKT's to an external java/hibernate class for the second phase. The second phase would be the geo-spatial search and it would hit the daily-generated table (and likely be pretty expensive). The plan seemed simple enough on the white board, eh?

One issue we immediately spotted was the scenario where a huge number of documents are returned by Alfresco (perhaps zero non-spatial search terms were provided). Running every Alfresco Document ID through its own personal hibernate query seemed just silly. One optimization we were considering was to make Alfresco a little more geo-savvy by using the WKT to compute quadrants (a misnomer used in our mapping group… perhaps *sections* is a better word?). These quadrants could be a singular aspect, computed on ingestion. Using these quadrants, it may be possible to narrow down search results by a 100 or 1000, etc., depending on the schema. It adds some overhead, but may be worth the time. Also, it may depend a lot on your data.

Anyway, the solution above seems spectacularly bad when compared to a straight PostGIS query, but we haven't though of an easier way to accomplish this without altering the Alfresco code base. Let me know how you progress. Or if you have better ideas. We likely won't know more until we are elbow-deep into it. . .

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

bnordgren — Thu, 03 Dec 2009 20:12:28 GMT

The more I think about this, the more important a proper declaration of "scope" becomes to me. Geospatial operations, while not rocket science, are very tedious and easy to implement incorrectly. There is a steep learning curve for geospatial newbies, which only gets steeper when the various geospatial metadata standards start coming into play.

Take a look at GeoNetwork, particularly SpatialFilter and its children. If I read this correctly, this heirerchy should serve as an adapter layer between geospatial filters and the Lucene search engine. Geospatial filters are defined in the GeoTools library, which in turn leverages the topology algorithms implemented in the Java Topology Suite (JTS). JTS is, of course, the Java "original" of the C++ port more commonly known as the Geometry Engine Open Source (GEOS) library, which is in turn the code which PostGIS uses to do the heavy lifting.

If we want integrated, unified searches of geospatial and nongeospatial characteristics within Alfresco quickly, I'd suggest carving out the family of adapter classes listed above into a standalone Lucene module. Of course, you still have to brew up a geospatial UI for searches and possibly result display.

Another, less invasive, alternative is to delegate all searches or maybe just geospatial searches to GeoNetwork. Under this scenario, Alfresco exposes all of its geospatial metadata via one of the mutually supported mechanisms (i.e., WebDAV) to GeoNetwork. GeoNetwork could then perform searches on Alfresco content. The site could be configured to allow users direct access to the GeoNetwork web UI, or it could hide the UI and expose search results via Alfresco's interface to OpenSearch. Within Alfresco, "geospatial metadata", in the XML format comprehensible to GeoNetwork, could be expressed as its own document type in the content model. A child association could be formed between "something in the repository" and the new geospatial metadata type. Alfresco's main task would then be to aggregate all the geospatial metadata it manages, ensure that "links to the actual content" are correct, inject whatever non-geospatial metadata it can from the associated object, and provide access to the file. Alfresco can search everything that's not geospatial, and GeoNetwork can possibly search everything. We can come up with an XML metadata template for a lat/lon point and/or a box. Maybe an importer based on GDAL which converts shapefiles to the XML format for more complex shapes.

Is there any possibility of using a portal to hide the fact that two different web apps are being used? I don't know much about portals.

The downside is that GeoNetwork appears to be written in French and I'm a spoiled English speaker. Installation guides and such are English friendly, but I'm not sure I want to look too closely at the code.

Does this sound rational? I'm not an Alfresco user yet, so I don't have a good feel for what it can and can't do. My perceptions of ECM in general and Alfresco in particular may suffer from the sales pitch where it is quite capable of washing my car, feeding the dogs, and building a spaceship all in one. Think its realistic or pie-in-the-sky?

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

jck — Wed, 20 Jan 2010 21:14:15 GMT

Hi,
could you please give an example of a geo-spatial query?

In which way a specialized storage engine can process it in comparison with usual database engines?

JC

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

mbailen — Mon, 25 Jan 2010 17:34:42 GMT

See the PostGIS site for information on a specialized storage engine: http://postgis.refractions.net/

You can find lots of queries in their doc. Example, page through http://postgis.refractions.net/documentation/manual-1.4/ch04.html#id2717073

bnordgren: we intend to revisit some of the Lucene solutions (e.g. Local Lucene), to see what sort of performance/functionality we can get with Alfresco. Not sure about GeoNetwork. I think we looked at it and determined that using it would render much of the DMS functionality of Alfresco useless. This is pretty much what you were driving at in your previous post. Alfresco adds a lot at ingestion time… e.g. meta-data extraction, version control, ftp-access, categorizing, etc., that we like.

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

bnordgren — Tue, 04 May 2010 21:20:33 GMT

bnordgren: we intend to revisit some of the Lucene solutions (e.g. Local Lucene), to see what sort of performance/functionality we can get with Alfresco. Not sure about GeoNetwork. I think we looked at it and determined that using it would render much of the DMS functionality of Alfresco useless. This is pretty much what you were driving at in your previous post. Alfresco adds a lot at ingestion time… e.g. meta-data extraction, version control, ftp-access, categorizing, etc., that we like.

Excellent. Please keep me updated. I missed the "Local Lucene" solution.

In terms of the "GeoNetwork" solution, GeoNetwork doesn't ingest anything, it merely catalogs it. It's mostly designed to poll geospatial servers in order to present a consilidated list of what "stuff" exists on which servers. If GeoNetwork were to be used, I think I'd suggest developing a CMIS backend to it so that it could poll CMIS repositories too. Presumably the query would look something like : "select all documents with geospatial properties which have been updated since the last time I polled you". "Geospatial Properties" would/should look like normal text properties to Alfresco, and should contain the geometry in well known text (WKT). Alternatively, complex geospatial properties could be represented as an "association to" a Geographic Markup Language file. I believe GeoNetwork could also harvest any non-geospatial properties attached to the CMIS objects, making mixed searches possible.

Whatever route you take, I'd love to hear how this turns out!

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

bnordgren — Fri, 04 Jun 2010 00:18:39 GMT

This is an interesting comment in the "local lucene" svn repository:

Revision 172 - Directory Listing
Modified Wed Apr 29 03:49:25 2009 UTC (13 months ago) by pjaol

Updated to latest version of lucene and solr
fixed a multireader issue
migrated to use spatial lucene instead of locallucene
locallucene is dead, long live spatial lucene‍‍‍‍‍‍‍

Which I think brings us back to the spatial extension for Lucene. But wait, take a look at this: http://www.jteam.nl/products/spatialsolrplugin.html. It even looks to be commercially supported if that's your style…

Re: Aspect (or property) with Geometry, Geo-Spatial Data type.

janvg — Thu, 16 Dec 2010 12:58:58 GMT

To answer your question yes it is feasible to add geo-spatial types. However as you have no doubt concluded the current version of Alfresco does not support these types and there would be issues with database support and the query engine.

This would make an interesting project for an extension to Alfresco.

How many people would make use geo-spatial searching if Alfresco supported it?

I do believe geo-spatial types should be on top of Alfresco's road map. It would be very interesting to add geo data when uploading a document , but even more important might be that one can tag documents with geo-data.