topic Re: Extracting XML meta data into aspects in Alfresco Archive

Extracting XML meta data into aspects

samuel_penn — Mon, 29 Sep 2008 16:29:01 GMT

Hi,I'm currently trying to get my head around the XML Meta Data extractor as described at http://wiki.alfresco.com/wiki/Metadata_Extraction.Something that isn't made clear, is what happens if the extractor is setup to write the data into a property field which only exists in an aspect? Is the aspect

Re: Extracting XML meta data into aspects

pmonks — Mon, 29 Sep 2008 19:17:45 GMT

what happens if the extractor is setup to write the data into a property field which only exists in an aspect? Is the aspect automatically added to the document, or does it have to already exist on the document?

This is exactly how it's intended to work, and in this case the aspect will get applied automatically by the XPath Metadata Extractor - it determines which aspect(s) to apply based on the target properties defined in the "mappingProperties" Spring bean property.

Also, is that the entire Spring configuration? If so, it appears that it's missing the metadata extractor registry Spring bean that enables metadata extraction for WCM content (see http://wiki.alfresco.com/wiki/Metadata_Extraction#Activating_Meta-data_Extraction_for_WCM for details).

Cheers,
Peter

Re: Extracting XML meta data into aspects

samuel_penn — Tue, 30 Sep 2008 07:38:54 GMT

what happens if the extractor is setup to write the data into a property field which only exists in an aspect? Is the aspect automatically added to the document, or does it have to already exist on the document?
This is exactly how it's intended to work, and in this case the aspect will get applied automatically by the XPath Metadata Extractor - it determines which aspect(s) to apply based on the target properties defined in the "mappingProperties" Spring bean property.

Okay, excellent. I figured that it could be doing something like that, but wasn't sure.

Any idea as to whether it gets applied whenever form content is saved to a user's sanbox, or just when it gets pushed to staging?

Also, is that the entire Spring configuration? If so, it appears that it's missing the metadata extractor registry Spring bean that enables metadata extraction for WCM content (see http://wiki.alfresco.com/wiki/Metadata_Extraction#Activating_Meta-data_Extraction_for_WCM for details).

No it's not - I only included the bits I changed, which was to replace the sample configs in the existing file. The avmMetadataExtracterRegistry, avmNodeService and avmMetadataExtracter are also defined at the top of the file.

Thanks,
Sam.

Re: Extracting XML meta data into aspects

samuel_penn — Tue, 30 Sep 2008 09:24:30 GMT

I've now got it working - I've gone back to removing all the namespaces from the various XML paths, and it is populating the properties in the user's sandbox and adding the aspect automatically. I've included the working definitions below just in case anyone finds them useful as another example.


<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE beans PUBLIC '-//SPRING//DTD BEAN//EN' 'http://www.springframework.org/dtd/spring-beans.dtd'>
 
<!–
   Sample configuration of a XmlMetadataExtracter in use within the WCM environment.
   
   This show how XML metadata extraction can be set up to extract metadata from different
   formats of XML.  It also shows how metadata can be extracted in WCM projects.
   
   Since: 2.1
   Author: Derek Hulley
–>
<beans>
   
   <!–
      In order to limit the number of extractors active for Web Content Management, a separate registry
      must be created for the extractors.
   –>
   <bean id="avmMetadataExtracterRegistry" class="org.alfresco.repo.content.metadata.MetadataExtracterRegistry" />
   
   <!–
      Configure the AVM services to broadcast the content update notifications.
   –>
   <bean id="avmNodeService" class="org.alfresco.repo.avm.AVMNodeService" init-method="init">
        <property name="dictionaryService">
            <ref bean="dictionaryService"/>
        </property>
        <property name="avmService">
            <ref bean="avmLockingAwareService"/>
        </property>
        <property name="policyComponent">
            <ref bean="policyComponent"/>
        </property>
        <property name="invokePolicies">
            <value>true</value>
        </property>
    </bean>
   <bean id="avmMetadataExtracter" class="org.alfresco.repo.avm.AvmMetadataExtracter" init-method="init">
      <property name="policyComponent">
         <ref bean="policyComponent"/>
      </property>
      <property name="extracterAction">
         <bean class="org.alfresco.repo.action.executer.ContentMetadataExtracter" >
            <property name="dictionaryService">
               <ref bean="dictionaryService"/>
            </property>
            <property name="nodeService">
               <ref bean="avmNodeService" />
            </property>
            <property name="contentService">
               <ref bean="contentService" />
            </property>
            <property name="metadataExtracterRegistry">
               <ref bean="avmMetadataExtracterRegistry" />
            </property>
            <property name="carryAspectProperties">
               <value>true</value>
            </property>
         </bean>
      </property>
   </bean>
 
   <bean id="extracter.xml.centrom.ArticleModelMetadataExtracter"
         class="org.alfresco.repo.content.metadata.xml.XPathMetadataExtracter"
         parent="baseMetadataExtracter"
         init-method="init" >
      <property name="mappingProperties">
         <bean class="org.springframework.beans.factory.config.PropertiesFactoryBean">
            <property name="properties">
               <props>
                  <prop key="namespace.prefix.lg">http://www.centrom.com/alfresco/localgov/model</prop>
                  <prop key="atoz">lg:atoz</prop>
               </props>
            </property>
         </bean>
      </property>
      
      <property name="xpathMappingProperties">
         <bean class="org.springframework.beans.factory.config.PropertiesFactoryBean">
            <property name="properties">
               <props>
                  <prop key="namespace.prefix.art">http://www.centrom.com/localgov/wcm/article</prop>
                  <prop key="atoz">/article/header/atoz/text()</prop>
               </props>
            </property>
         </bean>
      </property>
   </bean>
   
   
   <!–
      This selector examines the XML documents, executing the given XPath statements until a
      match is made.
   –>
   <bean id="extracter.xml.centrom.selector.XPathSelector"
         class="org.alfresco.repo.content.selector.XPathContentWorkerSelector"
         init-method="init">
      <property name="workers">
         <map>
            <entry key="/article">
               <ref bean="extracter.xml.centrom.ArticleModelMetadataExtracter" />
            </entry>
         </map>
      </property>
   </bean>
   
   <bean id="extracter.xml.centrom.XMLMetadataExtracter"
         class="org.alfresco.repo.content.metadata.xml.XmlMetadataExtracter"
         parent="baseMetadataExtracter">

      <property name="registry">
         <ref bean="avmMetadataExtracterRegistry" />
      </property>

      <property name="overwritePolicy">
         <value>EAGER</value>
      </property>
      <property name="selectors">
         <list>
            <ref bean="extracter.xml.centrom.selector.XPathSelector" />
         </list>
      </property>
   </bean>
</beans>
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Thanks,
Sam.

Re: Extracting XML meta data into aspects

steventux — Mon, 13 Oct 2008 10:08:47 GMT

Hi Sam,
I am a bit lost with the configuration steps needed to enable WCM Metadata Extraction.
Following the Wiki I've activated wcm-xml-metadata-extracter-context.xml (assuming this is done by renaming the file from *.xml.sample to *.xml, I haven't changed its directory location).
I've adapted the example you gave in this topic and changed the logging level for the XPathMetadataExtracter to debug so I should see content being processed.
I'm wondering if I am missing something vital as I see no log activity like the output you were getting (I see [metadata.xml.XPathMetadataExtracter] preRegister called).
Do I need to activate the custom-repository-context.xml and register the extracter there as well?
Also I understand the XPath parts of the configuration as they are self explanatory but I am unsure how to alter the mappingProperties suitably, is this simply a case of assigning an abbreviation to the full schema URI for later use in searching?
Thanks
Steve

Re: Extracting XML meta data into aspects

steventux — Mon, 13 Oct 2008 11:41:36 GMT

Sorry, please ignore these questions. Something very basic was wrong, the SVN build script wasn't copying the wcm-xml-metadata-extracter-context.xml file into the webapp.

Re: Extracting XML meta data into aspects

samuel_penn — Mon, 20 Oct 2008 15:14:45 GMT

All the above seems to be working okay, however I've now added a date field to my aspect which I need to be set to be the current date[1] when the aspect is added.

To do this, I've created a Java class which implements NodeServicePolicies.OnAddAspectPolicy, and which sets the property to be the current date/time when the aspect is first applied. After adding a bean definition which points to this class into custom-model-context.xml, I've tested this in the DM side of Alfresco and it works as aspected - adding the aspect causes the date to be set to the current date.

However, when the meta-data extractor adds the aspect in WCM, this code isn't run, and the date property isn't set. Is there something else that needs to be configured to get the policy to fire in WCM?

Thanks,
Sam.

[1] It needs to be writable, so though it will normally be the created date I need my own property. Being able to set a default value for a date property of 'now' would also work (and be easier), but as far as I can tell, this isn't possible.

Re: Extracting XML meta data into aspects

pmonks — Mon, 20 Oct 2008 20:04:42 GMT

The AVM doesn't support behavioural aspects, so the best way to accomplish this is to add the current date into the XML (via a dynamic include) and then extract it (via XML Metadata Extractor) into the aspect.

More generally, it's best not to think of AVM aspects as anything more than an intermediate, temporary storage area between XML element values and the Lucene indexes. In fact in a future release we hope to do away with all this aspect / XML Metadata Extractor stuff for Web Form backed content, and instead allow the indexing behaviour to be configured directly in the Web Form XSD (via annotations, much the same way that we allow the Web Form UI widgets to be configured).

Cheers,
Peter

Re: Extracting XML meta data into aspects

samuel_penn — Mon, 20 Oct 2008 20:28:00 GMT

The AVM doesn't support behavioural aspects, so the best way to accomplish this is to add the current date into the XML (via a dynamic include) and then extract it (via XML Metadata Extractor) into the aspect.

I had moved to this way of doing things because I couldn't see a way of getting the current date into the XML. I hadn't thought of doing a dynamic include - presumably this would set a default value for the form element. Seems a bit of a messy way of doing it however.

Forms also don't seem to support datetimes, which was another (less serious) downside to sticking the data into the form.

More generally, it's best not to think of AVM aspects as anything more than an intermediate, temporary storage area between XML element values and the Lucene indexes. In fact in a future release we hope to do away with all this aspect / XML Metadata Extractor stuff for Web Form backed content, and instead allow the indexing behaviour to be configured directly in the Web Form XSD (via annotations, much the same way that we allow the Web Form UI widgets to be configured).

In this case it was just a workaround for the (apparent) lack of good date support in web forms. Annotations to control indexing (plus maybe auto-population of field values from data sources?) would be nice however.

I'll give the dynamic include a try.

Thanks,
Sam.

Re: Extracting XML meta data into aspects

samuel_penn — Mon, 20 Oct 2008 21:26:00 GMT

That seems to work, with one problem - because Web Forms don't allow datetime values, I have it stored in two different fields (one date, one time). Is it possible to combine these two fields into a single datetime on the aspect during the metadata extraction?

Thanks,
Sam.

Re: Extracting XML meta data into aspects

pmonks — Mon, 20 Oct 2008 21:50:27 GMT

What specifically doesn't work about datetimes? Are you using the "xs:datetime" data type in your XML Schema?

The list of supported XML Schema structures and data types is at http://wiki.alfresco.com/wiki/Forms_Authoring_Guide#Overview_of_supported_XML_Schema_structures_and_data_types.

Cheers,
Peter

Re: Extracting XML meta data into aspects

samuel_penn — Mon, 20 Oct 2008 22:04:01 GMT

What specifically doesn't work about datetimes? Are you using the "xs:datetime" data type in your XML Schema?

Using a datetime causes an exception to be thrown when opening a form. If I use:


<xs:element name="expire" type="xs:datetime"/>
‍‍‍

Then I get the following error when creating a form:


org.alfresco.web.forms.FormProcessor$ProcessingException: org.alfresco.web.forms.xforms.FormBuilderException: error parsing schema: at line 27 column 59: src-resolve.4.2: Error resolving component 'xs:datetime'. It was detected that 'xs:datetime' is in namespace 'http://www.w3.org/2001/XMLSchema', but components from this namespace are not referenceable from schema document 'null'. If this is the incorrect namespace, perhaps the prefix of 'xs:datetime' needs to be changed. If this is the correct namespace, then an appropriate 'import' tag should be added to 'null'.
‍‍‍

The xs namespace is being used by the other elements, and they work fine.

That page you reference is why I assumed it was not supported, since under "Supported XML Schema Data Types in Alfresco 2.0 and higher" it is listed as unsupported. It is listed as supported earlier on the page, but since it's never worked for me, I always assumed the unsupported listing was the correct one.

Sam.

Re: Extracting XML meta data into aspects

pmonks — Mon, 20 Oct 2008 22:58:52 GMT

hmm…….strange! I just noticed that xs:datetime is listed as supported in the top table, and not supported in the bottom one.

Could I ask a big favour and have you raise a ticket for the lack of xs:datetime support in JIRA (http://issues.alfresco.com/), including the full exception stack trace from alfresco.log, then post the number back here?

Cheers,
Peter

Re: Extracting XML meta data into aspects

pmonks — Tue, 21 Oct 2008 02:39:23 GMT

Also, the enhancement request for the XSD driven indexing configuration is https://issues.alfresco.com/jira/browse/DW-5 - it'd be great if you could and/or comment on this if you feel it's a good idea.

Cheers,
Peter

Re: Extracting XML meta data into aspects

samuel_penn — Tue, 21 Oct 2008 07:33:02 GMT

hmm…….strange! I just noticed that xs:datetime is listed as supported in the top table, and not supported in the bottom one.

Could I ask a big favour and have you raise a ticket for the lack of xs:datetime support in JIRA (http://issues.alfresco.com/), including the full exception stack trace from alfresco.log, then post the number back here?

Actually, the problem is in the wiki page (which I tried to edit, but was defeated by the captchas). The clue was in another JIRA, which mentioned xs:dateTime not working in 2.9B. Sticking in a capital 'T', and it works (at least in 2.2.1). Given that other elements are listed with the correct case, someone who has better luck with captchas than me should probably fix the wiki.

Sam.

[1] https://issues.alfresco.com/jira/browse/ALFCOM-1798

Re: Extracting XML meta data into aspects

pmonks — Tue, 21 Oct 2008 18:42:33 GMT

Ah, so it's a typo in the wiki - I'll fix that now.

Are you seeing the issue described in https://issues.alfresco.com/jira/browse/ALFCOM-1798 on 2.2SP1?

Cheers,
Peter

Re: Extracting XML meta data into aspects

samuel_penn — Wed, 22 Oct 2008 08:03:47 GMT

Ah, so it's a typo in the wiki - I'll fix that now.
Are you seeing the issue described in https://issues.alfresco.com/jira/browse/ALFCOM-1798 on 2.2SP1?

No. Update and editing of the form works just fine. However, there does seem to be an issue with metadata extraction of dates. I am using the following configuration:


      <property name="supportedDateFormats">
          <list>
              <value>yyyy-MM-dd'T'HH:mm:ss</value>
              <value>yyyy-MM-dd</value>
          </list>
      </property>
‍‍‍‍‍‍‍‍

I have a xs:dateTime field which is being converted to a datetime property, and a xs:date field which is being converted to a date property. According to the metadata extraction page on the wiki, the above date formats should be tried in order until one succeeds. Trying the form with just one field, and just one date format, works fine.

However, it seems that regardless of the order of the two formats above, if both formats are listed then the xs:dateTime gets converted as a date - i.e. the time is always extracted as 0:00:00 as if the shorter format is always being matched. It seems I need both in there to be able to extract both fields, otherwise I get a date format error.

Sam.

Re: Extracting XML meta data into aspects

mcrocker — Thu, 06 Nov 2008 23:54:12 GMT

Hi im most definitely having the same issue.

I am trying to get your work around of using the xs:time to capture the time, but i havent been able to get it to display in any way other than publishTime: 1 January 1970 17:17…

i added the following code to the wcm-xml-metadata-extracter-context.xml but it seems that it doesn't get applied at all. <value>hh:mm:ss</value>

any advice ?

thanks,
Mat Crocker