topic Re: Lucene/SOLR Stemming Analyser in Alfresco Archive

Lucene/SOLR Stemming Analyser

stevegreenbaum — Tue, 29 Jul 2014 22:31:39 GMT

I am using the porter stemming analyser for d_content but noticed that stop words are not removed from the index for new documents I add. I added the stopword filter to schema.xml underneath the lowercase filter but the stop words still exist in the index. Is this the correct approach Is there ano

Re: Lucene/SOLR Stemming Analyser

kaynezhang — Thu, 07 Aug 2014 04:35:00 GMT

Adding a filter to alfrescoDataType file in schema.xml will have no effect on d_content data type,alfresco 4.x did not use analyer configured in schema.xml for d_content data type.
It will use data type Index Analyzers configured in

${SOLR_CONFIG_ROOT}/workspace-SpacesStore/alfrescoResources/alfresco/model/dataTypeAnalyzers__{your locale}.properties.‍

if there is no properties for you locale ,it will use the default AlfrescoStandardAnalyser.

Re: Lucene/SOLR Stemming Analyser

stevegreenbaum — Tue, 12 Aug 2014 19:57:01 GMT

Are you saying that schema.xml is not used at all for setting analysers or are you saying it just doesn't have an effective on d:content? So for which property types is the whitespace analyser that is declared in schema.xml used by Alfresco?

Is there a way to apply a filter to the analyser which is set in dataTypeAnalyzers__{your locale} so I can add the stopword filter to the portersnowball analyser?

Re: Lucene/SOLR Stemming Analyser

kaynezhang — Wed, 13 Aug 2014 02:38:44 GMT

1.In alfresco 4.x all fields are dynamic field,and all filed type are alfrescoDataType( you can see it in schema.xml file).
2.AlfrescoDataType will use SolrLuceneAnalyser as index analyser.
3.SolrLuceneAnalyser is an wrapper analyzer ,it will analyse properties according to the property definition.for example:
a)for some regular fields(FIELD_ID、FIELD_DBID） ,it will use fix analyzer (LongAnalyser、VerbatimAnalyser）;
b)and for d:content/d:text/d:mltext it will use MLAnalayser;
4.MLAnalayser will load analyzer according locale which is configured in

  ${SOLR_CONFIG_ROOT}/workspace-SpacesStore/alfrescoResources/alfresco/model/‍

,for exmaple for english locale it will use analyzers configured in dataTypeAnalyzers_en.properties file


   d_dictionary.datatype.d_text.analyzer=org.alfresco.repo.search.impl.lucene.analysis.AlfrescoStandardAnalyser
   d_dictionary.datatype.d_content.analyzer=org.alfresco.repo.search.impl.lucene.analysis.AlfrescoStandardAnalyser
   ‍‍‍‍

That is for d:text and d:cotent property org.alfresco.repo.search.impl.lucene.analysis.AlfrescoStandardAnalyser is used

If you want to use portersnowball analyser and apply a filter to it ,you can implement a new index analyzer that extends SnowballAnalyzer, in your analyzer override tokenStream method and call your StopFilter. And then configure your custom analyser in

${SOLR_CONFIG_ROOT}/workspace-SpacesStore/alfrescoResources/alfresco/model/dataTypeAnalyzers__{your locale}.properties
‍‍

Re: Lucene/SOLR Stemming Analyser

stevegreenbaum — Wed, 13 Aug 2014 15:52:32 GMT

Thank you! I appreciate your thoughtful response. Steve

I wanted to share a response I received via Alfresco support this morning. They are indicating that even if the stopwords are removed, a search using those stopwords will still find the document because a non-tokenized version is also stored (I assume even if you don't specify "Both" in the model for tokenization). The AlfrescoStandardAnalyser is supposed to remove stopwords, but I was finding that I could still search on them.

Here is their response "Alfresco implementaion of Solr every document is indexed on 2 different ways (so it uses twice space than lucene), one using the locale and its defined analyzer (and stopwords) and another raw idexation is done for the "crosslanguage" (multilingual). They are found in your case on the cross language search. As stop words are not the same across languages we leave all words in. as result … you cannot "not find" stopwords when you specifically search for them. "

Re: Lucene/SOLR Stemming Analyser

kaynezhang — Thu, 14 Aug 2014 01:58:00 GMT

I can't agree all .
I guess for "d:content/d:text/d:mltext" type, cross language search should be implemented like this:
For every definite locale , locale analyzer defined in dataTypeAnalyzers_**.properties will be used.
For the crosslanguage one,there also should be an analyzer to be used ,either it is server default locale's analyzer or just AlfrescoStandardAnalyser.
So either way ,we can implements your requirements by customizing your locale's analyzer and AlfrescoStandardAnalyser.

This is just my guess,I have not tested it,correct me if I'm wrong.