topic Re: Lucene performance/wildcard searches not working in Alfresco Archive

Lucene performance/wildcard searches not working

dbevacqua — Wed, 18 Oct 2006 11:27:31 GMT

we have a repo with about 400k nodes and 2.8M properties. A a very simple search on the lucene index for @cms\:name:researchis taking over half an hour to complete (I have not actually bothered to let it finish). There will be tens of thousands of matches (searches for more obscure words are taking

Re: Lucene performance/wildcard searches not working

marcus — Wed, 18 Oct 2006 14:14:31 GMT

I must have been typing my post when you posted this… http://forums.alfresco.com/viewtopic.php?t=3680

I get matches when I use quotes around the search term, but not if I leave the quotes off.

Re: Lucene performance/wildcard searches not working

dbevacqua — Wed, 18 Oct 2006 15:18:30 GMT

Hi Marcus

Looked at your posting. What happens if you do

@\{http\://www.lateralminds.com.au/model/name/1.0\}jurisdiction:'Aus*"

?

I ask because for me

@cms\:name:"nano*"

returns nothing, but

@cms\:name:"nanotechnology*"

returns what I would expect.

Re: Lucene performance/wildcard searches not working

andy — Wed, 18 Oct 2006 16:20:27 GMT

Hi

You can limit the query results by using the SearchParameters object to define your search. See setLimitBy and setLimit. There is config to enable this on the client/UI search.

The time is not spent doing the lucene query but evaluating permissions.
If you have 1M matches but can only see one you are likely to have a similar problem. Improving this is on the TODO list.

Lucene supports query styles where you know the tokens and those where you do not ….

See http://wiki.alfresco.com/wiki/Search#Understanding_tokenisation


TEXT:woof
‍‍‍

matches the token woof (which could have been generated from Woof etc). It is lowercased by default - but this depends on your tokeniser.

So


TEXT:*oof
TEXT:wo*
TEXT:*oo*
‍‍‍‍‍

will work.

Phrases, things in "" are tokenised for you. and do not support wildcards.
Again, this would be possible and is on the list of stuff that did not make 1.4.

In the examples, with the default config
@cms\:name:nano* should work but it depends on the tokens in the index …
@cms\:name:"nano*" will never work
@cms\:name:"nanotechnology*" should not work ….

How are you doing these searches?
Are you using the default tokeniser?

Regards

Andy

Re: Lucene performance/wildcard searches not working

marcus — Wed, 18 Oct 2006 22:44:20 GMT

Is there a way to browse the lucene entries stored for a node? I've tried using the Luke lucene browser, but don't know which folder in the lucene.indexes directory hold those for my node, or how to just show all the information for a noderef even if I did happen to stumble on the node I was interested in

Edit: The actual property value for the node is "Australia and New Zealand", and I'm not getting any match when doing

@lms\:jurisdiction:australia*
@lms\:jurisdiction:aust*
@lms\:jurisdiction:"Aust*"

but I do get matches with

@lms\:jurisdiction:"Australia and New Zealand"
@lms\:jurisdiction:"Australia*"

If I set this property in my model to not be tokenised, what will be the difference in the generated query from SearchContext.buildQuery? (I'll try this out later, but just in case my result differs from expected)

Re: Lucene performance/wildcard searches not working

dbevacqua — Thu, 19 Oct 2006 09:24:22 GMT

Hi Andy

Thanks for your quick response.

I have used limit and limitBy as you suggested, and noted what you said about permission evaluation. It did raise another issue, namely is it possible to delay this evaluation? I ask because I need to be able to do join type searches, where the intermediate ResultSets may be large, but the final one not so - in this case I think it makes sense to evaluate permissions on the final ResultSet. I notice that in PermissionEvaluationMode there is a commented value LAZY which may be connected.

As for the wildcard problem, I am searching using the search service e.g.


            s.setLanguage(SearchService.LANGUAGE_LUCENE);
            s.setLimitBy(LimitBy.FINAL_SIZE);
            s.setLimit(100);
            s.setQuery(query);
            s.addStore(storeRef);

            r = getSearchService().query( s );
‍‍‍‍‍‍‍‍‍

I am using the default tokeniser, and I still can't get any matches with wildcards.

I also get no matches searching on the TEXT field e.g

TEXT:nanotechnology

Have tried rebuilding the indexes but to no avail.

Any thoughts?

Thanks,

Dominic.

Re: Lucene performance/wildcard searches not working

andy — Fri, 20 Oct 2006 09:02:09 GMT

Hi

You can use luke (in Alfresco up to version 1.3). After 1.4 is more complicated.

You can query as you would normally - except for path.
@cms\:title:wo* should work (but not prefix wild cards as you will get the default lucene query parser).

You can check what tokens are available for each field using Luke.

To do joins you could use the "searchService" bean. This does not apply permissions. You can do your queries and joins etc…. you will then have to apply permissions on top. The easiest way is to add your own service and let the security layer filter a ResultSet.

The support for joins via lucene is not as neat as it could be.

Regards

Andy

Re: Lucene performance/wildcard searches not working

dbevacqua — Fri, 20 Oct 2006 10:46:55 GMT

Hi Andy

Took your advice on using the searchService bean and that has had the desired effect. Joins coming along nicely.

I had a go with Luke and noticed some interesting things:

1. the analyser is slightly different (org.apache.lucene.analysis.SimpleAnalyzer) - won't accept the one in the dictionary model (org.apache.lucene.analysis.standard.StandardAnalyzer), possibly a classpath issue.

2. wildcard searches (equivalent to @cms:name:nano*) work where they didn't using searchService (query parser problem?)

3. none of my fields are marked as indexed, tokenised or stored

4. none of my fields appear to have anything in them (the String value appears as "<not available>")

5. TEXT, ISNODE, ISROOT… have nothing in them

Not really sure where to look next. Any ideas?

Dominic

Re: Lucene performance/wildcard searches not working

dbevacqua — Fri, 20 Oct 2006 11:13:22 GMT

Similarly the ANCESTOR field appears not to contain anything, both in Luke and using queries that work using PARENT. All relevant nodes extend sys:container.

Re: Lucene performance/wildcard searches not working

dbevacqua — Fri, 20 Oct 2006 13:16:36 GMT

Also, I'm trying to get the contents of the PARENT and ANCESTOR fields from the ResultSetRows. Any pointers?
Dominic

Re: Lucene performance/wildcard searches not working

dbevacqua — Fri, 20 Oct 2006 13:29:02 GMT

found this note in the javadoc for ResultSetRow:

TODO: Support for other non attribute features such as parents and path

So I guess that answers my question

I'm guessing it's not that hard to do. Will look into having a go myself.

Dominic

Re: Lucene performance/wildcard searches not working

dbevacqua — Tue, 07 Nov 2006 18:57:29 GMT

got this working… sort of. See http://forums.alfresco.com/viewtopic.php?p=12723#12723.

Dominic