topic Re: Weird Search Result Issue in Alfresco Archive

Weird Search Result Issue

mcook — Thu, 29 Jan 2009 19:50:23 GMT

We are running alfresco version 2.9 in 4.2.2.GA on Linux 2.6.18-8.el5 (i386). We have created some custom scripts to manage driver documents. It checks to see if a driver has all the required documents on file and when they need updated (they have expired or will expire soon). We are currently ha

Re: Weird Search Result Issue

dhalupa — Thu, 29 Jan 2009 21:56:02 GMT

How many hits do you think your Lucene query typically returns? If the number of Lucene hits is large (close to 1000 or more) than ACL evaluation process could affect the number of found nodes.
There are few parameters which affect this behaviour
1. state of ACL cache before the repository is queried
2. size of ACL cache
3. system.acl.maxPermissionCheckTimeMillis defaults to 10000ms and defines maximum time that will be spent on ACL evaluation
4. system.acl.maxPermissionChecks defaults to 1000 and defines maximum number of ACL evaluations that will be performed for collection of hits that will be returned from Lucene query. If the number of hits exceedes this number, the rest will simply be discarded

Hope this helps,

Denis

Re: Weird Search Result Issue

mcook — Fri, 30 Jan 2009 14:56:34 GMT

Thank you for the advise. However, I do no think that is the cause of our current issue. Right now, the search is returning about 450 files, but in the future it could return many more. Ill look into changing those flags, and Ill hope this fixes things, but I remain skeptical that it will.

Re: Weird Search Result Issue

dhalupa — Fri, 30 Jan 2009 16:07:28 GMT

If I understood you correctly, this is some kind of system task which runs periodically. If this is the case, than there is one other thing you might consider. You could try using SearchService without transaction, security and audit interceptors applied. The id of that bean is "searchService" and in that case you will bypass acl evaluation completely. The only thing that you have to be careful about is that you have to create transaction boundaries yourself since it will not be handled by spring. I'm not sure how you will access this bean from JavaScript though, since I'm pretty sure that it is not injected into the JavaScript context.

Kind regards,

Denis

Re: Weird Search Result Issue

mcook — Wed, 04 Feb 2009 21:14:05 GMT

I have solved our issue, but I never found out the root cause of it. I would like to figure that out however, to prevent this from happening again. The documents were either not indexed or improperly indexed. A server restart solved the issue because by default Alfresco does some reindexing on startup. At least this is my best guess. Our server needs to remain up as much as possible (only restarted during maintenance windows); however, if this happens again we will inevitably require a restart unless another solution is found. Does anyone have any thoughts?

Re: Weird Search Result Issue

lotharm — Thu, 05 Feb 2009 09:32:26 GMT

How the daily is triggered? Perhaps it runs as guest user?

However, I would like to comment on the general approach you took, because we also took the same search-the-index-approach once, but concluded it is not the right way.
Querying the index for a certain property value (valid, expired or the like) to produce report-like lists seems like a reasonable idea. But it turns out that
the result list is not stable. That means, doing the same query twice, possibly gives a longer list. Possibly, because it is usually not visible on small systems, but with heavier load it might happen more often.The driving factors rise:
* a loaded system may hit the value of system.acl.maxPermissionCheckTimeMillis=10 seconds.
* a big result list may hit the value of system.acl.maxPermissionChecks=1000
* In a cluster setup, where every instance has its own index, a document might be missing for a short time in the index.
…and the result will just be cut. Very difficult to reproduce, too.

Using the index for something else than full text search is a debatable thing then. In my eyes, a lookup by a property value as above is not a thing for a full-text index, instead its a thing for the relational database.

I think Alfresco did a good job on the full text index. It is fast, respects transactions and the results for full-text search are well ranked. But it has to be used for what it was build for. By design the index is cutting the result list. This improves search speed and there is no need for the 10.000th result hit in a full text search. Did one ever go the second page on google?

Though, we solved our issue by using direct database queries through hibernate with success.

Hoping this gives some guidance,
lothar

Re: Weird Search Result Issue

mcook — Mon, 09 Feb 2009 18:04:47 GMT

Is it possible to do a hibernate query from the Javascript API? A quick google search did not turn up much.

Re: Weird Search Result Issue

dhalupa — Mon, 09 Feb 2009 18:47:53 GMT

How the daily is triggered? Perhaps it runs as guest user?
By design the index is cutting the result list.

It is not "index" (I guess you meant Lucene query) which is cutting result list but rather acl evaluation. Number of hits returned from Lucene is correct

Re: Weird Search Result Issue

dhalupa — Mon, 09 Feb 2009 18:49:02 GMT

Is it possible to do a hibernate query from the Javascript API? A quick google search did not turn up much.

No, it is not possible

Re: Weird Search Result Issue

lotharm — Mon, 09 Feb 2009 22:18:34 GMT

It is not "index" (I guess you meant Lucene query) which is cutting result list but rather acl evaluation.

To make it clear, I was talking about the Indexing-and-Searching component as a whole. That thingy you get as the SearchService bean, which one has to use in a black-box manner.
There is lucene hidden below of it, wrapped by a ACL-checker, which also cuts due to time constraints.

Number of hits returned from Lucene is correct

Not in a cluster environment. Some transactions might be missing in the local lucene index of an Alfresco instance.

Regards,
lothar

Re: Weird Search Result Issue

dhalupa — Mon, 09 Feb 2009 22:37:50 GMT

Not in a cluster environment. Some transactions might be missing in the local lucene index of an Alfresco instance.

Yes, and indexes might also get corrupted, but these are not not normal behaviors, I am referring to situation when indexes are not corrupted and are properly synchronized between cluster nodes. In that case Lucene query will always return a correct number of results, cutting will occur later during acl evaluation.

Re: Weird Search Result Issue

ebell — Mon, 04 May 2009 15:17:08 GMT

Would you mind showing some examples of using the direct database queries through hibernate?
I thought that direct access to the Alfresco database was not recommended. Did any further problems arise with your solution?
Would an XPath query based on node service provide consistent/stable results, or would that have the same problem with inconsistent results as Lucene did?

Re: Weird Search Result Issue

ebell — Wed, 06 May 2009 19:34:54 GMT

After some further investigation, I've changed

search.luceneSearch("TYPE:\""+BURRIS_DOC_TYPE+"\" +ASPECT:\""+EXPIREABLE_ASPECT+"\" -ASPECT:\""+SUPERCEDED_ASPECT+"\" -ASPECT:\""+TERMINATED_ASPECT+"\"");‍

search.xpathSearch("TYPE:\""+BURRIS_DOC_TYPE+"\" +ASPECT:\""+EXPIREABLE_ASPECT+"\" -ASPECT:\""+SUPERCEDED_ASPECT+"\" -ASPECT:\""+TERMINATED_ASPECT+"\"");‍

etc. in all instances in the javascript code.
It still appears that Lucene is being run to do part of the searching. However, as in the one post above, I'm looking to create a stable report every time. Am I heading in the right direction, or would using Hibernate be the way to go?