topic Re: Limitations of Lucene Results in Alfresco Archive

Limitations of Lucene Results

vanhoeck — Fri, 18 Jul 2008 15:21:40 GMT

Hi,I was testing the abilities of Lucene on an Alfresco server with about 10k files. When I run the query as Admin, I get 1000 results (intentionally didn't raise that limit). In the next step, I've added a new user (Homer) and I gave Homer one file in his home folder. When I ran the query as Homer,

Re: Limitations of Lucene Results

francescof — Fri, 18 Jul 2008 19:26:12 GMT

Your problem is most likely not caused by "system.acl.maxPermissionChecks" being set to "1000". You should raise your "system.acl.maxPermissionCheckTimeMillis" to the longest time you could wait for search results. It seems that your first test returned 1000 results in less than 10 seconds and then stopped because "maxPermissionChecks" was hit. Your second test probably stopped after hitting "maxPermissionCheckTimeMillis" without returning results because it didn't have enough time to go past the document visible to the current user.

In other words: with default settings Lucene filtering stops after 1000 checks or after 10 seconds (which one comes first). The only real solution to this problem is speeding up permission checks, since getting raw results out of Lucene indexes is already really fast. A temporary workaround for you could be setting a very high "maxPermissionCheckTimeMillis" value (and narrow down your search criteria as much as possible to reduce the raw resultset size).

If you are building a custom solution using Alfresco Foundation API and don't mind disabling permission checks on results you may try:

         
searchParameters.setLimitBy(LimitBy.NUMBER_OF_PERMISSION_EVALUATIONS);
searchParameters.setLimit(0);
‍‍‍‍

This disables every permission check and returns the whole raw result set.

Re: Limitations of Lucene Results

vanhoeck — Tue, 22 Jul 2008 07:41:44 GMT

Your problem is most likely not caused by "system.acl.maxPermissionChecks" being set to "1000". You should raise your "system.acl.maxPermissionCheckTimeMillis" to the longest time you could wait for search results. It seems that your first test returned 1000 results in less than 10 seconds and then stopped because "maxPermissionChecks" was hit. Your second test probably stopped after hitting "maxPermissionCheckTimeMillis" without returning results because it didn't have enough time to go past the document visible to the current user.

In other words: with default settings Lucene filtering stops after 1000 checks or after 10 seconds (which one comes first). The only real solution to this problem is speeding up permission checks, since getting raw results out of Lucene indexes is already really fast. A temporary workaround for you could be setting a very high "maxPermissionCheckTimeMillis" value (and narrow down your search criteria as much as possible to reduce the raw resultset size).

If you are building a custom solution using Alfresco Foundation API and don't mind disabling permission checks on results you may try:
         
searchParameters.setLimitBy(LimitBy.NUMBER_OF_PERMISSION_EVALUATIONS);
searchParameters.setLimit(0);
‍‍‍‍
This disables every permission check and returns the whole raw result set.

Thanks for your answer. I changed the time to 70000 and even 200000, but the one file did not appear. And even if it did work, I would still like to keep the time to search limited for growing numbers. Is there any way of knowing that Lucene has hit it's limits (some kind of overflow-check)? In that case, I can just return an error, asking for a more specific query.

Re: Limitations of Lucene Results

francescof — Tue, 22 Jul 2008 08:37:49 GMT

When hitting the lucene configured limits Alfresco's ACLEntryAfterInvocationProvider sets LimitBy.NUMBER_OF_PERMISSION_EVALUATIONS in result set metadata:


…
ResultSet resultSet = searchService.query(…);
…
LimitBy limitBy = resultSet.getResultSetMetaData().getLimitedBy();

if (LimitBy.NUMBER_OF_PERMISSION_EVALUATIONS.equals(limitBy)) {
   /* Search hit lucene configured limits… return the error here. */
   …
}
‍‍‍‍‍‍‍‍‍‍‍

Re: Limitations of Lucene Results

vanhoeck — Tue, 22 Jul 2008 13:11:58 GMT

Thanks!

Re: Limitations of Lucene Results

ddanninger — Wed, 21 Sep 2011 17:18:29 GMT

Is this also possible to handle that without using the Alfresco Foundation API?

with the javascript api / or the SOAP services or some general parameters in alfresco?

Re: Limitations of Lucene Results

nitinagarwal198 — Fri, 07 Oct 2011 06:40:22 GMT

How can I increase lucene result limit without modifying code?

Re: Limitations of Lucene Results

mrogers — Fri, 07 Oct 2011 07:30:10 GMT

Switch to Alfresco 4 and use Solr.

Re: Limitations of Lucene Results

nitinagarwal198 — Sat, 08 Oct 2011 05:23:31 GMT

Thanks for your reply.

I am using alfresco 3.2 community version and I don't want to upgrade it. So, can you please tell me how can I increase lucene result limit. Is there any configuration file for this?

Switch to Alfresco 4 and use Solr.

Re: Limitations of Lucene Results

mrogers — Sat, 08 Oct 2011 13:52:19 GMT

See the first post in this thread. On 3.2 I think that's all you can do.

Re: Limitations of Lucene Results

katnam21 — Fri, 19 Jun 2015 07:01:58 GMT

now we are in 5.0, how can we get all the results from lucene query.

i dont want to change the lucene limit in global properties, only in one case i need all the results, it might be 10k or 100k.

Cheers,
Mallesham