topic Re: Incorrect and partial total result by lucene query in Alfresco Archive

Incorrect and partial total result by lucene query

epaoletti — Fri, 27 Feb 2009 12:03:30 GMT

Hi,We are running the following lucene query via actionquery = "PATH:\"/app:company_home/cm:Test/cmocument//*\" AND @\\{http\\://www.alfresco.org/model/content/1.0\\}content.mimetype:application/pdf";So we would like to get the total number of pdf files under the "company_home/Test/Document" space

Re: Incorrect and partial total result by lucene query

derek — Fri, 27 Feb 2009 12:25:58 GMT

Add to your custom-repository.properties (this comes from repository.properties) and increase as appropriate.

#
# Properties to limit resources spent on individual searches
#
# The maximum time spent pruning results
system.acl.maxPermissionCheckTimeMillis=10000
# The maximum number of results to perform permission checks against
system.acl.maxPermissionChecks=1000

Re: Incorrect and partial total result by lucene query

epaoletti — Fri, 27 Feb 2009 13:45:29 GMT

Ok, We already done it,

The problems was that we increased the value but the time to wait for the query result was too long (about 60 seconds).
The time is too long for WEB interactive use.

At the moment, We have a total of 10.000/15.000 pdf documents to search.
I thinks it is not a critical number for Alfresco.
What's happen if we 'll manage 100.000 documents ?

Have you some suggestions in order to set the variables in the correct way to manage
10.000/15.000 documents ?
The machine is:
CPU Intel dual core 2,66 GHZ
Memory 2GB RAM

# The maximum time spent pruning results
system.acl.maxPermissionCheckTimeMillis=??????
# The maximum number of results to perform permission checks against
system.acl.maxPermissionChecks=????

Thanks in advance for your Help

Re: Incorrect and partial total result by lucene query

derek — Fri, 27 Feb 2009 15:58:42 GMT

15000 results for a user-driven query would be too much anyway.

If you have code that does the search, then you can set the search parameters:
org.alfresco.service.cmr.search.SearchParameters
and set the limit and limitBy properties on a per-query basis. You can also bypass security checks in your code by running as the system user or using the searchService instead of the SearchService.

Leave the default for user-driven searches as it is to prevent users from overloading the system with queries for thousands of documents.

Re: Incorrect and partial total result by lucene query

epaoletti — Mon, 02 Mar 2009 10:10:26 GMT

Thanks for your fast answer,

Now the full-text search philosophy is more clear to me.

Basically, I can never know the total number of contents(documents) in alfresco that correspond to a query criteria.
Is it Correct ?

This could be a problem for the end-user.
Have you some suggestions in order to provide end-user with it ?

Alfresco could be a good framework in order to develop vertical application but often you must know before
hown many objects you must manage (for instace to show it on the web interface).

Are there some alfresco internal counter variables containing the total number of contents(documents) in one "Space" ?

Thanks in advance

Re: Incorrect and partial total result by lucene query

derek — Mon, 02 Mar 2009 11:40:03 GMT

Hi,
We don't currently have space-related quotas built into the system. We're laying some groundwork in 3.2 that will make these types of calculations more efficient.
Regards

Re: Incorrect and partial total result by lucene query

dwilson — Mon, 20 Apr 2009 23:27:42 GMT

Hi,
We don't currently have space-related quotas built into the system. We're laying some groundwork in 3.2 that will make these types of calculations more efficient.
Regards

Derek, as it is now, what is the fastest way to find the count of the number of nodes for a query? For example, the number of nodes in a particular category or parent category?

Is there a way to do this, in say a webscript, without Lucene? (The total number might be larger than maxPermissionChecks)

I see in this post they talk of storing totals in another node, updated daily by workflow- which seems pretty kludgy. http://forums.alfresco.com/en/viewtopic.php?f=4&t=3677&p=11875&hilit=total+count#p11875

Thanks!

Re: Incorrect and partial total result by lucene query

derek — Tue, 21 Apr 2009 08:25:17 GMT

Derek, as it is now, what is the fastest way to find the count of the number of nodes for a query? For example, the number of nodes in a particular category or parent category?

dwilson,
The only way to efficiently find nodes in a category is by using a Lucene search, I'm afraid.

Re: Incorrect and partial total result by lucene query

dwilson — Tue, 21 Apr 2009 17:59:51 GMT

Derek, as it is now, what is the fastest way to find the count of the number of nodes for a query? For example, the number of nodes in a particular category or parent category?
dwilson,
The only way to efficiently find nodes in a category is by using a Lucene search, I'm afraid.

That is unfortunate, so in order to display this kind of information on our website, the only option without taking forever is to cache the totals?

e.g.:

Pet Category Links: (Totals)

Dogs (8,123)
Cats (12,328)
Birds (7,230)
Rabbits (418)
Hamsters (3,132)
… etc.

Re: Incorrect and partial total result by lucene query

vta — Wed, 28 Oct 2009 08:58:05 GMT

Hi.

> You can also bypass security checks in your code by running as the system …

It's possible using web services?

Thanks

Re: Incorrect and partial total result by lucene query

tcld0 — Tue, 01 Dec 2009 05:17:17 GMT

You can also bypass security checks in your code by running as the system user or using the searchService instead of the SearchService.

Hi,
could you please elaborate on how i can run a query that bypasses security checks? I'm writing an API into the repository that can be used by other systems to query for large batches of content nodes. I don't want there to be any limits on the number of records that could be returned, as the use cases specifically require the possibility of returning all nodes in the repository. Also, is there any way to know (programmatically) that a query has been cut short and has not returned all possible matching nodes due to some limit or timeout?

thanks

Re: Incorrect and partial total result by lucene query

dgenard — Tue, 01 Dec 2009 19:43:03 GMT

Hi epaoletti,
using the SearchService in Java, there is a way to detect that a limit (maxPermissionCheckTimeMillis or maxPermissionChecks) was reached.
We use this to display a warning for the current user.

Example code :


ResultSet results = this.getSearchService().query(sp);
if (LimitBy.NUMBER_OF_PERMISSION_EVALUATIONS == results.getResultSetMetaData().getLimitedBy()) {
    searchLimitReached = true;
    Utils.addErrorMessage(Application.getMessage(FacesContext.getCurrentInstance(), "warning_search_limit_reached"));
} else {
    searchLimitReached = false; 
}
‍‍‍‍‍‍‍‍‍

Denis