topic Re: How to exceed Lucene Limit Search? in Alfresco Archive

How to exceed Lucene Limit Search?

razieltd — Tue, 29 Mar 2011 14:30:03 GMT

Hi!Is it possible with a ws java-backed research by lucene an unlimited number of nodes?Now if i try to reseach in a store that contains 1100 nodes, the results list contains only 1000I found this codeearchParameters sp = new SearchParameters();sp.setLanguage(SearchService.LANGUAGE_LUCENE);sp.setQ

Re: How to exceed Lucene Limit Search?

mrogers — Tue, 29 Mar 2011 14:39:21 GMT

Because the number of permission checks and the time to search are also limited. You will find other threads discussing how to change this if you search these forums.

Re: How to exceed Lucene Limit Search?

razieltd — Wed, 30 Mar 2011 07:29:24 GMT

Thank you for the answare. But can i ask you a favor? can you post me some of these threads? Because when iam searching i found only solution that dont work (probably i did not understand :roll: ).

Now iam trying to implement in my webscript (java-baked) something like this


         int batchSize = 10;
         QueryConfiguration queryCfg = new QueryConfiguration();
         queryCfg.setFetchSize(batchSize);
         RepositoryServiceSoapBindingStub repositoryService = WebServiceFactory.getRepositoryService();
         repositoryService.setHeader(new RepositoryServiceLocator().getServiceName().getNamespaceURI(), "QueryHeader", queryCfg);
         Query query=new Query(SearchService.LANGUAGE_LUCENE,"PATH:\"/*\"");
         Store store=new Store();
         store.setScheme("archive");
         store.setAddress("SpacesStore");
         QueryResult result =repositoryService.query(store, query, true);
         //process the first query result
         String querySession = result.getQuerySession();
         while (querySession != null) {
             result = repositoryService.fetchMore(querySession);
             querySession = result.getQuerySession();
         }
         int size=(int)result.getResultSet().getTotalRowCount();
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

but dont work.

Thx a lot

Re: How to exceed Lucene Limit Search?

razieltd — Wed, 30 Mar 2011 07:51:13 GMT

??? ???

Re: How to exceed Lucene Limit Search?

darryl_staflund — Wed, 30 Mar 2011 18:49:37 GMT

Hi there,

Here is an example of how I disable this limit in Java:


package com.companyname.alfresco.common.daos.impl;

import static java.lang.String.format;

import java.util.List;

import org.alfresco.model.ContentModel;
import org.alfresco.service.cmr.repository.NodeRef;
import org.alfresco.service.cmr.repository.StoreRef;
import org.alfresco.service.cmr.search.LimitBy;
import org.alfresco.service.cmr.search.ResultSet;
import org.alfresco.service.cmr.search.SearchParameters;
import org.alfresco.service.cmr.search.SearchService;
import org.springframework.beans.factory.annotation.Required;
import org.springframework.stereotype.Component;

import com.companyname.alfresco.common.daos.SearchDao;

@Component
public final class SearchDaoImpl implements SearchDao {
   private SearchService searchService;

   @Required
   public void setSearchService(final SearchService service) {
      searchService = service;
   }

   @Override
   public List<NodeRef> searchForContentNode(final NodeRef nodeRef) {
      final SearchParameters params = getSearchParameters(nodeRef);

      ResultSet resultSet = null;

      try {
         resultSet = searchService.query(params);
         return resultSet.getNodeRefs();
      }

      finally {
         if (resultSet != null) {
            resultSet.close();
         }
      }
   }

   private SearchParameters getSearchParameters(final NodeRef nodeRef) {
      final SearchParameters params = new SearchParameters();
      params.addStore(StoreRef.STORE_REF_WORKSPACE_SPACESSTORE);
      params.setLanguage(SearchService.LANGUAGE_LUCENE);
      params.setQuery(format(QUERY_PATTERN, ContentModel.TYPE_CONTENT, nodeRef));
      params.setLimitBy(LimitBy.UNLIMITED);
      params.setLimit(0);
      params.setMaxPermissionChecks(100000);
      params.setMaxPermissionCheckTimeMillis(100000);
      params.setMaxItems(-1);
      return params;
   }
}


‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I hope it's helpful.

Darryl

Re: How to exceed Lucene Limit Search?

razieltd — Thu, 31 Mar 2011 07:57:46 GMT

Thank you so much!

but when i try to implement this code i receive this exception in my ws

Message:   org.alfresco.service.cmr.search.SearchParameters.setMaxPermissionChecks(I)V

Exception:   java.lang.NoSuchMethodError - org.alfresco.service.cmr.search.SearchParameters.setMaxPermissionChecks(I)V

:shock:   :shock:   :shock:

what can i do?

EDIT:

i receive this exception with these 3 methods:
            …
         params.setMaxPermissionChecks(100000);
         params.setMaxPermissionCheckTimeMillis(100000);
         params.setMaxItems(-1);

:shock:

Re: How to exceed Lucene Limit Search?

mrogers — Thu, 31 Mar 2011 10:25:43 GMT

What version are you using?

Re: How to exceed Lucene Limit Search?

razieltd — Thu, 31 Mar 2011 11:50:54 GMT

Alfresco 3.1

Re: How to exceed Lucene Limit Search?

razieltd — Mon, 04 Apr 2011 08:09:53 GMT

No solutions for this problem? There is another method that can i use for bypass the limit?

Thx :?

Re: How to exceed Lucene Limit Search?

razieltd — Thu, 07 Apr 2011 07:24:32 GMT

Ehmmm… someone have the solution? or another way for this?

Thx

Re: How to exceed Lucene Limit Search?

razieltd — Mon, 11 Apr 2011 11:56:15 GMT

Ok guys! if there is no way with this SearchParameters how can i fetch result set??

Thx

Re: How to exceed Lucene Limit Search?

razieltd — Wed, 13 Apr 2011 12:16:22 GMT

i see that there is another method in the SearchParameters

.setBulkFetch(true);
i think in combination with this other
.setBulkFetchSize(1000);

but dont work :?

It is correct to use these methods for Alfresco 3.1 ? and How can i use it?

Pls Somebody can Helpme?

Thx

Re: How to exceed Lucene Limit Search?

mrogers — Wed, 13 Apr 2011 12:44:13 GMT

You certainly don't want to play with bulk fetch size.

Have you tried setting
system.acl.maxPermissionCheckTimeMillis
system.acl.maxPermissionCheck

in your custom-repository.properties file?

And isn't it obvious that setMaxPermissionChecks does not exist on 3.1? What sort of development environment do you have that doesn't highlight that instantly? In particular make sure you are using the right version of the SDK for the code you are developing.

Re: How to exceed Lucene Limit Search?

razieltd — Wed, 13 Apr 2011 16:16:43 GMT

Thank you for the answare

If is it possible i want to do it without modify any file.
and yes it is obvious that setMaxPermissionChecks does not exist on 3.1, iam sorry in the previous post i explained bad :roll: . iam using 3.1SDK with eclipse :wink:

but now want can i do for break this limit without modify the properties file?

Thx a lot

Re: How to exceed Lucene Limit Search?

razieltd — Fri, 15 Apr 2011 07:16:52 GMT

So…. is it possible with 3.1??? and if yes how without modify any file?

Re: How to exceed Lucene Limit Search?

mrogers — Fri, 15 Apr 2011 07:40:21 GMT

No I don't think its possible without modifying your configuration. What's your objection to changing config?

Re: How to exceed Lucene Limit Search?

razieltd — Fri, 15 Apr 2011 17:29:20 GMT

Simple - i cant! Because we have several webscript for searching nodes and a lot of these are used by our users and this limit (1000 record) is useful. But some ws in particular this i'm developing in only for powerUser (not many around 5-7) and we need some statistics for example, and i cant get all node with a query lucene but only 1000 and in some spece there are 20000-30000.

For example if i want to know the exact number of nodes in a particular space i do my query lucene and i obtain the resultSet.size(). but if i have this limit what kind of research i can do? or there is another way to know this information?

iam thinking if is it possible to operate with the native code of lucene and bypass SearchParameter, searchService ect..
because when i do my search with searchService, at some point, there is some use of lib lucene that execute the query… :roll:

however thx for your help :wink:

Re: How to exceed Lucene Limit Search?

mrogers — Fri, 15 Apr 2011 17:36:21 GMT

It would probably be easier to upgrade to a more recent version of Alfresco

Re: How to exceed Lucene Limit Search?

mrogers — Fri, 15 Apr 2011 17:51:00 GMT

To find the number of nodes in a space there's no need to touch lucene at all!

Use the FileFolderService or NodeService directly.

Or run a SQL query directly. Or CQL.

Re: How to exceed Lucene Limit Search?

razieltd — Mon, 18 Apr 2011 07:53:00 GMT

:shock:

Thx!

I apologize for my ignorance :lol: can you show me how to use a query sql (tables)? I did not know that i can use in alfresco .

Thx so much