topic Re: Solr4 query threading beaviour in Alfresco Forum

Solr4 query threading beaviour

chkk — Fri, 20 Jan 2017 00:53:16 GMT

I have observed an unexpected behaviour in my Alfresco installation when doing advanced searches. The site has an complex advanced search form with about 30 criteria, many of them multi-valued picked from multi-selection lists.When I do a search with a couple of criteria selected with a user with ad

Re: Solr4 query threading beaviour

afaust — Sun, 22 Jan 2017 14:38:25 GMT

Are you sure the use of all the cores was really due to the query? SOLR will triggere a "new / updated content tracking" operation every 15 seconds for each SOLR core, which may also branch out to multiple threads when new content is found.

Setting alfresco.doPermissionChecks to false in solrcore.properties is not really a smart thing to do. It just means that permissions will not be checked on the SOLR tier, but they will still be checked on the Repository tier leading to way more performance overhead.

As I often tell people, a lot of the tuning advice should be taken with a grain of salt and a basic understanding of the correlation any setting may have on the performance. It would be great to know what you have tried as to judge the potential effect (benefit / damage) it could have.

What is the IO situation in your server? SOLR and DB are very sensitive to slow IO and you can easily tank the performance by not using a proper storage medium for the data.

I have used 201612 GA (5.2) for a while now locally (on my work laptop) and even with about 6 million documents loaded into it query time never reached double-digit seconds range.

Re: Solr4 query threading beaviour

chkk — Mon, 23 Jan 2017 06:46:32 GMT

Thanks for the answer!

Load is definitely associated to the query process, as the server is nearly idle when I don't start a query. And nobody else on the server, no new content created. However, it looks to me now that the load is not the SOLR search itself but the retrieval of the nodes in Alfresco after the initial search.

When testing for analysing the root cause I retrieved the JSON search results directly opening "https://<...url...>/share/page/dp/ws/faceted-search#searchTerm=&query=<...query...>&pageSize=25&maxResults=0&noCache=1485049983069&spellcheck=false" and noticed that if I set "maxResults" to something greater than 0, the query executes nearly instantly as it that case it only delivers the header information with the correct number of results but not the individual item list.

Setting alfresco.doPermissionChecks to false in solrcore.properties is not really a smart thing to do. It just means that permissions will not be checked on the SOLR tier, but they will still be checked on the Repository tier leading to way more performance overhead.

Required for our solution, users are allowed to search all content independent of access privileges and are only blocked from accessing full content, allowing them to request access if required.

This is implemented by a combination of the "doPermissionsChecks=false" mentioned above and running the "processResultsSinglePage" function in "search.lib.js" as admin to get all results and then clearing out some sensitive fields in the "getDocumentItem" function if the user is not in the admin role.

As I often tell people, a lot of the tuning advice should be taken with a grain of salt and a basic understanding of the correlation any setting may have on the performance. It would be great to know what you have tried as to judge the potential effect (benefit / damage) it could have

Java VM intial/max heap size increased from 2/4 GB to 8/16 GB
Different other JVM options for garbage collections tested, current settings as advised in Alfrecso 5.2 documentation
Caches for SOLR increased
Database tables optimized

Tested after each individual change, no real difference.

What is the IO situation in your server? SOLR and DB are very sensitive to slow IO and you can easily tank the performance by not using a proper storage medium for the data.

Tested with "hdparm -t", throughput is about 180 MB / s.

Thankful for any further input 🙂

Chris

Re: Solr4 query threading beaviour

afaust — Mon, 23 Jan 2017 07:26:51 GMT

chkk _ wrote:
When testing for analysing the root cause I retrieved the JSON search results directly opening "https://<...url...>/share/page/dp/ws/faceted-search#searchTerm=&query=<...query...>&pageSize=25&maxResults=0&noCache=1485049983069&spellcheck=false" and noticed that if I set "maxResults" to something greater than 0, the query executes nearly instantly as it that case it only delivers the header information with the correct number of results but not the individual item list.

Setting alfresco.doPermissionChecks to false in solrcore.properties is not really a smart thing to do. It just means that permissions will not be checked on the SOLR tier, but they will still be checked on the Repository tier leading to way more performance overhead.
Required for our solution, users are allowed to search all content independent of access privileges and are only blocked from accessing full content, allowing them to request access if required.

This is implemented by a combination of the "doPermissionsChecks=false" mentioned above and running the "processResultsSinglePage" function in "search.lib.js" as admin to get all results and then clearing out some sensitive fields in the "getDocumentItem" function if the user is not in the admin role.

Executing search as "admin" will NOT prevent overhead for permission checking on the Repository-tier when doPermissionCheck is disabled. First of all, "admin" is a normal user and does not short-circuit permission checks, and second of all "processResultsSinglePage" is too late to ensure that results won't be filtered incorrectly for your kind of customisation, as results are immediately permission checked / filtered as part of the "search.queryResultSet(queryDef)" call.

Re: Solr4 query threading beaviour

chkk — Mon, 23 Jan 2017 16:34:06 GMT

Authentication as admin seemed to work fine, any suggestions to avoid that overhead if we want public search but private full document access?

Re: Solr4 query threading beaviour

afaust — Mon, 23 Jan 2017 17:24:27 GMT

The use case itself does not make much sense: Why allow people to search/see all documents that they then may not be able to access at all? Something like this cannot really be handled efficiently with out-of-the-box Alfresco without messing with the permission model / SOLR ACL indexing.

If you want to continue with the current dirty hack, then you can use "System" as the runAs-user instead of "admin". At least that way the permission checks are properly short-circuited and overhead is minimized. Again, the runAs context needs to include the actual query call, not just the post processing...

Re: Solr4 query threading beaviour

chkk — Wed, 25 Jan 2017 07:28:44 GMT

Checked the code, actually that was already running as "System" ( ... AuthenticationUtil.getSystemUserName() ... ), so that was not the problem. I will try your suggestion to do the "RunAs" directly in the query call to see if that makes a difference.

The use-case for this scenario is that people may request access to the documents based on the search results: the search output is a custom table showing only a couple of metadata columns from the documents. The information in these columns is considered public, but the full document data is restricted based on the document's location and the user's group memberships.

So would it be better to "fake" public access at indexing time?

Re: Solr4 query threading beaviour

evaaa — Fri, 19 Feb 2021 12:18:16 GMT

@chkk wrote:
Thanks for the answer!
Load is definitely associated to the query process, as the server is nearly idle when I don't start a query. And nobody else on the server, no new content created. However, it looks to me now that the load is not the SOLR search itself but the retrieval of the nodes in Alfresco after the initial search.
When testing for analysing the root cause I retrieved the JSON search results directly opening "spyera<...url...>/share/page/dp/ws/faceted-search#searchTerm=&query=<...query...>&pageSize=25&maxResults=0&noCache=1485049983069&spellcheck=false" and noticed that if I set "maxResults" to something greater than 0, the query executes nearly instantly as it that case it only delivers the header information with the correct number of results but spyera the individual item list.
Setting alfresco.doPermissionChecks to false in solrcore.properties is not really a smart thing to do. It just means that permissions will not be checked on the SOLR tier, but they will still be checked on the Repository tier leading to way more performance overhead.
Required for our solution, users are allowed to search all content independent of access privileges and are only blocked from accessing full content, allowing them to request access if required.
This is implemented by a combination of the "doPermissionsChecks=false" mentioned above and running the "processResultsSinglePage" function in "search.lib.js" as admin to get all results and then clearing out some sensitive fields in the "getDocumentItem" function if the user is not in the admin role.
As I often tell people, a lot of the tuning advice should be taken with a grain of salt and a basic understanding of the correlation any setting may have on the performance. It would be great to know what you have tried as to judge the potential effect (benefit / damage) it could have
Java VM intial/max heap size increased from 2/4 GB to 8/16 GB
Different other JVM options for garbage collections tested, current settings as advised in Alfrecso 5.2 documentation
Caches for SOLR increased
Database tables optimized
Tested after each individual change, no real difference.
What is the IO situation in your server? SOLR and DB are very sensitive to slow IO and you can easily tank the performance by not using a proper storage medium for the data.
Tested with "hdparm -t", throughput is about 180 MB / s.
Thankful for any further input 🙂
Chris

You need more RAM and CPU