Solr4 query threading beaviour

[chkk]

I have observed an unexpected behaviour in my Alfresco installation when doing advanced searches. The site has an complex advanced search form with about 30 criteria, many of them multi-valued picked from multi-selection lists.

When I do a search with a couple of criteria selected with a user with admin rights, the search executes in about 11-15 seconds, which is not great (only a couple of thousand documents in the system) but acceptable. I followed tuning advice I found for the JVM, Solr4 and Postgres, did hardly make any difference.

When I execute exactly the same advanced search query as a non-admin user, the execution time is 55-60 seconds, which is way too slow.

When monitoring the server during query execution, I noticed to my surprise that the query by an admin is loading 3-4 CPU cores of the VM ("top" CPU load shown as 350%), while the non-admin query seems to run as a single thread (about 105% load shown in "top"). Which seems to explain the difference in execution time. 

I checked out Alfresco and Solr tuning guidelines, rebuilt the index, nothing made a difference.  One additional info: "alfresco.doPermissionChecks" is set to false in the solrcore properties as any user is allowed to search in this system. I did not find any Solr4 or Alfresco parmeters which would explain the threading behaviour to me.

This is on Alfresco 5.2.0 (Community 2016-11). Hosted on a VMware server with currently 4 cores and 24 GB memory assigned (used to be 8 GB, temporarily increased for testing for bottlenecks).

Thanks for any advice,

Chris

[afaust]

Are you sure the use of all the cores was really due to the query? SOLR will triggere a "new / updated content tracking" operation every 15 seconds for each SOLR core, which may also branch out to multiple threads when new content is found.

Setting alfresco.doPermissionChecks to false in solrcore.properties is not really a smart thing to do. It just means that permissions will not be checked on the SOLR tier, but they will still be checked on the Repository tier leading to way more performance overhead.

As I often tell people, a lot of the tuning advice should be taken with a grain of salt and a basic understanding of the correlation any setting may have on the performance. It would be great to know what you have tried as to judge the potential effect (benefit / damage) it could have.

What is the IO situation in your server? SOLR and DB are very sensitive to slow IO and you can easily tank the performance by not using a proper storage medium for the data.

I have used 201612 GA (5.2) for a while now locally (on my work laptop) and even with about 6 million documents loaded into it query time never reached double-digit seconds range.

[chkk]

Thanks for the answer!

Load is definitely associated to the query process, as the server is nearly idle when I don't start a query. And nobody else on the server, no new content created. However, it looks to me now that the load is not the SOLR search itself but the retrieval of the nodes in Alfresco after the initial search.

When testing for analysing the root cause I retrieved the JSON search results directly opening "https://<...url...>/share/page/dp/ws/faceted-search#searchTerm=&query=<...query...>&pageSize=25&maxResults=0&noCache=1485049983069&spellcheck=false" and noticed that if I set "maxResults" to something greater than 0, the query executes nearly instantly as it that case it only delivers the header information with the correct number of results but not the individual item list.

Setting alfresco.doPermissionChecks to false in solrcore.properties is not really a smart thing to do. It just means that permissions will not be checked on the SOLR tier, but they will still be checked on the Repository tier leading to way more performance overhead.

Required for our solution, users are allowed to search all content independent of access privileges and are only blocked from accessing full content, allowing them to request access if required.

This is implemented by a combination of the "doPermissionsChecks=false" mentioned above and running the "processResultsSinglePage" function in "search.lib.js" as admin to get all results and then clearing out some sensitive fields in the "getDocumentItem" function if the user is not in the admin role.

As I often tell people, a lot of the tuning advice should be taken with a grain of salt and a basic understanding of the correlation any setting may have on the performance. It would be great to know what you have tried as to judge the potential effect (benefit / damage) it could have

• Java VM intial/max heap size increased from 2/4 GB to 8/16 GB
• Different other JVM options for garbage collections tested, current settings as advised in Alfrecso 5.2 documentation
• Caches for SOLR increased
• Database tables optimized

Tested after each individual change, no real difference.

What is the IO situation in your server? SOLR and DB are very sensitive to slow IO and you can easily tank the performance by not using a proper storage medium for the data.

Tested with "hdparm -t", throughput is about 180 MB / s.

Thankful for any further input 🙂

Chris

[afaust]

chkk _ wrote:

When testing for analysing the root cause I retrieved the JSON search results directly opening "https://<...url...>/share/page/dp/ws/faceted-search#searchTerm=&query=<...query...>&pageSize=25&maxResults=0&noCache=1485049983069&spellcheck=false" and noticed that if I set "maxResults" to something greater than 0, the query executes nearly instantly as it that case it only delivers the header information with the correct number of results but not the individual item list.

Setting alfresco.doPermissionChecks to false in solrcore.properties is not really a smart thing to do. It just means that permissions will not be checked on the SOLR tier, but they will still be checked on the Repository tier leading to way more performance overhead.

Required for our solution, users are allowed to search all content independent of access privileges and are only blocked from accessing full content, allowing them to request access if required.

This is implemented by a combination of the "doPermissionsChecks=false" mentioned above and running the "processResultsSinglePage" function in "search.lib.js" as admin to get all results and then clearing out some sensitive fields in the "getDocumentItem" function if the user is not in the admin role.

Executing search as "admin" will NOT prevent overhead for permission checking on the Repository-tier when doPermissionCheck is disabled. First of all, "admin" is a normal user and does not short-circuit permission checks, and second of all "processResultsSinglePage" is too late to ensure that results won't be filtered incorrectly for your kind of customisation, as results are immediately permission checked / filtered as part of the "search.queryResultSet(queryDef)" call.

[chkk]

Authentication as admin seemed to work fine, any suggestions to avoid that overhead if we want public search but private full document access?

[afaust]

The use case itself does not make much sense: Why allow people to search/see all documents that they then may not be able to access at all? Something like this cannot really be handled efficiently with out-of-the-box Alfresco without messing with the permission model / SOLR ACL indexing.

If you want to continue with the current dirty hack, then you can use "System" as the runAs-user instead of "admin". At least that way the permission checks are properly short-circuited and overhead is minimized. Again, the runAs context needs to include the actual query call, not just the post processing...

[chkk]

Checked the code, actually that was already running as "System" ( ... AuthenticationUtil.getSystemUserName() ... ), so that was not the problem. I will try your suggestion to do the "RunAs" directly in the query call to see if that makes a difference.

The use-case for this scenario is that people may request access to the documents based on the search results: the search output is a custom table showing only a couple of metadata columns from the documents. The information in these columns is considered public, but the full document data is restricted based on the document's location and the user's group memberships.

So would it be better to "fake" public access at indexing time?

[evaaa]

---

@chkk wrote:

Thanks for the answer!

Load is definitely associated to the query process, as the server is nearly idle when I don't start a query. And nobody else on the server, no new content created. However, it looks to me now that the load is not the SOLR search itself but the retrieval of the nodes in Alfresco after the initial search.

When testing for analysing the root cause I retrieved the JSON search results directly opening "spyera<...url...>/share/page/dp/ws/faceted-search#searchTerm=&query=<...query...>&pageSize=25&maxResults=0&noCache=1485049983069&spellcheck=false" and noticed that if I set "maxResults" to something greater than 0, the query executes nearly instantly as it that case it only delivers the header information with the correct number of results but spyera the individual item list.

Setting alfresco.doPermissionChecks to false in solrcore.properties is not really a smart thing to do. It just means that permissions will not be checked on the SOLR tier, but they will still be checked on the Repository tier leading to way more performance overhead.

Required for our solution, users are allowed to search all content independent of access privileges and are only blocked from accessing full content, allowing them to request access if required.

This is implemented by a combination of the "doPermissionsChecks=false" mentioned above and running the "processResultsSinglePage" function in "search.lib.js" as admin to get all results and then clearing out some sensitive fields in the "getDocumentItem" function if the user is not in the admin role.

As I often tell people, a lot of the tuning advice should be taken with a grain of salt and a basic understanding of the correlation any setting may have on the performance. It would be great to know what you have tried as to judge the potential effect (benefit / damage) it could have

• Java VM intial/max heap size increased from 2/4 GB to 8/16 GB
• Different other JVM options for garbage collections tested, current settings as advised in Alfrecso 5.2 documentation
• Caches for SOLR increased
• Database tables optimized

Tested after each individual change, no real difference.

What is the IO situation in your server? SOLR and DB are very sensitive to slow IO and you can easily tank the performance by not using a proper storage medium for the data.

Tested with "hdparm -t", throughput is about 180 MB / s.

Thankful for any further input 🙂

Chris

---

You need more RAM and CPU