topic Re: Performance problems after a while that is not used in Alfresco Archive

Performance problems after a while that is not used

npasquetto — Wed, 20 Aug 2014 07:52:03 GMT

Hi,I've got a strange behavior in terms of performance in a webscript that consists in a Lucene query with some condition (by path, by date, by aspect and sometimes by some imap aspect metadata).The webscript is very slow when it is called after a while that is not used but is very very fast if it i

Re: Performance problems after a while that is not used

afaust — Wed, 20 Aug 2014 09:01:39 GMT

Hello,

a couple of reasons can influence the performance and there are multiple levels of caches to consider here, not just the Alfresco node and properties caches. E.g. Lucene can and will cache some index data necessary for performing the query. The database can and will cache some table data that even affects performance of queries with completely different results. And the OS may at a low level cache file system contents that Lucene or the DB use to execute queries when they have not cached data in memory.

What kind of magnitude of performance difference do you observe? How many results are we talking about here? What exactly does the Lucene query look like (keep in mind, PATH queries can be veerryyy expensive)?

Regards
Axel

Re: Performance problems after a while that is not used

npasquetto — Wed, 20 Aug 2014 10:07:00 GMT

<blockquote>What kind of magnitude of performance difference do you observe?</blockquote>
The difference is huge: 80 seconds for slow response, 2 seconds for fast response.

<blockquote>How many results are we talking about here?</blockquote>
Another thing that i forgot to specify in my post is that seems there is no relation between performance and number of results. Anyway we are talking of a maximum of 2000 results.

<blockquote>What exactly does the Lucene query look like (keep in mind, PATH queries can be veerryyy expensive)?</blockquote>
We have some files with the IMAP aspect, and we filter on metadatas as sentDate, sender, subject. All the files are organized in subspaces under a common spaces, so we introduce the PATH condition to limit the search, but in the path we need to use some wildcart to be able to go in subspaces. Could this condition affect the performance? If yes, I can remove it.

Re: Performance problems after a while that is not used

afaust — Thu, 21 Aug 2014 08:30:05 GMT

Ok - 80 vs 2 seconds is really a difference that are hard to explain away just by warm-up issues with some caches. Even Lucene index queries rarely rack up this kind of difference between cold+hot caches…

Does the webscript do anything else with the results of the Lucene query apart from generating the response via FTL? Can you insert timing measurements into the code of the web script to determine at what point in the execution what amount of time has elapsed?

Regards
Axel

Re: Performance problems after a while that is not used

npasquetto — Thu, 21 Aug 2014 12:38:57 GMT

<blockquote>Does the webscript do anything else with the results of the Lucene query apart from generating the response via FTL?</blockquote>
The webscript is very simple: based on the URL parameters it build a query string through a simple string concatenation; then there is a call to the search API

search.query(def);‍

where def is the object containing the search confinguration (query, language, sort…).

The result of the query will be assigned as variable in the model and passed to FTL to build the JSON response.

So there is no other elaboration except the query call.

Re: Performance problems after a while that is not used

npasquetto — Mon, 25 Aug 2014 10:18:46 GMT

<blockquote>Can you insert timing measurements into the code of the web script to determine at what point in the execution what amount of time has elapsed?</blockquote>
I've put some timing measurements in the code (query building, before the call to search API, after the call to search API, at the end of the script) and ~~I confirm that all the time is spent by the search API.~~ I noticed that sometimes the query is faster, a little more than 1 second, but I get the answer after 30 seconds. By raising the webscript immediately after, I get an instant response.

I noticed also that the clause is ignored if MaxItems use the admin user: there is no way to avoid this?

Re: Performance problems after a while that is not used

kaynezhang — Tue, 26 Aug 2014 03:11:54 GMT

How many memory did you allocate to JVM? have you tried to increase the heap size for the JVM?
Searching can use a lot of memory for a large index,. I guess if your JVM is not running with a large enough HEAP size then the JVM will pays the price of initializing caches at your first query at some time.

Re: Performance problems after a while that is not used

npasquetto — Tue, 26 Aug 2014 07:45:30 GMT

Thank you for your help.
<blockquote>if your JVM is not running with a large enough HEAP size then the JVM will pays the price of initializing caches</blockquote>
The JVM has a very large HEAP size, the server is very powerful.

But I've notice that, often the time of execution of the query seems low, but the time of response is high. There is a caching mechanism for the template?

Re: Performance problems after a while that is not used

kaynezhang — Tue, 26 Aug 2014 08:09:00 GMT

Yes,first query will cause the index caches to be warmed up( especially for sorting) and this is why the first query takes some time.
Where did you place your index on? keep the index on local disk will improve performence.
How much memory did you leave for operating system ? It seems os also needs some momory to cache index files.

Re: Performance problems after a while that is not used

npasquetto — Tue, 26 Aug 2014 12:33:36 GMT

<blockquote>this is why the first query takes some time</blockquote>
OK, but after some investigating, it seems that is not the query the problem.

for example:
<ol>
<li>First request: total time 30 sec, query time 2 sec</li>
<li>Second query (the same): total time 2,5 sec, query time 2 sec</li>
<li>Third query (completely different resultset): total time 2,5 sec, query time 2 sec</li>
</ol>
And in my cluster, I verified this behaviour in every single node.

Re: Performance problems after a while that is not used

kaynezhang — Wed, 27 Aug 2014 02:12:00 GMT

You mean most of the time is taken by FTL rendering process?

Re: Performance problems after a while that is not used

npasquetto — Wed, 27 Aug 2014 07:50:22 GMT

By the FTL rendering process or by another process before or after the execution of the controller.
But when the cache that is involved in this phase is warmed up, this process does not require so much time again.

Re: Performance problems after a while that is not used

kaynezhang — Wed, 27 Aug 2014 14:01:03 GMT

You can add some logging information at the beginning and at the end of your javascript controller to trace time consumption.


if(logger.isLoggingEnabled())
{
       logger.log("***");
}
‍‍‍‍‍‍

You also can estimate the time taken by ftl rending process using the log

Re: Performance problems after a while that is not used

npasquetto — Tue, 02 Sep 2014 07:07:35 GMT

Finally i've adopted a stupid solution: i've scheduled a dummy call to the web script every three minutes. I would like to know what kind of cache is involved in this process…