topic Re: CIFS slowdown in Alfresco Archive

CIFS slowdown

atariq — Mon, 05 Nov 2007 17:50:44 GMT

We've been going through a departmental testing phase of Alfresco 2.1 (with CIFS and NTLM) here at the office and things have generally been looking good so far.The way our repository has been setup now is to have the alf_data directory mapped over the network to a larger storage unit. I had made th

Re: CIFS slowdown

atariq — Mon, 12 Nov 2007 14:52:25 GMT

Upon further testing, it appears that the web client is inconsistent in its speed while opening the large directory in question. Sometimes it's fast (but not as fast as opening directories on a shared directory on a file server) and at other times it slows down to a crawl.

Again, thanks in advance for any tips on how to speed things up!

Re: CIFS slowdown

andy — Mon, 12 Nov 2007 15:06:50 GMT

Hi

The lucene indexes must be on a local disk. While this will change in the future, it is potentially unsafe for indexing and will give poor performance.

Andy

Re: CIFS slowdown

atariq — Mon, 12 Nov 2007 15:46:49 GMT

Thanks for the reply, Andy.

Is it possible to have the indexes on the local disk but the repository content elsewhere? Our reasons for having the repository on a separate server is two fold: easier backups, and better utilization of our SAN to provide much larger disk space.

Thanks, again.

Re: CIFS slowdown

andy — Tue, 13 Nov 2007 11:18:07 GMT

Hi

Yes, this is described on the wiki. There are properties in repository.properties.

Andy

Re: CIFS slowdown

atariq — Wed, 14 Nov 2007 20:11:03 GMT

I think I've found the documentation on the repository.properties file (http://wiki.alfresco.com/wiki/Technical_Tips#Externalising_a_Configuration_2) but I now have a quick question regarding this.

I've noticed that there are two files that refer to the repository properties:

1) …/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/repository.properties

which contains relative path names to the different components of the repository

and

2) …/tomcat/shared/classes/alfresco/extension/custom-repository.properties

which obviously contains custom settings for the rep. This is the file that I had edited in order for the repository to be redirected to the secondary server.

My question is: does one of these files override the other? Should one of them be 'commented' out? Or do both of them need to edited?

Much thanks in advance again!

Re: CIFS slowdown

atariq — Thu, 15 Nov 2007 07:52:04 GMT

Just to follow up, I edited both the custom-repository.properties and repository.properties files to have the lucene-indexes point to a local location. Of course, this meant that I had to move the lucene-indexes directory from the remote server to the local one accordingly.

After restarting Alfresco, I tried to access the large directory again through CIFS and unfortunately the speed is still the same – around 30 seconds to load every time it is accessed.

Do you think I have to recreate the indexes from scratch by reuploading the docs? While that would be quite inconvenient, if it would work then it would have to be done, I guess.

Otherwise, I'm running out of ideas. Thanks again for any help.

Re: CIFS slowdown

atariq — Wed, 28 Nov 2007 21:47:10 GMT

Hi again.

To follow up on this on-going issue, I've tried splitting the lucene indexes from the repository that is sitting on the external fileserver and moved the indexes to the local Alfresco server. This did not result in any noticeable increase in speed for the large directory.

Then, I deleted the lucene-indexes folder on the local server and recreated the indexes by setting the relevant flag in repository.properties from VALIDATE to FULL. After an hour or so, it was successfully completed. Sadly, though, this did not lead to any speeding up of the services either, so we are basically back to square one.

I'm wondering, though, whether an NFS share would be the root cause of this issue? Currently, the setup involves having the repository on an NFS share on a separate fileserver and it was directly after this move that we started seeing a significant slowdown (more specifically a slowdown on the largest directory).

Again, any insights would be greatly appreciated!

Thank you.

Re: CIFS slowdown

atariq — Mon, 10 Dec 2007 14:47:28 GMT

Any thoughts or suggestions on this would still be very much appreciated.

Are there known issues with NFS shares and the Alfresco repository?

Re: CIFS slowdown

jenglert — Mon, 10 Dec 2007 15:26:44 GMT

Do you think I have to recreate the indexes from scratch by reuploading the docs? While that would be quite inconvenient, if it would work then it would have to be done, I guess.

You can have the repository automatically reindex files by setting this property in the custom-repository.properties.

index.recovery.mode=FULL

This parameter basically deletes any existing indexes and rebuilds them. It seems like you have probably rebuilt the indexes when moving file systems anyways though.

Re: CIFS slowdown

jenglert — Mon, 10 Dec 2007 15:32:46 GMT

Any thoughts or suggestions on this would still be very much appreciated.

Are there known issues with NFS shares and the Alfresco repository?

We had problems like this quite a bit on a project we worked on. You are seeing inconsistent results because Lucene implements some internal caching (just like Oracle). When the results are in memory, the results can be returned quickly. When they are are on the HDD, it takes longer. Our solution was to enforce a maximum 500 files per directory limit. This worked pretty well for us. I'm not sure whether this would be a possibility for you?

Jim Englert
jenglert@tsgrp.com

Re: CIFS slowdown

atariq — Mon, 10 Dec 2007 17:30:29 GMT

Thanks for the response, Jim.

Yup, I learned about the reindexing strategy and rebuilt it successfully however it did not give any noticeable performance increase.

With regards to setting directory limits, that would be something I'd have to discuss further with the relevant department here at the office. That said, I'm pretty sure Alfresco should be able to handle this directory size as in previous tests of having the repository on the same server everything seemed to be just fine. It's only AFTER moving the repository over to an NFS share on our fileserver that the slowdown has started to occur.

Re: CIFS slowdown

jenglert — Mon, 10 Dec 2007 21:39:41 GMT

The actual location of the content should (hopefully) have no impact on search performance. Lucene might take longer to index the content initially but queries against Lucene should be unaffected by the underlying content store location. Your finding are consistent with what we found (thus mandating the 500 file limit). Maybe there is some sort of issue where path based Lucene queries (CIFS, the web interface, and WebDAV all use Lucene) are actually hitting the content store for some reason.

Problem is, that section of Alfresco code is likely very hard to follow to see if there might be an issue. I'll put it on the back burner though.

Re: CIFS slowdown

atariq — Tue, 11 Dec 2007 22:12:00 GMT

Thanks for following this thread.

An interesting note I've been informed of today is that a general slowdown is experienced anywhere on the repository whenever a content item is being saved. For instance, it was demonstrated to me that whenever a 1.5mb Excel spreadsheet was being saved via CIFS onto any directory whose file count was considerably less than 500, we were still seeing around 15 - 20 seconds of delay before it got saved. Of course, for directories on which sizable traffic is expected this becomes rather problematic. As such, I really wonder whether the file count limit is the culprit.

If I really run of ideas, I'm going to try to move the repository back to the Alfresco server and carry out some more tests. I'll keep you guys posted!

Thanks again.

Re: CIFS slowdown

andy — Fri, 21 Dec 2007 12:26:44 GMT

Hi

How does the performance of CIFS and FTP compare?

CIFS is a very chatty protocol and may well have some issues backed with NFS.

If your lucene indexes are stored locally then a search never goes to NFS. You should not have lucene indexes mounted over NFS (this is actually not supported in lucene until 2.1). Indexes in Alfresco over NFS will be supported in the future. At the moment they must be local or on some FS that is equavalent.

Is the CIFS performance consistent or is it much better the next time you open the directory.

It is possible that your CIFS client is going to get some meta data information for each file (which may end up pulling the content) - try changing what is displayed in the client (e.g. remove fils size info, thumbnails that kind of thing).

Andy

Re: CIFS slowdown

callermd — Fri, 25 Jan 2008 06:38:28 GMT

Using a local disk and lucene we are getting very slow CIFS browsing and saving. We have many directories with thousands of files plus large numbers of scanned images.

Running AC 2.1 under FC7.

Re: CIFS slowdown

loftux — Fri, 25 Jan 2008 07:18:24 GMT

This may be a windows explorer problem
http://www.ss64.com/nt/slow_browsing.html
Or try searching directly on microsoft support, you will find lots of issues there (tip: search on both smb and cifs)

Can you try mounting CIFS from a linux pc and se if you get the same response times listing files.

Anyway, thousands of files will take longer to list than just a few try segmenting into spaces. The actual save procedure should not be affected, unless you count the time it takes for the windows app save dialog to open an list the files already existing.

/Peter LÃƒÂ¶fgren

Re: CIFS slowdown

callermd — Fri, 11 Apr 2008 04:43:17 GMT

I found the query is causing the save slowdown..

select * from
alf_node_status nodestatus0_
inner join alf_node nodeimpl1_
on nodestatus0_.node_id=nodeimpl1_.id
where nodeimpl1_.id in
(select nodeimpl3_.id
from alf_child_assoc childassoc2_
inner join alf_node nodeimpl3_ on childassoc2_.child_node_id=nodeimpl3_.id
where childassoc2_.parent_node_id=1595175 and childassoc2_.is_primary=1);

It takes 7 seconds to execute this query on my machine.

Interestingly the subquery returns 0 rows.

Re: CIFS slowdown

callermd — Fri, 11 Apr 2008 05:29:06 GMT

The underlying cause is that the MySQL subquery optimizer is retarded:

http://bugs.mysql.com/bug.php?id=9090&thanks=3&notify=579

http://www.xaprb.com/blog/2006/04/30/how-to-optimize-subqueries-and-joins-in-mysql/#commen

Re: CIFS slowdown

callermd — Fri, 11 Apr 2008 05:57:01 GMT

And it looks like it is fixed in latest head

From Node.hbm.xml

   <query name="node.GetPrimaryChildNodeStatuses">
      select
         status
      from
         org.alfresco.repo.domain.hibernate.NodeStatusImpl as status,
         org.alfresco.repo.domain.hibernate.ChildAssocImpl as assoc
         join assoc.child as child
      where
         assoc.parent = arent and
         assoc.isPrimary = true and
         status.node = child
   </query>


This is the correct way of doing this.

My save time went from 7 seconds to less than one.