Disappearing Spaces

[macros]

Hi,

I'm wondering if someone else has run into the following problem:

We have a rather large but fairly static space structure.  In consists of around 40 spaces each with their own project spaces inside them and each of these project spaces containts roughly 12-20 sub spaces.  Of those 40 main spaces, one has got a fairly large number of projects, say around 200.  This is all running on the 3.0 Labs Edition of Alfresco on a CentOS machine using MySQL 5.  For backup purposes I mount the CIFS share locally on the Linux machine itself and run rsync to sync the data offsite.  The strange thing is that while rsync is running about 60 spaces inside the 200 project space structure disappear.  They're not accessible by either the web frond end or local CIFS drive. 

I can add a few things to this:
1) Its got nothing to do with rsync.  A find, or large copy from the CIFS drive produces the same result.
2) It happens on the Labs edition as well as the latest Enterprise version of alfresco (the trial version at least)
3) I thought it might have something to do with the Linux CIFS implementation so I tried mounting the drive via NFS.  For some reason unknown to me, the NFS drive never show 8 of my 200 spaces even when nothing else is happening on the server.  It is rather strange, in theory the spaces exists but an ls on linux doesn't show them, they do appear on the Web frontend though.  (again, this happens on both versions of alfresco)
4) The spaces doesn't need to contain any data, having the spaces themselves set up is enough for all this to occur.
5) The number of spaces disappearing varies depending on how busy the server is.  For instance, running an rsync and find simultaneously will cause more spaces to disappear.
6) When rsync/find or whatever stops, the spaces reappear in both CIFS and the Web front end.

Also, when I say disappear, I mean an ls on linux report 140ish spaces rather then 200 and the same goes for the web front end. 

So has anyone seen this or something similar?  Maybe have a solution?

[macros]

I've created a quick testcase for this:

Start with a blank new install of Alfresco Enterprise (trial version in my case). 
Once its started up, mount the drive locally on linux :

lindevel mnt # mount -t cifs //localhost/Alfresco /mnt/alfresco -o username=admin,password=admin‍

Next run the folling script on the local CIFS share:

#!/bin/bash

mkdir TestSpace
cd TestSpace
for i in $(seq 1 1 200)
do
  mkdir "00$i - Project $i"
  cd "00$i - Project $i"
  for a in $(seq 1 1 12)
  do
    mkdir "Subspace $a"
  done
  cd ..
done
cd ..
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

You should now have a space called TestSpace with 200 project spaces each with 12 spaces of their own.
Now try the following inside the newly created TestSpace space:

lindevel TestSpace # ls -1 | wc -l
203
lindevel TestSpace #
‍‍‍‍‍

This shows that there is currently 200 folders (plus the 3 desktop actions) inside the space. 

Now open another terminal, go to the same space and do the following:

lindevel TestSpace # find . -name "test" -print‍

You don't need to search for something that exists, anything will do. 

While this is running, go back to the other terminal and do the same command as above:

lindevel TestSpace # ls -1 | wc -l
148
lindevel TestSpace #‍‍‍

This is clearly not right, some of the spaces are not showing up at all.

Now try the following: (stop the find, make sure the server is not doing anything else perhaps by stopping/starting alfresco if you like)

lindevel mnt # mount -t nfs localhost:/alfresco /mnt/nfs
lindevel mnt # cd nfs
lindevel nfs # cd TestSpace/
lindevel TestSpace # ls -1 |wc -l
192
lindevel TestSpace #
‍‍‍‍‍‍‍‍

Again, this is not right as there are 8 spaces missing.  On some further examination, it appears as though every 23rd space is left out (counting from 0).

I seem to have been mistaken about this being reflected on the web front end though, all spaces stay visible throughout the tests above, it only disappears in CIFS.  NFS stays wrong no matter what you do.

[macros]

I've fixed both of the issues above.  Both were because of bugs in Alfresco.

NFS wasn't working because every 22nd space was enough to fill up an RPC packet, it skipped an entry before it moved on to the next lot.  (This error would probably have occured in CIFS as well in certain cases)
Of course if I knew that NFS (version 2 and 3) didn't support UTF-8 before I started fixing this I wouldn't have bothered, however, seeing that UTF-8 isn't supported, it forced me to fix the CIFS issue as well.

The CIFS problem was slightly more complicated and only occurs in very certain circumstances:
a.) You have to run something that causes a lot of search objects to be created/destroyed (ie. a find, rsync, something that traverses directories basically)
b.) While that is going on, use the same client machine that is doing the operation above, to create another search object but one that has to be continued.  Its likely that it will only complete its first iteration.

Not really sure what to do with the code, don't really want to paste it all here.  I suppose if someone at alfresco is interested they can pm/mail me and I'll send it along.

[mikeh]

Thanks for digging into this. We would definitely be interested in any fixes you've put together.

The easiest way to contribute is to raise a JIRA ticket and attach the source code to that.

Thanks,
Mike

[macros]

ALFCOM-2716 and ALFCOM-2717

Feel free to pay me. :wink:

Beer is indeed an acceptable currency.

[mikeh]

Thanks - we'll assign to the owner of those projects for review.

Mike