UploadContentServlet called from .NET fails for big files

[malfresco]

Hi,

Short scenario description:
Client machine: not relevant (anyway: CPU intel dual core 2200, RAM 2GB, SO 2008 server 32b)
Server machine: CPU intel 2x2 1800, RAM 4GB, SO open suse 11.0

Alfresco 3 labs(with derby)
- non default java_option Xmx increased to 1.5GB
   

I'll describe shortly my problem, anybody can help I'd appreciate.


I try to upload big files content to alfresco repository using a C# small application;

C#-pseudo snippets:________________________

string url = "http://NOT_LOCALHOST_MACHINEORT/alfresco/upload/workspace/SpacesStore/NODE_UUID/FILENAME";
url = url + ticketInfo;

HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Timeout = 1000 * 60 * 10;//it doesn't affect too much. if I set a very small value I'll get a time out
request.Method = "PUT";

Stream requestStream = request.GetRequestStream();

MYCOPYSTREAM_METHOD(requestStream, souceFileStream);
//CRITICAL POINT 1

requestStream.Flush();
requestStream.Close();

HttpWebResponse response = request.GetResponse() as HttpWebResponse;
//CRITICAL POINT 2

Console.WriteLine(response.StatusCode);

___________________________________


Explanations/PROBLEMS:

//CRITICAL POINT 1
Out of memory exception if the source file is => 1GB (the biggest file successfully uploaded was around 980MB big). I have tested MYCOPYSTREAM_METHOD using 2, 4 and 6 GB files copied from local to a remotely mapped drive and it works.

//CRITICAL POINT 2
Supposing I try to upload big file, but smaller than 1GB (e.g. 400MB), I get time out regardless how big is set the "request.Timeout" value and even the file content was successfully uploaded (I can see it, and it has the appropriate content, into /opt/alfresco/alf_data/contentstore/… folder)

NOTES:
MYCOPYSTREAM_METHOD uses byte array buffer (usually 1024 lenght, but customizable via property file) to iterativelly read from source and write to destination, and nor source nor destination are closed/disposed inside it.


Thank you in advance,

malfresco

[mrogers]

Is there a "Chunked" property that you can set to alter the behaviour?

[malfresco]

Thank you MRogers,

sorry for my delayed response, I was really busy last weeks with another issues. Yesterday I've checked your suggestion and…

The property exists and is named <request>.SendChunked

setting this property to true solves the issue described as CRITICAL POINT 2.

but the first issue is still unresolved

please let me know if there is any known "native" limitation of the alfresco/tomcat which could leads to that behaviour. Because for each file successfully uploaded using the web client, via browser (IE or Firefox) during my tests, I was able to upload it correctlly with my C# app too, and vice-versa. So I think is an server-side limitation. The bigger file uploaded was 980 MB (999MB file fails to upload in both scenario browser, C# app).

thanks,
malfresco

[mrogers]

Are you saying that you fixed the timeout issue by setting the chunked property?

I'm having great difficulty understanding your description of the issue.  Could you try again to say what the problem is without switching between different problems and scenarios ?

[malfresco]

Hi,

I'll describe one and just one issue:
NAME: C# client Alfresco fails to upload big files as alfresco content

CODE SNIPET:

       
…
url = "http://$SERVER:$PORT/alfresco/upload/workspace/SpacesStore/$NUUID/$FILENAME?ticket=$TICKET";
…
HttpWebResponse response = null;
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "PUT";
request.SendChunked = true;

Stream requestStream = request.GetRequestStream();
CopyStream(contentStream, requestStream);
requestStream.Flush();
requestStream.Close();
…


//CopyStream body
…
int bufferSize = Config.Configuration.BUFFER_SIZE, rest = 0;
byte[] buffer = new byte[bufferSize];

while ((rest = source.Read(buffer, 0, bufferSize)) > 0)
{
     destination.Write(buffer, 0, rest);//here is the point where Exception occurs!!!
     destination.Flush();
}
…


‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

It will throw an Exception when try to upload 1GB file.

the relevant trace of Exception

System.Net.ScatterGatherBuffers.AllocateMemoryChunk(Int32 newSize)
   at System.Net.ScatterGatherBuffers.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at AlfrescoComponent.Utils.General.CopyStream(Stream source, Stream destination)

My appoligize if I've confused you (one reason is my pour english), I'll try to avoid that in for the future.

Thanks in advance,
MAlfresco

[mrogers]

Is the at AlfrescoComponent.Utils.General.CopyStream(Stream source, Stream destination)
your code?  

Could you post it please.

[malfresco]

Yes is my code

You should know that I've tested the CopyStream method with signifiant bigger files (2GB, 4GB) and it works using "file-stream"s.
The issues occurs just when I try to copy into a stream provided by the WebRequest.

        /// <summary>
        /// Copies the source to destination stream, iteratively, using an byte[] buffer with customizable length.
        /// The length of the byte[] is given by Config.Configuration.BUFFER_SIZE.
        /// IMPORTANT: it dosen't close nor dispose any of the given streams. It just flushes the destination for each successfuly writed byte[].
        /// </summary>
        /// <param name="source">The source stream to copy from</param>
        /// <param name="destination">The destination stream to copy into</param>
        /// <returns>true if operation successfully ended</returns>
        public static bool CopyStream(Stream source, Stream destination)
        {
            bool success = true;
            try
            {
                int bufferSize = Config.Configuration.BUFFER_SIZE, rest = 0;
                byte[] buffer = new byte[bufferSize];

                while ((rest = source.Read(buffer, 0, bufferSize)) > 0)
                {
                    destination.Write(buffer, 0, rest);
                    destination.Flush();
                }
            }
            catch (Exception e)
            {
                //logger.ERR(e);
                success = false; ;
            }
            return success;
        }

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

here is the Configuration class

    public class Configuration
    {
        #region buffer size
        /// <summary>
        /// Stores the buffer size used for general stream-bytes operations, to avoid system memory limitation issues
        /// </summary>
        private static int _bufferSize = 0;
        /// <summary>
        /// accessor for the buffer size. The default value is 1024 if no customized provided (via properties)
        /// </summary>
        public static int BUFFER_SIZE
        {
            get
            {
                if (_bufferSize == 0)//no value set into the properties
                {
                    _bufferSize = 1024;
                }
                return _bufferSize;
            }
        }
        #endregion //buffer size
    }


‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍