topic Re: UploadContentServlet called from .NET fails for big files in Alfresco Archive

UploadContentServlet called from .NET fails for big files

malfresco — Mon, 15 Jun 2009 10:57:45 GMT

Hi,Short scenario description:Client machine: not relevant (anyway: CPU intel dual core 2200, RAM 2GB, SO 2008 server 32b)Server machine: CPU intel 2x2 1800, RAM 4GB, SO open suse 11.0 Alfresco 3 labs(with derby)- non default java_option Xmx increased to 1.5GB I'll describe shortly my problem, any

Re: UploadContentServlet called from .NET fails for big files

mrogers — Mon, 15 Jun 2009 11:35:57 GMT

Is there a "Chunked" property that you can set to alter the behaviour?

Re: UploadContentServlet called from .NET fails for big files

malfresco — Wed, 24 Jun 2009 06:01:40 GMT

Thank you MRogers,

sorry for my delayed response, I was really busy last weeks with another issues. Yesterday I've checked your suggestion and…

The property exists and is named <request>.SendChunked

setting this property to true solves the issue described as CRITICAL POINT 2.

but the first issue is still unresolved

please let me know if there is any known "native" limitation of the alfresco/tomcat which could leads to that behaviour. Because for each file successfully uploaded using the web client, via browser (IE or Firefox) during my tests, I was able to upload it correctlly with my C# app too, and vice-versa. So I think is an server-side limitation. The bigger file uploaded was 980 MB (999MB file fails to upload in both scenario browser, C# app).

thanks,
malfresco

Re: UploadContentServlet called from .NET fails for big files

mrogers — Wed, 24 Jun 2009 08:23:29 GMT

Are you saying that you fixed the timeout issue by setting the chunked property?

I'm having great difficulty understanding your description of the issue. Could you try again to say what the problem is without switching between different problems and scenarios ?

Re: UploadContentServlet called from .NET fails for big files

malfresco — Thu, 25 Jun 2009 06:49:08 GMT

Hi,

I'll describe one and just one issue:
NAME: C# client Alfresco fails to upload big files as alfresco content

CODE SNIPET:

       
…
url = "http://$SERVER:$PORT/alfresco/upload/workspace/SpacesStore/$NUUID/$FILENAME?ticket=$TICKET";
…
HttpWebResponse response = null;
HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest;
request.Method = "PUT";
request.SendChunked = true;

Stream requestStream = request.GetRequestStream();
CopyStream(contentStream, requestStream);
requestStream.Flush();
requestStream.Close();
…


//CopyStream body
…
int bufferSize = Config.Configuration.BUFFER_SIZE, rest = 0;
byte[] buffer = new byte[bufferSize];

while ((rest = source.Read(buffer, 0, bufferSize)) > 0)
{
     destination.Write(buffer, 0, rest);//here is the point where Exception occurs!!!
     destination.Flush();
}
…


‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

It will throw an Exception when try to upload 1GB file.

the relevant trace of Exception

System.Net.ScatterGatherBuffers.AllocateMemoryChunk(Int32 newSize)
   at System.Net.ScatterGatherBuffers.Write(Byte[] buffer, Int32 offset, Int32 count)
   at System.Net.ConnectStream.InternalWrite(Boolean async, Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
   at System.Net.ConnectStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at AlfrescoComponent.Utils.General.CopyStream(Stream source, Stream destination)

My appoligize if I've confused you (one reason is my pour english), I'll try to avoid that in for the future.

Thanks in advance,
MAlfresco

Re: UploadContentServlet called from .NET fails for big files

mrogers — Thu, 25 Jun 2009 08:50:47 GMT

Is the at AlfrescoComponent.Utils.General.CopyStream(Stream source, Stream destination)
your code?

Could you post it please.

Re: UploadContentServlet called from .NET fails for big files

malfresco — Thu, 25 Jun 2009 10:09:24 GMT

Yes is my code

You should know that I've tested the CopyStream method with signifiant bigger files (2GB, 4GB) and it works using "file-stream"s.
The issues occurs just when I try to copy into a stream provided by the WebRequest.



        /// <summary>
        /// Copies the source to destination stream, iteratively, using an byte[] buffer with customizable length.
        /// The length of the byte[] is given by Config.Configuration.BUFFER_SIZE.
        /// IMPORTANT: it dosen't close nor dispose any of the given streams. It just flushes the destination for each successfuly writed byte[].
        /// </summary>
        /// <param name="source">The source stream to copy from</param>
        /// <param name="destination">The destination stream to copy into</param>
        /// <returns>true if operation successfully ended</returns>
        public static bool CopyStream(Stream source, Stream destination)
        {
            bool success = true;
            try
            {
                int bufferSize = Config.Configuration.BUFFER_SIZE, rest = 0;
                byte[] buffer = new byte[bufferSize];

                while ((rest = source.Read(buffer, 0, bufferSize)) > 0)
                {
                    destination.Write(buffer, 0, rest);
                    destination.Flush();
                }
            }
            catch (Exception e)
            {
                //logger.ERR(e);
                success = false; ;
            }
            return success;
        }

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

here is the Configuration class



    public class Configuration
    {
        #region buffer size
        /// <summary>
        /// Stores the buffer size used for general stream-bytes operations, to avoid system memory limitation issues
        /// </summary>
        private static int _bufferSize = 0;
        /// <summary>
        /// accessor for the buffer size. The default value is 1024 if no customized provided (via properties)
        /// </summary>
        public static int BUFFER_SIZE
        {
            get
            {
                if (_bufferSize == 0)//no value set into the properties
                {
                    _bufferSize = 1024;
                }
                return _bufferSize;
            }
        }
        #endregion //buffer size
    }


‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍