topic Re: OutOfMemory errors while uploading files through web service in Alfresco Archive

OutOfMemory errors while uploading files through web service

sabsurendran — Fri, 27 Feb 2009 06:06:25 GMT

We are evaluating the performance of Alfresco for our project, and are trying to test the performance under heavy loads.We have 3 different java clients which use the Alfresco Webservice API to upload 100 files of 4 MB each to the same space. We frequently get OutOfMemory exceptions on the Java heap

Re: OutOfMemory errors while uploading files through web service

derek — Fri, 27 Feb 2009 08:14:14 GMT

Hi,
With WebServices, there is a very definite wrong and right way to upload files. You need to tweak your code to upload the content direct to the server over http, otherwise WebServices (and this is just how it is) will be sucking the binary data into memory before pushing it into Alfresco. Alfresco itself does not pull the binaries into memory first.
Regards

Re: OutOfMemory errors while uploading files through web service

sisao — Wed, 18 Mar 2009 13:26:28 GMT

since i've had problems of java heap space overflows that i've solved increasing the Xmx paramter for the alfresco lauch configuration i'd like you to be more precise about these "tweaks" and how can i understand if i'm going the right or the wrong way.
For example this is the upload method i use in my applications:

UpdateResult[] result = WebServiceFactory.getRepositoryService().update(cml);     
         Reference content = result[0].getDestination();
         byte[] bytes = ContentUtils.convertToByteArray(stream);
         ContentFormat contentFormat = new ContentFormat(tipo, "UTF-8");
         WebServiceFactory.getContentService().write(content, Constants.PROP_CONTENT, bytes, contentFormat);
         System.out.println("DONE");
         uuid=content.getUuid();‍‍‍‍‍‍‍

stream is the InputStream of the file i'm going to send to the repository which i guess take some space in the java heap and you cannot avoid this memory usage.
Regards.

Re: OutOfMemory errors while uploading files through web service

derek — Wed, 18 Mar 2009 16:32:40 GMT

You need to tweak your code to upload the content direct to the server over http

Re: OutOfMemory errors while uploading files through web service

sisao — Thu, 19 Mar 2009 14:37:22 GMT

Of course i've carefully read your previous post but i cant see a different way of uploading content to the repository via WS different from the one i've posted which is the result of the alfresco SDK examples…maybe i'm not getting the point
Regards.

Re: OutOfMemory errors while uploading files through web service

kbonnet — Wed, 15 Apr 2009 15:51:41 GMT

Derek,

The Alfresco Release Script for Kofax is also using the webservices. One of our client is using this setup, but ends up with OutOfMemoryErrors when posting large documents.

Could this have to do with the fact that the release script doesnt make the document available over htttp first, but instead wrapping it in the soap request?

I also read somewhere that Axis is only able to handle 32mb documents in soap requests. Are you familiar with such a boundary?

Thanks for your time.

Regards,

Koen Bonnet

Re: OutOfMemory errors while uploading files through web service

sisao — Thu, 16 Apr 2009 08:01:51 GMT

I also read somewhere that Axis is only able to handle 32mb documents in soap requests. Are you familiar with such a boundary?

I use WS APIs and i also send documents of 1GB of size without any java heap issues.
Check your application server startup parameters and alfresco's.

Regards.

Re: OutOfMemory errors while uploading files through web service

kbonnet — Thu, 16 Apr 2009 08:23:59 GMT

Hi Sisao,

Thanks for your reply. Are you writing your own webservices client? Are you pushing the document to alfresco first, as suggested by Derek as being the right way?

We have to deal with a webservice client which is part of kind of standard software, the release script. I dont see a way of altering this to push the document first.

Thanks for your help.

Koen Bonnet

Re: OutOfMemory errors while uploading files through web service

luca — Wed, 01 Jul 2009 09:21:40 GMT

Hi all,
does anyone found a solution?

I'm getting the same problem, but I can't figure it out!

Derek, can you give us some more explanation please? :wink:

Thanks,
Luca

Re: OutOfMemory errors while uploading files through web service

lakshya — Tue, 03 Nov 2009 03:51:33 GMT

Did anyone find the solution??

I am using SOAP API to upload files.


   private String writeContentInSpace(String contentName, String title,
         String description, String spaceName, String mimeType,
         String encoding, byte[] content, String author, String isbn,
         int pageCount, String classification, String clientName,
         String clientMail, String imageId) {
      try {
         // Start the session
         AuthenticationUtils.startSession(userName, password);
      } catch (AuthenticationFault fault) {
         fault.printStackTrace();
         return "Exception occured";
      }
      try {

         // Searching for the space UUID
         // Get a reference to the repository web service
         RepositoryServiceSoapBindingStub repositoryService;
         repositoryService = WebServiceFactory.getRepositoryService();
         // Get a reference to the space we have named
         Store storeRef = new Store(Constants.WORKSPACE_STORE, "SpacesStore");
         Node[] nodes = null;
         Reference reference = null;
         Predicate predicate = null;
         try {
            reference = new Reference(storeRef, null,
                  "/app:company_home/cm:" + spaceName);

            predicate = new Predicate(new Reference[] { reference }, null,
                  null);
            nodes = repositoryService.get(predicate);
         } catch (Exception ex) {
            System.out.println("Exception occured " + ex.getMessage());
            ex.printStackTrace();
         }
         // Searching for the UUID out of the result

         String spaceUUID = "";

         if (nodes[0] != null) {
            NamedValue[] properties = nodes[0].getProperties();
            for (int j = 0; j < properties.length; j++) {
               NamedValue nv = properties[j];
               if ("{http://www.alfresco.org/model/content/1.0}name"
                     .equals(nv.getName())
                     && nv.getValue().equals(spaceName)) {
                  System.out.println("In Space : " + nv.getValue());
               }
               if ("{http://www.alfresco.org/model/system/1.0}node-uuid"
                     .equals(nv.getName())) {
                  System.out.println("Space UUID: " + nv.getValue());
                  spaceUUID = nv.getValue();
               }
            }
         }

         // Create a reference to the parent where we want to insert content
         Reference referenceWrite = new Reference(storeRef, spaceUUID, null);

         ParentReference parentReference = new ParentReference(
               referenceWrite.getStore(), referenceWrite.getUuid(), null,
               Constants.ASSOC_CONTAINS, Constants.ASSOC_CONTAINS);
         // Create content
         String page_count_str = Integer.toString(pageCount);
         NamedValue[] newProperties = new NamedValue[] {
               Utils.createNamedValue(Constants.PROP_NAME, contentName),
               Utils.createNamedValue(Constants.PROP_DESCRIPTION,
                     description),
               Utils.createNamedValue(Constants.PROP_TITLE, title),
               Utils.createNamedValue(Constants.PROP_AUTHOR, author),
               Utils.createNamedValue(Constants.PROP_ISBN, isbn),
               Utils.createNamedValue(Constants.PROP_CLIENT_MAIL,
                     clientMail),
               Utils.createNamedValue(Constants.PROP_CLIENT_NAME,
                     clientName),
               Utils.createNamedValue(Constants.PROP_PAGE_COUNT,
                     page_count_str),
               Utils.createNamedValue(Constants.PROP_CLASSIFICATION,
                     classification),
         /* Utils.createNamedValue("IMAGE_ID", imageId), */
         };

         CMLCreate create = new CMLCreate("1", parentReference, null, null,
               null, Constants.TYPE_CONTENT, newProperties);

         CMLAddAspect addAspect = new CMLAddAspect();
         addAspect.setAspect(Constants.ASPECT_VERSIONABLE);
         addAspect.setProperty(newProperties);

         addAspect.setWhere(new Predicate(
               new Reference[] { referenceWrite }, null, null));
         addAspect.setWhere_id("1");

         CML cml = new CML();
         cml.setCreate(new CMLCreate[] { create });
         cml.setAddAspect(new CMLAddAspect[] { addAspect });

         // update
         UpdateResult[] results;
         results = WebServiceFactory.getRepositoryService().update(cml);
         // Set content
         ContentFormat format = new ContentFormat(mimeType.toString(),
               encoding);
         Content newContent;
         newContent = WebServiceFactory.getContentService().write(
               results[0].getDestination(), Constants.PROP_CONTENT,
               content, format);
         // System.out.println("UUID of uploaded content : " +
         // newContent.getNode().getUuid());
         System.out.println("File Uploaded.");

         // versioning(storeRef);

         return newContent.getNode().getUuid();

      } catch (Exception e) {
         // e.printStackTrace();
         return "Exception occured";
      } finally {
         // End the session
         AuthenticationUtils.endSession();
      }
}   
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Re: OutOfMemory errors while uploading files through web service

kebbs — Thu, 25 Feb 2010 20:34:05 GMT

Here is what worked for me…


            ContentFormat format = new ContentFormat(properties.getString("contentType"), "UTF-8");
            DataHandler attachmentFile = new DataHandler(new FileDataSource(file));
            // commented out since this results in an exception
            //_contentService._setProperty(Call.ATTACHMENT_ENCAPSULATION_FORMAT, Call.ATTACHMENT_ENCAPSULATION_FORMAT_DIME);
            _contentService.addAttachment(attachmentFile);
            _contentService.writeWithAttachment(obj.getNode().getReference(), Constants.PROP_CONTENT, format);
‍‍‍‍‍‍‍‍

Re: OutOfMemory errors while uploading files through web service

junieboy — Sun, 04 Apr 2010 13:26:32 GMT

Hi,
With WebServices, there is a very definite wrong and right way to upload files. You need to tweak your code to upload the content direct to the server over http, otherwise WebServices (and this is just how it is) will be sucking the binary data into memory before pushing it into Alfresco. Alfresco itself does not pull the binaries into memory first.
Regards

Hi Derek,

Good evening. It is very probable by this quarter that we start our development for Alfresco for our Document Management, and our communication would be through web services (more importantly "both saving of files and metadata"). We will be saving files from our system to Alfresco through the Alfresco web service API and basically, we'll develop a custom content model and all the metadata will be coming from our system will be attached to files and these metadata will be used in searching for these files in Alfresco.

My question is how reliable will this Alfresco web services API approach be since I'm seeing in this post that there's an issue saving files through web service? Also, I was having some question since I am seeing relatively limited Javadoc/API documentation regarding web services.

Also, could I ask you to please further elaborate the solution? How do we tweak the code such that we upload direct to the server over http? What Java API should be used to do this? Please share some examples or diagrams for our reference. Thanks a lot!

Best Regards,
Junie

Re: OutOfMemory errors while uploading files through web service

derek — Wed, 07 Apr 2010 10:39:06 GMT

Hi,
If you're starting from scratch, you should consider using CMIS as your starting point. There are servers available for testing directly e.g. http://cmis.alfresco.com/cmis.
Regards

Re: OutOfMemory errors while uploading files through web service

junieboy — Wed, 07 Apr 2010 14:09:15 GMT

Hi,
If you're starting from scratch, you should consider using CMIS as your starting point. There are servers available for testing directly e.g. http://cmis.alfresco.com/cmis.
Regards

Hi Derek,

Unfortunately we are using Alfresco 3.0 (the instance is already existing for sometime in our infrastructure and is already being used by our domains but with no web services integration) and I believe CMIS is only available at 3.3. So this is the first time we're doing it. Could I ask for more insights re my questions? Thanks a lot.

Regards.

Re: OutOfMemory errors while uploading files through web service

akoudrin — Wed, 13 Nov 2013 20:05:24 GMT

Could you please post all your includes to make the above work, please?