OutOfMemory errors while uploading files through web service

[sabsurendran]

We are evaluating the performance of Alfresco for our project, and are trying to test the performance under heavy loads.
We have 3 different java clients which use the Alfresco Webservice API to upload 100 files of 4 MB each to the same space. We frequently get OutOfMemory exceptions on the Java heap space, and tweaking the Java memory params has not helped.

This is what we have in alfresco.bat:
-Xms518m -Xmx1024m -Xss96k -XX:MaxPermSize=128m -server

The machine where Alfresco repository is hosted has 2 GB RAM. Does anyone know if we need to tweak something else?

[derek]

Hi,
With WebServices, there is a very definite wrong and right way to upload files.  You need to tweak your code to upload the content direct to the server over http, otherwise WebServices (and this is just how it is) will be sucking the binary data into memory before pushing it into Alfresco.  Alfresco itself does not pull the binaries into memory first.
Regards

[sisao]

since i've had problems of java heap space overflows that i've solved increasing the Xmx paramter for the alfresco lauch configuration i'd like you to be more precise about these "tweaks" and how can i understand if i'm going the right or the wrong way.
For example this is the upload method i use in my applications:

UpdateResult[] result = WebServiceFactory.getRepositoryService().update(cml);     
         Reference content = result[0].getDestination();
         byte[] bytes = ContentUtils.convertToByteArray(stream);
         ContentFormat contentFormat = new ContentFormat(tipo, "UTF-8");
         WebServiceFactory.getContentService().write(content, Constants.PROP_CONTENT, bytes, contentFormat);
         System.out.println("DONE");
         uuid=content.getUuid();‍‍‍‍‍‍‍

stream is the InputStream of the file i'm going to send to the repository which i guess take some space in the java heap and you cannot avoid this memory usage.
Regards.

[derek]

You need to tweak your code to upload the content direct to the server over http

[sisao]

Of course i've carefully read your previous post but i cant see a different way of uploading content to the repository via WS different from the one i've posted which is the result of the alfresco SDK examples…maybe i'm not getting the point 
Regards.

[kbonnet]

Derek,

The Alfresco Release Script for Kofax is also using the webservices. One of our client is using this setup, but ends up with OutOfMemoryErrors when posting large documents.

Could this have to do with the fact that the release script doesnt make the document available over htttp first, but instead wrapping it in the soap request?

I also read somewhere that Axis is only able to handle 32mb documents in soap requests. Are you familiar with such a boundary?

Thanks for your time.

Regards,

Koen Bonnet

[sisao]

I also read somewhere that Axis is only able to handle 32mb documents in soap requests. Are you familiar with such a boundary?

I use WS APIs and i also send documents of 1GB of size without any java heap issues.
Check your application server startup parameters and alfresco's.

Regards.

[kbonnet]

Hi Sisao,

Thanks for your reply. Are you writing your own webservices client? Are you pushing the document to alfresco first, as suggested by Derek as being the right way?

We have to deal with a webservice client which is part of kind of standard software, the release script. I dont see a way of altering this to push the document first.

Thanks for your help.

Koen Bonnet

[luca]

Hi all,
does anyone found a solution?

I'm getting the same problem, but I can't figure it out!

Derek, can you give us some more explanation please?  :wink:

Thanks,
Luca

[lakshya]

Did anyone find the solution??

I am using SOAP API to upload files.

   private String writeContentInSpace(String contentName, String title,
         String description, String spaceName, String mimeType,
         String encoding, byte[] content, String author, String isbn,
         int pageCount, String classification, String clientName,
         String clientMail, String imageId) {
      try {
         // Start the session
         AuthenticationUtils.startSession(userName, password);
      } catch (AuthenticationFault fault) {
         fault.printStackTrace();
         return "Exception occured";
      }
      try {

         // Searching for the space UUID
         // Get a reference to the repository web service
         RepositoryServiceSoapBindingStub repositoryService;
         repositoryService = WebServiceFactory.getRepositoryService();
         // Get a reference to the space we have named
         Store storeRef = new Store(Constants.WORKSPACE_STORE, "SpacesStore");
         Node[] nodes = null;
         Reference reference = null;
         Predicate predicate = null;
         try {
            reference = new Reference(storeRef, null,
                  "/app:company_home/cm:" + spaceName);

            predicate = new Predicate(new Reference[] { reference }, null,
                  null);
            nodes = repositoryService.get(predicate);
         } catch (Exception ex) {
            System.out.println("Exception occured " + ex.getMessage());
            ex.printStackTrace();
         }
         // Searching for the UUID out of the result

         String spaceUUID = "";

         if (nodes[0] != null) {
            NamedValue[] properties = nodes[0].getProperties();
            for (int j = 0; j < properties.length; j++) {
               NamedValue nv = properties[j];
               if ("{http://www.alfresco.org/model/content/1.0}name"
                     .equals(nv.getName())
                     && nv.getValue().equals(spaceName)) {
                  System.out.println("In Space : " + nv.getValue());
               }
               if ("{http://www.alfresco.org/model/system/1.0}node-uuid"
                     .equals(nv.getName())) {
                  System.out.println("Space UUID: " + nv.getValue());
                  spaceUUID = nv.getValue();
               }
            }
         }

         // Create a reference to the parent where we want to insert content
         Reference referenceWrite = new Reference(storeRef, spaceUUID, null);

         ParentReference parentReference = new ParentReference(
               referenceWrite.getStore(), referenceWrite.getUuid(), null,
               Constants.ASSOC_CONTAINS, Constants.ASSOC_CONTAINS);
         // Create content
         String page_count_str = Integer.toString(pageCount);
         NamedValue[] newProperties = new NamedValue[] {
               Utils.createNamedValue(Constants.PROP_NAME, contentName),
               Utils.createNamedValue(Constants.PROP_DESCRIPTION,
                     description),
               Utils.createNamedValue(Constants.PROP_TITLE, title),
               Utils.createNamedValue(Constants.PROP_AUTHOR, author),
               Utils.createNamedValue(Constants.PROP_ISBN, isbn),
               Utils.createNamedValue(Constants.PROP_CLIENT_MAIL,
                     clientMail),
               Utils.createNamedValue(Constants.PROP_CLIENT_NAME,
                     clientName),
               Utils.createNamedValue(Constants.PROP_PAGE_COUNT,
                     page_count_str),
               Utils.createNamedValue(Constants.PROP_CLASSIFICATION,
                     classification),
         /* Utils.createNamedValue("IMAGE_ID", imageId), */
         };

         CMLCreate create = new CMLCreate("1", parentReference, null, null,
               null, Constants.TYPE_CONTENT, newProperties);

         CMLAddAspect addAspect = new CMLAddAspect();
         addAspect.setAspect(Constants.ASPECT_VERSIONABLE);
         addAspect.setProperty(newProperties);

         addAspect.setWhere(new Predicate(
               new Reference[] { referenceWrite }, null, null));
         addAspect.setWhere_id("1");

         CML cml = new CML();
         cml.setCreate(new CMLCreate[] { create });
         cml.setAddAspect(new CMLAddAspect[] { addAspect });

         // update
         UpdateResult[] results;
         results = WebServiceFactory.getRepositoryService().update(cml);
         // Set content
         ContentFormat format = new ContentFormat(mimeType.toString(),
               encoding);
         Content newContent;
         newContent = WebServiceFactory.getContentService().write(
               results[0].getDestination(), Constants.PROP_CONTENT,
               content, format);
         // System.out.println("UUID of uploaded content : " +
         // newContent.getNode().getUuid());
         System.out.println("File Uploaded.");

         // versioning(storeRef);

         return newContent.getNode().getUuid();

      } catch (Exception e) {
         // e.printStackTrace();
         return "Exception occured";
      } finally {
         // End the session
         AuthenticationUtils.endSession();
      }
}   
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍