out of heap space when uploading large files

[alanbell]

I am using the python-cmis library to upload files to Alfresco community 3.4.b running on Ubuntu server.
where /mnt/import/bigfile is a 298MB file. The file can be imported successfully using the share interface or classic Alfresco web UI and small files of a few MB in size are uploaded by this code successfully.

the python code is as follows:

from cmislib.model import CmisClient
cmisClient = CmisClient('http://localhost:8080/alfresco/s/cmis', 'importuser','password')
repo = cmisClient.defaultRepository
ImportFolder = repo.getObjectByPath("/import")
instream=open("/mnt/import/bigfile","r")
d=ImportFolder.createDocument("bigfile",contentFile=instream)
‍‍‍‍‍‍‍‍

and it falls over leaving this error:

Traceback (most recent call last):
  File "mapimport.py", line 12, in <module>
    d=ImportFolder.createDocument("bigfile",contentFile=instream)
  File "/usr/local/lib/python2.6/dist-packages/cmislib-0.3-py2.6.egg/cmislib/model.py", line 2700, in createDocument
    result = self._cmisClient.post(postUrl, xmlDoc.toxml(), ATOM_XML_ENTRY_TYPE)
  File "/usr/local/lib/python2.6/dist-packages/cmislib-0.3-py2.6.egg/cmislib/model.py", line 243, in post
    self._processCommonErrors(result)
  File "/usr/local/lib/python2.6/dist-packages/cmislib-0.3-py2.6.egg/cmislib/model.py", line 296, in _processCommonErrors
    raise RuntimeException(error.status)
cmislib.exceptions.RuntimeException: 500‍‍‍‍‍‍‍‍‍‍

and in catalina.out (first error from a clean restart of Alfresco)

INFO: Server startup in 76334 ms
23:38:52,675  ERROR [extensions.webscripts.AbstractRuntime] Exception from executeScript - redirecting to status template error: 11230001 Wrapped Exception (with status template): 11230016 Failed to execute script 
'classpath*:alfresco/templates/webscripts/org/alfresco/cmis/children.post.atom.js': 11230015 Java heap space
org.springframework.extensions.webscripts.WebScriptException: 11230001 Wrapped Exception (with status template): 11230016 Failed to execute script 'classpath*:alfresco/templates/webscripts/org/alfresco/cmis/childre
n.post.atom.js': 11230015 Java heap space
        at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:749)
        at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:171)
        at org.alfresco.repo.web.scripts.RepositoryContainer$2.execute(RepositoryContainer.java:383)
        at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:377)
        at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:436)
        at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:466)
        at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:304)
        at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:333)
        at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:189)
        at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:58)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:619)
Caused by: org.alfresco.scripts.ScriptException: 11230016 Failed to execute script 'classpath*:alfresco/templates/webscripts/org/alfresco/cmis/children.post.atom.js': 11230015 Java heap space
        at org.alfresco.repo.jscript.RhinoScriptProcessor.execute(RhinoScriptProcessor.java:194)
        at org.alfresco.repo.processor.ScriptServiceImpl.executeScript(ScriptServiceImpl.java:282)
        at org.alfresco.repo.web.scripts.RepositoryScriptProcessor.executeScript(RepositoryScriptProcessor.java:102)
        at org.springframework.extensions.webscripts.AbstractWebScript.executeScript(AbstractWebScript.java:982)
        at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:86)
        … 24 more

Caused by: org.alfresco.error.AlfrescoRuntimeException: 11230015 Java heap space
        at org.alfresco.repo.jscript.RhinoScriptProcessor.executeScriptImpl(RhinoScriptProcessor.java:488)
        at org.alfresco.repo.jscript.RhinoScriptProcessor.execute(RhinoScriptProcessor.java:190)
        … 28 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:3209)
        at java.lang.String.<init>(String.java:215)
        at java.lang.StringBuffer.toString(StringBuffer.java:585)
        at com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:403)
        at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821)
        at org.apache.axiom.om.impl.builder.StAXBuilder.createOMText(StAXBuilder.java:245)
        at org.apache.axiom.om.impl.builder.StAXBuilder.createOMText(StAXBuilder.java:216)
        at org.apache.abdera.parser.stax.FOMBuilder.applyTextFilter(FOMBuilder.java:158)
        at org.apache.abdera.parser.stax.FOMBuilder.next(FOMBuilder.java:206)
        at org.apache.axiom.om.impl.llom.OMElementImpl.getNextOMSibling(OMElementImpl.java:265)
        at org.apache.axiom.om.impl.traverse.OMChildrenQNameIterator.hasNext(OMChildrenQNameIterator.java:69)
        at org.apache.axiom.om.impl.llom.OMElementImpl.getFirstChildWithName(OMElementImpl.java:223)
        at org.apache.abdera.parser.stax.FOMExtensibleElement.getExtension(FOMExtensibleElement.java:91)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
        at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:243)
        at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:66)

        at org.mozilla.javascript.gen.c9._c0(file:/opt/alfresco-3.4.b/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/cmis/children.post.atom.js:541)
        at org.mozilla.javascript.gen.c9.call(file:/opt/alfresco-3.4.b/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/cmis/children.post.atom.js)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2834)
        at org.mozilla.javascript.gen.c9.call(file:/opt/alfresco-3.4.b/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/cmis/children.post.atom.js)
        at org.mozilla.javascript.gen.c9.exec(file:/opt/alfresco-3.4.b/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/templates/webscripts/org/alfresco/cmis/children.post.atom.js)
        at org.alfresco.repo.jscript.RhinoScriptProcessor.executeScriptImpl(RhinoScriptProcessor.java:472)
        at org.alfresco.repo.jscript.RhinoScriptProcessor.execute(RhinoScriptProcessor.java:190)
        at org.alfresco.repo.processor.ScriptServiceImpl.executeScript(ScriptServiceImpl.java:282)
        at org.alfresco.repo.web.scripts.RepositoryScriptProcessor.executeScript(RepositoryScriptProcessor.java:102)
        at org.springframework.extensions.webscripts.AbstractWebScript.executeScript(AbstractWebScript.java:982)
        at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:86)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

so the python code gets an error 500 from Alfresco CMIS, I don't think there is anything wrong on the python side as such.
I tried editing /opt/alfresco-3.4.b/tomcat/scripts/ctl.sh to give java more heap size

    export JAVA_OPTS="-XX:MaxPermSize=512m -Xms2G -Xmx3G -Dalfresco.home=/opt/alfresco-3.4.b -Dcom.sun.management.jmxremote"‍

but that made no visible difference to the outcome. The server has 8GB of ram assigned to it and is virtualised.

Can anyone else reproduce this, or successfully upload large files to CMIS?
Is there something I can do to increase the heap size or tune things so that it works?
Is there another way I should be doing this?

[christophe]

Hello

I am having a similar issue while upload a 100Mb file with Apache Chemistry CMIS library.
I am using the AtomPub protocol.

Here is the stack trace client side :

Exception in thread "main" org.apache.chemistry.opencmis.commons.exceptions.CmisRuntimeException: Erreur Interne de Servlet
   at org.apache.chemistry.opencmis.client.bindings.spi.atompub.AbstractAtomPubService.convertStatusCode(AbstractAtomPubService.java:450)
   at org.apache.chemistry.opencmis.client.bindings.spi.atompub.AbstractAtomPubService.post(AbstractAtomPubService.java:568)
   at org.apache.chemistry.opencmis.client.bindings.spi.atompub.ObjectServiceImpl.createDocument(ObjectServiceImpl.java:109)
   at org.apache.chemistry.opencmis.client.runtime.SessionImpl.createDocument(SessionImpl.java:650)
   at org.apache.chemistry.opencmis.client.runtime.FolderImpl.createDocument(FolderImpl.java:87)
   at org.apache.chemistry.opencmis.client.runtime.FolderImpl.createDocument(FolderImpl.java:409)
   at com.mycompany.alfresco.cmis.main.create.CreatePolar.doit(CreatePolar.java:54)
   at com.mycompany.alfresco.cmis.main.AbstractAction.run(AbstractAction.java:12)
   at java.lang.Thread.run(Thread.java:662)
   at com.mycompany.alfresco.cmis.main.SimpleMain.main(SimpleMain.java:24)‍‍‍‍‍‍‍‍‍‍‍

server side :

11:20:57,216  ERROR [extensions.webscripts.AbstractRuntime] Exception from executeScript - redirecting to status template error: 07300001 Wrapped Exception (with status template): 
07302060 Failed to execute script 'classpath*:alfresco/templates/webscripts/org/alfresco/cmis/children.post.atom.js': 
07302059 Java heap space org.springframework.extensions.webscripts.WebScriptException: 
07300001 Wrapped Exception (with status template): 07302060 Failed to execute script 'classpath*:alfresco/templates/webscripts/org/alfresco/cmis/children.post.atom.js': 
07302059 Java heap space
        at org.springframework.extensions.webscripts.AbstractWebScript.createStatusException(AbstractWebScript.java:758)
        at org.springframework.extensions.webscripts.DeclarativeWebScript.execute(DeclarativeWebScript.java:171)
        at org.alfresco.repo.web.scripts.RepositoryContainer$2.execute(RepositoryContainer.java:383)
        at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:381)
        at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:436)
        at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:466)
        at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:304)
        at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:352)
        at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:189)
        at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:118)‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

the root cause is :

Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:3209)
        at java.lang.String.<init>(String.java:215)
        at java.lang.StringBuffer.toString(StringBuffer.java:585)
        at com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:403)
        at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821)
        at org.apache.axiom.om.impl.builder.StAXBuilder.createOMText(StAXBuilder.java:245)
        at org.apache.axiom.om.impl.builder.StAXBuilder.createOMText(StAXBuilder.java:216)
        at org.apache.abdera.parser.stax.FOMBuilder.applyTextFilter(FOMBuilder.java:158)
        at org.apache.abdera.parser.stax.FOMBuilder.next(FOMBuilder.java:206)
        at org.apache.axiom.om.impl.llom.OMElementImpl.getNextOMSibling(OMElementImpl.java:265)
        at org.apache.axiom.om.impl.traverse.OMChildrenQNameIterator.hasNext(OMChildrenQNameIterator.java:69)
        at org.apache.axiom.om.impl.llom.OMElementImpl.getFirstChildWithName(OMElementImpl.java:223)
        at org.apache.abdera.parser.stax.FOMExtensibleElement.getExtension(FOMExtensibleElement.java:91)
        at sun.reflect.GeneratedMethodAccessor979.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:155)
        at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:243)
        at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:66)
‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

It looks like Alfresco tries to put the whole content in memory and it leads to a OutOfMemoryError.

The server is running with +/- 980Mb of memory

When I am using the SOAP protocol, everything just works file (I am able to upload really large files (1Gb+).

I would like to use the AtomPub protocol as it is less verbose, but this issue is blocking as the application will be uploading large files.

Any though on that ?

[victorba]

Hello Christophe,

did you find a solution to upload large files using Atompub protocol? I have the same problem using Alfresco Community 4.2