Cannot upload a file larger than 4MB using CMIS

[ardamose123]

<strong>Alfresco versions:</strong> 4.2.d (community), 4.2.0 (enterprise).

<strong>Operating system:</strong> Ubuntu 12.04 to Ubuntu 13.10

<strong>Java major version:</strong> 7

I'm developing a web service that (among other things) uploads documents in a running Alfresco instance using Apache Chemistry. This is done in two parts: (1) the document is created along with its metadata with 0 bytes of content, then (2) the file contents are uploaded to the newly created document.

The file creation works well, but the content uploads do not work for files larger than 4MB. In this case, the

 /alfresco/service/script/org/alfresco/cmis/content.put‍

webscript complains about an non-existent file or directory (server side, just in case).

<blockcode>
2014-03-20 10:22:08,692  ERROR [extensions.webscripts.AbstractRuntime] [http-apr-9090-exec-10] Exception from executeScript - redirecting to status template error: java.io.IOException: No such file or directory
java.lang.RuntimeException: java.io.IOException: No such file or directory
    at org.alfresco.repo.web.scripts.RepositoryContainer$BufferedRequest$1.getInputStream(RepositoryContainer.java:998)
    at org.alfresco.repo.web.scripts.content.ContentSet.execute(ContentSet.java:175)
    at org.alfresco.repo.web.scripts.RepositoryContainer$3.execute(RepositoryContainer.java:446)
    at org.alfresco.repo.transaction.RetryingTransactionHelper.doInTransaction(RetryingTransactionHelper.java:447)
    at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecute(RepositoryContainer.java:508)
    at org.alfresco.repo.web.scripts.RepositoryContainer.transactionedExecuteAs(RepositoryContainer.java:546)
    at org.alfresco.repo.web.scripts.RepositoryContainer.executeScript(RepositoryContainer.java:358)
    at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:376)
    at org.springframework.extensions.webscripts.AbstractRuntime.executeScript(AbstractRuntime.java:209)
    at org.springframework.extensions.webscripts.servlet.WebScriptServlet.service(WebScriptServlet.java:132)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.alfresco.web.app.servlet.GlobalLocalizationFilter.doFilter(GlobalLocalizationFilter.java:61)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
    at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:1852)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: No such file or directory
    at java.io.UnixFileSystem.createFileExclusively(Native Method)
    at java.io.File.createTempFile(File.java:1879)
    at org.apache.chemistry.opencmis.server.shared.ThresholdOutputStream.expand(ThresholdOutputStream.java:142)
    at org.apache.chemistry.opencmis.server.shared.ThresholdOutputStream.write(ThresholdOutputStream.java:202)
    at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
    at org.alfresco.repo.web.scripts.RepositoryContainer$BufferedRequest.bufferInputStream(RepositoryContainer.java:871)
    at org.alfresco.repo.web.scripts.RepositoryContainer$BufferedRequest.access$800(RepositoryContainer.java:851)
    at org.alfresco.repo.web.scripts.RepositoryContainer$BufferedRequest$1.getInputStream(RepositoryContainer.java:994)
    … 29 more
</blockcode>

I investigated further about the

ThresholdOutputStream‍

class. It creates some temporary files in a given directory to do its job, if the file to be processed is over 4MB (by default) in size. Although I haven't found the exact place where it's attempting to create those files, I noticed that the

TempFileProvider‍

class is used to provide the directory. This directory is, in fact, $CATALINA_TEMP.

I created a simple Java-backed webscript that shows the permissions the

TempFileProvider‍

has over $CATALINA_TEMP. The script showed that all permissions (read, write, execute) were enabled. So, this is not a permissions problem.

I already tested the upload by changing the maxPostSize option for Tomcat, but to no avail.

I have a minimal test code (I think) that attempts to upload a local file to an Alfresco instance, in order to demonstrate this issue. However, no ZIP files can be uploaded here.

Is there any workaround to upload a file larger than 4MB to Alfresco? CMIS is preferred, but alternatives can be considered.

[kaynezhang]

I have many documents larger than 4M,they were all uploaded correctly using Apache Chemistry with AtomPub bingding.
There must be something wrong with your code or environment

[ardamose123]

Yes, there 'is' something wrong. The problem is 'what' is wrong. I've already discarded any proxy, permissions, and request size limits issues.

The exception is thrown from inside the mentioned Alfresco webscript. So, I'd like to know if there is something that webscript assumes about the environment, or if there's some other way to upload the contents of a document, even if it's not through CMIS, that bypasses that webscript.

Concerning the code, this is how the file is uploaded using CMIS:
<java>
contentStream = session.getObjectFactory().createContentStream(name, size, mime, newContent);
document.setContentStream(contentStream, true, true);
</java>

Concerning the environment, it's Ubuntu (from 12.04 to 13.10), with Alfresco installed from its installer into /opt using sudo. It's started using

sudo service alfresco start‍

. Nothing out of the ordinary.

[kaynezhang]

Following test code works fine in my environment,you can try it.

      String serverUrl = "http://localhost:8080/alfresco/api/-default-/public/cmis/versions/1.1/atom";
      String userName = "admin";
      String password = "admin";
      
      SessionFactory sessionFactory = SessionFactoryImpl.newInstance();
      Map<String, String> params = new HashMap<String, String>();
      params.put(SessionParameter.USER, userName);
      params.put(SessionParameter.PASSWORD, password);
      params.put(SessionParameter.ATOMPUB_URL, serverUrl);
      params.put(SessionParameter.BINDING_TYPE, BindingType.ATOMPUB.value());
      params.put(SessionParameter.OBJECT_FACTORY_CLASS,
            "org.alfresco.cmis.client.impl.AlfrescoObjectFactoryImpl");
      List<Repository> repos = sessionFactory.getRepositories(params);
      if (repos.isEmpty()) {
         throw new RuntimeException("Server has no repositories!");
      }
      Session session = repos.get(0).createSession();

      Map<String, Object> properties = new HashMap<String, Object>();
      properties.put(PropertyIds.OBJECT_TYPE_ID, "cmis:document,P:cm:titled");
      properties.put(PropertyIds.NAME, "test8");
      properties.put(PropertyIds.CREATED_BY, "admin");
      properties.put("cm:title", "Title8");
      properties.put("cm:description", "description8");


         AlfrescoFolder folder1 = (AlfrescoFolder) session.getObjectByPath("/folder1");
         File file = new File("C:\\filetoupload.pdf");
         InputStream fis = new FileInputStream(file);
         VersioningState vs = VersioningState.MAJOR;
         DataInputStream dis = new DataInputStream(fis);
         byte[] bytes = new byte[(int) file.length()];
         dis.readFully(bytes);
         ContentStream contentStream = new ContentStreamImpl(file
               .getAbsolutePath(), null, "application/pdf",
               new ByteArrayInputStream(bytes));
         
         AlfrescoDocument newDocument = (AlfrescoDocument) folder1
               .createDocument(properties, contentStream, vs);

‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

[ardamose123]

Thanks! I tried to use your code, but I cannot install the alfresco-opencmis-extension from Maven repository (there's something about the stax-ex artifact).

I tried with a local Alfresco Community 4.2.c installation and my code works fine there. Maybe it's something about 4.2.d. I'll report more findings on this issue.

[ardamose123]

I tested my content upload code in fresh installations of the following Alfresco versions:
<ul>
<li>Community 4.2.c: it works</li>
<li>Community 4.2.d: it doesn't work</li>
<li>Community 4.2.f: it doesn't work</li>
</ul>

It makes me believe something changed between those versions about the way the content.put webscript works between 4.2.c and 4.2.d.

[marco_pancioni]

Hello Ariel,

I've found the same problem on 4.2.f Community Edition when i try to upload via webscript a file larger than 4 MB.

After some hours of search i've found that Alfresco use a temp directory for manage file larger than a threshold (default value is 4 MB), in standard installation the Temp Directory is <strong>'/opt/alfresco-4.2.f/tomcat/temp/Alfresco'</strong>. In this directory the only one user that have the write permissions is the creator. You can try to change this permission to allow some other users to write inside.

In my case this change was not enough to solve the problem because after other inspection of Alfresco Source Code i've discover that for WebScripts ( and i think also for CMIS because the class used is the same), Alfresco use a subdirectory inside standard Temp Dir for store Temp File, this subdirectory is named <strong>"WebScripts"</strong>.
I've created manually this subdirectory and than I've solved my problem.

In the file repository.properties (under WEB-INF/classes/) you can change three properties:

webscripts.tempDirectoryName=WebScripts
# 4mb
webscripts.memoryThreshold=4194304
# 4gb
webscripts.setMaxContentSize=4294967296
‍‍‍‍‍‍‍

The first property <strong>webscripts.tempDirectoryName</strong> specify the name of subdirectory inside default Temp Dir where Webscript and CMIS will create the temp file when necessary.

The second property <strong>webscripts.memoryThreshold</strong> specify the threshold value after that Webscript and CMIS use a Temp File to store the content upload instead use of memory buffer.

The third property <strong>webscripts.setMaxContentSize</strong> specify the maximum size allowed in Alfresco Webscript to manage content file.

I hope this is usefull to solve your problem.

[kaynezhang]

You can try with Community 4.3.a to test if it works.You can download 4.3 from this address

http://dev.alfresco.com/downloads/nightly/dist/