Batch uploads fail in nuxeo 8.1 - how can I fix it?

[ssze_]

I'm trying to upload some videos via the import dialogue. In this case it were 4 videos (each of them 500 Mb). 3 uploads are running in parallel via the webinterface in Chrome (48.0). But after clicking the import button and waiting some time the following error notification appears "A problem occurred during import."

2016-03-01 17:15:29,156 ERROR [http-bio-0.0.0.0-8080-exec-35] [org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent] Error while executing automation batch
org.nuxeo.ecm.automation.TraceException:

****** chain ******
Name: FileManager.ImportWithMetaDataInSeam
Exception: OperationException
Caught error: Failed to invoke operation FileManager.Import
Caused by: org.nuxeo.ecm.core.api.PropertyException: Cannot get blob info for: org.nuxeo.ecm.core.api.impl.blob.FileBlob@c4f28ed2
****** Hierarchy calls ******
        org.nuxeo.ecm.automation.core.operations.services.FileManagerImport

        at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:234)
        at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:117)
        at org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent.execute(BatchManagerComponent.java:285)
        at org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent.execute(BatchManagerComponent.java:251)
        at org.nuxeo.ecm.webapp.action.ImportActions.importDocumentsThroughBatchManager(ImportActions.java:295)
        at org.nuxeo.ecm.webapp.action.ImportActions.importDocuments(ImportActions.java:247)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
        at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
        at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.nuxeo.ecm.platform.ui.web.util.NuxeoExceptionInterceptor.aroundInvoke(NuxeoExceptionInterceptor.java:80)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
        at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196)
        at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114)
        at org.nuxeo.ecm.webapp.action.ImportActions_$$_javassist_seam_23.importDocuments(ImportActions_$$_javassist_seam_23.java)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:335)
        at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:280)
        at org.jboss.el.parser.AstMethodSuffix.getValue(AstMethodSuffix.java:59)
        at org.jboss.el.parser.AstMethodSuffix.invoke(AstMethodSuffix.java:65)
        at org.jboss.el.parser.AstValue.invoke(AstValue.java:96)
        at org.jboss.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)
        at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
        at org.nuxeo.ecm.platform.ui.web.binding.MetaMethodExpression.invoke(MetaMethodExpression.java:106)
        at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:87)
        at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
        at javax.faces.component.UICommand.broadcast(UICommand.java:315)
        at org.nuxeo.ecm.platform.ui.web.binding.alias.UIAliasHolder.broadcast(UIAliasHolder.java:113)
        at org.nuxeo.ecm.platform.ui.web.component.holder.UIValueHolder.broadcast(UIValueHolder.java:95)
        at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:790)
        at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1282)
        at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81)
        at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
        at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
        at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.nuxeo.elasticsearch.web.sync.UIThreadMarker.doFilter(UIThreadMarker.java:48)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoThreadTrackerFilter.doFilter(NuxeoThreadTrackerFilter.java:43)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.core.management.jtajca.internal.Log4jWebFilter.doFilter(Log4jWebFilter.java:69)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.ui.web.rest.FancyURLFilter.doFilter(FancyURLFilter.java:120)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.mobile.filter.ApplicationRedirectionFilter.doNoRedirect(ApplicationRedirectionFilter.java:165)
        at org.nuxeo.ecm.mobile.filter.ApplicationRedirectionFilter.doFilter(ApplicationRedirectionFilter.java:95)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoRequestControllerFilter.doFilter(NuxeoRequestControllerFilter.java:142)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.webdav.service.WIRequestFilter.doFilter(WIRequestFilter.java:61)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:586)
        at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:51)
        at org.nuxeo.ecm.platform.ui.web.auth.oauth.NuxeoOAuthFilter.doFilter(NuxeoOAuthFilter.java:122)
        at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:49)
        at org.nuxeo.ecm.platform.ui.web.auth.oauth2.NuxeoOAuth2Filter.doFilter(NuxeoOAuth2Filter.java:83)
        at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:49)
        at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilter(NuxeoAuthenticationFilter.java:399)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:208)
        at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:274)
        at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoCorsFilter.doFilter(NuxeoCorsFilter.java:49)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:77)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:75)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.nuxeo.ecm.automation.OperationException: Failed to invoke operation FileManager.Import
        at org.nuxeo.ecm.automation.core.impl.InvokableMethod.invoke(InvokableMethod.java:189)
        at org.nuxeo.ecm.automation.core.impl.CompiledChainImpl.doInvoke(CompiledChainImpl.java:130)
        at org.nuxeo.ecm.automation.core.impl.CompiledChainImpl.invoke(CompiledChainImpl.java:116)
        at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:210)
        ... 115 more
Caused by: org.nuxeo.ecm.core.api.PropertyException: Cannot get blob info for: org.nuxeo.ecm.core.api.impl.blob.FileBlob@c4f28ed2
        at org.nuxeo.ecm.core.storage.BaseDocument.setValueBlob(BaseDocument.java:626)
        at org.nuxeo.ecm.core.storage.BaseDocument$BlobWriteContext.flush(BaseDocument.java:729)
        at org.nuxeo.ecm.core.api.DocumentModelFactory.writeDocumentModel(DocumentModelFactory.java:247)
        at org.nuxeo.ecm.core.api.AbstractSession.writeModel(AbstractSession.java:366)
        at org.nuxeo.ecm.core.api.AbstractSession.createDocument(AbstractSession.java:694)
        at org.nuxeo.ecm.platform.filemanager.service.extension.AbstractFileImporter.create(AbstractFileImporter.java:236)
        at org.nuxeo.ecm.platform.filemanager.service.FileManagerService.createDocumentFromBlob(FileManagerService.java:225)
        at org.nuxeo.ecm.automation.core.operations.services.FileManagerImport.run(FileManagerImport.java:73)
        at org.nuxeo.ecm.automation.core.operations.services.FileManagerImport.run(FileManagerImport.java:81)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.nuxeo.ecm.automation.core.impl.InvokableMethod.doInvoke(InvokableMethod.java:171)
        at org.nuxeo.ecm.automation.core.impl.InvokableMethod.invoke(InvokableMethod.java:177)
        ... 118 more
Caused by: java.io.FileNotFoundException: /mnt/data/DAM-Data01/nuxeo/nxserver/data/transientstores/default/YmF0Y2hJZC1lNzc2OTMzZi04MzYwLTRjZTYtYjYxMi0zYmFhYjUyYzFmZWZfMg==/005c707b-b2bb-4e46-adeb-1194b3250998 (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.nuxeo.ecm.core.api.impl.blob.FileBlob.getStream(FileBlob.java:169)
        at org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager.getBinary(AbstractBinaryManager.java:96)
        at org.nuxeo.ecm.core.blob.binary.DefaultBinaryManager.getBinary(DefaultBinaryManager.java:50)
        at org.nuxeo.ecm.core.blob.binary.BinaryBlobProvider.writeBlob(BinaryBlobProvider.java:97)
        at org.nuxeo.ecm.core.blob.BlobManagerComponent.writeBlob(BlobManagerComponent.java:281)
        ... 132 more

So it seems that the files get deleted after the upload. Single file uploads work though. Is there some setting to control the cleanup of the tmp folder? Or any other hint on tracing the cause of this issue are highly appreciated.

edit: Clustering is not enabled

##-----------------------------------------------------------------------------
## Clustering settings
##-----------------------------------------------------------------------------
#repository.clustering.enabled=false
#repository.clustering.delay=1000
#repository.binary.store=/path/to/some/shared/folder/for/binaries

Thanks. Stefan

[ataillefer_]

Hi, thanks for all these details.

[ssze_]

Thanks for the curl check.

[ataillefer_]

Increasing nuxeo.db.transactiontimeout should definitely work for the New Video case (stack clearly shows "Unable to commit