Batch uploads fail in nuxeo 8.1 - how can I fix it?

[ssze_]

I'm trying to upload some videos via the import dialogue. In this case it were 4 videos (each of them 500 Mb). 3 uploads are running in parallel via the webinterface in Chrome (48.0). But after clicking the import button and waiting some time the following error notification appears "A problem occurred during import."

2016-03-01 17:15:29,156 ERROR [http-bio-0.0.0.0-8080-exec-35] [org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent] Error while executing automation batch
org.nuxeo.ecm.automation.TraceException:

****** chain ******
Name: FileManager.ImportWithMetaDataInSeam
Exception: OperationException
Caught error: Failed to invoke operation FileManager.Import
Caused by: org.nuxeo.ecm.core.api.PropertyException: Cannot get blob info for: org.nuxeo.ecm.core.api.impl.blob.FileBlob@c4f28ed2
****** Hierarchy calls ******
        org.nuxeo.ecm.automation.core.operations.services.FileManagerImport

        at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:234)
        at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:117)
        at org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent.execute(BatchManagerComponent.java:285)
        at org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent.execute(BatchManagerComponent.java:251)
        at org.nuxeo.ecm.webapp.action.ImportActions.importDocumentsThroughBatchManager(ImportActions.java:295)
        at org.nuxeo.ecm.webapp.action.ImportActions.importDocuments(ImportActions.java:247)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
        at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
        at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.nuxeo.ecm.platform.ui.web.util.NuxeoExceptionInterceptor.aroundInvoke(NuxeoExceptionInterceptor.java:80)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
        at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196)
        at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114)
        at org.nuxeo.ecm.webapp.action.ImportActions_$$_javassist_seam_23.importDocuments(ImportActions_$$_javassist_seam_23.java)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:335)
        at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:280)
        at org.jboss.el.parser.AstMethodSuffix.getValue(AstMethodSuffix.java:59)
        at org.jboss.el.parser.AstMethodSuffix.invoke(AstMethodSuffix.java:65)
        at org.jboss.el.parser.AstValue.invoke(AstValue.java:96)
        at org.jboss.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)
        at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
        at org.nuxeo.ecm.platform.ui.web.binding.MetaMethodExpression.invoke(MetaMethodExpression.java:106)
        at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:87)
        at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
        at javax.faces.component.UICommand.broadcast(UICommand.java:315)
        at org.nuxeo.ecm.platform.ui.web.binding.alias.UIAliasHolder.broadcast(UIAliasHolder.java:113)
        at org.nuxeo.ecm.platform.ui.web.component.holder.UIValueHolder.broadcast(UIValueHolder.java:95)
        at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:790)
        at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1282)
        at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81)
        at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
        at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
        at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.nuxeo.elasticsearch.web.sync.UIThreadMarker.doFilter(UIThreadMarker.java:48)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoThreadTrackerFilter.doFilter(NuxeoThreadTrackerFilter.java:43)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.core.management.jtajca.internal.Log4jWebFilter.doFilter(Log4jWebFilter.java:69)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.ui.web.rest.FancyURLFilter.doFilter(FancyURLFilter.java:120)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.mobile.filter.ApplicationRedirectionFilter.doNoRedirect(ApplicationRedirectionFilter.java:165)
        at org.nuxeo.ecm.mobile.filter.ApplicationRedirectionFilter.doFilter(ApplicationRedirectionFilter.java:95)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoRequestControllerFilter.doFilter(NuxeoRequestControllerFilter.java:142)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.webdav.service.WIRequestFilter.doFilter(WIRequestFilter.java:61)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:586)
        at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:51)
        at org.nuxeo.ecm.platform.ui.web.auth.oauth.NuxeoOAuthFilter.doFilter(NuxeoOAuthFilter.java:122)
        at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:49)
        at org.nuxeo.ecm.platform.ui.web.auth.oauth2.NuxeoOAuth2Filter.doFilter(NuxeoOAuth2Filter.java:83)
        at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:49)
        at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilter(NuxeoAuthenticationFilter.java:399)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:208)
        at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:274)
        at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoCorsFilter.doFilter(NuxeoCorsFilter.java:49)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:77)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:75)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.nuxeo.ecm.automation.OperationException: Failed to invoke operation FileManager.Import
        at org.nuxeo.ecm.automation.core.impl.InvokableMethod.invoke(InvokableMethod.java:189)
        at org.nuxeo.ecm.automation.core.impl.CompiledChainImpl.doInvoke(CompiledChainImpl.java:130)
        at org.nuxeo.ecm.automation.core.impl.CompiledChainImpl.invoke(CompiledChainImpl.java:116)
        at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:210)
        ... 115 more
Caused by: org.nuxeo.ecm.core.api.PropertyException: Cannot get blob info for: org.nuxeo.ecm.core.api.impl.blob.FileBlob@c4f28ed2
        at org.nuxeo.ecm.core.storage.BaseDocument.setValueBlob(BaseDocument.java:626)
        at org.nuxeo.ecm.core.storage.BaseDocument$BlobWriteContext.flush(BaseDocument.java:729)
        at org.nuxeo.ecm.core.api.DocumentModelFactory.writeDocumentModel(DocumentModelFactory.java:247)
        at org.nuxeo.ecm.core.api.AbstractSession.writeModel(AbstractSession.java:366)
        at org.nuxeo.ecm.core.api.AbstractSession.createDocument(AbstractSession.java:694)
        at org.nuxeo.ecm.platform.filemanager.service.extension.AbstractFileImporter.create(AbstractFileImporter.java:236)
        at org.nuxeo.ecm.platform.filemanager.service.FileManagerService.createDocumentFromBlob(FileManagerService.java:225)
        at org.nuxeo.ecm.automation.core.operations.services.FileManagerImport.run(FileManagerImport.java:73)
        at org.nuxeo.ecm.automation.core.operations.services.FileManagerImport.run(FileManagerImport.java:81)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.nuxeo.ecm.automation.core.impl.InvokableMethod.doInvoke(InvokableMethod.java:171)
        at org.nuxeo.ecm.automation.core.impl.InvokableMethod.invoke(InvokableMethod.java:177)
        ... 118 more
Caused by: java.io.FileNotFoundException: /mnt/data/DAM-Data01/nuxeo/nxserver/data/transientstores/default/YmF0Y2hJZC1lNzc2OTMzZi04MzYwLTRjZTYtYjYxMi0zYmFhYjUyYzFmZWZfMg==/005c707b-b2bb-4e46-adeb-1194b3250998 (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.nuxeo.ecm.core.api.impl.blob.FileBlob.getStream(FileBlob.java:169)
        at org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager.getBinary(AbstractBinaryManager.java:96)
        at org.nuxeo.ecm.core.blob.binary.DefaultBinaryManager.getBinary(DefaultBinaryManager.java:50)
        at org.nuxeo.ecm.core.blob.binary.BinaryBlobProvider.writeBlob(BinaryBlobProvider.java:97)
        at org.nuxeo.ecm.core.blob.BlobManagerComponent.writeBlob(BlobManagerComponent.java:281)
        ... 132 more

So it seems that the files get deleted after the upload. Single file uploads work though. Is there some setting to control the cleanup of the tmp folder? Or any other hint on tracing the cause of this issue are highly appreciated.

edit: Clustering is not enabled

##-----------------------------------------------------------------------------
## Clustering settings
##-----------------------------------------------------------------------------
#repository.clustering.enabled=false
#repository.clustering.delay=1000
#repository.binary.store=/path/to/some/shared/folder/for/binaries

Thanks. Stefan

[ataillefer_]

Hi,

Is your Nuxeo configured in cluster?

If this is the case you need to follow the instructions given on this page: https://doc.nuxeo.com/display/ADMINDOC/Nuxeo+Clustering+Configuration#NuxeoClusteringConfiguration-T...

[ssze_]

No, we don't use the clustering. I left the default clustering settings commented out in the nuxeo.conf file.

[ataillefer_]

After the drag and drop of the 4 video files, for how long approximately are you waiting before you get the error?

[ssze_]

After clicking the import button nuxeo is working for up to 10 minutes until the import screen goes away. Then the message is displayed.

[ssze_]

I tried uploading via curl and API as described in https

[Florent_Guillau]

This fails after how long? Are there errors server-side?

[ssze_]

Hi, after reviewing the upload issues we see 4 different scenarios with cap 8.1

[ssze_]

Please see also the detailed curl output for the difference in the successful small and big video uploads. The strange thing is the 404 not found on the big file upload.

[ssze_]

Uploads in the UI still fail with "Processing of multipart/form-data request failed. Stream ended unexpectedly" errors.