Batch uploads fail in nuxeo 8.1 - how can I fix it?

ssze_ — Tue, 01 Mar 2016 16:30:54 GMT

I'm trying to upload some videos via the import dialogue. In this case it were 4 videos (each of them 500 Mb). 3 uploads are running in parallel via the webinterface in Chrome (48.0). But after clicking the import button and waiting some time the following error notification appears "A problem occurred during import."

2016-03-01 17:15:29,156 ERROR [http-bio-0.0.0.0-8080-exec-35] [org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent] Error while executing automation batch
org.nuxeo.ecm.automation.TraceException:

****** chain ******
Name: FileManager.ImportWithMetaDataInSeam
Exception: OperationException
Caught error: Failed to invoke operation FileManager.Import
Caused by: org.nuxeo.ecm.core.api.PropertyException: Cannot get blob info for: org.nuxeo.ecm.core.api.impl.blob.FileBlob@c4f28ed2
****** Hierarchy calls ******
        org.nuxeo.ecm.automation.core.operations.services.FileManagerImport

        at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:234)
        at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:117)
        at org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent.execute(BatchManagerComponent.java:285)
        at org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent.execute(BatchManagerComponent.java:251)
        at org.nuxeo.ecm.webapp.action.ImportActions.importDocumentsThroughBatchManager(ImportActions.java:295)
        at org.nuxeo.ecm.webapp.action.ImportActions.importDocuments(ImportActions.java:247)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.jboss.seam.util.Reflections.invoke(Reflections.java:22)
        at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56)
        at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.nuxeo.ecm.platform.ui.web.util.NuxeoExceptionInterceptor.aroundInvoke(NuxeoExceptionInterceptor.java:80)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44)
        at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68)
        at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107)
        at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196)
        at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114)
        at org.nuxeo.ecm.webapp.action.ImportActions_$$_javassist_seam_23.importDocuments(ImportActions_$$_javassist_seam_23.java)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:335)
        at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:280)
        at org.jboss.el.parser.AstMethodSuffix.getValue(AstMethodSuffix.java:59)
        at org.jboss.el.parser.AstMethodSuffix.invoke(AstMethodSuffix.java:65)
        at org.jboss.el.parser.AstValue.invoke(AstValue.java:96)
        at org.jboss.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276)
        at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105)
        at org.nuxeo.ecm.platform.ui.web.binding.MetaMethodExpression.invoke(MetaMethodExpression.java:106)
        at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:87)
        at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102)
        at javax.faces.component.UICommand.broadcast(UICommand.java:315)
        at org.nuxeo.ecm.platform.ui.web.binding.alias.UIAliasHolder.broadcast(UIAliasHolder.java:113)
        at org.nuxeo.ecm.platform.ui.web.component.holder.UIValueHolder.broadcast(UIValueHolder.java:95)
        at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:790)
        at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1282)
        at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81)
        at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
        at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198)
        at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
        at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.nuxeo.elasticsearch.web.sync.UIThreadMarker.doFilter(UIThreadMarker.java:48)
        at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
        at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoThreadTrackerFilter.doFilter(NuxeoThreadTrackerFilter.java:43)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.core.management.jtajca.internal.Log4jWebFilter.doFilter(Log4jWebFilter.java:69)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.ui.web.rest.FancyURLFilter.doFilter(FancyURLFilter.java:120)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.mobile.filter.ApplicationRedirectionFilter.doNoRedirect(ApplicationRedirectionFilter.java:165)
        at org.nuxeo.ecm.mobile.filter.ApplicationRedirectionFilter.doFilter(ApplicationRedirectionFilter.java:95)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoRequestControllerFilter.doFilter(NuxeoRequestControllerFilter.java:142)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.webdav.service.WIRequestFilter.doFilter(WIRequestFilter.java:61)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:586)
        at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:51)
        at org.nuxeo.ecm.platform.ui.web.auth.oauth.NuxeoOAuthFilter.doFilter(NuxeoOAuthFilter.java:122)
        at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:49)
        at org.nuxeo.ecm.platform.ui.web.auth.oauth2.NuxeoOAuth2Filter.doFilter(NuxeoOAuth2Filter.java:83)
        at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:49)
        at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilter(NuxeoAuthenticationFilter.java:399)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:208)
        at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:274)
        at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoCorsFilter.doFilter(NuxeoCorsFilter.java:49)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:77)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:75)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1079)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.nuxeo.ecm.automation.OperationException: Failed to invoke operation FileManager.Import
        at org.nuxeo.ecm.automation.core.impl.InvokableMethod.invoke(InvokableMethod.java:189)
        at org.nuxeo.ecm.automation.core.impl.CompiledChainImpl.doInvoke(CompiledChainImpl.java:130)
        at org.nuxeo.ecm.automation.core.impl.CompiledChainImpl.invoke(CompiledChainImpl.java:116)
        at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:210)
        ... 115 more
Caused by: org.nuxeo.ecm.core.api.PropertyException: Cannot get blob info for: org.nuxeo.ecm.core.api.impl.blob.FileBlob@c4f28ed2
        at org.nuxeo.ecm.core.storage.BaseDocument.setValueBlob(BaseDocument.java:626)
        at org.nuxeo.ecm.core.storage.BaseDocument$BlobWriteContext.flush(BaseDocument.java:729)
        at org.nuxeo.ecm.core.api.DocumentModelFactory.writeDocumentModel(DocumentModelFactory.java:247)
        at org.nuxeo.ecm.core.api.AbstractSession.writeModel(AbstractSession.java:366)
        at org.nuxeo.ecm.core.api.AbstractSession.createDocument(AbstractSession.java:694)
        at org.nuxeo.ecm.platform.filemanager.service.extension.AbstractFileImporter.create(AbstractFileImporter.java:236)
        at org.nuxeo.ecm.platform.filemanager.service.FileManagerService.createDocumentFromBlob(FileManagerService.java:225)
        at org.nuxeo.ecm.automation.core.operations.services.FileManagerImport.run(FileManagerImport.java:73)
        at org.nuxeo.ecm.automation.core.operations.services.FileManagerImport.run(FileManagerImport.java:81)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.nuxeo.ecm.automation.core.impl.InvokableMethod.doInvoke(InvokableMethod.java:171)
        at org.nuxeo.ecm.automation.core.impl.InvokableMethod.invoke(InvokableMethod.java:177)
        ... 118 more
Caused by: java.io.FileNotFoundException: /mnt/data/DAM-Data01/nuxeo/nxserver/data/transientstores/default/YmF0Y2hJZC1lNzc2OTMzZi04MzYwLTRjZTYtYjYxMi0zYmFhYjUyYzFmZWZfMg==/005c707b-b2bb-4e46-adeb-1194b3250998 (No such file or directory)
        at java.io.FileInputStream.open0(Native Method)
        at java.io.FileInputStream.open(FileInputStream.java:195)
        at java.io.FileInputStream.<init>(FileInputStream.java:138)
        at org.nuxeo.ecm.core.api.impl.blob.FileBlob.getStream(FileBlob.java:169)
        at org.nuxeo.ecm.core.blob.binary.AbstractBinaryManager.getBinary(AbstractBinaryManager.java:96)
        at org.nuxeo.ecm.core.blob.binary.DefaultBinaryManager.getBinary(DefaultBinaryManager.java:50)
        at org.nuxeo.ecm.core.blob.binary.BinaryBlobProvider.writeBlob(BinaryBlobProvider.java:97)
        at org.nuxeo.ecm.core.blob.BlobManagerComponent.writeBlob(BlobManagerComponent.java:281)
        ... 132 more

So it seems that the files get deleted after the upload. Single file uploads work though. Is there some setting to control the cleanup of the tmp folder? Or any other hint on tracing the cause of this issue are highly appreciated.

edit: Clustering is not enabled

##-----------------------------------------------------------------------------
## Clustering settings
##-----------------------------------------------------------------------------
#repository.clustering.enabled=false
#repository.clustering.delay=1000
#repository.binary.store=/path/to/some/shared/folder/for/binaries

Thanks. Stefan

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ataillefer_ — Tue, 01 Mar 2016 16:36:19 GMT

Hi,

Is your Nuxeo configured in cluster?

If this is the case you need to follow the instructions given on this page: https://doc.nuxeo.com/display/ADMINDOC/Nuxeo+Clustering+Configuration#NuxeoClusteringConfiguration-TransientStore

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ssze_ — Tue, 01 Mar 2016 16:42:26 GMT

No, we don't use the clustering. I left the default clustering settings commented out in the nuxeo.conf file.

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ataillefer_ — Tue, 01 Mar 2016 17:00:28 GMT

After the drag and drop of the 4 video files, for how long approximately are you waiting before you get the error?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ssze_ — Wed, 02 Mar 2016 12:35:40 GMT

After clicking the import button nuxeo is working for up to 10 minutes until the import screen goes away. Then the message is displayed.

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ssze_ — Thu, 03 Mar 2016 14:40:53 GMT

I tried uploading via curl and API as described in https

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Florent_Guillau — Thu, 03 Mar 2016 16:13:31 GMT

This fails after how long? Are there errors server-side?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ssze_ — Thu, 03 Mar 2016 21:04:31 GMT

Hi, after reviewing the upload issues we see 4 different scenarios with cap 8.1

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ssze_ — Thu, 03 Mar 2016 21:20:46 GMT

Please see also the detailed curl output for the difference in the successful small and big video uploads. The strange thing is the 404 not found on the big file upload.

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ssze_ — Mon, 07 Mar 2016 16:35:46 GMT

Uploads in the UI still fail with "Processing of multipart/form-data request failed. Stream ended unexpectedly" errors.

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ataillefer_ — Tue, 08 Mar 2016 13:50:18 GMT

Hi, thanks for all these details.

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ssze_ — Tue, 08 Mar 2016 14:44:52 GMT

Thanks for the curl check.

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

ataillefer_ — Tue, 08 Mar 2016 15:21:59 GMT

Increasing nuxeo.db.transactiontimeout should definitely work for the New Video case (stack clearly shows "Unable to commit

topic Batch uploads fail in nuxeo 8.1 - how can I fix it? in Nuxeo Forum

Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?

Re: Batch uploads fail in nuxeo 8.1 - how can I fix it?