topic Re: importing text file more than 32MB gives error "Caused by: org.bson.BsonSerializationException: Size 30831765 is larger than MaxDocumentSize 16793600." in Nuxeo Forum

importing text file more than 32MB gives error "Caused by: org.bson.BsonSerializationException: Size 30831765 is larger than MaxDocumentSize 16793600."

Santosh_Yadav — Mon, 04 Dec 2017 12:10:21 GMT

Hi, I am importing text file of size arroung 32MB. While importing file I am getting error as follows: Caused by: org.bson.BsonSerializationException: Size 30831765 is larger than MaxDocumentSize 16793600.

org.nuxeo.ecm.automation.OperationException: Failed to invoke operation FileManager.Import at org.nuxeo.ecm.automation.core.impl.InvokableMethod.invoke(InvokableMethod.java:188) at org.nuxeo.ecm.automation.core.impl.OperationChainCompiler$OperationMethod.invoke(OperationChainCompiler.java:128) at org.nuxeo.ecm.automation.core.impl.OperationChainCompiler$CompiledChainImpl.invoke(OperationChainCompiler.java:196) at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:119) at org.nuxeo.ecm.automation.core.impl.OperationServiceImpl.run(OperationServiceImpl.java:107) at org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent.execute(BatchManagerComponent.java:301) at org.nuxeo.ecm.automation.server.jaxrs.batch.BatchManagerComponent.execute(BatchManagerComponent.java:255) at org.nuxeo.ecm.webapp.action.ImportActions.importDocumentsThroughBatchManager(ImportActions.java:295) at org.nuxeo.ecm.webapp.action.ImportActions.importDocuments(ImportActions.java:247) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.seam.util.Reflections.invoke(Reflections.java:22) at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32) at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56) at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28) at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79) at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) at org.nuxeo.ecm.platform.ui.web.util.NuxeoExceptionInterceptor.aroundInvoke(NuxeoExceptionInterceptor.java:80) at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44) at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107) at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196) at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114) at org.nuxeo.ecm.webapp.action.ImportActions_$$javassist_seam_23.importDocuments(ImportActions$$_javassist_seam_23.java) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:335) at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:280) at org.jboss.el.parser.AstMethodSuffix.getValue(AstMethodSuffix.java:59) at org.jboss.el.parser.AstMethodSuffix.invoke(AstMethodSuffix.java:65) at org.jboss.el.parser.AstValue.invoke(AstValue.java:96) at org.jboss.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276) at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:105) at org.nuxeo.ecm.platform.ui.web.binding.MetaMethodExpression.invoke(MetaMethodExpression.java:130) at javax.faces.component.MethodBindingMethodExpressionAdapter.invoke(MethodBindingMethodExpressionAdapter.java:87) at com.sun.faces.application.ActionListenerImpl.processAction(ActionListenerImpl.java:102) at javax.faces.component.UICommand.broadcast(UICommand.java:315) at org.nuxeo.ecm.platform.ui.web.component.holder.UIValueHolder.broadcast(UIValueHolder.java:95) at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:790) at javax.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:790) at javax.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1282) at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:81) at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:198) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:646) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83) at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.nuxeo.elasticsearch.web.sync.UIThreadMarker.doFilter(UIThreadMarker.java:48) at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoThreadTrackerFilter.doFilter(NuxeoThreadTrackerFilter.java:43) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.nuxeo.ecm.core.management.jtajca.internal.Log4jWebFilter.doFilter(Log4jWebFilter.java:69) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.nuxeo.ecm.platform.ui.web.rest.FancyURLFilter.doFilter(FancyURLFilter.java:120) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoRequestControllerFilter.doFilter(NuxeoRequestControllerFilter.java:146) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.nuxeo.ecm.webdav.service.WIRequestFilter.doFilter(WIRequestFilter.java:61) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilterInternal(NuxeoAuthenticationFilter.java:630) at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:51) at org.nuxeo.ecm.platform.ui.web.auth.oauth.NuxeoOAuthFilter.doFilter(NuxeoOAuthFilter.java:122) at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:49) at org.nuxeo.ecm.platform.ui.web.auth.oauth2.NuxeoOAuth2Filter.doFilter(NuxeoOAuth2Filter.java:83) at org.nuxeo.ecm.platform.ui.web.auth.service.NuxeoAuthFilterChain.doFilter(NuxeoAuthFilterChain.java:49) at org.nuxeo.ecm.platform.ui.web.auth.NuxeoAuthenticationFilter.doFilter(NuxeoAuthenticationFilter.java:419) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.nuxeo.ecm.platform.web.common.requestcontroller.filter.NuxeoCorsFilter.doFilter(NuxeoCorsFilter.java:51) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.nuxeo.ecm.platform.web.common.exceptionhandling.NuxeoExceptionFilter.doFilter(NuxeoExceptionFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.nuxeo.ecm.platform.web.common.encoding.NuxeoEncodingFilter.doFilter(NuxeoEncodingFilter.java:75) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: org.bson.BsonSerializationException: Size 34523331 is larger than MaxDocumentSize 16793600. at org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:367) at org.bson.BsonBinaryWriter.doWriteEndDocument(BsonBinaryWriter.java:122) at org.bson.AbstractBsonWriter.writeEndDocument(AbstractBsonWriter.java:293) at com.mongodb.DBObjectCodec.encode(DBObjectCodec.java:133) at com.mongodb.DBObjectCodec.encode(DBObjectCodec.java:62) at com.mongodb.CompoundDBObjectCodec.encode(CompoundDBObjectCodec.java:48) at com.mongodb.CompoundDBObjectCodec.encode(CompoundDBObjectCodec.java:27) at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:63) at org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29) at com.mongodb.connection.InsertCommandMessage.writeTheWrites(InsertCommandMessage.java:101) at com.mongodb.connection.InsertCommandMessage.writeTheWrites(InsertCommandMessage.java:43) at com.mongodb.connection.BaseWriteCommandMessage.encodeMessageBodyWithMetadata(BaseWriteCommandMessage.java:129) at com.mongodb.connection.RequestMessage.encodeWithMetadata(RequestMessage.java:160) at com.mongodb.connection.WriteCommandProtocol.sendMessage(WriteCommandProtocol.java:220) at com.mongodb.connection.WriteCommandProtocol.execute(WriteCommandProtocol.java:101) at com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:67) at com.mongodb.connection.InsertCommandProtocol.execute(InsertCommandProtocol.java:37) at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168) at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289) at com.mongodb.connection.DefaultServerConnection.insertCommand(DefaultServerConnection.java:118) at com.mongodb.operation.InsertOperation.executeCommandProtocol(InsertOperation.java:76) at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:142) at com.mongodb.operation.BaseWriteOperation$1.call(BaseWriteOperation.java:134) at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:232) at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:223) at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:134) at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:61) at com.mongodb.Mongo.execute(Mongo.java:827) at com.mongodb.Mongo$2.execute(Mongo.java:810) at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:333) at com.mongodb.DBCollection.insert(DBCollection.java:328) at com.mongodb.DBCollection.insert(DBCollection.java:319) at com.mongodb.DBCollection.insert(DBCollection.java:289) at com.mongodb.DBCollection.insert(DBCollection.java:255) at com.mongodb.DBCollection.insert(DBCollection.java:239) at org.nuxeo.ecm.core.storage.mongodb.MongoDBRepository.createStates(MongoDBRepository.java:355) at org.nuxeo.ecm.core.storage.dbs.DBSCachingRepository.createStates(DBSCachingRepository.java:212) at org.nuxeo.ecm.core.storage.dbs.DBSTransactionState.save(DBSTransactionState.java:585) at org.nuxeo.ecm.core.storage.dbs.DBSSession.checkIn(DBSSession.java:532) at org.nuxeo.ecm.core.storage.dbs.DBSDocument.checkIn(DBSDocument.java:485) at org.nuxeo.ecm.core.versioning.StandardVersioningService.doCheckIn(StandardVersioningService.java:339) at org.nuxeo.ecm.core.versioning.VersioningComponent.doCheckIn(VersioningComponent.java:428) at org.nuxeo.ecm.core.api.AbstractSession.checkIn(AbstractSession.java:1740) at org.nuxeo.ecm.core.api.impl.DocumentModelImpl.checkIn(DocumentModelImpl.java:683) at org.nuxeo.ecm.core.versioning.StandardVersioningService.doAutomaticVersioning(StandardVersioningService.java:482) at org.nuxeo.ecm.core.versioning.VersioningComponent.doAutomaticVersioning(VersioningComponent.java:438) at org.nuxeo.ecm.core.api.AbstractSession.createDocument(AbstractSession.java:721) at org.nuxeo.ecm.platform.filemanager.service.extension.AbstractFileImporter.create(AbstractFileImporter.java:227) at org.nuxeo.ecm.platform.filemanager.service.FileManagerService.createDocumentFromBlob(FileManagerService.java:260) at org.nuxeo.ecm.automation.core.operations.services.FileManagerImport.run(FileManagerImport.java:76) at org.nuxeo.ecm.automation.core.operations.services.FileManagerImport.run(FileManagerImport.java:84) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.nuxeo.ecm.automation.core.impl.InvokableMethod.doInvoke(InvokableMethod.java:171) at org.nuxeo.ecm.automation.core.impl.InvokableMethod.invoke(InvokableMethod.java:176)

Re: importing text file more than 32MB gives error "Caused by: org.bson.BsonSerializationException: Size 30831765 is larger than MaxDocumentSize 16793600."

Kevin_Leturc — Mon, 04 Dec 2017 13:58:15 GMT

Hi,

It seems Nuxeo is trying to store your blob in the document itself, this is a Note specificity. Operation takes the NoteImporter to create the document from blob, which is configured as it:

<plugin name="NoteImporter"  
class="org.nuxeo.ecm.platform.filemanager.service.extension.NoteImporter" order="10">
  <filter>text/plain</filter>
  <filter>text/html</filter>
  <filter>application/xhtml+xml</filter>
  <filter>text/x-web-markdown</filter>
</plugin>

You have several options, set a mime type which is not in the list above and maybe more accurate. Or you can redefine the NoteImporter to exclude the mime type you use from its filter. As it, in both solutions, DefaultFileImporter will be used and a real file will be created on Nuxeo side instead of storing the file content in the document itself.

Re: importing text file more than 32MB gives error "Caused by: org.bson.BsonSerializationException: Size 30831765 is larger than MaxDocumentSize 16793600."

Santosh_Yadav — Tue, 05 Dec 2017 06:26:28 GMT

Hi,

Re: importing text file more than 32MB gives error "Caused by: org.bson.BsonSerializationException: Size 30831765 is larger than MaxDocumentSize 16793600."

Santosh_Yadav — Tue, 05 Dec 2017 07:09:23 GMT

2017-12-05 12

Re: importing text file more than 32MB gives error "Caused by: org.bson.BsonSerializationException: Size 30831765 is larger than MaxDocumentSize 16793600."

Kevin_Leturc — Tue, 05 Dec 2017 10:27:51 GMT

Hi,

From you first stack trace, I see you're using FileManager.Import operation to import your blobs. This operation relies on the file manager. This is a pluggable service where you define plugins according to the blob mime type, see the extension point documentation.

In your case you seem to upload blobs with the mime type text/plain (or it is deduced from the file extension). You can change the mime type when you're ulploading your blobs if one is more accurate than text/plain. How do you upload your blobs ? Do you use one of our clients ?

If you want to keep text/plain or you're not able to change the mime type in your upload code, you can contribute to Nuxeo server in order to disable the NoteImporter plugin for the corresponding mime type. You can find some documentation here.

Here's an example of xml contribution to put under NUXEO_HOME/nxserver/config/, let's name the file disable-note-importer-config.xml. This contribution will disable NoteImporter, so DefaultFileImporter will be used instead for the four mime types defined originally:

<?xml version="1.0"?>
<component name="org.nuxeo.platorm.filemanager.plugins.disable.note">
  <!-- we need this tag in order to contribute to Nuxeo after the default NoteImporter contribution -->
  <require>org.nuxeo.ecm.platform.filemanager.service.FileManagerService.Plugins</require>

  <!-- contribute to the related extension point -->
  <extension target="org.nuxeo.ecm.platform.filemanager.service.FileManagerService" point="plugins">
    <!-- point the NoteImporter in order to disable it -->
    <plugin name="NoteImporter" enabled="false"> 
      <!-- no need to declare filter again as we disable this plugin -->
    </plugin>
  </extension>
</component>

Re: importing text file more than 32MB gives error "Caused by: org.bson.BsonSerializationException: Size 30831765 is larger than MaxDocumentSize 16793600."

Santosh_Yadav — Tue, 05 Dec 2017 11:08:08 GMT

Hi,

Re: importing text file more than 32MB gives error "Caused by: org.bson.BsonSerializationException: Size 30831765 is larger than MaxDocumentSize 16793600."

Kevin_Leturc — Tue, 05 Dec 2017 17:00:41 GMT

Do you see the component above deployed during startup? Just checkin the component loading status / total metric should be enough.